Change search
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf
A System, Tools and Algorithms for Adaptive HTTP-live Streaming on Peer-to-peer Overlays
KTH, School of Information and Communication Technology (ICT), Software and Computer systems, SCS.
2013 (English)Doctoral thesis, comprehensive summary (Other academic)
Abstract [en]

In recent years, adaptive HTTP streaming protocols have become the de facto standard in the industry for the distribution of live and video-on-demand content over the Internet. In this thesis, we solve the problem of distributing adaptive HTTP live video streams to a large number of viewers using peer-to-peer (P2P) overlays. We do so by assuming that our solution must deliver a level of quality of user experience which is the same as a CDN while trying to minimize the load on the content provider’s infrastructure. Besides that, in the design of our solution, we take into consideration the realities of the HTTP streaming protocols, such as the pull-based approach and adaptive bitrate switching.

The result of this work is a system which we call SmoothCache that provides CDN-quality adaptive HTTP live streaming utilizing P2P algorithms. Our experiments on a real network of thousands of consumer machines show that, besides meeting the the CDN-quality constraints, SmoothCache is able to consistently deliver up to 96% savings towards the source of the stream in a single bitrate scenario and 94% in a multi-bitrate scenario. In addition, we have conducted a number of pilot deployments in the setting of large enterprises with the same system, albeit tailored to private networks. Results with thousands of real viewers show that our platform provides an average offloading of bottlenecks in the private network of 91.5%.

These achievements were made possible by advancements in multiple research areas that are also presented in this thesis. Each one of the contributions is novel with respect to the state of the art and can be applied outside of the context of our application. However, in our system they serve the purposes described below.

We built a component-based event-driven framework to facilitate the development of our live streaming application. The framework allows for running the same code both in simulation and in real deployment. In order to obtain scalability of simulations and accuracy, we designed a novel flow-based bandwidth emulation model.

In order to deploy our application on real networks, we have developed a network library which has the novel feature of providing on-the-fly prioritization of transfers. The library is layered over the UDP protocol and supports NAT Traversal techniques. As part of this thesis, we have also improved on the state of the art of NAT Traversal techniques resulting in higher probability of direct connectivity between peers on the Internet.

Because of the presence of NATs on the Internet, discovery of new peers and collection of statistics on the overlay through peer sampling is problematic. Therefore, we created a peer sampling service which is NAT-aware and provides one order of magnitude fresher samples than existing peer sampling protocols.

Finally, we designed SmoothCache as a peer-assisted live streaming system based on a distributed caching abstraction. In SmoothCache, peers retrieve video fragments from the P2P overlay as quickly as possible or fall back to the source of the stream to keep the timeliness of the delivery. In order to produce savings, the caching system strives to fill up the local cache of the peers ahead of playback by prefetching content. Fragments are efficiently distributed by a self-organizing overlay network that takes into account many factors such as upload bandwidth capacity, connectivity constraints, performance history and the currently being watched bitrate.

Place, publisher, year, edition, pages
Stockholm: KTH Royal Institute of Technology, 2013. , xix, 41 p.
Series
TRITA-ICT-ECS AVH, ISSN 1653-6363 ; 13:18
Keyword [en]
peer-to-peer, distributed caching, nat traversal, congestion control, adaptive HTTP streaming, live streaming
National Category
Computer Science
Research subject
SRA - ICT
Identifiers
URN: urn:nbn:se:kth:diva-134351ISBN: 978-91-7501-915-4 (print)OAI: oai:DiVA.org:kth-134351DiVA: diva2:666136
Public defence
2013-12-12, Sal D, Forum, KTH-ICT, Isafjordsgatan 39, Stockholm, 13:00 (English)
Opponent
Supervisors
Note

QC 20131122

Available from: 2013-11-22 Created: 2013-11-21 Last updated: 2016-01-21Bibliographically approved
List of papers
1. Mesmerizer: a effective tool for a complete peer-to-peer software development life-cycle
Open this publication in new window or tab >>Mesmerizer: a effective tool for a complete peer-to-peer software development life-cycle
2011 (English)In: 4th International ICST Conference on Simulation Tools and Techniques, 2011, 506-515 p.Conference paper, Published paper (Refereed)
Abstract [en]

In this paper we present what are, in our experience, the best practices in Peer-To-Peer (P2P) application development and how we combined them in a middleware platform called Mesmerizer. We explain how simulation is an integral part of the development process and not just an assessment tool. We then present our component-based event-driven framework for P2P application development, which can be used to execute multiple instances of the same application in a strictly controlled manner over an emulated network layer for simulation/testing, or a single application in a concurrent environment for deployment purpose. We highlight modeling aspects that are of critical importance for designing and testing P2P applications, e.g. the emulation of Network Address Translation and bandwidth dynamics. We show how our simulator scales when emulating low-level bandwidth characteristics of thousands of concurrent peers while preserving a good degree of accuracy compared to a packet-level simulator.

National Category
Computer Science
Identifiers
urn:nbn:se:kth:diva-92091 (URN)978-1-936968-00-8 (ISBN)
Conference
SIMUTools '11, 4th International ICST Conference on Simulation Tools and Techniques. Barcelona, Spain. March 21-25 2011
Note
QC 20120328Available from: 2012-03-26 Created: 2012-03-26 Last updated: 2013-11-22Bibliographically approved
2. Accurate and efficient simulation of bandwidth dynamics for Peer-To-Peer overlay networks
Open this publication in new window or tab >>Accurate and efficient simulation of bandwidth dynamics for Peer-To-Peer overlay networks
2011 (English)In: VALUETOOLS '11 Proceedings of the 5th International ICST Conference on Performance Evaluation Methodologies and Tools, ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering) , 2011, 352-361 p.Conference paper, Published paper (Refereed)
Abstract [en]

When evaluating Peer-to-Peer content distribution systems by means of simulation, it is of vital importance to correctly mimic the bandwidth dynamics behaviour of the underlying network. In this paper, we propose a scalable and accurate flow-level network simulation model based on an evolution of the classical progressive filling algorithm which follows the max-min fairness idea. We build on top of the current state of the art by applying an optimization to reduce the cost of each bandwidth allocation/deallocation operation on a node-based directed network model. Unlike other works, our evaluation of the chosen approach focuses both on efficiency and on accuracy. Our experiments show that, in terms of scalability, our bandwidth allocation algorithm outperforms existing directed models when simulating large-scale structured overlay networks. In terms of accuracy we show that allocation dynamics of our proposed solution follow those of the NS-2 packet-level simulator by a small and nearly constant offset for the same scenarios. To the best of our knowledge, this is the first time that an accuracy study has been conducted on an improvement of the classical progressive filling algorithm.

Place, publisher, year, edition, pages
ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering), 2011
Keyword
bandwidth dynamics simulation, flow-level simulation, peer-to-peer systems, progressive filling
National Category
Computer Systems
Identifiers
urn:nbn:se:kth:diva-134230 (URN)10.4108/icst.valuetools.2011.245721 (DOI)2-s2.0-84897392664 (Scopus ID)978-1-936968-09-1 (ISBN)
Conference
5th International ICST Conference on Performance Evaluation Methodologies and Tools, VALUETOOLS 2011, Cachan, France, 16 May 2011 through 20 May 2011
Note

QC 20131121

Available from: 2013-11-20 Created: 2013-11-20 Last updated: 2015-04-22Bibliographically approved
3. Natcracker: Nat combinations matter
Open this publication in new window or tab >>Natcracker: Nat combinations matter
2009 (English)In: Proceedings - International Conference on Computer Communications and Networks, ICCCN, San Francisco, CA, 2009Conference paper, Published paper (Other academic)
Abstract [en]

In this paper, we report our experience in working with Network Address Translators (NATs). Traditionally, there were only 4 types of NATs. For each type, the (im)possibility of traversal is well-known. Recently, the NAT community has provided a deeper dissection of NAT behaviors resulting into at least 27 types and documented the (im)possibility of traversal for some types. There are, however, two fundamental issues that were not previously tackled by the community. First, given the more elaborate set of behaviors, it is incorrect to reason about traversing a single NAT, instead combinations must be considered and we have not found any study that comprehensively states, for every possible combination, whether direct connectivity with no relay is feasible. Such a statement is the first outcome of the paper. Second, there is a serious need for some kind of formalism to reason about NATs which is a second outcome of this paper. The results were obtained using our own scheme which is an augmentation of currently-known traversal methods. The scheme is validated by reasoning using our formalism, simulation and implementation in a real P2P network. ©2009 IEEE.

Place, publisher, year, edition, pages
San Francisco, CA: , 2009
Keyword
Network address translators, P2P network, Computer simulation
National Category
Other Electrical Engineering, Electronic Engineering, Information Engineering Computer Engineering
Identifiers
urn:nbn:se:kth:diva-32889 (URN)10.1109/ICCCN.2009.5235278 (DOI)2-s2.0-70449103450 (Scopus ID)
Note
QC 20110426Available from: 2011-04-26 Created: 2011-04-26 Last updated: 2013-11-22Bibliographically approved
4. DTL: Dynamic transport library for peer-to-peer applications
Open this publication in new window or tab >>DTL: Dynamic transport library for peer-to-peer applications
2012 (English)In: Distributed Computing And Networking / [ed] Riccardo Reale, Roberto Roverso, Sameh El-Ansary and Seif Haridi, Springer Berlin/Heidelberg, 2012, 428-442 p.Conference paper, Published paper (Refereed)
Abstract [en]

This paper presents the design and implementation of the Dynamic Transport Library (DTL), a UDP-based reliable transport library, initially designed for - but not limited to - peer-to-peer applications. DTL combines many features not simultaneously offered by any other transport library including: i) Wide scope of congestion control levels starting from less-than-best-effort to high-priority, ii) Prioritization of traffic relative to other non-DTL traffic, iii) Prioritization of traffic between DTL connections, iv) NAT-friendliness, v) Portability, and vi) Application level implementation. Moreover, DTL has a novel feature, namely, the ability to change the level of aggressiveness of a certain connection at run-time. All the features of the DTL were validated using a controlled environment as well as the Planet Lab testbed.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2012
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 7129
Keyword
Application level, Controlled environment, Peer-to-peer application, Prioritization, Reliable transport, Runtimes
National Category
Computer and Information Science
Identifiers
urn:nbn:se:kth:diva-100009 (URN)10.1007/978-3-642-25959-3_32 (DOI)2-s2.0-84855724978 (Scopus ID)978-364225958-6 (ISBN)
Conference
13th International Conference on Distributed Computing and Networking, ICDCN 2012;Hong Kong; 3 January 2012 through 6 January 2012
Funder
Swedish e‐Science Research CenterICT - The Next Generation
Note

QC 20120802

Available from: 2012-08-02 Created: 2012-08-02 Last updated: 2014-01-27Bibliographically approved
5. Through the Wormhole: Low Cost, Fresh Peer Sampling for the Internet
Open this publication in new window or tab >>Through the Wormhole: Low Cost, Fresh Peer Sampling for the Internet
2013 (English)In: 13th IEEE International Conference on Peer-to-Peer Computing, IEEE P2P 2013 - Proceedings, IEEE , 2013, 6688707- p.Conference paper, Published paper (Refereed)
Abstract [en]

State of the art gossip protocols for the Internet are based on the assumption that connection establishment between peers comes at negligible cost. Our experience with commercially deployed P2P systems has shown that this cost is much higher than generally assumed. As such, peer sampling services often cannot provide fresh samples because the service would require too high a connection establishment rate. In this paper, we present the wormhole-based peer sampling service (WPSS). WPSS overcomes the limitations of existing protocols by executing short random walks over a stable topology and by using shortcuts (wormholes), thus limiting the rate of connection establishments and guaranteeing freshness of samples, respectively.We show that our approach can decrease the connection establishment rate by one order of magnitude compared to the state of the art while providing the same levels of freshness of samples. This, without sacrificing the desirable properties of a PSS for the Internet, such as robustness to churn and NAT-friendliness. We support our claims with a thorough measurement study in our deployed commercial system as well as in simulation.

Place, publisher, year, edition, pages
IEEE, 2013
Keyword
NAT-resilient gossip protocols, P2P Networks, Peer Sampling
National Category
Computer Systems
Identifiers
urn:nbn:se:kth:diva-134234 (URN)10.1109/P2P.2013.6688707 (DOI)000330658600013 ()2-s2.0-84893270274 (Scopus ID)978-147990521-8 (ISBN)
Conference
13th IEEE International Conference on Peer-to-Peer Computing, IEEE P2P 2013; Trento; Italy; 9 September 2013 through 11 September 2013
Note

QC 20131121

Available from: 2013-11-20 Created: 2013-11-20 Last updated: 2014-03-19Bibliographically approved
6. SmoothCache: HTTP-live streaming goes peer-to-peer
Open this publication in new window or tab >>SmoothCache: HTTP-live streaming goes peer-to-peer
2012 (English)In: Networking 2012, Springer Berlin/Heidelberg, 2012, Vol. 7290 LNCS, no PART 2, 29-43 p.Conference paper, Published paper (Refereed)
Abstract [en]

In this paper, we present SmoothCache, a peer-to-peer live video streaming (P2PLS) system. The novelty of SmoothCache is threefold: i) It is the first P2PLS system that is built to support the relatively-new approach of using HTTP as the transport protocol for live content, ii) The system supports both single and multi-bitrate streaming modes of operation, and iii) In Smoothcache, we make use of recent advances in application-layer dynamic congestion control to manage priorities of transfers according to their urgency. We start by explaining why the HTTP live streaming semantics render many of the existing assumptions used in P2PLS protocols obsolete. Afterwards, we present our design starting with a baseline P2P caching model. We, then, show a number of optimizations related to aspects such as neighborhood management, uploader selection and proactive caching. Finally, we present our evaluation conducted on a real yet instrumented test network. Our results show that we can achieve substantial traffic savings on the source of the stream without major degradation in user experience.

Place, publisher, year, edition, pages
Springer Berlin/Heidelberg, 2012
Series
Lecture Notes in Computer Science, ISSN 0302-9743 ; 7290
Keyword
caching, CDN, HTTP-Live streaming, peer-to-peer
National Category
Computer and Information Science
Identifiers
urn:nbn:se:kth:diva-99907 (URN)10.1007/978-3-642-30054-7_3 (DOI)2-s2.0-84861653146 (Scopus ID)978-364230053-0 (ISBN)
Conference
11th International IFIP TC 6 Networking Conference, NETWORKING 2012; Prague; 21 May 2012 through 25 May 2012
Funder
Swedish e‐Science Research CenterICT - The Next Generation
Note

QC 20120808

Available from: 2012-08-08 Created: 2012-08-02 Last updated: 2013-11-22Bibliographically approved
7. SmoothCache 2.0: the HTTP-Live peer-to-peerCDN
Open this publication in new window or tab >>SmoothCache 2.0: the HTTP-Live peer-to-peerCDN
(English)Manuscript (preprint) (Other academic)
Abstract [en]

In recent years, adaptive HTTP streaming protocols have become the de-factostandard in the industry for the distribution of live and video-on-demand contentover the Internet. This paper presents SmoothCache 2.0, a distributed cache foradaptive HTTP live streaming content based on peer-to-peer (P2P) overlays. Thecontribution of this work is twofold. From a systems perspective, to the best ofour knowledge, it is the only commercially deployed P2P platform which supportsrecent live streaming protocols based on HTTP as a transport and the concept ofadaptive bitrate switching. From an algorithmic perspective, the system describesa novel set of overlay construction and prefetching techniques that realize: i ) substantialsavings in terms of the bandwidth load on the source of the stream, and i i )CDN-quality user experience in terms of delay from the live playing point and themostly-watched bitrate. In order to support our claims, we conduct a methodicalevaluation encompassing a subset of thousands of real customers’ machines whichwho agreed to install an instrumented version of our software.

National Category
Computer and Information Science
Identifiers
urn:nbn:se:kth:diva-134333 (URN)
Note

QS 2013

Available from: 2013-11-21 Created: 2013-11-21 Last updated: 2013-11-22Bibliographically approved
8. On HTTP live streaming in large enterprises
Open this publication in new window or tab >>On HTTP live streaming in large enterprises
2013 (English)In: Computer communication review, ISSN 0146-4833, E-ISSN 1943-5819, Vol. 43, no 4, 489-490 p.Article in journal (Refereed) Published
Abstract [en]

In this work, we present a distributed caching solution which addresses the problem of efficient delivery of HTTP live streams in large private networks. With our system, we have conducted tests on a number of pilot deployments. The largest of them, with 3000 concurrent viewers, consistently showed that our system saves more than 90% of traffic towards the source of the stream while providing the same quality of user experience of a CDN. Another result is that our solution was able to reduce the load on the bottlenecks in the network by an average of 91.6%.

Keyword
content delivery network, distributed caching, http live, private networks
National Category
Computer and Information Science
Identifiers
urn:nbn:se:kth:diva-133408 (URN)10.1145/2486001.2491685 (DOI)000327465900061 ()2-s2.0-84883268151 (Scopus ID)978-145032056-6 (ISBN)
Conference
ACM SIGCOMM 2013 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communication, SIGCOMM 2013; Hong Kong; China; 12 August 2013 through 16 August 2013
Note

QC 20131104

Available from: 2013-11-04 Created: 2013-10-31 Last updated: 2017-12-06Bibliographically approved

Open Access in DiVA

fulltext(8019 kB)245 downloads
File information
File name FULLTEXT03.pdfFile size 8019 kBChecksum SHA-512
b4e9f7e175992bcb25871e4417fe7b1fcbf2267f179f4e8f49dbc970e5bebdb490ecc458284773779717ffaf070a3cbb64f7c5af9aa8725419b3a158770d95ed
Type fulltextMimetype application/pdf

Search in DiVA

By author/editor
Roverso, Roberto
By organisation
Software and Computer systems, SCS
Computer Science

Search outside of DiVA

GoogleGoogle Scholar
Total: 1177 downloads
The number of downloads is the sum of all downloads of full texts. It may include eg previous versions that are now no longer available

isbn
urn-nbn

Altmetric score

isbn
urn-nbn
Total: 654 hits
CiteExportLink to record
Permanent link

Direct link
Cite
Citation style
  • apa
  • harvard1
  • ieee
  • modern-language-association-8th-edition
  • vancouver
  • Other style
More styles
Language
  • de-DE
  • en-GB
  • en-US
  • fi-FI
  • nn-NO
  • nn-NB
  • sv-SE
  • Other locale
More languages
Output format
  • html
  • text
  • asciidoc
  • rtf