IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4...

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4, FOURTH QUARTER 2016 2887

Multipath Transmission for the Internet: A SurveyMing Li, Andrey Lukyanenko, Zhonghong Ou, Antti Ylä-Jääski,

Sasu Tarkoma, Matthieu Coudron, and Stefano Secci

Abstract—Smart devices equipped with multiple networkinterfaces are becoming commonplace. Nevertheless, even thoughmultiple interfaces can be used to connect to the Internet, theircapabilities have not been fully utilized yet because the defaultTCP/IP stack supports only a single interface for communication.This situation is now changing due to the emergence of multi-path protocols on different network stack layers. For example,many IP level approaches have been proposed utilizing tunnelingmechanisms for hiding multipath transmission from the trans-port protocols. Several working groups under IEEE and IETFare actively standardizing multipath transmission on the linklayer and transport layer. Application level approaches enablemultipath transmission capability by establishing multiple trans-port connections and distributing data over them. Given all theseefforts, it is beneficial and timely to summarize the state-of-the-art, compare their pros and cons, and discuss about the futuredirections. To that end, we present a survey on multipath trans-mission and make several major contributions: 1) we present acomplete taxonomy pertaining to multipath transmission, includ-ing link, network, transport, application, and cross layers; 2) wesurvey the state-of-the-art for each layer, investigate the prob-lems that each layer aims to address, and make comprehensiveassessment of the solutions; and 3) based on the comparison, weidentify open issues and pinpoint future directions for multipathtransmission research.

Index Terms—Multipath transmission, TCP-friendly, resourcepooling, packet reordering.

I. INTRODUCTION

THE INTERNET was originally designed as a “two-connected net” to guarantee that no single failure would

cause any non-failed portion of the network to lose connec-tivity [113]. In essence, any source-destination pair needs tomaintain more than one path to assure the reliability andresiliency of the network. Although the rich resources havebeen existing in the Internet, they have not been fully utilized

Manuscript received October 6, 2015; revised February 24, 2016; acceptedJune 9, 2016. Date of publication June 29, 2016; date of current versionNovember 18, 2016. This work was supported in part by the Academy ofFinland under Grant 278207, and in part by the HIIT Distributed and MobileCloud Systems Research Programme.

M. Li, A. Lukyanenko, and A. Ylä-Jääski are with the Departmentof Computer Science, Aalto University, Espoo 02150, Finland (e-mail:[email protected]; [email protected]; [email protected]).

Z. Ou is with the School of Computer Science, Beijing Universityof Posts and Telecommunications, Beijing 100876, China (e-mail:[email protected]).

S. Tarkoma is with the Department of Computer Science, University ofHelsinki, Helsinki 00014, Finland (e-mail: [email protected]).

M. Coudron and S. Secci are with Sorbonne Universités, UPMCUniversity of Paris 06, UMR 7606, LIP6, 75005 Paris, France (e-mail:[email protected]; [email protected]).

Digital Object Identifier 10.1109/COMST.2016.2586112

since the birth of the Internet. The reason lies in the fact that,by default, the conventional TCP/IP only uses a single “best”path according to certain routing metrics; the other availablepaths remain standby only for backup and recovery purposes.

Nonetheless, this situation has been changing in the pastfew years, which is indicated by several trends from the stan-dardization organization, academia, and industry. From thestandardization perspective, both IEEE and IETF are activeon concurrent multipath transmission. There have been severalworking groups dedicating on the standardization, for exam-ple [2], [3], [11], [39], [51], [56], [61], [87], [112], and [127].1

From the academia, hundreds of scientific articles revolvingaround multipath transmission have been published, coveringdifferent network stack layers on various aspects ranging frompacket reordering, scheduling to buffer management, fairness,Resource Pooling (RP). From the industry, several companieshave implemented their own link layer aggregation schemes,such as [15], [32], and [94]. Deutsche Telekom offers hybridaccess by bundling DSL-line with LTE in its portfolio [43].Tessares company [1] tried to develop new innovative networkservices on top of Multipath TCP (MPTCP). For example,its first product aims to aggregate the bandwidths of differ-ent infrastructure (LTE/DSL). In 2015, OVH company [138]announced a new product called Overthebox. This productcombines MPTCP and SOCKS proxies to enable users to bonddifferent DSL lines together. Apple has implemented a vari-ant of MPTCP on part of their Siri servers and allows iOS 7users to use it in their iPhones [4]. At IETF’93 in 2015, KTCorporation presented Gigapath, a commercial service whichcan achieve high bandwidth (800 Mbps and more) by combin-ing LTE and WiFi networks on Multipath TCP enabled smartphones [20].

There already exist a few survey articles focusing ondifferent aspects of multipath transmission. For exam-ple, [86], [108], [154], [174], and [193] mainly focused onthe control plane problem (i.e., multipath routing of how tocompute and select paths). Reference [183] covered the con-trol plane problem as well as the data plane problem (i.e.,how to split the flow on the chosen paths) in wired net-works. Reference [153] assumed multiple paths had beenestablished by routing protocols and focused on load dis-tribution in terms of traffic splitting and path selection.References [45] and [163] considered multipath transmissionin wireless and wired networks respectively. Reference [7]

1Reference [39] gives an overview of bandwidth aggregation mecha-nisms discussed in the context of Banana mailing list https://www.ietf.org/mailman/listinfo/banana.

1553-877X c© 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

2888 IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4, FOURTH QUARTER 2016

investigated the common features of various approachesand classified the features into layer-dependent and layer-independent features. In addition, [7] also abstracted commondesign patterns and proposed a unified networking architectureto enable mobile nodes to make context-aware decisions abouthow and when to use each or a combination of networks.

Nevertheless, to the best of our knowledge, this survey isthe first one to provide a holistic view on the data plane issuesof multipath transmission. We have surveyed papers from theyear 1975 to 2015, and investigated various research problemsfrom different layers, covering link layer, networking layer,transmission layer, application layer as well as cross layer. Theprimary research problems include packet reordering, fairnesscontrol, RP, Pareto-optimality and path diversity.

All frequently used acronyms in this survey are reported inthe Appendix.

A. Why Multipath?

As stated previously, the original Internet was designed as“two-connected net” with path diversity in mind. Nevertheless,computers with multiple network interfaces were not an imme-diate design priority at the early stage. Only the routers wereequipped with several physical network interfaces. However,the Internet has since then evolved significantly. For example,most servers have been equipped with more than one networkinterface nowadays. The abundance of network resources fromthe server domain has spurred the adoption of multipath trans-mission in data center networks. In the consumer electronicsdomain, the proliferation of mobile devices equipped with cel-lular (e.g., 3G and LTE) and WiFi interfaces, represented bysmart phones, brings with it a growing number of multi-homedhosts onto the Internet. Thus, there exists a mismatch betweensingle-path transport and the multitude of available networkpaths. Those multi-interface devices require multipath capa-bility to improve end-to-end communication performance andresilience.

Meanwhile, technological advancement has made multi-path transmission possible in theory. We investigate some ofthe major benefits as well as requirements, and give briefdescriptions as follows.

• Reliability: multipath transmission can enhance the reli-ability of data transfer because additional paths can keepthe connection alive in case of a failing or less performingpath. In wireless environment, reliability can be furtherimproved because signal interference is minimized due tothe use of heterogeneous wireless access techniques.

• Bandwidth aggregation: it is expected that the band-width aggregation can potentially multiply the experi-enced throughput by the number of available paths. Ifefficient bandwidth aggregation can be achieved in thismanner, a multi-homed device can obtain a much betterperformance.

• Fairness and RP: TCP fairness requires that a multi-path transmission protocol receives no larger bandwidthof the shared bottleneck link than a competing TCP flow.This is important because TCP is the dominant transportprotocol on the Internet. If new protocols acquire unfair

capacity, they tend to cause problems such as congestioncollapse. RP is a concept that changed the notions offairness in a way that made multipath communicationswidely acceptable in practice. Instead of handling perpath resource independently, the RP principle advocatesmaking improved use of multiple path resources by allow-ing separate paths to act as if they were a single largeresource. This principle is a significant step towards apractical multipath-aware end system.

• Pareto-optimality: it is a state of resource allocation inwhich there is no alternative state that would make somepeople better off without making anyone worse off. In thecase of multipath transmission, it means that upgradingsome regular single-path users to multipath ones can notreduce the throughput of other users with any benefit tothe upgraded users.

• Security: as data can be distributed over independentpaths, it will be more complex for a malicious entity tocapture the entire content.

B. Potential Blocking Points

Multipath transmission also has its own challenges we mustface. Some of the requirements mentioned in the last sec-tion can be seen as disadvantages as well. In this section,we investigate some of the potential blocking points from theperspective of deployment in practice.

• Packet reordering: it is difficult to schedule data packetsover heterogeneous paths without causing reordering andperformance penalties. A robust multipath transmissionsolution should be able to cope with any kind of pathheterogeneity, throughput fluctuations, or jitter. It shouldalso be able to deal with persistent reordering of datapackets. Otherwise, users would have less incentive toupgrade if the solution fails to work properly in certainnetwork environments.

• Fairness: traditionally, fairness has been one of the obsta-cles to the concurrent multipath transmission. It used tobe on a “per interface” basis which is unfair if there is acommon bottleneck later on. For example, simply utiliz-ing multiple flows would result in an unfair share of thebandwidth at the bottleneck; for example, n TCP flows getapproximately n times throughput as a competing singleTCP flow does.

• Compatibility: it is hard to implement a generic multi-path solution without modifying standardized protocolsor changing third-party network equipments. For exam-ple, on link layer dedicated setup (even equipment) isrequired on both sides. On other layers above the linklayer, either hosts or networks (sometimes both) need toupgrade in order to support multipath transmission.

• Pareto-optimality: MPTCP is the first multipathtransmission proposal which requires Pareto-optimality [101], [102] but there is little experiencehow well/often it respects this requirement. There areindeed cases where MPTCP may perform worse thannormal TCP due to path heterogeneity.

• Path diversity: it describes the ability of having multi-ple disjoint paths to reach a destination. Users expect to

LI et al.: MULTIPATH TRANSMISSION FOR THE INTERNET: A SURVEY 2889

TABLE ICLASSIFICATION OF THE RESEARCH WORK BASED ON THE INTERNET PROTOCOL LAYERS

obtain high throughput from the use of multiple paths.But if the paths (or partial of them) travel through ashared bottleneck link, the multiple flows can only getas much throughput as a comparable TCP flow does dueto the fairness guarantee. Currently, reliable bottleneckdetection is really hard in practice. No individual pathselection scheme can fit with all network environments.

• Security: multipath transmission has broken trust mod-els organizations placed in single network providers. Forexample, although traffic splitting makes sniffing harder,firewalls or gateways may miss part of the data deliveredover more than one network providers and thus becomeunable to analyze the flow. This would result in brokensecurity solutions including intrusion detection and dataleak prevention.

Note that this survey is neither limited to these problemsnor cover all of them. We present the scope of this survey inSection II.

C. Contributions

We provide a comprehensive survey of multipath transmis-sion, covering various aspects on different layers. Towards thatdirection, we make several key contributions and summarizethem as follows: (1) a complete taxonomy regarding multipathtransmission is presented, covering various protocol layersincluding link layer, network layer, transport layer, applica-tion layer and cross layer; (2) the state-of-the-art for eachlayer is surveyed, the problems addressed by layer specificapproaches are investigated, and comprehensive comparisonsamong them are made; (3) the standardization efforts from var-ious parties are summarized, including working groups fromIETF and IEEE; (4) by the means of comparison, open issuesare identified for future development of multipath transmis-sion. We believe this work will bring insights to the researchersand practitioners working in this field, and foster a set of newresearch towards different directions of multipath transmission.

D. Organization, Structure, and Research Problems

Grouping and discussing multipath transmission approachesaccording to their stack position are beneficial for researchersand practitioners to understand the benefits and trade-offs fromeach layer, and make an all-around decision. Therefore, wesurvey the state-of-the-art multipath transmission from layer-specific perspectives. Table I shows the classification of theresearch work according to the stack position.

The structure of the survey is organized as follows.Section II reviews a number of related surveys and layouts theposition of ours. In Section III, various approaches are clas-sified based on their network stack position and cross-layerapproaches are discussed separately (see Table I). Figure 1illustrates the structure and coverage of Section III. In eachdiscussion of the layer specific approaches, we investigate theproblems the approaches on that layer aim to address. Someproblems are common to all layers, such as the load sharingand packet reordering problems. Some are addressed only oncertain layers. For example, the fairness problem is addressedonly on IP and transport layers. Compared with other lay-ers, transport layer approaches have more problems to addressincluding RP, buffer impact, Pareto-optimality and path diver-sity. The discussion of approaches follows a chronologicalorder except that we group some research work which hassimilarity or progression. In addition, two tables are used tosummarize the key algorithms and approaches respectively.The approach table is connected to the key algorithm tableby the means of listing the key algorithms used in eachapproach as well as the intended network environments ofthe algorithms. Note that the same algorithms, which are usedon different layers, are not repeatedly described in differentkey algorithm tables. Instead, we only provide explanationin the table when the algorithm is first discussed. Followingthe discussion of the approaches on specific layers, we makea summary to present a comprehensive comparison fromfive perspectives. In Section IV, we point out the lessons that


Fig. 1. Structure of Section III and research problems (P) to address.

can be learned from this survey. In Section V, we pinpointfuture research directions. Finally, we conclude this survey inSection VI.

In this article, there are many abbreviations. To help readerstrack them easily, we provide a list of frequently usedacronyms in the Appendix.

II. SCOPE AND RELATED SURVEYS

First of all, we investigate multipath transmission in wiredand wireless networks but leave its discussion in sensor net-works out of the scope. For surveys on multipath transmissionin sensor networks, we refer the readers to [156]. In addi-tion, we focus solely on the data plane problem of how tosplit data on multiple paths and intentionally leave out allwork that focuses on multipath routing, i.e., the control planeproblem of how to compute and select the routes. We referthe readers to articles and recent surveys that cover suchwork [86], [108], [154], [174], [193]. Furthermore, the secu-rity issue of multipath transmission and P2P applications arealso out of the scope of this survey.

Compared with surveys on multipath routing, surveys onthe data plane problems are less popular. There are only a fewsurveys [7], [45], [67], [153], [154], [163], [183] that touchon the same topic as ours. In a somewhat old but still rele-vant survey, Ramaboli et al. [163] reviewed some bandwidthaggregation approaches in heterogeneous wireless networkswhich consist of a variety of integrated and jointly managedradio access technologies. They found that packet reorder-ing is the most dominant challenge because it can introduceundesirable delays for real-time applications and unnecessaryretransmissions for TCP applications. In this regard, theirsurvey focused mainly on the issues caused by packet reorder-ing and the approaches to address them accordingly. Thoseapproaches were classified into two groups according to their

adaptiveness to dynamic conditions: non-adaptive and adap-tive approaches. The former ones do not have the ability toadjust the resource allocation and traffic schedule in dynamicnetwork conditions. In contrast, the latter ones take varyingtraffic and link conditions into consideration in order to deriveoptimal resource allocation and scheduling decisions. In eachgroup, the approaches are further classified according to theirlayer position in TCP/IP protocol stack.

Prabhavat et al. [153] presented a literature review of vari-ous existing load distribution models for multipath networks,and classified the models in terms of their key functionalities:traffic splitting and path selection. A generalized multipathforwarding mechanism was used to discuss the two key func-tionalities without considering which protocol stack positionthe mechanism is implemented on. The paper did not addressrouting to establish multiple paths. Instead, it assumed thatmultiple paths had been established by routing techniques.

Domzał et al. [45] considered multipath transmission inwired networks. They classified the approaches based on threedifferent layers in which they operate, i.e., link layer, networklayer and transport layer. Specifically, on the link layer, theydiscussed a couple of multipath transmission approaches basedon Ethernet. On the network layer, they investigated variousrouting techniques that can be used for the construction andselection of multiple paths. On the transport layer, they gave abrief introduction of MPTCP. However, [45] lacks many keyapproaches added in this survey. For example, on the networklayer, all the tunneling based solutions are missing. On thetransport layer, they only discussed MPTCP but have left outall the other approaches.

Addepalli et al.’s [7] survey covers some aspects relatedto multipath transmission. The authors first reviewed someprotocols and architectures that enable heterogeneous network-ing support, and then abstracted common design patterns andproposed a unified networking architecture to enable mobilenodes to make context-aware decisions about how and when


TABLE IICOVERAGE OF RELATED SURVEYS (

√MEANS HAVING BEEN COVERED)

to use each or a combination of networks. However, the scopeof survey [7] is only partially overlapping with our work, inparticular the focus is mostly shifted to seamless mobility,multihoming, security and other aspects that are out of thescope of our work.

Habak et al. [67] investigated the common features of vari-ous approaches and classified the features into two categories:layer-dependent and layer-independent features. The layer-dependent features include, for example, the common researchproblems shared by the approaches on the same layer. In thediscussion of these features, they performed a layer by layeranalysis of the available approaches. The layer-independentfeatures include scheduling algorithms, estimation of inter-face/application characteristics, and networking models. Incontrast, in this survey, we argue that some seemingly layer-independent features are not completely layer independent. Forexample, certain scheduling algorithms are closely connectedto a specific layer so that they may perform differently ondifferent layers.

Singh et al. [183] surveyed both multipath routing and pro-visioning. In the discussion of provisioning multiple pathsbetween end hosts, they also followed a layer-based structure,from application layer to physical layer, to review the exist-ing approaches. Although [183] has a similar structure as oursurvey, they have several key differences: 1) Refernce [183]excluded multipath transmission in wireless networks whichare the primary environment for path diversity. Due to its focusonly on the wired networks, it missed many key referencesadded in this survey; 2) Refernce [183] lacked a detailed dis-cussion, for example, which scheduling algorithms are used toavoid reordering in the receiver and how well these algorithmsperform. Instead, this analysis is one of the main focuses inour survey; 3) compared to our survey, [183] has missed someimportant contents including whether the approaches consid-ered fairness, implemented the Pareto-efficiency and resourcepooling features, and what compatibility issues the approacheshad on each layer.

Qadir et al. [154] investigated multipath transmission onthe transport layer, despite their main focus on network-layermultipath routing. They organized their investigation by dis-cussing five questions relating to how the multiple paths areused. The five questions have covered: 1) the number of pathsto be used concurrently, 2) the configuration of how multiplepaths work together (backup or concurrent), 3) the load bal-ancing methods (static or dynamic), 4) the congestion controlmethods (coordinated or uncoordinated) and 5) the controllerentity which performs load balancing and traffic engineering

(host or network). In this survey, we organize the discussion ofmultipath transmission in a different way, for example, inves-tigating the various problems to address on different stackpositions (see Figure 1). We believe that these two differentperspectives on the same target are complementary and wouldgive readers more insights.

As we have discussed previously, most of the existing sur-veys have classified various solutions based on protocol layers(excluding [153]). In this regard, we use Table II to comparethe coverage of the previous surveys as well as this surveyon different protocol layers. In this survey, we summarize thekey algorithms used on all the other layers except the physi-cal layer (we refer the readers to [29] and [199] for multipathtransmission at the physical layer) and associate the algorithmsto the problems they were designed to address. We extract thescheduling algorithms and compare their efficiency in terms ofpacket reordering and load sharing capabilities without con-sidering their layer dependency. The approaches based on across-layer design are summarized and discussed in a separatesection. The approaches on different layers are also evaluatedfrom the viewpoint of compatibility capability. We also discussthe evolution of the research questions on multipath trans-mission and found that only the transport layer approachesfollowing a nice evolution.

III. MULTIPATH TRANSMISSION

Before we dig into the technical details of multipath trans-mission, we present the timeline of its milestones in Figure 2in order to give readers a general picture of its development.The first paper on TCP was published in 1974. In the follow-ing year, Dr. Maxemchuk proposed Dispersity Routing [126]in his Ph.D. dissertation to concurrently transmit data overmultiple paths. From that point onward, various forms of mul-tipath transmission have been proposed. For example, the ideaof building multipath capability into TCP was, to the best ofour knowledge, first suggested by Huitema [87] as an Internetdraft in IETF in 1995. In 2002, the first 3G network to go com-mercially live was launched in South Korea, which promotedthe proliferation of mobile devices equipped with multiplewireless interfaces. In 2006, Key et al. [100] used fluid-flowmodeling to demonstrate that multipath transport can providenot only robustness but also balanced congestion in a sta-ble manner. In the same year, Shakkottai et al. [176] used anon-cooperative pricing game to show that multihoming out-performs unihoming in terms of throughput and profit to theInternet service providers (ISPs). In 2008, Wischik et al. [200]


Fig. 2. Milestones in the evolution of multipath transmission. MT: Multipath Transmission, DCN: Data Center Network, MPTCP: Multipath TCP.

Fig. 3. Link aggregation between Ethernet switches. SW: Switch.

investigated the RP principle, which makes a collection ofresources behave like a single pooled resource. This princi-ple is a significant step towards a practical multipath-awareend system. From 2009, IETF started to define and stan-dardize MPTCP, which employs a coupled congestion controlalgorithm to achieve RP principle.

In the remainder of this section, the state-of-the-art multi-path transmission schemes are classified according to whichlayer of the protocol stack the proposed approach performs at:link layer, network layer, transport layer and application layer.

A. Link Layer Bonding

High end workstations and data centers can easily satu-rate existing Local Area Networks (LANs). On the link layer,multipath transmission is typically called bonding or linkaggregation because multiple physical channels are bundled(or aggregated) into a single logical channel. The primarygoal of link layer bundling is to coordinate multiple inde-pendent links between a fixed pair of systems, providing avirtual link with a larger bandwidth than what a single linkcan sustain. Figure 3 shows a simplified example of linkaggregation between two Ethernet switches (SW1 and SW6).These switches can obtain increased throughput by stripingdata across multiple interfaces.

In the following discussion, we use Table III and Table IVto summarize the key algorithms and approaches respectively.The approaches in Table IV are sorted in chronological order.The algorithms in Table III are sorted according to their order

mentioned in Table IV. Note that the algorithms in Table IIImay be not only adopted by approaches on the link layer, butmay also be used by those on other layers. In this survey,we will not elaborate the algorithms that have been discussedpreviously. This same rule is applicable for all other algorithmtables.

Multi-Link PPP (MP) [185], designed for IntegratedServices for Digital Network (ISDN), aggregates multiplelinks using the PPP protocol [181]. In order to detect frag-ment loss and disorder, MP uses a 4-byte re-sequencing header(RSH) for synchronization and detecting lost fragments at thereceiver. Therefore, a reorder buffer is required at the receiverto accommodate the out-of-order fragments caused by linkaggregation. MP suggests a Weighted Round Robin (WRR)scheduling scheme so that data can be distributed proportion-ally to the transmission rates of the links. To achieve this goal,two methods for fragmentation have been proposed. The firstone divides packets into segments with sizes proportional tothe transmission rates of different paths. The other methoddivides packets into many small equal sized fragments anddistributes the number of the fragments proportionally to thetransmission rates of different paths.

Adiseshu et al. [9] and Hari et al. [73] added a “strIPe”layer, a virtual IP interface below the IP layer and above thedata link layer, to aggregate multiple data links. The stripelayer implements the striping algorithm at the sender and thefair queuing algorithm at the receiver. The authors first showedhow a fair-queuing algorithm (FQA) can be transformed to afair load-sharing algorithm (FLSA), and then proposed thatthe FQA should run in a reversed manner of the load-sharingalgorithm in order to solve the load sharing issue with variablepacket size. It implies that identical equipment (or of the samevendor) is required on both sides of the aggregation. They alsodealt with the FIFO delivery issue for two separate cases. Forexample, if a RSH can be added to each packet, the issuecan be solved by using the additional reordering number; ifno header can be added, they proposed a way of synchro-nization in the event of frame loss to provide quasi-FIFOdelivery.

An implementation of MP in Wide Area Networks (WANs)was discussed by Snoeren [186]. They proposed a Link Quality


TABLE IIIKEY ALGORITHMS FOR LINK LAYER BONDING (SORTED ACCORDING TO THEIR ORDER MENTIONED IN TABLE IV)

TABLE IVSUMMARY OF LINK LAYER BONDING APPROACHES

Balancing (LQB) scheme to bundle multiple channels of thesame Wide-area Wireless Access Network (WWAN) technol-ogy. In order to adjust traffic striping across bundled linksaccording to their transmission capabilities, LQB adapts themaximum transmission unit (MTU) of each link in proportionto its available bandwidth (short-term averages of the observedthroughput). A link layer receive buffer is also required toreorder fragments.

FatVAP (an 802.11 driver design) [95] is another work ina wireless environment for link bonding. FatVAP aggregatesthe bandwidth available at multiple wireless access points(WAPs) that are worth connecting to and balances their loadsby scheduling traffic to different APs according to their avail-able bandwidth. In order to continue delivering the sum of thebandwidths available across all APs, FatVAP uses a constantestimation of both end-to-end and wireless bandwidth to reactto changes within a few seconds. Note that FatVAP uses a Per-Flow Allocation (PFA) strategy to distribute traffic over APs.For example, when a new flow arrives, FatVAP determineswhich AP to assign this flow to and records the mapping ina hash table. Subsequent packets in the flow are simply sentthrough the AP recorded in the hash table.

Within IEEE specifications, the Link Aggregation ControlProtocol (LACP) allows multiple links of Ethernet to be

aggregated together to form a Link Aggregation Group (LAG).As such, the Media Access Control (MAC) client can treat theLAG as a single link. LACP allows a network device to nego-tiate an automatic bundling of links by sending LACP packetsto the peer. LACP was initially released as 802.3ad [88] in2000. Nearly every network vendor quickly adopted this stan-dard over their proprietary standards. In 2008, the protocolwas transferred to the 802.1 group with the publication ofIEEE 802.1AX-2008 [2]. In LACP, a Frame Collector (FC)at the receiver is responsible for maintaining any frame order-ing constraint. In order to avoid frame reordering, the FrameDistributor (FD) at the sender transmits all frames that com-pose a given conversation2 only to a single link, which isa Per-Conversation Allocation (PCA) strategy (very similarto PFA strategy). Therefore, no frame reordering scheme orreordering buffer is required at the FC. In addition to theIEEE link aggregation standards, there are a number of propri-etary aggregation schemes, including EtherChannel [32] fromCisco, Aggregated Ethernet [94] from Juniper, Multi-LinkTrunking [15] from AVAYA. These proprietary aggregation

2A set of frames transmitted from one end station to another, where allof the frames form an ordered sequence, and where the communicatingend stations require the ordering to be maintained among the set of framesexchanged.


TABLE VSCHEMES USED ON IP LEVEL FOR BANDWIDTH AGGREGATION

schemes and IEEE 802.3ad standards are very similar andaccomplish the same goal.

Since the version 1.1 [33], OpenFlow has supported multi-link aggregation on layer-2. The specification of OpenFlowswitch has introduced Link Aggregation (LA) to obtain theability for one port to point to a group of other ports. UsingLACP for exchanging dynamic information between LA-supported devices, the OpenFlow controller has full controlover the switches on how frames are distributed and col-lected on multiple links. Nguyen-Duc et al. [135] investigatedthe operation of LA in OpenFlow switches and found thatLACP on OpenFlow switch provides a slightly lower through-put than the one on a conventional switch. Thus, OpenFlowswitches need to be further optimized to achieve equivalentperformance. Subedi et al. [188] presented an adaptive mul-tipath forwarding architecture in a layer-2 OpenFlow datacenter network. In the architecture, all-to-all forwarding pathsare set up proactively among the edge nodes. Aggregatedbandwidth is achieved by using all the available paths simul-taneously. To avoid the out-of-order delivery issue due tousing all available paths, a PCA style scheduling algorithm isused in [135] and [188]. Specifically, the algorithm excludesthe paths whose path length exceeds the shortest path lengthsignificantly.

In the last few years, there are notable new protocolsdesigned to support multipath forwarding at link-layer in IEEEand IETF standards, e.g., Shortest Path Bridging (SPB) [57](specified in the IEEE 802.1aq standard [3]) and IETFTransparent Interconnection of a Lot of Links (TRILL) [51].SPB and TRILL are potential successors of the Spanning Tree

Protocol (STP). SPB now supports multipath forwarding byusing Equal Cost Multipath (ECMP) and Equal Cost Tree(ECT) routing strategies. TRILL currently only supports mul-tipath forwarding by using ECMP routing strategy. In bothECMP and ECT strategies, if multiple equal cost paths arepresent towards a destination, network traffic is distributedover those multiple paths. In order to avoid reordering andpath MTU discovery problems, similar to FatVAP, both TRILLand SPB use per-flow multipath forwarding. Specifically,frames belonging to the same data flow take the same pathand frames belonging to other flows can take the otherpaths.

From Table III, we find that the main problems thealgorithms trying to address are load sharing and packetreordering. Among all the algorithms, the simplest one isRound Robin (RR) where the sender allocates fragmentsamong all the available links in equal portions and in anordered fashion. In the long run, RR scheduling provides afair share of fragments as long as these fragments are of thesame size. Nevertheless, the basic RR scheduling strategy israrely used in practice because it provides no load sharing witheither variable sized fragments or different link capacities. Inorder to solve these issues, the Weighted RR variations are themore widely used scheduling strategies.

The main advantage of link-layer bonding is that the signal-ing rate of the channel is relatively stable and can be utilized tomitigate reordering. However, link-layer approaches only workon a point-to-point link and even require dedicated Ethernetcards installed on both sides. Thus, they are not applica-ble in general scenarios of end-to-end communications where


TABLE VIKEY ALGORITHMS FOR IP LAYER BANDWIDTH AGGREGATION (SORTED ACCORDING TO THEIR ORDER MENTIONED IN TABLE VII)

the different domains involved are controlled by differentproviders.

B. IP Layer Bandwidth Aggregation

The IP layer, originally proposed to handle global address-ing and routing, is a natural candidate to host the multipathcapability to enhance end-to-end communication. A networkapproach has the advantage of being transparent to transportprotocols and applications, making wide spread deploymentmuch easier. In theory, each packet of a TCP flow can besent over a different path, and the IP protocol ensures that allpackets reach their destination. For example, Sun et al. [189]explored the use of multipath routing to reduce the filetransmission delay in a wireless network. Specifically, theyproposed taking advantage of packet level erasure code (e.g.,digital fountain code) to transmit data file with redundancyover a set of paths. They obtained the intuitive understandingof the trade-off between the code rate and delay reduction.Their research was made for a special network environmentwhere a source and destination pair has a rich set of identicaland disjoint paths, hence no packet reordering issue intro-duced. Nevertheless, in most practical network environments,when packets inside one connection taking more than one path,they can experience different propagation delay and arrive outof order. The TCP receiver sends duplicate acknowledgments(ACKs) to the sender, which causes the TCP sender mistakenlyinterprets packet reordering as packet loss. The results foundin [19], [104], and [203] show that TCP suffers significantperformance degradation due to frequent packet reordering.

Fig. 4. IP-in-IP tunneling between two multi-homed hosts.

Thus, the use of multiple paths with varying characteristicsdeteriorates the problem.

In the following discussion, the state-of-the-art is dividedinto three categories: IP-in-IP encapsulation, Network AddressTranslation (NAT) traversal, and Identity/locator split. Wesummarize their features in Table V and discuss each cate-gory according to the order they show in the table. Table VIand Table VII are used to summarize the key algorithms andapproaches respectively. The “Proxies or Updated Routers” inTable VII indicates the required number of proxies or updatedrouters.

1) IP-in-IP Encapsulation: A widely used IP layerapproach for aggregating bandwidth of multiple IP paths isto use tunneling mechanisms which transparently redirectpackets between two hosts on routing level. For example,Phatak et al. [148], [149] proposed using IP-in-IP encap-sulation [144] to split a data flow across multiple networkinterfaces. As shown in Figure 4, at the source (A), the trans-port layer assembles all packets as if they were going through


TABLE VIISUMMARY OF IP LEVEL BANDWIDTH AGGREGATION APPROACHES

A1 and addressed to B1. The packets going out on interfaceA2 get encapsulated in new IP packets each with an extraheader having destination B1 and source A2. Likewise, eachpacket going out on interface A3 can be encapsulated in a newIP packet having destination B2 and source A3. The destina-tion (B) can then recognize IP-in-IP packets and strip the outerheader. This leaves the original packets with source A1 anddestination B1 to be delivered up the network stack to TCPin a transparent manner. The same encapsulation scheme isused for tunneling in the mobile IP standard [145]. In orderto avoid fast retransmission, Phatak et al. [148], [149] used aWRR style scheduler, which distributes packets proportionallyto the effective rates of the paths.

Chebrolu and Rao [27] presented a network layer architec-ture to aggregate bandwidth on multiple paths for real-timeapplications. They made the assumption that an infrastructureproxy (like the Home-Agent in Mobile IP [146]) is aware ofthe multiple interfaces of the client, and tunnels the capturedpackets to the client using IP-in-IP encapsulation. The advan-tage of a proxy solution is that it is fully controllable andallows servers to remain unchanged and hide using multipleIPs from TCP. Chebrolu et al. [26] proposed a schedulingalgorithm, Earliest Delivery Path First (EDPF), to ensure thatpackets meet their playback deadlines by scheduling pack-ets based on the estimated delivery time of the packets. Toimprove the overall performance of IP-in-IP tunneling basedbandwidth aggregation by the means of minimizing packetreordering, Chebrolu et al. [26] proposed a two-prongedapproach. Firstly, a scheduling policy Packet-Pair based EDPFfor TCP applications (PET) was used to partition traffic ontodifferent paths. The design of PET has the same concept ofEDPF but with idealized delay and bandwidth values replacingthe estimates. Secondly, working together with the schedulingpolicy, a receiver-side Buffer Management Policy (BMP) wasused to delay forwarding the out-of-order packets to TCP andto detect losses, so that a variety of adverse effects can behidden.

Kim and Shin [104], [105] introduced PRISM, anotherproxy based approach that enables TCP to efficiently utilize

the WWAN connections from community members. The proxycan be a trusted party or a community member. PRISM usesa cross-layer approach that involves support from both trans-port and network layer. We classify PRISM as network layerapproach because the PRISM proxy, which is the main entityfor multipath support, is located on the network layer. PRISMuses a packet-scheduling algorithm, i.e., Adaptive Scheduler(ADAS), to maintain up-to-date path state. Using the up-to-state information, ADAS sends packets according to theirexpected arrival time (a variant of EDPF algorithm) to reducepacket reordering. ADAS also uses the path state to adjustpath weight by using the Additive Increase and MultiplicativeDecrease (AIMD) strategy from TCP, so that ADAS candynamically react to congestion from partial paths and controlthe amount of traffic to be allocated on those paths. Moreover,PRISM masks the effects of out-of-order delivery by identify-ing spurious duplicate ACKs and re-sequencing them so thata TCP sender receives correctly sequenced ACKs.

Lan and Li [109] designed a different proxy based mul-tipath network protocol called Enhancements for TCP On aMulti-homed mobile router (ETOM) that runs transparentlyto both clients and servers. ETOM involves two proxy com-ponents instead of one kind: MR (Mobile Router) and HA(Home Agent). The client and the MR (as well as the serverand the HA) use normal single-path connection, whereas allpackets traveling between MR and HA are IP-in-IP encap-sulated. ETOM uses a reordering buffer to eliminate packetreordering. For example, out-of-order packets are buffered atthe HA until the missing packets are received, and packets arethen sent out in order to the destination. ETOM also uses avariant of EDPF algorithm to further reduce packet reordering.Note that unlike other IP layer approaches, ETOM employsa subflow sequence number in the inner IP header to detectpacket loss between MR and HA.

2) NAT Traversal: Unlike the previous approach that relieson IP encapsulation, there exist several approaches takingadvantage of NAT instead of tunneling.

Rodriguez et al. [166] introduced MAR system (seeFigure 5), a commuter mobile access NAT router that provides


Fig. 5. MAR [166] system architecture where the MAR router is placed in public mobile vehicles and data traffic is sent from remote servers to localdevices. PDN: Public Data Network, AP: Access Point.

a set of local interfaces and a number of wide-area wirelessinterfaces. The former provides access to local mobile devicesand the latter accommodates a variety of wide-area wirelesstechnologies. The MAR router acts as a NAT box that islocated in the middle and translates IP addresses and portsof packets for two directions. A MAR router can work aloneor cooperate with a MAR proxy-server. With such a proxy-server, the Packet-Oriented Scheduling Mode (POSM) is usedwhere the packets of the same TCP flow can be deliveredover multiple paths and a MAR router can implement intelli-gent optimization including avoiding TCP 3-way handshake,slow-start, spurious timeouts and so on. When the proxy-serveris absent, the Flow-Oriented Scheduling Mode (FOSM) isused where a per-flow allocation strategy schedules all pack-ets belonging to the same TCP flow onto the same path. MARprovides an API that can accommodate any custom purpose-built scheduling protocol. But the scheduling protocol itself isnot part of the MAR architecture. MAR is also designed todetermine the weight that should be assigned to each inter-face to properly perform load balancing (e.g., dynamicallyshifts load from poor quality to better quality channels). Notethat the MAR router was supposed to be placed in movingvehicles, where users can use their devices for Web-browsingand audio/video streaming. Therefore, the traffic load-balancedis only in one direction (i.e., from remote servers to localdevices).

Manousakis and Famolari [122] proposed INTELiCON toallow devices to exploit wireless access diversity. At the send-ing side, a Packet Processing module manipulates the contentof packets, e.g., modifying IP headers and Piggy-Backing forControl Signaling (PBCS) (e.g., timestamp and customizedsequence numbers). At the receiving side, the piggy-backedinformation can be utilized to smooth out the arrival sequenceof incoming packets. The extra information is stripped outat the Packet Processing module before the packets are for-warded to the upper layer. Moreover, INTELiCON uses aDATA/ACK separation (SEP) scheme to reduce contention onshared media. For example, it transmits the DATA and ACKpackets on different paths.

Evensen et al. [52] proposed a Multilink Proxy (MLP)that makes use of a NAT proxy to rewrite the default des-tination IP address and port to the address of the other

additional interfaces. The client does the inverse address trans-lation of the packets arriving at non-default interfaces andforwards the packets internally. In order to mitigate packetreordering, Evensen et al. [52] uses a WRR based sched-uler in the NAT to distribute packets according to estimatedthroughput ratio of available paths.

Habak et al. [68] proposed OSCAR architecture that worksin a distributed environment. An OSCAR-enabled node canshare and use the bandwidth available from its OSCAR-enabled neighbors to connect to both legacy and OSCAR-enabled servers. OSCAR has a NAT module at both sides ofa connection. At the sender, the NAT module replaces thesource and destination IP addresses with the used IPs fortransmission. Upon receiving a packet, the NAT module at thereceiver reverses the source and destination IPs by replacingthem with the negotiated ones before delivering the packet toTCP. When a connection goes through a shared neighbor to alegacy server, the neighbor also needs to have a NAT modulethat conducts the address translation operation. OSCAR uses aPacket Reordering Module (PRM) to handle packet reorderingissues. Specifically, it delays the packets and their ACKs onboth sides respectively before forwarding them to the upperlayer. OSCAR has two scheduling modes: POSM and FOSM.FOSM is used if the server is a legacy server, where PFA isused. POSM is used if the server is OSCAR-enabled such thata WRR style scheduler is used.

Some of the IP-in-IP encapsulation and NATbaed approaches, e.g., [26], [27], [52], [104], [105],[109], and [166], assume the presence of a proxy infrastruc-ture in the network. Nevertheless, such approaches work onlyfor plain-text TCP communication and fail in the presenceof IPsec encryption or authentication mechanisms. WhenTCP packets are protected with IPsec, the proxy is not ableto observe or modify the packet headers. Next we discusscertain bandwidth aggregation approaches that use IPSecencapsulation for tunneling.

3) Identity/Locator Split: Host Identity Protocol(HIP) [131]3 and Site Multihoming by IPv6 Intermediation(shim6) [137] have been proposed and implemented to

3HIP may not be considered as a strict IP layer approach; however, itsfunctions related to multipath transmission are best suited to this layer.


provide multihoming support for failover with the possibilityof flow-based load balancing. shim6 is architecturally relatedto HIP in that they both introduce an additional addressinglayer to allow changing IP addresses on network interfaces,while keeping constant transport-layer identifiers. These twoprotocols enable IP packet flows to dynamically changepaths in the presence of link failure. Thus, they naturallyshield the presence of multiple paths from transport andapplication layers, presenting only the global identity ofthe peer host. Nevertheless, HIP and shim6 do not supportsimultaneous multipath transmission without additionalextensions.

SIMA [150] is an extension of HIP to use multihoming forassigning separate TCP connections independently to differentpaths. Like FatVAP [95], TRILL [51], SPB [57], MAR [166]and OSCAR [68], SIMA also uses PFA multipath forward-ing strategy where flow bonding rules are created to definethe usage of the local interfaces. SIMA does not define anyadditional sending or receiving policies to mitigate reorderingissue, instead it uses the IPsec Encapsulating Security Payload(ESP) packet processing unit built in HIP to handle eachdata packet, as specified in [93]. Gurtov and Polishchuk [65]designed and implemented Multipath HIP (mHIP), a multi-path scheduler based on HIP, to distribute traffic over multipleavailable paths. Utilizing a EDPF scheduling algorithm, theystriped packets within a TCP connection to multiple pathsto mitigate packet reordering. Nevertheless, they found thatEDPF algorithm is only effective against packet reorderingwith stable paths in terms of bandwidth and delay. In order toreact to dynamic path characteristics, a Marking Technique isused as a part of the multipath congestion avoidance scheme,so that changes of path characteristics can be detected inone Round-Trip Time (RTT).

Polishchuk and Gurtov [151] proposed a TCP-friendlycongestion control algorithm for mHIP to prevent stealingbandwidth from legacy TCP flows at the shared bottleneck.Specifically, they proposed a two-level congestion controlscheme (removing the Marking Technique): per-path conges-tion control, and global congestion control on top of it. Theglobal congestion controller coordinates the individual per-path controllers and balances traffic load among the pathsbased on their available capacity. The per-path controllersare connected so that the aggregated congestion windowis the sum of per-flow congestion windows. The goal ofthis twofold congestion control scheme is to automaticallyredirect traffic from congested paths to the ones that haveavailable capacity. The concept of joint congestion controlalgorithm adopted in [151] is also used by certain trans-port layer approaches (which will be discussed in the nextsection). Thus, the concern is that the reordering and conges-tion avoidance algorithms used on the IP layer (or betweenIP and TCP, like HIP) may need to repeatedly design addi-tional mechanisms that are already existing on the transportlayer.

When ESP is used with HIP, a 64-bit sequence numbermust be used. Therefore, HIP based bandwidth aggregationapproaches such as SIMA [150] and mHIP [65], [151] all havea double sequence space design. However, instead of being

used for packet reordering, the additional sequence number inHIP is used for the purpose of anti-replay.

Unlike HIP and shim6 which focus on host-level identityand locator separation, Locator/Identifier Separation Protocol(LISP) [56] is an identity and locator separation protocolworking on the network-level to improve the scalability ofthe routing system. LISP creates two numbering spaces anduses two IP addresses: Endpoint Identifiers (EIDs) (assigned toend-hosts) and Routing Locators (RLOCs) (typically assignedto border routers). To achieve the separation of identificationand localization, LISP follows a map-and-encapsulate scheme.Specifically, upon reception of a packet from the local networkto an outer EID, the border router is responsible for look-ing up and retrieving the mapping (from a mapping system)between EID and RLOC and this process is invisible to theendpoints. Then the router encapsulates the packet with a LISPheader and an outer IP header with the destination RLOC asthe destination IP address. When the packet reaches the borderrouter assigned with the destination RLOC, the router decap-sulates the outer headers and forwards the inner packet to thedestination EID. LISP has two metrics to support multipathtransmission: RLOC priority and RLOC weight. If equal pri-ority is sent on the RLOCs, the RLOC weight could be usedfor the load-balancing ratio. Under such a setting, an IP-levelaggregate flow (e.g., the same destination prefix) would usedifferent paths.

Locator/Identifier Separation Protocol - Hybrid Access(LISP-HA) [127] is a mechanism to provide simultaneoushybrid access (e.g., DSL-line and LTE ) based on LISP tech-nology in both upstream and downstream direction. LISP byitself has basic capabilities to support hybrid access with staticload balancing. However, static load balancing may lead tostatistical variations [128] so that some paths are already over-loaded while others are underutilized. Instead, LISP-HA canperform dynamic per-flow load-balancing, which increases theefficiency of hybrid access. The basic idea is to obtain feed-back about path-specific packet loss and delay, and leveragethis information for improved load balancing. In addition,LISP-HA also supports dynamic per-packet load-balancing.Currently, the challenge is the packet reordering problem inthe case that paths have different delay.

As summarized in Table VI, packet reordering is one of themain challenges for all IP layer approaches. These approachesuse various scheduling algorithms to minimize the reorderingeffect. In Table VII, we have several observations. First, mostof the approaches were proposed either for mobile networks orwireless networks. Second, there are two primary schedulingschemes: EDPF and WRR. Third, some buffer managementstrategies are used to compensate for the inefficiency of thescheduling algorithm in the scenario of dynamically changingnetworks.

C. Transport Layer Multipath Transmission

Compared with IP layer based approaches, transport layerapproaches have certain inherent benefits because congestioncontrol can be used as a mechanism for resource allocationin a network. At this layer, end-systems can easily obtain


TABLE VIIICOMPARISON OF PROBLEMS ADDRESSED BY SCTP AND TCP BASED MULTIPATH TRANSMISSION APPROACHES

information about each path: capability, latency, loss rate andcongestion state. This information can then be used to reactto congestion in the network by moving traffic away fromthe congested paths. Current connection-oriented transportprotocols, e.g., TCP, Stream Control Transmission Protocol(SCTP) [187], and Datagram Congestion Control Protocol(DCCP), transmit data only over a single path between asource and a destination at any given time. Numerous attemptshave been made to tune these existing transport protocols formultipath capability. Currently, Concurrent Multipath Transfer(CMT) for TCP and SCTP are in the process of IETFstandardization.

Like bandwidth aggregation on network layer, concurrentmultipath transmission at the transport layer introduces anincrease in the occurrence of packet reordering due to differentpath characteristics, including run-time throughput, RTT, loss,and error. Specifically, if a connection is striped over mul-tiple network paths, the overall throughput may potentiallybe even worse than the throughput available on any one ofthe paths [58], [140]. There are two main causes of it. Thefirst comes from the impact of heterogeneous RTT. For exam-ple, TCP expects a first-in-first-out delivery of packets throughthe network. Packet reordering at the receiver results in thereception of duplicate ACKs at the sender. The sender willfast retransmit the “missing” packet that may still be on itsway over a high RTT path. Due to the misunderstanding ofpacket reordering, the overall throughput may degrade signif-icantly. The second cause is the receive buffer blocking dueto path heterogeneity or path failing. We provide more detailabout the receive buffer blocking problem in later discussionon CMT-SCTP.

In this section, we classify the state-of-the-art according totheir base protocols, in the order of SCTP and TCP. In eachdiscussion of SCTP and TCP based approaches, we furtherdivide them into two categories: with and without consid-ering fairness. In Table VIII, we make a comparison of thegeneral problems addressed by approaches in each category.In addition to the fairness issue, we also analyze the bufferimpact, Pareto-efficiency, and path diversity of the TCP basedapproach MPTCP. In the end, we give a comparison betweentwo representative approaches based on SCTP and TCP (i.e.,CMT-SCTP and MPTCP).

Table IX and Table X are used to summarize the key algo-rithms and approaches of SCTP based multipath transmissionrespectively. Likewise, Table XI and Table XII are used tosummarize the key algorithms and approaches of TCP basedmultipath transmission respectively.

1) SCTP Based on Multipath Transmission: A standardSCTP is a general-purpose, connection-oriented unicast trans-port protocol. An SCTP connection denotes an associa-tion. The user data is segmented into units of so calledDATA chunks,4 which are identified by unique TransmissionSequence Numbers (TSNs).5 The Selective Acknowledgment(SACK) [125] mechanism is used as default to acknowl-edge received data chunks and report gaps (i.e., missing datachunks indicated by their TSNs) to the sender. In this section,we divide the approaches into two groups differentiated bywhether they have considered fairness.

a) Multipath transmission without considering fairness:Unlike TCP, SCTP was designed with multihoming in mindthat an SCTP association allows multi-homed source and des-tination endpoints. Nevertheless, SCTP uses only one primarypath and switches to another path for retransmission of lostpackets, or as a backup in the case of failure from the primarypath. Note that SCTP uses a single buffer structure on bothendpoints, but maintains several states per destination: sepa-rate congestion window (cwnd), slow start threshold (ssthresh),retransmission timer, and RTT estimate. Several SCTP exten-sions, such as BA-SCTP [13], W-SCTP [24], CMT-SCTP [11],[89]–[91], [120], LS-SCTP [5], [6], WiMP-SCTP [85], cmp-SCTP [118], mSCTP-CMT [21] and FPS-SCTP [129], enableSCTP to transmit data over multiple paths simultaneously.

Argyriou and Madisetti [13] proposed BA-SCTP, a band-width aggregation protocol based on SCTP. BA-SCTP imple-ments a mechanism for identifying bottlenecks that are sharedby flows from the same aggregate connection. Based on thismechanism, BA-SCTP performs a unified congestion controlalgorithm for the flows that share the same bottleneck (wename this algorithm UCCSB) instead of applying congestioncontrol for each flow separately. This design guarantees that

4Corresponding to segments in TCP.5TSN serves the same function in SCTP as the sequence number does

in TCP.


TABLE IXKEY ALGORITHMS FOR SCTP BASED CONCURRENT MULTIPATH TRANSMISSION (SORTED ACCORDING TO THEIR ORDER MENTIONED IN TABLE X)

BA-SCTP flows are fair with the other TCP flows sharing thesame bottleneck. BA-SCTP employs a WRR style schedulingstrategy, a congestion window based data allocation strategy

where each subflow pulls data from the shared sending bufferwhenever the subflow has congestion window space to senddata. This strategy assumes the congestion window to be a true


TABLE XCONCURRENT MULTIPATH TRANSFER PROTOCOLS BASED ON SCTP

representative of the bandwidth-delay product of the subflowcompared to an estimated product. In the rest of the paper,we use WRR-PULL to denote congestion window based dataallocation strategy.

Casetti and Gaiotto [24] proposed W-SCTP, aWestwood [124] flavored SCTP to exploit bandwidthaggregation. The authors believed that Westwood stylecongestion control could fully exploit the advantages ofbandwidth estimation which could be utilized for trafficallocation among multiple flows. W-SCTP uses a EDPF stylescheduler that chooses the path for next packet by predictingwhether it can deliver the packet the fastest to the destination.

Iyengar et al. [89], [91] proposed integrating CMT capabil-ity into SCTP, namely CMT-SCTP. CMT-SCTP utilizes themultihoming feature from SCTP to correctly transfer databetween multi-homed end hosts. They identified three neg-ative side-effects of CMT, and proposed algorithms to solvethem accordingly. First, they proposed a Split Fast RetransmitChangeover Aware Congestion Control (SFR-CACC) algo-rithm to eliminate the unnecessary fast retransmissions byusing a different interpretation of SACK information. Second,they used a congestion window (cwnd) growth algorithm totrack the earliest outstanding TSN per destination and updatethe cwnd, even in the absence of new cum ACKs. Third,they proposed a new Delayed ACK algorithm for CMT-SCTP,namely Delayed ACK for CMT (DAC). The algorithm allowsthe receiver to delay sending ACK of an out-of-order segment.In [90] and [91], they proposed five retransmission policies forCMT. They demonstrated the occurrence of spurious retrans-missions with all of those policies, and proposed amendmentalgorithms to avoid them.

There has been a considerable amount of work on thecore CMT-SCTP [91] to overcome its defect and improveperformance. Liu et al. [120] found that all of the five retrans-mission policies may cause throughput degradation due to

Fig. 6. HLB: the receive buffer cannot accommodate other chunks any morebefore the arrival of the head-of-line chunk (chunk 1). HLB: Head-of-LineBlocking.

receive buffer blocking. This blocking problem is also namedHead-of-Line Blocking (HLB). Figure 6 illustrates a simplifiedexample of it. As shown in the figure, a CMT-SCTP receivermaintains a single receive buffer which is shared across twosub-association flows in an association. The C1 (chunk 1) istransmitted through the path 2 and is lost due to traffic conges-tion or path failure. During the time period of C1’s retransmis-sion, the receive buffer cannot accommodate any other packetsdue to flow control so that the overall throughput degrades.To solve the problem, Liu et al. [120] proposed a compoundparameter retransmission policy named RTX-LCS. It limits theretransmission path selection by considering three commonconditions: cwnd, ssthresh, and loss rate. Adhari et al. [8]and Dreibholz et al. [49], [50] examined the challenges ofCMT-SCTP over dissimilar paths. They identified the issuesof sender and receiver queue blocking, which may lead topoor overall performance. In order to improve performance,Dreibholz et al. [49] proposed multiple mechanisms accord-ingly, including Buffer Splitting, Chunk Rescheduling, andSmart Fast Retransmission. Buffer Splitting is used to avoidone path occupying too much buffer space, which preventsother paths from sending out new chunks. Chunk Rescheduling


TABLE XIKEY ALGORITHMS FOR TCP BASED CONCURRENT MULTIPATH TRANSMISSION (SORTED ACCORDING TO THEIR ORDER MENTIONED IN TABLE XII)

copes with the problem of certain delayed or lost chunksstalling the whole transmission. Smart Fast Retransmissiondeals with spurious fast retransmission bursts. For example,

it does not consider chunks being moved from another pathin the decision about fast retransmissions on the new path.In [8], they presented an optimized buffer handling technique


TABLE XIICONCURRENT MULTIPATH TRANSFER PROTOCOLS BASED ON TCP

to further improve performance. Specifically, they proposed touse the shared buffer space dynamically so that a faster pathcan have the possibility to send more data by granting it morebuffer space. In this paper, we use Buffer Splittingv2 to denotethis updated version of Buffer Splitting mechanism. Moreover,in [50], they proposed using the Multi-streaming feature ofSCTP to mitigate the HLB problem. Specifically, each mes-sage is assigned an identifier to indicate a stream. With thisidentifier, the protocol only needs to restore the sequence ofmessages belonging together. Hence, after a packet loss, onlythe messages of the affected streams have to be delayed torestore the sequence. The other messages can be processedimmediately without delay.

The authors in LS-SCTP [5], [6] and cmpSCTP [118]proposed separating the association flow control from perpath congestion control (denoted as FCCS). The conges-tion control is performed per path, whereas the flow controlis performed per association. In order to achieve this goal,LS-SCTP uses two different sequence numbers. The first oneis the Association Sequence Number (ASN) that is used toreorder the received data at the receiver association buffer.The second one is the Path Sequence Number (PSN) that isused for reliability and congestion control on each path. Thescheduling module of LS-SCTP uses the current congestionwindow (cwnd) of each path as an estimate of its currentbandwidth-delay product. For example, it assigns data to the


paths according to the cwnd/RTT of each path. cmpSCTP dis-tributes data over available paths based on real-time bandwidthestimation of each path. Similar to LS-SCTP, cmpSCTP alsodistributes the data on the available paths based on the esti-mation of the available bandwidth of each path. Thus, they alluse a WRR style data scheduler.

Sarkar [172] proposed Concurrent Multipath TCP(cmpTCP), an extension of SCTP. cmpTCP splits packetsconcurrently over all available paths from a shared sendingbuffer. cmpTCP maintains a virtual retransmission queue(RTxQ) on each path to control the number of outstandingbytes on the path. The receiver sends back ACKs on the samepath on which the packets are received. These two designsmay help to ignore spurious gap reports and eliminate unnec-essary packet retransmissions. Sarkar [172] also developeda Markov model in cmpTCP to estimate the data transportrate on each path when the transmission has reached a steadystate. cmpTCP uses a WRR style scheduler by consideringthe number of outstanding bytes and congestion window size.

Huang and Tsai [85] proposed Wireless Multi-Path SCTP(WiMP-SCTP), which devised two data transmission modes,i.e., Data-striping Mode and Data-duplicating Mode, for multi-path transmission in multiple wireless access networks. Whenthe network status is good, the Data-striping Mode is selectedto aggregate bandwidth. On the other hand, when the networkstatus becomes bad, the Data-duplicating Mode is selectedto increase destination reachability. To switch between thetwo modes, a mode selection scheme that determines thestatus of these multiple paths was proposed. Specifically, itdesigned a HEARTBEAT scheme where heartbeat chunksare sent periodically on the paths. The transmission errorcounter increases when a heartbeat is not acknowledgedwithin one retransmission timeout interval. If the numberof transmission error counter plus the number of consecu-tive retransmissions exceeds a certain threshold, the Data-duplicating Mode is switched on. Otherwise, the Data-stripingmode is used. Nevertheless, Huang and Tsai [85] did notpresent a complete explanation of the scheduling algorithmused by WiMP-SCTP. They only mentioned that the sendertransmits data as soon as the corresponding receive win-dow at the receiver side allows data to be sent. We spec-ulate that this is a variant of the WRR style schedulingalgorithm.

Budzisz et al. [21] proposed an mSCTP-CMT protocol toinvestigate the applicability of using CMT-SCTP to distributedata between two paths during the handover transition pro-cess. They emphasized the consequence of a sender-introducedreordering and its effect on congestion control. The authorsfound that in CMT-SCTP the receive buffer may be filled without-of-order data caused by complete or short-term failuresduring handover. Although handover is out of the scope ofthis paper, the handover scenario is very similar to the worstcase in CMT where a path experiences a long delay suddenly.To solve this problem, they proposed using Potentially Failed(CMT-PF) scheme that a path experiencing a single timeoutis marked as “potentially failed” and no further data trans-mission is allowed on that path. They utilized a heartbeatscheme to probe whether the potentially-failed path has got

(a) (b)

Fig. 7. RP (a) a single path shares its resource fairly to competing flows (b)multiple paths are treated as a single pooled resource. RP: Resource Pooling.

back to a positive state in the case of successful heartbeatacknowledgement.

Mirani et al. [129] proposed a multipath Forward PredictionScheduling (FPS) for SCTP, namely FPS-SCTP. In order toreduce the number of out-of-order packets, it estimates thearrival time of each packet in advance and decides whichpacket is to be sent through a certain path so that the packetscan arrive at the destination in order. They used roughly a halfof the RTT on a subflow to estimate the one trip time fromdata leaving the send buffer to being received at the receiver.This prediction is an approximation considered as too coarsebecause previous studies have shown that a majority of Internetconnections experience latency asymmetry [141], [197].

b) Multipath transmission considering fairness: Thework discussed previously on CMT-SCTP performs indepen-dent congestion control on each path, and considers little aboutthe fairness against other single-path flows. For example, inthe case of n CMT-SCTP paths, the association will get ntimes the bandwidth share of a competing non-CMT SCTP orTCP flow over the same bottleneck.

The concept of RP [200] is a milestone for multipath trans-mission aggressiveness control. In the context of multipathtransmission, it makes a collection of resources behave like asingle resource by balancing traffic across multiple paths. Asshown in Figure 7, when several TCP flows compete for a sin-gle path, TCP can share the path’s capacity fairly among them.When the flows go through more than one path, the paths aretreated as a single pooled resource. With appropriate coordi-nation, traffic can move away from more congested paths toless congested ones, and larger bursts can be accommodated.

Dreibholz et al. [48] proposed a RP congestion controlfor CMT-SCTP denoted as CMT/RP-SCTP by combiningCMT-SCTP with the concept of RP. The goal of CMT/RP-SCTP is to improve the data throughput while still remainingfair to concurrent single-path flows on the shared bottle-neck. For example, when two paths are used concurrentlyfor data transmission and they share a bottleneck link, theoverall throughput obtained by a CMT/RP-SCTP associa-tion should be similar as that of a standard SCTP. However,CMT/RP-SCTP assumes similar paths, i.e., paths having verysimilar characteristics in terms of bandwidth, delay and lossrate. Dreibholz et al. [47] proposed an updated version ofCMT/RP-SCTP (denoted as CMT/RPv2-SCTP) to overcomethe limitations of CMT/RP-SCTP by considering path band-widths. They studied the behavior of CMT/RPv2-SCTP ondissimilar paths and found that CMT/RPv2-SCTP achieves thegoals of RP. Furthermore, they observed that compared withMPTCP (which will be discussed later), CMT/RPv2-SCTP


distributes bandwidth to flows equally when possible regard-less of the number of paths used for transport.

Note that both CMT/RP-SCTP and CMT/RPv2-SCTP applyRP principle only during the congestion control phase.Shailendra et al. [175] argued that this strategy is not opti-mum on heterogeneous networks with wireless links becauselosses in wireless links may happen because of reasons otherthan congestion. Hence they considered the resources to be as asingle pool of resources during the congestion detection phaseas well. To achieve this goal, they proposed a BandwidthEstimation Based Resource Pooling (BERP) algorithm whichapplies the RP principle based upon the bandwidth estimatesobtained by observing the data flow on the paths.

2) TCP Based Multipath Transmission: In contrast toSCTP, which was designed with multihoming support innature, TCP is unaware of multiple interfaces and allowsonly a single IP address per endpoint. Nevertheless, TCPhas dominated the Internet traffic and has sparked a lot ofinterests in enabling TCP to support simultaneous multipathtransmission. In this section, we divide the approaches infour groups. The grouping principle is influenced by theresearch issues the approaches aim to address, such as fair-ness, buffer impact on performance, Pareto-optimality andpath diversity. Although some approaches may cover morethan one research issue, we only discuss them in the groupwhich we believe the approaches are best fit into. In ourstudy, we found that SCTP and MPTCP share many sim-ilar issues and certain algorithms. At the end of this sec-tion, we summarize their common features as well as theirdifferences.

a) Multipath transmission without considering fairness:Magalhaes and Kravets [121] proposed Reliable MultiplexingTransport Protocol (R-MTP), which is a rate-based reliabletransport protocol multiplexing data across multiple networkinterfaces (i.e., a WRR style scheduler). It relies on explicitbandwidth probing via the packet pair method [99] to esti-mate bandwidth in order to adjust the rate on the availablepaths accordingly. For example, it measures packet inter-arrival times and jitter to sense bandwidth scarcity. Theprobing period should occur on a fine time-scale to reflectthe fluctuation of the available bandwidth.

Lee et al. [111] supported two transmission modes in theirwork: FOSM and POSM. In POSM, they investigated multipleschemes to address the spurious retransmissions by modify-ing two TCP operations: 1) Increasing the Fast RetransmitThreshold (IFRT) and 2) enabling Delayed ACKs for out-of-order packets as well as sending immediately ACKs forretransmitted packets. The second modified operation is likean advanced version of DAC used in [89] and [91]. Thus, wename it DACv2 in this paper. IFRT makes the TCP senderwait for more than triple duplicate ACKs, which reduces thenumber of the fast retransmission and the fast recovery events.DACv2 enhances performance because when ACKs are beingdelayed, new packets may fill the gap and change the out oforder packets in order.

Parallel TCP (pTCP) [82]–[84] functions as a wrap-per around a modified version of TCP. It opens multipleTCP flows, one for each interface in use. pTCP performs

data-striping across multiple micro-flows (TCP flows) by con-sidering their bandwidth difference. Specifically, pTCP usescwnd/RTT ratio, a WRR-PULL scheduler, to allocate trafficproportionally to path capacity. In addition, pTCP has severalother strategies addressing specific problems. For example, thecongestion window could be an over-estimate especially justbefore congestion occurs. This can result in an undesirablehold up of data in subflows. Instead of reassigning data to othersubflows later on, pTCP uses a Delayed Binding strategy toadapt to instantaneous changes in path capacity. Specifically,it pulls data from the shared sending buffer only when thedata is scheduled to send out immediately through a subflow.In order to avoid an overflow of the receive buffer, pTCP usesa Packet Re-striping strategy to retransmit a packet through adifferent subflow instead of the subflow which transmitted thatpacket earlier, and uses a Redundant Striping strategy to send aduplicated packet on one subflow to another. Moreover, pTCPuses SACK feedback mechanism to recover a lost packet in amuch shorter time period.

Reception Control Protocol (RCP) [81], [106] is a receiver-centric transport protocol with a minimized sender design. Thereceiver controls all the key functions in RCP. To supportCMT, a multi-state extension of RCP, i.e., Radial RCP (R2CP),was proposed. R2CP maintains one RCP pipe (the same as aTCP flow) per end-to-end path with congestion control beinghandled by individual RCP pipes. Traffic is scheduled to eachRCP pipe based on the (estimated) time the requested segmentwill arrive through the concerned pipe (a variant of EDPF).Note that each RCP pipe maintains a local sequence numberspace internally to facilitate loss detection and recovery. Thelocal sequence number can be converted to the global sequencenumber, and vice versa.

Chen et al. [28] proposed M-TCP that uses a DuplicateTransmission mode for the lossy wireless environment withhigh interference. In this transmission mode, multiple copiesof the same packet are sent on different paths so that thechance that all copies are lost is much reduced. Unfortunately,they only present sending-side modification without address-ing duplicate ACKs due to multiple copies of the samepacket.

Rojviboonchai and Hitoshi [167] proposed a multipathTransmission Control Protocol (M/TCP). M/TCP uses One-Way-Trip Time (OWTT) [169], a similar method as FPS, atthe sender to estimate the delay time of the forward path andreverse path separately in order to calculate per path RTOtimer. In addition, M/TCP employs two mechanisms to dealwith packet loss. In the case of fast retransmission, M/TCPuses Duplicate Transmission policy, i.e., duplicating the miss-ing segment and sending each copy through all paths so thata quick and reliable retransmission can be desirable; in thecase of timeout retransmission, the missing segment on a flowis sent through the other flows. Moreover, M/TCP uses twoalgorithms for the receiver to transmit an ACK in the case ofCMT. Namely, using Duplicated ACK algorithm, an M/TCPreceiver sends an ACK immediately upon the receipt of a datasegment to more than one path; using Duplicated & DelayedACK, the receiver transmits an ACK for every other data seg-ments through more than one path. Rojviboonchai et al. [168]


proposed R-M/TCP as an extension of their previous workM/TCP. R-M/TCP is a rate-based M/TCP that performs con-gestion control in a rate-based and loss avoidance manner(we use RCC to denote it) to avoid packet loss by adjustingthe congestion window before buffer overflows. Specifically,R-M/TCP schedules data packets in a WRR manner while itestimates the queue length at the bottleneck link. If the queuelength grows beyond a predefined threshold, the sender recal-culates a new congestion window to achieve a fair share at thebottleneck.

Cetinkaya and Knightly [25] proposed an OpportunisticMultipath Scheduler (OMS) that follows a traffic splittingpolicy that favors low-delay high-throughput paths opportunis-tically for a short term variations in path quality. To avoidviolating path weights of the routing protocol and potentiallyleading to oscillation, OMS ensures that over longer timescales traffic is split according to the ratios determined bythe routing protocol.

Dong et al. [46] proposed concurrent TCP (cTCP) that usesa single congestion window to control the global throughputand a single sending buffer to be shared among all paths.It uses Credit-Weighted Round-Robin (a variant of WRR) asthe scheduling algorithm. Each time an ACK comes back tothe sender, the capacity estimation of that path is updated,and a new sending credit (similar to the congestion windowsize) is added to the sender. The new credit is for all thepaths combined, and it is further divided into each path. cTCPuses the path credit (similar to the path capacity) to distributedata among the available paths. Furthermore, cTCP adopts aduplicated ACK classifier that handles packet reordering bydifferentiating whether a duplicated ACK is likely caused byCMT or a real duplicated ACK.

Sharma et al. [178], [179] proposed Multi-Path LOss-Tolerant protocol (MPLOT) to provide multipath transmissionon multiple heterogeneous, highly lossy paths. MPLOT useserasure based Forward Error Correction (FEC) packet coding.The major benefit of packet coding stems from its ability tocompensate for missing packets from redundancy. This makesdata transmission over lossy networks robust and efficient.To counter against packet reordering, MPLOT estimates pathparameters (i.e., loss rate, capacity, and RTT) continuously toprovide adaptive FEC coding. In particular, MPLOT performslatency-aware packet mapping, a variant of EDPF. For exam-ple, it maps packets that are not required immediately to pathswith long delays, while mapping the more immediately use-ful packets to paths with short delays. MPLOT uses ExplicitCongestion Notification (ECN) [164] to distinguish congestionlosses from those due to faulty/lossy links.

Wang et al. [198] proposed a segment-based adaptive JointSession Scheduling (JOSCH) mechanism. The main goal isto restrain the delay difference among multiple Radio AccessNetworks (RANs) by means of allocating the traffic to dif-ferent RANs dynamically with reasonable ratios. Specifically,JOSCH obtains network conditions by a segment-based feed-back approach, where a “segment” is defined as a predeter-mined size of data block. Its size is configurable accordingto the delay sensitivities of different services. After each seg-ment transmission, the receiver sends feedback to the sender.

According to the feedback, the sender adjusts traffic allocationdynamically according to the estimated transmission rates anddelays.

Tsao and Sivakumar [191] argued that aggregated band-width of two wireless interfaces (3G and WiFi) is a SimpleAggregation due to path heterogeneity. For example, the lowbandwidth interface (e.g., 3G with 100-500Kbps) can onlyachieve negligible bandwidth compared to the high band-width interface (e.g., WiFi with 2-54Mbps). They proposeda super-aggregation to achieve performance that is betterthan the sum of throughput achievable through each of theinterfaces individually by the means of three mechanisms:Selective Offloading, Proxying, and Mirroring. In spite of thefact that an interface may have a relatively small amount ofbandwidth, these mechanisms can provide considerable per-formance improvement. Specifically, the Selective Offloadingmechanism is to receive TCP data segments over a compa-rably high-speed WiFi and return ACKs over a low-speed3G path to address self-contention in WiFi networks. TheProxying mechanism, following IP-in-IP encapsulation, allowsa 3G path to notify the TCP sender about blackout eventson the WiFi path. The Mirroring mechanism (i.e., DuplicateTransmission) establishes an addition TCP connection through3G to fetch the missing segments due to random loss on theWiFi path.

b) Multipath transmission considering fairness: Similarto the early development of SCTP based CMT, at the earlystage, TCP based multipath transmission was only used forutilizing multiple TCP flows with intelligent scheduling algo-rithms to mitigate packet reordering. Nevertheless, it wasfound that simply utilizing multiple TCP flows concurrentlyat a bottleneck would result in a fairness issue, i.e., an unfairshare of the bandwidth at a bottleneck link. For example,NewReno [77] is the most common TCP congestion controlvariant as it yields an equal share of the congested link. Thisequal share outcome of NewReno results in an unfair share ofthe bandwidth if more than one TCP flow is active for a sin-gle multipath transmission connection at the bottleneck link.From the literature review we have made, we find that mul-tipath transmission approaches based on TCP in recent yearshave also started to make fairness a necessary feature.

To the best of our knowledge, mTCP proposed byZhang et al. [204] is among the earliest proposals thathave taken fairness issue into consideration for TCP basedmultipath transmission. To address the fairness issue, theyproposed not establishing multiple flows through the same bot-tleneck. For example, to alleviate the aggressiveness problem,Zhang et al. [204] integrated a shared congestion detec-tion mechanism into mTCP so as to identify and suppresssubflows that traverse the same set of congested links. Forexample, mTCP detects shared congestion by examining thecorrelations among the fast retransmit times of the subflows.mTCP also uses a scheduler in a WRR manner. For exam-ple, it maintains a counter, i.e., pipei, to represent the numberof outstanding packets on the ith path. pipei is incrementedby 1 when the sender either sends or retransmits a packetover the ith path, and is decremented by 1 when an incomingACK indicates that a packet previously sent has been received.


Fig. 8. The architecture of MPTCP, which has the application compatibilityby keeping the standard socket API to legacy applications. MPTCP: MultipathTCP, src: Source IP address, dst: Destination IP address, sp: Source Port, dp:Destination Port, APP: Application.

The sender associates a score, i.e., pipei/cwndi, for each path.The path with the minimum score has priority to send the nextpacket.

Instead of avoiding shared bottlenecks, Honda et al. [79]proposed Bidimensional-Probe Multipath Congestion Control(BMC) to address the fairness issue. Specifically, BMC usesa Weighted Congestion Control (WCC) approach that appliesthe weight to each subflow so that the throughput of eachsubflow is in proportion to its weight. In addition, WCC main-tains the sum of the weight so that a bundle of subflows inthe multipath connection is kept as aggressive as one TCPflow. WCC can achieve not only fair resource allocation atthe shared bottleneck, but also RP [200] along the disjointbottlenecks. For example, multiple different connections canobtain fair resource allocation across distinct bottlenecks.

Approximately in 2009, MPTCP [60], [158], [162] was pro-posed with the fairness property as well as RP feature inmind. Specifically, MPTCP, under discussion of IETF, has thefollowing set of goals to achieve:

• Improve throughput: MPTCP should perform at least asa single TCP flow running on the best path.

• Do no harm: MPTCP subflows should not take morecapacity than a single TCP flow would get at a sharedbottleneck.

• Balance congestion: MPTCP should utilize the leastcongested path the most.

Figure 8 shows the architecture of MPTCP. It is a majorextension to TCP and allows a pair of hosts to use severalpaths to exchange the segments that carry the data from a sin-gle connection. MPTCP presents a standard TCP socket APIto the upper layer so that legacy applications can run uponMPTCP transparently. A Coupled Congestion Control (CCC)algorithm, Linked Increase Algorithm (LIA) [159], is used toguarantee fair resource allocation on multiple paths and pro-vide RP feature among them. Its RP feature can shift trafficaway from more congested paths to less congested ones. Inaddition to the joint congestion control algorithm, MPTCP alsohas a few other design features. For example, MPTCP addsconnection-level sequence numbers in order to reassemble thedata stream in-order from multiple subflows. A Data Sequence

Signal (DSS) option [62] specifies a full mapping from theconnection-level sequence number to the subflow sequencenumber. In the early stage of MPTCP, a PUSH style WRRscheduling strategy is used where the scheduler tries to fillall subflows when there is data coming from the application.Later, MPTCP adopts a WRR-PULL manner scheduler [17], asimilar design adopted by pTCP [82]–[84] where the applica-tion data stored in a shared connection-level sending bufferis pulled by subflows whenever they have space in theircongestion window. Both PUSH and WRR-PULL schedulingstrategies are variants of WRR. Their difference lies in the factthat WRR-PULL strategy uses less time waiting in a subflowqueue before its actual transmission on the wire. Although thistime period seems minor, the path properties may change dur-ing that time. For the latest development of MPTCP in IETF,we refer readers to RFCs such as [61], [61], and [159].

Raiciu et al. [161] were the first proposing a natural evo-lution of data center transport from TCP to MPTCP. Theydemonstrated that MPTCP could efficiently and seamlessly useavailable bandwidth, provide improved throughput and betterfairness compared to single path TCP. The same authors fur-ther investigated what caused these benefits in [157]. Theyfound that using MPTCP allows to rethink data center net-works and approach them with a different mindset as to therelationship between transport protocols, routing, and topol-ogy. One of the challenges of deploying MPTCP in data cen-ters is the Incast effect, a behavior of MPTCP as well as TCPthat results in the gross under-utilization of link capacity incertain many-to-one traffic patterns [40], [132], [147], [190].Incast collapse is not specific to MPTCP, but is inheritedfrom TCP. Li et al. [117] investigated how to share networkresources among different MPTCP flows by performing anadditional weighted congestion control based on the coupledcongestion control mechanism.

Although LIA ensures MPTCP subflows to be no moreaggressive than a competing TCP flow, LIA is not able todifferentiate whether the subflows share the same bottleneckor different ones. This may cause sub-optimal performance ofMPTCP. Hassayoun et al. [76] proposed a Dynamic WindowCoupling (DWC) algorithm to address it by detecting dis-tinct bottlenecks and only coupling those that share a commonbottleneck, while allowing other subflows to use separate con-gestion control. For example, DWC uses a loss congestionevent to trigger an alert while using either the Delay or Losscongestion event to group subflows. Singh et al. [182] pro-posed an extension of the DWC algorithm, denoted as EDWC.This extension uses a delay congestion to trigger an alert whileeither delay or loss congestion is used to group and couplesubflows. The concept behind DWC and EDWC algorithms issimilar to that of UCCSB algorithm used in BA-SCTP [13]because they both identify the shared bottleneck and guaranteethat the aggregated flows on the bottleneck is TCP-friendly.

Diop et al. [44] proposed QoS-oriented MPTCP that takesadvantage of the two sub-layers architecture of MPTCP touse Quality of Service (QoS) techniques for multimedia appli-cations over multiple paths. MPTCP originally offers a fullyreliable and fully ordered service. Nevertheless, full reliabil-ity may not be required by certain multimedia applications.


Diop et al. [44] investigated the QoS benefits induced bythe implementation of the Partial Reliability [12] feature inMPTCP for interactive video applications. Partial Reliabilityis an important concept for multimedia transmission over IPnetworks and is defined as the possibility to not recover lossesunder a threshold in order to improve QoS.

Peng et al. [142], [143] presented a fluid model to investi-gate a few existing congestion control algorithms designed forMPTCP, and identified design criteria that guarantees the exis-tence, uniqueness, and stability of system equilibrium. Theycharacterized algorithm parameters for TCP friendliness andproved that there is an inevitable trade-off between respon-siveness and friendliness. Based on the study, in [143] theyproposed a new congestion control algorithm, Balia (balancedlinked adaptation). This algorithm generalizes existing algo-rithms and strikes a good balance among TCP-friendliness,responsiveness, and window oscillation.

Lim et al. [119] proposed MPTCP-MA to improve MPTCPperformance during intermittent path connectivity in wirelessenvironment. MPTCP-MA exploits MAC-Layer informationto estimate path status, and suspends/releases a path based onthe estimation. By quickly detecting path failure and recov-ery, MPTCP-MA can avoid unnecessary losses and utilizerecovered paths more quickly.

c) Buffer impact on MPTCP: Although MPTCP wasdesigned with several merits such as fairness, RP, and Pareto-optimality in mind, buffer size has significant impact onMPTCP performance. This problem stems from the packetreordering issue due to heterogeneous path characteristics. Wenow discuss the related work which has explicitly examinedthe impact of buffer size on MPTCP. Note that the impact ofbuffer size is not limited to MPTCP but on all other approachesusing POSM.

Barré et al. [17] evaluated the impact of heterogeneouspaths on the receive buffer and aggregated throughput. Theexperiment result shows that losses on one subflow have alimited impact on the performance of the other subflows.Nevertheless, this observation is based on the assumptionthat the reordering buffer is big enough to accommodate allthe out-of-order data. Nguyen et al. [133], [134] evaluatedthe performance of MPTCP in terms of load sharing andthroughput optimization with and without LIA respectively.The results show that the context of mismatched path char-acteristics has a great impact on the performance of MPTCPwith constrained receive buffers. Kim et al. [103] proposeda reordering scheme that considers packet scheduling algo-rithm at the sender to reduce the receive buffer. The mainidea is to estimate packet arrival time and schedule pack-ets accordingly. The sender maintains a per path time tableincluding calculated values of receiving time at the receiverfrom now for each packet. When the sender has opportunityto send a new packet, it chooses the path that can deliver thepacket faster than others. Thus, it is a EDPF style schedulingalgorithm.

The impact of buffer size on MPTCP performance has alsobeen observed in [37], [38], and [114]–[116], which proposedpacket coding based approaches to address it. For example,Li et al. [114] proposed NC-MPTCP that introduces packet

coding to some but not all subflows. The regular subflowsdeliver original packets while the coding subflows deliverlinear coded packets. The coded packets are used to compen-sate for the lost and much delayed packets in order to avoidreceive buffer blocking. They used an out-of-order schedulerthat calculates the expected packet arriving time by takingRTT, throughput, and loss ratio into account. Thus, the pack-ets that are sent out of order are expected to arrive at thereceiver in order. This scheduling algorithm is the same asthe FPS algorithm used in [129]. Li et al. [115] proposedSC-MPTCP to mitigate the packet reordering issue with con-strained receive buffer. In SC-MPTCP, they proposed to makeuse of coded packets only as redundancy to compensatefor expensive retransmissions while minimizing the encod-ing/decoding operations. The redundancy is provisioned inboth proactive and reactive manners. Specifically, SC-MPTCPtransmits proactive redundancy first and then delivers the origi-nal packets. The proactive redundancy is continuously updatedaccording to the estimated aggregate retransmission ratio. Inorder to avoid the proactive redundancy being underestimated,a pre-blocking warning mechanism is utilized to retrieve thereactive redundancy from the sender. Cui et al. [37], [38] pro-posed applying the fountain code for multipath scheduling tomitigate the impact of path heterogeneity. They also designeda data allocation algorithm based on the expected packet arriv-ing time and decoding demand to coordinate the transmissionsof different subflows. Li et al. [116] dealt with the packetreordering issue from a different perspective. They demon-strated that the traditional Delayed ACK mechanism can leadto significant performance degradation in the presence of time-outs. Thus, they proposed a New Delayed ACK (NDA) forMPTCP aiming to remove the Minimum RTO constraint atthe sender while to reserve the Delayed ACK function at thereceiver.

Raiciu et al. [160] proposed schemes of opportunisticretransmission and penalizing slow subflows to avoid thereordering problem. If a subflow holds up a packet at thetrailing edge of the receive window, the opportunistic retrans-mission allows the sender to resend the packet that is pre-viously sent on another subflow. This scheme is similar tothe Packet Re-striping used in [81]–[84] and [106] and Pre-blocking warning used in [115]. These three similar schemesare used in different scenarios but for the same purpose.Opportunistic retransmission is used only if the connection isreceive-window limited. Packet Re-striping is employed in thecase of path capacity fluctuations. Pre-blocking warning istriggered if the proactive redundancy is underestimated. Thepenalizing scheme of [160] is used to slow down the slowsubflows. For example, if a subflow has caused too manyout-of-order packets in the reordering buffer, the congestionwindow of that subflow is reduced by half and its slow-start threshold is set to the current congestion window. Butif that subflow has been in the slow-start phase already, thereordering problem may become worse because the penaliza-tion mechanism will set its slow-start threshold to a smallervalue. Paasch et al. [140] proposed improving the penalizationmechanism by adjusting the slow-start threshold only whena subflow is not in its slow-start phase. However, they also


identified that the penalization mechanism is far from perfectbecause a subflow at full sending speed may still overflow thereceive buffer while another subflows is in slow-start.

Ferlin et al. [58] argued that the opportunistic retrans-mission scheme does not reduce the effect of extreme RTTheterogeneity. Instead, they proposed a Dynamic Relative PathScoring (DRePaS) algorithm to dynamically score the pathsrelative to the best path and adapt the scheduling accordingly.Specifically, when the score of a path is less than a threshold,no payload is scheduled over that path until its score is largerthan the threshold measured by probing traffic. Note that thestandard MPTCP uses the congestion window as an estimationof path capacity. In contrast, Ferlin et al. [58] believed that thesmoothed in-flight data on each path reflects the behavior ofthe connection more dynamically than the congestion window.

Chen et al. [30] explored the performance of MPTCP overwireless networks. In order to avoid performance degradation,they set the receive buffer up to 8 MB, which is not feasiblein practice for many devices. Shamszaman et al. [177] ana-lyzed the feasibility of MPTCP for big data applications. Theyfound that constrained receive buffer leads to poor perfor-mance of MPTCP. Zhou et al. [206] proposed CWA-MPTCPthat examines the goodput of MPTCP with bounded receivebuffers. They found that if the paths have similar end-to-enddelays, the MPTCP goodput is near optimal, otherwise thegoodput will be degraded significantly. For a wireless environ-ment, they proposed a Congestion Window Adaptation (CWA)algorithm that can adjust the congestion window dynamicallyfor each TCP subflow so as to mitigate the variation of end-to-end path delay, maintaining similar end-to-end delays overmultiple paths. The primary idea behind CWA is that a largedelay ratio indicates that the high-delay path is overloaded.Its congestion window needs to be decreased to relieve traf-fic and reduce path delay. For wired environment with stableend-to-end delay they proposed using a delay-aware schedul-ing algorithm to predict the receiving sequence, i.e., a FPSmanner scheduler, so that packets can arrive at the receiver inorder.

Paasch et al. [139] designed and implemented a genericmodular scheduler framework that enables testing of differ-ent schedulers for Multipath TCP. Using this framework, theyevaluated different schedulers for MPTCP and provided anin-depth performance analysis. They identified the impact ofscheduling decisions on the performance of MPTCP and illus-trated the underlying root cause for the observed behavior.For example, they discovered that a bad scheduling deci-sion triggers two packet reordering effects. First, EDPF basedscheduler is more efficient than simple RR in terms of avoid-ing the HLB problem. Second, receive-window limitation mayprevent the subflows from being fully utilized.

In the last few years, several articles visited how to usedelay-aware scheduling algorithms in MPTCP to improve thereceive buffer utilization. Yang and Amer [202] used one-way communication delay of a TCP connection to design anMPTCP scheduler that transmits data out-of-order over multi-ple paths such that their arrival is in-order. Le and Bui [110]dealt with the packet reordering problem of MPTCP usinga Forward-Delay-based packet scheduling algorithm. Its main

idea is that the sender distributes packets via multiple pathsaccording to their estimated forward delay and throughput dif-ference. This scheduling algorithm is an advanced version ofFPS because it took throughput difference into consideration.Due to the latency asymmetry on the Internet, obtaining agood one-way delay is difficult. Therefore, Coudron et al. [35]proposed relying to the one-way delay (OWD) difference(�OWD) of multiple subflows to improve the scheduling.Their estimator functions more like OWTT than FPS because itestimates not only forward delay difference but also backwarddelay difference.

The MPTCP performance is not only impacted by the receivebuffer but also by the send buffer. Yang and Amer [201] foundthat in an MPTCP connection with several high-BDP subflows,send buffer blocking can occur and seriously decrease theoverall throughput. They introduced Non-Renegable SelectiveAcknowledgments (NR-SACKs) to MPTCP. The idea is thatonce a data packet has been sacked, it can’t be removedfrom the receive buffer. Thus, the sender can free the sackeddata sooner than the advance of the MPTCP level cumulativeacknowledgement. Arzani et al. [14] found that the send buffersize has significant impact on the performance of MPTCP. Forexample, MPTCP provides higher performance gains with alarger send buffer. However, they did not propose any methodto address the problem.

d) Pareto-efficient MPTCP: Khalili et al. [101], [102]demonstrated that MPTCP is not Pareto-optimal because theyfound that MPTCP users can be excessively aggressive towardTCP users over congested paths even without any benefit tothe MPTCP users. They attributed the problem to the LIAof MPTCP. To deal with the problem, they proposed anOpportunistic Linked Increases Algorithm (OLIA) as an alter-native for LIA and proved that OLIA is Pareto-optimal andsatisfies the three design goals of MPTCP. Like LIA, OLIAis a window-based congestion-control algorithm that couplesthe increase of congestion windows and uses unmodified TCPbehavior in the case of loss. The increase part of OLIA has twoterms. The first term provides the Pareto optimality. The sec-ond term guarantees non-flappiness6 as MPTCP with LIA andresponsiveness (i.e., the rate of algorithm convergence). OLIAalso compensates for different RTTs by adapting the windowincreases as a function of RTTs. However, Singh et al. [182]found that OLIA of MPTCP still has performance issues. Theypresented Adapted Opportunistic Linked Increases Algorithm(AOLIA) to ensure controlled aggressiveness of the MPTCPsubflows. In order to minimize the packet reordering delay,they also proposed a Push-Pull-Hybrid (PSPLH) schedulerwhere Pull strategy is used to allocate data segments to mul-tiple flows, and Push strategy is used to tune the size of thesegments dynamically.

e) Path diversity for MPTCP: By default, MPTCP uses a“fullmesh” manager to create static full-mesh of possible net-work paths among the available IP addresses (see Figure 9).This path management may not only lead to a large numberof subflows being established but also ignore the benefits the

6Flappiness means that MPTCP would use one path almost exclusively fora while, then flip to another path, and then repeat.


Fig. 9. Static full-mesh of possible network paths between two MPTCPenabled hosts: A1:B1, A1:B2, A2:B1, A2:B2, A3:B1, A3:B2.

path diversity could offer. van der Pol et al. [194] combinedMPTCP and OpenFlow to dynamically exploit path diversity(choose disjoint paths) between two endpoints to improve sta-bility (in the case of partial path failure) and obtain higherthroughput. Specifically, in [194] OpenFlow is used to dis-cover the topology of the network, calculate multiple disjointpaths and configure these paths. MPTCP is used to distributethe traffic across the selected paths.

In contrast to limiting the number of subflows [194],Coudron et al. [34], [36] exploited the path diversity in cloudnetworks from a different perspective, for example, creatingmore subflows over disjoint WAN paths (under the assump-tion that the WAN capacity is the throughput bottleneck).To achieve this goal, Coudron et al. [36] proposed a cross-layer approach which combines MPTCP and LISP. LISP isused to gather information and give hints to MPTCP aboutthe WAN paths diversity between two MPTCP endpoints.MPTCP could use the path manager “ndiffports” to createadditional LISP-enabled subflows which share the same sourceIP with different TCP ports. In their another work [34], apartfrom MPTCP and LISP, Coudron et al. [34], [36] introducedone more protocol TRILL to support Ethernet level multipathtransmission. For example, MPTCP employs coordinated con-gestion control to achieve RP, LISP handles path diversity atthe external data center packets between border nodes, andTRILL deals with multipath Ethernet-layer communications.

f) Comparison between CMT-SCTP and MPTCP: SCTPand TCP are the main base protocols widely used to supportmultipath transportation. Currently, CMT-SCTP and MPTCPare in the focus of the IETF and academia. In the followingdiscussion, we make a comparison of them.

Although CMT enabled SCTP shares the same issues withMPTCP in terms of fairness, reordering, and retransmis-sion policies, moving legacy applications from TCP to SCTPinvolves a number of challenges such as making SCTP workthrough NATs, the need to modify applications, and the lackof an easy way to negotiate SCTP versus TCP between aclient and a server. None of the issues are insurmountable,but together they make adoption of SCTP as a TCP alterna-tive a challenge. From the previous discussion, we found thatMPTCP instead of CMT-SCTP has become the main streamof multipath transportation solution. In this section, we discussthe reason behind it.

By comparing the key algorithms used by CMT-SCTP andMPTCP (see Table X and Table XII), we found that the twoprotocols have shared the same or similar key algorithms.

Therefore, algorithm is not the primary factor that determineswhich protocol could become the mainstream because thealgorithms are not specific to any base protocols but couldbe used on any of them with minor adaptation. We believethat the reason mainly comes from their difference in termsof backward compatibility and sequence number design. Forexample, unlike SCTP which modifies the interfaces of legacyTCP to applications, MPTCP presents a single TCP interfaceto the application layer (see Figure 8). This seemingly minordifference makes MPTCP compatible with all legacy appli-cations. As such, the implementation of multipath in TCP,which dominates Internet traffic, is a much more attractivedeployment strategy.

The sequence space design is another primary difference. Tomake a fair comparison, we assume that SCTP has been widelydeployed as TCP has, otherwise CMT-SCTP packets would bedropped by legacy middleboxes. As summarized in Table X,CMT-SCTP keeps using one single sequence space as SCTPdoes. Therefore, a new SACK mechanism and its interpretationaccordingly are required to avoid spurious retransmissions.Moreover, single sequence space makes bytestream discon-tinuous on multiple paths so that certain middleboxes maybreak such associations [160]. In contrast, in MPTCP thesequence numbers carried in the TCP headers are separateon each path so that the interpretation of out-of-order packetsand ACKs remain the same as before. Hesmans et al. [78] andHonda et al. [80] found that MPTCP could traverse most ofthe middleboxes because of its double sequence space design.Furthermore, with checksums MPTCP can detect middleboxesinterference and fallback to legacy TCP. To allow clients tobenefit from MPTCP in its early deployment (e.g., servers havenot upgraded to support MPTCP), Detal et al. [41] proposed aprotocol converter, MIMBox, to translate MPTCP to TCP andvice versa. MPTCP is not only little influenced by middleboxesbut is even extended to explicitly add specified middleboxesin the middle of an ongoing communication. For example,the proposed solution in [136] used MPTCP to implementconnection acrobatics (i.e., the ability to explicitly redirectconnections via a middle point and the ability to migrate theendpoint of a connection). Therefore, MPTCP connections canbe redirected to middleboxes located anywhere in the Internetto improve services like load balancing, DDoS filtering andanycast.

One more difference between CMT-SCTP and MPTCP liesin the path management strategy. By default, MPTCP createsa full-mesh of possible network paths among the available IPaddresses, whereas CMT-SCTP only uses pairs of addressesto set up communication paths (creating only one additionalpath per additional source address). Becke et al. [18] foundthat MPTCP’s path management strategy performs signifi-cantly better than that of CMT-SCTP in the case of asymmetricpaths.

D. Application Layer Multipath Capability

Provisioning multipath capability at the application layerhas received a lot of attention because the approaches arealmost independent of the underlying access technologies and


TABLE XIIIKEY ALGORITHMS FOR APPLICATION LAYER MULTIPATH CAPABILITY (SORTED ACCORDING TO THEIR ORDER MENTIONED IN TABLE XIV)

TABLE XIVCONCURRENT MULTIPATH TRANSFER APPLICATIONS

Fig. 10. Multipath transmission on application layer. It is assumed that eachhost has two interfaces. src: Source IP address, dst: Destination IP address,sp: Source Port, dp: Destination Port, APP: Application.

network-layer routing. It is a common practice that an appli-cation establishes multiple transport connections, binds themto different IP addresses, and distributes the data in propor-tion to the available path capacity over these connections(see Figure 10). To reassemble the data delivered over differ-ent connections, each packet or a group of packets are usuallyassigned additional sequence numbers.

In the rest of this section, we also divide the approachesinto four groups. The first group discusses approaches usingthe same path. The second group discusses approaches usingdifferent paths. The third group investigates approaches basedon HTTP in order to highlight the importance of the combi-nation of HTTP and multipath transmission. The fourth groupexploits middleware approaches. Table XIII and Table XIVare used to summarize the key algorithms and approachesrespectively.

1) Multiple Connections Over the Same Path: In the earlystage of research work on application layer multipath transmis-sion, the focus was on bandwidth aggregation using multipleTCP connections over the same physical path. For instance,Allman et al. [10] developed a new application called XFTP,a modified version of FTP [152], to improve the poor perfor-mance of TCP over long-fat satellite channels. XFTP createsmultiple TCP connections. To send a file, XFTP divides thefile into records, reads the file from local storage one recordat a time, and sends each record over whichever connec-tion has resource available for transmission. To reassemblethe records into the correct order, XFTP uses an additional 4-byte sequence number to each record. GridFTP [92] is anotherextension of the FTP protocol implemented for bulk data trans-fer, where parallel TCP connections are created to increase thethroughput in a bottleneck link. Specifically, GridFTP divides


the data to be transferred into multiple portions and transferseach portion with a separate TCP connection. When competingwith non-GridFTP connections over a bottleneck link, theGridFTP connections will be less likely to be selected todrop their packets. Hacker et al. [72] examined the effectsof using parallel TCP flows to improve end-to-end networkperformance. They found that in the absence of congestion,the use of parallel TCP connections is equivalent to usinga large Maximum Segment Size (MSS) on a single con-nection. In addition, they addressed the question of howto select the maximum number of connections to maximizethe overall throughput while avoiding congestion. For exam-ple, if the selected value is too large, the aggregate flowmay cause network congestion and throughput will not beoptimized.

2) Multiple Connections Over Different Paths: Theapproaches mentioned above aim to increase applicationthroughput by using multiple TCP connections through thesame physical path. Nevertheless, if they are used for strip-ing data over different physical paths, the reordering issueat the receiver would render them inefficient because theydo not take into consideration the reordering issue causedby heterogeneous paths. In the 2000s, researchers started toseek solutions to provide bandwidth aggregation over differ-ent physical paths. A simple approach to achieve this goalis to directly add support for multiple interfaces to a givenapplication by opening multiple TCP sockets (one for eachactive interface), and performing striping of data across dif-ferent sockets. If the interfaces of a client are connected toindependent networks, the simultaneous use of multiple pathscan achieve a total throughput close to the sum of all thethroughput from individual interfaces.

Given that popular files are often replicated on multipleservers, it becomes natural for clients to connect in parallel toseveral mirror servers to retrieve a file (i.e., many-to-one fash-ion). Golubchik et al. [64] investigated the potential benefitsof an application layer multipath streaming approach betweena set of senders and a receiver. They found that multipathstreaming exhibits better loss characteristics than single pathstreaming. Rodriguez and Biersack [165] described a parallel-access (PA) scheme that allows users to fetch different portionsof a file from multiple servers at the same time and reassem-ble the file locally. The PA scheme allows dynamic loadsharing among all servers so that faster servers will deliverbigger portions of a file while slower ones will deliver smallerportions.

Shiwen et al. [123] proposed Multiflow Real-time TransportProtocol (MRTP) which is a multipath transmission exten-sion to Realtime Transport Protocol (RTP) [173] for real-timeapplications. MRTP supports multimedia services by explor-ing multipath transport in mobile ad hoc networks, where linkbandwidth may fluctuate and paths are unreliable. The authorsstudied the impact of traffic partitioning on congestion at bot-tleneck links and found that the bandwidth utilization of abottleneck node could be much improved by two strategies(thinning and striping [22]). Furthermore, they showed ana-lytically that most of the performance improvement can beachieved with a few paths (e.g., two or three paths), while

only marginal improvement is gained by further increase inthe number of paths.

Wang et al. [195], [196] proposed Dynamic MPath-Streaming (DMP), a scheme for live streaming over multipleTCP connections. DMP allocates packets over multiple pathsaccording to their current throughput. DMP does not use anexplicit probing scheme for bandwidth estimation on eachpath. Instead, it uses the WRR-PULL scheduler to allow eachconnection to pull data from a shared queue whenever ithas opportunity to send data. Thus, the paths with higherthroughput deliver more packets than others.

Tullimas et al. [192] proposed MultiTCP for multimediastreaming. It aims at providing resilience against short terminsufficient bandwidth due to traffic bursts by using multi-ple TCP connections for the same application. MultiTCP isa “smart” application that allows the application to controlthe desired sending rate during congestion periods. MultiTCPachieves rate control by the means of adjusting the receiverwindow.

Zhang et al. [205] proposed a general framework ofmultipath transport system based on application-level relay(MPTS-AR), currently under the standardization within theIETF [112]. This framework defines three logical entities andtwo protocols. The entities include user agent, relay server,and relay controller. The protocols are OpenPath and MPTP(Multipath Transport Protocol), which are used in controlplane to manage relay paths and in data plane to facilitate mul-tipath data transport respectively. However, they left a few keyfunctions we concern the most out of the scope. For example,how to split the original data stream into several substreams,how to mitigate the reordering issue at the receiving side, howto provide path diversity among all available paths, and howto obtain the performance of overlay paths are all out of thescope.

3) HTTP Based Multipath Media Streaming: In additionto multipath approaches for specific applications, HTTP [59]with multipath capability is currently one of the most commonprotocols for streaming video on the Internet through multiplepaths. Kaspar et al. [96], [97] and Evensen et al. [53]–[55]presented HTTP-based methods for downloading mul-timedia content simultaneously over multiple networkinterfaces.

Kaspar et al. [97] proposed an HTTP-based on-demandstreaming service over multiple wireless access networks.They presented a proof-of-concept implementation of a pro-gressive download service, which uses HTTP Range RetrievalRequest (HTTP-RRR) capability [59] to download specificsegments of a file from a media server. The drawback ofthe range retrieval request is that each request must be han-dled sequentially by the server before the client can send thenext request. Thus, an average time overhead of one RTTis introduced for each request. In order to avoid waiting foreach response, in [96] they presented an improved version oftheir work by using an additional HTTP capability, i.e., HTTPRequest Pipelining (HTTP-RP) [59]. The request pipeliningfunction of HTTP allows a client to make multiple requestssimultaneously. In [97], they studied the effect of file seg-mentation on the buffer requirements and found that there


exists an optimal segment size for which the aggregation effi-ciency is maximized. In [96], they found that due to the use ofrequest pipelining, very small segments can provide efficientthroughput aggregation.

Evensen et al. [55] introduced an adaptive, WRR-PULLbased scheduler that achieves smooth playback by schedul-ing requests for video segments of different quality lev-els over multiple interfaces simultaneously. Like their pre-vious work in [97], they still utilized HTTP-RRR andHTTP-RP functions to support multipath transmission. Inorder to avoid video deadline misses, they proposed an addi-tional request scheduler in [55]. The scheduler is mainlyused for estimation of the aggregated throughput for cho-sen video quality level and request of segments over theavailable interfaces. However, the weakness of the requestscheduler is that the segments are divided into fixed-sizedsubsegments, which may lead to low performance with con-strained receive buffer. Evensen et al. [53], [54] proposedimproving the request scheduler by loosing the segmentsize constraint. For example, the segment sizes are dynam-ically calculated on the fly based on the capability ofeach path.

4) Session Layer Multipath Capability: Unlike applicationlayer approaches discussed previously, some approaches openmultiple TCP flows without any change to existing applica-tions by providing specialized middleware or virtual socketsat the session layer between the application and transport layer.

Sivakumar et al. [184] proposed PSockets (Parallel Sockets).PSockets is a library that transparently partitions upper layerdata into multiple transport streams through the same phys-ical path. The principal idea is to split data equally acrossseveral open sockets without application upgrade. PSocketshas the same Application Programming Interfaces (APIs) asthose of a regular socket. Note that this work was publishedin the year 2000. Back then, the TCP window size had to betuned manually for high-speed networks at both the source andthe destination in order to achieve the maximum throughput.Differently, PSockets allowed applications to achieve the bestperformance without tuning the window size.

Hasegawa et al. [74], [75] proposed an Arrival-Time match-ing Load-Balancing (ATLB) layer between the applicationlayer and TCP layer. ATLB consists of a distributed data trans-fer method and a path-failure detection/recovery method. Inorder to mitigate the reordering effect at the receiver, ATLBcalculates the data arrival time for each path. It considers thetime that data segments spend in the TCP queue at a senderand the time needed for data segments to pass through thenetwork.

Qureshi et al. [155] presented a prototype system Tavarua.Tavarua is a middleware for providing network striping capa-bility to applications with high demands on uplink throughput.Note that Tavarua runs upon UDP. In an effort to mitigatethe impact of variations in bandwidth, an application can usefeedback information to estimate the maximum available datatransmission rate. Then the bit-rate at which the video isencoded is adapted dynamically. The middleware also han-dles low-level issues related to the network interfaces (e.g.,congestion control, disconnections, and reconnections).

Sakakibara et al. [171] proposed a Socket-level BandwidthAggregation Mechanism (SBAM) to offer aggregated band-width. SBAM is located at the socket layer (close to TCP)so that it can collect system resources efficiently. For exam-ple, it has a network monitoring function to collect delayand available bandwidth of each path. Using this informa-tion, the traffic scheduler decides the amount of data to fillthe bandwidth-delay product of each path.

Like the other middleware approaches, Parallel TCPTransfers Helper (PATTHEL) [16] also provides APIs forapplications. The difference of PATTHEL lies in two facts.First, PATTHEL incorporates a separate data connection andcontrol connection, where the control connection is createdfirst to manage the other data connections for the entire com-munication period. Second, PATTHEL is a cross-layer protocolbecause it adds an entrance to the routing table in order todeliver data over a certain channel.

Miyazaki and Oguchi [130] examined how much receivebuffer is needed in various scenarios and found that the buffersize is proportional to the ratio of the bandwidth of the twointerfaces. A larger bandwidth difference leads to a biggerreceive buffer and vice versa. The scheduling algorithm usedin [130] is EDPF.

Habak et al. [70], [71] proposed a Deplorable BandwidthAggregation System (DBAS) middleware architecturefor multi-interface enabled devices. Like the workin [68], [111], and [166], DBAS also supports both FOSMand POSM. In FOSM where the server is not DBAS enabled,DBAS schedules different connections to the interfaces suchthat a connection can be assigned to only one of the availableinterfaces. If both sides are DBAS enabled, POSM is usedso that each packet can be scheduled independently on a dif-ferent interface. To make better scheduling decisions, DBASestimates the characteristics of the applications dynamicallybased on their behavior and stores them in a database forhistory track. DBAS focuses on the actual implementationof the basic core system. The authors presented an extendedwork based on DBAS, a Green DBAS (G-DBAS) [69] tobalance overall throughput with energy consumption. Forexample, they introduced a new utility based scheduler thattakes energy consumption of each interface into account inorder to balance the trade-off between maximizing throughputand minimizing power consumption. Note that G-DBASonly works in FOSM. OPERETTA [66] is an extension ofG-DBAS to support POSM.

Application layer approaches split a single file or bytestream into segments that are transmitted concurrently overdifferent paths. These kind of approaches seem to be simplein the sense that the applications are in full control of the strip-ing decisions. Thus, it does not require any protocol changeat lower layers so that clients and servers can find an optimalway to collaborate. Nevertheless, the complexity and over-head at the application layer are considerable. For example, anapplication-level sequence number has to be included in eachof the application defined headers. Meanwhile, the applicationhas to explicitly ensure that the application layer data units,which carry unique application-level sequence numbers, donot get fragmented during transmission. Moreover, a dedicated


TABLE XVEVALUATION OF END-TO-END PACKET REORDERING ALGORITHMS

resequencing mechanism is required to reassemble the dataat the receiver. In practice, different paths may have diversecharacteristics, and the striping ratio may not exactly matchthe ratio of data rate from different paths. A large receivebuffer (on the application level) is required to accommodatethe out-of-order data. Finally, in order to split intelligently, theapplication has to implement a bandwidth estimation mech-anism redundantly despite the same mechanism has beenemployed by TCP through its congestion control mechanism.

The middleware approaches are very similar to application-layer approaches. They also face the same challenges, forexample, the reordering issue. The advantage of middlewareapproaches is that although it still requires client and server-side modifications, applications usually are not required to beupgraded.

E. Summary

In this section, we summarize the issues that are common tothe approaches from all layers we have covered in this survey.Specifically, we first discuss the packet reordering problemand how effective the widely used scheduling algorithms areto mitigate it. After that, we present the approaches whichhave adopted the cross-layer design. Next, we compare theapproaches’ compatibility capability, which is inherently deter-mined by their stack positions. At last, we summarize theresearch problem evolution on each layer.

1) Packet Reordering: When packets travel through differ-ent paths which may have mismatched characteristics, theymay arrive at the destination out of order. All the presentedapproaches deal, to some extent, with the reordering issue onthe layer which they are located at. If they ignore or haveno control over the reordering mechanism on the transportlayer, their approach may suffer from performance degrada-tion because of the misinterpretation of out-of-order packets.Table XV lists the primary algorithms used to mitigate packetreordering without considering which layers they are locatedat. We group and sort them according to their effectiveness interms of packet reordering and load-sharing capability. We usefour levels (++, +, o, -) to grade the mechanisms only for gen-eral and relative evaluation. The level ++ indicates the mostefficiency. + comes second, and then o and -. A scheme graded- does not imply that it is useless. Instead, that scheme maywork well either in certain network context or with additionalbuffer management.

The first group includes PCA [2], [88], [135], [188],PFA [51], [57], [68]–[71], [95], [111], [127], [150], [166],and Multi-streaming [50]. They are the most effective mech-anisms which can completely eliminate packet reorderingincurred by multipath transmission because the data units

required sequencing at the destination are assigned only tothe same path. However, a multipath transmission protocolwith them usually performs worse than without them in termsof load sharing. For example, if the number of flows isless than that of available paths, there would exist pathswhich cannot be fully utilized. Therefore, we give - to theirload-sharing capability due to their lower performance thanaverage.

FPS [37], [38], [74], [75], [110], [114], [115], [129],[202], [206] breaks the in-order scheduling rule at the source.Whenever a path has opportunity to send a new packet, itestimates that path’s capacity and chooses a new data blockaccordingly so that out-of-order sent out packets could arriveat the receiver in-order. We grade FPS a + to its packet reorder-ing capacity because it concerns most on whether the packetsarrive at the receiver, and a ++ to its load-sharing capacitybecause fully utilizes each path’s capacity.

In EDPF [24], [81], [103], [106], [130], [178], [179] andPET [27], the scheduler first sends data on a path with lowestRTT until it has filled its congestion window. Then, the data issent on the subflow with the next lowest RTT. We grade themo and + to their reordering and load-sharing capacities respec-tively because EDPF and PET preferentially schedule data onthe available path with lowest RTT. The larger the RTT differ-ence is, the more out-of-order packets arrive at the destination.Moreover, EDPF and PET consider only path bandwidthand latency, ignoring packet loss rate caused by congestion.Therefore, it may achieve sub-optimal load-balancing in thepresence of high losses or heavy congestion.

WRR [5], [6], [10], [13], [16], [17], [46], [52]–[56], [66],[68]–[71], [74]–[76], [82]–[85], [96], [97], [101], [102], [118],[121], [148], [149], [155], [161], [162], [165]–[168], [171],[172], [185], [186], [195], [196], [198], [204] is the mostwidely used scheduling algorithm on all layers with maxi-mizing the overall throughput as its first priority. It couldachieve the goal if the multiple paths have similar charac-teristics. Otherwise, it would cause significant out-of-orderpackets at the receiver because it considers little about itsimpact on packet reordering. Therefore, WRR works betteron link layer than other layers because link layer has rela-tively stable link state. Like FPS, WRR could fully utilize theavailable path capacity. Based on the discussion, we give o toWRR’s packet reordering capacity and ++ to its load-sharingcapability.

In practical network environments where path characteris-tics may change dynamically, the scheduling algorithms exceptPFA, PCA and Multi-streaming may fail to counter againstpacket reordering. Several additional mechanisms have beenproposed to work coherently with the scheduling algorithms,such as ACK manipulation [46], [68], [89], [104], [111], [116],[122], [167], buffering management [8], [26], [49], [68], [109],packet coding [38], [114]–[116], [178], blocking warning andfast retrieving [81]–[84], [106], [115], [160], slow-path penal-ization [58], [140], [160] and so on. The comparison of variouscombinations of the scheduling algorithms is out of the scopeof our knowledge because they may be used in various networkenvironments, triggered by different conditions, and supportedby diverse assumptions.


TABLE XVICROSS-LAYER SUPPORT FOR MULTIPATH TRANSMISSION

To make a conclude on the discussion of packet reorder-ing, the choice of the best scheduling policy dependson path characteristics. There is no single schedulingmechanism which can handle all scenarios. To adapt todifferent network environments, several approaches, suchas [68], [70], [71], [111], and [166], could support both flowlevel scheduling and packet level scheduling strategies.

2) Layer-Dependent Scheduling Algorithms: In ourprevious discussion, we could find that many scheduling algo-rithms are shared by different approaches on various layers.For example, FPS variants are used on transport (includ-ing [37], [38], [110], [114], [115], [129], [202], and [206])and application layers (including [74] and [75]). EDPFvariants are used on network layer (including [27], [65],[104], [105], and [109]), transport layer (including [24], [81],[103], [106], [178], and [179]) and application layer (includ-ing [130]). WRR variants are used from link to applicationlayers. However, we argue that it does not imply that thesescheduling algorithms are layer-independent. Instead, they aremostly layer-dependent.

Running a scheduling algorithm usually requires measuringbandwidth, delay, loss, or jitter to provide best effort or evenQoS guarantees. No single measurement on a certain layercould always give accurate measurements. The measurementsmay even vary on different layers. Although how to measurethose metrics is an entire problem unto its own, the effi-ciency of the algorithm is closely connected to the correctness

of the measurements. Therefore, we argue that the schedul-ing algorithms which rely on the estimation of certain pathcharacteristics are layer-dependent.

We discuss WRR as an example to support our argument.The same principle is applicable to other algorithms whichrely on the estimation of path characteristics. WRR is the mostwidely used scheduling algorithm. Although each layer adoptsmany variants of the WRR algorithm, its effectiveness highlydepends on how correctly the path capability is estimated in adynamic fashion. For example, using the congestion windowsize on the transport layer to estimate the path capability is amore lightweight and accurate way than those using variousprobing methods on other layers. Furthermore, WRR worksmuch better on link layer than upper layers because link layerhas more stable link status.

3) Cross-Layer Support: In this survey, we define that anyattempt to violate the TCP/IP reference model is considereda cross-layer design. Among the approaches we have dis-cussed previously, several of them have explicitly involvedcross-layer interaction for purposes of estimating path statusto avoid packet delays and losses, scheduling traffic over mul-tiple paths according to their capacity, exploring path diversityto obtain high throughput, achieving better QoS for multimediaapplications, and so on. Due to these benefits the cross-layerdesign may offer, there has been increased interest in proto-cols with interactions between various layers of the networkstack. In the rest of this section, we discuss the cross-layer


TABLE XVIICOMPATIBILITY EVALUATION (

√MEANS BEING SUPPORTED)

approaches based on our previous discussion and give a fewobservations on them without our own judgment. Instead, werefer readers to Kawadia and Kumar’s [98] work which callsfor a cautionary approach to cross-layer design. Although [98]examined the issue of cross-layer design in wireless networks,we believe the same principle is also applicable for multipathtransmission.

Table XVI shows approaches based on a cross-layer design.The “Base Layer” indicates the layer where the data splittingis initiated, and the “Additional Layers” indicate which lay-ers are required to provide support to the base layer. Fromthe table, we observe that the transport and application lay-ers are the main base layers. From 2005 to 2013, it wasdrawn most of the attention to implement the base layeron the application level. Some applications obtain low layerinformation to optimize their behavior in terms of interfaceselection, load balancing, and energy efficiency. The infor-mation includes throughput history and smoothed RTT fromtransport level, routing table from IP level, and link status(e.g., energy consumption, available bandwidth, and bit errorrate). Some applications can even change the low layer proto-col behaviors such as changing TCP window size to controlthe throughput, modifying the routing table to optimize pathselection, and disconnecting/reconnecting certain interfaces forenergy efficiency or partial failure. In most recent years, ithas become more attractive to use transport layer as the baselayer with additional support from network and link layers.Although transport layer approaches have advantages fromthe congestion control mechanism, they lack the choice ofpath diversity to free the constraint of fairness control. Thepath optimization support from network and link layers cancompensate for the weakness in network and Ethernet levelsrespectively. In addition, some transport layers can suspend orrelease a path based on the estimated MAC information.

We can also find that the interaction of cross-layer designmay not be limited to adjacent protocol layers. Instead, itallows vertical communication to take place between nonad-jacent layers. The cross-layer design approaches are actuallynot limited to the ones listed in Table XVI because althoughmany approaches above the network layer did not mention orspecify how packets are delivered through multiple interfaces,the cross-layer support from routing function on network layermay be assumed implicitly.

4) Compatibility: In this section, we evaluate the com-patibility capacity of the approaches on different layers.The evaluation is made in general because there may beexceptions in certain approaches. Table XVII presents theevaluation where we separate network layer approaches into

three categories according to how many proxies required ineach category. We use MPTCP and CMT-SCTP to representthe transport layer approaches and evaluate their compatibilityseparately. Likewise, we also separate session layer approachesfrom application layer approaches.

Application compatibility means that the lower layerchanges do not require the legacy applications to be upgraded.Obviously, the link layer and all the network layer approachesare compatible with the legacy applications because they donot change the socket interface between the legacy applicationsand transport layer protocols. MPTCP presents a standard TCPsocket API to the application so that legacy applications canrun upon MPTCP transparently. However, the legacy appli-cations running on TCP have to upgrade in order to takeadvantage of CMT-SCTP. Session layer approaches usuallykeep the same socket API to applications (e.g., PSockets [184])so that they require no changes to the legacy applications.Application layer approaches have a serious application com-patibility issue because the multipath transmission propertyneeds to be implemented for specific applications.

An updated approach has backward compatibility if it caneither work with its communicating peer which uses thestandard approach or automatically fallback to the standardapproach if the communicating peer does not support thenew features. Link layer approaches generally do not main-tain the backward compatibility because most of them areproprietary approaches and require dedicated setup on bothsides. Therefore, we mark that the link layer approaches arenot backwards compatible. Network layer approaches with noproxy usually employ some of the fields in protocol headers(e.g., [65], [68], [122], and [148]–[150]) to negotiate multipleIP addresses to be used or piggy-back information so as to pro-vide backward compatibility. Network layer approaches withone proxy has backward compatibility because the communi-cating peer is unaware of the multipath transmission betweenthe client and the proxy. The network layer approaches requir-ing two proxies are also backwards compatible because bothsides are unaware of the multipath transmission between thetwo proxies. MPTCP is backwards compatible with plain TCPbecause it can fallback to single-path TCP if the commu-nicating host does not support the extensions. CMT-SCTPrelated articles did mention at all whether it can fallback toSCTP if the server is not CMT-SCTP enabled. We believeCMT-SCTP could also fallback to SCTP if the server doesnot support concurrent multipath transfer. But anyway, CMT-SCTP has inherited the backward compatibility issue of SCTPitself. For example, if the server is only aware of TCP opera-tions, a CMP-SCTP client may fail to create connection with


TABLE XVIIIRESEARCH PROBLEMS ON EACH LAYER (

√CMEANS HAVING TRIED TO ADDRESS)

the server. Session layer and application layer approaches aregenerally backwards compatible with the legacy applications.When connecting to a server, a client application usually spec-ifies two different TCP ports, a probe one and a fallback one.First, the client tries to establish a connection using the probeport to check whether the server is upgraded. If the operationfails, another attempt is made by using the fallback port tocreate a standard connection.

The compatibility with existing middleboxes, such as fire-walls and NATs, affects whether the “new” packets are ableto traverse the legacy middleboxes. First of all, link layerapproaches generally would not have this issue because thecommunication paths or trees are well designed between dedi-cated multipath-aware devices. Most network layer approaches(with and without proxies) split bytestream over multiplepaths. Therefore, the single sequence space across more thanone path leaves gaps in the sequence space seen on any indi-vidual path, which may upset certain middleboxes. To solvethis issue, the double sequence space design was proved tobe an effective solution [80], [160]. However, how to use thedouble space also influenced the outcome. For example, HIPbased multipath transmission adopts double sequence space.But the additional sequence space is used for the purpose ofanti-replay instead of resequencing. In contrast, MPTCP pack-ets, which carry double sequence numbers for resequencing ontwo levels, can traverse most of the middleboxes. CMT-SCTPpackets may fail to traverse through certain middleboxes dueto its single sequence space design. Session and applicationlayer approaches create multiple standalone TCP flows so thattheir TCP flows can travel through various middleboxes asnormal TCP does.

Host compatibility means whether the approaches requirechanges in hosts. We found that all approaches need changeson hosts except the network layer approaches with two proxiesbecause the multipath transmission between the two proxiesare unknown to both communicating peers.

Infrastructure compatibility means whether an approachneeds additional network infrastructure such as NAT box andproxy. We found that only the network layer approaches withthe proxy support require additional infrastructure.

According to the previous discussion, it is hard to imple-ment a generic multipath solution which can satisfy all thecompatibility requirements. As summarized in Table XVII,we have a few more observations. For example, MPTCPand session layer approaches have more compatibility sup-port than others. CMT-SCTP has inherited SCTP’s applicationcompatibility issue, which becomes the major obstacle of its

real deployment. Network layer approaches lack discussion onbackward compatibility.

5) Evolution of Research Problems: Table XVIII presentsthe research problems the approaches on each layer havetried to address. In the early stage of multipath transmissionresearch, most approaches emphasized only bandwidth aggre-gation with various scheduling and packet reordering algo-rithms. Few of them considered the fairness and RP features.Today, these two problems as well as Pareto-optimality prob-lem have become challenges along with the revolutionarydevelopment of multipath transmission.

The fairness requirement on multipath transmission wasunclear in the beginning. For example, the early research onmultipath transmission focused on bandwidth aggregation bytaking advantage of the resources through multiple interfaces.The target in the research community matched the potentialexpectation of end users because an end user can benefitfrom the aggregated bandwidth if they have paid for bothaccesses. Thus, the fairness emphasized the fairness of eachindividual subflow; for example, each subflow gets as muchbandwidth as a standalone TCP flow does. In recent years,the research focus was on the fairness of the multiple sub-flows as a whole at shared bottlenecks. The principle is that amultipath transmission should behave as a single TCP flow atshared bottlenecks. Coordinated congestion control algorithmsare used as a powerful tool to achieve it.

The concept of RP [200] was proposed in 2008. The earlyapproaches before it also proposed using less congested pathsmore, which is one of the RP principles. From the conges-tion control algorithm viewpoint, the difference between theseapproaches and the approaches after 2008 is that the latterones use coordinated congestion control algorithms instead ofindependently tuning each path’s congestion control behavior.

The Pareto-optimality is a state of resource allocation inwhich there is no alternative state that would make somepeople better off without making anyone worse off. MPTCPwith LIA [17], [60], [158], [162] fails to satisfy the Pareto-optimality because upgrading some regular TCP users toMPTCP can reduce the throughput of other users without anybenefit to the upgraded users. OLIA [101], [102] as an alter-native for LIA could make MPTCP Pareto-optimal and satisfythe three design goals of MPTCP.

In Table XVIII, we observe that the multipath transmissionfollows an evolutionary way mostly on the transport layer.We believe this is determined by the stack position at whichthe proposed approaches are located. For example, the fair-ness, RP, and Pareto-optimality features are achieved by the


means of congestion control, which as default is managed onthe transport layer. The link layer and application layer can-not intervene congestion control without breaking the protocolstack layered structure. In our literature review, mHIP [151] isthe only protocol which provides fairness feature on networklayer. However, mHIP is actually located on a middle layerbetween network and transport layers. Therefore, because ofits closeness to transport layer, mHIP is able to manipulatethe congestion control operations. The link layer and networklayer can provide path diversity for cross-layer approaches.For example, OpenFlow and TRILL can provide Ethernet levelpath diversity [34], [194], and LISP can provide routing levelpath diversity [34], [36].

IV. LESSONS LEARNED

From the evolution of end-to-end multipath transmissionviewpoint, we observe that multipath has become increasinglypopular at the transport layer with features such as load balanc-ing, fairness control, congestion control and Pareto-optimality.As a major extension to TCP which has not changed verymuch in the last decades, MPTCP has attracted more and moreattention in recent years. We refer to Figure 2, Table XVII andTable XVIII to summarize the lessons of its development wehave learned from this survey. We believe that they are valu-able lessons that others should learn in order to make theirproposals into practice.

As shown in Figure 2, MPTCP was proposed at an oppor-tune time to draw the experience gathered in previous workand correcting the past mistakes. For example, MPTCP wasdesigned with the backward compatibility to legacy applica-tions and middleboxes (see Table XVII), which makes MPTCPinstead of CMT-SCTP a big step towards being widely accept-able. In addition to solving the compatibility issue, MPTCPgoes further in addressing issues in fairness and RP by jointincrease and decrease rules (e.g., the coupled congestion con-trol algorithm LIA [162]). Furthermore, OLIA [102] andAOLIA [182] even make MPTCP meet the requirements ofPareto-efficiency (see Table XVIII). Apart from the technicalaspects, we believe the following factors also contribute tothe success of MPTCP: kernel implementation in Linux, sup-port from Internet standards organization (e.g., IETF), activeand public academic community,7 and incentives from indus-try [1], [4], [20]. According to the previous discussion, weuse the following key words to summarize the key effortswhich has driven the development of MPTCP: collecting previ-ous experience, correcting past mistakes, continuously findingand solving new challenges, implementation, standardization,research community and industrial incentives.

Although MPTCP is a promising protocol for multipathtransmission, it still has room for improvement. We give a fewrecommendations as follows. First, MPTCP needs more “mar-keting” work to get more support from industry. Currently,only a few companies (e.g., [1], [20], and [138]) make prod-ucts or services on MPTCP. Although Apple implementedMPTCP (in a backup mode) in iOS 7, later iOS upgrades

7MPTCP has an active mail list ([email protected]) forsharing experience.

did not support it any more. Secondly, although MPTCP hasbeen implemented in Linux kernel, people have to manuallyinstall it (e.g., by the means of apt-repository or compilingand installing from source) before trying it. It would becomemuch convenient if some mainstream Linux distributions canadd the MPTCP kernel directly in their releases. Thirdly, muchwork has been done to improve the performance of MPTCPin terms of packet reordering and Pareto-efficiency. But theyare still open questions and need more technical improvement.

V. OPEN QUESTIONS

We have identified a few open questions and present themas follows.

• Multipath transmission in Information-CentricNetworking (ICN): in contrast with the host-centricparadigm based on perpetual connectivity and theend-to-end principle, ICN was proposed to make thenetwork content centric allowing nodes to requestcontent that is then delivered by the network to them. Inthis new networking paradigm, information retrieval ispull-based, driven by user requests, point-to-multipointand intrinsically coupled with in-network caching. InICN, a content item can be replicated in more than onenode. There is an increased interest in adapting multipathtransport control to ICN in [23], [42], and [170] sothat the delivery of the content can follow more thanone path to reach the user. The benefits of multipathin ICN include increased resilience and decreased loadto content repositories. The combination of ICN andmultipath transport brings new challenges in terms ofbalancing the performance maximization and networkcost minimization. A multipath solution for ICN needs totake into account that the content sources are unknownin advance and may vary over time, and that in-networkcaching may impact the variability of path length andthe associated delivery time.Currently, there are not many existing multipath trans-mission approaches suitable for ICN. A strategy layer hasbeen proposed for Content Centric Networks (CCN) thatmake decisions pertaining to the multipath selection pro-cess. Naive multipath strategies have been reported tonegatively impact CCN efficiency [170]. An analyti-cal model has been proposed for evaluating differentICN multipath forwarding strategies. According to themodel, a good forwarding strategy that maximizes thereceive-rate should control the pending interests injectedin the different paths so as to fill the capacity of theirpipelines [42]. Joint multipath congestion control andrequest forwarding have been investigated in [23] withthe twofold objective of maximizing user throughput andminimizing overall network cost. Relevant research top-ics for multipath ICN include forwarding strategies forwireless and dynamic networks, joint design of forward-ing strategies and cache replacement policies, and routingprotocol support for multipath.

• Context-aware scheduling: we refer Table XV to sup-port our discussion on this open question. As shown in


the table, no individual algorithm wins from both thepacket reordering and load sharing capability. This resultimplies that there is no single algorithm that is one-size-fits-all. The efficiency of an algorithm depends on howmuch it fits to the network environment. For example,although an algorithm may drive better results than othersin certain network environment, it cannot beat others allthe time in a dynamically changing network. Each algo-rithm has its intended network environment in which towork the best. Currently, most work has been using onesingle scheduling algorithm all the time, which woulddefinitely lead to low performance in certain networkconditions. Therefore, context-aware scheduling policyis required to dynamically detect the network contextand switch to the best performing scheduling algorithmaccordingly. The concept of context-aware schedulingis not new. Some initial work has been conducted.For example, WiMP-SCTP [85] has two data transmis-sion scheduling algorithms used for different networkconditions. When the network condition is good, thedata-striping algorithm is selected to aggregate band-width. When the network condition becomes bad, thedata-duplicating algorithm is switched on to increase des-tination reachability. In [166] and [68], there are twoscheduling algorithms which are used for legacy des-tination and updated destination respectively. For moresimilar work, we refer readers to [70], [71], and [111].

• Richer API of transport services: if the application wereto explicitly control the congestion control algorithms bythe means of APIs provided by transport protocols, thenit would not only know everything the transport layerknows but also what the application knows (e.g., theworkload and application content type), which helps inmaking optimal decisions. This is one of the goals theTAPS (Transport Services) work group (WG) in IETFplans to achieve. For example, the TAPS WG will identifythe services provided by existing IETF transport protocolsand congestion control mechanisms as well as networkrequirements of APIs. The application layer approachescould then use the standard APIs to control mechanismunderneath.

• Multicast meets multipath: multicast traffic over theInternet is growing steadily with the increasing numberof demanding applications. Many of them require certainQoS guarantees, and demand that the network resourcebe utilized in an efficient way. To achieve these goals,the multipath transmission could be used to effectivelysplit multicast traffic over multiple paths at the edge ofthe multicast tree. For example, the future of mobile con-tent delivery may use multicast networks for audio/videostreaming applications. The last hop of the multicast net-works would be based on various wireless technologies(e.g., WiFi, 3G, and LTE). Due to the fact that an indi-vidual wireless path may be unreliable and be unable toprovide required bandwidth, multipath transmission forthe last hop would improve its resilience and throughput.Generally, the challenges include packet reordering andin-network caching issues. We believe that packet coding

schemes [31], [63], [107], [180] can be potential solutionsto them.

• Heterogeneity of multipath: there are many differentnetwork environments for multipath transmission. Forexample, in modern data centers, there are usually morethan one path available between any pair of endpoints.The same applies to access networks, the core ofthe Internet, and Internet Service Providers (ISPs).Nevertheless, these domains are usually autonomous andisolated from each other. There are specialized networkentities, e.g., border gateway, to guarantee the autonomyof each domain. The downside of this network topologyis that even though there might be abundant multipathresources available locally within each domain, the glob-ally available multi-path resources are limited to theborder gateways. Thus, how to break the border of eachautonomous domain to enrich the multipath resources sig-nificantly is not only a technical problem, but also amanagement problem. It requires efforts from multipledomains, including ISPs, policymakers and end users.

• Specialized use of multipath transmission: most of theexisting approaches aim to obtain higher throughput fromthe use of multipath transmission. However, from a user’sperspective, boosting the throughput of a multi-homedmobile device may not be the first-priority goal all thetime. Instead, some users may be willing to use cheapsubscriptions in order to save a few dollars. Others wouldrather use the low energy-consumption interface(s) tosave the battery life. Therefore, multipath transmissionstrategies which take price and energy into considerationshould be provided to users so that they can choose fromdifferent transmission strategies to satisfy their demands.

VI. CONCLUSION

Conventional TCP/IP always uses a single “best” pathaccording to certain routing metrics, even if there may bemore than one path between two endpoints. This behaviorresults in under-utilization of the available network resources.The proliferation of mobile devices equipped with multipleinterfaces, represented by smart phones, brings with it a grow-ing number of multi-homed hosts onto the Internet. Thus, thisdeteriorates the mismatch between single-path transport andthe multitude of available network paths. Multipath transmis-sion comes into the picture as a natural solution with severalsalient features, such as reliability, fairness, RP and Pareto-optimality. In this article, we make a comprehensive surveyabout the state-of-the-art multipath transmission approaches,intending to provide researchers and practitioners with insight-ful observations. We hope this survey will inspire a series ofnew research work in this field.

APPENDIX

LIST OF ACRONYMS

AOLIA Adapted Opportunistic Linked IncreasesAlgorithm

ATLB Arrival-Time matching Load-Balancing


BERP Bandwidth Estimation Based ResourcePooling

BMP Buffer Management PolicyCMT Concurrent Multipath TransferDAC Delayed ACK for CMTDBAS Deplorable Bandwidth Aggregation SystemECMP Equal Cost MultipathECT Equal Cost TreeEDPF Earliest Delivery Path FirstFOSM Flow-Oriented Scheduling ModeFPS Forward Prediction SchedulingHIP Host Identity ProtocolHLB Head-of-Line BlockingHTTP-RP HTTP Request PipeliningHTTP-RRR HTTP Range Retrieval RequestLACP Link Aggregation Control ProtocolLIA Linked Increase AlgorithmLISP Locator/Identifier Separation ProtocolMPTCP Multipath TCPNAT Network Address TranslationOLIA Opportunistic Linked Increases AlgorithmPCA Per-Conversation AllocationPET Packet-Pair based EDPF for TCP applicationsPFA Per-Flow AllocationPOSM Packet-Oriented Scheduling ModeRP Resource PoolingRR Round RobinRTT Round-Trip TimeSACK Selective AcknowledgmentSCTP Stream Control Transmission ProtocolSPB Shortest Path BridgingTRILL Transparent Interconnection of a Lot of LinksWRR Weighted Round Robin.

REFERENCES

[1] Hybrid Internet Access Bonding. [Online]. Available:http://www.tessares.net/solutions/dsl-lte

[2] IEEE Standard for Local and Metropolitan Area Networks—LinkAggregation, IEEE Standard 802.1AX-2008, pp. 1–163, 2008.

[3] IEEE Standard for Local and Metropolitan Area Networks—MediaAccess Control (MAC) Bridges and Virtual Bridged Local AreaNetworks—Amendment 20: Shortest Path Bridging, IEEE Standard802.1aq-2012, pp. 1–340, Jun. 2012.

[4] iOS: Multipath TCP Support in iOS 7. (2015). [Online]. Available:https://support.apple.com/en-us/HT201373

[5] A. Abd El Al, T. Saadawi, and M. Lee, “Improving throughput andreliability in mobile wireless networks via transport layer bandwidthaggregation,” Comput. Netw., vol. 46, no. 5, pp. 635–649, Dec. 2004.

[6] A. Abd El Al, T. Saadawi, and M. Lee, “LS-SCTP: A bandwidth aggre-gation technique for stream control transmission protocol,” Comput.Commun., vol. 27, no. 10, pp. 1012–1024, Jun. 2004.

[7] S. Addepalli, H. G. Schulzrinne, A. Singh, and G. Ormazabal,“Heterogeneous access: Survey and design considerations,”Dept. Comput. Sci., Columbia Univ., New York, NY,USA, Tech. Rep. CUCS-28-13, 2013. [Online]. Available:http://dx.doi.org/10.7916/D8QJ7F8P

[8] H. Adhari, T. Dreibholz, M. Becke, E. P. Rathgeb, and M. Tüxen,“Evaluation of concurrent multipath transfer over dissimilar paths,” inProc. IEEE Workshops Int. Conf. Adv. Inf. Netw. Appl. (WAINA), 2011,pp. 708–714.

[9] H. Adiseshu, G. Parulkar, and G. Varghese, “A reliable and scalablestriping protocol,” in Proc. Appl. Technol. Archit. Protocols Comput.Commun., vol. 26. Stanford, CA, USA, 1996, pp. 131–141.

[10] M. Allman, H. Kruse, and S. Ostermann, “An application-level solutionto TCP’s satellite inefficiencies,” in Proc. 1st Int. Workshop SatelliteBased Inf. Services (WOSBIS), 1996.

[11] P. Amer et al., “Load sharing for the stream control transmission pro-tocol (SCTP),” draft-tuexen-tsvwg-sctp-multipath-10 (Experimental),Internet-Draft, IETF, Nov. 2015.

[12] P. D. Amer, C. Chassot, T. J. Connolly, M. Diaz, and P. Conrad,“Partial-order transport service for multimedia and other applications,”IEEE/ACM Trans. Netw., vol. 2, no. 5, pp. 440–456, Oct. 1994.

[13] A. Argyriou and V. Madisetti, “Bandwidth aggregation with SCTP,”in Proc. IEEE Glob. Telecommun. Conf. (GLOBECOM), vol. 7.San Francisco, CA, USA, 2003, pp. 3716–3721.

[14] B. Arzani, A. Gurney, S. Cheng, R. Guerin, and B. T. Loo, “Impact ofpath characteristics and scheduling policies on MPTCP performance,”in Proc. 28th Int. Conf. Adv. Inf. Netw. Appl. Workshops (WAINA),Victoria, BC, Canada, May 2014, pp. 743–748.

[15] AVAYA. (2011). Multi-Link Trunking. [Online]. Available:https://downloads.avaya.com/css/P8/documents/100134063

[16] A. Baldini, L. De Carli, and F. Risso, “Increasing performances ofTCP data transfers through multiple parallel connections,” in Proc.IEEE Symp. Comput. Commun. (ISCC), Sousse, Tunisia, Jul. 2009,pp. 630–636.

[17] S. Barré, C. Paasch, and O. Bonaventure, “MultiPath TCP: From theoryto practice,” in Proc. 10th Int. IFIP TC 6 Conf. Netw. (NETWORKING),Valencia, Spain, 2011, pp. 444–457.

[18] M. Becke et al., “Comparison of multipath TCP and CMT-SCTP basedon intercontinental measurements,” in Proc. IEEE Glob. Commun.Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 1360–1366.

[19] E. Blanton and M. Allman, “On making TCP more robust to packetreordering,” ACM SIGCOMM Comput. Commun. Rev., vol. 32, no. 1,pp. 20–30, 2002.

[20] O. Bonaventure. (2015). In Korean, Multipath TCP is PronouncedGIGA Path. [Online]. Available: http://blog.multipath-tcp.org/blog/html/2015/07/24/korea.html

[21] L. Budzisz, R. Ferrús, F. Casadevall, and P. Amer, “On concurrentmultipath transfer in SCTP-based handover scenarios,” in Proc. IEEEInt. Conf. Commun. (ICC), Dresden, Germany, 2009, pp. 1–6.

[22] D. Bushmitch, S. Panwar, and A. Pal, “Thinning, striping and shuffling:Traffic shaping and transport techniques for variable bit rate video,” inProc. IEEE Glob. Telecommun. Conf. (GLOBECOM), vol. 2. Taipei,Taiwan, Nov. 2002, pp. 1485–1491.

[23] G. Carofiglio, M. Gallo, L. Muscariello, M. Papalini, and S. Wang,“Optimal multipath congestion control and request forwarding ininformation-centric networks,” in Proc. 21st IEEE Int. Conf. Netw.Protocols (ICNP), Göttingen, Germany, 2013, pp. 1–10.

[24] C. Casetti and W. Gaiotto, “Westwood SCTP: Load balancingover multipaths using bandwidth-aware source scheduling,” in Proc.Veh. Technol. Conf. (VTC), vol. 4. Los Angeles, CA, USA, 2004,pp. 3025–3029.

[25] C. Cetinkaya and E. W. Knightly, “Opportunistic traffic scheduling overmultiple network paths,” in Proc. 23rd Annu. Joint Conf. IEEE Comput.Commun. Soc. (INFOCOM), vol. 3. Hong Kong, 2004, pp. 1928–1937.

[26] K. Chebrolu, B. Raman, and R. R. Rao, “A network layer approach toenable TCP over multiple interfaces,” Wireless Netw., vol. 11, no. 5,pp. 637–650, 2005.

[27] K. Chebrolu and R. R. Rao, “Bandwidth aggregation for real-timeapplications in heterogeneous wireless networks,” IEEE Trans. MobileComput., vol. 5, no. 4, pp. 388–403, Apr. 2006.

[28] J. Chen, K. Xu, and M. Gerla, “Multipath TCP in lossy wire-less environment,” in Proc. IFIP 3rd Annu. Mediterr. Ad Hoc Netw.Workshop (Med-Hoc-Net), Bodrum, Turkey, 2004, pp. 263–270.

[29] X. Chen, A. Jukan, A. C. Drummond, and N. L. S. da Fonseca,“A multipath routing mechanism in optical networks with extremelyhigh bandwidth requests,” in Proc. IEEE Glob. Telecommun.Conf. (GLOBECOM), Honolulu, HI, USA, 2009, pp. 1–6.

[30] Y.-C. Chen et al., “A measurement-based study of multipath TCP per-formance over wireless networks,” in Proc. ACM Conf. Internet Meas.Conf. (IMC), Barcelona, Spain, 2013, pp. 455–468.

[31] P. A. Chou, Y. Wu, and K. Jain, “Practical network coding,” in Proc.Allerton Conf. Commun. Control Comput., Monticello, IL, USA, 2003.

[32] Cisco. (2003). EtherChannels. [Online]. Available: http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3550/software/release/12-1_13_ea1/configuration/guide/3550scg/swethchl.html

[33] OpenFlow Consortium, “Openflow switch specification/version1.1.0,” Feb. 2011. [Online]. Available: http://archive.openflow.org/documents/openflow-spec-v1.1.0.pdf


[34] M. Coudron, S. Secci, G. Maier, G. Pujolle, and A. Pattavina, “Boostingcloud communications through a crosslayer multipath protocol archi-tecture,” in Proc. IEEE SDN Future Netw. Services (SDN4FNS), Trento,Italy, 2013, pp. 1–8.

[35] M. Coudron, S. Secci, and G. Pujolle, “Differentiated pacing on mul-tiple paths to improve one-way delay estimations,” in Proc. IFIP/IEEEInt. Symp. Integr. Netw. Manag. (IM), Ottawa, ON, Canada, May 2015,pp. 672–678.

[36] M. Coudron, S. Secci, G. Pujolle, P. Raad, and P. Gallard, “Cross-layercooperation to boost multipath TCP performance in cloud networks,”in Proc. IEEE 2nd Int. Conf. Cloud Netw. (CloudNet), San Francisco,CA, USA, Nov. 2013, pp. 58–66.

[37] Y. Cui, L. Wang, X. Wang, H. Wang, and Y. Wang, “FMTCP: A foun-tain code-based multipath transmission control protocol,” IEEE/ACMTrans. Netw., vol. 23, no. 2, pp. 465–478, Apr. 2015.

[38] Y. Cui, X. Wang, H. Wang, G. Pan, and Y. Wang, “FMTCP: A foun-tain code-based multipath transmission control protocol,” in Proc. 32ndIEEE Int. Conf. Distrib. Comput. Syst. (ICDCS), Macau, China, 2012,pp. 366–375.

[39] M. Cullen et al., “Problem statement: Bandwidth aggregationfor Internet access,” draft-zhang-banana-problem-statement-01(Informational), Internet-Draft, IETF, Nov. 2015.

[40] T. Das and K. M. Sivalingam, “TCP improvements for data centernetworks,” in Proc. 5th Int. Conf. Commun. Syst. Netw. (COMSNETS),Bengaluru, India, 2013, pp. 1–10.

[41] G. Detal, C. Paasch, and O. Bonaventure, “Multipath in the mid-dle(box),” in Proc. Workshop Hot Topics Middleboxes Netw. Funct.Virtualization (HotMiddlebox), Santa Barbara, CA, USA, 2013,pp. 1–6.

[42] A. Detti, C. Pisa, and N. B. Melazzi, “Modeling multipath forward-ing strategies in information centric networks,” in Proc. IEEE Conf.Comput. Commun. Workshops (INFOCOM WKSHPS), Hong Kong,Apr. 2015, pp. 324–329.

[43] Deutsche Telekom. (2014). Hybrid Access—A Promising Approachto Increase Bandwidth of DSL-Lines. [Online]. Available:http://www.laboratories.telekom.com/public/english/newsroom/news/pages/hybrid-access.aspx

[44] C. Diop, G. Dugue, C. Chassot, and E. Exposito, “QoS-orientedMPTCP extensions for multimedia multi-homed systems,” in Proc.26th Int. Conf. Adv. Inf. Netw. Appl. Workshops (WAINA), Fukuoka,Japan, Mar. 2012, pp. 1119–1124.

[45] J. Domzał et al., “A survey on methods to provide multipath trans-mission in wired packet networks,” Comput. Netw., vol. 77, pp. 18–41,Feb. 2015.

[46] Y. Dong, N. Pissinou, and J. Wang, “Concurrency handling in TCP,”in Proc. 5th Annu. Conf. Commun. Netw. Services Res. (CNSR),Fredericton, NB, Canada, May 2007, pp. 255–262.

[47] T. Dreibholz, M. Becke, H. Adhari, and E. P. Rathgeb, “On the impactof congestion control for concurrent multipath transfer on the transportlayer,” in Proc. 11th IEEE Int. Conf. Telecommun. (ConTEL), Graz,Austria, 2011, pp. 397–404.

[48] T. Dreibholz, M. Becke, J. Pulinthanath, and E. P. Rathgeb, “ApplyingTCP-friendly congestion control to concurrent multipath transfer,” inProc. 24th IEEE Int. Conf. Adv. Inf. Netw. Appl. (AINA), Perth, WA,Australia, 2010, pp. 312–319.

[49] T. Dreibholz, M. Becke, E. P. Rathgeb, and M. Tuxen, “On the use ofconcurrent multipath transfer over asymmetric paths,” in Proc. IEEEGlob. Telecommun. Conf. (GLOBECOM), Miami, FL, USA, 2010,pp. 1–6.

[50] T. Dreibholz, R. Seggelmann, M. Tüexen, and E. Rathgeb,“Transmission scheduling optimizations for concurrent multipathtransfer,” in Proc. 8th Int. Workshop Protocols Future Large ScaleDiverse Netw. Transp. (PFLDNeT), vol. 8. Lancaster, PA, USA, 2010,pp. 1–6.

[51] D. Eastlake, M. Zhang, A. Ghanwani, V. Manral, and A. Banerjee,“Transparent interconnection of lots of links (TRILL): Clarifications,corrections, and updates,” IETF, Fremont, CA, USA, RFC 7180,May 2014.

[52] K. Evensen et al., “A network-layer proxy for bandwidth aggre-gation and reduction of IP packet reordering,” in Proc. IEEE34th Conf. Local Comput. Netw. (LCN), Zürich, Switzerland, 2009,pp. 585–592.

[53] K. Evensen et al., “Improving the performance of quality-adaptivevideo streaming over multiple heterogeneous access networks,” in Proc.2nd Annu. ACM Conf. Multimedia Syst., Santa Clara, CA, USA, 2011,pp. 57–68.

[54] K. Evensen et al., “Using bandwidth aggregation to improve the perfor-mance of quality-adaptive streaming,” Signal Process. Image Commun.,vol. 27, no. 4, pp. 312–328, 2012.

[55] K. Evensen, T. Kupka, D. Kaspar, P. Halvorsen, and C. Griwodz,“Quality-adaptive scheduling for live streaming over multiple accessnetworks,” in Proc. 20th Int. Workshop Netw. Oper. Syst. Support Digit.Audio Video, Amsterdam, The Netherlands, 2010, pp. 21–26.

[56] D. Farinacci, D. Lewis, D. Meyer, and V. Fuller, “The locator/IDseparation protocol (LISP),” IETF, Fremont, CA, USA, RFC 6830,Feb. 2013.

[57] D. Fedyk, P. Ashwood-Smith, D. Allan, A. Bragg, and P. Unbehagen,“IS-IS extensions supporting IEEE 802.1 aq shortest path bridging,”IETF, Fremont, CA, USA, RFC 6329, 2012.

[58] S. Ferlin, T. Dreibholz, and O. Alay, “Multi-path transport over het-erogeneous wireless networks: Does it really pay off?” in Proc. IEEEGlob. Commun. Conf. (GLOBECOM), Austin, TX, USA, Dec. 2014,pp. 4807–4813.

[59] R. Fielding et al., “Hypertext transfer protocol–HTTP/1.1,” IETF RFC2616, Fremont, CA, USA, Jun. 1999.

[60] A. Ford, C. Raiciu, M. Handley, and S. Barre, “TCP exten-sions for multipath operation with multiple addresses,”IETF Internet-Draft, May 2009. [Online]. Available:https://tools.ietf.org/html/draft-ford-mptcp-multiaddressed-00

[61] A. Ford, C. Raiciu, M. Handley, S. Barré, and J. Iyengar, “Architecturalguidelines for multipath TCP development,” IETF, Fremont, CA, USA,RFC 6182, Mar. 2011.

[62] A. Ford, C. Raiciu, M. Handley, and O. Bonaventure, “TCP extensionsfor multipath operation with multiple addresses,” IETF, Fremont, CA,USA, RFC 6824, Jan. 2013.

[63] C. Fragouli, D. Lun, M. Médard, and P. Pakzad, “On feedback fornetwork coding,” in Proc. 41st Annu. Conf. Inf. Sci. Syst. (CISS),Baltimore, MD, USA, 2007, pp. 248–252.

[64] L. Golubchik et al., “Multi-path continuous media streaming: What arethe benefits?” Perform. Eval., vol. 49, nos. 1–4, pp. 429–449, 2002.

[65] A. Gurtov and T. Polishchuk, “Secure multipath transport for legacyInternet applications,” in Proc. 6th Int. Conf. Broadband Commun.Netw. Syst., Madrid, Spain, 2009, pp. 1–8.

[66] K. Habak, K. A. Harras, and M. Youssef, “OPERETTA: An opti-mal energy efficient bandwidth aggregation system,” in Proc. 9thAnnu. IEEE Commun. Soc. Conf. Sensor Mesh Ad Hoc Commun.Netw. (SECON), Seoul, South Korea, 2012, pp. 121–129.

[67] K. Habak, K. A. Harras, and M. Youssef, “Bandwidth aggregationtechniques in heterogeneous multi-homed devices: A survey,” Comput.Netw., vol. 92, pp. 168–188, Dec. 2015.

[68] K. Habak, K. A. Harras, and M. Youssef, “OSCAR: A collaborativebandwidth aggregation system,” arXiv preprint arXiv:1401.1258, 2014.

[69] K. Habak, M. Youssef, and K. A. Harras, “G-DBAS: A greenand deployable bandwidth aggregation system,” in Proc. IEEEWireless Commun. Netw. Conf. (WCNC), Shanghai, China, 2012,pp. 3290–3295.

[70] K. Habak, M. Youssef, and K. A. Harras, “DBAS: A deployablebandwidth aggregation system,” in Proc. 5th Int. Conf. New Technol.Mobility Security (NTMS), Istanbul, Turkey, 2012, pp. 1–6.

[71] K. Habak, M. Youssef, and K. A. Harras, “An optimal deploy-able bandwidth aggregation system,” Comput. Netw., vol. 57, no. 15,pp. 3067–3080, 2013.

[72] T. J. Hacker, B. D. Athey, and B. Noble, “The end-to-end performanceeffects of parallel TCP sockets on a lossy wide-area network,” in Proc.IEEE Int. Parallel Distrib. Process. Symp. (IPDPS), 2002, pp. 434–443.

[73] A. Hari, G. Varghese, and G. Parulkar, “An architecture for packet-striping protocols,” ACM Trans. Comput. Syst., vol. 17, no. 4,pp. 249–287, Nov. 1999.

[74] Y. Hasegawa, I. Yamaguchi, T. Hama, H. Shimonishi, and T. Murase,“Improved data distribution for multipath TCP communication,” inProc. IEEE Glob. Telecommun. Conf. (GLOBECOM), St. Louis, MO,USA, Nov. 2005, p. 5.

[75] Y. Hasegawa, I. Yamaguchi, T. Hama, H. Shimonishi, and T. Murase,“Deployable multipath communication scheme with sufficient perfor-mance data distribution method,” Comput. Commun., vol. 30, no. 17,pp. 3285–3292, 2007.

[76] S. Hassayoun, J. Iyengar, and D. Ros, “Dynamic window couplingfor multipath congestion control,” in Proc. 19th IEEE Int. Conf. Netw.Protocols (ICNP), Vancouver, BC, Canada, 2011, pp. 341–352.

[77] T. Henderson, S. Floyd, A. Gurtov, and Y. Nishida, “The NewRenomodification to TCP’s fast recovery algorithm,” IETF, Fremont, CA,USA, RFC 6582, 2012.


[78] B. Hesmans, F. Duchene, C. Paasch, G. Detal, and O. Bonaventure,“Are TCP extensions middlebox-proof?” in Proc. Workshop Hot TopicsMiddleboxes Netw. Funct. Virtualization, Santa Barbara, CA, USA,2013, pp. 37–42.

[79] M. Honda, Y. Nishida, L. Eggert, P. Sarolahti, and H. Tokuda,“Multipath congestion control for shared bottleneck,” in Proc.Workshop Protocols Fast Long-Distance Netw. (PFLDNeT), 2009,pp. 19–24.

[80] M. Honda et al., “Is it still possible to extend TCP?” in Proc.ACM SIGCOMM Conf. Internet Meas. Conf., Berlin, Germany, 2011,pp. 181–194.

[81] H.-Y. Hsieh, K.-H. Kim, Y. Zhu, and R. Sivakumar, “A receiver-centrictransport protocol for mobile hosts with heterogeneous wireless inter-faces,” in Proc. 9th Annu. Int. Conf. Mobile Comput. Netw. (MobiCom),San Diego, CA, USA, 2003, pp. 1–15.

[82] H.-Y. Hsieh and R. Sivakumar, “A transport layer approach for achiev-ing aggregate bandwidths on multi-homed mobile hosts,” in Proc. 8thAnnu. Int. Conf. Mobile Comput. Netw. (MobiCom), Atlanta, GA, USA,2002, pp. 83–94.

[83] H.-Y. Hsieh and R. Sivakumar, “pTCP: An end-to-end transport layerprotocol for striped connections,” in Proc. 10th IEEE Int. Conf. Netw.Protocols, Paris, France, 2002, pp. 24–33.

[84] H.-Y. Hsieh and R. Sivakumar, “A transport layer approach for achiev-ing aggregate bandwidths on multi-homed mobile hosts,” WirelessNetw., vol. 11, nos. 1–2, pp. 99–114, 2005.

[85] C.-M. Huang and C.-H. Tsai, “WiMP-SCTP: Multi-path transmissionusing stream control transmission protocol (SCTP) in wireless net-works,” in Proc. 21st IEEE Int. Conf. Adv. Inf. Netw. Appl. Workshops(AINAW), vol. 1. Niagara Falls, ON, Canada, 2007, pp. 209–214.

[86] X. Huang and Y. Fang, “Multiconstrained QoS multipath routing inwireless sensor networks,” Wireless Netw., vol. 14, no. 4, pp. 465–478,2008.

[87] C. Huitema, “Multi-homed TCP,” IETF Internet-Draft (Expired),May 1995. [Online]. Available: https://tools.ietf.org/html/draft-huitema-multi-homed-01

[88] IEEE. (2000). IEEE 802.ad Link Aggregation Task Force. [Online].Available: http://www.ieee802.org/3/ad/

[89] J. Iyengar, K. Shah, P. Amer, and R. Stewart, “Concurrent multipathtransfer using SCTP multihoming,” in Proc. Int. Symp. Perform. Eval.Comput. Telecommun. Syst. (SPECTS), 2004.

[90] J. R. Iyengar, P. D. Amer, and R. Stewart, “Retransmission policies forconcurrent multipath transfer using SCTP multihoming,” in Proc. 12thIEEE Int. Conf. Netw. (ICON), vol. 2. Singapore, 2004, pp. 713–719.

[91] J. R. Iyengar, P. D. Amer, and R. Stewart, “Concurrent multipathtransfer using SCTP multihoming over independent end-to-end paths,”IEEE/ACM Trans. Netw., vol. 14, no. 5, pp. 951–964, Oct. 2006.

[92] J. Lee et al., “Applied techniques for high bandwidth data transfersacross wide area networks,” in Proc. Int. Conf. Comput. High EnergyNuclear Phys., Beijing, China, 2001.

[93] P. Jokela, R. Moskowitz, and P. Nikander, “Using the encapsulatingsecurity payload (ESP) transport format with the host identity protocol(HIP),” IETF, Fremont, CA, USA, RFC 5202, Apr. 2008.

[94] Juniper. (2014). Aggregated Ethernet. [Online]. Available:http://www.juniper.net/documentation/en_US/junos14.1/topics/task/configuration/link-aggregation-cli.html

[95] S. Kandula, K. C.-J. Lin, T. Badirkhanli, and D. Katabi, “FatVAP:Aggregating AP backhaul capacity to maximize throughput,” inProc. 5th USENIX Symp. Netw. Syst. Design Implement., vol. 8.San Francisco, CA, USA, 2008, pp. 89–104.

[96] D. Kaspar, K. Evensen, P. Engelstad, and A. F. Hansen, “UsingHTTP pipelining to improve progressive download over multiple het-erogeneous interfaces,” in Proc. IEEE Int. Conf. Commun. (ICC),Cape Town, South Africa, 2010, pp. 1–5.

[97] D. Kaspar et al., “Enhancing video-on-demand playout over multipleheterogeneous access networks,” in Proc. 7th IEEE Consum. Commun.Netw. Conf. (CCNC), Las Vegas, NV, USA, 2010, pp. 1–5.

[98] V. Kawadia and P. R. Kumar, “A cautionary perspective on cross-layerdesign,” IEEE Wireless Commun., vol. 12, no. 1, pp. 3–11, Feb. 2005.

[99] S. Keshav, “A control-theoretic approach to flow control,” ACMSIGCOMM Comput. Commun. Rev., vol. 25, no. 1, pp. 188–201, 1995.

[100] P. Key, L. Massoulié, and D. Towsley, “Combining multipath routingand congestion control for robustness,” in Proc. 40th Annu. Conf. Inf.Sci. Syst., Princeton, NJ, USA, 2006, pp. 345–350.

[101] R. Khalili, N. Gast, M. Popovic, and J.-Y. Le Boudec, “MPTCPis not Pareto-optimal: Performance issues and a possible solution,”IEEE/ACM Trans. Netw., vol. 21, no. 5, pp. 1651–1665, Oct. 2013.

[102] R. Khalili, N. Gast, M. Popovic, U. Upadhyay, and J.-Y. Le Boudec,“MPTCP is not Pareto-optimal: Performance issues and a possible solu-tion,” in Proc. 8th Int. Conf. Emerg. Netw. Exp. Technol. (CoNEXT),Nice, France, 2012, pp. 1–12.

[103] H. A. Kim, B.-H. Oh, and J. Lee, “Improvement of MPTCP per-formance in heterogeneous network using packet scheduling mecha-nism,” in Proc. 18th Asia Pac. Conf. Commun. (APCC), Oct. 2012,pp. 842–847.

[104] K.-H. Kim and K. G. Shin, “Improving TCP performance over wire-less networks with collaborative multi-homed mobile hosts,” in Proc.3rd Int. Conf. Mobile Syst. Appl. Services, Seattle, WA, USA, 2005,pp. 107–120.

[105] K.-H. Kim and K. G. Shin, “PRISM: Improving the performance ofinverse-multiplexed TCP in wireless networks,” IEEE Trans. MobileComput., vol. 6, no. 12, pp. 1297–1312, Dec. 2007.

[106] K.-H. Kim, Y. Zhu, R. Sivakumar, and H.-Y. Hsieh, “A receiver-centric transport protocol for mobile hosts with heterogeneous wirelessinterfaces,” Wireless Netw., vol. 11, no. 4, pp. 363–382, 2005.

[107] R. Koetter and M. Médard, “An algebraic approach to network coding,”IEEE/ACM Trans. Netw., vol. 11, no. 5, pp. 782–795, Oct. 2003.

[108] M. Kun, Y. Jingdong, and R. Zhi, “The research and simulationof multipath-OLSR for mobile ad hoc network,” in Proc. IEEE Int.Symp. Commun. Inf. Technol. (ISCIT), vol. 1. Beijing, China, 2005,pp. 540–543.

[109] K.-C. Lan and C.-Y. Li, “Improving TCP performance over anon-board multi-homed network,” in Proc. IEEE Wireless Commun.Netw. Conf. (WCNC), Shanghai, China, 2012, pp. 2961–2966.

[110] T.-A. Le and L. X. Bui, “Forward delay-based packet schedulingalgorithm for multipath TCP,” arXiv preprint arXiv:1501.03196, 2015.

[111] Y. Lee, I. Park, and Y. Choi, “Improving TCP performance in mul-tipath packet forwarding networks,” J. Commun. Netw., vol. 4, no. 2,pp. 148–157, Jun. 2002.

[112] W. Lei, W. Zhang, and S. Liu, “A framework of multipathtransport system based on application-level relay (MPTS-AR),”IETF Internet-Draft (Experimental), Jul. 2015. [Online]. Available:https://tools.ietf.org/html/draft-leiwm-tsvwg-mpts-ar-04

[113] L. Kleinrock, “An early history of the Internet,” IEEE Commun. Mag.,vol. 48, no. 8, pp. 26–36, Aug. 2010.

[114] M. Li, A. Lukyanenko, and Y. Cui, “Network coding based multipathTCP,” in Proc. IEEE Conf. Comput. Commun. Workshops (INFOCOMWorkshop), Orlando, FL, USA, Mar. 2012, pp. 25–30.

[115] M. Li, A. Lukyanenko, S. Tarkoma, Y. Cui, and A. Ylä-Jääski,“Tolerating path heterogeneity in multipath TCP with bounded receivebuffers,” Comput. Netw., vol. 64, pp. 1–14, May 2014.

[116] M. Li, A. Lukyanenko, S. Tarkoma, and A. Ylä-Jääski, “Thedelayed ACK evolution in MPTCP,” in Proc. IEEE Glob. Commun.Conf. (GLOBECOM), Atlanta, GA, USA, 2013, pp. 2282–2288.

[117] M. Li, A. Lukyanenko, S. Tarkoma, and A. Ylä-Jääski, “MPTCP incastin data center networks,” China Commun., vol. 11, no. 4, pp. 25–37,Apr. 2014.

[118] J. Liao, J. Wang, and X. Zhu, “cmpSCTP: An extension of SCTPto support concurrent multi-path transfer,” in Proc. IEEE Int. Conf.Commun. (ICC), Beijing, China, 2008, pp. 5762–5766.

[119] Y.-S. Lim, Y.-C. Chen, E. M. Nahum, D. Towsley, and K.-W. Lee,“Cross-layer path management in multi-path transport protocol formobile devices,” in Proc. Annu. Joint Conf. IEEE Comput. Commun.Soc. (INFOCOM), Toronto, ON, Canada, 2014, pp. 1815–1823.

[120] J. Liu, H. Zou, J. Dou, and Y. Gao, “Rethinking retransmission pol-icy in concurrent multipath transfer,” in Proc. Int. Conf. Intell. Inf.Hiding Multimedia Signal Process. (IIHMSP), Harbin, China, 2008,pp. 1005–1008.

[121] L. Magalhaes and R. Kravets, “Transport level mechanisms for band-width aggregation on mobile hosts,” in Proc. 9th IEEE Int. Conf. Netw.Protocols, Riverside, CA, USA, 2001, pp. 165–171.

[122] K. Manousakis and D. Famolari, “INTELiCON: A framework for thesimultaneous utilization of multiple interfaces and its application onTCP,” in Proc. Int. Wireless Commun. Mobile Comput. Conf. (IWCMC),Aug. 2008, pp. 976–981.

[123] S. Mao, D. Bushmitch, S. Narayanan, and S. S. Panwar, “MRTP:A multiflow real-time transport protocol for ad hoc networks,” IEEETrans. Multimedia, vol. 8, no. 2, pp. 356–369, Apr. 2006.

[124] S. Mascolo, L. A. Grieco, R. Ferorelli, P. Camarda, and G. Piscitelli,“Performance evaluation of Westwood+ TCP congestion control,”Perform. Eval., vol. 55, nos. 1–2, pp. 93–111, 2004.

[125] M. Mathis, J. Mahdavi, S. Floyd, and A. Romanow, “TCP selectiveacknowledgment options,” IETF RFC 2018, Fremont, CA, USA, Oct.1996. [Online]. Available: https://tools.ietf.org/html/rfc2018


[126] N. F. Maxemchuk, “Dispersity routing in store-and-forward net-works,” Ph.D. dissertation, Univ. Pennsylvania, Philadelphia,PA, USA, 1975. [Online]. Available: http://repository.upenn.edu/dissertations/AAI7524101

[127] M. Menth, A. Stockmayer, and M. Schmidt, “LISP hybrid access,”draft-menth-lisp-ha-00 (Experimental), Internet-Draft, IETF, Jul. 2015.

[128] J. Milbrandt, K. Humm, and M. Menth, “Adaptive bandwidth allo-cation: Impact of routing and load balancing on tunnel capacityrequirements,” in Proc. Next Gener. Internet Design Eng. (NGI),Valencia, Spain, 2006, pp. 318–325.

[129] F. H. Mirani, N. Boukhatem, and M. A. Tran, “A data-schedulingmechanism for multi-homed mobile terminals with disparate link laten-cies,” in Proc. 72nd IEEE Veh. Technol. Conf. Fall (VTC), Ottawa, ON,Canada, 2010, pp. 1–5.

[130] E. Miyazaki and M. Oguchi, “Evaluation of middleware for bandwidthaggregation using multiple interface in wireless communication,” Int.J. Adv. Netw. Services, vol. 4, nos. 3–4, pp. 343–352, 2012.

[131] R. Moskowitz, P. Nikander, P. Jokela, and T. Henderson, “Host identityprotocol,” IETF, Fremont, CA, USA, RFC 5201, Apr. 2008.

[132] D. Nagle, D. Serenyi, and A. Matthews, “The Panasas ActiveScalestorage cluster: Delivering scalable high bandwidth storage,” in Proc.ACM/IEEE Conf. Supercomput., Pittsburgh, PA, USA, 2004, p. 53.

[133] S. C. Nguyen and T. M. T. Nguyen, “Evaluation of multipath TCPload sharing with coupled congestion control option in heterogeneousnetworks,” in Proc. Glob. Inf. Infrastruct. Symp. (GIIS), 2011, pp. 1–5.

[134] S. C. Nguyen, X. Zhang, T. M. T. Nguyen, and G. Pujolle, “Evaluationof throughput optimization and load sharing of multipath TCP in het-erogeneous networks,” in Proc. 8th Int. Conf. Wireless Opt. Commun.Netw. (WOCN), Paris, France, 2011, pp. 1–5.

[135] T. Nguyen-Duc et al., “Investigating the performance of link aggrega-tion on OpenFlow switches,” in Testbeds and Research Infrastructure:Development of Networks and Communities. Cham, Switzerland:Springer, 2014, pp. 194–202.

[136] C. Nicutar, C. Paasch, M. Bagnulo, and C. Raiciu, “Evolving theInternet with connection acrobatics,” in Proc. Workshop Hot TopicsMiddleboxes Netw. Funct. Virtualization, Santa Barbara, CA, USA,2013, pp. 7–12.

[137] E. Nordmark and M. Bagnulo, “Shim6: Level 3 multihoming shimprotocol for IPv6,” IETF, Fremont, CA, USA, RFC 5533, Jun. 2009.[Online]. Available: http://www.hjp.at/doc/rfc/rfc5533.html/

[138] OVH Company. (2015). Overthebox. [Online]. Available:https://www.ovhtelecom.fr/overthebox

[139] C. Paasch, S. Ferlin, O. Alay, and O. Bonaventure, “Experimentalevaluation of multipath TCP schedulers,” in Proc. ACM SIGCOMMWorkshop Capacity Sharing Workshop, Chicago, IL, USA, 2014,pp. 27–32.

[140] C. Paasch, R. Khalili, and O. Bonaventure, “On the benefits of applyingexperimental design to improve multipath TCP,” in Proc. 9th ACMConf. Emerg. Netw. Exp. Technol. (CoNEXT), Santa Barbara, CA, USA,2013, pp. 393–398.

[141] A. Pathak, H. Pucha, Y. Zhang, Y. C. Hu, and Z. M. Mao, “A mea-surement study of Internet delay asymmetry,” in Passive and ActiveNetwork Measurement (LNCS 4979). Heidelberg, Germany: Springer,2008, pp. 182–191.

[142] Q. Peng, A. Walid, and S. H. Low, “Multipath TCP algorithms: Theoryand design,” ACM SIGMETRICS Perform. Eval. Rev., vol. 41, no. 1,pp. 305–316, 2013.

[143] Q. Peng, A. Walid, J. Hwang, and S. H. Low, “Multipath TCP:Analysis, design and implementation,” IEEE/ACM Trans. Netw.,vol. 24, no. 1, pp. 596–609, Feb. 2016. [Online]. Available:http://arxiv.org/abs/1308.3119

[144] C. Perkins, “IP encapsulation within IP,” IETF, Fremont, CA, USA,RFC 2003, Oct. 1996.

[145] C. Perkins, “IP mobility support for IPv4, revised,” IETF, Fremont,CA, USA, RFC 5944, Nov. 2010.

[146] C. E. Perkins, “Mobile IP,” IEEE Commun. Mag., vol. 35, no. 5,pp. 84–99, May 1997.

[147] A. Phanishayee et al., “Measurement and analysis of TCP throughputcollapse in cluster-based storage systems,” in Proc. 6th USENIX Conf.File Stor. Technol. (FAST), vol. 8. Berkeley, CA, USA, 2008, pp. 1–14.

[148] D. S. Phatak and T. Goff, “A novel mechanism for data streamingacross multiple IP links for improving throughput and reliability inmobile environments,” in Proc. 21st Annu. Joint Conf. IEEE Comput.Commun. Soc. (INFOCOM), vol. 2. New York, NY, USA, 2002,pp. 773–781.

[149] D. S. Phatak, T. Goff, and J. Plusquellic, “IP-in-IP tunneling toenable the simultaneous use of multiple IP interfaces for network levelconnection striping,” Comput. Netw., vol. 43, no. 6, pp. 787–804, 2003.

[150] S. Pierrel, P. Jokela, and J. Melen, “Simultaneous multi-access exten-sion to the host identity protocol,” draft-pierrel-hip-sima-00 (Workin Process), Internet-Draft, IETF, Jun. 2006. [Online]. Available:https://tools.ietf.org/html/draft-pierrel-hip-sima-00

[151] T. Polishchuk and A. Gurtov, “Improving TCP-friendliness and fairnessfor mHIP,” Infocommun. J., vol. 3, no. 1, pp. 26–34, 2011.

[152] J. Postel and J. Reynolds, “File transfer protocol,” IETF, Fremont, CA,USA, RFC 959, Oct. 1985.

[153] S. Prabhavat, H. Nishiyama, N. Ansari, and N. Kato, “On load distribu-tion over multipath networks,” IEEE Commun. Surveys Tuts., vol. 14,no. 3, pp. 662–680, 3rd Quart. 2012.

[154] J. Qadir, A. Ali, K.-L. A. Yau, A. Sathiaseelan, and J. Crowcroft,“Exploiting the power of multiplicity: A holistic survey of network-layer multipath,” IEEE Commun. Surveys Tuts., vol. 17, no. 4,pp. 2176–2213, 4th Quart. 2015.

[155] A. Qureshi, J. Carlisle, and J. Guttag, “Tavarua: Video streaming withWWAN striping,” in Proc. 14th Annu. ACM Int. Conf. Multimedia,Santa Barbara, CA, USA, 2006, pp. 327–336.

[156] M. Radi, B. Dezfouli, K. A. Bakar, and M. Lee, “Multipath routingin wireless sensor networks: Survey and research challenges,” Sensors,vol. 12, no. 1, pp. 650–685, 2012.

[157] C. Raiciu et al., “Improving datacenter performance and robustnesswith multipath TCP,” in Proc. ACM SIGCOMM Conf., vol. 41. Toronto,ON, Canada, 2011, pp. 266–277.

[158] C. Raiciu, M. Handley, and D. Wischik, “Coupled multipath-aware con-gestion control,” IETF Internet-Draft, Oct. 2009. [Online]. Available:https://tools.ietf.org/html/draft-raiciu-mptcp-congestion-00

[159] C. Raiciu, M. Handly, and D. Wischik, “Coupled congestion control formultipath transport protocols,” IETF, Fremont, CA, USA, RFC 6356,Oct. 2011.

[160] C. Raiciu et al., “How hard can it be? Designing and implement-ing a deployable multipath TCP,” in Proc. 9th USENIX Conf. Netw.Syst. Design Implement. (NSDI), vol. 12. San Jose, CA, USA, 2012,pp. 399–412.

[161] C. Raiciu et al., “Data center networking with multipath TCP,” in Proc.9th ACM SIGCOMM Workshop Hot Topics Netw., Monterey, CA, USA,2010, p. 10.

[162] C. Raiciu, D. Wischik, and M. Handley, “Practical congestion con-trol for multipath transport protocols,” Univ. College London, London,U.K., Tech. Rep., 2009.

[163] A. L. Ramaboli, O. E. Falowo, and A. H. Chan, “Bandwidth aggre-gation in heterogeneous wireless networks: A survey of currentapproaches and issues,” J. Netw. Comput. Appl., vol. 35, no. 6,pp. 1674–1690, 2012.

[164] K. Ramakrishnan, S. Floyd, and D. Black, “The addition of explicitcongestion notification (ECN) to IP,” IETF, Fremont, CA, USA,RFC 3168, Sep. 2001.

[165] P. Rodriguez and E. W. Biersack, “Dynamic parallel access to repli-cated content in the Internet,” IEEE/ACM Trans. Netw., vol. 10, no. 4,pp. 455–465, Aug. 2002.

[166] P. Rodriguez, R. Chakravorty, J. Chesterfield, I. Pratt, and S. Banerjee,“MAR: A commuter router infrastructure for the mobile Internet,” inProc. 2nd Int. Conf. Mobile Syst. Appl. Services, Boston, MA, USA,2004, pp. 217–230.

[167] K. Rojviboonchai and A. Hitoshi, “An evaluation of multi-pathtransmission control protocol (M/TCP) with robust acknowledgementschemes,” IEICE Trans. Commun., vol. 87, no. 9, pp. 2699–2707, 2004.

[168] K. Rojviboonchai, T. Osuga, and H. Aida, “R-M/TCP: Protocol forreliable multi-path transport over the Internet,” in Proc. 19th Int. Conf.Adv. Inf. Netw. Appl. (AINA), vol. 1. Taipei, Taiwan, 2005, pp. 801–806.

[169] K. Rojviboonchai, N. Watanabe, and H. Aida, “One-way-triptime (OWTT) measurement and retransmission policy for congestioncontrol in M/TCP,” in Proc. Annu. Conf. IPSJ, Okinawa, Japan, 2002.

[170] G. Rossini and D. Rossi, “Evaluating CCN multi-path interest for-warding strategies,” Comput. Commun., vol. 36, no. 7, pp. 771–778,2013.

[171] H. Sakakibara, M. Saito, and H. Tokuda, “Design and implementationof a socket-level bandwidth aggregation mechanism for wireless net-works,” in Proc. 2nd Annu. Int. Workshop Wireless Internet, Boston,MA, USA, 2006, p. 11.

[172] D. Sarkar, “A concurrent multipath TCP and its Markov model,” inProc. IEEE Int. Conf. Commun. (ICC), Istanbul, Turkey, Jun. 2006,pp. 615–620.


[173] H. Schulzrinne, S. Casner, R. Frederick, and V. Jacobson, “RTP: Atransport protocol for real-time applications,” IETF, Fremont, CA,USA, RFC 3550, Jul. 2003.

[174] K. Sha, J. Gehlot, and R. Greve, “Multipath routing techniques in wire-less sensor networks: A survey,” Wireless Pers. Commun., vol. 70, no. 2,pp. 807–829, May 2013.

[175] S. Shailendra, R. Bhattacharjee, and S. K. Bose, “Improving congestioncontrol for concurrent multipath transfer through bandwidth estimationbased resource pooling,” in Proc. 8th Int. Conf. Inf. Commun. SignalProcess. (ICICS), Singapore, 2011, pp. 1–5.

[176] S. Shakkottai, E. Altman, and A. Kumar, “The case for non-cooperativemultihoming of users to access points in IEEE 802.11 WLANs,” inProc. 25th IEEE Int. Conf. Comput. Commun. (INFOCOM), Barcelona,Spain, 2006, pp. 1–12.

[177] Z. U. Shamszaman, S. S. Ara, and I. Chong, “Feasibility considerationsof multipath TCP in dealing with big data application,” in Proc. Int.Conf. Inf. Netw. (ICOIN), Bangkok, Thailand, Jan. 2013, pp. 708–713.

[178] V. Sharma, S. Kalyanaraman, K. Kar, K. Ramakrishnan, andV. Subramanian, “MPLOT: A transport protocol exploiting multipathdiversity using erasure codes,” in Proc. 27th IEEE Conf. Comput.Commun. (INFOCOM), Phoenix, AZ, USA, 2008, pp. 121–125.

[179] V. Sharma, K. Kar, K. K. Ramakrishnan, and S. Kalyanaraman, “Atransport protocol to exploit multipath diversity in wireless networks,”IEEE/ACM Trans. Netw., vol. 20, no. 4, pp. 1024–1039, Aug. 2012.

[180] H. Shojania and B. Li, “Parallelized progressive network codingwith hardware acceleration,” in Proc. 15th IEEE Int. Workshop Qual.Service, Evanston, IL, USA, 2007, pp. 47–55.

[181] W. Simpson, “The point-to-point protocol (PPP),” IETF, Fremont, CA,USA, RFC 1661, Jul. 1994.

[182] A. Singh, M. Xiang, A. Konsgen, C. Goerg, and Y. Zaki,“Enhancing fairness and congestion control in multipath TCP,” inProc. 6th Joint IFIP Wireless Mobile Netw. Conf. (WMNC), Dubai,United Arab Emirates, 2013, pp. 1–8.

[183] S. K. Singh, T. Das, and A. Jukan, “A survey on Internet multipathrouting and provisioning,” IEEE Commun. Surveys Tuts., vol. 17, no. 4,pp. 2157–2175, 4th Quart. 2015.

[184] H. Sivakumar, S. Bailey, and R. L. Grossman, “PSockets: The casefor application-level network striping for data intensive applicationsusing high speed wide area networks,” in Proc. ACM/IEEE Conf.Supercomput. (CDROM), Dallas, TX, USA, 2000, Art. no. 37.

[185] K. Sklower, B. Lloyd, G. McGregor, D. Carr, and T. Coradetti, “ThePPP multilink protocol (MP),” IETF, Fremont, CA, USA, RFC 1990,Aug. 1996.

[186] A. C. Snoeren, “Adaptive inverse multiplexing for wide-area wirelessnetworks,” in Proc. Glob. Telecommun. Conf. (GLOBECOM), vol. 3.Rio de Janeiro, Brazil, 1999, pp. 1665–1672.

[187] R. Stewart, “Stream control transmission protocol,” IETF, Fremont,CA, USA, RFC 4960, Sep. 2007.

[188] T. N. Subedi, K. K. Nguyen, and M. Cheriet, “OpenFlow-basedin-network layer-2 adaptive multipath aggregation in data centers,”Comput. Commun., vol. 61, pp. 58–69, May 2015.

[189] J. Sun, Y. Wen, and L. Zheng, “On file-based content distribution overwireless networks via multiple paths: Coding and delay trade-off,” inProc. IEEE INFOCOM, Shanghai, China, 2011, pp. 381–385.

[190] A. S.-W. Tam, K. Xi, Y. Xu, and H. J. Chao, “Preventing TCP incastthroughput collapse at the initiation, continuation, and termination,”in Proc. 20th IEEE Int. Workshop Qual. Service, Coimbra, Portugal,2012, pp. 1–29.

[191] C.-L. Tsao and R. Sivakumar, “On effectively exploiting multiplewireless interfaces in mobile hosts,” in Proc. 5th Int. Conf.Emerg. Netw. Exp. Technol. (CoNEXT), Rome, Italy, 2009,pp. 337–348.

[192] S. Tullimas, T. Nguyen, R. Edgecomb, and S.-C. Cheung, “Multimediastreaming using multiple TCP connections,” ACM Trans. MultimediaComput. Commun. Appl., vol. 4, no. 2, p. 12, 2008.

[193] I. Van Beijnum, J. Crowcroft, F. Valera, and M. Bagnulo, “Loop-freeness in multipath BGP through propagating the longest path,” inProc. IEEE Int. Conf. Commun. Workshops (ICC), Dresden, Germany,2009, pp. 1–6.

[194] R. van der Pol et al., “Experiences with MPTCP in an intercontinen-tal OpenFlow network,” in Proc. 29th TERENA Netw. Conf. (TNC),Maastricht, The Netherlands, 2013, pp. 1617–1624.

[195] B. Wang, W. Wei, Z. Guo, and D. Towsley, “Multipath live streamingvia TCP: Scheme, performance and benefits,” in Proc. 3rd ACM Int.Conf. Emerg. Netw. Exp. Technol. (CoNEXT), New York, NY, USA,2007, Art. no. 11.

[196] B. Wang, W. Wei, Z. Guo, and D. Towsley, “Multipath live streamingvia TCP: Scheme, performance and benefits,” ACM Trans. MultimediaComput. Commun. Appl., vol. 5, no. 3, p. 25, 2009.

[197] F. Wang, Z. M. Mao, J. Wang, L. Gao, and R. Bush, “A measurementstudy on the impact of routing events on end-to-end Internet path per-formance,” in Proc. Conf. Appl. Technol. Architect. Protocols Comput.Commun. (SIGCOMM), Pisa, Italy, 2006, pp. 375–386.

[198] X. Wang, Z. Feng, D. Fan, Y. Xue, and V. Le, “A segment-basedadaptive joint session scheduling mechanism in heterogeneous wire-less networks,” in Proc. 70th IEEE Veh. Technol. Conf. Fall (VTC),Anchorage, AK, USA, 2009, pp. 1–5.

[199] Y. Wen and V. W. S. Chan, “Ultra-reliable communication over vul-nerable all-optical networks via lightpath diversity,” IEEE J. Sel. AreasCommun., vol. 23, no. 8, pp. 1572–1587, Aug. 2005.

[200] D. Wischik, M. Handley, and M. B. Braun, “The resource poolingprinciple,” ACM SIGCOMM Comput. Commun. Rev., vol. 38, no. 5,pp. 47–52, 2008.

[201] F. Yang and P. Amer, “Non-renegable selective acknowledgments(NR-SACKs) for MPTCP,” in Proc. 27th Int. Conf. Adv. Inf. Netw. Appl.Workshops (WAINA), Barcelona, Spain, Mar. 2013, pp. 1113–1118.

[202] F. Yang and P. Amer, “Work in progress: Using one-way commu-nication delay for in-order arrival MPTCP scheduling,” in Proc. 9thInt. Conf. Commun. Netw. China (CHINACOM), Maoming, China,Aug. 2014, pp. 122–125.

[203] M. Zhang, B. Karp, S. Floyd, and L. Peterson, “RR-TCP: A reordering-robust TCP with DSACK,” in Proc. 11th IEEE Int. Conf. Netw.Protocols, Atlanta, GA, USA, 2003, pp. 95–106.

[204] M. Zhang, J. Lai, A. Krishnamurthy, L. L. Peterson, and R. Y. Wang,“A transport layer approach for improving end-to-end performance androbustness using redundant paths,” in Proc. Gen. Track USENIX Annu.Tech. Conf., Boston, MA, USA, 2004, pp. 99–112.

[205] W. Zhang, W. Lei, S. Liu, and G. Li, “A general framework ofmultipath transport system based on application-level relay,” Comput.Commun., vol. 51, pp. 70–80, Sep. 2014.

[206] D. Zhou, W. Song, and M. Shi, “Goodput improvement for multipathTCP by congestion window adaptation in multi-radio devices,” in Proc.IEEE Consum. Commun. Netw. Conf. (CCNC), Las Vegas, NV, USA,2013, pp. 508–514.

Ming Li received the M.S. degree in computerscience from the Beijing University of Posts andCommunications, China, in 2008, and the Ph.D.degree in computer science and engineering fromAalto University, Finland. His research interestsinclude network security, network architecture, andnetwork protocols.

Andrey Lukyanenko received the M.Sc. degreesin computer science from the University of Kuopioin 2005, the M.Sc. degree in mathematics fromPetrozavodsk State University in 2005, and the Ph.D.degree in computer science from the Universityof Helsinki in 2010. From 2006 to 2010, hewas a Researcher in the networking group withthe Helsinki Institute for Information Technology.Since 2010, he has been a Post-Doctoral Researcherwith Aalto University, with the DatacommunicationSoftware Group. In 2013, he was on a research

visit to ICSI/UC Berkeley. He has worked on problems related to backoffprotocols in IEEE802.11, security with host identity protocol, problems ofdenial-of-service attacks, and reputations in peer-to-peer networks. Duringhis research, he actively uses game theory, queuing theory, and data analy-sis methods. His research interests include information centric networking,datacenter architecture, in-network caching mechanisms, and future Internettechnologies.


Zhonghong Ou received the Ph.D. degree fromthe University of Oulu, Finland. Since 2016, hehas been an Associate Professor with the Schoolof Computer Science, Beijing University of Postsand Telecommunications, China. From 2010 to2015, he was a Post-Doctoral Researcher withAalto University, Finland. In 2010, he was aVisiting Scholar with Internet Real-Time Laboratory,Columbia University, USA. In 2013, he was aVisiting Scholar with Intel Labs, USA. He has awide spectrum of research interests, including virtu-

alization and cloud computing platforms, cloud storage, energy optimizationfor mobile platforms, and big data analytics.

Antti Ylä-Jääski was with Nokia from 1994 to2004 in several research and research manage-ment positions with focus on future Internet, mobilenetworks, applications, services and service archi-tectures. He is the Vice Head of the DepartmentComputer Science, Aalto University. He has super-vised 231 master’s thesis and 21 doctoral disserta-tions during his professorship in Aalto Universityfrom 2002 to 2015. He has published over 100peer-reviewed articles and holds several internationalpatents. His current research interest is focused on

mobile and cloud computing, crowdsourcing, and energy efficient computingand communications.

Sasu Tarkoma received the Ph.D. degree in com-puter science from the University of Helsinki. Hehas managed and participated in national and inter-national research projects with the University ofHelsinki, the Helsinki University of Technology, andthe Helsinki Institute for Information Technology.He has lectured several courses on middleware anddata communications. His research interests includemiddleware and distributed computing.

Matthieu Coudron received the degree in telecom-munications engineering from Telecom SudParis,France, in 2012. He is currently pursuing the Ph.D.degree with University Pierre and Marie Curie (ParisVI, Sorbonne Universites), France. He works onInternet multipath forwarding.

Stefano Secci received the Laurea degree intelecommunications engineering from Politecnico diMilano, in 2005, and a dual Ph.D. degrees in com-puter networks from Politecnico di Milano andTelecom ParisTech, France, in 2009. He has beenan Associate Professor with the University Pierreand Marie Curie (Paris VI, Sorbonne Universites),France, since 2010. He also worked as a ResearchFellow with NTNU, George Mason University,Ecole Polytechnique de Montreal, and Politecnicodi Milano, and as a Network Engineer with Fastweb

Italia. He has been the current Chair of the Internet Technical Committee,joint between the IEEE Communication Society and the Internet Society, since2013. His work spans network optimization, protocol design, Internet rout-ing, and traffic engineering. His current research interests are about Internetresiliency and cloud networking.

IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4...

Documents

Transcript of IEEE COMMUNICATIONS SURVEYS & TUTORIALS, VOL. 18, NO. 4...