ECN (Explicit Congestion Notification) in TCP/IP

Additional topics below:
Obsoleted by RFC 3168,
Evaluations of ECN,
Proposed Modifications of ECN,
Implementation and Deployment of ECN,
ECN in Simulators,
Notes and Email,
Research on ECN and Related Work,
Alternate proposals for the ECN field in IP,
ECN with MPLS,
Active Queue Management and ECN,
Talks,
Open issues for ECN,
Approaches in other architectures: FECN, BECN, and EFCI,
Archival documents.

The Addition of Explicit Congestion Notification (ECN) to IP.
Ramakrishnan, K.K., Floyd, S., and Black, D.
RFC 3168, Proposed Standard, September 2001. Local copy: (postscript, text).
RFC 3168 obsoletes RFC 2481, and several internet drafts listed below.
Wetherall, D., Ely, D., and Spring, N.,
Robust ECN Signaling with Nonces,
RFC 3540, Experimental, June 2003.
Adding Explicit Congestion Notification (ECN) Capability to TCP's SYN/ACK Packets,
A. Kuzmanovic, A. Mondal, S. Floyd, and K.K. Ramakrishnan,
RFC 5562, Experimental, June 2009. Local copy: (text, pdf). ECN web page. Simulation scripts.
The earlier SIGCOMM paper: The Power of Explicit Congestion Notification (PDF), A. Kuzmanovic, SIGCOMM, August 2005.
Section 9.1 of RFC 3168 discusses IP packets encapsulated in IP, but we do not have a separate document updating RFC 2003, "IP Encapsulation within IP", for ECN compatibility.
The ECN BOF at the L.A. IETF, March 1998.
Floyd, S.,
TCP and Explicit Congestion Notification (compressed postscript, pdf).
ACM Computer Communication Review, V. 24 N. 5, October 1994, p. 10-23.
[This issue of CCR incorrectly has "1995" on the cover instead of "1994".] Abstract.
Problems with non-ECN-compatible equipment in the network: There are several (broken) deployed TCP implementations and routers that don't respond to SYN packets that use the ECN-related flags for negotiating ECN-capability.
A March 2007 report at IETF also reports on problems of routers that crash when a TCP data packet arrives marked with either ECT(0) or ECT(1), representing a packet from an ECN-Capable connection.

Obsoleted by RFC 3168:

A Proposal to add Explicit Congestion Notification (ECN) to IP (local copy, in text or postscript).
Ramakrishnan, K.K., and Floyd, S.,
RFC 2481, January 1999, Experimental. Obsoleted by RFC 3168, Proposed Standard.
TCP with ECN: The Treatment of Retransmitted Data Packets (text, postscript).
Floyd, S. and Ramakrishnan, K.K.,
Internet-draft, November 2000. Obsoleted by RFC 3168, Proposed Standard.
IPsec Interactions with ECN ( text, postscript).
Floyd, S., David Black, and K.K. Ramakrishnan.
Internet-draft, December 1999. Obsoleted by RFC 3168.
ECN Interactions with IP Tunnels,
Floyd, S., K.K. Ramakrishnan, and Black, D..
Internet draft, October 2000. Obsoleted by RFC 3168.

Evaluations of ECN:

S. Floyd, Notes on Evaluating ECN, April 2002.
Marek Malowidzki, ECN is Fine - But Will It Be Used?, PDCN'03
"The paper ... presents possible threats to ECN and shows the gain malicious ECN sources could achieve from ECN subverting."
In particular, the paper outlines a scenario of macro-scale attacks, where a company uses two access routers, one at each site, to use ECN inappropriately on the path between the two sites.
Marek Malowidzki, Simulation-based Study of ECN Performance in RED Networks, SPECTS'03.
"The main assessment factor used in our test, the aggregate goodput of all TCP sources, in most cases was the same for both ECN and non-ECN cases."
[As suggested in Notes on Evaluating ECN, one relevant metric for ECN is that of occasionally-high per-packet delay due to packet drops (or equivalently, for unreliable transport such as multimedia using DCCP, of packet drop rates). -SF]
S. Alessandrini and Tiziana Ferrari, Experimental Evaluation of TCP Performance with ECN Optimization, 2002.
K. Pentikousis and H. Badr, An Evaluation of TCP with Explicit Congestion Notification, Annals of Telecommunications, Mid-2003.
"ECN is an effective mechanism for the timely and efficient conveying of congestion information. It leads to fewer packet drops, but does not necessarily lead to improved transfer times, and thus goodput, under uniform conditions. On the other hand, in no case does ECN seem to lead to an actual degradation in performance."
K. Pentikousis, H. Badr, and B. Kharmah, TCP with ECN: Performance Gains for Large Transfers, 2001.
"Our results show that TCP performance improves significantly with ECN both for individual flows and the network as a whole."
[Some of bad performance without ECN is due to the use of Reno TCP in these simulations. - SF]
Jamal Hadi Salim and Uvaiz Ahmed,
Performance Evaluation of Explicit Congestion Notification (ECN) in IP Networks,
RFC 2884, July 2000.

Proposed Modifications of ECN:

M. Kwon and S. Fahmy, TCP Increase/Decrease Behavior with Explicit Congestion Notification (ECN), Proceedings of IEEE ICC (Symposium on High-Speed Networks), V. 4, pp. 2335-2340, April 2002.
"We investigate the effect of TCP Explicit Congestion Notiﬁcation (ECN) with a new response strategy that is more aggressive in the short term, but preserves TCP long term behavior, without modifying the router marking rate. A less aggressive ECN decrease gives more incentives for end systems to become ECN-compliant, as ECN serves as an early warning sign in this case."

Implementation and Deployment of ECN:

Microsoft Vista supports ECN, but it is disabled by default. Implementation report from March 2007.
MAC OS X:
Leopard 10.5.0 implements ECN, controlled by the variables "net.inet.tcp.ecn_negotiate_in" and "net.inet.tcp.ecn_initiate_out". Reported by Rui Paulo, 2007.
Linux:
- Linux 2.4 has full ECN support, including ECN TCP. January 2001.
- Linux 2.3: The Linux 2.3 kernel includes the router code for ECN. May 1999.
FreeBSD: Rui Paulo added TCP ECN end-host support to FreeBSD 8.0. July 2008.
NetBSD supports ECN at the end host. Reported by Rui Paulo, 2006.
ALTQ:
Kenjiro Cho added ECN router support to ALTQ, which runs on FreeBSD/NetBSD/OpenBSD. February 2002.
KAME includes ALTQ and supports ECN for IPv6 and IPsec.
Test results of ALTQ/RED performance, January 1998.
Cisco:
Cisco supports ECN starting from 12.2(8)T. August 2006.
IBM's AIX:
IBM supports ECN in AIX at the end hosts (TCP level) from release 5.1. May 2001.
ECN is off by default, and can be turned on by a runtime network option 'tcp_ecn', with the command: # no -o tcp_ecn=1
Nortel:
Nortel's Open IP Environment 2.1 contains router ECN support. November 2000.
Sun:
Sun has ECN support in Solaris 9. 2002.
One needs to say "ndd -set /dev/tcp tcp_ecn_permitted 2" for an end-host to ask for ECN.
Tcpdump: Jamal Hadi Salim, Tcpdump patch to print ECN info, May 1998.
Floyd, S., Guidelines for testing the ECN implementation in a router, May 2001.
In 2002, David Moore from CAIDA reported that in measurements at one link, 0.1% of the packets had the CE codepoint set. Either this codepoint was being used for some other purpose, or there is some deployment of ECN capability in routers as well as in TCP stacks.

ECN in Simulators:

The ns-2 simulator implements ECN with TCP. There are two validation test suites for the ECN implementation in ns, "test-all-ecn" and "test-all-ecn-ack". The validation test suite is described in the note ECN Implementations in the NS Simulator (postscript, PDF), and in the file "test-suite-ecn.txt" in the ns directory tcl/test.
The note on Ns Simulator Tests for Random Early Detection (RED) Gateways, October 1996, shows a simple simulation using ECN with RED queue management and Reno TCP. This test can be run in ns-2 with the command "ns test-suite-red.tcl ecn" in the directory tcl/test.
In ns-2, a simple simulation with one Tahoe TCP connection that is ECN-Capable and a second Tahoe TCP connection that is not ECN-Capable can be run with the command "ns test-suite-tcp.tcl ecn" in the directory tcl/test.
A brief discussion about the model of ECN in the ns-2 simulator.

Notes and Email:

Floyd, S., ECN and defenses against evil applications, December 1, 1997. Addendum, Sally Floyd, January 1998.
Floyd, S., ECN, TCP, and congestion on the ACK-path , December 16, 1997.
Blake, S., ECN and IPv4 Header Checksum Recalculation, February 12, 1998.
Floyd, S., ECN and Security Concerns, March 18, 1998.

Research on ECN and Related Work:

K. K. Ramakrishnan and Raj Jain, A Binary Feedback Scheme for Congestion Avoidance in Computer Networks, ACM Transactions on Computer Systems, Vol.8, No.2, pp. 158-181, May 1990.

Hari Balakrishnan, Venkata Padmanabhan, and Randy H. Katz, The Effects of Asymmetry on TCP Performance, Proc. Third ACM/IEEE MobiCom Conference, Budapest, Hungary, Sep 1997.
This paper includes a proposal for Ack Congestion Control, for responding to ACK packets that have had the ECN bit set.
Prasad Bagal, Shivkumar Kalyanaraman, Bob Packer, Comparative study of RED, ECN and TCP Rate Control, Technical Report, March 1999.
This paper evaluates RED, ECN, and a scheme which explicitly and transparently controls TCP rate (Packeteer TCP rate control). The paper indicates marked improvements in fairness moving from RED through ECN to TCP rate control.

Alternate proposals for the ECN field in IP, or for other ECN-like semantics:

We welcome research about proposals for alternate uses of the ECN field that could be associated with different diff-serv classes of traffic.

RFC 4774: Specifying Alternate Semantics for the Explicit Congestion Notification (ECN) Field, RFC 4774, Best Current Practice, November 2006. Local copy: (postscript, text).
"This document discusses some of the issues in defining alternate semantics for the ECN field, and specifies requirements for a safe co-existence in an Internet that could include routers that do not understand the defined alternate semantics."
L. Andrew, S. Hanly, S. Chan and T. Cui, Adaptive Deterministic Packet Marking, IEEE Comm. Letters, 10(11):790-792, Nov 2006.
"An efficient method is presented for signaling link price information using single-bit marks... A marking scheme with improved compatibility with current ECN (RFC 3168) is also proposed."
Nandita Dukkipati and Nick McKeown, Rate Control Protocol, 2005-2006.
"RCP has two components: (1) End-host congestion control layer that sits between IP and TCP/UDP... (2) Each router maintains a single fair-share rate per link. Each packet carries the rate of the bottleneck link."
Y. Xia, L. Subramanian, I. Stoica, and S. Kalyanaraman, One More Bit Is Enough, SIGCOMM 2005.
"In this paper, we design and implement a simple, low-complexity protocol, called Variable-structure congestion Control Protocol (VCP), that leverages only the existing two ECN bits for network congestion feedback, and yet achieves comparable performance to XCP, i.e., high utilization, low persistent queue length, negligible packet loss rate, and reasonable fairness."
A. Karnik and A. Kumar, Performance of TCP Congestion Control with Explicit Rate Feedback, CCR, February 2005.
"We consider a modification of TCP congestion control in which the congestion window is adapted to explicit bottleneck rate feedback; we call this RATCP (Rate Adaptive TCP)... We believe that ... ECN bit itself can be used to provide the rate feedback."
S Kunniyur and R. Srikant, End-to-End Congestion Control Schemes: Utility Functions, Random Losses and ECN Marks, IEEE/ACM Transactons on Networking, October 2003.
"ECN marking levels can be designed to nearly eliminate losses in the network by choosing the marking level independently for each node in the network."
S. Kunniyur, AntiECN Marking: A Marking Scheme for High Bandwidth Delay Connections, to appear in Proceedings, ICC, May 2003.
"We describe a simple scheme that uses feedback from underutilized high capacity links to allow a TCP connection to aggressively increase its sending rate. The feedback is in the form of a single bit in the packet header and is given per packet."
Arjan Durresi, Mukundan Sridharan, Chunlei Liu, Mukul Goyal and Raj Jain, Multilevel Explicit Congestion Notification, presented at SCI2001.
This paper proposes "multilevel ECN, which conveys more accurate feedback information about the network congestion status than the current two levels ECN".
R.J. Gibbens and F.P. Kelly, Resource pricing and the evolution of congestion control, Automatica 35, 1999.
This paper argues that ``by appropriately marking packets at overloaded resources and by charging a fixed small amount for each mark received, end-nodes are provided with the necessary information and the correct incentive to use the network efficiently.''
Microsoft Research's web page on Congestion Pricing and a Distributed Game also uses ECN for shadow prices which convey the marginal cost of congestion back to the user.
Koenraad Lavens, Peter Key and Derek McAuley, An ECN-based End-to-end Congestion-control Framework: Experiments and Evaluation.
Steven Low and others, Optimization Flow Control.
This page outlines a modified version of Optimization Flow Control that uses packet marking to convey to a source the information it needs to optimally adjust its rate.
S. Kalyanaraman, S. Arora, K. Wanglee, G. Guarriello and D. Harrison, A One-bit Feedback Enhanced Differentiated Services Architecture, work-in-progress, Apr. 1998.
This document proposes the use of one bit in the DS-byte to facilitate ECN-type control in the differentiated services architecture.
Teunis J. Ott, ECN Protocols and the TCP Paradigm, May 1999 (PDF).

ECN with MPLS:

ECN has not yet been standardized for use with MPLS, but see the draft below by Shayman and Jaeger. I am not aware of any efforts to standardize ECN for use with GRE.

M. Shayman and R. Jaeger, Using ECN to Signal Congestion Within an MPLS Domain. Internet-draft draft-shayman-mpls-ecn-00.txt, November 2000.
"The purpose of the present draft is to propose a mechanism whereby ECN can be used to provide congestion notifications to ingress LSRs when MPLS tunnels become congested. At the same time, the ability to provide end-to-end congestion notifications is retained."
Ramakrishnan, Floyd, and Davie, A Proposal to Incorporate ECN in MPLS, internet-draft draft-mpls-ecn-00.txt, June, 1999 (now expired). Local copy: ( text).
This draft has expired, and is not being pursued further.

Active Queue Management and ECN

The RED Web Page.

Talks:

Talk on A Proposal to add Explicit Congestion Notification (ECN) to IPv6 and to TCP (postscript, pdf), Munich IETF, Sally floyd, joint work with K. K. Ramakrishnan, August 11-15, 1997.
Second talk on A Proposal to add Explicit Congestion Notification (ECN) to IPv6 and to TCP (postscript, pdf), TCP Implementors Working Group, Washington IETF, December 8, 1997.

Open issues for ECN:

What are the quantitative benefits of ECN for TCP? What new dynamics does ECN add to TCP, in terms of competition between ECN-capable and non-ECN-capable traffic, the robustness of ECN in the presence of dropped ACK packets, performance with multiple congested gateways, etc.?
What are the benefits of ECN for rate-adaptive realtime traffic with fixed or adaptive playback times? What would be the risks in adding ECN to the UDP API, for ECN-capable traffic applications that use end-to-end congestion control? Would "lying about ECN-capability" be significantly more attractive to misbehaving users than simply adding another layer of FEC to an application without end-to-end congestion control?
What are the benefits of ECN in reliable multicast, for sessions using end-to-end congestion control? What are the relative benefits and costs of ECN relative to simply using more FEC?

Approaches in other architectures: FECN, BECN, and EFCI

FECN and BECN are Explicit Congestion Notification mechanisms for Frame Relay, based on the instantaneous queue size (as opposed to the average queue size).
ATM's Explicit Forward Congestion Indication (EFCI).

Archival documents:

Floyd, S., Experiments using ECN bits in the IPv6 and IPv4 headers, January 1998.
Floyd, S., Implementing ECN in TCP, January 29, 1998.
Email on current status, Sally Floyd and K. K. Ramakrishnan, November 19, 1997.
Floyd, S., ECN - one bit or two?, November 19, 1997.
Floyd, S., ECN vs. Source Quench, November 20, 1997. Addendum on Source Quench, May 7, 1998.

Return to [ Sally Floyd] [ K. K. Ramakrishnan].
Thanks for Simon Leinen and Reiner Ludwig for contributions to these web pages.
Last modified: June 2009. Links checked March 2008.