Jul
08

Generally, flow-control is a mechanics allowing the receiving party of a connection to control the rate of the sending party. You may see many different implementations of flow-control technologies at different levels of OSI model (e.g. XON/XOFF for RS232, TCP sliding window, B2B credits for Fibre Channel, FECN/BECN for Frame-Relay, ICMP source-quench message, etc). Flow-Control allows for explicit feedback loop and theoretically implementing loss-less networks that avoid congestion.

For the original Ethernet technology on half-duplex connections there was no possibility of implementing explicit flow control, since only one side could send frames at time. However, you may still remember the Cisco’s so-called “back-pressure” feature on some of the older switches, e.g. Cisco Catalyst 1924. The idea was that switch may send barrage of dummy frames on a half-duplex link, effectively preventing the attached station from transmitting information at given moments of time.

Things changed with full-duplex point-to-point connections. The idea of IEEE 802.3X flow-control was to allow a station on a point-to-point link sending a special “PAUSE” frame (destined to a reserved multicast MAC address 01:80:C2:00:00:01). This frame would signal the other end of the connection to pause transmission for a certain amount of time, which was is specified in the frame. (Note that the PAUSE frame uses MAC LLC encapsulation). The attached device could be a host station or another switch – it does not matter. Upon receiving the PAUSE frame, the sender should stop the traffic flow and either buffer the packets or drop them. The paused switch may in turn propagate the PAUSE frame further upstream toward the sending node, and thus congestion notification will propagate end-to-end.

There were two main problems with the 802.3x flow-control. Firstly, there were interoperability problems with different vendors implementing the PAUSE functionality. Secondly, the PAUSE frame mechanics was too simple to implement granular policies, for example based on per-VLAN or per-class basis. This being said, mixing PAUSE functionality with per-class services was impossible – it would stop sending of all traffic, regardless of QoS marking. This is why it’s typically advised to turn off 802.3X flow control whether you enable MLS QoS on a Cisco switch be it 3550 or 3560. By default flowcontrol is disabled and you can only enable a Cisco switch to receive PAUSE frames, but not to send them.

Use the following commands to set up and verify your current flow-control settings:

SW1:
interface GigabitEthernet0/1
 flowcontrol receive on

giga1-sw-ats64#show interfaces gigabitEthernet 0/1 flowcontrol
Port       Send FlowControl  Receive FlowControl  RxPause TxPause
           admin    oper     admin    oper
---------  -------- -------- -------- --------    ------- -------
Gi0/1      Unsupp.  Unsupp.  on       on          0       0

Even though the simple 802.3X flow-control idea did not work out well, the evolving body of DCB (data-center bridging) standards uses similar, yet more granular mechanism to implement lossless Ethernet transmissions. Lossless Ethernet is important for storage transportation protocols such as FCoE and is one of the cornerstones of the evolving data-center Ethernet standard.

About Petr Lapukhov, 4xCCIE/CCDE:

Petr Lapukhov's career in IT begain in 1988 with a focus on computer programming, and progressed into networking with his first exposure to Novell NetWare in 1991. Initially involved with Kazan State University's campus network support and UNIX system administration, he went through the path of becoming a networking consultant, taking part in many network deployment projects. Petr currently has over 12 years of experience working in the Cisco networking field, and is the only person in the world to have obtained four CCIEs in under two years, passing each on his first attempt. Petr is an exceptional case in that he has been working with all of the technologies covered in his four CCIE tracks (R&S, Security, SP, and Voice) on a daily basis for many years. When not actively teaching classes, developing self-paced products, studying for the CCDE Practical & the CCIE Storage Lab Exam, and completing his PhD in Applied Mathematics.

Find all posts by Petr Lapukhov, 4xCCIE/CCDE | Visit Website


You can leave a response, or trackback from your own site.

13 Responses to “802.3x Flow-Control”

 
  1. Dirk-Jan van Helmond says:

    Nice article Petr! Recently with all the FCoE and datacenter ethernet fuzz and the demand for a lossless fabric, PAUSE frames are back. The 802.1Qbb workinggroup is working on a per-priority PAUSE mechanism, overcoming the problems you’ve described.

  2. Hi Petr,

    Thanks for the great post.

    I once ‘broke’ a network by including “flowcontrol receive desired” in a macro applied to the configuration of all ‘uplinks’.

    Please can you clarify something for me –> Can ethernet flowcontrol ‘break’ TCP’s mechanisms for flow control in any way?
    Somebody once mentioned it to me and it had me thinking…

    Richard

  3. Jody says:

    We ran into an issue with our 6500 (which was configured by default for send flow control to be “desired”). When interfacing with a Nortel Passport, the link wouldn’t pass traffic. Once we disabled flow control, things worked fine.

  4. To: Richard Bannister

    TCP uses somewhat sophisticated “end-to-end” flow-control model based on sliding-window concept. This model implicitly accounts for the RTT between the endpoints, available end-to-end bandwidth and buffer space utilization at each endpoint. Note that TCP interprets packet loss as a sign of congestion in the transmission path.

    802.3x Flow-Control is based on a “hop-by-hop” local concept, e.g. it only takes in account the loading of the link between a stations and a switch. Moreover, it’s up to NIC driver to decide whether to send the PAUSE frame and how long to hold the locally connected sender, and this process probably does not involve consulting with TCP/IP stack.

    Therefore, as you can see, 802.3x flow-control may affect TCP performance in the following manner:

    1) Changing the effective RTT between the endpoints by introducing artificial delay
    2) Causig excessive packet drops and resulting in TCP slowing the sending rate
    3) Possibly introducing TCP flows “synchronization” issue due to tail-end dropping behavior

    However, while this surely affects TCP end-to-end flow control, it’s not necessarily affecting it in a bad way. On contrary, it may even improve TCP performance – but this should be a subject to a separate study.

    The following IEEE abstract demonstrates that people have been concerned with this interaction and even made some studies already.

    http://ieeexplore.ieee.org/iel5/9789/30874/01431675.pdf

  5. Hi Petr,

    Thanks very much for going to the effort it must have taken to put your answer together.

    Your explanation is very easy to understand (and ‘picture’), I will have a look at the pdf (url) this evening whilst I’m studying.

    Many thanks again,

    Richard

  6. [...] Internetwork Expert Blog – Petr discusses the history of 802.3x flow control. [...]

  7. [...] of Internetwork Expert posted a good article about why 802.3X pause frame should be disabled here Filed Under: Blog, DesignTagged: data centre, Design, Ethernet, [...]

  8. Sai Lwin Thu says:

    Thanks petr, nice post.

  9. Olivier says:

    Hi Peter,

    Thanks for this post but I still have a question regarding this feature.
    I installed an EtherConnect Line 55Mbps between one Cisco Switch and an Huawei Switch.
    I heard from the Telecom Technician that I should enable FlowControl on both sides in order to limit the rate of Traffic below 55 Mbps instead of the physical Port speed (100Mbps). Is it correct ? Or does TCP already makes this job ?

    Thanks in advance,

    Regards,
    Olivier

  10. RajaSekhar says:

    Is there a scenario, if a switch receives a PAUSE frame, does it drop packets till that PAUSE timer expiry.

    Thanks,

  11. nice says:

    thank you very much

 

Leave a Reply

Categories

CCIE Bloggers