Terabit Ethernet Explained

Terabit Ethernet (TbE) is Ethernet with speeds above 100 Gigabit Ethernet. The 400 Gigabit Ethernet (400G, 400GbE) and 200 Gigabit Ethernet (200G, 200GbE)^[1] standard developed by the IEEE P802.3bs Task Force using broadly similar technology to 100 Gigabit Ethernet^{[2] [3]} was approved on December 6, 2017.^{[4] [5]} On February 16, 2024 the 800 Gigabit Ethernet (800G, 800GbE) standard developed by the IEEE P802.3df Task Force was approved.^[6]

The Optical Internetworking Forum (OIF) has already announced five new projects at 112 Gbit/s which would also make 4th generation (single-lane) 100 GbE links possible.^[7] The IEEE P802.3df Task Force started work in January 2022 to standardize 800 Gbit/s and 1.6 Tbit/s Ethernet. ^[8] In November 2022 the IEEE 802.3df project objectives were split in two, with 1.6T and 200G/lane work being moved to the new IEEE 802.3dj project. The timeline for the 802.3dj project indicates completion in July 2026. ^[9]

History

Facebook and Google, among other companies, have expressed a need for TbE.^[10] While a speed of 400 Gbit/s is achievable with existing technology, 1 Tbit/s (1000 Gbit/s) would require different technology.^{[2] [11]} Accordingly, at the IEEE Industry Connections Higher Speed Ethernet Consensus group meeting in September 2012, 400 GbE was chosen as the next generation goal.^[2] Additional 200 GbE objectives were added in January 2016.

The University of California, Santa Barbara (UCSB) attracted help from Agilent Technologies, Google, Intel, Rockwell Collins, and Verizon Communications to help with research into next generation Ethernet.^[12]

As of early 2016, chassis/modular based core router platforms from Cisco, Juniper and other major manufacturers support 400 Gbit/s full duplex data rates per slot. One, two and four port 100 GbE and one port 400 GbE line cards are presently available. As of early 2019, 200 GbE line cards became available after 802.3cd standard ratification.^{[13] [14]} In 2020 the Ethernet Technology Consortium announced a specification for 800 Gigabit Ethernet.^[15]

200G Ethernet uses PAM4 signaling which allows 2 bits to be transmitted per clock cycle, but at a higher implementation cost.^[16] Cisco introduced an 800G Ethernet switch in 2022.^[17] In 2024, Nokia routers with 800G Ethernet were deployed.^[18]

Standards development

The IEEE formed the "IEEE 802.3 Industry Connections Ethernet Bandwidth Assessment Ad Hoc", to investigate the business needs for short and long term bandwidth requirements.^{[19] [20] [21]}

IEEE 802.3's "400 Gb/s Ethernet Study Group" started working on the 400 Gbit/s generation standard in March 2013.^[22] Results from the study group were published and approved on March 27, 2014. Subsequently, the IEEE 802.3bs Task Force^[23] started working to provide physical layer specifications for several link distances.^[24]

The IEEE 802.3bs standard was approved on December 6, 2017.^[4]

The IEEE 802.3cd standard was approved on December 5, 2018.

The IEEE 802.3cn standard was approved on December 20, 2019.

The IEEE 802.3cm standard was approved on January 30, 2020.

The IEEE 802.3cu standard was approved on February 11, 2021.

The IEEE 802.3ck and 802.3db standards were approved on September 21, 2022.

In November 2022 the IEEE 802.3df project objectives were split in two, with 1.6T and 200G/lane work being moved to the new IEEE 802.3dj project

• Original IEEE P802.3df Objectives
• Updated IEEE P802.3df Objectives to reduce scope to 800G Ethernet using 100G physical lanes
• IEEE P802.3dj Objectives for 1.6 Tbit/s Ethernet and PHYs that employ 200 Gbit/s lanes
• IEEE P802.3dj Objectives updated in May 2023 to include 200G/lane backplane Ethernet
• IEEE P802.3dj Objectives updated in January 2024 to include additional PHY types

The IEEE 802.3df standard was approved on February 16, 2024.

IEEE project objectives

Like all speeds since 10 Gigabit Ethernet, the standards support only full-duplex operation. Other objectives include:

1. Preserve the Ethernet frame format utilizing the Ethernet MAC
2. Preserve minimum and maximum frame size of current Ethernet standard
3. Support a bit error ratio (BER) of 10⁻¹³, which is an improvement over the 10⁻¹² BER that was specified for 10GbE, 40GbE, and 100GbE.
4. Support for OTN (transport of Ethernet across optical transport networks), and optional support for Energy-Efficient Ethernet (EEE).

802.3bs project

Define physical layer specifications supporting:

• 400 Gbit/s Ethernet
  • at least 100 m over multi-mode fiber (400GBASE-SR16) using 16 parallel strands of fiber each at 25 Gbit/s^{[25] [26]}
  • at least 500 m over single-mode fiber (400GBASE-DR4) using 4 parallel strands of fiber each at 100 Gbit/s^{[27] [28]}
  • at least 2 km over single-mode fiber (400GBASE-FR8) using 8 parallel wavelengths (CWDM) each at 50 Gbit/s^{[29] [30]}
  • at least 10 km over single-mode fiber (400GBASE-LR8) using 8 parallel wavelengths (CWDM) each at 50 Gbit/s^{[30] [31]}
  • 8 and 16 lane chip-to-chip/chip-to-module electrical interfaces (400GAUI-8 and 400GAUI-16)
• 200 Gbit/s Ethernet
  • at least 500 m over single-mode fiber (200GBASE-DR4) using 4 parallel strands of fiber each at 50 Gbit/s^{[32] [33]}
  • at least 2 km over single-mode fiber (200GBASE-FR4) using 4 parallel wavelengths (CWDM) each at 50 Gbit/s
  • at least 10 km over single-mode fiber (200GBASE-LR4) using 4 parallel wavelengths (CWDM) each at 50 Gbit/s
  • 4 or 8 lane chip-to-chip/chip-to-module electrical interfaces (200GAUI-4 and 200GAUI-8)

802.3cd project

• Define four-lane 200 Gbit/s PHYs for operation over:
  • copper twin-axial cables with lengths up to at least 3 m (200GBASE-CR4).
  • printed circuit board backplane with a total channel insertion loss of ≤ 30 dB at 13.28125 GHz (200GBASE-KR4).
• Define 200 Gbit/s PHYs for operation over MMF with lengths up to at least 100 m (200GBASE-SR4).

802.3ck project

• 200 Gbit/s Ethernet
  • Define a two-lane 200 Gbit/s Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on 100 Gbit/s per lane optical signaling (200GAUI-2 C2M)
  • Define a two-lane 200 Gbit/s Attachment Unit Interface (AUI) for chip-to-chip applications (200GAUI-2 C2C)
  • Define a two-lane 200 Gbit/s PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (200GBASE-KR2)
  • Define a two-lane 200 Gbit/s PHY for operation over twin axial copper cables with lengths up to at least 2 m (200GBASE-CR2)
• 400 Gbit/s Ethernet

• • Define a four-lane 400 Gbit/s Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on 100 Gbit/s per lane optical signaling (400GAUI-4 C2M)
  • Define a four-lane 400 Gbit/s Attachment Unit Interface (AUI) for chip-to-chip applications (400GAUI-4 C2C)
  • Define a four-lane 400 Gbit/s PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (400GBASE-KR4)
  • Define a four-lane 400 Gbit/s PHY for operation over twin axial copper cables with lengths up to at least 2 m (400GBASE-CR4)

802.3cm project

• 400 Gbit/s Ethernet
  • Define a physical layer specification supporting 400 Gbit/s operation over 8 pairs of MMF with lengths up to at least 100 m (400GBASE-SR8)
  • Define a physical layer specification supporting 400 Gbit/s operation over 4 pairs of MMF with lengths up to at least 100 m (400GBASE-SR4.2)

802.3cn project

• 200 Gbit/s Ethernet
  • Provide a physical layer specification supporting 200 Gbit/s operation over four wavelengths capable of at least 40 km of SMF (200GBASE-ER4) ^[34]
• 400 Gbit/s Ethernet
  • Provide a physical layer specification supporting 400 Gbit/s operation over eight wavelengths capable of at least 40 km of SMF (400GBASE-ER8)

802.3cu project

• Define a four-wavelength 400 Gbit/s PHY for operation over SMF with lengths up to at least 2 km (400GBASE-FR4)
• Define a four-wavelength 400 Gbit/s PHY for operation over SMF with lengths up to at least 6 km (400GBASE-LR4-6) ^[35]

802.3cw project

• Provide a physical layer specification supporting 400 Gbit/s operation on a single wavelength capable of at least 80 km over a DWDM system (400GBASE-ZR)^[36] Dual polarization 16-state quadrature amplitude modulation (DP-16QAM) with coherent detection is proposed.^[37]

802.3db project

• 200 Gbit/s Ethernet
  • Define a physical layer specification that supports 200 Gbit/s operation over 2 pairs of MMF with lengths up to at least 50 m (200GBASE-VR2)
  • Define a physical layer specification that supports 200 Gbit/s operation over 2 pairs of MMF with lengths up to at least 100 m (200GBASE-SR2)
• 400 Gbit/s Ethernet
  • Define a physical layer specification that supports 400 Gbit/s operation over 4 pairs of MMF with lengths up to at least 50 m (400GBASE-VR4)
  • Define a physical layer specification that supports 400 Gbit/s operation over 4 pairs of MMF with lengths up to at least 100 m (400GBASE-SR4)

'IEEE P802.3db 100 Gb/s, 200 Gb/s, and 400 Gb/s Short Reach Fiber Task Force'

802.3df project

• Adds 800G Ethernet rate and specifies port types using existing 100G per lane technology

IEEE P802.3df Objectives for 800 Gbit/s Ethernet and 400G and 800G PHYs using 100 Gbit/s lanes

802.3dj project

• Adds 1.6T Ethernet rate and specifies port types using new 200G per lane technology

IEEE P802.3dj Objectives for 1.6 Tbit/s Ethernet and 200G, 400G 800 Gb/s, and 1.6 Tb/s PHYs using 200 Gbit/s lanes

200G port types

Name	Standard	Status	Media	Connector	Transceiver Module	Reach in m	(⇆)	(→)	(→)	Notes
colspan="11"
	(CL120F/G)				N/A	0.25	4	N/A	2	PCBs
	(CL163)					1	4	N/A	2	PCBs; total insertion loss of ≤ 28 dB at 26.56 GHz
	(CL162)				N/A	2	4	N/A	2
	(CL167)	QSFP QSFP-DD SFP-DD112		4	1	2
	(CL167)	QSFP QSFP-DD SFP-DD112		4	1	2
colspan="11"

Notes and References

1. Web site: IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives.
2. Web site: Network boffins say Terabit Ethernet is TOO FAST: Sticking to 400Gb for now. .
3. http://www.fibre-systems.com/feature/board-optics-beyond-pluggables On-board optics: beyond pluggables
4. Web site: [STDS-802-3-400G] IEEE P802.3bs Approved! ]. IEEE 802.3bs Task Force . 2017-12-14.
5. Web site: High-Speed Transmission Update: 200G/400G. 2016-07-18.
6. Web site: [802.3_B400G] IEEE P802.3df Standard Approved! ]. 2024-02-25 . www.ieee802.org.
7. Web site: OIF Launches CEI-112G Project for 100G Serial Electrical Links . Businesswire . 30 Aug 2016.
8. Web site: 802.3df Public Area .
9. https://www.ieee802.org/3/dj/projdoc/timeline_3dj_231128.pdf
10. Web site: Michael . Feldman . Facebook Dreams of Terabit Ethernet . February 3, 2010 . HPCwire . Tabor Communications, Inc..
11. Web site: Dare We Aim for Terabit Ethernet? . Light Reading . UBM TechWeb . Craig . Matsumoto . March 5, 2010.
12. Web site: The Terabit Ethernet Chase Begins . Craig Matsumoto . Light Reading . October 26, 2010 . December 15, 2011 .
13. Web site: Cisco 4 x 100 Gbit/s interface.
14. Web site: Alcatel-Lucent boosts operator capacity to deliver big data and video over existing networks with launch of 400G IP router interface.
15. Web site: Rebranded Ethernet Technology Consortium Unveils 800 Gigabit Ethernet .
16. Web site: Micron Spills on GDDR6X: PAM4 Signaling For Higher Rates, Coming to NVIDIA's RTX 3090. Ryan. Smith. www.anandtech.com.
17. Web site: Cisco powers up Nexus switch, offers 800GB optic modules .
18. Web site: NL-ix deploys a range of Nokia service routers . 16 January 2024 .
19. IEEE Seeks Data on Ethernet Bandwidth Needs . Stephen Lawson . May 9, 2011 . PC World . May 23, 2013 .
20. Web site: IEEE Industry Connections Ethernet Bandwidth Assessment . IEEE 802.3 Ethernet Working Group . July 19, 2012 . 2015-03-01 .
21. Web site: Terabit Ethernet could be on its way . Perle . Max Burkhalter Brafton . May 12, 2011 . December 15, 2011 .
22. Web site: 400 Gb/s Ethernet Study Group . Group web site . IEEE 802.3 . May 23, 2013 .
23. http://www.ieee802.org/3/bs/ IEEE 802.3bs Task Force
24. Web site: Objectives . IEEE 802.3bs Task Force . Mar 2014 . 2015-03-01.
25. http://www.ieee802.org/3/bs/public/14_11/king_3bs_02a_1114.pdf 100 m MMF draft proposal
26. Web site: 400GBase-SR16 draft specifications.
27. http://www.ieee802.org/3/bs/public/15_07/trowbridge_3bs_03_0715.pdf IEEE 802.3 Ethernet Working Group Liaison letter to ITU-T Questions 6/15 and 11/15
28. http://www.ieee802.org/3/bs/public/15_01/welch_3bs_01a_0115.pdf 400G-PSM4: A Proposal for the 500 m Objective using 100 Gbit/s per Lane Signaling
29. http://www.ieee802.org/3/bs/public/15_07/cole_3bs_01a_0715.pdf 400 Gb/s 8x50G PAM4 WDM 2km SMF PMD Baseline Specifications
30. http://www.ieee802.org/3/bs/public/15_01/kojima_3bs_01a_0115.pdf Baseline Proposal for 8 x 50G NRZ for 400GbE 2 km and 10 km PMD
31. Web site: 400 GbE technical draft specifications.
32. http://www.ieee802.org/3/bs/public/16_03/NGOATH_3bs_Objectives_16_0316.pdf IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives
33. http://blog.siemon.com/standards/category/ieee/httpblog-siemon-comstandardsp941 IEEE 802.3bs 200/400 Gb/s Ethernet (Standards Informant)
34. http://www.ieee802.org/3/cn/proj_doc/3cn_Objectives_181113.pdf
35. https://www.ieee802.org/3/cu/Objectives_Approved_Sept_2019.pdf
36. http://www.ieee802.org/3/cw/proj_doc/3cw_Objectives_190911.pdf
37. https://www.ieee802.org/3/ct/public/tf_interim/20_0227/dambrosia_3cw_01_200227.pdf