Terabit Ethernet Explained

Terabit Ethernet (TbE) is Ethernet with speeds above 100 Gigabit Ethernet. The 400 Gigabit Ethernet (400G, 400GbE) and 200 Gigabit Ethernet (200G, 200GbE)^[1] standard developed by the IEEE P802.3bs Task Force using broadly similar technology to 100 Gigabit Ethernet^{[2] [3]} was approved on December 6, 2017.^{[4] [5]} On February 16, 2024 the 800 Gigabit Ethernet (800G, 800GbE) standard developed by the IEEE P802.3df Task Force was approved.^[6]

The Optical Internetworking Forum (OIF) has already announced five new projects at 112 Gbit/s which would also make 4th generation (single-lane) 100 GbE links possible.^[7] The IEEE P802.3df Task Force started work in January 2022 to standardize and Ethernet. ^[8] In November 2022 the IEEE 802.3df project objectives were split in two, with 1.6T and 200G/lane work being moved to the new IEEE 802.3dj project. The timeline for the 802.3dj project indicates completion in July 2026. ^[9]

History

Facebook and Google, among other companies, have expressed a need for TbE.^[10] While a speed of is achievable with existing technology, would require different technology.^{[2] [11]} Accordingly, at the IEEE Industry Connections Higher Speed Ethernet Consensus group meeting in September 2012, 400 GbE was chosen as the next generation goal.^[2] Additional 200 GbE objectives were added in January 2016.

The University of California, Santa Barbara (UCSB) attracted help from Agilent Technologies, Google, Intel, Rockwell Collins, and Verizon Communications to help with research into next generation Ethernet.^[12]

As of early 2016, chassis/modular based core router platforms from Cisco, Juniper and other major manufacturers support full duplex data rates per slot. One, two and four port 100 GbE and one port 400 GbE line cards are presently available. As of early 2019, 200 GbE line cards became available after 802.3cd standard ratification.^{[13] [14]} In 2020 the Ethernet Technology Consortium announced a specification for 800 Gigabit Ethernet.^[15]

200G Ethernet uses PAM4 signaling which allows 2 bits to be transmitted per clock cycle, but at a higher implementation cost.^[16] Cisco introduced an 800G Ethernet switch in 2022.^[17] In 2024, Nokia routers with 800G Ethernet were deployed.^[18]

Standards development

The IEEE formed the "IEEE 802.3 Industry Connections Ethernet Bandwidth Assessment Ad Hoc", to investigate the business needs for short and long term bandwidth requirements.^{[19] [20] [21]}

IEEE 802.3's "400 Gb/s Ethernet Study Group" started working on the generation standard in March 2013.^[22] Results from the study group were published and approved on March 27, 2014. Subsequently, the IEEE 802.3bs Task Force^[23] started working to provide physical layer specifications for several link distances.^[24]

The IEEE 802.3bs standard was approved on December 6, 2017.^[4]

The IEEE 802.3cd standard was approved on December 5, 2018.

The IEEE 802.3cn standard was approved on December 20, 2019.

The IEEE 802.3cm standard was approved on January 30, 2020.

The IEEE 802.3cu standard was approved on February 11, 2021.

The IEEE 802.3ck and 802.3db standards were approved on September 21, 2022.

In November 2022 the IEEE 802.3df project objectives were split in two, with 1.6T and 200G/lane work being moved to the new IEEE 802.3dj project

• Original IEEE P802.3df Objectives
• Updated IEEE P802.3df Objectives to reduce scope to 800G Ethernet using 100G physical lanes
• IEEE P802.3dj Objectives for Ethernet and PHYs that employ lanes
• IEEE P802.3dj Objectives updated in May 2023 to include 200G/lane backplane Ethernet
• IEEE P802.3dj Objectives updated in January 2024 to include additional PHY types

The IEEE 802.3df standard was approved on February 16, 2024.

IEEE project objectives

Like all speeds since 10 Gigabit Ethernet, the standards support only full-duplex operation. Other objectives include:

1. Preserve the Ethernet frame format utilizing the Ethernet MAC
2. Preserve minimum and maximum frame size of current Ethernet standard
3. Support a bit error ratio (BER) of 10⁻¹³, which is an improvement over the 10⁻¹² BER that was specified for 10GbE, 40GbE, and 100GbE.
4. Support for OTN (transport of Ethernet across optical transport networks), and optional support for Energy-Efficient Ethernet (EEE).

802.3bs project

Define physical layer specifications supporting:

• Ethernet
  • at least 100 m over multi-mode fiber (400GBASE-SR16) using 16 parallel strands of fiber each at ^{[25] [26]}
  • at least 500 m over single-mode fiber (400GBASE-DR4) using 4 parallel strands of fiber each at ^{[27] [28]}
  • at least 2 km over single-mode fiber (400GBASE-FR8) using 8 parallel wavelengths (CWDM) each at ^{[29] [30]}
  • at least 10 km over single-mode fiber (400GBASE-LR8) using 8 parallel wavelengths (CWDM) each at ^{[30] [31]}
  • 8 and 16 lane chip-to-chip/chip-to-module electrical interfaces (400GAUI-8 and 400GAUI-16)
• Ethernet
  • at least 500 m over single-mode fiber (200GBASE-DR4) using 4 parallel strands of fiber each at ^{[32] [33]}
  • at least 2 km over single-mode fiber (200GBASE-FR4) using 4 parallel wavelengths (CWDM) each at
  • at least 10 km over single-mode fiber (200GBASE-LR4) using 4 parallel wavelengths (CWDM) each at
  • 4 or 8 lane chip-to-chip/chip-to-module electrical interfaces (200GAUI-4 and 200GAUI-8)

802.3cd project

• Define four-lane PHYs for operation over:
  • copper twin-axial cables with lengths up to at least 3 m (200GBASE-CR4).
  • printed circuit board backplane with a total channel insertion loss of ≤ 30 dB at 13.28125 GHz (200GBASE-KR4).
• Define PHYs for operation over MMF with lengths up to at least 100 m (200GBASE-SR4).

802.3ck project

• Ethernet
  • Define a two-lane Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on per lane optical signaling (200GAUI-2 C2M)
  • Define a two-lane Attachment Unit Interface (AUI) for chip-to-chip applications (200GAUI-2 C2C)
  • Define a two-lane PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (200GBASE-KR2)
  • Define a two-lane PHY for operation over twin axial copper cables with lengths up to at least 2 m (200GBASE-CR2)
• Ethernet

• • Define a four-lane Attachment Unit interface (AUI) for chip-to-module applications, compatible with PMDs based on per lane optical signaling (400GAUI-4 C2M)
  • Define a four-lane Attachment Unit Interface (AUI) for chip-to-chip applications (400GAUI-4 C2C)
  • Define a four-lane PHY for operation over electrical backplanes an insertion loss ≤ 28 dB at 26.56 GHz (400GBASE-KR4)
  • Define a four-lane PHY for operation over twin axial copper cables with lengths up to at least 2 m (400GBASE-CR4)

802.3cm project

• Ethernet
  • Define a physical layer specification supporting operation over 8 pairs of MMF with lengths up to at least 100 m (400GBASE-SR8)
  • Define a physical layer specification supporting operation over 4 pairs of MMF with lengths up to at least 100 m (400GBASE-SR4.2)

802.3cn project

• Ethernet
  • Provide a physical layer specification supporting operation over four wavelengths capable of at least 40 km of SMF (200GBASE-ER4) ^[34]
• Ethernet
  • Provide a physical layer specification supporting operation over eight wavelengths capable of at least 40 km of SMF (400GBASE-ER8)

802.3cu project

• Define a four-wavelength PHY for operation over SMF with lengths up to at least 2 km (400GBASE-FR4)
• Define a four-wavelength PHY for operation over SMF with lengths up to at least 6 km (400GBASE-LR4-6) ^[35]

802.3cw project

• Provide a physical layer specification supporting operation on a single wavelength capable of at least 80 km over a DWDM system (400GBASE-ZR)^[36] Dual polarization 16-state quadrature amplitude modulation (DP-16QAM) with coherent detection is proposed.^[37]

802.3db project

• Ethernet
  • Define a physical layer specification that supports operation over 2 pairs of MMF with lengths up to at least 50 m (200GBASE-VR2)
  • Define a physical layer specification that supports operation over 2 pairs of MMF with lengths up to at least 100 m (200GBASE-SR2)
• Ethernet
  • Define a physical layer specification that supports operation over 4 pairs of MMF with lengths up to at least 50 m (400GBASE-VR4)
  • Define a physical layer specification that supports operation over 4 pairs of MMF with lengths up to at least 100 m (400GBASE-SR4)

'IEEE P802.3db 100 Gb/s, 200 Gb/s, and 400 Gb/s Short Reach Fiber Task Force'

802.3df project

• Adds 800G Ethernet rate and specifies port types using existing 100G per lane technology

IEEE P802.3df Objectives for Ethernet and 400G and 800G PHYs using lanes

802.3dj project

• Adds Ethernet rate and specifies port types using new per lane technology.
• Objectives for Ethernet and 200, 400, and PHYs using lanes.^[38]

200G port types

Name	Standard	Status	Media	Connector	Transceiver Module	Reach in m	(⇆)	(→)	(→)	Notes
colspan="11"
	(CL120F/G)				N/A	0.25	4	N/A	2	PCBs
	(CL163)					1	4	N/A	2	PCBs; total insertion loss of ≤ 28 dB at 26.56 GHz
	(CL162)				N/A	2	4	N/A	2
	(CL167)	QSFP QSFP-DD SFP-DD112		4	1	2
	(CL167)	QSFP QSFP-DD SFP-DD112		4	1	2
colspan="11"
	(CL120F/G)					0.25	8	N/A	4	PCBs
	(CL163)					1	8	N/A	4	PCBs; total insertion loss of ≤ 28 dB at 26.56 GHz
	(CL162)				N/A	2	8	N/A	4	Data centres (in-rack)
	(CL167)	QSFP-DD		8	1	4
	(CL167)	QSFP-DD		8	1	4
	(CL124)				QSFP-DD OSFP		8	1	4
	(CL124)				QSFP-DD OSFP		8	1	4
					QSFP-DD OSFP		8	1	4
	(CL151)				QSFP-DD OSFP		2	4	4	Multi-Vendor Standard[39]
	(CL151)				QSFP-DD		2	4	4
					QSFP-DD		2	4	4	Multi-Vendor Standard[40]
400GBASE-ZR	802.3cw (CL155/156)				QSFP-DD OSFP		2	1	2	59.84375 Gigabaud (DP-16QAM)
colspan="11"

Notes and References

1. Web site: IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives.
2. Web site: Network boffins say Terabit Ethernet is TOO FAST: Sticking to 400Gb for now. .
3. Web site: On-board optics: beyond pluggables | Fibre Systems. www.fibre-systems.com.
4. Web site: [STDS-802-3-400G] IEEE P802.3bs Approved! ]. IEEE 802.3bs Task Force . 2017-12-14.
5. Web site: High-Speed Transmission Update: 200G/400G. 2016-07-18.
6. Web site: [802.3_B400G] IEEE P802.3df Standard Approved! ]. 2024-02-25 . www.ieee802.org.
7. Web site: OIF Launches CEI-112G Project for 100G Serial Electrical Links . Businesswire . 30 Aug 2016.
8. Web site: 802.3df Public Area .
9. Web site: Adopted IEEE P802.3dj Timeline . 2023-11-28 . www.ieee802.org.
10. Web site: Michael . Feldman . Facebook Dreams of Terabit Ethernet . February 3, 2010 . HPCwire . Tabor Communications, Inc..
11. Web site: Dare We Aim for Terabit Ethernet? . Light Reading . UBM TechWeb . Craig . Matsumoto . March 5, 2010.
12. Web site: The Terabit Ethernet Chase Begins . Craig Matsumoto . Light Reading . October 26, 2010 . December 15, 2011 .
13. Web site: Service Provider Network and Technology Services. Cisco.
14. Web site: Alcatel-Lucent boosts operator capacity to deliver big data and video over existing networks with launch of 400G IP router interface.
15. Web site: Rebranded Ethernet Technology Consortium Unveils 800 Gigabit Ethernet .
16. Web site: Micron Spills on GDDR6X: PAM4 Signaling For Higher Rates, Coming to NVIDIA's RTX 3090. Ryan. Smith. www.anandtech.com.
17. Web site: Cisco powers up Nexus switch, offers 800GB optic modules .
18. Web site: NL-ix deploys a range of Nokia service routers . 16 January 2024 .
19. IEEE Seeks Data on Ethernet Bandwidth Needs . Stephen Lawson . May 9, 2011 . PC World . May 23, 2013 .
20. Web site: IEEE Industry Connections Ethernet Bandwidth Assessment . IEEE 802.3 Ethernet Working Group . July 19, 2012 . 2015-03-01 .
21. Web site: Terabit Ethernet could be on its way . Perle . Max Burkhalter Brafton . May 12, 2011 . December 15, 2011 .
22. Web site: 400 Gb/s Ethernet Study Group . Group web site . IEEE 802.3 . May 23, 2013 .
23. Web site: IEEE P802.3bs 400 Gb/s Ethernet Task Force. www.ieee802.org.
24. Web site: Objectives . IEEE 802.3bs Task Force . Mar 2014 . 2015-03-01.
25. Web site: 100 m MMF draft proposal.
26. Web site: 400GBase-SR16 draft specifications.
27. Web site: IEEE 802.3 Ethernet Working Group Liaison letter to ITU-T Questions 6/15 and 11/15.
28. Web site: 400G-PSM4: A Proposal for the 500 m Objective using per Lane Signaling.
29. Web site: 400 Gb/s 8x50G PAM4 WDM 2km SMF PMD Baseline Specifications.
30. Web site: Baseline Proposal for 8 x 50G NRZ for 400GbE 2 km and 10 km PMD.
31. Web site: 400 GbE technical draft specifications.
32. http://www.ieee802.org/3/bs/public/16_03/NGOATH_3bs_Objectives_16_0316.pdf IEEE 802.3 NGOATH SG Adopted Changes to 802.3bs Project Objectives
33. Web site: IEEE 802.3bs 200/400 Gb/s Ethernet (Standards Informant).
34. Web site: Adopted Objectives . www.ieee802.org.
35. Web site: Approved Objectives . www.ieee802.org.
36. Web site: IEEE P802.3cw Adopted Objectives . www.ieee802.org.
37. Web site: Review of Decisions in P802.3ct related to P802.3cw . www.ieee802.org.
38. https://www.ieee802.org/3/dj/projdoc/objectives_P802d3dj_240314.pdf IEEE P802.3dj
39. Web site: Nowell . Mark . 400G-FR4 Technical Specification . 100glambda.com . 100G Lambda MSA Group . 26 May 2021.
40. Web site: Nowell . Mark . 400G-LR4-10 Technical Specification . 100glambda.com . 100G Lambda MSA Group . 26 May 2021.