Optimizing Power Consumption in High-End Routers

Introduction

The last few decades have seen exponential growth in the bandwidths of high-end routers and switches. As the bandwidths of these systems increased, so did the power consumption. To reduce the carbon footprint and keep power delivery and cooling costs low, it's crucial to minimize the energy these systems consume.

In this article, I examine various components inside a high-end router and how they contribute to overall power consumption. I'll also explore deeper into the techniques used by high-end networking ASIC vendors to optimize the power per gigabit per second of bandwidth.

This article serves as an excellent introduction for beginners and a refresher for networking enthusiasts.

High-End Routers - Basics

These typically come in two form factors - standalone or modular systems. A standalone router is typically a 1RU (rack unit) to 3 RU high box with a fixed number of ports in its front panel. It is mostly used in small to medium-sized enterprise networks or inside data centers.

As networking ASICs pack more and more bandwidth, the throughputs of these standalone systems are reaching upwards of 14.4Tbps. A 14.4Tbps system optimized for 400G port density would require the front panel to accommodate 36 x 400G ports, which could take up a majority of the front panel area. Routers larger than 14.4Tbps would often require 800G optics to saturate the system bandwidth fully.  

A modular system consists of line cards and fabric cards. Line cards contain one or two networking ASICs that receive traffic from the front panel network ports. These ASICs can talk to all the switch fabric cards in the backplane through high-speed serdes and backplane connectors. This provides any-to-any connectivity where a network port from a line card can send/receive traffic from any other line card in the system. These systems usually come in 4-20 slot configurations. They have a much larger scale and allow flexibility for the customers to upgrade the bandwidth by purchasing line cards based on their needs. It is not uncommon to find line cards exceeding 14.4Tbps density these days. With 8-slot chassis, this translates to 115 Tbps of system bandwidth! At these scales, delivering power to various components inside the line and fabric cards and cooling (removing the heat generated by these components) is a challenge. 

Router Components

To better understand router power, it is important to comprehend the functions and power requirements of different components within the system as they collectively contribute to the total power.

Front Panel/Optical Modules 

There are optical cages near the front panel for connecting to optical modules. These optical modules carry network traffic to and from the system. Optical modules consume a significant amount of power at higher speeds. Power consumed by these modules varies widely depending on the type of module and the reach (how long the optical signals can travel without signal degradation). In a 14.4Tbps line card with 36 x 400G ports, the optical modules themselves could consume between 500-860W power when fully populated and loaded. Similarly, a 28.8Tbps line card with 36 x 800G ports would require ~1100W power for the optical modules. 

Reducing the cost/power during optical transmission has been a hot topic for research past decade. There is continuous innovation on that front, with some vendors offering Silicon Photonics transceivers that integrate discrete components in a photonic integrated circuit to reduce area/cost and power. PAM4 signaling for higher data rates, low power mode when not actively transmitting, and improved lasers, photodiodes, modulators, and DSP circuitry have all contributed to power reduction in optics. As a result, when going from 400G to 800G optics for a specific reach, the power only increased by 1.5x, as shown in the table above.

(article continues at link)