High-speed backplane interconnect

Vladimir Stojanovic
(with slides from J. Zerbe, P. Desai, R. Kollipara)
Outline

 Inside the router

 Backplane channel problem
 What can backplane designer do about it
 What can IC designer do about it
 Scaling the system to 10-100Tb/s
Inside the Router
Passive Switch Cards:
Line Cards:
Backplane 2 to 4 per System
8 to 16 per System

MEM

MEM

MEM

MEM
MEM

MEM

MEM

MEM
SerDes Crossbar

TM/
Optics SerDes MAC NPU Fabric
IF SerDes

 Past  Present
 OC-12  OC48
 622 MHz LVDS  2.5 Gbps serial
parallel
 10GigE
 GigE
 1.25 Gbps serial  XAUI (3.125 Gbps) serial
Serial Links in Networking Systems
Line Card:

MAC/ NPU/ Switch

Optics Framer TM Fabric IF

XAUI SPI4.2 CSIX Backplane

4, 3.125 Gbps Proprietary 8 to 16
Serial Links of 1-3.2Gbps
Serial Links
Switch Card:
32 to 64
Switch
Backplane
Crossbar
Serial Links
IC
(1-3.2 Gbps)
Backplane interconnect path

Package
to board
Chip transition
Line card Package
trace
Backplane connector Line card
Backplane trace via

Backplane via

 There are many components on the signal path,

potential source of problems
RaSer X Link Features
20 bit
Parallel Serializer Tx Tx Link TX PLL
Interface
1-10 Gbps

Eq
RefClk PLL RX

20 bit
Parallel Deserializer and Rx Rx Link
Interface CDR
1-10 Gbps

Eq
Process 0.13 CMOS
Power 40mW / Gb
Area 1mm2
2-PAM Range 2 – 6.4 Gb/s

4-PAM Range 5 – 10 Gb/s

I/O Driver Scheme (Example)

Vtt Vtt
50  50 

Zo = 50 
Tx Rx
Zo = 50 

 Impedance-controlled (CML) I/Os

 Integrated terminations
 Adjustable output-voltage/common mode
System Issues

 Goal – Increase Router Throughput

 Limitations
 Backplane channel
 Power
 Mechanical/Physical density constraints
 Backplane and linecard routing density
 Connector pin density
 Package I/O density
Outline

 Inside the router

 Backplane channel problem
 What can backplane designer do about it
 What can IC designer do about it
 Scaling the system to 10-100Tb/s
Backplane Component Effects

PCB only

PCB + Connectors

PCB, Connectors,
Via stubs & Devices
Deterministic Noise
0
Attenuation [dB]

pulse response
THROUGH
-10
0.8
-20

-30 0.6 Tsymbol=160ps

NEXT
-40 0.4
FEXT
-50
0.2
-60
0
0 2 4 6 8 10
frequency [GHz] 0 1 2 3
ns

 Inter-symbol interference
 Dispersion (skin-effect, dielectric loss) - short latency
 Reflections (impedance mismatches – connectors, via
stubs, device parasitics, package) – long latency
XTALK and reflections
Far-end XTALK (FEXT)

Desired signal

Reflections

Near-end XTALK (NEXT)

 Primary reflection sources are at the connector/backplane

transition
 Grouped in time – as a function of backplane length
Backplane channel variations
0

Attenuation [dB]
-10 9" FR4

-20

-30 26" FR4

-40
9" FR4,
-50 via stub

-60 26" FR4,

via stub
0 2 4 6 8 10
frequency [GHz]

 Variability in trace length, routing layer and via stub

 Significantly different transfer functions even within the same backplane
 Test Backplane Example
 Trace lengths: 1.5”, 9”, 14”, 20” and 32” FR4 Cross Section
 Effective number of signal layers: 13
 Effective number of total layers: 28

Dielectric material FR-4 Nelco Roger

6000 4350

Dielectric constant 4.2 4 3.6

Loss tangent (1 MHz) 0.016 0.005 0.0035
Loss tangent (1 GHz) 0.017 0.007 0.0035
Thickness 0.295" 0.299" 0.297"
Backdrilling - A Solution to the Stub
Effect
Stub Effect Eye Pattern Analysis
(2.5 Gbits/sec FR-4)

MAX STUB MIN STUB

Stub Effect Eye Pattern Analysis
(5.0 Gbits/sec FR-4)

MAX STUB MIN STUB

Stub Effect Eye Pattern Analysis
(12.0 Gbits/sec FR-4)

MAX STUB MIN STUB

Connector design

GBX
Teradyne
Connector Density

Teradyne’s GbX™ Connector

Differentia
Card Bandwidth/linear inch
l
Pitch (at 6.25 Gbps)
Pairs/inch
1.25" min.
5 pair 69 431 Gb
(30 mm)
1.00" min.
4 pair 55 343 Gb
(24.7mm)
.80" min.
3 pair 41 256 Gb
(20 mm)
.575" min.
2 pair 27.5 171 Gb
(14 mm)
Reducing Crosstalk within the
Connector
D/C Shield

Cross talk is
reduced in the
mating interface
by surrounding
each pair with a
ground shield

Mated pair B/P Shield

Backplane Connector Considerations

 Many connector types:

 Teradyne: VHDM, HSD, GbX, …
 Tyco: HS3, HMZd, …
 FCI: Metral 2000, 3000, 4000, …
 3M/Harting: HSHM, …
 ERNI: ERmetZd, ErmetXT, …
 Issues
 Loss, impedance profile, crosstalk, skew
 Foot print: routability, pin density, via impedance
 Single-ended and differential
 Press-fit and SMT
 10Gbps Test Package Design Example

 Ceramic BGA
 Wire-bonded
 4-Layer
 1 mm pitch

Source: Designed for Rambus by Kyocera

Example of a really good backplane

Works with simple 15” FR4

OC192 xcvr

20” Roger
Outline

 Inside the router

 Backplane channel problem
 What can backplane designer do about it
 What can IC designer do about it
 Scaling the system to 10-100Tb/s
Loss : Equalize to Flatten Response

 Channel is band-limited
 Equalization : boost high-frequencies relative to lower
frequencies
Receiver Linear Equalizer

 Amplifies high-frequencies
attenuated by the channel D
… D D
 Digital or Analog FIR filter WL
W1
 Issues WL-1
+
 Also amplifies noise!
 Precision
 Tuning delays (if analog)
 Setting coefficients
 Adaptive algorithms such as
LMS H(s)

freq
Transmitter Linear Equalizer

 Attenuates low-frequencies
 Need to be careful about output
amplitude : limited output power
 If you could make bigger swings
you would
 EQ really attenuates low-
frequencies to match high
frequencies
 Also FIR filter : D/A converter
 Can get better precision than Rx
 Issues H(s)

 How to set EQ weights?

 Doesn’t help loss at f freq
Transmit Linear Equalizer :
Single Bit Operation

0.7
Unequalized
Equalization Pulse
End of Line
0.5
Voltage

0.3

0.1

-0.1

-0.3
0.0 0.3 0.6 0.9 1.2
time (ns)
Decision Feedback Equalization (DFE)

 Don’t invert channel…

just remove ISI
Decision (slicer)
 Know ISI because already Feed-forward EQ

received symbols FIR filter

-
 Doesn’t amplify noise
 Requires a feed-forward FIR filter
equalizer for precursor Feed-back EQ

ISI
 Reshapes pulse to
eliminate precursor
DFE Example
Transmit and Receive Equalization

RX
DATA
TX
DATA

TAP SEL
LOGIC
3

 Transmit and receive equalizers are

combined to make a range restricted DFE
 Tx equalizer functions as the feed-forward filter
 Rx equalizer restricted in performance of loop
Tx & Rx Equalization Ranges

RX Equalizer
5-17 taps after main
Pick any 5 taps

TX Driver/Equalizer : 5 taps
1(pre)+1(main)+3(post)
Pulse Amplitude Modulation

 Binary (NRZ) is 2-PAM  4-PAM uses 4-levels to send

 2-PAM uses 2-levels to send 2 bits per symbol
one bit per symbol  Each level has 2 bit value
 Signaling rate = 2 x Nyquist  Signaling rate = 4 x Nyquist

00
0 0
01

11
1 1

10
 When Does 4-PAM Make Sense?
Nyquist Frequency (GHz)
0.0 1.0 2.0 3.0 4.0 5.0

-20db
|H(f)|

-40db

-60db

 First order : slope of S21

 3 eyes : 1 eye = 10db
 loss > 10db/octave : 4-PAM
should be considered
Example : 5Gbps Over 26” FR4
With No Equalization
Example : 5Gbps Over 26” FR4
Correct Tx Equalization
Example : 5Gbps Over 26” FR4
Under Equalized
Example : 5Gbps Over 26” FR4
Over Equalized
26” FR4 Bot 3.125Gbps, 2P noEQ
26” FR4 Bot 3.125Gbps, 2P w/EQ
26” FR4 Bot 6.4Gbps, 2P w/3G EQ
26” FR4 Bot 6.4Gbps, 2P w/EQ
26” FR4 Top 6.4Gbps, 2P w/EQ
26” FR4 Top 6.4Gbps, 4P w/EQ
26” Nelco6k-cb Top 10Gbps, 4P
26” Nelco6k-cb Top 6.4Gbps, 2P
Scaling the router throughput
 System (Tot. throughput ~2.5Tb/s)
 8-16 Line Cards
 <40Gbs / LC 3-6Gbs links
 40mW/Gbs Link power in 0.13um
 Speedup ~2x
 #links at switch card ~ 200
 Limitations
 #diff pairs at switch card 16*40Gbs/6Gbs*2*2 ~ 400
 Switch card power from links ~200*6Gbs*40mW/Gbs ~ 50W
 Connector density 50diff pairs/inch: (tot length=400/50= 8” )
 BP/LC routing pitch 0.050”
 Num. Layers (BP=13, LC=4) 100diff pairs/layer = 5” LC routing width
 Package ball pitch (1mm/200um)
Scaling the router throughput
 System (Tot. throughput ~100Tb/s)
 100 Line Cards
 1Tbs / LC 10Gbs links
 4mW/Gbs Link power in 0.065um
 Speedup ~1x
 #links at switch card ~ 10k
 Limitations
 #diff pairs at switch card 20k
 Switch card power from links in 0.13um
~10k*10Gbs*4mW/Gbs ~ 400W
 Connector density 50diff pairs/inch: (tot length=20k/50= 400” )
 BP/LC routing pitch 0.050”
 Num. Layers (BP=13, LC=4) 5k diff pairs/layer = 250” LC routing width
 Package ball pitch (1mm/200um)