x5-2 System v3

X5-2(L) Transfer of Information
System Overview v3
Joseph Ervin
System Lead Engineer
X86 Systems Development
August 22, 2014
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal
Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal 3
Program Agenda
1 X5-2(L) Overview
2 NVM-Express Overview
3 X5-2(L) Architecture
4 X5-2(L) Components
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Confidential – Internal/Restricted/Highly Restricted 4
Program Agenda
1 X5-2(L) Overview
X5-2 Server Family – What’s new vs. X4-2?
• Haswell-EP CPUs, 24x DDR4 Memory vs. 16x DDR3. DDR4 is faster, lower power than
DDR3.
• Use of traditional vertical backplanes in place of the modular 4-disk horizontal
backplane.
• SAS expander attached to 12-disk and 24-disk backplane, dramatically simpler system
cabling.
• Only one HBA type - New “Aspen” RAID HBA, with SAS3 speed of 12Gbit/sec, up from
SAS2 speed of 6Gbit/sec. Uses a SuperCap instead of Li-Ion battery, remotely mounted
to cool location.
• NVM-Express (NVMe) 2.5” flash devices for high performance.
• Minor Chassis modifications to accommodate storage configs and PCI-Express
Specification changes.
Oracle Server X5-2 Family
• Four (4) unique offerings
– Based on the Intel Grantley-EP server architecture
– New 1U and 2U chassis designs – only slight modifications from X4-2.
• Oracle X5-2 (1 RU)
– 8x2.5” disks, plus optional 4x NVMe 2.5”, and optional DVD.
• Oracle X5-2L (2 RU)
– 8x2.5” disks, plus optional DVD and optional 4xNVMe 2.5”
– 12x3.5” disks
– 24x2.5” disks, plus optional 4x NVMe 2.5”
Oracle Server X5-2
• Compute
– 2x Intel Haswell-EP E5-2600 v3 CPUs
– 24x DDR4-2133 RDIMMs, LR-DIMMs
– Up to 768GB DRAM • Availability
• I/O – Hot-swap disks
– 4x PCIe Gen3 slots 2 x16, 2 x8 – HW RAID 0, 1, 5, 6, 10, 50, 60
– 8x 2.5” SAS HDDs/SSDs, four of which can – Hot-swap redundant 600W PSUs
optionally be NVMe SSDs
• Management
– Optional DVD
– Oracle ILOM Service Processor
– 4x on-board 10Gb Ethernet RJ45 Ports
– 4x +2 Internal USB 2.0 ports • Oracle Solaris, Oracle Linux, Oracle VM,
Windows, VmWare
Oracle Server X5-2L Common Features
• Compute
– 2x Intel Haswell-EP E5-2600 v3 CPUs
– 24x DDR4-2133 RDIMMs, LR-DIMMs
– Up to 768GB DRAM
• Compute
– 6x PCIe Gen3 slots, 2 x16, 4 x8
– 4x on-board 10Gb Ethernet RJ45 ports
• Management • Availability
– Oracle ILOM Service Processor – Hot-plug Disks
– HW RAID 0, 1, 5, 6, 10, 5, 60
• Oracle Solaris, Oracle Linux, Oracle – Hot-swap redundant 1000W PSUs
VM, VmWare, Windows – Hot-swap redundant fans
Oracle Server X5-2L Disk Configurations
• 8 disk
– Up to 8x SAS HDDs, SSDs
– Up to 4x NVMe SSDs
– Optional DVD
• 12 disk
– Up to 12x 3.5” HDDs
– Plus two rear 2.5” HDDs
• 24 disk
– Up to 24x 2.5” HDDs/SSDs
– Up to 4 NVMe SSDs
– Plus two rear 2.5” HDDs
Motherboard Comparison: X5-2 versus X4-2
X4-2 X5-2
2 CPU x 12 cores (30M cache) 2 CPUs x 18 cores (45M cache)

CPU/Memory DDR3-1600 (16 slots), QPI @ 8GT/s DDR4-2133 (24 slots), QPI 2 9.6 GT/s
Up to 512GB @ 1600MT/sec Up to 768 GB @ 2133 MT/sec
PCI-Express Gen3 PCI Express Gen3
I/O One x 16 slot, others x8 Two x16 slots, others x8
Four 10GbE RJ45 Four 10GbE RJ45
Service
Aspeed AST-2300 Emulex Pilot3
Processor
Program Agenda
1 X5-2(L) Overview
NVM-Express Technology
• NVM-Express is an emerging flash storage technology with very high
performance relative to SAS SSDs.
– NVMe drives follow the same 2.5” form factor as previous enterprise 2.5” drives
– Drive/Backplane connector (SFF-8639) was modified to add additional pins required
for NVMe signaling.
• NVM-Express uses PCI-Express signaling
– 8GT/sec, x4 interface per drive. (Four times the wire-count for NVMe versus SAS)
– Approximately 4GB/sec full-duplex to the drive.
• NVMe drives also exist in other form factors
– PCI-Express Add-in cards (AICs)
– M.2 cards used in laptops and tablets
Hot-Plug with NVM-Express SFF Drives
• NVM-Express drives are native PCI-Express devices
– No HBA required – this is good because it reduces latency between main memory and
the NVMe SFF device.
– No HBA to provide isolation between I/O errors at the drive interface hardware and
the host system – This is bad because it means that errors on the electrical link are
directly seen by the system hardware and OS.
• Hot-plug of an NVMe SFF drive is PCI-Express Hot-plug
– Requires careful treatment by the OS to ensure that the PCI-Express link is quiesced
before drive removal or insertion.
• Failure to prepare a drive properly for removal will crash the system.
NVM-Express in X5-2(L)
• X5-2 1U and X5-2L 2U 8-disk and 24-disk servers optionally support 4x
NVMe SSDs in Oracle Linux and Oracle Solaris only, and not for boot.
• PCI-Express switch card is installed into motherboard PCIe slots
– Switch provides fan-out and electrical re-timing function for PCI-Express signaling,
connecting to the CPU complex with 8-lanes of Gen3 PCIe, and provides four x4
downstream links (one per NVMe drive)
– SAS-like cables carry PCI-Express from switch card to disk backplane along the
centerline of the motherboard.
• OS-specific hot-plug procedures must be followed. (refer to OS TOI
presentation)
• Watch for the blue-LED before pulling the drive!
NVM-Express 2.5” SFF SSD
PCI-Express Switch Card
NVM-Express Cable Routing in X5-2
NVMe
PCI-Express Cables
Switch Card
NVM-Express Cabling in the X5-2 1U
HBA
SuperCap
SAS Cables NVMe Cables
Program Agenda
1 X5-2(L) Overview
X5-2 System Block Diagram
• Delta’s w.r.t. X3-2 and X4-2
– Max CPU core count increases 12 -> 18
– DIMM count increases 16 -> 24
– Wellsburg replaces Patsburg
– Slot 2 is x16 versus x8 on X3-2/X4-2
– New Pilot 3 Service Processor
– SAS3 support in HBA and Disk Backplane
– NVM-Express Support
X5-2 Chassis Front
X5-2 Storage Device Nomenclature
NVMe Bay Label Bay Shared with PCIe Slot# PCIe bus #
NVMe0 HDD2 10 0x96
NVMe1 HDD3 11 0x98
NVMe2 HDD4 12 0x94
NVMe3 HDD5 13 0x92
X5-2 Chassis Rear
X5-2 Exploded View
1) Disk Drives
2) Front Indicator Module
3) Disk Backplane
4) SuperCap (for the HBA)
5) Fans
6) Motherboard
7) CPU
8) CPU Heat-sink
9) Coin Battery
10) USB Flash Drives
11) Air Baffle
12) Top Cover
13, 15, 16) PCIe Cards
14) HBA
17) PCI-Express Risers
18) DDR4 DIMMs
19) A256 PSUs
20) Chassis Base
21) DVD
X5-2L System Block Diagram
• Delta’s w.r.t. X3-2 and X4-2
– Max CPU core count increases 12 -> 18
– DIMM count increases 16 -> 24
– Wellsburg replaces Patsburg
– x16 supported on Slots 3 & 4, versus just
on Slot 1 on X3-2/X4-2.
– New Pilot 3 Service Processor replaces
AST-2300.
– SAS3 support in HBA and Disk Backplane
– NVM-Express Support
X5-2L 8-disk Chassis Front
X5-2L 8-disk Storage Device Nomenclature
NVMe0 HDD2 10 0x96
NVMe1 HDD3 11 0x98
NVMe2 HDD4 12 0x94
NVMe3 HDD5 13 0x92
X5-2L 12-disk Drive Numbering
X5-2L 24-disk Drive Numbering
NVMe0 HDD3 10 0x96
NVMe1 HDD4 11 0x98
NVMe2 HDD19 12 0x94
NVMe3 HDD20 13 0x92
X5-2L Chassis Rear
X5-2L Exploded View
1) Front HDD/SSD Drives
2) Disk Backplane
3) Left Front Indicator Module
4) Base Chassis
5) Motherboard
6) Coin Battery
7) Fans
8) CPU Heat-sink
9) DDR4 DIMMs
10) Air Baffle
11) Top Cover
12) HBA w/SuperCap
13) PCIe Cards
14) Rear I/O Riser
15) USB Flash Drives
16) Rear HDDs
17) Rear Disk Backplane
18) A258 PSUs
19) Right Front Indicator Module
20) DVD
12-disk and 24-disk SAS and NVMe Cabling
• Significant improvements in cabling compared to X3-2(L) and X4-2(L)
• Integration of the SAS expander to the disk backplane simplifies the
SAS cabling dramatically compared to X3-2(L) and X4-2(L).
– Just two cables from the HBA to the Expander.
– Cables run up the side of the chassis from HBA.
• SAS HBA SuperCap mounts remotely on top of the air baffle above P0
DIMMs D0-D6. Cool location avoids service-life issues.
• PCIe slot selection for PCIe switch card optimized for straight cabling
from the PCIe Switch Card to the disk backplane.
12-disk and 24-disk SAS Cabling
Program Agenda
1 X5-2(L) Overview
X5-2(L) FRUs and CRUs
• Hot-service CRUs • FRUs
– Fan Modules, – CPUs
– SAS HDD/SSD – SAS HBA
– NVMe SSD storage – PCIe Switch Card
– Power Supplies – Disk Backplanes (front and rear)
• Cold Service CRUs – SAS and NVMe Cables
– DIMMs – Motherboard
– PCIe Risers and Cards (except HBA – Front Indicator Modules
and PCIe Switch Card)
– DVD
– Coin Battery
– USB Storage devices (thumbdrives)
CPUs and DIMMs
• CPUs
– Intel® Xeon® E5-2699 v3 18-core 2.3 GHz, 45 MB, 9.6 GT/s QPI, DDR4-2133, 145 W
• DIMMs
– 8 GB DDR4-2133 RDIMM
– 16 GB DDR4-2133 RDIMM
– 32 GB LR DDR4-2133 LR-DIMM
HDDs and SSDs
• HDDs
– 600 GB 10000 rpm 2.5-inch SAS-2 HDD with marlin bracket
– 4 TB 7200 rpm 3.5-inch SAS-2 HDD with coral bracket
– 600 GB 10000 rpm 2.5-inch SAS-3 HDD with marlin bracket (Post-RR)
– 1.2 TB 10000 rpm 2.5-inch SAS-3 HDD with marlin bracket (Post-RR)
– 4 TB 7200 rpm 3.5-inch SAS-3 HDD with coral bracket (Post RR)
• SSDs
– 1.6TB 2.5-inch NVMe PCIe 3.0 SSD mix-use with Marlin bracket
– 400 GB 2.5-inch SAS3 SSD with marlin bracket
PCI-Express I/O Cards
• HBAs and Fibre Channel
– Sun Storage 12 Gb SAS PCIe RAID HBA, internal: 8 port and 1GB memory (Aspen-INT)
– Sun Storage 12 Gb SAS PCIe HBA, external: 8 port (Estes-EXT)
– Sun Storage Dual 16 Gb Fibre Channel PCIe Universal HBA, Qlogic (Ganymede-Q)
– Sun Storage Dual 16 Gb Fibre Channel PCIe Universal HBA, Emulex (Ganymede-E)
• Infiniband
– Sun Dual Port QDR InfiniBand Host Channel Adapter for PCIExpress Gen 3 (CX3)
• Networking
– Sun Dual Port 10 GbE PCIe 2.0 Low Profile Adapter, Base-T (Twinville)
– Sun Dual 10GbE SFP+ PCIe 2.0 Low Profile adapter (Niantic)
– Oracle 10 Gb and 40 Gb Ethernet Adapter, 2 QSFP+ connectors (Post-RR) (Fortville)

x5-2 System v3

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

x5-2 System v3

Uploaded by

Copyright:

Available Formats

X5-2(L) Transfer of Information

2 CPU x 12 cores (30M cache) 2 CPUs x 18 cores (45M cache)

SAS Cables NVMe Cables

You might also like