Proceedings oI the International MulticonIerence on ISBN 978-83-60810-22-4

Coputer Science and InIoration echnology, pp 567 571 ISSN 1896-7094

-The second generation of Intel Xeon proces-
sors based on Core Microarchitecture and 45nm process tech-
nology bring not only a new level of performance but also sig-
nificant improvement in power characteristics Continuous
performance improvement and power efficiency are the para-
digms for most Data Centers today and are also the challenges
that will not go away anytime soon The increasing energy costs
have made Data Centers even more energy conscious especial-
ly those with older facilities that lack the power and thermal
capacity to expand and keep pace with the growing needs and
reuirements This paper will describe how far the industry has
progressed and evaluates some of the challenges we are facing
with new 45 nm Intel Xeon processors and some of the solu-
tions that have been developed
I INODCION
AA Centers and large HPC installations are beco-
ing increasingly ore expensive to power and cool
According to the S EPA, Data Centers and servers in
the consued about 15 oI the S total electricity con-
suption in 2006 he average annual power costs Ior a
10000 square-eter Data Center is around 6 illion SD
Soe large Data Centers require as uch energy as a sall
town, and consue 5MW energy Ior power and cooling
hese Iigures continue to rise as copute needs grow, densi-
ty increases, and power and cooling deands clib Many
Data Centers are reaching their Iull capacity Ior which they
were designed Gartner says 50 oI data centers will have a
shortage oI power in the current year
D
II PEFOMANCE PE WA CONSIDEAION
Building an energy eIIicient solution Ior High PerIor-
ance Coputing requires an energy eIIicient central pro-
cessing unit (CP) Pure perIorance is iportant but we
need to always consider the iplications on power, when
easuring the perIorance oI a syste ore and ore we
look Ior the best ratio oI power and perIorance II the pow-
er consuption is related to the dynaic capacitance, the
square oI the voltage with which the transistors and I/O
buIIers are supplied and the Irequency at which the
transistors and signals switch then we can express:
Power Dynaic Capacitance x oltage
2
x Frequency
aking into account perIorance and power equations,
CP designers need to balance instruction per clock (IPC)
eIIiciency Iro one side and voltage/Irequency Iro the oth-
er to oIIer a coproise oI perIorance and power eIIicien-
cy oI the processor Microprocessor design criteria are no
longer Iocused on ust pure perIorance, but rather on deliv-
ering leadership in both raw perIorance and in perIor-
ance per watt In choosing the ost energy-eIIicient pro-
cessor, it is also iportant to consider the relationship be-
tween so called heral Design Power (DP) and processor
Irequency For a speciIic processor Iaily, the processor
with the highest Irequency and cache will typically provide
the highest perIorance However, it will also have the
highest DP he processor with the second highest Ire-
quency typically provides soewhat slower perIorance but
has a uch lower DP For exaple, decreasing the Ire-
quency Iro 316 GH to 30 GH ight reduce coputing
perIorance by less than 10, but decrease power con-
suption by as uch as 30-40 (Iro 120 W to 80 W) In
this situation individual tasks will run slower due to the low-
er CP Irequency but the total perIorance per energy will
be uch higher
III NEW 45 NM AND HIGH- DIEECIC ECHNOOG: HE NEW
INNOAION FO HE BES PEFOMANCE PE WA POCESSO
CHAACEISIC
Intel has achieved the biggest breakthrough in transistor
technology through the use oI HaIniu Hi- and etal
gates in its 45n silicon process he 45n Hi-k silicon
technology provides iproved transistor switching speeds
which, when cobined with Intel icroarchitecture en-
hanceents, delivers higher clock speeds and greater perIor-
978-83-60810-14-9/09/2500 2009 IEEE 567
New Multi-Core Intel Xeon Processors help design nergy fficient
Solution for High Performance Computing
Pawe Gepner
EMEA Platform Architecture Specialist
Intel Corporation
pawel.gepnerintel.com
David Fraser
EMEA Regional Applications Manager
Intel Corporation
davidlIraserintelco
Micha F owalik
Market Analvst
Intel Corporation
michal.f.kowalikintel.com
aIa ylan
KDM Director
TASK Acaemic Computer Center
traItaskgdapl
568 POCEEDINGS OF HE IMCSI OME 4, 2009
ance Ior a given power level his process technology also
enables lower leakage transistors which Iurther beneIits
power eIIiciency he Intel 45 n Hi-k silicon technology
provides nearly 2x ore transistors than 65 n technologies
to eIIiciently add new capability, perIorance, and Ieatures
Intels 45 n Hi-k next generation eon processors take ad-
vantage oI these beneIits to deliver up to 12 MB cache, ore
than 3 GH core speeds, greater overall perIorance and
leading energy eIIiciency he High-k aterial is based on
an eleent called HaIniu, replacing silicon dioxide In ad-
dition the transistor gate is ade up oI two types oI etals,
replacing silicon HaIniu, eleent 72 on the periodic ta-
ble, is a highly elastic, corrosion resistant and cheically
siilar to irconiu Intel is using HaIniu to replace sili-
con dioxide in its 45 n transistor because it is a thicker a-
terial and this signiIicantly reduces electrical leakage and
has the beneIit oI higher capacitance which is desirable Ior
transistor perIorance hese innovations iprove peak
perIorance Ior a given CP power envelope he new 45
n process technology based ulti-core processor systes
changed the dynaics oI the arket and enable new innova-
tive designs delivering high perIorance with an optiied
power characteristic
I AE OF NEW 45 NM MI-COE EON POCESSOS FO
ENEG EFFICIEN HPC SSEMS
Gdansk Acadeic Coputer Center AS has been us-
ing Intel based clusters since 2000 he Iirst installation was
based on Pentiu III eon Iollowed by an Intel Itaniu
2 based syste which was subsequently upgraded to Dual-
Core Itaniu 2 (Montecito) In 2007 a new syste based on
uad-Cores Intel eon 5345 (Clovertown) was deployed
In 2008 AS has been testing new platIors based on 45
n uad-Core Intel eon 5462 (Harpertown) as well as
uad-Core Intel eon 5420 energy-eIIicient version oI
Harpertown his work has been necessary to validate the
next potential platIor oI choice Since March 30th 2009
when Intel introduced a copletely new platIor based on
uad-Core Intel eon 5500 Iaily (Nehale-EP) AS
started evaluation oI the new platIor as a candidate Ior
their next generation cluster
In order to deterine the perIorance and attractiveness
oI a coputer syste, AS uses a inpack benchark
he inpack benchark is an industry standard Ior HPC
perIorance evaluation and generates workloads which are
very intensive and siulate the extree scenario to those
running in typical HPC centers inpack was chosen as the
deIault benchark, because it is also one oI the ost eIIec-
tive ethods oI stressing the theral envelope to the liit
he inpack result divided by the aount oI power provides
a easure oI inpack per watt his result is quite iportant,
since it indicates how uch processing power is provided
Ior each watt consued by a running syste
Figure 1 shows Iive evaluated systes the Iirst one is
based on 65 n Intel eon 5345 the second is using the Intel
eon 5462 supporting 1600 FSB the third is utiliing Intel
eon 5420 energy eIIicient version and the Iourth one is
based on sae Intel eon 5420 processor but uses diIIerent
platIor with DD2 eory interIaces not FB-DIMM as it
was the case in 3 Iirst platIor Finally the last syste is
based on new Intel eon 5560 with DD3 eory interIace
operating at 1066MH and Iro an architecture perspective
it represents a new class oI syste with an integrated eo-
ry controller and Intel uickPath Interconnect
We can see the perIorance oI the new Intel eon 5560
based syste is siilar to the syste based on Intel eon
5462 but Iro power reduction point oI view this is 20
less power consued 313W versus 385W
In this case perIorance parity is driven by the nature oI
benchark used Ior the testing environent as inpack is
heavily dependent on CP clock Irequency and in both cas-
es clocks are the sae 28GH, so results are very siilar
inpack is a Iloating-point benchark that solves a dense
syste oI linear equations in parallel he etric produced
is Giga-FOPS or billions oI Iloating point operations per
second inpack perIors operations called Factoria-
tion hese are highly parallel and store ost oI their work-
ing data set on processor cache It akes relatively Iew reI-
erences to eory Ior the aount oI coputation it per-
Iors so an integrated eory controller does not play such
an iportant role in this benchark scenario When we con-
sider any other benchark or real HPC application then the
Intel eon 5500 Iaily deonstrates 20-80 perIorance
iproveent versus Intel eon 5400 Iaily
his reduction oI 20 power consuption was achieved
with typical production processor Iro the Intel eon 5500
Iaily with a 95W theral envelope II we consider using
the ow oltage version Intel eon 5520 or Intel eon
5506 with a 60W theral envelope then on the platIor
level we can save 70W, reducing platIor consuption to
243W
Energy-eIIicient 60-watt uad-Core Intel eon 5520 pro-
cessor delivers a 40 decrease in power Iro those current-
ly used in the AS GAEA syste (Intel eon
5345) his energy eIIicient version oI the Intel eon pro-
cessor requires ust 15 watts oI power Ior each core It
provides siilar perIorance on inpack to the 65 n
63
75
68 66
76
405
385
320
280
313
0
50
100
150
200
250
300
350
400
450
2xCTN
2.33/1333
2xHTN
2.8/1600
2xHTN -LV
2.5/1333
2xHTN -LV
2.5/1333 on
DDR2
2xNHL-EP
2.8 /DDR3
1066
GFLOPS watt

Fig. 1 Evaluated platforms and their performance and power.
PAWE GEPNE E A: NEW MI-COE INE EON POCESSOS HEP DESIGN 569
uad-Core Intel eon 5345 processors but sets a new stan-
dard in energy eIIiciency
In the best scenario the reduction in power is not only re-
lated to the new generation oI the CP but it beneIits also
Iro the new platIor architecture and oI the new Intel
5520 chipset he platIor brings new power capabilities as
part oI Intel Intelligent Power echnology One Ieature oI
which is Intel Node Manager echnology, which enables
users to set power levels and ake sure their server systes
do not exceed these thresholds his technology is an Intel
provided Iirware stack running on a icrocontroller that is
ebedded into the Intel 5520 chipset, which links the BMC,
power supply and CP sensors to support this dynaic plat-
Ior power control As we can see on Figure 2 the new plat-
Ior based on uad-Core Intel eon 5500 also reduces the
idle power by 50 vs Intel uad-Core eon 5400 platIor
Iaily hese are exaples oI capabilities that allow the In-
tel eon 5500 platIors to adapt and be the ost versatile
platIor to support a range oI workloads, environents and
operating scenarios he Peak Power reduction helps when
systes are Iully utilied but Idle Power reduction iprov-
ing the overall consuption oI power in the Data Center
when the systes are not utilied and are in waiting ode
0
100
200
300
400
nte Xeon 5462 baed yte nte Xeon 5560 baed yte
Peak Power oad W) de Power W)

Fig. 2 Peak Power and Idle Power reduction.
he platIor and processor related iproveents the In-
tel eon 5500 Iaily brings has helped to bring a signiIicant
reduction in power by providing:
5x the nuber oI operating states (15 p-states with
eon 5500 vs 3 p-states with eon 5300)
5x lower CP idle power (10W with eon 5500, 50W
with eon 5300)
5x Iaster transition between power states (2 icro-
seconds with eon 5500, 10 icro-seconds with eon
5300)
he Intel eon 5500 processor has a new eory subsys-
te which includes an integrated eory controller sup-
porting up to 18 DIMMs oI DD3 eory which provides
3x the bandwidth (64GB/sec, copared to 21GB/sec) Iro
the previous generation Intel eon 5400 With support Iro
the new Intel eon 5520 chipset, and Intel uickPath Inter-
connect, oIIering up to 256 GB/sec bandwidth per link an
iproveent oI 24x the bandwidth (512 GB/sec PI bi-di-
rectional, copared to 21GB/sec FSB bi-directional) over
the previous generation Intel eon platIor
Also perIorance enhancing technologies ipleented
in Intel eon 5500 such as:
Intel urbo Boost echnology Increase perIorance
by increasing processor Irequency and enabling Iaster
speeds when conditions allow
Intel Hyper-hreading echnology Increase perIor-
ance Ior threaded applications by running two data threads
in each processor core, delivering greater throughput and re-
sponsiveness
Copleented by power technologies such as:
Integrated power gates allows power control oI each
CP core, enabling idle cores to go to near ero power inde-
pendently and can be controlled autoatically or anually -
Figure 3
Autoated low power states More and lower CP
power states, reduced latency during transitions between
power states, and new power anageent capabilities on
eory and I/O
Allowing uad-Core Intel eon 5500 Iaily based plat-
Iors to deliver great balance between perIorance and
power consuption

Fig. 3 Integrated power gates
he new platIor is also ready to scale with the processor
generations oday the platIor is ready Ior Intel eon
5500 but will also support the next generation 32n based
processors
DIING ENEG EFFICIENC A HE PAFOM AND DAA
CENE EE
he ove to ulti-core processors enables ongoing i-
proveents in overall perIorance, without a subsequent in-
crease in processor power consuption For AS, adding
additional processing power also eans adding additional
eory (AM), as ost oI ASs applications have a re-
quireent oI 2 GB oI eory per processing core he
eory has becoe a signiIicant Iactor in power calcula-
tions Depending on the technology, eory odules con-
sue between 5 - 10 watts per GB his gives 80 - 120
watts Ior eory subsyste alone he syste tested by
AS, based on the Intel 5100 and Intel 5520 chipsets used
DD2 and DD3 eory odules and this contributed to
570 POCEEDINGS OF HE IMCSI OME 4, 2009
Iurther power savings in the region oI 40-50 watts per
syste
ASs delivered syste perIorance has increased by
570 ties since 2000 Iro 67 GFOPS to 38 FOPS At
the sae tie power increased ore than 10 ties Iro 20
W to 216 W Figure 4 shows the ratio oI perIorance
per watt Ior all generations oI systes in AS Acadeic
Coputer Center
LlNPACK}watt
3.35
8.55
30
176.71
nte Pentu
Xeon 700 MHz)
nte tanu2 1.3
GHz)
nte tanu2
DuaCore 1.4 GHz)
nte Xeon 5345
2.33 Ghz)

Fig . 4 Performance per watt for all TASKs generation of sys-
tems
Building energy eIIicient data centers is a ore coplex
proble then siply choosing the best energy eIIicient CP
icroarchitecture and platIor Nevertheless these two ele-
ents are critical he industry is looking Ior other cople-
entary technologies and techniques eg power anage-
ent to build ore sophisticated and ore energy eIIicient
solution Power anageent controls the platIor power
based on actual workload and iniiing the power con-
suption not associated with coputing Others techniques
are looking Ior ways to iniie wasted power during con-
version and transission, odeling airIlow to identiIy and
address key airIlow probles, using barriers and custo
cabinets to control the airIlow
he new introduced Intel Data Center Manager, which is
a soItware application that can be use to extend platIor
power control to the rack level his will enable soItware
anageent tools to aggregate data, reports trends and an-
age power at the rack or datacenter level his is one exa-
ple oI any types oI soItware that will take advantage oI In-
tel Node Manager echnology Intel Data Center Manager
(DCM) build on Intel Node Manger and custoers existing
anageent consoles to aggregate node data across the en-
tire rack or Data Center to track etrics, historical data and
provide alerts to I operators his allows Data Center an-
agers to establish group level power policies to liit con-
suption while dynaically adapting to changing server
loads he wealth oI data and control that DCM provides al-
lows Data Centers to increase rack density, anage power
peaks, and right sie the power and cooling inIrastructure
AS oved to the new Data Center in the beginning oI
2008, beIore this ove they were studying air cooling issues
and developing design ethodologies based on 15 years op-
erational experience AS progressively developed an ap-
proach, incorporating new techniques and then ipleent-
ing it into their Data Center
AS studys includes several innovations and
considerations:
AS odeled airIlow to identiIy and address key air-
Ilow probles
hey were considering using barriers and custo cabi-
nets to control airIlow and increase the air conditioning air-
Ilow eIIiciency
AS analyed cobined ultiple techniques to
achieve high densities servers (twin boards as well as
blades)
ASs results showed that it is possible to use eIIective
air cooling techniques to achieve uch greater power densi-
ties than they previously considered hey suppose that
these results will Iurther stiulate the discussion to use air
or water cooling by other Data Centers
During the Data Center design process, AS engineers
repeatedly Iaced the sae questions:
What is the axiu air cooling capacity that we
should build into the Data Center based on the anticipated
Iuture density and heat characteristics oI blade and other
servers
What are the liits oI air cooling and when should we
start planning to use an alternative cooling ethod
Can our Data Centers accoodate the servers that
will be produced during the next Iive to seven years
AS validated any oI the diIIerent solutions and plat-
Ior types, blade servers are the ost proising Iro a
theral and perIorance point oI view odays blade sys-
te solutions can generate a heat load oI 14 W to 25 W
per cabinet Many existing and planned Data Centers are
not ready to support this density but AS Data Center has
been designed to sustain it ASs experience shows that
the ost appropriate layout is repeating rows oI racks side-
by-side with alternating cold aisles and hot aisles he cold
aisle supplies cool air to the servers, with each rack dis-
charging into a hot aisle shared with the next row oI servers
aised Iloors provide cool supply air to the cold aisles with
overhead returns to the air conditioning syste Ior the war
return air In this hot conIiguration, changeable nubers oI
servers can Iit into each rack based on any Iactors cooling
capability, power availability, network availability, and Iloor
loading capability here is a lot oI debate that 14 W racks
will need suppleental cooling or liquid cooling to be able
to handle these types oI loads his is not the case in AS
study High density can be cooled successIully with standard
hot-aisle / cold-aisle design he AS HPC Data Center is
easily carrying the high-density racks with no recirculation
probles On-going internal analysis ade by AS shows
also that supporting 30 W racks is realistic with air cool-
ing
I CONCSION
Multi-core processors becoe the standard Ior delivering
greater perIorance, iproved perIorance per watt, and
PAWE GEPNE E A: NEW MI-COE INE EON POCESSOS HEP DESIGN 571
add new capabilities Ior server platIors AS has beneIit-
ed Iro the ove to ulti-core processors with a ore eIIi-
cient icroarchitecture Intel uad-Core processors based
on the Intel Core icroarchitecture deliver about Iive ties
ore copute power per watt than previous generations
his is enabling AS to provide IiIteen ties ore co-
pute power than they have previously been able to oIIer on
their earlier generation systes As the nuber oI cores per
die is expected to rise, ulti-core technology sees to be a
proising way to Iurther reduce power consuption and in-
crease the perIorance per watt ratio Finally new platIors
can be instilled in high density Data Centers with air cool-
ing
EFEENCES
1 eport to Congress on Server and Data Center Energy EIIiciency Pub-
lic aw 109-431, S EPA, ENEG SA Progra, pp 10-11,
Aug 2007
2 M oy: House Green ights EPA Data Centers Study, Internetnews-
co, uly 2006
3 Gartner Says 50 Percent oI Data Centers Will Have InsuIIicient Power
and Cooling Capacity by 2008, Gartner Inc press release, Nov 2006
4 P Gepner, MF owalik: Multi-Core Processors: New Way to
Achieve High Syste PerIorance PAEEC 2006, pp 9-13, Sep
2006
5 A Hirstius, S arp, A Nowak: Strategies Ior increasing data centre
power eIIiciency CENopenlab, Feb 2008
6 O Wechsler, Inside Intel Core Microarchitecture: Setting New
Standards Ior Energy-EIIicient PerIorance, echnology Intel
Magaine
7 D Garday, D Costello, Air-Cooled High-PerIorance Data Centers:
Case Studies and Best Methods, echnology Intel Magaine