Re-Thinking Data Center Power Delivery: Regulating Series-Connected Voltage Domains in Software

Josiah McClurg, Yujia Zhang, Jeff Wheeler, and Robert Pilawa-Podgurski

Department of Electrical and Computer Engineering University of Illinois at Urbana-Champaign Urbana, Illinois 61801 E-mail: jcmcclu2@illinois.edu

AbstractIn this paper, the fundamental advantages of seriesconnected voltage domains are briey discussed in light of existing power distribution architectures for data centers. The key technical challenges of this architecture are addressed; and results from a working prototype system are presented as a proof of concept. In particular, a straightforward softwareonly solution is presented which eliminates the need for power conversion hardware between the distribution bus and server. Based on power-aware load balancing of web trafc and server frequency, this design is shown to provide adequate voltage regulation for a cluster of four web servers sharing the same line current. Compared with measurements of maximum server performance, the series-connected conguration shows only a minor reduction in net throughput. As a step toward improving system reliability and performance, a hardware supplement for over-voltage protection is presented along with measurements verifying its operation at full server load.

such as the Cisco StackPowerTM system are already available on the market, as companies transition the power distribution of data centers and network switch rooms to a more efcient, clustered conguration. A. Motivation: Removing Rather than Reducing Per-Server Conversion Losses However, AC or DC, parallel or clustered, there will always be some quantity of power lost in conversion if the server cannot operate directly off the bus voltage. In all such congurations, the quantity of power lost between the bus and the server will increase in proportion to the number of servers in the system. Moreover, even at very high net efciencies, the cost of power conversion loss can be quite signicant. For example, if the power supplies of each server were to operate at 90% efciency 100% of the time, the conversion losses between server and bus in a 1000-server data center at full load (about 300W per server) would be around 300 kW. At the 2012 average commercial electricity price in Illinois (8.23 cents per kilowatt-hour), this amounts to an annual cost of around $216,000. Compared to traditional power distribution architectures, the primary advantage of series-connected voltage domains is that by stacking voltages to the level of the bus, it is possible to do away with off-bus DC/DC converters altogether, as is shown in Figure 1b. Assuming DC loads that consume identical power, and an appropriately-sized stack, voltage scaling occurs automatically as a result of Kirchhoffs circuit laws, and conversion losses between server and bus are nonexistent. In addition to the direct decrease in total conversion losses, the act of eliminating off-bus power converters reduces the size and heat production of each server module. Cooling is one of the primary challenges associated with maintaining a highlyefcient modern data center [6] and internal server layout is a critical parameter in thermal management [7] of each server module. While server power supply units (PSUs) rarely generate signicant compared to the servers they power, the PSU is frequently one of the most bulky components within a server enclosure. Thus, its removal will have a direct impact

I. I NTRODUCTION Power distribution optimization for data centers is an exciting topic in power systems research. Not only do data centers have a high impact on industry and society as a whole, but the eld is a rich frontier of technical opportunity with a great deal of theoretical room for efciency improvement. Data centers are complex systems with a wide variety of interdependent electrical loads. However, efciency optimization has mainly focused on improving cooling and power conversion equipment because these represent the largest power consumers of the system aside from the servers themselves [1]. While a DC power distribution architecture offers several technical advantages over an AC distribution system [2], there remains some controversy over the preferred distribution bus type for data centers, due to the relatively high net efciency of the best combination of commercially-available AC transformers, AC UPSs, and AC/DC server power supplies [3]. In either case, system efciency can be improved from the common parallel architecture of Figure 1a by clustering multiple loads together [4]. A simple example of such clustering is shown in Figure 1c. One fundamental reason for the increased efciency of the clustered power pool architecture is that the variability of the load is decreased and the external power supply can be sized to operate at its maximum efciency for a greater percentage of time [5]. Some commercial products

978-1-4673-5602-2/13/$31.00 2013 IEEE

147

380V DC distribution bus

Cluster test and control unit

Server cluster with distributed UPS

48V DC power supply

12V + 120V AC or 380V DC distribution bus 12V + Server PSU + 12V DC motherboard ... ... + + 12V DC motherboard Network switch 12V DC motherboard Trafc generation Power-aware load scheduler

12V

12V

Fig. 2.
...

Server stack architecture.

Server PSU + 12V DC motherboard + 12V DC bus + + 12V DC motherboard ... + 12V DC motherboard 12V DC motherboard

(a) Standard parallel conguration

(b) Stacked cluster conguration

120V AC or 380V DC distribution bus

(c) Parallel cluster conguration Fig. 1. Power delivery architectures.

on reducing the cooling requirements and geographical area required for data centers of the future. B. Challenge: Reliability and Maintenance Considering that traditionally-encountered factors such as temperature and server workload can still present serious difculties in data center reliability [8], it is not surprising that the series-connected power distribution architecture brings along with it a set of reliability and maintenance concerns. Cascaded failure is a particularly important problem to be

+ -

Cluster PSU

addressed in the case of stacked server clusters. Because all servers share the same line current, a single malfunctioning server node may deprive the other servers of the current they require to operate. In the best case, this may cause the other servers to shut down, resulting in a temporary interruption in server operation. In the worst scenario, the failure of a single server could result in a cascaded overvoltage condition, which could cause irreversible hardware damage to the servers. In order to mitigate the risk of hardware damage to the servers in the case of software failure, an independent hardware protection solution is required. To this end, this paper presents implementation results from an overvoltage protection prototype, which makes use of a distributed uninterruptable power supply (UPS) and custom power electronics to detect and bypass nonfunctioning servers in order to maintain safe stack operation in the event of single or multiple server failures. C. Potential Challenge: Communication Across Voltage Domains At the chip level, high-speed communication across voltage domains can pose a signicant design challenge due to the latency introduced by the level shifting circuitry [9]. One might expect similar difculties for whole-server systems, or that special communication hardware might need to be installed in order to implement a stacked server architecture. While such additional design effort may still be needed in some particular applications of series connected voltage domains, the primary communication technologies utilized in data centers already exceed the architectures voltage isolation needs. Standards-compliant network interfaces for Ethernet guarantee at least 1.5 kV DC isolation [10] [11], and optical ber networks are inherently isolated communication channels at levels well beyond the 380 V that would likely be encountered in a full server stack. In addition to reducing the design costs associated with architecture implementation, the fact that existing communication hardware can be re-used without modication in a stacked voltage domain conguration adds substantial value to this architecture.

148

D. Challenge: Load-Side Voltage Regulation Since each server load is designed to operate within a limited range of input voltages and each server in a cluster shares the same line current, the power consumed by each server must be balanced such that a constant voltage is maintained across each server. Previous work in poweraware scheduling such as [12] and [13] has demonstrated the capability of reducing average power consumption without sacricing computational performance. Work such as [14] has demonstrated the operation of voltage regulation of low-power series-connected voltage domains using external differential power processing hardware. The experimental work shown in this paper will apply the concept of power-aware scheduling to perform load-side voltage regulation with software alone on 120 W network servers. This regulation is implemented in two independent but complementary software contexts. A centralized algorithm handles long-term uctuations in voltage difference between servers by intelligently distributing an incoming network load among the cluster nodes. On the other hand, short-term uctuations in voltage are corrected by running a simple communicationless distributed frequency scaling algorithm on each server. Figure 2 provides a system-level overview of the setup. II. M ETHODS : L OAD -S IDE VOLTAGE R EGULATION Both the centralized and the distributed algorithms regulate server voltage by adjusting the power consumption of each server. For control purposes, the server voltage is considered to be inversely proportional to power consumption. To see why increasing server load reduces server voltage, consider that for each server i, the power Pi is related to the voltage Vi and server impedance Zi according to Pi = Vi2 Zi (1)
Digital multimeter

Server stack

Tra Trafc generation and control unit

Ethernet switch

Battery UPS

Main power line

Patch box

Power switches

Fig. 3.

Experimental setup.

From Equation (1), power and impedance are inversely related so that increasing the power load of server i results in a decrease in Zi . From Equation (3), this ultimately results in a decrease in Vi . III. E XPERIMENTAL R ESULTS Figures 2 and 3 describe the experimental stacked server cluster prototype that has been implemented. The voltage inputs of four 12 V, 120 W Optiplex workstations running Ubuntu Linux are connected in series, each with a battery at the server input. Collectively, the batteries may be thought of as a short-lifetime distributed UPS [15]. In addition to assisting server startup during the power on sequence, the UPS is included in this prototype to mitigate stability issues that result from the high rate of server power uctuation and the relatively low granularity and speed of the current software control loops. Supplying power to the stack is a digitally-controllable DC supply. The servers receive their work load across the network in the form of HTTP requests from the trafc generation and control unit. The control unit is also responsible for reading the voltages of each server node by polling a digital multimeter. While most industry-standard servers are equipped with an Intelligent Platform Management Interface card or other builtin hardware capable of directly measuring the servers own voltage [16], the server nodes in this implementation emulate this functionality through a network socket connection to the control unit. It is important to note that no control messages are passed among servers or between servers and control unit. Thus, the centralized and decentralized voltage regulation softwares can be seen to operate independently. Experimental results demonstrate adequate voltage regulation under each of these modes separately, and show improved operation when both are used concurrently. A. Work Allocation 1) Implementation: The power-aware load scheduling algorithm is implemented as a straightforward proportional control. Figure 4 gives a detailed conceptual view of the

Now, consider the case that N of these servers have been N stacked to the bus voltage Vs such that 1 Vi = Vs and Ii = Ij i, j {1 . . . N }. The shared line current I is then given by I= Vs
N 1

Zi

(2)

All other factors being equal, an instantaneous change in the power consumption of node i then corresponds to a change in impedance Zi , such that Zi,new = Zi + Zi . If the change in power is small compared to the total power consumption such N that Zi << 1 Zi , it is easy to see from Equation (2) that Inew I . However, because the sum of the server voltages must remain constant, the steady-state voltage of node i will be given by Vi,new = (Zi + Zi )Inew IZi + I Zi (3) = Vi + I Zi

149

Load Allocation Start

Trafc for server 1 Trafc for server 2 Trafc for server 3 Trafc for server 4

Node 1 HTTP trafc Node 2 Master Node 3

12.4V

12.6V

Read initial voltage

12.2V

Spawn work allocation threads

Node 4

12.7V

Fig. 5.
0.2 0.1 0 15 10 5 0 1 0 1 Voltage imbalance detected 0 1

Load balancing architecture.

Work Allocation Coefcient

Read voltage

Voltage list

Send rate and CPU difculty of HTTP requests in proportion with voltage 2 3 4 5

Time between jobs (s)

CPU time for job (s)

CPU Utilization Coefcient Processing previous requests CPU utilization directly proportional to number of new requests

Update work allocation

Work Allocations

Highest voltage 2nd highest 3rd highest Lowest voltage

2 1 .5 .3

0 .5 1 3

2 Voltage

Voltage inversely proportional to CPU utilization 0.5 0

Fig. 4.

Work allocation procedure.

Time unit 3

Fig. 6.

Ideal balancing behavior.

work allocation procedure. The software uses a multi-threaded environment to separate the voltage measurement from the trafc generation and routing. The main thread of the control unit software polls a multi-channel data acquisition unit every 300 milliseconds to obtain voltage readings across each server terminal. After the initial voltage has been read, four trafc generation threads are initiated, with each thread corresponding to a single server node. The trafc generated is in the form of HTTP requests with a known CPU time for the servers to process. While future work may incorporate an algorithm to obtain an estimate of CPU time remotely, this implementation assumes foreknowledge of server CPU time. This is accomplished through a custom web application running on each server node. Upon receipt of an HTTP request, each servers web application loads the CPU for the time period specied in the HTTP request received from the control unit. Relative power consumption is controlled by adjusting the CPU time and request frequency to each server. A straightforward discrete-valued control scheme is used to make this adjustment. After each set of four voltage measurements, the servers are assigned a CPU time and request frequency which corresponds to their voltage ranking in relation to the other server nodes. For a concrete example of the trafc balancing algorithms basic operation, consider Figure 5 and Figure 6. Node 4 has the highest voltage, and therefore receives the largest average

CPU load, while Node 3 has the lowest voltage, and therefore receives the smallest demand on its CPU. 2) Measurements: The top portion of Figure 7 demonstrates that regulation via network job scheduling is able to maintain each voltage of a four-server stacked cluster within a reasonably narrow range (approximately 0.5 V at steady state). In order to observe system response characteristics, the supply voltage is increased before minute 9 and is lowered between minutes 10 and 11. The work allocation coefcient displayed in the lower graph of Figure 7 represents the ratio of requested CPU time to the seconds between successive request. While regulation in this manner is within acceptable bounds for safe server operation, the Node 4 voltage does often rise above the recommended maximum value of 12.5 V. Thus, it is also clear that the regulation can (and should) be made to operate within tighter bounds. B. Per-Server Frequency Scaling Algorithm 1) Implementation: In addition to maintaining a more netuned control of voltage, it is also important that the servers maintain a safe operating voltage in the case of a communication failure, or the lack of incoming network trafc. These two requirements are met simultaneously by implementing a local regulation and over-voltage protection software on each server. This software, described by Figure 8, performs frequency

150

Server voltage (50 sample average) 12.8 Voltage 12.6 12.4 12.2 12 7 8 9 10 11 12 13

Work allocation coefcient (20 sample average) 100 Work allocation 80 60 40 20 7 8 9 10 Time (minutes) 11 Node Node Node Node 1 2 3 13 4

Fig. 7. Load balancing only. All web servers running at maximum frequency of 3.2 GHz.
Software Start Server reads voltage

No, below lower limit

Is the voltage within safe operation range? Yes

No, above upper limit

Reduce CPU frequency No

Increase CPU frequency

Is server running failsafe load? No Yes Reduce failsafe server load Is CPU frequency at maximum? Yes Increase failsafe server load Yes Continue? No End

scaling to improve voltage regulation. It is also capable of articially increasing server power consumption in order to mitigate an over-voltage condition. Power consumption measurements taken for one of the stacks servers at different frequencies, running four different types server loading benchmarks are shown in Figure 9. Two observations of this data are used to inform the straightforward control strategy described below. First, the fact that power consumption at full load (be it CPU, I/O, memory, or disk) depends on the frequency in a linear fashion provides strong motivation for using frequency scaling as the primary means of voltage regulation. Second, the fact that idle power consumption remains constant, regardless of frequency inspired the concept of introducing a failsafe load, capable of increasing server power consumption if an overvoltage occurs in the absence of incoming network trafc. Upon regulation start, the server sets up an SSH socket connection to the control node in order to monitor its own voltage readings as those readings are received from the digital multimeter. Based on these readings, the software performs frequency scaling using the Linux kernel infrastructure cpufrequtils. Because increasing CPU frequency increases the power consumption of the server, the frequency is increased at a constant rate until the voltage drops below 12.5 V or the frequency reaches maximum. Likewise, if the voltage drops below the recommended minimum of 12.2 V, the frequency is decreased until the voltage rises again or the minimum frequency is reached. In the case that the frequency is at maximum and the voltage remains above 12.5 V, the software enters an over-voltage protection mode, where the CPU is loaded to articially increase power consumption until a safe operating voltage is restored. 2) Measurements: Figure 10 shows the server voltages when they are running the distributed regulation software alone, under a randomized trafc load. Between minute 26 and 27, the power supply voltage is increased in order to observe the system response. Compared to the results of Figure 7 (work allocation alone), it is clear that the frequency scaling approach achieves a tighter voltage regulation. The CPU utilization coefcient referred to in the lower portion of Figures 10 and 11 is given by Equation (4), and corresponds roughly to the power consumption of the node. fGHz 1 Tidle T 100% (4)

Fig. 8.

Distributed voltage regulation procedure.

Power Consumption of Optiplex Sx280 Server 140 120 100 80 60 40 20 0 Idle

CPU only I/O I/O and Memory 1.2 GHz 1.6 GHz 2.4 GHz 400 MHz 800 MHz 3.2 GHz 2.8 GHz 2 GHz I/O, Memory, and Disk

Fig. 9.

Frequency and power consumption of Optiplex workstations.

As would be expected, the regulation exhibits more favorable properties when both distributed and centralized regulation softwares are operated at the same time. As is observed in Figure 11, the regulation of the end nodes (1 and 4) are noticeably improved compared to Figure 10. The behavior from minutes 15 to 18 demonstrates that a high degree of robustness has been achieved with this software-only solution for regulating voltage of a stacked server cluster. From minute 15 to 16, the total power supply voltage is increased beyond the recommended voltage. All four servers respond by increasing their CPU utilization coefcients

Power (W)

151

Server voltage (50 sample average) 12.8 Voltage 12.6 12.4 12.2 12 24.5 25 25.5 26 26.5 27 27.5 28 Node Node Node Node 1 2 3 4

CPU utilization coefcient (50 smpl avg) 400 CPU utilization 300 200 100 0 24.5 25 25.5 26 26.5 Time (seconds) 27 27.5 28

Fig. 10.

Power-aware frequency scaling, randomized load.

Server voltage (50 sample average)

12.8 Voltage 12.6 12.4 12.2 12 14 15 16 17 18 19 20 21 22 23 Node Node Node Node 1 2 3 4

relative information processing speed. In particular, a CPU operation here refers to the execution of a Python script which generates a random number and performs a oating point multiplication. As each HTTP request is sent by the trafc generation and control node, the latency for that request is recorded, along with the servers count of its own number of CPU operations. After 400 seconds of operation, the total number of CPU operations is divided by the total latency to attain a measure of net throughput for the conguration. The trafc pattern used for this measurement is generated by running the centralized voltage regulation algorithm on the stacked conguration, and is played back under the parallel conguration. The results of this measurement are summarized in Table I. Computational throughput is given in millions of operations per second and reduction in throughput is taken to be the percent decrease in throughput from the parallel-connected conguration. Despite the fact that this implementation introduces relatively frequent CPU overhead and makes no effort to optimize CPU performance, it is interesting to note that there is only a minor reduction (around 8%) in computational throughput, compared to the parallel case.
TABLE I T HROUGHPUT MEASUREMENTS . Stacked (Mops/s) 0.534 0.729 0.606 0.500 2.368 Parallel (Mops/s) 0.644 0.658 0.711 0.576 2.589 Reduction in Throughput 17.0% -10.7% 14.8% 13.3% 8.5%

CPU utilization coefcient (50 smpl avg) 400 CPU utilization 300 200 100 0 14 15 16 17 18 19 Time (seconds) 20 21 22

Server 1 Server 2 Server 3 Server 4 Total

23

D. Hardware Protection Circuit

Fig. 11. Load balancing and power-aware frequency scaling.

such that no server voltage rose above 12.7 volts. From minutes 16 to 18, the total power supply voltage is lowered below the recommended voltage. In this case, all servers respond by decreasing their operating frequency to maintain a voltage above the 12.2 volt minimum. C. Efciency and Throughput Considerations The notion that removing power converters entirely will result in a large reduction in overall system power losses for many-server stacks is highly intuitive. However, due to the computational overhead and the fact that frequency scaling was managed for voltage regulation rather than for CPU performance, power savings in this implementation are expected to come at the price of decreased computational throughput. To calculate computational throughput, each server is congured to record and report the number of repeated CPU operations it is able to perform during its requested CPU load time. In this case CPU operation is not intended to correspond precisely to a standard benchmark metric, but is used solely for a convenient representation of the servers

As a step toward increasing the robustness and performance of the stacked voltage-domain server cluster, the hardware protection circuit shown in Figure 12 has been implemented. The measurement shown in Figure 14 of the protection circuits terminal voltages under a fault condition illustrates the behavior of the voltage between nodes N + 1 and N 1 in the case of node N failure. Between seconds 10 and 18 (normal operation), the full line current passes through the server load. Between seconds 18 and 24, regulation is no longer maintained (as would be the case in the event of a software crash). At 24 seconds, the load voltage crosses the 14 V threshhold and the protection process begins. At this point, the server and its associated UPS is immediately switched out, effectively reducing the line current to zero. The other servers are able to continue operating off of their UPSs during the period of zero line current. Shortly following the switch-out, the line current is brought back to its full value in a linear fashion over 8 seconds, through closed-loop control (discrete PID) of the power FET labeled Slow Short in Figure 14. During this time, the control node is notied of the fault condition and automatically reduces the output of the

152

From GND, node N 1 To Vcc , node N

16 14
Clamp Voltage Sense

Fault response for 130 watt load Reaches voltage threshold Protection begins

Slow Short Server

12 10 Voltage (V)

Voltage ramps down

MCU (msp430) Low Power Isolation (ADuM5402) To Power Supply 3.3V Shutoff + +

8 6 4 2 0 2 0 5 10 Turn on

Vdd , node N GND, node N To Vcc , node N + 1

Protection script is removed 15 20 25 Time (Seconds) 30 35 40

Fig. 12.

Protection circuit schematic.

Fig. 14. Transient response of slow shorting function of protection circuit.

Server 4 Protection

Server 2 Protection

Server 3 Protection

Server 1 Protection Bypass Circuit

Controller

Fig. 13.

Protection circuit breadboard prototype.

implemented with no additional hardware and only minor throughput reduction. Secondly, the system specications (in terms of voltage regulation and throughput) that have been achieved by the basic algorithms presented here provide practical justication for anticipating a high performance server stack with minimal additional hardware requirements. It is hoped that the results presented in this paper will provide practical impetus and inspiration toward realizing the goal of efcient and reliable power delivery for tomorrows data centers. R EFERENCES

stacks main power supply in accordance with the number of functioning servers. After the line current has been restored, an SCR clamp is turned on for additional safety. IV. C ONCLUSION In contrast to previous research such as [17] and [18] regarding hardware power processing of series-connected voltage domains for low power digital circuits, this research presents a proof of concept that high-power digital loads such as web servers can be regulated in software alone. The software regulation utilizes a centralized algorithm to schedule incoming network load and implements an independent distributed algorithm to scale frequency and provide failsafe server loading. Throughput measurements are made to verify that server performance is not substantially degraded. Given the high requirements for data center reliability and performance [19], this paper also presents a simple design for protection circuit hardware which allows prevents cascaded overvoltage in the case of software failure. The results of the four-server cluster presented in this paper are important for two main reasons. First, the stacked voltage architecture has been demonstrated to be viable for a high power application that is sensitive to voltage imbalance. Of particular interest is that this conguration has been

[1] R. Brown, A. to Save Energy, I. Incorporated, E. Incorporated, and U. E. P. Agency, Report to congress on server and data center energy efciency: Public law 109-431, 2008. [Online]. Available: http://www.escholarship.org/uc/item/74g2r0vg [2] G. Allee and W. Tschudi, Edison redux: 380 vdc brings reliability and efciency to sustainable data centers, IEEE Power Energy Mag., vol. 10, no. 6, pp. 5059, Nov. 2012. [3] N. Rasmussen and J. Spitaels, A quantitative comparison of high efciency ac vs. dc power distribution for data centers, White Paper #127 of APC Inc, 2007. [4] X. Fan, W.-D. Weber, and L. A. Barroso, Power provisioning for a warehouse-sized computer, in Proceedings of the 34th annual international symposium on Computer architecture, ser. ISCA 07. New York, NY, USA: ACM, 2007, pp. 1323. [Online]. Available: http://doi.acm.org/10.1145/1250662.1250665 [5] R. Bianchini and R. Rajamony, Power and energy management for server systems, Computer, vol. 37, no. 11, pp. 6876, nov. 2004. [6] C. Bash and G. Forman, Cool job allocation: measuring the power savings of placing jobs at cooling-efcient locations in the data center, in 2007 USENIX Annual Technical Conference on Proceedings of the USENIX Annual Technical Conference, 2007, p. 16. [7] E. Frachtenberg, A. Heydari, H. Li, A. Michael, J. Na, A. Nisbet, and P. Sarti, High-efciency server design, in Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis, ser. SC 11. New York, NY, USA: ACM, 2011, p. 127. [8] D. Economou, S. Rivoire, C. Kozyrakis, and P. Ranganathan, Fullsystem power analysis and modeling for server environments, in In Proceedings of Workshop on Modeling, Benchmarking, and Simulation, 2006, pp. 7077.

153

[9] U. Ogras, R. Marculescu, P. Choudhary, and D. Marculescu, Voltagefrequency island partitioning for gals-based networks-on-chip, in Design Automation Conference, 2007. DAC 07. 44th ACM/IEEE, june 2007, pp. 110115. [10] R. White, Electrical isolation requirements in power-over-ethernet (poe) power sourcing equipment (pse), in Applied Power Electronics Conference and Exposition, 2006. APEC 06. Twenty-First Annual IEEE, march 2006, pp. 14. [11] IEEE standard for information technologytelecommunications and information exchange between systemslocal and metropolitan area networksspecic requirements part 3: Carrier sense multiple access with collision detection (csma/cd) access method and physical layer specications - section one, IEEE Std 802.3-2008, pp. c1597, 2008. [12] H. Aydin, R. Melhem, D. Mosse, and P. Mejia-Alvarez, Power-aware scheduling for periodic real-time tasks, Computers, IEEE Transactions on, vol. 53, no. 5, pp. 584600, May 2004. [13] V. Sharma, A. Thomas, T. Abdelzaher, K. Skadron, and Z. Lu, Poweraware QoS management in web servers, in Real-Time Systems Symposium, 2003. RTSS 2003. 24th IEEE, 2003, p. 6372. [14] P. S. Shenoy and P. T. Krein, Differential power processing for DC systems, IEEE Trans. Power Electron., vol. 28, no. 4, pp. 1795 1806, Apr. 2013.

[15] B. Weihl, E. Teetzel, J. Clidaras, C. Malone, J. Kava, and M. Ryan, Sustainable data centers, XRDS, vol. 17, no. 4, pp. 812, Jun. 2011. [Online]. Available: http://doi.acm.org/10.1145/1961678.1961679 [16] L. C. Pouchard, J. D. Dobson, and S. W. Poole, Collecting sensor data for high-performance computing: A case study, in International Conference on Parallel and Distributed Processing Techniques and Applications, 2010. [17] P. Shenoy, S. Zhang, R. Abdallah, P. Krein, and N. Shanbhag, Overcoming the power wall: Connecting voltage domains in series, in Energy Aware Computing (ICEAC), 2011 International Conference on, Dec. 2011, pp. 1 6. [18] P. Shenoy, I. Fedorov, T. Neyens, and P. Krein, Power delivery for series connected voltage domains in digital circuits, in Energy Aware Computing (ICEAC), 2011 International Conference on, 30 2011-dec. 2 2011, pp. 16. [19] M. Wiboonrat, An optimal data center availability and investment trade-offs, in Software Engineering, Articial Intelligence, Networking, and Parallel/Distributed Computing, 2008. SNPD 08. Ninth ACIS International Conference on, Aug. 2008, pp. 712719.

154