Transcript Architecture for Modular Data Centers

Where Does the Power go in DCs
& How to get it Back
Foo Camp 2008
2008-07-12
James Hamilton
email :[email protected]
web: http://mvdirona.com
blog: http://perspectives.mvdirona.com
Agenda
• Power is the important measure
– Power drives costs in that Data Center costs are
80% providing power and cooling infrastructure
– Increasing concern about DC power consumption
– Work done/watt
• Power In: Power Distribution & Optimizations
• Servers: Critical Load & Optimizations
• Heat Out: Mechanical Systems & Optimization
7/12/2008
http://perspectives.mvdirona.com
2
Power Distribution: Utility to CPU
• Power Conversions to server (each roughly 98%)
– High (115kVAC) to medium(13.2kVAc) [differs by geo]
– Uninterruptable Power Supply & Generators:
• Running at 13.2VAC
• UPS: can be rotary or battery
– Good ones in 97% range. Much more common 93 to 94%
– Common: rectify to DC, trickle to batteries, then invert to AC (~93%)
– No loss at generators (please don’t start them: ~130gallons/hour * 10 or so)
– 13.2kVACto 480VAC
– 480VAC to 208VAC
• Conversions in Server to CPU & Memory:
– Power Supply: 208VAC to 12VDC (80% common, ~95%
affordable)
– VRM: 12VDC to ~1.5VDC (80% common, 90% affordable)
7/12/2008
http://perspectives.mvdirona.com
3
Power Redundancy at Geo-Level
• Over 20% of entire DC costs is in power redundancy
– Batteries able to supply up to 15 min at some facilities
– N+2 generation (2.5MW) at over $2M each
• Instead, use more smaller, cheaper data centers
• Eliminate redundant power & bulk of shell costs
• Average UPS in the 93% range
– Over 1MW wasted in 15MW facility
7/12/2008
http://perspectives.mvdirona.com
4
Power Distribution Optimization
• Rules to minimize power distribution losses:
1.
2.
3.
4.
5.
Avoid conversions (Less transformer steps & efficient or no UPS)
Increase efficiency of conversions
High voltage as close to load as possible
Size voltage regulators (VRM/VRDs) to load & use efficient parts
DC distribution potentially a small win
•
•
With regulatory issues
Two interesting approaches:
–
–
480VAC (or higher) to rack & 48VDC (or 12VDC) within
480VAC to PDU and 277VAC to load
•
1 leg of 480VAC 3-phase distribution
• Common design: 44% lost in distribution
– 1*.98*.98*.93*.98*.8*.8 => 56% (~4.4MW lost on 10MW total)
– Affordable technology: 1*.99*.99*.95*.95 => 88% (~1.2MW total)
7/12/2008
http://perspectives.mvdirona.com
5
Critical Load Optimization
• Power proportionality is great but “off” is even better
– Today: Idle server consumes ~60% power of full load
– Industry secret: “good” data center server utilization around ~30%
– Off requires changing workload location
• What limits 100% dynamic workload distribution?
– Networking constraints
• VIPs can’t span L2 nets, ACLs are static, manual configuration, etc.
– Data Locality
• Hard to efficiently move several TB & workload needs to be close to data
– Workload management:
• Scheduling work over resources optimizing for power with SLA constraint
• Server power management
– Most workloads don’t fully utilize all resources on server
– Need ability to shut off or de-clock unused server resources
– Very low power states recover more quickly
• Move from 30% utilization to 80%
7/12/2008
http://perspectives.mvdirona.com
6
CEMS: Thin Slice Computing
• Cooperating Expendable Micro-Slice Servers
– Correct system balance problem with less-capable CPU
• Too many cores, running too fast for memory, bus, disk, …
– Power consumption scales with cube of clock frequency
• Goal: ¼ the price & much less than ½ the power
– Utilize high-volume client parts in server environment
– Goal: 20 to 50W at under $500
– 1U form factor or less with service-free design
• Longer term goals
– High-density, shared power supply & boot disk
– Eliminate non-server required components
– Establish viability of service free designs
7/12/2008
http://msblogs/JamesRH
7
Conventional Mechanical Design
• Server fans (from components to air)
• CRACs: (from air to chilled water)
– Air moving over long distance expensive
– Air control often poor with hot/cold mixing
• Secondary water circuit (variable flow)
• Primary water circuit (fixed flow)
– Water side economizer & A/C evaporator
• Condensate circuit
– A/C condenser
– Water side economizer
– Cooling tower
7/12/2008
http://perspectives.mvdirona.com
8
Mechanical Optimization
• Simple rules to minimize cooling costs:
1.
2.
3.
4.
5.
•
Best current designs have water close to load but don’t use
direct water cooling
–
•
•
Raise data center temperatures
Tight control of airflow with short paths
Cooling towers rather than A/C
Air side economization (open the window)
Low grade, waste heat energy reclamation
Lower heat densities could be 100% air cooled but density trends
suggest this won’t happen
Common mechanical designs: 24% lost in cooling
Assume reduction to 1/3 current
–
7/12/2008
24% to 8% for 16% savings
http://perspectives.mvdirona.com
9
Summary
• Some low-scale facilities incredibly bad
• Assuming current high-scale installation:
– Power distribution savings ~32%
• Save 8% in power distribution to server
• Save further 24% power distribution losses in server
– Cooling Savings: ~16%
• Conservatively estimate 1/3 the power using air-side economization
• 24% loss down to 8% for a 16% power savings
– Server Utilization: ~90%
• Move from 30% to 80% through DC-wide workload scheduling
• 30% load @ 60% of full load power to 80% load @ 100% of full load power
• 2.6x work at 1.7x more power for a gain of 90%
– Cooperative, Expendable, Micro-slice Servers: ~12%
• ½ the power but less capable server (most workloads are memory or disk I/O bound)
• Conservatively assume .8x work done .5x power => 30% savings
• 4.0x gains in work done/watt look attainable:
– 1*1.32*1.16*1.90*1.30 => 3.8x (some overlap between CEMS & power dist savings)
• Power is #3 expense in DC behind server h/w, power distribution & cooling
– Data center capital expense savings nearly 100% driven by power
• Reductions in power reduce capex, opex & is good for environment
7/12/2008
http://perspectives.mvdirona.com
10
Slides
• These Slides:
– http://mvdirona.com/jrh/TalksAndPapers/JamesRH_DCPo
werSavingsFooCamp08.ppt
• Perspectives Blog:
– http://perspectives.mvdirona.com
7/12/2008
http://msblogs/JamesRH
11