Transcript IPP - Planning and Optimising the Green IT DC

Green IT Business Transformation Seminar
Planning & Optimising the Green IT Datacentre:
Design, Operation & Management Best
Practices, Technologies & Challenges
Pierre Ketteridge, IP Performance Ltd
1
Introduction
Yes! Of course…
…but only with careful planning, design
and management!
2
Introduction
• The direct carbon impact (ie Carbon Footprint)
of Data Centres on the environment is almost
exclusively related to power consumption
• Data Centres do not (when properly designed
and managed) vent hot air or polluting gases
into the atmosphere – cooling should be a
‘closed system’
• There may be indirect carbon impacts through
staffing levels, travel to and from site,
operational maintenace and housekeeping
3
Introduction
15% of business power consumption is
accounted for by Data Centres & ICT…
Cooling
IT
Components
Power
Distribution
…Lighting accounts for 1-3%,
dependent on whether LO operation
is implemented or not
4
Cooling
Cooling falls into two categories:
• Air Cooling
• Liquid (water) Cooling
5
Cooling> Air Cooling
Air Cooling
The traditional way of cooling a Data Centre
Computer Room:
• CRAC (Computer Room Air Conditioner)
• Water Chiller
• Cold Aisle/Hot Aisle Configuration
6
Cooling> Air Cooling
Inherent limitations of CRAC-based Air
Cooling Systems:
• CRAC capacity needs to be 30% greater than the actual
demand
• Limitations in cooling (5kW – 7kW per rack)
• N+1 active equipment resilience/redundancy drives
efficiency of cooling system down further
7
Cooling> Air Cooling
Some Easy-to-Implement Air Cooling
Optimisation Suggestions:
• Hot Aisle/Cold Aisle Arrangement
• Cold Aisle Containment
• Blanking Panels
• Raised Floor Brush Strips
• Underfloor, Inter- and Intra-rack Cable Management
• Free Air Cooling
8
Cooling> Air Cooling> Hot Aisle/Cold Aisle
• With no hot aisle/cold aisle
arrangement, returning heated
air mixes with the CRAC-cooled
air and cooling to the DC CR
equipment is impaired. There is
also the issue of bypass cold
airflow, which can impact chiller
operation.
• With a hot aisle/cold aisle
arrangement, chilled air is forced
out into the front-of-cabinet facing
cold aisles, across the equipment
surface, and warm air is
channeled out into the rear-ofcabinet facing hot aisle for return
to the chiller/CRAC.
9
Cooling> Air Cooling> Hot Aisle/Cold Aisle
• Ineffective positioning of CRACs impair the
airflow around the DC CR.
• CRACs along the side walls are too close to the
equipment racks, and will cause the airflow to
bypass the floor vents in those cold aisles.
• Place cooling units at the end of the equipment
rows, not mid-row.
• CRACs should be aligned with the hot aisles
to prevent hot/cold aisle airflow crossover,
which apart from increasing the temperature of
air supply to the rack fronts but also can trigger
the cooling unit to throttle back, reducing
cooling overall.
• Limit maximum cooling unit throw distance to
50'
10
Cooling> Air Cooling> Hot Aisle/Cold Aisle
Separation of High-density Racks
• Air cooling systems become ineffective
when high-density racks are co-located
• “Borrowing” of adjacent rack cooling
capacity is not possible in this
configuration
• An alternative (other than self-contained
cooling) is to spread out high-density
racks to maintain the cooling averages
• Obviously this is not always practical –
witness the prevalance of blade server and
virtualisation technologies – two to five
times the per rack power draw of
traditional servers
11
Cooling> Air Cooling> Cold Aisle Containment
Cold Aisle Containment
• Very simple to deploy / Retrofit
• Hot and cold aisles physically
separated
• Greater watts per rack approx 10kW
• Over sizing of the CRAC is reduced
• CRAC efficiency is increased due to a
higher delta T
• CRAC fan speed can be reduced which
provides:
- Reduced running costs
- Increased MTBF
12
Cooling> Air Cooling> Blanking Panels
• Reduction and stabilization of
equipment air-intake temperatures
• Elimination or reduction of the number
and severity of hotspots
• Increased availability, performance,
and reliability of IT equipment, especially
in the top one-third of the equipment
cabinet
• Elimination of exhaust air
recirculation within the cabinet,
optimising cooling and reducing
energy consumption and OpEx
• Deferral of CapEx (additional cooling
capacity)
• The potential of greening the data
center by reducing its carbon footprint
13
Cooling> Air Cooling> Raised Floor Brush Strips
Raised Floor Brush Grommets
• Cable openings allow approx. 60%
of conditioned air to escape
• Use brush grommets to seal every
cabling entry point
• Increases static pressure in the
under-floor plenum - ensures that
the DC airflow remains at a
pressure above atmospheric
• Extend reach of Hot Aisle/Cold Aisle
system
• Self-sealing and interwoven closure system
• Brush grommets can be installed as DC is commissioned, or retro-fitted
• No changes to existing wiring configuration
• Fits into the raised floor tiles prior to cabinet installation
• Simple
• Inexpensive
14
Cooling> Air Cooling> Cable Management
Cable Management – Intra-rack, Inter-rack
and underfloor
•
•
•
•
•
Airflow within racks is also affected by unstructured cabling arrangements
Deployment of high-density servers creates new problems in cable management
Cut data cables and power cords to the correct length – use patch panels where
appropriate
Equipment power should be fed from rack-mounted PDUs
Raised floor/subfloor plenum ducting carries other services apart from airflow:
– Data cabling, power cabling, water pipes/fire detection & extinguishing systems
•
•
•
Remove unnecessary or unused cabling - old cabling is often abandoned
beneath the floor – particularly in high churn/turnover Co-Lo facilities
Spread power cables out on the subfloor - under the cold aisle to minimize airflow
restrictions
Run subfloor data cabling trays at the stringer level in the hot aisle - or at an
“upper level” in the cold aisle, to keep the lower space free to act as the cooling
plenum
15
Cooling> Air Cooling> Free Air Cooling
What is Free Cooling?
Roof-Mounted Free
Air Cooler
Chiller Unit
DC CRAC
16
Cooling> Air Cooling> Free Air Cooling
Average UK Temperatures
25
Average Day
Average Night
15
10
Dec
Oct
Sep
Aug
Jul
Jun
May
Apr
Mar
-5
Feb
0
Nov
5
Jan
Degrees C
20
Month
17
Cooling> Air Cooling> Free Air Cooling
Budgetary Example – Projected Cost of Running the
System for a Year
200C0
20 C
0
20 C
150C
Not using the Free Cooler
• Chiller Capacity
150 kW
• Energy needed to run the chiller
62 kW
• Numbers of Hours running per year
8784
• Cost per kWh
£0.0784
Total Cost of Running per Year
£42,697.00
100% free cooling 70% of the year
• Chiller capacity
• Energy needed to run the chiller
• Numbers of hours running per year
• Cost per kWh
• Cost of running the chiller
• Cost of running Free Cooling (10.4kw)
Total Cost of Running per Year
150 kW
62 kW
2580
£0.0784
£12,540.00
£ 5,058.00
£17,599.00
18
Cooling> Liquid Cooling
High Density Data Centres and Liquid Cooling
• When going above 10kW per rack a new,
more targeted/directed cooling method is
required
• Most common methods is Water Cooling
19
Cooling> Liquid Cooling
So What is Liquid – or Water – Cooling?
• Delivery of chilled water to multiple heat exchange
points from a central unit
• The central unit circulates water from the buildings
existing chilled water loop
• Heat exchange units in rear doors (one per cabinet,
capacity 30kW) or side doors (2 x dual cabinet
resilience, 2 x 15kW)
• Heat is carried away in the water - air is ejected back
out into the DC at the same temperature it entered the
rack - zero thermal footprint
20
Cooling> Liquid Cooling
Why Use Water Cooling?
• Water 3,500 times more thermally efficient than air
• Air cooling only delivers 5-7kW of cooling per rack (10kW
with hot aisle/cold aisle arrangement)
• High Density DCs place increasing power and thermal
control demands on the infrastructure
• Blade servers - up to 80 servers in a standard 42u cabinet
– and anything from 80 to 800 virtual machines!
• Fully-loaded blade server rack can produce 25Kw of heat
• Water Cooling can deliver 30kW of cooling to a fullyloaded 42u rack
21
Cooling> Liquid Cooling
Adding the benefits of Free Cooling, some CapEx/OpEx
implications of Water Cooling:
• Water cooling has a slightly higher install cost (more terminations/
pipe work)…but greater kW per sq ft gives us…
– 35-45% reduction in required real estate
– 15-30% reduction lower in overall construction costs
– 10-20% reduction on total annual fan power consumption
– 12-14% reduction in power delivered to mechanical chilled
water plant
• For an average efficiency data centre, annual savings of £22,000
and £80,000 for small and large data centres respectively
• Significant when the design life of the data centre is 10 years
• Reduction in energy is a reduction in costs and also a reduction in
your carbon footprint
22
Network Components
Active Equipment (Networking)
• Switches
• Routers
• Appliances
– Load balancers
– Caching/Proxying
– Bandwidth Management
– Application Acceleration & Optimisation
23
Network Components> Ethernet LAN Switches
Data Centre Switch Requirements
•
•
•
•
•
•
•
•
Port density
Feeds & Speeds
Performance
Functionality
Capabilities
Feature set
Resilience/Redundancy
Security
Price
Power consumption/Heat output
24
Network Components>
Components Ethernet LAN Switches
Data Centre Switch Requirements
• High port density per chassis
• Low power consumption
Optimised for the environment
• Availability
• High performance
• Low latency
Optimised for the application
25
Network Components>
Components Ethernet LAN Switches
26
Network Components>
Components Ethernet LAN Switches
Ethernet Switch Power Consumption A Comparative Example: 15,000 User Network
Solution
ALU Configuration
Cisco Configuration
LAN edge 48 ports
216 x OS6850
216 x Catalyst 3750
LAN edge 24 ports
160 x OS6850
160 x Catalyst 3750
40 x OS9000
40 x Catalyst 4500
8 x OS9000 Chassis
8 x Catalyst 6500
LAN aggregation
• LAN core
Total
Delta Power Consumption
Cisco/A-L
54 kW/h
48kW/h
102 kW/h
Across an installed network base of 15,000 ports, it was possible to save
102 kW/h, resulting in:
• Lower Power Consumption
• Less Cooling Equipment
• Smaller Batteries
• Smaller Data Centers
27
Network Components> WAN Routers
Routers
• Look at power consumption figures/thermal output
• Deploy shared WAN architecture – MPLS, VPLS, IP VPNs
• Investigate leveraging and integrating bandwidth
optimisation and application acceleration technologies
28
Network Components> Appliances> Load Balancing
LAN/WAN Optimisation Appliances
…an area where we can make a difference, in the way in which technologies
are deployed to optimise LAN/WAN bandwidth usage and availability of backend servers.
An excellent example would be application delivery, traffic management and
web server load balancers:
• High Performance through acceleration techniques
• High Availability
29
Network Components> Appliances> DPI Bandwidth Management
More LAN/WAN Optimisation Options…
DPI Bandwidth Management solutions:
• Inspection, Classification, Policy Enforcement
and Reporting on all traffic:
– Identification - application signature, TCP/UDP port,
protocol, source/destination IP addresses, URL
– Classification – CoS/ToS (IP Prec/Diffserv
CodePoint/DSCP); user-defined QoS policy
– Enforcement based on user-defined policy
– Reporting – RT and long-term – extremely valuable
for SLAs/SLGs in DC environments
30
Network Components> Appliances> WAN Optimisation
LAN/WAN Optimisation Options (cont’d)
WAN optimisation and application acceleration:
• Usually deployed as a reverse proxy device
• Provides some form of bandwidth management
• Protocol optimisation – making LAN protocols more latency-tolerant
– eg. TCP handshake spoofing
• Object caching
– Files, videos, web content, locally cached and served
•
Byte caching
– Repetitive traffic streams, hierarchically indexed and tagged (inline only)
•
Compression
– (inline only)
•
Proxy support for common protocols
– HTTP, CIFS, SSL (HTTPS), FTP, MAPI, P2P, MMS, RTSP, QT, TCPTunnel, DNS etc
31
Network Components> Appliances> WAN Optimisation
LAN/WAN Optimisation Options (cont’d)
WAN optimisation and application acceleration:
• Reverse Proxy
• Bandwidth Management
• Protocol optimisation – for latency-intolerant
LAN protocols
– eg. TCP handshake spoofing
• Object caching
• Byte caching
• Compression (inline only)
• Proxy support for all/most common protocols
32
Infrastructure Management
Managing the Data Centre Infrastructure
“Lights Out” operation requires…
• Little or no human intervention
•Exceptions:
• Planned maintenance
• Fault rectification/management (emergency
maintenance/repair)
• Physical installs/removals
• Housekeeping (cable management, MAC)
• Cleaning
• How are you going to control it? How are you going to
manage it?
33
Infrastructure Management
Remote Control and Management
• RDC, VNC – In Band Management
• Console Servers – Out of Band Management
• KVM switching (local/remote)
• KVM/IP switching & USB2 VM Remote Drive Mapping
• IPMI Service Processor OOB Management
• Intelligent Power Management (iPDUs)
34
Infrastructure Management
Service Processor
Management
(Closed Loop InBand
or Out-of-Band) –
IPMI, iLO, DRAC etc
SMASH CLP
Intelligent Power
Management
(iPDU)
Console Server
Management
(Routers, Switches,
Appliances)
KVM/KVM-over-IP
(Servers, Blade Servers, Management
PCs, Appliance Management Devices)
VNC/RDC
5
35
Summary
Summary - Cooling
• Data Centre “Greening” is mainly down to managing
power consumption
• Cooling is the biggest consumer of power (50%)
• Optimise your air-cooled CRAC system:
–
–
–
–
–
Cold Aisle/Hot Aisle arrangement
Cold Aisle containment
Blanking Panels
Raised floor/underfloor brush strips/grommets
Free air cooling system
36
Summary
Summary – Cooling (Cont’d)
• If deploying high-density bladeservers/virtualisation,
consider water-cooling (max kW/hr cooling rises from 510kW/hr to 30kW/hr)
• Targeted control
• Even distribution of cooling
• Full (42u) rack utilisation
• Zero thermal footprint – design flexibility
• Remember free air cooling reduces costs further
• Real Estate savings
37
Summary
Summary - Active Equipment (Networking)
Switches:
• high port density, low power consumption, PSU disconnect/fanless
operation
• Extrapolate power consumption over entire port count
Routers:
• Modular architecture, high density, low power consumption
• Make full use of available bandwidth
– Shared services: IP VPN, point-to-multipoint or meshed MPLS
– Use/honour QoS marking
– Deploy Bandwidth optimisation techniques
38
Summary
Summary - Active Equipment
(Networking) – Cont’d
Appliances:
• Load Balancing – Maximise performance,
utilisation and availability of server
resources
Maximise performance,
• DPI Bandwidth Management
utilisation and availability
of WAN resources
• WAN Optimisation
39
Summary
Summary – Infrastructure Management
• Remote Infrastructure Control and Management enables “lights-out”
operation
• Remote console management gives CLI access to network
infrastructure – routers, switches, firewalls, other network
optimisation appliances
• KVM-over-IP allows remote, distributed control of server and
workstation systems
• Service Processor Management allows remote control and
management of system processor and environmental
monitors/controls
• Intelligent Power Management enables remote monitoring, control
and management of PDUs, UPS and battery backup resources
40
Close
THANK YOU
Pierre Ketteridge, IP Performance Ltd
[email protected]
[email protected]
www.ip-performance.co.uk
41