Transcript Calcolo per LHCb Italia

The 2001 Tier-1
prototype for LHCb-Italy
Vincenzo Vagnoni
Genève, November 2000
Outline





2001 LHCb Italian Tier-1
prototype architecture
Linux diskless nodes howto
Network attached storage
Tests made in Bologna
Conclusions
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
2001 LHCb Italian Tier-1
protoype

15 CPU 2001 (700 SI95) Tier-1 computer
farm (at the beginning of the year, with
further increase if we can show that more
CPU power is needed for the italian groups)



15 single-processor motherboards, 256 MB
RAM each or more, rack-mounted, diskless, with
redundant power supply and cooling.
1 TB IDE disk array in RAID-5 configuration,
hosted in a NAS (Network Attached Storage)
unit (also to be increased if needed).
100 Mbps Ethernet Switch.
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
WAN
Router
Switch
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Motherboards
NAS
Network Attached Storage
2001 Tier-1 Computer Farm Prototype
Components (Motherboards,
Racks, NAS, Switches)
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Advantage of diskless nodes




Easy Operating System installation and maintainance

Adding a machine to the farm requires to run a simple script on
the disk server

The Operating Systems are centralized and accessible through
the file system of the disk server
Enhanced disk fault tolerance

The disk server can have large hard disk arrays with redundant
information by using a RAID 5
No need of UPS battery for the client nodes

Only the disk server needs UPS, no damage if a client is
powered off
System compactness

The nodes are basically simple motherboards with integrated
ethernet cards

They can be arranged in racks, ~25 nodes in only 1 m2.
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Working concept (using for
example a PXE BootROM)




At boot time the diskless client sends its ethernet card MAC
address by making a bootp (or dhcp) broadcast request over the
LAN

The diskless client ethernet controller must be equipped with a
BootROM (for example a PXE, Pre-eXecution Environment,
compliant one)
The bootp (or dhcp) server replies assigning the IP address to the
client and passing the file name of the boot code on the server

Using a PXE BootROM the boot code is a small loader
(pxelinux.bin), provided with the syslinux package, also able to
read a configuration file on the server with some additional
information (i.e. the arguments to pass to the linux kernel)
The client makes a TFTP connection to the server asking to
download the PXE loader and executes it
The loader tftp-downloads the linux kernel, and executes it
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Further details…




The linux kernel needs to be compiled with root-over-NFS
support to be able to mount the root filesystem through
NFS
The Ethernet Card driver must be compiled resident into
the kernel (in principle could be also provided by an initial
ramdisk)
The clients can share some directories of the linux
filesystem (/usr, /opt, …), but need some others to be
private (at least /etc, /dev, /var, /tmp)
Optionally some non-shareable directories (for example
/dev, /var, /tmp) can be mounted on ramdisks
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Network Attached Storage
OpenNAS RS15 RaidZone










File Server supports NFS (also Windows SMB, but
fortunately we don’t need this guy)
Built-in Web Server and DNS Server
R3 RAID5 (Redundancy with Rapid Recovery)
Java-based remote interface for all OpenNAS functions,
including disk array configuration
15 hot-swappable drives bays, with 15 Ultra ATA/100 drives
(80 GB each) + 1 hot-swappable spare
Dual PIII 800MHz CPUs and 256 MB ECC RAM
Dual 100BaseT network connections (Gigabit optional)
Dual redundant 300W hot-swappable power supplies
Possibility of two RAIDZONE disk array expansions (up to 3
TB)
Cost: $20850 (RS15) + optional $19595 (1 TB expansion)
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Tests made in Bologna



Hardware setup
 4 PC HP Vectra, single processor PIII 500 MHz, 256 MB
RAM, one acting as root filesystem disk server and three
as diskless clients
 1 PC HP Kayak, dual processor PIII 733 MHz, 256 MB
RAM, as an additional client with different architecture
Software setup
 Linux RedHat 6.2, upgraded with kernel 2.2.16, or
optionally (to be choosen at boot time) Linux SuSE 6.4
Tests made to evaluate:
 Complexity of installation and administration
 Performance of the system (especially interesting due to
the absence of the system swap area, even if in principle
a network based one could be arranged)
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Installation tests results



Installation trivial
 Once the server system is configured, adding a new
client takes 1 minute
 All the installations are by default identical
No additional network activity observed (monitored directly
into the switch)
 The OSes rarely need to access non cached information
on the server (server disk almost completely inactive)
Robust system
 Simulated network failures for few seconds do not
interfere with the OS work
 Powering off the client machines (without a shutdown!)
doesn’t create any problem at all (no need to check the
disk data integrity at reboot)
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Performance test results





Performance tested by running two HERA-B MonteCarlo
simulation (ARTE) jobs on a dual processor 733 MHz, 256
MB RAM, HP Kayak with diskless installation (running Linux
SuSE 6.4 in this case)
Performance compared with an identical machine with nondiskless installation and local disk swap area
Identical performances! The MC jobs were tuned to allocate
~100 MB RAM each
Even with 256 MB RAM, two heavy jobs can run together
without using a local disk swap area at all
Conclusion: no problem for a single-processor motherboard
equipped with 256 MB RAM running one single job
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni
Conclusions





The 2001 LHCb Tier-1 prototype
architecture is presented
Based on diskless nodes configuration with
a Network Attached Storage disk server
and a 100 Mbps switched ethernet
System proof of principle tested
This kind of system can be an interesting
contribution to the LHC computing
community
It saves time, room and money!
The 2001 Tier-1 prototype for LHCb-Italy
Vincenzo Vagnoni