Transcript Объединенный институт ядерных исследов

Perspectives of JINR
Grid-infrastructure
V.V. Ivanov, Gh. Adam, V.V. Korenkov, T.A. Strizh, P.V. Zrelov
Laboratory of Information Technologies
Joint Institute for Nuclear Research
Participating in LHC, RHIC, TEVATRON…
In future: FAIR, ILC …
LHC-2008
FAIR-2015
ILC – after 2020 (?)
V. Ivanov (LIT, JINR)
The JINR participation in
the LHC experiments
and in other large-scale
projects asked for a
substantial increase of
its networking and
information resources as
well as the deployment
of a large volume of
work toward the
development of the JINR
Grid-segment and its
integration in the
Russian Gridinfrastructure RDIG
(Russian Data Intensive
Grid).
The Laboratory of Information
Technologies (LIT) is responsible for
the computing infrastructure
development in JINR and hosts the
Central Information and Computing
Complex (CICC) of the Institute.
Further perspective tasks follow from the decision of the JINR Committee of
the Plenipotentiary Representatives, concerning the radical improvement of
the JINR computer telecommunication links with major partner laboratories
in the JINR Member States during the years 2010-2015 and the development
of a unified Grid-environment of the JINR Member States.
This is a cornerstone task of the JINR seven-year plan on 2010–2016 within
the direction ”Networks. Computing. Computational Physics”.
V. Ivanov (LIT, JINR)
Development of the JINR
Grid-environment – 2010-2016
Three levels
within the JINR
Gridenvironment
can be
distinguished:
network,
resource, and
applied.
APPLIED
RESOURCE
NETWORK
V. Ivanov (LIT, JINR)
Cooperation
Collaborators from the JINR Member States (protocols of cooperation with
INRNE (Bulgaria), ArmeSFo (Armenia), FZK Karlsruhe GmbH (Germany),
IHEPI TSU (Georgia), NC PHEP BSU (Belarus), KFTI NASU (Ukraine), etc.).
Grants of Plenipotentiaries of the JINR Member States: Poland, Bulgaria,
Slovakia, Romania. Agreement on Joint Research Programme
“Meshcheryakov-Hulubei” LIT-Romania.
BMBF grant “JINR Network Security, Failover Measures, Network monitoring”.
CERN-JINR Cooperation Agreement on several topics, and JINR in the LCG
Project (participation of JINR in the LCG project).
JINR-South Africa Cooperation Agreement - project “Development of Grid
segment for the LHC experiments”.
Participation in joint projects: Enabling Grids for E-sciencE (EGEE -a project
co-funded by the European Commission through the Sixth Framework
Programme, CERN-INTAS project and "SKIF-GRID“ project of the
Program of the Belarusian-Russian Union State as a part of joint
propositions of NASB and Federal Agency of Science and Innovations of
Russian Federation “The development and use of hardware and software
in GRID-technologies and advanced supercomputer systems SKIF in 20072010 (SKIF-GRID)”.
Joint research grants from RFBR and Ministry of Science and Education with
MSU and SINP MSU, Saratov University, IM SB RAS, IMM RAS, TU Gomel
(Belorussia).
V. Ivanov (LIT, JINR)
June 2009 - two important projects completed
Two large-scale very important projects have been completed:
-- JINR - Moscow 20Gbps telecommunication channel
-- JINR Central Information and Computing Complex major upgrade.
Comprehensive presentation was done at the meeting of the Programme
Advisory Committee for Particle Physics, on June 10.
All organizations involved in the project realizations, together with the
representatives of Ministry of Communications, Federal Agency for Science
and Innovation, attended the Presentation.
Our community colleagues and prospective users - residents of the special
economical zone - also took part in the event.
V. Ivanov (LIT, JINR)
JINR - Moscow 20Gbps
telecommunication channel
Collaboration with RSCC, RIPN, MSK-IX,
JET Infosystems, Nortel
In perspective, the used DWDM and 10Gb Ethernet technologies allow the
creation of up to 80 channels of 10 Gbps each, resulting in a throughput of
up to 800 Gbps.
V. Ivanov (LIT, JINR)
JINR Central Information and Computing Complex
In 2008, the total CICC
performance was 1400 kSI2K,
the disk storage capacity 100 TB
At present, the CICC
performance equals 2300 kSI2K
and the disk storage capacity
500 TB
PLAN
2008
2009
2010
CPU
(kSI2K)
1250
1750
2500
Disk
(TB)
400
800
1200
Tapes
(TB)
-
100
200
V. Ivanov (LIT, JINR)
JINR Local Area Network Backbone (LAN)
• Comprises 6579 computers



and nodes,
Users – 3593,
IP – 7633,
Remote VPN users (Lanpolis,
Contact, TelecomMPK) - 1248;
• High-speed transport (1Gbps)
(Min. 100 Mbps to each PC);
• Controlled-access (Cisco PIX525 firewall) at network
entrance;
• Partially isolated local traffic
(8 divisions have own
subnetworks with Cisco
Plans:
Catalyst 3550 as gateways);
Step-by-step modernization of the
• General network authorization
JINR Backbone – transfer to 10 Gbps system involves many services
Development and modernization of
(AFS, batch systems, Grid,
the control system of the JINR
JINR LAN remote access, etc.)
highway network
V. Ivanov (LIT, JINR)
The plan for the JINR LAN development for the years 2010-2016
foresees several general purpose tasks as well as specific site-related
ones. A selective enumeration points to the:
JINR Laboratories
First
stage
JINR Laboratories
• upgrade of the JINR backbone to a
data transfer rate of 10 Gbps and
gradual connection to it of various JINR
laboratories;
• increase of the performance of the LAN
central telecommunication node kernel;
• increase of the LAN protection at the
hardware level;
• wireless access to the LAN within the
JINR territory;
• implementation of new solutions and
methods for the traffic control,
information security, development of
effective tools for control and the quality
of service of JINR LAN;
• development of the system of
protection of the JINR LAN resources
from non-authorized access.
V. Ivanov (LIT, JINR)
JINR Central Information and Computing Complex
dCache
Servers
Users
servers
CICC
Servers
CICC
Servers
6
5
1
4
2
2
HP
20
Т-platforms
20
JINR Backbo ne
2 x 1 GbE
(2 x 10 GbE - plan)
2
2
A1-12
82 TB
B1-8
144 TB
1 HP Procurve 5412zl-966
2 Hp Procurve 2810-24 G
B9-16
144 TB
RAID
3 Hp Procurve 2810-48 G
3
2
Super Blades
5 x 10
Twins
20 X 2
4 Cisco Catalyst 2970
5 Cisco Catalyst 3750
6 HP Procurve 3500yl-24G
12 12 12 12 12 12
APS
Т-platforms
20
Batch Cluster
V. Ivanov (LIT, JINR)
64
KW
JINR CICC structural scheme
V. Ivanov (LIT, JINR)
LHC Computing Grid Project
The protocol between CERN, Russia and JINR on participation
in LCG Project was approved in 2003. MoU on Worldwide LHC
Computing Grid (WLCG) was signed by JINR in October 2007
The tasks of the JINR in the WLCG:
•WLCG-infrastructure support and development at JINR;
• participation in WLCG middleware testing/evaluation,
• participation in Service/Data Challenges,
• grid monitoring and accounting tools development;
• FTS-monitoring and testing;
• participation in ARDA activities in coordination with experiments;
• JINR LCG portal support and development;
• HEP applications;
• MCDB development;
• support of JINR Member States in the WLCG activities;
• User & Administrator training and education.
V. Ivanov (LIT, JINR)
JINR in the LCG/EGEE

support and development WLCG/EGEE infrastructure;

participation in middleware testing/evaluation,

participation in Data and Service Challenges,

grid monitoring and accounting system development;

FTS-monitoring and testing

MCDB development;

Participation in ARDA activities in coordination with
experiments;

HEP applications;

User & Administrator Training and Education

support of JINR member states in the GRID activities.
V. Ivanov (LIT, JINR)
JINR WLCG infrastructure
CICC comprises:
65 servers
4 interactive nodes
960 computing nodes, Xeon 5150, 8GB RAM (GEthernet)
(160 computing nodes, Xeon X5450, 16GB RAM, InfiniBand).
Site name: JINR-LCG2
Internal CICC network – 1Gbit/sec
Operating system -Scientific Linux CERN 4.6 (transfer to Scientific Linux
SL release 5.2.) ;
Middleware version GLITE-3.1
File Systems – AFS (the Andrew File System) for user Software and
home directories is a world-wide distributed file system. AFS permits to
share easily files in an heterogeneous distributed environment (UNIXes,
NT) with a unique authentication scheme (Kerberos).
dCache- for data.
User registration system – Kerberos 5 ( AFS use Kerberos 5 for
authentication )
V. Ivanov (LIT, JINR)
JINR WLCG infrastructure
JINR provides the following services in the WLCG environment:
Basic services:
- Berkley DB Information Index (top level BDII);
- site BDII;
- 2 x Computing Element (CE);
- Proxy Server (PX);
- 2 x Workload Management System (WMS);
- Logging&Bookkeeping Service (LB);
- RGMA-based monitoring system collector server (MON-box);
- LCG File Catalog (LFC);
- Storage Element (SE), dCache 400 TB, 4 x gridftp door, 14 x pool;
- 4 x User Interface (UI), installed in AFS.
Special Services - VO boxes for ALICE and for CMS; ROCMON;
PPS and testing infrastructure - Pre-production gLite version;
Software for VOs: dCache xrootd door, AliROOT, ROOT, GEANT packages for ALICE;
ATLAS packages; CMSSW packages for CMS and DaVinchi, Gauss packages for
LHCb.
V. Ivanov (LIT, JINR)
dCache in JINR
Internet / GRID
JINR Backbone
local
Protocols gFTP, SRM, XROOT
lxfs07
lxfs07
lxfs07
lxpub01
Admin
WNs
lxfs07
lxfs07
lxfs07
wn…
DCAP
Doors
500 TB
lxfs07
PNFS
lxfs07
lxfs07
lxfs07
lxfs71
V. Ivanov (LIT, JINR)
RAID
lxfs07
lxfs07
lxfs07
rda02
Pools
Frames for Grid cooperation of JINR in 2009











Worldwide LHC Computing Grid (WLCG);
Enabling Grids for E-sciencE (EGEE);
RDIG Development (Project of FASI)
CERN-RFBR project “GRID Monitoring from VO perspective”
BMBF grant “Development of the GRID-infrastructure and tools
to provide joint investigations performed with participation of
JINR and German research centers”
“Development of Grid segment for the LHC experiments” was
supported in frames of JINR-South Africa cooperation
agreement;
NATO project "DREAMS-ASIA“ (Development of gRid EnAbling
technology in Medicine&Science for Central ASIA);
JINR-Romania cooperation Hulubei-Meshcheryakov programme
Project "SKIF-GRID" (Program of Belarussian-Russian Union
State).
Project GridNNN (National Nanotechnological Net)
We work in close cooperation and provide support to our
partners in Armenia, Belarus, Bulgaria, Czech Republic, Georgia,
Germany, Poland, Romania, South Africa, Ukraine, Uzbekistan.
V. Ivanov (LIT, JINR)
Development of the JINR
Grid-environment – 2010-2016
Network level:
links between Moscow and Dubna on the basis of state-of-the-art
technologies DWDM and 10Gb Ethernet.
JINR Local area network :
JINR High-speed backbone construction – 10Gbps
Resource level:
requirements of the LHC experiments stimulate the development of a global
Grid-infrastructure, together with the resource centers of all the cooperating
organizations. First of all, this is of primary concern for such large research
centers as the JINR. To reach effective processing and analysis of the
experimental data, further increase in the JINR CICC performance and disk
space is needed.
V. Ivanov (LIT, JINR)
The RDIG infrastructure
Now the RDIG infrastructure comprises 15 Resource Centers
with > 7000 kSI2K CPU and > 1850 TB of disc storage.
• RDIG Resource
Centres:
– ITEP
– JINR-LCG2
– Kharkov-KIPT
– RRC-KI
– RU-Moscow-KIAM
– RU-Phys-SPbSU
– RU-Protvino-IHEP
– RU-SPbSU
– Ru-Troitsk-INR
– ru-IMPB-LCG2
– ru-Moscow-FIAN
– ru-Moscow-GCRAS
– ru-Moscow-MEPHI
– ru-PNPI-LCG2
– ru-Moscow-SINP
•
•
RDIG monitoring&accounting
http://rocmon.jinr.ru:8080
Monitoring – allows to keep an eye on parameters of
Grid sites' operation in real time
Accounting - resources utilization on Grid sites by
virtual organizations and single users
Monitored values
CPUs - total /working / down/ free / busy
Jobs - running / waiting
Storage space - used / available
Network - Available bandwidth
Accounting values
Number of submitted jobs
Used CPU time
Totally sum in seconds
Normalized (with WNs productivity)
Average time per job
Waiting time
Totally sum in seconds
Average ratio waiting/used CPU time
per job
JINR CICC
Physical memory
V. Ivanov (LIT, JINR)
Average per job
Production Normalised CPU time per
EGEE Regions (June-August 2009)
V. Ivanov (LIT, JINR)
RDIG Normalised CPU time per Site and per VO
(June-September 2009)
V. Ivanov (LIT, JINR)
Production Normalised CPU
time per EGEE site for VO
LHC (June – August 2009)
GRID-site
CPU time Num CPU time/N
-----------------------------------------------------------------------FZK-LCG
6,039,304
8620
700
INFN-T1
3,493,208
2862
1220
CERN-PROD
3,309,795
6812
485
CYFRONET-LCG
3,198,111
2384
1341
PIC
3,037,977
1337
2272
JINR
2,938,039
960
3060
IN2P3-CC
2,668,914
4544
587
RAL-LCG2
2,629,889
2532
1038
UKI-GLASGOW
2,610,116
1912
1365
UKI-LT2-IC-HEP
2,533,230
960
2638
V. Ivanov (LIT, JINR)
24
JINR-LCG2 in RDIG
JINR - 40%
of total RDIG
CPU time
Normalised CPU time
per VO at JINR site
V. Ivanov (LIT, JINR)
Further participation in the WLCG project










Support and development of the JINR WLCG-segment
in frames of global WLCG infrastructure;
Participation in Service/Data Challenges in coordination
with LHC experiments;
WLCG middleware testing/evaluation;
Grid-monitoring the LCG-infrastructure at JINR and
other sites of the Russian Tier2 cluster;
Evaluation of new Grid technologies in context of their
usage in WLCG;
MCDB development (data base structure; basic
modules;
Participation in ARDA activities in coordination with
experiments;
JINR WLCG web-portal improvement;
User’s support to stimulate their active usage of WLCG
resources (courses, lectures, trainings, publication of
user guides in Russian);
Support of JINR member states in WLCG activities.
V. Ivanov (LIT, JINR)
Applied level developments of the
JINR Grid-environment (I)
1. The applied level covers the user applications working in a
virtual organization (VO) environment which comprises
both users and owners of computing resources.
2. In the existing Grid-systems, a VO defines a collaboration of
specialists in some area, who combine their efforts to
achieve a common aim.
3. The virtual organization is a flexible structure that can be
formed dynamically and may have a limited life-time.


VOs working within the WLCG project are the VOs on the LHC
experiments - ATLAS, CMS, Alice, LHCb, the first three being
carried out with the noticeable and direct participation of the
JINR. Nowadays, as a Grid-segment of the EGEE/RDIG, the
JINR CICC supports computations of the virtual organizations
registered in RDIG ( LHC experiments, BioMed, PHOTON,
eEarth, Fusion, HONE, Panda.
In the future, as the interest arises at a large-scale level, VOs
can be organized at JINR in the fields of nuclear physics and
condensed matter physics and, most probably, in the new
promising direction related to the research of the nanostructure
properties.
V. Ivanov (LIT, JINR)
Applied level developments of the
JINR Grid-environment (II)





The creation of new VOs gets
possible and necessary under
maturation of the algorithmic
approaches to the problem solution,
the development of corresponding
mathematical methods and tools:
methods and tools for simulation of
physical processes and analysis of
experimental data software and
computer complexes for
experimental data processing;
numerical methods, algorithms and
software for modeling complex
physical systems;
methods, algorithms and software of
computer algebra;
new computing paradigms;
adaptation of specialized software for
solving problems within the Gridenvironment.
V. Ivanov (LIT, JINR)
Formation of the Grid environment of the
JINR-Member States
Creation of the Grid environment of the JINR-participating countries
will allow to bring out the research underway at JINR in the field of
high energy physics, nuclear physics, solid state and condensed
matter physics, biophysics, biosensor (nano)technologies, etc., on a
qualitatively new level.
Basic directions:

Creation of a working group of the representatives of the JINRparticipating countries for the development of the main directions of
the activity toward forming a joint Grid environment.

Definition, together with the JINR Member States, of the basic
directions of activity toward forming the joint Grid environment.

Preparation of corresponding multi-side agreements.

Development of the network level of the Grid environment of the
JINR Member States.

Development of the resource level of the Grid environment of the
JINR Member States.

Formation of the applied level of the Grid environment of the JINR
Member States.

Preparation and implementation of joint projects.
V. Ivanov (LIT, JINR)
Thank you for your attention !
V.Ivanov (LIT, JINR)