Transcript EELA Documents

E-infrastructure shared between Europe and Latin America
gLite Information System
Valeria Ardizzone
INFN-Catania
1° EELA Grid School
Itacuruçà Island,
State of Rio De Janeiro, Brazil
Outline
E-infrastructure shared between Europe and Latin America
• Grid Information Systems Overview
• Information Systems in gLite
• GLUE Schema
– Grid elements
• LCG Information System
– Hierarchy
– LDAP model
• R-GMA in depth
–
–
–
–
–
Overview
Server Architecture
Producer and Consumer model
Query and Storage Types
R-GMA Browser
• References
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Grid Information Systems Overview
E-infrastructure shared between Europe and Latin America
• Collect information of grid resources
– Discovering new added resources
– Monitoring load and health status
• Publish these information
– Periodically updated
– Well know data model
• Used by
– Users searching a concrete resource
– WMS allocating and managing jobs
– Other monitoring services
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
gLite Information Systems
E-infrastructure shared between Europe and Latin America
• Basic data model.
– Grid Laboratory Uniform Environment (GLUE) Schema.
• Two architectures in glite3
– LCG Information System
 BDII over Globus MDS.
 OpenLDAP interface.
 Preserves LCG-2 compatibility.
– Relational Grid Monitoring Architecture (R-GMA)




Based on the GMA standard from the Grid Global Forum
Information in SQL relational databases
Web Services.
Future replacement of LCG IS.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
E-infrastructure shared between Europe and Latin America
GLUE Schema
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
GLUE Schema
E-infrastructure shared between Europe and Latin America
• A schema of objects and attributes describing Grid
resources and its relationships.
– Originally a EU-DataTAG and US-iVDGL coordinated effort.
– Current participants: EGEE, OSG, Globus and NorduGrid.
– A way to describe Grid info
 Statically and dynamically supplied
 Hierarchically represented
 Independently of the framework (LDAP, XML, SQL…)
• Actual release (1.2) is mapped into
– LDAP
– XML
– Old ClassAd (used by Condor Matchmaking)
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Site Element
E-infrastructure shared between Europe and Latin America
*
Site
Service
A collection of resources owned by
the same organization and
managed by the same
administrator. Contains info on the
location, the administrator, the web
homepage and so on.
1
1
The description of a deployed Web
Service. Contains the URI endpoint
of the WS, the WSDL document, the
list owners and so on.
1
*
*
StorageElement
Cluster
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Cluster Element
E-infrastructure shared between Europe and Latin America
Cluster
A set of heterogeneous resources.
Contains information on shared
temporary directories.
1
1
*
SubCluster
A set of similar resources. Contains
the number of Logical and Physical
CPUs.
Host
Contains detailed static information
of the type of hosts and related
installed software. Data deal with
the type of CPU architecture,
memory sizes, the operating system
installed as well as the type of
network adapter. Furthermore it
contains some information on
performance mesures obtained by
executing well known benchmark
softwares.
1
*
Location
*
ComputingElement
Information on installed softwares,
their path and version
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Computing Element
E-infrastructure shared between Europe and Latin America
ComputingElement
Info
Abstraction of a queue of jobs
Static information on the resource
that deal with the type of Loca
scheduler adopted, the default
Storage Element and so on.
Policy
*
Contains info on configuration
policies. MaxWallClockTime,
MaxRunningJobs, MaxCPUTime . . .
VOview
View for a given Virtual
Organization. Contains authorization
details for VO members and the
amount of available resources.
AccessControlPolicyBase
Set of rules defining access control
policy rules
*
Job
Information on jobs in this queue, its
owner, its local and global ID and its
status
State
Dynamic information on the status
of this queue such as the number of
free CPUs and the Estimated
Traversal Time (ETT)
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
E-infrastructure shared between Europe and Latin America
LCG Information System
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
LCG Information System Levels
E-infrastructure shared between Europe and Latin America
• Resource level: Grid Resource Information Server (GRIS)
– One GRIS on top of each CE, SE, RB, MyProxy (no WNs).
– Sensors and scripts get status of concrete resources statically (e.g.
GlueCEUniqueID) or dynamically (e.g. GlueCEStateWaitingJobs)
• Site level: Grid Index Information Server (GIIS)
– Compiles all the information of the different GRISes in a site.
– LCG-2 recommends using a BDII instead of a GIIS
 Improves robustness and stability.
 Called the site BDII.
• Top level: Berkeley DB Information Index (BDII)
– Keeps all Grid information about the VOs (generally only one).
– Stores information from local BDIIs or GIISes in its database.
– Only queries sites that are included in a configuration file.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
LCG Information System Hierarchy
E-infrastructure shared between Europe and Latin America
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
A LDAP Model
E-infrastructure shared between Europe and Latin America
• Way of collecting info
– Pull model (higher level servers periodically query lower level
servers)
– All servers are based on LDAP
 Inherit hierarchical structure (tree-like)
 LDAP Data Information Format (LDIF)
• Users get info with
– Generic applications
 ldapsearch (MDS: 2135 and BDII:2170 ports)
 Graphical (BDII web; LDAP GUIs)
 Always can get information about specific resources (maybe more
up-to-date) by querying directly the site BDIIs, GIISes or GRISes.
– Querying VO info with lcg-infosites or lcg-info
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
E-infrastructure shared between Europe and Latin America
Relational-GMA
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Grid Monitoring Architecture(GMA)
E-infrastructure shared between Europe and Latin America
• The Producer stores its
location (URL) in the
Registry.
PRODUCER
Store location
• The Consumer looks up
producer URLs in the
Registry.
• The Consumer contacts
the Producer to get all
the data or the Consumer
can listen to the
Producer for new data.
REGISTRY
Transfer Data
CONSUMER
Lookup location
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
R-GMA within Testbed
E-infrastructure shared between Europe and Latin America
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
R-GMA: Schema-Registry-Mediator
E-infrastructure shared between Europe and Latin America
R-GMA Server
VIRTUAL DATABASE
SCHEMA : it holds the
names and definitions of all
of the tables in the virtual
database,
and
their
authorization rules.
TABLE 1,Producer P1 details
TABLE 1, Colum defs
TABLE 2, Colum defs
TABLE 3, Colum defs
TABLE 4, Colum defs
SCHEMA
TABLE 2,Producer P1 details
TABLE 2,Producer P2 details
TABLE 2,Producer P3 details
TABLE 3,Producer P2 details
TABLE 3,Producer P1 details
TABLE 3,Producer P3 details
MEDIATOR
REGISTRY
REGISTRY: It holds the
details of all producers that
are publishing to tables in
the virtual database and it
also holds the details of
“continuous” consumers.
MEDIATOR: a set of rules for
deciding
which
data
providers to contact for any
given query.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
R-GMA: Producer-Consumer
E-infrastructure shared between Europe and Latin America
Producers: are the data providers for the virtual database. Writing data into the
virtual database is known as publishing, and data is always published in complete
rows, known as tuples. There are three types of producer: Primary, Secondary and
On-demand.
R-GMA Server
VIRTUAL DATABASE
P1
SQL “INSERT”
P2
TABLE 1, Colum defs
TABLE 2,Producer P1 details
TABLE 2, Colum defs
TABLE 2,Producer P2 details
TABLE 3, Colum defs
TABLE 2,Producer P3 details
TABLE 4, Colum defs
TABLE 3,Producer P2 details
SCHEMA
TABLE 3,Producer P1 details
MEDIATOR
P3
TABLE 1,Producer P1 details
C1
SQL “SELECT”
TABLE 3,Producer P3 details
REGISTRY
C2
Consumer: represents a single SQL SELECT query on the virtual database. The
query is matched against the list of available producers in the Registry. The
consumer service then selects the best set of producers to contact and sends the
query directly to each of them, to obtain the answer tuples.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Producer Types
E-infrastructure shared between Europe and Latin America
• Primary Producer
Queries
User
Code
Producer
API
Producer
Service
Control
and
inserted
tuples
Tuples
C
Tuple
Storage
• Secondary Producer
P
SELECT *
Tuples
Queries
User
Code
Producer
API
Control only
Producer
Service
Tuples
C
Tuple
Storage
• On-Demand Producer
User
Code
Queries
Tuples
Queries
User
Code
Producer
API
Control only
Producer
Service
Tuples
C
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Query and Storage Types
E-infrastructure shared between Europe and Latin America
•
•
•
Continuous: as soon as new data becomes
available it is broadcast to all interested parties.
Latest: correspond to intuitive idea of current
information.
History: return time sequenced data.
P1
TABLE 1,Producer P1 details
TABLE 2,Producer P1 details
TABLE 2,Producer P2 details
TABLE 2,Producer P3 details
TABLE 3,Producer P2 details
Tuple-store can be in Memory or Database
TABLE 3,Producer P1 details
TABLE 3,Producer P3 details
REGISTRY
P1
LATEST RETENTION PERIOD (LRP) and
HISTORY RETENTION PERIOD (RTP)
Latest-store
allow producers to periodically purge old tuples,
and to give a precise meaning to the “current
state”.
Continuous&History-store
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Continuous
E-infrastructure shared between Europe and Latin America
Producer API
Insert
Producer Servlet
Schema
TableName
SQL “CREATE TABLE”
SQL “INSERT”
Value 1
TableName
Column
Value 2
TableName
Value 1
Value 2
Registry
Continuous
TableName
Consumer API
SQL “SELECT”
TableName
Value 1
Query
Result Set
URL
Predicate
Consumer Servlet
TableName
TableName
Value 1
Value 2
UK
RAL
Alice
Value 2
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
History or Latest
E-infrastructure shared between Europe and Latin America
Producer API
Insert
Producer Servlet
Schema
TableName
SQL “CREATE TABLE”
SQL “INSERT”
Value 1
TableName
Column
Value 2
TableName
Value 1
Value 2
Registry
Query
TableName
Consumer API
SQL “SELECT”
TableName
Value 1
Query
Result Set
URL
Predicate
Consumer Servlet
TableName
TableName
Value 1
Value 2
UK
RAL
Alice
Value 2
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
R-GMA APIs
E-infrastructure shared between Europe and Latin America
• APIs exist in Java, C, C++.
– For clients (servlets contacted behind the scenes)
• They include methods for…
– Creating consumers
– Creating primary and secondary producers
– Setting type of queries, type of produces, retention
periods, time outs…
– Retrieving tuples, inserting data
– …
• You can create your own Producer or Consumer.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
The R-GMA Browser
E-infrastructure shared between Europe and Latin America
• The easiest way to try out R-GMA.
– It is installed on the machine running the Registry and
Schema:
https://egris198.eela.ufrj.br:8443/R-GMA
• Using the Browser you can do the following.
– Browse the tables in the schema.
– Look at the table definitions.
– See all the available producers for a table.
– Query a table.
– Query only selected producers.
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
https://egris198.eela.ufrj.br:8443/R-GMA
E-infrastructure shared between Europe and Latin America
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
References
E-infrastructure shared between Europe and Latin America
• BDII exercises on EGRIS Wiki:
http://egris200.eela.ufrj.br/twiki/bin/view
• GLUE Schema
– http://glueschema.forge.cnaf.infn.it/
• LCG-2 User Guide
– https://edms.cern.ch/file/454439//LCG-2-UserGuide.html
• gLite 3.0 User Guide
– https://edms.cern.ch/file/722398/1.1/gLite-3-UserGuide.pdf
• R-GMA home page
– http://www.r-gma.org/
• R-GMA in EGEE
– http://hepunx.rl.ac.uk/egee/jra1-uk/
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006
Questions…
E-infrastructure shared between Europe and Latin America
1° EELA Grid School, Itacuruçà Island, State of Rio De Janeiro (Brazil), 04-15.12.2006