Transcript transparencies

Enabling Grids for E-sciencE
DGAS: State of the art.
Andrea Guarise
CNAF ,12 Ottobre 2006
www.eu-egee.org
INFSO-RI-508833
A Grid accounting infrastructure
Enabling Grids for E-sciencE
A generic grid accounting infrastructure should cover the requirements of
tree type of customers: Sites, VOs and their users, grid administrators.
While fulfilling the requirements it must also be secure, flexible and scalable.
grid layer
Users/VO layer
grid accounting database
VO accounting database
Site layer
Site accounting database
Resource’s layer
Usage Metering
INFSO-RI-508833
Web reports:
-Aggregate info
-Sites load
-Anonymised statistics
-Aggregate VO info
-Group/Role info
-Detailed User info
-Job level info
-Aggregate site info
-VO (with role/group)
usage on the site.
-Detailed Resource Usage
info
-Job level info
[email protected]
2
TAG: glite-dgas_R_3_1_4
Enabling Grids for E-sciencE
• Metering system:
– The standard sensors: gianduia, pushd and outOfBand have
been improved with many bug fixes.
– The new sensor: dgas-urcollector is furnished as well. It can be
used on CE’s that implements the unfied log file for accounting.
(Best known as Maarten log file)
• HLR server:
– Site HLR and VO HLR are available. Since previous releases
there has been many improvementes to enanche usability and
fix minor problems.
• Query interfaces:
– There are both a local and a remote CLI query interface that can
be used to query the dataset for single job or for aggregate
inforamtion. Both interfaces have been enanched since previous
release to enanche usability and increase the functionalities.
• Regression testsuite.
– A brand new testsuite is available and installed with the HLR. It
helps the system administrator to diagnose most common
problems and misconfiguration,
INFSO-RI-508833
[email protected]
3
Usage record
Enabling Grids for E-sciencE
Grid Job, Gianduia
GridJob, URCollector
Local or Grid, outOfBand
gridJobId
YES
YES
NO
startDate
YES
YES
YES
endDate
YES
YES
YES
userGridId
YES
YES
NO
resourceGridId
YES
YES
NO
vomsFQAN
YES
YES
NO
Virtual Organisation
YES
YES
YES
lrmsId
YES
YES
YES
localUserId
YES
YES
YES
localGroup
YES
YES
YES
cpuTime
YES
YES
YES
wallClockTime
YES
YES
YES
Phisycal Memory
YES
YES
YES
Virtual Memory
YES
YES
YES
SpecInt2K
YES
YES
NO
SpecFloat2K
YES
YES
NO
Cost
YES
YES
NO
lrmsHost + lrmsQueue
YES
YES
YES
INFSO-RI-508833
[email protected]
4
Site HLR
Enabling Grids for E-sciencE
The site HLR is a service designed to manage a set of ‘accounts’ for the
Computing Elements of a given computing site.
For each job executed on a Computing Element (or a on local queue), the
Usage Record for that job is stored on the RDBMS of the site HLR.
Each site HLR can:
• Receive Usage Records from the registered Computing Elements.
• Answer to site manager queries such as:
– Datailed job list queries (with many search keys: per user, VO, FQAN ,CEId…)
– Aggregate usage reports, such as per hour, day, month…, with flexible search
criteria.
•
•
Optionally forward Usage Records to APEL database.
Optionally forward Usage Records to a VO specific HLR.
GOC
Site layer
Site HLR
Resource’s layer
Usage Metering
INFSO-RI-508833
-Aggregate site info
-VO (with role/group)
usage on the site.
-Detailed Resource Usage
info
-Job level info
[email protected]
5
VO HLR
Enabling Grids for E-sciencE
The VO HLR is a service designed to manage a set of ‘accounts’ for the
Users belonging to a Virtual Organisation.
The VO HLR receives a copy of the Usage Records for the jobs executed on
computing sites that authorise the site HLR to forward the records to the
User/VO HLR
Each VO HLR can be queried to obtain information about user Jobs (both
detailed and aggregate views).
Unprivileged users are authorised to acces just to their own records, while
authorised people (VO managers), can access to broader set of records.
Users/VO layer
VO HLR
Site layer
Site HLR
Resource’s layer
Usage Metering
INFSO-RI-508833
-Aggregate VO info
-Group/Role info
-Detailed User info
-Job level info
-Aggregate site info
-VO (with role/group)
usage on the site.
-Detailed Resource Usage
info
-Job level info
[email protected]
6
dgas2apel
Enabling Grids for E-sciencE
• DGAS Usage Records tables can be converted into APEL
LCGRecords table structure.
• This is done with a translation tool nemed ‘dgas2apel’ that
simply translates DGAS Usage Records into a local (to the
HLR) instance of the LCGRecords table.
• This instance of the table can then be sent to the GOC
APEL database. This can be performed using the already
existing APEL producer that uses RGMA to send the sites
LCGRecords table to the GOC.
• The translation tool is already tested and working.
• The version shipped with this version of DGAS is functional
and approved by UK people. Unfortunately there are still
bugs on the APEL side that prevents this to be used.
• Bugs are being investigated by D.Kant.
• This version of dgas2apel do not send the user DN with the
records. Future version will be able to do that since it is
now possible for APEL to encrypt them.
INFSO-RI-508833
[email protected]
7
Soon (?) available features
Enabling Grids for E-sciencE
• There are two important features currently implemented but that still
need (in depth) testing before showing up into a production release:
– Full VOMS integration in query authorisation.
– 2nd level HLR service: service that can be populated with records
originating from other HLRs. (Not covered here, see specific talk.)
• Full VOMS integration in query authorisation:
– Major requirement from the experiments: be able to define at VO
level a ‘Role’ such as ‘vo-manager’, defined in the VOMS
certificate of a given user:
 (e.g. /atlas/Role=vomanager/Group=NULL)
 This information must be used by the accounting system to let that user
access the information for the VO.
– The HLR will have 4 level of authorisation that can be mapped to
wathever string the VO will decide to use instead of ‘vomanager’
as a Role:
 voManager: Access to all the information for the VO of the user sepcified in
the FQAN.
 groupManager: Access to all the information for the Group specified in the
FQAN.
 normalUser: Access to her own detailed records + aggregate records for
the whole VO.
INFSO-RI-508833
[email protected]
8
Conclusion.
Enabling Grids for E-sciencE
• Further information about DGAS can be found at:
– DGAS website:
http://www.to.infn.it/grid/accounting/
– DGAS User Guides:
https://edms.cern.ch/cedar/plsql/doc.info?cookie=3881073&document_id=
571271&version=1
INFSO-RI-508833
[email protected]
9