Transcript EELA Documents

E-infrastructure shared between Europe and Latin America
Usage of R-GMA
and
Grid Application Monitoring
Valeria Ardizzone
INFN-Catania
1° EELA Grid School
Itacuruçà Island,
State of Rio De Janeiro, Brazil
Outline
E-infrastructure shared between Europe and Latin America
• Usage of R-GMA command line interface
• Usage of R-GMA Web Browser like a Consumer
• Usage of R-GMA cli tool into jobs
• Usage of R-GMA API to create Producer and
Consumer
• Usage of Producer and Consumer within application
• References
E-infrastructure shared between Europe and Latin America
R-GMA command line tool
R-GMA Command Line Tool
E-infrastructure shared between Europe and Latin America
•
Before you start the R-GMA command line tool make sure you have a proxy
certificate :
> voms-proxy-init --voms gilda
Your identity: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Valeria Ardizzone/[email protected]
Enter GRID pass phrase:
Creating temporary proxy ............................................................ Done
Contacting voms.ct.infn.it:15001[/C=IT/O=GILDA/OU=Host/L=INFN
Catania/CN=voms.ct.infn.it/[email protected]] "gilda" Done
Creating proxy .................................... Done
Your proxy is valid until Fri Jun 16 08:20:14 2006
•
Run the command and you should receive the following message on startup:
> rgma
Welcome to the R-GMA virtual database for Virtual Organisations.
================================================================
Your local R-GMA server is:
https://egris198.eela.ufrj.br:8443/R-GMA
You are connected to the following R-GMA Registry services:
https://egris198.eela.ufrj.br:8443/R-GMA/RegistryServlet
You are connected to the following R-GMA Schema service:
https://egris198.eela.ufrj.br:8443/R-GMA/SchemaServlet
Type "help" for a list of commands.
rgma>
Entering Command
E-infrastructure shared between Europe and Latin America
• Commands are entered by typing at the > rgma
prompt and hitting ‘enter’ to execute the command.
• A history of the commands executed can be accessed
using the Up and Down arrow keys.
• To search a command from history use CTRL-R and
type the first few letters of the command to recall.
• Command autocompletion is supported (use Tab
when you have partly entered a command).
General Commands
E-infrastructure shared between Europe and Latin America
• help and help <command>
Display general help information or specific for a command.
• exit or q
Exit from R-GMA command line interface.
• show
…...
• set
…...
• clear history
Clear the current session history of commands executed.
• write history <filename>
Write the session command history to a file.
• write results <filename>
Write query results to a file.
• read <filename>
Show all information about the structure of a table
Batch mode
E-infrastructure shared between Europe and Latin America
The command line tool can be used in batch mode in three
ways:
• rgma –c <command> [ -c <command> …]
Executes the command and exit.
• rgma –f <filename>
Executes commands in file sequentially and exit.
• Commands embedded in a shell script:
#!/bin/sh
$RGMA_HOME/bin/rgma <<EOF
set query latest
select Service_Name, Status FROM ServiceStatus
WHERE
Status != 0
EOF
Table Commands
E-infrastructure shared between Europe and Latin America
• Show tables
Display the name of all tables existing in the Schema
• Describe <tablename>
Show all information about the structure of a table
• create table
Create a table in the R-GMA virtual database schema.
• drop table <tablename>
Delete a table in the R-GMA virtual database schema.
Exercise 1: create a table
E-infrastructure shared between Europe and Latin America
rgma> create TABLE ITATutTable(COD_Test INT PRIMARY KEY, Application
VARCHAR(20), Status VARCHAR(10), PercStatus INT, Owner VARCHAR(25))
rgma> show tables
…….
| NetworkUDPPacketLoss
| NetworkFileTransferThroughput
| ITATutTable
|
|
|
rgma> describe ITATutTable
+-----------------+------------+-------------------------+----------------+------------------+
| Column name
| Type
| Primary key | Can be NULL |
+-----------------+-------------+------------------------+----------------+------------------+
| COD_Test
| INTEGER
| Yes
| No
|
| Application
| VARCHAR(20)
| No
| Yes
|
| Status
| VARCHAR(10)
| No
| Yes
|
| PercStatus
| INTEGER
| No
| Yes
|
| Owner
| VARCHAR(25)
| No
| Yes
|
| MeasurementDate
| DATE
| No
| No
|
| MeasurementTime
| TIME
| No
| No
|
+-----------------+------------+-------------------------+----------------+------------------+
Producer Properties
E-infrastructure shared between Europe and Latin America
• Using the command line tool you may work with one producer at a
time.
• The current producer type can be displayed using:
rgma>show producer
• Set the latest retention period for tuples published by the producer
rgma>set producer latestretentionperiod|lrp <time value> [<units>]
•
Set the history retention period for the producer. If the producer does
not support history queries this command has no effect.
rgma>set producer historyretentionperiod|hrp <time value> [<units>]
• Producer which handles the INSERT statement. The SQL INSERT
statement may be used to add data to the system:
rgma> INSERT INTO <Tablename> VALUES (’a’, ’b’, ’c’, ’d’)
Exercise 2: create a Producer
E-infrastructure shared between Europe and Latin America
rgma> set producer latest
Producer type : continuous latest
rgma> set producer latestretentionperiod 50 seconds
Set producer LRP to 50 seconds
rgma> set producer historyretentionperiod 2 minutes
Set producer HRP to 2 minutes
rgma>describe ITATutTable
rgma> insert INTO ITATutTable
values(001,'TestProducer','Start',10,'Valeria');
Inserted 1 row into ITATutTable
Consumer Properties (1)
E-infrastructure shared between Europe and Latin America
The behaviour of Consumer varies according to the type of query
being executed. In R-GMA there are three basic types of query:
LATEST Queries: only the most recent tuple for each
primary key
HISTORY Queries: all historical tuples for each
primary key
CONTINUOUS Queries: returns tuples continuously as
they are inserted.
• The type of query can be changed using the SET QUERY command
as follow:
rgma> SET QUERY LATEST | CONTINUOUS | HISTORY
• The current query type can be displayed using
rgma> SHOW QUERY
Consumer Properties (2)
E-infrastructure shared between Europe and Latin America
• The maximum age of tuples to return can also be controlled. To limit
the age of latest or historical tuples use the MAXAGE property.
rgma> SET MAXAGE <time> seconds|minutes|hours|days
• The current maximum tuple age can be displayed using
rgma> SHOW MAXAGE
• To disable the maximum age, set it to none:
rgma> SET MAXAGE none
Consumer Properties (3)
E-infrastructure shared between Europe and Latin America
• The final property affecting queries is timeout.
– For a latest or history query the timeout exists to prevent a problem
(e.g. network failure) from stopping the query from completing.
– For a continuous query, timeout indicates how long the query will
continue to return new tuples. Default timeout is 1 minute and it can be
changed using
rgma>SET TIMEOUT <time> seconds|minutes|hours|days
• The current timeout can be displayed using
rgma>SHOW TIMEOUT
Consumer Properties (4)
E-infrastructure shared between Europe and Latin America
• Querying data uses the standard SQL SELECT statement:
rgma> SELECT * FROM <table_name>
• Set the output format for results. 'table' formats the results in a table,
tsv outputs tab-separated results and csv outputs comma-separated
results:
rgma> SET output table|tsv|csv
output csv:
1,TestProducer,Start,10,Valeria,2006-12-07,22:07:36,
set output tsv:
1
TestProducer
Start 10
Valeria 2006-12-07
22:07:36
Exercise 3: create a Consumer
E-infrastructure shared between Europe and Latin America
rgma> set query latest
Set query type to latest
rgma> set maxage 1 minutes
Set max age to 1 minutes
rgma> set timeout 50 seconds
Set timeout to 50 seconds
rgma> set output tsv
Set output format to 'tsv‘
rgma>select Application,Status FROM ITATutTable
TestProducer Start
TestProducer Step1
TestProducer Step2
Exercise 4: Producer & Consumer
E-infrastructure shared between Europe and Latin America
Continuos Producer and Consumer:
(NOTE. Open 2 rgma client tool, one for Consumer the other for Producer.)
•
Consumer’s client:
rgma> set query continuous
rgma> set timeout 50 seconds
rgma> set maxage 30
rgma> set output csv
rgma> select * from ITATutTable
rgma>
• Producer’s client:
rgma> set producer continuous
rgma> insert INTO ITATutTable values(004,'TestProducer','Step3',30,'Valeria');
rgma> insert INTO ITATutTable values(005,'TestProducer','Step4',40,'Valeria');
rgma> insert INTO ITATutTable values(006,'TestProducer','Step5',50,'Valeria');
rgma> insert INTO ITATutTable values(007,'TestProducer','Step6',60,'Valeria');
rgma> write history Prod_comm.rgma
Exercise 5: Producer & Consumer
E-infrastructure shared between Europe and Latin America
Continuos Producer and Consumer:
(NOTE. Open 1 rgma client tool only for Consumer. Open one shell
for Producer.)
• Consumer’s client:
rgma> set query continuous
rgma> set timeout 60 seconds
rgma> set output csv
rgma> select * from ITATutTable
• Producer’s shell:
rgma -c “set producer continuous”
rgma -c “insert INTO ITATutTable values(004,‘A','Step1',100,'Valeria');”
rgma -c “insert INTO ITATutTable values(005,‘B','Step2',100,'Valeria');”
Exercise 6: Producer & Consumer
E-infrastructure shared between Europe and Latin America
Continuos Producer and Consumer:
(NOTE. Open 1 rgma client tool only for Consumer. Open one
shell for Producer.)
• Consumer’s client:
rgma> set query continuous
rgma> set timeout 60 seconds
rgma> write results Results.rgma
rgma> select * from ITATutTable
• Producer’s shell:
rgma -f Prod_comm.rgma
E-infrastructure shared between Europe and Latin America
R-GMA Web Browser
https://egris198.eela.ufrj.br:8443/R-GMA
E-infrastructure shared between Europe and Latin America
3
1
2
https://egris198.eela.ufrj.br:8443/R-GMA
E-infrastructure shared between Europe and Latin America
3
5
4
1
2
E-infrastructure shared between Europe and Latin America
4
3
1
2
E-infrastructure shared between Europe and Latin America
Jobs using R-GMA commands
script.sh
E-infrastructure shared between Europe and Latin America
#!/bin/sh
echo "I'm doing application A...now I finished."
$RGMA_HOME/bin/rgma <<EOF
insert INTO ITATutTable values(001,'A','Step1',100,'Valeria');
EOF
echo "I'm doing application B...now I finished."
$RGMA_HOME/bin/rgma <<EOF
insert INTO ITATutTable values(002,'B','Step2',100,'Valeria');
EOF
echo "Bye bye"
my.JDL file
E-infrastructure shared between Europe and Latin America
Type = "Job";
JobType = "Normal";
Executable=“/bin/sh";
Arguments = “script.sh";
StdOutput="stdout.log";
StdError="stderr.log";
InputSandbox={“script.sh"};
OutputSandbox={"stdout.log","stderr.log"};
Exercise 7
E-infrastructure shared between Europe and Latin America
>edg-job-list-match myJDL.jdl
>edg-job-submit –o jobID myJDL.jdl
>edg-job-status –i jobID
>edg-job-get-output –i jobID
Continuos Consumer Properties:
>rgma
rgma> set query continuous
rgma> show query
rgma> set timeout 3 minutes
rgma> select * from ITATutTable
E-infrastructure shared between Europe and Latin America
R-GMA API to create
Producer and Consumer
User Producer and Consumer
E-infrastructure shared between Europe and Latin America
API available for Java, C, C++ and Python
Users may by-pass API if they wish, but API is the easiest way to use R-GMA services
Use Case for Monitoring Application
E-infrastructure shared between Europe and Latin America
• Ingredients:
• An application, called A, that does something and
produce data;
• A producer object used by A application to publish its
relevant data;
• A script to prepare the environment and launch the A
application;
• A JDL;
• A rgma client or R-GMA Web Browser to act like a
Consumer.
My Producer (Java): PP (1)
E-infrastructure shared between Europe and Latin America
Producer
Properties
Type: PRIMARY
Storage type: MEMORY
Termination Interval: 20 (minutes)
Query type: CONTINUOUS
Latest Retention Period: 60 (minutes)
History Retention Period: 60 (minutes)
. . . . . . . . . .
ProducerProperties props = null;
if (producerType.equals("CONTINUOUS"))
{
props = new ProducerProperties(Storage.MEMORY, 0);
}
else if (producerType.equals("LATEST"))
{
props = new ProducerProperties(Storage.DATABASE, ProducerProperties.LATEST);
}
else if (producerType.equals("HISTORY"))
{
props = new ProducerProperties(Storage.DATABASE, ProducerProperties.HISTORY); }
else
{
System.err.println("Invalid producer type (" + producerType + ").");
System.exit(1);
}
My Producer (Java): PP (2)
E-infrastructure shared between Europe and Latin America
. . . . . . . . . .
PrimaryProducer pp = null;
ResourceEndpoint endpoint = null;
Try
{
ProducerFactory factory = new ProducerFactoryStub();
TimeInterval ti = new TimeInterval(20, Units.MINUTES);
pp = factory.createPrimaryProducer(ti, props);
String predicate = "WHERE Owner = '" + owner + "'";
TimeInterval historyRP = new TimeInterval(60, Units.MINUTES);
TimeInterval latestRP = new TimeInterval(60, Units. MINUTES);
pp.declareTable(tableName, predicate,historyRP,latestRP);
...
My Producer (Java)(3)
E-infrastructure shared between Europe and Latin America
. . . . . . . . . .
public void insert(String status, int percstat){
String insert = "INSERT INTO ITATutTable (COD_Test, Application,
Status, PercStatus,Owner) VALUES (" + cod +",'" + appname +"','"
+ status + "'," + percstat + ",'" + owner + "')";
try {
pp.insert(insert);
} catch
. . . . . . . . . .
public void close()
{
try {
pp.close();
}
Application: A
E-infrastructure shared between Europe and Latin America
public class A {
public static void main(String[] args){
long startTime = System.currentTimeMillis() / 1000;
if (args.length != 3) {
System.err.println("Usage: java A <Cod_Test> <Owner> <producerType>");
System.exit(1); }
PP mytest = new PP(new Integer(args[0]).intValue(),args[1],args[2]);
for( int i = 0, j = 0; i <= 10; i++, j+=10)
mytest.insert("Step"+i,j);
mytest.close();
long endTime= System.currentTimeMillis() / 1000;
System.out.println("resourceId=" + mytest.getResourceID() );
System.out.println("startTime=" + startTime);
System.out.println("endTime=" + (System.currentTimeMillis() / 1000));
}}
Script: startAPP.sh (1)
E-infrastructure shared between Europe and Latin America
if [ -z "$RGMA_HOME" ]
then
export RGMA_HOME=$EDG_LOCATION
fi
export LD_LIBRARY_PATH=$RGMA_HOME/lib:$EDG_LOCATION/externals/lib
#Java Environment
if [ "$X509_USER_PROXY" != "" ]
then
securityProp="-DX509_USER_PROXY=$X509_USER_PROXY"
else
securityProp="-DTRUSTFILE=$TRUSTFILE"
fi
CLASSPATH="$CLASSPATH:$RGMA_HOME/share/java/glite-rgma-apijava.jar:$RGMA_HOME/share/java/glite-rgma-stubs-servletjava.jar:$GLITE_LOCATION/share/java/glite-securitytrustmanager.jar:$GLITE_LOCATION/share/java/glite-security-utiljava.jar:$GLITE_LOCATION/externals/share/java/bcprov-jdk14122.jar:$GLITE_LOCATION/externals/share/java/log4j.jar:."
export CLASSPATH
Script: startAPP.sh (2)
E-infrastructure shared between Europe and Latin America
COMM_EXE=A.class
chmod +x $COMM_EXE
# Arguments
cod=$1
owner=$2
queryType="$3"
if [ "$JAVA_HOME" != "" ]
then
COMM_JAVA=$JAVA_HOME/bin/java
else
COMM_JAVA=java
fi
$COMM_JAVA -classpath $CLASSPATH:. -DRGMA_HOME=$RGMA_HOME $securityProp A $cod
$owner $queryType
if [ $? -ne 0 ]
then
echo "Program $cod reported failure." 1>&2
exit 1
fi
JDL with User Producer Application
E-infrastructure shared between Europe and Latin America
Type = "Job";
JobType = "Normal";
Executable=“/bin/sh";
Arguments = "startAPP.sh 1001 Valeria CONTINUOUS";
StdOutput="stdout.log";
StdError="stderr.log";
InputSandbox={"startAPP.sh","pp.class“,”A.class”};
OutputSandbox={"stdout.log","stderr.log"};
More information
E-infrastructure shared between Europe and Latin America
• R-GMA overview page.
http://www.r-gma.org
• R-GMA documentation in EGEE
http://hepunx.rl.ac.uk/egee/jra1-uk/
• R-GMA in E-GRIS Site
https://egris198.eela.ufrj.br/R-GMA
• R-GMA API in E-GRIS Wiki Site
E-infrastructure shared between Europe and Latin America
Questions…