Transcript IT-division

IASA IT division
Marios Chatziangelou
Technical Manager
Division key areas
Division expertise: focused on three main areas:
• (Inter)National IT services (EU/National projects)
• Institutional-level IT services
• Software Development
2
Activities
An overview of the IT team/division activities:
ITdivision
Network
Operations
Center
Collaboration
Services
Helpdesk
Clustering
Grid
Computing
(1st-2nd line support )
Institutional IT
Services
Multithreaded
applications
Parallel
programming
Cloud
Computing
Inter-National IT
Services
(EU/National
projects)
HPC
Software
development
Client/Server
applications
Web
applications/
services
Embedded/RT
SW
Databases
Digital
Repositories
Drivers
Databases
3
(Inter)National IT services
Introduction
• EU-funded projects
• i.e. GRIDCC, EGEE I-II-III, EGI, PRACE etc.
• Nationally-level projects
• i.e. HellasGrid I-II, HellasHPC etc.
• Current responsibilities/active roles (Grid-Computing)
• EGI task leader for User Community Technical Services
• Regional Coordinator for the operations in the South Eastern Europe
• Country representative/coordinator for HellasGrid
• Experienced WP leader in numerous EU projects
• Preparation of EU proposals (related to Grid and/or Infrastructures)
4
Grid Computing (1)
Joint Research Unit on the EGI-InSpire project - the biggest, 4-year, panEuropean project on Grid Computing (total project budget aprox. 100M€)
• Operation of two production-level Grid Clusters
since 2005 (HG-02-IASA and GR-06-IASA)
• Major stakeholder in HellasGrid
(Greek National Grid Initiative)
• Since 2005 our clusters handled:
> 2.6M CPU hours; >1M concrete jobs
• Extensive knowledge of gLite, UNICORE, ARC
middleware
• Testing and certification of grid/distributed
services and middleware
• Site & Region monitoring
(stand-alone/hierarchical mode mainly using
open-source frameworks i.e. Nagios, Ganglia, Cacti etc.)
• Experienced in distributed systems security
5
Grid Computing (2)
Infrastructure:
Computing/storage
>300 cores
>20TB reliable storage
Networking
1Gigabit connectivity
CISCO, eXtreme & 3COM stack switches
General
>150kBTU - A/C
>100kVA – UPS
>160kVA – outdoor power-generator
permanent connection to the
UoA power-safe network
6
Grid Computing (3)
Software:
• Operating Systems
(Scientific Linux/Redhat, Debian, Fedora …)
• Grid Middleware
(gLite 3.2)
• Batch systems/Schedulers
(OpenPBS/PBS, LSF, Condor, MAUI
– experimenting on SGE on Grid)
• Information System
(OpenLDAP based architecture)
• Mass deployment/configuration
(Quattor, OSCAR, Rocks)
• Grid/Cluster Monitoring
•Updates management
(Nagios, Ganglia)
(Pakiti)
•Network monitoring
• Central logging
(SNMP-MRTG)
(syslog-ng)
7
Digital Repositories (1)
Development & hosting of the EGI-InSpire International Software Repository
[http://repository.egi.eu/]
(joint effort with GRNET, AUTH, ICCS)
Release management of *all* Grid Middleware Distros
(gLite, ARC, UNICORE,
UMD [Unified Middleware Distribution])
Full support of the EGI release management process
•Quality criteria verification process
•Stage-Rollout process
•Production level software provision
8
Digital Repositories (2)
Other software handled/managed by the Repo:
• Globus
• Grid Operations Tools
• NAGIOS Grid monitoring suite
• European Certificate Authorities packages
• Jpackage (mirroring)
• g-Eclipse
• Ganga
• Migrating Desktop
and many other ……
9
Digital Repositories (3)
Three-tier architecture:
Data Tier
• ESciDoc (fedora-commons distributed repos arch)
• Web services based API (SOAP & REST)
Business Tier
• TAVERNA workflow management system
• Java & perl combined implementation
• Web services based API (SOAP & REST)
• Request Tracker (RT) for SW Release workflow
(deposit & verification processes)
Presentation Layer
• For end-users
EPrints & DuraSpace implementation
• For automated mass package installation
Multiple (hundreds) YUM/APT repositories provision
10
International databases (1)
IASA is responsible for the Development/Hosting/Maintenance of the
International Grid Applications Database [http://appdb.egi.eu]
The EGI Applications Database (AppDB) is a full-edged, community driven web
portal, meant to act as
• a point-of-reference for scientic applications and tools available within the
EGI infrastructure (over 260 registered Applications and Tools)
• registry of persons involved in the development of said software (over 500
registered scientists/developers)
Main aims are to:
• promote the significance of EGI/Grid in the e-Science
• facilitate the reuse of applications and expertise that is available in NGIs
• help create a vibrant user community, with information owing inwards
directly from involved individuals - developers and end-users alike
Descendant of the EGEE Applications Registry portal, developed and provided by
INFN and IASA. Since the EGI era, the AppDB came under sole IASA responsibility
11
International databases (2)
[http://appdb.egi.eu]
12
International databases (3)
Technically speaking ….
Zend Framework
• PHP basis for the Model-View-Controller (MVC) architectural pattern
• Supports Table Data Gateway design patterns
• Integrates with the dojo toolkit and jQuery
dojo toolkit
• High-end UI experience through advanced widgets
jQuery
•Application" look-and-feel through full use of AJAX
•Use of jSON for data interchange
MySQL data back-end
• InnoDB storage engine (triggers, referential integrity)
• Extended use of optimized Views (Table/Row Data Gateway DP)
•SOAP/REST web-services for integration with external sources
13
HPC & Cloud computing
High Performance Computing:
• Participation to the international project PRACE-1P
• Major stakeholder of the HellasHPC consortium (IASA, ICCS, AUTH,
UPatras, led by GRNET)
Mainly focused on:
• Benchmark suite for Tier-0 procurement
• Integration of codes into the applications benchmark suite
• Synthetic benchmark suite
• Performance analysis tools
Cloud Computing (experimental phase)
•Mainly focused on offering Grid services/resources over Cloud.
14
Institutional-level IT services
•
•
•
•
Local Network Operations Center
Provision of many Collaboration services
Operation of a Helpdesk
Provide clustering services
Local Network Operations Center
• Our Local NOC offers the following services:
– eMailing services: MTA (Postfix), (s)IMAP (courier), Web mail (squirrel
& horde), central AV (ClamAV), centrall anti-spamming control
(spamassasign), mailing lists (mailman)
– Ldap catalog for local users auth/authz
– Virtual Private Network (CA based auth/authz)
– DNS services (4 domains, 2 private – 2 public)
– Web services (two Web servers on a Round-Robin mechanism)
– A local Certification Authority (IASA CA)
– Central firewall system
• Gigabit network connectivity
• All of our systems are centrally monitored:
– Nagios, Gaglia, Cacti
• Central logging facility it is also available
– Syslog-ng, with advanced filters enabled for immediate alerting
Collaboration Services & Helpdesk
Collaboration services
• Wiki services (MediaWiki, Twiki)
• TRAC/SVN for covering the SW development life cycle.
• Open-xChange facility (an open-source MS exchange replacement)
• Internal Document Management System
• 2 TB of reliable Samba store (internal) with “on access” antivirus scanning
(ClamAV)
• eGroupware collaboration suite
And many other….
Helpdesk
• Main aim, is to support and keep track of, our local users request.
• The Helpdesk workflow, It is fully covered by
the Request Tracker (RT) ticketing system.
• More than 65.000 tickets since 2005, originated from:
– Local users (requests)
– Automated alerting mechanisms
Clustering
• Main aim: Provide, secure and reliable computational and storage
resources to our local (institutional) scientific communities.
• In terms of IT resources, the division offers:
– One cluster with: Computing >120 cores – Storage >10TB
• Obligations:
– Software installation & parameterization
• Operating system (installation, updates, fine-tunning)
• Libraries, Applications, Tools etc. (i.e. lapack, blas, fft, octave,
ROOT, NS2, FLUKA, MEEP, …. )
– Application Porting & Support
• Coding assistance – debugging – benchmarking
• MPICH/OpenMPI, OpenMP support for parallel applications
– System/Cluster parameterization
• User management
• Adjust the reservation policy based on local users needs
– Cluster Monitoring
• disk usage, cpu and memory utilization, networking, security ….
– And many other …
Software Development
• Indicative projects/developments
– Driver Application for DAQ (Visual C/C++, multithreaded, )
– Applications Database (php/Zend,Javascript/dojo,MySQL)
– Digital Repository (perl, MySQL, php)
– National Grid Hardware Registry (php, Javascript/AJAX, perl, MySQL)
– Software Discovery Engine (php, perl, MySQL)
– Institutional ERP system (Java, MySQL)
– Driver developments – PCI, RS232, TCP/IP, USB PCI-X ….
– Numerous, production-level applications for hardware/software
communication, integration and data acquisition (C/C++, Qt, posix
threads)
– and many other ….
Skills/Expertise (1)
• Deep knowledge on every phase through the software development lifecycle process, including:
– Software requirements gathering
– Preliminary design
– Detailed design
– Coding
– Unit testing
– Integration testing
– Qualification testing
• Deep knowledge in a software quality assurance techniques, based on both
ISO and CMM (Capability Maturity Model) standards.
Skills/Expertise (2)
• Advanced coding skills and experience. Some, indicative, coding
languages/frameworks we are focused in, are:
C/C++
Python
Javascript
Fortran
Php/JSP
HTML
Shell scripting (bash, tch etc)
Ada
XML (XSD/DTD)
Perl
JAVA
Many other …
• Parallel/multithreaded applications development
• Web application/service development
• Databases: MySQL, PostgreSQL, Oracle (development and/or
administration skills)
Affiliations (1)
• The IASA IT team, keep strong relation with many Greek, as well as
International, academic institutes
National:
– National and Kapodistrian University of Athens (NKUA)
– National Technical University (NTUA)
– Institute of Communications and Computer Systems (ICCS)
– Greek Research NETwork (GRNET)
– Foundation for Research & Technology (FORTH)
– University of Ioannina (UoI)
– Research Academic Computer Technology Institute (RACTI)
– Aristotle University of Thessaloniki (AUTH)
Affiliations (2)
International :
– CyI - Cyprus Institute
– CERN - European Organization for Nuclear Research
– INFN - Istituto Nazionale di Fisica Nucleare
– LIP - Laboratório de Instrumentação e Física Experimental de Partículas
– UPV - Universidad Politécnica de Valencia
Questions & answers
Thank you!! ….
24