Transcript unLHCLoggingDB

UNICOS LHCLoggingDB
Josef Hofer EN/ICE/SCD
Agenda
• The LHC Logging Database
• Purpose of the LHCLogging component
• Basic concepts
• Advanced concepts
• Logging configuration
• Logging detail information
• System integrity
• Current status / Next steps
The LHC Logging Database
• What is the LHC Logging Database? (LDB)
• Long term data storage and biggest DB at CERN
• How much data does it contain?
• ~ 1 million signals from different sources
• ~ 1 TB of new data per week
• > 20 years of filtered data
• How is the database used?
• > 700 clients (onsite and offsite)
• > 100 custom applications analysing data
• Generic visualisation tools: e.g. “TIMBER”
• Data insertion and extraction via
• PL/SQL
• Java APIs
The LHC Logging System
Purpose of the LHCLogging component
• Ensure that data is being transferred
from PVSS to LDB
• Provide an easy way of configuration
to specify which data shall be transferred to LDB
• Display detail information for monitoring
to see which data is currently being transferred
• Trigger alarms
if data transfer to LDB does not work
unLHCLogging
manager
Local Archive
ValArch
PVSS System
C++ Application
HTTP
XML
Java API
Basic concepts LHCLogging “old”
LHC Logging DB
Data flow
Basic concepts LHCLogging “old”
UI-Panels:
unLHCLogging
manager
Local Archive
ValArch
PVSS System
HTTP
XML
Java API
Configuration
& diagnostics
C++ Application
Configuration & Diagnostics
LHC Logging DB
Data flow
Why do we have to reinvent the wheel ?
• Maintenance of “old” logging requires
a lot of effort
• Performance was not sufficient for QPS
• Gain robustness by eliminating number of
failure points
Basic concepts LHCLoggingDB
Employ existing and maintained
mechanism used by measurement DB
No Java application server!
No custom PVSS manager!
LDB data
transfer mgr.
DB Link
PVSS System
Oracle RDB Archive
or MDB
LHC Logging DB
Data flow
Basic concepts LHCLoggingDB
Employ existing and maintained
mechanism used by measurement DB
No Java application server!
Configuration
& diagnostics
No custom PVSS manager!
PL/SQL
API
UI-Panels:
LDB data
transfer mgr.
DB Link
PVSS System
Oracle RDB Archive
or MDB
LHC Logging DB
Configuration & Diagnostics
Data flow
Advanced concepts LHCLoggingDB
UI-Panels:
Configuration
& diagnostics
LDB data
transfer mgr.
DB schema 1
DB schema 2
DB schema 3
PVSS Systems
Oracle RDB Archive
or MDB
Configuration & Diagnostics
Data flow
Advanced concepts LHCLoggingDB
Data transfer is performed by “transfer jobs”
RDB jobs
LDB data
transfer mgr.
• Up to 16 transfer jobs per database
• Each job manages multiple elements
• Jobs run every 5 minutes:
• Query value changes on RDB Archive
• Push archived values to Logging DB
• Caution: Max. chunk size per element!
DB Link
Oracle RDB Archive
or MDB
LHC Logging DB
Configuration & Diagnostics
Data flow
Advanced concepts LHCLoggingDB
MDB:
LVS:
TS
Value
2014/05/19 02:34:15
0.074
2014/05/19 02:33:59
0.068
2014/05/19 02:33:17
0.072
2014/05/19 02:32:12
0.069
2014/05/19 02:28:16
0.065
TS job run
TS last logged
Recs
2014/05/19 02:35:00
2014/05/19 02:30:00 2014/05/19 02:28:16
4
2014/05/19 02:25:00 2014/05/19 02:24:22
4
2014/05/19 02:20:00 2014/05/19 02:17:12
3
2014/05/19 02:15:00 2014/05/19 02:14:01
5
*LVS: Log Variable Status
Advanced concepts LHCLoggingDB
function: TransferJob():
do every 5 minutes
for all elements e assigned to job j
TS_Latest = getLatestTsFromLvs(e);
data = queryDataFromMDB(e, TS_Latest, TS_Now);
TS_Latest = pushDataToLDB(data);
setLatestTsToLvs(e, TS_Latest);
end function;
function: TS pushDataToLDB(data):
if data.len() > 3000 then
transfer(data,0,2999);
TS = data[2999].getTS();
else
transfer(data,0,data.len()-1);
TS = data[data.len()-1].getTS();
end if
return(TS);
end function;
Advanced concepts LHCLoggingDB
• Currently available RDB Archives
NAME
DESCRIPTION
SCADAR
Production systems EN/ICE
QPSR
Production systems QPS only
TEST
Test systems for development
Logging configuration
• Possible types of configuration
• Register new elements for logging
• Update already registered elements
• Unregister elements from logging
• Change settings of transfer manager
• Create/Import/Export configuration files
Register new elements for logging
• Use standard logging configuration file
in .text or .xml format
[_ValueArchive_0]
dpe=dpname10.actVal
alias=
name=HeatingCircuit10:FlowTemperature
description=Flow temp. heating circuit 10
hierarchy=<node name="HC"><node name="FLOW"></node></node>
format=
dpe=dpname11.actVal
alias=
name=HeatingCircuit11:FlowTemperature
description=Flow temp. heating circuit 11
hierarchy=<node name="HC"><node name="FLOW"></node></node>
format=
Register new elements for logging
• Select config file and click “Check” to verify
1
Register new elements for logging
• Enter additional information
Register new elements for logging
• Click on “Append” to register data
Logging detail information
• List of all registered elements
Logging detail information
• Transfer history of selected element
Logging detail information
• Transfer job details of selected job
Logging detail information
• Element history of selected element
System Integrity RDB
• Change RDB system integrity plugin to
• Avoid rash killing of the archive manager
• Employ a simple way of checking
• Provide additional functions which will be
used by system integrity of LHCLoggingDB
System Integrity RDB
•
•
•
•
•
Perform system integrity checks in a polling loop
Kill manager only if it is blocking longer than a specific time
Monitor state of database connection and buffer usage
Do a round trip check for several test data points
Set alarm data point according to the output of each check
System Integrity LHCLoggingDB
• Create new plugin to be able to:
• Ensure that all transfer jobs are running
• Check if all registered elements are being
processed by the responsible transfer job
• Check if transfer jobs are able to handle the
amount of data that is produced by each signal
(max. chunk size should never be reached)
• Do a round trip check for specific test data points
(only one per data category to not kill the DB)
System Integrity LHCLoggingDB
•
•
•
•
•
Perform system integrity checks in a polling loop
Take RDB system integrity status into account
Check transfer history by searching for recent timestamps
Do a round trip check for several test data points
Set alarm data point according to the output of each check
Current status / Next steps
• Completed tasks
• Display detail information for monitoring
• Verification and Import of configuration files
(.text and .xml)
• Current tasks
• Develop system integrity plugin to automatically
check logging and trigger alarms on error
• Next tasks
• Implement possibility to update or delete already
registered elements (cooperation with BE/CO)
• Implement creation & export of config-files
• Improve usability