Transcript kekcont-furukawa4

Accelerator Controls at KEK
< kazuro.furukawa @ kek.jp >
Accelerator Controls at KEK
Mainly KEKB and Linac Evolution at Tsukuba Site
Kazuro Furukawa, KEK
KEKB and Linac Control Groups
<[email protected]>
Oct. 2009.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
1
Accelerator Controls at KEK
Several aspects of Evolution of the
Accelerator Controls at the KEK
Communication Networks
Equipment Controllers
EPICS Environments
Scripting Languages
Summary
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
2
Accelerator Controls at KEK
Mt. Tsukuba
J-PARC
(at Tokai Site)
KEKB
PF-AR
ATF
STF
cERL
PF
Linac
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
3
Accelerator / Control Systems
Accelerator Controls at KEK
Accelerators/Control Systems in KEK
Operational Presently
Linac, PF, PF-AR, ATF, KEKB, J-PARC
Under Construction
STF, cERL
EPICS
KEKB, …
Linac
J-PARC
PF
Controls Meeting, NSRRC, Taiwan
PF-AR
EPICS
Group
KEKB
ATF
Kazuro Furukawa, KEK, Oct.2009.
4
Accelerator / Control Systems
Accelerator Controls at KEK
Control Systems
12GeV Proton Synchrotron (PS): Hardwire + Mini-computers
2.5GeV Electron Linac (Linac): 8 Mini-computers + >200 Micro-computers,
Optical Networks
Photon Factory (PF): Mini-computers -- Workstations
TRISTAN ~33GeV: ~20 Mini-computers, CAMAC + NODAL Interpreter
Upgraded PS: VME/VersaDOS + MAP
Upgraded Linac: Unix servers, VME, PLC, CAMAC + TCP/IP + Home-grown
RPC, Tcl/Tk, Gateway to EPICS
ATF: VMS, CAMAC + V-System (Vista)
KEKB: VME, CAMAC, VXI, ARCnet + EPICS + Python, SADscript
PF-AR: the Same architecture as KEKB
PF: Many of the Components with EPICS
STF, cERL, SuperKEKB, …
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
5
Accelerator / Control Systems
Accelerator Controls at KEK
PF, PF-AR, ATF, STF
PF-AR
Mostly the same environment as KEKB
Many CAMAC installations
PF
Moved to EPICS environment
Mainly with Linux-VME
ATF
Vista Controls environment with CAMAC
Linux and socket environment
EPICS devices
STF
Test facility for ILC
Linux, ATCA test, PLC, …
cERL
Being built for ERL development
Will share the same environment with STF
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
6
Accelerator / Control Systems
Accelerator Controls at KEK
J-PARC at Tokai Canpus
HD
(hadron)
50GeV MR
MLF
(0.75MW)
(neutron
& muon)
3GeV RCS
NU
(25Hz,1MW)
(neutrino)
CCR
(Central
Control Room)
Linac
(181MeV
)
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
7
Accelerator / Control Systems
Accelerator Controls at KEK
J-PARC Controls
We started the design in 1998
But nobody was dedicated at the beginning
EPICS was chosen
The same reason as KEKB, EPICS was successful at KEKB
IP/Ethernet-only field network was chosen
It was successful at Linac
Device support
Development was started with Network-based device supports
WE7000, FAM3 PLC, EMB-Lan etc.
Later, Integrated into NetDev by Odagiri
Mixed application environment with Java, SAD, XAL, Python
Good practice for inter-institute developments with different
cultures
Under commissioning, user experiments in 2010
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
8
Accelerator / Control Systems
Accelerator Controls at KEK
EPICS Software Environment
Accelerator
Linac
RCS
MR
OPI Applications
Basic
High-Lvl.
IOC/VME
OS,H/W
Drivers
(slightly old)
Java
XAL VxWorks
PowerPC
/JCE
Adv7501
- VME I/O Modules
mainly by Advanet
- TeraDev for PLC
Java
VxWorks
- VME I/O Modules
mainly by Advanet
- TeraDev for PLC
Linux
(Network Devices)
-NetDev for PLC,
BPMC, EMB-LAN
-WE7000 Drivers
+MEDM
+MEDM
MEDM
(or EDM)
Controls Meeting, NSRRC, Taiwan
SAD
SAD
Python
PowerPC
Adv7501
Intel-based
GE Funac
and Sanritz
Kazuro Furukawa, KEK, Oct.2009.
9
KEKB and Linac Accelerator
Accelerator Controls at KEK
Mt. Tsukuba
B-factory: Electron/Positron
Asymmetric Collider for CP-violation Study
~3km Dual-rings: Electron(8GeV - 1.4A) /
Positron(3.5GeV - 1.8A)
J-PARC
KEKB
PF-AR
ATF
STF
PF
Linac
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
10
KEKB and Linac Accelerator
Accelerator Controls at KEK
KEKB and Linac
KEKB B-factory: Electron/Positron
Asymmetric Collider for CP-violation Study
~3km Dual-rings: Electron(8GeV - 1.4A) /
Positron(3.5GeV - 1.8A)
Stable and Robust Operation
Many Active Operation Parameters
Importance of Controls
Linac:
~600m, 50Hz
8GeV 2nC Electron, 3.5GeV 1.2nC Positron
Beam switchings for PF and PF-AR rings
Increase of Luminosity with Crab Cavities
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
11
KEKB and Linac Accelerator
Accelerator Controls at KEK
Increase of the Luminosity
percent by percent
Feb.2005
Continuous
Injections
May.2000
Apr.2003
Dual Bunch e+
Now, Collision
with Crab Cavities and
with Simultaneous Injection
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
12
KEKB / Linac Controls
Accelerator Controls at KEK
KEKB and Linac Control Systems
Linac
Controls Upgrade, (1990~)1993
De-facto (and International) Standards, IP-only Networks
No long Shutdown for KEKB upgrade
3.5-times Energy increase, 10-times current increase
Three indirect User Facilities (KEKB, PF, PF-AR)
Fewer resources
KEKB
5-year Shutdown after TRISTAN, 1994-1998
Precision requirements were much different for KEKB
Complete transition of Controls
from Nodal at TRISTAN to EPICS+SADscript at KEKB
from Energy frontier to Luminosity frontier
Basically Single-user (Belle)
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
13
Linac Network
Accelerator Controls at KEK
Communication Network at Linac
Fiber-optic Networks (1982~1996)
Because of High-power modulators for rf systems
~30 Loops to connect ~300 equipment controllers
However, the fiber-optic Technology was not mature enough yet
Often Failed and Loop Topology made it difficult to identify the trouble
All IP network (1993~)
Still all Fiber-optic
(Faster Ethernet enables shorter packets and less failures)
Inherited at J-PARC Controls as well
Gradual Transition of Technologies
From FDDI + 10Base-FL to 1000Base-LX + 100Base-Fx
Redundancy (1996~)
At more than 50 Ethernet links
Helped continuous operation in spite of a failure at night
Redundant Transceivers, then Rapid Spanning-tree and HSRP/VRRP
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
14
KEKB Network
Accelerator Controls at KEK
Communication Network at KEKB
TRISTAN
Token Ring and CAMAC Serial highways
Token ring between mini-computers
CAMAC serial highways to equipment controllers
KEKB
IP Network for EPICS
FDDI+10BaseT to GbE+100Base-Tx
Single broadcast domain
ARCNet for equipment controllers
More than 200 network segments
MXI-1/2 for VXI-based frames
20 segments, 200 frames
Keep some CAMAC Serial highways
10 highways, 50 crates
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
15
Linac Controller
Accelerator Controls at KEK
Equipment Controllers at Linac
1982~(1997) (1st generation)
300 microprocessor-based controllers
Linked together with home-grown fiber-optic network
1993~now (upgrade of controls)
160 PLCs (programmable logic controller)
Linked via only Fiber-optic Ethernet/IP
Control communication with servers and program development
1995~ (upgrade for KEKB)
Direct Fiber-optic Ethernet/IP to each Controllers
30 VXI for rf measurement
7 VME / 10 CAMAC for Timing (will retire soon)
20 VME for Beam monitors, etc (will retire soon)
2007~ (upgrade for 50Hz beam switching)
17 (increasing) VME for fast “event” handling, timing, llrf controls, etc.
24 Oscilloscopes with WindowsXP IOC for 100 BPMs
10Gs/s, 50Hz acquisition, local processing with 25 calibration parameters/BPM
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
16
KEKB Controller
Accelerator Controls at KEK
Equipment Controllers at KEKB
TRISTAN
Mostly CAMAC
Equipment group responsibility: CAMAC module and outside
KEKB
100 VME/IOC without Analog processing
200 VXI/MXI mainframes for 900 BPMs
50 CAMAC crates are kept for rf and vacuum
ARCNet boards for Magnet ps. settings, and others
GPIB for Magnet ps. readback, and other measurements
PLCs for Magnet interlocks, and others
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
17
EPICS at Linac
Accelerator Controls at KEK
EPICS Transition at Linac
Home-grown RPC at Linac (1990~/1993~)
At end of old mini-computer support
Not complete transition to EPICS yet at Linac
Mixed EPICS devices and gateways to middleware and applications
LynxOS Transition was developed (1994~1996)
To cover both RPC and EPICS with pthread, posix
Mostly working, Failed to get funding for Hardware/Software upgrade
Gateways to EPICS in several ways
Software-only IOC and Gateway (Clients to both RPC/CA)
Portable Channel Access Server of EPICS-3.12 (1995~)
Soft-IOC with device support to Linac RPC (2002~)
Redundant EPICS environment as well (2008~)
Real IOCs are increasing
PLC(rf,vacuum,magnet) and Linux, Oscilloscope(bpm) with Windows,
VME(event-based llrf and timing)
RPC servers read EPICS IOCs, EPICS gateways read RPC servers
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
18
EPICS at KEKB
Accelerator Controls at KEK
EPICS Transition at KEKB
Some candidates discussed after Nodal at TRISTAN
RPC/CORBA based control design
Reflective memory (hardware shared memory) design
No other choice than EPICS for KEKB
No man-power for control system software
The choice at SSC
International collaboration was attractive
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
19
KEKB Controls
Accelerator Controls at KEK
KEKB Control System (Hardware)
GbE Fiber Optic Networks
Single Broadcast Domain for now
Central Control Room and 26 Local Control Rooms
VME/IOC
~100 VME/IOC mostly with PowerPC CPU
Fieldbus
VXI thru MXI for BPM Instrumentations
CAMAC for rf and Vacuum (inherited from TRISTAN)
~180 ArcNet network segments for Magnet Power Supplies, and other field
Controllers
GPIB for Instrumentations, RS232C, Modbus+ for PLCs
Host Computers
(HP-UX/PA-Risc,) Linux/x86 Controls Server
(3 Tru64/Alpha with TruCluster)
Several Linux
Many MacOSX
(Solaris/Sparc for VxWorks)
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
20
KEKB Controls
Accelerator Controls at KEK
KEKB Control System (Software)
EPICS 3.13.1 mainly, with increasing 3.14.9
VxWorks 5.3.1 mainly, with increasing 5.5.1
Hope to upgrade EPICS/VxWorks Shortly
VxWorks 6.7 is also under testing
IOC Development
CapFast, VDCT, Perl, SADscript for Database Configuration
Oracle as a backend Database (Partially)
Operational Application Development
MEDM(DM2k) for Startup
Python/Tk for Equipment Controls
SADScript/Tk for Beam Operation, etc
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
21
KEKB Controls
Accelerator Controls at KEK
IOC and Records
There are Several Groups in KEKB
Each IOC is Assigned to a Group
MG, RF, VA, BM, etc
About 100 in Total
About 250k Records in Total
Most Groups Follow Naming Convention
Accelerator GroupName DeviceType
DeviceName
Property
Some EPICS Databases are Generated from
Oracle Database
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
22
KEKB Controls
Accelerator Controls at KEK
IOC Configuration
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
23
KEKB Controls
Accelerator Controls at KEK
IOC Database with Oracle
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
24
KEKB Controls
Accelerator Controls at KEK
Magnet Controls
~5000 Magnets, ~2500 Power Supplies
No One-to-one Correspondence
Basic Controls thru ARCnet
Voltmeter Scanner for Analog Read-out
PLC for Interlocks
IOC Manages All Controls
Interlock Status, On/Off, etc
Energy - Field - Current
Synchronous Operation
for Tune change, Orbit Correction, etc.
Standardization
Special Large Subroutine Record
“hugesub” Record and C Routine
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
25
KEKB Controls
Accelerator Controls at KEK
Magnet Controls
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
26
Linac Controls
Accelerator Controls at KEK
Linac Controls
KEKB = Factory Machine => Reliable Operation
Controls should be Robust and Flexible
~1000 devices and ~10000 signals
Frequent Beam Mode Switches
Four very Different Beam Modes, 360 times/day
Now 20ms switching with event-based controls
Precise Controls of Beam Parameters
Energy, Orbit, Emittance, Charge, Energy spread, Timing, etc.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
27
Linac Controls
Accelerator Controls at KEK
History and Design Concept
 History
 1978-1982: Construction of First Computer-controlled
System with 8 mini-computers, >200 micro-computers,
>30 optical loop networks
 1989-1992: Design of the next system
 1993-1997: Installation and expansion for KEKB
 Design Concept
 Use of International and/or de-facto Standards
 Use of Optical IP Networks for every Device controllers
 No new field Networks, only IP Network (to be inherited by J-PARC)
 Both of above should make future upgrade easier
 (EPICS was not available widely at that time)
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
28
Linac Controls
Accelerator Controls at KEK
Physical Structure
Multi-tier, Multi-hardware, Multi-client, …
Touch Panel Interface
X-Window Interface for Operation
EPICS Gateway to KEKB-Ring
MS-Windows Interface
Central Network (FDDI, Ethernet)
Device Manager
Main Computer Systems (Unix)
Equipment Level Network (Optical Ethernet)
VME's (~30)
PLC's (~150)
Beam Monitor
Timing
etc.
RF
Magnet
Vacuum
Controls Meeting, NSRRC, Taiwan
CAMAC's (~15)
Timing
VXI's (~30)
PC / GPIB / RS232C
RF Monitor
Gun
Beam Monitor
others
Kazuro Furukawa, KEK, Oct.2009.
29
Linac Controls
Accelerator Controls at KEK
Multi-tier Logical Structure
Advanced Beam Operations & Beam Study
Upper Level Servers
Engineering
Operations
Routine Operations
Middle Level Servers
Lower Level Servers
Network Based Hardware Controllers
Accelerator Equipment
Electron / Positron Beams
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
30
Linac Controls
Accelerator Controls at KEK
Software Architecture (old)
 Base control software structure for Multi-platform





any Unix, OS9, LynxOS (Realtime), VMS, DOS, Windows, MacOS
TCP - UDP General Communication Library
Shared-Memory, Semaphore Library
Simple Home-grown RPC (Remote Procedure Call) Library
Memory-resident Hash Database Library
 Control Server software




Lower-layer servers (UDP-RPC) for control hardware
Upper-layer server (TCP-RPC) for accelerator equipment
Read-only Information on Distributed Shared Memory
Works redundantly on multiple servers
 Client Applications
 Established applications in C language with RPC
 Many of the beam operation software in scripting language,
 Tcl/Tk
 SADscript/Tk
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
31
Linac Controls
Accelerator Controls at KEK
Why EPICS in my case
 We made too much effort on duplicate development on many
control systems
 Our goal is to achieve high performance in the accelerator
and the physics experiments
 Reuse of available resources is preferable
 Devices in Linac have been modernized, and development of
EPICS device supports became possible
 Anyway we need interface to down-stream accelerators esp.
KEKB
 Want to merge several archive formats in Linac
 May expect (?) man-power from other groups
 May contribute to world-wide EPICS collaboration
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
32
Linac Controls
Accelerator Controls at KEK
Building EPICS Gateway
 Common Control System at the Top (of Linac and Ring)
 Needs too much resources
 Port EPICS onto our VME/OS9-LynxOS (1994)
 Failed to get support/budget for LynxOS at Linac
 (EPICS Maintenance with an unsupported Platform ?)
 Special Gateway Software, which interfaces to both the Linac Controls and
EPICS IOCs as a Client
 Built to ensure the feasibility at 1995
 Portable Channel Access Server
 Implemented with EPICS 3.12 and being used on HP-UX since 1996
 It is being used for several application software including Alarm display
 Software IOC with 3.14
 Being used and being extended on Linux since 2003
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
33
Linac Controls
Accelerator Controls at KEK
Use of Existing EPICS IOC (Gateway IOC)
Software availability
Portable Channel Access Server was not available at around 1995
Channel Access Server Emulation with Available Software Components
New gateway software which is clients to the both Linac and EPICS, and group
of EPICS soft records
Real-time Operation is possible both ways using Monitors
Tested for Magnet Controls
MEDM panels were written
EPICS CA Clients
Linac Console
Gateway
C h a n n e l Ac c e s s P r o t o c o l
Linac
Controls
Controls Meeting, NSRRC, Taiwan
Gateway
IOC
Ring
IOC's
Kazuro Furukawa, KEK, Oct.2009.
34
Linac Controls
Accelerator Controls at KEK
Portable Channel Access Server (PCAS)
 Protocol Conversion
 Client to Linac Controls with Home-grown RPC and Cache Memory,
Interface to Upper-level Servers (not directly to Lower-level Hardware
Servers)
 Server to EPICS environment, with some Name wrapping
 Implemented for Linac in 1996 for Magnets, RF, Beam Instrumentations
 >4000 Records are available
 Write-access Possible,
normally Read-only
 Still used for
Linac Console
KEKB Unified Alarm,
Operation Status, etc.
EPICS CA Clients
C h a n n e l Ac c e s s P r o t o c o l
CA
server
CA Server
Linac
Controls
Controls Meeting, NSRRC, Taiwan
Ring
IOC's
Kazuro Furukawa, KEK, Oct.2009.
35
Linac Controls
Accelerator Controls at KEK
Soft IOC
 IOCcore is available on Unix in EPICS 3.14
 We have Tru64unix, Linux, HP-UX
 Simple
 IOCcore hides the complexity of Channel Access, etc
 We design the device support to Upper-level Linac Servers, as we access to
hardware in normal IOC
 All standard EPICS facilities are available
 Alarms, Operation Limits, Links, Periodic processing, Monitors, etc.
 Implemented for Linac on Linux since 2003
 For RF, Beam Instrumentation, Vacuum, etc.
 >5000 Records are available and extending
 Most records are archived in Channel Archiver and KBlog
 KBlog is used to analyze correlations between Linac/Ring
 (Developed Java viewer of the archive at One Time)
 Channel Archiver/Viewer from SNS is used for Linac Internals
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
36
Linac Controls
Accelerator Controls at KEK
General Comparisons
Symmetry
Gateway IOC is Symmetric between outside and inside of EPICS
Accessing from/to EPICS goes thru the same Gateway
Others are (somewhat) asymmetric
Name Resolution
PCAS can resolve names dynamically (at run-time)
Consumes less memory (?)
SoftIOC has to be prepared with static database
May be expected to give better response
Can be impossible for a large installations
Database processing and associate fields
SoftIOC provides EPICS database Facilities like Limits, Alarms, Links, etc.
If we archive them, Archive Deadband is most necessary
Implementation of Gateway
SoftIOC is relatively straight forward
Simply adding device supports
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
37
Linac Controls
Accelerator Controls at KEK
Application software
 Most records from the Linac Soft IOC are archived both in
Channel Archiver and in KBlog
 KBlog is used to analyze correlations between Linac/Ring
 (Developing Java viewer of the archive)
 KEKB Alarm is connected to Linac PCAS
 May migrate to Linac SoftIOC
(Linac PCAS is currently based on EPICS 3.12)
 Some other applications utilize PCAS as well
 (Many others access Linac Controls directly now)
 Small number of Records are going thru Gateway IOC,
historically
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
38
Linac Controls
Accelerator Controls at KEK
Performance
EPICS Gateway and Channel Archiver
are Running on Linux 2.4.20 (Redhat) with Intel
Xeon 2.4GHz and Memory of 2GB
About 10% of CPU usage
Monitors/Archives all of ~2200 Channels (partial in Kblog)
Can process 5400~6600 Channel Access Requests over
Network
Archive size is about 400MB/day (300MB/day in Kblog)
Both Channel Archiver and KBlog collect Data
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
39
Archiver
Accelerator Controls at KEK
Archiver/Logger
Linac
Several archivers with group-dependent filters
Replaced with two EPICS archivers (2000~)
Channel archiver, with Java viewer, and Web-based viewer
KEKBlog, SADscript-based viewer
Both ~500MB/day, Dynamic ADEL changes
KEKB
KEKBlog, since 1998
Once there was a plan to replace it with Channel Archiver
Data conversion, no much performance difference
Only ADEL-based filter
~4GB/day
SADscript-based viewer, one of the most-used applications in controls
With Data analysis capability, easy manipulations
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
40
Scripting Language
Accelerator Controls at KEK
Scripting Languages
Heavy use because of rapid prototyping
Linac
(1992~) Tcl/Tk as Test tools on Unix
(1997~) Tcl/Tk as Main Operator Programming Tool
Mostly replaced Windows/VisualBasic-based environment
(Now) Mixture of Tcl/Tk, SADscript/Tk, Python/Tk
SADscript has most accelerator design capability
Covers most features like MATLAB, Mathematica, XAL, MAD
KEKB
(Nodal interpreter and Fortran covered everything at TRISTAN)
Python covers many areas which is not covered by medm
SADscript is used by operators and physicists everyday
Realization of novel ideas in hours
Only some ideas are effective, so rapid prototyping is important
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
41
KEKB and Linac Operation
Accelerator Controls at KEK
SADScript
Accelerator Modeling Environment
MAD-like Environment was created during TRISTAN
Needs for Conditionals, Flow-controls, Data manipulations, Plot, GUI
Mathematica-like Language introduced as a front-end
Not Real Symbolic Manipulation (Thus fast)
Data Manipulations (Fit, FFT, …), List Processing (Mathematica-like)
EPICS CA (Synchronous and Asynchronous)
CaRead/CaWrite[ ], CaMonitor[ ], etc.
Tk Widget
Canvas Draw and “Plot”
KBFrame on top of Tk
Greek Letters, Compound widgets, etc
Relational Database
Inter-Process Communication (Exec, Pipe, etc)
System[ ], OpenRead/Write[ ], BidirectionalPipe[ ], etc.
Beam Operation with Full Accelerator Modeling Capability
Online and offline under the same environments
Also Used for non-Accelerator Applications (Archiver viewer, Alarm handler, etc.)
Comparable to XAL, MATLAB, but very different architecture
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
42
KEKB and Linac Operation
Accelerator Controls at KEK
SADScript
Mathematica-like Language
Not Real Symbolic Manipulation (Fast)
EPICS CA (Synchronous and Asynchronous)
CaRead/CaWrite[ ], CaMonitor[ ], etc.
(Oracle Database)
Tk Widget
Canvas Draw and Plot
KBFrame on top of Tk
Data Processing (Fit, FFT, …)
Inter-Process Communication (Exec, Pipe, etc)
System[ ], OpenRead/Write[ ], BidirectionalPipe[ ], etc.
Greek Letter
Full Accelerator Modeling Capability
Also Used for non-Accelerator Applications
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
43
KEKB and Linac Operation
Accelerator Controls at KEK
SADScript
Data manipulation/plot example
FFS;
w=KBMainFrame["w1",fm,Title->"t1"];
$DisplayFunction=CanvasDrawer;
W1=Frame[fm];
c1=Canvas[w1,Width->600,Height->400,
Side->"top"];
Canvas$Widget=c1;
data = {{0,0}, {1,1}, {2,5}, {3,8}, {4,10}, {5,7}, {6,4}, {7,2}, {8,0}, {9,2}}
fit = FitPlot[data,a Sin[x b + c] + d, x, {a,5},{b,1},{c,1},{d,5},
FrameLabel->{"X","Y"}];
phase = StringJoin["Phase : ", (c/.fit[[1]]) 180/Pi, " Deg."];
f1=KBFComponentFrame[w1,Add->{KBFText[Text->phase]}];
TkWait[];
Exit[];
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
44
SADscript
Accelerator Controls at KEK
Virtual Accelerator in SADscript
A Example in KEKB
most Beam Optics Condition is
maintained in the Optics Panel
Other Panels Manipulate Parameters
Communicating with the Optics Panel
(Oide, Koiso, Ohnishi et al)
===>
Tune Measurement/Changer
Controls Meeting, NSRRC, Taiwan
Optics Panels
Kazuro Furukawa, KEK, Oct.2009.
45
KEKB and Linac Operation
Accelerator Controls at KEK
KEKB Commissioning Groups
Formation of Commissioning Group (KCG)
Linac Commissioning (LCG)
7 from Linac
~10 from Ring
KEKB Ring Commissioning Group (KCG)
All LCG
~20 from Ring
Several from Detector (BCG)
Commissioning software
base was formed during
Linac Commissioning
(1997~)
Tcl/Tk, SAD/Tk , Python/Tk
KEKB
Commissioning
Linac
Group
Commissioning
Group
KEKB Ring
Controls Meeting, NSRRC, Taiwan
Linac
Kazuro Furukawa, KEK, Oct.2009.
46
KEKB and Linac Operation
Accelerator Controls at KEK
KEKB Alarm Panel
KEKB Alarm Main Panel covers Linac Alarms as well.
~10,000 Records are Monitored in One Panel. Detailed alarm
information/history is available in a separate panel
Linac
Ring
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
47
KEKB and Linac Operation
Accelerator Controls at KEK
Beam Optics Panels in SAD
Beam Optics Matching and Optimization Panels
in SADscript
Some Parameters goes
thru EPICS Gateways,
others directly to Linac
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
48
KEKB and Linac Operation
Accelerator Controls at KEK
KEKB Operation Panel Examples
Tune Measurement
and Tune Changer
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
49
KEKB and Linac Operation
Accelerator Controls at KEK
Virtual Accelerator in KEKB
Tune/Optics Server
Keep A Model of Real Accelerator
Can Change Tune, Chromaticity, etc, on Request
by Other Panels
Act as a Virtual
Accelerator
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
50
KEKB and Linac Operation
Accelerator Controls at KEK
Example Virtual Accelerator
Virtual Accelerator may Provide
the Both Fake Steerings and Fake BPMs
Maybe with Simulated Errors/Noises
Orbit Correction Application may Work
On Those Fake Information
Operation Panel
Orbit Correction
Controls Meeting, NSRRC, Taiwan
Real Acc.
Virtual Acc.
Optics
Steerings
BPMs
Noises
Optics
Steerings
BPMs
Errors
Kazuro Furukawa, KEK, Oct.2009.
51
KEKB and Linac Operation
Accelerator Controls at KEK
Virtual Accelerator with EPICS
Fake Accelerator Implementation
With EPICS Channel Access
Operation Panel
Simulator
In A Single SAD Application
Built-in Simulator in Operation Panel
Only SAD Applications
Separate Simulator (Virtual Accelerator)
Needs Some Switching Mechanism
Real Accelerator
Operation Panel
Switch
Real Acc.
Virtual Acc.
Separate Simulator (Virtual Accelerator)
In EPICS Semantics (EPICS Simulation Server)
Any Operation Panel (not only SAD)
SAD Simulation Server should Act as
EPICS Channel Access Server
Controls Meeting, NSRRC, Taiwan
Operation Panel
(Any EPICS Client)
Switch
Real Acc.
Virtual Acc.
Kazuro Furukawa, KEK, Oct.2009.
52
KEKB and Linac Operation
Accelerator Controls at KEK
Virtual Accelerator
Other Implementation Possibilities
Upper Level Protocol Like
CORBA
Used in Several Lab.
Cdev
Common Applications
May be Used in LHC (?)
cdev
cdev (or CORBA)
May Cover Systems
Not On EPICS
cdev
cdev
cdev
cdev
Linac
Ring
Other
Facilities
Other
Facilities
Not Covered in This Talk
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
53
KEKB and Linac Operation
Accelerator Controls at KEK
EPICS Simulation Mode
EPICS Database - Simulation Mode
A set of fields to support simulation are supplied on all hardware input records.
SIMM = YES makes this record a simulation record.
A link to a database value to put the record into simulation mode is specified
in the SIML. A non-zero number puts the record into simulation mode.
SVAL is the engineering units value used when the record is in simulation mode.
SIOL is a database location from which to fetch SVAL when the record is in
simulation mode.
SIMS is the alarm severity of the record if it is in simulation mode.
That is, EPICS Records may have Proxy Records
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
54
KEKB and Linac Operation
Accelerator Controls at KEK
EPICS Simulation Mode
SIMM - Simulation Mode
»This field has either the value YES or NO. By setting this field to YES, the record can
be switched into simulation mode of operation. While in simulation mode, input will be
obtained from SIOL instead of INP.
SIML - Simulation Mode Location
»This field can be a constant, a database link, or a channel access link. If SIML is a
database or channel access link, then SIMM is read from SIML. If SIML is a constant
link then SIMM is initialized with the constant value but can be changed via dbPuts.
SVAL - Simulation Value
»This is the record's input value, in engineering units, when the record is switched into
simulation mode, i.e. when SIMM is set to YES.
SIOL - Simulation Value Location
»This field can be a constant, a database link, or a channel access link. If SIOL is a
database or channel access link, then SVAL is read from SIOL. If SIOL is a constant link
then SVAL is initialized with the constant value but can be changed via dbPuts.
SIMS - Simulation Mode Alarm Severity
»When this record is in simulation mode, it will be put into alarm with this severity and a
status of SIMM.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
55
KEKB and Linac Operation
Accelerator Controls at KEK
Simulation Mode
EPICS Simulation Mode Simple Example
Tests Logic without Hardware
AI Record
.INP
.SIML
.SIOL
AO Record
A/D
D/A
Card
Programmable
Power Supply
.VAL
.SIML
.OUT
.VAL
(Simulation
Mode
Switch)
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
56
KEKB and Linac Operation
Accelerator Controls at KEK
SAD as EPICS Simulator
Implementing a Virtual Accelerator
SAD Simulator in Channel Access Server
Serves Channel Values Requested by Channels (Records)
in Simulation Mode (SIOL),
Acting as a Channel Access Server
Slightly more Difficult to Implement (at the First Stage)
SAD Simulator in Channel Access Client
Provides Channel Values Needed by Channels (Records)
in Simulation Mode (SVAL)
Easier to Implement (?)
Needs Some Studies
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
57
KEKB and Linac Operation
Accelerator Controls at KEK
Channel Access Server
Using Portable Channel Access Server
Needs Interface from Server-side Channel Access Library
to SAD
Written in C++
EPICS CA Clients
C hannelA ccess P rotocol
EPICS CA Records
Using IOC Core of EPICS-3.14 (or Later)
C hannelA ccess P rotocol
Needs Device Support for SAD
CA server
Maybe Easier
SAD
Engine
Using Intermediate Soft Records
SAD may act as an EPICS Client
Maybe Easier
SAD
Engine
EPICS CA Records
Gateway
C hannelA ccess P rotocol
Gateway
IOC
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
58
KEKB Controls
Accelerator Controls at KEK
KEKBLOG and ZLOG
KEKBlog/kblog Archiver is Used from the
Beginning of the Commissioning
~2GB / day
Several Viewer Tools
Very often Used to Analyze the Operation Status
Zlog Operation Log
Zope, Python, PostgreSQL
EPICS direct inputs / Human operator inputs
Mostly In Japanese
Figure/Picture Storing
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
59
Accelerator Controls at KEK
Near Future
SADscript
Will be maintained, but should look more at XAL - CSS
EPICS
Still many hopes waiting to be realized
More integration between control systems
PLC usage
Embedded IOC, IEC61131-3 Standards (?)
FPGA usage
More embedded controllers / instrumentations
VME will be kept, TCA will be installed
VME may decrease
More and more reliability considerations
Surveillance, Testing environments, Redundancy, etc.
More operation side developments
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
60
Accelerator Controls at KEK
Recent Improvements
PLCs with Embedded EPICS (Linux)
from Ethernet/IP-only to Channel-Access-only
Event system introduction
Single fiber to distribute globally synchronized, 10ps timing,
50Hz interrupts, data, etc
Another control layer besides EPICS
EPICS-embedded Oscilloscopes (Windows)
FPGA-based EPICS-embedded controllers (Linux)
Zlog operation log improvements
Used also at J-PARC, RIKEN, and BINP
Reliability improvement studies
Redundant IOC, ATCA and EPICS, Test systems
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
61
Accelerator Controls at KEK
Collaboration
Please help us in above new fields
Let’s help each other
Asian activities on EPICS should be kept
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
62
Accelerator Controls at KEK
Summary
Control systems at KEK are evolving based on changes and
advances in
in accelerator design concepts
in available technologies
Control system design needed balances between many aspects
Control Architecture Has Changed
Tried to establish unified controllers (before 15 years ago)
Tried to use only Ethernet/IP networks (15 to 5 years ago)
Trying to use (only) EPICS-embedded controllers (now)
EPICS and Scripting Languages brought great success to the
both KEKB and Linac Beam Operations
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
63
Accelerator Controls at KEK
Thank you
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
64
Accelerator Controls at KEK
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
65
Accelerator Controls at KEK
Backup slides
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
66
Accelerator Controls at KEK
KEKB & Linac
Further Electron-Positron Collider Experiments at KEKB
Contributed Nobel Prize to Kobayashi-Maskawa
Maintenance Difficulties
In Software and Hardware after 10-years of Operation
Transition from CAMAC to PLC, etc.
Transition to Newer versions of Software
After Stable Usage of VxWorks-5.3.1, EPICS-3.13.1.
Still Intensive Use of Scripting Languages
SAD-script, Python, Tcl.
Zlog operation log improvements
Used also at RIKEN, J-PARC, and BINP
Adding New Devices
PLC with Embedded Linux/EPICS
For Improved Machine Performance
New Hardware like Linux-embedded
PLC Controller (F3RP61)
for Beam-mask, Pulsed-quad, etc.
EPICS-3.14.9, Linux-2.6 , Procserv, Pcmon, Asyn
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
67
Accelerator Controls at KEK
Linac & PF & KEKB
Simultaneous Continuous Injection to PF, KEKB-HER and KEKB-LER
50Hz Beam Pulses are Shared between 3 Rings
With very different Beam Properties, in Energy, Charge, etc.
50Hz Beam Instrumentation (Beam Position Monitor)
Only Passive Components other than Oscilloscope (Tek-DPO7104)
Windows-embedded (3GHz Intel), EPICS-3.14.9, VC++
One Oscilloscope reads 2-5 BPMs, 24 Oscilloscopes Installed
Synchronized 100-BPM Read-out
Introduction of Event System, EVG230-EVR230RF from MRF
10 EVR’s Installed, 1/3 of Old Timing Stations Replaced
EVG & Timing
VxWorks-5.5.1, EPICS-3.14.9, (Gave-up with RTEMS)
Event drives Low-level RF in VME, BPM Oscilloscopes over Network
Gun Parameters, Pulsed Magnets, Kickers, etc are Controlled 50Hz
Beam Pattern Rules on Client Script, can be Downloaded every second
More Development Needed
Flavoured Beam Feedback Systems
Event System Integrity Monitor
EVR & LLRF
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
68
Accelerator Controls at KEK
(previous) PLC usage at KEK
At Linac
We enforced that all the controllers should be connected over
IP/Ethernet
PLC was much cost-effective compared with VME
if the speed requirement allows
Products from OMRON, Mitsubishi, Yokogawa were installed
Only Yokogawa (FAM3) increased, because maintenance capability over
network was better
Ladder software downloadable over IP/Ethernet
(Recently Mitsubishi also added that feature)
150 PLCs used at Linac for RF, Magnets, Vacuum, Safety, etc
At J-PARC
Many installations with the same reason as Linac
At KEKB
Used indirectly at many devices, over serial or GPIB links
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
69
Accelerator Controls at KEK
Software management for PLC
Ideal at the beginning
Separate software development at control group, at
equipment group, or at industrial company
Later, integration test IP/Ethernet
Logic management
Same logics could be placed at ladder software, in EPICS
database/sequencer (or in high-level applications)
Speed requirement
Closed loop over Ethernet was slow, sometimes un-reliable
Socket-based interrupts were possible, but complicated
Thus, hoped to run EPICS on PLC
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
70
Accelerator Controls at KEK
Vacuum Controller Internal
Magnet Controller Internal
Safety Controller
Touch Panel Display for RF
Controls Meeting, NSRRC, Taiwan
RF Controller Internal
Kazuro Furukawa, KEK, Oct.2009.
71
Accelerator Controls at KEK
EPICS on PLC
VxWorks was available on PLC (Yokogawa, Mitsubishi)
We use VME for realtime performance with VxWorks
License management of vxWorks …
Yokogawa starts to provide Linux (2.6) on PLC CPU
Brave enough to choose open source environment
We negotiate with Yokogawa to remove any license issues
Odagiri/KEK, Uchiyama/SHI, Yamada/KEK made much effort to realize
the EPICS implementation
Takuya-Nakamura/MSC tailored the environment for KEKB
Procserv, pcmon, NFS, …
Three of them are used in KEKB operation
Beam mask controller and Pulsed-quad controller
It already ran for three months without any troubles/stops
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
72
Accelerator Controls at KEK
F3RP61
Linux 2.6.24
PPC 533MHz
128Mbyte RAM
100BaseTx x 2
USB
IEEE1394
Serial
Beam mask controller
PCI
I/O Bus for FAM3 Module Interface
Software development environment
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
73
Accelerator Controls at KEK
Event System
Quasi-simultaneous Injection
to KEKB-HER, KEKB-LER, and PF
2.5GeV to 8GeV, 0.1nC to 10nC
Stable stored beam current at three rings
Should improve collision tuning with Crab cavities at KEKB
Should improve the quality of experimental data at PF
Fast switching of many device parameters
In 20ms / 50Hz
Should be reliable because beam power is much different
MRF Series-230 Event Generator / Receiver
VxWorks 5.5.1, MVME5500, EPICS 3.14.9 (Originally with RTEMS but…)
Timing precision less than 10ps (TD4 provides 3ps)
Multi-mode fiber, and single-mode fiber for longer distance
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
74
Accelerator Controls at KEK
Basic synchronization outside of EVG
Control
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
75
Accelerator Controls at KEK
Event system configuration, autumn 2008
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
76
Accelerator Controls at KEK
Beam mode pattern generation
Every pulse (every 20ms) corresponds to a beam mode.
10 different beam modes are defined (for KEKB e+, etc).
One beam mode may contain many event codes.
About 50 event codes are defined.
Some events correspond to many functions, and others to specific
devices.
Beam pattern buffer length (n) can be 2 to 500 (20ms x 500 = 10 seconds).
A new pattern is loaded at the end of the previous pattern.
Otherwise, the pattern repeats forever.
Pattern generator software arbitrates requests from downstream rings.
There are many pattern rules due to pulse device features and limitations.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
77
Accelerator Controls at KEK
Beam mode pattern generation
Manual pattern designer
Controls Meeting, NSRRC, Taiwan
A version for current operation
Kazuro Furukawa, KEK, Oct.2009.
78
Accelerator Controls at KEK
LLRF
Timing and analog signals are essential for absolute energy,
energy spread, and dual-bunch energy equalization.
Signals can be switched pulse-by-pulse.
Driver klystrons (SB), energy tuner klystron (KL), and
sub-harmonic bunchers (SH) are managed
by the event system.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
79
Accelerator Controls at KEK
BPM
DPO7104 can acquire data in 50Hz .
Beam modes are recognized by events through network.
Clients can monitor data of an interested beam mode.
100 BPMs are synchronized.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
80
Accelerator Controls at KEK
Parameters
Parameters switching via Event system
RF Timing x~35
LLRF x~11
Gun voltages, fast delays, x4
Pulsed magnets x~12
Injection system x~4
BPM over channel access x~100
Basically sufficient for fast beam mode switching
More parameters next year
Integrity monitors
Improved slow beam feedback, fast feedback, etc.
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
81
Accelerator Controls at KEK
Embedded EPICS with FPGA
Suzaku/atmark-techno
FPGA Vertex-4
PPC Linux-2.6
EPICS 3.14
J-PARC MPS
KEKB Magnet
Linac RF
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
82
Accelerator Controls at KEK
J-PARC MR MPS Operational
Akiyama,
Nakagawa,
et al.
Several
Different
Interfaces
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
83
Accelerator Controls at KEK
Thank you
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
84
Accelerator Controls at KEK
Thank you
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
85
Linac Controls
Accelerator Controls at KEK
Linac; Physical Structure
Multi-tier, Multi-hardware, Multi-client, …
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
86
Linac Controls
Accelerator Controls at KEK
Linac; Software Architecture
 Base control software structure for Multi-platform





any Unix, OS9, LynxOS (Realtime), VMS, DOS, Windows, MacOS
TCP - UDP General Communication Library
Shared-Memory, Semaphore Library
Simple Home-grown RPC (Remote Procedure Call) Library
Memory-resident Hash Database Library
 Control Server software




Lower-layer servers (UDP-RPC) for control hardware
Upper-layer server (TCP-RPC) for accelerator equipment
Read-only Information on Distributed Shared Memory
Works redundantly on multiple servers
 Client Applications
 Established applications in C language with RPC
 Many of the beam operation software in scripting language,
 Tcl/Tk
 SADscript/Tk
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
87
Available Technologies
Accelerator Controls at KEK
Network with only IP/Ethernet
The policy chosen when we upgrade Linac in 1993
Make network management simpler
Faster switches, routing, network-booting, etc.
Avoid Hardware failure and analysis effort with old field network
Home-grown field networks need much dedicated man-power
Cost for optical Ethernet went down at around 1995
Linac has high-power modulator stations, noise source
Nowadays many facilities have this policy with GbE
J-PARC controls basically followed this
More and more intelligent network devices
ex. Oscilloscopes with Windows/3GHz-Pentium built-in
Even EPICS IOC, MATLAB, or others can be embedded
Network components can be replaced one-by-one
Security consideration will be more and more important
Controls Meeting, NSRRC, Taiwan
Kazuro Furukawa, KEK, Oct.2009.
88