Transcript eVLBI Progress

e-EVN developments in 2006
Arpad Szomoru
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Outline
•
•
•
•
•
•
The past
Current status
Expansion of e-EVN
EXPReS: first results
Connectivity improvements
The future
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-VLBI Milestones
20
num_badorder
25000
No LBE
num_lost
18
16
20000
14
12
15000
10
8
10000
6
4
5000
2
0
0
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
20
40
60
80
100
120
Transfer number
140
160
180
0
200
No. Lost
No. Out of order
• September 2002:
• 2 X 1 Gbit Ethernet links to
JIVE
• Demonstrations at
igrid2002 and ER2002
• UDP data rates over 600
Mbit/s
e-VLBI Milestones: 2003
Onsala
Sweden
Chalmers
University of
Technology,
Gothenburg
• May 2003: First use of
FTP for VLBI session
fringe checks.
• November 2003:
Cambridge –
Westerbork fringes
detected, only 15
minutes after
September: e-VLBI
July:10 Gbit•access
• October 2003:observations
first light on were
data transfer
GEANT-Surfnet
Westerbork – JIVE
made.1 Gb/s
between
Bologna
• 6 X 1 Gbit links
to JIVE
• November 2003: Onsala
and JIVE – connection
300Mb/s • 64Mb/s, with
disk Observatory
Space
buffering at JIVE
only. connected at
(Sweden)
1Gb/s.
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-VLBI Milestones: 2004
800
Mbps
600
On-Dw
Dw-On
Tr-Dw
Dw-Tr
Bo-Dw
Dw-Bo
400
200
0
m
ax
m
in
• June 2004: network stress
test (iperf) involving
December 20 2004:
• September• 2004:
Bologna, Torun, Onsala
of JBO
• January 2004: Disk
First e-EVN connection
science
and JIVE • September 2004:
to Manchester
at 2 x
buffered
e-VLBI
• March
2004: first realsession (Ar, Cm,
Tr,
• April 2004: ThreeFour telescope
real1 Gb/s
WestfordOn,
Wb)
• On, Wb,time
Cmfringes
at 128Mb/s
telescope,• real-time
(Ar,
Haystack June 2004: Torun time e-VLBI
• e-VLBI test with Tr,
for first GGAO
e-VLBIto
image
•
Spectral
line
fringes at 64Mb/s
connected at 1Gb/s. Cm, Tr, Wb)
and Jb
Intercontinental
observations On
at 32
Wb).
• On –• Wb
fringes(On,
at Jb, real• First fringes to
Ar at • Jb - Tr fringes at
Wf -On,
Mb/s
256Mb/stime •fringes,
First
real-time
EVN
5th EVN-NREN
32 Mb/s
32 meeting,
Mb/sZaandam, October 2006, A. Szomoru, JIVE
256Mb/s
image at 32Mb/s
e-VLBI Milestones: 2005
• March 2005: e-VLBI science
session
• Spring 2006: Metsahovi
• January 2005: Huygens • First continuum science connected at 10Gb/s
February
2005: network
observations at 128 and 64
descent tracking,
salvage
transfer test (BWCTL)
Mb/s, involving
6 radio2005: trench
• Summer
of Doppler experiment
employing various
network (Wb,
telescopes
Jb,mile”
Cm,
forAr,
“last
• Use of dedicated
monitoring tools On,
involving
Tr)
connection to Medicina
lightpath Australia-JIVE,
Jb, Cm, On, Tr, Bologna
dug
data transferred
at
~450
and JIVE
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Mb/s
1 Gbps
10 Gbps
155 Mbps
EVN Symposium 2004, A. Szomoru, JIVE
2.5 Gbps
Why bother? (change is bad…)
• Target of Opportunity - unscheduled observations triggered
by sudden astronomical events. This capability will become
much more important when LOFAR comes online
• Adaptive Observing - Use e-VLBI as a finder experiment
• Or, e-VLBI sessions a few days apart, adapt schedules for
later observations based on results (rapid results on large
sample, focus in detail on best candidates)
• Automatic Observing - small number of telescopes
observing for extended periods doing spectral line
observations of large galactic samples
• Interface with other real-time arrays – e-MERLIN,
LOFAR, SKA.. Also function as SKA-pathfinder
• Bandwidth no longer limited by magnetic media: 10Gbps
technology already becoming mainstream
• Because we can…
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Recent developments
• Regular science/test sessions throughout the year
• First open calls for e-VLBI science proposals
• First science run completely lost, but, first ever real-time
fringes to Mc (128 Mbps)
• Second and third science runs: many hours of smooth
sailing at 128 Mbps. No excitement, no drama. JIVE
becoming an observatory?
• Fourth run: 16 hours at 256 Mbps. However, nearly 25% of
time lost to technical problems..
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
First e-EVN Astronomy Publications
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Current status
• Technical tests:
•
•
•
•
6-station fringes at 256 Mbps
first European 512 Mbps fringes (Jb and Wb, May 18)
3-station 512 Mbps fringes (Cm, Wb, On, August 21)
first fringes using new 5 GHz receiver at Mc
• Current connectivity:
• Ar: 64 Mbps in the past, but <32 Mbps this year
• European telescopes: 128 Mbps always, 256 Mbps
often, 512 Mbps to Wb, Jb and On
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Tr connectivity bottleneck – (partially) solved
Black Diamond 6808 switches:
New interfaces (10GE) system in old architecture (1GE)
• Originally 8x1GE interface per card
• 10GE NIC served by 8 x 1GE queues
• Queuing regime – RR (packet based) and flow-based
Flow based:
Max. flow capacity – 1Gbit/s – backround traffic.
There is no known reordering workaround to solve this problem.
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-VLBI to South America? SMART-1
SMART-1 factsheet
Testing solar-electric propulsion and other deep-space technologies
Name SMART stands for Small Missions for Advanced Research in
Technology.
Description SMART-1 is the first of ESA’s Small Missions for Advanced
Research in Technology. It travelled to the Moon using solar-electric
propulsion and carrying a battery of miniaturised instruments.
As well as testing new technology, SMART-1 is making the first
comprehensive inventory of key chemical elements in the lunar surface. It
is also investigating the theory that the Moon was formed following the
violent collision of a smaller planet with Earth, four and a half thousand
million years ago.
Launched 27 September 2003
Status Arrived in lunar orbit, 15 November 2004. Conducting lunar orbit
science operations.
Notes SMART-1 is the first European spacecraft to travel to and orbit
around the Moon.
This is only the second time that ion propulsion has been used as a
mission's primary propulsion system (the first was NASA's Deep Space 1
probe launched in October 1998).
SMART-1 is looking for water (in the form of ice) on the Moon.
To save precious xenon fuel, SMART-1 uses 'celestial mechanics', that is,
techniques such as making use of 'lunar resonances' and fly-bys.
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-VLBI to Sourth America (2)
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
And other continents..
Australia:
• Telescopes connected
• PCEVN-Mk5 interface needed
China:
• Shanghai Observatory connected at 2.5Gbps
• Connection via TEIN (622Mbps), ORIENT?
Issues with CERNET, CSTNet
• Direct lightpath Hong Kong-Netherlight?
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Hybrid networks in the Netherlands..
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Switch from Cisco to
Nortel/Avici equipment has
been completed: for now, 7
* 1 Gbps, ultimately 16 * 1
Gbps lightpaths + 10 Gbps
IP connection
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
..and across Europe: GÉANT2 network upgrade
Outline
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
EXPReS: getting underway
SA1: new hires at JIVE; two software engineers, one network
engineer (finally!), one e-VLBI postdoc
Inclusion of e-MERLIN telescopes in e-EVN
Operational improvements (deliverable driven):
• Robustness
• Reliability
• Speed
• Ease of operation
• Station feedback
And still, pushing data rates, protocols, UDP, Circuit TCP?
Get rid of fairness… Better usage of available bandwidth.
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-VLBI control interface
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Interface to station Mk5s
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Runtime control
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Integrating fringe display
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Data status monitor
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Streamlining of post processing
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Web-based Post-processing
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Ongoing
New control computers (Solaris AMD servers)
• Cut down dramatically on (re-)start time
• Powerful code development platform
• Tightening up of existing code
Other hardware upgrades:
• SX optics (fibres + NICs), managed switch at JIVE
• Mark5A→B: motherboards, memory, power supplies, serial
links, CIBs
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
FABRIC: (Huib Jan van Langevelde)
• Distributed software correlation
• High bandwidth data transport (On part of e-MERLIN @ 4Gbps)
• Two new hires at JIVE
And coming..
SCARIe:
• Collaboration with SARA and UvA
• Distributed software correlation using Dutch grid
• Lambda switching, dynamical allocation of lightpaths,
collaboration with DRAGON project
• JIVE postdoc hired, still looking for UvA postdoc
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
FABRIC components
observing schedule
in VEX format
DBBC
VSI
PC-EVN
#2
field system
controls antenna
and acquisition
VSIe??
on??
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
GRID
resources data
user correlator
parameters
earth orientation
parameters
resource allocation
and routing
correlator control
including model
calculation
FABRIC
=
The GRID
output
data
Connectivity improvements
Martin Swany
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
•
•
•
•
•
2 Heavy duty gamer PCs
Tyan Thunder K8WE Motherboards
Dual AMD Opteron 2.4GHz processors
4GB RAM
2 1Gb PCI-Expres Nics
• First one at Torun, back-to-back to Mark5
• Second one located at Poznan Supercomputing Centre
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Protocol work in Manchester:
(Richard Hughes-Jones, Ralph Spencer & collaborators)
• Protocol Investigation for eVLBI Data Transfer
• Protocols considered for investigation include:
•
•
•
•
•
•
TCP/IP
UDP/IP
DCCP/IP
VSI-E RTP/UDP/IP
Remote Direct Memory Access
TCP Offload Engines
• Work in progress – Links to ESLEA UK e-science
• Vlbi-udp – UDP/IP stability & the effect of packet loss on correlations
• Tcpdelay – TCP/IP and CBR data
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Protocols (1) Mix of High Speed and Westwood TCP (Sansa)
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Protocols (2)
Circuit TCP (Mudambi,
Zheng and Veeraraghavan)
Meant for Dedicated End-toEnd Circuits, fixed
congestion window
No slow start, no backoff:
finally, a TCP rude enough
for e-VLBI?
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Protocols (3)
Home-grown version of CTCP using pluggable TCP congestion
avoidance algorithms in newer Linux kernels (Mark Kettenis)
Rock-steady 780 Mbps transfer using iperf from Mc to JIVE
Serious problem with new version of Mk5A software under
newer kernels
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
e-EVN: the future
• Aim: 16 * 1 Gbps production e-EVN network
• IP: not possible/affordable
• 10 Gbps lightpath across Europe: currently ~20k€/year
• Lightpaths across GÉANT terminating at JIVE
• If possible, all the way from telescopes. If not,
overprovisioned IP connections from telescopes to
GÉANT, lightpaths from there on.
• Guaranteed bandwidth, possibility to use ethernet frames,
no more need to worry about congestion..
• Towards a true connected-element interferometer
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Proposed connection Surfnet-JIVE
External LPs
N GE
Mk5
16 GE
OME
network
OME
network
Dynamic capabilities
through DRAC
10 G IP
5th EVN-NREN meeting, Zaandam, October 2006, A. Szomoru, JIVE
Switch
Mk5
Mk5