Transcript Some Sample Commercial Network Costs

e-VLBI:
Connecting the World’s Radio Telescopes
with High-Speed Networks
Alan R. Whitney
MIT Haystack Observatory
Westford, Massachusetts, USA
Traditional VLBI
The Very-Long Baseline
Interferometry (VLBI)
Technique
(with traditional data recording
on magnetic tape or disk)
The Global VLBI Array
(up to ~20 stations can be used
simultaneously)
VLBI for Astronomy
• Science Missions
– Development of the very early universe
– Formation and evolution of galaxies
– Black holes in galactic cores
– Stellar nurseries
– Evolution and formation of elements
VLBI for Astronomy
• Combines data from global telescope array to
synthesize a very large, distributed antenna
• Highest resolution technique available to astronomers –
tens of microarcseconds (corresponds to resolving
dimples on a golf ball at distance 5000 km!)
• Allows detailed studies of the most distant objects in
the Universe
• Data is derived from national and international array of
large radio telescopes
• Disadvantage: Very large amount of data and huge
amount of computing required to make a single image
VLBI astronomy example
Galaxy NGC6251
• Distance: ~500Mlight-yrs
• 1 parsec = 3.262 light-yrs
= ~0.1arcseconds
• Allows observations to
focus in on energetic core
VLBI astronomy example
Quasars, hotspots, polarization
Chautauqua 2001
Resolution: ~10 marcsec
VLBI for Geodesy
• Fundamental VLBI measurement is time-of-arrival
difference of signals between telescopes in array;
typical single-measurement precision is <10 picosec
• Highest precision (few mm) technique available for
global tectonic measurements
• Highest precision Earth Orientation and Earth Rotation
measurements
• Earth-rotation measurements are important for military/civilian
navigation
• Stable reference frame formed by distant quasars
• Fundamental calibration for GPS constellation within Celestial
Reference Frame
• Highest spatial and time resolution of Earth’s motion in
space for the study of Earth’s interior
• International observing program includes ~35 stations
around the world, coordinated by International VLBI
Service
• In South American, stations in Concepcion, Chile and
VLBI astronomy
example
Fortaleza,
Brazil are regular contributors
Plate Tectonic Motion from VLBI measurements
VLBI astronomy example
Correlation between Earth Rotation and
Atmospheric Angular Momemtum
VLBI astronomy
example
Conclusion:
Primary driver of variations in Earth rotation rate is weather!
Disadvantages of Traditional ‘Record & Ship’ VLBI
• Long interval between data collection and results (typically weeks
to months)
• Uncertainty of proper equipment operation during experiment
• Expensive media (tape, disks) sometimes lost or damaged in transit
• Limited bandwidth limits sensitivity (few Gbps per station is
practical maximum)
A Little History: Mark 4 VLBI data system
16-station VLBI correlator at JIVE in The Netherlands
Enter ‘e-VLBI’ –
Electronic transmission of VLBI data over high-speed
global networks!
e-VLBI is not new!
• 1979 – OVRO-Haystack 1 Mb/station; 2400-baud modems (Mark III)
• 1980’s – JPL DSN for EOP; 500 kb/s sample rate buffered to disc,
56 kbps transfer over land lines
• 1995 – Japanese Keystone project; 256 Mbps over dedicated fiber
• 1999 – Europe; 1 Mb/station over Internet using ftp
• 2000 - Japan develops 1 Gbps e-VLBI system within Japan over
dedicated links
• 1977 – NRAO-Algonquin – 20 Mb/sec real-time satellite link
But the advent in global high-speed networks is completing
changing the game!
Scientific Advantages of e-VLBI
• Bandwidth growth potential for higher sensitivity
– VLBI sensitivity (SNR) proportional to square root of Bandwidth
resulting in a large increase in number of observable objects
(only alternative is bigger antennas – hugely expensive)
– e-VLBI bandwidth potential growth far exceeds recording capability
(practical recordable data rate limited to few Gbps)
• Rapid processing turnaround
– Astronomy
• Ability to study transient phenomena with feedback to steer observations
– Geodesy
• Higher-precision measurements for geophysical investigations
• Better Earth-orientation predictions, particularly UT1, important for
military and civilian navigation
Practical Advantages of e-VLBI
• Increased Reliability
– remove recording equipment out of field
– remote performance monitor & control capability in near real-time
• Lower Cost
– Automated Operation Possible
• eliminates manual handling and shipping of storage media
– Real-time or near-real-time Processing
• forestalls growth of storage-capacity requirements with bandwidth growth
– Elimination of recording-media pool (millions of $’s!)
• Avoid unexpected media-shipping interruptions and losses
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
– Mark 5 system developed at MIT Haystack Observatory with support from several
national and international institutions
• Demonstrate e-VLBI in feasibility experiment
– ~700 km link between Haystack Observatory and NASA/GSFC
• Develop specialized e-VLBI data-transport formats and protocols
– Develop international standard for e-VLBI data format
– New IP-based protocol tailored to operate in shared-network ‘background’ to
efficiently using available bandwidth
• Extend e-VLBI to national and global VLBI community
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Mark 5 VLBI Disk-Based Data System
• 1 Gbps continuous recording/playback to/from set of 8 inexpensive (ATA) disks
• Developed at MIT Haystack Observatory with multi-institutional support
• Mostly COTS components
• Two removable ‘8-pack’ disk modules in single 5U chassis
• With currently available 250GB disks – capacity of single ‘8-pack’ 2.0TB;
expected to increase to 3.0TB by early 2005 at cost of ~$0.5/GB
• GigE connection for real-time and quasi-real-time e-VLBI operations
• Inexpensive: <$20K
• ~75 Mark 5 systems now installed at stations and correlators
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Bossnet 1 Gbps e-VLBI
demonstration experiment
Westford
(Fall 2002)
Haystack
• 788 Mbps e-VLBI transfer achieved over
shared IP infrastructure, but took much
tuning
• Full report at www.haystack.edu/e-vlbi
(correlator)
Future
Initial experiment
USNO
(correlator)
NASA/GSFC
Westford-GGAO e-VLBI results
• First near-real-time e-VLBI experiment demonstrated on 6 Oct 02
– Recorded data at 1152 Mbps on Westford-GGAO baseline
– GGAO disk-to-disk transfer at average 788 Mbps transfer rate
• Direct data transfer experiment demonstrated on 24 Oct 02
– Direct transfer of GGAO data to disk at Haystack at 256 Mbps
– Immediate correlation with Westford data
– Nominal fringes observed
• Conclusion
– e-VLBI at near Gbps speeds over ordinary shared networks is possible
but still difficult
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
VSI-E
• VSI-E = VLBI Standard Interface for e-VLBI
• Follows on heels of VSI-H and VSI-S specifications over last 3 years
• Goal is to allow compatible interchangeable of data between
heterogeneous VLBI data sytems
• VSI-E currently under development by international VSI committee
• RTP protocol has been chosen
• Draft specification currently under discussion.
• Goal: Complete VSI-E specification by mid-2004
• Prototype software will be available June 04
RTP Capabilities
•
RTP provides an Internet-standard format for:
–
–
–
–
–
–
•
•
Transmission of sampled analog data
Dissemination of session information
Monitoring of network and end system performance (by participants and third parties)
Adaptation to varying network capability / performance
Message Sequencing / reordering
Multi-cast distribution of statistics, control and data
RTP allows the reuse of many standard monitoring / analysis tools
RTP seen as internet-friendly by the network community:
–
attention to efficiency
•
–
attention to resource constraints
•
–
protocol designed to have minimum overhead for in-band data
won't use up all your bandwidth with control information
attention to scaling issues
RTCP Capabilities
• Monitors network’s real-time data delivery performance
• Statistics collected from receivers
• Information delivered to
–Senders (adapt to prevailing conditions)
–Network management (identifies faults, provisioning problems)
• Adaptive, bandwidth-limited design
Possible VSI-E Topologies
New Application-Layer Protocols for e-VLBI
• Based on observed usage statistics of networks such as Abilene, it is
clear there is much unused capacity
• New protocols are being developed which are tailored to e-VLBI
characteristics; for example:
– Can tolerate some loss of data (perhaps 1% or so) in many cases
– Can tolerate delay in transmission of data in many cases
• ‘Experiment-Guided Adaptive Endpoint’ (EGAE) strategy being
developed at Haystack Observatory under 3-year NSF grant:
– Will ‘scavenge’ and use ‘secondary’ bandwidth
– ‘Less than best effort’ service will not interfere with high-priority users
– Dr. David Lapsley has joined Haystack staff to lead this effort
Typical bit-rate statistics on Abilene network
1.0
Usage >20Mbps less than 1% of the time
0.1
0.01
0.001
100
500
Mbps
Conclusion: Average network usage is only a few % of capacity
Typical distribution of heavy traffic on Abilene
1.0
0.9
0.8
0.7
<10% of ‘bulk’ transfers exceed ~100 secs
200
400
1000
secs
Conclusion: Heavy usage of network tends to occur in bursts of <2 minutes
New Transport Protocols for e-VLBI
• TCP is very inefficient is there are virtually any losses on a network
due either to network sharing or due to physical packet losses
– e-VLBI is particularly sensitive due to typical long RTT’s
• UDP can used, but is sometimes ‘unfriendly’ to other users
• New transmission protocols are being developed which are much more
aggressive, but fair – some examples:
–
–
–
–
–
FAST (Caltech)
Tsunami (Indiana University)
UDT (UDP-based Data Transfer; Univ. of Illinois)
High-speed TCP (Sally Floyd)
XCP (Explicit Control Protocol; Dina Katabi, MIT)
Acceptable Latencies for e-VBLI
•
Latency delays must not exceed the ability of the correlator to buffer the data
•
Near real-time operation
– At 1 Gbps, disk buffers of large size can be used for; in this case latency is not a
real issue.
– At 10 Gbps, disk buffers are not economically feasible and will require real-time
correlation
•
Real-time operation
– At 1 Gbps, a buffer size of ~0.5GB is currently available from each station;
a latency delay of up to ~2 seconds is acceptable
– At 10 Gbps, it is likely that larger buffers will need to be built to accommodate
latency delays up to at least 1 second
•
Optically switched networks with dedicated wavelengths may eventually put
the latency issue to rest as high-performance applications are able to use
dedicated-lambda facilities.
Elements of e-VLBI Development
• Develop e-VLBI-compatible data system
• Demonstrate e-VLBI in feasibility experiment
• Develop specialized e-VLBI data-transport formats and protocols
• Extend e-VLBI to national and global VLBI community
Westford-to-Kashima e-VLBI experiments
•
First Westford/Kashima experiment
conducted on 15 Oct 02
–
–
Data recorded on K5 at Kashima and
Mark 5 at Westford at 256 Mbps
Files exchanged over Abilene/GEMnet
networks
•
–
–
–
•
Nominal speed expected to be ~20
Mbps, but achieved <2 Mbps for
unknown reasons - investigating
File formats software translated
Correlation on Mark 4 correlator at
Haystack and PC Software correlator at
Kashima
Nominal fringes obtained
Kashima data is now regularly
transmitted to Haystack for processing
Westford to Sweden ‘Real-time’ experiments
•
‘Real-time’ experiments conducted
in March/April 2004
– Data transmitted and processed in
real-time (i.e. no disk buffering);
data transmitted directly from
stations to correlator
– First experiment at 32 Mbps due to
temporary low-speed connection to
Haystack
– Plan to extend these experiments
to at least 512 Mbps with GigE
links
UT1 ‘Intensive’ e-VLBI
•
•
•
•
Daily ~1 hour VLBI sessions between
Kokee Park, Hawaii and Wettzell, Germany
are used for UT1 measurements
Data are time sensitive since they are used
for predicting UT1
Currently requires ~4 day turnaround
shipping media
These measurements are an ideal candidate
for routine e-VLBI
– Short daily session collect <100 GB of data
– Even 100 Mbps will allow transfer in a few
hours
•
Work now in progress to make necessary
connections
– Network being organized from Kokee Park
to USNO;
connection speed OC-3
– Data from Mark 5 system in Wettzell will be
carried over dedicated fiber at ~30Mbps to
Univ. of Regensberg, then to GEANT
First e-VLBI to South America!
•
•
•
First e-VLBI data transmitted from TIGO in Concepion, Chile to Haystack
Observatory in April 2004
Two scans of ~1.5GB each were transmitted at an average rate of 1.0-1.5 Mbps
Hope to continue this effort and upgrade the link!
A sample of international connections
• Possibilities for international connections
–
–
–
–
–
Surfnet – U.S. to Europe at 10 Gbps
TransPAC/APAN – U.S. to Japan at 2.5 Gbps x2
GEMnet – U.S. to Japan at 2.5 Gbps (privately operated by NEC)
AMPATH – connections to telescopes in Chile and Brazil
AARNET – Recently announced 2x10 Gbps connections to Hawaii and
U.S.
– IEEAF –Europe/U.S./Japan link at 10 Gbps
– A sample of others under construction
•
•
•
•
•
GLORIAD – connecting China and Russis
TEIN – Paris to Seoul
EUMEDCONNECT – Europe to Mediterranean
NeDAP – Europe to Russia
ALIS – Europe to Latin and South America
– And many others………..
622 Mbps +10 Gbps l
Transoceanic donations to IEEAF (in red)
Credit: IEEAF
TransPAC Network
(planned upgrade in 2004 to 2xGigE plus OC-48)
Credit: J. Williams, IU
AMPATH: Research and Education Network and
International Exchange Point for the Americas




Launched in March 2000 as a
project led by Florida International
University (FIU), with industry
support from Global Crossing (GX),
Cisco Systems, Lucent
Technologies, Juniper Networks
and Terremark Worldwide
Enables wide-bandwidth digital
communications between the
Abilene network and 10 National
Research and Education Networks
(NRNs) in South and Central
America, the Caribbean and Mexico
Provides connectivity to US
research programs in the region
AMPATH is a project of FIU and the
National Science Foundation’s
Advanced Networking Infrastructure
& Research (ANIR) Division
Note: VLBI telescopes currently in Chile
and Brazil
Credit: Julio Ibarra, FIU
GÉANT:
The connectivity at 10 Gbps
10 Gbps
2.5 Gbps
EE
LV
34
155
UK
155
LU
NL
155
155
BE
FR
SE
IE
LT
45
SE - PoP for Nordunet
155
DE
ES
PT
CZ
0
SK
622
CH
IT
AT
HU
622
34
HR
BG
622
GR
34
45
CY
155
SI
34
622
PL
IL
34
RO
Aarnet: SXTransport Project in 2004
 Connect Major Australian Universities to 10 Gbps Backbone
 Two 10 Gbps Research Links to the US
 Aarnet/USLIC Collaboration on Net R&D Starting Now
Credit: George McLaughlin, AARNET
GLORIAD: Global Optical Ring (US-Ru-Cn)
“Little Gloriad” (OC3) Launched January 12; to OC192 in 2004
Abilene - Upgrade Completed!
Credit: Internet2
National Light
Rail Footprint
SEA
POR
SAC
NYC
CHI
OGD
DEN
SVL
CLE
FRE
PIT
BOS
WDC
KAN
NAS
STR
LAX
RAL
PHO
SDG
WAL
OLG
ATL
NLR
DAL
JAC
15808 Terminal, Regen or OADM site
Fiber route
Starting
Up Now
Initially 4x10 Gb
Wavelengths
Future: to
40x10Gb Waves
 Transition beginning now to optical, multi-wavelength R&E networks.

Also Note: XWIN (Germany); IEEAF/GEO plan for dark fiber in Europe
What is the future of e-VLBI?
•
•
•
Global connectivity is increasing at
a very rapid rate
e-VLBI is being aggressively
developed in the U.S., Europe and
Japan and will likely become
standard procedure within the next
decade.
In South America, the ALMA
project in Chile will be a large
driver for high-bandwidth
communications to the rest of the
world.
Bandwidth Growth of Int’l
HENP Networks (US-CERN
Example)
 Rate of Progress >> Moore’s Law. (US-CERN Example)









9.6 kbps Analog
(1985)
64-256 kbps Digital
(1989 - 1994)
[X 7 – 27]
1.5 Mbps Shared
(1990-3; IBM)
[X 160]
2 -4 Mbps
(1996-1998)
[X 200-400]
12-20 Mbps
(1999-2000)
[X 1.2k-2k]
155-310 Mbps
(2001-2)
[X 16k – 32k]
622 Mbps
(2002-3)
[X 65k]
2.5 Gbps l
(2003-4)
[X 250k]
10 Gbps l
(2005)
[X 1M]
A factor of ~1M over a period of 1985-2005
(a factor of ~5k during 1995-2005)
 HENP has become a leading applications driver,
and also a co-developer of global networks;
Credit: Harvey Newman, Caltech
What are the problems?
• Biggest problem: ‘Last-mile’ connection of telescopes
–
–
–
–
Most telescopes are deliberately built in remote locations
Biggest single obstacle is physical cost of laying fiber
Cost of terminal equipment is rapidly diminishing, even for 10 Gbps
A concerted international effort must be made to connect every major
telescope in the world to high-speed network
• Some countries require buying services from service providers, which
is very expensive; lighting dark fiber is far cheaper, if possible
e-VLBI is one among many applications
• HENP – high-energy physics community is currently heaviest user of
international networks, mainly in dissemination of very large data files
from major HENP facilities in Europe and U.S.
• Astronomy in general – optical telescopes are now sending highresolution images in real-time to remote observers around the world;
NVO will be among the world’s largest distributed database
• Education – tremendous opportunities here
–
–
–
–
Remote interactive learning from world experts in all fields
Demonstration of advanced surgical procedures
Easy international collaboration for all disciplines
Master classes in the performing arts, with remote students having access
to the world’s great performers and teachers
– Access to global databases and information of all kinds will help level the
playing field between rich and poor nations.
Summary of Impact of e-VLBI Program
• Opens new doors for national and international astronomical and
geophysical research.
• Represents an excellent match between modern Information
Technology and a real science need.
• Motivates the development of a new shared-network protocols that will
benefit other similar applications.
• Drives an innovative IT research application and fosters a strong
international science collaboration.
The End
Thank you!