Transcript January Webinar: How to Integrate FileCatalyst Java Applets

May Webinar: 10Gbps transfers
Presenter: Christian Charette, John Tkaczewski
Duration: 30 minutes
Presenter Introduction
John Tkaczewski
• Co-Founder and President of Unlimi-Tech
Software
• Oversees creative and technical vision for
FileCatalyst Webmail and FileCatalyst
Workflow
• Blog: http://www.filecatalyst.com/blog
Christian Charette
• Director of Software Development
• Project lead on 10 Gbps research
2 | © Copyright FileCatalyst, 2013
Agenda
• FileCatalyst Direct Overview
• Quick refresher why TCP doesn’t work in high speed networks
• 10Gbps test infrastructure
– Ie: what are we using to perform the demo?
• Watch the data fly!
• Questions (please type them in as we go)
3 | © Copyright FileCatalyst, 2013
FileCatalyst Direct Overview
•
•
•
•
Software architecture is Client-Server.
One side acts as the main file repository (FileCatalyst Server)
One or many clients connect to server, and push/pull data
Clients include:
– automated interfaces (API, CLI, Javascript, C++)
– web components (applets)
– feature rich applications (HotFolder, Express, mobile)
• Primarily written in Java, works on every major platform
(Windows, Linux, Mac OSX, iOS, Android, and more)
• Transport protocol utilizes proprietary UDP, with error
correction and congestion control built into the application
layer (instead of relying on TCP network layer)
4 | © Copyright FileCatalyst, 2013
Putting 10 Gbps in context
• 10 Gbps transfers an average BlueRay disk (26GB) in under 30
seconds
• Cost of 10 Gbps dedicated lines is dropping – becoming cost
effective to many companies (~10K/month US to UK)
• 1 Gbps fiber connections becoming available to residential
consumers
5 | © Copyright FileCatalyst, 2013
TCP vs UDP for packet loss & latency
• Quick refresher: TCP can’t cut it.
•
Data provided as part of Photonics North 2012 presentation, FTP data uses FileZilla client
connecting to PureFTPD
Transfer speed on 100mbps link
Internet average 0.6% packet loss
100
90
Average Transfer Speed (mbps)
80
FileCatalyst 0.0% loss
70
FileCatalyst 0.1% loss
60
FileCatalyst 0.5% loss
FileCatalyst 1.0% loss
50
FileCatalyst 5.0% loss
40
FTP 0.0% loss
30
FTP 0.1% loss
FTP 0.5% loss
20
FTP 1.0% loss
10
FTP 5.0% loss
0
0
2
5
10
25
50
75
100
Network latency on link (ms)
6 | © Copyright FileCatalyst, 2013
150
200
250
300
TCP vs UDP for packet loss & latency
• TCP cannot handle packet loss -- even for networks with small latency.
• High latency networks fall apart under real world conditions.
Transfer speed on 1gbps link
Internet average 0.6% packet loss
1000
900
Average Transfer Speed (mbps)
800
FileCatalyst 0.0% loss
700
FileCatalyst 0.1% loss
600
FileCatalyst 0.5% loss
FileCatalyst 1.0% loss
500
FileCatalyst 5.0% loss
400
FTP 0.0% loss
300
FTP 0.1% loss
FTP 0.5% loss
200
FTP 1.0% loss
100
FTP 5.0% loss
0
0
2
5
10
25
50
75
100
Network latency on link (ms)
7 | © Copyright FileCatalyst, 2013
150
200
250
300
TCP vs UDP for packet loss & latency
• You cannot transfer files at 10 Gbps with TCP. It just doesn’t
work.
Transfer speed on 10gbps link
10000
9000
Average Transfer Speed (mbps)
8000
FileCatalyst 0.0% loss
7000
FileCatalyst 0.1% loss
6000
FileCatalyst 0.5% loss
FileCatalyst 1.0% loss
5000
FileCatalyst 5.0% loss
4000
FTP 0.0% loss
3000
FTP 0.1% loss
FTP 0.5% loss
2000
FTP 1.0% loss
1000
FTP 5.0% loss
0
0
2
5
10
25
50
75
100
Network latency on link (ms)
8 | © Copyright FileCatalyst, 2013
150
200
250
300
10 Gbps transfers
• 10 Gbps research project started in May 2011.
• Project completed 6 months later with Nov 2011 release of
FileCatalyst Direct v3.0, able to transfer at full 10gbps speed.
• 1½ years of successful deployments of 2-10 Gbps in the field.
• Tested in 2012 by independent researchers (ex-CEO of
competitor Tixel GmbH), validated as most stable transfer
results between 5 managed transport solutions.
Comparison of Contemporary Solutions for High Speed Data Transport on WAN 10 Gbit/s Connections, Dmitry Kachan, Eduard Siemens,
Vyacheslav Shuvalov
9 | © Copyright FileCatalyst, 2013
What are we using to run this demo?
FileCatalyst HotFolder
Ares x.x.x.58
10 | © Copyright FileCatalyst, 2013
FileCatalyst Server
Hermes x.x.x.59
What are we using to run these tests?
• Machines utilize standard components circa 2011.
• Each machine costs approx. $5000 to put together.
– Intel Westmere 5600 series (used in demo testbed)
– AMD Magny-Cours 6100 Opterons
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
11 | © Copyright FileCatalyst, 2013
What are we using to run these tests?
•
•
•
•
Machine has 48GB total, 36GB reserved for RAMFS (so 12GB for OS + FileCatalyst).
For the purpose of these tests, going to read/write to RAM drive.
Why? Want to remove disks as part of the equation for demo to simplify things
(everybody has different IO solutions in place).
If your disks can handle the speeds, our software can transfer it.
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
12 | © Copyright FileCatalyst, 2013
What are we using to run these tests?
• In demo, to simulate WAN environment, a Linktropy box from
Apposite Technologies is used
– Control link speed, packet loss, and link latency
WAN Emulator
Apposite Technologies
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
13 | © Copyright FileCatalyst, 2013
So what are we going to see?
WAN Emulator
Apposite Technologies
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
14 | © Copyright FileCatalyst, 2013
So what are we going to see?
• (1) Going to test most basic scenario:
– Transfer static file from point A to B
– 30GB file… slightly larger than the average Blue Ray disk.
WAN Emulator
Apposite Technologies
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
15 | © Copyright FileCatalyst, 2013
So what are we going to see?
• (2) Going to test standard live media scenario:
– Growth files (5) simulate live video feeds saved to system
– Move files in real-time to destination as files are output
WAN Emulator
Apposite Technologies
FileCatalyst HotFolder
Ares x.x.x.58
FileCatalyst Server
Hermes x.x.x.59
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Dual Intel Xeon 5690
48 GB of RAM
(36 GB as RAMDRIVE )
Intel 10Gbps cards
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
Ubuntu 10.04.1 LTS
Java 1.7 64-bit
FileCatalyst v3.2
16 | © Copyright FileCatalyst, 2013
What are we using to run these tests?
• Linktropy setup to simulate LA -> NY transfers
– Latency of 60 ms
– 0.1% packet loss
• Configurations used for single file transfer @ 10gbps:
–
–
–
–
–
–
–
–
Server & HotFolder each using 4GB of RAM
MTU 9000 (packet 8972 bytes)
Blocks: 10MB
Number of sender threads: 50
Sender sockets: 5
Receiver sockets: 3
RTT congestion control (because test scenario utilizes packet loss)
Congestion control set to 7 (more aggressive)
17 | © Copyright FileCatalyst, 2013
DEMOS
DEMOS
18 | © Copyright FileCatalyst, 2013
Upcoming Events:
- NYC Digital Asset Managers, (May 29th)
- Broadcast Asia, Singapore (June 18-21)
- IBC, Amsterdam (Sept 13-17)
- Interop New York (Oct 2-3)
- Broadcast India, Mumbai (Oct 9-11)
And of course don’t forget our monthly webinar....
Thank you for attending our May webinar!
Questions? Please type in your questions.