Transcript A Java File Transfer Framework (and More) Using Quartz

A Java File Transfer Framework (and
More) Using Quartz
Jared Lynem
Amway Corporation
[email protected]
Hello
 I’m Jared
 We’ll talk about:




File transfers: requirements, reasons, concepts
A framework for file transfers
A way to schedule them
Other quick victories
 Links to resources at the end of this document
 Feel free to contact me

[email protected]
Requirements
 The first business data “integrations”
 Moving paper files from one office to another
 Ditto machines! Crazy.
 Offices like on Mad Men
 Fast forward to early digital integrations
 Moving data files from one server to another
 Bread and butter, reliable, easy
Requirements – What?
 What – select and move files based on
location, file name
 When – based on a schedule or as a process
step
 Where – multiple targets, multiple sources,
archiving, PCI/PII considerations
 How – Protocols: FTP, SFTP, FTPS, CIFS,
VPN; Encryption: PGP
Requirements – Why?
 Why indeed.
 Not our problem, just do as the customer
requests.
 Seriously though, fight for the best design.
Reasons
 Why should we use file transfers?





To communicate securely with external entities
To interact with legacy applications
Because they’re simple and easy to support
When you don’t need data transformation
When someone need to review the data
 Beware of PCI/PII requirements!
Reasons
 Why develop a framework?
 Oracle SOA Suite “FTP Adapter” doesn’t do
what you might think it does.
 Oracle Data Integrator FTP/SFTP setup is
clunky, not easily extensible
 Often times, many very similar integrations are
required
Concepts - Java
 Object oriented programming
 Native language for Weblogic Application
Server
 Relatively easy, JDeveloper is a great IDE
 Much, much faster than SOA tools
 Widely used, plenty of knowledgeable
developers
 Oracle: “Resistance is futile”
Concepts – Transfer Protocols
 FTP – Old ‘n reliable
 Credential based login, or anonymous
 Not encrypted
 Easy to implement, widely supported
 SFTP – “FTP” over SSH
 Encrypted transfer using Secure Shell
 Allows for additional authentication measures
Concepts – Transfer Protocols
 FTPS – FTP with encryption over SSL
 Support for multiple authentication methods
 Support for public key certificates (SSL)
 CIFS (aka SMB) – Used by Windows
 Allows for Active Directory or Kerberos
authentication
 Handy when FTP isn’t set up on Windows
servers
Concepts – Security
 Transport Level Security (See above)
 Data Level Security – PGP (Pretty Good
Privacy)
 Offers encryption in a wide variety of
algorithms and strengths
 Offers data compression, integrity checking,
and message authentication
 Unbreakable!
The Framework
 Most file transfers follow a similar design:
1.
2.
3.
4.
5.
Trigger condition (sometimes)
Source(s) for files
Encryption/decryption/renaming (sometimes)
Target(s) for files
Archiving (sometimes)
The Framework - Goal
 To make these common tasks configurable
via XML elements
 To allow custom tasks to work seamlessly
with common tasks to minimize
development time
The Framework – JAXB
 The XML part is easy:
 Create an XML schema with elements for each
step’s configuration
 Use JAXB (built-in to JDeveloper) to create
matching Java objects
 Use these objects as “properties” for your step
implementations
 Create a mapping between XML element types
and Java classes to execute
The Framework – Trigger Condition
 A trigger condition decides whether the
transfer should occur
 Typically a “trigger file” is used, placed in a
directory near where the actual file will be
 Useful for preventing accidental transfer
during file write
 Configurations: FTP location,
authentication, file name
The Framework – Source config
 FTP location, authentication, remote
directory, file names or file patterns
 Java supports regular expressions, so use
them
 Not all FTP servers support glob
expressions
The Framework – local work
 Encryption config
 Public key
 Whether to delete unencrypted file afterwards
(yes)
 Decryption config
 Private key
 Passphrase
The Framework – Target Config
 Multiple target elements, each with:
 FTP location, authentication
 Remote directory
The Framwork - Archiving
 Considerations
 Local vs. remote
 Append timestamps, other clerical tasks
 Archiving sensitive data
The Framework – Custom task
 Create your own Java class to do some
complicated work (i.e., building a file from
a database query)
 XML configuration:
 Class name
 Name/Value properties used by the class
Putting it All Together – Task Interface
 A “Task” needs:
 A definition: settings, configurations
 A context: what happened before?
 A result: something that can be interpreted and
acted upon
Task Interface – Task Definition
 Say, one of the XML elements you just
defined
 The Task knows what to do with it
Task Interface - Context
 A way to keep track of what happened in
previous steps
 Most obvious example is, what files are we
working with?
 Doesn’t need to be too complex
Task Interface – Result
 Three values
 Success/Failure
 Whether to continue
 Message for grateful support person
One More Interface - Plan
 Just a way to build a list of tasks from a
given XML configuration
 Execute them in order, checking the result
after each task

Bonus points: have “Plan” implement “Task”
phew
 Hard part is over
 Lunch soon!
I feel ya, bro
Quartz
 Well known (among Java nerds) library for
scheduling jobs
 Free as in speech, beer
 Well documented, tons of features
 Runs in a clustered environment
 Free web-app UIs exist
 You can pay for support if you want
Quartz – Runs stuff
 Use Quartz if:




You want to run something every five minutes
You want to run something every day
You want to run something on demand
You want to run something every 15 minutes
from noon to 2pm on the fourth day of each
month and every second Tuesday
An Alternative to Quartz
 Weblogic Job Scheduler





Built-in to Weblogic =)
Not portable =(
Not widely used =(
More complicated setup =(
Documented by Oracle =(
Quartz – Concepts
 Job – interface for Java classes, with only and
“execute” method. Quartz can run any class that
implements Job.
 Trigger – Tells the schedule when to run
 Groups – Group Jobs for better control
 Listener – Your own optional class that gets
notified when things run (for logging purposes)
 Job Data Map – Configurations or data passed in
when the Job is run.
Putting the FTP Framework in Quartz
 Create a Java class that implements Job (so
Quartz can run it)
 Schedule the Job, put your XML file in the
Job Data Map so it is accessible by your
new class.
 Have the class set up a Plan of Tasks based
on the XML file, and execute it.
Bonus - Quartz/SOA Interaction
 Quartz -> SOA
 Use the javax.xml.soap libraries to create a simple web
service Job
 Now you can schedule your SOA composites!
 SOA -> Quartz
 Create a simple class in your scheduler project that
triggers a job on-demand in Quartz based on an input
job name.
 In JDeveloper, right-click and select “Create Web
Service”
 Now you can kick-off scheduler jobs from SOA!
You’ve been a great audience
 Takeaways:
 Java is your best bet for file transfers if you’re
using Oracle products for integration
 Abstract Source/Target/Encryption tasks since
they’re all pretty much the same
 XML is a good format for configurations since
it’s extensible and easy to import to Java
Thanks OWMUG
 Takeaways:
 Quartz is a powerful, free scheduling tool used
by many large organizations
 Quartz can interact with SOA via web services,
too (or anything Java can interact with- ODI,
AS400, SQL, etc.)
select * from QUESTIONS
Libraries and Links (all free!)
 Quartz - http://quartz-scheduler.org/
 Apache Java FTP/FTPS client library http://commons.apache.org/net/
 JSCH Java SFTP client library –
http://www.jcraft.com/jsch/
 CIFS/SMB client library - http://www.samba.org/
 Bouncy Castle Java PGP library http://www.bouncycastle.org/
 JWatch Quartz Manager UI http://code.google.com/p/jwatch/