Transcript Slide - aplts

S ome
S ecrets of
S uccessful
S cripting and
S tatistics
h arvey
h ahn
Arlington Heights Memorial Library
Arlington Heights, Illinois
© 2009 by Harvey E. Hahn
Presenter
Harvey E. Hahn
Former Manager,
Technical Services,
Arlington Heights
Memorial Library
Arlington Heights, Illinois
Program overview
• Part 1
• Tips gained from 4 years and over 25,000
lines of AutoIt code (as well as 12 years of writing
152 OCLC macros, including III telnet applications)
• Part 2
• Brief “whet your appetite” introduction to
the J programming language for
processing exported Millennium data
Some
Scripting
“Secrets”
Purpose
• To share helpful patterns, tips, and
techniques that you can use in your
scripts to automate Millennium
What is a script?
• One or more groups of instructions
that perform a sequence of window
actions,
often entering, modifying, and/or
deleting data,
imitating what a person would perform
What is “dumb” scripting?
• Sends text, keystrokes, and mouse
clicks to one or more windows
“blindly”
• Does not (and cannot) “react” to what is
happening on the screen
• Millennium keystroke “macros” are
simple examples of this
What is “smart” scripting?
• “Aware” of which window is currently
displayed
• Can “react” appropriately to screen
behavior
• Sends text, keystrokes, and mouse
clicks to one or more windows
“intelligently”
• Can “read” textual data from the screen
and act upon it in some way
Why use scripts?
• Computer can do what it’s good at
• Speed
• Repetition
• People can do what they’re good at
• Judgment / Decision making
Result :
• Increased efficiency
• Increased productivity
Script language comparison
• Cost
• AutoIt, AutoHotKey, VBScript, and Expect are
free
• OCLC Macro Language is freely available to
OCLC customers only
• Macro Express costs $40 per computer
• GUI capabilities
• AutoIt, AutoHotKey, and Macro Express do
windows
• VBScript, Expect , and OML do not
Other helpful freeware tools
• FRHED (FRee Hex EDitor)
http://www.kibria.de/frhed.html
• IrfanView (“EAR-fun-view”) (image viewer)
http://www.irfanview.com/
• Any text editor (Notepad, Notepad+, NoteTab,
Win32Pad, etc.)
FRHED
IrfanView
Successful scripting “secrets”
•
•
•
•
•
Typing keystrokes
“Where am I?”
Window coordinates
Clicking buttons / tabs / icons
WYSIWYG—not! (getting window data)
Typing keystrokes
• Known issue with Microsoft Windows:
• A script can send keystrokes faster than
Windows can process them
• Programmers need to insert delays after
Send commands to permit Windows to
catch up
• The delay value should be at least 2/10ths
of a second, possibly more
Typing keystrokes
• For a Send command to work
successfully, a Sleep command needs
to be paired with it:
Send( “John{ENTER}” )
Sleep( 200 )
Send( “Smith{ENTER}” )
Sleep( 200 )
Typing keystrokes
• More efficient approach:
Func Type( $Text )
Send( $Text )
Sleep(400) ; add pause after sending text
EndFunc
...
Type( “John{ENTER}” )
Type( “Smith{ENTER}” )
Typing keystrokes
• Millennium scripting tip:
Always first try using a keyboard shortcut
(before trying mouse clicks) because it’s
the easiest and most reliable technique
Typing keystrokes
• Millennium keyboard equivalents:
#100743 – function keys
#100742 – general shortcut keys
#100535-100542 – window controls and
associated shortcut keys
Typing keystrokes
Millennium keyboard equivalents
Typing keystrokes
Millennium keyboard equivalents
Typing keystrokes
Millennium keyboard equivalents
“Where am I?”
• Getting the title of the active
window
Why?
Identify various windows (error,
processing, get info from user, etc.)
“Where am I?”
• Waiting for a window to appear
Why?
• Script must wait until a given window is
“ready for action”
• Script might need to check for possible
appearance of an “optional” window
• This is one of the most important aspects
of GUI scripting!
“Where am I?”
• Waiting for a window to appear
Tip:
• Watch for possible “surprise” appearance
of an error message window
• These are undocumented!
“Where am I?”
• Waiting for a window to appear
How?
• Keep checking to see when the title of the
window has changed (but sometimes
multiple windows have the same title!), or
• Wait until the window is “active” (focus)
(most common approach)
“Where am I?”
• Waiting for a window (or part of a
window) to change
Why?
• Script must wait until a given window is “ready for
action”
• Script must wait until part of a given window has
changed and is “ready for action”
• This is one of the most important aspects of GUI
scripting!
“Where am I?”
• Waiting for a window (or part of a
window) to change
How?
(in AutoIt)
• (1) Determine a rectangular area to check
• (2) Take a “snapshot” of the area with
PixelCheckSum
• (3) Keep taking “snapshots” until the pixel
checksum for the area changes (compared with
the original value)
“Where am I?”
• Getting pixel colors of text, background,
or images (“worst case” scenario)
AutoIt syntax:
$hexcolor = Hex( PixelGetColor( $x, $y ), 6 )
0 =
FFFFFF =
RRGGBB
black
white
“Where am I?”
• Distinguish windows with same titles
“C”
“D”
“Where am I?”
• Using patterns of pixels to identify
characters or positions on the screen is
why it is so important to maximize
windows and thus stabilize coordinates
• Coordinates being “off” by 1 or 2 pixels
can make all the difference in the world!
Window coordinates
• The screen is the entire desktop
• A window is placed somewhere on the
screen
• A Millennium window usually covers the
entire screen
Window coordinates
• Coordinates are zero-based from the
upper left corner of a window or screen
• Coordinates inside of a window stay the
same, regardless of where a window is
located on the screen
Window coordinates
98
79
98
79
Window coordinates
• Millennium scripting tip:
A window must be maximized for
consistent coordinates
Example of AutoIt syntax:
WinSetState( $WinName, "", @SW_MAXIMIZE )
NOTE:
This command will even maximize
windows which have no maximize button!
Window coordinates
• Millennium scripting tip:
The screen resolution affects coordinates!
Always indicate your screen resolution in
the script preliminaries—this helps others
when using or adapting your script
Window coordinates
Window coordinates
Clicking buttons/tabs/icons
• “Wrapper” function approach:
Func ClickAt( $x, $y )
MouseClick( "left", $x, $y, 1, 0 )
Sleep(400) ; add pause after clicking mouse
EndFunc
...
ClickAt( 400, 300 )
Clicking buttons/tabs/icons
• Clicking depends on the desired
coordinates of the mouse pointer on a
screen or window
• Coordinates may be relative to:
• a given window
• the full screen
Clicking buttons/tabs/icons
Window-based coordinates
Clicking buttons/tabs/icons
Screen-based coordinates
Clicking buttons/tabs/icons
• To keep coordinates inside of a window
stable, they should always be relative to
that window (not to the screen)
Clicking buttons/tabs/icons
• To resolve any screen/window conflicts,
offsets are needed from the screen
edges to a window’s upper left corner
(“origin”)
Clicking buttons/tabs/icons
Y
offset
X offset
Clicking buttons/tabs/icons
• Match coordinate references to the type
of justification being used
• Add offsets to left edge for left-justified text,
tabs, icons, etc.
• Subtract offsets from right edge for rightjustified text, tabs, icons, etc.
Clicking buttons/tabs/icons
Clicking buttons/tabs/icons
Clicking buttons/tabs/icons
WYSIWYG—not! (getting window data)
• “Reading” data off the screen is often
the most difficult and challenging task
when writing scripts to automate
Millennium
WYSIWYG—not! (getting window data)
• Select (highlight) specific text
How?
• Use the standard Windows “select all” command
(Ctrl-a)
• Depending on the screen context, this may select
the contents of a single row or the contents of an
entire bib, item, order, or patron record
WYSIWYG—not! (getting window data)
• Use the clipboard
AutoIt syntax:
$text = ClipGet() ; get data from clipboard
ClipPut( $text ) ; put data onto clipboard
Note:
It is standard practice to clear the clipboard before
copying any data to it
WYSIWYG—not! (getting window data)
Example
Func GetHighlightedData()
ClipPut( “” ) ; always clear the clipboard first!
TypeCtl ( “c” ) ; copy highlighted data to the clipboard
TypeCtl ( “c” ) ; duplicate needed for reliability (why??)
$HighlightedData = ClipGet() ; put data into a variable
Return $HighlightedData
EndFunc
...
TypeCtl( “a” ) ; select (highlight) all editable text
$text = GetHighlightedData()
WYSIWYG—not! (getting window data)
• Tables
Algorithm
• Choose a row, using Home, End, Up, or Down
Rapidly type a row number or
Click a row in the table
• Copy the auto-highlighted data into a string
• Parse the string, looking for specific data
or
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
check for this text
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
• Millennium scripting tip:
• To find a highlighted row in a table, don’t
loop through every pixel in a column,
looking for the color
• Note: Cell data in tables is 16 pixels apart. For speed
purposes, check just every 8th (or even 16th) pixel in a
blank area of a given data column
WYSIWYG—not! (getting window data)
• Dropdown lists
Elements in these lists cannot be “read” by scripts
• Identifiable elements
• There is known data that can be used to
distinguish and choose elements in the list
• Unidentifiable elements
• No data in the elements in the list can be used
to distinguish or choose any of the elements
WYSIWYG—not! (getting window data)
• Dropdown lists
• Identifiable elements
• “If the list item has a bold letter preceding it,
you can select the list item by pressing the
letter.” (Manual # 100541)
Algorithm
• Choose the list (usually an Alt-key combination)
• Send( “c” ) or Send( “i” ) or Send( $choice )
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
No letters, yet works!
Note: first element is NOT default!
WYSIWYG—not! (getting window data)
• Dropdown lists
• Unidentifiable elements
Its ordinal position in the list is the only way to
refer to an element
Algorithm
• Choose the list (usually an Alt-key combination)
• “Hard code” the list choices into the script
• Based on the desired choice, use Home, End,
and the Up and Down arrows to move the
highlight to the corresponding list element
• Send( “{ENTER}” ) to choose that element
WYSIWYG—not! (getting window data)
WYSIWYG—not! (getting window data)
END key moved to last element
WYSIWYG—not! (getting window data)
• Fixed-field tables and variable fields
(screen editor variations)
Algorithm
•
•
•
•
Click anywhere within the table or record
Select all of the data (Ctrl-a)
Copy the data into a string
Parse the string, looking for specific data
Getting item data
Getting item data
Getting item data
Getting item data
Getting item data
hex 20 = space
Getting item data
Getting item data
Getting item data
hex 09 = TAB
(not spaces)
hex 0d 0a = CRLF
(end of line)
Getting bib data
Getting bib data
Getting bib data
Getting bib data
Getting bib data
Getting bib data
hex 20 = space
Getting bib data
Getting bib data
Getting bib data
hex 09 = TAB
(not spaces)
Getting bib data
hex 7c =
subfield_delimiter
(pipe)
hex 0d 0a = CRLF
(end of line)
hex 09 = TAB
(not spaces)
Changing/deleting data
• Fixed fields:
• Click at end of fixed field table
• Ctrl-Home to beginning of first fixed field
• "tab to a field" approach
Changing/deleting data
• Variable fields (top-down):
• Click at end of fixed field table
• Tab once to first variable field
• "read a line at a time" approach (downward)
(using Home and Shift-End to get each line)
Changing/deleting data
• Variable fields (bottom-up):
• Ctrl-End to end of last variable field
• Home to beginning of field
• "read a line at a time" approach (upward)
(using Home and Shift-End to get each line)
Changing/deleting data
• Millennium scripting tip:
To replace data in a variable field, rather
than fiddle with insertions and such, it's
sometimes easier to delete the field and
then add the modified version as a new
field
Next-to-last scripting “secret”
• Using a script recorder is NOT cheating!
• Enables you to establish a starting point
from which you can adapt as necessary
Final scripting “secret”
• Reuse code written by yourself and
others -- there’s no point in reinventing
the wheel!
• 2009 Conference program web site:
FavFuncs.au3
REVIEW using Data Exchange
REVIEW using Data Exchange
OPTIONAL:
check title for correct window
REVIEW using Data Exchange
OPTIONAL:
check pixels for correct screen
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
send keystroke (DOWN key)
REVIEW using Data Exchange
send keystroke (END key)
REVIEW using Data Exchange
send keystroke (ENTER key)
REVIEW using Data Exchange
click mouse
REVIEW using Data Exchange
send keystroke (TAB key)
REVIEW using Data Exchange
send keystroke (SHIFT-HOME key)
REVIEW using Data Exchange
send keystrokes (IP address)
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
send keystrokes:
Alt-u user_name
Alt-p password
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
wait for window to appear
then wait for window to disappear
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
send keystroke (DOWN key)
REVIEW using Data Exchange
send keystroke (identifier--“a” key)
REVIEW using Data Exchange
click mouse
REVIEW using Data Exchange
compare if this is desired filename
if not, send DOWN key, and
compare next filename
REVIEW using Data Exchange
REVIEW using Data Exchange
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
wait for window to appear
then wait for window to disappear
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
send keystroke
REVIEW using Data Exchange
send keystroke
alternative keystroke
REVIEW using Data Exchange
OPTIONAL:
send keystroke
REVIEW using Data Exchange
OPTIONAL:
send keystroke
REVIEW using Data Exchange
OPTIONAL:
send keystroke
Why two parts in this talk?
• My vision:
To combine the Innovative system,
AutoIt scripts, J scripts, and one’s local
PC to create a powerful fusion of
technologies for creating, managing,
manipulating, and interpreting
Innovative data
Some
Statistics
“Secrets”
Purpose
• To introduce the J programming
language as a powerful alternative and
addition to Excel for handling exported
Millennium data
What is “J”?
• An improved derivative of APL
• Invented in 1989 by Ken Iverson, who
also invented APL in early 1960’s
• Availability (free):
http://www.jsoftware.com
What is “J”?
• Like APL, J operates on an entire array
as a single entity rather than thinking
of the array as a collection of separate
elements
• J views nearly everything as an array
What is “J”?
• Terms for arrays of increasing rank:
•
•
•
•
atom – a single element (rank 0)
list – one or more atoms (rank 1)
table – one or more lists (rank 2)
report – one or more tables (rank 3+)
Analogy of J to geometry
• atom
point
• list
line
• table
plane
(0 dimensions)
(1 dimension)
(2 dimensions)
• report cube, hypercube, etc.
(3+ dimensions)
Another J analogy
• atom
• list
• table
• report
What is “J”?
• Expanding upon the arithmetic symbols
(+, -, etc.), J uses nearly all symbols
(and some letters) on an ASCII
keyboard to represent operations on
arrays
• Each symbol also has 2 inflected forms,
appending either a period or a colon
What is “J”?
• Elements in a programming command
can be any of these parts of language:
•
•
•
•
•
nouns
verbs
adverbs
conjunctions
gerunds
What is “J”?
• There are only two basic command
formats—everything else is a variation
on these:
•
•
verb noun
noun verb noun
(monadic - 1 arg)
(dyadic - 2 args)
• Note: A noun argument can actually contain multiple
arguments or parameters, if needed; a verb can be a
J primitive or something you (or others) have written
(because J is extensible)
What is “J”?
• Both forms resolve to a noun, which
can be substituted in a larger command:
( noun verb noun ) verb ( verb noun )
noun
verb
noun
What is “J”?
• Often, but certainly not always, the
righthand noun argument is data and
the lefthand noun argument (if any)
controls how the verb acts on the data
What is “J”?
• The key difference in these noun
arguments from other programming
languages is that not only can they be
single numeric or literal values but they
can also be entire arrays!
• This is the main “secret” of J’s power
What is “J”?
• A second “secret” of J’s power is that
the explicit array looping needed in
other languages is built into J’s
commands
• This considerably simplifies the writing
of scripts and helps mentally focus on
the “big picture”
What is “J”?
• Because J is so terse, often a whole
program (or portions) can be written
sideways rather than vertically!
• Here is an example of a whole program
within a single line of code (frequencies
of subject headings in a catalog):
sh7=.|:((|:~.(/:{)sh),(<&>#;.1~:(/:{)sh))
What is “J”?
What is “J”?
• Downsides:
• Looks like “Martian”
• Rather “geeky”
• Huge amount of documentation, but very
difficult to find specific information
• Steep learning curve may intimidate
beginners
• Major paradigm shift in thinking
What is “J”?
• Upsides:
• Can do a lot with very little code
• Availability of a very complete suite of
“assisting” applications, most written in J
(plotting, forms and controls, spreadsheet,
database, regular expressions, etc.)
• Size of arrays is limited only by amount of
memory and disk space available
• So-called “boxed” arrays can contain
mixed numeric and literal data
Examples of J primitives
i. n
|.
|:
/:~
~.
,
=:
create list of integers from 0 to n-1
reverse order of items in array
transpose (flip) rows/cols in table
sort items in list in ascending order
dedupe items in list
append two sets of items
global assignment ( =. is local )
Simple J program/script
br3=: ~. /:~ (br1,br2)
• br1 and br2 are two lists of bib rec numbers that have already
been read from disk
• Meaning of script (executed right-to-left):
•
•
•
•
Append br2 to br1 to form a single large list
Sort this list
Dedupe the list
Assign (globally) the list to variable br3
More examples of J
• You can give your own names to verbs:
] revfile1=: 1 + i.10
1 2 3 4 5 6 7 8 9 10
] revfile2=: 2 3 5 8
2 3 5 8
remove=: -. NB. usually known as “not”
] result=: revfile1 remove revfile2
1 4 6 7 9 10
More examples of J
• You can create your own verbs:
shfreq=: 3 : 0
NB. “x” is LH arg, “y” is RH arg of verb:
sh2=.
sh3=.
sh4=.
sh5=.
sh6=.
(/:{) y
# ;.1 ~: sh2
|: ~. sh2
< every sh3
|: (sh4,sh5)
NB.
NB.
NB.
NB.
NB.
sorted list
freq’s
unique SH’s
counts
combine SH’s & counts
)
• Use new verb like any other J verb:
freqlist=: shfreq sh
NB. sh = file of SH’s
More examples of J
• You can use verbs that others have created:
updatewith is a verb written by someone
proficient in J at my request
(see updatemasterfile.ijs on 2009 Conference program web site)
It appends a column of data to an existing
(“master”) table, matching row data according
to the keys in the first column of both files
(this provides a workaround for III’s zero suppression in reports)
More examples of J
• Example of monthly statistics use:
load '~user\temp\updatemasterfile.ijs'
master=: jpath '~user\data\master.csv'
newdata=: jpath '~user\data\month01.csv'
master updatewith newdata
• Note: updatemasterfile.ijs script contains 2 verbs with 21 lines of
code
More examples of J
• Example of use with patron checkouts:
load '~user\temp\updatemasterfile.ijs'
master=: jpath '~user\data\master.csv'
newdata=: jpath '~user\data\snapshot01.csv'
master updatewith newdata
• Note: this can be subsequently used to calculate circulation
statistics between two or more “snapshot” dates
More examples of J
• It’s also possible to “GUI-fy” the
previous simple scripts by having two
dropcombo boxes of file directories that
a staff member could choose from—
without having to know or learn J
themselves!
Appetite whetted for J?
• J has enough array and mathematical power
to keep you learning for a lifetime
• J can be used independently of Excel
• J can be used in conjunction with Excel
(there are also direct interfaces with Excel)
• I hope that I’ve interested you enough in J so
that you can…
J
Pick up the torch
and go for it!
More information
•
---- AHML web site ---(as long as it’s available)
http://www.ahml.info/oml
• Links: Scripts Using AutoIt
Scripts Using J
(including my “Beginning J”)
More information
• ---- WILIUG presentations ---• More about AutoIt scripting (Jun 2006):
http://www.wiliug.org/meetings/2006/6_06.html
• More about the J language (Jun 2009):
http://www.wiliug.org/meet.html
More information
• ------ Discussion groups -----• Script language forums:
http://www.autoitscript.com/forum/index.php?
http://www.autohotkey.com/forum/
http://pgmacros.invisionzone.com/ (Macro Express)
• J language forums/groups:
http://www.jsoftware.com/forums.htm
http://groups.google.com/group/J-Programming
Email addresses
• After May 2009:
[email protected]
or
[email protected]
Thanks for attending!
Q&A