Transcript Python Advanced (Powerpoint)

Python Programming
by
Chris Seddon
Copyright ©2000-15 CRS Enterprises Ltd
1
Python Programming
1.Introduction to Python
2.The Basics
3.More on Data Types
4.Functions
5.Classes and Objects
6.Modules and Packages
7.Installing Packages
8.Files
9.Attribute Lookup
10. Advanced Techniques
11. Decorators
12. Regular Expressions
Copyright ©2000-15 CRS Enterprises Ltd
13.Exception Handling
14.Unit Test
15.Inheritance
16.Threading
17.Extending and Embedding
18.Meta Classes
19.Oracle Database
20.Operator Overloading
21.Web Application Frameworks
22.JSON and XML
23.Miscellaneous Topics
3
1
Introduction to Python
IronPython
Copyright ©2000-15 CRS Enterprises Ltd
6
What is Python?
Python is a Dynamic Scripting Language
not a static language like C++, C# and Java
easy to learn
no compile step
can be run a line at a time inside an interpreter (REPL)
Supports both Object Oriented and Functional styles
often described as an agile language
ideal for rapid prototyping and testing
often used as a plug-in to a C++ or Java system
extensive library
Python Virtual Machine
garbage collection
reference counted objects
Copyright ©2000-15 CRS Enterprises Ltd
7
Advantages of Python
Python is often used ...
to develop small programs and scripts
simple programs can be created very quickly
... but can be used for large-scale programming projects
sophisticated module and packaging scheme
extensive object oriented support
Python has a modern design
more natural coding style than other languages
highly portable
Copyright ©2000-15 CRS Enterprises Ltd
8
What is Jython?
Jython is an implementation of Python
that compiles to Java byte code
runs on any JVM
highly portable
thread safe
Jython currently supports the Python syntax at level 2.5
Jython Libraries
uses Java libraries because of the JVM
NOT standard Python libraries
Copyright ©2000-15 CRS Enterprises Ltd
9
What is Iron Python?
.NET framework version of Python
runs in the CLR
only used on Windows systems (and mono)
Iron Python currently supports the Python syntax at level 2.7
Iron Python Libraries
uses .NET libraries and Python libraries
Copyright ©2000-15 CRS Enterprises Ltd
10
Everything is Interpreted
Python is an interpreted language
no compile step as in Java and C++
each time Python code is run it is interpreted afresh into byte code
unless byte code cached internally
Code can also be entered interactively in the REPL
myfile.py
compile
Python
byte code
interpret
Python
interpreter
Target
Hardware
Copyright ©2000-15 CRS Enterprises Ltd
11
Performance
Python is interpreted
much slower than a compiled language such as C/C++
increased design and coding flexibility more than makes up for performance
loss
Python libraries usually written in C/C++
a large proportion of the Python library is written in C/C++
as is the interpreter itself
mitigates poor performance of pure Python
Python has modules to measure performance
profile, cProfile
Copyright ©2000-15 CRS Enterprises Ltd
12
Invoking Python
1. From the Command Line
code is entered one line at a time
see the results immediately
ideal way to learn Python
2. Use Python Script files
interpreted by the Python runtime
source files can be combined using modules for larger applications
3. IDEs
e.g. PyDev plug-in for Eclipse
PyDev is very mature (10+ years development)
full debugging support
Copyright ©2000-15 CRS Enterprises Ltd
13
Python Distributions
Python home page
http://www.python.org/
Major Releases
Python 3.4.3
February, 2015
Python 3.4.2
October, 2014
Python 3.4.1
May, 2014
Python 3.4.0
March, 2014
Python 2.7.9
December, 2014
Python 2.7.8
July, 2014
Python 2.7.7
June, 2014
Python 2.7.6
November, 2013
Jython 2.7 Beta 4
February, 2015
Jython 2.7 Beta 3
August, 2014
Iron Python 2.7.5
December, 2014
Copyright ©2000-15 CRS Enterprises Ltd
14
Python Resources
Python docs
http://www.python.org/doc/
http://docs.python.org/modindex.html
ActiveState Python
http://www.activestate.com/activepython/
Python Tutorial by Guido van Rossum
http://www.python.org/doc/2.2.3/tut/tut.html
Python Cookbook
http://oreilly.com/catalog/9780596007973/
Copyright ©2000-15 CRS Enterprises Ltd
15
Books
Python Essential Reference, Fourth Edition
By: David M. Beazley
Publisher: Addison-Wesley Professional
The Definitive Guide to Jython: Python for the Java™
Platform
By: Josh Juneau; Jim Baker; Victor Ng; Leo Soto; Frank Wierzbicki
Publisher: Apress
Expert Python Programming: Learn best practices to
designing, coding, and distributing your Python
software
By: Tarek Ziadé
Publisher: Packt Publishing
Copyright ©2000-15 CRS Enterprises Ltd
16
2
The Basics
Copyright ©2000-15 CRS Enterprises Ltd
20
A "Hello World" Example
Trivial one liner in Python
you don’t even require a semi-colon!
print "Hello World!"
Copyright ©2000-15 CRS Enterprises Ltd
21
Comments and Indentation
# for comments
Indentation instead of braces
# this is a comment
x = 100
y = 200
if x + y < 500:
print "sum is less than 500"
colon
indent
Copyright ©2000-15 CRS Enterprises Ltd
22
Input
Input is converted to a string
may need to use a cast
x = int(raw_input("Please enter an integer: "))
x=x+1
Please enter an integer: -5
Copyright ©2000-15 CRS Enterprises Ltd
23
if Statements
x = int(raw_input("Please enter an integer: "))
if x < 0:
print 'Negative'
Please enter an integer: -5
Negative
Copyright ©2000-15 CRS Enterprises Ltd
24
if Statements
x = int(raw_input("Please enter a positive integer: "))
if x > 0:
print 'x is positive'
print 'the square of x is', x * x
print 'the cube of x is', x * x * x
print 'End of if-statement'
Please enter a positive integer: 66
x is positive
the square of x is 4356
the cube of x is 287496
End of if-statement
Copyright ©2000-15 CRS Enterprises Ltd
25
if-else Statements
x = int(raw_input("Please enter an integer: "))
if x < 0:
print 'Negative'
else:
print 'Not negative'
Please enter an integer: 5
Not negative
Please enter an integer: -5
Negative
Copyright ©2000-15 CRS Enterprises Ltd
26
if-elif-else Statements
x = int(raw_input("Please enter an integer: "))
if x < 0:
print 'Negative'
elif x == 0:
print 'Zero'
elif x == 1:
print 'One'
else:
print 'Greater than one'
Copyright ©2000-15 CRS Enterprises Ltd
Please enter an integer: -3
Negative
Please enter an integer: 0
Zero
Please enter an integer: 1
One
Please enter an integer: 7
Greater than one
27
Conditional if Statements
x = 100
result = (-1 if x < 0 else 1)
print result
x = -200
result = (-1 if x < 0 else 1)
print result
1
-1
Copyright ©2000-15 CRS Enterprises Ltd
28
for Statements
for x in (10,17,24,31,38,45,52,59,66,73,80,87,94):
print x,
print
for x in range(10,100,7):
print x,
10 17 24 31 38 45 52 59 66 73 80 87 94
10 17 24 31 38 45 52 59 66 73 80 87 94
Copyright ©2000-15 CRS Enterprises Ltd
29
for-else Statements
for x in (1,2,3,4,5,6):
print x,
else:
print "only get here if all iterations succeed ..."
for x in (1,2,3,4,5,6):
print x,
if x > 3:
break
else:
print "only get here if all iterations succeed ..."
1 2 3 4 5 6 only get here if all iterations succeed ...
1234
Copyright ©2000-15 CRS Enterprises Ltd
30
while Statements
formula = 0
x=0
while formula < 1000:
x=x+1
formula = 2*x*(x + 1)
print x, formula
Copyright ©2000-15 CRS Enterprises Ltd
1 4
2 12
3 24
4 40
5 60
6 84
7 112
8 144
9 180
10 220
11 264
12 312
13 364
14 420
15 480
16 544
17 612
18 684
19 760
20 840
21 924
22 1012
31
Integers/Long
Unlimited precision
decimals, octal and hexadecimal
x = 300
x = 053
x = 0xFF
# decimal
# octal
# hex
x=1
for n in range(1,50000):
...
x=x*n
8640372151906087336721162787027740701980046703537693007160454743
print x
0881028943029710890094952575983102831942332098801078767873403184
3812303645836587362298825375962390412214818730218601896401786005
4217856498897415140571330537697495383717507271695713173805885868
6051525163484354006340593710441714431751346890331014890001250991
3262093824861142024508385572268159407796321465886842319191296757
8078562876393851213501459437877878620141455136735126924004144819
7911187026793326350625732189630574843722324400961845552066777053
7106542144272488920866258356532973475712072964180507234862689087
6456030420157452090095580448550503063205853759254527374217561203
7061462858200453655527745928196559852331864722970592646723758606
...
Copyright ©2000-15 CRS Enterprises Ltd
32
Booleans
Values
True, False
Operators
and, or
x = True
y = False
z = x or y
print z
True
z = x and y
print z
print type(z)
False
<type 'bool'>
Copyright ©2000-15 CRS Enterprises Ltd
33
Floating Point
13 significant figures on 32 bit implementation
no exact representation
# floating point
x = 1e6 + 0.000001e-4
format = "%32.20g"
print type(x)
print (format % x)
<type 'float'>
1000000.0000000001
Copyright ©2000-15 CRS Enterprises Ltd
34
Complex Numbers
floating types
real and imaginary parts
can be extracted separately
x = (+2.5-3.4j) - (-1.4+1.0j)
print type(x)
print x
print x.real
print x.imag
Copyright ©2000-15 CRS Enterprises Ltd
<type 'complex'>
(3.9-4.4j)
3.9
-4.4
35
None
special type
NoneType
x = 120
print type(x)
print x
x = None # not defined
print type(x)
print x
<type 'int'>
120
<type 'NoneType'>
None
240
if (x == None):
x = 240
print x
Copyright ©2000-15 CRS Enterprises Ltd
36
3
More On Data Types
Immutable Types
int/long
float
tuple
frozen set
Mutable Types
list
dict
set
Copyright ©2000-15 CRS Enterprises Ltd
40
Object References
All data types are objects
x is an object reference (pointer)
can point to any type of object
int
float
bool
NoneType
str
str
list
tuple
dict
x = 200
x = 5.734
x = True
x = None
x = 'Hello'
x = "Goodbye"
x = [2,3,5,7,11,13,17]
x = (2,3,5,7,11,13,17)
x = { 'Tom':23000, 'Sue':24500, 'Zoe':21800}
Copyright ©2000-15 CRS Enterprises Ltd
41
Immutable Types
Several types are immutable
int, float, str, NoneType, tuple
state cannot be modified after object is created
Immutable objects are often useful because they are
inherently thread-safe
What happens when you try to change an immutable
object?
id(x) = unique id of
object pointed at by x
x = 100
print id(x)
10053412
x=x+1
print id(x)
10053400 x points to a different object
Copyright ©2000-15 CRS Enterprises Ltd
42
Mutable Types
Several types are mutable
list, dict, class
Operations change state
id of object doesn't change
x = [2,3,5,7,11,13,17]
print id(x)
x.append(23)
print id(x)
print x
Copyright ©2000-15 CRS Enterprises Ltd
43143336
43143336
[2, 3, 5, 7, 11, 13, 17, 23]
43
Strings ...
immutable
single or double quotes
triple quoting for multi-line strings
x = 'hello'
y = "from"
z = """the
planet
earth"""
print type(x)
print x, y, z
<type 'str'>
hello from the
planet
earth
HELLO FROM THE
PLANET
EARTH
phrase = x + " " + y + " " + z
print phrase.upper()
Copyright ©2000-15 CRS Enterprises Ltd
44
... Strings
Many string manipulation methods available
original string is unchanged
new string returned
s = "---abc:xyz:123---"
t = s.lower()
t = s.lstrip("-")
t = s.replace(":","@")
t = s.rstrip("-")
t = s.split(":")
t = s.strip("-")
t = s.swapcase()
t = s.upper()
Copyright ©2000-15 CRS Enterprises Ltd
---abc:xyz:123--abc:XYZ:123-----abc@XYZ@123-----abc:XYZ:123
['---abc', 'XYZ', '123---']
abc:XYZ:123
---ABC:xyz:123-----ABC:XYZ:123---
45
Printing
Use the format method on strings
x = 6.5
y = 1208.5632
z = 200
x=6.5, y=1208.5632, z=200
x= 6.5000, y=1208.5632, z=200.0000
print "x={0}, y={1}, z={2}".format(x, y, z)
print "x={0:8.4f}, y={1:8.4f}, z={2:8.4f}".format(x, y, z)
Copyright ©2000-15 CRS Enterprises Ltd
46
Tuple
immutable
can be nested
note syntax for a single item tuple
x1 = ()
x2 = (1,)
x3 = (1,2,3)
x4 = (1, "mixed", 2, "tuple")
x5 = ((1,2),(3,4))
Copyright ©2000-15 CRS Enterprises Ltd
47
List
mutable
can be nested
x1 = [ ]
x2 = [1]
x3 = [1,2,3]
x4 = [1, 'mixed', 2, 'tokens']
x5 = [[1,2],[3,4]]
Copyright ©2000-15 CRS Enterprises Ltd
48
Memory Layout
mylist = [ (10,20), [30,40] ]
list
mylist
tuple
10
Copyright ©2000-15 CRS Enterprises Ltd
list
20
30
40
49
Using range
use range to iterate a fixed number of times
for x in range(1,10):
print x,
123456789
Copyright ©2000-15 CRS Enterprises Ltd
50
Modifying Lists
Lists can be extended
with another list
List can be appended
adds an element to a list
list1 = [10, 20, 30]
list2 = [55, 56, 57]
list2.extend(list1)
print list
[55, 56, 57, 10, 20, 30]
list2.append(list1)
print list2
[55, 56, 57, 10, 20, 30, [10, 20, 30]]
Copyright ©2000-15 CRS Enterprises Ltd
51
Accessing Lists and Tuples
only lists can be modified
tuples are immutable
mylist = [10, 20, 30]
mytuple = (10, 20, 30)
y = mytuple[0]
y = mytuple[1]
y = mytuple[2]
x = mylist[0]
x = mylist[1]
x = mylist[2]
mylist[1] = 99
mylist.append(40)
print mylist
[10, 99, 30, 40]
Copyright ©2000-15 CRS Enterprises Ltd
52
Casting between Lists and Tuples
use a cast to convert
tuple( ... )
list( ...)
theList = [2, 3, 5, 7, 11, 13, 17]
myTuple = tuple(theList)
myList = list(myTuple)
print myTuple
print myList
Copyright ©2000-15 CRS Enterprises Ltd
(2, 3, 5, 7, 11, 13, 17)
[2, 3, 5, 7, 11, 13, 17]
53
List Methods
To remove an element
use del or pop()
To check for existence
use in
myList = [1,2,3,4,5,6]
del myList[3]
item = myList.pop(3)
print myList
[1, 2, 3, 6]
a=3
if a in myList:
print "list contains a" list contains a
Copyright ©2000-15 CRS Enterprises Ltd
54
Slices
Sequences support slicing
selects a range of elements
x[1:3] selects the second through third elements of x
the end index is always one past the selection
assignment slicing
replaces multiple elements
x[3:5] = (5,4)
colors = ["red", "blue", "green", "white", "black"]
print colors[1]
print colors[1:3]
print colors[1:]
print colors[-3:-1]
print colors[4:1:-1]
colors[2:4] = ("purple", "cyan")
print colors[0:];
Copyright ©2000-15 CRS Enterprises Ltd
blue
['blue', 'green']
['blue', 'green', 'white', 'black']
['green', 'white']
['black', 'white', 'green']
['red', 'blue', 'purple', 'cyan', 'black']
55
Dictionary
like a database table
also called hash, associative array
key:value pairs
keys usually a string
keys must be immutable
very efficient lookups
empty = { }
months = {"Jan":1, "Feb":2, "May":5, "Dec":12}
seasons = {"Spring":("Mar","Apr","May"), "Summer":("Jun","Jul","Aug")}
print type(empty)
print months["Dec"]
print seasons["Summer"]
Copyright ©2000-15 CRS Enterprises Ltd
<type 'dict'>
12
('Jun', 'Jul', 'Aug')
56
Working with Dictionaries
Dictionaries are mutable
modify existing entries
add new entries
Key Order is indeterminate
because of hashing algorithm
salary = {
"zak": 34000,
"sara": 27000,
"pedro": 52000,
"jo": 12500,
"zoe": 66000
}
print salary["pedro"]
salary["jo"] = 15000; # modify existing
salary["peter"] = 35000 # add new entry
Copyright ©2000-15 CRS Enterprises Ltd
57
Initialization
Use { } or constructor
salary1 = {
"zak": 34000,
"sara": 27000,
"pedro": 52000,
"kilas": 12500,
"zoe": 66000
}
salary2 = dict( zak
= 34000,
sara
= 27000,
pedro
= 52000,
kilas
= 12500,
zoe
= 66000)
Copyright ©2000-15 CRS Enterprises Ltd
58
Inserting or Updating
New elements ...
are added
salary["sara"] = 28000
Existing elements ...
are updated
salary["sara"] = 29000
Copyright ©2000-15 CRS Enterprises Ltd
59
Removing Keys
Use del statement
del salary["zak"]
or
Use pop
salary.pop("zak")
Copyright ©2000-15 CRS Enterprises Ltd
60
Removing Values
Assign key to None
previous value gets garbage collected
salary["sara"] = None
Copyright ©2000-15 CRS Enterprises Ltd
61
Common Functions ...
Extract all the keys, and all the values
# create list of keys
theKeys = salary.keys()
# create list of values
theValues = salary.values()
# get key value pairs as a list
list_of_key_value_pairs = salary.items()
Copyright ©2000-15 CRS Enterprises Ltd
62
... Common Functions
Work with keys, values or both
# print all the keys
for key in salary.iterkeys():
print key,
print
# print all the values
for value in salary.itervalues():
print value,
print
# print all <key,value> pairs
for key, value in salary.iteritems():
print key, value
Copyright ©2000-15 CRS Enterprises Ltd
63
Set and FrozenSet ...
unordered collections of unique elements
Set (mutable)
FrozenSet (immutable)
myset = set(("Monday", "Tuesday"))
myset.add("Wednesday")
myset.add("Thursday")
myset.add("Friday")
print myset
if "Wednesday" in myset:
print "member"
myset.remove("Wednesday")
if "Wednesday" not in myset:
print "not a member"
Copyright ©2000-15 CRS Enterprises Ltd
set(['Friday', 'Tuesday', 'Thursday',
'Wednesday', 'Monday'])
member
not a member
64
... Set and FrozenSet
Set supports mathematical operations
union, intersection and symmetric difference
sequence1 = 1, 3, 5, 7, 5, 4, 3, 2, 1
sequence2 = 2, 4, 6, 8, 6, 4, 3, 2, 1
set1 = set(sequence1)
set2 = set(sequence2)
print(set1)
set([1, 2, 3, 4, 5, 7])
print(set2)
set([1, 2, 3, 4, 6, 8])
unionSet = set1.union(set2)
intersectionSet = set1.intersection(set2)
differenceSet = set1.symmetric_difference(set2)
print(unionSet)
set([1, 2, 3, 4, 5, 6, 7, 8])
print(intersectionSet)
set([1, 2, 3, 4])
print(differenceSet)
set([5, 6, 7, 8])
Copyright ©2000-15 CRS Enterprises Ltd
65
4
Functions
Copyright ©2000-15 CRS Enterprises Ltd
68
Functions
square is an object reference
not the name of the function
the function is an object
all objects are anonymous
def square(n):
print n * n
square(5)
square(6)
square(7)
square
Copyright ©2000-15 CRS Enterprises Ltd
25
36
49
print n * n
69
Function Aliases
You can point two or more references at a function
or point the reference elsewhere
as in fib = 42
def fib(n):
a, b = 0, 1
while b < n:
print b,
a, b = b, a+b
print
fib(1000)
1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987
f = fib
f(100)
fib = 42
1 1 2 3 5 8 13 21 34 55 89
Copyright ©2000-15 CRS Enterprises Ltd
70
return Statements
use return statements to ...
pass data back to the calling program
return a tuple or list if you need to return more than one piece of data
def average(a, b):
return (a + b)/2.0
x = average(5, 7)
print x
6.0
x = average(5.5, 7.7)
print x
6.6
print average(10.5, 12.5)
11.5
Copyright ©2000-15 CRS Enterprises Ltd
71
Function with Mutable Data
With mutable data ...
no need to use a return statement
no need to assign the return in the calling program
Mutable data can be modified in the function
changes are seen in the calling program
def addOneElement(m):
m.append(99)
mylist = [12, 15, 18]
addOneElement(mylist)
print mylist
[12, 15, 18, 99]
Copyright ©2000-15 CRS Enterprises Ltd
72
Function with Immutable Data
Object references are copied when a function is called
the addresses are copied
and copied on return
With immutable data you must ...
use a return statement
assign the return in the calling program
def swap(a,b):
return b,a
x = 100
y = 200
x,y = swap(x,y)
print x, y
200 100
Copyright ©2000-15 CRS Enterprises Ltd
73
How not to Swap
no_swap fails because
no return statement
no assignment in the calling program
def no_swap(a,b):
temp = a
a=b
b = temp
x = 100
y = 200
no_swap(x,y)
print x, y
100 200
Copyright ©2000-15 CRS Enterprises Ltd
74
Swapping Lists
Lists are mutable data, so ...
no need to use a return statement
no need to assign in the calling program
def swap(theList):
theList [0], theList[1] = theList[1],theList[0]
mylist = [100,200]
swap(mylist)
print mylist
[200, 100]
Copyright ©2000-15 CRS Enterprises Ltd
75
Default Parameter Values
Default parameters can be either:
positional or
named
def Display(a, b=10, c=100):
print ("a=%6.1f " % a), ("b=%6.1f " % b), ("c=%6.1f" % c)
Display(19, 6.2, 4.8)
Display(17)
Display(17, 21)
Display(17, c=0)
Copyright ©2000-15 CRS Enterprises Ltd
a=
a=
a=
a=
19.0
17.0
17.0
17.0
b=
b=
b=
b=
6.2
10.0
21.0
10.0
c= 4.8
c= 100.0
c= 100.0
c= 0.0
76
Variadic Functions ...
Use * to define a variable number of positional
parameters
get packed into a tuple
Use * in call flatten positional parameters
def f(*args):
print 'args:', args
a = (3,4,7,8)
b = [3,4,7,8]
args: (1, 3, 4, 7, 8, 9)
args: (1, (3, 4, 7, 8), 9)
args: (1, [3, 4, 7, 8], 9)
args: (1, 3, 4, 7, 8)
args: (1, 3, 4, 7, 8)
f(1,3,4,7,8,9)
# pass multiple args
f(1,a,9)
# pass int, tuple, int
f(1,b,9)
# pass int, list, int
f(1,*a) # treat tuple as multiple args
f(1,*b) # treat list as multiple args
Copyright ©2000-15 CRS Enterprises Ltd
77
... Variadic Functions
Use ** to define a variable number of named parameters
get packed into a dict
Use ** in call flatten named parameters
def f(x,y,**params):
print 'x,y:', x, y
print 'params:', params
print params['Dec']
hash = { 'a':1, 'b':2 }
f(13,7, May=5, Jan=1, Feb=2, Dec=12, **hash)
x,y: 13 7
params: {'a': 1, 'b': 2, 'Feb': 2, 'May': 5, 'Jan': 1, 'Dec': 12}
12
Copyright ©2000-15 CRS Enterprises Ltd
78
Lambda Functions
Alternative to using def
but is an expression not a statement
Often used to defined functions in line
that will not be called from elsewhere
sum = lambda x, y: x + y
print sum(1,2)
print sum(5,9)
print sum(10,20)
3
14
30
Copyright ©2000-15 CRS Enterprises Ltd
equivalent definition
def sum(x, y):
return x + y
79
Stub Functions
Use stub functions when ...
you are not ready to define the function
often used in test driven development
def myStub(x, y):
pass
function will be defined later
myStub(5,6)
myStub(1,10)
myStub(0,7)
Copyright ©2000-15 CRS Enterprises Ltd
80
5
Classes and Objects
Classes
what is a class?
what is an object?
Methods
defining
calling
Attributes
for objects
for classes
Copyright ©2000-15 CRS Enterprises Ltd
84
The Object Model
Object-oriented programming
modeling real-world objects
model described by classes
model implemented by creating
objects
Copyright ©2000-15 CRS Enterprises Ltd
85
Classes
Methods
are functions
Attributes
are object references
superclass
reference to self
operations
methods
class Yacht(object):
attributes
def __init__(self, bearing, speed):
self.bearing = bearing
self.speed = speed
def display(self):
print self.bearing, self.speed
Copyright ©2000-15 CRS Enterprises Ltd
86
The Object Class
object
All classes have a single root
class object
but only for new style classes
Person
Object
defines methods inherited by all Python
classes
Employee
SalesPerson
Copyright ©2000-15 CRS Enterprises Ltd
87
Creating Objects
Use the class name to create a new object
returns an object reference
the object is not directly accessible
Object destroyed by the VM
when no references exist
class Person:
...
object references
Copyright ©2000-15 CRS Enterprises Ltd
john = Person()
mary = Person()
88
Object Attributes
Object attributes can be
defined on the fly
can apply to different objects
Object b1 has
height
width
color
Object b2 has
height
width
Use pass to define a empty
class
useful for trying things out
Copyright ©2000-15 CRS Enterprises Ltd
class Box:
pass
b1 = Box( )
b2 = Box( )
b1.height = 100
b1.width = 50
b1.color = "red"
b2.height = 100
b2.width = 50
89
Manipulating Objects ...
constructor
create objects
call methods
class Point:
def __init__(self, x = 0, y = 0):
self.x = x
self.y = y
def display(self):
print "Point is at", self.x, self.y
def moveBy(self, dx, dy):
self.x += dx
self.y += dy
p1 = Point(5,7)
p2 = Point(15,17)
p3 = Point(25,27)
p1.moveBy(1,1)
p2.moveBy(1,10)
p3.moveBy(1,100)
p1.display()
p2.display()
p3.display()
Copyright ©2000-15 CRS Enterprises Ltd
Point is at 6 8
Point is at 16 27
Point is at 26 127
90
Class Attributes
Shared by all objects in
class
class attribute
class Point:
count = 0
object
attribute
def __init__(self, x = 0, y = 0):
self.x = x
self.y = y
Point.count = Point.count + 1
def display(self):
print "Point({0}) is at {1},{2}".format(id(self), self.x, self.y)
normal
method
static(class)
method
def moveBy(self, dx, dy):
self.x += dx
self.y += dy
def HowManyPoints(self):
print "There are {0} Point object(s)\n".format(Point.count)
Copyright ©2000-15 CRS Enterprises Ltd
91
Memory Layout
Point
K
__bases__
__dict__
V
‘count’
3
K
p1
V
__class__
__dict__
Copyright ©2000-15 CRS Enterprises Ltd
‘x’
100
‘y’
200
92
6
Modules and Packages
Copyright ©2000-15 CRS Enterprises Ltd
96
Modules and Packages
Programs can be organized into modules and packages
many modules are available in the Python and Java libraries
Modules
single file
usually Python code
but can be written in other languages such as C/C++
Packages
collection of modules grouped using a common package name
helps resolve namespace conflicts
A package is defined by creating a directory
with the same name as the package
place modules in this directory
can be extended to include sub-packages within sub-directories
Copyright ©2000-15 CRS Enterprises Ltd
97
What is a Module
Importing a file
executes all the code in that file
often just defines functions
myfunctions.py
import myfunctions
myfunctions.f1()
myfunctions.f2()
myfunctions.f3()
myfunctions.f4()
def f1():
pass
def f2():
pass
def f3():
pass
def f4():
pass
Copyright ©2000-15 CRS Enterprises Ltd
98
Module Test Code
Importing a file ...
causes all code to executed
but what about test code?
Wrap all test code ...
inside a __name__ test
__name__ is only set to
__main__
if we are the main program
code will not be executed via an import
Copyright ©2000-15 CRS Enterprises Ltd
def f1():
def f2():
def f3():
def f4():
pass
pass
pass
pass
if __name__ == "__main__":
f1()
f2()
f3()
f4()
99
Importing
4 ways to import
mypackage
mysubpackage
import mypackage.mysubpackage.MyClass
x1 = mypackage.mysubpackage.MyClass()
x1.f1()
from mypackage import mysubpackage
x2 = mysubpackage.MyClass()
x2.f2()
public class MyClass {
public MyClass() { ... }
public void f1() { ... }
public void f2() { ... }
public void f3() { ... }
public void f4() { ... }
}
import mypackage.mysubpackage.MyClass as theClass
x3 = theClass()
x3.f3()
from mypackage.mysubpackage import MyClass as theClass
x4 = theClass()
x4.f4()
Copyright ©2000-15 CRS Enterprises Ltd
100
Packages
Import
loads all modules together
executes __init__.py
for each package
import sys
sys.path.append("lib")
# load both packages together
import mypackage.mysubpackage
x = mypackage.mysubpackage.ModuleA.A()
x.f();
loading mypackage
x.g();
loading mysubpackage
x.h()
this is A.f() ...
this is A.g() ...
this is A.h() ...
Copyright ©2000-15 CRS Enterprises Ltd
mypackage/__init__.py
mysubpackage/__init__.py
ModuleA.py
101
Module Search Path
When loading modules
the interpreter searches the list of directories in sys.path
first entry in sys.path can be an empty string
refers to the current working directory
other entries in sys.path may consist of
directory names, .zip archive files, and .egg files
order determines which modules are loaded
import sys
sys.path.append('mylib')
Copyright ©2000-15 CRS Enterprises Ltd
102
Displaying Symbol Tables ...
To display symbol table of main program
use the global dictionary
Iterate through the dictionary using iteritems()
# use globals to get a list of symbols in main module
# Python returns a StringMap which needs to be
# converted to a dictionary
symbols = dict(globals())
#print all the symbols in module
for key, value in symbols.iteritems():
print key, value
Copyright ©2000-15 CRS Enterprises Ltd
103
... Displaying Symbol Tables
To display symbol table of a module
use the module's dictionary
Iterate through the dictionary using iteritems()
#print all the symbols in myfunctions module
for key, value in myfunctions.__dict__.iteritems():
print key, value
Copyright ©2000-15 CRS Enterprises Ltd
104
7
Installing Packages
Copyright ©2000-15 CRS Enterprises Ltd
108
Installing Third-Party Libraries
Definitive resource for locating third-party libraries and
extensions to Python is
the Python Package Index (PyPI)
http://pypi.python.org
Installing third-party modules is usually straightforward
use pip or easy_install
use platform-native installer
use a download a zip that comes with setup.py file
Copyright ©2000-15 CRS Enterprises Ltd
109
Python Module Downloads
Pypi repository (cheese shop)
http://pypi.python.org/pypi
Download package, extract and then run
e.g. install pip
wget https://pypi.python.org/packages/source/p/pip/pip-6.0.8.tar.gz
gunzip pip-6.0.8.tar.gz
tar xvf pip-6.0.8.tar
cd pip-6.0.8
python setup.py install
Copyright ©2000-15 CRS Enterprises Ltd
110
Pip
Pip is a package management system
used to install and manage software packages
Installation
Python 2.7.9+ and Python 3.4+ include pip by default
earlier versions had separate installers
pip install requirements
Collecting requirements
Downloading requirements-0.1.tar.gz
Installing collected packages: requirements
Running setup.py install for requirements
Successfully installed requirements-0.1
Copyright ©2000-15 CRS Enterprises Ltd
111
Install Packages with Dependencies
Pip works with package versions:
$ pip install SomePackage
$ pip install SomePackage==1.0.4
$ pip install 'SomePackage>=1.0.4'
# latest version
# specific version
# minimum version
Wheel is a pre-built archive:
greatly speeds up installation compared to building and installing from
source archives
Pip prefers Wheels where they are available, but otherwise defaults to
source archives
$ pip install SomePackage-1.0-py2.py3-none-any.whl
Copyright ©2000-15 CRS Enterprises Ltd
112
Pip Commands
Uninstall a package
pip uninstall SomePackage
List installed packages
pip list
Show details of a package
pip show sphinx
Query for Packages
pip search "query"
Copyright ©2000-15 CRS Enterprises Ltd
113
Compiling from Source
Some packages need to be compiled before installation
may need to control which C++ compiler is used
1. Create a config file
in distutils directory called distutils.cfg
2. Edit file to specify compiler
distutils.cfg
[build]
compiler=mingw32
Copyright ©2000-15 CRS Enterprises Ltd
114
Pip behind a Proxy
install behind a proxy
use HTTP_PROXY or HTTPS_PROXY environment variables
HTTP_PROXY = http://<myproxyhost>:<port>
HTTPS_PROXY = https://<myproxyhost>:<port>
e.g.
export HTTP_PROXY = http://www.mydomain.com:8080
pip install somepackage
Copyright ©2000-15 CRS Enterprises Ltd
115
8
Files
Opening and closing
Reading and writing
Modes
read only
read-write
append
Iterating through
Random Access
seek
Copyright ©2000-15 CRS Enterprises Ltd
118
File Objects
open
creates a file object
close
destroys a file object
myFile = open('myfile.txt', 'w')
myFile.close()
Copyright ©2000-15 CRS Enterprises Ltd
119
Iterating through a File
Use for-in statement
to process every line
loop terminates on end of file
f = open("hello.txt")
for line in f:
print line
Copyright ©2000-15 CRS Enterprises Ltd
120
Using with Statement
with statement
automatic clean up
avoid forgetting to close a file
old way
f = open("hello.txt")
try:
for line in f:
print line
finally:
f.close()
from __future__ import with_statement
# This isn't required in Python 2.6+
with ("hello.txt") as f:
for line in f:
print line
Copyright ©2000-15 CRS Enterprises Ltd
121
Coping with Errors
If file I/O fails
Error object thrown
belongs to IOError class
try:
f = open("data/unknown.txt", "r+t")
except IOError,error:
print "File open failed!"
error object
print error
error class
Copyright ©2000-15 CRS Enterprises Ltd
122
Read Methods
Simple to read from files
f = open('myfile.txt', 'r')
data = f.read()
f.close()
read entire file
f = open('myfile.txt', 'r')
data = f.read(100)
f.close()
f = open('myfile.txt', 'r')
f.readline()
f.readline()
f.readline()
f.close()
read a line at a time
(empty string returned on EOF)
read 100
bytes from file
Copyright ©2000-15 CRS Enterprises Ltd
123
Write Methods
Simple to write to files
f = open("data/out1.txt", "w")
f.write("line 1\nline 2\n")
f.close()
write a string
sequence = ("line 1\n", "line 2\n", "line 3\n", "line 4\n")
f = open("data/out2.txt", "w")
f.writelines(sequence)
f.close()
write a sequence of
lines
Copyright ©2000-15 CRS Enterprises Ltd
124
File Modes
read only
use "r"
read and write
use "r+"
or "w+" to empty file on opening
write only
use "w"
file will be empty on opening
appending
use "a"
binary
use "b" for binary and "t" for text (default)
random access
choose appropriate mode and then use seek()
Copyright ©2000-15 CRS Enterprises Ltd
125
Writing Binary Data
Open file with mode 'b'
myFile = open('data/myfile2.bin', 'wb')
myFile.write("\x5F\x9D\x3E");
myFile.close()
Copyright ©2000-15 CRS Enterprises Ltd
126
Reading Binary Data
Use ord() and hex(ord()) for conversions
myFile = open('data/myfile2.bin', 'rb')
# read bytes
ch1 = myFile.read(1) # read 1 byte
ch2 = myFile.read(1)
ch3 = myFile.read(1)
# convert to ascii (decimal)
d1 = ord(ch1)
d2 = ord(ch2)
d3 = ord(ch3)
# convert to ascii (hex)
h1 = hex(ord(ch1))
h2 = hex(ord(ch2))
h3 = hex(ord(ch3))
myFile.close()
Copyright ©2000-15 CRS Enterprises Ltd
127
Random Access
seek(offset, whence)
whence = 0
whence = 1
whence = 2
=> from start of file
=> from current position
=> from end of file
myFile = open('myfile.txt', 'w')
myFile.seek(40, 0)
myFile.write("ABCDEFGH");
myFile.seek(140, 0)
myFile.write("ABCDEFGH");
myFile.seek(240, 0)
myFile.write("ABCDEFGH");
myFile.close()
Copyright ©2000-15 CRS Enterprises Ltd
128
Reading a File into a Dictionary
You can read an entire file into a Dictionary
and then perform lookups
stdcodes = {}
f = open("codes.txt", "r+t")
read a file line by line, splitting each
line and then adding to dictionary
for line in f:
[code, place] = line.rstrip().split(' ', 1)
stdcodes[place] = code
now try searching the dictionary
placeRequested = raw_input("Enter place: ")
try:
print "The code for " + placeRequested + \
" is " + stdcodes[placeRequested]
Town: Washington
except KeyError:
The code for Washington is 0191
print "code not found"
Copyright ©2000-15 CRS Enterprises Ltd
129
Sorting a File
stdcodes = {}
f = open("codes.txt", "r+t")
read a file line by line, splitting each
line and then adding to dictionary
for line in f:
[code, place] = line.rstrip().split(' ', 1)
stdcodes[place] = code
list = [ ]
list = stdcodes.keys()
list.sort()
for place in list:
print place, stdcodes[place]
Copyright ©2000-15 CRS Enterprises Ltd
extract the keys into an
array then sort the array
now step through the sorted list looking
up each entry in the original dictionary
130
9
Attribute Lookup
Copyright ©2000-15 CRS Enterprises Ltd
134
Attribute Lookup ...
1. __getattribute__()
always called if present
2. direct dictionary lookup
use dictionaries
3. non direct dictionary lookup
if descriptor protocol used
call descriptor methods
else
search dictionaries
print A.x
# dotted class lookup
o = A()
print o.x
# dotted attribute lookup
print o.__dict__['x'] # direct dictionary lookup
4. lookup failure
call __getattr__ or __setattr__
Copyright ©2000-15 CRS Enterprises Ltd
135
... Getting Algorithm for Objects
a.__getattribute__()
exists
no
object.__getattribute__() gets called
yes
a.__class__[‘x’]
is a
descriptor
call
a.__getattribute()
a = A()
a.x
no
yes
a.__dict__[‘x’]
exists
invoke one of
a.__class__[‘x].__get__()
a.__class__[‘x’].__set__()
a.__class__[‘x’].__del__()
no
yes
a.__class__.__dict__[‘x’]
exists
invoke a.__dict__[‘x’]
no
yes
a.__getattr__()
exists
invoke
a.__class__.__dict__[‘x’]
Copyright ©2000-15 CRS Enterprises Ltd
yes
no
invoke
a.__getattr__()
raise
AttributeException
136
__getattribute__ Method
Only works with new style classes
not used by old style classes
Called unconditionally
to implement attribute accesses for instances of the class
__getattr__() not called unless you raise an AttributeError
To avoid infinite recursion
call the base class method to access any attributes:
object.__getattribute__(self, name)
class Point(object):
def __getattribute__(self, name):
print "*** getting", name
return object.__getattribute__(self, name)
Copyright ©2000-15 CRS Enterprises Ltd
137
Direct Dictionary Lookup
Look up via the class dictionary
always direct - no overrides
Look up via the instance dictionary
always direct - no overrides
class A:
...
A.__dict__['x']
# class dictionary lookup
o = A()
o.__dict__['x']
# attribute dictionary lookup
Copyright ©2000-15 CRS Enterprises Ltd
138
Dotted Lookup without Descriptors
Look up class attribute
search class dictionary (A.__dict__)
search __mro__ base classes dictionaries
Look up instance attribute
search instance dictionary
search class dictionary (A.__dict__)
search __mro__ base classes dictionaries
A.x
o = A()
o.x
Copyright ©2000-15 CRS Enterprises Ltd
# dotted class lookup
# dotted attribute lookup
139
Descriptors ...
The descriptor protocol defines 3 methods:
__get__()
__set__()
__delete__()
Any class implementing any of the above methods is a
descriptor
define the attributes of your class to be references to instances of a descriptor
do not try to use descriptors with object attribute
nothing happens
Copyright ©2000-15 CRS Enterprises Ltd
140
... Descriptors
Define the class implementing the descriptor interface
no all methods are required
class Trace(object):
def __init__(self, name):
self.name = name
def __get__(self, obj, objtype):
print "GET:" + self.name + " = " +
str(obj.__dict__[self.name])
return obj.__dict__[self.name]
def __set__(self, obj, value):
obj.__dict__[self.name] = value
print "SET:" + self.name + " = " +
str(obj.__dict__[self.name])
Copyright ©2000-15 CRS Enterprises Ltd
141
... Descriptors
Use the descriptor interface in your class
descriptor invoked by dotted attribute access: A.x or a.x
descriptor reference must be stored in the class dict, not instance dict
descriptor not invoked by dictionary access: Point.__dict__['x']
class Point(object):
x = Trace("x")
y = Trace("y")
def __init__(self, x0, y0):
self.__dict__['x'] = x0
self.__dict__['y'] = y0
def moveBy(self, dx, dy):
self.x = self.x + dx
self.y = self.y + dy
Copyright ©2000-15 CRS Enterprises Ltd
142
... Descriptors
Layout in memory
Trace
name
K
Point
V
__bases__
__dict__
Trace
‘x’
name
‘y’
K
p1
‘x’
‘y’
V
__class__
__dict__
Copyright ©2000-15 CRS Enterprises Ltd
‘x’
100
‘y’
200
143
... Descriptors
Invoking the descriptor
object class will redirect attribute lookup only if ...
class defines descriptors
otherwise object's dictionary is used
Direct access to object's __dict__
bypasses descriptor
p1 = Point(15, 25)
p1.x = 20
p1.y = 35
result = p1.x
p2 = Point(16, 26)
p2.x = 30
p2.moveBy(1, 1)
Copyright ©2000-15 CRS Enterprises Ltd
SET:x = 20
SET:y = 35
GET:x = 20
SET:x = 30
GET:x = 30
SET:x = 31
GET:y = 26
SET:y = 27
144
Property Descriptors
Allow you to define methods as properties
property implements the interface behind the scenes
getter and setter methods invoked by assignment
class Point(object):
def getX(self):
print "getting x"
return self._x
def setX(self, value):
print "setting x"
self._x = value
p = Point()
p.x = 55 # calls setX()
a = p.x # calls getX()
del p.x # calls delX()
def delX(self):
print "deleting x"
del self._x
x = property(getX, setX, delX)
Copyright ©2000-15 CRS Enterprises Ltd
145
Handling Undefined Methods
Calling an undefined method causes ...
an exception to be thrown ...
Unless a class defines a __getattr__() method
called by Python interpreter
class MyClass:
def __getattr__(self, method):
print "method not found"
return self.myDefault
def myDefault(self): print "default()"
def f(self): print "f..."
def g(self): print "g..."
a = MyClass()
a.f()
a.g()
a.dummy()
f...
g...
method not found
default()
# method doesn't exist
Copyright ©2000-15 CRS Enterprises Ltd
146
Get Attribute Lookup Failures
If class or instance attribute lookup fails
__getattr__ is called if it exists
class A(object):
x = 100
# used if attribute not found by lookup
def __getattr__(self, name):
self.__dict__[name] = 0 # used to avoid recursion
return self.__dict__[name]
a1 = A()
a1.x = 200
a2 = A()
print a1.x
print a2.x
print a2.y
# attribute found in instance
# attribute found in class
# attribute not found
Copyright ©2000-15 CRS Enterprises Ltd
147
Attribute Assignment
1. __setattr__() exists ...
call when an attribute assignment is attempted
2. otherwise
instance dictionary is updated
__setattr__() can assign to an instance attribute using:
use self.__dict__[name] = value for old style classes
use object.__setattr__(self, name, value) for new style classes
but don't execute ...
self.name = value
... causes a recursive call
Copyright ©2000-15 CRS Enterprises Ltd
X
def __setattr__(self, name):
self.name = name # recursive call
148
The 5 Intrinsic Class Attributes
__name__
__doc__
__module__
__dict__
__bases__
name of the class (read only)
class documentation string
module in which the class is defined
map of all class attributes
tuple of all base classes
class Derived(Base):
"This is the Derived class"
def Display(self):
pass
print Derived.__name__
print Derived.__doc__
print Derived.__module__
print Derived.__dict__
print Derived.__bases__
Copyright ©2000-15 CRS Enterprises Ltd
Derived
This is the Derived class
__main__
{'Display': <function Display at ...
(<class __main__.Base at ...
149
The 2 Intrinsic Object Attributes
__class__
the object's class
__dict__ map of all the object's attributes
class Derived(Base):
"This is the Derived class"
def Display(self):
pass
x = Derived()
x.color = "red"
x.width = 10
print x.__class__
print x.__dict__
__main__.Derived
{'color': 'red', 'width': 10}
Copyright ©2000-15 CRS Enterprises Ltd
150
10
Advanced Language Techniques
Copyright ©2000-15 CRS Enterprises Ltd
154
Lambda functions
Used to build on the fly methods
builds a function at run time
only input parameters and return allowed
no function body
def do_operation(op, a, b):
return {'+':
'-':
'*':
'/':
}[op]()
lambda: a + b,
lambda: a - b,
lambda: a * b,
lambda: a / b
print do_operation('+', 2, 5)
Copyright ©2000-15 CRS Enterprises Ltd
155
Decorators
Decorators allow you to inject or modify code
in functions and classes
Introduced with @ symbol
@staticmethod
@classmethod
@myFunction
@myClass
def trace(fn):
def enhance(x):
print "calling function with parameter x =", x
return fn(x)
return enhance
@trace
def h(x):
return x * x
print h(7)
Copyright ©2000-15 CRS Enterprises Ltd
calling function with parameter x = 7
49
156
Iterators
Create a class with __iter__() and next() methods
behave like built in iterator objects
__iter__() called at start of iteration
normally returns self
next() called for each iteration
for f in Fibonacci():
print f,
class Fibonacci:
def __init__(self):
self.x,self.y = 0,1
def __iter__(self):
return self
def next(self):
if self.x > 10000:
raise StopIteration # end of iteration
self.x, self.y = self.y, self.x + self.y
return self.x
Copyright ©2000-15 CRS Enterprises Ltd
157
Generators
Generators are used to generate a sequence of objects
often used as iterators
uses the yield keyword
Create a class
with __iter__() method and your generator method
generator uses yield to return an object and suspend
generation ends when function returns (no yields left)
class Fibonacci:
def __init__(self):
f = Fibonacci()
self.x,self.y = 0,1
def __iter__(self):
# get generator
return self
g = f.myGenerator()
def myGenerator(self):
# call the generators next() method
while self.x < 10000:
print g.next()
yield self.x
self.x, self.y = self.y, self.x + self.y
return
Copyright ©2000-15 CRS Enterprises Ltd
158
Filters
Filters define a predicate function
returns True/False
allow filtering of a sequence
def primesFilter(x):
result = True
if x % 2 == 0: result = False
if x % 3 == 0: result = False
if x % 5 == 0: result = False
if x % 7 == 0: result = False
if x % 11 == 0: result = False
if x % 13 == 0: result = False
return result
sequence = range(14, 200)
# filter out all non primes
primes = filter(primesFilter, sequence)
print primes
Copyright ©2000-15 CRS Enterprises Ltd
159
Maps
Maps apply a function to every item in a sequence
def cube(x):
return x * x * x
# set up a sequence
sequence = range(1, 20)
# apply a map to entire sequence
cubes = map(cube, sequence)
for value in cubes:
print "%6i" % value,
Copyright ©2000-15 CRS Enterprises Ltd
160
Comprehensions
List comprehensions provide a concise way to create
lists
without resorting to use of map(), filter() and/or lambda
def cube(x):
return x * x * x
# set up a sequence
sequence = range(1, 20)
# apply a comprehension to entire sequence
cubes = [cube(x) for x in sequence]
for value in cubes:
print "%6i" % value,
# just print the sequence
cubes = [x for x in sequence]
for value in cubes:
print "%6i" % value,
Copyright ©2000-15 CRS Enterprises Ltd
161
Introspection
Introspection allows you to investigate methods and
attributes of a class at runtime
dir(obj)
returns all methods and attributes
callable(method)
checks if a method
p = MyClass()
print "methods and attributes:", dir(p)
methodList = [method for method in dir(p) if callable(getattr(p, method))]
print "methods:", methodList
print "attributes: ", p.__dict__
print "class:", p.__class__
print "doc string:", p.__class__.__doc__
Copyright ©2000-15 CRS Enterprises Ltd
162
Pickling
Pickling allows serializing objects to and from disk
persistent objects
import pickle
# pickle object
foo = Foo(66)
f = open("test.dat", "w")
pickle.dump(foo, f)
f.close()
class Foo:
def display(self):
print self.x
def __init__(self, x0):
self.x = x0
# unpickle object
f = open("test.dat", "r")
x = pickle.load(f)
x.display()
f.close()
Copyright ©2000-15 CRS Enterprises Ltd
163
exec and eval Statements
exec
x = 10
y = 20
codeFragment = raw_input("Please enter some code (e.g. 'print x + y'): ")
exec codeFragment
Please enter some code (e.g. 'print x + y'): print x * y + 1
201
eval
x = 10
y = 20
codeFragment = str(input("Please enter an expression (e.g. 'x + y'): "))
result = eval(codeFragment)
print "result is: ", result
Please enter an expression (e.g. 'x + y'): x * y
result is: 200
Copyright ©2000-15 CRS Enterprises Ltd
164
11
Decorators
built-in decorators
user defined decorators
Copyright ©2000-15 CRS Enterprises Ltd
168
Static Methods
@staticmethod
used with class attributes and methods
can be called either on the class or on an instance
avoid having to pass self as first parameter
class MyClass:
# static data
hits = 0
@staticmethod
def hit():
MyClass.hits = MyClass.hits + 1
m = MyClass()
MyClass.hit() # call through class
m.hit() # call through an instance
Copyright ©2000-15 CRS Enterprises Ltd
169
Class Method
@classmethod
can be used as an alternative way of defining a constructor
permits multiple constructors
class passed as first parameter - instead of self
class Point:
@classmethod
def initialize1(theClass, x, y):
p = theClass()
p.x = x
p1 = Point.initialize1(11, 26)
p.y = y
p2 = Point.initialize2(20)
return p
@classmethod
def initialize2(theClass, a):
p = theClass()
p.x = a
p.y = a
return p
Copyright ©2000-15 CRS Enterprises Ltd
170
Property Decorator
@property
used to define a computed read only property
class Circle(object):
def __init__(self, radius):
self.radius = radius
# computed properties (and hence read only)
@property
def area(self):
circle = Circle(10.0)
return math.pi*self.radius**2
@property
def perimeter(self):
return 2*math.pi*self.radius
Copyright ©2000-15 CRS Enterprises Ltd
print circle.radius
print circle.area
print circle.perimeter
171
Property Setter Decorator
@property.setter
make a method appear to be a property
class Circle(object):
def __init__(self, radius):
self.theRadius = radius
@property
def radius(self):
return self.theRadius
circle1 = Circle(10.0)
circle2 = Circle(20.0)
circle3 = Circle(30.0)
@radius.setter
circle1.radius = 15.0
def radius(self, value):
if not isinstance(value, float):
raise TypeError("Must be a float")
self.theRadius = value
Copyright ©2000-15 CRS Enterprises Ltd
172
User Defined Decorators
Decorators ...
take an input function
and return a different function
modifying the behavior of the input function
def trace(fn):
def enhance(x):
print "calling " + fn.func_name + "(" + str(x) + ")"
return fn(x)
return enhance
@trace
def square(x):
return x * x
print square(4)
Copyright ©2000-15 CRS Enterprises Ltd
calling square(4)
16
# calls trace, not square
173
Decorators with Parameters
Decorators can have parameters
def log(level):
def logit(fn):
def enhance(x):
message = "calling " + fn.func_name + "(" + str(x) + ")"
if(level == logging.DEBUG): logging.debug(message)
if(level == logging.INFO):
logging.info(message)
if(level == logging.WARNING):
logging.warning(message)
if(level == logging.ERROR): logging.error(message)
if(level == logging.CRITICAL):
logging.critical(message)
return fn(x)
@log(logging.WARNING)
return enhance
def square(x):
return logit
return x * x
logging.basicConfig(level=logging.WARNING)
print square(4)
WARNING:root:calling square(4)
16
Copyright ©2000-15 CRS Enterprises Ltd
174
Nested Decorators
Decorators can be nested
def bold(fn):
def decorate():
return "<b>" + fn() + "</b>"
return decorate
def uk(fn):
def decorate():
fields = fn().split('/')
date = fields[1] + "/" + fields[0] + "/" + fields[2]
return date
return decorate
@bold
def getBoldDate():
return getDate()
@uk
def getUkDate():
return getDate()
@bold
@uk
def getBoldUkDate():
return getDate()
<b>6/16/2013</b>
print getBoldUkDate()
def getDate():
now = datetime.datetime.now()
return "%d/%d/%d" % (now.day, now.month, now.year)
Copyright ©2000-15 CRS Enterprises Ltd
175
12
Regular Expressions
Copyright ©2000-15 CRS Enterprises Ltd
178
Anchors and Selections
selection indicates which characters to match
[abc]
any one character from list (a OR b OR c)
[a-m]
any one character from range (a OR b OR ... OR m)
[^abc]
any one character not in list (NOT a NOR b NOR c)
. any one character
\s any white space character
\S any NON white space character
\d any digit
\D any NON digit
\w any alphanumeric
\W any NON alphanumeric
anchors indicate context
^ start of string
$ end of string
\b on a word boundary
\B NOT on a word boundary
Copyright ©2000-15 CRS Enterprises Ltd
179
Repeat Counts
repeat counts apply to the previous expression
* 0 or more repeats
+ 1 or more repeats
? 0 or 1 repeats
{n,m}
n to m repeats
{n,}
at least n repeats
{n} exactly n repeats
Examples
[aeiou]*
[0-9]+
[0-9]{3,5}
\.?
0 or more vowels
1 or more digits (\d+)
3 to 5 digits
0 or 1 periods
\. matches a period
the backslash negates the special meaning of the pattern that follows
Copyright ©2000-15 CRS Enterprises Ltd
180
Pattern Matching Functions
match
match from the start of text
search
match anywhere in the text
import re
test = "-------ABC------------"
pattern = r"\w+"
pattern = re.compile(pattern)
match = pattern.match(test)
print match
None
match = pattern.search(test)
print match.group()
ABC
Copyright ©2000-15 CRS Enterprises Ltd
181
Splitting
Perhaps we want to split a string using a regular
expression as a delimiter
pattern.split(text)
import re
pattern = re.compile(r"\s*;\s*")
text = "aaa ; bbb ;ccc ;
ddd ; eee"
list = pattern.split(text)
print list
['aaa', 'bbb', 'ccc', 'ddd', 'eee']
Copyright ©2000-15 CRS Enterprises Ltd
182
Substitution
Perhaps we want to substitute something for the matched
pattern? Use
pattern.sub(replacement, text)
import re
pattern = re.compile(r"\s*;\s*")
text = "aaa ; bbb ;ccc ;
ddd ; eee"
newText = pattern.sub("---", text)
print newText
aaa---bbb---ccc---ddd---eee
Copyright ©2000-15 CRS Enterprises Ltd
183
Using Comments
Always a good idea to use comments
regular expressions are difficult to read at the best of times
import re
string = "AAAA1111BBBB2222CCCC3333DDDD";
pattern = r"""
^
# start of line
(.*?)
# 0 or more characters
# non greedy
(\d+) # 1 or more digits
(.*)
# 0 or more characters
$
# end of line
"""
compiledPattern = re.compile(pattern, re.VERBOSE)
result = compiledPattern.search(string)
Copyright ©2000-15 CRS Enterprises Ltd
184
More Advanced Searches
Use ( ) in pattern to capture data
import re
text = "---111122223333333333334444555566667777---"
pattern = "2+(3+)4+(5+)6+"
# search for pattern
pattern = re.compile(pattern)
result = pattern.search(text)
# print results
print "full match: ", result.group(0)
print "capture pattern 1: ", result.group(1)
print "capture pattern 2: ", result.group(2)
print "all captures: ", result.groups()
Copyright ©2000-15 CRS Enterprises Ltd
185
Repeating Patterns
Use findall() and finditer()
to repeat searches
import re
text = "AB12CD34EF56GH"
pattern = r"(\d+)"
# find all occurrences of pattern
matcher = re.findall(pattern, text)
print matcher
# iterate through finding the pattern
for matcher in re.finditer(pattern, text):
print matcher.groups(0)
Copyright ©2000-15 CRS Enterprises Ltd
['12', '34', '56']
('12',)
('34',)
('56',)
186
Greedy Patterns
Greedy Patterns are the default
Use ? to use non greedy matching
text = "AAAA1111BBBB2222CCCC3333DDDD"
greedyPattern = r"^(.+)(\d+)(.+)$"
nonGreedyPattern = r"^(.+?)(\d+)(.+)$"
^(.+)(\d+)(.+)$
Greedy:
<AAAA1111BBBB2222CCCC333><3><DDDD>
^(.+?)(\d+)(.+)$
Non-Greedy: <AAAA><1111><BBBB2222CCCC3333DDDD>
Copyright ©2000-15 CRS Enterprises Ltd
187
Look Ahead and Look Behind
(?= )
(?! )
(?<= )
(?<! )
peek ahead matching RE
peek ahead not matching RE
peek behind matching RE
peek behind not matching RE
\w+[.](?!bat)\w+$
?!bat
not "bat"
filename.ext
Copyright ©2000-15 CRS Enterprises Ltd
188
Building Patterns
Translation patterns can be built from hashes
e.g. translate all occurrences of mon to Monday, tue to Tuesday etc
import re
def replaceGroup(matcher):
key = matcher.groups(1)
return days[key[0]]
days = { 'mon' : 'Monday', 'tue' : 'Tuesday',
'wed' : 'Wednesday', 'thu' : 'Thursday',
'fri' : 'Friday' }
pattern = "(" + "|".join(days.keys()) + ")"
compiledPattern = re.compile(pattern)
print "pattern: ", pattern
text = '''The course starts on mon, continues on tue,
but wed is the last day'''
for matcher in re.finditer(compiledPattern, text):
text = compiledPattern.sub(replaceGroup, text, 1)
print text
Copyright ©2000-15 CRS Enterprises Ltd
189
13
Exception Handling
Copyright ©2000-15 CRS Enterprises Ltd
192
try-except Statements
array = [1,2,3,4,5,6]
exception thrown here
try:
for x in array:
print x, array[x]
exception type
print "Exiting try block"
except IndexError,target:
print "... entering except block"
print target
this is never executed
object reference
23
34
45
56
6 ... entering except block
list index out of range
Copyright ©2000-15 CRS Enterprises Ltd
193
try-except-else Statements
from math import sqrt
x = int(raw_input("Enter positive integer: "))
try:
root = sqrt(x)
except:
print "... entering except block"
else:
print "... entering else block"
print root
exception occurs
no exception
Enter positive integer: 25
... entering else block
5.0
Enter positive integer: -25
... entering except block
Copyright ©2000-15 CRS Enterprises Ltd
194
try-finally Statements
array = [1,2,3,4,5,6]
1 2 3 4 5 6 ... entering finally block
3 5 ... entering finally block
IndexError: list index out of range
try:
for x in array:
print array[x - 1],
finally:
print "... entering finally block"
guaranteed to be called
even if exception occurs
try:
for x in array:
print array[x * 2],
finally:
print "... entering finally block"
Copyright ©2000-15 CRS Enterprises Ltd
195
raise Statements
array = [1,2,3,-4,5,6]
try:
exception type
for x in array:
if x < 0:
raise ValueError, "array index is negative!"
print x,
object reference
except ValueError, value:
print
print "... entering except block: " + str(value)
123
... entering except block: array
index is negative!
Copyright ©2000-15 CRS Enterprises Ltd
196
Unified try except finally
In Python 2.5+ you can combine finally and except
not allowed in previous versions
try:
block-1 ...
except Exception1:
handler-1 ...
except Exception2:
handler-2 ...
else:
else-block
finally:
final-block
Copyright ©2000-15 CRS Enterprises Ltd
197
Nested Exceptions
def f1():
print "Entering f1"
f2()
print "Leaving f1"
never
executed
def f2():
print "Entering f2"
f3()
print "Leaving f2"
regarded as part
of the try block
Entering f1
Entering f2
Entering f3
Some exception
def f3():
print "Entering f3"
raise Exception("Some exception")
print "Leaving f3"
try:
f1()
except Exception, reason:
print reason
Copyright ©2000-15 CRS Enterprises Ltd
198
Standard Exceptions
Exception
StopIteration
StandardError
BufferError
ArithmeticError
FloatingPointErro
OverflowError
ZeroDivisionError
AssertionError
AttributeError
EnvironmentError
IOError
OSError
EOFError
ImportError
LookupError
IndexError
KeyError
Copyright ©2000-15 CRS Enterprises Ltd
MemoryError
NameError
UnboundLocalError
ReferenceError
RuntimeError
NotImplementedError
SyntaxError
IndentationError
TabError
SystemError
TypeError
ValueError
UnicodeError
UnicodeDecodeError
UnicodeEncodeError
UnicodeTranslateError
199
User Defined Exceptions
Always subclass Exception
class MyError(Exception):
def __init__(self, value):
self.value = value
try:
My exception occurred, value: 4
raise MyError(2*2)
except MyError as e:
print 'My exception occurred,
value:', e.value
Copyright ©2000-15 CRS Enterprises Ltd
200
Controlling Tracebacks
Use the traceback module
provides a standard interface to extract, format and print stack traces of Python
programs
exactly mimics the behavior of the Python interpreter when it prints a stack
trace
traceback.print_exc(limit, file)
prints limit entries in a stack trace to a file. If the parameters are omitted the
full traceback is output to the console
Copyright ©2000-15 CRS Enterprises Ltd
201
When Cleanup Raises ...
What happens if exception is raised in handler?
original exception can get lost
def doit():
# some nested function encounters a problem
raise RuntimeError('original error ...')
def myfunct():
try:
doit()
except:
# if errorCorrection() fails
# it will mask out the original exception
errorCorrection()
Copyright ©2000-15 CRS Enterprises Ltd
202
... When Cleanup Raises
Perform the cleanup in a finally block
the original exception gets preserved
def doit():
# some nested function encounters a problem
raise RuntimeError('original error ...')
def doit():
def myfunct():
# some nested function encounters a problem
try:
raise RuntimeError('original error ...')
doit()
except:
try: # re-raise the original error
raise
finally:
# handle any problems locally
try:
errorCorrection()
except:
traceback.print_exc()
Copyright ©2000-15 CRS Enterprises Ltd
203
Logging Exceptions
Always wise to log an exception
don't swallow exceptions
sometimes it is best to log and then rethrow
def do_work():
# if exception encountered in library code
# we might not know what to do about it,
# so log error and rethrow
raise Exception('do_work() failed')
try:
# perform some work
do_work()
except Exception, e:
# handle the error in your code
# because you know what you want to do about it
logging.error('problem in library code ...')
print "handling exception generated by library code ..."
Copyright ©2000-15 CRS Enterprises Ltd
204
assert Statements
def CalculateQuartile(percent):
assert type(percent).__name__ == 'int'
assert percent >= 0 and percent <= 100
quartile = 1
if percent > 25:
print CalculateQuartile(34)
quartile = 2
print CalculateQuartile(104)
if percent > 50:
quartile = 3
if percent > 75:
2
quartile = 4
assert percent >= 0 and percent <= 100
return quartile
AssertionError
Copyright ©2000-15 CRS Enterprises Ltd
205
14
Testing
Unittest framework
DocTest
PyTest
Nose
Copyright ©2000-15 CRS Enterprises Ltd
208
Components of Unittest
test fixture
code run to prepare for one or more tests
e.g. start a server
any associate cleanup actions
test case
unit of testing
implemented as a method of test class
test suite
a collection of test cases
used to aggregate tests that should be executed together.
test runner
component which orchestrates the execution of tests
provides output relating to the success or failure of the tests
Copyright ©2000-15 CRS Enterprises Ltd
209
Structure of Test Class
derive from unittest.TestCase
setUp call before each test
tearDown called after each test
import unittest
execute tests by calling
unittest.main()
test names must begin with
test...
class testPoint(unittest.TestCase):
def setUp(self):
# called before each test executes
def testA(self): ...
def testB(self): ...
def testC(self): ...
def testD(self): ...
if __name__ == '__main__':
unittest.main()
Copyright ©2000-15 CRS Enterprises Ltd
210
Structure of a Test Case
Use assertEqual()
to check the output of a test is correct
other functions available
def testMoveBy2(self):
"""moveBy test(2)"""
self.point.moveBy(5, 2)
self.assertEqual(self.point.display(), "8,7")
FAIL: moveBy test(2)
---------------------------------------------------------------------Traceback (most recent call last):
File "C:\_Eclipse\Python\Unit Test\unit-testing-1.py", line 29, in testMoveBy2
self.assertEqual(self.point.display(), "8,7")
AssertionError: '8,6' != '8,7'
Copyright ©2000-15 CRS Enterprises Ltd
211
Test Functions
assertEqual()
check for an expected result
assert_()
verify a condition
assertRaises()
verify that an expected exception gets raised
Copyright ©2000-15 CRS Enterprises Ltd
212
Test Suites
Tests can be grouped into suites
def suite():
suite = unittest.TestSuite()
suite.addTest(testPoint('testConstructor'))
suite.addTest(testPoint('testMoveBy1'))
suite.addTest(testPoint('testDistance1'))
return suite
if __name__ == '__main__':
mySuite = suite()
unittest.TextTestRunner(verbosity=0).run(mySuite)
Copyright ©2000-15 CRS Enterprises Ltd
213
DocTest
DocTest module searches for text that look like interactive
Python sessions
executes those sessions to verify that they work exactly as shown
Tests are embedded in code
easy to define tests
can cause clutter
Copyright ©2000-15 CRS Enterprises Ltd
214
Simple Example
Use doctest.testmod(...)
many parameters, but usually just use verbose=True
def cube(x):
"""
The cube function returns x * x * x
>>> cube(3)
27
>>> cube(-1)
-1
"""
return x*x*x
if __name__ == '__main__':
import doctest
doctest.testmod(verbose = True)
Copyright ©2000-15 CRS Enterprises Ltd
215
Formatting
Tests can be fuzzy
... used to omit some results
whitespace not relevant
if ELLIPIS and NORMALIZE_WHITESPACE specified
def squares(a, b):
"""
returns all the squares in range a..b
>>> squares(1,10) # doctest:+ELLIPSIS +NORMALIZE_WHITESPACE
[1, 4, ..., 100]
"""
answer=[]
for i in range(a,b+1):
answer.append(i*i)
return answer
Copyright ©2000-15 CRS Enterprises Ltd
216
Failures
def sumsquares(n):
"""
with line numbers
1+4+9+16 = 30 (not 30457)
and erroneous result
>>> sumsquares( 4 )
30457
"""
Trying:
sum = 0
sumsquares( 4 )
for i in range(n+1):
Expecting:
sum += square(i)
30457
return sum
**********************************************************************
Failures reported
File "C:\_Eclipse\Advanced Python\src\Unit Test Framework
Failed example:
sumsquares( 4 )
Expected:
30457
Got:
30
Copyright ©2000-15 CRS Enterprises Ltd
217
Exceptions
Exceptions can be catered for in the tests
use ... to suppress unnecessary information
def square(x):
"""
This function squares a number
>>> square(-3)
9
>>> square(16)
Traceback (most recent call last):
...
ValueError: input too large
"""
if x > 10:
raise ValueError('input too large')
else:
return x*x
Copyright ©2000-15 CRS Enterprises Ltd
218
PyTest
PyTest runs as a separate program
inspects code for methods beginning with test_ and runs them
If you need to install PyTest, use:
easy_install -U py
To run tests, go to a command prompt and type:
py.test myExample.py
Copyright ©2000-15 CRS Enterprises Ltd
219
Simple Example
Add tests as methods in your modules
use assert statements
def square(x):
return x * x
def test_squares():
assert square(6) == 36 # succeeds
assert square(10) == 99 # fails
Copyright ©2000-15 CRS Enterprises Ltd
220
Multiple Tests
Use iterators to create multiple tests
Use setup_class()
class TestSquare:
for initial test conditions
def setup_class(self):
self.m = maths.Maths()
def is_square(self, n, expected):
self.m.setX(n)
assert self.m.square() == expected
def test_squares(self):
# set up 5 tests (the last one fails!)
inputs = ( 4, 7, 10, 20, 25)
outputs = (16, 49, 100, 400, -1)
for i, o in zip(inputs, outputs):
yield self.is_square, i, o
Copyright ©2000-15 CRS Enterprises Ltd
221
Nose
Nose is not a test methodology
simplifies calling other tests
Nose will search out and run tests defined as
functions
methods of a class
unit tests
python script
command line
import nose
nose.run()
nosetests -v
Copyright ©2000-15 CRS Enterprises Ltd
222
15
Inheritance
Inheritance
new style classes and object
subtyping
Abstract Classes
interfaces
OO contracts
Polymorphism
the good
the bad (duck typing)
Copyright ©2000-15 CRS Enterprises Ltd
226
Subclassing
A class can inherit methods and state from another class
the new class is called the subclass or derived class
the original class is called the superclass or base class
The subclass
is a superset of the superclass
inherits all the methods and state of the superclass
can add extra methods and state of its own
cannot remove and methods and state of the superclass
methods in the superclass can be overridden by the
subclass
the subclass is defined in the terms of the differences from
the superclass
Copyright ©2000-15 CRS Enterprises Ltd
227
The Object Class
object
All classes have a single root
object
only for new style classes
Person
Object
defines methods inherited by all Python
classes
Employee
SalesPerson
Copyright ©2000-15 CRS Enterprises Ltd
228
Constructors
Subclass makes explicit call to CTORs in superclass
class Person(object):
def __init__(self, name):
self.name = name
class Employee(Person):
def __init__(self, name, id):
Person.__init__(self, name)
self.id = id
class SalesPerson(Employee):
def __init__(self, name, id, region):
Employee.__init__(self, name, id)
self.region = region
Employee IS A Person
SalesPerson IS A Employee
SalesPerson IS A Person
p = Person("Anne")
e = Employee("Beth", 7468)
s = SalesPerson("Carol", 4712, "NE")
Copyright ©2000-15 CRS Enterprises Ltd
229
The Substitution Rule
Employee class
extends the Person class
Person
SalesPerson class
extends the Employee class
Employee substitutes for
Person object
SalesPerson substitutes for
Employee object
Person object
Copyright ©2000-15 CRS Enterprises Ltd
Superclass
extends
Employee
Subclass
extends
SalesPerson
230
Overriding Superclass Methods
Subclass can override any method of the superclass
superclass method hidden from client code
e = Employee("Beth", 7468)
e.display()
class Employee(Person):
def display(self):
Person.display(self)
# alternatively search super classes for display method
# super(Employee, self).display() # only for new style classes
print "id=", self.id,
Copyright ©2000-15 CRS Enterprises Ltd
231
Concrete and Abstract Classes
Concrete class
all methods are fully implemented
may have instance variables
objects can be instantiated
Abstract class
some methods are not implemented
some methods are implemented
may have instance variables
objects cannot be instantiated
Hoisting
abstract classes used to hold "common" code from subclasses
A concrete subclass
must implement all abstract methods of its superclass
Copyright ©2000-15 CRS Enterprises Ltd
232
Abstract Classes ...
from abc import ABCMeta, abstractmethod, abstractproperty
class Shape:
__metaclass__ = ABCMeta
@abstractmethod
def display(self): pass
@abstractproperty
def name(self): pass
Copyright ©2000-15 CRS Enterprises Ltd
233
... Abstract Classes
class Circle(Shape):
def __init__(self, name):
self.theName = name
def display(self):
print "Circle", self.name
@property
def name(self):
return self.theName
non abstract
methods and
instance
variables
automatically
available
must implement
all abstract
methods
c = Circle("my-circle")
print c.name
c.display()
Copyright ©2000-15 CRS Enterprises Ltd
234
Interfaces
An interface
similar to an abstract class, but ...
all methods are abstract
no instance variables allowed
final class variables permitted (constants)
Can't be instantiated
A concrete class can
implemented multiple interfaces
multiple inheritance model
extend only one class
multiple inheritance model
Copyright ©2000-15 CRS Enterprises Ltd
235
Interfaces as Contracts
Interfaces for the Class Implementer
the interface defines all the methods and signatures for which the class
designer must provide implementations
Interfaces for the Class User
the class user can write code assuming that implementations will be provided
for all the methods in the interface
The Contract
each interface defines a contract between the class user and class
implementer
separates interface from implementation
essential for good object oriented design
Copyright ©2000-15 CRS Enterprises Ltd
236
Polymorphic Methods
An (optional) abstract method
defined in interfaces and abstract classes
not appropriate for the superclass to provide an implementation
implemented by a subclasses
Implementations
each implementing subclass can provide its own implementation
there can be 'many forms' of these implementations
each implementation is called a Polymorphic method
Copyright ©2000-15 CRS Enterprises Ltd
237
Polymorphism
Polymorphic collections
include objects of any subclass
can call any of the methods of the superclass
class Person(object):
def Eat(self): pass
def Drink(self): pass
def Sleep(self): pass
class Employee(Person):
def Eat(self): pass
def Drink(self): pass
def Sleep(self): pass
def NightOut(p):
p.Drink()
p.Drink()
p.Eat()
p.Drink()
p.Sleep()
class SalesPerson(Employee):
def Eat(self): pass
def Drink(self): pass
def Sleep(self): pass
Copyright ©2000-15 CRS Enterprises Ltd
238
Duck Typing
Python allows polymorphic methods from classes not in
the hierarchy
not really sensible
drawback of Python's weak (duck) typing
class Person(object):
def Eat(self): pass
def Drink(self): pass
def Sleep(self): pass
def NightOut(p):
p.Drink()
p.Drink()
p.Eat()
p.Drink()
p.Sleep()
class Employee(Person): ...
class SalesPerson(Employee): ...
p = Person()
e = Employee()
s = SalesPerson()
class Horse:
def Eat(self): pass
def Drink(self): pass
def Sleep(self): pass
NightOut(p)
NightOut(e)
NightOut(s)
h = Horse()
NightOut(h)
Copyright ©2000-15 CRS Enterprises Ltd
239
16
Threading
Copyright ©2000-15 CRS Enterprises Ltd
242
A Thread
Multiple executions of the same process
‘parallelism' within a process
A thread has its own:
Stack
Each thread shares
Heap
Static data
Process resources (open files etc)
Process
Thread A
Thread B
Thread C
Thread scheduling performed by kernel
co-operative
pre-emptive
Copyright ©2000-15 CRS Enterprises Ltd
243
Why Use Multithreading?
Most applications need to perform parallel tasks
updating complex graphical information
waiting for user input
writing data to or reading data from sockets
performing callback operations
Single-threaded programs
each task is sequential
tasks can't be run together
inefficient
Multithreaded programs
each task is run in parallel
harder to program
efficient
Copyright ©2000-15 CRS Enterprises Ltd
244
Thread Objects and Threads
A thread object represents a thread
A thread is not the same as the thread object
The thread object is a helper object
has several methods to facilitate thread management
most methods of the thread object are invoked by the calling thread
often, only the run() method is executed by the new thread
start
run
Thread Object
calling thread
Copyright ©2000-15 CRS Enterprises Ltd
new thread
245
Python Threads
Python has no built in support for threads
must use Thread module
based on the Java thread model
threads have no priorities
Two strategies
new thread executes a callback managed by the Thread class
the callback can be a function or a class with __call__()
derive from Thread class and define a run() method for the thread to
execute
Newly created thread
runs on the callback to completion
can't be suspended
Copyright ©2000-15 CRS Enterprises Ltd
246
01 creating threads
use callback managed by Thread class
can pass args to the new thread
from threading import Thread
def myfunc(name):
for i in range (1, 50):
sys.stdout.write(name)
time.sleep(random.random() * 0.1)
# define a callback function - to be called via start()
thread1 = Thread(target=myfunc, args=("1",))
thread2 = Thread(target=myfunc, args=("2",))
thread1.start()
thread2.start()
# start the threads
thread1.join() # wait for threads to complete
thread2.join()
Copyright ©2000-15 CRS Enterprises Ltd
247
02 using runnable
Inherit from Thread class and define run() method
acts as the callback function
m1 = MyClass("1")
m2 = MyClass("2")
from threading import Thread
# create a runnable class
class MyClass(Thread):
def __init__(self, name):
Thread.__init__(self)
self.name = name
# start calls run()
m1.start()
m2.start()
# wait for threads to complete
m1.join()
m2.join()
def run(self):
for i in range (1, 50):
sys.stdout.write(self.name)
time.sleep(random.random() * 0.1)
...
Copyright ©2000-15 CRS Enterprises Ltd
248
03 callable classes as callbacks
define __call__() as callback
from threading import Thread
# create a callable class
class MyClass:
def __call__(self, name):
for i in range (1, 50):
sys.stdout.write(name)
time.sleep(random.random() * 0.1)
...
m1 = MyClass()
m2 = MyClass()
# define a callback class
# __call__() to be called via start()
t1 = Thread(target = m1, args = ("1",))
t2 = Thread(target = m2, args = ("2",))
t1.start()
t2.start()
t1.join()
t2.join()
Copyright ©2000-15 CRS Enterprises Ltd
249
04 comparing techniques
Callable classes are the most flexible
you can pass parameters to the new thread
by position, as a list, or as a dictionary
Callback function the simplest
t1 = Thread(group = None,
target = m1,
name = None,
args = (),
kwargs = {'name':'1'})
t2 = Thread(target = m2, args = ("2",))
t3 = Thread(None, threadFunction, None, ("3",))
Copyright ©2000-15 CRS Enterprises Ltd
250
05 locking
locks are required to synchronize threads
always release locks in a finally block
just in case an exception occurs
with construct does this automatically
from threading import Lock
lock = Lock() # create lock
from threading import Lock
# locks should be released in finally block
lock.acquire()
try:
for i in range (1, 50):
sys.stdout.write(name)
time.sleep(random.random() * 0.1)
finally:
lock.release()
lock = Lock() # create lock
Copyright ©2000-15 CRS Enterprises Ltd
with lock:
for i in range (1, 50):
sys.stdout.write(name)
time.sleep(random.random() * 0.1)
251
06 sharing data
shared resources need locking
count1 is incremented outside a lock - unreliable
count2 is incremented inside a lock - reliable
class MyClass:
def __call__(self, name):
global lock, count1, count2
for i in xrange(0, 20*1000*100):
count1 += 1
lock.acquire()
count2 += 1
lock.release()
Copyright ©2000-15 CRS Enterprises Ltd
252
07 events
Use event objects as signals
thread can wait() on signal
another thread can set() the signal
class MyClass:
def __call__(self, name):
global event
print(name + " waiting for event");
event.wait()
print("\t" + name + " proceeding after event");
event = Event() # create event object
m1 = MyClass()
t1 = Thread(target = m1, args = ("1",))
t1.start()
time.sleep(15)
event.set()
Copyright ©2000-15 CRS Enterprises Ltd
253
08 semaphores
Semaphore provide a shared lock
define a count (number of threads given simultaneous access to a resource)
count is decrement each time semaphore acquired (>= 0)
from threading import Thread, BoundedSemaphore
class MyClass:
def __call__(self, name):
global semaphore
semaphore.acquire()
time.sleep(5)
semaphore.release()
semaphore = BoundedSemaphore(3)
m1 = MyClass()
t1 = Thread(target = m1, args = ("1",))
t1.start()
t1.join()
Copyright ©2000-15 CRS Enterprises Ltd
254
Using Immutable Objects
An object is considered immutable
if its state cannot change after it is constructed.
Maximum reliance on immutable objects
is widely accepted as a sound strategy for creating simple, reliable multithreaded code.
Since they cannot change state
they cannot be corrupted by thread interference or observed in an inconsistent
state
no need to use locks
Copyright ©2000-15 CRS Enterprises Ltd
255
Global Interpreter Lock ...
Global Interpreter Lock (GIL) is a mutex that ...
prevents multiple native threads from executing byte codes at same time
necessary mainly because CPython's memory management is not thread-safe
other features have grown to depend on the guarantees that the GIL enforces
C/C++ extensions must be GIL-aware
to be thread safe
GIL is controversial and a bottleneck
prevents multithreaded CPython programs from taking full advantage of
multiprocessor systems in certain situations
potentially blocking or long-running operations, such as I/O, image processing,
and NumPy number crunching, happen outside the GIL
Copyright ©2000-15 CRS Enterprises Ltd
256
... Global Interpreter Lock
GIL degrades performance even when it is not a bottleneck
system call overhead is significant, especially on multicore hardware
two threads calling a function may take twice as much time as a single thread calling the
function twice
can cause I/O-bound threads to be scheduled ahead of CPU-bound threads
prevents signals from being delivered
Stackless Python - no GIL
an alternative Python Implementation that does not use the C stack
green threads managed by the interpreter - not the O/S
can run hundreds of thousands of tiny tasks, called "tasklets", in a single thread
completely compatible with Standard Python - just adds some functionality
Jython and IronPython - no GIL
can fully exploit multiprocessor systems
Copyright ©2000-15 CRS Enterprises Ltd
257
17
Extending and Embedding Python
C/C++ interface to Python
using SWIG
Copyright ©2000-15 CRS Enterprises Ltd
260
Calling a C/C++ Module from Python
Calling a C Module from Python:
C Part
1. Write the C routines
2. Define the Module’s Method Table
3. Define the Initialization Function
Python Part
1. Write setup.py
2. Build module:
python setup.py build
Testing
1. import C module
2. call C methods as in regular Python code
Copyright ©2000-15 CRS Enterprises Ltd
261
C Part
#include <Python.h>
// 1. C routines
static PyObject* say_hello(PyObject* self, PyObject* args) {
// implementation
}
// 2. method table
static PyMethodDef HelloMethods[] = {
{"say_hello", say_hello, METH_VARARGS, "Greet somebody."},
{NULL, NULL, 0, NULL}
};
// 3. initialization function
PyMODINIT_FUNC
inithello(void) {
(void) Py_InitModule("hello", HelloMethods);
}
Copyright ©2000-15 CRS Enterprises Ltd
262
Python Part
Write setup.py
setup.py
from distutils.core import setup, Extension
module1 = Extension('hello', sources = ['hellomodule.c'])
setup (name = 'PackageName',
version = '1.0',
description = 'This is a demo package',
ext_modules = [module1])
Build Module
set PATH=C:\Python25;C:\Program Files\pythonxy\mingw\bin
python setup.py build --compiler=mingw32
Copyright ©2000-15 CRS Enterprises Ltd
263
Testing
Make C module available in Python code
Add build directory to PYTHONPATH
import new module
C module now behaves as any other module
import sys
sys.path.append("build/lib.win32-2.5")
import hello
hello.say_hello("World")
Copyright ©2000-15 CRS Enterprises Ltd
264
Using SWIG
SWIG is a generator program
simplifies creating new modules from C/C++ code
use C/C++ data structures
don't need to rest to Py* functions
Steps Required
1. Write C code (no Python classes required)
2. Write interface file
3. Build module
4. Test module in Python
Copyright ©2000-15 CRS Enterprises Ltd
265
Coding
Write C code
hellomodule.c
#include <stdio.h>
void say_hello(const char* name) {
printf("Hello %s!\n", name);
}
Write interface file
hello.i
%module hello
extern void say_hello(const char* name);
Copyright ©2000-15 CRS Enterprises Ltd
266
Build and Test
Build
set PATH=C:\Python25;C:\Program Files\pythonxy\swig;
C:\Program Files\pythonxy\mingw\bin
swig -python hello.i
gcc -fpic -c hellomodule.c hello_wrap.c -IC:\Python25\include
gcc -shared -LC:\Python25\libs hellomodule.o
hello_wrap.o -lpython25 -o _hello.pyd
Test
import hello
hello.say_hello("World")
Copyright ©2000-15 CRS Enterprises Ltd
267
18
Meta Classes
Copyright ©2000-15 CRS Enterprises Ltd
270
‘type’ Metaclass
type metaclass can create ‘normal’ classes
def __init__(self, x):
self.x = x
def printX(self):
print self.x
dictionary of the attributes of the class
type(name, bases, dict)
className = 'TestB'
tuple of the base classes of the class
bases = (object,)
dictionary = {
name of the class
'__init__': __init__,
'printX' : printX
}
TestB = type(className, bases, dictionary)
Copyright ©2000-15 CRS Enterprises Ltd
271
Class Factories ...
Choose your own metaclass to create your classes
class MyClassFactory(type):
y = 100
def __init__(cls, name, bases, dct):
print "Creating class %s using CustomMetaclass" % name
print "with dictionary: "
displayUserDefinedPartsOfDictionary(dct)
bases = (object,)
super(MyClassFactory, cls).__init__(name, bases, dct)
def __new__(cls, name, bases, dct):
print "Allocating memory for class", name
dct['y'] = MyClassFactory.y
# attribute created
dct['f'] = lambda self: self.x + self.y; # method created
# use type to allocate
return type.__new__(cls, name, bases, dct)
Copyright ©2000-15 CRS Enterprises Ltd
272
... Class Factories
__metaclass__ attribute selects the metaclass
subclasses inherit metaclass attributes and methods
class BaseClass(object):
__metaclass__ = MyClassFactory
x = 44
# this class will inherit attributes and methods
# defined by the meta-class
# this uses the class factory of its superclass
class SubClass(BaseClass):
def h(self):
print self.x
Copyright ©2000-15 CRS Enterprises Ltd
273
Object Attribute Resolution Order
Object lookup ignores meta classes
object
ClassA
ClassB
b (object)
Copyright ©2000-15 CRS Enterprises Ltd
274
Class Attribute Resolution Order
Class lookup includes meta classes
object
type
ClassA
Meta=ClassA
ClassB
MetaClassB
Copyright ©2000-15 CRS Enterprises Ltd
275
19
Oracle Database
Copyright ©2000-15 CRS Enterprises Ltd
278
Connect
Connect to database and retrieve a cursor object
import cx_Oracle
def ConnectToOracle():
connection = cx_Oracle.connect(
user="java",
password="oracle",
dsn="XE")
return connection
def GetCursor(connection):
cursor = cx_Oracle.Cursor(connection)
return cursor
Copyright ©2000-15 CRS Enterprises Ltd
279
Cursors
Used to parse and execute SQL and fetch results
usually only a single cursor is required
multiple cursors can be defined
Parse
cx_Oracle.Cursor.parse(statement)
Execute
cx_Oracle.Cursor.execute(statement, parameters)
Fetch
cx_Oracle.Cursor.fetchall( )
cx_Oracle.Cursor.fetchmany(rows)
cx_Oracle.Cursor.fetchone( )
Copyright ©2000-15 CRS Enterprises Ltd
280
Data Types
ORACLE
cx_Oracle
Python
VARCHAR2
NVARCHAR2
LONG
STRING
str
CHAR
FIXED_CHAR
str
NUMBER
NUMBER
int
FLOAT
NUMBER
float
DATE
DATETIME
datetime.datetime
TIMESTAMP
TIMESTAMP
datetime.datetime
CLOB
CLOB
cx_Oracle.LOB
BLOB
BLOB
cx_Oracle.LOB
Copyright ©2000-15 CRS Enterprises Ltd
281
Select Statements
Use cursor to execute a SELECT query on database
def PrintSalaryTable(connection):
cursor = GetCursor(connection)
SQL = "SELECT * FROM SALARY_TABLE"
cursor.execute(SQL)
for name, salary in cursor.fetchall():
print name, salary
connection = ConnectToOracle()
PrintSalaryTable(connection)
Copyright ©2000-15 CRS Enterprises Ltd
282
Insert Statements
Use cursor to execute INSERT on database
def InsertRow(connection, name, salary):
try:
SQL = "INSERT INTO SALARY_TABLE (Name, Salary) " + \
"VALUES ('" + name +"'" + \
"
,'" + str(salary) +"')"
cursor = GetCursor(connection)
cursor.execute(SQL)
cursor.close()
connection.commit()
except Exception, reason:
connection.rollback()
print reason
connection = ConnectToOracle()
InsertRow(connection, "George", 15000.0)
Copyright ©2000-15 CRS Enterprises Ltd
283
Stored Procedures
Names used in Stored Procedure must be used in Python
see next slide
create or replace PROCEDURE Increase_Salary
(theName IN varchar2,
Increase IN float,
parameter names
NewSalary IN OUT float)
AS
BEGIN
UPDATE SalaryTable
SET SALARY = SALARY + Increase
WHERE NAME = theName;
SELECT SALARY INTO NewSalary
FROM SalaryTable
WHERE NAME = theName;
END;
Copyright ©2000-15 CRS Enterprises Ltd
284
Stored Procedures
Use cursor to execute StroredProcedure
def UseStoredProcedure(connection, name, increase):
try:
cursor = GetCursor(connection)
newSalary = cursor.var(cx_Oracle.NUMBER)
result = cursor.callproc('INCREASE_SALARY', \
[name, increase, newSalary])
print "result=", result
print "newSalary=", newSalary
print "newSalary=", newSalary.getvalue()
connection.commit()
except Exception, reason:
connection.rollback()
print reason
connection = ConnectToOracle()
UseStoredProcedure(connection, name="Charles", increase=10000.0)
Copyright ©2000-15 CRS Enterprises Ltd
285
Bind Variables ...
Bind Python names to StoredProcedure names
def method(cursor, name, salary):
named_params = { 'aName':name, 'aSalary':salary }
SQL = 'UPDATE SALARY_TABLE SET salary=:aSalary WHERE name=:aName'
cursor.execute(SQL, named_params)
print cursor.bindnames()
def UseBindVariables(connection, name, salary):
try:
cursor = GetCursor(connection)
method(cursor, name, salary)
connection.commit()
except Exception, reason:
connection.rollback()
print reason
connection = ConnectToOracle()
UseBindVariables(connection, "Charles", 10005.0)
Copyright ©2000-15 CRS Enterprises Ltd
286
20
Operator Overloading
Intrinsic Attributes
for classes and objects
Dynamic Statements
exec and eval
Operator Overloading
numeric types
list types
Copyright ©2000-15 CRS Enterprises Ltd
290
Operator Overloading Numeric Types
class Time:
def __init__(self, hrs, min):
self.hrs = hrs
self.min = min
def __add__(self, other):
hrs = self.hrs + other.hrs
min = self.min + other.min
if min >= 60:
hrs = hrs + 1
min = min - 60
return Time(hrs, min)
t3 = t1 + t2
def Display(self):
print "Time is", self.hrs, self.min
Copyright ©2000-15 CRS Enterprises Ltd
291
Operator Overloading Numeric Types
class Time:
def __init__(self, hrs, min):
self.hrs = hrs
self.min = min
def __iadd__(self, other):
self.hrs = self.hrs + other.hrs
self.min = self.min + other.min
if self.min >= 60:
self.hrs = self.hrs + 1
self.min = self.min - 60
return self
t1 += t3
def Display(self):
print "Time is", self.hrs, self.min
Copyright ©2000-15 CRS Enterprises Ltd
292
Operator Overloading Numeric Types
class Time:
def __init__(self, hrs, min):
self.hrs = hrs
self.min = min
def __add__(self, other):
hrs = self.hrs + other.hrs
min = self.min + other.min
if min >= 60:
hrs = hrs + 1
min = min - 60
return Time(hrs, min)
t1 = Time(5,30)
t2 = Time(3,30)
t3 = t1 + t2
t3.Display()
def __iadd__(self, other):
t1 += t3
self.hrs = self.hrs + other.hrs
t1.Display()
self.min = self.min + other.min
if self.min >= 60:
self.hrs = self.hrs + 1
self.min = self.min - 60
return self
Time is 9 0
Time is 14 30
def Display(self):
print "Time is", self.hrs, self.min
Copyright ©2000-15 CRS Enterprises Ltd
293
Operator Overloading other Types
Emulate other types
Built-In Objects
Sequences
Slices
Emulate Maps
Special Methods to be Written
e.g. for Maps
__len__(self)
__getitem__(self,key)
__setitem__(self,key,value)
__delitem__(self,key)
Copyright ©2000-15 CRS Enterprises Ltd
294
Operator Overloading List Types
class MyList:
def __init__(self, p1, p2):
self.theList = [ ]
self.theList.append(p1)
self.theList.append(p2)
t1 = MyList(1,2)
t2 = MyList(3,4)
t3 = t1 + t2
def __add__(self, other):
result = [ ]
for item in self.theList:
result.append(item)
for item in other.theList:
result.append(item)
return result
t1: [1,2]
t2: [3,4]
t3: [1,2,3,4]
Copyright ©2000-15 CRS Enterprises Ltd
295
Overloading [ ] using __getitem__
class mydata:
def __init__(self):
self.data = "ABCDEFG"
def __getitem__(self, i):
return self.data[i]
B
ABCDEFG
object = mydata()
theChar = object[1]
print theChar;
for item in object:
print item,
Copyright ©2000-15 CRS Enterprises Ltd
296
21
Web Application Frameworks
Django
Pyramid
Flask
Copyright ©2000-15 CRS Enterprises Ltd
300
Django
Good points
the most famous Python web framework
easy to get started with
documentation focused on beginners
comes with a lot of integrated functionality
can develop large scale applications
supports decorators in views
has its own templates
Jinja2 developed as a superset of Django templates
Not so good points
has a fixed way of doing things
model breaks down somewhat if you want to use other python packages not
integrated with Django
e.g. possible to use SQLAlchemy instead of the Django ORM, but
it’s certainly not straightforward
Copyright ©2000-15 CRS Enterprises Ltd
301
Pyramid
Good points
can develop large scale applications
flexible core set of modules
if you need anything else, like an ORM you can integrate it
yourself
ideal for apps which go beyond CRUD
supports decorators in views
no restrictions on template engines
Not so good points
not really a framework for the beginner
there is a learning curve
Copyright ©2000-15 CRS Enterprises Ltd
302
Flask
Good points
simplicity
can develop applications quickly
supports decorators in views
uses Jinja2 as template engine
Not so good points
not designed for very large and complex websites
can’t use other template engines
chameleon, mako
Copyright ©2000-15 CRS Enterprises Ltd
303
MVT
Model/View/Template
Variant of MVC
Model used for specifying database schema
or other method of gathering data
often contains the business logic
View used for
managing the presentation
Django uses this for business logic
Template assists with the presentation
use one of the many template engines
Copyright ©2000-15 CRS Enterprises Ltd
304
Popular Web Template Engines
Template engines offer high
reusability of markup code
Includes:
to incorporate snippets of content common to site
e.g. footer, scripts, styles, etc.
Master Layout:
skeleton of the site content with placeholders
e.g. sidebar, horizontal menu, content, etc.
Widgets:
snippets of reusable markup
e.g. list item, button, etc.
Copyright ©2000-15 CRS Enterprises Ltd
chameleon
cheetah
django
jinja2
mako
durusworks
spitfire
tenjin
tornado
web2py
wheezy.template
305
Routing in Pyramid
Decorators used to configure routing
name specifies what to request on the browser
default route has no name
from pyramid.view import view_config
@view_config(renderer="home.pt")
def home(request):
return {}
@view_config(renderer="pages.pt", name="page1")
def view1(request):
return { "firstName": "Chris",
"lastName": "Seddon“ }
@view_config(renderer="pages.pt", name="page2")
def view2(request):
return { "firstName": "John",
"lastName": "Smith“ }
Copyright ©2000-15 CRS Enterprises Ltd
306
Templates
Template expanded to HTML
<h1>Football Teams and Grounds: Grounds</h1>
<div>
<li>
Amsterdam ArenA - Amsterdam
</li>
<div metal:use-macro="layout">
<li>
<div metal:fill-slot="content">
Wembley - London
</li>
<li tal:repeat="g theGrounds">
<li>
${g.ground} - ${g.city}
Nou Camp - Barcelona
</li>
</li>
</div>
<li>
Hampden Park - Glasgow
</div>
</li>
loop
</div>
<ul>
<li><a href="/">Home</a></li>
<li><a href="/teams">Teams</a></li>
<li><a href="/grounds">Grounds</a></li>
<li><a href="/both">Teams and Grounds</a></li>
</ul>
Copyright ©2000-15 CRS Enterprises Ltd
307
22
JSON and XML
Copyright ©2000-15 CRS Enterprises Ltd
310
JSON Parsing
object
unordered set of name/value pairs
{ color: "red", value: "#f00" }
array
ordered collection of values
[ 100, 500, 300, 200, 400 ]
Copyright ©2000-15 CRS Enterprises Ltd
311
Working with JSON
Object with embedded arrays and objects
{ "products":
[
{ "company":"Company1", "name":"Prod1", "prices":
[
{"price":123.2},{"price":123.6}
],
"serialNumber":"12345"
},
{ "company":"Company2", "name":"Prod2", "prices":
[
{"price":234.2},{"price":234.6}
],
"serialNumber":"67890"
}
]
}
Copyright ©2000-15 CRS Enterprises Ltd
312
JSON Encoder
For user defined types
include class and module names
class MyEncoder(json.JSONEncoder):
def default(self, obj):
# Convert objects to a dictionary of their representation
d = {}
d['__class__'] = obj.__class__.__name__
d['__module__'] = obj.__module__
d.update(obj.__dict__)
return d
Copyright ©2000-15 CRS Enterprises Ltd
313
JSON Decoder
For user defined types
with class and module names included
class MyDecoder(json.JSONDecoder):
def __init__(self):
json.JSONDecoder.__init__(self, object_hook=self.dict_to_object)
def dict_to_object(self, d):
if '__class__' in d:
class_name = d.pop('__class__')
module_name = d.pop('__module__')
module = __import__(module_name)
class_ = getattr(module, class_name)
args = dict( (key.encode('ascii'), value) for key, value in d.items())
instance = class_(**args)
return instance
Copyright ©2000-15 CRS Enterprises Ltd
314
XML Parsing
XML is more verbose than JSON
but more common in industry
<?xml version="1.0"?>
<Book>
<Title>Sherlock Holmes</Title>
<Author>Arthur Conan Doyle</Author>
</Book>
ElementTree module will parse XML, but
has no namespace support
Copyright ©2000-15 CRS Enterprises Ltd
315
XML Namespaces
Prefix is bound to a URL
URL doesn't have to exist
you don't have to use a URL
Prefix acts as a variable
URL is its value
the value can change
<?xml version="1.0"?>
<Book xmlns:lib="http://www.library.com">
<lib:Title>Sherlock Holmes</lib:Title>
<lib:Author>Arthur Conan Doyle</lib:Author>
</Book>
lxml is namespace aware
built on top of ElementTree
Copyright ©2000-15 CRS Enterprises Ltd
316
Using the Default Namespace
defines the namespace for unqualified tags
<?xml version="1.0"?>
<Book xmlns="http://www.library.com">
<Title>Sherlock Holmes</Title>
<Author>Arthur Conan Doyle</Author>
</Book>
Copyright ©2000-15 CRS Enterprises Ltd
317
ElementTree
DOM (in memory) parser
namespace support
Example shows how to find author's first and last names
from xml.etree.ElementTree import *
from xml.etree.ElementTree import _namespace_map
filename = "../xml/book.xml"
tree = parse(filename)
root = tree.getroot()
author = tree.find("{http://www.demo.com/book}author");
print "first=", author.get("first")
print "last=", author.get("last")
Copyright ©2000-15 CRS Enterprises Ltd
318
lxml
Extension to ElementTree
DOM and SAX parsers
XPath support
XSLT support
Example shows DOM parsing
from lxml import etree
doc = etree.parse('xml/book.xml')
print(etree.tostring(doc, pretty_print=True))
Copyright ©2000-15 CRS Enterprises Ltd
319
HTML Parsing
from HTMLParser import HTMLParser
# create a subclass and override the handler methods
class MyHTMLParser(HTMLParser):
def handle_starttag(self, tag, attrs):
print "Encountered a start tag:", tag
def handle_endtag(self, tag):
print "Encountered an end tag :", tag
def handle_data(self, data):
print "Encountered some data :", data
# instantiate the parser and fed it some HTML
parser = MyHTMLParser()
parser.feed('<html><head><title>Test</title></head>'
'<body><h1>Parse me!</h1></body></html>')
Copyright ©2000-15 CRS Enterprises Ltd
320
Beautiful Soup
from BeautifulSoup import BeautifulSoup
# use mechanize to scrape spans from BBC news page
br = mechanize.Browser()
br.open("http://www.bbc.co.uk/news")
response = br.follow_link(text_regex=r"UK")
data = br.response().read()
# pick out anchors that are tagged with the story class
soup = BeautifulSoup(data)
tags = soup.findAll("a", "story")
newSoup = BeautifulSoup()
for tag in tags:
tag.insert(0, "<br>")
newSoup.insert(0, tag)
data = str(newSoup)
Copyright ©2000-15 CRS Enterprises Ltd
321
23
Miscellaneous Topics
PDF files
Excel
Zip Files
GUI Frameworks
Plotting
Image Processing
Copyright ©2000-15 CRS Enterprises Ltd
324
Working with PDF Files
PyPDF2
replaces PyPDF
open-source pure-Python libraries which concentrate on manipulation of
existing PDF instances
ReportLab
industrial-strength library largely focused on precise creation of PDF
open-source, no-charge base, but ...
"ReportLab PLUS" extension is not free
PDFMiner
tool for extracting information from PDF documents
focuses entirely on getting and analyzing text data
Copyright ©2000-15 CRS Enterprises Ltd
325
Working with Excel
Use the modules: xlrd and xlrt
works with Linux and Mac
not just Windows
Best place to start when working with Excel files:
http://www.python-excel.org
Excellent set of examples:
http://www.simplistix.co.uk/presentations/europython2009excel.zip
http://www.simplistix.co.uk/presentations/python-excel.pdf
Copyright ©2000-15 CRS Enterprises Ltd
326
Zip Files
zipfile
provides tools to create, read, write, append, and list a ZIP file
does not currently handle multi-disk ZIP files
handles ZIP files that use the ZIP64 extensions (> 4 GByte)
supports decryption of encrypted files in ZIP archives
cannot create an encrypted file
decryption is extremely slow as it is implemented in native Python
rather than C
import zipfile
zipfile.ZipFile("gp444win32.zip", "r").extractall()
Copyright ©2000-15 CRS Enterprises Ltd
327
GUI Frameworks: Tkinter
Tkinter is Python's de-facto standard GUI package
thin object-oriented layer on top of Tcl/Tk
Tkinter wiki:
from Tkinter import *
http://tkinter.unpythonic.net/wiki/
def main():
root = Tk()
root.title("Hello World")
root.minsize(width=300, height=300)
mainframe = Frame(master=root)
mainframe.pack()
draw(mainframe)
root.mainloop()
def draw(frame):
Button(frame,
text = "Hello World",
command = lambda : exit()).pack()
main()
Copyright ©2000-15 CRS Enterprises Ltd
328
GUI Frameworks: PyQt
Python bindings
for the Qt cross-platform GUI/XML/SQL C++ framework
PyQt provides bindings for Qt 4 and Qt 5
choice of licences: GPL or commercial
Contains over 620 classes that cover:
graphical user interfaces
XML handling
network communication
SQL databases
Web browsing
Copyright ©2000-15 CRS Enterprises Ltd
329
GUI Frameworks: wxPython
Python bindings
implemented as a Python extension module (native code)
that wraps the popular wxWidgets cross platform GUI library written in C++
wxPython is Open Source
free to use
source code is available for anyone to look at and modify
wxPython is a cross-platform toolkit
same program will run on multiple platforms without modification
supported platforms are 32-bit Microsoft Windows, most Unix or unix-like
systems, and Macintosh OS X.
Copyright ©2000-15 CRS Enterprises Ltd
330
Accessing the Web
urllib/urllib2
provides a high-level interface for fetching data across the World Wide Web
urlopen() function
is similar to the built-in file open() function, but accepts Universal
Resource Locators (URLs)
can only open URLs for reading
no seek operations
import urllib
Requests
f = urllib.urlopen("http://www.ibm.com")
print f.read()
alternative to urllib
claims to be much easier to use
quote: ‘Python’s standard urllib2 module provides most of the HTTP
capabilities you need, but the API is thoroughly broken. It was built for a
different time — and a different web. It requires an enormous amount of
work (even method overrides) to perform the simplest of tasks.’
Copyright ©2000-15 CRS Enterprises Ltd
331
More Modules for the Web
webbrowser
high-level interface to display Web-based documents
import webbrowser
url = 'http://www.ibm.com'
webbrowser.open_new_tab(url)
mechanize
library is used for automating interaction with websites.
automatically stores and sends cookies
follows redirects
follows links and submits forms
keeps track of the sites that you have visited as a history
import mechanize
br = mechanize.Browser()
br.open("http://www.bbc.co.uk/news")
response = br.follow_link(text_regex=r"UK")
data = br.response().read()
Copyright ©2000-15 CRS Enterprises Ltd
332
Selenium
Used to write tests for web applications
simple API to write functional/acceptance tests using Selenium WebDriver
Works with Firefox, IE, Chrome ...
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("http://www.python.org")
assert "Python" in driver.title
elem = driver.find_element_by_name("q")
elem.send_keys("selenium")
elem.send_keys(Keys.RETURN)
assert "Google" in driver.title
driver.close()
Copyright ©2000-15 CRS Enterprises Ltd
333
Plotting
matplotlib
plotting library which producing publication quality figures in a variety of
hardcopy formats and interactive environments across platforms
see gallery: http://matplotlib.org/gallery.html
gnuplot
older library with Python bindings
very extensive capabilities
import matplotlib.pyplot as plt
redCircles = "ro"
plt.plot([1,2,3,4], [1,4,9,16], redCircles)
plt.ylabel("squares")
plt.show()
Copyright ©2000-15 CRS Enterprises Ltd
334
Python Imaging Library
PIL
adds image processing capabilities to your Python interpreter
library supports many file formats and provides powerful image processing and
graphics capabilities
http://www.pythonware.com/products/pil/
Python Imaging Library Handbook
very useful examples
http://effbot.org/imagingbook/pil-index.htm
Copyright ©2000-15 CRS Enterprises Ltd
335