Chapter 6 Application Layer
Download
Report
Transcript Chapter 6 Application Layer
Computer Networks
An Open Source Approach
Chapter 6: Internet Services
Chapter 6: Application Layer
1
Content
6.1 Issues
6.2 Domain Name System
6.3 E-Mail
6.4 World Wide Web
6.5 File Transfer Protocol
6.6 Simple Network Management Protocol
6.7 Voice over IP (VoIP)
6.8 Streaming
6.9 Peer-to-Peer Applications (P2P)
6.10 Summary
Chapter 6: Application Layer
2
6.1 Issues
How ports work?
How servers start?
Classification of servers
Characteristics of application layer protocols
Chapter 6: Application Layer
3
How Ports Work?
Well known ports: 0~1023
Registered ports: 1024~49151
Dynamic and/or private ports: 49152~65535
Client 1 Machine
Server Machine
User Agent
Outbound
Port
FTP Client
2880
Client 2 Machine
Listening
Ports
Server Daemons
21
FTP Daemon
23
Telnet Daemon
User Agent
Outbound
Port
25
Mail Daemon
Web Browser
8752
80
HTTP Daemon
Chapter 6: Application Layer
4
How Servers Start?
Is a server better to run as standalone or via (x)inetd?
Standalone:
requires explicit restarting for configuration changes to take
effect
crashed standalone server will probably stay unnoticed and
the service will become unavailable
better performance
(x)inetd:
reads configuration files each time a client attempts to
access their information
fresh copies will be spawned on demand
Chapter 6: Application Layer
5
Classification of Servers
Concurrent connectionless server
Concurrent connection-oriented server
Iterative connectionless server
Iterative connection-oriented server
Chapter 6: Application Layer
6
Underlying Transport Protocols
Application
Application layer protocol
Underlying transport protocol
electronic mail
SMTP, POP3, IMAP
TCP
remote terminal access
Telnet
TCP
file transfer
FTP
TCP
web
HTTP
TCP
web caching
ICP
typically UDP
name resolution
DNS
typically UDP
network file system
NFS
typically UDP
network management
SNMP
typically UDP
routing protocol
RIP
typically UDP
Internet telephony
proprietary (e.g. Vocaltec)
typically UDP
streaming multimedia
proprietary (e.g. RealNetworks)
typically UDP
P2P
proprietary (e.g. BitTorrent, eDonkey)
UDP for queries and TCP for data transfer
Chapter 6: Application Layer
7
Iterative Connectionless Server
Server
Clients
socket
socket
bind
sendto
recvfrom
requests
process
recvfrom
sendto
responses
repeat as
needed
repeat
infinitely
close
Chapter 6: Application Layer
8
Concurrent Connection-Oriented Server
Server
socket
bind
Client
listen
socket
repeat
infinitely
connection request
connect
accept
fork
close
(listening)
write
request
close
(accepting)
server process
read
process
write
reply
read
repeat as
needed
repeat as
needed
Close
(accepting)
close
child process
Chapter 6: Application
Layer
9
Characteristics of Application Layer
Protocols
Variable message formats and length
Variable data types
Message formats and length of application layer protocols
vary with different applications and requirements
Messages can be transmitted in textual or non-textual
formats.
E.g., a Web server replies textual Web pages and binary
images
Statefulness
The server retains information of the session with the client
E.g., an FTP server remembers the client’s current working
directory
Chapter 6: Application Layer
10
6.2 Domain Name System
Domain name space and domain delegation
Zones and name servers
Zone data files and resource records
Resolvers and Name resolution
Open source: BIND
Chapter 6: Application Layer
11
Top Level Domains
Domain
com
Description
Commercial organizations, such as Intel (intel.com).
org
Non-profit organizations, such as WWW consortium (w3.org).
gov
Government organizations, reserved for U.S government such as
National Science Foundation (nsf.gov).
edu
Educational organizations, such as UCLA (ucla.edu).
net
Networking organizations, such as Internet Assigned Numbers Authority
which maintains the DNS root servers (gtld-servers.net) .
int
Organizations established by international treaties between governments.
For example, International Telecommunication Union (itu.int).
Mil
Reserved exclusively for the United States Military. For example, Network
Information Center, Department of Defense (nic.mil).
Two-letter
The two-letter country code top level domains (ccTLDs) are based on the
country code ISO 3166-1 two-letter country codes. Examples are tw (Taiwan), uk
(United Kingdom).
arpa
Mostly unused now, except for the in-addr.arpa domain, which is used to
maintain a database for reverse DNS queries.
Others
Such as .biz (business), .name (for individuals), .info (similar with .com).
Chapter 6: Application Layer
12
Domain Delegation
Easier
management
Load distribution
root domain
tw
org
edu
com
com
edu
uk
co
nthu
nctu
nsysu
csie
cis
ee
cis.nctu.edu.tw
Chapter 6: Application Layer
13
Zones and Name Servers
Differences between a zone and a domain
Zone data files in a name server
A zone contains some part of the domain name space
A domain is a subtree in the domain name space
A name server may be authoritative for multiple zones
Contains Resource Records (RR) describing all the hosts
within that zone
Six major types of RR: “SOA”, “NS”, “A”, “PTR”, “CNAME” and
“MX”
Master and slave name servers
Availability and load balance
Data duplication by zone transfer
Request from slave
Notify by master
Chapter 6: Application Layer
14
Master/Slave Name Servers
1. Redundancy for heavy load
2. Availability achieved
slave
df of A
master (A) | slave (B)
zone
transfer
df of A
and B
zone
transfer
clients
master
A large domain
name space
df of B
clients
zone A
zone B
df: zone data file
Chapter 6: Application Layer
15
Resource Records
SOA (Start Of Authority)
NS (Name Server)
Specify a best name server to be authoritative (master) for
a zone
Ex: cis.nctu.edu.tw. 86400 IN SOA cisserv.cis.nctu.edu.tw.
Options for refresh, retry, expire ..etc are supported
Specify name servers for a domain
Ex: cis.nctu.edu.tw. 86400 IN NS cisserv.cis.nctu.edu.tw.
A (Address)
Map names to addresses,
Example (multi-homed):
linux.cis.nctu.edu.tw. 86400 IN A 140.113.168.127
linux.cis.nctu.edu.tw. 86400 IN A 140.113.207.127
Chapter 6: Application Layer
16
Resource Records (cont)
CNAME (Canonical Name)
PTR (Pointer)
Create name-to-name alias
Ex: www.cis.nctu.edu.tw. 86400 IN CNAME cache.cis.nctu.edu.tw.
Point addresses to names (canonical name only)
Ex: 10.23.113.140.in-addr.arpa. 86400 IN PTR laser0.cis.nctu.edu.tw.
MX (Mail Exchanger)
Provide mail-routing information
The one of best (lowest) preference value is chosen
cis.nctu.edu.tw. 86400 IN
MX
0 mail.cis.nctu.edu.tw.
cis.nctu.edu.tw. 86400 IN
MX
10 mail1.cis.nctu.edu.tw.
Chapter 6: Application Layer
17
Name Resolution
Query type
Resolution method
Forward query – name-to-address
Reverse query – address-to-name
Iterative (by referral, commonly adopted)
Recursive (site-by-site recursion)
Caching for speeding up the lookup process
Expire according to TTL
Tradeoff between performance and consistency
Chapter 6: Application Layer
18
Name Resolution (Iterative)
1
query for
“www.dti.gov.uk”
client
(resolver)
local
name
server
www.dti.gov.uk. 5M IN A 164.36.253.20
www.dti.gov.uk. 5M IN A 164.36.164.20
root
name server
2
3
4
“uk”
name server
5
6
”gov.uk”
name server
7
8
“dti.gov.uk”
name server
Chapter 6: Application Layer
other candidate
name servers
19
Message Format
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 bit
ID
Q
R
Opcode
A
A
T
C
R
D
R
A
Reserved
QDCOUNT
Rcode
Header
ANCOUNT
NSCOUNT
ARCOUNT
Question
Answer
Authority
Additional
Chapter 6: Application Layer
20
Execution Flowchart of Named
Install a handler
for signals
Cache initialization
Command line
parsing
chroot and
daemonize
1. Task manager
2. Timer manager
Create managers
3. Socket manager
Initialize server
configurations and data
structures
Wait for signals
For possible restart or
termination signals
Chapter 6: Application Layer
21
Chapter 6: Application Layer
22
Basic Data Structure in Named
viewlist
view2
zone_table
zone_table
…
zone1
Red/Black Tree
for zones
…
view1
zone3
viewN
NULL
zone_table
zone3
zone2
zone4
zone8
RR1
zoneN
RR2
…
Red/Black Tree
of RRs for zone4
RRN
Chapter 6: Application Layer
23
Domain Information Gopher (DIG)
; <<>> DiG 9.2.0 <<>> www.nctu.edu.tw
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26027
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3
;; QUESTION SECTION:
;www.nctu.edu.tw.
IN
A
;; ANSWER SECTION:
www.nctu.edu.tw.
259200 IN
;; AUTHORITY SECTION:
nctu.edu.tw.
259200 IN
nctu.edu.tw.
259200 IN
nctu.edu.tw.
259200 IN
;; ADDITIONAL SECTION:
ns.nctu.edu.tw.
259200 IN
ns2.nctu.edu.tw.
259200 IN
ns3.nctu.edu.tw.
259200 IN
A
140.113.250.5
NS
NS
NS
ns.nctu.edu.tw.
ns2.nctu.edu.tw.
ns3.nctu.edu.tw.
A
A
A
140.113.250.135
140.113.6.2
163.28.64.11
Chapter 6: Application Layer
24
6.3 E-Mail
Introduction
Message Formats
Internet Mail Protocols
Open Source: Qmail
Chapter 6: Application Layer
25
Introduction
What is E-mail?
A method of sending messages from one user to
another via computer networks
Internet mail addressing:
format: [email protected]
e.g. [email protected]
Chapter 6: Application Layer
26
Introduction (cont.)
Components of Internet mail system:
Mail User Agent (MUA)
Mail Transfer Agent (MTA)
Mail Delivery Agent (MDA)
Mail Retrieval Agent (MRA)
Sender's Machine
Sending MUA
Local Mail Server
SMTP
Forwarding MTA
Remote Mail Server
SMTP
POP/IMAP
Receiving MTA
System Call
TCP MDA
Recipient's Machine
Receving MUA
System Call
Local MDA
Write
Mailbox
Read/Write
MRA
Chapter 6: Application Layer
27
Message Formats
Internet Message Format (RFC 822)
Multipurpose Internet Mail Extensions (RFC
2045-2049)
Chapter 6: Application Layer
28
RFC 822 – Internet Message Format
A message consists of:
An envelope which contains information needed
to accomplish transport and delivery
Contents which compose the object to be
delivered to the recipient
Chapter 6: Application Layer
29
RFC 822 – Internet Message Format
(cont.)
Common message header fields:
Type
Field
Originator From:
Reply-To:
Receiver To:
Trace
Meaning
The person(s) who wished this message to be sent
Provides a general mechanism for indicating any
mailbox(es) to which responses are to be sent
The primary recipients of the message
Cc:
The secondary recipients of the message
Bcc:
Additional recipients of the message
Received:
A copy of this field is added by each transport service
that relays the message
Return-Path: This field is added by the final transport system that
delivers the message to its recipient
Reference Message-ID: Contains a unique identifier generated by the mail
transport on the originating system
In-Reply-To: Previous correspondence which this message
answers.
Other
Subject:
Provides a summary, or indicate the
message.
nature, of the
Date
Date:
Supplies the date and time the mail was sent
Extension X-anything: It is used to implement additional features that have
not yet made it into an RFC, or never will.
Chapter 6: Application Layer
30
RFC 822 – Internet Message Format
(cont.)
An example of message header:
From: "Dr. Ying-Dar Lin" <[email protected]>
To: [email protected]
Date: Mon, 30 Jul 2001 02:42:34 +0800 (CST)
Subject: paper
Return-Path: <[email protected]>
Received: (from root@localhost) by mail.cis.nctu.edu.tw (8.11.4/8.11.4) id f6TIgct33454 for
[email protected]; Mon, 30 Jul 2001 02:42:38 +0800 (CST) (envelope-from
[email protected])
Received: from cissun53.cis.nctu.edu.tw ([email protected] [140.113.23.53]) by
mail.cis.nctu.edu.tw (8.11.4/8.11.4av) with ESMTP id f6TIgaI33445 for <[email protected]>;
Mon, 30 Jul 2001 02:42:36 +0800 (CST) (envelope-from [email protected])
Received: (from ydlin@localhost) by cissun53.cis.nctu.edu.tw (8.9.3/8.9.3) id CAA16191 for juvenia;
Mon, 30 Jul 2001 02:42:34 +0800 (CST)
Message-Id: <[email protected]>
X-Virus-Scanned: by AMaViS perl
X-UIDL: F5%#!88!"!>RG"!l!G!!
Chapter 6: Application Layer
31
MIME
What is MIME?
MIME is a specification for enhancing the
capabilities of conventional Internet message
format .
MIME enables email messages to have:
textual header information and message bodies in
character sets other than 7-bit ASCII
multiple objects within a single message
binary or application-specific files
multimedia files, such as images, audio, and video
files
Chapter 6: Application Layer
32
MIME (cont.)
MIME message header:
Field
Description
MIME-Version:
Describes the version of the MIME message
format
Content-Type:
Describes the MIME content type and subtype
Content-Transfer-Encoding: Indicates the encoding method for transmission
Content-ID:
Allows a body of information to refer to another
Content-Description:
Possible description for a body of information
Chapter 6: Application Layer
33
MIME (cont.)
The MIME content type set:
Type
Subtype(s)
Important parameters
text
plain, html
charset
multipart
mixed, alternative, parallel,
boundary
digest
message
application
RFC 822, partial,
Id, number, total, access-type,
external-body
expiration, size, permission
octet-stream, postscript, rtf,
type, padding
pdf, msword
image
jpg, gif, tiff, x-xbitmap
none
audio
basic, wav
none
video
mpeg
none
Chapter 6: Application Layer
34
MIME (cont.)
Possible values for the Content-Transfer-Encoding:
field:
Quoted-Printable
Base64
7bit
8bit
Binary
X-Encoding
Chapter 6: Application Layer
35
MIME (cont.)
An example of MIME message:
From: 'Ching-Ming Tien' <[email protected]>
To: [email protected]
Subject: Cover
MIME-Version: 1.0
Content-Type: image/jpg;
name=cover.jpg'
Content-Transfer-Encoding: base64
Content-Description: The front cover of the book
<.....base64 encoded jpg image of cover...>
Chapter 6: Application Layer
36
Internet Mail Protocols
Simple Mail Transfer Protocol (RFC 2821)
Post Office Protocol (RFC 1939)
Internet Message Access Protocol (RFC
2060)
Chapter 6: Application Layer
37
Simple Mail Transfer Protocol
What is SMTP?
A standard host-to-host mail transport protocol and
traditionally operates over TCP on port 25
Chapter 6: Application Layer
38
Simple Mail Transfer Protocol (cont.)
Important SMTP commands:
Command
Description
HELO
Greet the receiver with the name
MAIL FROM:
Indicates the sender, could be spoofed too
RCPT TO:
Indicates the recipient
DATA
Indicates the mail data, terminated by a " . " in a single line
RSET
Reset the session
QUIT
Close the session
SMTP replies:
Response
Description
2xx
Command accepted and processed.
3xx
General flow control.
4xx
Critical system or transfer failure.
5xx
Errors with the SMTP command.
Chapter 6: Application Layer
39
Simple Mail Transfer Protocol (cont.)
An SMTP transaction scenario:
Chapter 6: Application Layer
40
Post Office Protocol
What is POP?
A protocol used to retrieve e-mail from a mail server.
POP3 Session States:
Authorization:
1.
Must log in with password before entering transaction state.
Transaction:
2.
Client can request actions of server, get mail for example.
Update:
3.
Updates mail box to reflect actions taken in transaction
state.
Chapter 6: Application Layer
41
Post Office Protocol (cont.)
Minimal POP3 commands:
Command
Description
Session state
USER name
Identifies the user to the server
AUTHORIZATION
PASS string
Enters user password
AUTHORIZATION
STAT
Gets the number of messages in and TRANSACTION
octet size of maildrop
LIST [msg]
Gets the size of one or all messages
RETR msg
Retrieves a message from the maildrop. TRANSACTION
DELE msg
Marks the msg as deleted from the TRANSACTION
maildrop.
NOOP
No operation.
RSET
Resets all messages that are marked as TRANSACTION
deleted to unmarked.
QUIT
Terminates the session.
Chapter 6: Application Layer
TRANSACTION
TRANSACTION
AUTHORIZATION,
UPDATE
42
Post Office Protocol (cont.)
POP3 replies:
+OK, -ERR
Example POP
session:
Chapter 6: Application Layer
43
Internet Message Access Protocol
What is IMAP?
A replacement for the POP3 protocol
Differences between IMAP4 and POP3:
IMAP4 allows messages being stored and
manipulated on the mail system
POP3 only allows users to download their
messages and store and manipulate messages
on the client’s machines
Chapter 6: Application Layer
44
Internet Message Access Protocol
(cont.)
Four states in the IMAP4 server:
Non-authenticated state
Authenticated state
When a pre-authenticated connection starts,
When acceptable authentication credentials have been provided
After an error in selecting a mailbox
Selected state
When a connection is established between the server and client
When a mailbox has been successfully selected
Logout state
When the client asks to exit the server
Chapter 6: Application Layer
45
Internet Message Access Protocol (cont.)
IMAP4 command summary:
Session state
Commands
Any
CAPABILITY, NOOP, LOGOUT
Non-authenticated
AUTHENTICATE, LOGIN
Authenticated
SELECT, EXAMINE, CREATE, DELETE, RENAME,
SUBSCRIBE, UNSUBSCRIBE,
STATUS, APPDNED
Selected
LIST,
LSUB,
CHECK, CLOSE, EXPUNCGE, SEARCH, FETCH,
STORE, COPY UID
Chapter 6: Application Layer
46
Internet Message Access Protocol (cont.)
An IMAP4 transaction scenario:
Chapter 6: Application Layer
47
Chapter 6: Application Layer
48
Open Source: qmail
Introduction to qmail:
qmail is a secure, reliable, efficient, simple MTA
designed for Unix-like operating systems
qmail is a replacement for the sendmail
qmail is the second most common SMTP server
qmail has by far the fastest growth of any SMTP
server
Chapter 6: Application Layer
49
qmail System Structure
Core modules of qmail:
Module
Description
qmail-smtpd
Receive a message via SMTP
qmail-inject
Preprocess and send a message
qmail-queue
Queue a message for delivery
qmail-send
Deliver messages from the queue
qmail-clean
Clean up the queue directory
qmail-lspawn
Schedule local deliveries
qmail-local
Deliver or forward a message
qmail-rspawn
Schedule remote deliveries
qmail-remote
Send a message via SMTP
qmail-pop3d
Distribute message via POP3
Chapter 6: Application Layer
50
qmail Data Flow
MTA
remote mail
server
remote mail
client
MUA
SMTP
qmail system
MUA
qmail-smtpd
MTA
local mail
client
qmail-inject
qmail-queue
MDA
qmail-send
qmail-clean
qmail-rspawn
qmail-lspawn
qmail-remote
qmail-local
mailbox/maildir
MRA
SMTP
MTA
qmail-pop3d
POP3
Chapter
Layer mail
remote
mail 6: Application remote
MUA
server
client
51
qmail Control Files
Control files are in /var/qmail/control
Some control files of qmail:
Control
Default
Used by
me
FQDN of various
Description
Default for many control files
system
rcpthosts
(none)
qmail-smtpd Domains that
message for
locals
me
qmail-send
Domains that qmail deliver locally
defaultdomain
me
qmail-inject
Default domain name
plusdomain
me
qmail-inject
Added to any host name that
ends with a plus sign
qmail-send
Virtual domains and users
virtualdomains (none)
Chapter 6: Application Layer
qmail
accept
52
qmail Queue Structure
The qmail queue directory is in /var/qmail/quque
Subdirectories in the qmail queue and contents :
Subdirectory
Contents
bounce
Permanent delivery errors
info
Envelope sender addresses
intd
Envelopes under construction by qmail-queue
local
Local envelope recipient addresses
lock
Lock files
mess
Message files
pid
Used by qmail-queue to acquire an i-node number
remote
Remote envelope recipient addresses
todo
Complete envelopes
Chapter 6: Application Layer
53
How Messages Pass Through the qmail
Queue
Chapter 6: Application Layer
54
6.4 World Wide Web
Introduction
Web Naming and Addressing
Web Data Formats
Hypertext Transfer Protocol
Open Source: Apache
Chapter 6: Application Layer
55
Introduction
What is WWW?
A system of Internet servers that support specially
formatted documents.
How does it work?
1: DNS query
DNS Server
2: TCP 3-way handshake
Web
Browser
Web Server
3: HTTP request
4: HTTP response
Chapter 6: Application Layer
56
Web Naming and Addressing
Uniform Resource Identifier (RFC 2396)
Uniform Resource Locator (RFC 1738)
Uniform Resource Name (RFC 2141)
http:
ftp:
gopher:
etc.
URLs
urn:
URNs
URNs
Chapter 6: Application Layer
57
Uniform Resource Identifier
What is URI?
URI syntax:
A compact string of characters for identifying an abstract or
physical resource.
Absolute URI: <scheme>:<scheme-specific-part>
Generic URI: <scheme>://<authority><path>?<query>
URI examples:
http://speed.cis.nctu.edu.tw/~ydlin/index.html#Books
http://www.google.com/search?q=linux
ftp://ftp.cis.nctu.edu.tw/Documents/IETF/rfc2300~2399/rfc2396.txt
mailto: [email protected]
news: comp.os.linux
telnet://bbs.cis.nctu.edu.tw/
../icons/logo.gif
Chapter 6: Application Layer
58
Uniform Resource Locator
What is URL?
A compact string representation of the location for a
resource that is available via the Internet
URL syntax:
<service>//<user>:<password>@<host>:<port>/<url-path>
Service
Description
ftp
File Transfer protocol
http
Hypertext Transfer Protocol
gopher
The Gopher protocol
mailto
Electronic mail address
news
USENET news
nntp
USENET news using NNTP access
telnet
Reference to interactive sessions
wais
Wide Area Information Servers
file
Host-specific file names
prospero
Prospero Directory Service
Chapter 6: Application Layer
59
Uniform Resource Locator (cont.)
Some URL examples:
http://www.cis.nctu.edu.tw/chinese/ccg/titleMain.gif
ftp://john:[email protected]/projects/book.txt
nntp://news.cis.nctu.edu.tw/cis.course.computernetworks/5238
telnet://mail.cis.nctu.edu.tw:110/
Chapter 6: Application Layer
60
Uniform Resource Name
What is URN?
A name that identifies a resource of unit of
information independent of its location
URN syntax:
URN examples:
<URN> ::= "urn:" <NID> ":" <NSS>
NID: Namespace Identifier
NSS: Namespace Specific String
urn:path:/A/B/C/doc.html
urn:ans:cis.nctu.edu.tw/ydlin/Resource
urn:isbn:0-201-56317-7
URN resolutioin:
http://www.isbn.com/0-201-56317-7
Chapter 6: Application Layer
61
Web Data Formats
Evolution of Web Data Formats
SGML
HTML
XML
XHTML
SGML
HTML
XML
XHTML
Chapter 6: Application Layer
62
Standard Generalized Markup
Language
What is SGML?
A system for organizing and tagging elements of a
document.
Characteristics of SGML
Descriptive Markup
Types of Document
Data Independence
Chapter 6: Application Layer
63
HyperText Makeup Language
What is HTML?
A brief history of HTML
The authoring language used to create
documents on the World Wide Web.
HTML 2.0, 3.0, 3.2, 4.0
HTML document structure
HTML version information
Header
Body
Chapter 6: Application Layer
64
Extensible Markup Language
What is XML?
A pared-down version of SGML, designed
especially for Web documents.
Why XML?
How to use XML?
Traditional data processing
Document-driven programming (DDP)
Archiving
Binding
Chapter 6: Application Layer
65
Extensible HyperText Markup
Language
What is XHTML?
A hybrid between HTML and XML specifically
designed for Net device displays.
Why XHTML?
Using XHTML with other W3C tag sets:
XHTML for structural markup of documents
SMIL for multimedia
MathML for mathematics
SVG for scalable vector graphics
XForms for smart web forms
Chapter 6: Application Layer
66
Hypertext Transfer Protocol
What is HTTP?
HTTP Conversation
Client Request
Server Response
HTTP 1.1
Chapter 6: Application Layer
67
What is HTTP?
A comprehensive addressing scheme
Client-Server Architecture
The HTTP protocol is connectionless and
stateless
An extensible and open representation for
data types
Chapter 6: Application Layer
68
HTTP Conversation
Client request
Request headers
Request body
Server status
Response headers
Requested data
Disconnected
Chapter 6: Application Layer
69
Client Request
HTTP request methods:
Method
OPTIONS
GET
HEAD
POST
PUT
DELETE
TRACE
CONNECT
Meaning
A request for information about the communication options available
for the specified URI.
Requests a document from the server.
Like GET, except only the headers are returned.
Sends data to some handler indicated by the URI.
Requests that the data in the body section be stored at the specified
URI.
Requests that the specified resource be deleted.
For debugging purposes; lets the client see what’s being received on
the other end.
Reserved for future use.
Chapter 6: Application Layer
70
Client Request
Request example:
Chapter 6: Application Layer
71
Server Response
Response example:
Server status codes:
Class of code
1xx
2xx
3xx
4xx
5xx
Meaning
Informational.
Indicates that a client request was completed successfully.
Indicates that the request was redirected for some reason.
Indicates that there was an error on the client end.
Indicates that there was an error on the server end.
Chapter 6: Application Layer
72
Chapter 6: Application Layer
73
HTTP 1.1
HTTP 1.1 features:
Persistent connections
Chunked encoding
byte ranges
Caching
Non-IP virtual hosting
Chapter 6: Application Layer
74
Web Caching
Web caching is a mechanism to expedite
document downloading in WWW.
To achieve maximum satisfaction from Web
caching, some aspects need to be
considered.
Candidates to be cached
Content replacement
Cache coherence
Copyright
Reserved 2010
Chapter
6: Application
Layer
75
Transparent Proxy
A cache server can also act as a proxy server
With port redirection, transparent proxy does
not require manual configuration
Two types of transparent proxy
gateway
HTTP
requests
integrated within a gateway
standalone server box
iptables
dest. port
= 80
Squid proxy/cache
server
(1)
HTTP
requests
router/switch
policy route or
switch rules
dest. port
= 80
Squid box
(2)
Chapter
Copyright
6: Application
Reserved 2010
Layer
76
Open Source: Apache
Introduction to Apache:
Open-Source Web server originally based on
NCSA server
Available on over 160 varieties of Unix -- and
Windows NT
Over 58% of Internet Web servers run Apache or
an Apache derivative
Chapter 6: Application Layer
77
Chapter 6: Application Layer
78
Chapter 6: Application Layer
79
Chapter 6: Application Layer
80
Chapter 6: Application Layer
81
Chapter 6: Application Layer
82
Chapter 6: Application Layer
83
Ration of sendfile ( ) to
ap_invoke_handler ( )
File size
1 kb
4 kb
16 kb
64 kb
256 kb
1024 kb
# of calling sendfile ( )
1
1
1
2
7
15
Total execution time (μs) 37
of sendfile ( )
37
42
78
215
527
Time ratio of sendfile ( ) 35%
to ap_invoke_handler ( )
38%
40%
53%
77%
87%
Chapter 6: Application Layer
84
Apache Server Life Cycle
On Unix systems, Apache creates multiple processes to handle
requests.
The Windows and OS/2 ports are multithreaded..
Chapter 6: Application Layer
85
The Request Processing Cycle
Chapter 6: Application Layer
86
6.5 File Transfer Protocol
File transfer service
Operation model
Inside the connections
Open source: wu-ftpd
Chapter 6: Application Layer
87
File Transfer Service
Goal
File sharing
Data replication for backup
Efficiency and reliability during transmission
Use client-server model based on TCP/IP
Authenticated and anonymous accesses
Relationship between FTP and Telnet
Access an ftp server with browser
ftp://ftp.cis.nctu.edu.tw (anony)
ftp://[email protected] (authen)
Chapter 6: Application Layer
88
Some Application FTP Commands
Command
OPEN
Description
Connect to a remote host
CAT
View a file in a remote host
GET
Retrieve files in a remote host
RENAME
RM
QUIT
Change the name of a file in a remote host
Delete a file in a remote host
Terminate an FTP session
Chapter 6: Application Layer
89
Operation Model
PORT: Send the IP and port of the client to which the data is retrieved
Control connection
(initiated by client)
FTP commands
"PORT h1,h2,h3,h4,p1,p2" Listen on port
"L" (L=21)
FTP replies
File System
Listen on port
"p1,p2"
Send/Receive data
Client
File System
Server
Data connection
(initiated by server)
Client as both control host and receiver
Chapter 6: Application Layer
90
Inside the Connections
Establishing control/data connections
Active Mode
Passive Mode
Control connection initiated by client
Data connection initiated by server
When client is behind a firewall
Both control/data connections are initiated by client
FTP Reply
Example FTP session
Error Recovery
Chapter 6: Application Layer
91
Active/Passive Mode
Active mode
Request
(Connect to me at port P of host H)
Initiate a data connection
Control
client
server
firewall
Data
Passive mode
Request
(Ask server to listen)
Reply
(Listening on port P of host H)
Initiate a data connection
client
server
firewall
Chapter 6: Application Layer
92
Some Proper FTP Commands
Description
Command
Type
USER
Send the user name
Access Control
PASS
Send the password
Access Control
PORT
Send the IP and port of the client to which the data is
retrieved
Transfer
Parameter
PASV
Tell the server to listen on a data port rather than initiate a
data connection
Transfer
Parameter
RETR
Ask server to transfer a copy of the requested file to the
client
File service
STOR
Cause the server to accept and receive the data and store
it as a file
File service
RNFR
Specify the path of a source file to rename from
File service
RNTO
Specify the path of a destination file to rename to
File service
ABOR
Tell the server to abort the previous command and the
corresponding data transfer
File service
Chapter 6: Application Layer
93
FTP Reply
Reply
Description
Type
1yz
The requested action is being initiated; expect another
reply before proceeding with a new command.
2yz
The requested action has been successfully completed.
Positive Complete
reply
3yz
The command has been accepted, but the requested
action is being held, waiting for further information from
another command.
Positive Intermediate
reply
4yz
The command is not accepted the and the requested
action did not take place. The action can be requested
again.
Transient Negative
Completion reply
5yz
Similar with 4yz, except that the error condition is
permanent so that the action cannot be requested again.
Chapter 6: Application Layer
Positive Preliminary
reply
Permanent Negative
Completion reply
94
Error Recovery
The restart mechanism
Sender inserts ‘marker’ (used to identify the checkpoint)
in the data stream
Receiver marks the position of the marker and reply the
latest marker position of both sender and receiver to user
When error, user issues ‘restart’ with the position of the
marker to the sender
* User (control host) and receiver may/may not exist in the same machine
Chapter 6: Application Layer
95
Example FTP Session
STATUS:>
STATUS:>
Connecting to www.cis.nctu.edu.tw (ip = 140.113.166.122)
Socket connected. Waiting for welcome message...
220 www.cis.nctu.edu.tw FTP server (Version wu-2.6.0(1) Mon Feb 28 10:30:36 EST 2000) ready.
COMMAND:>
USER www
331 Password required for www.
COMMAND:>
PASS ********
230 User www logged in.
COMMAND:>
TYPE I
200 Type set to I.
COMMAND:>
REST 100
350 Restarting at 100. Send STORE or RETRIEVE to initiate transfer.
COMMAND:>
REST 0
350 Restarting at 0. Send STORE or RETRIEVE to initiate transfer.
COMMAND:>
pwd
257 "/home/www" is current directory.
COMMAND:>
TYPE A
200 Type set to A.
COMMAND:>
PORT 140,113,189,29,10,27 tell the server where to connect to
200 PORT command successful.
COMMAND:>
LIST
retrieve directory listing
150 Opening ASCII mode data connection for /bin/ls. File status okay; about to open data connection
…….list of files….
COMMAND:>
TYPE I
200 Type set to I.
COMMAND:>
PORT 140,113,189,29,10,31
200 PORT command successful.
COMMAND:>
RETR test
retrieve the file “test”
150 Opening BINARY mode data connection for test (5112 bytes).
Chapter 6: Application Layer
96
Open Source: wu-ftpd
Introduction to wu-ftpd
Features and configurations
Originally developed at Washington University
Most popular ftp daemon
Maintained by WU-FTPD Development Group.
Virtual ftp servers
On-the-fly compression
Important configuration files
Inside wu-ftpd
Chapter 6: Application Layer
97
Inside wu-ftpd
start ftp server
with some options
read ACL files
no (under (x)inetd)
stand-alone?
yes
service
initialization
fork off
(parent exits)
no
connection
accepted?
reverse DNS check
listen to
requests
parse & execute
commands
yes
fork a handler
loop until termination
signal is received
Chapter 6: Application Layer
98
Virtual FTP Servers
1. Manage two or more FTP servers in one machine
2. Support guest groups for different virtual FTP servers
clients
FTP server
ftp.site2.com.tw
A rule segment
in ftpaccess
Configuration file of
each virtual server
…
ftp.site3.com.tw
Lookup the ftpaccess file
ftp.site1.com.tw
# Virtual Server setup for ftp.site1.com.tw
virtual ftp.site1.com.tw root /var/ftp/virtual/site1
virtual ftp.site1.com.tw banner /var/ftp/virtual/site1/banner.msg
virtual ftp.site1.com.tw logfile /var/log/ftp/virtual/site1/xferlog
# Virtual Server setup for ftp.site2.com.tw
…
Chapter 6: Application Layer
99
On-the-Fly File Compressions
Server compress a file (or directory) right before
retrieved by users
Example
User ynlin logged in.
Logged in to wwwpc.cis.nctu.edu.tw.
ncftp /home/ynlin > ls
1.tar.gz
Desktop/
ucd-snmp-4.2.1/
ncftp /home/ynlin > get ucd-snmp-4.2.1.tar.gz
ucd-snmp-4.2.1.tar.gz:
7393280 bytes 552.83 kB/s
ncftp /home/ynlin >lls -l
drwxr-xr-x 24 gis88559 gis88
3584 Oct 8 12:18 .
drwxr-xr-x 88 root gis88
2048 Sep 10 17:48 ..
-rw-r----- 1 gis88559 gis88 7393280 Oct 8 12:18 ucd-snmp-4.2.1.tar.gz
Chapter 6: Application Layer
100
Important Configuration Files
File name
Description
ftpaccess
Used to configure the operations of the ftp
daemon.
ftpconversions Specify the postfix of a retrieved file and its
corresponding operations.
ftphosts
Used to deny/allow some hosts to login as
certain accounts.
ftpservers
List the virtual servers and the corresponding
directories containing their own configuration
files.
Chapter 6: Application Layer
101
6.6 Simple Network Management Protocol
Background
Architectural framework
MIB
SMI
SNMP, SNMPv2, SNMPv3
Open source: NET-SNMP
Chapter 6: Application Layer
102
Background
Control over network systems are demanded
Small tools: ping, traceroute, netstate..etc (base
on ICMP)
SNMP:
Remote control without being physically attached to
managed entities
Exchange management information between network
devices
Short history
SNMPv1 (1989)– Management framework
SNMPv2 (1993)– Functionality enhancement
SNMPv3 (1998)– Security add-on
Chapter 6: Application Layer
103
Architectural Framework
Management station
Agent
Managed device
Managed object (specified in MIB)
Management protocols
management
station
trap or response
SNMP
poll or request
agent &
managed
device in one
machine
master
agent
MIB
Protocols simpler than SNMP
subagent
(managed device ) MIB
MIB
Chapter 6: Application Layer
MIB
104
Management Information Base (MIB)
Tree-like virtual information store
Identified by object identifiers
Extensible in “experimental” and “private” branches
Register at IANA (http://www.iana.org/)
MIB-II: for network management of TCP/IP-based
internets (RFC 1213)
Chapter 6: Application Layer
105
MIB-II
….
iso (1)
org (3)
….
dod….
(6)
….
internet (1)
snmpv2 (6)
security (5)
private (4)
experimental (3)
….
mgmt (2)
directory (1)
mib-2 (1)
snmp (11)
transmission (10) egp (8) udp (7) tcp (6)
icmp (5)
ip (4) at (3)
interface (2)
system (1)
OBJECT IDENTIFIER: 1.3.6.1.2.1.4
Chapter 6: Application Layer
106
Object Groups in Mib-2
Group
system
interface
Description
General information about the managed system
Configuration information and statistics of each physical interface
at
Address translation between network address and physical address
ip
Information of implementation and operation of IP in a local system.
For example, routing table, default TTL.
icmp
Information about the implementation and operation of ICMP. For
example, number of messages ICMP sent and received.
tcp
Information about the implementation and operation of TCP. For
example, the number of maximum and active connections in the
system.
udp
Information about the implementation and operation of UDP. For
example, the number of datagrams sent.
egp
Information about the implementation and operation of EGP.
tranmission
snmp
Related information and statistics of different transmission schemes.
Information about the accesses and errors of SNMP operations.
Chapter 6: Application Layer
107
Structure of Management Information (SMI)
Define the structure of a particular MIB
Place restrictions on the types of the objects allowed in the MIB
Abstract Syntax Notation one (ASN.1)
Exchange of information between application components in
different systems
Syntax – integer, octet string, object identifier
Encode the object values (by Basic Encoding Rule)
Define the way data is represented during transmission
Three categories of data types
Simple: Primitive ASN.1 data types
Application-wide: Special data types used in particular Applications
Simply constructed: table, row
Chapter 6: Application Layer
108
TCP Connection Table (in MIB-II)
-- the TCP Connection table
-- The TCP connection table contains information about this
-- entity's existing TCP connections.
ACCESS read-write
STATUS mandatory
DESCRIPTION
"The state of this TCP connection.."
tcpConnTable OBJECT-TYPE
::= { tcpConnEntry 1 }
SYNTAX SEQUENCE OF TcpConnEntry
tcpConnLocalAddress OBJECT-TYPE
ACCESS not-accessible
SYNTAX IpAddress
STATUS mandatory
ACCESS read-only
DESCRIPTION
STATUS mandatory
"A table containing TCP connection-specific information."
DESCRIPTION
::= { tcp 13 }
"The local IP address for this TCP connection. In the case of a
tcpConnEntry OBJECT-TYPE
connection in the listen state which is willing to accept connections for any IP
SYNTAX TcpConnEntry
interface associated with the node, the value 0.0.0.0 is used."
ACCESS not-accessible
::= { tcpConnEntry 2 }
STATUS mandatory
tcpConnLocalPort OBJECT-TYPE
DESCRIPTION
SYNTAX INTEGER (0..65535)
"Information about a particular current TCP connection. An
ACCESS read-only
object of this type is transient, in that it ceases to exist when (or soon after)
STATUS mandatory
the connection makes the transition to the CLOSED state."
DESCRIPTION
INDEX { tcpConnLocalAddress,
"The local port number for this TCP connection."
tcpConnLocalPort,
::= { tcpConnEntry 3 }
tcpConnRemAddress,
tcpConnRemAddress OBJECT-TYPE
tcpConnRemPort }
SYNTAX IpAddress
::= { tcpConnTable 1 }
ACCESS read-only
TcpConnEntry ::=
STATUS mandatory
SEQUENCE {
DESCRIPTION
tcpConnState INTEGER,
"The remote IP address for this TCP connection."
tcpConnLocalAddress IpAddress,
::= { tcpConnEntry 4 }
tcpConnLocalPort INTEGER (0..65535),
tcpConnRemPort OBJECT-TYPE
tcpConnRemAddress IpAddress,
SYNTAX INTEGER (0..65535)
tcpConnRemPort INTEGER (0..65535)
ACCESS read-only
}
STATUS mandatory
tcpConnState OBJECT-TYPE
DESCRIPTION
SYNTAX INTEGER {
"The remote port number for this TCP connection."
closed(1), listen(2), synSent(3), synReceived(4)
::= { tcpConnEntry 5 }
established(5), finWait1(6), finWait2(7), closeWait(8),
lastAck(9), closing(10), timeWait(11), deleteTCB(12) }
Chapter 6: Application Layer
109
TCP Connection Table (tabular view)
tcpConnTable (1.3.6.1.2.1.6.13)
tcpConnEntry = (x.1)
tcpConnState
tcpConnLocalAddress
(x.1.1)
(x.1.2)
x.1
x.1
Listen
0.0.0.0
23
0.0.0.0
0
Listen
0.0.0.0
161
0.0.0.0
0
x.1
closeWait
127.0.0.1
161
127.0.0.1
1029
140.113.88.164
23
140.113.88.174
3082
x.1 established
tcpConnLocalPort tcpConnRemoteAddress tcpConnRemotePort
(x.1.3)
(x.1.4)
(x.1.5)
INDEX
Chapter 6: Application Layer
110
Basic Operations in SNMP
PDU
GetRequest
GetNextRequest
SetRequest
GetResponse
Trap
GetBulkRequest
InformRequest
Descriptions
Version
Retrieve the value of a leaf object
V1
Get the object lexicographically next to the one specified
V1
Set (update) a leaf object with a value
V1
Response for GetRequest (value) or SetRequest (ACK)
V1
Issued by agent to notify the management station of
some significant event asynchronously
V1
Retrieve large blocks of data, such as multiple rows in a
table.
V2
Allows one MS to send trap information to another MS
and receive a response
V2
PDU: Basic data unit in SNMP operations
MS : Management Station
Variable-binding list: A list of variables and corresponding values in a PDU
Chapter 6: Application Layer
111
TCP Connection Table (lexicographical
tcpConnTable (1.3.6.1.2.1.6.13=x)
view)
Traverse the tree using
Depth First Search
tcpConnEntry = (x.1)
…
tcpConnState
tcpConnLocalAddress
tcpConnLocalPort
(x.1.1)
(x.1.2)
(x.1.3)
0.0.0.0
23
(x.1.1.0.0.0.0.23.0.0
.0.0.0)
(x.1.2.0.0.0.0.23.0.0
.0.0.0)
(x.1.3.0.0.0.0.23.0.0
.0.0.0)
Listen
0.0.0.0
161
(x.1.1.0.0.0.0.161.0.0
.0.0.0)
(x.1.2.0.0.0.0.161.0.0
.0.0.0)
(x.1.3.0.0.0.0.161.0.0
.0.0.0)
Listen
closeWait
127.0.0.1
161
(x.1.1.127.0.0.1.161.
127.0.0.1.1029)
(x.1.2.127.0.0.1.161.
127.0.0.1.1029)
(x.1.3.127.0.0.1.161.
127.0.0.1.1029)
established
140.113.88.164
23
(x.1.1.140.113.88.164.23.
140.113.88.174.3082)
(x.1.2.140.113.88.164.23.
140.113.88.174.3082)
(x.1.3.140.113.88.164.23.
140.113.88.174.3082)
Chapter 6: Application Layer
112
User-based Security Model (USM, RFC2574)
Four major threats
Modification of Information (between agent and MS)
Masquerade (pretend to be an authorized user)
Disclosure (eavesdropping)
Message Stream Modification (reorder, delay, replay)
Solutions
Timeliness module (for 1st, 3rd and 4th threats)
Authentication protocol—use MD5 (for the 2nd threat)
Privacy protocol—Use DES (for 3rd)
Chapter 6: Application Layer
113
View-based Access Control Model (VACM
RFC 2575)
Access control
MIB view-- a collection of MIB objects
Elements in the model
Group– Categorization of managers
securityLevel– Help distinguish the access rights of a
group
Context– A collection of managed object accessible by an
snmp entity.
MIB view– A subset of a particular context
Access policy– Decide the final access rights of a group
to a context
Chapter 6: Application Layer
114
An SNMP Entity and its Component (RFC 2571)
SNMP entity
SNMP engine
Dispatcher
Message
Processing
Subsystem
Security
Subsystem
Access
Control
Subsystem
Application(s)
Command
Generator
Notification
Receiver
Proxy
Forwarder
Command
Responder
Notification
Originator
Other
Chapter 6: Application Layer
115
Comparisons Between Three SNMP Versions
Version
Description and improvement
SNMPv1
1. Define the SMI (RFC1155)
2. A more concise MIB definition (RFC 1212)
3. SNMP framework and its related operations (RFC 1157)
4. Concept of security (authentication) is proposed
SNMPv2
1. Improved SMI (support 64-bit counter, and other types of address than IP)
2. Inclusion of ‘GetBulkRequest’ to improve the efficiency for retrieving large
blocks of data
3. ‘InformRequest’ for communication between management stations
SNMPv3
1. Security and administration add-on’s (RFC2571)
2. Multi-version SNMP message processing and dispatch capability (RFC 2572)
3. The five type of applications within an SNMP engine (RFC 2573)
4. User-based security model (RFC 2574)
5. View-based access control (RFC 2575)
Chapter 6: Application Layer
116
Open Source: Net-SNMP
Introduction to Net-SNMP
Some commands for query, set and trap
Extensible architecture
Ways of extending the MIB
Subagent protocol– AgentX (RFC2741)
Build and include a private MIB
Inside Net-SNMP
Snmpd and snmptrapd
Chapter 6: Application Layer
117
Introduction
Started at CMU, moved to UCDavis (1995) and is
now based at SourceForge (2000~)
What it provides:
An extensible agent
SNMP library for further development
Tools to get or set information from SNMP agents
Tools to generate and handle SNMP traps
Support SNMP V1, V2 and V3
Runs on many Unix-like systems and Windows
Chapter 6: Application Layer
118
Some Commands for Query, Set and Trap
Name
Description and example
PDU used
SNMPGET
Retrieve the value of a leaf object using get
GetRequest
SNMPSET
Set (update) a leaf object with a value
SetRequest
SNMPBULKGET
Get multiple objects at a time. Possibly
under different subtrees
GetBulkReques
t
SNMPWALK
Explore all the objects under a subtree
of the MIB
GetNextReques
t
SNMPTRAP
Uses the TRAP Request to send information
to a network manager. More than one object
identifiers can be applied as arguments
SNMPSTATUS
Used to retrieve several important statistics
from a network entity. Errors will also be
reported, if any
SNMPNETSTAT
Displays the values of various networkrelated information retrieved from a remote
system using the SNMP protocol
Chapter 6: Application Layer
Trap
119
Command Line Examples
$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0
system.sysContact.0 = [email protected]
$ snmpset -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0 s gis88559
system.sysContact.0 = gis88559
$ snmpget -v 3 -u ynlin -l authNoPriv -a MD5 -A ynlinsnmp localhost system.sysContact.0
system.sysContact.0 = gis88559
$ /usr/local/bin/snmpbulkwalk -v 3 -u ynlin -l authNoPriv –a MD5 -A ynlinpasswd localhost system
system.sysDescr.0 = Linux ynlin2.cis.nctu.edu.tw 2.4.14 #5 SMP 週四 11月 22 23:6
system.sysObjectID.0 = OID: enterprises.ucdavis.ucdSnmpAgent.linux
system.sysUpTime.0 = Timeticks: (30411450) 3 days, 12:28:34.50
system.sysContact.0 = gis88559
system.sysName.0 = ynlin2.cis.nctu.edu.tw
system.sysLocation.0 = ynlin2
system.sysORLastChange.0 = Timeticks: (0) 0:00:00.00
system.sysORTable.sysOREntry.sysORID.1 = OID: ifMIB
system.sysORTable.sysOREntry.sysORID.2 = OID: .iso.org.dod.internet.snmpV2.snmpB
system.sysORTable.sysOREntry.sysORID.3 = OID: tcpMIB
system.sysORTable.sysOREntry.sysORID.4 = OID: ip
system.sysORTable.sysOREntry.sysORID.5 = OID: udpMIB
....
Chapter 6: Application Layer
120
Snmpd and Snmptrapd
Snmpd
Process snmp packets and respond with the required
information
Can be either master agent or subagent
Basic access control with a port/addr pair
Load dynamical MIB module at start
Snmptrapd
Receive and log trap messages
Configured to launch an external program when receiving
a trap matching a particular object identifier
Chapter 6: Application Layer
121
Processing flow inside Net-SNMP
Start the server with
required options
init_agent()
[agent/snmp_vars.c]
init_snmp()
[agent/snmp_api.c]
init_master_agent()
[agent/snmp_agent.c]
while (netsnmp_running)
receive() [agent/snmpdt.c]
snmp_select_info()
[snmplib/snmp_api.c]
count = select(numfds, &fdset, 0, 0, tvp)
snmp_read()
[snmplib/snmp_api.c]
netsnmp_check_outstanding_agent_requests()
[agent/snmp_agent.c]
Chapter 6: Application Layer
122
Data Structure of a Session
Chapter 6: Application Layer
123
6.7 VoIP
Introduction
Elements in a VoIP environment
Underlying protocols
H.323
SIP
Challenges for VoIP technology
Open source: openphone
Chapter 6: Application Layer
124
Introduction
Three ways of transferring voice
Circuit switched
Voice over frame relay
Voice over IP
Pros and cons of VoIP compared with PSTN
Inexpensive cost
Simplicity
Less bandwidth consumption
Extensibility
Chapter 6: Application Layer
125
VoIP Environment (H.323 adopted)
Gateway
Router
PSTN
Network
IP Network
MCU
H.323
gatekeeper
H.323
terminal
Telephone
Packet switched network
Circuit switched network
Chapter 6: Application Layer
126
H.323 Protocol Stack
Control
Data
A / V Control
control (by GK)
Audio Video
G.711
Q.931
H.245 T.120
G.723 H.263
RTCP
G.729
RAS
(H.225.0)
RTP
TCP
UDP
IP
Recommendation
RAS(H.225.0)
Description
Stands for Registration, Admission and Status.
H.245
Control protocol for capability negotiation and channel setup.
Q.931
Used for call setup and teardown between two terminals.
T.120
Data protocols for multimedia conferencing. (application sharing,
whiteboarding)
RTP/RTCP
Used for real time traffic synchronization and transportation.
Chapter 6: Application Layer
127
Setup Procedure of an H.323 Call
Registration and admission
RAS
Call setup
Q.931
Terminal capability negotiation ,channel
setup and master-slave detection
Stable call established and proceeds
H.245
RTP/RTCP
Close channel
H.245
Call teardown
Q.931
Disengagement
Chapter 6: Application Layer
RAS
128
Session Initialization Protocol (SIP)
SIP
Targeted to replace the H.323
Simplicity
Compatibility
Characterize a multimedia session
With supplementary SDP/SAP protocols
HTTP-like characteristics
Text-based protocol
Message syntax and header fields identical to HTTP/1.1
Client-server scheme
Chapter 6: Application Layer
129
Elements in an SIP Environment
Redirect
server
Location
server
Internet
User Agent
Local
Client (UAC) proxy server
Remote
proxy server
Chapter 6: Application Layer
User Agent
Server (UAS)
130
Protocol Stacks of SIP
Control plane
Data plane
SIP
Multimedia Traffic
SAP/SDP
RTP
TCP
RTCP
UDP
IP
Chapter 6: Application Layer
131
Operators and Reply Codes in SIP
Operators
Description
INVITE
Invite a user to a call
ACK
Confirmation for the final response
BYE
Terminate a call between endpoints
CANCEL
Terminate the search for a user or request for a call
OPTIONS
Features supported for a call
REGISTER
Register current location of the client with location server
INFO
Use for mid-session signaling
Reply Code
Description
1xx (Informational)
Trying, ringing and queued
2xx (Successful)
The request was successful
3xx (Redirection)
Give information about the receiver’s new location
4xx (Request Failures)
Failure responses from a particular server
5xx (Server Failures)
Failure responses given when a server itself has erred
6xx (Global Failures)
Busy, decline, requests not acceptable
Chapter 6: Application Layer
132
Example Operation in SIP
1. UAC INVITE UAS with the SIP URL
2. Destination IP address
known => Send the request directly to the destination
unknown => Redirected to the local proxy server (with
location server).
3. In redirection mode, send back the callee’s new location
4. In proxy mode, forward the request to the destination.
5. The UAS answers 200 (OK), then the UAC ACKs
6. Session established
Chapter 6: Application Layer
133
H.323 V.S. SIP
Simplicity in operation and implementation
Message encoding
Capability exchange
Supported data types
Capability of handling a conference
Chapter 6: Application Layer
134
Open Source: Asterisk
An integrated PBX (Private Branch eXchange)
system
PSTN
Networks
Asterisk PBX Server
ATA
Adaptor
PC Softphone
SIP phone
Analog Phone
Chapter 6: Application Layer
135
Block Diagram
Asterisk provides a framework to build a customized
VoIP system.
Flexible to add/remove modules to establish the basic
transport service
Asterisk framework
HTTP
PBX
Channel
CDR Engine
SNMP
Module Loader
RTP
Framer
Chapter 6: Application Layer
136
Data Structures
Two concepts, context and extension, are
introduced
Inside a
Asterisk PBX
Context1
...
...
Ext1
Ext2
...
Context2
Ext1
Ext2
Chapter 6: Application Layer
137
Call flow
init_manager()
[main/manager.c]
action_originate()
[main/manager.c]
ast_pbx_outgoing_exten() [main/pbx.c]
__ast_request_and_dial()
[main/pbx.c]
yes
First call in this
channel?
ast_pbx_start()
[main/pbx.c]
pbx_thread()
[main/pbx.c]
no
ast_pbx_run()
[main/pbx.c]
__ast_pbx_run()
[main/pbx.c]
__ast_pbx_run()
[main/pbx.c]
Chapter 6: Application Layer
138
Chapter 6: Application Layer
139
6.8 Streaming
Introduction to streaming
Architecture and components for streaming
Availability
Short fetching time
Avoid storage overhead
Live broadcasting
Compression scheme
Protocol stack for streaming
QoS and synchronization mechanism
Open source: Darwin streaming server
Chapter 6: Application Layer
140
Architecture and Components for Streaming
Internet
Server
Client
Transport
Protocols
Video/Audio
raw data
Transport
Protocols
QoS Control
Video Files
QoS Control
Audio Files
synchronization
Video Decoder
Chapter 6: Application Layer
Audio Decoder
141
Compression Schemes
Temporal or spatial
Lossless or lossy
Temporal – describe changes between frames
Spatial – describe similar patterns in a frame
Recoverability when decompressed
GIF (lossless) and JPG (lossy)
Symmetrical or Asymmetrical
The time for compression and decompression
According to the load of the server
Chapter 6: Application Layer
142
Protocol Stacks of Streaming
RTSP (Real Time Streaming Protocol)
VCR-Style remote control functionalities
Establish and control streams
Description file before a session
Methods in a session (SETUP, PLAY, TEARDOWN)
Reuse of HTTP concepts (Ex: ASCII messages)
HTTP
Mostly by buffering
Low performance
Control plane
HTTP
RTSP
RTCP
TCP
Data plane
RTP
UDP
IP
Chapter 6: Application Layer
143
QoS and Synchronization Mechanism
QoS mechanism
Source-based rate control
Receiver-based rate control
Synchronization mechanism
Intra-stream synchronization
Inter-stream synchronization
Continuity of audio/video data frames inside a
stream
Consistency between cooperative streams
Inter-object synchronization
Consistency between streams and other objects like
text and still images
Chapter 6: Application Layer
144
Open Source: Darwin Streaming Server
Open source version of Apple's QuickTime
Streaming Server (QTSS)
Deliver streaming media with RTP and RTSP
Provide both live and on-demand programs
Supports many formats: H.264/MPEG-4 AVC,
MPEG-4 Part 2, 3GP and MP3
Provide Web-based administration, authentication,
relay support, and integrated broadcaster
administration, etc.
Chapter 6: Application Layer
145
Block Diagram
Two parts: core server and modules
Core server provides task scheduling
Module provides specific functions
Chapter 6: Application Layer
146
Data Structures
Chapter 6: Application Layer
147
Call Flow: Task Handling
Chapter 6: Application Layer
148
RTSP Handling State Transition Diagram
Chapter 6: Application Layer
149
6.9 Peer-to-Peer Applications (P2P)
Introduction
P2P Architectures
Performance Issues of P2P Applications
Case Study: BitTorrent
Open source: Vuze (BitTorrent Client)
Chapter 6: Application Layer
150
Introduction to P2P
A distributed network architecture in which
participants act as both a client and a server
Participants construct a virtual overlay network at
the application layer on top of the underlying IP
network
P1
Node: peers
Edge: TCP/UDP connection
P2
Overlay Network
P1
R1
Physical Network
R2
Chapter 6: Application Layer
R3
P2
151
Introduction to P2P
Operations in P2P systems
Joining the P2P overlay network,
Resource discovery
The most challenging problem for P2P applications
Resource retrieval
Chapter 6: Application Layer
152
P2P Architectures
Way of forming a P2P overlay network could
be classified into three categories
Centralized
Decentralized and unstructured
Decentralized but structured
Chapter 6: Application Layer
153
Centralized P2P
Operation overview
2. Query&
Response Directory
1. upload index
1. upload index
Peer
Peer
Peer
3. Download file
Chapter 6: Application Layer
154
Centralized P2P
Utilizes a central directory server for locating
objects in the P2P network
Peers join by registering to the directory server
Peers inform the directory server of objects to be
shared
A peer sends query to the directory server to
search an object
The peer receives reply from the directory server
The peer selects one or more peer in the reply to
download the object directly from
Example: Napster
Chapter 6: Application Layer
155
Centralized P2P
Advantages
Simple
Easy to implement
Support various kinds of search such as keyword,
full-text, and metadata search
Disadvantages
Not a true P2P system as it relies on a central
directory server which becomes performance
bottleneck, single point of failure, not scalable and
vulnerable to DoS attacks
Chapter 6: Application Layer
156
Decentralized and Unstructured P2P
To get rid of server, this approach floods query
messages to peers to search for shared objects
Limited-scope flooding is adopted to reduce flooding
messages
A query hit message is returned along the reverse
path back to the inquirer
Download
Query
Query
Query Hit
Query Hit
Query
Example: Gnutella
Query
Query Hit
Query
Chapter 6: Application Layer
157
Decentralized and Unstructured P2P
Join procedure
A peer needs to know at least one of the peers
already on the overlay network.
The peer sends a join message to a peer already
on the overlay.
The existing peer then replies its identity as well
as a list of its neighbors. It may also forward the
join message to its neighbors.
Upon receiving join reply messages, the
newcomer knows more peers on the overlay.
Chapter 6: Application Layer
158
Decentralized and Unstructured P2P
Hierarchical overlay with super peers
Flooding is apparently not scalable
FastTrack adopts a hierarchical overlay
A super peer acts as a local directory database
which stores the indexes of objects shared by
ordinary peers
Two-level hierarchical overlay
The lower level adopts the central
server approach
The upper level (super peers) adopts the
decentralized and unstructured approach.
Chapter 6: Application Layer
query reply
159
Decentralized and Unstructured P2P
Advantages
Fully distributed
Reliable, fault-tolerant
No single point of failure
Disadvantages
Excessive query traffic make it not scalable
May fail to find content that is actually in the
system
Super peer may become overloaded or been
attacked
Chapter 6: Application Layer
160
Decentralized but Structured
Combine the distributed directory service with
an efficient query routing scheme
Key ideas
For distributed directory service, a hash function
maps peers and objects into the same address
space so that objects can be deterministically
assigned to peers in a distributive manner.
For efficient query routing, peers are organized
into a structured overlay based on their positions
in the address space.
Chapter 6: Application Layer
161
Decentralized but Structured
Operations overview
Each peer generates its own node ID by a predefined
hash function.
For each object being held and to be shared, the peer
generates the object ID by the same or another hash
function.
For each object, the peer sendd a register message to
the node that has the node ID same as the object’s ID.
If a peer wants to query an object, it uses the hash
function to generate the object ID and sends the query
message to the node that hosts the object’s ID.
Chapter 6: Application Layer
162
Decentralized but Structured
Message routing (use Chord as an example)
Key idea: have each peer maintain a specially
designed routing table such that every peer could
forward the arriving message to a neighboring
peer with node ID that is further closer to the
destination.
Consider a 10-node Chord overlay in a 6-bit
address space
Chord views its address space as a onedimensional circular space such that peers in the
space form a ring overlay.
Chapter 6: Application Layer
163
Message Routing in Chord
The routing table in Chord is called a finger
table.
For an m-bit address space, the finger table
of a node with ID=x consists of at most m
entries and the i-th entry points to the first
node with ID following the ID of x+2i-1 modulo
2m, for 1≤i≤m.
Chapter 6: Application Layer
164
Finger Table of Chord
Finger table of node N8, where m =6.
N1
Finger table
N8+1 N15
N8
N8+2 N15
+1
N56
N8+4 N15
N8+8 N20
N8+16 N30
+2
N8+32 N42
+4
N51
N15
+8
N47
N20
+32
+16
N42
N38
N30
Chapter 6: Application Layer
165
Routing a Query Message
Routing a query message for object 54 from N8
N1
Finger table
N8
lookup(54)
N8+1 N15
N8+2 N15
N8+4 N15
K54 N56
N8+8 N20
N8+16 N30
N8+32 N42
Finger table
N51+1 N56
N51+2 N56
N51+4 N56
N51
N51+8 N1
N15
N51+16 N8
N51+32 N20
N47
N20
Finger table
N42
N42+1 N47
N42+2 N47
N42+4 N47
N42+8 N51
N38
N30
N42+16 N1
N42+32 N15
Chapter 6: Application Layer
166
Performance Issues of P2P Applications
Free Riding
Flash Crowd
Topology Awareness
NAT Traversal
Churn
Security
Copyright Infringement
Chapter
Copyright
6: Application
Reserved 2010
Layer
167
Free Riding
Scalability of P2P systems relies on the
contribution from peers
A common solution is to implement some
incentive mechanisms.
If a peer only consumes but contributes little or no
resources, it becomes a free rider of the system.
85% of peers share no files in Gnutella in 2005
For example, tit-for-tat in BitTorrent.
Other solutions: reward-based and creditbased mechanisms
Chapter 6: Application Layer
168
Flash Crowd
A sudden, unanticipated growth in the
demand of a particular object
Issues
e.g., a new release of a DVD video or mp3 file
how to deal with a sudden large amount of query
messages
how long to find and download the object within a
short time period
Solutions
Cache, duplicating popular objects
Chapter 6: Application Layer
169
Topology Awareness
A virtual link in a P2P overlay could be a
long end-to-end connection across continents
or a short one within a local area network
How to avoid serious topology mismatch
Solutions
Many route-proximity and neighbor-proximity
enhancements for P2P overlay systems have
been proposed based on RTT measurement,
preference of routing domain or ISP, or
geographical information.
Chapter 6: Application Layer
170
NAT Traversal
Basic requirement for a P2P system is to
provide peers with NAT traversal
mechanisms
If both peers are behind NAT devices, they cannot
connect to each other without help from other
peers or STUN servers
Solutions
In most cases, NAT traversal is solved by relay
peers or super peers that have public IP
addresses
Chapter 6: Application Layer
171
Churn
Churn refers to the phenomenon that peers
dynamically join and leave the system at will.
high churn rate seriously affects the stability and
scalability of a P2P system.
e.g., a high churn rate may cause a tremendous
overlay maintenance overhead and dramatic routing
performance degradation in DHT-based system
Solutions
Avoid a rigid structure or relation among peers
Peers maintain a list of potential neighbors for quick
and dynamic neighbor replacement
Chapter 6: Application Layer
172
Security
Security issues
P2P programs with back hole (Trojan Horse),
spurious content, leaking of files not to be shared.
Solutions to content pollution
protect the content with message digest such as
MD5
In BitTorrent, the MD5 digest of each piece of a shared
file is stored in the metadata file
peer reputation system
object reputation system
Chapter 6: Application Layer
173
Copyright Infringement
Sharing copyrighted objects through P2P
systems is a serious problem which hinders
the promotion and usage of P2P systems.
Not only P2P users are responsible for
copyright infringement, so are the companies
that host P2P applications
Especially in the case where P2P systems will not
be able to exist without their serves, e.g., Napster.
Chapter 6: Application Layer
174
Case Study: BitTorrent
BitTorrent (BT) is originally designed by Bram Cohen in
2001
Well thought-out protocol with several unique features
use tit-for-tat as an incentive mechanism to cope with free
riders
use out-of-band search to avoid copyright infringement issue
use pull-based swarming for load balancing
use hash check to prevent propagation of spurious pieces
after a peer has successfully downloaded a file, it becomes a
seeder to distribute the file
Chapter
Copyright
6: Application
Reserved 2010
Layer
175
BT Terminologies
A file is cut into pieces of a fixed size.
A piece is further divided into chunks, the basic
data unit for a peer to request for content.
The integrity of a piece is protected by a SHA-1
A peer becomes a seeder if it has successfully
downloaded the file.
A tracker for each file or group of files to be
shared. The tracker tracks the downloading
peers and seeds, and coordinates the file
distribution among peers.
Chapter 6: Application Layer
176
BT Operation Overview
Web Server
Source
Web page with Link
to .torrent
publish
1. get .torrent
Downloading peer
Tracker
2. get
announce
3. response peer
Seeder
list
4. get piece
5. get piece
Downloading peer
Chapter 6: Application Layer
177
Piece Selection
Random first piece selection
Rarest first policy
For the first few pieces, the client just randomly
selects a piece to download.
Selects the most scarce piece to download first
End-game mode
To speed up the completion of a file download at
the end, a peer with only a few pieces missing will
send requests for all missing pieces to all the
peers
Chapter 6: Application Layer
178
Peer Selection
Choking/unchoking
Optimistic unchoking
Choking refers to a temporal refusal to upload to a peer.
At the beginning, all peers are chocked
Tit-for-tat algorithm selects a fixed number of peers from which the
peer downloaded most to unchoke
new peer needs to move its first step when initially joined the
system
select one peer at random
Anti-snubbing
If a peer is choked by all of its peers (snubbed), it is better to run
optimistic unchoking more often to explore more peers that are
willing to cooperate.
Chapter 6: Application Layer
179
Open source: Vuze (BitTorrent Client)
Many BT client programs are open-source
Some popular client programs
Vuze
uTorrent
BitComet
Chapter
Copyright
6: Application
Reserved 2010
Layer
180
Core Packages
Most of Vuze’s core packages
are located under the
.\com\aelitis\azureus\core
directory
Chapter 6: Application Layer
181
Data Structure
Classes for the peer and piece objects
Chapter 6: Application Layer
182
Algorithm Implementation
Main program for controlling the piece and peer
selection is the PEPeerControlImpl class
Chapter 6: Application Layer
183
Implementation of Peer and Piece Selection
Unchoking algorithms are implemented in
DownloadingUnchocker.java and
SeedingUnchocker.java
tit-for-tat is implemented in calculateUnchokes()
Optimistic unchoking is implemented in
UnchokerUtil.getNextOptimisticPeer()
getRequestCandidate() defined in
PiecePickerImpl.java is the core method for
deciding which block to download
Chapter 6: Application Layer
184
6.10 Summary
General issues
Major application protocols
DNS, SMTP, POP3, IMAP4, HTTP, FTP, SNMP
Real time, multimedia application protocols
well-known ports, concurrent connection-oriented
servers and iterative connectionless servers,
statefulness/statelessness
SIP, RTP, RTCP, RTSP
P2P protocols
Napster, Gnutella, DHT, Chord, BT
Chapter 6: Application Layer
185