Transcript pptx - Dongsu Han

NETWORK PROGRAMMING
INTRO TO DISTRIBUTED SYSTEMS
FALL 2015
Some material taken from publicly available lecture slides
including Srini Seshan’s and David Anderson’s
Dongsu Han
L3-socket
Today’s Lecture

Lecture
 Transport
protocols: UDP and TCP
 Version control using SVN

TA-led programming session
 Editor,
socket programming
 Debugging with gdb

Announcement
 Programming
Assignment 1 is out. (Due in two weeks--추석)
Review of Previous Lectures

What is a distributed system? (L1)
 Definition
 Example

What are some of the important decisions made in the Internet
architecture? (L2)
 Narrow
 Why?:
waist: Network provides minimal functionality (best-effort)
to connect existing networks that are heterogeneous
 Stateless
architecture: no per-flow state inside network
 Why?
 For
survivability
Principles






P1: IP over all network (narrow waist)
P2: Layering
Principle 3: Fate Sharing
Principle 4: Soft-state
Principle 5: End-to-End Argument  Next lecture
Principle 6: network layer provides one simple service: best
effort datagram (packet) delivery
Principle 4: Soft-state
5

Soft-state (as oppose to hard-state)
 Announce
state
 Refresh state
 Timeout state


Penalty for timeout – poor performance
Robust way to identify communication flows (firewall)
 Possible

mechanism to provide non-best effort service
Helps survivability
Goal 2: Types of Service
6


Principle 6: network layer provides one simple service: best effort datagram (packet)
delivery
 All packets are treated the same
Network layer (IP) provides “best effort” service.


Other functions are implemented at the end-host at the transport layer (TCP/UDP)



Relatively simple core network elements
Building block from which other services (such as reliable data stream) can be built
Contributes to scalability of network
Issues: No QoS support assumed from below
 Hard to implement without network support
 QoS is an ongoing debate…
Summary
7

Successes: IP on everything!

Drawbacks…
but perhaps they’re totally
worth it in the context of the
original Internet. Might not have
worked without them!
“This set of goals might seem to be
nothing more than a checklist of all the
desirable network features. It is
important to understand that these
goals are in order of importance, and
an entirely different network
architecture would result if the order
were changed.”
Transport Protocols

Lowest level end-to-end protocol.
Header generated by sender is interpreted
only by the destination
 Routers view transport header as part of the
payload; they are not supposed to see the
transport header.

7
7
6
6
5
5
Transport
Transport
IP
IP
IP
Datalink
2
2
Datalink
Physical
1
1
Physical
router
8
What does the transport layer do?

No per-flow (connection) state in the network
 Transport
layer maintains the connection state. Transport layer identifies an
end-to-end connection.

How does the transport layer identify an end-point (application)?
De-multiplexing: Port numbers




Network layer gets you to the host.
A port identifies the sending or receiving socket (application).
A socket is one endpoint of a two-way communication between two
programs running on the network. A socket is bound to a port number
so that the transport layer can identify the application that data is
destined to be sent.
Identifying two end-points: An endpoint = <IP address, Port #>
Source endpoint = <SrcIPAddr, SrcPort>
 destination endponit = <DestIPAddr, DestPort>

Protocol and Port Demultiplexing
11


Multiple choices at each layer
Applications open a socket (identified by the port number)
Port 80
Port 22 Port 80 Port 22
TCP
UDP
Network
IP
TCP/UDP
IP
NET1
NET2
…
NETn
Type Field
=IP
Protocol Field Port Number
= TCP
= 80
Server and Client
Server and Client exchange messages over the
network through a common Socket API
Clients
Server
TCP/UDP
user
space
ports
Socket API
TCP/UDP
IP
IP
Ethernet Adapter
Ethernet Adapter
12
kernel
space
hardware
Transport protocols

UDP
 Provides

only integrity and de-multiplexing
TCP adds…
UDP: User Datagram Protocol [RFC 768]
14


“bare bones” Internet transport
protocol, “Best effort” service
UDP segments may be:
Lost, duplicated (bug, spurious retransmission a
t L2)
 Delivered out of order to app


Connectionless:
No handshaking between UDP sender, receiver
 Each UDP segment handled independently

Why is there a UDP?
• No connection establishment
(which can add delay)
• Simple: sender, receiver stat
e is minimal
• Small header
• No congestion control: UDP
can blast away as fast as de
sired
UDP, cont.
15

Often used for streaming multimedi
a apps (e.g., Skype)
Length, in
 Loss
tolerant
 Rate sensitive

Other UDP uses (why?):
bytes of UDP
segment,
including
header
32 bits
Source port #
Dest port #
Length
Checksum
 DNS

Reliable transfer over UDP
 Must
be at application layer
 Application-specific error recovery
Application
data
(message)
UDP segment format
Transport protocols

UDP
 Provides

only integrity and de-multiplexing
TCP adds…
 Connection-oriented
 Reliable
(error control)
 Ordered byte stream (in-order delivery)
 Bi-directional byte-stream
 Flow and congestion controlled
TCP

Protocol implemented entirely at the ends
 Fate

sharing
Abstraction provided
 Connection
oriented protocol (hand-shake or connection establishment)
 In-order byte-stream

Implemented functionality (next lecture)
 Flow
control
 Reliability (error control)
 Reassembly (in-order delivery)
 Congestion control
TCP Client Example
18
/* get the server host entry */
hp = gethostbyname(argv[1]);
if (hp == NULL) {
perror("gethostbyname");
return -1;
}
/* create stream socket */
sockfd = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (sockfd < 0) {
perror("socket");
return -1;
}
/* set the destination address */
daddr.sin_family = AF_INET;
memcpy(&daddr.sin_addr.s_addr, hp->h_addr, hp->h_length);
daddr.sin_port = htons(PORT);
TCP Client Example
19
/* connect to the destination */
if (connect(sockfd, (struct sockaddr *)&daddr, sizeof(struct sockaddr_in))) {
close(sockfd);
perror("connect");
return -1;
}
inet_ntop(AF_INET, &daddr.sin_addr, s, sizeof(s));
printf("client: connecting to %s\n", s);
/* receive message sent from the server */
if ((numbytes = recv(sockfd, buf, MAXDATASIZE-1, 0)) == -1) {
perror("recv"); exit(1);
}
/* print the received message */
buf[numbytes] = '\0';
printf("client: received '%s'\n", buf);
close(sockfd);
Difference in abstraction

UDP
 Recv(len)
call gives you a packet (the payload part)
len
len
UDP segment (packet)

UDP segment (packet)
TCP
establishment (hand-shake)  Will see in the TA led session.
Application
 Recv(len) call give you a sequence of bytes
send()
 Connection
len
Application
Recv()
len
Byte stream
Byte stream
Incoming stream
TCP– What goes on underneath
Byte stream
Packet (1~7)
Packet (8~10)
Packet (9~11)
Receive buffer
Packet (8~10)
I need seq# 8.
User Datagram Protocol(UDP):
An Analogy
UDP





Postal Mail
Single socket to receive messages
No guarantee of delivery
Not necessarily in-order delivery
Datagram – independent packets
Must address each packet





Single mailbox to receive letters
Unreliable 
Not necessarily in-order delivery
Letters sent independently
Must address each reply
Example UDP applications
Multimedia, voice over IP
22
Transmission Control Protocol
(TCP): An Analogy
Telephone Call
TCP




Reliable – guarantee delivery
Byte stream – in-order delivery
Connection-oriented – single socket per
connection
Setup connection followed by data
transfer




Guaranteed delivery
In-order delivery
Connection-oriented
Setup connection followed by
conversation
Example TCP applications
Web, Email, Telnet
REVISION CONTROL & MAKE
Overview

Revision control systems
 Motivation
 Features
 Subversion

primer
Make
 Simple
gcc
 Make basics and variables
 Testing

Useful Unix commands
Dealing with large codebases

Complications
 Code
synchronization
 Backups
 Concurrent modifications

Solution: Revision Control System (RCS)
 Store
all of your code in a repository on a server
 Metadata to track changes, revisions, etc…
 Also useful for writing books, papers, etc…
Basic RCS features (1/2)

Infinite undo
 go

Automatic backups
 all

back to previous revisions
your code ever, forever saved
Changes tracking
 see
differences between revisions
Basic RCS features (2/2)

Concurrency
 Multiple

Snapshots
 Save

people working at the same time
stable versions on the side (i.e. handin)
Branching
 Create
diverging code paths for experimentation
Typical RCS workflow
1.
2.
3.
4.
5.
6.
Create repository on RCS server
Checkout the repository to your local machine
Work locally: create, delete and modify files
Update the repository to check for changes other people might ha
ve made
Resolve any existing conflicts
Commit your changes to the repository
RCS implementations

Concurrent Versions System (CVS)


Concurrency, versioning of single files
Subversion (SVN)

File and directory moving and renaming, atomic commits
Git


Modern, de-facto standard
Concurrent edits (1/2)
John
File v1
David
Both check out file
Edits (lines 30-40)
Edits (lines 1-20)
commit
File v2
Update
Merged automatically!
Concurrent edits (2/2)
John
File v1
David
Both check out file
Edits (lines 10-20)
Edits (lines 1-20)
commit
File v2
Update
Conflict needing manual inte
rvention!
Resolving conflicts

When changes can’t be merged automatically

SVN gives you 3 files:




file.mine : your file
file.rx : the remote file
file : original file with marked conflicts
You can



Discard your changes
Discard others’ changes
Go over each conflict and arbitrate as appropriate
Interacting with SVN

Command line


Readily available on your lab machines
Graphical tools



Easier to use
Subclipse for Eclipse gives IDE/SVN integration
tortoiseSVN
실습 server



IP: 143.248.141.40, 143.248.141.42~143.248.141.44
ID: ee<your student id> (e.g., ee20110111)
Password: ee324EE#@$
Svn password


20100456 63900
20100667 26834
20100967 48013
20110198 810
20110227 39617
20110319 16411
20110468 57700
20110548 64302
20110968 30860
20120184 19424
20120277 12976
20120508 17278
20120677 40029
20120973 61034
20121092 42821
20130098 42807
20130185 30791
20130234 13448
20130363 14421
20130394 20796
20130451 13008
20130477 53705
20130478 45122
20130505 4805
20130842 742
20140055 20389
20140244 45336
20140544 23793
20153347 5044
Command line SVN example (1/2)
$ svn co –-username=<<student id>> https://ina.kaist.ac.kr/svn/test
$ cd test
$ vim <<student id>>.txt
$ vim <<student id>>.
$ svn add <<student id>>.txt
$ svn commit -m 'adding a text file with my student id‘
$ svn update
$ cd ../
$ svn cp trunk tags/<<student id>>_submission
$ svn commit -m 'making a tag of the trunk for submission!‘
$ svn up
$ echo -e ”<<student id>>" >> student.txt
$ svn commit –m “added my student id”
Command line SVN example (2/2)
Revision control lets you note (and then see) what you changed:
> svn log student.txt
r986 | ntolia | 2006-08-01 17:13:38 -0400 (Tue, 01 Aug 2006) | 6 lines
This allows the sp to get rid of chunks early before a transfer is complete.
Useful when a file is requested in-order and the file size > mem cache size
> svn diff -r 1:2 student.txt
Index: file
===================================================================
--- file (revision 1)
+++ file (revision 2)
@@-1,2+1,3@@
This isatestfile
-It startedwithtwolines
+It nolongerhastwolines
+it hasthree
General SVN tips





Update, make, test, only then commit
Merge often (svn up)
Comment commits
Avoid commit races
Modular design avoids conflicts
Know more


subversion.tigris.org for SVN software & info
svnbook.red-bean.com for SVN book
Makefile

You are on your own.
Make



Utility for executable building automation
Saves you time and frustration
Helps you test more and better
Simple gcc
If we have files:
• prog.c: The main program file
• lib.c: Library .c file
• lib.h: Library header file
% gcc -c prog.c -o prog.o
% gcc -c lib.c -o lib.o
% gcc lib.o prog.o -o binary
gcc flags
• Useful flags
1. -g: debugging hook
2. -Wall: show all warnings
3. -Werror: treat warning as errors
4. -O0, -O1, -O2, -O3: optimization level
5. -DDEBUG: macro for DEBUG (#define DEBUG)
• Avoid using dangerous optimizations that could affe
ct correctness
More gcc
% gcc -g -Wall -Werror -c prog.c -o prog.o
% gcc -g -Wall -Werror -c lib.c -o lib.o
% gcc -g -Wall -Werror lib.o prog.o -o binary
This gets boring, fast!
Makefile basics
• Build targets
target: dependency1 dependency2 ...
unix command (start line with TAB)
unix command
Makefile example
binary: lib.o prog.o
gcc -g -Wall lib.o prog.o -o binary
lib.o: lib.c
gcc -g -Wall -c lib.c -o lib.o
prog.o: prog.c
gcc -g -Wall -c prog.c -o prog.o
clean:
rm *.o binary
Makefile variables (1/7)
• Variables
CC = gcc
CFLAGS = -g -Wall -Werror
OUTPUT = binary
Makefile variables (2/7)
binary: lib.o prog.o
gcc -g -Wall lib.o prog.o -o binary
lib.o: lib.c
gcc -g -Wall -c lib.c -o lib.o
prog.o: prog.c
gcc -g -Wall -c prog.c -o prog.o
clean:
rm *.o binary
Makefile variables (3/7)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
$(OUTPUT): lib.o prog.o
$(CC) $(CFLAGS) lib.o prog.o -o binary
lib.o: lib.c
$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c
$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:
rm *.o $(OUTPUT)
Makefile variables (4/7)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
$(OUTPUT): lib.o prog.o
$(CC) $(CFLAGS) lib.o prog.o -o binary
lib.o: lib.c
$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c
$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:
rm *.o $(OUTPUT)
Makefile variables (5/7)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
OBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)
$(CC) $(CFLAGS) $(OBJFILES) -o binary
lib.o: lib.c
$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c
$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:
rm *.o $(OUTPUT)
Makefile variables (6/7)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
OBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)
$(CC) $(CFLAGS) $(OBJFILES) -o binary
lib.o: lib.c
$(CC) $(CFLAGS) -c lib.c -o lib.o
prog.o: prog.c
$(CC) $(CFLAGS) -c prog.c -o prog.o
clean:
rm *.o $(OUTPUT)
Makefile variables (7/7)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
OBJFILES = lib.o prog.o
$(OUTPUT): $(OBJFILES)
$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c
# $<: dependency (%.c)
# $@: target (%.o)
$(CC) $(CFLAGS) -c $< -o $@
clean:
rm *.o $(OUTPUT)
Simple Test Script (1/2)
% ./server 6667 &
% cat testfile.01 | ./testscript.py
% cat testfile.02 | ./testscript.py
% killall -9 server
Simple Test Script (2/2)
#/bin/sh
echo “Starting server on port 6667.”
./server 6667 &
SERVERPID = $!
echo “Running test files.”
cat testfile.01 | ./testscript.py
cat testfile.02 | ./testscript.py
echo “Killing server process.”
kill $(SERVERPID)
Augmenting the Makefile for testing (1/2)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
OBJFILES = lib.o prog.o
all: $(OUTPUT)
$(OUTPUT): $(OBJFILES)
$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c
# $<: dependencies (%.c)
# $@: target (%.o)
$(CC) $(CFLAGS) -c $< -o $@
clean:
rm *.o $(OUTPUT)
Augmenting the Makefile for testing (2/2)
CC = gcc
CFLAGS = -g -Wall
OUTPUT = binary
OBJFILES = lib.o prog.o
all: $(OUTPUT)
test
$(OUTPUT): $(OBJFILES)
$(CC) $(CFLAGS) $(OBJFILES) -o binary
%.o: %.c
# $<: dependencies (%.c)
# $@: target (%.o)
$(CC) $(CFLAGS) -c $< -o $@
test: $(OUTPUT)
sh ./testscript.sh
clean:
rm *.o $(OUTPUT)
Using the Makefile
% make
% make test
% make clean
• Know more
Google
– “makefile example”
– “makefile template”
– “make tutorial”
– Chapter 3 of Dave Andersen’s notes
Extra: useful Unix commands (1/2)
• find “func_name” in files
% grep -r func_name .
• replace “bad_func_name” to “good_func_n
ame”
% sed -e “s/bad_func_name/good_func_name/g”\
prog.c > prog.c.new
Extra: useful Unix commands (2/2)
• find a file named “prog.c”
% find -name prog.c
• download files from the Internet
% wget http://address/to/file.tar.gz
• untar and unzip a file
% tar xzvf file.tar.gz
Assignment




You have access to svn repository at
https://ina.kaist.ac.kr/svn/<student id>
However, we will change your password (to some unknown integer).
Luckily, we made a server that gives you a hint.
When your client asks whether the username and the password match,
the server can tell you whether the password provided is less than,
equal to, or greater than your real password.
Assignment I

Protocol is defined in the document. (See course web page).
Query
Server (provided by the instructor)
Client
Response


Part I: Assume the network is reliable. (due 9/14)
Part II: The server may drop a packet. (due 9/24)