Transcript Chapter Eight : File Management

Chapter Eight : File Management
•
•
•
•
•
•
•
The File Manager
Interacting With File Manager
File Organization
Physical Storage Allocation
Data Compression
Access Methods
Levels in File Management
System
• Access Control Verification
Module
Fixed Length
Records
Contiguous
Storage
Non-contiguous
Storage
Variable Length
Records
Indexed
Storage
Sequential or Direct File Access
Understanding
Operating Systems
1
The File Manager
• File Manager controls every file in system which is a
complex job.
• Efficiency depends on:
– how system’s files are organized (sequential, direct, or
indexed sequential).
– how they’re stored (contiguously, noncontiguously, or
indexed).
– how each file’s records are structured (fixed-length or
variable-length).
– how access to these files is controlled .
Understanding
Operating Systems
2
Responsibilities of File Manager
1. Track where each file is stored.
2. Determine where and how files will be stored.
– Efficiently use available storage space.
– Provide efficient access to files.
3. Allocate each file when a user has been cleared for access
to it, then record its use.
4. Deallocate file when it is returned to storage.
– Communicate its availability to others waiting for it.
Understanding
Operating Systems
3
Important Definitions
• Field -- group of related bytes that can be identified by
user with name, type, and size.
• Record -- group of related fields.
• File (flat file) -- group of related records that contains info
used by specific application programs to generate reports.
• Database -- groups of related files that are interconnected
at various levels to give flexible access to users.
– Appears to File Manager to be a type of file.
Understanding
Operating Systems
4
Definitions - 2
• Program files contain instructions.
• Data files contain data.
• Directories -- listings of file names and their attributes.
• Every program and data file accessed by computer system,
and every piece of computer software, is treated as a file.
• File Manager treats all files exactly same way as far as
storage is concerned.
Understanding
Operating Systems
5
Interacting With File Manager
• Users communicates with File Manager via specific
commands that may be either embedded in user’s program
or submitted interactively by user.
• Embedded commands:
– OPEN & CLOSE pertain to availability of file for
program invoking it.
– READ & WRITE are I/O commands.
– MODIFY – specialized WRITE command for existing
data files that allows for appending/rewriting records.
Understanding
Operating Systems
6
Interactive Commands
• CREATE & DELETE -- deal with system’s knowledge of
file.
• SAVE -- first time used, a file is actually created.
• OPEN NEW -- within a program indicates file must be
created.
• OPEN…FOR OUTPUT -- creates file by making entry for
it in directory & finding space for it in secondary storage.
• RENAME -- allows users to change name of existing file.
• COPY – allows user to make duplicate copies of existing
files.
Understanding
Operating Systems
7
Commands Are Device-Independent
• Interface commands designed to be as simple as possible
to use.
– Lack detailed instructions to run device where file is stored.
– Device independent.
• To access a file, user doesn’t need to know its exact
physical location on disk pack or storage medium.
• Each logical command broken down into sequence of lowlevel signals that
– Trigger step-by-step actions performed by device.
– Supervise progress of operation by testing device’s status.
Understanding
Operating Systems
8
Typical Volume Configuration
• Each secondary storage unit (removable or non-removable)
is considered a volume.
– Each volume can contain several files called multifile
volumes.
– Some files are extremely large and are contained in
several volumes called multivolume files.
• Generally, each volume in system is given name.
– File Manager writes name & other descriptive info on
easy-to-access place on each unit.
Understanding
Operating Systems
9
Master File Directory (MFD)
• MFD stored immediately after volume descriptor
– Lists names & characteristics of every file contained in volume.
– File names refer to program files, data files, and/or system files.
– Subdirectories, if supported.
– Remainder of volume is used for file storage.
• Early OS supported only a single directory per volume.
– Created by File Manager.
– Contains names of files, usually organized in alphabetical, spatial,
or chronological order.
– Simple to implement and maintain.
– Some major disadvantages
Understanding
Operating Systems
10
Volume Descriptor
Creation Date
Date when volume was created
Pointer to Directory Area
Indicates first sector where directory
is stored
Pointer to File Area
Indicates first sector where file is
stored
File System Code
Used to detect volumes with incorrect
formats
Volume News
User-allocated name
11
Some Major Disadvantages of Single
Directory Per Volume
1. Takes long time to search for an individual file, especially
if MFD was organized in an arbitrary order.
2. If user has many small files stored in volume, directory
space fills before disk storage space fills. User told “disk
full” when only directory full.
3. Users can’t create subdirectories to group related files.
4. Multiple users can’t safeguard files from other users
browsing file lists ‘cause entire directory listed on request.
5. Each program in entire directory needs unique name.
• E.g., Only 1 person using directory can name program
PROG1.
Understanding
Operating Systems
12
About Subdirectories
• Semi-sophisticated File Managers create MFD for each
volume with entries for files & subdirectories.
• Subdirectory created when user opens account to access
computer.
– MFD entry flagged to indicate subdirectory with unique
properties.
• Improvement from single directory scheme.
• Still can’t group files in a logical order to improve
accessibility & efficiency of system.
Understanding
Operating Systems
13
Subdirectories Can Be Implemented As an
Upside-down Tree
• Today’s File Managers allow users to create subdirectories
so related files are grouped together.
– Extension of previous two-level directory structure.
• Tree structures allow system to efficiently search
individual directories due to fewer entries in each.
• Path to requested file may lead through several directories.
• When user wants to access specific file, file name is sent to
File Manager. File Manager searches MFD for user's
directory. Then searches user's directory & any
subdirectories for requested file & location.
Understanding
Operating Systems
14
File Descriptor
Each file entry in every directory contains info describing file:
1. File name—usually represented in ASCII code.
2. File type—organization and usage that are dependent on system (e.g.,
Files and directories).
3. File size—size is kept here for convenience.
4. File location—identification of first physical block (or all blocks)
where file is stored.
5. Date and time of creation.
6. Owner.
7. Protection information—access restrictions based on who is allowed
to access file and what type of access is allowed.
8. Record size —its fixed size or its maximum size, depending on type
of record
Understanding
Operating Systems
15
File Names
• Absolute file name (complete file name) – long name
that includes all path info.
• Relative file name – short name seen in directory listings.
– Selected by user when file is created.
– E.g., ACCOUNT ADDRESSES, TAXES 2001, or AUTOEXEC.
• Extension – 2-3 character name used to identify type of
file or its contents.
– Separated from relative name by a period.
– E.g., CPP, BAS, BAT, COB, & EXE signal to system to use
specific compiler or program to run these files.
– E.g., TXT, DOC, OUT, MIC, & KEY created by applications or
by users for own identification.
Understanding
Operating Systems
16
File Naming Conventions
• Can vary in length from 1 or more characters.
• Can include letters of alphabet & digits.
• Every OS has specific rules that affect length of relative
name & types of characters allowed.
– E.g., MS-DOS allows 1-8 alphanumeric character
names without spaces.
– More modern OS allow names with dozens of
characters including spaces.
• Try to select descriptive relative names that readily identify
file contents/purpose of file.
Understanding
Operating Systems
17
Base and Current Directories Used by File
Manager to Locate Files
• File Manager selects base directory for user when
interactive session begins.
– All file operations requested by that user start here.
• Then, user selects subdirectory (current directory or
working directory).
– Thereafter, files presumed to be located in current directory.
• Whenever file accessed, user types in relative name & File
Manager adds proper prefix.
• As long as users refer to files in working directory, can
access them without entering complete name.
Understanding
Operating Systems
18
File Organization : Record Format
1. Fixed-length records – easiest to access directly.
– Most common type & ideal for data files.
– Record size critical (too small – truncation; too large – wastes
space).
2. Variable-length records -- difficult to access directly because hard to
calculate exactly where record is located.
– Don’t leave empty storage space & don’t truncate any characters.
– Frequently used in files accessed sequentially (e.g,. text files,
program files) or files using index to access records.
– File descriptor stores record format, how it’s blocked, & other
related info.
Understanding
Operating Systems
19
Physical File Organization
•
Concerned with how records are arranged &
characteristics of medium used to store it.
•
On magnetic disks, files can be organized as:
1. Sequential
2. Direct
3. Indexed sequential.
Understanding
Operating Systems
20
Characteristics Considered When
Selecting File Organization
• Volatility of data—frequency with which additions &
deletions made.
• Activity of file—% records processed during a given run.
• Size of file.
• Response time—amount of time user is willing to wait
before requested operation is completed.
Understanding
Operating Systems
21
Sequential Record Organization
• Easiest to implement because records are stored &
retrieved serially, one after other.
• To speed process some optimization features may be built
into system.
– E.g., select a key field from record & then sort records
by that field before storing them.
– Aids search process.
– Complicates maintenance algorithms because original
order must be preserved every time records added or
deleted.
Understanding
Operating Systems
22
Direct Record Organization
(Random Organization)
• Uses direct access files which can be implemented only on
direct access storage devices.
• Give users flexibility of accessing any record in any order
without having to begin search from beginning of file.
• Records are identified by their relative addresses (their
addresses relative to beginning of file).
– Logical addresses computed when records are stored
& again when records are retrieved.
– Use hashing algorithms.
Understanding
Operating Systems
23
Advantages of Direct Access Organization
• Fast access to records.
• Can be accessed sequentially by starting at first relative
address & incrementing it by one to get to next record.
• Can be updated more quickly than sequential files because
records quickly rewritten to original addresses after
modifications.
• No need to preserve order of the records, so adding or
deleting them takes very little time.
Understanding
Operating Systems
24
Collisions Are a Problem With Direct
Access Organization
• Several records with unique keys may generate same
logical address (collision).
• Program generates another logical address before
presenting it to File Manager for storage.
• Colliding records stored in overflow area via links.
• File Manager handles physical allocation of space.
• Maximum file size established when created & eventually
file is full or too many records are stored in overflow area.
• Programmer must reorganize & rewrite file.
Understanding
Operating Systems
25
Indexed Sequential Record Organization
• Combines best of sequential & direct access.
• Created & maintained through Indexed Sequential Access
Method (ISAM) software package.
• Doesn’t create collisions because it doesn’t use result of
hashing algorithm to generate a record’s address.
– Uses info to generate index file through which records retrieved.
• Divides ordered sequential file into blocks of equal size.
– Size determined by File Manager to take advantage of physical
storage devices & to optimize retrieval strategies.
• Each entry in index file contains highest record key &
physical location of data block where this record, &
records with smaller keys, are stored.
Understanding
Operating Systems
26
Indexed Sequential - 2
• To access any record in file, system begins by searching index file &
then goes to physical location indicated at that entry.
• Overflow areas are spread throughout file
– Existing records can expand & new records are in close physical &
logical sequence.
– Last-resort overflow area is located apart from main data area but
is used only when the other overflow areas are completely filled.
• When retrieval time becomes too slow, file has to be reorganized..
• Allows both direct access to a few requested records & sequential
access to many records for most dynamic files.
• A variation of indexed sequential files is B-tree.
Understanding
Operating Systems
27
Physical Storage Allocation
• File Manager must work with files not just as whole units
but also as logical units or records.
• Records within file must have same format but can vary in
length.
• Records are subdivided into fields.
– Structure usually managed by application programs, not
OS.
• When we talk about file storage, we’re actually referring to
record storage .
Understanding
Operating Systems
28
R1
R2
Block #
1
Recs
R3
R4
R5
R6
(a) Unblocked, fixed-length
records
R1
R2
R3
Block
2
(b) Blocked, fixed length
records
R1
R1
Length
R1
R2
R2
Length
#
R2
Block Block #
R1
1
Size
Recs. Len.
Understanding
Operating Systems
(c) Unblocked, variablelength records
#
R1
R3
R2
Len.
(d) Unblocked, variablelength records
R2
(e) Blocked, variable-length
records
29
Contiguous Storage
• Records stored one after other.
• Any record can be found & read once starting address &
size are known, so directory is very streamlined.
• Direct access easy – every part of file is stored in same
compact area.
• Files can’t be expanded unless there’s empty space
available immediately following it.
– Room for expansion must be provided when file is created.
• Fragmentation occurs (slivers of unused storage space).
– Can compact & rearrange files.
– Files can’t be accessed while compaction is taking place.
Understanding
Operating Systems
30
Noncontiguous Storage
• Allows files to use any storage space available on disk.
• File’s records are stored in a contiguous manner if enough
empty space.
• Any remaining records, & all other additions to file, are
stored in other sections of disk (extents).
– Linked together with pointers.
– Physical size of each extent is determined by OS (e.g.,
256 bytes).
Understanding
Operating Systems
31
Linking File Extents
1.
2.
•
•
Linking at storage level – each extent points to next one in
sequence.
– Directory entry consists of file name, storage location of first
extent, location of last extent, & total number of extents, not
counting first.
Linking at directory level – each extent listed with its physical
address, size, & pointer to next extent.
• A null pointer indicates that it's last one.
Eliminate external storage fragmentation & need for compaction.
Don’t support direct access because no easy way to determine exact
location of specific record.
Understanding
Operating Systems
32
Indexed Storage
• Allows direct record access by bringing pointers linking
every extent of that file into index block.
• Every file has its own index block (addresses of each disk
sector that make up the file)
– Lists each entry in same order in which sectors linked .
• When a file is created, pointers in index block set to null.
• As each sector is filled, pointer set to appropriate sector
address.
– Address is removed from empty space list & copied
into its position in index block.
Understanding
Operating Systems
33
Indexed Storage - 2
• Supports both sequential & direct access.
• Doesn’t necessarily improve use of storage space because
each file must have index block.
• For larger files with more entries, several levels of indexes
can be generated.
– To find a desired record, File Manager accesses first
index (highest level), which points to a second index
(lower level), which points to an even lower level index
& eventually to data record.
Understanding
Operating Systems
34
Data Compression
• Several techniques (3) used to save space in files.
• System must be able to distinguish between compressed &
uncompressed data.
• Trade-off: storage space gained, but processing time lost.
1. Records with repeated characters can be abbreviated.
– E.g., fixed-length field with short name & many blank characters;
replaced with variable-length field & special code to indicate #
blanks truncated.
ADAMSbbbbbbbbbb  ADAMSb10
300000000
 3#8
Understanding
Operating Systems
35
Data Compression: Repeated Terms
2. Repeated terms compressed by using symbols to represent
each of most commonly used words in the database.
– E.g., in a university’s student database common words
like student, course, teacher, classroom, grade, &
department could each be represented with single
character.
Understanding
Operating Systems
36
Data Compression : Front-end
Compression
3. Front-end compression used for index compression.
– For example, student database where the students’
names are kept in alphabetical order could be
compressed
Understanding
Operating Systems
Original list
Compressed list
Smith, Betty
Smith, Betty
Smith, Gino
7Gino
Smith, Donald
7Donald
Smithberger, John
5berger, John
Smithbren, Ali
6ren, Ali
Smithco, Rachel
5co, Rachel
Smither, Kevin
5er, Kevin
Smithers, Renny
7s, Renny
Snyder, Katherine
1nyder, Katherine
37
Access Methods
• Access methods dictated by a file’s organization
• Most flexibility is allowed with indexed sequential files
and least with sequential.
– File organized in sequential fashion can support only
sequential access to its records, & these records can be
of fixed or variable length.
– File Manager uses the address of last byte read to
access the next sequential record.
– Current byte address (CBA) must be updated every
time a record is accessed.
Understanding
Operating Systems
38
Sequential Access
• For sequential access of fixed-length records, CBA
updated by incrementing it by record length (RL), which is
constant:
CBA = CBA + RL
• For sequential access of variable-length records, File
Manager adds length of record (RLk) plus number of bytes
used to hold record length (N) to CBA.
CBA = CBA + N + RLk
Understanding
Operating Systems
39
Direct Access & Fixed-Length Records
• If file is organized in direct fashion, accessed easily in
direct or sequential order if have fixed-length records.
• For direct access with fixed length records, CBA
computed directly from record length & desired record
number RN (info provided through READ command)
minus one:
CBA=(RN–1) * RL
Understanding
Operating Systems
40
Direct Access & Variable-Length Records
• Virtually impossible to access a record directly because address of
desired record can’t be easily computed.
• To access a record, File Manager must do sequential search through
records.
– If File Manager saves address of last record accessed, can do halfsequential read through file. When next request arrives it could
search forward from CBA.
– Or File Manager can keep table of record numbers & their CBAs.
Search table for exact storage location of desired record.
• To avoid this problem, many systems force users to have files
organized for fixed-length records if want direct access to records.
Understanding
Operating Systems
41
Access of Records in Indexed Sequential
File
• Accessed either sequentially or directly,
• Either CBA computations apply but with one extra step.
– Index file must be searched for pointer to block where data stored.
– Because index file is smaller, kept in main memory & quick search
to locate block where desired record is located.
– Block retrieved from secondary storage & beginning byte address
of record calculated.
• In systems with several levels of indexing, index at each
level must be searched before computing CBA.
– Entry point to this type of data file is usually through index file.
Understanding
Operating Systems
42
Levels in a File Management System
•
•
•
Efficient management of files
can’t be separated from
efficient management of
devices that house them.
A wide range of functions must
be organized for I/O system to
perform efficiently.
Each level implemented by
using structured & modular
programming techniques,
which also set up a hierarchy.
Understanding
Operating Systems
Basic File System
Access Control Module
Logical File System
Physical File System
Device Interface Module
Device
43
Basic File System
•
Highest level module that passes info to logical file
system, which notifies physical file system, which works
with Device Manager.
•
Activates access control verification module to verify
that this user is permitted to perform this operation with
this file.
Understanding
Operating Systems
44
Access Control Verification Module
• Any file can be shared.
• Saves space & allows for synchronization of data updates.
• Improves efficiency of system's resources, because if files
are shared in main memory, I/O operations reduced.
• However, integrity of each file must be safeguarded
– Control over who is allowed to access file and what
type of access is permitted.
– READ only, WRITE only, EXECUTE only, DELETE
only, or some combination.
Understanding
Operating Systems
45
File Access Control Methods
•
Each file management system has own file access control
method.
1.
2.
3.
4.
Access control matrix
Access control lists
Capability lists
Lockword control.
Understanding
Operating Systems
Most
Common Methods
46
Access Control Matrix
•
•
•
•
Intuitively appealing & easy to implement.
Works well only for systems with few files & few users.
In matrix each column identifies a user & each row identifies a file.
Intersection of row & column has access rights for that user to that file.
User 1
User 2
User 3
User 4
User 5
File 1
RWED
R-E-
----
RWE-
--E-
File 2
----
R-E-
R-E-
--E-
----
File 3
----
RWED
----
--E-
----
File 4
R-E-
----
----
----
RWED
File 5
----
----
----
----
RWED
R = Read Access
W = Write Access
E = Execute Access
D = Delete Access
Understanding
Operating Systems
- = Access Not Allowed
47
Access Control Lists
• Modification of access control matrix technique.
• Each file is entered in list & contains names of users allowed to access
it & type of access permitted.
• To shorten list, only those who may use file are named; those denied
any access are grouped under global heading such as WORLD.
• Or shorten by putting every user into a category:
– SYSTEM – system personnel with unlimited access to all files.
– OWNER – absolute control over all files created in own account.
– GROUP – all users belonging to appropriate group have access.
– WORLD – all other users in system; default access types given by
File Manager.
Understanding
Operating Systems
48
Access Control List Example
File
Access
File 1
USER1 (RWED), USER2 (R-E-), USER4 (RWE-),
USER5 (--E-), WORLD (----)
File 2
USER2(R-E-), USER3 (R-E-), USER4 (--E-), WORLD (---)
File 3
USER2(RWED), USER4 (--E-), WORLD (----)
File 4
USER1(R-E-), USER5(RWED), WORLD(----)
File 5
USER5(RWED), WORLD (----)
Understanding
Operating Systems
49
Capability Lists
• Lists every user and files to which each has access.
• Requires less storage space than an access control matrix.
• Easier to maintain than an access control list when users
are added or deleted from system.
User
Access
User1
File1 (RWED), File4 (R-E-)
User2
File1 (R-E-), File2 (R-E-), File3 (RWED)
User3
File2 (R-E-)
User4
File1 (RWE-), File2 (--E-), File3 (--E-)
User5
File1 (--E-), File4 (RWED), File5 (RWED)
Understanding
Operating Systems
50
Lockword Control
• Lockword is similar to a password but protects a single
file.
– When file created, owner protects it via lockword
– Stored in directory but isn’t revealed with directory listing.
– User must provide correct lockword to access protected file.
• Require smallest amount of storage for file protection.
• Can be guessed by hackers or passed on to unauthorized
users.
• Generally doesn’t control type of access to file.
– Anyone who knows lockword can read, write, execute, or delete
file.
Understanding
Operating Systems
51
Terminology
•
•
•
•
•
•
•
•
•
•
•
•
access control list
access control matrix
capability list
complete file name
current byte address (CBA)
current directory
data compression
data file
database
device independent
direct access files
direct record organization
Understanding
Operating Systems
•
•
•
•
•
•
•
•
directory
extension
extents
file
file descriptor
fixed-length record
hashing algorithm
indexed sequential record
organization
• key field
• lockword
• logical address
52
Terminology - 2
•
•
•
•
•
•
•
•
•
logical address
master file directory (MFD)
relative address
relative file name
sequential record organization
subdirectory
variable-length record
volume
working directory
Understanding
Operating Systems
53