Transcript Chapter 1: Computers: A First Look - CS

Chapter 9:
Multimedia
Most of the outside information entering our
brains, enters through our eyes or our ears. They
are, indeed, input devices through which our world
communicates with us.
The Computer Continuum
9-1
Multimedia

In this chapter:
•
•
•
•
•
•
•
What is multimedia?
How are pictures stored within the computer?
What can be done with digitized pictures?
How can pictures be created using a computer and software?
What is the importance of vocal communication?
How is speech stored and played back by the computer?
What are some problems the computer has when analyzing
speech?
• What areas are computers used in audio communication?
The Computer Continuum
9-2
What is Multimedia?

Multimedia has become interactive
• Interactive multimedia: The use of media such as text,
graphics, animation, video, and audio in an interactive way
that allows a participant to control it.
• To understand the value of interactive multimedia, we will
examine visual and audio concepts in detail.
The Computer Continuum
9-3
Visual Media:
Manipulating Images

Previously created images (Modifying or processing
images that already exist)
• Three types of techniques used to manipulate images:
– Minor Processing Techniques
– Enhancement
– Restoration
The Computer Continuum
9-4
Visual Media:
Manipulating Images

Common example of
digitized images: Satellite
weather maps.
• GEOS-7 satellite picture
taken on Aug. 24, 1992.
• Outlines created by a
computer program.
• Program color coded the
picture to give information
about the hurricane’s wind
speed.
The Computer Continuum
9-5
Visual Media:
Manipulating Images

Landsat 4 Satellite:
• Uses 3 of the 7 frequencies
available from the Thematic
Mapper of the satellite.
• Like having a camera with
different filters, each letting
a different frequency
through (including infrared).
The Computer Continuum
9-6
Visual Media:
Manipulating Images

Image enhancement
• A type of digital image
processing whose goal is to
highlight or enhance
particular aspects of an
image or change an image’s
structure.
• Common example of image
enhancement: Falsecoloring: An image
enhancement technique that
consists of changing the
colors of an image or
assigning colors to various
parts of an image.
The Computer Continuum
9-7
Visual Media:
Manipulating Images

Image restoration
• A type of digital image
processing whose goal is to
eliminate known, but
unwanted image flaws or
degradations.
The Computer Continuum
9-8
Visual Media:
Manipulating Images

Image enhancement example:
• It makes you wonder if
photographs can be trusted.
The Computer Continuum
9-9
Visual Media:
Creating Images

Artists follow rules (or intentionally don’t follow rules)
to create original images:
• Hiding parts of things that wouldn’t normally be visible.
• Sources of light that illuminate the objects.

Creating images using the computer
• Line images.
• Solid forms to 3D.
• Animation and video.
The Computer Continuum
9-10
Visual Media:
Creating Images

Creating Line Images
• What are some uses of line drawings?
– Map making.
– Architectural drawings.
– Business graphs.
– Plans for building machinery, aircraft, furniture-in-a-box.
– City population density maps.
• How can line images be created?
– Using drawing programs such as Adobe Illustrator.
• Have drawing related features including geometric
shapes, free-hand tools, fills, patterns, text.
The Computer Continuum
9-11
Visual Media:
Creating Images

Bit-mapped or Raster graphical
images
•
•
•
Constructed of individual pixels.
Only crude manipulation can be
done (modification of each pixel).
Retains “bit-mapped” appearance.

Object-oriented or Vector
graphic images
•
•
•
Stored in the computer as lines,
curves, or geometric shapes.
Formulas are used to draw
(circles, twice the size, are still
circles!)
Objects can be moved or
modified easily.
The Computer Continuum
9-12
Visual Media:
Creating Images

Designing 3-dimensional pictures drawn in 2dimensions
•
•
•
•
Curved surfaces
Color
Texture
Shading
The Computer Continuum
9-13
Visual Media:
Creating Images

One of the classic problems in computer
graphics:
• Hidden-line problem: Concerns
itself with how to hide the outlines
or surfaces of a solid object that
shouldn’t be seen from the direction
of the observation.
• Example: Studies of an airline
pilot in action.
The Computer Continuum
9-14
Visual Media:
Creating Images

Perspective: The quality that allows 3-dimensional images to be
drawn on 2-dimensional surfaces and yet retain the look of a 3dimensional image.
The Computer Continuum
9-15
Visual Media:
Creating Images

Shading: A technique used to give the appearance of illumination
by some combination of light sources.
The architecture program used to draw this building has a 3D description
of the building It can produce pictures from any angle.
The Computer Continuum
9-16
Visual Media:
Creating Images

Shading flat surfaces of the head (as in b) then blending the
surfaces to give a more natural look (as in c).
a
b
The Computer Continuum
c
9-17
Visual Media:
Creating Images

Texture: A property of a surface. It is observed and identified
through the reflection of the light off the surface.
The Computer Continuum
9-18
Visual Media:
Animation and Video

Animation:
• Early animation started with pictures flipping hand-cranked
players.
• Animation 30 years ago:
– Cartoons had to have the figures drawn by artists.
– The animator would make them move while taking pictures,
frame by frame.
• Animation now:
– Computers are used to create full length feature cartoons.
• Toy Story
• Individual frames stored on the computer.
• Special player programs can be used to view them on the
computer screen or recorded on videotape using special
hardware and software.
The Computer Continuum
9-19
Visual Media:
Animation and Video

Jurassic Park dinosaurs
• Wire frames were created in the computer to depict dinosaurs.
• Each wire frame is moved by the computer which calculates
positions of each part viewed from a certain angle and
distance.
• The computer covered the surfaces of the wire frames with a
texture layer.
– Computer takes into account sources of light illuminating
the figure.
The Computer Continuum
9-20
Visual Media:
Animation and Video

Morphing: A
graphics technique
used in animation
• A beginning image
will distort and
change in a
predetermined
number of frames
into a final image.
The Computer Continuum
9-21
Visual Media:
Animation and Video

Digitizing and Manipulating Video
• Digital video hardware systems: Systems capable of capturing
and manipulating video.
• Analog video uses standards developed over 50 years ago.
– Each differs in the number of horizontal lines that make
up a single video frame and in the number of frames per
second that are displayed on the TV set.
• European broadcast standards (PAL).
• United States broadcast standards (NTSC).
» 30 frames per second.
» VHS.
The Computer Continuum
9-22
Visual Media:
Animation and Video

Video capture: The process of digitizing an analog TV signal.
• Done using hardware that produces a digital form of the
analog TV signal.
– Uses the VCR to feed the video signal into a digital
hardware card that plugs into the computer.
– Each frame is converted into a bitmapped image.
– The digital video signal is compressed.
• Eliminates data that our eye doesn’t see or that is
hardly noticed.
• Codec: A scheme for coding & decoding large
amounts of data. (The data is economical and
uncomplicated to store.)
The Computer Continuum
9-23
Visual Media:
Animation and Video

Common codecs:
• Motion Picture Experts Group (MPEG)
– MPEG-1
•
•
•
•
Developed to handle slower media such as videoCDs.
Deteriorates when high-speed action occurs.
Roughly the quality of VHS.
70 minutes with sound fits on a CD-ROM disc.
– MPEG-2
• Full-screen codec standard that brings twice the resolution
of VHS.
• Four times the resolution of MPEG-1.
• Optimized for higher demands of broadcast, satellite, and
DVD.
The Computer Continuum
9-24
Visual Media:
Animation and Video

Common codecs: (continued)
• Motion Picture Experts Group (MPEG)
– M-JPEG
• Requires more memory space than MPEG-2
• Each frame is digitized separately using JPEG compression.
» Makes it easier to edit.
• As we move from analog to digital broadcast, coding and decoding
will become unnecessary. (Everything will already be digital!)
The Computer Continuum
9-25
Visual Media:
Animation and Video

DVD: The Digital Versatile Disc
• DVD: An optical disk storage technology that looks like the
standard CD, but is faster and has a much larger storage
capacity.
– Two forms:
• DVD-Video Discs: Intended as a replacement of
VHS.
• DVD-ROM: Intended as a computer storage medium.
– Digital Video Players:
• Software products are available that will play back
video coded with one of these codec systems.
» QuickTime, ActiveMovie
The Computer Continuum
9-26
Visual Media:
Animation and Video

Streaming media: Information in one of many different visual or
audio forms that is sent from a server in packets to the requesting
computer.
• Contains parts of the medium that may be a recording or an
actual live event.
• The information is played back in the order sent, starting soon
after a few packets arrive, but before all of them have been
sent.
The Computer Continuum
9-27
Audio Media:
Human Speech

The importance of vocal communication:
• It is the fastest method of communication.
• Most communication is vocal.
• Researchers believe that human intelligence is closely linked
to the development of vocal communication and language.
The Computer Continuum
9-28
Audio Media:
Human Speech

Creating Speech: Recorded Speech
• Recording and digitizing individual words and phrases.
– Common uses:
• Talking cash registers.
• Automated national telephone information service.
– Problems with recorded speech:
• Creating whole sentences of human speech would
take a huge amount of computer memory.
• Several versions of each word would need to be
digitized to account for pitch and volume levels (for
punctuation and variations of the same word).
The Computer Continuum
9-29
Audio Media:
Human Speech

Creating Speech: Speech Synthesis
• Speech synthesis: The electronic production of sounds and
sound patterns that closely resemble human speech.
– Uses phonemes: The fundamental sounds of any given
human language.
– Three additional factors that have an affect on how a word
or phrase sounds:
• Inflection - The rising and falling pattern of pitch.
• Duration - The time spent on or the time between
individual phonemes.
• Elision - Splicing phonemes together so that when
one ends, the other begins. The connection will sound
natural.
The Computer Continuum
9-30
Audio Media:
Human Speech

Creating Speech: Speech Synthesis
• The sounds produced are electronically generated.
– The sounds simulate those produced by the human vocal
tract.
• Can produce both major types of human sounds:
» Voiced sounds: Sounds produced by the
vibration of vocal cords in conjunction with
specific positioning of teeth, tongue, and lips.
Include all vowels and some consonants.(d, g, b)
» Voiceless sounds: Characterized by the lack of
vibration of the vocal cords. (s, k, t)
The Computer Continuum
9-31
Audio Media:
Human Speech

Recognizing Spoken Words
• More difficult for the computer than speech synthesis.
• Voiceprint: A visual plot of frequency versus time of sound
produced by a human speaker.
– No two voiceprints are completely identical.
• Human voice is determined at least by:
» Pitch: The number of cycles per second of a particular
sound’s vibration.
» Resonance: The reverberation or amplification of the
voice in the cavities of the vocal tract.
• Problem: Because each individual has uniquely shaped vocal
cords and resonating chambers, each has a uniquely individual
voice.
The Computer Continuum
9-32
Audio Media:
Human Speech

Recognizing Speech
• Disjointed speech: Words spoken one at a time with silence
between words.
– Software programs are available for home use.
– Each person goes through a recording session to train it.
– Total vocabulary to fewer than 1000 words spoken by an
individual.
• Problems:
– Speaking in a disjointed manner is unnatural.
– Any changes in the person’s voice (head cold, sinus
infection) may make the speech unrecognizable.
The Computer Continuum
9-33
Audio Media:
Human Speech

Recognizing Speech
• Continuous speech: Words spoken in a continuous stream of
sounds, usually with no pauses between words.
– More difficult for the computer to understand. Computer
has difficulty when one word ends and next begins.
The Computer Continuum
9-34
Audio Media:
Music

The Computer and Recorded
Music
• Compact Disc Technology
– CDs don’t wear out like
records and magnetic tape.
• After numbers are
recorded, they don’t
change.
• Scratching the surface
can affect quality of
sound.
– Compact discs can be
produced from a master
disc very inexpensively.
The Computer Continuum
9-35
Audio Media:
Music

The Computer and Recorded Music
• DAT (Digital Audio Tape) technology
– Advantages of DAT:
• Binary information is recorded onto magnetic tape.
• DAT can easily record sound information.
• Perfect duplication of CDs can be made onto tape.
The Computer Continuum
9-36
Audio Media:
Music

Recorded Audio Files and Formats
• Individual standards have been developed for special purposes
or for specific computers.
– Sound files can be huge. Sound quality affects size.
– The four most common sound file formats:
WAV: Originally developed with Windows 3.1
AU: Developed by SunAudio for UNIX systems but
now supported by Windows and Macintosh.
AIFF: Originated on the Macintosh computer.
MP3: (MPEG - level 3) Can be played on Windows,
Macintosh, and UNIX systems.
The Computer Continuum
9-37
Audio Media:
Music

Hardware and Software Needs
• Playing back Web sound
– Need a sound card.
• Newer “multimedia” computers have sound cards
included.
– Need external speakers.
• Receiving sound files from the Web.
– Must be a type of file your browser recognizes.
• Most new browsers handle Web sound.
The Computer Continuum
9-38
Audio Media:
Music

MIDI: A Revolution in Music
• MIDI (Musical Instrument Digital Interface): Musical
standard created in 1982 that makes possible communication
of audio information
– Between keyboards, drum pads, etc.
– To a computer-controlled mixer.
– Each instrument connected to a MIDI bus usually has a
small special-purpose computer with some memory.
• MIDI bus: The main communications channel for
MIDI information that can consist of signals sent to
as many as 16 different instruments or groups of
instruments.
The Computer Continuum
9-39
Audio Media:
Music

All MIDI instruments have three-pronged connectors
that are used to connect other MIDI instruments or
computers.
• Thru port: Sends MIDI signals obtained from the IN port on to other
MIDI instruments.
• OUT port: Sends out MIDI signals.
• IN port: Receives MIDI signals.
• The three connectors necessary for a MIDI interface:
The Computer Continuum
9-40
Audio Media:
Music

Several ways to use a MIDI instrument:
• By itself.
– A MIDI synthesizer can connect to an amplifier with
speakers.
• Using one MIDI instrument as a master to a second, acting as
a slave.
– Can not only play its own notes, but can also control the
slave’s notes.
• In conjunction with a MIDI sequencer.
– MIDI sequencer: A device, usually a computer, that
records and plays back MIDI signals.
– Can be either a computer with sequencing software or a
dedicated sequencer.
The Computer Continuum
9-41
Internet Music and Audio Net

Three techniques for both downloading sound to your
computer and listening to live audio over the WWW.
1. Streaming audio: A request to hear music is made by clicking
on a button. The Website sends back the sound which you
hear immediately. You don’t receive a copy of the sound. (You
may need to add a plug-in to hear the sound.)
– Plug-in: A piece of software that is added to your browser
to give it desired capability.
2. Sound file is sent over the WWW. It allows you to store the
sound file on your computer.
3. MIDI files: Created by a performer on a MIDI instrument.
Needs a MIDI player (available on the Internet.)
The Computer Continuum
9-42