Transcript metadata-harvesting

Metadata Harvesting - OAI-PMH
Thanks to Carl Lagoze, Christoper Gutteridge
The Open Archives Initiative (OAI) and the Protocol for
Metadata Harvesting (OAI-PMH)
Open Archives?
• An open archive is an archive which exports its
metadata in a useful way (OAI-PMH)
• Most also make some or all content available for
free
Origins of the OAI
“The Open Archives Initiative has been
set up to create a forum to discuss and
solve matters of interoperability between
electronic preprint solutions, as a way to
promote their global acceptance. “
(Paul Ginsparg, Rick Luce & Herbert Van de Sompel - 1999)
What is the OAI now?
“The OAI develops and promotes interoperability
standards that aim to facilitate the efficient
dissemination of content.” (from OAI mission statement)
 Technological framework around OAI-PMH protocol
 Application independent
 Independent of economic model for content
Also … a community and a “brand”
Where does the OAI fit?
NSDL
Metadata
Repo.
DLESE Earth Science
Digital Library
DSpace
EPrints
OAI
OAIster
Search
Service
CiteSeer
arXiv
Library of Congress
What is OAI-PMH
• The Open Archives Initiative Protocol for Metadata
Harvesting
• A way of asking an archive about the stuff it’s got in it.
• This allows services to provide searches and other
functionality across the metadata from many archives.
• XML over HTTP
OAI and Open Access
• There is “A” difference
– Open Archives Initiative
– Open Access
• The OAI is not tied to a particular political agenda
- technical focus
• BUT… the OAI provides functionality that is
essential for many Open Access proposals
OAI-PMH
 PMH -> Protocol for Metadata Harvesting
http://www.openarchives.org/OAI/2.0/openarchivesprotocol.htm
• Simple protocol, just 6 verbs
• Designed to allow harvesting of any XML (meta)data (schema described)
• For batch-mode not interactive use
Service
Provider
(Harvester)
Protocol requests (GET, POST)
XML metadata
Data Provider
(Repository)
What Questions can you ask
via OAI-PMH?
•
•
•
•
•
•
Identify
GetRecord
ListIdentifiers
ListMetadataFormats
ListRecords
ListSets
OAI for discovery
R1
R2
User
?
R3
R4
Information islands
OAI for discovery
Service layer
R1
R2
User
Search
service
R3
R4
Metadata harvested by service
OAI for XYZ
Service layer
R1
R2
User
XYZ
service
R3
R4
Global network of resources exposing XML data
OAI-PMH Data Model
resource
item has
identifier
all available metadata
about this sculpture
Dublin Core
metadata
MARC21
metadata
branding
metadata
item
records
record has identifier + metadata format + datestamp
OAI and Metadata Formats
• Protocol based on the notion that a record can be
described in multiple metadata formats
• Dublin Core is required for “interoperability”
• Extended to include XML compound object
formats: e.g., METS, DIDL
– http://www.dlib.org/dlib/december04/vandesompel/12va
ndesompel.html
OAI-PMH and HTTP
•
OAI-PMH uses HTTP as transport
– Encoding OAI-PMH in GET
• http://baseURL?verb=<verb>&arg1=<arg1Val>...
• Example:
http://an.oa.org/OAIscript?
verb=GetRecord&
identifier=oai:arXiv.org:hep-th/9901001&
metadataPrefix=oai_dc
•
Error handling
 all OK at HTTP level? => 200 OK
 something wrong at OAI-PMH level? => OAI-PMH error (e.g.
badVerb)
•
HTTP codes 302 (redirect), 503 (retry-after), etc. still available to
implementers, but do not represent OAI-PMH events
OAI-PMH verbs
Verb
metadata
about the
repository
harvesting
verbs
Function
Identify
description of archive
ListMetadataFormats
metadata formats supported by archive
ListSets
sets defined by archive
ListIdentifiers
OAI unique ids contained in archive
ListRecords
listing of N records
GetRecord
listing of a single record
most verbs take arguments: dates, sets, ids, metadata formats
and resumption token (for flow control)
What Questions can you ask
via OAI-PMH?
•
•
•
•
•
•
Identify
GetRecord
ListIdentifiers
ListMetadataFormats
ListRecords
ListSets
Identify
• Who are you?
• What kind of stuff do you contain?
• What is the copyright of your data and your
metadata?
“A collection-level description”
GetRecord
• Give me the metadata of a single record!
ListRecords
• Give me the metadata of all your records!
• May be limited by the date a record was last
modified
• May be limited to a subset of the archive (e.g. only
physics related records, but only if supported by
archive)
ListIdentifiers
• Give me a list of all your records!
• May be limited by date record was last modified
• May be limited to a subset of the archive (e.g. only
physics related records, but only if supported by
archive)
ListMetadataFormats
• What metadata formats can you supply?
All archives must supply Dublin Core but may supply
other metadata formats too.
ListSets
• What subsets of your records may I ask for?
Some archives define subsets, by subject, by rights
etc. e.g. Physics related records, or public domain
items or peer-reviewed items.
So how does a service query
an archive?
• The first time it asks for ALL records.
• Then, every so often (day, week…) it asks for
everything that’s changed since it last asked.
CogPrints
(GNU EPrints)
1600 Records
www.orgprints.org
(GNU EPrints)
264 Records
arXiv
(custom software)
230,000 Records
D-Space @ MIT
(D-Space Software)
769 Records
Harvester #1
(Psychology Service)
500 Cogprints
169 D-Space
Harvester #3
(General Service)
230,000 arXiv
769 D-Space
264 OrgPrints
1600 CogPrints
150,162 “Improved” records
from physics aggregator
Harvester #2
(Physics Aggregator)
150,000 arXiv
162 D-Space
Day 1
Archive
Service A
1403 records
Give me everything!
OK!
(1403 records)
Harvester
1403 records
Day 2
Give me all records which were
added or changed since yesterday
Archive
Service A
1501 records
Archive
Service B
123 records
OK!
(102 new records,
4 deleted records,
23 changed records)
Give me everything
in set “physics”
OK!
(15 records)
Harvester
1403 records
1501
records
15 records
OK!
(0 new records,
1 record changed)
Day 3
Give me all records which were
added or changed since yesterday
Archive
Service A
1490 records
Give me everything in set
“physics” which were
added or changed since
yesterday.
Archive
Service B
123 records
OK!
(25 new records,
36 deleted records,
3 changed records)
Harvester
1501 records
1490
records
15 records
What are these records?
• Dublin Core
–
–
–
–
–
–
Title
Creator
Date
Description
Identifier (URL)
…
• Very simple, but more useful than plain text.
Dublin Core in OAI
• Dublin Core required!
– You must provide Dublin Core data via OAI, so that all
harvesters can use your data.
– You may also provide any other metadata formats you
want to (MARC, AMF, one you-made-up etc.)
http://www.openarchives.org/tools/tools.html
Identify verb
Information about the repository, start any harvest with Identify
<Identify>
<repositoryName>Library of Congress 1</repositoryName>
<baseURL>http://memory.loc.gov/cgi-bin/oai</baseURL>
<protocolVersion>2.0</protocolVersion>
<adminEmail>[email protected]</adminEmail>
<adminEmail>[email protected]</adminEmail>
<deletedRecord>transient</deletedRecord>
<earliestDatestamp>1990-02-01T00:00:00Z</earliestDatestamp>
<granularity>YYYY-MM-DDThh:mm:ssZ</granularity>
<compression>deflate</compression>
GetRecord - Normal response
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH>
….namespace info not shown here
<responseDate>2002-0208T08:55:46Z</responseDate>
<request verb=“GetRecord”… …>http://arXiv.org/oai2</request>
<GetRecord>
<record>
<header>
<identifier>oai:arXiv:cs/0112017</identifier>
<datestamp>2001-12-14</datestamp>
<setSpec>cs</setSpec>
<setSpec>math</setSpec>
note no HTTP encoding
</header>
of the OAI-PMH request
<metadata>
…..
</metadata>
</record>
</GetRecord>
</OAI-PMH>
Error/exception response
<?xml version="1.0" encoding="UTF-8"?>
<OAI-PMH>
<responseDate>2002-0208T08:55:46Z</responseDate>
<request>http://arXiv.org/oai2</request>
<error code=“badVerb”>ShowMe is not a valid OAI-PMH verb</error>
</OAI-PMH>
Same schema for all responses,
including error responses.
with errors, only the correct
attributes are echoed in
<request>
Identifiers
• Items have identifiers (all records of same item share
identifier)
• Identifiers must have URI syntax Unless you can recognize
a global URI scheme, identifiers must be assumed to be local
to the repository
• Complete identification of a record is
baseURL+identifier+metadataPrefix+datestamp
• <provenance> container may be used to express
harvesting/transformation history
Selective Harvesting
• RSS is mainly a “tail” format
• OAI-PMH is more “grep” like
• Two “selectors” for harvesting
– Date
– Set
• Why not general search?
– Out of scope
– Not low-barrier
– Difficulty in achieving consensus
Datestamps
•
All dates/times are UTC, encoded in ISO8601, Z notation: 1957-0320T20:30:00Z
•
Datestamps may be either fill date/time as above or date only (YYYY-MMDD). Must be consistent over whole repository, ‘granularity’ specified in
Identify response.
•
Earlier version of the protocol specified “local time” which caused lots of
misunderstandings. Not good for global interoperability!
Harvesting granularity
•
mandatory support of YYYY-MM-DD
•
optional support of YYYY-MM-DDThh:mm:ssZ (must look at
Identify response)
•
granularity of from and until agrument in
ListIdentifier/ListRecords must match
Sets
• Simple notion of grouping at the item level to support
selective harvesting
– Hierarchical set structure
– Multiple set membership permitted
– E.g: repo has sets A, A:B, A:B:C, D, D:E, D:F
If item1 is in A:B then it is in A
If item2 is in D:E then it is in D, may also be in D:F
Item3 may be in no sets at all
Record headers
• header contains set membership of item
<record>
<header>
<identifier>oai:arXiv:cs/0112017</identifier>
<datestamp>2001-12-14</datestamp>
<setSpec>cs</setSpec>
<setSpec>math</setSpec>
</header>
<metadata>
…..
eliminates the need for the “double
</metadata>
harvest” 1.x required to get all records
</record>
and all set information
resumptionToken
•
Protocol supports the notion of partial responses in a very simple way:
Response includes a ‘token’ at the which is used to get the next chunk.
•
Idempotency of resumptionToken: return same incomplete list when
resumptionToken is reissued
•
while no changes occur in the repo: strict
•
while changes occur in the repo: all items with unchanged datestamp
•
optional attributes for the resumptionToken: expirationDate,
completeListSize, cursor
Harvesting strategy
• Issue Identify request
– Check all as expected (validate, version, baseURL, granularity,
comporession…)
• Check sets/metadata formats as necessary (ListSets,
ListMetadataFormats)
• Do harvest, initial complete harvest done with no from and
to parameters
• Subsequent incremental harvests start from datastamp that
is responseDate of last response