Transcript "Metadata"? - World Bank

Salvador Bayarri
Consultant, World Bank
[email protected]






The SDI architecture model
Software components: the SDI stack
Basic services
Metadata
SDI access control
Current trends in SDI technology

SDI uses a three-tier service-oriented
model
Standardized
Source:
mapit.biz
Same thing with a component-oriented view:
GIS
Clients
Access to transformed
data, pictures, maps, reports,
multi-media content
Metadata search and retrieval for
data and services
Catalogs
Metadata update
Service chaining:
search, display,
access, e-commerce,
….
Geo-processing
and catalog Services
Middleware
Direct
data
access
Other data
Content
Repositories
e.g., administrative,
statistical, env. reporting
Coverages
Features
Servers

The use of a communication layer based on
standardized Web services ensures:
◦ Transparent access to data, regardless of format,
technology… but we still need to harmonize the
data content
◦ Interoperability between different brands of
servers and clients
◦ Distributability of components in a network. The
SDI philosophy is that each responsible
organization maintains and publishes its
datasets. Multiple clients, catalogs, etc. are
possible




Spatial DB = Relational DB + …
◦ Geometry representation. Standard is SFS (Simple
Feature Specification)
◦ Spatial Index. Speeds up search and operators, sorting
objects by location
◦ Spatial operators. Can be invoked in SQL sentences to
measure, perform geometric operations (e.g.
intersection, reprojection), create new objects, etc.
Essential for data quality assurance.
Needed for large datasets, many simultaneous
accesses (e.g. concurrent editing)
Relational databases also used for metadata
Files still used for raster data (images,
elevation grids)

Example of spatial table
Encoded geometry field
SDI stack: Spatial Databases

Simple Feature Specification (OGC) for SQL

Well-Known Binary representation for simple
features (OGC)

Open-source (OS) implementations
◦ SpatiaLite (extension to sqLite), supported
by qGIS
◦ MySQL spatial extensions (not fully
standard)
◦ PostGIS (extension to PostgreSQL)





Excellent support for standards
Widely supported in OS GIS, ArcGIS, Intergraph…
Used by GeoNode
Friendly administration tools (pgAdmin)
Interesting extensions (pgRoute for routing)
The whole UK Ordnance Survey is stored in
PostGIS (£1,000,000 savings compared to Oracle
Spatial), as well as the national topographic data
of France for the Institut Géographique National

Proprietary implementations
◦ Oracle Spatial
◦ MS SQL Server 2008 and later
◦ ArcSDE (now ArcGIS Server
Basic)
 Wraps spatial or non-spatial DBs
for use in ESRI software, getting
good performance
 Some support in OS GIS, but…
 In general, not useful if you need
to use non-ESRI software with the
DB
SDI stack: GeoServers

A variety of OS and proprietary technologies
support OGC standards for geoservices
SDI stack: GeoServers


MapServer (OS). Excellent performance, quality
and symbology for Map services (image)
GeoServer (OS). Excellent performance and
support for Feature services (data), great
integration with PostGIS, even for remote
editing. Easy administration.
◦ Used in GeoNode
SDI stack: Catalog Servers

Geonetwork (OS)
◦ Reference OGC implementation for catalog service
◦ Includes powerful Web metadata editing and
administration
◦ Extensively used in catalog networks (UN, many SDIs)
◦ Used in GeoNode
SDI stack: Catalog Servers

ESRI’s Geoportal Server
◦ Free and Open source, but…
◦ Meant to be used with ArcGIS Online and ArcGIS for Server
SDI stack: Web clients

Intergraph’s SDI Portal – Geospatial Portal
◦ Bundles OpenLayers with GeoExt UI to provide
ready-to-use components
SDI stack: Web clients

Usually combine:
◦ A live-map component (map viewer) which can:
 Display data from standard Web geoservices
 Navigate, query data…
 Sometimes provide data editing
◦ Web user-interface UI technology for legend,
search, symbology, tools, etc.
 Today these are based on AJAX, Flex, Silverlight or
other highly interactive framework for Web apps
SDI stack: Web clients

OpenLayers (OS)
◦
◦
◦
◦
◦
Most used map viewer
Supports Google Maps and other map providers
Excellent performance, using tiled images
Supports feature data, editing with WFS-T
Used in GeoNode via the GeoExplorer component
SDI stack: Web clients

GeoExt (OS)
◦ Based on JavaScript toolkit ExtJS, applied to Web
geoportals
◦ Highly interactive table of contents, menus, tools,
dialogs, etc.
◦ Integrates OpenLayers. It easily extended and
customized
◦ Used in GeoNode via the GeoExplorer
SDI stack: Web clients

ESRI Web Mapping API
◦ Bundles ESRI’s map components with JavaScript,
Flex or Silverlight UI. Includes Web editing
functionality.
SDI stack: desktop clients

GIS (Geographical Information Systems)
◦ Heavy-duty data analysis and processing are still run
in desktop applications
◦ All of them support OGC standards, so they can use
Web geoservices and catalogs as source for data (but
compliance and compatibility is not always good)
◦ The best practice is not to download data locally, but
work with remote data that is official and up-to-date
◦ The SDI model, together with GIS, can also be used to
implement information systems within organizations,
sharing common data and metadata
SDI stack: desktop clients
qGIS (FOSS) – We’ll use this one in the workshop
SDI stack: desktop clients
uDIG (FOSS)
SDI stack: desktop clients
ArcGIS
SDI stack: desktop clients
GeoMedia

Web Map Service (WMS)
◦ Get maps as images
◦ Get attribute information about features in a specific map location

Web Feature Service (WFS)
◦ Get vector data with geometry and attributes based on filters. Not
meant for large data downloads!
◦ Remote feature editing (WFS-T)

Web Coverage Service (WCS)
◦ Similar to WFS, for raster data (multiband images, elevation grids DEMs)

Catalog Service for Web (CSW)
◦ Search and read metadata
◦ Harvest metadata to other catalogs
◦ Remotely edit medatada

Web Map Service (WMS) requests:
◦ GetCapabilities: returns the description of the service (layers,
styles, coordinate systems…) as XML text
http://neowms.sci.gsfc.nasa.gov/wms/wms?version=1.1.1&service=WMS&request=GetC
apabilities
http://www.snitcr.org/cgibin/wms?map=snit.map&VERSION=1.1.1&SERVICE=WMS&REQUEST=GetCapabilities
◦ GetMap: returns an image (jpeg, png) of a map composed of
one or more layers, in the requested coordinate system. The
layers and style (legend) are defined in the server but can be
selected and changed (the legend) in the request
http://neowms.sci.gsfc.nasa.gov/wms/wms?VERSION=1.1.1&REQUEST=GetMap&LAYER
S=MOD14A1_E_FIRE&WIDTH=640&HEIGHT=320&FORMAT=image/png&SRS=EPSG:4326
&BBOX=-180.0,-90.0,180.0,90.0
http://196.40.0.82/cgibin/mapserv?map=c:\ms4w\wms\biodiversidad_basp10v4geog50k.map&LAYERS=ARE
AS_SILVESTRES_PROTEGIDAS%2CETIQUETAS&TRANSPARENT=true&VERSION=1.1.1&FOR
MAT=image%2Fpng&SERVICE=WMS&REQUEST=GetMap&STYLES=&SRS=EPSG%3A97057
&BBOX=247206.87468627,1128978.9530588,427828.99937255,1309601.0777451&
WIDTH=256&HEIGHT=256

Web Map Service (WMS) requests:
◦ GetFeatureInfo request: returns the attributes of one or
more objects of a specified layer at a certain location in the
map. Meant as a ‘click-on’ query on a map viewer
http://demo.opengeo.org/geoserver/wms?LAYERS=topp:tasmania_water_bodies&QUERY_
LAYERS=topp:tasmania_water_bodies&STYLES=&SERVICE=WMS&VERSION=1.1.1&REQUES
T=GetFeatureInfo&BBOX=141.5115,-43.933,150.8015,39.288&FEATURE_COUNT=10&HEIGHT=256&WIDTH=512&FORMAT=image/gif&INFO_FO
RMAT=text/html&SRS=EPSG:4326&X=258&Y=188
The output format can be changed (plain text, XML…) to process it in
our client application.

WMS GetFeatureInfo Exercise
◦ Try it live at
http://openlayers.org/dev/examples/getfeatureinfocontrol.html
◦ Use your Web Browser to spy on network requests and
responses
 Chrome: Tools /Developer Tools / Network tab
 Firefox: Tools / Web Developer / Web Console

Web Feature Service (WFS) requests:
◦ GetCapabilities: returns the description of the service
(layers, filter types, coordinate systems…) as XML text
http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=
GetCapabilities
◦ DescribeFeatureType: returns a description of the feature
for a specified layer
http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request
=DescribeFeatureType&typeName=usa:states

Web Feature Service (WFS) requests:
◦ GetFeature: returns the features that comply with a certain
condition (filter), based on location or attribute values (it is not
meant for massive data dowload!)
http://v2.suite.opengeo.org/geoserver/ows?version=1.1.0&service=WFS&request=GetF
eature&typeName=usa:states&maxFeatures=10&filter=
<ogc:Filter xmlns:ogc="http://www.opengis.net/ogc">
<ogc:BBOX>
<ogc:PropertyName>the_geom</ogc:PropertyName>
<gml:Envelope xmlns:gml="http://www.opengis.net/gml" srsName="EPSG:4326">
<gml:lowerCorner>-89.92773475911 32.386718678814</gml:lowerCorner>
<gml:upperCorner>-89.04882850911 33.265624928814</gml:upperCorner>
</gml:Envelope>
</ogc:BBOX>
</ogc:Filter>
<?xml version="1.0" encoding="UTF-8"?>
<wfs:FeatureCollection numberOfFeatures="1">
<gml:featureMembers>
<usa:states gml:id="states.20">
<usa:the_geom>
<gml:MultiSurface srsDimension="2">
<gml:surfaceMember>
<gml:Polygon>
<gml:exterior>
<gml:LinearRing>
response in
<gml:posList> -81.759758 33.195232000000004 ... </gml:posList>
</gml:LinearRing>
GML format
</gml:exterior>
</gml:Polygon>
</gml:surfaceMember>
</gml:MultiSurface>
</usa:the_geom>
<usa:STATE_NAME>South Carolina</usa:STATE_NAME>
<usa:PERSONS>3486703.0</usa:PERSONS>
</usa:states>
</gml:featureMembers>
</wfs:FeatureCollection>
GetFeature
Test this live at
http://openlayers.org/dev/examples/getfeature-wfs.html

Web Feature Service (WFS) requests for remote
editing (WFS-T):
◦ LockFeature: prevents other clients from editing selected
features while we work at them
◦ Transaction: creates, modifies or deletes features. Each
transaction consists Insert, Update, and Delete elements,
performed in order

Not all servers or clients implement these requests!

Web Coverage Service (WCS):
◦ Similar to WMS but returns raster data with
multiple band information, elevation values, etc.

Catalog Service for the Web (CSW):
◦ Provides remote access to a DB of metadata
records (catalog)




DescribeRecord - returns info about the format of metadata records
GetRecords - search for records, returning record IDs
GetRecordsById - returns records, specified by their ID
Harvest (optional) - create/update metadata by asking the server to 'pull'
metadata from somewhere (other CSW server, a Web folder…)
 Transaction (optional) - create/edit metadata by 'pushing' the metadata to
the server

Catalog Service for the Web (CSW):
◦ GetCapabilities and DescribeRecord examples:
http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=Get
Capabilities
http://www.fao.org/geonetwork/srv/en/csw?service=CSW&request=Des
cribeRecord
◦ GetRecords example:
http://www.fao.org/geonetwork/srv/en/csw?REQUEST=GetRecords&SER
VICE=CSW&VERSION=2.0.2&OUTPUTSCHEMA=http://www.isotc211.org/
2005/gmd&CONSTRAINTLANGUAGE=CQL_TEXT&RESULTTYPE=results&T
YPENAMES=csw:Record
A constraint or filter is usually added to get only the metadata that we
are searching (for instance, containing certain words, or for a certain
geographic area)
Basic SDI Services
The CSW service, through the Harvest request, allows the
federation and integration of metadata catalogs,
automatically collecting metadata from some catalogs to
another (for instance, to specific organizations to a
national catalog)
Metadata
editor
Web or
desktop client
Local/entity
Catalog
Server
National
Catalog
Server
Metadat
a DB
Harvest
Metadat
a DB
Basic SDI Services
How they all work together… (Animation)
Geoportal, GIS
client
Search
Viewer
Web
Service
File
Metadata
Metadata
Metadata
File
Add WFS
WMSlayer
layer
Data
CSW request QueryWMS result
visualization
(image)
Data
download
http://catalog.smi-q.gob.ec
Main Catalog
Metadata
WFS result
WMS
request
WFS
(data)request
CSW Result
http://server.dep.gob.ec/maps?map1
http://server.dep.gob.ec/wfs?layer1
Servidor
entidad
Entity
metadata
Entity
metadata
Entity
data
Metadata
Geodata
Archivo
http://server.dep.gob.ec/files/data.dat

Direct file download
◦ Use it for geodata download instead of WFS or WCS, which
should be only for limited query
◦ Use it for documents and other files that cannot be accessed
through standard Web services
◦ The download address can be written in the metadata, as we
do for Web services

News and alerts (RSS, GeoRSS, Atom…)
◦
◦
Conventional Web clients can get updated messages and
news
Users can subscribe to receive them in a variety of readers



Metadata is a formal data
documentation and it is critical to data
discovery
A metadata record is a set of
information fields, usually presented as
an XML document, which captures the
basic characteristics of data as an
information resource
It represents the who, what, when,
where, why and how of the resource
What is "Metadata"?
Metadata as seen in Geonetwork
What is "Metadata"?
Metadata as seen in ESRI’s ArcCatalog



Organize and maintain an organization's
internal investment in spatial data
Provide information about an organization's
data holdings to data catalogs,
clearinghouses, and portals (it is the entry
point of an SDI and for each SDI node)
Provide information to process and interpret
data received through a transfer from an
external source




Facilitates data maintenance
Facilitate data discovery
Enables reuse of data
Informs potential users of inappropriate uses
of a dataset



Exchanging records -> Use standards!
Creating metadata takes time and patience
-> Use templates and tools that
automatically extract or set metadata (like
GeoNode). Imperfect metadata is better
than none
Maintaining metadata is essential, especially
for Web services and downloads (beware of
URL changes!) -> set up processes or tools
for automatic checkups

Some standards define the metadata content
(fields, values, mandatory v. optional)
◦ The most comprehensive is ISO 19115



From this standards, custom profiles can be
defined for specific countries, themes, etc.
‘Old’ metadata standards like FGDC or
Dublin Core are now profiles of ISO 19115
Other standards define appropriate
exchange formats
◦ The most accepted is ISO 19139, which defines an
XML-based format
• Important for data search
• Information in abstract: data model
– Entity types and their attributes and the
domains from which attribute values
may be assigned
– The names and definitions of features,
attributes, and attribute values
•
•
•
•
How to access the information?
Multiple transfer options are possible
Contact for physical media
Fees
•
•
•
•
•
Positional and attribute accuracy
Completeness and consistency,
Sources of information
Methods used to produce the data
Appropriate uses (scales)
• Language, date, update status
• Who is responsible for the metadata?
Metadata content:
Access and use constraints



Use constraint statements can be crafted to express
scale, geographic, or temporal limitations to the data
Liability statements should be written by legal staff to
ensure that the legal requirements for use of the data
are fully outlined
In general, it is far better to publish your dataset with
limitations within your metadata than to later attempt
to generate them in response to an inquiry or lawsuit.

Use constraints and disclaimers
◦ Web clients typically include liability limitations
(disclaimers) and licenses that users must accept
◦ But geoservices can be accessed directly!


Therefore, use constraints must be
specified in the metadata and
GetCapabilities response
And services must be protected from nonauthorized access
◦ Within an organization: generic IP filtering and
firewalls
◦ External services: need an authentication
mechanism

There are generic mechanisms for ServiceOriented Architectures
◦ Based on XML standards like SAML and XACML
– However, these
standards are not
easily used in SDIs
because
• Current geoservices do
not use SOAP for
passing information
• Server and client
software should be
modified to comply



Specific SDI
solutions have
been proposed
and used
Existing clients
and server don’t
have to be
modified, only
‘intercepting’
components
need to be
added
Access can be
controlled for
specific layers,
scales,
features
See
http://www.eurogeographics.org/sites/default/files/070504_SecurityInSDI_conter
and
attributes
ra_V2.pdf


In practice, however, most systems use simple
HTTP authentication that provides a simple on/off
swith for services depending on the user
credentials
Web browsers have a built-in implementation to
ask for user and password

Gazetteer services (search location by
name), based on WFS-G profile

Web Processing Service (WPS). Standard for
discovering and calling remote processing
modules
◦ Used for typical GIS analysis like spatial analysis,
running distributed models
◦ Used for operations that are not easy to
standardize in a specific Web service
 Spatial Coordinates transformation
 Geocoding (address finding)
See, for instance:
http://www.cartociudad.es/wps/WebProcessingService?se
rvice=WPS&request=GetCapabilities
Current trends in SDI
technology
Tiled maps
Cached and tiled
Map Services
(WMTS, WMSC…)
Current trends in SDI
technology
Tiled maps

Static maps can be pre-calculated and tiled
at different scales, like a raster pyramid
◦ Tiles can be directly server through tile services like
WMTS, WMS-C, Google Earth…
◦ Pre-calculated tiles can be used to quickly respond to
conventional WMS requests
◦ An example is the GeoWebCache technology, used by
GeoNode, which can be integrated with regular WMS
servers like GeoServer
Current trends in SDI
technology
Tiled maps
Current trends in SDI
technology
Tiled maps


A problem with tiled map providers like
Google, Bind, OpenStreetMap, is that they
only provide tiles with a specific ad-hoc
coordinate system (used to be
EPSG:900913, now it is officially EPSG:3857)
It is difficult to overlay this data layer on
other coordinate systems
Current trends in SDI
technology
Remote on-line
editing with
WFS-T
Current trends in SDI
technology
Remote
collaborative editing
(OpenStreetMap)
Current trends in SDI
technology
Data capture and
visualization in
mobile devices
Current trends in SDI
technology
Remote geoprocesses (coordinate
transformation, geocoding) with
WPS
Current trends in SDI
technology
Real-time data
integration (RSS,
Common Alert
Protocol)
Current trends in SDI
technology
Standard Web services for
reading and controling
remote sensors (SWE,
SOS…)
Current trends in SDI
technology
Integration with SOLAP
data mining
Current trends in SDI
technology
SDI Web services in the
cloud
Current trends in SDI
technology
New data sources
(LiDAR, georeferenced
Video & street-level
images)