Transcript VALIDATION OF REMOTE SENSING CLASSIFICATIONS: a case of

VALIDATION OF REMOTE
SENSING CLASSIFICATIONS:
a case of Balans classification
Markus Törmä
STRUCTURE OF THESIS
1. Introduction
2. Methods for validation
3. Case: Validation of Balans classification
4. Success of validation
5. Conclusions
BALANS CLASSIFICATION
• The goal of the project
was to develop a
streamlined production
line for creation of land
cover information using
medium resolution satellite
data
• The production of land
cover classification
covering Baltic Sea
drainage basin was tested
BALANS CLASSIFICATION
Organisations:
• Metria Miljöanalys
• Finnish Environment Institute
• Novosat Oy
• GRID-Arendal
• GRID-Warsaw
• Swedish Meteorological and Hydrological
Institute.
BALANS CLASSIFICATION
Classes:
•
•
•
•
•
•
•
•
•
Artificial surfaces
Agricultural areas
Coniferous forest
Deciduous forest
Bare rock
Glaciers and perpetual snow
Other seminatural areas
Wetlands
Water bodies
BALANS CLASSIFICATION
IRS WiFS:
• Channels: RED and NIR
• Spatial resolution: 188m
BALANS CLASSIFICATION
Reference datasets:
•
•
•
•
•
•
•
•
Finnish National Land Use and Forest Classification
Swedish Terrain Type Classification
CORINE Land Cover classification
FIRS (Forest Information from Remote Sensing project)
regions and stratas
Agricultural areas from Baltic Sea Region GIS, Maps and
Statistical Database
Baltic Sea Drainage Basin Watershed Areas from Baltic Sea
Region GIS, Maps and Statistical Database
World Atlas of Agriculture
Digital Elevation Model (DEM), slope and wetness index
BALANS CLASSIFICATION
Classification:
• Unsupervised classification of individual WiFSimages
• Interpretation of clusters using reference data
• Second unsupervised classification to unclassified
areas
• Some classes like artificial surfaces and
agricultural areas are taken directly from reference
data if possible
VALIDATION
Geometric validation:
• The accuracy of a geometrically rectified
image
Thematic validation:
• The correspondence between the class label
assigned to a image pixel and its true class
on the field
GEOMETRIC VALIDATION
•
•
•
•
•
Validation points from 12 test areas
Residual = reference crd - classification crd
Water areas as reference data:
Finland: Digital map 1:250 000, 4 areas
Poland: Topographic map 1:100 000, 6
areas
• Sweden: Topographic map 1:50 000 and
1:100 000, 2 areas
GEOMETRIC VALIDATION
• Water areas from classification result and reference data
were compared using INREC-software (VTT)
GEOMETRIC VALIDATION
• Areal distribution of geometric control points
GEOMETRIC
VALIDATION
• Residual plots of
Fuinnish, Polish and
Swedish test areas
GEOMETRIC VALIDATION
• Geometrically Balans classification was
successful
Finland
Poland
Sweden
ALL
Emean
55.3
53.3
63.4
56.2
Nmean
-2.0
6.6
-22.0
-2.9
ERMSE
79.2
116.9
98.5
95.5
NRMSE
51.3
79.2
44.4
59.3
Planimetric
94.4
141.2
108.0
112.6
RMSE
THEMATIC VALIDATION
• In order to get some idea about the mixing
between classes and accuracy in the different parts
of database
• Check points using reference material
• Classes of check points in reference material and
classification result were compared
• Number of check points per class was indended to
be about 100 per country, but was as low as 20 in
some case
THEMATIC VALIDATION
Reference material
• Finland
Digital map 1:250000
Base map 1:20000
3 Landsat-7 ETM-images
• Poland
Corine Land Cover classification
Topographic map 1:100 000
• Sweden
Topographic map 1:50 000
Topographic map 1:100 000
2 Landsat-5 TM-images
THEMATIC VALIDATION
Sampling
• Aim: 100 or more reference points per class per
country
• Finland and Sweden:
-Visual interpretation of Landsat image or map
-It was tried to place reference point to an area
with size more than five hectares in order to
eliminate the possibity of border and mixed pixels
and minimize the effect of geometric errors
THEMATIC VALIDATION
Sampling
• Poland:
-GRID Warsaw
-Center points of areas larger than 100 hectares
from Corine landcover
-Random sampling for classes artificial surfaces,
agricultural areas, deciduous and coniferous
forests
-Classes water and wetlands were augmented
using points taken from topographic map
-No reference points for class seminatural areas
THEMATIC VALIDATION
Different class-combinations were studied:
• A: Original classes
• B: Coniferous and deciduous forest classes
are combined to one forest class
• C: Agricultural and seminatural areas are
combined to one class
• D: One forest class and agricultural and
seminatural areas are combined
THEMATIC VALIDATION
• Following information was computed from
sampling points:
•
•
•
•
•
•
•
•
•
Statistics of reference and classified data
Error matrix
Overall accuracy
Interpretation accuracies of classes and their mean value
Target accuracies of classes and their mean value
Average accuracies of classes and their mean value
Mapping accuracy of classes and their mean value
Kappa coefficient of classification
Tau coefficient of classification using equal a’priori probabilities for
classes
• 95% confidence intervals for overall, interpretation, target and average
accuracies
THEMATIC VALIDATION
• Thematic accuracy of Balans classification is not that
good
• Overall accuracy of original classes is 69.7% when
wetlands are not included and 65.4% with wetlands
• Water is classified very well
• Artificial surfaces reasonably well
• Otherwise from average (agricultural land) to very poor
(seminatural areas)
• The most problematic classes coniferous forest and
seminatural areas
SUCCESS OF VALIDATION
• Geometric validation was successful and provides
unbiased view to geometric errors of Balans
classification
• Validation was based on check points measured from
independent material (digital maps and Landsat image)
• Number of check points was rather large, 294
• There were 12 test areas in Finland, Poland and Sweden
so check points cover Balans classification area quite
well
• The effect of transformations between different
coordinate systems was not evaluated
SUCCESS OF VALIDATION
• Thematic validation was unsuccessful
• Sampling design was driven by the availability of
free reference data and lack of time
• Effects of mixed pixels and geometric
transformations were tried to minimize
-visual
interpretation
from
middle
of
homogeneous areas
• Due to the sampling process the performed
sampling is not probability sampling, more like
convenience sampling.
SUCCESS OF VALIDATION
• Finland
points for deciduous forest and wetlands
unreliable due to reference data
• Poland
points were based on CORINE which was used in
classification
points for seminatural areas were missing
• Sweden
two small test areas
SUCCESS OF VALIDATION
Lessons:
• Aim of thematic validation should be stated
clearly
• Sampling should be based on probability
sampling
• Large pixel, there is need for the validation
methods for soft classifications
VALIDATION OF OTHER
CLASSIFICATIONS
• Should classification and thematic validation
use hard or soft methods?
• Simple, economic, unbiased and spatially
comprehensive sampling method
VALIDATION OF OTHER
CLASSIFICATIONS
Spatial resolution vs. size of land cover polygons
• Determine optimal spatial resolution (SR) for scene
If SR of instrument smaller than optimum SR, use hard
classification and validation
If SR of instrument larger than optimum SR, use fuzzy
classification and validation
• Decide acceptable proportion of mixed pixels, i.e. pixels
containing more than one land cover type
If the average size of land cover polygons is known then the
required spatial resolution of remote sensing instrument can be
estimated.
VALIDATION OF OTHER
CLASSIFICATIONS
• Simulated effect of
the size of land
cover polygon to the
proportion of mixed
pixels
• Solid line: small
change causes pixel
to be labeled as
mixed
• Dashed line: some
amount of change
(10%) in pixel
values is tolerated
VALIDATION OF OTHER
CLASSIFICATIONS
• Sampling should be random, cover studied area
and all classes
• Stratified random sampling
• If not cost-efficient, include clustering
• Sampling points from ground visits, aerial
photographs, satellite images or existing maps
• More than one preson should interprete the
sampling points
CONCLUSIONS
• Geometric accuracy of Balans classification
was good
• Geometric validation method OK
• Thematic accuracy of Balans classification
poor
• Due to sampling, it is dangerous to generalize
thematic validation outside validation points
• Thematic accuracy most likely optimistically
biased