Post on 24-Jun-2022
Learning Spatial Filters forMultispectral Image Segmentation
Devis Tuia∗, Gustavo Camps-Valls∗,Remi Flamary∗∗, Alain Rakotomamonjy∗∗
∗Image Processing Laboratory (IPL)Universitat de Valencia, Spain
∗∗LITIS EA 4108, Universite de Rouen76800 Saint Etienne du Rouvray, France
August 24, 2010
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 1 / 14
Introduction Multispectral Image segmentation
Multispectral Image segmentation
I In multispectral images we have highspatial variability of the spectralsignature
I VHR images allows us betterrecognition, but noisy maps
I Strong intraclass variance, higher thaninterclass
I Including spatial, not only spectralinfo, is mandatory!
⇒ Spatial regularization
image ground thruth
0 100 200 300 400 500 600 700 800 900 1000200
0
200
400
600
800
1000
1200
1400
1600
red bandN
IR b
and
Class “Buildings’ (red)against the others (blue)’
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 2 / 14
Introduction Spatial Filtering
Spatial FilteringSpatial filtering solves such problems:
I Mathematical morphology [Benediktsson et al., 2005, Tuia et al., 2010]
I Geometrical features [Inglada, 2007]
I Composite kernels with spatial filters [Camps-Valls et al., 2006]
SVMK(xiw, xj
w) K(xir, xj
r) K(xic, xj
c) K(xis, xj
s)
K(xi, xj)
But remain the problem of defining
I what kind of features,
I at what scale, ...
In this paper we propose tolearn the spatial filter that maximizes separability of the classes
in a SVM framework.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 3 / 14
Large Margin Filtering Definitions
Definitions
I X ∈ Rr1×r2×d is an image containing r1 × r2 pixels ∈ Rd .
I Xi,j,k = Xp,k ais the kth component of pixel p = (i , j).
I 2D convolution filter band-by-band:
Xp,k =
f ,f∑u=1,v=1
Fu,v,k Xp+(u,v)−f0,k
where f0 = f /2 and F ∈ Rf×f×d .
I RBF kernel between filtered pixels:
Kp,q = k(Xp,., Xq,.) = exp
(−||Xp,. − Xq,.||2
2σ2
), (1)
where σ is the kernel width or bandwidth.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 4 / 14
Large Margin Filtering Optimization Problem
Optimization Problem
Large Margin Filtering [Flamary et al., 2010]
ming,F
1
2‖g‖2 +
C
n
∑p∈Sl
H(Yp, g(Xp,.))
︸ ︷︷ ︸SVM objective function
+ λΩ(F)︸ ︷︷ ︸Filter regularization
(2)
where:
I H(Yp, g(Xp,.) = max(0, 1− Yp · g(Xp,.)) is the SVM hinge loss.
I C and λ are the regularization parameters.
I Ω(·) is a 3D Frobenius Norm: Ω(F) =∑f ,f ,d
u,v,k F2u,v,k
I g(·) is the SVM decision function:
g(Xp,.) =∑q∈Sl
αqYqk(Xq,., Xp,.) + b, (3)
where αp are the dual variables of problem.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 5 / 14
Large Margin Filtering Optimization Problem
Solving the problem
Approach
I Convex problem for a fixed F.
I We can always find the optimal decision function g∗ for a fixed F.
I Do gradient descent on F and stay in the optimal g∗ space[Bonnans and Shapiro, 1998]:
minF
J(F) = minF
J ′(F) + λΩ(F) (4)
with:
J ′(F) = ming
1
2‖g‖2 +
C
n
∑p∈Sl
H(Yp, g(Xp,.))
(5)
Algorithm [Flamary et al., 2010]
I Conjugate Gradient descent on F + linesearch.
I Solve a SVM at each cost calculation in the linesearch.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 6 / 14
Results Dataset
Dataset and experimental setup
Dataset
I VHR QuickBird image of the city of Zurich, Switzerland.
I 7 classes, difficult to discriminate ’buildings’ classes (’residential’ vs ’commercial’).If merged, difficulty to discriminate ’buildings’ and ’roads’
Compared Models
1. SVM pixel classifier.
2. AvgSVM, averaged pixel classifier.
3. WinSVM, classification of a window of pixels.
4. KF-SVM, Large margin filtering.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 7 / 14
Results Binary classification
Binary Classification
Method Class Training #Class AUC KappaPixels Pixels
SVMResidential ∼ 5000 2000
0.904 0.638AvgSVM 0.916 0.689WinSVM 0.947 0.730KF-SVM 0.938 0.742
SVMBuildings∗ ∼ 4000 1000
0.938 0.706AvgSVM 0.946 0.779WinSVM 0.970 0.807KF-SVM 0.974 0.815
∗ Pixels from classes ‘Residential’ and ‘Commercial’.
Results
I The estimated Area Under the ROC Curve (AUC) and Kappa coefficient arecomputed.
I Improving over the SVM classification and average Filtering.
I Similar results between KF-SVM and WinSVM (slightly better Kappa).
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 8 / 14
Results Multiclass classification
Multiclass classification
Method Classes Filtersize
TrainingPixels
[%]OA Kappa
SVM7 9 ∼ 5000
75.11 0.685AvgSVM 83.68 0.796WinSVM 82.98 0.785KF-SVM 85.32 0.816
SVM6∗ 9 ∼ 5000
83.04 0.772AvgSVM 89.48 0.860WinSVM 91.71 0.889KF-SVM 91.45 0.885∗ Pixels from classes ‘Residential’ and ‘Commercial’.
Results
I WinSVM and KF-SVM give similar results and both outperform SVM and AvgSVM.
I But with KF-SVM, only pixels are classified.
I Optimal preprocessing done by filtering.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 9 / 14
Results Visualization
Segmentation Visualization
Image SVM KF-SVM
Results
I Smoothed borders thanks to the filtering.
I ’Commercial Buildings’ (red class) detectedover ’Buildings’ (orange class).
I What about the filters ? ⇒ Fourier transform
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 10 / 14
Results Visualization
Filter Visualization (1)
Class: Houses, Residential buildingsMagnitude of the FT for different components
Blue filter for Houses
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
2
4
6
Green filter for Houses
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
2
4
6
Red filter for Houses
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
2
4
6
Near Infrared filter for Houses
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
2
4
6
I Low pass but larger band (houses are small).
I Green, Red and InfraRed are selected.
(Orange)
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 11 / 14
Results Visualization
Filter Visualization (2)
Class: Commercial buildingsMagnitude of the FT for different components
Blue filter for Commercial
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
10
20
Green filter for Commercial
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
10
20
Red filter for Commercial
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
10
20
Near Infrared filter for Commercial
Inverse frequency (m)
Inve
rse
fre
qu
en
cy (
m)
50 10 5 2
50
10
5
2
0
10
20
I Low pass but small band to detect largebuildings.
I Red is the most discriminant feature.
(Red)
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 12 / 14
Conclusion
Conclusion
Large Margin Filtering
I Method to learn jointly a SVM pixel classifier and a spatial filtering.
I Large margin spatial filtering/Preprocessing.
I Possibility to use other classifier after filtering, e.g. GMM.
I Visualization for the filtering, no black box approach.
Future Work
I Propose other regularization terms.
I Going local, a global filter is limited.
I Test the method in hyperspectral images, where stacking approaches fail.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 13 / 14
References
References
[Benediktsson et al., 2005] Benediktsson, J., Palmason, J. A., and Sveinsson, J. R. (2005).
Classification of hyperspectral data from urban areas based on extended morphological profiles.
IEEE Trans. Geosci. Remote Sens., 43(3):480–490.
[Bonnans and Shapiro, 1998] Bonnans, J. and Shapiro, A. (1998).
Optimization problems with pertubation : A guided tour.
SIAM Review, 40(2):202–227.
[Camps-Valls et al., 2006] Camps-Valls, G., Gomez-Chova, L., Munoz-Marı, J., Vila-Frances, J., and Calpe-Maravilla, J. (2006).
Composite kernels for hyperspectral image classification.
IEEE Geosci. Remote Sens. Lett., 3(1):93–97.
[Flamary et al., 2010] Flamary, R., Labbe, B., and Rakotomamonjy, A. (2010).
Large margin filtering for signal sequence labeling.
In IEEE Conference on Acoustic, Speech and Signal Processing ICASSP, Austin, Texas, USA.
[Inglada, 2007] Inglada, J. (2007).
Automatic recognition of man-made objects in high resolution optical remote sensing images by SVM classification of geometric image features.
ISPRS Journal of Photogrammetry Rem. Sens., 62:236–248.
[Tuia et al., 2010] Tuia, D., Ratle, F., Pozdnoukhov, A., and Camps-Valls, G. (2010).
Multi-source composite kernels for urban image classification.
IEEE Geosci. Remote Sens. Lett., Special Issue ESA EUSC, 7(1):88–92.
Devis Tuia et al (IPL, LITIS) Learning Spatial Filters August 24, 2010 14 / 14