Geo-based Automatic Image Annotation

Hatem MoussellySergieh

INSA de Lyon7, Avenue Jean-Capelle

69621 Villeurbanne, Francehatem.mousselly-

sergieh@insa-lyon.fr

Mario DöllerUniversity of Passau

Innstrasse 4394032 Passau,Germanymario.doeller@uni-

passau.de

Elöd Egyed-ZsigmondINSA de Lyon

7, Avenue Jean-Capelle69621 Villeurbanne, France

elod.egyed-zsigmond@insa-lyon.fr

Gabriele GianiniUniversity of Milanvia Bramante, 65

26013 Crema, Italygabriele.gianini@unimi.it

Harald KoschUniversity of Passau

Innstrasse 4394032 Passau,Germanyharald.kosch@uni-

passau.de

Jean-Marie PinonINSA de Lyon

7, Avenue Jean-Capelle69621 Villeurbanne, Francejean-marie.pinon@insa-

lyon.fr

ABSTRACTA huge number of user-tagged images are daily uploadedto the web. Recently, a growing number of those imagesare also geotagged. These provide new opportunities for so-lutions to automatically tag images so that efficient imagemanagement and retrieval can be achieved. In this paperan automatic image annotation approach is proposed. Itis based on a statistical model that combines two differentkinds of information: high level information represented byuser tags of images captured in the same location as a newunlabeled image (input image); and low level informationrepresented by the visual similarity between the input im-age and the collection of geographically similar images. Tomaximize the number of images that are visually similar tothe input image, an iterative visual matching approach isproposed and evaluated. The results show that a significantrecall improvement can be achieved with an increasing num-ber of iterations. The quality of the recommended tags hasalso been evaluated and an overall good performance hasbeen observed.

Categories and Subject DescriptorsH.4 [Information Systems Applications]: Miscellaneous;I.4 [Image Processing and Computer Vision]: Appli-cations

General TermsAlgorithms, Experimentation, Performance

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.ICMR ’12, June 5-8, Hong Kong, ChinaCopyright c©2012 ACM 978-1-4503-1329-2/12/06 ...$10.00.

KeywordsImage Annotation, Geotagging, Image Retrieval, StatisticalModels

1. INTRODUCTIONRecent technological development has allowed an easy ac-

cess to digital photography devices, such as digital cam-eras and smart phones. Everyday, individuals are producinglarge amounts of images and upload them to the web. Forexample, Flickr1 hosts about 6 billion images. As a result,image organization and retrieval became more challengingand efficient solutions are required.

Research in content based image retrieval (CBIR) [6] [5]addresses the problem by identifying, combining and com-paring different kinds of low level image features, such ascolor histogram, texture, etc. Although this alleviates theproblem, CBIR alone is still unable to cope with the se-mantic notions of user queries. In fact, most CBIR systemsfollow a query by example (QBE) scheme which is still fartoo less spread than the conventional keyword-based imagesearch.To narrow the semantic gap, annotating images with key-words, known as tags is widely used. Tagging enables eas-ier management and retrieval of images. Recently, in addi-tion to the huge amount of user tagged images which areavailable on the web, the number of images which are pro-vided with location information ”geotagged” is increasing.Geotagged images contain in their Exif descriptors [16] thecoordinates (latitude and longitude) of the location of theircapture. Currently, Flickr2 hosts more than 171 million geo-tagged items, while a statistic of the year 2007 shows thatPanoramio3 hosts 2 million geotagged images.

The high availability of user- and geotagged images pro-vides opportunities to develop new tools for automatic imageannotation. This can be done by combining the high level

1http://news.softpedia.com/news/Flickr-Boasts-6-Billion-Photo-

Uploads-215380.shtml2http://www.flickr.com/map

3http://blog.panoramio.com/2007/06/2-million-photos-in-

panoramio.html

information represented by user tags and CBIR solutionswhich work on image low level features. For better under-standing, consider the following scenario.Assume that a user took a photo of the building of Institutede France (Fig. 4a) with his smart phone. The GPS re-ceiver of the phone calculates the latitude and the longitudeof the location and the data are stored in the Exif descriptorof the produced image. Later on, the user wants to uploadthis image to his favorite social website but he doesn’t havetime to tag it, or he doesn’t even know about the place andthe contents of the image. The user can send the imageto a tag recommendation system which extracts the GPScoordinates of the image. Then, the system applies a geo-based search on online image databases, such as Flickr tofind user-tagged images which were taken by other users inthe surroundings of Institute de France. From the retrievedset, the system tries to find images in which the buildingof Institute de France appears. For this purpose, the sys-tem searches for images from the retrieved collection whichare visually similar to the input image. Finally, the tags ofthe visually matching images can be analyzed to produceannotation suggestions for the input image.

To address each of the steps presented in the above im-age annotation scenario, a statistical model for automaticimage annotation is proposed. It combines high level (usertags) and low level information extracted from communityimage to annotate a new image taken in the same location.To maximize the size of the candidate tag collection an it-erative image matching approach is provided. It increasesthe number of visually similar images and, as a result, thenumber of the collected tags. This has a direct influenceon the quality of the provided tags. First, additional candi-date tags can be discovered. Second, form statistical pointof view, the usage pattern of user tags can be better esti-mated in a larger tag collection. To reduce the runtime ofthe visual matching, an image clustering approach based onimage low level feature similarity is introduced. Finally, dif-ferent evaluation studies for estimating the effectiveness ofthe introduced solutions are provided.

The rest of the paper is organized as follows: in the nextsection related work is reviewed. In section 3 a detailed de-scription of the annotation approach is presented. In section4 different evaluation scenarios are provided. The paper isconcluded and future work is discussed in section 5.

2. RELATED WORKThis work is related to search-based image annotation ap-

proaches. It exploits geo- and user-tagged image datasetsand proposes a probabilistic model for tag recommendation.This model combines low level image features and user tagsto produce word-image relevance values which can be usedto rank the recommended tags. In the following, we reviewimage annotation works which use similar techniques.

The authors in [18] introduce a hybrid probabilistic modelfor automatic image tagging. The model combines CPAM(Colored Pattern Appearance Model) [15] image featureswith textual information provided by the user as initial tags.The goal is to extend the set of provided initial tags by rec-ommending new ones.

In [11] user-tagged community photos are exploited to pro-vide a tag ranking approach. Hereby, the importance of thetag is determined by the number of visual neighbors of theinput image which are annotated with that tag. The vi-

sual neighbors are identified based on three image features,namely, color correlgoram, color texture moment and RGBcolor moment.

The tag recommendation approach presented in [17] gen-erates tag rankings by combining three kinds of tag corre-lations is presented in: tag co-occurrence in a large Flickrimage dataset, tag correlation calculated based on visuallanguage model (VLM), and image-tag conditioned tag cor-relation.

The image annotation approach proposed in [12] combinesdifferent search schemes to estimate the relation between aword and an unlabeled image. For this purpose, keyword-based image retrieval is used to calculate the likelihood ofobtaining a certain image given a certain keyword. Afterthat, the set of candidate annotations is expanded by apply-ing a linear combination of two kinds of search-based wordcorrelations. A statistical correlation that uses the wordsas input to Google image searcher and calculates ”Normal-ized Google Distance” [4] based on the provided results. Theother correlation is based on the visual consistence amongthe resulted images returned by the search.

A geo-based image annotation approach for landmark pho-tos is proposed in [7]. It uses multi-level clustering approachthat exploits geographical as well as visual features to iden-tify the category of an input image and the nearest visualclusters. A caption for the input image is generated formthe set of most frequently used tags in the correspondingvisual cluster.

Among the above introduced works only [7] consider geo-tagged community photos. In fact, our approach share sim-ilar ideas, however, we have a broader focus than only pro-viding captions for landmark photos. We provide a solidframework for tag recommendation that exploits visual, ge-ographical and different tag occurrence measures to provideranked tag suggestions, while image captions produced in [7]depend only on the usage frequency of the tags in the visualcluster.

The statistical model provided here is similar to the oneproposed by [18]. However, our approach considers the ge-ographical context of the input image and uses a differentfamily of low level image features. Furthermore, we don’texpect any further input for our approach except the inputimage, while initial tags are considered as perquisite to gen-erate tag recommendations in [17]. In [18] the absence ifinitial tags leads to a pure CBIR search.

Finally, no one of the approaches presented in this sectionconsiders the effect of the volume of the image dataset on thequality of the suggested tags. In this paper we deal with thisproblem and propose an iterative visual matching solutionto find additional visual correspondences for an input image.We also show how the size of the visual cluster influences thequality of the produced tags.

3. TAG RECOMMENDATION APPROACHThe proposed approach consists of three phases (see Fig.1).

A geo-based search in which the geographical coordinatesof an input image are used to find a collection of user-contributed images that are taken in the same geographicalarea. Hereby, the web image databases Flickr and Panoramioare used since they contain a large amount of geo- and user-tagged images and provide a freely available API. Next, avisual matching is applied on the retrieved image collectionto identify images that are visually similar to the input im-

Geotagged, unlabelled Image Iu

Tag Analysis and

Ranking

Iterative Visual Image

Matching

Tag Recommendations

User-tagged Images in the same location

as Iu

User-tagged images in the same location & visually similar to Iu

Geotagged image databases

Tags

Tags

Tags

Tags

Tags

CTags

CTags

Tags

Lat. & lon. of Iu

CTags = Candidate Tags correspond only to tags of the visually similar images

Visual Features of I

u

Phase I Phase III

Geo-based image search

Phase II

Figure 1: Three-phases automatic tag recommendation ap-proach

age. Finally, user-tags of the visual correspondences areanalyzed and used to generate tag recommendations for theinput image. The output of each phase is integrated into aprobabilistic model that generates scores (ranks) represent-ing how suitable is a candidate tag for the input image.

3.1 Problem StatementLet Iu be an unlabeled image taken at a location (lt, ln)

where lt and ln represent the latitude and the longitude ofthe location respectively. Let C = {I1, I2, ..., Im} be a setof images taken in the same geographical area as Iu, suchthat ∀Ii ∈ C , distance(Iu, Ii) < d, where distance(Iu, Ii) isthe geographical distance between the locations of captureof the two images and d is a predefined distance threshold.Furthermore, suppose that members of C are annotated withwords taken from the lexicon W = {w1, w2, ..., wn}. Eachword in W can be used at most for one time to annotatethe same image, however, the same word can be used toannotate more than one image.The introduced components and their relationships can bevisualized by an undirected graph G(V,E) where the verticescorresponds to elements from V = W ∪ C (the union of wordand image sets). Edges connect words and images and aredefined as E = {(w, I)|w ∈W ∧ I ∈ C}. An edge indicatesthat a word has been used to annotate the linked image. Theproblem of recommending a tag to an input image can beexpressed by finding edges between the vertex representingthe input image and the set of vertices which correspond towords. Such edges are represented as dashed lines in Fig.2. To achieve that, an edge can be drawn between Iu and aword w if there is an image Ii ∈ C which is visually identicalto Iu (i.e. a copy of Iu) and annotated with w. However,this assumption is too strict and to loosen it, we assumethat an edge can be drawn between Iu and a word w ifsome kind of similarity relationship exists between Iu andan image Ii which is annotated with w. There are differentpossibilities to derive such a relationship: in this paper, wewill use Bayesian methods to derive a relationship from thevisual similarity between the images Iu and Ii.

3.2 Pseudo-generative Statistical ModelWe frame the problem within a probabilistic model based

on Bayes’ Rule (a.k.a. Total Probability Law). Paradig-matically the Bayes’ Rule is used in settings where, from agenerative model– linking some causes (endowed with an a-priori probability) to some effects – one wants to get theprobability of the effects given a cause. This generativemodel is typically represented by means of a tree, where

w1

wi

wn

I1

Ij

Im

Iu

Inferred Word- Image Relevance

Observed Word-Image Relevance

Figure 2: Graph representation of word-image relevance

the leaves correspond to the effects, (or by means of a morecompact graph where the effects can be shared among dif-ferent causes). Complying to this paradigm, we can identify,for sake of convenience, the input image Iu with the root ofthe tree, and the words wi with the leaves, whereas the im-ages Ij play the role of intermediate nodes, ”caused” by theroot and ”causing”the leaves. Within this pseudo-generativemodel the input image Iu can be thought to ”yield” each ofthe images Ij ∈ C, with a probability value P (Ii|Iu); inturn images from C can be thought to ”yield” the words wi

– which annotate them – with probability P (wi|Ij). Thismodel in the compact version is shown in the graph of Fig.3.

w2

P(Iu)

P(I1|w

1)

P(Ij|w

1)

P(Ij|w

2)

P(Im

|wn)

P(Iu|I

1)

P(Iu|I

2)

w1

wn

I1

Ij

Im

Iu

P(Iu|I

j)

wi

P(Im

|w2)

Figure 3: A Bayesian model for tag recommendation. Iurepresent the input image, Ij ∈ C is the collection of imagesfound in the same geographical area as Iu and wi ∈W arethe set of tags associated with members of C

Let us assume that the values of the conditional prob-abilities P (Ij |Iu) and P (wi|Ij) are available, then we candefine the strength of the relationship of Iu with a word wi

to be equal to the conditional probability P (wi|Iu), which,given an input image Iu, can be computed by means of theBayes’Rule as

P (wi|Iu) =

m∑j=1

P (Ij |Iu)P (wi|Ij)

This corresponds to the sum over all the paths leading fromthe root image Iu to the leaf word wi, of the path’s probabil-ity computed by chain product. Therefore, by construction,only words that are reachable from the input image are con-sidered as candidate tags. Once the probabilities P (wi|Iu)have been computed for all the candidate tag words relevantto a specific input image, the probability values provide anatural ordering of the importance/appropriateness of thewords to the input image, and few of them, the top rankingones, as actual input image tags.

The assumption of the availability of the values of theconditional probabilities P (Ij |Iu) and P (wi|Ij) needs fur-ther consideration: it is a strong assumption, however it canbe easily relaxed. Strictly speaking neither the generativeprocess from Iu to the Ij ’s nor the generative model fromthe Ij ’s to the wi’s are known or defined precisely, hencethe above conditional probabilities cannot be known exactly.However we are not interested in probabilities ”per-se”, butrather in probability values as indicators used eventually forranking the different candidate tag words (for appropriate-ness with respect to a specific input image). Therefore evenquantities proportional to (or simply monotonically depen-dent on) those probabilities will suite the task, because theywill not change the ordering. Furthermore if the probabil-ity gaps between pairs of images (P (Ij |Iu) − P (I ′j |Iu) withIj , I

′j ∈ C) relevant to an input image are wide enough, then

even slightly distorting functions or indicators correlatedwith the P (.|Iu) (proxies of the P (.|Iu)) can suite the task.In an analogous way proxies of the P (.|Iu) can can be used intheir place if the probability gaps between pairs of candidatetag words describing a tagged image (P (wi|Ij) − P (w′

i|Ij)with wj , w

′j ∈ W (Ii)) are wide enough . For those reasons

even if the conditional probabilities P (Ij |Iu) and P (wi|Ij)are not directly available to us, we will adopt the above de-scribed ranking procedure: in place of the probabilities wewill use proxy quantities – respectively an image-to-imagesimilarity measure and a word-to-image importance mea-sure – which are introduced in the next subsection.

3.2.1 Image-to-Image SimilarityThe term P (Ii|Iu) represents the probability of generat-

ing the image Ii from the input image Iu which was taken inthe same geographical neighborhood. There exist differentkinds of information that can be exploited and combined tocalculate this probability. For example, some of this infor-mation can be extracted from user profile similarities andother from the geographical context of the image pair, suchas the geographical distance. In this paper, we only considerimage visual similarity based on low-level image features toestimate this probability.

Visual SimilarityImages which are taken in a certain location vary accordingto the camera perspective, zoom, light conditions, etc. Cor-respondingly, finding visually similar images requires match-ing the images based on low level features that are robustagainst changes in the scale, rotation, skew and illumination.Recently, various approaches for image matching based onfinding correspondences among distinctive points, called in-terest points, have shown a great success in wide range ofapplications. The general approach can be described in athree step process. First, a detector is used to identify in-terest points in the image, such as corners or blobs. Next,

every interest point is represented by a distinctive featurevector known as descriptor. Finally, correspondences aredetermined based on the distance (Euclidean distance orMahalanobis distance) between the two descriptors. Amongthe different image matching algorithms that follow this ap-proach we selected SURF (Speeded Up Robust Features)[2]. The SURF detector and descriptor are scale and rota-tion invariant. A special characteristic of SURF is also thatit uses a small sized descriptor. This allows fast matchingand makes the algorithm suitable for online applications.To achieve this, SURF uses integral images to reduce thecomputation required by a ’Hessian’ detector. Addition-ally, SURF descriptor is built from Haar-wavelet responseswithin the interest point neighborhood and has a size of only64 dimensions. In [8] it has been shown that SURF outper-forms other algorithms from the same family, namely SIFT[13] and PCA-SIFT [9]. The evaluation shows that SURFis faster than the other approaches and provides matchingresults comparable to that of SIFT.To estimate the term P (Ii|Iu), the visual similarity betweenthe two images is calculated. For this purpose, the SURFdescriptor of each image is extracted and common inter-est points are determined. The similarity of the image paircan be calculated by computing the Dice’s (intersection overunion) similarity coefficient, which compares the number ofcommon elements with the total elements of two sets. TheDice’s coefficient is defined as follows:

Sim(Ii, Ij) =2× |IPs(Ii) ∩ IPs(Ij)||IPs(Ii)|+ |IPs(Ij)|

(1)

where:- IPs(Ii): The set of interest points extracted from Ii.- IPs(Ij): The set of interest points extracted from Ij .- IPs(Ii) ∩ IPs(Ij) The set of common interest points. Itconsists of interest point pairs which have minimum Eu-clidean distance between their descriptor vectors4.

Now a proxy of the probability P (Ii|Iu) can be obtainedby normalizing the visual similarity between Ii and Iu ac-cording to the total similarity between Iu and all imagesform C.

P (Ii|Iu) =Sim(Ii, Iu)∑mj=1 Sim(Ij , Iu)

(2)

Iterative Image MatchingApproaches for interest point based image matching fallshort of discovering similarity between two versions of thesame image, for example, if the image is rotated by an an-gle exceeding some threshold. In our scenario, this results inloosing images which are visually similar to the input image,thus, additional candidate tags can also be missed.

The recall of image matching that is based on interestpoints similarities can be improved by applying the match-ing algorithm repetitively. The idea is very simple, givenan input image, visual correspondences that are found bythe matching algorithm can be used as a new input to fur-ther applications of the algorithm. This results in findingan increasing number of visually similar images, thus, thematching recall can be improved. To better understand theidea refer to Fig. 4. An input image shown in Fig. 4a is

4We used a Java implementation of the algorithm using the standard

settings as described by [2]. The implementation is provided by EugenLabun and is available at: http://homepages.thm.de/˜elbn98/imagej-surf/

(a) Input image

(b) A direct match

(c) Indirect matching images

Figure 4: Iterative interest point-based image matching

visually matched based on the SURF descriptor with imagesfound in the same geographical location. One of the foundmatchings is shown in Fig. 4b. Although other images (e.g.Fig. 4c) are also visually similar to the input image theyhave not been discovered by the matching algorithm. Forthis purpose, a new visual matching is performed using theimage in Fig. 4b as input. This results in finding addi-tional matching images as can be seen in Fig. 4c. Thoseimages are, in turn, visually similar to the initial input im-age. The introduced matching approach can be representedin a tree structure5 as it is shown in Fig. 5. The root ofthe tree corresponds to the original input image. Visuallysimilar images which are connected with a single edge areconsidered as a direct match. The iterative matching im-plies an increased computation which reaches its maximumwhen a naive method is applied in which all the matchingimages from a previous iteration are used as input for thenext iteration. To reduce the computation cost, input ofthe next matching iterations have to be wisely selected. Itcan be noticed that visual correspondences of an input im-age can intersect in the sets of interest points which theyshare with the input image. Therefore, it can be sufficientfor the next matching iteration to only consider images thatmatch the input image according to different set of interestpoints. To achieve that, we propose clustering the visualcorrespondences of an input image based on the commoninterest points which they share with the input image. Con-sequently, only one representative image from each cluster

5This tree representation only holds if visual similarities among im-

ages resulting from matching the input image are ignored. Otherwise,the structure corresponds to a graph.

Input Image Possible image

clusters

Direct matching image Matching image from

the first iteration Cluster representative image

Figure 5: Tree representation of iterative image match-ing with clustering considerations. The rounded rectanglearound an image group indicates that those images matchthe input image according to the same (sub)set of inter-est points. Therefore, only one of them, i.e., the clusterrepresentative will be used as input for the next matchingiteration.

can be used as input for the next matching iteration (Fig.5). Creating the clusters also implies comparing image de-scriptors. However, the number of compared descriptors issharply reduced because only the descriptors of the interestpoints that correspond to the input image are considered. Ingeneral, this number corresponds to tens of interest pointsand in most cases to less than ten, while the whole numberof interest points extracted from an image can exceed thou-sands.The effectiveness of the iterative image matching approachand the trade off between using the naive and the clusteringapproach is evaluated in section 4.1.

3.2.2 Candidate Word ImportanceThe term P (wi|Ij) can be interpreted as an indicator to

how mutually discriminative are an image and a word. Wedirectly derive the proxy value for this term form the modelseen in Fig. 3 using a simple frequency based approach asfollows:

P (wi|Ij) =1

Number of words annotating Ij(3)

This measure assumes a uniform probability for all wordsconcerning a given image. This value increases when asmaller number of tags are used to annotate the image. The

value ofm∑

j=1

P (wi|Ij) can be considered as a weighted voting

for the word. Each vote for a word wi given by an image Ijis scaled by the inverse frequency of the words annotatingthat image.In this paper we aim to compare the above presented mea-sure to a TF-IDF (Term Frequency-Inverse Document Fre-quency) based approach. In a similar manner to [14], anadapted version of the TF-IDF measure can be used to esti-mate the word probability while considering a set of imagesannotated with it.Since we assume that each word can be used only once toannotate a given image, one to one correspondence betweenan image and a document results in the same term frequency

value for all tags annotating that image, i.e., one to the totalnumber of tags. To deal with this problem, a more realisticmeasure for term frequency can be defined as follows: letR ⊂ C be the set of visual neighbors of the input image andRw ⊂ R the subset of visual neighbors which are annotatedwith w. We consider Rw as one document and calculate theterm frequency of the word w as follows:

TF (w) =|Rw||R| (4)

The inverse document frequency for w is calculated in atraditional way. Assume that each image in C correspondsto a document and let Cw ⊂ C indicate the subset of imagesin C which are annotated with w, the inverse documentfrequency can be calculated as follows:

IDF (w) = log2(|C||Cw|

) (5)

Now, the TF-IDF importance of a word w can be estimatedas:

P (wi|Ij) =TF (w).IDF (w)

log2|C|(6)

The term log2|C| is used for normalization.

3.2.3 User Related IssuesSince tags are contributed from users, the importance of

a tag as a recommended annotation candidate can be deter-mined by its usage pattern by groups of users. Generally,users tend to use the same set of words to annotate their im-age collections even if the images share a little or no visualsimilarity. This results in increasing the usage frequency ofsome words in a way that does not reflect its importancefor automatic annotation. To address this problem, we pro-pose scaling the importance value of a candidate word bya user’s factor: U . In this respect, we propose two waysto calculate U . One way is to take into consideration thewhole set of unique users who tagged images captured in agiven geographical area. In this case, U can be calculated asthe proportion of unique users who annotated images usingsome word wi to the total number of unique users:

Uall users =number of unique users using wi

total number of unique users(7)

On the other hand, U can be estimated by only consideringunique users who used wi. In this case, U can be given as:

Uusers of wi =number of unique users using wi

total number of occurrences of wi(8)

In the next sections we will show how the quality of the rec-ommended tag can vary according to the provided measuresfor word importance as well as the user considerations.

4. EVALUATIONIn this section different evaluation scenarios are presented.

In the first part, we will show the effect of an iterative im-age matching on improving the matching recall. Hereby, wecompare two variations of the algorithm: a naive and a clus-tering based approach. The evaluation reports the achievedprecision and recall and the runtime taken by each method.The second part is concerned with evaluating the annota-tion performance. For this purpose, we created our groundtruth and used a subset of the NUS-WIDE [3] dataset andcalculated the average precision and recall.

4.1 Effectiveness of Iterative Image MatchingThe ground truth for this test consists of groups of images

crawled from Flickr, Google Images and the European Cities50K dataset [1]. Each group of the datasets contains imageswhich have the same contents but vary in scale, rotation, andillumination. Moreover, the dataset expands over images ofdifferent categories, such as outdoor, indoor, buildings, na-ture etc. In total, we gathered 69 such groups with averageof 70 images each.From each group an image is selected and matched againstthe remaining images of the same group. The decision aboutthe visual similarity between two images is done based onthe number of the interest points they share. It has beenproposed that image matching approaches which are basedon finding interest points correspondences provide a preci-sion of 1 by using 5 common interest points as a threshold[7][10]. To exactly determine this threshold, we matched theinput images to a group of 2000 images covering differentcategories. The results showed that a matching thresholdof 4 common interest points produced zero false positives(precision of 1%) for almost all input images (the results arenot show due to space limitations).We adopted this matching threshold (4 interest points) andused it evaluate the matching recall under different numberof iterations. In the iterative approach the matching im-ages are used as input for further iterative applications ofthe same matching algorithm. The selection of the input im-ages for each matching iteration is done in two ways: a naiveone which considers all images found by a previous iteration,and a clustering based method which uses one representativeimage from each cluster as input. We used agglomerative hi-erarchal clustering to build the clusters. Agglomerative clus-tering is convenient to our case since we don’t have to fix thenumber of the clusters and it is efficient since the numberof the images that will be clustered before each matchingiteration is small. After building the clusters, a random se-lection of a representative image from each cluster is appliedto determine the inputs of the next matching iteration.

Fig. 6a shows the average matching recall for 0 (corre-sponds to a single matching without iterations) to 4 iter-ations. It can be seen, that the iterative approach resultsin significant recall improvement with an increasing numberof iterations. Furthermore, the experiments show that for76% of the cases 3 iterations were enough to achieve themaximum recall. On the other hand, the clustering basedapproach shows slight drop in the average recall compared tothe naive approach, however, it provides better performance(Fig. 6b). The average runtime of the clustering approachis less than that of the naive approach and this differenceincreases by an increasing number of iterations. For exam-ple, by four iterations the clustering based approach is morethan three times faster than the naive approach with a lossof about 0.065 in the recall.

4.2 Quality of Recommended TagsThe performance of the proposed annotation approach is

evaluated by a leave-one-out cross-validation (LOOCV) ap-proach: using a human-produced ground truth set of taggedimages we compare the tags automatically generated withthe actual ones. For this purpose, we created a dataset of100 images and annotated them manually. Each image isgeotagged and described with 21 tags on average. For eachtest image the annotation approach is applied and the pre-

0.625

0.745

0.8460.855

0.567

0.690

0.7720.791

0.5

0.6

0.7

0.8

0.9

Recall

0.313 0.313

0.0

0.1

0.2

0.3

0.4

No Iterations 1 Iteration 2 Iterations 3Iterations 4Iterations

Average Recall - Naive Avg Recall - Clustering

Recall

(a) Average recall at different matching iterations

533.832

637.177

400

500

600

700

(seconds)

71.241

137.536

43.710

66.436

158.289

204.440

0

100

200

300

1 Iteration 2 Iterations 3Iterations 4Iterations

Average Runtime - Naive Average Runtime - Clustering

Tim

e(seconds)

(b) Clustering-based vs. naive iterative matching:Average matching runtime needed by each approachat different matching iterations

Figure 6: Evaluation of iterative image matching approach

cision and recall are calculated by comparing the producedtags to the reference tags. After that, the average of preci-sion and recall are calculated for the total set of test images.To show the effect of the produced ranking, the average pre-cision and recall are demonstrated at different annotationlengths.

Word Importance / User Related FactorWeighted Voting/ - TF-IDF/ -Weighted Voting/Uall users TF-IDF/Uall users

Weighted Voting/Uusers of w TF-IDF/Uusers of w

Table 1: Configurations of the annotation approach

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 5 10 15 20 25 30

Pre

cisi

on

Number of tags (annotation length)

Weighted Voting /no users

Weighted Voting /all users

Weighted Voting /users of w

TF-IDF /no users

TF-IDF /all users

TF-IDF /users of w

Figure 7: Average precision under different configurationsand using ground truth of 100 images and with 0 iterations

The annotation approach can be configured according totwo parameters. First, the word importance which can becalculated in two different ways: a weighted voting (Equa-tion 3) and TF-IDF based approach (Equation 6). The sec-ond parameter is the user related factor U which can alsocan be calculated in two ways (Equations 7 and 8). In to-tal, the two parameters can be combined in 6 different ways(Table 1).Fig. 7 and 8 show that combination between the weighted

0

0.1

0.2

0.3

0.4

0.5

0 5 10 15 20 25 30

Re

call

Number of tags (annotation length)

Weighted Voting /no users

Weighted Voting /all users

Weighted Voting /users of w

TF-IDF /no users

TF-IDF /all users

TF-IDF /users of w

Figure 8: Average recall under different configurations andusing ground truth of 100 images and with 0 iterations

voting and the user factor Uall users provides the best re-call and precision. This is a followed by TF-IDF combinedwith the same user factor. Additionally, we evaluated theannotation approach with a group of 88 images taken fromNUS-WIDE dataset. NUS-WIDE database contains about50 thousands geotagged images. Each image in the databaseis annotated with user tags in a uncontrolled manner, thus,the provided tag are somehow noisy. We cleaned a subsetof the tags by removing irrelevant ones such numbers, stopwords and user names. Fig. 9 and 10 show how the an-notation performance using the weighted voting approachwithout user consideration under 0 and 1 iteration. The re-sults show a clear improvement in the precision and recallwhen the iterative approach is applied. This is due to thefact, that the iterative matching approach results in findingadditional user-tagged images. Consequently, the size of thecandidate tags set as well as the accuracy of the producedtag ranking increase.

5. CONCLUSIONSIn this paper we presented an approach for automatic im-

age annotation. It exploits geotagged images contributed

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Pre

cisi

on

Number of tags (annotation length)

Non-iterative Image Matching

Image Matching with 1 Iteration

Figure 9: Average precision for a subset of NUS-WIDEdataset

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25

Re

call

Number of tags (annotation length)

Non-iterative Image Matching

Image Matching with 1 Iteration

Figure 10: Average recall for a subset of NUS-WIDE dataset

to the web to automatically annotate new untagged im-ages captured in the same location. We showed how usercontributed images can be geographically filtered, visuallymatched, and how their tags can be used to annotate newimages. The extracted information are combined in a flexi-ble statistical model which is able to provide an image-wordrelevance score. A proposal is made to improve the visualmatching by applying additional matching iterations and asolution for improving the performance based on image clus-tering is provided. Finally, the quality of the produced tagsis evaluated by a human created ground truth. In our futurework, we will consider performing further evaluation scenar-ios on larger datasets. Additionally, we aim to extend theprobabilistic model to consider further image-word, image-image and word-word relations. We will also consider en-hancing the performance of the provided approach so thatit can fit an online system.

6. REFERENCES[1] Y. Avrithis, G. Tolias, and Y. Kalantidis. Feature map

hashing: Sub-linear indexing of appearance and globalgeometry. In in Proceedings of ACM Multimedia (Fullpaper) (MM 2010), Firenze, Italy, October 2010.

[2] H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool.Speeded-up robust features (surf). Comput. Vis.Image Underst., 110:346–359, June 2008.

[3] T.-S. Chua, J. Tang, R. Hong, H. Li, Z. Luo, andY.-T. Zheng. Nus-wide: A real-world web image

database from national university of singapore. InProc. of ACM Conf. on Image and Video Retrieval(CIVR’09), Santorini, Greece., July 8-10, 2009.

[4] R. L. Cilibrasi and P. M. B. Vitanyi. The googlesimilarity distance. IEEE Trans. on Knowl. and DataEng., 19:370–383, March 2007.

[5] R. Datta, D. Joshi, J. Li, and J. Z. Wang. Imageretrieval: Ideas, influences, and trends of the new age.ACM Comput. Surv., 40:5:1–5:60, May 2008.

[6] J. Eakins, M. Graham, and J. I. S. C. T. A.Programme. Content-based image retrieval, 1999.

[7] G. J. F. Jones, D. Byrne, M. Hughes, N. E. O’Connor,and A. Salway. Automated annotation of landmarkimages using community contributed datasets and webresources. In T. Declerck, M. Granitzer,M. Grzegorzek, M. Romanelli, S. M. Ruger, andM. Sintek, editors, SAMT, volume 6725 of LectureNotes in Computer Science, pages 111–126. Springer,2010.

[8] L. Juan and O. Gwun. A comparison of sift, pca-siftand surf. International Journal of Image Processing(IJIP), 3(5), 2010.

[9] Y. Ke and R. Sukthankar. Pca-sift: A more distinctiverepresentation for local image descriptors. ComputerVision and Pattern Recognition, IEEE ComputerSociety Conference on, 2:506–513, 2004.

[10] Y. Ke, R. Sukthankar, and L. Huston. Efficientnear-duplicate detection and sub-image retrieval. InACM Multimedia, volume 4, page 5, 2004.

[11] X. Li, C. G. M. Snoek, and M. Worring. Learningsocial tag relevance by neighbor voting. IEEETransactions on Multimedia, 11(7):1310–1322, 2009.

[12] J. Liu, B. Wang, M. Li, Z. Li, W. Ma, H. Lu, andS. Ma. Dual cross-media relevance model for imageannotation. In Proceedings of the 15th internationalconference on Multimedia, MULTIMEDIA ’07, pages605–614, New York, NY, USA, 2007. ACM.

[13] D. G. Lowe. Distinctive image features fromscale-invariant keypoints. Int. J. Comput. Vision,60:91–110, November 2004.

[14] M. Naaman, S. Ahern, R. Nair, and T. Rattenbury.How flickr helps us make sense of the world: contextand content in community-contributed mediacollections. In In Proceedings of the 15th InternationalConference on Multimedia (MM2007, pages 631–640.ACM, 2007.

[15] G. Qiu. Image coding using a coloured patternappearance model. In Visual Communication andImage Processing, 2001.

[16] T. Tachibanaya. Description of exif file format. URLhttp://park2. wakwak.com/tsuruzoh/Computer/Digicams/exif-e. html.February, 2001.

[17] L. Wu, L. Yang, N. Yu, and X.-S. Hua. Learning totag. In 18th International World Wide WebConference, pages 361–361, April 2009.

[18] N. Zhou, W. K. Cheung, G. Qiu, and X. Xue. Ahybrid probabilistic model for unified collaborativeand content-based image tagging. IEEE Trans.Pattern Anal. Mach. Intell., 33:1281–1294, July 2011.