Conflation of expert and crowd reference data to validate global binary thematic maps
With the unprecedented availability of satellite data and the rise of global binary maps, the collection of shared reference data sets should be fostered to allow systematic product benchmarking and validation. Authoritative global reference data are generally collected by experts with regional knowledge through photo-interpretation. During the last decade, crowdsourcing has emerged as an attractive alternative for rapid and relatively cheap data collection, beckoning the increasingly relevant question: can these two data sources be combined to validate thematic maps? In this article, we compared expert and crowd data and assessed their relative agreement for cropland identification, a land cover class often reported as difficult to map. Results indicate that observations from experts and volunteers could be partially conflated provided that several consistency checks are performed. We propose that conflation, i.e., replacement and augmentation of expert observations by crowdsourced observations, should be carried out both at the sampling and data analytics levels. The latter allows to evaluate the reliability of crowdsourced observations and to decide whether they should be conflated or discarded. We demonstrate that the standard deviation of crowdsourced contributions is a simple yet robust indicator of reliability which can effectively inform conflation. Following this criterion, we found that 70% of the expert observations could be crowdsourced with little to no effect on accuracy estimates, allowing a strategic reallocation of the spared expert effort to increase the reliability of the remaining 30% at no additional cost. Finally, we provide a collection of evidence-based recommendations for future hybrid reference data collection campaigns.
Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
---|---|
Format: | article biblioteca |
Language: | eng |
Subjects: | U30 - Méthodes de recherche, A01 - Agriculture - Considérations générales, C30 - Documentation et information, cartographie de l'utilisation des terres, cartographie de l'occupation du sol, analyse de données, données spatiales, Observation satellitaire, photo-interprétation, approche participative, http://aims.fao.org/aos/agrovoc/c_9000100, http://aims.fao.org/aos/agrovoc/c_9000094, http://aims.fao.org/aos/agrovoc/c_15962, http://aims.fao.org/aos/agrovoc/c_379bbe9f, http://aims.fao.org/aos/agrovoc/c_9000182, http://aims.fao.org/aos/agrovoc/c_16375, http://aims.fao.org/aos/agrovoc/c_9000119, |
Online Access: | http://agritrop.cirad.fr/590246/ http://agritrop.cirad.fr/590246/1/Waldner_RSE_2018.pdf |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
id |
dig-cirad-fr-590246 |
---|---|
record_format |
koha |
institution |
CIRAD FR |
collection |
DSpace |
country |
Francia |
countrycode |
FR |
component |
Bibliográfico |
access |
En linea |
databasecode |
dig-cirad-fr |
tag |
biblioteca |
region |
Europa del Oeste |
libraryname |
Biblioteca del CIRAD Francia |
language |
eng |
topic |
U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 |
spellingShingle |
U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 Waldner, François Schucknecht, Anne Lesiv, Myroslava Gallego, Javier See, Linda Pérez-Hoyos, Ana d'Andrimont, Raphaël de Maet, Thomas Laso Bayas, Juan Carlos Fritz, Steffen Leo, Olivier Kerdiles, Hervé Díez, Mónica Van Tricht, Kristof Gilliams, Sven Shelestov, Andrii Lavreniuk, Mykola Simoes, Margareth Ferraz, Rodrigo P.D. Bellon De La Cruz, Beatriz Bégué, Agnès Hazeu, Gerard Stonacek, Vaclav Kolomaznik, Jan Misurec, Jan Verón, Santiago R. de Abelleyra, Diego Plotnikov, Dmitry Mingyong, Li Singha, Mrinal Patil, Prashant Zhang, Miao Defourny, Pierre Conflation of expert and crowd reference data to validate global binary thematic maps |
description |
With the unprecedented availability of satellite data and the rise of global binary maps, the collection of shared reference data sets should be fostered to allow systematic product benchmarking and validation. Authoritative global reference data are generally collected by experts with regional knowledge through photo-interpretation. During the last decade, crowdsourcing has emerged as an attractive alternative for rapid and relatively cheap data collection, beckoning the increasingly relevant question: can these two data sources be combined to validate thematic maps? In this article, we compared expert and crowd data and assessed their relative agreement for cropland identification, a land cover class often reported as difficult to map. Results indicate that observations from experts and volunteers could be partially conflated provided that several consistency checks are performed. We propose that conflation, i.e., replacement and augmentation of expert observations by crowdsourced observations, should be carried out both at the sampling and data analytics levels. The latter allows to evaluate the reliability of crowdsourced observations and to decide whether they should be conflated or discarded. We demonstrate that the standard deviation of crowdsourced contributions is a simple yet robust indicator of reliability which can effectively inform conflation. Following this criterion, we found that 70% of the expert observations could be crowdsourced with little to no effect on accuracy estimates, allowing a strategic reallocation of the spared expert effort to increase the reliability of the remaining 30% at no additional cost. Finally, we provide a collection of evidence-based recommendations for future hybrid reference data collection campaigns. |
format |
article |
topic_facet |
U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 |
author |
Waldner, François Schucknecht, Anne Lesiv, Myroslava Gallego, Javier See, Linda Pérez-Hoyos, Ana d'Andrimont, Raphaël de Maet, Thomas Laso Bayas, Juan Carlos Fritz, Steffen Leo, Olivier Kerdiles, Hervé Díez, Mónica Van Tricht, Kristof Gilliams, Sven Shelestov, Andrii Lavreniuk, Mykola Simoes, Margareth Ferraz, Rodrigo P.D. Bellon De La Cruz, Beatriz Bégué, Agnès Hazeu, Gerard Stonacek, Vaclav Kolomaznik, Jan Misurec, Jan Verón, Santiago R. de Abelleyra, Diego Plotnikov, Dmitry Mingyong, Li Singha, Mrinal Patil, Prashant Zhang, Miao Defourny, Pierre |
author_facet |
Waldner, François Schucknecht, Anne Lesiv, Myroslava Gallego, Javier See, Linda Pérez-Hoyos, Ana d'Andrimont, Raphaël de Maet, Thomas Laso Bayas, Juan Carlos Fritz, Steffen Leo, Olivier Kerdiles, Hervé Díez, Mónica Van Tricht, Kristof Gilliams, Sven Shelestov, Andrii Lavreniuk, Mykola Simoes, Margareth Ferraz, Rodrigo P.D. Bellon De La Cruz, Beatriz Bégué, Agnès Hazeu, Gerard Stonacek, Vaclav Kolomaznik, Jan Misurec, Jan Verón, Santiago R. de Abelleyra, Diego Plotnikov, Dmitry Mingyong, Li Singha, Mrinal Patil, Prashant Zhang, Miao Defourny, Pierre |
author_sort |
Waldner, François |
title |
Conflation of expert and crowd reference data to validate global binary thematic maps |
title_short |
Conflation of expert and crowd reference data to validate global binary thematic maps |
title_full |
Conflation of expert and crowd reference data to validate global binary thematic maps |
title_fullStr |
Conflation of expert and crowd reference data to validate global binary thematic maps |
title_full_unstemmed |
Conflation of expert and crowd reference data to validate global binary thematic maps |
title_sort |
conflation of expert and crowd reference data to validate global binary thematic maps |
url |
http://agritrop.cirad.fr/590246/ http://agritrop.cirad.fr/590246/1/Waldner_RSE_2018.pdf |
work_keys_str_mv |
AT waldnerfrancois conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT schucknechtanne conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT lesivmyroslava conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT gallegojavier conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT seelinda conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT perezhoyosana conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT dandrimontraphael conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT demaetthomas conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT lasobayasjuancarlos conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT fritzsteffen conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT leoolivier conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT kerdilesherve conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT diezmonica conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT vantrichtkristof conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT gilliamssven conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT shelestovandrii conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT lavreniukmykola conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT simoesmargareth conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT ferrazrodrigopd conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT bellondelacruzbeatriz conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT begueagnes conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT hazeugerard conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT stonacekvaclav conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT kolomaznikjan conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT misurecjan conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT veronsantiagor conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT deabelleyradiego conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT plotnikovdmitry conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT mingyongli conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT singhamrinal conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT patilprashant conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT zhangmiao conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps AT defournypierre conflationofexpertandcrowdreferencedatatovalidateglobalbinarythematicmaps |
_version_ |
1792499606240624640 |
spelling |
dig-cirad-fr-5902462024-01-29T01:28:26Z http://agritrop.cirad.fr/590246/ http://agritrop.cirad.fr/590246/ Conflation of expert and crowd reference data to validate global binary thematic maps. Waldner François, Schucknecht Anne, Lesiv Myroslava, Gallego Javier, See Linda, Pérez-Hoyos Ana, d'Andrimont Raphaël, de Maet Thomas, Laso Bayas Juan Carlos, Fritz Steffen, Leo Olivier, Kerdiles Hervé, Díez Mónica, Van Tricht Kristof, Gilliams Sven, Shelestov Andrii, Lavreniuk Mykola, Simoes Margareth, Ferraz Rodrigo P.D., Bellon De La Cruz Beatriz, Bégué Agnès, Hazeu Gerard, Stonacek Vaclav, Kolomaznik Jan, Misurec Jan, Verón Santiago R., de Abelleyra Diego, Plotnikov Dmitry, Mingyong Li, Singha Mrinal, Patil Prashant, Zhang Miao, Defourny Pierre. 2019. Remote Sensing of Environment, 221 : 235-246.https://doi.org/10.1016/j.rse.2018.10.039 <https://doi.org/10.1016/j.rse.2018.10.039> Conflation of expert and crowd reference data to validate global binary thematic maps Waldner, François Schucknecht, Anne Lesiv, Myroslava Gallego, Javier See, Linda Pérez-Hoyos, Ana d'Andrimont, Raphaël de Maet, Thomas Laso Bayas, Juan Carlos Fritz, Steffen Leo, Olivier Kerdiles, Hervé Díez, Mónica Van Tricht, Kristof Gilliams, Sven Shelestov, Andrii Lavreniuk, Mykola Simoes, Margareth Ferraz, Rodrigo P.D. Bellon De La Cruz, Beatriz Bégué, Agnès Hazeu, Gerard Stonacek, Vaclav Kolomaznik, Jan Misurec, Jan Verón, Santiago R. de Abelleyra, Diego Plotnikov, Dmitry Mingyong, Li Singha, Mrinal Patil, Prashant Zhang, Miao Defourny, Pierre eng 2019 Remote Sensing of Environment U30 - Méthodes de recherche A01 - Agriculture - Considérations générales C30 - Documentation et information cartographie de l'utilisation des terres cartographie de l'occupation du sol analyse de données données spatiales Observation satellitaire photo-interprétation approche participative http://aims.fao.org/aos/agrovoc/c_9000100 http://aims.fao.org/aos/agrovoc/c_9000094 http://aims.fao.org/aos/agrovoc/c_15962 http://aims.fao.org/aos/agrovoc/c_379bbe9f http://aims.fao.org/aos/agrovoc/c_9000182 http://aims.fao.org/aos/agrovoc/c_16375 http://aims.fao.org/aos/agrovoc/c_9000119 With the unprecedented availability of satellite data and the rise of global binary maps, the collection of shared reference data sets should be fostered to allow systematic product benchmarking and validation. Authoritative global reference data are generally collected by experts with regional knowledge through photo-interpretation. During the last decade, crowdsourcing has emerged as an attractive alternative for rapid and relatively cheap data collection, beckoning the increasingly relevant question: can these two data sources be combined to validate thematic maps? In this article, we compared expert and crowd data and assessed their relative agreement for cropland identification, a land cover class often reported as difficult to map. Results indicate that observations from experts and volunteers could be partially conflated provided that several consistency checks are performed. We propose that conflation, i.e., replacement and augmentation of expert observations by crowdsourced observations, should be carried out both at the sampling and data analytics levels. The latter allows to evaluate the reliability of crowdsourced observations and to decide whether they should be conflated or discarded. We demonstrate that the standard deviation of crowdsourced contributions is a simple yet robust indicator of reliability which can effectively inform conflation. Following this criterion, we found that 70% of the expert observations could be crowdsourced with little to no effect on accuracy estimates, allowing a strategic reallocation of the spared expert effort to increase the reliability of the remaining 30% at no additional cost. Finally, we provide a collection of evidence-based recommendations for future hybrid reference data collection campaigns. article info:eu-repo/semantics/article Journal Article info:eu-repo/semantics/publishedVersion http://agritrop.cirad.fr/590246/1/Waldner_RSE_2018.pdf text Cirad license info:eu-repo/semantics/restrictedAccess https://agritrop.cirad.fr/mention_legale.html https://doi.org/10.1016/j.rse.2018.10.039 10.1016/j.rse.2018.10.039 info:eu-repo/semantics/altIdentifier/doi/10.1016/j.rse.2018.10.039 info:eu-repo/semantics/altIdentifier/purl/https://doi.org/10.1016/j.rse.2018.10.039 info:eu-repo/grantAgreement/EC/FP7/603719//(EU) Stimulating Innovation for Global Monitoring of Agriculture and its Impact on the Environment in support of GEOGLAM/SIGMA info:eu-repo/grantAgreement/EC/FP7/617754//(EU) Harnessing the power of crowdsourcing to improve land cover and land-use information/CROWDLAND info:eu-repo/grantAgreement/ERC/FP7/603719//(EU) Stimulating Innovation for Global Monitoring of Agriculture and its Impact on the Environment in support of GEOGLAM/SIGMA info:eu-repo/grantAgreement/ERC/FP7/617754//(EU) Harnessing the power of crowdsourcing to improve land cover and land-use information/CROWDLAND |