A new metric to assess the predictive accuracy of multinomial land cover models
Aim: The earth's land cover is often represented by discrete classes, and predicting shifts between these classes is a major goal in the field. One increasingly common approach is to build models that predict land cover classes with probabilities rather than discrete outcomes. Current assessment approaches have drawbacks when applied to these types of models. In this paper we present a new metric, which assesses agreement between model predictions and observations, while correcting for chance agreement. Location: Global. Methods: κmultinomial is the product of two metrics: the first component measures the agreement in the ranks of the predicted and observed classes, the other specifies the certainty of the model in the case of discrete observations. We analysed the behaviour of κmultinomial and two alternative metrics: Cohen's Kappa (κ) and an extension of the area under receiver operating characteristic Curve to multiple classes (mAUC) when applied to multinomial predictions and discrete observations. Results: Using real and synthetic datasets, we show that κmultinomial - in contrast to κ - can distinguish between models that are very far off versus slightly off. In addition, κmultinomial ranks models higher that predict observed classes with an onaverage higher probability. In contrast, mAUC gives the same score to models that are perfectly able to discriminate among classes of outcomes regardless of the certainty with which those classes are predicted. Main conclusions: With κmultinomial we have provided a tool that directly uses the multinomial probabilities for accuracy assessment. κmultinomial may also be applied to cases where model predictions are evaluated against multiple sets of observations, at multiple spatial scales, or compared to reference models. As models develop we assess how well new models perform compared to the real world.
| Main Authors: | , , |
|---|---|
| Format: | Article/Letter to editor biblioteca |
| Language: | English |
| Subjects: | Cohen's kappa, Kappa multinomial, Land cover, Model predictive accuracy, Multinomial models, Multiple class AUC, Validation, |
| Online Access: | https://research.wur.nl/en/publications/a-new-metric-to-assess-the-predictive-accuracy-of-multinomial-lan |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| id |
dig-wur-nl-wurpubs-520135 |
|---|---|
| record_format |
koha |
| spelling |
dig-wur-nl-wurpubs-5201352024-12-04 Douma, Bob Cornwell, William K. van Bodegom, Peter M. Article/Letter to editor Journal of Biogeography 44 (2017) 6 ISSN: 0305-0270 A new metric to assess the predictive accuracy of multinomial land cover models 2017 Aim: The earth's land cover is often represented by discrete classes, and predicting shifts between these classes is a major goal in the field. One increasingly common approach is to build models that predict land cover classes with probabilities rather than discrete outcomes. Current assessment approaches have drawbacks when applied to these types of models. In this paper we present a new metric, which assesses agreement between model predictions and observations, while correcting for chance agreement. Location: Global. Methods: κmultinomial is the product of two metrics: the first component measures the agreement in the ranks of the predicted and observed classes, the other specifies the certainty of the model in the case of discrete observations. We analysed the behaviour of κmultinomial and two alternative metrics: Cohen's Kappa (κ) and an extension of the area under receiver operating characteristic Curve to multiple classes (mAUC) when applied to multinomial predictions and discrete observations. Results: Using real and synthetic datasets, we show that κmultinomial - in contrast to κ - can distinguish between models that are very far off versus slightly off. In addition, κmultinomial ranks models higher that predict observed classes with an onaverage higher probability. In contrast, mAUC gives the same score to models that are perfectly able to discriminate among classes of outcomes regardless of the certainty with which those classes are predicted. Main conclusions: With κmultinomial we have provided a tool that directly uses the multinomial probabilities for accuracy assessment. κmultinomial may also be applied to cases where model predictions are evaluated against multiple sets of observations, at multiple spatial scales, or compared to reference models. As models develop we assess how well new models perform compared to the real world. en application/pdf https://research.wur.nl/en/publications/a-new-metric-to-assess-the-predictive-accuracy-of-multinomial-lan 10.1111/jbi.12983 https://edepot.wur.nl/414484 Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation Wageningen University & Research |
| institution |
WUR NL |
| collection |
DSpace |
| country |
Países bajos |
| countrycode |
NL |
| component |
Bibliográfico |
| access |
En linea |
| databasecode |
dig-wur-nl |
| tag |
biblioteca |
| region |
Europa del Oeste |
| libraryname |
WUR Library Netherlands |
| language |
English |
| topic |
Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation |
| spellingShingle |
Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation Douma, Bob Cornwell, William K. van Bodegom, Peter M. A new metric to assess the predictive accuracy of multinomial land cover models |
| description |
Aim: The earth's land cover is often represented by discrete classes, and predicting shifts between these classes is a major goal in the field. One increasingly common approach is to build models that predict land cover classes with probabilities rather than discrete outcomes. Current assessment approaches have drawbacks when applied to these types of models. In this paper we present a new metric, which assesses agreement between model predictions and observations, while correcting for chance agreement. Location: Global. Methods: κmultinomial is the product of two metrics: the first component measures the agreement in the ranks of the predicted and observed classes, the other specifies the certainty of the model in the case of discrete observations. We analysed the behaviour of κmultinomial and two alternative metrics: Cohen's Kappa (κ) and an extension of the area under receiver operating characteristic Curve to multiple classes (mAUC) when applied to multinomial predictions and discrete observations. Results: Using real and synthetic datasets, we show that κmultinomial - in contrast to κ - can distinguish between models that are very far off versus slightly off. In addition, κmultinomial ranks models higher that predict observed classes with an onaverage higher probability. In contrast, mAUC gives the same score to models that are perfectly able to discriminate among classes of outcomes regardless of the certainty with which those classes are predicted. Main conclusions: With κmultinomial we have provided a tool that directly uses the multinomial probabilities for accuracy assessment. κmultinomial may also be applied to cases where model predictions are evaluated against multiple sets of observations, at multiple spatial scales, or compared to reference models. As models develop we assess how well new models perform compared to the real world. |
| format |
Article/Letter to editor |
| topic_facet |
Cohen's kappa Kappa multinomial Land cover Model predictive accuracy Multinomial models Multiple class AUC Validation |
| author |
Douma, Bob Cornwell, William K. van Bodegom, Peter M. |
| author_facet |
Douma, Bob Cornwell, William K. van Bodegom, Peter M. |
| author_sort |
Douma, Bob |
| title |
A new metric to assess the predictive accuracy of multinomial land cover models |
| title_short |
A new metric to assess the predictive accuracy of multinomial land cover models |
| title_full |
A new metric to assess the predictive accuracy of multinomial land cover models |
| title_fullStr |
A new metric to assess the predictive accuracy of multinomial land cover models |
| title_full_unstemmed |
A new metric to assess the predictive accuracy of multinomial land cover models |
| title_sort |
new metric to assess the predictive accuracy of multinomial land cover models |
| url |
https://research.wur.nl/en/publications/a-new-metric-to-assess-the-predictive-accuracy-of-multinomial-lan |
| work_keys_str_mv |
AT doumabob anewmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels AT cornwellwilliamk anewmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels AT vanbodegompeterm anewmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels AT doumabob newmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels AT cornwellwilliamk newmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels AT vanbodegompeterm newmetrictoassessthepredictiveaccuracyofmultinomiallandcovermodels |
| _version_ |
1819147263478333440 |