On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices
ABSTRACT A key step for any modeling study is to compare model-produced estimates with observed/reliable data. The original index of agreement (also known as original Willmott index) has been widely used to measure how well model-produced estimates simulate observed data. However, in its original version such index may lead the user to erroneously select a predicting model. Therefore, this study compared the sensibility of the original index of agreement with its two newer versions (modified and refined) and provided an easy-to-use R-code capable of calculating these three indices. First, the sensibility of the indices was evaluated through Monte Carlo Experiments. These controlled simulations considered different sorts of errors (systematic, random and systematic + random) and errors magnitude. By using the R-code, we also carried out a case of study in which the indices are expected to indicate that th empirical Thornthwaite’s model produces poor estimates of daily reference evapotranspiration in respect to the standard method Penman-Monteith (FAO56). Our findings indicate that the original index of agreement may indeed erroneously select a predicting model performing poorly. Our results also indicate that the newer versions of this index overcome such problem, producing more rigorous evaluations. Although the refined Willmott index presents the broadest range of possible values, it does not inform the user if a predicting model overestimate or underestimate the simulated data, resulting in no extra information regarding those already provided by the modified version. None of the indices represents the error as linear functions of its magnitude in respect to the observed process.
Main Authors: | , , , |
---|---|
Format: | Digital revista |
Language: | English |
Published: |
Instituto Agronômico de Campinas
2018
|
Online Access: | http://old.scielo.br/scielo.php?script=sci_arttext&pid=S0006-87052018000200394 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
id |
oai:scielo:S0006-87052018000200394 |
---|---|
record_format |
ojs |
spelling |
oai:scielo:S0006-870520180002003942019-05-16On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indicesPereira,Heloisa RamosMeschiatti,Monica CristinaPires,Regina Célia de MatosBlain,Gabriel Constantino modified index of agreement refined index of agreement model performance ABSTRACT A key step for any modeling study is to compare model-produced estimates with observed/reliable data. The original index of agreement (also known as original Willmott index) has been widely used to measure how well model-produced estimates simulate observed data. However, in its original version such index may lead the user to erroneously select a predicting model. Therefore, this study compared the sensibility of the original index of agreement with its two newer versions (modified and refined) and provided an easy-to-use R-code capable of calculating these three indices. First, the sensibility of the indices was evaluated through Monte Carlo Experiments. These controlled simulations considered different sorts of errors (systematic, random and systematic + random) and errors magnitude. By using the R-code, we also carried out a case of study in which the indices are expected to indicate that th empirical Thornthwaite’s model produces poor estimates of daily reference evapotranspiration in respect to the standard method Penman-Monteith (FAO56). Our findings indicate that the original index of agreement may indeed erroneously select a predicting model performing poorly. Our results also indicate that the newer versions of this index overcome such problem, producing more rigorous evaluations. Although the refined Willmott index presents the broadest range of possible values, it does not inform the user if a predicting model overestimate or underestimate the simulated data, resulting in no extra information regarding those already provided by the modified version. None of the indices represents the error as linear functions of its magnitude in respect to the observed process.info:eu-repo/semantics/openAccessInstituto Agronômico de CampinasBragantia v.77 n.2 20182018-06-01info:eu-repo/semantics/articletext/htmlhttp://old.scielo.br/scielo.php?script=sci_arttext&pid=S0006-87052018000200394en10.1590/1678-4499.2017054 |
institution |
SCIELO |
collection |
OJS |
country |
Brasil |
countrycode |
BR |
component |
Revista |
access |
En linea |
databasecode |
rev-scielo-br |
tag |
revista |
region |
America del Sur |
libraryname |
SciELO |
language |
English |
format |
Digital |
author |
Pereira,Heloisa Ramos Meschiatti,Monica Cristina Pires,Regina Célia de Matos Blain,Gabriel Constantino |
spellingShingle |
Pereira,Heloisa Ramos Meschiatti,Monica Cristina Pires,Regina Célia de Matos Blain,Gabriel Constantino On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
author_facet |
Pereira,Heloisa Ramos Meschiatti,Monica Cristina Pires,Regina Célia de Matos Blain,Gabriel Constantino |
author_sort |
Pereira,Heloisa Ramos |
title |
On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
title_short |
On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
title_full |
On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
title_fullStr |
On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
title_full_unstemmed |
On the performance of three indices of agreement: an easy-to-use r-code for calculating the Willmott indices |
title_sort |
on the performance of three indices of agreement: an easy-to-use r-code for calculating the willmott indices |
description |
ABSTRACT A key step for any modeling study is to compare model-produced estimates with observed/reliable data. The original index of agreement (also known as original Willmott index) has been widely used to measure how well model-produced estimates simulate observed data. However, in its original version such index may lead the user to erroneously select a predicting model. Therefore, this study compared the sensibility of the original index of agreement with its two newer versions (modified and refined) and provided an easy-to-use R-code capable of calculating these three indices. First, the sensibility of the indices was evaluated through Monte Carlo Experiments. These controlled simulations considered different sorts of errors (systematic, random and systematic + random) and errors magnitude. By using the R-code, we also carried out a case of study in which the indices are expected to indicate that th empirical Thornthwaite’s model produces poor estimates of daily reference evapotranspiration in respect to the standard method Penman-Monteith (FAO56). Our findings indicate that the original index of agreement may indeed erroneously select a predicting model performing poorly. Our results also indicate that the newer versions of this index overcome such problem, producing more rigorous evaluations. Although the refined Willmott index presents the broadest range of possible values, it does not inform the user if a predicting model overestimate or underestimate the simulated data, resulting in no extra information regarding those already provided by the modified version. None of the indices represents the error as linear functions of its magnitude in respect to the observed process. |
publisher |
Instituto Agronômico de Campinas |
publishDate |
2018 |
url |
http://old.scielo.br/scielo.php?script=sci_arttext&pid=S0006-87052018000200394 |
work_keys_str_mv |
AT pereiraheloisaramos ontheperformanceofthreeindicesofagreementaneasytousercodeforcalculatingthewillmottindices AT meschiattimonicacristina ontheperformanceofthreeindicesofagreementaneasytousercodeforcalculatingthewillmottindices AT piresreginaceliadematos ontheperformanceofthreeindicesofagreementaneasytousercodeforcalculatingthewillmottindices AT blaingabrielconstantino ontheperformanceofthreeindicesofagreementaneasytousercodeforcalculatingthewillmottindices |
_version_ |
1756375459446128640 |