Detailed definitions of some of the more technical terms and a comparison of the mathematical properties of the analysed evaluation metrics are found in Table 2.

Below are the equations of four simple evaluation metrics (variables a, b, c and d according to Table 1). First, the hit rate (H, also termed sensitivity) measures the ratio of true presences to the sum of true presences and omission errors while completely ignoring commission errors and the number of true absences. Second, the false positive rate (F), equal to 1–specificity, measures the ratio of commission errors to the sum of commission errors and true absences. Third, bias (also termed bias score or frequency bias) measures the ratio of commissions to omissions and helps to identify over-/under-predicting models. And fourth, prevalence (also termed base rate) measures the ratio of presences (both predicted and omitted) to all cells and hence expresses how common within the study area a species is, according to the available data.

H = a/(a + c) (1)

F = b/(b + d) (2)

bias = (a + b)/(a + c) (3)

prevalence = (a + c)/(a + b + c + d) (4)

TSS measures the difference between H and F and was first developed as Peirce’s skill score in meteorology (Peirce 1884). It was later introduced to other fields, including the field of SDM, where it replaced kappa (Cohen 1960) and its strong unimodal response to prevalence (Allouche et al. 2006).

TSS = H – F (5)

ORSS measures skill compared to a random prediction, is a synonym of Yule’s Q (1900) and was introduced to meteorology by Stephenson (2000). ORSS provides equal error weighting but rapidly converges to one even for imperfect predictions (Woodcock 1976) and hence, requires significance testing to quantify skill and to discern real skill from chance (Stephenson 2000).

ORSS = (ad – bc)/(ad + bc) (6)

Ferro and Stephenson (2011) developed the Symmetric Extremal Dependence Index (SEDI) as an improvement on earlier work by Stephenson et al. (2008) and Hogan et al. (2009). SEDI featured greatly reduced sensitivity to prevalence, while retaining most beneficial properties of its predecessors (Ferro and Stephenson 2011). This includes asymptotic equitability, i.e. the ability to distinguish random and skilled predictions at smaller than infinite sample sizes (Hogan et al. 2010). Yet, SEDI is not applicable if any of the four cells in the confusion matrix equals zero (Ferro and Stephenson 2011) since log (0) yields infinity. Overfitted or misspecified models in these instances, however, can still be interpreted by adding an infinitely small number to those cells containing zeros. Our implementation of SEDI (see Supplementary Information for repository link) also issues a character string indicating if such approximations were used and how to best interpret the result.

(7)

Using two types of extreme prediction settings (“Optimistic” and “Pessimistic”) and two types of typical species prevalence settings (“Differential Bias” and “Changing Bias”), we investigated the response of TSS, ORSS and SEDI to increasing cell number total in the confusion matrix, varying commission error rates, omission error rates and species prevalence. These settings were each divided into two theoretical scenarios: “Incorrectly Optimistic” (IO) and “Correctly Optimistic” (CO), “Correctly Pessimistic” (CP) and “Incorrectly Pessimistic” (IP), “Commission Bias” (CB) and “Omission Bias” (OB) and “Low Commission Rate” (LC) and “High Commission Rate” (HC). For each of the eight scenarios (Table 3), we prepared twenty cases, i.e. confusion matrices, to be evaluated (Table 3). In “Changing Bias” scenarios (LC and HC), the term ‘logistic’ was used to describe model fit which improved with background point totals and cell number totals but plateaued below perfect fit.

Although not mutually exclusive, each scenario is designed to reflect signals that could have arisen from biological signals or artifacts, thereby revealing how susceptible model evaluation metrics are to conflating the two. Below, we briefly describe all scenarios and Fig. 1 visualises selected example cases. Across the theoretical cases in the different scenarios, the cell number total (a + b + c + d) approximately ranges from 1240 to 42560 with increasingly large step size (Table 3) to allow the analysis of model evaluation across a large range of modelling conditions without requiring an overly large number of cases. The R code to reproduce our analysis is available upon request.

Figure 1.

Potential spatial distributions of confusion matrix categories corresponding to values of the True Skill Statistic (TSS), the Odds Ratio Skill Score (ORSS) and the Symmetric Extremal Dependence Score (SEDI) for selected scenario cases.

Scenarios IO, CO, CP and IP were designed to demonstrate how evaluation metrics at essentially constant extreme levels of prevalence react to an increasing cell number total in the confusion matrix. The biological component of these scenarios is analogous to specialist or generalist species that have a constant prevalence of 0.5% or 99.5% of the study area. The artefactual component is related to the implications of study area increases for the number of background points and total number of cells and their effect on the calculation of evaluation metrics. Scenario IO was characterised by large numbers of commission errors as it evaluated an extreme incorrectly optimistic modelling prediction (over-prediction) when true species prevalence is equal to 0.5%, reflecting extreme specialisation or rarity and under increasing background size. Scenario CO was identical to scenario IO in its extreme prediction. However, as true species prevalence was equal to 99.5% (reflecting extremely low specialisation), it no longer resembled an over-prediction and was consequently dominated by true presences. Scenario CP evaluated an extreme correctly pessimistic prediction when true species prevalence was equal to 0.5%, reflecting a high degree of ecological specialisation and species presence was only predicted for a small proportion of the study area, under increasing background size. This scenario was characterised by large numbers of true absences. Scenario IP was identical to scenario CP in its extreme prediction but dominated by false absences since true species prevalence was now equal to 99.5%, turning it into a gross under-prediction.

Scenarios CB and OB were designed to reveal the effect of bias on evaluation metrics under decreasing prevalence (~17% to ~0.5%) as the study area increased. In these scenarios, evaluation metrics should consistently penalise model predictions according to the degree of their bias, across the whole range of prevalence. Scenario CB was more optimistic (more commission errors and more predicted presences) than scenario OB (more omissions and fewer predicted presences). Therefore, the two scenarios together can be seen as a test of transpose symmetry.

Scenarios LC and HC examined the response of evaluation metrics to changes in bias while model fit (i.e. the number of true positives) and the total number of cells increased as prevalence decreased (~17% to ~0.5%). More specifically, the number of observations was held constant in both scenarios, while the numbers of true positives and omissions increased and decreased, respectively. However, at the same time, commission errors became more frequent. In other words, the bias of the model changed together with prevalence and the size of the study area. Scenarios LC and HC differed only in their rate of commission errors which was three times higher in scenario HC than in scenario LC. The biological component could represent increasing specialisation of a given species as the study extent increases; whereas the artefactual component could represent resultant increases in model fit as increasing specialisation makes for easier characterisation (Jiménez-Valverde et al. 2008).

TSS shows a strong response to increased study area size and, hence, confusion matrix cell number totals and rapidly converges to zero, rendering indifferent useful and random models beyond cell number totals in the confusion matrix of approximately 30,000 cells. SEDI shows only a moderate response and converges much later to zero. Of note, H completely fails in this respect since both incorrectly optimistic predictions (IO) and correctly pessimistic predictions (CP) converge to one and zero, respectively and yield scores very similar to those of their correct (CO) and incorrect (IP) counterparts. Finally, SEDI has stronger discriminatory power than TSS at intermediate study areas yet, only ORSS is expected to correctly assess model performance as study area size converges to infinity (Fig. 2a, b).

Figure 2.

Plots of the values of hit rate (H), the True Skill Statistic (TSS), the Odds Ratio Skill Score (ORSS) and the Symmetric Extremal Dependence Score (SEDI) for all eight scenarios. Panels a, b, c, d display scenarios “Incorrectly Optimistic” and “Correctly Optimistic”, “Correctly Pessimistic” and “Incorrectly Pessimistic”, “Commission Bias” and “Omission Bias” and “Low Commission Rate” and “High Commission Rate”. In panels a and b, the x-axis denotes the log of the total number of cells, i.e. the size of the study area, whereas in panels c and d, the x-axis denotes prevalence (%).

TSS quickly converges with H and always favours over-predictions to under-predictions. However, the degree to how much over-predictions are favoured increases as prevalence decreases. Although SEDI also favours over-predictions, it does so to a much smaller degree and is not significantly affected by prevalence. Just as in scenarios CO and CP, ORSS rapidly converges to one (Fig. 2c). Under increasing study area and background point totals and so decreasing prevalence, TSS converges quickly with H for the most realistic scenarios (LC and HC in Fig. 2d) and models that predict increasing amounts of both true presences and commission errors, as omission errors, decrease. At higher levels of prevalence, TSS can still discern the quality of models differing only in their rate of commission errors, but once prevalence falls below approximately 2.5%, their difference becomes indistinguishable. SEDI can assess the quality of models differing only in their rate of commission errors as prevalence decreases to almost zero. As in previous scenarios, since ORSS rapidly converges to one, model scores differing only in their rate of commission errors become indistinguishable even faster than when using TSS. In addition, the proportional difference between LC and HC SEDI scores and TSS scores are lower at the start though increase as prevalence decreases (Fig. 3). This indicates that, although TSS identifies differences in commission error levels at high prevalence (greater than approximately 7%) better than SEDI, the reverse is true at the low prevalence levels typically encountered in presence-background modelling.

Figure 3.

Proportional difference between scenarios “Low Commission Rate” (LC) and “High Commission Rate” (HC) for (in dark green) the Symmetric Extremal Dependence Score (SEDI) vs. the True Skill Statistic (TSS), (in orange) the Odds Ratio Skill Score (ORSS) vs. TSS and (in blue) SEDI vs. ORSS. The black, horizontal, dashed line represents equal differentiation. As there are slight differences in prevalence between scenarios LC and HC, the x-axis shows the mean prevalence for given cases across both scenarios.

By grossly over- or under-predicting the distribution of a hypothetical target species, we observed the response of evaluation metrics to extreme biases in less realistic scenarios. These extreme scenarios, however, have also shown that discernment of strongly and weakly performing models greatly differs amongst evaluation metrics and modelling conditions. While these scenario results support the use of ORSS for large study extents, because of its rapid convergence to one, even for imperfect predictions (Woodcock 1976), significance testing is required in order to determine the quality of and to allow comparisons across models (Stephenson 2000). Therefore, SEDI is preferable as it allows direct assessments of model quality across a much larger spectrum of study extents. Further, SEDI assessments are made with higher discriminatory power than TSS, which rapidly converges to zero for extreme predictions.

These scenarios have been designed to reflect common modelling conditions in order to observe the response of evaluation measures to differential (CB and OB) and changing (LC and HC) biases, under decreasing prevalence as the size of the study area and, hence, the number of background points increased. Analysis of these scenarios revealed very distinct responses to the differing modelling conditions. Results for scenarios CB and OB and LC and HC suggest the use of SEDI since: 1) TSS encourages over-predictions due to its strongly biased treatment of errors which increases as prevalence decreases; 2) TSS quickly loses the discriminatory power to differentiate between models, differing only in their commission rate as it always converges to H; and 3) ORSS converges to one so rapidly (Stephenson 2000) that such differences vanish at even higher prevalence levels than when using TSS.

Our analysis reaffirms the importance of selecting model evaluation metrics corresponding with modelling questions and conditions (Woodcock 1976). Important modelling conditions include the extent of the study area (Termansen et al. 2006), prevalence (Doswell et al. 1990; Stephenson 2000; Somodi et al. 2017; Leroy et al. 2018) and the relative severity of model error types due to varying degrees of biological and artefactual causes. Important biological factors, related to the extent of the study area and species prevalence, include the degree of specialisation within the given landscape (Jiménez-Valverde et al. 2008); population sinks (Guisan and Thuiller 2005); and species equilibrium (Václavík and Meentemeyer 2012). This latter factor is also important in the context of invasive species (Ward 2007). Artifacts can originate from the cell number total in the confusion matrix which is largely driven by background points and hence dependent on modelling resolution (Seo et al. 2008) and study extent. Artefactual signals can also be caused by prevalence and its interactions with sample-size (Somodi et al. 2017) and the degree of confidence in absence data (Leroy et al. 2018) which is influenced by species mobility (Jaberg and Guisan 2001) and a multitude of other factors.

Our results suggested a limited capacity of TSS to provide consistent performance comparisons across varying modelling conditions. This is worrying because TSS may yield misleading estimates of model fidelity, which can lead to the selection of inadequate models. Although it may be tempting to assume that researchers would recognise anomalous conditions where TSS scores are misleading (such as those presented here), this is not necessarily the case – as demonstrated by the broad and seemingly uncritical application of TSS in presence-background SDM over the last decade. Complications, owing to the relative inability of TSS to provide information on commission errors as prevalence approaches zero, are more nuanced. Ultimately, such complications are only problematic in as much as commission errors matter, which depends on bias > 1, the question, the available data and the biology of the species. In fact, it is widely acknowledged that even absence data from professional surveys have greater degrees of both sampling and ecological uncertainty than presence data (MacKenzie et al. 2017). Therefore, one could argue that errors of omission, i.e. predicting absences where there are species observations, are far more grievous than commission errors in model evaluation, particularly when using presence-background data. This is because commission errors can be relatively obscure and should be given less weight than omission errors in model evaluation (Braunisch and Suchant 2010; Liu et al. 2013). Taking the above reasoning to its logical extreme then, commission errors are irrelevant when considering pseudo-absence data.

While the above reasoning is persuasive, simply ignoring commission errors in presence background data by limiting evaluation to H, is not a viable option under all but a small subset of questions, modelling conditions and biological assumptions. More specifically, doing so would be incongruent with the biological circumstances, sampling realities and the intents of most modelling initiatives. Further, evaluation scores would become more vulnerable to artificial inflation. From a biological perspective, model evaluation metrics that ignore commission errors are equivalent to assuming that all background points are locations where the species is present but unobserved. That is, assuming that observed presence locations may represent the subset of relative high use or occupied conditions within local settings (Elith et al. 2011). While this may be a good assumption for situations where habitat use of a generalist species is considered (particularly if survey effort has been uniform across the entire study area), it is unrealistic and impractical for many modelling initiatives that attempt to characterise either the fundamental or realised niche (Elith and Leathwick 2009), based on incomplete sampling regimes.

Furthermore, even when the above biological assumptions and survey prerequisites are valid, explicitly choosing to ignore commission errors further assumes that unobserved locations are irrelevant—an assumption that is seldom the case since these locations may correspond to low population density areas (Guisan and Thuiller 2005). Models (and subsequent thresholds) that commit lower numbers of commission errors may better differentiate between high and low population density areas, a highly desirable characteristic for assessing population status (Tôrres et al. 2012). Of course in rarer situations, strictly considering omission errors in model evaluation may be desirable such as when assessing the impacts of decisions on vulnerable species (Karl et al. 2000). In addition, ignoring commission errors can lead to unintended consequences as seen in Fig. 2a where H is deceptively high for incorrectly optimistic outputs, i.e. over-predicting models. Although commission errors should be weighted less than omission errors in most SDM initiatives, as accomplished by both TSS and SEDI, this does not mean that they are irrelevant or become irrelevant when prevalence decreases. Exceptions to this are very specific circumstances.

This study demonstrated that more consistent commission error weighting (as with SEDI) also circumvents a number of potentially artefactual signals as prevalence approaches zero. We also discussed the relative inability of TSS to compare performance across modelling conditions. For these reasons, whereas maximising TSS may be instrumental when presence-absence thresholds are required (Liu et al. 2013), TSS may not perform well in presence-background model evaluations (Somodi et al. 2017; Leroy et al. 2018). Unless circumstances require down-weighting of commission errors as prevalence decreases, SEDI’s ability to take into account information on both error types across a wide range of modelling conditions makes it a better choice for low prevalence conditions, characteristic of presence-background modelling approaches.

The use of similarity measures as an alternative to TSS has recently been suggested by Leroy et al. (2018). However, similarity measures are only applicable when there is a known truth such as when modelling virtual species (see Hirzel et al. 2001; Leroy et al. 2016 for an introduction to virtual species). One might also consider the use of bootstrapping and related techniques (Efron 1983) such as sub-sampling in model evaluation (Verbyla and Litvaitis 1989), i.e. the repeated sub-sampling of background points to the number of presences and averaging of the resulting scores. Sub-sampling would remove the bias caused by large numbers of background points towards true absences from confusion matrices altogether, albeit at the cost of being computationally intensive. Therefore, alternative measures such as SEDI or, for some specific cases, ORSS, appear superior as they are neither limited to virtual species nor costly in terms of computation.