Calculate proper scores

Calculates proper scores for probabilistic predictions

Usage

proper_score(type, obs, ...)

Arguments

type

A string indicating the type of score to calculate. One of "se", "ae", "ds", or "interval". See Details below.

obs

A vector of observations

...

Additional named arguments needed for each type of score, see Details below.

Value

A vector of calculated scores, one for each observation

Details

Each score type takes additional arguments that define the prediction model information to be used in the score calculations. Unless specified otherwise, the extra arguments should either be scalars (applying to all the predictions) or vectors of the same length as the obs vector.

se

Squared error score. Requires mean, defining the prediction mean

ae

Absolute error score. Requires median, defining the prediction median

ds

Dawid-Sebastiani score. Requires mean and sd, defining the mean and standard deviation of the predictions

interval

Squared error score. Requires lwr and upr, defining the lower and upper prediction interval endpoints, and alpha, defining the prediction error probability targeted by the prediction intervals

The scores \(S_{type}(F_i,y_i)\) are defined as in the lecture notes, with obs[i] equal to \(y_i\) and each prediction distribution \(F_i\) defined by additional named arguments.

Examples

# Five realisations of N(3, 3^2)
obs <- rnorm(5, mean = 3, sd = 3)

# One prediction for each observation. Only one of the predictions describes
# the true model
F_mean <- 1:5
F_median <- 1:5
F_sd <- 1:5
F_lwr <- F_mean - F_sd * qnorm(0.9)
F_upr <- F_mean - F_sd * qnorm(0.1)

# Compute the scores
data.frame(
  se = proper_score("se", obs, mean = F_mean),
  ae = proper_score("ae", obs, median = F_median),
  ds = proper_score("ds", obs, mean = F_mean, sd = F_sd),
  interval = proper_score("interval", obs,
                          lwr = F_lwr, upr = F_upr, alpha = 0.2)
)
#>            se        ae       ds  interval
#> 1  4.84057444 2.2001306 4.840574 11.748893
#> 2  3.11858352 1.7659512 2.165940  5.126206
#> 3 53.46228520 7.3117908 8.137478 42.360671
#> 4  1.03370707 1.0167139 2.837195 10.252413
#> 5  0.01831741 0.1353418 3.219609 12.815516