13 Unfinished Odds and Ends
13.1 Probability, Odds, and Logits
13.1.1 Odds
Most of us are comfortable with the notion of probability, but we find odds to be a little harder to understand. In everyday language, people use probability and odds interchangeably. Technically, they are related, but not at all the same thing.
When the odds of winning are 3 to 1, there will be 3 wins for every loss.
As seen in Figure 13.1, the relationship between probability and odds is not linear. To convert an odds ratio to probability:
p=\frac{odds}{1 + odds}
Thus, 3 to 1 odds is:
<- 3 / 1
odds
<- odds / (1 + odds)
p p
[1] 0.75
Or, 2 to 3 odds is:
<- 2 / 3
odds <- odds / (1 + odds)
p p
[1] 0.4
To convert probability to odds, one can simply use this formula:
Odds = \frac{p}{1-p}
If the probability of an event is .8, then the odds are:
<- .8
p <- p / (1 - p)
odds odds
[1] 4
Probabilities bounded by 0 and 1, inclusive. Odds have a minimum of 0 but have no upper bound (See Figure 13.1).
In many statistical applications, we need to convert a probability to a value that has neither upper nor lower bounds. A logit maps probabilities onto real numbers from negative to positive infinity (See Figure 13.2).
A logit is the log-odds of probability.
\text{logit}\left(p\right)=\ln\left(\frac{p}{1-p}\right)
13.1.2 W Scores
In the same way that we transform z-scores to various kinds of standard scores, we can transform logits to other kinds of scales. The most prominent in psychological assessment is the W-score (AKA Growth Score Values), developed by Woodcock & Dalh (1971). An accessible discussion of its derivation can by found in Benson et al. (2018).
If ability \theta is in logits, then W is calculated like so:
W = \frac{20}{\ln(9)}\theta+500
Although the coefficient \frac{20}{\ln(9)} may seem strange, it has some desirable features. It is equivalent to 20 times the base-9 logarithm of e:
20 \log_9(e) = \frac{20}{\ln(9)} \approx 9.1024
This unusual coefficient yields some nice round probabilities of answering an item correctly when the person’s ability and the item’s difficulty differ by 0, 10, or 20 points (See Figure 13.3).
The probability p of answering an item correctly depends on the difference between the item’s difficulty W_D and the person’s ability W_A. Specifically, the relationship in Figure 13.3 works according to this equation:
p = \left(1 + 9^{\left(W_D-W_A\right)/20}\right)^{-1}
13.2 Relative Proficiency
One of the benefits of item response theory models is that ability is expressed on the same scale as item difficulties. This feature allows us to make predictions about how like a person with a particular ability level will correctly answer an item of a particular difficulty.
The Relative Proficiency Index (RPI) was first used in the Woodcock Reading Mastery Tests (Woodcock, 1973). The RPI answers the following question:
When the average same-age peer with ability \mu_\theta has probability p of answering an item, what is the probability of answering correctly for a person of ability \theta?
The RPI can be calculated like so:
RPI = \left(1+e^{-\left(\theta-\mu_{\theta}+\text{logit}(p)\right)}\right)^{-1} \begin{align*} \theta &= \text{Ability level of person (in logits)}\\ \mu_{\theta} &= \text{Average ability level (in logits) of a reference group}\\ p &= \text{Probability a person with ability } \mu_\theta \text{ will answer correctly}\\ \text{logit}(p) &= \ln\left(\frac{p}{1-p}\right) = p \text{ converted to logits} \end{align*}
The psycheval package can calculate the RPI using the the rpi
function. By default, the primary inputs x
and mu
are assumed to be on the W scale, and the criterion p is .90.
Suppose a person’s ability corresponds to a W-score of 460 and same-age peers have an average W score of 500:
library(psycheval)
rpi(x = 460, mu = 500)
0.1
This difference of 40 W score points means that when typical same-age peers have a 90% chance of answering an item correctly (a common benchmark for mastery), this person has a 10% chance of answering the item correctly.
If x
and mu
are logits, then you can specify scale = 1
like so:
rpi(x = 1, mu = 0, scale = 1)
0.9607297
The RPI works nicely for documenting deficits, but for gifted students, the RPI is quite high, often near 1. In such cases, we can also calculate the probability a person with ability \mu_\theta can answer an item that a person with ability \theta has a probability p_\theta of answering correctly:
RPI_{\text{reversed}} = \left(1+e^{-\left(\mu_\theta-\theta+\text{logit}(p_\theta)\right)}\right)^{-1}
Suppose a person has a W score of 550, which is 50 points higher than typical same-age peers. The standard RPI will give a value close to 1:
rpi(x = 550, mu = 500)
0.999543
This means that when same-age peers have a 90% chance of answering the item correctly, this person is almost certain to answer it correctly. Unfortunately, this fact does not convey the degree of giftedness in an evocative manner.
To get a better sense of how far advanced this person is compared to the performance of typical same-age peers, we can reverse the RPI like so.
rpi(x = 550, mu = 500, reverse = TRUE)
0.03571429
This means that when this person has a .9 probability of answering an item correctly, the typical same-age peer has about a .036 probability of answering it. Thus, this person is capable of completing tasks that are quite difficult for typical same-age peers.
The standard RPI refers to a proficiency level of .9, but the rpi
function can calculate the relative proficiency index at any criterion level. For example:
rpi(x = 550, mu = 500, criterion = .10)
0.9642857
This means that when a typical same-age peer has a .10 probability of answering an item correctly, this person will answer it correctly with a .96 probability.
An Excel spreadsheet that calculates this “generalized” RPI can be found here.
13.3 Thresholds
The traditional threshold for diagnosing intellectual disability is 70. If a person’s observed IQ \left(r_{XX}=.97\right) is 68, what is the probability that the person’s true score is 70 or less?
More generally, given an observed score X with reliability coefficient r_{XX}, what is the probability that the associated true score T is less than or equal to threshold \tau?
When we predict the true score T with a specific value of X, the estimated true score \hat{T} is:
\hat{T}=r_{XX}\left(X-\mu_X\right)+\mu_X
The standard error of the estimate \sigma_{T-\hat{T}} in this prediction is:
\sigma_{T-\hat{T}}=\sigma_X\sqrt{r_{XX}-r_{XX}^2}
If we have reason to assume that the prediction errors are normally distributed, the probability that the true score T is less than or equal to threshold \tau can be calculated using the standard normal cumulative distribution function \Phi like so:
P(T\le\tau)=\Phi\left(\frac{\tau-\hat{T}}{\sigma_{T-\hat{T}}}\right)
Using this formula, we can see that our hypothetical person with IQ = 68, the probability that the true score is 70 or lower is about .66. Figure 13.4 shows the probabilities for all values near the diagnostic threshold of 70.
As a shortcut to using the formulas displayed above
13.3.1 Multivariate Thresholds
To diagnose intellectual disability, we need a standardized measure of intellectual functioning (usually an IQ test) and well-validated measure of adaptive functioning. Suppose our two measures correlate at r = .40. The reliability coefficient of the IQ is r_{iq}=.97, and the reliability coefficient of the adaptive behavior composite is r_{ab}=.95. Both measures have a mean of \mu=100 and a standard deviation of \sigma=15.
Suppose that a person with IQ = 68 has an adaptive behavior composite of 67. What is the probability that both true scores are 70 or lower?
The vector of observed scores is:
X=\{68, 67\}
The vector of reliability coefficients:
r_{XX}=\{.97,.95\}
The correlation matrix is:
R_X=\begin{bmatrix} 1&.97\\ .97&1 \end{bmatrix}
The vector of means is:
\mu_X=\{100,100\}
The vector of standard deviations is:
\sigma_X=\{15,15\}
The observed covariance matrix is:
\Sigma_X=\sigma_X^\prime R_X\sigma_X
The true score covariance matrix is the same as the observed score covariance matrix except that the diagonal of \Sigma_X is multiplied by the vector of reliability coefficients r_{XX}:
\Sigma_T=\Sigma_X \circ \begin{bmatrix}.97&1\\1&.95\end{bmatrix}
The cross-covariances between X and T also equal \Sigma_T.
We can use equations from Eaton (2007, p. 116) to specify the conditional means and covariance matrix of the true scores, controlling for the observed scores:
\begin{align} \mu_{T\mkern 2mu\mid X}&=\mu_X+\Sigma_T\Sigma_X^{-1}\left(X-\mu_X\right)\\ \Sigma_{T\mkern 2mu\mid X}&=\Sigma_T-\Sigma_T\Sigma_X^{-1}\Sigma_T^\prime \end{align}
We can imagine that the true scores conditioned on the observed scores are multivariate normal variates:
\left(T\mid X\right)\sim\mathcal{N}\left(\mu_{T\mkern 2mu\mid X}, \Sigma_{T\mkern 2mu\mid X}\right)
We can estimate the probability that both true scores are 70 or lower using the cumulative distribution function of the multivariate normal distribution with upper bounds of 70 for both IQ true scores and adaptive behavior true scores. Under the conditions specified previously, Figure 13.5 shows that the probability that both scores are 70 or lower is about .53.
14 Multivariate Thresholds and True Scores
<- .96
r_xx <- c("IQ", "IQ_true")
v_name <- 70
threshold <- matrix(c(1,rep(r_xx,3)), nrow = 2, dimnames = list(v_name,v_name)) * 15^2
sigma <- c(IQ = 100, IQ_true = 100)
mu
<- tibble(lower = list(c(IQ = -Inf, IQ_true = -Inf),
p c(IQ = -Inf, IQ_true = threshold),
c(IQ = threshold, IQ_true = -Inf),
c(IQ = threshold, IQ_true = threshold),
c(IQ = -Inf, IQ_true = -Inf),
c(IQ = threshold, IQ_true = -Inf),
c(IQ = -Inf, IQ_true = -Inf),
c(IQ = -Inf, IQ_true = threshold)),
upper = list(c(IQ = threshold, IQ_true = threshold),
c(IQ = threshold, IQ_true = Inf),
c(IQ = Inf, IQ_true = threshold),
c(IQ = Inf, IQ_true = Inf),
c(IQ = threshold, IQ_true = Inf),
c(IQ = Inf, IQ_true = Inf),
c(IQ = Inf, IQ_true = threshold),
c(IQ = Inf, IQ_true = Inf)),
outcome = c("TP", "FP", "FN", "TN", "P", "N", "D+", "D-"),
p = pmap_dbl(list(lower = lower, upper = upper), \(lower, upper) mvtnorm::pmvnorm(lower, upper, mean = mu, sigma = sigma) |> as.vector())) |>
select(outcome,p) |>
deframe()
p
TP FP FN TN P N
0.017460927 0.005289205 0.003152489 0.974097379 0.022750132 0.977249868
D+ D-
0.020613417 0.979386583
tibble(Statistic = c("Sensitivity",
"Specificity",
"PPV",
"NPV",
"Overall Accuracy",
"Prevalence",
"Selection Ratio",
"Positive Likelihood Ratio",
"Negative Likelihood Ratio",
"True Positive Rate",
"False Positive Rate",
"True Negative Rate",
"False Negative Rate"),
Value = c(p["TP"] / p["D+"],
"TN"] / p["D-"],
p["TP"] / p["P"],
p["TN"] / p["N"],
p["TP"] + p["TN"],
p["D+"],
p["P"],
p["TP"] / p["D+"]) / (p["FP"] / p["D-"]),
(p["FN"] / p["D+"]) / (p["TN"] / p["D-"]),
(p["TP"],
p["FP"],
p["TN"],
p["FN"]
p[ ))
# A tibble: 13 × 2
Statistic Value
<chr> <dbl>
1 Sensitivity 0.847
2 Specificity 0.995
3 PPV 0.768
4 NPV 0.997
5 Overall Accuracy 0.992
6 Prevalence 0.0206
7 Selection Ratio 0.0228
8 Positive Likelihood Ratio 157.
9 Negative Likelihood Ratio 0.154
10 True Positive Rate 0.0175
11 False Positive Rate 0.00529
12 True Negative Rate 0.974
13 False Negative Rate 0.00315
tibble(statistic = c("Sensitivity",
"Specificity",
"PPV",
"NPV"),
lowerx = list(c(IQ = -Inf, IQ_true = -Inf),
c(IQ = threshold, IQ_true = -Inf),
c(IQ = -Inf, IQ_true = -Inf),
c(IQ = -Inf, IQ_true = threshold)),
upperx = list(c(IQ = threshold, IQ_true = Inf),
c(IQ = Inf, IQ_true = Inf),
c(IQ = Inf, IQ_true = threshold),
c(IQ = Inf, IQ_true = Inf)),
lower = list(c(IQ = -Inf, IQ_true = -Inf),
c(IQ = -Inf, IQ_true = threshold),
c(IQ = -Inf, IQ_true = -Inf),
c(IQ = threshold, IQ_true = -Inf)),
upper = list(c(IQ = Inf, IQ_true = threshold),
c(IQ = Inf, IQ_true = Inf),
c(IQ = threshold, IQ_true = Inf),
c(IQ = Inf, IQ_true = Inf)),
value = pmap_dbl(list(lowerx = lowerx,
upperx = upperx,
lower = lower,
upper = upper), tmvtnorm::ptmvnorm,
mean = mu,
sigma = sigma)) |>
select(statistic, value)
# A tibble: 4 × 2
statistic value
<chr> <dbl>
1 Sensitivity 0.847
2 Specificity 0.995
3 PPV 0.768
4 NPV 0.997
list(
list(
statistic = "Sensitivity",
lowerx = c(IQ = -Inf, IQ_true = -Inf),
upperx = c(IQ = threshold, IQ_true = Inf),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = threshold)
),list(
statistic = "Specificity",
lowerx = c(IQ = threshold, IQ_true = -Inf),
upperx = c(IQ = Inf, IQ_true = Inf),
lower = c(IQ = -Inf, IQ_true = threshold),
upper = c(IQ = Inf, IQ_true = Inf)
),list(
statistic = "PPV",
lowerx = c(IQ = -Inf, IQ_true = -Inf),
upperx = c(IQ = Inf, IQ_true = threshold),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = threshold, IQ_true = Inf)
),list(
statistic = "NPV",
lowerx = c(IQ = -Inf, IQ_true = threshold),
upperx = c(IQ = Inf, IQ_true = Inf),
lower = c(IQ = threshold, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = Inf)
),list(
statistic = "True Positives",
lowerx = c(IQ = -Inf, IQ_true = -Inf),
upperx = c(IQ = threshold, IQ_true = threshold),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = Inf)
),list(
statistic = "False Positives",
lowerx = c(IQ = -Inf, IQ_true = threshold),
upperx = c(IQ = threshold, IQ_true = Inf),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = Inf)
),list(
statistic = "True Negatives",
lowerx = c(IQ = threshold, IQ_true = threshold),
upperx = c(IQ = Inf, IQ_true = Inf),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = Inf)
),list(
statistic = "False Negatives",
lowerx = c(IQ = threshold, IQ_true = -Inf),
upperx = c(IQ = Inf, IQ_true = threshold),
lower = c(IQ = -Inf, IQ_true = -Inf),
upper = c(IQ = Inf, IQ_true = Inf)
)|>
) map(\(l) {
::ptmvnorm(
tmvtnormlowerx = l$lowerx,
upperx = l$upperx,
lower = l$lower,
upper = l$upper,
mean = mu,
sigma = sigma
|>
) as.vector() |>
`names<-`(l$statistic)
|>
}) unlist() |>
enframe()
# A tibble: 8 × 2
name value
<chr> <dbl>
1 Sensitivity 0.847
2 Specificity 0.995
3 PPV 0.768
4 NPV 0.997
5 True Positives 0.0175
6 False Positives 0.00529
7 True Negatives 0.974
8 False Negatives 0.00315
14.1 Structure
Exploratory factor analysis
Confirmatory factor analysis
Structural equation modeling
14.2 Reliability
Retest reliability
Alternate-form reliability
Split-half reliability
Internal consistency
- Cronbach’s Alpha
- McDonald’s Omega
Conditional reliability (IRT)
14.3 Validity
Face validity
Content validity
Criterion-oriented validity
- Concurrent validity
- Predictive validity
Discriminant validity
Convergent validity
Incremental validity
Construct validity
14.4 Profiles
Difference scores
Outliers
Highest-lowest scores
Multivariate profile shape
Conditional profiles