A Chi-Square test of close fit in covariance-based SEM

TLDR: If you can assume close fit for the RMSEA, there is no reason why you cannot for a Chi-Square test in SEMs. The method to do this is relatively simple, and may cause SEM practitioners to reconsider the Chi-Square test.

When assessing the fit of structural equation models, it is common for applied researchers to dismiss the \(\chi^2\) test because it will almost always detect a statistically significant discrepancy between your model and the data, given a large enough sample size. This is because, almost always, our models are approximations of the data. If our model-implied covariance matrix actually matched the sample covariance matrix within sampling variability, the \(\chi^2\) test would not be statistically significant regardless of sample size.

Because of the sensitivity of the \(\chi^2\) test to large sample sizes, practitioners often rely on other fit indices like the RMSEA, CFI, and TLI - all of which are based on the \(\chi^2\). For the RMSEA, MacCallum, Browne and Sugawara (1996)¹ specified values of .05 and .08 as indicating close and mediocre² fit respectively. And in lavaan, you automatically get a test of close fit for the RMSEA with confidence intervals and a p-value. This test actually uses the \(\chi^2\) distribution, and there is no reason why one cannot perform a \(\chi^2\) test of close or mediocre fit depending on one’s standards.³

The sections that follow may include details that not everyone would like to read about, you can skip to the bottom of the page for annotated lavaan code for how to compute a \(\chi^2\) test of close or mediocre fit.

So the formula for the RMSEA is:

$$\sqrt{\frac{\chi^2-df}{df(N-1)}}$$

where \(\chi^2\) is the \(\chi^2\) test statistic of your model, \(df\) is your model degrees of freedom, and \(N\) is sample size.

If your model fit the data perfectly, the numerator, \(\chi^2-df\), is zero; this is the hypothesis the standard \(\chi^2\)-test tests. And to test this hypothesis, it uses the \(\chi^2\)-distribution. If we want to perform a test of close fit on the RMSEA, we do not assume a nil null distribution for the \(\chi^2\). Instead, we use the non-central \(\chi^2\) distribution with a non-centrality parameter that corresponds to an RMSEA of .05. The idea is we accept some level of misspecification, and we use a distribution that corresponds to this level of misspecification. Lavaan reports the result of this test as one of the fit statistics.

For those who are not familiar with non-central distributions, they are the general family of distributions to which the distributions we are familiar with belong. For example, the \(t\)-test assumes a nil (zero) null effect so we use the non-central \(t\)-distribution, with an expected value (and non-centrality parameter) of zero. This distribution is what we call the \(t\)-distribution. If we want to create confidence intervals without assuming a nil effect, we can actually use a \(t\)-distribution while specifying its non-centrality parameter \(\lambda\). It is the distribution when the null of zero is false. The Wikipedia introduction to this topic for the \(t\)-distribution is decent.

So how does this help us? The non-centrality parameter \(\lambda\) for the RMSEA test in lavaan is actually the \(\chi^2-df\) value that corresponds to an RMSEA of .05. In math:

$$RMSEA = \sqrt{\frac{\chi^2-df}{df(N-1)}}$$

$$RMSEA^2 = \frac{\chi^2-df}{df(N-1)}$$

$$RMSEA^2 \times df(N-1) = \chi^2-df$$

Since \(\chi^2-df\) is \(\lambda\), then:

$$\lambda = RMSEA^2 \times df(N-1)$$

So for a test of close fit, \(\lambda\) is:

$$RMSEA^2 \times df(N-1) = .05^2 \times df(N-1) = .0025 \times df(N-1)$$

And for a test of mediocre fit, \(\lambda\) is:

$$RMSEA^2 \times df(N-1) = .08^2 \times df(N-1) = .0064 \times df(N-1)$$

Note that lavaan may do things a little differently.⁴

Hence, given a model degrees of freedom, and sample size, we can calculate the non-centrality parameter \(\lambda\). And given \(\lambda\), a \(\chi^2\) value and the degrees of freedom for the model, we can calculate the p-value for a test of close or mediocre fit.

The R syntax for this is:

pchisq(Chi-sq-value, degrees-of-freedom, non-centrality-parameter, FALSE)

library(lavaan)
data("HolzingerSwineford1939")
# model syntax for a bifactor model with the HolzingerSwineford1939 dataset
# eliminating visual factor resolves Heywood case
writeLines(syntax <- paste(
  paste("g =~", paste0("x", 1:9, collapse = " + ")),
  # paste("visual =~", paste0("x", 1:3, collapse = " + ")),
  paste("textual =~", paste0("x", 4:6, collapse = " + ")),
  paste("speed =~", paste0("x", 7:9, collapse = " + ")),
  sep = "\n"
))

g =~ x1 + x2 + x3 + x4 + x5 + x6 + x7 + x8 + x9
textual =~ x4 + x5 + x6
speed =~ x7 + x8 + x9

summary(hs.fit <- cfa(syntax, HolzingerSwineford1939, std.lv = TRUE,
                      orthogonal = TRUE), fit.measures = TRUE)

lavaan (0.5-23.1097) converged normally after  25 iterations

  Number of observations                           301

  Estimator                                         ML
  Minimum Function Test Statistic               42.291
  Degrees of freedom                                21
  P-value (Chi-square)                           0.004

Root Mean Square Error of Approximation:

  RMSEA                                          0.058
  90 Percent Confidence Interval          0.032  0.083
  P-value RMSEA <= 0.05                          0.276

pchisq(q = 42.291, df = 21, ncp = 0, lower.tail = FALSE)

[1] 0.003867178

(ncp.close <- .0025 * 21 * (301 - 1))

[1] 15.75

pchisq(q = 42.291, df = 21, ncp = ncp.close, lower.tail = FALSE)

[1] 0.2740353

(ncp.med <- .0064 * 21 * (301 - 1))

[1] 40.32

pchisq(q = 42.291, df = 21, ncp = ncp.med, lower.tail = FALSE)

[1] 0.9199686