Hierarchical Covariance Estimation Approach to Meta-Analytic Structural Equation Modeling

Manuscript

There is a non-pay-walled version below, but the version of record is available at the Structural Equation Modeling, http://www.tandfonline.com/10.1080/10705511.2022.2142128.

Here’s the PDF and OSF repository with simulation and data analyses code in R.

Also, some of the methods in this paper are now implemented in the minorbsem R package: website, r-universe.

Some points about the paper from a Twitter thread I just typed up, and some things I would change at the bottom:

Summary twitter thread

The paper does a few things I like. I assume random effects MASEM is a given and lean on a Wishart-based approach I first saw in a really good paper by Hao Wu & Michael Browne in Psychometrika https://ncbi.nlm.nih.gov/pmc/articles/PMC5439333/. First heard of their paper at a talk by Paul de Boeck at OSU.

The random-effects MASEM returns an RMSEA that is a measure of dispersion of the different population covariance matrices in the meta-analysis from the “shared covariance matrix”. A reviewer suggested a meta-regression of this dispersion, I think that improved the paper nicely.

I also assumed that minor factors influence the shared population covariance matrix, so the approach returns a CRMR-type index capturing the influence of minor factors. That connects the paper to something else of interest: modeling SEMs w/ assumed misspecification

Several people have made the point that the true pop covariance matrix probably contains the influence of minor factors such that even if we have the right major factor configuration, we may still have poor model fit e.g. MacCallum & Tucker (1991) https://proquest.com/openview/40e6469da2ae2ec65eaea16757cada31/1?pq-origsite=gscholar&cbl=60977

So how about models that try to capture the influence of minor factors? Wu&Browne did this with frequentist SEMs, I feel their work should be studied more. Muthen & Asparouhov (2012) do this in Bayesian SEMs, assuming the influence of minor factors known a-priori :(

I think it’s a generally useful approach because trying to capture the influence of “minor factors” or modelling the degree of misspecification largely and rightly increases the uncertainty about point estimates.

Another way of putting this: most of our SEMs adjust for sampling error, but ignore the uncertainty due to fitting incorrect models. If you account for that uncertainty, standard errors tend to become much larger especially when the major factor configuration is less adequate.

Some of this stuff about uncertainty due to misspecification shows up in the current paper, but it’s kind of core to another paper I’m working on that just got what read like favourable 1st round reviews.

Returning to the current paper, something I’d like to do is write out using Wishart distribution for MASEM beginning with fixed-effects, random-effects then dependent covariance matrices. There’s something accepted at AERA on this, that I’d like to expand on.

Things I would change

1. The way the fit index is computed. It’s first computed unstandardized then it is standardized in an unappealing way. I’d modify equation 6, such that each ${\psi }_{ij}\sim N(0,{\tau }_{\psi }\sqrt{{\sigma }_{ii}{\sigma }_{jj}})$ such that ${\tau }_{\psi }$ is already standardized.
2. The reviewers were initially skeptical of SBC and that was partly my fault. So I had to utilize some statistical tests for SBC that I think are pretty reasonable, if reviewers are not satisfied with simple histograms. And by addressing point 1, I should be able to use a wider (less-constrained) prior for ${\tau }_{\psi }$ in the SBC implementation.