A Bayesian hierarchical framework to evaluate policy effects through quasi-experimental designs

Gianluca Baio

Department of Statistical Science | University College London

g.baio@ucl.ac.uk

https://gianluca.statistica.it

https://egon.stats.ucl.ac.uk/research/statistics-health-economics

https://github.com/giabaio https://github.com/StatisticsHealthEconomics

@gianlubaio@mas.to @gianlubaio

New challenges in the estimation of the impact of policies

XLIII Jornadas de Economia de la Salud, San Cristóbal de la Laguna, Tenerife, Spain

27 June 2024

Check out our departmental podcast “Random Talks” on Soundcloud!

Follow our departmental social media accounts + magazine “Sample Space”

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) \color{white}{= \alpha_0 + \alpha_1 w_i + \sum_{k=1}^K \beta_k X_{itk} + \phi_{0}t + \phi_{1}t w_i + \delta_0 z_{t} + \delta_1 w_i z_t + \gamma_i + \lambda_t} \,[+\ldots]$

$g(\cdot)$ is a suitable link function

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) = \alpha_0 + \alpha_1 w_i \color{white}{+ \sum_{k=1}^K \beta_k X_{itk} + \phi_{0}t + \phi_{1}t w_i + \delta_0 z_{t} + \delta_1 w_i z_t + \gamma_i + \lambda_t} [+\ldots]$

$g(\cdot)$ is a suitable link function
$\alpha_0$ and $(\alpha_0+\alpha_1)$ are the the baseline intercept in the controls and exposed, respectively

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) = \alpha_0 + \alpha_1 w_i + \sum_{k=1}^K \beta_k X_{itk} \color{white}{+ \phi_{0}t + \phi_{1}t w_i + \delta_0 z_{t} + \delta_1 w_i z_t + \gamma_i + \lambda_t} [+\ldots]$

$g(\cdot)$ is a suitable link function
$\alpha_0$ and $(\alpha_0+\alpha_1)$ are the the baseline intercept in the controls and exposed, respectively
$\bm{\beta}=(\beta_1,\ldots,\beta_K)$ quantifies the effect of the confounders $\bm{X}_{it}$

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) = \alpha_0 + \alpha_1 w_i + \sum_{k=1}^K \beta_k X_{itk} + \phi_{0}t + \phi_{1}t w_i \color{white}{+ \delta_0 z_{t} + \delta_1 w_i z_t + \gamma_i + \lambda_t} [+\ldots]$

$g(\cdot)$ is a suitable link function
$\alpha_0$ and $(\alpha_0+\alpha_1)$ are the the baseline intercept in the controls and exposed, respectively
$\bm{\beta}=(\beta_1,\ldots,\beta_K)$ quantifies the effect of the confounders $\bm{X}_{it}$
$\phi_{0}$ and $(\phi_{0}+\phi_{1})$ are the linear time trend in the controls and exposed, respectively

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) = \alpha_0 + \alpha_1 w_i + \sum_{k=1}^K \beta_k X_{itk} + \phi_{0}t + \phi_{1}t w_i + \delta_0 z_{t} + \delta_1 w_i z_t \color{white}{+ \gamma_i + \lambda_t} [+\ldots]$

$g(\cdot)$ is a suitable link function
$\alpha_0$ and $(\alpha_0+\alpha_1)$ are the the baseline intercept in the controls and exposed, respectively
$\bm{\beta}=(\beta_1,\ldots,\beta_K)$ quantifies the effect of the confounders $\bm{X}_{it}$
$\phi_{0}$ and $(\phi_{0}+\phi_{1})$ are the linear time trend in the controls and exposed, respectively
$\delta_0$ and $(\delta_0+\delta_1)$ model the change in the outcome pre-post intervention in the controls and the exposed, respectively

Interrupted time series

General model structure

$y_{it}$ is the outcome for unit $i=1\ldots,I$ and time $t=1,\ldots T\geq 2$
$\bm{X}_{it}=(X_{it1},\ldots,X_{itK})$ is a vector of $K$ covariates (confounders)
$z_{t}=1$ if the intervention is being applied at time $t$ and 0 otherwise
$w_{i}=1$ if unit $i$ is exposed and 0 if it is a control

$g\left(\E[Y_{it}]\right) = g(\mu_{it}) = \alpha_0 + \alpha_1 w_i + \sum_{k=1}^K \beta_k X_{itk} + \phi_{0}t + \phi_{1}t w_i + \delta_0 z_{t} + \delta_1 w_i z_t + \gamma_i + \lambda_t \,[+\ldots]$

$g(\cdot)$ is a suitable link function
$\alpha_0$ and $(\alpha_0+\alpha_1)$ are the the baseline intercept in the controls and exposed, respectively
$\bm{\beta}=(\beta_1,\ldots,\beta_K)$ quantifies the effect of the confounders $\bm{X}_{it}$
$\phi_{0}$ and $(\phi_{0}+\phi_{1})$ are the linear time trend in the controls and exposed, respectively
$\delta_0$ and $(\delta_0+\delta_1)$ model the change in the outcome pre-post intervention in the controls and the exposed, respectively
$\bm{\gamma}=(\gamma_1,\ldots,\gamma_I)$ are a set of unit-specific random effects to model dependencies across the units
$\bm\lambda=(\lambda_1,\ldots,\lambda_T)$ quantifies the time trends for the overall period, modelled using autoregressive structures, which will allow to increase the model flexibility
$[+\ldots]$ (optional) additional structured effect (e.g. to model spatial correlation)

Hostile environment

Results

we found evidence of greater psychological distress in people from Black Caribbean backgrounds than White participants after implementation of the Immigration Act 2014 (MD 0·67, 95% CrI 0·06 to 1·28). This effect persisted for several years, shown by the absence of difference over time since implementation of the Immigration Act 2014. We also found evidence that the Black Caribbean group had a further increase in psychological distress relative to White participants after the Windrush scandal media coverage commenced in 2017 (MD 1·28, 95% CrI 0·34 to 2·21).

Apart from Black Caribbean (where the immediate effect of the policy is strong), there’s an increase in mental distress between the two periods and post media seems to have a bigger effect across all ethnic groups

included ethnic group, the exposure period, all aforementioned confounders, a linear fixed effect for time (by year), a linear fixed effect for time since the start of each exposure period (by year), random effects to model residual temporal confounding (by year), and residual spatial confounding (by local authority area). These random effects accounted for variation not captured by our measured fixed effects. We specified weakly informative priors for all model parameters; these allow one to stabilise the inference while not imposing overbearing restrictions on the parameters’ values. We fitted the interrupted time series model using integrated nested Laplace approximations (INLA) through the R-INLA package

We then repeated this analysis, stratified first by whether participants were born in the UK, and second by net household income, to investigate effect moderation of these variables on the association between exposure to the hostile environment policy and subsequent media coverage and mental ill health. Finally, we repeated the primary analysis with three sensitivity analyses: first, without the use of weights to assess the effect of our weighting strategy; second, on participants with complete case data; and third, to assess the effect of non-response, by restricting our analysis to participants who responded at least once in each exposure period

1 / 17

A Bayesian hierarchical framework to evaluate policy effects through quasi-experimental designs Gianluca Baio Department of Statistical Science | University College London g.baio@ucl.ac.uk https://gianluca.statistica.it https://egon.stats.ucl.ac.uk/research/statistics-health-economics https://github.com/giabaio https://github.com/StatisticsHealthEconomics @gianlubaio@mas.to @gianlubaio New challenges in the estimation of the impact of policies XLIII Jornadas de Economia de la Salud, San Cristóbal de la Laguna, Tenerife, Spain 27 June 2024 Check out our departmental podcast “Random Talks” on Soundcloud! Follow our departmental social media accounts + magazine “Sample Space”