Initial setup

rm(list=ls())
library(ggplot2)
library(effects)
library(effsize)
rnd = function(x,digits=2){ return(format(round(x,digits),nsmall=digits)) }
load("interaction_datasets.RData")
tsz=25

Fully open code at: https://github.com/psicostat/workshops/tree/main/Interaction%20introduction%202024




WHAT’S IN A “Cohen’s f”?

Cohen’s f is often used for expressing effect size in interactions. General formula for Cohen’s f is like this:

\[Cohen's f = \sqrt{\frac{R^2}{1 - R^2}}\]

However, for interactions we should consider what is added by the interaction alone (above and beyond the main effects):

\[Cohen's f = \sqrt{\frac{R^2_1 - R^2_0}{1 - R^2_1}}\]

Cohen’s f = 0.25 (about Cohen’s f 2 = 0.06) is often taken as “medium” effect size. In fact, Cohen’s f = 0.40 (about Cohen’s f 2 = 0.15) is also taken as “medium” sometimes. Note that Cohen’s f = 0.25 requires R2 = 0.06, corresponding to about r = 0.24, while Cohen’s f = 0.40 requires R2 = 0.15, corresponding to about r = 0.39. So, it makes more sense to say that Cohen’s f = 0.25 is “medium”.

Anyways… what does that mean in actual interactions? The problem is that there are infinite cases that lead to the same effect size (e.g., Cohen’s f) in interactions.

All examples below will present about Cohen’s f ≈ 0.25 (Cohen’s f 2 ≈ 0.06).

This is also a useful resource: https://lakens.github.io/statistical_inferences/06-effectsize.html#effect-sizes-for-interactions




Example: Fully crossed interaction (2 x 2)

May happen in well-designed experimental studies, but otherwise infrequent in real life (?). Here, each main effect is small or even null, and the interaction accounts for most or all explained variance.

In model without interaction, R2 = 0.01. In model with interaction, R2 = 0.07.
Overall, Cohen’s f = 0.25 (Cohen’s f 2 = 0.06).

But… a simpler way to understand the above effect may be via Standardized Mean Differences (SMDs, such as Cohen’s d):

  • the effect size (SMD) for X when W = 0 is Cohen’s d = 0.65 (cell B - A).

  • the effect size (SMD) for X when W = 1 is Cohen’s d = -0.37 (cell D - C).




Example: Treatment-like effect (2 x 2)

Imagine a situation like a treated-vs-control-group experiment with pre-test and post-test. In one condition (w = 0) x has not effect over y, while in the other (w = 1) x has a clearly visible effect over y. The interaction coefficients tells you (as usual) the difference-between-differences across conditions.

In model without interaction, R2 = 0.10. In model with interaction, R2 = 0.15.
Overall, Cohen’s f = 0.25 (Cohen’s f 2 = 0.06).

But note how much the effects must vary across conditions to get there:

  • the effect size (SMD) for X when W = 0 is Cohen’s d = 0.09 (cell B - A).

  • the effect size (SMD) for X when W = 1 is Cohen’s d = 1.03 (cell D - C).

…is this really a medium effect?! 😱




Example: Treatment-like effect (2 x 3)

Imagine a situation like a treated-vs-control-group experiment with pre-test, post-test, and follow-up. Not much different from before, and it’s a “familiar” situation. However, there are 2 interaction coefficients here; using the default contrasts, they tell you how the difference between levels of w (e.g., groups) varies from x = “00” to x = “01” (1st interaction coefficient), and from x = “00” to x = “10” (2nd interaction coefficient).

In model without interaction, R2 = 0.20. In model with interaction, R2 = 0.25.
Overall, Cohen’s f = 0.25 (Cohen’s f 2 = 0.06).

But note how much the effects must vary across conditions to get there:

  • the effect size (SMD) at time x = “00” (cell D - A) is Cohen’s d = 0.01.

  • the effect size (SMD) at time x = “01” (cell E - B) is Cohen’s d = 1.00.

  • the effect size (SMD) at time x = “10” (cell F - C) is Cohen’s d = 1.04.

…is this really a medium effect?! 😱


Let’s try Superpower on this case, with some “plausible” assumptions:

# devtools::install_github("arcaldwell49/Superpower")
library(Superpower)
(design = ANOVA_design(design="3w*2b",
             n=30/2,
             mu=c(.00,.00,1.30,.30,1.20,.10),
             sd=1,
             r=.50)
)

## The design is specified as: 3w*2b
## Summary of means (mu) and standard deviations (SD)
##    mu SD  a  b
## 1 0.0  1 a1 b1
## 2 0.0  1 a1 b2
## 3 1.3  1 a2 b1
## 4 0.3  1 a2 b2
## 5 1.2  1 a3 b1
## 6 0.1  1 a3 b2
## 
## Correlation Matrix
##       a1_b1 a1_b2 a2_b1 a2_b2 a3_b1 a3_b2
## a1_b1   1.0   0.0   0.5   0.0   0.5   0.0
## a1_b2   0.0   1.0   0.0   0.5   0.0   0.5
## a2_b1   0.5   0.0   1.0   0.0   0.5   0.0
## a2_b2   0.0   0.5   0.0   1.0   0.0   0.5
## a3_b1   0.5   0.0   0.5   0.0   1.0   0.0
## a3_b2   0.0   0.5   0.0   0.5   0.0   1.0
pow = ANOVA_power(design)
## Power and Effect sizes for ANOVA tests
##           power effect_size
## anova_b    63.3      0.1868
## anova_a    99.0      0.2908
## anova_b:a  82.3      0.1852
## 
## Power and Effect sizes for pairwise comparisons (t-tests)
##                       power effect_size
## p_a_a1_b_b1_a_a1_b_b2   5.6   -0.006196
## p_a_a1_b_b1_a_a2_b_b1  99.8    1.364028
## p_a_a1_b_b1_a_a2_b_b2  14.2    0.296238
## p_a_a1_b_b1_a_a3_b_b1  98.8    1.244667
## p_a_a1_b_b1_a_a3_b_b2   5.9    0.096413
## p_a_a1_b_b2_a_a2_b_b1  92.2    1.350039
## p_a_a1_b_b2_a_a2_b_b2  18.5    0.317151
## p_a_a1_b_b2_a_a3_b_b1  87.4    1.242589
## p_a_a1_b_b2_a_a3_b_b2   6.2    0.104481
## p_a_a2_b_b1_a_a2_b_b2  76.7   -1.040732
## p_a_a2_b_b1_a_a3_b_b1   6.5   -0.108645
## p_a_a2_b_b1_a_a3_b_b2  88.3   -1.242163
## p_a_a2_b_b2_a_a3_b_b1  67.2    0.937406
## p_a_a2_b_b2_a_a3_b_b2  11.7   -0.207264
## p_a_a3_b_b1_a_a3_b_b2  79.9   -1.135023
## 
## 
## Within-Subject Factors Included: Check MANOVA Results



Example: Group moderate continuous effect (2 x continuous)

Imagine studying how a relationship between x and y is different across two populations.

In model without interaction, R2 = 0.57. In model with interaction, R2 = 0.59.
Overall, Cohen’s f = 0.25 (Cohen’s f 2 = 0.06).

But note how much the effects must vary across conditions to get there:

  • the x-y correlation in sample w = 0 is Pearson’s r = 0.30

  • the x-y correlation in sample w = 0 is Pearson’s r = 0.66

…is this really a medium effect?! 😱




Example: Continuous relationship moderated by another continuous variable (continuous x continuous)

Imagine studying how a relationship between x and y is different across different levels of a continuous moderator z.

In model without interaction, R2 = 0.15. In model with interaction, R2 = 0.20.
Overall, Cohen’s f = 0.25 (Cohen’s f 2 = 0.06).

The graphical representation above, however, is a little bit messy. We could simplify that by keeping just 3 lines at interpretable points (e.g., z-score of the moderator W equal -1, 0, +1):

Getting the estimates of the x-y relationship at given points of w, like in the previous case, can be done by looking at the coefficient “x” in the model summary, knowing that the “x:w” coefficient means how the effect of x on y varies for every unitary +1 increase of the moderator w.

Let’s have a look at the summary:

summary(fit1)
## 
## Call:
## lm(formula = y ~ x * w, data = dcc)
## 
## Residuals:
##     Min      1Q  Median      3Q     Max 
## -3.1749 -0.6784 -0.0200  0.7015  2.7791 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)    
## (Intercept)  0.04000    0.04648   0.861     0.39    
## x            0.33167    0.04700   7.056 6.08e-12 ***
## w            0.30607    0.04717   6.489 2.18e-10 ***
## x:w          0.25116    0.04590   5.472 7.20e-08 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 1.016 on 476 degrees of freedom
## Multiple R-squared:  0.1976, Adjusted R-squared:  0.1925 
## F-statistic: 39.07 on 3 and 476 DF,  p-value: < 2.2e-16

In fact, y is not really on a z-score metrics here. Its residual SD that is about 1.0. Due to the very strong effects of x, w, and their interactions, the overall SD of y is 1.13.

So, the model coefficients here cannot be easily and directly interpreted in a correlation-like fashion.

Perhaps it’s better to look at the actual standardized coefficients:

lm.beta::lm.beta(fit1)
## 
## Call:
## lm(formula = y ~ x * w, data = dcc)
## 
## Standardized Coefficients::
## (Intercept)           x           w         x:w 
##          NA   0.2910461   0.2677671   0.2260456