Multilevel model of accuracy
Params:
params
$dv_var
[1] "ACC.pli"
$nsim
[1] 10000
$plot_ymax
[1] 0.04
$plot_yshift
[1] -0.006
Read in data
DV_VAR = params$dv_var
all.dat = read.csv('data/1_scored.csv')
all.dat$Subject = factor(all.dat$Subject)
all.dat$dv = all.dat[,DV_VAR]
# Remove regular ospan, which has substantially lower accuracy
# due to verification requirements
dat = subset(all.dat, !task %in% 'Ospan.reg')
# Mark high and low interference conditions
low_int = c('spOspan.noVer', 'Ospan.scram.noVer', 'Rspan.names.long', 'Rspan.names.short', 'Ospan.reg')
dat$interference = ifelse(dat$task %in% low_int, 'low', 'high')
Models
dat$cond = paste(dat$interference, dat$trialtype)
contrasts(dat$trialtype) <- c(0,1) # similarity increment
Model with recall predictions for each interference:trialtype explicit
fit.mlm = lmer(dv ~ 0 + cond + (1 | task:Subject) + (1 | task), data=dat)
summary(fit.mlm)
Linear mixed model fit by REML ['lmerMod']
Formula: dv ~ 0 + cond + (1 | task:Subject) + (1 | task)
Data: dat
REML criterion at convergence: -2157.2
Scaled residuals:
Min 1Q Median 3Q Max
-1.3324 -0.6772 0.0022 0.0164 6.1485
Random effects:
Groups Name Variance Std.Dev.
task:Subject (Intercept) 8.103e-06 0.002847
task (Intercept) 0.000e+00 0.000000
Residual 1.408e-04 0.011866
Number of obs: 368, groups: task:Subject, 184; task, 12
Fixed effects:
Estimate Std. Error t value
condhigh D 0.0157077 0.0011531 13.623
condhigh S 0.0008267 0.0011531 0.717
condlow D 0.0084877 0.0014381 5.902
condlow S 0.0002572 0.0014381 0.179
Correlation of Fixed Effects:
cndhgD cndhgS cndlwD
condhigh S 0.054
condlow D 0.000 0.000
condlow S 0.000 0.000 0.054
Same model contrast coded for similarity benefit
fit.mlm.con = lmer(dv ~ 0 + interference/trialtype + (1 | task:Subject) + (1 | task), data=dat)
summary(fit.mlm.con)
Linear mixed model fit by REML ['lmerMod']
Formula: dv ~ 0 + interference/trialtype + (1 | task:Subject) + (1 | task)
Data: dat
REML criterion at convergence: -2157.2
Scaled residuals:
Min 1Q Median 3Q Max
-1.3324 -0.6772 0.0022 0.0164 6.1485
Random effects:
Groups Name Variance Std.Dev.
task:Subject (Intercept) 8.103e-06 2.847e-03
task (Intercept) 1.286e-18 1.134e-09
Residual 1.408e-04 1.187e-02
Number of obs: 368, groups: task:Subject, 184; task, 12
Fixed effects:
Estimate Std. Error t value
interferencehigh 0.015708 0.001153 13.623
interferencelow 0.008488 0.001438 5.902
interferencehigh:trialtype1 -0.014881 0.001586 -9.385
interferencelow:trialtype1 -0.008230 0.001978 -4.162
Correlation of Fixed Effects:
intrfrnch intrfrncl intrfrnch:1
interfrnclw 0.000
intrfrnch:1 -0.688 0.000
intrfrncl:1 0.000 -0.688 0.000
Same model contrast coded for interference benefit
fit.mlm.int = lmer(dv ~ 0 + trialtype/interference + (1 | task:Subject) + (1 | task), data=dat)
summary(fit.mlm.int)
Linear mixed model fit by REML ['lmerMod']
Formula: dv ~ 0 + trialtype/interference + (1 | task:Subject) + (1 | task)
Data: dat
REML criterion at convergence: -2157.2
Scaled residuals:
Min 1Q Median 3Q Max
-1.3324 -0.6772 0.0022 0.0164 6.1485
Random effects:
Groups Name Variance Std.Dev.
task:Subject (Intercept) 8.103e-06 0.002847
task (Intercept) 0.000e+00 0.000000
Residual 1.408e-04 0.011866
Number of obs: 368, groups: task:Subject, 184; task, 12
Fixed effects:
Estimate Std. Error t value
trialtypeD 0.0157077 0.0011531 13.623
trialtypeS 0.0008267 0.0011531 0.717
trialtypeD:interferencelow -0.0072200 0.0018433 -3.917
trialtypeS:interferencelow -0.0005695 0.0018433 -0.309
Correlation of Fixed Effects:
trltyD trltyS trltD:
trialtypeS 0.054
trltypD:ntr -0.626 -0.034
trltypS:ntr -0.034 -0.626 0.054
Why is task variance estimated to be 0?
Sanity check, injecting noise at task level. Note the accurate task variance estimates.
tmp_dat = ddply(dat, .(task), transform, dv = dv + rnorm(1, sd=.1))
fit.mlm2 = lmer(dv ~ 0 + cond + (1 | task:Subject) + (1 | task), data=tmp_dat)
summary(fit.mlm2)
Linear mixed model fit by REML ['lmerMod']
Formula: dv ~ 0 + cond + (1 | task:Subject) + (1 | task)
Data: tmp_dat
REML criterion at convergence: -2090.2
Scaled residuals:
Min 1Q Median 3Q Max
-1.6073 -0.5913 -0.0478 0.1259 6.0911
Random effects:
Groups Name Variance Std.Dev.
task:Subject (Intercept) 7.608e-06 0.002758
task (Intercept) 4.848e-03 0.069625
Residual 1.408e-04 0.011866
Number of obs: 368, groups: task:Subject, 184; task, 12
Fixed effects:
Estimate Std. Error t value
condhigh D 0.007134 0.024644 0.290
condhigh S -0.007747 0.024644 -0.314
condlow D -0.052182 0.034842 -1.498
condlow S -0.060412 0.034842 -1.734
Correlation of Fixed Effects:
cndhgD cndhgS cndlwD
condhigh S 0.998
condlow D 0.000 0.000
condlow S 0.000 0.000 0.998
Another Sanity check, looking at task variance from ANOVA standpoint. Note that the F-value for task is 1 (no between task var beyond subject var)
fit.aov = aov(dv ~ interference + task + Error(task:Subject), data=dat)
Warning in aov(dv ~ interference + task + Error(task:Subject), data =
dat): Error() model is singular
summary(fit.aov)
Error: task:Subject
Df Sum Sq Mean Sq F value Pr(>F)
interference 1 0.001330 0.0013296 8.522 0.00398 **
task 10 0.001741 0.0001741 1.116 0.35295
Residuals 172 0.026836 0.0001560
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Error: Within
Df Sum Sq Mean Sq F value Pr(>F)
Residuals 184 0.04047 0.0002199
Confidence Intervals
Computing bootstrap confidence intervals ...
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
2.5 % 97.5 %
sd_(Intercept)|task:Subject 0.000000000 0.005372848
sd_(Intercept)|task 0.000000000 0.002296895
sigma 0.010654498 0.012857745
condhigh D 0.013512107 0.017953509
condhigh S -0.001439155 0.003081612
condlow D 0.005676607 0.011272283
condlow S -0.002592041 0.002964397
Computing bootstrap confidence intervals ...
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 1 from bobyqa: bobyqa --
maximum number of function evaluations exceeded
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
2.5 % 97.5 %
sd_(Intercept)|task:Subject 0.000000000 0.005414811
sd_(Intercept)|task 0.000000000 0.002338520
sigma 0.010634333 0.012819305
interferencehigh 0.013443174 0.017981917
interferencelow 0.005790776 0.011298175
interferencehigh:trialtype1 -0.017947443 -0.011723088
interferencelow:trialtype1 -0.012161966 -0.004442335
Cohen's d
Here, I divided group differences by either the residual variance, or between-subject variance + residual variance.
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in checkConv(attr(opt, "derivs"), opt$par, ctrl =
control$checkConv, : Model failed to converge: degenerate Hessian with 1
negative eigenvalues
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
Warning in optwrap(object@optinfo$optimizer, ff, x0, lower = lower,
control = control$optCtrl, : convergence code 3 from bobyqa: bobyqa -- a
trust region step failed to reduce q
$d_high
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 10000 bootstrap replicates
CALL :
boot.ci(boot.out = booted, type = c("norm", "perc"), index = ii)
Intervals :
Level Normal Percentile
95% (-1.0853, -0.6714 ) (-1.1124, -0.6969 )
Calculations and Intervals on Original Scale
$d_low
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 10000 bootstrap replicates
CALL :
boot.ci(boot.out = booted, type = c("norm", "perc"), index = ii)
Intervals :
Level Normal Percentile
95% (-0.7233, -0.2485 ) (-0.7334, -0.2636 )
Calculations and Intervals on Original Scale
$d_sub_high
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 10000 bootstrap replicates
CALL :
boot.ci(boot.out = booted, type = c("norm", "perc"), index = ii)
Intervals :
Level Normal Percentile
95% (-1.0664, -0.6705 ) (-1.0863, -0.6884 )
Calculations and Intervals on Original Scale
$d_sub_low
BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS
Based on 10000 bootstrap replicates
CALL :
boot.ci(boot.out = booted, type = c("norm", "perc"), index = ii)
Intervals :
Level Normal Percentile
95% (-0.7125, -0.2483 ) (-0.7202, -0.2604 )
Calculations and Intervals on Original Scale
Plotting
Means and Standard Errors
The following `from` values were not present in `x`: Ospan.reg
p +
geom_rect(aes(x=NULL, y=NULL, shape=NULL,xmin=xmin, xmax=xmax, ymin=ymin, ymax=ymax),
color='black', fill='white', data=group_annot) +
geom_text(aes(shape=NULL, color=NULL, x=text.x, y=text.y, label=label),
show_guide=FALSE, data=group_annot) + pub_theme + colors + shapes
Scale for 'colour' is already present. Adding another scale for 'colour', which will replace the existing scale.
Scale for 'shape' is already present. Adding another scale for 'shape', which will replace the existing scale.
ymax not defined: adjusting position using y instead
title: "1_mlm.R" author: "machow" date: "Wed Jan 13 12:31:53 2016"