This release adds choice-level likelihood fitting, multi-block
(learning-trajectory) designs, the kachergis_priming
trajectories, uncfam_general(), which nests the
uncfam() family, canonical options for
rescorla_wagner() and kalman_filter(), and a
softmax choice rule for any model. It also corrects several datasets and
models, so most results differ from 0.4.0:
xsl_datasets change for every model: its
filtering, frequency / contextual-diversity and 4AFC conditions were
corrected, two duplicate conditions were dropped, and the Suanda (2014)
and Koehne (2013) condition means moved to held-out datasets (see
Dataset changes). It now has 47 conditions, all with item-level accuracy
and a response_matrix.get_group_model_fit() and
get_crossvalidated_model_fit() search a different
space.tilles() now follows Tilles & Fontanari (2013),
rescorla_wagner() applies each object’s prediction error to
that object (it had reduced to co-occurrence counting), and
bayesian_decay() applies its choice exponent on conditions
with a test.rescorla_wagner() and kalman_filter() gain
options that restore the canonical forms of these models. The defaults
reproduce the current models exactly.
outcomes = "all" (both models) treats every object as
an outcome on every trial, with target 1 (or lambda) if present and 0 if
absent, as in Rescorla & Wagner (1972). A word’s associations with
objects that are absent when it is heard then weaken: discriminative
learning (Ramscar et al., 2010).cues = "objects" (rescorla_wagner())
reverses the direction of prediction, so objects predict words, as in
Ramscar et al.’s (2010) feature-label ordering account.covariance = "full" (kalman_filter()) is
the Kalman filter of the associative-learning literature (Dayan &
Kakade, 2001; Kruschke, 2008). Each object has a weight vector over
words with a full covariance, the present words share one prediction
error, and co-present words’ weights become negatively correlated,
giving backward blocking and explaining away. The default filter is one
independent filter per word-object pair, which can’t make associations
interact.phi (both models) replaces Luce choice with a softmax
choice rule with inverse temperature phi. Softmax is
defined for the negative associations these models can learn.xsl_run() scores a
model with a phi parameter by softmax over its (mean)
associations, on every scoring path and in the likelihoods, as it
already applies ch_dec. New softmax_weights()
gives the weights, and choice_probabilities() gains
phi.rescorla_wagner_softmax,
rescorla_wagner_discriminative (absent objects as outcomes)
and kalman_filter_full (full covariance, absent objects as
outcomes). Group-fit to xsl_datasets, each fits about as
well as uncfam() (weighted SSE .57, .61 and .56, against
.56 for uncfam() and 1.63 for baseline() and
the defaults under Luce choice). Softmax alone brings
baseline() to .88.xslControl() gains
objective and lapse.
objective = "binomial" scores how many participants got
each test item right.objective = "multinomial" scores which object each
participant chose, using the full confusion matrices in
response_matrix.lapse (default .01, used only by the likelihoods) mixes
in uniform guessing among an item’s alternatives, so a model that is
certain doesn’t get an infinitely bad fit when someone chooses
otherwise. Naming "lapse" in xsl_fit()’s
free fits it, as does fit_lapse = TRUE in
get_group_model_fit() and
get_crossvalidated_model_fit().xsl_run() returns each
condition’s loglik and the total loglik and
nll. xsl_fit() minimizes nll and
reports loglik, n_obs, k,
aic and bic.
cross_validated_group_fits() adds in-sample and held-out
train_nll and test_nll.objective = "sse", is unchanged.choice_probabilities() gives, for each test item, the
alternatives offered, the model’s probability of choosing each, and how
many participants chose each. The correct alternative’s probability is
exactly the accuracy xsl_run() scores.xslData‘s accuracy can now be a matrix with
one row per block, and its response_matrix a list with one
matrix per block. xsl_run() scores block b on
a run trained on b repetitions of the sequence. This works
for every model, including batch models and those without a
reps loop, and with test specifications and both likelihood
objectives. For every deterministic model with a reps loop,
the result is exactly what continuous training records after each block.
The fit’s perf has one row per block, sse and
loglik sum over blocks, and blocks holds each
block’s fit. choice_probabilities() gains
block. Each block is a separate run, so for stochastic
models the blocks’ predictions come from independent simulated
learners.kachergis_priming: three 4-block learning
trajectories (164 participants), each studying one sequence four times
with an 18AFC test after each block. Two of the sequences are
xsl_datasets conditions (3_x8_369_4x4,
freq369-3x3loCD), with different participants; the third is
the original 4x4 sequence. Registered in
xsl_holdout_datasets. See
data-raw/add_kachergis_priming.R.xsl_datasets condition now carries a
response_matrix: for each test item, the number of
participants choosing each object. This is the full confusion matrix
behind accuracy, which it reproduces exactly, so models can
be fit to which objects learners chose (e.g. by multinomial likelihood)
rather than only to proportion correct. It is extracted from the raw
response files (data-raw/raw_responses.R,
data-raw/fix_freqcd_add_responses.R).
data-raw/extract_response_matrices.R reports how each
condition was matched to its raw data.uncfam_general(), a single biased associative
model that nests the uncfam() family, so variants can be
compared as nested models (fix or free parameters) rather than as
separate functions. A pair’s share of the trial’s weight is
M^gamma * exp(B * (U(w) + U(o))) (normalized over the
trial), plus eps times elimination support, and the trial’s
weight is scaled by (relative object entropy)^kappa; finite
K replaces deterministic allocation with sampled attention.
Its defaults reproduce uncfam(). Setting
gamma = 0, uncertainty = "novelty", a free
gamma, a free eps, kappa = 1 or a
finite K reproduces, respectively,
uncfam(variant = "uncertainty-only"),
uncfam(variant = "novelty"), uncfam_gamma(),
uncfam_elimination(), uncfam_attention() and
uncfam_sampling(). Each reproduces the original exactly (to
the last bit, and draw for draw for sampling) on all 47
xsl_datasets conditions; this is checked in
tests/testthat/test-uncfam_general.R.
uncfam_predictive() is not nested, since its
prediction-error update is unnormalized. Registered in
show_models()/xsl_model_registry(), fitting
X, B, C, gamma, eps, kappa with K = Inf. It
also has an optional short-term trace: with rho > 0,
every update also goes into a second matrix that decays by
Cs per trial, and familiarity becomes
(M + rho * S)^gamma, so recently strengthened pairings draw
extra attention. Uncertainty, elimination and test performance still use
M, and rho = 0 (the default) is
unchanged.uncfam(), uncfam_gamma(),
uncfam_elimination(), uncfam_attention() and
uncfam_sampling() are now thin wrappers around
uncfam_general()’s model, replacing about 450 lines of
duplicated code. Their results are identical to the last bit, and draw
for draw for uncfam_sampling(). This was checked on all 47
xsl_datasets conditions over two training blocks and with a
start matrix, and a stored reference
(tests/testthat/fixtures/uncfam_family_reference.rds) keeps
it checked. Their parameters, names and the registry are unchanged.
uncfam_sampling() now records performance after every
training block (control$reps), like the rest of the family,
rather than only after the last. It also now applies
control$test_noise.uncfam_general() now declares support for
control$start_matrix, which it already used.xsl_fit(), cross_validated_group_fits(),
get_group_model_fit() and
get_crossvalidated_model_fit() gain n_restarts
and seed. n_restarts keeps the best of several
DEoptim runs, and $restarts reports every run’s best SSE.
Some models’ objectives have poor basins that single runs often land in:
about a third of single runs for uncfam_predictive(). The
group and cross-validated helpers default to the model’s registry
setting: 5 restarts for uncfam_predictive, 3 for
tilles, otherwise 1. seed makes fits
reproducible.update_params() and the fitting functions accept
free, the names of the parameters being fit, so any subset
of a model’s parameters can be fit with the rest fixed. Previously only
the leading parameters could be fit. Fitted bestmem is now
named by parameter. Every registry entry names its free parameters.show_models() now lists every model in the registry.
This adds four variants used in the model comparison paper:
uncfam_novelty, uncfam_uncertainty_only,
uncfam_strength (familiarity only, B = 0) and
fazly_threshold.xsl_datasets
(filt*E_*L; Kachergis, Yu, & Shiffrin, 2012) were
scored wrongly, and held only half their tests. Each word co-occurs in
training with both its own object and its partner’s object (e.g. w1 with
o1 and o7). It was tested twice, once with the partner object absent
(AtoA, measuring w1-o1) and once with its own object absent (AtoB,
measuring w1-o7). xsl_datasets held only the AtoA
accuracies, scored as a choice among all 12 objects, so a model’s
association with the absent partner object counted against it, and
filt0E_* (symmetric pairs) was capped at .5. Each condition
now has 24 test items scored by mafc_test(): the 12 AtoA
items, then the 12 AtoB items. In the raw data, no participant ever
chose the absent object on either test. Two participants in each
*_6L condition had test codes shifted by 6, and are now
excluded (n_subj 27 -> 23; the stored 27 was a per-word
count inflated by the miscoding), which changes those conditions’ AtoA
accuracies slightly. This changes all fits involving these
conditions. See data-raw/fix_filtering_tests.R and
?xsl_datasets.xsl_datasets are rebuilt from the original analysis data
(data-raw/data/freqCD_item_data.Rdata), giving exact
accuracies and correct sample sizes. This changes fits involving
these conditions..
n_subj had been
attached to the wrong conditions. It is now 104 (was 26) for
freq369-3x3hiCD, 66 (62) for freq369_36mx, 40
(66) for freq369_39mx and 62 (40) for
cont_div6-12.3_x8_369_4x4 and freq369-3x3loCD had been
pooled with the first test block of the “priming” experiments. Those
participants trained on the same sequences but were tested four times,
and the pooled values could not be reproduced. Both conditions now hold
only their original experiments: n_subj 30 (was 74) and 34
(was 102), with item accuracies changing by up to .27 and .51.orig_3x3 was condition 224 (same training,
identical accuracies from the same 36 participants) with
n_subj 104.orig_4x4’s 26 original participants are a subset of
condition 223’s 77, and its training sequence differs from
223’s only in words 4 and 16 trading places between trials
11 and 24.data-raw/fix_freqcd_add_responses.R.xsl_datasets listed test
alternatives that participants were not shown. This changes fits
of every model involving conditions 211 and 214. See
data-raw/fix_4afc_tests.R.
214 (3x4 6/8 4AFC): 175 of its 648 human responses, on
all 18 words, were to objects not among the listed alternatives. All of
them are among condition 213’s alternatives for the same
word (and no other condition’s), so 214’s participants were evidently
tested with 213’s displays. 214’s test is now 213’s. Its
accuracy and response_matrix are
unchanged.211 (1x3 6/18 4AFC), word 17: the alternatives listed
object 17 twice and omitted foil 2, which participants chose; now
c(17, 16, 5, 2).xsl_datasets is now to one of
its item’s listed alternatives (checked in
tests/testthat/test-data.R).xsl_datasets to new
standalone held-out datasets, suanda2014 and
koehne2013, registered in
xsl_holdout_datasets, as in the original CogSci bake-off.
Only their condition-level mean accuracies are available, so they can’t
inform item-level group or cross-validated fits; they’re for testing
generalization of mean accuracy. xsl_datasets now has 47
conditions, all with item-level accuracy and a
response_matrix. See
data-raw/move_condition_means_to_holdout.R.xslData‘s validator now expects one
accuracy value (and one response_matrix row)
per test item, i.e. per test trial when test is given,
rather than per word. The filtering conditions’ 24-item tests failed the
old check.mafc_test() (and xslData test
specifications) accept an optional correct element giving
each test trial’s correct object; it defaults to the tested word’s own
object, as before.rescorla_wagner() applied each trial’s prediction
errors to the wrong associations. The errors, one per present object,
were added to the trial’s word x object submatrix as a vector, which R
recycles down the columns, so word i received the error for the trial’s
i-th object, for every object on the trial. Because training files list
each trial’s words and objects in matching order, each word got its own
referent’s error added equally to all the trial’s objects. Luce choice
over its associations was then exactly its co-occurrence proportion,
which is why rescorla_wagner() predicted the same
accuracies as baseline() in most designs. Each object’s
error now goes to that object’s associations, matching a cell-by-cell
implementation of Rescorla & Wagner (1972)
(tests/testthat/test-rescorla_wagner.R), and cue
competition works (e.g. blocking). This changes all
rescorla_wagner() results and fits. Fit to
xsl_datasets, the corrected model still ends up close to
baseline() (its learning rate goes to near 0, where it
counts co-occurrences), because it learns only about objects present on
a trial.fits/v05/specs.R), so
get_group_model_fit() and
get_crossvalidated_model_fit() search the same space.
This changes their results..
uncfam()’s own optimum (B ~ 14 on
xsl_datasets).fazly(): lambda in [1e-10, 2], beta in [2, 20000].bayesian_decay(): each parameter in [1e-5, 10].propose_but_verify(), guess_and_test() and
pursuit(): lower bounds just above 0.rescorla_wagner() now fits beta,
C and lambda with alpha fixed at
1, since alpha and beta only enter as a
product.multi_sampling()’s registry bounds were attached to the
wrong parameters. Its parameters are ordered (C, X, B, K), but the
bounds were written for (X, B, C, K), so decay C was fit in
[.01, .5].tilles() now follows Tilles & Fontanari (2013) in
three places where it departed from the paper. It matches an
independent, literal transcription of the paper’s eqs 1-13 and procedure
(tests/testthat/helper-tilles-literal.R) to 1e-12 on all 47
xsl_datasets conditions, including at the parameter bounds.
This changes all tilles() results and
fits..
alpha is recomputed before every trial for every
word seen so far, as the paper’s procedure (p. 4) specifies. Words
absent from a trial use it in eqs 12-13. They used to keep the
alpha from the last trial they appeared on, and a word seen
only on trial 1 kept alpha_0.201-222), eq 1 divided by the number of words
and eq 2 by the number of new words, so a word’s associations summed to
as much as 3.95 rather than 1 (eq 10). When a new word appears with no
new objects (73 trials in 15 conditions, where its referent was shown
earlier as a foil), eq 2’s b share now goes to all of the
trial’s objects; it used to be dropped.206 at any
parameters, and in most of the 3x4-type conditions at
b = 1).b = 1 and x = alpha_0 = 1
no longer stop with “no associations left”.alpha_0 = .9855), SSE over the 47 conditions is .6466
(.6399 with only the eq 7 normalization fix below, .6125 before either),
and model accuracy changes by up to .135 for a word. Refitting with
v05’s settings gives SSE .6061 at x = .124, b = .129,
alpha_0 = .979, the same as with only the eq 7 fix
(.6060).tilles()’s eq 7 now normalizes the entropy of a word’s
associations to [0, 1], as Tilles & Fontanari (2013) specify, by
dividing shannon_entropy() (in bits) by
log2(n_o) rather than by log(n_o) in nats. The
mismatch came from the original 2013 implementation, so the published
tilles() results ran on it too. This changes all
tilles() results and fits, except at
alpha_0 = 1, where eq 7’s alpha is 1 whatever the entropy.
Because alpha now stays in [alpha_0, 1],
tilles() gives valid probabilities at every interior
parameter vector. On all 47 xsl_datasets conditions at 1682
parameter vectors, all 72944 runs with x, b and alpha_0 in
(0, 1) were valid, including the 23808 that had stopped with
“associations became negative” (see below), and no association went
negative even transiently. The only remaining stops were the boundary
cases b = 1 and x = alpha_0 = 1, which no
longer occur (see above). At the v05 group-fit parameters (x = .129, b =
.124, alpha_0 = .985), SSE over the 47 conditions rises
from .6125 to .6399. Refitting gives .6060 at almost the same parameters
(x = .129, b = .126, alpha_0 = .979). When only one object
has been seen, eq 7’s normalized entropy is now 0 instead of 0/0.tilles() no longer returns NaN, or fails with “missing
value where TRUE/FALSE needed” inside shannon_entropy(), at
small alpha_0. Instead it stops with a clear error,
tilles(): associations became negative by trial .... The
cause is eq 7, carried over from the original 2013 implementation: it
normalizes entropy in bits (log2) by log(n_o)
in nats. Its uncertainty term can then reach 1/log(2)
(about 1.44) rather than 1, so alpha falls below 0 when
alpha_0 is below about .31. Eqs 8-13 can then push
associations negative, and the next eq 7 entropy of a negative row is
NA. tilles() now stops when a word’s associations are
negative at its next eq 7 (where NaN used to start), or when final
associations are negative (previously returned silently as negative
probabilities). Separately, at the boundary values b = 1 (a
novel word appearing with no novel objects gets no association) and
x = alpha_0 = 1, a word could be left with no associations,
giving the same NA (no longer, see above); this stopped with
tilles(): word ... has no associations left by trial ....
Every run that returned finite, nonnegative associations is unchanged,
bit for bit. This was checked on all 47 xsl_datasets
conditions at 553 parameter vectors, including every one in the v05
refit’s error log. Fitting already scored errors and NA results as an
infinitely bad fit, so fits change only where tilles()
previously returned negative probabilities, which are now also scored as
infinitely bad. The normalization itself is corrected in the entry
above, so small alpha_0 no longer causes this; the checks
remain for parameters outside [0, 1] and for the boundary cases.tilles() no longer errors on training data with a
single trial. Its trial loop was
for (t in 2:length(data$words)), which for one trial is
2:1 = c(2, 1) and indexed a nonexistent second
trial. Results with two or more trials are unchanged.bayesian_decay()’s choice exponent ch_dec
was ignored on conditions with a test (the 4AFC conditions
211-214 and the 12 filtering conditions): xsl_run() applied
it as a power choice rule only on the get_perf() path,
while mafc_test() scored test trials with plain Luce
choice. predict_referent() and mafc_test()
gain a d argument (the choice exponent; NULL,
the default, leaves associations unchanged), and xsl_run()
passes a model’s ch_dec to both paths. This changes
bayesian_decay() fits involving any of these 16
conditions. Models without ch_dec are unchanged,
bit for bit.get_tp(), get_fscore(), and so
get_roc(), plot_roc() and
get_roc_max(), read a gold lexicon’s numeric word and
object ids as matrix positions rather than labels, scoring the wrong
cells whenever a matrix’s labels weren’t 1..n in order (matrices from
xsl_run() are, so its results were unaffected). They now
index by label.get_roc() returned NA at every threshold (and
get_roc_max() -Inf, with a warning) for a
matrix with any all-zero row, a word with no associations, which models
such as pursuit() and propose_but_verify()
often leave. Such a word now selects no referent.uncfam_elimination(), a variant of
uncfam() with learning by elimination (mutual exclusivity).
Each on-trial pair’s mutual confidence P(o|w) * P(w|o) is
computed from the current associations, and a pair gets elimination
support to the extent that the trial’s other words and objects are
already accounted for by each other. That support, weighted by a free
parameter eps, is added to uncfam()’s
familiarity-and-uncertainty share of the trial’s associative weight
before renormalizing; eps = 0 reproduces
uncfam() exactly. In uncfam(), an
already-known pair on a trial draws weight away from the
trial’s other pairs; with elimination, knowing one word directs learning
to the remaining word-object pairing. Motivated by the initial-accuracy
experiment, where learners were more accurate on an initially mis-paired
word the more of its study trials it shared with an initially accurate
word. Registered in
show_models()/xsl_model_registry()uncfam_gamma(), a variant of
uncfam() that decouples familiarity’s contribution to a
trial’s attentional allocation from uncertainty’s.
uncfam()’s allocation weight is
exp(B * entropy) * assocs – entropy gets a free exponent
(B), but familiarity (assocs) enters linearly
with an implicit weight of 1 and no free parameter of its own.
uncfam_gamma() adds one parameter, gamma,
raising familiarity to a free power instead (assocs^gamma);
gamma = 1 reproduces uncfam() exactly.
Cross-validated against all of xsl_datasets, it beats plain
uncfam() in every fold of a 5-fold split (mean test SSE
0.48 vs. 0.56), with a best-fit gamma around 0.3
(familiarity’s pull dampens with diminishing returns). A
separate analysis fitting this same free-gamma substrate to
active cross-situational word learning (learners choosing which items to
see named next, rather than a fixed passive sequence) instead found a
best-fit gamma around 2 (amplified, not dampened) – see
?uncfam_gamma for the full comparisonminerva2(), todam(), and
rem(), three distributed/episodic memory models ported from
standalone prototypes that predated this package (Hintzman’s MINERVA2,
1984/1986; Murdock’s TODAM, 1982; Shiffrin & Steyvers’ REM, 1997)
and were never wired into it or tested. All three represent
words/objects as random feature vectors rather than tallying a single
word-by-object association, and turn their native similarity/odds score
into a Luce-choice-ready matrix via an exponentiated
(X-temperature) or likelihood-ratio-based choice rule so
they plug into get_perf()/mafc_test() like
every other model here. Porting surfaced real bugs in the prototypes,
fixed along the way (see each model’s @details/source
comments): minerva2()‘s prototype compared a 2D
echo-content vector against D-dim object vectors (silent recycling, not
an error) instead of just the echo’s object half; todam()’s
prototype computed trial-1 attention weights as
cos_sim(item, 0) = NaN, poisoning memory from the first
trial on, and both todam() prototypes called the
long-removed as.real(); rem()’s prototype
never actually implemented REM at all (it declared a geometric
feature-vector generator but its model/test functions were an unmodified
copy of the MINERVA2 prototype’s continuous-Gaussian code) –
rem() here is a fresh adaptation of Shiffrin &
Steyvers’ storage/odds equations to this package’s associative task.
Benchmarking (see
?minerva2/?todam/?rem performance
notes) found rem()’s raw per-simulation likelihood-ratio
odds span several orders of magnitude (a lucky match on one rare feature
value can dominate), which would otherwise let a single lucky simulation
hijack xsl_run()’s sum-across-simulations aggregation;
rem() now row-normalizes each simulation’s choice weights
before returning them. All three are registered in
show_models()/xsl_model_registry(), fitting
only their choice-rule temperature (X) or storage
parameters (g, u, c) by default –
D/w (feature-vector dimensionality) are fixed
representational hyperparameters, not free cognitive parameters. Speed
varies widely: on the full xsl_datasets (53 conditions) at
n_sim = 500, minerva2() takes ~30s,
todam() ~8 min, and rem() ~20 min –
todam()/rem() are impractical to
xsl_fit() at full DEoptim settings
(NP = 100, itermax = 100 would be tens of thousands of
evaluations) without sharply reducing
n_sim/NP/itermax, as the
package’s own fitting examples already do for slow modelsxsl_datasets from Kachergis, Yu, & Shiffrin (2009) –
the same paper behind the existing
1_max_temp_spat_cont_orig/4_no_spat_orig_max_tc
conditions. data-raw/XSL-dataset-fields.csv documented 6
“overlap” conditions (3 matched pairs of a control and non-control order
at the same overlap degree) that were never pulled in; only the 3
control-order ones (1olap3tr_contr,
temp_cont_1olap2tr_contr,
temp_cont_2olap2tr_contr) have real subject-level accuracy
data findable anywhere in the repo (in
data-raw/agg_data/x13_1-21.txt, per
export_data.R’s own comments mapping its generic condition
labels to these named orders). The other 3 have order files but no
discoverable human data, and are deliberately left out rather than
guessed. See
data-raw/add_temporal_contiguity_overlap.Rbenitez2020 (a named
list of 7 xslData objects, one per condition x age-group
combination) from Benitez et al. (2020), “The temporal structure of
naming events differentially affects children’s and adults’
cross-situational word learning” (data: osf.io/2hmxr).
data-raw/Benitez2020-rep/Benitez2020_XSLdata_extraction.R
already extracted training orders, 2AFC test trials, and response-count
matrices from the raw OSF data but stopped short of building
xslData objects (a literal # ToDo in the
file); data-raw/add_benitez2020.R picks up from there. 8
symmetric word-object pairs, 12 training trials, 8-item 2AFC test;
unlike the other standalone datasets, real per-word accuracy is
recoverable directly from the provided response matrices (not just an
overall mean) – structurally it could join xsl_datasets
directly, but is kept standalone by choice, as held-out
generalization-test data (see xsl_holdout_datasets)xsl_holdout_datasets, a registry of every dataset
in the package that is not part of xsl_datasets
and therefore never seen by
get_group_model_fit()/get_crossvalidated_model_fit()
– i.e. everything available for testing a fitted model’s generalization
to genuinely held-out data, along with how to score each one (most need
a different scoring approach than xsl_datasets’s standard
diagonal convention, which is exactly why they’re standalone;
benitez2020 and kachergis_initial_accuracy are
the two directly-scorable exceptions, kept out of the fitting pool
specifically to serve as held-out data)vlach_debrock2017
(Vlach & DeBrock, 2017, Journal of Memory and Language,
“Remember dax? Relations between children’s cross-situational word
learning, memory, and language abilities”). 12 symmetric word-object
pairs, 2 pairs per training trial, 36 trials, with pairs varying in
whether their 6 repetitions are massed or spaced (n = 47 children, ages
2-5). The source only reports an overall mean/SD across all pairs and
participants, not a per-word breakdown, so accuracy is
NA throughout rather than fabricated – kept standalone (not
part of xsl_datasets) for the same reason as
kachergis2012_highlighting/gangwani2011_category.
See ?vlach_debrock2017 and
data-raw/add_vlach_debrock2017.Rgangwani2011_category
(Experiment 1 of Gangwani, Kachergis & Yu, “Simultaneous
Cross-situational Learning of Category and Object Names”; an
undergraduate-journal version is Gangwani & Kachergis, Indiana
Undergraduate Journal of Cognitive Science 5, 2010). A hierarchical
design: each training trial shows 2 objects and names 3 words – the two
objects’ own 1-to-1 names plus one 1-to-many “category” label naming
those two objects and two others seen on other trials – so every object
carries both a basic name and a category label, and the paper’s finding
is that learners violate mutual exclusivity by learning both. A named
list of three xslData objects (one per training block: ad
hoc / natural / ad hoc), sharing one 36-trial co-occurrence schedule.
Its 15 x 12 (word x object) association matrix breaks
xslData’s diagonal-scoring convention the same way
kachergis2012_highlighting does, so it is kept standalone;
score it with score_gangwani2011_category() in the new
tests/bakeoff_comparison/gangwani2011_category_fit.R, which
reads 12-AFC name accuracy, 3-AFC category accuracy, and the paper’s
headline ME-violation measure straight off any model’s matrix. See
?gangwani2011_category and
data-raw/add_gangwani2011_category.Rcontrol$start_matrix is no longer silently ignored.
Only some models initialized from it; the rest dropped it without
warning (so e.g. a familiarization phase passed as a start matrix simply
vanished for them). Models now declare
supports_start_matrix in xslMod(), and
xsl_run() errors if a start matrix is given to one that
doesn’t, or if it doesn’t match the training data’s dimensions.
baseline(), decay(),
rescorla_wagner(), softmax_rl(),
uncfam_sampling(), and multi_sampling() now
support it, joining the uncfam() family,
propose_but_verify(), and pursuit(). The
remaining models (fazly(), guess_and_test(),
tilles(), bayesian_decay(),
kalman_filter(),
fgt2009()/fgt2009_rsa(),
minerva2()/todam()/rem()) have no
single association matrix a start matrix maps onto (or would need a
model-specific convention, e.g. a variance as well as a mean), so prior
learning should be passed to them as training trials, as
kachergis_initial_accuracy does. A custom model built with
xslMod() must set supports_start_matrix = TRUE
to receive one. Runs without a start matrix are unchangeduncfam_gamma(), uncfam_elimination(),
minerva2()/todam()/rem();
vlach_debrock2017, benitez2020,
xsl_holdout_datasets), and adds two sections: passing prior
learning to a model as training trials vs. a start_matrix,
and testing a fitted model’s generalization on the held-out datasets.
Its description of the dataset structure used field names
(trials, objs, perf) that no
longer exist – so its create_cooc_matrix() example was
silently printing an empty matrix – and said words/objects must be
referred to by index, which no longer holdsxsl_fit() with a parallel DEoptim backend
(DEoptim.control(parallelType = "parallel")) silently
returned bestval = Inf when control was passed
as a variable
(e.g. ctrl <- xslControl(n_sim = 100); xsl_fit(..., control = ctrl)):
control was never forced, so the objective shipped to the
worker processes carried an unevaluated promise that each worker tried
to evaluate in its own empty global environment, every evaluation
errored, and the error was converted to Inf.
control is now forced before fitting, and
xsl_fit() warns when every evaluation of a fit failed
instead of returning an all-Inf result silently. Serial
fits were unaffected.create_cooc_matrix() looped over the number of distinct
words rather than the number of trials: with more words than
trials it errored, and with fewer (e.g. its own documented example,
xsl_datasets[[1]]: 18 words, 36 trials) it silently tallied
only the first trials. It also indexed by raw label (the same issue as
#14). Both fixed; plot_training_trials() now indexes by
label too. No model uses either function, so no fits changepropose_but_verify() and pursuit() drew
new hypotheses with sample(tr_o, ...). On a trial with a
single object, tr_o is one value, and when that value is an
integer k, R’s sample(k, n) draws from
1:k rather than returning k – so a word seen
alone with an object usually hypothesized a different object,
one not even on the trial (propose_but_verify()
hypothesized the right object ~1/8 of the time in an 8-object toy). Both
now index into tr_o
(tr_o[sample.int(length(tr_o), n, replace = TRUE)], as
guess_and_test() already did). No xsl_datasets
condition has single-object trials, so fits to it are unchanged;
kachergis_initial_accuracy (18 single-object
familiarization trials per condition),
kachergis2012_highlighting’s words-as-cues condition (28
trials naming two words with one object), and the naturalistic corpora
(rollins_corpus, fm_corpus) are affected.
Integer-labelled data (e.g. kachergis_initial_accuracy) was
affected before this release too; the corpora were not – they reached
these models as string labels, for which sample() behaves
as intended – until #14 began handing models integer positions, so for
the corpora this is a regression in 0.4.0.m[tr_w, tr_o]) rather than by position in the sorted
vocabulary, which is only correct for labels that are exactly
1..N (#14): a vocabulary with a gap crashed (“subscript out
of bounds”), and a raw value that still fell in range silently updated
the wrong row. Fixed once, in xsl_run(): models now receive
each dataset’s words/objects as positions, and the original labels are
restored on the returned matrix, perf, and trajectory. Test
trials are scored by label, and a start_matrix with
dimnames is aligned to the data by label. Checked against every registry
model on every bundled dataset (23 models x 75 datasets, fixed seed):
1721 of 1725 runs are bit-identical, including all of
xsl_datasets and the standalone datasets, whose labels are
already 1..N. The 4 that differ are
propose_but_verify() and pursuit() on
rollins_corpus/fm_corpus, where the fix
corrects a real error: both compared a column position against
the trial’s object labels, which can never match for character
labels, so on the corpora every hypothesis was treated as disconfirmed
and none was ever verified. (Handing these models integer positions also
exposed the single-object sample() bug above on the same
corpora; both fixes are needed for correct corpus results.) The
propose_but_verify and pursuit results
previously in tests/bakeoff_comparison/corpus_report.md
were therefore invalid (including pursuit’s reported collapse on FM);
the report and vignette("corpora") have been rerun on the
fixed codeuncfam_predictive() now caps associations at
beta (1), its stated maximum, as well as flooring them at
0. Its unnormalized update could overshoot far above beta
on a word’s first exposure: the prediction error is then
~beta, and it is scaled by exp(B * entropy) of
the word’s row, which update_known() spreads over every
object seen so far – so a word first seen after many others could reach
an association in the thousands in a single trial (e.g. ~4600 for the
18th of 18 sequentially introduced words at X = .8, B = 1.6, C = .7).
That association then dominated the word’s row and collapsed its
entropy, locking in the first pairing it was seen with; in the
initial-accuracy study this made the model predict learners would choose
an initially inaccurate word’s familiarization referent 3-4x as often as
they did. This changes uncfam_predictive()’s
fits wherever the overshoot occurred, including the corpus-fit
gains reported for it in
tests/bakeoff_comparison/corpus_report.md (not rerun
here)get_perf() now treats negative associations as 0 before
applying the Luce choice rule (as predict_referent()
already did), and answers a word whose associations are then all 0 at
chance (1 / ncol(m)) rather than returning
NaN. Error-driven models (rescorla_wagner(),
kalman_filter()) can learn negative associations, for which
diag(m) / rowSums(m) gave per-item “probabilities” outside
[0, 1] – which DEoptim could exploit when fitting. In the
initial-accuracy study, fitted rescorla_wagner() had
negative associations for about half of participants and out-of-range
predictions for 25 of 61. This changes those models’
fits wherever negative associations occurredkachergis_initial_accuracy put the
familiarization pairing on the diagonal (the
xslData “correct referent” slot), but the experiment’s
test-correct answer is always the study pairing. For every
initially inaccurate item these differ, so any model’s accuracy on those
items was actually its probability of choosing the wrong,
familiarization object. Its per-item accuracy was also
keyed by init_word_ind (the word originally familiarized
with the tested word’s correct object) rather than by the tested word.
Both are fixed: objects are now indexed by the word they are studied
with, so the study pairing is the diagonal and an initially inaccurate
word’s familiarization object is off-diagonal, and accuracy
is keyed by the tested word. It now also excludes the 4 participants
whose median test response time was under 400 ms (as the accompanying
manuscript does), leaving 34 High IA and 27 Low IA participants.
tests/bakeoff_comparison/kachergis_initial_accuracy_fit.R
and kachergis_initial_accuracy_persubject_fit.R are updated
and rerun accordinglykachergis_initial_accuracy’s familiarization trials
were in word-index order; they now follow the order participants
actually saw them (the same for every participant, recovered from the
raw familiarization records). Order matters for the
uncfam() family, whose first-exposure update depends on how
many objects have already been seen via update_known() –
for uncfam_predictive() it determined how far a word’s
familiarization association overshot beta before the cap
above. The bakeoff scripts are rerun accordinglyuncfam():
uncfam_attention() (BAM+Attention), which scales a trial’s
learning rate by that trial’s object uncertainty relative to the running
mean (implementing Fitneva & Christiansen 2015’s system-level
attention account), and uncfam_predictive()
(BAM+Prediction), which replaces the normalized update rule with an
item-level, Rescorla-Wagner-style prediction-error term (floored at 0 to
prevent the unnormalized update from diverging over trials)xsl_datasets:
kachergis2012_highlighting (Kachergis, 2012, CogSci) and
kachergis_initial_accuracy (an unpublished MTurk
initial-accuracy manipulation).
kachergis2012_highlighting‘s training trials are genuine
simultaneous 2-cue trials (2 words + 1 object, or 2 objects + 1 word,
per trial, matching the paper’s Figure 2), read directly from the real
per-trial ordering file for the real N=67 dataset behind the paper’s
published statistics – confirmed by reproducing its exact reported
response proportions and chi-square values (an earlier version of this
dataset, built from a different, superficially similar file before that
real data was located, did not reproduce them, and has been replaced).
Each condition replicates the classic 3-role highlighting structure (PE,
PL, I) twice; words-as-cues’ PL items have their real target at a
different index than their own word index, and its I items have two
legitimate targets – neither fits xslData’s
one-correct-object-per-word accuracy convention, so only
its two PE items get a real accuracy value (the rest are
NA). Objects-as-cues has no such issue and gets a real
accuracy value for all 4 of its items. Because of the partial-NA
accuracy vector, kachergis2012_highlighting is kept
standalone: an NA would silently poison every other
dataset’s aggregate fit in
get_group_model_fit()/get_crossvalidated_model_fit(),
which sum/average SSE across all of xsl_datasets.
kachergis_initial_accuracy has no such blocker but is kept
standalone for now. See their ?help pages and
data-raw/add_kachergis2012_highlighting.R/add_kachergis_initial_accuracy.R
for full construction details, and
tests/bakeoff_comparison/kachergis2012_highlighting_fit.R
for a custom scorer that fits a model against all four of the paper’s
reported proportions (PE-E, PL-L, I-E, I-L) directly off the association
matrix, the way the paper’s own (non-package) fitting procedure didkalman_filter(), a Kalman-filter generalization
of rescorla_wagner()’s error-driven update: each
word-object association is tracked as a Gaussian belief (mean and
variance), giving an adaptive learning rate that starts high under
uncertainty and shrinks with confidence, rather than a fixed rate (Dayan
& Kakade, 2000; Kruschke, 2008)softmax_rl(), a Q-learning-style reinforcement
learning model. Unlike every other model in the package (which updates
every word-object pair presented together on a trial), it makes a
discrete choice per word – sampling a candidate referent from a softmax
policy over its current values, then updating only that guess via a
TD/delta-rule reward (1 if the guess was present in the scene, 0
otherwise). This mirrors
propose_but_verify()/pursuit()’s
propose-and-verify logic but replaces their threshold-based keep/discard
rules with graded value learning and explicit softmax explorationxsl_datasets, which is scored by SSE against a human
accuracy vector these don’t have): rollins_corpus – the
CHILDES/Rollins corpus the Frank, Goodman & Tenenbaum (2009) model
was fit to (619 mother-infant utterances, 416 words, 22 objects) – and
fm_corpus – the larger Frank, Tenenbaum & Fernald
corpus (4763 utterances, 1122 words, 30 objects), which additionally
carries the speaker’s hand-coded referential intention per utterance
(fm_corpus$intents). Each is a list bundling the training
xslData ($data) with a gold-standard lexicon
($gold, plus $gold_variants for FM) to be
scored with
get_fscore()/get_roc()/get_tp()
rather than SSE. Gaze/hand attentional cues in the FM source are not
imported. See data-raw/add_wurwur_corpora.Rpredict_referent(), a model-agnostic trial-level
prediction: given any model’s word-object matrix, a heard word, and the
objects present, it returns the probability distribution over those
objects
(P(object | word) proportional to m[word, object] * prior(object),
normalized over the objects present) – the quantity to compare against a
participant’s choice on a referent-selection trial. Supports a
non-uniform prior (set an entry to 0 to remove an object
the speaker has no epistemic access to) and a
pragmatic = TRUE mode that resolves the objects with a
one-step Rational Speech Act pragmatic listener
(rsa_listener()/rsa_speaker(), also exported),
yielding strong mutual exclusivity when a heard word is lexically
ambiguous and a competitor word names one of the candidates.
mafc_test() is now a thin wrapper over it (behavior
unchanged; gains a ... passthrough for
pragmatic/threshold)fgt2009() and fgt2009_rsa(), a
pure-R port of the intentional Bayesian word-learning model of Frank,
Goodman & Tenenbaum (2009) and its Rational Speech Act (RSA)
pragmatic extension, from the Python package wordlearn
(github.com/mcfrank/wurwur). Unlike every other model in the package –
which tally word-object associations incrementally, trial by trial –
this is a batch model: it does joint MCMC inference
(blocked per-object Gibbs plus MI-biased single-edge Metropolis moves)
over the whole corpus at once, asking which lexicon best explains
all the data, so xslFit$matrix holds posterior
edge marginals P(word names object | corpus) rather than a
running tally. It therefore has no learning trajectory and ignores
control$reps. The scoring kernel is verified to match the
Python reference to numerical precision.
gamma/kappa default to the values used for the
analogous cross-situational simulation in the source package
(gamma = 1, since every word in a controlled XSL experiment
is a referential label). The lexicon-size prior alpha
scales with corpus size and its fit-vs-alpha curve is
single-peaked, so it is not wired into
xsl_model_registry()’s DEoptim bounds like the other
models; use the new fgt2009_sweep_alpha() helper instead
(which takes an optional gold argument to report the
best-threshold F-score against a gold lexicon at each
alpha, for corpora with no human accuracy vector)rollins_corpus/fm_corpus, score the learned
matrix against a gold lexicon with
get_fscore()/get_roc()/get_roc_max(),
and choose fgt2009()’s alpha by sweeping. It
ends with a quick model comparison (bayesian_decay and
fazly recover the gold lexicon best on both corpora, with
fgt2009 close behind and highest-precision). The comparison
script and its outputs are in
tests/bakeoff_comparison/corpus_gold_comparison.R (every
model at registry defaults); a companion,
corpus_gold_fits.R, fits each model’s parameters to each
corpus (parallel DEoptim maximizing gold-lexicon F, since these corpora
have no accuracy vector; fgt2009() by an alpha
sweep). Fitting lifts every model – dramatically for the ones whose
registry defaults were tuned on controlled experiments
(tilles +0.34, uncfam_predictive +0.15/+0.32)
– and bayesian_decay fit to the FM corpus reaches F = 0.56.
tests/bakeoff_comparison/corpus_report.md is a rendered
write-up of all of thisxsl_run() is now far more memory-frugal on long inputs
(corpora). Previously it built the full list of every simulation’s
xslFit – each carrying a word-by-object matrix per
training trial – and only reduced them at the end, so a stochastic
model on the 4763-utterance FM corpus at n_sim = 50 needed
tens of GB and simply failed. Now: (1) the per-trial trajectory
(xslFit$traj) is only recorded when
xslControl(keep_traj = TRUE) – it is off by default and is
not used by any function in the package; (2) xsl_run()
accumulates the summed matrix one simulation at a time rather than
holding all n_sim at once; (3) the full per-simulation list
(fits[[i]]$sims) is dropped unless
xslControl(keep_sims = TRUE), replaced by
fits[[i]]$responses, a compact n_sim x n-words
matrix of each simulated participant’s final per-word accuracy.
matrix, perf, and sse are
numerically unchanged.uncfam_sampling()/multi_sampling(): every
trial was allocating two full voc_sz x ref_sz matrices and
computing a full outer product, even though only the handful of cells at
[tr_w, tr_o] are ever nonzero. Entropy, weighting, and the
update are now computed on that small per-trial submatrix instead.
Behaviour-preserving – the sample() call’s probability
vector is still built at full ref_sz length (zero outside
tr_o), so the RNG draw sequence, and therefore every
existing fit, is unchanged. 1.17x faster on a small lab condition, 1.97x
on rollins_corpus (bigger vocabularies benefit more, since
that’s the dimension the fix targets)shannon_entropy() computed sum(p) twice
(once for the <= 0 check, again for normalization); now
computed once and reusedupdate_known()’s which(...)-based
zero-filling replaced with direct logical-vector indexing, removing five
which() calls per trialupdate_known() computed
which(m[w, fam_objects] == 0) – positions within
the fam_objects-filtered subset – and then indexed
m[w, zeros] as if those were absolute column positions,
silently filling startval into the wrong cell whenever
fam_objects (or fam_words, for the symmetric
object-side fill) was a proper subset of all columns/rows –
i.e. whenever some word or object was still wholly unfamiliar. Affects
all 5 models that call it (uncfam(),
uncfam_attention(), uncfam_predictive(),
uncfam_sampling(), multi_sampling()). In
practice this only ever mis-set a handful of near-zero cells during
early training that get overwritten by real learning signal soon after –
verified to make no difference to the final matrix on any of
xsl_datasets’ 53 conditions or fm_corpus, and
a single cell’s startval on rollins_corpus.
Fixed as part of the logical-indexing rewrite above.guess_and_test() errored (if with a zero-
or multiple-length condition at the disconfirmation step) on any
utterance that repeats a word – common in a naturalistic corpus,
impossible in a controlled trial – because the repeated word was visited
twice and the first pass could clear or double-set its hypothesis. It
now de-duplicates a trial’s words and objects. Its disconfirmation check
also compared a column position to the object labels
(only equivalent when objects are labelled 1..N), and it
applied new hypotheses to need_hypoths[w] rather than the
actual stored word store[w] (setting hypotheses for the
wrong words whenever only a subset rolled below the storage threshold);
both are fixed.tilles() coerced each trial’s words/objects with
as.integer() (it indexes its state by position), turning
any non-integer label into NA and leaving the matrix
all-zero; it also never set dimnames on its matrix, so the
result could not be scored against a gold lexicon by label. It now maps
labels to positions with match() and names its matrix.
xsl_datasets results are unchanged (the labels there are
already 1..N).softmax_rl() checked its sampled guess against the
trial’s objects with proposal %in% tr_o, comparing a column
position (1..ref_sz) against the objects’
labels. For xsl_datasets the object labels are
1..N so it happened to work, but on any dataset whose
objects have non-integer (or non-contiguous) labels – e.g. either
bundled corpus – every reward was 0 and the learned Q-matrix stayed
identically zero. It now maps the objects present to positions
(match(tr_o, ref)) before the check, and skips a trial with
no objects present. xsl_datasets results are
unchanged.uncfam_sampling() and multi_sampling()
errored (sample(): “too few positive probabilities”) on any
dataset containing an utterance with no objects present – e.g. the ~320
non-referential utterances in fm_corpus, which made both
models unable to run on a naturalistic corpus. Such a trial now leaves
the association matrix unchanged apart from decay (as
uncfam() already did), by skipping the per-word sampling
when its weights are all zero and only applying the normalized update
when something was sampled. Behaviour on datasets without objectless
trials (all of xsl_datasets) is unchanged.tilles() was essentially unfittable – three separate
bugs (not the single one originally suspected) made a quick DEoptim
search return Inf for every draw: (1) its
alpha term was recomputed fresh every trial instead of
persisting per-word across trials, so words absent from the current
trial (needed by the “other old words” update) always had
alpha = NA; (2) an internal matrix was sized
voc_sz x voc_sz instead of voc_sz x ref_sz, a
copy-paste bug that only breaks on asymmetric datasets (word count !=
object count); (3) a flux computation divided by a quantity that can
legitimately be zero (a previously-seen word paired with an entirely
new-to-it set of objects), producing Inf * 0 = NaN. One
dataset failed on 100% of a 40-draw sweep regardless of parameters
before these fixes; after them, a 100-draw sweep shows a residual ~2%
failure rate confined to extreme parameter-space tails, on par with
other models’ known numerical edgesshow_models(),
show_datasets(), get_group_model_fit(), and
get_crossvalidated_model_fit() (now actually exported and
working)cross_validated_group_fits(),
get_train_test_split()) for the current
xslMod/xslData architecturexsl_datasets (44 -> 53): 2
from Kachergis, Yu, & Shiffrin (2009) (temporal contiguity), 3 from
Suanda, Mugwanya, & Namy (2014), and 4 from Koehne, Trueswell, &
Gleitman (2013)xslData() no longer rejects data with no
response_matrix supplied – this was breaking the vignette,
most function examples, and
get_example_ambiguous_condition()/get_example_unambiguous_condition()get_group_model_fit()/get_crossvalidated_model_fit()
called uncfam_sampling(), multi_sampling(),
tilles(), and fazly() with incorrect parameter
names, causing them to error for those modelsupdate_params() was dropping the parameter list’s
names/type, which made xsl_run() error for every model
except bayesian_decay() whenever it was called after
fitting (e.g. from xsl_fit() or cross-validation)xsl_fit() now treats a model erroring or returning
NA/NaN SSE for a given parameter draw as an
infinitely bad fit instead of aborting the whole optimization runtilles() mislabeling its model name as
"rescorla_wagner"create_cooc_matrix() test had an incorrect expected
valueplot_training_trials() now gives an informative error
if the optional viridis/gganimate packages
aren’t installed, instead of a raw namespace-loading errorxsl_run() checked !is.null(dat$test) to
decide whether to use mafc_test(), but
xslData()’s test defaults to
list() (not NULL), so any dataset built
without explicit test trials – including the package’s own
get_example_ambiguous_condition()/get_example_unambiguous_condition()
– was silently routed through mafc_test(mat, list()). That
returned numeric(0) rather than erroring (an R quirk:
1:0 counts down instead of being empty, and
vec[i] <- numeric(0) truncates the vector rather than
erroring), producing a false sse = 0 “perfect fit” with no
warning. Now checks length(dat$test) > 0.guess_and_test() had an inverted guard clause
(if (length(which(m[w,]==1))) next) that made its
hypothesis-disconfirmation step permanently unreachable, since it was
only reached for words that already have a stored hypothesis
(guaranteeing the guard’s condition was always true). Hypotheses were
therefore never disconfirmed once formed, silently disabling a core part
of the model (“…on later encounters, learners attempt to retrieve this
hypothesis from memory and test it against a new context, updating it
only if it is disconfirmed”). Found by comparing the package’s fits
against the original pre-package research code’s fits for the same
44-condition dataset.