OECDdata: discovery, careful selection, and reproducible analysis

A reliable workflow

Discover a dataflow, inspect its dimensions and dictionaries, restrict the request, inspect status and units, and only then calculate statistics. The eurodata package inspires the workflow, but names, classes and identities are OECD-native. The default route is SDMX REST v2. See the package README for current protocol references and a detailed interface and migration guide.

All evaluated chunks in this vignette are offline. The following object contains invented teaching observations, not actual OECD statistics.

x <- oecdExample()
x[, .(REF_AREA, TIME_PERIOD, value_, flags_)]
#>     REF_AREA TIME_PERIOD value_ flags_
#>       <char>      <char>  <num> <char>
#>  1:      USA     2024-01  100.0      A
#>  2:      USA     2024-02  101.5      A
#>  3:      USA     2024-03  102.0      A
#>  4:      USA     2024-04  103.2      A
#>  5:      USA     2024-05  102.9      A
#>  6:      USA     2024-06  104.1      P
#>  7:      FRA     2024-01  100.0      A
#>  8:      FRA     2024-02   99.4      A
#>  9:      FRA     2024-03  100.3      A
#> 10:      FRA     2024-04  101.2      A
#> 11:      FRA     2024-05  102.5      A
#> 12:      FRA     2024-06  103.0      P

Observations, dimensions and attributes

attr(x, "Dimensions")
#> [1] "FREQ"        "REF_AREA"    "MEASURE"     "GROUP"       "TIME_PERIOD"
attr(x, "OECDAttributes")
#> [1] "OBS_STATUS" "UNIT_MULT"
str(x)
#> Classes 'OECDDataset', 'data.table' and 'data.frame':    12 obs. of  9 variables:
#>  $ FREQ       : chr  "M" "M" "M" "M" ...
#>  $ REF_AREA   : chr  "USA" "USA" "USA" "USA" ...
#>  $ MEASURE    : chr  "IX" "IX" "IX" "IX" ...
#>  $ GROUP      : chr  "00" "00" "00" "00" ...
#>  $ TIME_PERIOD: chr  "2024-01" "2024-02" "2024-03" "2024-04" ...
#>  $ value_     : num  100 102 102 103 103 ...
#>  $ flags_     : chr  "A" "A" "A" "A" ...
#>  $ OBS_STATUS : chr  "A" "A" "A" "A" ...
#>  $ UNIT_MULT  : chr  "0" "0" "0" "0" ...
#>  - attr(*, ".internal.selfref")=<pointer: 0x0000023d2d14cf60> 
#>  - attr(*, "RequestedDatasetCode")= chr "TEST,DEMO,1.0"
#>  - attr(*, "OECDDatasetCode")= chr "TEST,DEMO,1.0"
#>  - attr(*, "DownloadTime")= POSIXct[1:1], format: "2024-01-01"
#>  - attr(*, "SourceURL")= chr "synthetic example bundled with OECDdata"
#>  - attr(*, "Cached")= logi FALSE
#>  - attr(*, "Dimensions")= chr [1:5] "FREQ" "REF_AREA" "MEASURE" "GROUP" ...
#>  - attr(*, "OECDAttributes")= chr [1:2] "OBS_STATUS" "UNIT_MULT"

value_ is numeric. Dimension codes remain strings or, with the default options, factors. Numeric-looking identifiers retain leading zeros. flags_ preserves OBS_STATUS without translating it into another provider’s status vocabulary. The original attribute is retained too.

series_keys <- setdiff(attr(x, "Dimensions"), "TIME_PERIOD")
x[, .(observations = .N, missing = sum(is.na(value_))), by = series_keys]
#>      FREQ REF_AREA MEASURE  GROUP observations missing
#>    <char>   <char>  <char> <char>        <int>   <int>
#> 1:      M      USA      IX     00            6       0
#> 2:      M      FRA      IX     00            6       0
x[, .N, by = OBS_STATUS]
#>    OBS_STATUS     N
#>        <char> <int>
#> 1:          A    10
#> 2:          P     2

An incomplete series is not automatically filled with zeros. An attribute such as UNIT_MULT is not automatically applied: verify whether the reported value and the unit metadata require scaling in your analysis.

Safe reshaping and plotting

This synthetic example contains one frequency, measure and group per country. Check that simplification before dropping those columns.

stopifnot(x[, uniqueN(FREQ)] == 1L, x[, uniqueN(MEASURE)] == 1L,
          x[, uniqueN(GROUP)] == 1L,
          !anyDuplicated(x, by = c("TIME_PERIOD", "REF_AREA")))
dcast(x, TIME_PERIOD ~ REF_AREA, value.var = "value_")
#> Key: <TIME_PERIOD>
#>    TIME_PERIOD   FRA   USA
#>         <char> <num> <num>
#> 1:     2024-01 100.0 100.0
#> 2:     2024-02  99.4 101.5
#> 3:     2024-03 100.3 102.0
#> 4:     2024-04 101.2 103.2
#> 5:     2024-05 102.5 102.9
#> 6:     2024-06 103.0 104.1

one <- x[REF_AREA == "USA"][order(TIME_PERIOD)]
plot(as.Date(paste0(one$TIME_PERIOD, "-01")), one$value_, type = "l",
     xlab = "Month", ylab = "Synthetic index", main = "Invented example")

An adjacent-period change

Sorting is necessary but not sufficient. Check that observations are exactly one month apart, so a gap is not misinterpreted as one month’s growth.

copy(x) %>%
  .[, month_id := as.integer(substr(TIME_PERIOD, 1, 4)) * 12L +
                  as.integer(substr(TIME_PERIOD, 6, 7))] %>%
  .[order(month_id), `:=`(previous = shift(value_),
                        previous_month = shift(month_id)), by = series_keys] %>%
  .[, growth_pct := fifelse(!is.na(previous) & previous != 0 &
                             month_id - previous_month == 1L,
                           100 * (value_ / previous - 1), NA_real_)] %>%
  .[, .(REF_AREA, TIME_PERIOD, growth_pct)]
#>     REF_AREA TIME_PERIOD growth_pct
#>       <char>      <char>      <num>
#>  1:      USA     2024-01         NA
#>  2:      USA     2024-02  1.5000000
#>  3:      USA     2024-03  0.4926108
#>  4:      USA     2024-04  1.1764706
#>  5:      USA     2024-05 -0.2906977
#>  6:      USA     2024-06  1.1661808
#>  7:      FRA     2024-01         NA
#>  8:      FRA     2024-02 -0.6000000
#>  9:      FRA     2024-03  0.9054326
#> 10:      FRA     2024-04  0.8973081
#> 11:      FRA     2024-05  1.2845850
#> 12:      FRA     2024-06  0.4878049

Discovery and metadata (online, not evaluated)

OECDdata::find("health expenditure")
OECDdata::find(education, expenditure)
browseDataList(grepl("employment", `Dataset name`, ignore.case = TRUE))

code <- "CLI"
s <- oecdStructure(code)
s$flow
s$dimensions
s$labels$REF_AREA
importLabels("REF_AREA", dataset = code)
describe(code, wide = TRUE)

A codelist describes permissible values. An Actual content constraint can establish reported marginal availability, but not that every combination of values is observed. describe() and importMetabase() fail explicitly when that coverage cannot be established. availability = "allowed" is an explicit alternative for dictionary-based inventories, not a substitute presented as observed coverage.

Short codes and dataset identity (online, not evaluated)

A short code is matched against the current catalogue, not a hard-coded alias list. Ambiguity is an error. Agency and dataflow-prefix selectors can resolve it without making the dataset code harder to read. Resolve once for reuse:

ref <- oecdDataset("CLI", agency = "OECD.SDD.STES", prefix = "DSD_STES")
ref
importLabels(OECDDimCode = "REF_AREA", dataset = ref)

# A query preview fetches metadata but does not download observations.
oecdQuery("CLI", filters = list(
  .dataset = list(agency = "OECD.SDD.STES", prefix = "DSD_STES"),
  REF_AREA = "USA", FREQ = "M", TIME_PERIOD = c("2023-01", "2023-12")
))

The reserved .dataset block selects agency, dataflow namespace and/or version; it is not a set of series dimensions. prefix matches the dataflow ID prefix before @. The actual referenced DSD is read from the server metadata. All other filter entries retain their ordinary dimension semantics. A selector-only block leaves all observations unrestricted. A qualified reference avoids the catalogue lookup, and explicit selector/reference conflicts are errors.

A small download (online, not evaluated)

cli <- importData(code, list(REF_AREA = c("USA", "FRA"), FREQ = "M",
                             TIME_PERIOD = c("2023-01", "2023-12")))
attr(cli, "SourceURL")
attr(cli, "OECDDatasetCode")

The two time bounds define an inclusive range. An unspecified dimension is a wildcard, so the result may include multiple measures or transformations. Never collapse them into a single line or aggregate solely by country and date without checking the other key dimensions.

Caching and provenance

Requests are cached in a session temporary directory by default. Persistent storage is opt-in. Offline mode accepts expired but verified cache entries; missing or corrupt entries cause explicit errors. Cached data retain their original download time. A structure version is not a statistical vintage.

attributes(x)[c("RequestedDatasetCode", "OECDDatasetCode", "DownloadTime",
                "SourceURL", "Dimensions")]
#> $RequestedDatasetCode
#> [1] "TEST,DEMO,1.0"
#> 
#> $OECDDatasetCode
#> [1] "TEST,DEMO,1.0"
#> 
#> $DownloadTime
#> [1] "2024-01-01 UTC"
#> 
#> $SourceURL
#> [1] "synthetic example bundled with OECDdata"
#> 
#> $Dimensions
#> [1] "FREQ"        "REF_AREA"    "MEASURE"     "GROUP"       "TIME_PERIOD"

oecdOptions(cache_dir = tools::R_user_dir("OECDdata", "cache"))
# First make a successful online request with the desired metadata and filters.
# Then use offline = TRUE with the same request.
saveRDS(list(data = cli, provenance = attributes(cli)), "cli-snapshot.rds")

HTTP failures remain structured errors. HTTP 204 is an explicit empty result with a warning. The client never converts an arbitrary 404 or malformed response into an apparently successful empty dataset.