---
title: "Cross-round harmonisation"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Cross-round harmonisation}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
```

Comparing indicators across multiple survey rounds is essential for longitudinal policy evaluation. However, structural differences across the survey instruments pose a major barrier. This vignette walks through how `ihsMW` streamlines cross-round harmonisation and analysis.

## 1. The Harmonisation Challenge

Over the years, the Malawi Integrated Household Survey (IHS) questionnaires have evolved. Variables are added, retired, renamed, or relocated to different modules. 

For instance, the household size indicator is named `hhsize` in some files, `hh_size` in others, or is represented by variables counting household members. Similarly, nominal consumption expenditure variables and agricultural crop names frequently change, making direct cross-round comparisons error-prone and tedious.

## 2. The Crosswalk

To resolve this, `ihsMW` bundles a static crosswalk database containing mappings for over 6,400 variables across IHS2, IHS3, IHS4, IHS5, and IHS6. It acts as a translation layer, mapping round-specific variable names to consistent, harmonised names.

You can inspect the crosswalk programmatically:

```{r eval=FALSE}
library(ihsMW)

# Read the crosswalk
crosswalk <- read.csv(system.file("extdata", "ihs_crosswalk.csv", package = "ihsMW"))
head(crosswalk)
```

## 3. Basic Harmonisation

Use `ihs_harmonise()` to standardise column names in raw dataframes:

```{r eval=FALSE}
library(haven)

# Load IHS4 raw household module
ihs4_raw <- read_dta("path/to/IHS4/hh_mod_a_filt.dta")

# Harmonise to standard names
ihs4_clean <- ihs_harmonise(ihs4_raw, round = "IHS4")
```

## 4. Multi-round Panel Work

To compile a cross-round dataset for longitudinal analysis, load the data from each round, harmonise them separately, and bind them together:

```{r eval=FALSE}
library(dplyr)

# Load and harmonise IHS4
ihs4_raw <- read_dta("path/to/IHS4/hh_mod_a_filt.dta")
ihs4_harm <- ihs_harmonise(ihs4_raw, round = "IHS4")

# Load and harmonise IHS5
ihs5_raw <- read_dta("path/to/IHS5/hh_mod_a_filt.dta")
ihs5_harm <- ihs_harmonise(ihs5_raw, round = "IHS5")

# Load and harmonise IHS6
ihs6_raw <- read_dta("path/to/IHS6/hh_mod_a_filt.dta")
ihs6_harm <- ihs_harmonise(ihs6_raw, round = "IHS6")

# Bind rows - ihs_harmonise adds an `ihs_round` column automatically
pooled_data <- bind_rows(ihs4_harm, ihs5_harm, ihs6_harm)
```

Before pooling, check how many of your variables actually survive across every round you intend to use. Coverage is far from universal:

```{r}
library(ihsMW)

cw <- ihs_crosswalk_check(verbose = FALSE)
table(cw$n_rounds_avail)
```

Only a small core of variables appears in all five rounds. Everything else needs a narrower round window or a concept-level judgement call from you.

## 4a. What changed in IHS6

IHS6 (2024/25) is largely continuous with IHS5, but three differences matter:

- the design **stratum column was renamed** from `stratum` to `strata`. `ihs_svydesign()` and `ihs_panel_ids()` both handle this, but hard-coded `stratum` references in your own scripts will break;
- **`adulteq` (adult equivalents) is not present** in the IHS6 consumption aggregate, so per-adult-equivalent measures need to be constructed yourself;
- consumption aggregates are published in **July 2024 prices**, not the earlier base. Use `ihs_deflate()` to put every round on a common basis before comparing.

Some IHS6 crop and unit codes are also new and have no published NSO conversion factor yet; `ihs_convert_units()` returns `NA` for those rows and names the combinations it could not map.

## 5. Merging Modules

Within a single survey round, information is split across multiple modules (e.g., household demographics, agriculture, food consumption). Use `ihs_merge()` to merge these dataframes:

```{r eval=FALSE}
# Load household demographics and crop harvest modules
hh_demog <- read_dta("path/to/IHS5/hh_mod_a_filt.dta") |> ihs_harmonise("IHS5")
hh_agri <- read_dta("path/to/IHS5/ag_mod_i.dta") |> ihs_harmonise("IHS5")

# Merge modules - automatically detects common ID columns (e.g., case_id)
merged_data <- ihs_merge(hh_demog, hh_agri)
```

`ihs_merge()` checks if the join type results in unexpected row expansion and issues a warning if many-to-many joins occur.

## 6. Price Deflation

When comparing monetary values (e.g., household consumption, crop sales) across different years, nominal values must be deflated to account for inflation. `ihs_deflate()` uses bundled Malawi Consumer Price Index (CPI) data to convert nominal values to real values, with 2019 (IHS5 baseline) as the default reference year:

```{r eval=FALSE}
# Deflate expenditure variables to 2019 real prices
real_data <- ihs_deflate(
  data = pooled_data,
  value_cols = c("consumption_nominal", "food_exp_nominal")
)
# This creates new columns: `consumption_nominal_real` and `food_exp_nominal_real`
```

## 7. Quality Checks

To verify that the crosswalk mappings are valid and check how many variables are successfully mapped, use `ihs_crosswalk_check()`:

```{r eval=FALSE}
ihs_crosswalk_check()
```

Additionally, `ihs_panel_ids()` returns the standard ID columns (e.g. household ID, individual ID, enumeration area ID, strata, weights) for a given round to help you construct panel keys or verify design structures:

```{r eval=FALSE}
# Get standard ID columns for IHS5
ihs_panel_ids("IHS5")
```