Skip to contents

Measure agreement between LLM codes and human codes

Source: R/code_agreement.R
code_agreement.Rd

Validates automated coding against a human-coded subsample. For each theme it reports raw agreement, Cohen's kappa with a confidence interval, and, treating the human codes as the reference, the LLM's precision, recall and F1 score.

Usage

code_agreement(
  llm_coded,
  human_coded,
  id_col = NULL,
  themes = NULL,
  conf_level = 0.95
)

Arguments

llm_coded

A data frame returned by llm_auto_code().

human_coded

A data frame with the same ID column and the human codes in columns named after the themes (logical, or 0/1).

id_col

Name of the ID column (a string). Defaults to the ID column used in llm_auto_code().

themes

Character vector of theme columns. Defaults to the codebook stored by llm_auto_code().

conf_level

Confidence level for the kappa interval.

Value

A tibble with one row per theme: n units compared, prevalence of the theme for each coder, percent agreement, kappa with kappa_low and kappa_high, and precision, recall and f1. Kappa is NA when it is undefined (both coders give every unit the same code).

Details

Units are matched by id_col, which must be unique in both data frames (use a segment ID when coding at segment level). Units missing from either data frame, or with an NA code from either coder, are left out of that theme's statistics.

The kappa interval uses the large-sample standard error \(\sqrt{p_o(1 - p_o) / (n(1 - p_e)^2)}\), which is approximate for small samples. As a rough guide, kappa above 0.6 is commonly read as substantial agreement and above 0.8 as near-perfect.

Examples

df <- mock_mixed_data(60, seed = 1)
coded <- llm_auto_code(df, user_id, transcript_feedback,
                       codebook = c("UI_Confusion", "Positive_Experience"),
                       coder = mock_coder())

# Pretend a human coded a subsample, using the hidden profile:
human <- data.frame(
  user_id = df$user_id[1:30],
  UI_Confusion = df$profile[1:30] == "frustrated",
  Positive_Experience = df$profile[1:30] == "efficient"
)
code_agreement(coded, human)
#> # A tibble: 2 × 11
#>   theme              n prevalence_llm prevalence_human agreement kappa kappa_low
#>   <chr>          <int>          <dbl>            <dbl>     <dbl> <dbl>     <dbl>
#> 1 UI_Confusion      30          0.433            0.433         1     1         1
#> 2 Positive_Expe…    30          0.333            0.333         1     1         1
#> # ℹ 4 more variables: kappa_high <dbl>, precision <dbl>, recall <dbl>, f1 <dbl>