Cleaning log

Simulated data, not real patients. FLUID-ICU is an invented study: every value was produced by a computer program with a fixed random seed. It does not describe any real patient, hospital or ICU. Any manuscript written from it must say that the data are simulated and must never be submitted as real research.

SIMULATED DATA: NOT REAL PATIENTS. This log belongs to the teaching dataset FLUID-ICU.

Raw file fluid_icu_raw.csv (identical content in fluid_icu_raw.xlsx): 623 rows, 29 columns, as received from ICU A-D
Clean file fluid_icu_clean.csv / .rds: 620 patients, 32 variables
Cleaning script R/01_clean.R (each step below is one block of the script)
Check The script rebuilds the clean file from the raw file and confirms it is identical, value for value and byte for byte
Rule The raw file is never edited. Every change is made by the script and written down here.

Problems found and what we did

Step Problem found How many What we did Why
1 Column names typed like a spreadsheet: spaces, brackets, capitals (Weight (kg), MAP (mmHg), VFD 28) 29 columns Converted to short lower-case names with clean_names(); renamed 4 (e.g. map_mm_hg to map_mmhg, vfd_28 to vent_free_days_28) Names without spaces are easy to type and the same in every script
2 Duplicate records: the same patient entered twice 3 IDs (FLU-139, FLU-274, FLU-574): 623 rows for 620 patients Checked that the two copies were identical in every column, then kept one copy Each patient must be counted once
3 ICU name typed in different ways (icu a, ICU-A, ICUA, trailing space) 14 rows Standardised to ICU A, ICU B, ICU C, ICU D Otherwise R sees 11 “different” ICUs
4 Admission date in three formats: 2023-03-14, 14/03/2023, 14-Mar-2023 365 / 164 / 94 rows Read each format explicitly and converted all to one date (YYYY-MM-DD); 0 dates failed Mixed formats are read wrongly or not at all; 03/04 could be March or April
5 Sex coded 9 ways (Male, M, m, male, Female, F, f, female, Female) 200 rows not Male/Female Recoded to Female / Male One variable, one coding
6 Yes/No variables mixed Yes, No, Y, N, yes, no, 1, 0 (ICU B sends 1/0) 8 variables; about 1,900 cells not Yes/No Recoded to No / Yes; checked that no value was lost 1 and Yes must mean the same thing
7 All columns read as text all Converted numbers to numbers (whole numbers as integers) Needed for calculations
8 Missing-value codes used as if they were numbers: lactate 999 and -99, APACHE II -99, MAP 999 Lactate 21 (12 x 999, 9 x -99), APACHE II 6, MAP 4 Set to missing (NA) A lactate of 999 would wreck the mean; missing is missing
9 Units: ICU C reported lactate in mg/dL, the other ICUs in mmol/L (ICU C median 23.8 vs about 2.5 elsewhere) 126 values (all ICU C lactates) Divided ICU C values by 9.01 (1 mmol/L = 9.01 mg/dL), 1 decimal; ICU C median became 2.65 One unit for one variable; found by comparing medians by ICU
10 Impossible values (outside the allowed range in the data dictionary) 3: FLU-529 age 210; FLU-277 weight 7 kg; FLU-282 day-2 output -2990 ml Set to missing and raised a query with the site. The patient stays in the study We cannot guess the true value (21? 71?); deleting the patient would bias the sample
10b Range checks on all other variables (SOFA, APACHE II, lactate, MAP, AKI stage, VFD, ICU stay, follow-up time; survivors followed to day 28) 0 problems None Documents that the checks were done
11 Infection source as free text 5 categories Fixed category order (Respiratory, Urinary, Abdominal, Skin/soft tissue, Other) Same order in every table
12 Derived variables needed (not in the raw file) 3 new variables Computed cum_fluid_balance_d3_ml_kg = (fluid in d1-3 - fluid out d1-3) / weight; positive_balance_d3 (> 0 ml/kg); aki_any (KDIGO stage >= 1) Derived values are calculated by the script, never typed by hand

Missing data after cleaning (real missing values, reported in the papers)

Variable Missing Reason
lactate_mmol_l 93 (15.0%) Not measured (more often in less severe patients and in ICU D)
apache_ii 6 (1.0%) Not recorded
map_mmhg 4 (0.6%) Not recorded
age 1 Impossible value set to missing (query pending)
weight_kg 1 Impossible value set to missing (query pending)
fluid_out_d2_ml 1 Impossible value set to missing (query pending)
cum_fluid_balance_d3_ml_kg, positive_balance_d3 2 Follows from the missing weight and output above

Queries sent to sites (simulated)

Query Patient Variable Value in raw file Status
Q-001 FLU-529 (ICU B) Age 210 Open; set to missing
Q-002 FLU-277 (ICU C) Weight 7 kg Open; set to missing
Q-003 FLU-282 (ICU A) Fluid out, day 2 -2990 ml Open; set to missing
Q-004 ICU C Lactate unit mg/dL Confirmed by site lab; converted

Teaching note: in a real study the site would correct the source record and the database, and the cleaning script would be re-run. Here the queries stay open so that the clean file is fixed for the course.