Foundations of Clinical Research · Level 1 – Session 4 · Bias & effect: can we trust it, how big is it?

Week 4 · Saturday · online, 19:30-22:30

Recap quiz on Session 3 (10 min)
4.1 Bias, confounding and causation (40 min)
4.2 Measuring disease and association, with a live 2x2 demo (25 min)
Break (15 min), then two 2x2 calculations together (15 min)
4.3 Beyond single studies: tests, meta-analysis, appraisal (30 min)
Proposal builder 4: exposure, outcomes, biases (40 min), then wrap-up (5 min)

Session 4 opening (19:30). Last Saturday we chose how to answer a question. Today two questions about any result: can we trust it (bias, confounding, causation) and how big is it (risk, risk ratio, odds ratio, risk difference, number needed to treat)? Then we look beyond single studies: diagnostic tests, meta-analysis and appraisal tools.

Open with the exit question from Session 3: show three of the answers you received in the chat or the form (anonymously). The question was about doctors choosing restrictive fluids for some patients: a cohort, so ‘associated’. It leads straight into confounding.

How we work online (repeat briefly): quick answers in the poll, other answers from two or three people called on with the microphone on, ‘raise hand’ to speak, every exercise done by the whole class together.

Pace: the only arithmetic is division in a 2x2 table, and the spreadsheet checks every answer.

Timing: 1 minute. Clock: 19:30 recap quiz (10), 19:40 section 4.1 (40), 20:20 section 4.2 with the demo (25), 20:45 break (15), 21:00 two calculations together (15), 21:15 section 4.3 (30), 21:45 proposal builder 4 (40), 22:25 wrap-up (5), 22:30 end.

Recap quiz: three questions on Session 3

Q1. Patients with sepsis who died are compared with survivors for the fluid they received on days 1-3. Design?
Q2. Which design randomises whole intensive care units (ICUs) instead of patients?
Q3. Restrictive vs usual fluids: mortality difference 95% CI -1 to +3 points. Margin +4 points. Result?
Then re-read your own Proposal Part 2a (design, setting, population): 3 min

How we run it

Answer in the poll
Results on screen, then the answer
One volunteer explains why

OUTPUT Poll answers, and your Part 2a ready to extend

Key words of the day

Today’s vocabulary. Each word comes with a picture or an example

Can we trust it?

Bias, confounding
Effect modification
Reverse causation

How big is it?

Risk ratio (RR), odds ratio (OR)
Risk difference (RD)
Number needed to treat (NNT)

Tests

Sensitivity, specificity
Positive and negative predictive values
ROC curve and the area under it

Many studies

Forest plot, heterogeneity
Risk-of-bias tools

KEY MESSAGE Trust first, then size: a precise answer can still be a biased one.

4.1 Bias, confounding and causation

Bias is systematic error. Size does not fix it.

Selection, information, confounding, effect modification, and from association to causation. (40 minutes)

Random error versus bias

KEY MESSAGE A bigger sample shrinks random error but leaves bias untouched.

Use the dartboard. Left: what we want, close to the truth and tightly grouped. Middle: random error, centred on the truth but scattered. A larger sample pulls the darts together. Right: bias, tightly grouped but in the wrong place. A larger sample just makes us more confident in the wrong answer.

Teaching point: confidence intervals (Session 6) describe random error only. A narrow CI says nothing about bias.

Misconception: ‘a study of 10,000 patients cannot be biased’. Large registry studies can be very precise and still biased.

Ask the class (poll): which of our two papers is more at risk of random error, and which of bias? Expected answer: Paper 1, Boulet 2024 (48 patients) for random error. Paper 2, Hyun 2023 (observational) for bias, especially confounding. Both are in the course folder, Course/Papers.

Timing: 2 minutes.

Selection bias: who got into the study?

KEY MESSAGE If the sample differs from the target population, results may not transfer.

Selection bias happens when the people who enter (or stay in) a study differ systematically from the people we want to learn about.

Example from Paper 2 (Hyun): only patients who stayed at least three days in the ICU with fluid data were included (2,516 of 11,981 registry patients). The sickest patients who died early, and the mildest who left early, were left out. The result applies to ‘patients still in the ICU on day 3’, not to all sepsis patients.

Other common sources: volunteers, patients who refuse consent, loss to follow-up, single referral centres.

Misconception: ‘selection bias only matters in surveys’. It affects every design, including RCTs (strict exclusion criteria limit who the result applies to).

Ask the class (call on two or three people, microphone on): in Paper 1 (Boulet), 1,201 patients were screened and 50 included. What does that tell you? Expected answer: trial patients were highly selected. Check whether they look like your patients before applying the result.

Timing: 2.5 minutes.

Information bias: how things were measured

Recall bias

Cases search their memory harder than controls
e.g. families of patients who died recall the days before admission in more detail
Fix: use records made before the outcome

Observer bias

An assessor who knows the group judges differently
e.g. ‘ready for ICU discharge’ in an unblinded fluid trial
Fix: blinded assessor, objective outcome

Measurement bias

The tool or the record is wrong in a systematic way
e.g. urine output under-recorded on night shifts
Fix: standard definitions, checked charts

KEY MESSAGE Measure exposure and outcome the same way in every group.

Information (measurement) bias: the data are wrong in a systematic way.

Recall bias: in a case-control study, cases (or their families) search their memory harder than controls, so exposures look more common in cases.

Observer bias: an assessor who knows the group may judge a soft outcome differently, for example ‘ready for discharge’ or ‘delirium present’. In Boulet clinicians knew the allocation.

Measurement bias: a scale that always reads 0.5 kg high, or urine output that is under-recorded on night shifts. For our fluid question this matters directly: missing urine output makes the fluid balance look more positive than it is. If it differs between groups, it distorts the comparison.

Misconception: ‘hard outcomes like death are immune’. Death is reliably recorded, but the timing and cause of death can still be misclassified.

Ask the class (call on two or three people, microphone on): how would you reduce observer bias in an unblinded ICU fluid trial? Expected answer: an objective outcome, a blinded outcome assessor, standard definitions and training.

Timing: 2.5 minutes.

Confounding by indication: the ICU fluid example

SOFA: Sequential Organ Failure Assessment score

KEY MESSAGE A confounder is linked to the exposure and causes the outcome.

This is the most important slide of 4.1. Point to each arrow with the cursor.

Severity of illness pushes clinicians to give more fluid (left arrow). Severity also causes death (right arrow). So patients who got more fluid die more often, even if the fluid itself did nothing.

This special case is called confounding by indication: the reason for giving a treatment is itself linked to the outcome.

Three conditions for a confounder: associated with the exposure, a cause of the outcome, and not on the causal pathway between them.

Misconception: ‘a confounder is any variable that differs between groups’. It must also affect the outcome.

Ask the class (call on two or three people, microphone on): which bedside signs make you give more fluid and predict death? Expected answer: low blood pressure, high lactate, high vasopressor dose, mottled skin. Each one is a candidate confounder.

Timing: 3.5 minutes.

Confounding in numbers: the association disappears

Hypothetical data for teaching. RR: risk ratio.

KEY MESSAGE Compare like with like: within each severity level, fluid makes no difference.

Hypothetical numbers, chosen to make the point. All patients together: 42% of high-fluid patients die versus 18% of low-fluid patients, a crude risk ratio (RR) of 2.33. It looks as if fluid kills.

Now split by severity. Mild illness: 10% versus 10%. Severe illness: 50% versus 50%. Within each level the RR is 1.0.

The reason: 80 of the 100 high-fluid patients were severely ill, but only 20 of the 100 low-fluid patients. Severity, not fluid, explained the crude difference.

Teaching point: this is what ‘adjusting for severity’ does: it compares like with like. Session 6 shows how papers and models report it. Next slide: what it means when the strata do NOT agree.

Misconception: ‘after adjustment the remaining association is causal’. Only if every important confounder was measured well, which is never guaranteed in observational data.

Ask the class (call on two or three people, microphone on): what would you need to measure at baseline in an observational fluid study to do this? Expected answer: the main severity markers: SOFA score, lactate, vasopressor dose, age, comorbidity.

Timing: 3.5 minutes. Check the arithmetic aloud: 2 + 40 = 42 deaths of 100, 8 + 10 = 18 of 100.

Confounding or effect modification?

Hypothetical data for teaching. RR: risk ratio. ARDS: acute respiratory distress syndrome.

KEY MESSAGE Confounding is a bias to remove. Effect modification is a finding to report.

Both appear when we split the data into strata, but they mean opposite things.

On the left, confounding (our fluid example): the strata agree with each other (RR 1.0 and 1.0) but differ from the crude result (2.33). The third factor (severity) created a false association. Fix: adjust, and report the adjusted estimate.

On the right, effect modification, also called interaction (hypothetical): a restrictive strategy might help patients with ARDS (RR 0.60) but make no difference without ARDS (RR 1.00). The strata differ from each other. This is real biology, not bias: report each subgroup. Do not average it away.

Rule of thumb: strata similar to each other but different from the crude = confounding. Strata different from each other = effect modification.

Misconception: ‘every subgroup difference is real’. Subgroups must be pre-specified and few. With many subgroups, some differ by chance. Session 6 shows interaction in a model.

Ask the class (poll): restrictive fluids, RR 0.95 in patients with normal kidneys and 0.60 in patients with kidney injury. Confounding or effect modification? Expected answer: effect modification (the strata differ from each other).

Timing: 3 minutes.

How designs fight bias

Build the protection into the design. Repair what is left in the analysis

Randomise and conceal

Chance decides the group
Balances known and unknown confounders
Nobody can predict the next allocation

Blind

Patients, clinicians, assessors, analysts
Protects how outcomes are measured
If impossible: blind the assessor

Restrict, match, adjust

Restrict: narrow eligibility
Match: pair similar patients
Adjust: statistics in the analysis
Measured confounders only

KEY MESSAGE Only randomisation protects against confounders you never measured.

Randomisation balances what we know and what we do not

R: randomisation

KEY MESSAGE By chance, both groups get a similar mix, even of factors nobody measured.

The red figures are severely ill patients (a known confounder). The gold stars are a hidden factor nobody measured: a genetic trait, frailty, a clinician’s gut feeling.

Randomisation sends patients to group A or B by chance, so on average both groups receive the same mix of severe patients and of stars. Here each group gets 5 severe and 3 starred patients.

Teaching point: balance is on average. With very small trials chance imbalances happen. Boulet found baseline creatinine higher in the intervention group. With 24 per arm that is not surprising.

Misconception: ‘randomised means the groups are identical’. They are similar on average. Table 1 of the paper lets you check.

Ask the class (call on two or three people, microphone on): why does this work better with 500 patients than with 50? Expected answer: with larger numbers, chance imbalances average out.

Timing: 1.5 minutes.

Concealment before, blinding after

Before: allocation concealment

Nobody can predict the next allocation
Paper 1 (Boulet): a central computer
Protects who gets into each group
Always possible, always required

After: blinding in Paper 1 (Boulet 2024)

Patients: blinded
Clinicians: knew the group
Outcome assessors: knew the group
Statisticians: blinded

KEY MESSAGE Concealment protects who gets in. Blinding protects how outcomes are measured.

Allocation concealment happens before randomisation: the person recruiting cannot know or predict the next allocation (central computer, not envelopes on the desk). Without it, a clinician could steer sicker patients away from the new treatment: selection bias inside a trial.

Blinding happens after: who knows the group? In Boulet the patients and the statisticians were blinded, the clinicians and investigators were not, hence ‘single-blind’.

Teaching point: in fluid and many ICU trials, blinding the clinician is often impossible. Then blind the outcome assessor and choose an objective outcome.

Misconception: ‘single, double, triple blind’ have fixed meanings. They do not. Papers should state exactly who was blinded.

Ask the class (poll): is concealment possible when blinding is not? Expected answer: yes. Concealment is always possible and always required.

Timing: 2 minutes.

Matching helps, but only for what was measured

Paper 2 (Hyun 2023). SOFA: Sequential Organ Failure Assessment score

1:1 matching

Each positive-balance patient paired with a similar negative-balance patient
483, 373 and 392 pairs for days 1, 2 and 3

Measured: balanced

Age, sex, SOFA score, septic shock
Comorbidities, infection site
Fluid output

Not measured: may still differ

Type of fluid, nutrition
The clinician’s bedside judgement
This is residual confounding

KEY MESSAGE Matching and adjustment cannot balance what nobody recorded.

Hyun matched each positive-balance patient to a negative-balance patient with a similar propensity score, built from measured factors: age, sex, SOFA score, septic shock, comorbidities, infection site, fluid output and others. This gave 483, 373 and 392 matched pairs for the day-1, day-2 and day-3 analyses.

The authors themselves list what was not measured: type of fluid and nutritional support. The clinician’s bedside judgement of how sick a patient looks is never fully captured. Whatever is left is called residual confounding.

Teaching point: this is why a cohort, however well matched, says ‘associated with’, and why the question ‘does restriction save lives?’ still needs a large RCT.

Misconception: ‘matched groups are as good as randomised groups’. Check Table 1, then ask what is not in Table 1.

Ask the class (call on two or three people, microphone on): name one unmeasured factor that might push clinicians to give more fluid and predict death. Expected answer: clinical impression of shock, mottled skin, a failing heart, a goals-of-care discussion.

Timing: 2 minutes.

Internal validity first, then external validity

Internal validity

Is the answer right for the people studied?
Randomisation
Concealment and blinding
Complete follow-up
Good measurement

External validity

Does it apply to my patients?
Boulet: age 18-85. My ICU: many over 85?
Boulet: no dialysis. My ICU: patients on dialysis?
Boulet: 3 ICUs in one country
Boulet: only 4% of screened patients enrolled

KEY MESSAGE First: is it right for the people studied? Then: does it apply to mine?

Internal validity: is the answer correct for the people who were in the study? This is what randomisation, concealment, blinding, complete follow-up and good measurement protect.

External validity (generalisability): does the answer apply to my patients in my setting? Compare the trial population with yours. Paper 1 (Boulet) included adults aged 18-85 without dialysis or severe malnutrition, in 3 ICUs of one country, and only about 4% of screened patients.

Teaching point: internal validity comes first. A biased result generalises nowhere. But a perfectly valid trial in highly selected patients may not help your unit.

Misconception: ‘an RCT applies to everyone’. It applies most clearly to patients like those enrolled.

Ask the class (call on two or three people, microphone on): name one patient group in your ICU who would not have entered Boulet’s trial. Expected answer: patients over 85, patients on chronic dialysis, those admitted more than 24 h after starting vasopressors.

Timing: 2 minutes.

Reverse causation: which came first?

What we assume

A positive fluid balance harms the kidneys
Kidney injury then leads to death

What may really happen

Failing kidneys make less urine
Less urine out makes the balance positive
The arrow runs the other way

How to check

Which came first?
Measure the exposure before the outcome starts
Use a time lag
Prefer a cohort or an RCT to a cross-sectional study

KEY MESSAGE Before claiming X causes Y, make sure Y did not cause X.

Reverse causation: the ‘outcome’ actually causes the ‘exposure’.

ICU example: we assume a positive fluid balance causes kidney injury and death. But failing kidneys produce less urine, and less urine out means a more positive balance. The arrow may run the other way. This is why Hyun matched patients on fluid output as well as input.

Other examples: people with early, undiagnosed illness stop exercising (so inactivity looks harmful). Patients who are deteriorating get more tests and more drugs.

How to check: make sure the exposure was measured before the outcome began, use a time lag, prefer cohort or trial designs over cross-sectional ones.

Misconception: ‘reverse causation only matters in cross-sectional studies’. It can affect cohorts too when the exposure is measured while the outcome is already developing.

Ask the class (call on two or three people, microphone on): patients in shock receive more vasopressors and die more often. Which way could the arrow run? Expected answer: both ways. The drugs might cause harm, but deepening shock is the reason they are increased.

Timing: 2.5 minutes.

From association to causation: the Bradford Hill viewpoints

Viewpoint	What it asks	Applied to Paper 2 (Hyun 2023)
Strength	Is the effect big?	Moderate (HR about 1.6 to 2.1)
Consistency	Seen in other studies and places?	Yes, in other cohorts
Specificity	One cause, one effect? (the weakest)	Not useful here
Temporality	Did the cause come first? (the only must)	Balance before death, but reverse causation is possible
Dose-response	More exposure, more effect?	Not shown
Plausibility	Is there a biological mechanism?	Yes: fluid overload, oedema, organ congestion
Coherence	Does it fit what we already know?	Mostly
Experiment	Does removing the exposure reduce the outcome?	Trials such as Boulet: no effect on deaths shown yet
Analogy	Do similar causes have similar effects?	Fluid overload harms in other settings

HR: hazard ratio. Causation is a judgement built from many clues. Temporality is the only must.

In 1965 Austin Bradford Hill listed nine viewpoints to judge whether an association is likely to be causal. They are not a checklist to score. They are clues.

Go down the table quickly. Stop at temporality (the only essential one) and experiment (the strongest, given by an RCT).

Apply to Hyun (right-hand column): strength moderate (hazard ratio, HR, about 1.6-2.1). Consistency with other cohorts yes. Temporality: balance measured before death, but reverse causation is possible. Dose-response not shown. Experiment: trials such as Boulet have not shown a mortality effect. Conclusion: possible, not proven.

Misconception: ‘if most criteria are met, causation is proven’. They raise or lower our confidence. They never prove.

Ask the class (poll): which viewpoint does the RCT provide that no cohort can? Expected answer: experiment.

Timing: 4.5 minutes.

Quick quiz: Spot the bias

Families of patients who died recall the days before ICU admission in more detail than families of survivors
In an unblinded fluid trial, the doctor who knows the group decides when a patient is ‘ready for ICU discharge’
A fluid study includes only patients who stayed in the ICU for 3 days or more
Patients with a higher lactate get more fluid and die more often

Vote for one

Selection bias
Recall bias
Observer bias
Confounding

OUTPUT One bias named for each scenario, and one fix said aloud

Run it as a four-question poll (one per scenario). After each result, call on one or two people to suggest a fix, microphone on. The same four scenarios are in the Session 4 exercise sheet (PDF), Part A.

Answers. 1: recall bias. Fix: use exposure data recorded before the outcome was known (charts, prescription records). Ask both groups the same structured questions. 2: observer bias. Fix: blinded assessor, objective discharge criteria. 3: selection bias (the sickest, who died early, are missing). Fix: include all eligible patients from admission and report how many were excluded and why. 4: confounding by indication. Fix: randomise, or measure lactate and other severity markers and adjust or match.

Timing: 5 minutes. Keep it fast: this is a check, not a lecture. This closes the 40 minutes of 4.1.

4.2 Measuring disease and association

How common? How much more likely?

Prevalence, incidence, risk, and the 2x2 table behind RR, OR, RD and NNT. (25 minutes)

Prevalence and incidence: the bathtub

KEY MESSAGE Prevalence = how many have it now. Incidence = how many new cases over time.

Water flowing in from the tap = new cases = incidence. Water in the tub at this moment = existing cases = prevalence. Water leaving through the drain = patients who recover or die.

Worked example on the right (another example, from a ward): 20 of 200 ward patients have a pressure injury today, prevalence 10%. Of the 180 without one, 9 develop a new injury over the next week, incidence 5% per week. Note the denominator for incidence: only people at risk (those without the condition at the start).

Teaching point: a long-lasting condition can have a high prevalence with a low incidence (diabetes). A short one can have a high incidence but low prevalence (common cold, short delirium episodes).

Misconception: using prevalence and incidence interchangeably. Incidence always has a time period attached.

Ask the class (call on two or three people, microphone on): in our ICU, ‘how many patients have a positive fluid balance today’ is prevalence or incidence? And ‘how many develop kidney injury in their first week’? Expected answer: prevalence, incidence.

Timing: 2.5 minutes.

Risk in two groups, as pictures

Hypothetical trial for teaching: 100 patients per group

KEY MESSAGE Risk = people with the outcome / people in the group.

The 2x2 table: one tool for every measure

Hypothetical trial	Died	Survived	Total	Risk
Restrictive	a = 20	b = 80	a + b = 100	a / (a + b) = 20%
Usual care	c = 30	d = 70	c + d = 100	c / (c + d) = 30%

Always the same layout: exposure in rows, with the exposed (treated) group on top. Outcome in columns, with ‘outcome yes’ (died) on the left. Then a, b, c and d always mean the same thing.

Four measures from one 2x2 table

Worked example: restrictive 20/100 died, usual care 30/100 died

Risk ratio (RR)

[a/(a+b)] / [c/(c+d)]
20% / 30% = 0.67
Risk is one third lower

Risk difference (RD)

a/(a+b) - c/(c+d)
20% - 30% = -10 points
10 fewer deaths per 100

Number needed to treat (NNT)

1 / |risk difference|
1 / 0.10 = 10
Treat 10 to prevent 1 death

Odds ratio (OR)

(a x d) / (b x c)
(20 x 70) / (80 x 30) = 0.58
Odds, not risk

KEY MESSAGE Ratios tell you how strong. Differences tell you how many patients.

Go card by card, always back to the 2x2 table.

RR = 0.20 / 0.30 = 0.67: the risk of death is one third lower with restriction. RR = 1 means no difference. Below 1 protective, above 1 harmful.

RD = 0.20 - 0.30 = -0.10: 10 fewer deaths per 100 patients treated. Also called absolute risk reduction (ARR = 10%).

NNT = 1 / 0.10 = 10: treat 10 patients to prevent one death. If the exposure is harmful the same calculation gives the number needed to harm (NNH).

OR = (20 x 70) / (80 x 30) = 1400 / 2400 = 0.58. Further from 1 than the RR. The next slides explain why.

Misconception: a relative effect sounds bigger. ‘One third lower’ and ‘10 fewer per 100’ are the same result. Patients and policymakers need the absolute version.

Ask the class (call on two or three people, microphone on): if the baseline risk were 3% instead of 30%, with the same RR of 0.67, what would the NNT be? Expected answer: risk 2% vs 3%, difference 1%, NNT = 100.

Timing: 5 minutes.

NNT (number needed to treat) in pictures

KEY MESSAGE NNT = 1 / risk difference: how many to treat for one to benefit.

When is the odds ratio close to the risk ratio?

KEY MESSAGE OR is close to RR only when the outcome is rare (under about 10%).

Here the exposure truly doubles the risk (RR = 2, green line). The red line is the OR for the same data as the outcome becomes more common.

When the outcome is rare (left, shaded), OR is about 2.1, practically the same as RR. At a baseline risk of 20%, OR = 2.7. At 40%, OR = 6. The OR always lies further from 1 than the RR.

Teaching point: case-control studies and logistic regression give ORs. When a paper reports an OR for a common outcome, such as ICU mortality of 20-40%, do not read it as ‘X times the risk’.

Misconception: ‘OR 2 means twice the risk’. Only for rare outcomes.

Ask the class (call on two or three people, microphone on): in our worked example (deaths of 20% and 30%), OR was 0.58 and RR 0.67. Why do they differ? Expected answer: death is common in this population, so the OR exaggerates the effect.

Timing: 1.5 minutes.

Live demo: the 2x2 calculator

The instructor shares the screen: Course/Resources/tools/2x2_calculator.xlsx, sheet Association
Enter a, b, c, d from the worked example: 20, 80, 30, 70
Read the risks, RR, RD, NNT and OR, and the plain-language lines
Change the deaths to 2 and 3 of 100: watch the OR approach the RR
Point at the red line: case-control data give an OR only

Watch for

Same layout as the slide
Relative vs absolute effect
Rare outcome: OR close to RR

OUTPUT Everyone has seen every measure computed from one table

Script: Course/Demos/s4_demo.md. File: Course/Resources/tools/2x2_calculator.xlsx (pre-filled with the worked example). Participants only watch. They can download the file from the course folder afterwards.

Teaching point: the spreadsheet does the arithmetic. The human does the interpretation. Read the plain-language cells aloud.

Key moment: change a to 2 and c to 3 (b = 98, d = 97). RR stays 0.67 but OR is now 0.66, almost identical. The NNT jumps to 100. Same relative effect, very different absolute effect.

Fallback if screen sharing fails: type the 2x2 table into the chat or a shared document and calculate RR and NNT by hand with the class. The numbers are chosen to be easy.

Misconception to raise: the calculator will happily give a ‘risk’ for case-control data. It is meaningless there: only the OR is valid.

Timing: 8 minutes. The Diagnostic sheet is shown briefly in 4.3. Then the break. This closes the 25 minutes of 4.2.

Break

15 min

Exercise: two 2x2 calculations, together

Session 4 exercise sheet (PDF sent before the session), Part C: draw each 2x2 table first, exposure in rows
Calculation 1 (a fluid trial, 200 per group): risks, RR, RD, NNT, OR. Answer each in the poll
Calculation 2 (a fluid case-control study): OR only. Poll: can we calculate an RR?
The instructor checks each answer live with the 2x2 calculator
A volunteer says each result in one plain sentence

Remember

Risk = a / (a + b)
RR = risk / risk
NNT = 1 / risk difference
OR = (a x d) / (b x c)

OUTPUT Both calculations solved together, each with one plain sentence

Whole-class exercise. Sheet: Course/Practicals/s4_exercises.md, Part C (PDF). Key: Course/Solutions/s4_solutions.md, Part C.

Calculation 1 (hypothetical trial of restrictive vs usual fluids, 200 patients per group): 20 vs 30 deaths by day 28. Poll each step: risks 10% and 15%, RR = 0.67. RD = -5 percentage points. NNT = 20. OR = (20 x 170) / (180 x 30) = 0.63. Teaching point: the same RR (0.67) as the worked example, but the baseline risk is lower (15% instead of 30%), so the NNT doubles from 10 to 20.

Calculation 2 (hypothetical case-control study): 40 patients with sepsis who needed new dialysis (cases) and 80 who did not (controls), same ICUs. Positive fluid balance on day 3 in 24 of 40 cases and 16 of 80 controls. OR = (24 x 64) / (16 x 16) = 6.0. RR cannot be calculated because the investigators chose the number of cases and controls. Then ask (call on two or three people, microphone on): could the arrow run the other way here? Yes: failing kidneys make less urine, so the balance becomes positive (reverse causation, slide 16).

Rhythm: 7 minutes per calculation (poll each answer, type the table into the calculator on the shared screen), 1 minute to close. Most mistakes are layout, not arithmetic: check that everyone drew exposure in rows.

If the class is quick: the sheet has one more calculation on another topic (Part C, extra practice) for self-study.

4.3 Beyond single studies

One study is rarely the answer.

Diagnostic tests, systematic reviews and meta-analysis, and how to appraise them. (30 minutes)

Diagnostic accuracy: sensitivity and specificity revisited

KEY MESSAGE Sensitivity catches the sick. Specificity clears the healthy.

Back to the 2x2 table, now for a test. 100 patients, 20 truly have the disease.

Sensitivity: of the 20 who have the disease, the test finds 18 = 90%. A highly sensitive test rarely misses disease, so a negative result helps rule it out.

Specificity: of the 80 who do not have it, 64 test negative = 80%. A highly specific test rarely raises false alarms, so a positive result helps rule it in.

Preview of the next slide: of the 34 positive tests, only 18 truly have the disease, a positive predictive value of 53%. It depends on how common the disease is.

Misconception: ‘a 90% sensitive test means a positive patient has a 90% chance of disease’. No. That is the positive predictive value, which here is only 53%.

Ask the class (poll): for a screening test that must not miss sepsis, which comes first: sensitivity or specificity? Expected answer: sensitivity.

Timing: 2.5 minutes.

Same test, different setting: PPV depends on prevalence

PPV, NPV: positive and negative predictive values, the chance that a positive or a negative result is right

KEY MESSAGE A positive result means much less when the disease is rare.

The same test (sensitivity 90%, specificity 80%) used on 500 people in two settings.

ICU, where 20% have the disease: 90 true positives, 80 false positives, so a positive result is right 90 / 170 = 53% of the time (positive predictive value, PPV).

Screening, where 2% have it: 9 true positives but 98 false positives: PPV 9 / 107 = 8%. More than 9 in 10 positives are false alarms. Meanwhile the negative predictive value (NPV) rises to 99.7%.

Teaching point: sensitivity and specificity belong to the test. PPV and NPV belong to the test in A population. Never borrow a PPV from a study done in a different setting.

Misconception: ‘a good test is good everywhere’. Screening healthy populations with a moderately specific test creates many false positives and many unnecessary investigations.

Ask the class (call on two or three people, microphone on): why do screening programmes use a second, more specific test before acting? Expected answer: to remove the false positives created by low prevalence.

Optional (1 minute): on the shared screen, show the ‘PPV at another prevalence’ box on the Diagnostic sheet of the 2x2 calculator, then type 2% and 20% (Course/Demos/s4_demo.md, Part 2).

Timing: 3 minutes.

The ROC curve and the AUC

ROC: receiver operating characteristic. AUC: area under the curve. Simulated test.

KEY MESSAGE Each cut-off trades sensitivity for specificity. AUC sums up the whole curve.

Many tests give a number (lactate, procalcitonin, a risk score). Every cut-off gives one pair of sensitivity and specificity. The ROC curve plots all of them: sensitivity up, false-positive rate (1 - specificity) across.

A low cut-off catches almost every case but raises many false alarms (top right). A high cut-off gives few false alarms but misses cases (bottom left). The best cut-off depends on the cost of missing a case versus the cost of a false alarm.

The area under the curve (AUC, also called the c-statistic) summarises discrimination: 0.5 = coin toss (the dashed line), 1.0 = perfect. This simulated test has AUC 0.80.

Misconception: ‘AUC tells you which cut-off to use’. It does not. It describes the whole curve. The same AUC appears again in Session 6 for prediction models.

Ask the class (poll): for ruling out sepsis at triage, would you choose a low or high lactate cut-off? Expected answer: low, for high sensitivity, accepting more false alarms.

Timing: 2.5 minutes.

The PRISMA flow: from 1,300 records to 9 studies

PRISMA: Preferred Reporting Items for Systematic reviews and Meta-Analyses. RCT: randomised controlled trial.

KEY MESSAGE A PRISMA flow shows how studies were found, screened and excluded, with reasons.

Illustrative numbers. Every systematic review should show this diagram (PRISMA 2020).

Identification: 1,240 database records plus 60 from trial registers = 1,300, 300 duplicates removed.

Screening: 1,000 titles and abstracts screened, 920 excluded, 80 full texts read, 71 excluded with reasons (38 not RCTs, 21 wrong population, 12 no mortality data, which add up to 71).

Included: 9 studies, all in the meta-analysis.

Teaching point: two reviewers screen independently, and the protocol is registered beforehand (PROSPERO, the international register of systematic review protocols). The diagram lets a reader judge whether relevant studies were lost.

Misconception: ‘more included studies = a better review’. Strict, explicit inclusion criteria matter more than numbers.

Ask the class (call on two or three people, microphone on): why search trial registers and not only PubMed? Expected answer: to find unpublished or negative trials, because of publication bias (Session 2, grey literature).

Timing: 2.5 minutes.

Reading a forest plot in four steps

RR: risk ratio, CI: confidence interval

KEY MESSAGE Squares = studies, lines = CIs, diamond = pooled result, vertical line = no effect.

Hypothetical trials, fixed-effect (inverse-variance) pooling. Read it in four steps. (1) Find the vertical line of no effect: RR = 1. Left of it favours restrictive fluids. Right favours usual care. (2) Each square is one study’s RR. Its size shows its weight (Trial 4 is largest, 38%). (3) Each horizontal line is that study’s 95% CI. If it crosses 1, that study alone is inconclusive. Here every single trial crosses 1. (4) The diamond is the pooled estimate: RR 0.79 (95% CI 0.64-0.98). Its width is the CI. It does not cross 1.

Teaching point: pooling several inconclusive trials can give a precise answer. That is the strength of meta-analysis. In words: about 21% lower relative risk of death, though the upper limit (0.98) is close to no effect.

Misconception: ‘a trial whose CI crosses 1 shows no effect’. It shows uncertainty, not absence of an effect.

Do not ask questions yet: the whole-class task at the end of 4.3 uses this same plot.

Timing: 3.5 minutes.

Heterogeneity: do the studies agree?

I-squared (i²): the share of the variation between studies that is beyond chance

KEY MESSAGE High I-squared means the studies disagree: find out why before trusting the diamond.

Left: the six trials from the previous slide point the same way. Their CIs overlap, I-squared = 0%. Pooling is sensible.

Right: six other hypothetical trials disagree. Some clearly favour restriction, others point the other way. I-squared = 80%: about 80% of the variation between studies reflects real differences rather than chance. The random-effects diamond is wide and crosses 1 (RR 0.74, 95% CI 0.47-1.15).

Rough guide (Cochrane): I-squared around 0-40% might not be important, 30-60% moderate, 50-90% substantial, 75-100% considerable. The ranges overlap on purpose: judge it with the plot.

Teaching point: with high heterogeneity the most useful question is why: different patients, doses, timing, outcome definitions or risk of bias? A single pooled number may hide this.

Misconception: ‘a random-effects model fixes heterogeneity’. It only widens the CI. It does not explain anything.

Ask the class (call on two or three people, microphone on): give one reason fluid-restriction trials might disagree. Expected answer: different populations (sepsis vs surgery), different restriction protocols, different timing.

Timing: 3 minutes.

Which risk-of-bias tool for which design?

If the study is a…	Appraise it with	Report it with	Course example
Randomised trial	RoB 2	CONSORT	Paper 1 (Boulet)
Non-randomised study of an intervention	ROBINS-I	TREND or STROBE
Cohort or case-control study of an exposure	Newcastle-Ottawa Scale (or ROBINS-E)	STROBE	Paper 2 (Hyun)
Diagnostic accuracy study	QUADAS-2	STARD
Systematic review	AMSTAR 2	PRISMA

RoB 2: Cochrane risk-of-bias tool for randomised trials. ROBINS-I / -E: risk of bias in non-randomised studies of interventions / exposures. QUADAS-2: for diagnostic accuracy studies. AMSTAR 2: for systematic reviews. The reporting guidelines are covered in Session 8.

Risk-of-bias tools are structured ways to ask ‘could the design or conduct have distorted this result?’. Each design has its own tool. A one-page version of this table is in the course folder: Course/Resources/Risk_of_Bias_Tool_Chooser.pdf.

Randomised trials: RoB 2 (Cochrane), used for Paper 1 (Boulet). Non-randomised studies of interventions (before-after, cohorts comparing treatments): ROBINS-I. Cohort and case-control studies of exposures: the Newcastle-Ottawa Scale is still widely used. ROBINS-E is its newer, more rigorous alternative (used for Paper 2, Hyun). Diagnostic accuracy studies: QUADAS-2. Systematic reviews themselves: AMSTAR 2.

The third column links forward: each design also has a reporting guideline (CONSORT for trials, TREND or STROBE, STROBE for observational studies, STARD for diagnostic accuracy, PRISMA for reviews), covered in Session 8.

Teaching point: reporting guidelines check what was reported. Risk-of-bias tools judge how trustworthy the result is. A well-reported study can still be at high risk of bias.

Misconception: ‘add up the stars and you get quality’. Scores hide which domain failed. Judgements by domain are more useful.

Ask the class (poll): you are appraising a before-after quality-improvement study of a sepsis bundle. Which tool? Expected answer: ROBINS-I.

Timing: 2.5 minutes.

The last question: are the conclusions supported by the data?

Questions 7-9 of the 10-question appraisal checklist used in Session 8 (Course/Resources)

Words match the design

‘Caused’ needs randomisation
Observational: ‘associated with’
Pilot: feasibility, not effect

Numbers match the words

Effect size with its 95% CI
Relative and absolute effect
No ‘trend’ for p = 0.42

Honest limits, no spin

Primary outcome reported first
Main limitations stated
Conclusion based on the primary result

KEY MESSAGE Read the conclusion last, and check it against the design and the numbers.

This closes the appraisal block and links to the 10-question checklist used in Session 8 (‘Be the reviewer’): Course/Resources/Critical_Appraisal_Checklist.pdf.

Words vs design: Boulet’s conclusion (‘did not reduce the overall fluid balance at day 5’) matches its pilot RCT design. Hyun’s abstract says ‘associated’, but its title says ‘Impact’, a stronger word than a cohort supports.

Numbers vs words: is the main result given as an effect size with a 95% CI? Is a non-significant result (p = 0.42) described honestly, without ‘a trend towards’?

Spin: is the primary outcome reported first and the conclusion based on it, rather than on a favourable secondary outcome or subgroup?

Ask the class (call on two or three people, microphone on): find one word in a title or abstract that overclaims. Expected answer: ‘impact’, ‘effect’, ‘reduces’ or ‘prevents’ in an observational study.

Timing: 2 minutes.

Exercise: read the forest plot, pick the tool

Session 4 exercise sheet (PDF), Part E: the forest plot of six fluid trials, also on the shared screen
Poll: which trial carries the most weight?
Poll: does any single trial show a clear effect?
Poll: does the diamond cross RR = 1? Say the result in one sentence
Poll: the risk-of-bias tool for Paper 1 (Boulet), Paper 2 (Hyun) and a review of fluid trials

Reminder

Square = one study
Line = 95% confidence interval
Diamond = pooled result
Tools: see the table two slides back

OUTPUT One class sentence on the pooled result, and three tools named

Whole-class exercise. Sheet: Course/Practicals/s4_exercises.md, Part E (PDF). Key: Course/Solutions/s4_solutions.md, Part E. Go back to the forest-plot slide on the shared screen while the class answers.

Answers: Trial 4 has the most weight (38%). No single trial excludes RR = 1: all CIs cross it. The diamond (RR 0.79, 95% CI 0.64-0.98) does not cross 1: pooled, restriction was associated with about 21% lower relative risk of death in these (hypothetical) trials, with the upper limit close to no effect. Call on two or three people, then type the best sentence on the shared screen. Tools: RoB 2 for Boulet. Newcastle-Ottawa (or ROBINS-E) for Hyun. AMSTAR 2 for a review.

If time is short, do the diamond sentence and the tools only.

Common error: reading the diamond’s centre without its width. Always say the CI.

Timing: 8 minutes. This closes the 30 minutes of 4.3. Then move to the proposal builder.

Outcomes: one primary, defined by what, how and when

The running example (observational version)

What

Death from any cause
Not ‘mortality improved’

How

Hospital record
Phone call if discharged

When

Day 28 after ICU admission
The same day for every patient

2-3 secondary outcomes

Acute kidney injury by day 7
Ventilator-free days to day 28
ICU length of stay

KEY MESSAGE Decide the primary outcome before you see the data, and report it first.

Proposal builder 4 starts here (40 minutes in all, the figure shown on the builder banner and in the agenda: this slide 2 minutes, then 38 minutes of builder work).

The primary outcome is the one result the study is designed and sized for. Define it three ways: what (death from any cause, not ‘mortality improved’), how (hospital record, a phone call if discharged), when (day 28 after ICU admission). Secondary outcomes get the same three parts (e.g. acute kidney injury by day 7 using the standard KDIGO criteria, Kidney Disease: Improving Global Outcomes).

Paper 1’s primary outcome (Boulet): cumulative fluid balance (ml/kg) at day 5: clear what, how and when. Paper 2’s (Hyun): mortality at day 28.

Misconception: ‘more primary outcomes = more chances of a result’. That is exactly the problem: multiple primaries invite cherry-picking (spin, Session 2).

Ask the class (call on two or three people, microphone on): rewrite ‘improve patient outcomes’ as a primary outcome for the fluid question. Expected: e.g. ‘new dialysis by day 28 after ICU admission, from the ICU chart’.

Timing: 2 minutes.

Proposal builder 4: exposure, outcomes and biases

Watch: the instructor writes Part 2b live for the running example (15 min)
Exposure or intervention, and comparator: exactly what, how much, when
One primary outcome (what, how, when) and 2-3 secondary outcomes
Two main biases, a concrete fix for each, and ‘caused’ or ‘associated’
Your turn: draft the same for your own question (16 min, quiet individual work)
Three volunteers read one bias and its fix aloud for feedback (7 min)

Check your draft

Comparator stated?
Only one primary outcome?
Outcome has a time point?
Confounding by indication?
A concrete fix for each bias?

OUTPUT Your own Proposal Part 2 complete (2a from Session 3, 2b today). Finish it as homework

Worksheet: Session 4 exercise sheet (PDF), Part F, and the proposal template (Course/Templates/Research_Proposal_Template.docx, Part 2). Everyone writes their own proposal. There are no groups.

Rhythm (38 minutes): 15 minutes in which the instructor fills in Part 2b for the running example on the shared screen, thinking aloud, 16 minutes in which participants draft their own Part 2b quietly (cameras optional, questions in the chat), 7 minutes in which three volunteers read one bias and its fix aloud and the instructor comments.

Model answer for the running example (observational version), to write live: exposure: positive cumulative fluid balance (ml/kg) at 72 h after ICU admission. Comparator: zero or negative balance. Primary outcome: death from any cause by day 28 (hospital record). Secondary: acute kidney injury (KDIGO) by day 7, ventilator-free days to day 28, ICU length of stay. Biases: confounding by severity (record SOFA score, lactate, vasopressor dose and adjust or match), selection (consecutive patients, report exclusions), information (pre-defined fluid definitions, double-check 10% of charts). Claim: ‘associated with’. Full model in Course/Solutions/s4_solutions.md, Part F.

Common problems: no comparator, three primary outcomes. A bias named without a concrete mitigation (‘we will be careful’ is not a mitigation).

Timing: warn at 10 minutes and 2 minutes before the end of the individual slot.

Session 4 in one slide

4.1 Trust

Bias: size does not fix it
Confounding: remove it
Effect modification: report it

4.2 Size

Relative: RR, OR
Absolute: RD, NNT
OR close to RR only if rare

4.3 Many studies

PPV depends on prevalence
Forest plot: read the diamond
Right tool for the design

Your proposal

Part 2 complete
Caused or associated?

KEY MESSAGE Trust first, then size, and check that the words match the design.

Wrap-up and homework

Exit question (chat or 1-question form): ICU patients who received more fluid died more often. Give one reason other than the fluid
Why: it checks today’s key idea. The instructor reads every answer and opens Session 5 with them
Homework: finish your own Proposal Part 2 (exposure, comparator, outcomes, biases)
Optional: exercise sheet Parts B and D (effect modification, PPV in two settings)
Next Saturday (Session 5): data, and the course dataset FLUID-ICU

OUTPUT One exit answer per person, in the chat or the form

Extra slides: Session 4

Use if time allows, or for self-study

More practice (other examples): which bias?

A. Only patients who survived to hospital discharge are interviewed about their ICU care
B. The nurse who knows each patient’s group scores their pain
C. A scale that always reads 0.5 kg too high is used on one ward only

Choose

Selection bias
Observer bias
Measurement bias

OUTPUT One bias for each (answers in the notes)

Other examples: find the confounder

In each, a third factor drives both the exposure and the outcome

Antibiotics and death

More antibiotics, more deaths?
Confounder: severity of infection
Sicker patients get more drugs

Dietitian and weight loss

Seen by a dietitian, more weight loss?
Confounder: malnutrition at admission
The reason for referral

Vasopressors and arrhythmia

Vasopressors, more arrhythmia?
Confounder: depth of shock
Partly a drug effect too

KEY MESSAGE Ask: why did these patients receive the exposure? That reason may be the confounder.

Likelihood ratios (LR) and reporting diagnostic studies

Same test: sensitivity 90%, specificity 80%

LR+ (positive test)

sensitivity / (1 - specificity)
0.90 / 0.20 = 4.5
A positive result is 4.5 times as likely in the sick

LR- (negative test)

(1 - sensitivity) / specificity
0.10 / 0.80 = 0.13
An LR above 10 or below 0.1 changes decisions most

Report and appraise

STARD: reporting checklist and flow diagram
QUADAS-2: risk of bias
Test it on the patients who will use it

KEY MESSAGE Likelihood ratios do not change with prevalence. Predictive values do.

Extra: Concept only. Likelihood ratios combine sensitivity and specificity into one number that tells you how much a result should change your suspicion.

LR+ = 0.90 / 0.20 = 4.5: a positive result is 4.5 times as likely in someone with the disease as in someone without. LR- = 0.10 / 0.80 = 0.13: a negative result is about 8 times less likely in the sick. Rough rule: LR+ above 10 or LR- below 0.1 changes decisions a lot. Values near 1 change little.

Unlike PPV, likelihood ratios do not depend on prevalence, so they travel between settings better.

Reporting: STARD (Standards for Reporting Diagnostic accuracy studies: 30 items and a participant flow diagram). Appraisal: QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies). Watch for spectrum bias: a test evaluated in very sick patients versus healthy volunteers looks better than it will in real, mixed patients.

Misconception: ‘diagnostic accuracy studies are just a 2x2 table’. The design questions (who was tested, was the reference standard applied to everyone, was it read blind) matter as much as the numbers.

Timing: 1.5 minutes. The Advanced Modules offer diagnostic-accuracy analysis hands-on.

The RoB 2 traffic-light plot

RoB 2: Cochrane risk-of-bias tool for randomised trials. D1-D5: its five domains. Illustrative judgements.

KEY MESSAGE Judge each domain. The overall risk is set by the worst domain.

Extra: Illustrative judgements. RoB 2 asks signalling questions in five domains: D1 randomisation process (sequence and concealment), D2 deviations from intended interventions (blinding, adherence), D3 missing outcome data, D4 measurement of the outcome, D5 selection of the reported result.

Each domain is judged low risk (green +), some concerns (yellow -) or high risk (red x). The overall judgement is driven by the worst domain: Trial C is high risk because of D2.

Reviews show this plot so readers can see which trials drive the pooled result and whether excluding high-risk trials changes it (a sensitivity analysis).

Apply to Paper 1, Boulet (discuss, do not score formally): D1 likely low (central computer). D2 some concerns, as clinicians knew the group. D4 low for an objective outcome such as fluid balance.

Misconception: ‘unblinded = high risk’. Not if the outcome is objective and the deviations are small. It depends on the domain questions.

Timing: 1.5 minutes. Session 8 uses our simplified 10-question checklist. The Advanced Modules practise the full tools.

Worked example: prevalence or incidence?

Today

200 ward patients, 20 have a pressure injury
Prevalence 10%

At risk

180 patients without an injury

One week

9 develop a new injury

Incidence

9 / 180 = 5% per week
Denominator: only those at risk

KEY MESSAGE Incidence always has a time period and only counts people at risk.

All property rights and copyright are reserved.

This presentation contains proprietary data, information, and materials developed by Neudata for dedicated use only and may not be communicated, copied, reproduced, distributed, published, or cited, in whole or in part, without the prior written consent of Neudata. Where explicit written permission has been granted, appropriate attribution must be included, followed by “by courtesy of Neudata”.

Any unauthorized use or infringement of these materials may give rise to legal action and claims for damages, without prejudice to any other rights of Neudata, including rights relating to patents, trademarks, or other forms of intellectual property protection.