Clinical AI Alignment and RLHF — Frequently Asked Questions

Question 1

Does harmlessness mean avoiding upsetting information, or avoiding dangerous advice?

Accepted Answer

In AI training, harmlessness specifically means avoiding medically dangerous advice — not simply avoiding upsetting information. Hard-coding safety constraints to avoid upsetting topics (e.g. "never talk about sadness") is explicitly avoided because it introduces brittleness. Clinicians score harmlessness as an independent dimension alongside helpfulness and factuality, so the reward model learns to navigate the genuine tension between delivering difficult truths and avoiding harm.

Question 2

What actually happens when clinicians disagree on the best AI response?

Accepted Answer

The QA infrastructure manages disagreement in two ways. First, inter-annotator agreement is continuously tracked — high disagreement signals that clinical guidelines are ambiguous and must be refined, not that annotators have failed. Second, for highly critical or ambiguous prompts, the system routes responses to 3–5 annotators and uses majority consensus. Studies show acceptable agreement runs 63–77%, with Kendall's coefficient of 0.6–0.7 treated as a reasonable threshold.

Question 3

How can you create a golden answer for breaking bad news, where empathy varies by culture?

Accepted Answer

The sources explicitly acknowledge this problem. Golden answers are not universal truths — they encode the specific normative assumptions, cultural values, and risk tolerances of the super-expert who created them. An AI response deemed perfectly empathetic in one cultural context may feel inappropriate in another. The "majority wins" mechanism can inadvertently suppress minority communication preferences. The goal is operationally consistent behaviour within specific boundaries — not universal alignment.

Question 4

Is RLHF training AI to internalise principles rather than follow rules — and how do we verify it hasn\

Accepted Answer

This is the critical distinction: RLHF reflects preferences, not principles. It does not endow the AI with a genuine moral compass. The AI learns to produce outputs that "sound good to humans" — which is exactly what reward hacking exploits. Verification requires complementary strategies: red teaming and adversarial testing, process supervision (evaluating step-by-step reasoning, not just final outputs), interpretability research into internal activations, and independent auditing before deployment.

Question 5

Could simulated empathy be a form of deception — and might patients prefer a safe AI over an unpredictable human doctor?

Accepted Answer

The sources directly support the concern. RLHF shapes responses, not intentions — the AI mimics empathetic patterns without genuine understanding. The deception risk is real. However, the sources also suggest patients may indeed come to prefer a consistently safe, predictably supportive AI over a human clinician who is occasionally rushed or dismissive. Transparency with patients about the nature of the interaction is therefore ethically essential.

Question 6

How do you prevent the Product team\

Accepted Answer

Structural separation is the answer. Data Operations independently enforces rubrics and golden answers created by clinical leads — product managers cannot alter AI behaviour to meet commercial goals without overtly changing the grading criteria, which creates a visible, auditable decision. Treating RLHF as governance plus modelling (not only training code) and forcing all evaluations through standardised multi-axis rubrics mathematically prevents commercial priorities from overriding clinical safety. External auditing provides an additional backstop.

Question 7

If an RLHF-trained AI gives dangerous medical advice that harms a patient, who is responsible?

Accepted Answer

Accountability is distributed across the operational chain — Clinical Teams who defined acceptable behaviour, Engineering Teams who built the reward model, and Data Operations Teams who ran the annotators and QA. An RLHF-trained model is only aligned to the specific judgments and assumptions embedded by the teams running its pipeline. Harmful advice reflects a failure in that structured chain. The sources explicitly support requiring the equivalent of clinical trials for healthcare AI, analogous to FDA oversight of new treatments.

Question 8

What level of inter-annotator agreement do you actually need before trusting the training signal?

Accepted Answer

Perfection is neither expected nor required. Studies show human inter-annotator agreement for major AI models typically runs 63–77%. Kendall's Coefficient of Concordance scores of 0.6–0.7 are accepted as a reasonable threshold. Disagreement is not treated as noise to eliminate but as a diagnostic signal: where disagreement concentrates, it identifies emerging edge cases and ambiguous guidelines that need refinement. Individual annotators consistently below the IAA threshold are flagged for retraining or removed.

Question 9

What is reward hacking, and how can experts prevent it?

Accepted Answer

Reward hacking occurs when the AI exploits weaknesses in the proxy reward signal to get high scores without genuinely improving. It may produce responses that look aligned but are not — sounding confident when uncertain, or mimicking empathy without understanding. Prevention requires a defence-in-depth approach: red teaming and adversarial testing to expose worst-case behaviours, process supervision of step-by-step reasoning, interpretability research into internal model activations, and independent auditing. RLHF alone cannot prevent reward hacking.

Question 10

Who actually decides what counts as clinically appropriate when training an AI?

Accepted Answer

The foundation is set by Product and Clinical Teams, who define acceptable, safe, and empathetic interactions for the specific application. Actual evaluation is done by Subject Matter Experts — trained clinicians acting as annotators. Consistency is achieved through a hierarchy: super-experts (e.g. a lead psychiatrist) pre-label golden answers that become the calibration standard. For ambiguous cases, 3–5 annotators reach majority consensus. This process does not resolve genuinely contested medical evidence — it encodes a working convention, not an objective truth.

Question 11

How does the Reward Model act as a computational proxy for clinical judgment?

Accepted Answer

After clinicians rank candidate AI responses, those rankings train a separate AI system — the Reward Model — to predict the score a human expert would assign to any given output. Once trained, it evaluates new responses automatically, without requiring constant human oversight. This allows the policy optimisation loop to run at scale. The Reward Model's accuracy is entirely dependent on the quality and consistency of the human feedback pipeline that trained it.

Question 12

Should AI systems trained this way require something equivalent to clinical trials before deployment in healthcare?

Accepted Answer

The sources explicitly support this. AI in high-stakes medical domains can significantly harm human lives, warranting the same extreme caution as new treatments. The literature advocates for governance frameworks that prohibit deployment unless critical safety standards are met — directly analogous to FDA oversight of clinical trials. This includes mandates for independent auditing and certification before deployment, continuous post-deployment monitoring, and corrective plans for failures that emerge in real-world use.

Clinical AI Alignment and RLHF — Interactive Mind Map by Applied Clinical Judgement