In healthcare, what matters most is often difficult to measure. Patient well-being, quality of life, dignity, autonomy, and clinical judgment are central to good care, yet they rarely fit neatly into a single metric. Artificial intelligence (AI) systems, however, depend on measurable objectives. As a result, healthcare AI is frequently optimized using proxy indicators such as readmission rates, length of stay, treatment adherence, or diagnostic accuracy. While these metrics can provide valuable signals, they do not fully capture the broader goals of patient-centered care.
The challenge of alignment is especially important in healthcare, where many AI applications are considered high-risk under the EU AI Act because of their potential impact on patient safety and fundamental rights. Ensuring that AI systems optimize for meaningful patient outcomes rather than narrow performance metrics is therefore not only a technical challenge, but also a matter of clinical quality, risk management, and business governance. Aligning healthcare AI requires healthcare organizations to move beyond rigid, quantifiable operational targets and establish governance structures that ensure AI systems remain consistent with patient-centered values, clinical objectives, and organizational responsibilities.
AI systems can appear autonomous because they generate strategies that were not explicitly programmed by developers. However, the actual autonomy of settings to end goals like a human is not the case. AI systems do not formulate their own goals, intentions, or motivations. Instead, their objectives are mathematically defined through objective functions that specify what the system is designed to optimize. Behaviors emerge as a consequence of optimizing these predefined objectives within a given environment.
What may appear to be goal setting is more accurately understood as the discovery of an unexpected strategy for maximizing the original objective. While the objective itself is established by humans, the pathway toward achieving it is often determined by the AI system. This capacity to identify novel and sometimes unforeseen solutions introduces an important need for transparency, monitoring, and continuous human oversight.
In healthcare, this raises a critical question: how should objectives be designed to ensure that AI systems support the best possible patient outcomes? Goals should reflect not only medical priorities but also ethical principles. A narrowly defined objective, such as maximizing short-term survival rates, may inadvertently overlook factors such as quality of life, patient preferences, fairness, equity, or long-term health outcomes. As a result, effective objective functions should balance multiple dimensions of patient well-being rather than optimize a single metric in isolation.
Achieving this balance is challenging because healthcare decisions involve complex ethical considerations that must be translated into quantitative optimization criteria. Important factors such as patient autonomy, quality of life, fairness, compassion, and clinical judgment are often difficult to capture in a mathematical objective function. Consequently, even a well-designed system may optimize measurable outcomes while failing to fully advance the broader ethical goals that healthcare professionals seek to uphold.
The importance of careful goal design becomes evident when considering how AI systems respond to the objectives they are given. In complex healthcare environments, AI systems may discover highly effective strategies for maximizing predefined metrics that nevertheless conflict with the underlying intentions of clinicians or patients. This phenomenon, commonly referred to as reward hacking, occurs when a system exploits loopholes in its objective function to achieve high rewards while deviating from the broader purpose of the task. Rather than reflecting genuine autonomy, such behavior demonstrates how limitations in goal specification can lead systems to pursue optimization strategies that are technically successful but clinically or ethically undesirable.
Alignment is the problem of getting an AI system to pursue what we actually intend, rather than what we managed to write down. It is one of the central research problems in artificial intelligence and a priority at the frontier AI companies, where it is mostly discussed with future, more powerful models in mind. In healthcare, it is already visible in systems deployed today. “Good health” cannot be written into a loss function, so developers specify something measurable that tends to move with it: length of stay, readmissions, cost, diagnosis code. A model trained on such a target learns the target, including every way it differs from what we care about, and repeats that difference across every patient it scores.
There are three things in play: the goal we intend, the target we specify, and what the model actually learns. The gap can open between any two of them.
Outer misalignment is a gap between the goal and the target. The model learns its target faithfully; the target was wrong. Sometimes this is visible. Obermeyer et al. (2019) studied a commercial algorithm used to decide which patients received extra care, applied to millions of people in the US. Its target was future healthcare costs, as a stand-in for health needs. Because less money was historically spent on black patients at the same level of illness, the algorithm ranked them healthier than they were. Sometimes it is hidden inside a target that looks right. The Epic Sepsis Model was trained to predict sepsis, but its labels were sepsis as clinicians recorded it, shaped by when they suspected it and started treatment. At an external site, the model missed about two thirds of sepsis cases while alerting on nearly a fifth of admissions (Wong et al., 2021). Kamran et al. (2024) found that when only data from before treatment started was used, its performance dropped to no better than chance. The model did what its target asked: it recognized clinicians’ suspicion, not the patient’s condition.
Inner misalignment is a gap between the target and what the model learns. The target is sound, but the model relies on something else that happened to predict it during training. Zech et al. (2018) trained a pneumonia detector on chest X-rays from several hospitals. It learned to recognize which hospital an image came from, because pneumonia rates differed between sites.
Bias and poor generalization of new settings are among the best-known problems of clinical AI. Seen through alignment, they are a logical consequence of the gap. The gap can show in two ways. Within the data the model was built on, it appears as bias: systematic error that may hit all patients or some groups more than others, as Obermeyer’s cost target did to black patients. When the setting changes, through a new hospital, population, protocol, or scanner, it appears as a distribution shift. Epic’s labels reflected local practice, so the model faltered at a hospital where practice differed. Zech’s shortcut broke at new sites, and the COVID-19 detectors studied by DeGrave et al. (2021) looked accurate until they were tested at new hospitals. Either kind of misalignment can produce either symptom.
Headline performance is measured against the specified target, on data like the training data. A model can therefore score well while misaligned in either way.
Testing a model at other sites can show that it fails, but not why. For inner misalignment, the why lies in what the model relies on, and recognizing a shortcut requires knowing what it should rely on instead: a pneumonia detector should look at the lungs, not at a hospital marking. Interpretability methods show what a model relies on; clinical knowledge judges whether that is right. This is how DeGrave and colleagues found models reading text markers and image borders. The same knowledge can also be built into the model before training. Instead of letting a model learn freely from whatever patterns predict the target, developers can constrain it with explicit knowledge of the domain: which inputs are relevant, how clinical concepts relate, what may influence what. A model that cannot rely on irrelevant patterns cannot learn shortcuts through them. Detection and prevention draw on the same source: explicit knowledge of what should matter.
Outer misalignment is harder to recognize. Knowing what should matter helps when a model strays from its target, but not when the target itself is wrong. The model does what its target asks, so inspecting it shows nothing wrong, and every standard check, from validation to monitoring to explanation, measures it against that same target. When outer misalignment does show, it looks like an ordinary bias or generalization problem, and the cause stays hidden. It can only be found with a reference outside the target. Obermeyer’s bias came to light by comparing the scores with independent measures of illness and when the authors replaced cost with such a measure as the target, the share of black patients selected for extra care more than doubled. Epic’s reliance on clinicians’ suspicion, by looking at when its predictions came relative to treatment. Human oversight is meant to supply that outside reference, but the clinician often has nothing to check the output beyond their own judgment. In practice, they defer, and automation bias follows.
Organizations already run on proxies, and AI inherits the gaps in its targets and turns them into decisions about patients, at scale. The costs land on the organization: patient harm, liability, failed deployments and, under the EU AI Act, compliance exposure.
An organization deploying AI has two questions to answer. Did the model learn its target? That can be tested at other sites and inside the model. Is the target the right one? That cannot be answered from the model or its metrics, only by asking what the target measures and what it leaves out. The safest target is a measurable patient outcome, not a record of what clinicians or the system did.
Alongside the AI systems an organization procures and validates; staff increasingly use general-purpose language models directly, for summaries, letters and clinical questions. These models face the same gap in another form: their target is plausible text, not true text, and where the two come apart, they produce fluent answers that are wrong. No validation process sits between such an output and the decision it informs; the only outside reference is the person using it. That adds a third question: does the user know what the model answers, and when their own judgment is needed? That is what AI literacy should provide.
Because the usual checks are built on the target, an organization can be compliant on paper while its AI drifts from the patient. The objective an AI system is given is a strategic decision, not a technical detail.
Governance begins with a precise statement of intended use: the clinical or operational problem, target population, care setting, intended user, input data, output, and decision the output is expected to influence. The organization should document both the measurable target and the outcome that the target is intended to represent. For example, if cost is used as a proxy for health need, the assessment must test whether cost is sufficiently related to need across relevant patient groups and care pathways. This makes objective selection a clinical and strategic decision rather than a technical implementation detail.
Multi-Objective Alignment:Incorporating soft constraints, ethical guidelines, and subjective patient preferences alongside hard clinical metrics.
Purpose should also be multi-dimensional. Accuracy may matter, but so do patient benefit, safe failure modes, equity, usability, clinical workload, timeliness, and patient preferences. A system can meet its technical target while failing its healthcare purpose if it creates unnecessary alerts, delays of care, shifts work to clinicians, or improves a proxy without improving outcomes.
Value Alignment Beyond Accuracy: Why medical AI cannot just be evaluated on accuracy but must be evaluated on value alignment and safe failure modes.
The intended use determines the applicable regulatory pathway. Where AI performs a medical purpose, organizations should assess whether it qualifies as medical-device software and identify the relevant obligations under the MDR or FDA SaMD framework. AI Act classification should be assessed alongside, not instead of, medical-device status, data-protection duties and professional requirements. Controls should be incorporated into the quality management system. ISO 13485 provides the organizational structure for controlled design, supplier management, documentation, and corrective action. IEC 62304 supports disciplined software lifecycle processes, including maintenance and change control. ISO 14971 provides a method for identifying hazards, evaluating risks, implementing controls, and assessing residual risk. AI-specific concerns such as data shift, bias, opaque failure modes, and model updates should be translated into these established processes rather than managed in isolation.
Evidence should show more than performance against a retrospective benchmark. It should establish analytical validity, clinical validity, and practical utility in the intended setting. Evaluation should test relevant subgroups, external sites, workflow conditions and failure scenarios, and should compare model performance with the actual patient or operational outcome the system is expected to improve. The business case should use the same evidence discipline. Expected value should account for implementation costs, integration, training, workflow redesign, false alerts, additional diagnostics, maintenance and monitoring. Procurement should not proceed on the basis of headline accuracy when the organization cannot explain how the system will improve care, reduce avoidable work, or support a reimbursable service.
Deployment is a controlled change to a care process. Before going-live, the organization should define who receives the output, how it enters the workflow, what action is expected, when escalation is required, and what happens if the system is unavailable or wrong. Local validation is necessary because patient populations, clinical protocols, prevalence, equipment and documentation practices may differ from the environment in which the model was developed. Clinician adoption depends on relevance, usability, and trust. Human oversight is meaningful only when users have sufficient time, information, authority, and competence to challenge an output. Training should explain what the system predicts, what it does not predict, known limitations, appropriate reliance, and reporting routes. Accountability must remain explicit; adding a human reviewer does not resolve unclear responsibility or transfer liability away from the deploying organization.
Human-in-the-Loop vs. Human-over-the-Loop:Redefining the clinician's role not as an executor of AI output, but as a critical evaluator of the AI's implicit assumptions.
Most healthcare organizations procure AI rather than develop it. Supplier assurance is therefore a core governance activity. Assessment should cover intended-use fit, regulatory status, quality-system maturity, clinical evidence, data provenance, subgroup performance, cybersecurity, interoperability, explainability appropriate to the use case, and the supplier’s approach to updates and incident management. Contracts should define responsibilities for validation, regulatory compliance, data protection, change notification, audit access, service levels, liability, incident support, post-market monitoring and termination. Buyers should also require enough information to assess whether an update changes performance, intended use or risk. A CE mark, authorization or vendor's assurance package can support due diligence, but it does not replace local assessment of workflow fit, patient population, residual risk and organizational readiness.
Monitoring should follow the system throughout its operational life. Organizations should track technical performance, data and model drift, subgroup effects, overrides, near misses, complaints, cybersecurity events, and changes in clinical practice. They should also monitor whether the intended patient or operational outcome improves. A stable model score can conceal a failing deployment if clinicians ignore the output, the workflow changes, or the proxy becomes less connected to the desired outcome. Post-market monitoring should connect with vigilance, complaint handling, corrective and preventive action, supplier management, and periodic benefit-risk review. It should also include business measures such as adoption, time saved or added, avoided costs, reimbursement performance, and realized return on investment. These measures should never displace patient safety, but they are essential for deciding whether the system should be scaled, modified, restricted, or retired.
Predefined thresholds should trigger investigation, restriction, rollback, retraining, supplier escalation, or suspension. Changes to models, data pipelines, interfaces, clinical protocols or intended use should pass through documented change control and, where relevant, renewed validation and regulatory assessment. Governance bodies should have clear authority to stop use when evidence no longer supports the benefit-risk balance. Executive ownership is necessary because failures cross functional boundaries. Harm may arise from model design, procurement, integration, staffing, workflow, training, or incentives. A multidisciplinary decision structure should therefore include clinical leadership, quality and regulatory affairs, safety, legal, privacy, cybersecurity, procurement, finance, IT and representatives of the people who use or are affected by the system.
The core governance test is not merely “Does the AI perform as specified?” It is “Does the specified objective, embedded in this workflow and monitored over time, continue to produce an acceptable benefit-risk balance for patients and the organization?”
Alignment is not something that can be solved entirely inside the model. It is both a technical and an organizational problem.
AI governance should not be treated as a stand-alone compliance program or reduced to the requirements of the EU AI Act. It should operate through the broader healthcare quality and regulatory ecosystem: clinical governance, quality management, patient safety, medical-device oversight, data protection, cybersecurity, professional accountability, and enterprise risk management. The central challenge is maintaining alignment between what an AI system optimizes, the care process in which it operates, the organization's objectives, and the outcomes that matter to patients.
That challenge extends across the entire lifecycle. Intended use, objectives, and evidence must be critically evaluated before deployment; workflow integration, safe failure modes, and clinician responsibilities must be validated before go-live; and performance, adoption, incidents, drift, and patient outcomes must be continuously monitored thereafter. Human oversight is meaningful only when clinicians have the information, time, authority, and competence to challenge outputs and when escalation paths, intervention mechanisms, and accountability structures are clearly defined.
This is also a business discipline. Weak governance increases the likelihood of failed deployments, patient harm, regulatory exposure, unclear liability, low clinician adoption, duplicated work, procurement lock-in, unreimbursed services, and costly remediation. Strong governance improves investment decisions by linking clinical benefit and safety to implementation cost, workflow impact, reimbursement potential, contractual responsibility, and realized return. It enables organizations to discontinue unsuitable projects early, scale systems that demonstrate value, and retire those whose benefit-risk profile or business case no longer holds.
Because most healthcare organizations procure rather than develop AI, governance cannot end at procurement. Acquiring an AI system does not outsource accountability. Organizations remain responsible for determining whether a system is appropriate for their patients, workflows, and risk environment, and for ensuring that its performance and impacts remain acceptable throughout its operational life.
The ultimate governance question is therefore not whether an AI system performs as specified, but whether the objective it has been given continues to serve patients, clinicians, and the organization in practice. Healthcare organizations should govern AI not as a technology purchase, but as a continuously monitored clinical and business intervention for which they remain accountable.
Joy is an AI researcher and intern at Nemko Digital. Drawing on her academic research on the EU AI Act and global AI regulations, she focuses on the relationship between AI, trust in technology, and societal impact across different cultural contexts. Her work centers on supporting the safe and responsible adoption of innovative AI technologies across a wide range of sectors, like environmental sustainability, healthcare, and organizational efficiency.
Gijs Danoe is a PhD researcher at the University Medical Center Groningen, working on the alignment of AI in medicine. Trained as a computational linguist, he builds clinical language models and is first author of ERNIE, an outbreak detection system published in npj Digital Medicine and awarded the NHG Science Prize. He translates this research into AI literacy workshops for healthcare professionals.