This article was first published in The Mandarin on 29th October 2026. My thanks to Peter Gearin and the editorial team.
The Commonwealth Ombudsman’s investigation into the Department of Health, Disability and Ageing’s assessment algorithm, which prevented staff from being able to override automated outcomes, concluded that a reasonable person would lack confidence that the system is fair and consistent. It is therefore unsurprising that there was a five-fold increase in appeals compared to the previous year. As the APS gears up to further integrate AI and algorithms as part of the decision-making process, ensuring that human beings have effective oversight, and that agencies can demonstrate this, becomes a requirement of good governance.
Federal policy on the use of AI across the APS contains a mixture of mandatory requirements, desirable principles, and outcomes. Significantly, it does not mandate that the question of whether AI systems are in effect replacing human judgement is systematically measured. Without such a requirement in place, AI systems risk becoming the de facto decision maker, with no rigorous way to identify if and when this is occurring. Instead of having the decision-making processes enhanced by AI, we may be handing decisions over to AI, jeopardising fairness and accountability in the provision of government services.
This risk is repeatedly acknowledged. As an overarching principle, APS officers need to be able to justify and take ownership of decisions and advice. More specifically, departments are asked to measure how human oversight is performing in the decision-making process. Various methods are put forward, with the only mandatory requirement being staff training, and other approaches being recommended, such as performance assessments, interviews, live observations, peer reviews, random audits, and feedback loops for incorrect AI outcomes. These policy positions recognise that human oversight is a requirement to correct for poor outcomes, and that this necessitates active steps to ensure effective implementation.
It is questionable whether these approaches are able to provide the rigorous assurance that would be desirable. Undertaking training by itself does not guarantee that it is having an impact on the decision-making process when it actually counts. Similarly, human beings will alter their behaviour when they know they are being observed or assessed, meaning that these reports come with a degree of distortion. Furthermore, engagement with peers and the undertaking of audits may very well repeat the same mistakes that they seek to catch, as peers and auditors are given the same information as officers, but crucially lack certainty as to what constitutes the correct decision. Additionally, feedback loops tell us nothing about the poor outcomes we are failing to recognise, or what proportion of decisions they constitute. In effect, what we are missing is a robust approach to ensure that we have understood how AI systems are impacting decisions, yet at the same time we are insisting on accountability.
Ensuring that people have reached the correct conclusion from the information they are presented with is not a new challenge. The seeding of test cases, whereby scenarios with known correct outcomes are presented as part of the officer’s caseload, goes a long way to remediating the weaknesses identified in approaches suggested by AI policies. For example, at airports, virtual images of weapons and explosives are added to the x-ray pictures of passenger bags to determine if staff are correctly judging threats. This way, determining performance becomes clearer, as we can identify what proportion of outcomes are incorrect, and which cases this applies to. We also negate the risk of peer reviews and random audits reaching the same incorrect conclusions as the officer being tested, as the correct outcome is known beforehand. Moreover, because officers know that seeded test cases can be a part of their regular workload, this encourages increased attentiveness across all of their work, rather than simply altering behaviour during performance assessments, interviews, and live observations. It also allows us to surface other insights, such as the impact of training strategies on performance, how fatigue and caseload come to impact outcomes, and if an officer’s performance is changing over time. Importantly, seeded test cases should be designed to present no risk of impacting real-world outcomes.
Having effective oversight of AI requires officers to not only be able to accurately judge what is the correct outcome, but to also have the preparedness to override AI recommendations when they believe they are wrong. Each of these factors can show up as errors, but have markedly different causes, and call for very different solutions. In the case of the inability of the officer to correctly judge cases, this could come from poor training or tools, lack of experience, or simply being given irrelevant information in the decision-making process. Contrastingly, a reluctance to override AI when an officer suspects it is wrong can come from the belief that AI rarely, if ever, makes mistakes, that overriding involves extra work such as the recording of a justification, and the very human reality that being wrong in the company of AI feels less lonely than risking being wrong by yourself. One cause calls for better systems, tools and training, the other for increased individual authority and an environment that does not overly punish wrong decisions. Disentangling these factors is important so that we not only detect if human oversight is operating effectively, but how to go about making improvements.
This has been addressed through a framework that has often been employed in psychology called Signal Detection Theory. In essence, it allows us to distinguish between officers who override both incorrect and correct recommendations at similar rates, indicating poor judgement, and officers who rarely override recommendations, but are usually correct when they do so, suggesting deference. What this means is that through the application of seeded test cases, coupled with the Signal Detection Theory framework, accountable officials can have hard data which provides assurance of effective human oversight, as well as a clearer understanding of why things may be going wrong, and therefore how to correct them. In many cases, data collation and analysis could be automated, giving departments a way to diagnose and improve at both the organisational and individual level, without these numbers becoming performance targets.
One of the most important benefits may be helping us to be clear about what really matters on a human level. It is self-evident that the cost of a wrong decision in the form of inspecting a bag at the airport is minor compared to the catastrophe of allowing explosives on a plane. Every real-world decision involves degrees of ambiguity, but every environment varies greatly in which kind of errors we are willing to tolerate, how cautious or liberal we are in our approach. Every department and officer is making this trade-off, whether they are aware of it or not. The combination of seeded test cases and Signal Detection Theory allows us to be clearer on how this is operating in practice at both the organisational and individual level, and for this to be explicitly acknowledged and accounted for. As AI systems and policies are rolled out across the APS, without having the right tools and frameworks in place, we will simply lack the ability to know if human oversight is working effectively, and whether it reflects what we as a society deem to be important. This brings into question the fairness and accountability that we seek in the provision of government services.