Human, all too human: accounting for automation bias in generative large language models

Introduction

The latest advancements in Artificial Intelligence (AI), notably generative large language models (LLMs) and their widespread availability through chatbots such as OpenAI’s ChatGPT have not only demonstrated their immense conversational capabilities but have also exposed the public to risks of developing undue reliance on the chatbot’s generated output. In fact, due to their highly persuasive and human-like answers to the users’ questions or instructions, users may be prone to overrelying on the chatbot’s generated output in their decision-making processes.¹

However, the technical foundation of such technology, which is predicting the next word according to statistical occurrences in a specific context, makes them intrinsically prone to generating incorrect, albeit plausible, information. Depending on the specific context of application, substantial harm may occur if the generated incorrect information is used by the user to make a decision.

Regardless of the factual correctness of the resulting output of such conversational chatbots, their highly performative abilities to generate human-like text and execute user’s instructions in natural language pave the way for future cognitive automation. In this sense, in the interaction between a human and an AI system, the anthropomorphic features coupled with uncertain levels of system accuracy contribute to the development of inappropriate trust levels, potentially resulting in overreliance. Therefore, the issue of users’ trust in such tools shall be properly addressed to avoid overreliance, as further explained in ‘Human, all too human: the paradox of cognitive automation’ section.

A few examples of (over)reliance on existing LLM-driven applications stem from the use of ChatGPT by legal practitioners.² For instance, in early 2023, a lawyer quoted verbatim in a legal brief several legal judgments suggested by ChatGPT in its 3.5 version, not integrated into the real-time web search engine Bing Chat by Microsoft, except that the cases were made up, and their justification to have acted in good faith was to no avail.³ As a consequence, the lawyers and their law firm faced a $5,000 fine for overrelying on the chatbot’s outcome, as well as for not exercising due human (and professional) oversight. More recently, two Colombian judges are reported to have used ChatGPT in drafting their judgments.⁴ Despite stating that the chatbot was not used as the sole basis for ruling, mindful of the experience of the two US lawyers, questions around potential automation bias that might have caused the judges to regard ChatGPT as a trustworthy source of knowledge spontaneously arise.⁵

This article delves into the capabilities and limitations of generative LLMs, focusing specifically on chatbots’ anthropomorphic features and the tendency to generate factually incorrect outputs as possible drivers for user overreliance. It then critiques a solely technical approach to improving accuracy, arguing this can perversely increase automation bias.⁶ Moreover, an analysis is presented of the EU’s Artificial Intelligence Act,⁷ requiring human oversight, highlighting its inadequacy in addressing such cognitive biases, in contrast with the EU General Data Protection Regulation’s (GDPR)⁸ principles of meaningful human intervention. Ultimately, the article advances a comprehensive socio-technical framework integrating human factors for a human-centered approach to AI development and deployment: promoting user literacy of AI systems, ensuring appropriate automation levels, and implementing interface designs that counter reliance on potentially inaccurate outputs. Rather than overemphasizing fallible human oversight, the article advocates accountability measures across the entire LLM value chain to calibrate user trust in these systems.

Capabilities and limitations of generative LLMs

The recent trend in Natural Language Processing (NLP) has been towards the development of foundation models that have general-purpose abilities, achieving a range of tasks not explicitly encoded in the training data.⁹ Foundation models have significantly advanced the field of NLP due to their generality and adaptability.¹⁰

Currently, generative foundation models trained to generate text based on a narrowly defined task, such as predicting the next word in a sentence, have become important for machine learning language tasks, triggering increased research in tasks such as summarization, dialogue generation, question answering, and adaptation techniques.¹¹ First, the transformer architecture, comprising self-attention and feedforward layers linked through the residual connections,¹² has been introduced and extensively used in language models.¹³ Such architecture has facilitated fine-tuning with smaller datasets, bypassing the necessity for task-specific designs.¹⁴

Among the reasons for this change are improved computing power and larger available datasets. Recent advancements stem from the ability to augment the size of such models in terms of parameter count,¹⁵ and volume of training data.¹⁶ These models are also better at learning from a few examples or even without any examples, compared to smaller language models,¹⁷ which simplifies the development of task-specific LLMs by reducing the required process to effective prompt design, that is instructing the model on how to perform to achieve the desired result.

Scholars have suggested that LLMs might show emergent abilities.¹⁸ In the context of LLMs, this means a proportionate correlation between the model’s increase in scale and its respective abilities.¹⁹ Examples of strategies favouring emerging abilities of LLMs are specialized prompting. This allows the model to better generalize on previously unseen tasks thanks to effective word choices in the prompts,²⁰ chain-of-thought prompting,²¹ which guides language models to produce a sequence of intermediate steps before giving the final answer by merely specifying in the prompt to ‘think step by step’, and methods for fine-tuning language models on a mix of tasks framed as instructions.

New frontiers for cognitive automation

Due to their ability to generate high-quality, human-like text with high fidelity, LLMs bring about potential benefits across different sectors.²² For example, expert evaluations indicate that LLMs are generally capable of understanding complex medical queries, frequently recalling specialized knowledge, and executing significant reasoning processes, thanks to chain-of-thought prompting for effective enhancement of performance and interpretability in medical reasoning.²³ Another example shows the capabilities of LLMs in legal reasoning, as well as automation of certain routine tasks legal professionals perform daily, such as searching for relevant laws, organizing information, identifying common issues, conducting basic legal analysis, and generating standardized language in correspondence, documents, contracts, and legal briefs.²⁴ The same assumptions can be extended to the field of economics.²⁵

These LLMs are facilitating not only routine cognitive tasks such as decision-making and planning but also higher-order functions such as creative thinking, paving the way for an era of advanced cognitive automation.²⁶ This highlights the potentially disruptive nature of LLMs concerning tasks that were once exclusively human.²⁷ In essence, AI is typically understood as the capability of computers to emulate intelligent behaviour, including activities like reasoning, strategizing, planning, and decision-making, tasks traditionally associated with human intellect.²⁸ Under the broader umbrella of AI, LLMs can automate specific cognitive tasks such as text generation, question answering, reasoning, and research through their ability to produce text in response to prompts. Furthermore, they enhance human capabilities, enabling more productive and insightful interactions between humans and computers.²⁹

Risks of LLMs from the human–computer interaction perspective

While the potential for new emergent abilities in scaled-up language models is likely to occur, this is also unpredictable and uncharted.³⁰ Current limitations prevent these models from achieving peak performance on some tasks, but continual scaling of models may ultimately result in instances of new and unpredictable emergent abilities.³¹

Regardless of whether LLMs indeed possess emergent abilities, or are just a result of certain metrics or statistics for performance evaluation, and not a natural consequence of scaling,³² since the global deployment of ChatGPT, users realized that such tools do not come without potentially harmful effects, opening the discussion around the risks posed by LLMs. Therefore, computer science scholars have devised a common taxonomy of risks.³³ Divided into roughly six areas, ranging from discrimination, hate speech, and toxicity to environmental harms, misinformation, and other malicious uses,³⁴ while legal scholars are still devising a common taxonomy of risks from a regulatory perspective.³⁵ It is important thus to stress that LLM capabilities also come with the challenge and risk of dealing with untruthfulness, bias, toxicity, backdoor vulnerabilities, inadvertent deception, or harmful content synthesis even as language models scale and evolve.

Although all LLM-related risks are worth addressing, and will indeed require extensive research to devise the appropriate mitigation strategy, the scope of this article focuses specifically on the risks stemming from the human computer interaction (HCI) dimension of LLMs. By resembling and mimicking human interaction, especially if integrated into dialogue-type applications, LLMs may display an increased potential for user influence, deception, or manipulation,³⁶ also given the difficulty in detecting machine-generated text³⁷ and the need for caution in developing such detection tools,³⁸ such as watermarking.³⁹

The following analysis explores possible reasons for such misuse of LLMs for cognitive automation and how that contributes to generating harmful consequences of the AI-augmented decision-making process.

Anthropomorphism

Recent studies on the use of LLMs by legal professionals tend to present their capabilities with terms that very much allude to a certain anthropomorphism of such systems.⁴⁰ Take, for instance, the news that ‘GPT Takes the Bar Exam’⁴¹ or even ‘GPT-4 Passes the Bar Exam’⁴². These alone might give the impression that LLMs, deployed in the form of a chatbot, such as OpenAI’s ChatGPT, can perform legal cognitive tasks at the same degree as, if not better than, a legal practitioner.

Even before the advent of LLMs, scholars have been concerned with the sociotechnical dimension of AI systems in the decision-making process, considering their characteristics of autonomy, learning, and inscrutability.⁴³ Research suggests that the more human-like a system appears, the higher the likelihood of the user attributing human traits and capabilities to it, which can also increase perceived credibility.⁴⁴

The generative capabilities of LLMs can be viewed as autonomous behaviour in the sense that, although primarily trained to predict the next word in a text corpus, LLMs, especially if deployed through a chatbot interface, can inadvertently interpret and exhibit human-like agency. Such as the goals, beliefs, or intentions of the original human text author.⁴⁵ As anticipated, this may be a byproduct of attributing anthropomorphic features to such models.⁴⁶ For instance, generative LLMs may be implemented into human–machine dialogue systems, which can be referred to as ‘conversational agents’.⁴⁷ Given the essentially human nature of communication through natural language, users tend to anthropomorphize such conversational agents,⁴⁸ falsely attributing them with human-like traits such as empathy or a coherent identity.

Furthermore, attributing human-like characteristics to language agents could lead to shifting responsibility from developers to the technology itself,⁴⁹ a phenomenon referred to as ‘agency laundering’,⁵⁰ obscuring developer responsibilities and reducing their accountability, thus posing ‘computers as scapegoats’ and forming a barrier to ensuring their liability.⁵¹

As the study of human-generative AI relationships is in its early stages but steadily gaining momentum, it underscores the idea that the anthropomorphism of these technologies may contribute to the establishment of unduly high trust and reliance.⁵²

Hallucinations

In their extraordinary generative textual capabilities, LLMs are intrinsically prone to generating factually incorrect outputs.⁵³ In technical jargon, the term ‘hallucination’ is used to refer to instances where the language model produces unfaithful or nonsensical text. Misleadingly appearing fluent and natural.⁵⁴ This phenomenon occurs due to various factors, including data inconsistencies and model training issues, impacting the model’s performance by reducing its accuracy and reliability.⁵⁵ In other words, LLMs generate inaccurate information due to a lack of sources for the generated content or out-of-date, or otherwise low-quality, training data.

Users of conversational AI, and even professionals using fine-tuned expert systems, risk blindly trusting these sources of information. Given their anthropomorphic features on the one hand, and the potential for factual inaccuracy on the other, this overreliance can lead to harmful outcomes.⁵⁶ To tackle these issues, the proposed solutions heavily rely on human-centric approaches, the shortcomings of which are addressed in the following section.

Human, all too human: the paradox of cognitive automation

While anthropomorphism can enhance trust in AI systems by making them seem more relatable, the occurrence of hallucinations—whether detected or undetected—undermines their reliability, which in turn diminishes trust. This creates a paradox: efforts to build trust through human-like qualities can backfire if reliability issues like hallucinations persist. As a result, trust calibration becomes essential to fully leverage the potential of LLMs for cognitive automation, requiring a balance between the system’s perceived reliability and its actual performance.⁵⁷

Although trust calibration has been extensively examined in the field of automation to address both automation aversion,⁵⁸ and automation bias,⁵⁹ despite the inherent susceptibility of human decision-making to cognitive biases arising from the use of decision heuristics,⁶⁰ its application and understanding in the context of generative LLMs is still under study.⁶¹

Currently, there are no specially designed technical interventions aimed at reducing overdependence on LLMs, and none have been empirically proven to decrease over-trust.⁶² Research has been conducted in the domain of HCI emphasizing the significant role trust plays in users’ behaviours, accounting for their decision to continue using an automated system or their acceptance of the output from a machine.⁶³ Some scholars suggest, such as Parasuraman and Riley,⁶⁴ that excessive reliance on automation represents an aspect of misuse stemming from various forms of human error. An example of such an error is automation bias: the tendency to favour recommendations from automated systems while disregarding information from other sources due to decision biases and a failure of monitoring. Another example is confirmation bias: the tendency to favour information that aligns with prior assumptions, beliefs, and values.⁶⁵

As such, automation bias, resulting from excessive trust in the (anthropomorphized) generative AI system, creates an accountability gap,⁶⁶ which occurs when there is no clear or justified candidate to hold accountable and no one is willing or capable of assuming the responsibility.⁶⁷ When human decision-makers incur automation bias, they end up delegating the decision to the AI system itself, which cannot be properly held accountable. This phenomenon is referred to as ‘computers as scapegoats’, whereby the human in control attributes agency to the computer, which cannot legally bear the consequences for its potentially flawed decision-making.⁶⁸

In the following paragraphs, I analyse—and critically assess—the proposed solutions from both the technical and the policy perspectives.

Human-in-the-loop: the techno-solutionist

In the previous section, the issues of anthropomorphism and hallucinations were mentioned, along with their impact on trust calibration. While it is true that the intention behind intelligent decision aids is to minimize human error and workload, designers nonetheless need to recognize that incorporating higher levels of automation without accounting for human cognitive limitations and biases can lead to new errors in system operation, ultimately impacting the success or failure of the system itself.⁶⁹

For instance, technical scholarship has advanced numerous solutions to reduce an LLM’s tendency to produce factually incorrect outputs. This is generally attempted through techniques involving grounding the model’s response into external knowledge.⁷⁰ Such a technique, known as ‘Retrieval Augmented Generation’ (RAG), involves strategies like prompting,⁷¹ leveraging the LLM’s self-critique capacity,⁷² refining the input to achieve the desired output,⁷³ or increasing its reasoning capacity.⁷⁴ By augmenting the LLMs with external knowledge, it is claimed that LLMs minimize factual inconsistency,⁷⁵ even with real-time verification of possible hallucinations.

I conventionally refer to these approaches as ‘human-in-the-loop’, since humans, and more specifically the system’s designers, will decide whether to introduce such RAG solutions and which databases to leverage for augmented retrieval.

However, trying to solve the hallucination problem from a purely technical perspective may bring about even more adverse effects.⁷⁶ Counterintuitively perhaps, the more performative a system becomes, namely in its capacity to generate plausible answers and reliable answers validated by data from reliable external databases, the higher the risk of automation bias due to increased perception of trust in the system.⁷⁷ Even more so if we also consider the anthropomorphic features of such conversational chatbots.

Some scholars address the issue of overreliance on anthropomorphized chatbots from a perspective of misinformation, or manipulation, since overreliance on such conversational tools can not only make people prone to believe inaccurate information but can also be abused to subtly change or manipulate people’s behaviours.⁷⁸ It may also be partially resolved by sector-specific regulations imposing an extra duty of care onto users of an AI in their decision-making process,⁷⁹ such as a doctor or a lawyer.⁸⁰ However, where such an extra, legally imposed, duty of care does not exist, one needs to resort to a different risk mitigation strategy. Any attempt therefore to tackle the anthropomorphism issue would also require a ‘human-in-the-loop’ decision at the system’s design phase.

Nonetheless, to truly grasp the potential of human augmentation through AI deployment in decision-making, which may outperform both humans and AI systems alone, it is crucial to be able to detect incorrect AI recommendations to preserve human agency. ⁸¹ For instance, for classification models, it has been proposed that augmenting transparency,⁸² or explainability,⁸³ of the model’s recommendation might help tackle automation bias, with mixed or often unsatisfactory results.⁸⁴

Paradoxically, again, explanations can contribute to overestimating the AI’s output, particularly when such explanations, accompanying an incorrect output, are detailed enough to mimic human-like reasoning.⁸⁵ The assumption is that users would analytically evaluate each explanation to determine which suggestions to accept, but people seem to avoid such cognitive effort and instead use explanations as an overall signal of system competence or expertise, increasing trust and potential overreliance.⁸⁶

With generative LLMs, things are even more complicated since studies are less extensive due to their relatively recent deployment on a broader scale. Moreover, due to their intrinsically less explainable nature and tendency to hallucinate, there are fewer possibilities to verify the accuracy and correctness of the LLM’s output.⁸⁷

Therefore, simply reducing the risk of hallucination without accounting for anthropomorphism might further increase trust due to the higher reliability of the system’s outputs, leading to inflated perceptions of the conversational agent’s competencies and underestimating the risks of surrendering effective control. Therefore, it may exacerbate the risk of automation bias, since the LLM can not only automate certain cognitive tasks, but also automate them beyond what is delegated or dictated by the human decision-maker.⁸⁸ Thus posing a safety risk as models could harbour inaccurate beliefs, adverse intent, or even follow misaligned goals.⁸⁹

Human oversight: the regulatory optimist

Current regulatory initiatives across the globe, from the EU Artificial Intelligence Act⁹⁰ (hereinafter referred to as the ‘AI Act’) to the US-proposed NIST AI Risk Management Framework, adopt a risk-based approach to AI regulation. This entails, from a policymaking perspective, regarding harms caused by AI as risks and adopting risk mitigation strategies to reduce their impact to a threshold of societal acceptability.

Risk is typically assessed as the probability of a harmful event multiplied by its severity, which generally entails a quantitative evaluation. However, in terms of AI-related risks, these evaluations take on a more qualitative character, considering the impact of such systems on fundamental rights. Therefore, addressing these risks proactively from a purely quantitative perspective can be challenging.⁹¹

From these premises, it is clear how these challenges are further exacerbated in the case of foundation models, which by definition are subject to various downstream applications, whose impact on safety or fundamental rights are not identifiable ex ante.⁹² Understanding the context, such as the target industry, task, end-user, and model architecture, is crucial for defining and assessing harm in any application, emphasizing the importance of cross-communication and collaboration among model developers, deployers, and end-users in crafting a comprehensive risk evaluation and risk mitigation strategy.

As proof of the difficulty in regulating ex ante risks arising from certain AI systems, consider that the original text of the AI Act, upon the European Parliament’s proposition, was amended to include specific requirements for foundation models, now general-purpose AI models (GPAI). As a result, Articles 51–56 were introduced, addressing specifically providers of GPAI. This is to acknowledge that a GPAI ‘displays significant generality and is capable of competently performing a wide range of distinct tasks […] and that [it] can be integrated into a variety of downstream systems or applications’, according to the definition provided in Article 3(63).⁹³ Leaving aside the differentiation related to systemic risks, Article 53 obliges providers of these GPAI to draw up and keep up-to-date technical documentation, and make available information and said documentation to downstream AI providers to enable them to have a good understanding of the capabilities and limitations of the GPAI and to comply with their obligations pursuant to the AI Act.⁹⁴

While the state of the art is indeed constantly advancing in the attempt to find new and more effective solutions from the technological side, from the policy side, another potential safeguard is to require the exercise of human oversight.⁹⁵ Relevant to the analysis at hand, Article 14 of the AI Act generally imposes that AI systems shall be designed to allow appropriate oversight. This entails that the natural persons in charge of human oversight of high-risk AI systems understand the system’s capabilities and limitations, monitor its operations for signs of anomalies, and address these promptly while also being aware [emphasis added] of potential automation bias that may result in overreliance on the system’s output (Article 14(4)(b)). Furthermore, Article 14 specifies that they should be capable of interpreting the system’s output correctly and have the discretion to disregard its use, reverse its output, or intervene in its operation, including the ability to halt the system’s operation via a ‘stop’ button or similar procedure.

As for the actors involved along the AI system’s supply chain, Article 16 imposes the obligation to providers of AI to ‘ensure that natural persons to whom human oversight of high-risk AI systems is assigned are specifically made aware of the risk of automation or confirmation bias’, while Article 26 specifies the obligations of deployers, who shall implement human oversight, referring to the requirements laid down in Article 14, and ensure competence, proper qualification, training, and resources for the natural persons in charge of the AI system’s supervision. This is corroborated by recital 73 of the AI Act, which regards as ‘essential, as appropriate,’ that human oversight includes mechanisms to guide the natural person to make ‘informed decisions’.

However, Articles 14, 16, and 26 are directly applicable to high-risk AI systems and do not automatically apply to GPAI, unless such models are deployed in the high-risk scenarios provided for in Article 6(1) or listed in Annex III under Article 6(2).⁹⁶

To close this framework, consider also that high-risk AI systems are subject to transparency requirements under Article 13 and recitals 27 and 72 of the AI Act, which mandate transparent outputs and detailed specifications to enable users to properly interpret the capabilities and limitations of the high-risk systems. Article 13(3)(d) specifies that transparency requirements must first and foremost be met in the instructions for use, which shall also contain information regarding human oversight, mentioning Article 14, including technical tools to allow for output explanations to facilitate appropriate human supervision.

Lessons from the GDPR: the notion of ‘meaningful’ human intervention

Aiming to minimize risks to health, safety, and especially fundamental rights, Article 14 of the AI Act, concerning human oversight, echoes Article 22 of the GDPR.⁹⁷ The former applies when a high-risk AI system provides information or recommendations for decisions to be taken by natural persons, whereas the latter applies whenever such a decision, involving personal data, is based solely on automated processing. For the scope and purposes of this article, the provisions in Article 22 of the GDPR offer valuable insights in the context of LLM-based AI systems, not so much from a risk-management perspective, but rather from a fundamental rights protection viewpoint, by introducing the notion of ‘meaningful’ human intervention.⁹⁸

EU data protection law allows individuals to question decisions taken through automated means that significantly impact their lives. In force since May 2018, the GDPR mandates transparency about the purpose of an algorithm and its decision-making data, along with the right not to be subject to solely automated decision-making.⁹⁹ In addition, Article 22 of the GDPR provides the data subject the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her.¹⁰⁰

From a risk management perspective, the wording of Article 22 suggests two different approaches for automated decision-making depending on the basis for processing.

First, in the cases of solely automated decision-making for the purpose of contract performance,¹⁰¹ or based on explicit consent,¹⁰² it is the data controller who shall implement measures to safeguard the data subject’s rights, freedoms, and legitimate interests. In these cases, Article 22(3) specifies that the data controller shall implement suitable measures to safeguard the data subjects’ rights and freedoms and legitimate interests, at least the right to obtain human intervention [emphasis added] on the part of the controller, to express their point of view, and to contest the decision. In other words, human intervention is regarded as the minimum option for such measures to be considered suitable for the purpose of safeguarding—at least formally—the rights, as well as allowing them to express their views and to contest the decision.

Secondly, in cases of automated processing that is authorized by Union or Member State laws,¹⁰³ suitable measures safeguarding the data subjects’ rights and freedoms are to be regarded as the necessary precondition for the lawful implementation of automated decision-making. In other words, if no safeguards are laid down by Union or Member State laws, or if such safeguards are not suitable to protect the data subjects’ rights, then the data subjects preserve their right not to be subject to solely automated decision-making. Despite not mentioning the right to obtain human intervention, scholars have argued that such measures should nonetheless apply to Article 22(2)(b) for the contestability of decisions made through automated means.¹⁰⁴

Moreover, recital 71 of the GDPR specifies that, in any case, suitable safeguards should include specific information to the data subject and the right to obtain human intervention, to express his or her point of view, to obtain an explanation of the decision reached after such assessment, and to challenge the decision. Additionally, Articles 13(2)(f) and 14(2)(g), depending on whether the personal data have been obtained by the data subject or not, mandate that information regarding the existence of automated decision-making shall be provided, in which case ‘meaningful’ information about the logic involved, as well as the significance and the envisioned consequences of data processing in such context, shall be provided to the data subject.

However, the safeguards provided in Article 22, together with Articles 13 and 14 of the GDPR, may not constitute adequate protection against automation bias, unless the control of the human decision-maker is ‘meaningful’. Such safety measures should be implemented taking automation bias into account, refraining from adopting a blind human-in-the-loop approach as a panacea for all bad automated decisions. In fact, human intervention can act, as it has already been proven, as a mere ‘rubber-stamp’ for the decision that would otherwise be ‘solely’ automated.¹⁰⁵ For further confirmation, the Court of Justice of the EU in the SCHUFA case guards against potentially harmful practices of automated decision-making—in the case at hand credit scoring—where the final decision is made by a third party based on the outputs of the automated processing.¹⁰⁶ Similar cases have been filed in the US.¹⁰⁷

In the context of LLM-based AI systems deployed for decision-making, anthropomorphism, and the tendency to hallucinate may exacerbate the risks of automation bias. Not only hindering the efficacy of human oversight but also rendering the decision de facto solely automated whenever personal data are involved. Although Article 22 of the GDPR may not ultimately be applicable either because no personal data are processed, or because the decision is not formally ‘solely’ automated,¹⁰⁸ it is nonetheless useful as a guiding principle from a regulatory perspective for the interpretation of Article 14 of the AI Act. After all, the ratio legis of Article 22, interpreted in the light of recital 71, is precisely to avoid automation bias and guarantee the right to contest a decision made through automated means, in compliance with the principle of the rule of law.¹⁰⁹

Therefore, for human oversight to be considered meaningful, rather than a mere token gesture, the party responsible for the AI system must ensure the human reviewer has adequate authority, expertise, and information to evaluate and potentially override algorithmic decisions. This includes access to the full range of system inputs and outputs to enable a comprehensive analysis when deciding. This aligns with the Article 29 Working Party (now European Data Protection Board) guidelines on Article 22 of the GDPR, which interpreted ‘solely’ as a ‘lack of meaningful human intervention’.¹¹⁰

Given the challenges faced by both of these EU regulations, to avoid an excess of regulatory optimism in the human decision-maker, and to avoid indulging in techno-solutionism, human factors come into play.

Human factors: the socio-technical realist

For all the considerations above, it is difficult to conceptualize AI risk management without considering both the social and technical dimensions, as well as how these components interact. While automation can lower decision-making costs, it also raises due process concerns about a lack of notice and the ability to contest decisions, especially when private entities design unregulated code that controls certain public decision-making.¹¹¹

Therefore, effective AI governance requires examining not just the technology itself but also its embedding in social systems and human contexts, along with the complex interplay between these spheres.¹¹² I discuss some of these ‘human factors’ in the following subsections.

AI literacy

From a cognitive psychology viewpoint, overreliance may derive from the users’ lack of AI literacy,¹¹³ domain expertise,¹¹⁴ and lack of task familiarity,¹¹⁵ which also determine their attitudes towards AI systems. The level of knowledge regarding an AI system has a significant impact on individuals’ attitudes, as it can lead to either underestimation or overestimation of the system’s capabilities,¹¹⁶ ultimately resulting in the potential misuse, disuse, or abuse of automation.¹¹⁷ Theories of trust highlight that trust-based actions and behaviours, such as the adoption of suggestions, are influenced not only by a favourable perception of the trustee but also by factors like one’s inclination to trust and situational awareness.¹¹⁸

Against this background, it is possible to devise strategies to assess AI literacy during the initial interaction and adjust the subsequent interactions accordingly.¹¹⁹ As an example, showing both correct and incorrect recommendations may help users develop appropriate reliance and critical assessment skills.¹²⁰

To the point, the final version of the AI Act includes specific provisions in Article 4 to ensure a sufficient level of AI literacy of the users of such systems. A definition of AI literacy is provided in Article 3(56): it consists of ‘skills, knowledge and understanding that allow providers, deployers and affected persons, […] to make an informed deployment of AI systems, as well as to gain awareness about the opportunities and risks of AI and possible harm it can cause’. Measures to foster AI literacy should consist of teaching basic notions and skills about the functioning, risks, and benefits of AI systems. As such, according to recitals 20 and 91 of the AI Act, besides promoting skills and knowledge, AI literacy is also viewed as a means to ensure compliance and enforcement of the regulatory provisions.

Appropriate levels of automation (for proper augmentation)

Automation was traditionally included to reduce human error in decision-making.¹²¹ However, one needs to account for the degree of discretion required by the context-specific decision-making process. As such, only routine tasks requiring little cognitive functioning can be subject to high levels of automation, while other cognitive tasks requiring more discretion shall be subject to lesser automation based on the risk level.

This may depend on the role the automated system plays in the decision-making process. For instance, some scholars have presented a qualitative model for automation. Where automation can be applied to four types of functions: information acquisition; information analysis; decision and action selection; and action implementation. Each function can have varying degrees of automation on a continuum from low to high automation.¹²²

Other studies show that people often reduce their effort in group settings where tasks are shared.¹²³ This tendency also appears in HCIs, as computers are increasingly seen as team members.¹²⁴ When tasks are split between humans and automated or computerized systems, people may shift responsibility onto these systems and feel less need to exert strong individual effort.¹²⁵ Thus, in human–AI collaboration, unique human knowledge and effectiveness can decrease over time.¹²⁶

Appropriate augmentation entails deciding which tasks to automate versus those that necessitate human judgment. This requires a more nuanced approach beyond the traditional allocation of tasks between humans and machines according to the so-called MABA–MABA (acronym for Men Are Better At - Machines Are Better At) approach, where humans excel in detecting minimal visual or acoustic energy, perceiving patterns of light or sound, improvising with flexible procedures, storing and recalling large amounts of information over long periods, inductive reasoning, and exercising judgment. Whereas machines surpass humans in responding rapidly and precisely to control signals with great force, performing repetitive routine tasks, briefly storing and completely erasing information, deductive reasoning and computational abilities, and managing highly complex operations simultaneously.¹²⁷

Therefore, there is a need for a new definition of protocols for human–AI system interactions, also accounting for human factors in the design process by directing companies to consider the role of device users, the use environment, and the device’s user interface (UI).¹²⁸ Automation, just as emergent capabilities of LLMs, does not only have quantitative implications. Such as how many tasks are automatable or how much time can the human decision-maker save? It also determines qualitative shifts capable of transforming people’s practice. For this reason, successful automation requires cooperative system architectures that allow strategic human oversight,¹²⁹ grounded in human factors research.¹³⁰

Adequate UI and design choices for human–AI interaction

As noted above, several strategies can promote better understanding and appropriate reliance during regular system use, hence implemented in the UI design to adjust the user experience (UX) accordingly. Some suggest implementing cognitive forcing functions during regular use.¹³¹ However, such an approach shall tailor interfaces based on user motivation levels, which may mitigate this performance gap while still reducing overreliance.¹³² Moreover, providing effective explanations of the system’s outputs,¹³³ is shown to help users develop appropriate reliance.¹³⁴ ChatGPT’s UI, for instance, although providing external sources through Bing, does not directly display those sources, but merely reports the quotation mark throughout the generated text. A better design choice, implementing cognitive forcing functions, would be to display the sources used to enable users to assess their reliability and relevance first, then show the generated response.¹³⁵

The accountable human(s)

The traditional view that either humans or machines hold sole responsibility for errors is mismatched with modern human–AI collaborative systems. Where both contribute to the decision-making process.¹³⁶ For this reason, responsibility should be allocated more diversely, recognizing the transformed assemblages created when AI assists human judgment.¹³⁷ To enable meaningful oversight, minimum standards must grant humans sufficient agency over automated systems, rather than just ‘rubber-stamping’ machine decisions. Truly shared responsibility that distributes accountability across human and algorithmic inputs is necessary to properly scrutinize and regulate these socio-technical decision processes.¹³⁸

From the policy side, several criteria to identify such cases of quasi-automation have been proposed by Wagner.¹³⁹ Based on the amount of time allotted for task performance (‘the lower the amount of time assigned to the human operator, the more likely it is to be quasi-automated’); the operator qualifications (‘the less qualified the individual is to fulfill a specific task, the more likely it is to be quasi-automated’); the degree of liability (‘the greater the amount of legal liability a human operator is assigned for failure; the more likely that humans are engaged in the process merely to ensure they can take liability if the automated system fails’); the availability of support systems (‘higher levels of psycho-social support or other forms of support are likely to be an indicator that speaks against quasi-automation’); the adaptability to operators (‘the more a human operator must adapt to the system, instead of the system being designed to serve the operator, the more the system is quasi-automated’); the access to information; and the degree of human agency to diverge from automated recommendations.¹⁴⁰

In this context, the notion of human oversight is as fragmented. In fact, among 41 policy documents requiring human oversight,¹⁴¹ Green identifies three possible approaches: (i) restricting ‘solely’ automated decisions; (ii) emphasizing human discretion; (iii) requiring ‘meaningful human input’, all of which are in line with the GPDR’s lessons, taught through the safeguards set in Article 22. Nevertheless, he argues that:

[h]uman oversight policies position frontline human operators as the scapegoats for algorithmic harms, even though algorithmic errors and injustices are typically due to factors over which frontline human overseers have minimal agency, such as the system design and the political goals motivating implementation.¹⁴²

Crootof, Kaminski, and Price II¹⁴³ have argued that the original version of the AI Act inadequately addressed hybrid human–AI systems by over-emphasizing the human oversight role, which may overload the human deployer or unfairly assign blame to them for system failures. Rather than appropriately distributing responsibility between the human and the AI components of the decision-making process. Luckily, things may have changed. In fact, unlike the April 2021 draft of the AI Act, where no obligations for human oversight were imposed directly on a deployer, who was only expected to users must simply follow the instruction manual provided by the AI provider,¹⁴⁴ now Article 26 and recital 93 clarify that risks related to AI systems might arise from the design of the system itself, and places—perhaps excessive—burden onto the system’s deployer to prevent violations of fundamental rights.

While it is reasonable, and even desirable, that the end users, acting as the deployer of the AI system under the AI Act, are held accountable for the system’s use, the same holds true for all actors involved in the AI system’s value chain. It is precisely the sense of accountability that creates a common thread among all the humans in the loop for responsible AI innovation and—unsurprisingly—functions as the most effective risk mitigation strategy against automation bias.

Studies indicate that an increased sense of accountability causes highly experienced workers to exhibit greater algorithm aversion.¹⁴⁵ Another study indicates that when operators believe that they are responsible for monitoring an automated system’s performance, they are motivated to employ careful oversight strategies, resulting in fewer automation-related errors.¹⁴⁶ A study specific to AI system designers shows that their perceived accountability for an AI system’s outcomes is directly proportionate to their tendency to design the system to be less autonomous, more interpretable, and more adept at learning to correct errors.¹⁴⁷ Furthermore, it appears that accountability is also regarded as a determining factor in balancing the need for innovation against the need for safety.¹⁴⁸ On the contrary, lesser perceived accountability, encouraged by higher levels of automation, diminishes the moral agency of the human decision-maker and favours the shift of responsibility to the autonomous system.¹⁴⁹ This can be determined at the very human–computer interface level in decision support systems, where human factors must be taken into account as an aspect of risk management,¹⁵⁰ to avoid the creation of a moral buffer, that is a form of psychological distancing, that allows people to ethically distance themselves from their actions.¹⁵¹

Even in the advancements in state-of-the-art NLP, humans will always be in the loop, from the data gathering to the design of the UI and the final decision-making process. Future applications of generative LLMs in decision-making processes are expected to range from the legal sector,¹⁵² the judiciary,¹⁵³ finance,¹⁵⁴ the medical sector,¹⁵⁵ and hiring.¹⁵⁶ This points in the direction of holding all actors involved accountable for how the system was designed.¹⁵⁷ However, humans in the loop may serve different roles,¹⁵⁸ and all the considerations above point towards a clearer definition of roles and responsibilities along the AI system’s value chain.¹⁵⁹ This includes the responsibility of designers of such systems, who would need to account for the risk mitigation strategies against automation bias at a product design level.¹⁶⁰ Specifically, they would need to adopt the known state-of-the-art techniques stemming from the automation literature, such as cognitive forcing functions at the UI, domain-specific fine-tuning of LLMs, or grounding techniques such as Retrieval Augmented Generation. This would have a double beneficial effect of avoiding the shifting of responsibility entirely onto the system under the ‘computer as scapegoats’ barrier, since the human oversight would be practically included by design, on the one hand, and holding the final decision-maker responsible for decisions in which the AI system had a fundamental impact, on the other hand.¹⁶¹

Discussion

Note that my analysis was limited to the current state-of-the-art of generative LLMs and did not consider more specific context-based approaches. Nonetheless, the findings of this article call for a sociotechnical approach that extends beyond the mere balancing of benefits and risks of a new technology. While it recognizes the potential benefits of AI in automating routine tasks, it is mindful of the risks of overreliance and automation bias at a more profound societal level. Achieving the desired balance is crucial for harnessing the full potential of AI in augmenting human decision-making without compromising human agency. For this reason, I envision that future research will focus on developing more dynamic and resilient regulatory frameworks that adapt to the evolving nature of AI technologies, along with in-depth studies on user interactions with AI systems.

Concluding remarks

This article examined the capabilities and limitations of generative LLMs for cognitive automation from a sociotechnical perspective. It highlighted how their remarkable conversational abilities might induce the user to overrely on their output in the decision-making process. However, given their intrinsic tendency to generate incorrect factual outputs, the so-called ‘hallucinations’, alongside the potential for anthropomorphism in such AI systems, I addressed the issue of how to properly calibrate the user’s trust in the AI system, while also accounting for cognitive biases, such as automation bias. After critiquing a purely technical solution based on the principle of ‘human-in-the-loop’, I also showed the weakness of the regulatory requirement of ‘human oversight’. Finally, I introduced and presented a more holistic sociotechnical perspective that also takes ‘human factors’ into account.

If generative LLMs were ultimately to be used for cognitive automation, I argued that an appropriate level of automation shall be determined, ensuring that the user possesses sufficient knowledge of the AI system’s capabilities and limitations, and implementing cognitive forcing functions at the UI level.

Conclusively, instead of emphasizing the role of human oversight or relying on techno-solutionism, accountability along the entire AI value chain shall be implemented, involving the providers of the foundation model, the developers of the HCI platform, and the final user.

Funding

This research was financially supported by the European Union's Horizon 2020 research and innovation programme under the following grant agreements: ‘SoBigData++: European Integrated Infrastructure for Social Mining and Big Data Analytics’ GA 871042 and by the project ‘Biorobotics Research and Innovation Engineering Facilities “IR0000036” – CUP J13C22000400007’.

Declarations

This article was accepted as a non-archival paper and presented at the 2024 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’24), 3–6 June 2024, in Rio de Janeiro, Brazil. The non-archival option is offered to avoid precluding future submission of these papers to discipline-specific journals. Non-archival papers will only appear as abstracts in the proceedings if they are accepted. For further information, please visit the conference’s webpage: https://facctconference.org/2024/cfp.

Conflict of interest statement

None declared.

Footnotes