The foundational 2004 human-factors review arguing that trust guides how people rely on automation, and that the design goal is appropriate reliance on an imperfect system, not maximum trust in it.
Why it matters
Two decades before LLM agents, this review named the problem the behavioral layer still works on: “Automation is often problematic because people fail to rely upon it appropriately.” Both directions are failures. Relying on a system where it is weak and ignoring it where it is strong are the same miscalibration seen from opposite sides.
The mechanism is social: “Because people respond to technology socially, trust influences reliance on automation.” And trust does its heaviest work exactly where understanding runs out: the paper argues that trust guides reliance “when complexity and unanticipated situations make a complete understanding of the automation impractical.” That is the default condition of anyone using an agent today.
Builder read
You are not just shipping capability; you are shipping the signals a person uses to decide when to lean on it. The review’s conceptual model covers “the dynamics of trust, the role of context, and the influence of display characteristics”: what the interface shows shapes reliance, which is the human-factors ancestor of Confidence and Disclosure design.
Trust also forms through more than argument. The paper describes automation characteristics influencing trust “through analytic, analogical, and affective processes,” so an explanation panel is one channel among several: track record, resemblance to familiar systems, and feel all move reliance too. The paper’s stated application is “improved designs of systems that require people to manage imperfect automation.” Every agent is one of those systems.
Exec read
The KPI this paper hands you is appropriate reliance, not user trust scores. A product that maximizes trust in a system that is sometimes wrong is manufacturing the failure case; the goal is users who lean on the system where it is strong and check it where it is not. This review is the scholarly root of the “trust calibration” language now common in AI strategy decks, and it frames trust as something design influences, which makes reliance a product decision rather than a user trait.
Caveats
This is a 2004 review and conceptual model of automation, not an experiment and not a study of LLM agents. It integrates prior research on the automated systems of its era; carrying its claims to conversational, tool-using agents is analogy, not evidence, and the paper itself discusses “the challenges of extrapolating the concept of trust in people to trust in automation.”
The full text sits behind the publisher’s paywall. This note is verified against the journal landing page abstract only, so the paper’s fuller vocabulary for kinds and qualities of trust is deliberately not summarized here.
Source
Primary: Trust in Automation: Designing for Appropriate Reliance (Lee and See, Human Factors 46(1), 50-80, 2004). DOI: 10.1518/hfes.46.1.50_30392.
Related
- Cognitive Forcing Functions - overreliance measured, and one design lever that reduced it
- Human-AI Complementarity - what appropriate reliance buys: teams that beat either party alone
- Trust Scaffolding - the site’s working concept for building reliance deliberately