The Trust Trap

If you have been using AI for any amount of time, you have probably discovered that the relationship is not quite the same as the one you had with a simple piece of software. And despite all the science fiction promises, it is certainly not a trusted colleague or a friend.

Because it exhibits qualities from both of these camps, you cannot relate to it like Excel, and you cannot relate to it like a person, and either way you do not get the most out of it. It can be frustrating, sometimes spectacularly so.

So I have been on a mission to work out how to get the most out of working with AI, and how to do that in a way where you still feel relaxed and in control. Through a mix of observation, research and consulting in industry, I have come to identify some specific cognitive dynamics that hold most of us back from a sustainable, healthy and productive relationship with AI.

I call this the trust trap. Pretty consistently, when I show it to anyone who has been working with AI for a while, they say: yes, that is me.

Start near the top. You get good results, so you trust the AI more. You hand over more, you practise less, you scrutinise less. Soon you begin to lose the ability to tell what merely looks good from what actually is good. Then the results fall over, and your trust falls with them. As you take the work back, your judgement comes back too. You can hold the work to account again, you get good results again, and the cycle repeats.

One of the ways out of this model has been studied since the early days of automation, when people first started to realise that de-skilling was happening. You can stop using the tool entirely. Things do come back, and fairly quickly.

Unfortunately it is not much of an option now. It comes with a certain feeling of guilt, or your boss says you really ought to be using AI, there is a lot of potential here. And if you are only ever in fits and starts with the tool, you are not learning to use it well either.

Two things move as you ride that cycle. Cognitive load is how much of your capacity is being eaten into: your ability to think, to hold several things at once, and to solve complex problems. Cognitive debt is the de-skilling itself, in big ways or small.

The big version is obvious. Use a calculator long enough and you stop practising arithmetic. But there are small versions everywhere. Any time AI sets up a file system for you, creates the directories and names the files, and you let it do the deciding about what goes where, you have to go back and reconstruct what it was thinking before you can check what is in those files. That is a small amount of cognitive debt, incurred just by letting it name and structure things for you.

What I propose, and what the research supports, is that externalisation improves results across all kinds of domains. Simple checklists. To-do lists. The wider idea of distributed cognition, where the artefacts you create and then update over time carry some of the load for you.

And the more you handle the load while staying in control, checking the work and checking it through tooling, the more you ride the good part of the trust cycle rather than the trap.

Four dimensions shook out of this. How much you keep in control, which we call cognitive sovereignty: staying independent of the tool rather than dependent on it. How much you trust the tool, whether or not that trust is always merited. And the cognitive load and the cognitive debt you might be incurring.

So we measured it. We surveyed over two hundred knowledge workers, with questions that map specifically onto load, debt, sovereignty and trust, and plotted the result. There is a continuum, of course. To be able to talk about the dynamics at play, we gave the regions labels: Partnership, Vigilance, Execution and Pragmatism.

One of the findings that was interesting is that these four descriptors are not job role dependent. Coming from a long background in software development, I would certainly have thought that technical roles would have a different posture profile from the general population. That is simply not true.

Across 197 respondents, technical and non-technical knowledge workers spread across the four postures with a largest gap of nine points, chi-square p = 0.46. At this sample size, that is a null result. So this is not telling you what your job is. It is telling you where you land right now, in relation to your trust and your cognitive sovereignty.

We modelled this as a finite state machine: a set of states and transitions across the positions you might occupy inside the trap. As we did, something interesting fell out. Because we modelled a delay in recognising that bad work is bad and that good work is good, and a delay in reclaiming your sovereignty, the model oscillates. It produces a kind of elliptical orbit, an oval shape that will look familiar.

Overlay that orbit onto the plot of the evidence we found and it tracks the same region, the same shape, at about one standard deviation. That is within the core group of where people typically land.

Two things follow. The oscillation is self-sustaining, so with no intervention, cycling is what this model does by default. And the fact that a single simulated worker covers roughly the spread of the whole population begs a question, which we are now testing directly by re-polling the people who took the assessment earlier.

You can run the machine yourself, and see the detail behind it, on the posture machine page.

What this may indicate is that the spread is not permanent to a person. The assessment is designed to give you a sample of a moment in time, of how you are doing now. People may move across these postures over time.

So it helps to treat the postures as neither inherently good nor bad. We always think of up and to the right as the goal, but in reality each of them has a potential dark side. You can cede sovereignty inappropriately. You can lend trust inappropriately. And by finding better ways to relate to the tools, using external methods, systems and approaches, you can mitigate the fundamental downsides of each.

There is a lot more that we are doing with all of this, mapping which kinds of externalisation work best. Because you are not only externalising your own outcomes and the next step for whichever agents you might be delegating work to. The AI itself is also externalising. Beyond its working memory within a session, it needs to create documents and artefacts for itself.

So how do you keep all of that in sync? How do you take enough control of the situation that you and the AI are quite literally on the same page as you work? We are proposing taxonomies for exactly how to do this, and for improving executive function: your own ability to plan and carry out steps in relationship with AI, so that AI is helping you while you retain control and stay out of debt.

We are working on definitions of effectiveness with AI that make sense and that can even be quantified, so you can start to understand both sides of the equation. Yes, doing more. But also staying relaxed and in control, doing the right things, and being able to trust, even though AI is imperfect, that you and the AI together have done those things correctly.

DRIVE is the simple model we use for the phases of working with AI in the loop, so that a bank of best practice becomes something you can apply intuitively, in the moment.

Ultimately the goal is to support people in getting out of this trust trap. To be able to use external tools and approaches, rituals and systems, that they can rely on whether or not AI is reliable.