An interactive model of the sovereignty-trust loop (“trust trap“) in human-AI co-working: what happens to a person’s discernment, and their trust in a tool, when recognising the results of co-working are imperfect and delayed.
What this is
A simulator that assumes trust in the tool is updated from results that arrive late. The ability to judge the tool’s output is a competence which decays (cognitive debt) if discernment is not exercised. Holding on to work costs effort (cognitive load) as does reclaiming it after letting it go.
Findings
Those assumptions turn out to produce a cycle (oscillatory attractor) rather than converging. That is not something we tuned the model to do. From five unrelated starting states, including one beginning in total collapse, it converges to the same orbit, at the same amplitude every time. The cycle is robust and self-perpetuating.
Implications
In the “Trust Trap” we theorise from observation and literature that this loop continues without intervention. Our research is into what interventions most successfully break out of that loop into virtuous cycles, both in terms of output and in terms of the cognitive wellbeing of the human involved. The strongest direction so far is that externalisation, i.e. using a mix of systems and rituals to both keep track of outcomes and states of play outside the human’s head, and the promotion of working-memory information into durable reference on the AI side, will yield the best results on both sides of that system.