AI Agents Shouldn’t Go Where Humans Cannot See

Since my original worries about agentic AI in October last year, advances in their capabilities have been prolific. Tools such as OpenAI’s Codex, Anthropic’s Cowork, and open-source projects like OpenClaw are bringing powerful agentic capabilities within the reach of non-developers. Many are enthusiastically heralding the arrival of a golden agentic age.

My previous advice was always to keep a human-in-the-loop and ensure there is a kill switch, but, as I will explain, this is now impossible in the very scenarios that need it most.

After six months – a long time in AI – my guidance needs an upgrade.

The twin perils of complexity and consequence

Not all agents are created equal, and I find it helpful to think of them in two dimensions: complexity and consequence. In this scheme, complexity is closely related to whether a human can understand an agent’s operations in a timely way, and consequences are defined by what it is capable of doing.

Until recently, most agentic deployments were low-complexity and low-consequence. For example, agents for personal productivity live relatively benignly in mailboxes and file servers, where, even if they delete or alter important items, there is usually a recovery path (even if it is an embarrassing manual apology).

Their counterparts powering internal corporate processes are gaining traction at higher levels on both scales. Nonetheless, they tend to remain outside core operations and away from customers. Most problems, if they occur, never breach the company perimeter.

Even narrowly scoped external agents for automated sales and service generally exhibit manageable complexity and tend to escalate to a human operator long before serious consequences occur. That said, poorly designed offerings in this space have already invented policies, offered disproportionate refunds, insulted customers, and resisted escalation safeguards.¹

However, my real concerns start once agentic AI takes root in core operations, where it becomes interconnected with third-party systems. This is when complexity and consequences begin to increase rapidly.

Activities such as orchestrating payments, triggering actions in the physical world, making important commercial decisions, or providing regulated advice sit at the very top of the consequences scale – yet all are already under development by major companies.

Some may argue this is old news. SaaS systems have been doing similar activities for years, so AI agents are just an upgrade to something tried and tested.

However, as I have argued previously, traditional software is deterministic and bounded: once tested, behaviour remains predictable even across complex systems. Generative AI is different. It is probabilistic, and identical inputs can produce different outputs. When multiple agents interact,unpredictability compounds, which is why human oversight is so vital.

And so, we reach the heart of the problem: from a human-in-the-loop perspective, the most advanced AI agents now exceed our cognitive capacity to oversee them. Their opacity,² speed of operation, volumes of data, and interactions with unknown third-party entities (often other agents) are simply beyond our ability to comprehend then act.

In this context, “the loop” can only be decoded by another machine – we can no longer truly be part of it.

Elevating humans above the loop

In analogous situations, complementary systems have been developed to monitor the loop and report out to humans in ways that give them back control. Think of safety systems in factories and security applications on laptops that translate incomprehensible system interactions into warnings that can be acted upon. Humans are repositioned from being “in” the loop to being “on” (or above) it instead.

The problem is that AI labs have delivered sophisticated agentic capabilities before the control technologies required to manage them. In history, we have seen the devastating effects of this latency in aircraft design (e.g., Air France Flight 447), nuclear power plants (e.g., Three Mile Island), and algorithmic trading (e.g., the Flash Crash) – to name just three chilling examples.

The safety cavalry is coming (slowly)

The good news is that automated control innovations that put humans back in charge are in the works. These include automated systems that detect deviations from an expected “behavioural signature”; self-auditing agents that provide transparency; LLMs that monitor other LLMs (aka “LLM as judge”); and finally, “watermarking” to ensure behaviour can be traced to a specific identifiable agent should something go wrong.

Of these, monitoring agents are now a serious prospect, and I have first-hand knowledge of encouraging results with “LLM as judge”, but the other innovations are earlier in their lifecycle, and I think we are at least a year away from where we need to be.

You can’t go where you can’t see

My (very patient) driving instructor used to say, “You can’t go where you can’t see”. He was talking about sharp corners and blind elevations, but his advice is also good for business leaders considering deploying consequential agents.

More concretely, I would suggest:

1. Begin with simple agents focused on personal productivity. The workbench is often available in existing corporate software

2. Progress to internal back-office processes to achieve scale. This step has a significant potential payback and may be as far as many companies want to go at this stage

3. Consider tightly scoped sales and service agents with robust escalation

4. Only move into high-complexity, high-consequence agentic development when the business case is compelling and even then, only with a very experienced team

Agentic AI is rapidly becoming the productivity unlock we have all been waiting for. However, the common assumption that a human-in-the-loop ensures safety is now obsolete for high-complexity, high-consequence agents. In these cases, machine oversight is needed to re-empower humans to govern. Adapting the common nomenclature, this can be described as “human-on-and-machine-in-the-loop” – HOMIL.

In this context, the leadership challenge is knowing when to accelerate – and when to pause until governance technology catches up.

For most companies, the Roman maxim “festina lente” (“make haste slowly”) may be the right agentic philosophy for now.

If you'd like to listen to an audio version of this article, please click here.

Footnotes

1 These are all things people do too, but we generally have mature governance methods to handle human failures.

2 Arising from the internal structure of large neural networks and their high-dimensional reasoning processes.