Lifting the curtain on enterprise AI agents
The illusion of agency and what's really going on under the hood
TL;DR
Most so-called enterprise AI “agents” aren’t truly agentic in the classical sens: they don’t autonomously sense, plan, and act with their own goals. Instead, they’re tightly controlled systems that execute (mostly) pre-defined processes using large language models to manage dialogue and language generation.
Enterprise AI agents are:
Goal-directed, not autonomous; their goals are explicitly given.
Rule-following, not planning independently; workflows are predefined and/or explicit instructions are given.
Tool-bound, not self-directed; actions are limited to approved functions.
The illusion of agency comes from:
Dialogue management (deciding what to say, when, and how).
Contextual responses (sounding natural and adaptive).
This illusion is exactly what makes them effective—they feel smart, but behave predictably. And that’s by design, because enterprises need reliability, consistency, and control.
Ultimately, the value of enterprise AI agents lies not in autonomy, but in delivering better user experiences; experiences that are more natural, flexible, and efficient than legacy scripted systems.
OK, there’s the summary, now let’s get into the details…
Let’s begin with the assumption that you’re familiar with the concept of AI agents. Let’s also assume that this new generation of AI automation has peaked your interest. You might well be considering what relevance this has to your business. Are these systems ready for prime time? How do they actually work? Do they perform better than rule-based NLU systems?
The aim of this piece is to show you how the leading enterprise AI automation platforms are approaching the whole agentic AI thing. It is intended to open your eyes a little as to what agentic AI is, in an enterprise setting, what’s really going on behind the scenes and how, in fact, there’s probably not as much agency going on as you may think… And why that’s a good thing.
For those of you that follow me, you’ll know that I've got a bit of a personal gripe with agentic AI on a semantic level. I’ve written pretty deeply about what the accepted definition of an AI agent is and how the vast majority of what people call AI agents today are nothing like actual software agents.
So what are they, then? Do they sense, plan, act and evaluate autonomously? Or is there something else going on?
What do enterprise AI agents do?
Most of what the enterprise AI vendors are talking about when they refer to agents, is a software program that uses large language models and generative AI to do a combination (or all) of the following:
Understand what a user has said
Marry what was said to either a question or an action (retrieve information based on retrieval augmented generation or route to another agent or program to fulfil the action)
Manage some degree of dialogue, as in, what questions to ask or information to gather to progress through a workflow
Select the right tool that should be used for a given function, based on a list of tools and descriptions on how to execute the action
I’ll unpack each of these phases and dive a bit deeper into the typical components of an agent later. But first, we should consider how ‘agentic’ these things really are.
How agentic do you want your AI agents to be?
Part of the concern that most enterprises have about the agentic AI approach is in the reliability of these things. Will they do the same thing every time? Do you want a new plan to be drawn-up for every user need when you have perfectly good (or at least existing!) business processes already? Will they get things wrong? And how often?
Thankfully, the way to mitigate this, ironically, is to wrap enough control around the AI models as you can, so that you reduce (not remove entirely) the risk of it doing undesirable things.
If I could sum up the general enterprise agentic AI approach so far, it’s trying as hard as it can to de-agentify agents, whilst still keeping enough agency in spirit to be able to call it agentic.
I’ll explain exactly what I mean here, but first, let’s quickly align ourselves on what an AI agent is, in the first place. Then, I’ll show you how enterprise AI platforms are (rightfully so) steadily de-agentifying them in a (so far) successful effort to make them enterprise-ready.
Where’s the agency?
Where the agency comes from in an enterprise AI agent depends on how you define agent. I prefer to use the classic definition of agent, which is:
“A system situated within and a part of an environment that senses that environment and acts on it, over time, in pursuit of its own agenda and so as to effect what it senses in the future”
If you accept this as the definition, then you’d have to say that the current enterprise approach isn’t fully deserving of the full ‘agent’ label, but it perhaps shares enough of the qualities to be considered in some way ‘agentic’.
For example: your agent might exist within a call centre. That’s its environment. It’s listening and transcribing the incoming audio, and attempting to understand it, which is sensing. Then, there’s an action performed, based on the business workflow. So far so good. However, it wouldn’t necessarily have its own agenda, and nor would the impact of actions taken today affect what it’ll do tomorrow (without human involvement).
What about autonomy?
You’ll likely hear the word autonomy used in tandem with the word agency. Autonomy can be defined as the ability to set and pursue your own goals. Many AI providers will tell you that their agents have autonomy, when they don’t. Not really. And that’s a good thing. You don’t actually want autonomy in your AI systems, most of the time.
So let’s have a look at the key capabilities of enterprise AI agents, how they work and determine the degree of autonomy or agency they have (and that you need) to accomplish results.
How enterprise agentic AI platforms work
Most of the enterprise AI platforms that have agentic capabilities work in a very similar way. They consist of Goals, Planning and Actions.
Goal-setting
The starting point for most enterprise AI platforms today is in specifying a goal and (often) a role for the agent to adopt. For example “You are a customer service representative and an expert in mortgages. Your goal is to help the user find the best mortgage deal for them, based on their circumstances.”
The goal-setting component tells the agent what its job is and sets the scene for how it should behave.
Is there any autonomy in goal-setting?
If a system is given a goal, then it doesn’t exhibit autonomy. It exhibits goal-directed behaviour. The vast majority of AI systems in businesses have to have a goal. Otherwise, what's it gonna do? You don't want an AI system coming-up with its own goals, do you?
And so if you have a goal-directed system (which every enterprise AI agent platform I’ve seen to date has), you don’t have an autonomous system. You've taken away a little bit of agency from your agent.
That’s not a problem at all and it’s clearly what you want. It’s just not fully autonomous.
Planning
The next area of concern for designing your agent is in determining how it should accomplish its goal. A typical agent, according to the accepted definition above, will have autonomy over its planning. That is to say that it’ll decide on, and create, a plan of action, on its own, without external or pre-defined guidance.
So, for an enterprise AI agent, if you're going to give it a goal, then do you want it to decide how to solve that goal? Sometimes, maybe. But I would argue that, most of the time, you probably don’t want that at all.
See, if your goal is to do something like change an address, or freeze a credit card, or add somebody onto an insurance policy, then why would you have a probabilistic model decide the best way to do that every single time? Your business has a specific process already, doesn’t it? All you want is to follow that process, don’t you?
In enterprise AI platforms, the planning portion consists of the designer or developer providing instructions for the agent to follow to accomplish its goal. Typically, this would be a written description of a business process, critical information that should be gathered, important logic to be considered and the tools that it should use.
Is there agency in the planning phase?
If you're going to describe what the process is - even if that process is described in language and uses a probabilistic model to navigate it, which most AI agent platforms do - you're still giving it explicit instructions. You’re still giving it pre-determined rules. And if that’s the case, then you're removing another degree of agency.
Again, there’s nothing wrong with that, and it’s exactly what you’d want to be able to do: follow instructions.
Acting
Finally, the thing needs to do something. It needs to take an action. So in terms of acting, an autonomous agent needs to act as it sees fit to accomplish its goal.
For an enterprise AI agent, what most platforms allow you to do is to provide the tools that your agent needs to do its job. In the provision of those tools, you’ll provide a description of what the tool is, how it can be used, when it should be used and what information needs to be collected from the user (or another tool) before it can be used. Most leverage the tool-calling capability of large language models to do this.
How much agency is involved in Acting?
This approach seems to be becoming the standard (until MCP potentially supplants it), but, again, there’s not a great deal of agency involved other than determining when it’s the right time to use the tool. And because you’re providing explicit instructions here about what to do, when and how, you’re removing yet another degree of agency from your agent.
Reliability and risk reduction
If your concern is in the reliability of agentic AI systems, I’m not going to make any guarantees here. I’m not a technology vendor and have nothing to sell you on. The reality is that generative AI is probabilistic and so there’s always a chance that it’ll make a mistake.
However, you might recognise a pattern here, which is that enterprise AI platforms, rightfully so, are attempting to reduce risk by actually removing elements of autonomy, and by manipulating probabilities to promote consistency, like a weighted dice.
Now, they can’t really tell you that because that’s not cool. It’s far better to ride the agentic wave and create the impression that there’s magic going on.
Yet if you look at the way that most of these platforms are structured, you’ll see that there are explicit instructions given at every point in time, with the explicit direction of what should happen when, and what to do if something doesn’t or can't happen.
The reality is, then, that most enterprise agentic AI platforms at the moment are, in a way, deterministic. By that I mean that the business rules have been pre-determined. The risk or reliability issue comes because we’re describing the rules in language instead of code, and then encouraging a probabilistic technology to follow them.
The illusion of agency comes not from what the agent does (follow rules to accomplish goals), but in how it does it.
Let me explain.
Where the real agency in enterprise AI agents comes from
I think that most people think that, because the technology is probabilistic, there must be some agency involved. But the only agency you want, and the only real agency that most enterprise AI platforms provide, is fundamentally that the thing won’t say the same thing every time.
You do want it to do the same thing every time, but you don’t want it to say the same thing every time.
That sounds simple, but the implications of that are pretty stark.
See, when you script every single response in your chatbot, that means that you can only answer questions that you have a scripted response for. That also means that you’ll only plan for, build and allow for conversational pathways that correspond with the responses that you’ve written.
It means that you’ve made big assumptions about what your users will say and the order in which they’ll say them. Generative AI doesn’t suffer from those limitations.
Therefore, the two things that enterprise AI agents do that are giving the illusion of agency are:
Dialogue management, which is the ability to handle the back and forth of the conversation in a natural and seemingly intuitive manner. Here, because you’re not scripting the dialogue line for line, there is a degree of ‘making it up’ by leveraging the LLM’s ability to generate language. Given that what it says isn’t scripted, then you could argue that this falls under the umbrella of agency. Also, part of dialogue management is deciding what questions to ask, in what order, and what to do if there’s a switch of topic, an expandable sequence (like a question asked), or if grounding is required (conversation grounding, or repair, not model grounding). You will definitely give the agent instructions about what information it needs to collect, but you’ll lean on the model to generate the specific questions and handle the back and forth of conversation until it’s collected what it needs.
Contextual responses, which are the ability to respond to users in a way that’s reflective of their previous utterance and the general context of the conversation. It’s closely related to dialogue management, but the distinction is worth noting. You could have an LLM manage your dialogue, but still have pre-scripted responses. Contextual responses include things like mirroring, where you reflect back to the user something they’ve said to show understanding. For example, if a user says “Can I add my grandma to my insurance policy?”, then you might respond with “Sure, you can add your grandma to your policy. All you need to do is…” This shows the user that what they’ve said has been understood and contributes to the natural flow of conversation, giving the allure of agency.
And that’s all it takes. If you have an AI agent that is given an explicit goal, and explicit instructions, with explicit tools, and constrained within an inch of its life, but you let it determine when to ask which questions, how it should ask them, and how it should respond, you’ll find that you have the most fluid and seemingly ‘agentic’-feeling conversation you’ve ever had. Even if the ‘agency’ is an illusion, who cares, as long as the experience works?
The benefits of enterprise agentic AI platform approaches
Therefore, if you want to understand the real benefit of generative AI agents compared to a rule-based, deterministic AI solution, it’s that the generative one ought to provide a much better user experience. It’s as simple as that.
A better experience means a less effortful journey. A less effortful journey means a higher probability of success. And a higher probability of success means a greater chance of ROI. And isn’t that the whole point?
Your AI Ultimatum
So where does that leave us? With a better understanding, hopefully, that what’s being sold as agentic AI in the enterprise is often a carefully structured orchestration of deterministic rules wrapped in probabilistic language.
For those investing in agentic AI, the real question isn’t how autonomous is it, it’s how valuable is it? And how good is the experience?
Here’s your AI Ultimatum, then: how long will you stick to your pre-scripted, assumption-based experiences? How long will you battle with conflicting intents and overfit ML models before trying something a little different? I’s wager it’s only a matter of time.