Skip to content

Why Current AI Systems are not very good to work with

In this post, we will explore how to think about Humans working with AI systems.

There has been a lot of discourse as to how humans are supposed to work with AI systems. Since the introduction of Large Language Models (LLMs) and AI Agents powered by LLMs, AI systems have been introduced as collaborators in different jobs. In this post I will be using software engineering as my main example, but I believe this applies to other fields as well.

The current model of Human-AI collaboration looks like this

graph LR
    Human -- instructions --> Agent -- action --> Workspace;
    Human -- action --> Workspace;

and sometimes like this, in the case of vibe coding

graph LR
    Human -- instructions --> Agent -- action --> Workspace;

In either case, having a workspace where the artefacts of the collaboration live is an important element as it allows for any actor to augment it at any point in time. A real example of this would be a software engineer and an AI agent working together on the same codebase

graph LR
    SoftwareEngineer -- instructions --> AIAgent -- action --> Codebase;
    SoftwareEngineer -- action --> Codebase;

At a later time, there could be more or different software engineers or AI Agents working on the same codebase.

graph LR
    SoftwareEngineer1 --> AIAgent1 --> Codebase;
    SoftwareEngineer2 --> AIAgent2 --> Codebase;
    SoftwareEngineer1 --> Codebase;
    SoftwareEngineer2 --> Codebase;
    SoftwareEngineer3 --> Codebase;
    AIAgent4 --> Codebase;

At a later time, we could even pass the workspace over to some other party like a program manager, the point is that having the workspace in the setup makes this possible.

The most important thing during a collaboration is the ability to intervene. This model makes that possible, but variations of the model could have different levels of intervention. Although all variations of the model technically support intervention, the capabilities of the actors to empathise greatly limits the practicality of it.

To illustrate this point we would look at a variation of the model where the Agent is a code generation tool.

graph LR
    SoftwareEngineer --> CodeGenerationTool --> Codebase;

In this model it can be very problematic to intervene later as code generation tools that were not built with intervention in mind do not produce code that is easy for humans to read. An example is minified JavaScript code that has variables, functions and classes with single character names.

// Example of normal javascript code
function calculateSum(firstNumber, secondNumber) {
    const result = firstNumber + secondNumber;
    return result;
}

const finalValue = calculateSum(10, 20);
console.log("The sum is: " + finalValue);

// Example of equivalent minified javascript code
function a(n, t) {return n + t}const v = a(10, 20);console.log("The sum is: " + v);

This is where empathy is important, even though the engineer can still technically intervene, sometimes their ability to actually do so is limited because the artefacts within the workspace where not created with them in mind.

It is important to constrain agents to keep Humans in mind when generating artefacts in the workspace because it enables intervention. Intervention is what makes collaboration possible, and collaboration is important because of ambiguity of requirements. The point of this model is to get work done, the work that needs to be done is usually described as a set of requirements. The main problem is that requirements are not usually well-defined.

Requirements describe something that is yet to exist, and it is hard to describe the requirements fully without creating the thing you wish to describe. Basically, requirements fundamentally have less information than the thing they are describing, and the process of executing them bridge this gap by leveraging the experience of the actors. For example, Product owners leverage the experience of the Software Engineers to build products that don't exist in the forms that they imagine them.

This means for any task there would always be requirements, and there would always be ambiguity in those requirements. So the only way to bridge that gap is to leverage the experience or intelligence capabilities of the actors. This means requirements will almost always be subjective and almost impossible to get in one shot.

This leaves us with only one way out, iteration. To disambiguate the requirements, they need to become living documents that clarify ambiguities as they come up. And the actors within the model need to be able to take in new information as they come up to make better decisions within the workspace

graph LR
    SoftwareEngineer <-- instructions --> AIAgent -- action --> Codebase;
    SoftwareEngineer -- action --> Codebase;
    TaskOwner <-- feedback --> Requirements;
    Requirements <-- feedback --> AIAgent;
    Requirements <-- feedback --> SoftwareEngineer;

The arrows in the diagram indicate iteration. The feedbackloop on the left side inform the actions on the right side.

This means the model is an online learning environment where new information could come up at any time, and the actors need to integrate it into their understanding so they can make better decisions. Note that the requirements changing over time can make the feedback a non-stationary distribution. The AI Agent is in a peculiar position as it needs to factor in feedback from the Requirements and input instructions from the Software Engineer.

So where does this leave us? We have a good model of collaboration with AI systems but their ability to collaborate is heavily dependent on how good they are at learning on the fly. Everything I have described above are insights derived from Human-Human collaboration settings, an AI Agent should essentially be like another human collaborator with extra features. I hope this post makes clear why current AI systems as of March 2026 are severely limited.

Summary

  1. We described a model of Humans and AI working together to do some work
  2. We noted that even though the model technically allows for intervention it can sometimes be limited by the actor's capabilities
  3. We made a case for why intervention is important because of the ambiguous nature of requirements
  4. We concluded that the only guarantee to disambiguate requirements and get the desired outcome requested is through iteration
  5. We also concluded that the effectiveness of the collaboration is dependent on the online learning qualities of the actors

Thanks for taking the time to read this post. See you in the next one. Bye.