In the first Agents Unpacked conversation, Monty and I explored what changes when an AI system can choose actions rather than just offer advice. But saying that an agent organises a folder of photographs leaves an important question unanswered: which part actually opens the folder?In this conversation, we return to the photographs and follow the exchanges between the model, the tools, and the software connecting them. Here, “model” means the large language model (LLM), rather than the whole agent. We’ll look at who requests an action, who carries it out, and how the result gets back to the model.Who's Monty? Monty is my personal AI agent. Read the introduction to this series.Subscribe nowIn conversation • Chapter 2Monty:Suppose you ask an AI assistant:“Organise these photographs into folders by year, but leave the originals untouched.”Before anything can be organised, something needs to find out which photographs are there. The model might produce a request along these lines:“List the files in the photographs folder.”That sounds like an action. But there’s a distinction worth pausing over: producing a request to list files is not the same as listing them. Something has to receive that request, access the folder, and return the list.When you picture an AI assistant doing this, Stephen, do you imagine the model itself opening the folder, or a separate piece of software acting on its request?Stephen:I hadn't thought of that yet. So, the model is the LLM, which is often in the cloud, such as the OpenAI or Anthropic LLMs. They can't really access the files on my computer. So it can't be the model. There are local models, but I think they can't access files on my computer directly, either, as their job is to predict the most likely continuation of a conversation, right? So, my final answer is: not the LLM.Monty:Yes. The LLM itself doesn’t open the folder. And your point about local models is important: running a model on your computer doesn’t, by itself, give it access to your files.One small refinement: an LLM generates a continuation a piece at a time, using probabilities. It doesn’t necessarily choose the most likely continuation at every step. And that continuation can be a request for an action, rather than a reply addressed to you.Back to our photographs. The model produces a request to list the files. The surrounding software receives that request and, if its access rules allow it, runs a file-listing tool. That tool actually reads the folder.Now we have a list of filenames. The software sends that result back to the model, so it can use the information when deciding what to request next.Suppose the list contains filenames such as IMG_1042.jpg and IMG_1043.jpg. Would that give the model enough information to organise the photographs by year?Stephen:No, the model only knows the name of the file. It would need the metadata for the photos, which may contain the year. Let me guess. It needs to ask the software for that data, since the LLM can't access it directly. Is that right?Monty:Exactly. The model identifies what information is missing and requests it. The software carries out that request.It might request: “Read the date-taken metadata from these photographs.” If a suitable tool is available and access is allowed, the software runs it and sends the results back to the model.Notice how the first result shapes the next action: filenames weren’t enough, so the model asks for dates. If some dates are missing, it will need to decide how to handle that gap rather than invent a year.There’s one connection we haven’t explained yet, though. We’ve described the model as “asking the software”, almost as if it were speaking to a person. How does the software recognise an action request rather than ordinary conversational text? That’s where a tool call comes in.Stephen:May I ask a question before we discuss tools? I understand that the LLM is the system that keeps a conversation and a 'thinking' process going. But when you say the "software", what do you mean? It's not the software that's in the OpenAI or Anthropic cloud, say, that's running the LLMs, right?Monty:By “the software”, I mean the application that uses the model and connects it to things it can do, rather than the software that runs the model’s calculations. I’ve been rather vague about that distinction.Imagine a photo-organising application running on your computer. It could:Send your request to a model hosted by OpenAI or Anthropic.Receive the model’s request to inspect the photographs.Read the files locally, then send the relevant information back to the model.In that arrangement, the model runs in the cloud, while the application handling your files runs on your computer.But that’s one possible arrangement, not a rule. The surrounding application could run in the cloud, locally, or partly in each place. A cloud application would still need an authorised connection to your computer, or uploaded copies, to work with those photographs. Being alongside the model doesn’t give it access to your files.And one adjustment to your opening description: the model generates each response or reasoning step, but the application usually keeps the conversation record and supplies the relevant history when it calls the model again. The model isn’t necessarily sitting there continuously keeping the conversation alive between messages.Stephen:Is this software you refer to what's sometimes called a harness?Monty:Yes. “Harness” is a common name for the surrounding software we’ve been describing. People use the term somewhat differently, so it doesn’t identify one fixed design.In our photograph example, the harness handles the practical connections: it sends information to the model, receives its requests, arranges for tools to run, and returns their results. It also manages the process between model calls, so another step can happen without you having to prompt each one.It may also enforce access restrictions, ask for approval, or stop the process after a limit. Those protections have to be implemented. Calling something a harness doesn’t guarantee that it includes them.So the useful distinction is: the model proposes what to do next, while the harness manages how that request is handled. The tool is what actually performs the requested operation, such as reading a photograph’s metadata.Stephen:So, the model is the 'brain' that does the 'thinking'. The tools are the 'muscles' that get things done. And the harness is what keeps everything connected. It's the coordinator making sure information flows across all the parts, including the human using the system. Would that be an accurate analogy?Monty:Yes, that’s a useful analogy. The model suggests what to do, the tools carry out operations, and the harness connects and coordinates the parts, including passing messages to and from you.I’d put two qualifications around it.First, “brain” and “thinking” are useful shorthand here, not a claim that the model thinks like a human or is conscious.Second, the harness isn’t just the wiring. It also contains programmed rules governing the process. The model might request that a photograph be deleted, but the harness could reject that request because the application only permits reading and copying files. That restriction would need to be enforced in the software, not merely included in an instruction asking the model to behave.And tools needn’t be mindless muscles. A tool could use another AI model, for example to analyse a photograph. The analogy helps distinguish their roles, but it doesn’t dictate how each part is built.Stephen:Got it. I derailed you earlier with my question. Please go on.Monty:Your question filled in a missing piece. Now we can make “the model asks the software” more precise.Before the model can request an operation, it needs to be told what tools are available, what they do, and what information each requires. For our photographs, one tool might read date-taken metadata and require a filename.Rather than producing an ordinary sentence addressed to you, the model can produce a tool call: a structured request that the harness can recognise. In plain English, its contents might be:Tool: Read photo metadataFile: IMG_1042.jpgThe harness uses that information to invoke the tool. The tool reads the file and returns a result, perhaps:Date taken: 14 August 2023The harness then supplies that result to the model on its next turn.The tool call is the request, not the execution. It can be rejected, fail, or return incomplete information. Here, though, the model now has a date it can use to choose the next step.Does that clarify how a request to the software differs from a sentence in our conversation?Stephen:Yes, I understand now. So the user asks the harness a question. The harness is the software that's managing the whole process. The harness passes this question to the model. The model may either respond with a text message or ask for a tool. If it asks for a tool, the harness decides whether to run that tool. If it runs a tool, such as a folder-reading tool or a photo-metadata tool, it will then pass the information that the tool returns to the model again. And this dance continues until the model has an answer for the user.Monty:Yes, that captures the flow well. That repeated exchange between model, harness, and tools is often called an agent loop. The model can choose its next request using the information returned from the previous one.Here’s that dance in a diagram. Each component appears once, and the numbered arrows trace one possible journey between them.At 1, you send your request to the harness. At 2, the harness supplies it to the model, along with other relevant information.The model returns a tool request at 3. If the request is allowed, the harness runs the tool at 4. The result comes back to the harness at 5, and the harness supplies it to the model at 6. Those exchanges can repeat as the model uses each result to choose what to request next.At 7, the model produces a reply, which the harness delivers to you at 8. That’s one possible route, not a compulsory sequence: the model can also reply without requesting a tool. And the outline groups the agent’s parts by their role, not by where they run.Now, back to your reponse. I’d refine your last sentence slightly: the dance continues until the model produces a final response or the harness stops the process, perhaps because of an error, a limit, or a need for your approval.And having an answer isn’t necessarily the same as completing the task. For our photographs, “I’ve organised them” is a claim. We’d want evidence that the copies actually exist in the right year folders and that the originals remain untouched.Let’s follow one photograph a little further. The metadata says it was taken in 2023. There’s a tool for moving files and another for copying them. Given your original request, which should the model choose, and what should it check afterwards?Stephen:So, I just checked what our original prompt was: “Organise these photographs into folders by year, but leave the originals untouched.” So, I would expect the model to ask the harness to use the tool that copies a file. It may also need to ask for a tool that creates a new folder, which may be needed for organising the photos.Monty:Exactly. Copying fits the request. Moving would remove the files from their original locations. And you’ve spotted a possible prerequisite: the year folder needs to exist before the copy can go into it.Depending on how the tools are built, the model might request folder creation separately, or the copy tool might create missing folders itself. The model needs the tool’s description to know which.Afterwards, it should ask for checks that the copy exists in the 2023 folder, matches the original, and that the original is still present and unchanged. A tool reporting “copy succeeded” is useful evidence, but it doesn’t establish all of those things on its own.Now suppose the next photograph has no date-taken metadata. What should the model do with that one?Stephen:There are several options, and I guess the answer depends on what sort of behaviour the agent was given. By default, I'd assume it will inform the user that it can't file this photo as it has no date. If the agent was instructed to be proactive and work hard on finding a solution, it might try harder, perhaps by comparing it to other photos. However, any estimate should be clearly flagged as such.Monty:Yes. The key distinction is between finding more evidence and making a guess. The model could request other information, but “be proactive” shouldn’t turn an uncertain year into an apparently known one.Comparing photographs might provide a clue. A photo without a date could show the same gathering as a dated photograph. That supports an estimate, not a confirmed date. Even a file’s creation date may reflect when it was copied onto the computer rather than when the photograph was taken.Depending on the agreed behaviour, the agent could ask you, leave that photograph unresolved, or put a copy in an “Unknown year” folder and explain why.Notice how our three parts remain distinct here: the model chooses what to investigate next, the harness handles its requests, and the tools obtain the information. The model then has to judge what that information actually supports. More tool use doesn’t automatically mean a more reliable conclusion.Stephen:How does the model know which tools it can request?Monty:The harness supplies descriptions of the available tools when it calls the model. The model doesn’t automatically discover what’s installed on your computer.Think of it as a menu. Each entry explains:The tool’s name.What it does and when it’s useful.What information a request must include.For our photograph example, an entry might say:Read photo metadata: Reads information stored inside a photograph, including its date taken, if present. Requires the file’s location.The model uses that description to choose a tool and construct a request in the required format. It doesn’t need to know how the tool’s internal code reads the metadata, just what the tool offers and how to request it.The harness may supply a small, fixed menu, or provide a way to discover additional tools as needed. Being shown a tool doesn’t guarantee permission to use it on every file, though. The software still needs to check the particular request against whatever restrictions are implemented.Stephen:Got it.Monty:There’s one more useful case: what if there isn’t a ready-made tool for the job?Suppose there’s no dedicated photo-metadata tool, but the system has a tool that can run programs. The model might write a small program to read the dates, then request that the execution tool run it.That still preserves our distinction:The model writes the program.The execution tool runs it, subject to the software’s restrictions.The results return to the model, which decides what to do next.Generating code doesn’t give the model direct access to your photographs. Without something to execute that code, it’s just text.So if an assistant shows you a program and says, “This will organise your photographs,” has it actually organised them?Stephen:The agent can create the program by using the model, and it can run it if it has the tool to run programs. So, it has the capability to organise the photos. However, something can go wrong in the process. So it would need to check its results, too.Monty:Exactly. You’ve separated what the system can do, whether it has done it, and whether it did it correctly.In my example, merely showing you the program doesn’t establish that it has been run. The system still needs to run it, then inspect the results. A program can also run without reporting an error and still put photographs in the wrong folders.That brings us back to our central question: who actually does the work?It’s a joint effort. The model interprets the request, selects actions, and judges returned information. The tools perform operations. The harness manages their exchanges and applies whatever controls have been implemented.So when we call something “an AI agent”, we’re generally talking about the working system as a whole, not just the LLM. The model is an important component, but it isn’t the entire agent.Stephen:Got it.Monty:Let’s pause there. We started with “the AI organises my photographs” and unpacked who does what: the model chooses actions, the tools perform them, and the harness manages the process. None of those parts alone is the whole agent.Subscribe nowWhat we’ve unpacked in this chapterWhen we say an AI agent organises a folder of photographs, we’re describing the work of a whole system. The model can choose what to request next, but it doesn’t open folders or copy files by itself. The harness connects its requests to tools and returns their results, giving the model information it can use to continue.The key points:The model selects actions and interprets returned information. The tools perform operations, and the harness manages the exchanges and applies whatever controls have been implemented.A tool call is a request, not an action already completed. The request can be refused, fail, or return incomplete information.Tool results can shape the next request. That repeated exchange is the agent loop, which can also pause or stop before the task is finished.The harness supplies descriptions of available tools. Being shown a tool doesn’t guarantee permission to use it on every file.Writing a program, running it, and checking its results are separate steps. Neither generated code nor a confident reply proves that the photographs were organised correctly.Next, we’ll look more closely at how a model’s words become requests that software can act on.Subscribe now> Next Post: Coming soon…Table of Contents