How I Tackle Building AI Agents

By day I help clients build custom AI chatbots in the ecommerce space, by night I am creating agents to help with storytelling (e.g. reviewing screenplays I write). Here is how I currently approach designing AI agents.

What is an Agent?

Oooh! Agents! Sounds impressive! So advanced and complex! Before going too deep, a very quick introduction to what an AI agent is, assuming you are familiar with tools like ChatGPT.

Here is an example agent definition:

You are a helpful agent, an expert in the area of Lego.

An agent definition can be that simple.

In practice, you start adding to the agent with additional guidance. For example, “If asked about anything other than Lego, politely reply that you only respond to questions related to Lego.” Or “If you don’t know the answer to the question, do not reply bluntly with ‘I don’t know!’, instead reply with suggestions where they may find an answer.”

We at work, have built agents with hundreds of lines of rules due to the level of control wanted by the client. I am still in the process of building agents for my home hobby project, so I suspect they will grow a bit more before they are ready for prime time!

Further, agents can have connections to other data sources and external systems, then use them to perform some action as requested by the user. For example, to pull up tracking data on an order in response to a request. They can also talk to other agents, so you can build focused agents rather than broad agents, which can improve the reliability of results.

Designing Agents

I normally design agents in close collaboration with clients. The clients know their customers best, so are the best source of information as to what is needed. I normally start with the following steps.

1. Confirm the business problem the client is trying to solve, and ideally metrics to measure the problem, so we can objectively measure if the AI solution when put in place helps.

2. Ask the client what sorts of questions they want answered, and what are good (and bad) examples of responses.

3. Ask the client what data they think needs to be used to formulate the answers. The biggest effort in most AI projects I have seen is data collection.

4. Explore the designed UI. It is now possible to embed UI widgets in chat experiences, for example. Chat is introducing new challenges in UX design.

An Example Screenplay Critic Agent

For my evening hobby projects, I am building a toolchain to help create a AI video generated series on YouTube — multiple episodes with a story running through the episodes. For a screenplay critic agent, my answers for the above questions might be:

1. Problem: Improve the engagement levels of episodes by using AI to review screenplays and identify issues in the screenplay. (Personally, I am not interested in AI writing screenplays.)

2. Q&A: Questions to answer are topics like “Are there any inconsistencies in the sub-plots introduced by recent edits” or “does John talk in the same way throughout the series”

3. Data: The data needed would obviously include the screenplay, but I am also trying to collect notes (seasons, episode arcs, character development beats, and more) and provide that to the AI as well.

4. UI: ChatGPT provides a simple chat interface with Markdown formatting, but many chatbots are starting to explain more visually engaging output. It could be embedded in the UI of a web app, it could have links to other pages so the user can navigate to other views, there are many interesting UI experiences possible. Which is most logical for users needs understanding of the needs. For a writing assistant tool for example, users are used to typing requests, so I am starting with fairly traditional output.

Some people ask “why use chat when we already have great navigation patterns with a mouse”. Its a great question. I support and personally use both. Navigation with menus I do find faster than typing sentences into chat. But I also find more advanced requests are possible, like the above - tell me projects with at least two episodes. The results have URLs I can click on to navigate to that project.

And yes, I did add voice control as well. Why? Because!

My next steps are to start planning out what agents to build, which includes a planning agent. I need to provide them a bit more context first however — data for the agent to leverage.