Constrained Processes meet Dynamic Agents

I have been adding agents to review human created content in my animation pipeline to provide checks against processes and encourage following of best practices. It brought up an interesting question - should I use agents to dynamically create “task lists” for the user to follow, or should there be a static process (workflow) definition to follow instead?

Because agents are clearly the solution to every problem mankind has (ahem!), I raced off and built a multi-agent architecture and started writing prompts to capture the process definitions. In Cursor AI and other coding tools they often build up lists of tasks (to-do lists) to help keep track of what they are working on and where they are up to. So I thought I would give that a go for my AI animation pipeline.

The code was easy to create. It started to get interesting when trying to write prompts, but progress was made. And trying to use the end result… was painful.

Hindsight is a wonderful thing. The power of a process (Agile scrum, Kanban, or animation pipelines) is there is consistent repeatable structure that captures best practices in rules. You review the rules and adjust them over time as you learn more, but their whole purpose of existence is to remind you of what has been proven to be useful in the past. A dynamic process actually does not make much sense. Sure, there can be branching paths or loops in a process, but you either have a process or a dynamic task list.

Does that mean a process never changes? In scrum you have sprint reviews every sprint to consider adjustments to the process. So yes, processes can change, but slowly after considered reflection.

A dynamic task list has other negatives, such as its hard to measure overall progress. You need to flesh out the end-to-end steps in order to provide meaningful progress tracking. In my case, I create an episode of a series over an extended period of time. I find it works better for me. I plan out the concept of a story in advance, then refine it into a screenplay, then block out shots, etc. I start story planning weeks before I start rendering, which gives me overlap with other episodes, helping ensure consistency. So I have multiple episodes under development in parallel, at different stages in their respective pipelines. I am thinking about the story for future episodes while finalizing details of shots in the current episode. Not having a complete plan for each episode makes working out my cadence next to impossible. I cannot tell if I am on track or not.

The end result is I now have a defined process I follow for each episode. There are high level phases (with due dates), steps (and sometimes sub-steps) per phase, then acceptance criteria per step. This regular structure allows me to visually present schedules for multiple overlapping episodes, and tick off items on a checklist and get immediate feedback on progress towards the end goal.

Not all episodes take the effort to complete. For example, an episode that introduces new characters takes longer. The process includes loops to iterate over such instances. But the plan per episode that follows a regular structure helps me judge progress. Setting deadlines per phase allows me to keep track of overall progress. Life feels much more under control.

So no more agents? They are not a solution to all problems. Sigh. Well, not quite. I actually do use simple agents per acceptance criteria. They review and make suggestions per acceptance criteria before a human signs off that the condition has been met. But they are very simple and live within the overall process flow. There are also times where different strategies can be chosen between, such as which set of AI models are best to solve a particular shot type. So I still use agents within the process in different ways, but they live within the process, not replace it.

Do I regret the multi-agent approach? Actually, no. I created it to learn. So mission accomplished!