Ordinary Animator for ComfyUI Users

Ordinary Animator is designed to help novice storytellers construct a story then publish it as a video series, on a platform such as YouTube. It includes a range of AI tools to review and advise on content, as well as leveraging ComfyUI for image and video generation.

Why ComfyUI? Because I find in order to convey deeper emotions and mood, you occasionally need greater control, beyond what more closed options make possible. Particular shots may need special tools or workflows, and ComfyUI offers a huge range of models all on the one platform, with deep access to tweak settings when needed. And as a side bonus, ComfyUI can be run on your desktop (with many free models available) or in the cloud on 96GB GPUs.

This blog focuses on the ComfyUI touch points rather than storytelling support. It walks through a number of Ordinary Animator features that may be interesting to existing ComfyUI users.

Note: Ordinary Animator is an evolving work in progress, not a product today. I have been building it for myself, but am exploring if it could be useful to others as well. Feedback welcome!

Cloud and Local Connections

Ordinary Animator works by collecting metadata about your project (such as character descriptions) and then using that information to submit AI image and video rendering jobs to a cloud or local ComfyUI installation. Ordinary Animator can connect to your Comfy Cloud account given an API key, or to a local installation by installing a Chromium browser extension (such as Google Chrome or Microsoft Edge) to deal with web browser security issues.

Modern web browsers include security checks to stop malicious access to local resources on your computer. However, Ordinary Animator wants to be able to connect to your local ComfyUI installation. A simple browser extension forwards requests from ordinaryanimator.com to your local ComfyUI service on port 8188. This allows Ordinary Animator (and no other domain name) get API access.

You can control which ports it uses to reach the local ComfyUI service.

Installing new Model Files

I run ComfyUI locally and in the cloud using Comfy's cloud offering. I frequently use my local machine for experiments and tweaking but lengthy video generation jobs do push the machine, so the cloud offering comes in handy. I also wanted Ordinary Animator to be used by users who did not (or could not) run ComfyUI locally.

So, instead, I store all input and output media assets in cloud storage folders ("media galleries"). When a user tries to run a workflow locally or in the cloud, I copy the media asset to the required file system if not already present. I then watch for output files created by ComfyUI jobs and save them (with the workflow) back in cloud storage.

The end result is users can flip between the cloud and local at any time to run a workflow. All media file movement is performed on behalf of users. And users can preview all of the generations along with their generation history.

Model Selection

Once ComfyUI is connected, the next step is to decide which models you want to use in your project.

Most users don't want to tweak settings - they want to get on and tell their story. So Ordinary Animator uses AI to help users both structure a story AND the visual presentation, recommending models and workflows to turn that story into life. To support this, I am building up a database of knowledge base articles on what models are best to solve common problems. There are many people investing a lot more time exploring models than I can afford, so my articles tend to summarize and aggregate information discovered and shared by others (with acknowledgements). There are a lot of bright, creative people out there! This database will keep growing over time as new content becomes available.

Here is an example chat session. Notice that the responses use knowledge of the current project settings, not just what is available. Ordinary Animator actually includes GenAI AND 3D model support (although most of the 3D support has been temporarily disabled). I started work on Ordinary Animator using 3D models and animation (hence "Animator" in its name) with GenAI for background skyboxes, but more recently I have been focusing on GenAI for all aspects of videos. 3D is still interesting however, but now for posing models to feed into AI image generators.

The database is then hooked up with a LLM (I am currently using a combination of Gemini and Claude models hosted on Google Cloud infrastructure). Many of these articles I plan to turn into blog posts as well. See the chat on the right guiding the user through available models.

The end result is you can sit down and ask the chat session recommended models to achieve your desired goal. "I want to create an anime series using live motion recordings for reference".

Local Model Installation

One side feature I added, since I was building up a knowledge base, was also keep track of what files different models need. This includes model checkpoints, VAEs, LoRAs, text encoders, etc. If you pick a model that is currently not installed locally, if I managed to track down the original source of the files it works with, I offer a single click download and install into the correct directory for the user. Chromium based browsers include a "File System API" allowing the web app to ask for permission to access a local directory, and if granted it can install files into that directory. So the user does not have to work out the destination, or move file afterwards - they just click and the file is downloaded directly into the correct directory. (The user must browse to the 'models' directory once so it knows where to put files under. After that it remembers that directory.)

To use it, click on the download button on the model cards, select the "models" folder on your computer (it is remembered if already set).

It will show which files are needed and which are already available locally.

Turning on a model or LoRA then clicking Open Settings takes you to a page where you can see enabled models when menu items to download the missing files, or search for the same file in a different local directory.

Workflow Selection

Ordinary Animator has the concept of a workflow library per project. Projects can be shared between users, so all assets for a project are stored in the projects. Ordinary Animator provides a library of prebuilt workflows (the library is still being fleshed out), plus an AI Agent using Anthropic's Claude models for editing ComfyUI workflow definitions. Instead of pre-generating all possible workflows, an existing workflow can be adjusted by an agent to add a LoRA to be loaded.

The end result is a project can use provided workflows, or a project can load up custom workflows to be used on that project (including those generated by AI agents).

Recipes

Ordinary Animator puts another concept on top of workflows, called recipes. A recipe is a sequence of operation steps, where running ComfyUI workflow is a common type of operation. Other operations include built in interactive operations like cropping images with a mouse.

The chips in the top right corner of recipes are internal scores kept for different recipes. If you star a media file (indicating you like it), the steps used to create the image are given a boost automatically behind the scenes in the cookbook. The more a recipe produces useful output, the more often it is recommended.

Asset Management

As mentioned above, inside Ordinary Animator there are "media galleries" (directories) for storing media files to group related files together. For example, a directory is created for each character, location, prop, scene, and shot in a project. I sneak a bit of metadata into workflows so generated files when stored with their workflows know which media gallery they belong to. For example, you might create a close up of a face, then out-paint to a full body shot, and finally create a character reference sheet. The media gallery retains the fully journey the user went through, with the original workflows used.

By saving all the workflows, input file references, and outputs, Ordinary Animator can show the history used to create a file, with all generations. This can be useful to reload later, for all media types. Ordinary Animator uses the same approach as Comfy Cloud, using hashing the contents of media files to form their filename, meaning any duplicate is automatically identified and slotted into the dependency graph.

The dog on a motorcycle with removed background, when loaded as a recipe, restores all the steps that were used to create the media file. This allows the previous workflows to be rerun if desired, or modified. The recipe is shows as a series of steps on the right.

If you are familiar with ComfyUI, you might know that PNG files retain a copy of the workflow inside the output image. This is not used by Ordinary Animator. Instead, Ordinary Animator retains the original workflows behind the scenes. There are two problems here:

• The workflows you supply ComfyUI APIs are not the same as workflows you save from the ComfyUI UI - some nodes are removed, so the original workflow is not exactly preserved in the PNG files.

• Not all media file formats preserve the workflow, such as video, audio, and text files.

So Ordinary Animator saves away the original workflows alongside media files to retain full fidelity of inputs.

Job Queue and Costs

Ordinary Animator provide a view of the local and cloud ComfyUI job queue. You can check up on background jobs without leaving the web interface, and it does its best to hookup up with billing information in ComfyUI so you can plan out your spend better.

Today, this is not 100% reliable. The ComfyUI APIs don't associate billing with a specific job, so instead I use time to watch expenditure and and jobs were run, to try and associate costs with jobs. But it does get it wrong sometimes if you have a lot of jobs running or have a long queue. Hopefully the Comfy Cloud APIs will improve here in the future.

Model Sensitive Prompt Generation

When writing a story, you want to express the story in your terms. So, of course, we support model sensitive prompt generation - the original information is converted into a prompt based on the model preferences (stored in the KB database). Want to use a different model? It automatically generates a different prompt. This is achieved by keeping prompt writing instructions for the AI for every model. Ordinary Animator remembers for each workflow what model it uses, so is able to convert the input context (e.g. character descriptions) to the specific format preferred by the model, on behalf of the user.

Conclusions

This was a quick run through of integration touch points between Ordinary Animator and ComfyUI. It is currently a hobby project … that grew bigger than originally planned. If you are interested to learn more, you can find the Discord server link on ordinaryanimator.com. Regardless, it has been a fun and educational experience getting the tool this far.