6:15 PM

I am exploring assembling scenes for computer animation from text based on AI. This is partly because I would like to speed up my workflow, and partly as a learning exercise to understand the challenges of applying AI to real life problems. This time I am looking at GPT to see how it can be used to tackle the problem.

To summarize the problem, how to convert arbitrary free text into a restricted vocabulary of commands that software can process. Software cannot have new options added on the fly - it has a well-defined set of operations it can support and options it understands. The output text could be JSON (which is easily machine processible), or text that follows a limited grammar that can be tokenized and parsed, much like a programming language.

Text to Command

My first experiment was to try out the OpenAI "Text to Command" playground, which is very close to what I am after. The following is an example of what I am trying to do:

Convert this text to a programmatic command:

Here is what the playground returned.

Output: CHARACTER Hank LOCATION desk POSE standing FACE angry LOOKAT window SPEECH ""

Not bad! It is only a single character, but it got the idea of what I was after pretty well.

So I tried to "fine tune" the model so it knows what to generate without providing examples every time. But what to provide as training data? Th example and output? The full prompt and output? I could not find any examples, so I decided to try the example text as the prompt and the output as the completion example. The OpenAI tool to check the input file started coming up with recommendations:

• Don't repeat the same prefix ("Convert this text to a programmatic command:") for each sample.

• Don't use lots of words with uppercase because GPT has not been trained with upper case text much.

But I hit my first road bump. I had (incorrectly) assumed the "text-davinci-003" model used in the playground is the same as the "davinci" base model that I can fine tune. Bzzzt! My inability to find training examples from others turns out to be something simple: you cannot fine tune the "text-davinci-003" model.

Text Explanation

So, my next attempt was to look at "text explanation" using ChatGPT alone. "Text explanation" includes capabilities such as asking questions about a piece of text, summarizing text (pull out the highlights), classifying text (including sentiment analysis), and extracting entities. Because I am after structured data to be extracted from the text, question answering and extracting entities appeared to be the most relevant.

For example, with question answering I could provide the text and then ask a series of questions such as "who is in the shot?", "where is <X> located?", "what pose does <X> have?" and so forth. I did not find this very useful however as the responses were free text. Adding extra phrases such as "in at most 2 words" helped it be more precise, but it was not ideal - I had to send many questions via separate requests.

So I moved on to try entity extraction. My first attempt was:

From the text below describing a shot in a screenplay, extract the following entities in the following format:

This returned the following:

People: Sam, Hank

Not bad, but it is frequently too wordy for what I wanted. So I tried to give it a restricted vocabulary of words that I wanted extracted. (See the description of Expressions and Pose below.)

From the text below describing a shot in a screenplay, extract the following entities in the following format:

The response was better, closer to my needs.

People: Sam, Hank

I continued on to the next scene, wondering how much of the previous conversation would automatically flow through. This raised a problem however in that the length of the instructions is eating into the memory of previous shots. ChatGPT has a fixed size buffer of how much it feeds into subsequent requests.

Add the text below to the next shot in the screenplay, then extract the following entities in the following format:

This returned the following:

People: Sam, Hank, Liana

Note the use of "and" sneaking in ("Hank and Liana(standing)"). Well, I can handle that. The camera description however needs to be tightened up.

After a few more trials however I noticed it start return values not in the lists I provided. It might provide "smiling" when I told it to use "happy". That may be  better choice, but is problematic because it means my code later has to deal with unexpected values.

But adding all this specific details into the problem is annoying. There is a length limit for prompts, so adding more and more detail to the prompt leaves me less space for content. So my next try was to reword it to try and be more concise.

The text below is a single shot of a screenplay. For each person in the shot, list the following:
Name: <name of the person>

I did not include the camera request yet, but the output was pretty good!

Name:

Another example, trying to tighten up the location description.

The text below is a single shot of a screenplay. For each person in the shot, list the following:
Name: <name of the person>

This returned

Name: Liana

Err, great. The location descriptions are much more useful. But instead of listing characters per attribute, it listed attributes per character. Don't you love consistency? I still had not included camera positioning. So I added some more text to the prompt.

Also for each camera shot, include:

The response was, you guessed it, structured differently again. (It did however pick up an additional line of dialogue.)

Shot 1:

It has decided that what I wrote as a single shot should be broken down into a series of shots! They were actually pretty good decisions where to do the breaks. But I want that artistic control, so I plan to tweak the wording more to be more precise what I am after.

Conclusions

The purpose of this blog was to explore how AI can be used to extract directions from text. A few approaches exist that show promise, but its by no means a solved problem. You might not be interested in this precise problem, but understanding the capabilities of the various AI libraries around can be useful as working out which of many available paths to take may ultimately be the most important question to answer. For myself, the GPT models show promise, but I will need to refine my prompts further, and it is annoying to have to repeat so much instructional text per request.

I like the idea of training a model to avoid repeating instructions for my request per call. I have been told that to fine tune a model, you need to give it 200 to 300 samples minimum for it to start following what you tell it. Not a small effort for an experiment. And its training a base model, not the model pre-trained for commands. So I hope to give this a go, but it might take a while to get all the text created.

Without fine turning, carefully wording and restructuring prompts (referred to as "prompt engineering") can still get you a long way. I need to work on the length and preciseness of the instructions. It feels wasteful to have to keep repeating lengthy instructions per shot description (and you pay based on number of input tokens).

Another challenge I face is I want to extract more information than the above. I want to know what characters were looking at, I want to know any mentioned leg, arm, and hand positions beyond the basic poses. So the instructions are going to get even longer.

But it is interesting that with a very simple API (text in, text out) you can get access to an API with pretty powerful capabilities.