What Prompt Engineering Isn't

Prompt Engineering is like building a bridge that has to handle different physics each time someone drives over it.

Prompt engineering doesn’t provide a repeatable way to use AI models.

You’ve likely seen courses sold with thousands of prompts and industry professionals discussing amazing prompt engineering techniques. Technologists excited about ChatGPT have spent a lot of time creating engineered prompts. With so much excitement, how is it possible that prompt engineering isn’t providing a repeatable practice or series of techniques we can rely on?

To understand how Prompt Engineering is coming up short, we need to start with the AI research discussed in one of the latest episodes of our podcast. Researchers are looking into ways of using AI models that get the best results. They do rigorous testing, collect their results, and publish their findings for others to read and discuss.

For example, one study showed that posing a question as:

Question:

Answer:

gives you better results than using

Q:

A:

Examples like this are perfect for improving prompts to get better results. Looking at this research, you can begin to piece together a general approach for prompting. Sometimes this can be just enough to overcome challenges you’ve encountered with AI models and get the result you’re looking for. But the problem is these models are changing every month, and this research may not be relevant to a later version.

Changes to models may not seem like a big deal, but they fundamentally change how AIs think. Worse, we only have a small glimpse into how these models are changing. OpenAI, in particular, changed its models significantly last week to add functions as a new capability. While it’s hard to understand how much prompts will be impacted by this change, we can see the impact of these sorts of changes clearly when we look at image generation.

DALL·E 2 images using the same prompt ("a closeup of a bearded man's face wearing headphones in retro colors, synthwave style, 2d digital vector art") a few months apart. The style has changed dramatically, and the effect I liked was no longer present using the same prompts.

I was a huge fan of DALL·E when it was first released, but my interest quickly soured after a month of using it. Updates completely changed how the model generated images, and many prompts that previously made amazing art for me suddenly had really poor results. I was devastated and eventually moved on to Midjourney, which, while changing and releasing multiple versions, continues to support older versions that consistently produce expected results.

While it’s tempting to stick with a version of an AI model that works for you, you’ll eventually miss out on new capabilities. If we look at the results for the same Midjourney image generation below, we can see a dramatic shift in style. This upsets our ability to prompt for the same thing, but as new versions are released, new appealing aspects of the presentation are added.

The wide variety of styles that appeared as Midjourney evolved. V3 has a special place in my heart because I loved the style of that version for abstract art. None of the following versions have given me the same feeling that was felt in that version despite their ability to do many things better. No amount of prompt engineering is likely to give me the same results.

When we think of engineering, we might think of bridge construction. An engineer applies principles, standards, and techniques to work on a complex problem and create an expected output: a bridge. But that thinking starts to break down for AI because the tools we build with are changing, and the result is often a loose solution rather than a fixed statistical answer. It’s like building a bridge that has to handle different physics each time someone drives over it.

So what do we do? How do we handle the change that will continue to happen as new AI versions are released?

We need to do research. Every time a new model is released, we must re-validate what we know. Rather than focus on specific skills or prompt methods, we must learn from solutions to problems.

For example: How do you get an AI to give a more direct or less direct answer?
Knowing some good ways to change the answer returned gives you a better idea of how to direct a model. Record what you tried, try to measure its effectiveness, and see if you can reproduce it when the next version is released.

This trial and error process is the journey of an early adopter. There won’t be any best practices for quite a while, so try not to focus too much on one way of doing things and keep flexible. Many changes will happen, and there will be many new ways to build things and converse with AIs. If you’re brave enough, there’s still an amazing space to explore with AIs, but don’t be surprised if the road gets rougher the further you go.