The Codans made some really smart design choices for Coda AI and specifically with respect to prompt design. Here are just a a few off the top of my head:
-
Prompt fields allow simple rich-text formatting of texts that will become large in certain cases. It’s nice to be able to add structure to prompt texts. Will such formatting help or hinder AI outcomes, I am uncertain. I suspect they are stripped of formatting before being shipped off to complete the inferencing process.
-
Prompts may contain multiple references to formulas. Simply use the equals (=) key to invoke the formula editor at any location in the prompt field. This makes it very easy to inject data into prompts. There are limits to the volume of data you can use for inferencing, but that will change as LLMs expand the prompt window size.
-
Prompts can be abstracted. A prompt field can be a formulaic reference, e.g., the prompt itself can be fabricated elsewhere in the document. This makes it possible to create reusable prompt patterns that serve as templates.
These and many other prompting features provide significant agility for Makers to craft elegant AI solutions. After all, LLMs need your data if you want to take full advantage of AGI’s benefits. Coda AI paves this path with an effortless way to bind data to AI prompts.
Here’s an example.
In this prompt, I’m pulling in data from adjacent table fields to proved the context for this inference. The abstraction of data in this prompt is both an advantage and a curse. The advantages are obvious; the curse, not so much.
If prompts contain abstractions, testing them must occur in the environment where they are created.
This is to say that you can’t test your prompts in another document or tool unless the abstracted dependencies can be resolved. Otherwise, your prompt will be incomplete and such tests will fail.
Prompt development is like code development. I’ve been vocal about language as code and the need to treat prompts no different than lines of code. Prompt engineering must embrace change management much the way we version and test code for applications.
Prompt engineering may require tools.
- What will you use to compose your prompts?
- Where will you store all your prompts?
- How will you test and track the performance of the prompts you develop?
The abstraction features in Coda AI make it possible develop prompts in nicely organized tables. This is a huge advantage until you realize your prompt needs data that exists where the prompt will actually be executed. Oops!
This is the curse. We want to engage in prompt engineering by leaning into good development practices, but in our zest for elegance and reusability, we paint ourselves into a corner. The more we abstract data and prompt texts, the smaller the corner becomes.
What’s the Problem?
Coda has the chops to perform well with regard to prompt engineering process, testing, and workflows to corral the tasks of testing and iterating better prompts. It has super-strengths to assist in the development of better prompts because at its core, it is a word factory.
I’ve actually used Coda’s innate abilities as a prompt management platform to test AI queries and improve them for an FAQ chat system.
However, as elegant and productive as this application is, the prompts stand alone; they have no moving parts. Users ask questions and the inferences are performed against the trained corpus (which is also managed in Coda).
If I tried to use this prompt testing approach for applications that involve dynamic data injections into the prompts, the app would need to be moved into the environment where the data exists, or the data would need to exist where the prompt testing app exists.
I predict prompt management and reusability is going to be a thing for every Maker because almost every person will soon encounter a situation that results in poor outcomes with no apparent solution. Makers will become frustrated, and without a clear approach that leads us out of prompt cul-de-sacs, our envisioned AI advantages for many use cases will die at Knots Landing.
Indeed, the long lists of prompt options provided in Coda’s UI exist because Codans saw the writing on the wall -
If we don’t nudge them by literally putting words in their AI prompt editor, it will not end well for most Makers.
The embedded prompts are simple and helpful - to a point. Beyond that point is a land of AI disenchantment.
What’s the Remedy?
I don’t have one [yet] and I’m not sure there is one. I have a sense that tools that address prompt engineering challenges need to be abstracted themselves; they need to operate as a veneer over Coda (and other applications). Ideally, we should be able to create and manage prompts independently of any single app or any specific data. This is a big challenge.
In the meantime, perhaps a good tactical move is to think carefully about how you craft prompts such that they can be improved with subtle changes all while able to roll back and revert from prompt missteps.
This prompt is fairly complex. It produces a customer survey analysis for a single question. It required 27 iterations; 12 of them related to the embedded formula and the remaining 15 were subtle modifications to the text.
As this prompt was developed, there were at least 11 roll-backs to earlier versions of the prompt. How was I able to roll-back? It’s not pretty and I won’t pretend my development process is ideal or ready to apply in new contexts. However, you can surmise from my initial observations about complete prompt abstraction that I probably used another table to manage the iterations.
PrompTunr is a prompt tuning table that includes essential details about the prompt as well as a field that includes all the variants I created to reach a satisfactory (production) version that my app would use. This is what I meant earlier when I said -
It [Coda] has super-strengths to assist in the development of better prompts because at its core, it is a word factory.
As each new prompt variant is created from a previous variant, it allows me to preserve earlier versions. Each version;'s outcome (using complete prompt abstraction) is displayed in the prompt variants table which serves as the prompt workbench.
Despite the clumsiness of this approach, it certainly provides a pathway to prompt development with a little development science so I don’t go insane as I try to bend the will of LLMs to be truthful and produce expected outputs.
Takeaway
Good AI is often less about the AI and more about good information management. Prompts should be treated like code - versioned, documented, and organized in ways where all meta-data is leveraged to reach your objectives.