Prompt Compass
Give a prompt rewrite a test it can fail.
I built and released Prompt Compass, then developed local comparisons that check whether a rewrite preserves the original task.
- My role
- Independent creator · workflow and tool development
- Period
- 2026
- Delivery stage
- Public prerelease · later local evaluation examples
The problem
Improving a prompt is easy to confuse with improving its wording. If a rewrite drops a required field, a constraint, or a review condition, the resulting answer can sound better while failing the original task.
My contribution
I built Prompt Compass to help an AI assistant inspect a request, identify missing requirements, choose an improvement route, and compare prompt versions. The tool provides structured analysis; the host assistant handles the conversation and final wording.
Make the required behavior explicit.
The analysis builds a prompt contract: what the person wants, which constraints matter, and what a useful output must contain. It checks for missing instructions and common failure patterns, then helps route the request into an appropriate improvement workflow.
Reasoning and tradeoffs
Why this mattered
The comparison has to know what to preserve. The schema example makes that concrete: output format, required fields, and an escalation condition each serve a different part of the task.
The options
A shorter instruction can leave those details implicit. The fixture uses that alternative directly, so the comparison has a specific loss to detect instead of a vague preference for one writing style.
The tradeoff
Explicit requirements take more space and still need review for consistency. They give the comparison something inspectable, but they are not a guarantee that the model will comply.
Give a polished rewrite a test it can fail.
One regression fixture starts with a prompt that requires valid JSON, four named fields, and a rule that sets suggested_response to needs_review when confidence falls below 0.7. Its shorter alternative asks only for a clean structured answer. The expected winner is the original: the rewrite loses the schema and review condition. This is a deliberately designed test case, useful for checking whether the comparison protects the task.
Reasoning and tradeoffs
Why this mattered
The shorter version removes the explicit schema and fallback. In this fixture, the comparison should prefer the original because the task requirements have become less specific.
The options
The original fixture deliberately compares the detailed instruction with “Return a clean structured answer with the best response for the user.” That is the alternative under test, not a reported customer mistake.
The tradeoff
The preservation check is simple and inspectable: it looks for each required term in the revised prompt. It can catch these dropped field names, but keeping the words would not prove that the resulting instruction preserves their meaning.
Keep the tool inside a useful conversation.
A structured check is most useful when it helps the person resolve the next uncertainty. In the example session, the request is clarified before it becomes a feature specification. I separated the local tool’s analysis and routing from the assistant’s conversation, so the workflow could make its reasoning about requirements visible.
Reasoning and tradeoffs
Why this mattered
A person testing Auto Flow is asking about the tool itself. Automatically treating that request as a prompt to improve would change their task before the conversation had addressed the bug report.
The options
The ordinary improvement route would analyze and rewrite the request. The implementation includes a separate meta-task branch that suppresses Auto Flow and lets the assistant handle the QA or configuration request directly.
The tradeoff
The branch depends on recognizing the request correctly. It keeps this boundary explicit, but its examples are a narrow test of routing behavior rather than proof that every ambiguous request is classified correctly.
What the work produced
A released tool with inspectable prompt checks.
I released Prompt Compass as a public beta with installable packages, source downloads, and versioned releases. Later local evaluation work added explicit comparisons for preserved requirements, including the schema-loss fixture shown here.
Why it matters for your project
I can help make your prompt requirements explicit and give the team a concrete way to compare alternatives.
Evidence and delivery stage
Public beta.3 release, April 27, 2026; original implementation and May 2026 local evaluation fixtures. Later local features are distinct from the April release. The example is a designed regression test, not a customer incident or an independent effectiveness benchmark.
What would you like AI to help with?
Tell me about the task and what a useful result would look like. We can find a concrete starting point.