Prompt Engineering for Production AI: Beyond "Just Ask Nicely"

---

title: "Prompt Engineering for Production AI: Beyond "Just Ask Nicely"" date: "2026-06-20" author: "Daniyal Alam" tags: ["AI", "LLM", "prompt engineering", "production", "backend"] excerpt: "Prompt engineering in production is a software discipline, not a creative exercise. Here's how I ship reliable LLM features at scale."

Prompt engineering in a playground is forgiving — you iterate until you like the result, then move on. Prompt engineering in production is a different discipline entirely: your prompt is a contract between your system and the model, and breaking that contract at 3 AM costs you money, users, and sleep. After shipping LLM-powered features inside multiple production products at DanixSoft, I want to share what actually works beyond the tutorials.

Why Production Prompt Engineering Is Different

In a notebook, a slightly wrong answer is an annoyance. In production, it is a bug. The differences compound fast:

• Determinism expectations: Your users expect consistent behavior. The playground forgives variance; a customer-facing feature does not.
• Scale and cost: A prompt that is 200 tokens longer than it needs to be, multiplied by a million calls per month, is a real infrastructure cost.
• Failure modes are silent: A model that generates plausible-but-wrong JSON will not throw an exception — it will corrupt your database quietly.
• Maintenance surface: Prompts rot. The model updates, your product evolves, and a prompt written in January may silently regress by March.

Treat every prompt like production code. Version it, test it, observe it.

Structuring Prompts: Roles, Instructions, Context, Examples

The single biggest upgrade I made to our LLM pipeline was getting religious about prompt structure. A well-structured prompt has four distinct layers.

System vs. User Roles

The system message defines the model's identity and the rules of engagement. The user message carries the actual request. Never mix them up. If you put behavioral constraints in the user turn, a sufficiently creative user can argue with them or override them. The system prompt is the law; the user turn is the case.

System: You are a senior code reviewer. You only review code for correctness, 
        security, and performance. You never write new features.
        Always respond in valid JSON matching the provided schema.

User: Review the following Python function: [code]

Instructions Before Context

Put your behavioral instructions before the context you want the model to reason over. Models attend more reliably to instructions that appear before a large blob of context. Putting your output format spec at the very end of a 4,000-token system prompt is asking for it to be ignored.

One Instruction, One Sentence

Compound instructions fail compound prompts. Instead of "Be concise but thorough and also avoid technical jargon unless necessary," write three separate, testable directives:

1. Keep responses under 150 words.
2. Cover the most important risk in the first sentence.
3. Define any technical term you use.

Each one can be independently evaluated in your test suite.

Few-Shot vs. Zero-Shot: When Each Wins

Zero-shot prompts are cheaper and faster to maintain. Use them when the task is genuinely general-purpose and the model already has strong priors — summarization, translation, simple classification.

Few-shot examples are worth the token cost when:

• The output format is non-standard or domain-specific.
• The tone or style must be precise (matching your brand voice, for instance).
• The task requires a reasoning pattern the model does not apply by default.

My rule: start zero-shot, measure accuracy, add shots only when the measurement tells you to. Three good examples beat ten mediocre ones. Bad examples actively hurt performance — the model anchors on them.

Enforcing Structured Output

This is where most teams lose a week debugging. Here is how I handle it.

JSON Schema and Tool Calling

Modern frontier APIs (Claude, GPT-4o, Gemini) support constrained output via tool calling or native JSON mode. Use them. Do not ask a model to "respond in JSON" in free text — use the schema enforcement the API provides.

Below is a TypeScript example using the Anthropic SDK with tool calling to extract structured data from an unstructured report:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const extractionTool = {
  name: "extract_risk_summary",
  description: "Extract a structured risk summary from an analyst report.",
  input_schema: {
    type: "object" as const,
    properties: {
      risk_level: {
        type: "string",
        enum: ["low", "medium", "high", "critical"],
        description: "Overall risk classification.",
      },
      key_risks: {
        type: "array",
        items: { type: "string" },
        maxItems: 5,
        description: "Top risks identified, max 5 items.",
      },
      recommended_action: {
        type: "string",
        description: "One-sentence recommended next step.",
      },
    },
    required: ["risk_level", "key_risks", "recommended_action"],
  },
};

async function extractRisk(reportText: string) {
  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 512,
    tools: [extractionTool],
    tool_choice: { type: "any" }, // force tool use
    system:
      "You are a risk analyst. Extract structured risk data using the provided tool. Be conservative — if uncertain, classify higher.",
    messages: [{ role: "user", content: reportText }],
  });

  const toolUse = response.content.find((block) => block.type === "tool_use");
  if (!toolUse || toolUse.type !== "tool_use") {
    throw new Error("Model did not call the extraction tool.");
  }

  // Type-safe, validated by the schema at the API layer
  return toolUse.input as {
    risk_level: string;
    key_risks: string[];
    recommended_action: string;
  };
}

The schema does two things at once: it constrains the model's output and documents your contract. Any engineer reading this file understands exactly what they get back.

Parsing Safely

Even with schema enforcement, parse defensively. Validate the response against your expected types before passing it downstream. In Python, use Pydantic. In TypeScript, use Zod. Never assume the output is what you asked for just because the API said the call succeeded.

Evals and Regression Testing

A prompt without tests is a time bomb. I run three types of checks on every prompt we ship:

1. Unit evals: Controlled inputs with known expected outputs. Automated, run in CI on every prompt change.
2. LLM-as-judge: For subjective quality (tone, helpfulness), use a separate model call with an explicit rubric to score the output. Cheap and surprisingly reliable.
3. Golden set regression: A curated set of real production inputs with manually verified outputs. If a prompt change degrades performance on more than 5% of the golden set, the change does not ship.

The tooling here has matured significantly. Frameworks like LangSmith, Braintrust, and PromptFoo all support prompt versioning and eval pipelines. Pick one and commit to it.

Guardrails and Input/Output Validation

Defense in depth matters. I layer guardrails at three points:

• Input sanitization: Strip or escape content that could inject instructions. A user who pastes "Ignore previous instructions and..." into your form should hit a wall, not your system prompt.
• Output validation: Run a lightweight classifier or regex check on the model's response before returning it to the user. If the output contains something it should never contain (PII patterns, competitor names, hallucinated URLs), catch it before it ships.
• Fallback paths: Define what happens when validation fails. Retry with a stricter prompt? Surface a generic response? Log and escalate? Whatever it is, define it explicitly. Silence is not a strategy.

Prompt Versioning and Observability

Every prompt in our system has a version string. When we ship a new version, we log which version produced each response. This lets us:

• Roll back a prompt independently of a code deploy.
• A/B test prompt versions with real traffic.
• Correlate user complaints with specific prompt changes.

Store prompts in your codebase, not in a database field that someone edits ad hoc. Treat them like configuration — in version control, reviewed, tested before merge. Tools like LangSmith or even a simple prompt registry table with version, hash, and deployed_at columns get you most of the way there.

Instrument every LLM call with: model name, prompt version, input token count, output token count, latency, and whether the output passed validation. Without this telemetry, you are flying blind.

Temperature and Determinism

Temperature controls randomness. For extraction, classification, and structured output tasks: use temperature: 0. You want the most probable token at every step, not creative variance.

For generative tasks (marketing copy, creative brainstorming): a temperature between 0.7 and 1.0 is reasonable. For everything in between, measure. Do not guess.

Also set top_p conservatively for deterministic tasks. And be aware: even at temperature: 0, model outputs are not guaranteed to be identical across different batches or API versions. Do not hardcode expected strings in tests — test semantics, not literals.

Cost and Token Discipline

Token cost is an engineering constraint, not an afterthought. Habits I enforce on every LLM feature:

• Measure before optimizing: Log token counts from day one. You cannot optimize what you do not measure.
• Trim context aggressively: Only send what the model needs for the current task. Retrieval-augmented generation helps — fetch the three most relevant chunks, not the entire knowledge base.
• Cache where possible: Anthropic's prompt caching, OpenAI's cached inputs — if your system prompt is long and static, cache it. The savings are substantial at scale.
• Choose the right model for the task: Routing simple classification tasks to a smaller, cheaper model and reserving a flagship model for complex reasoning cuts costs dramatically without sacrificing quality on either end.

Key Takeaways

• Prompt engineering for production is a software engineering discipline: version, test, and observe your prompts like code.
• Separate system and user roles cleanly; put instructions before context.
• Use schema-enforced output (tool calling / JSON mode) and validate the response before trusting it downstream.
• Build an eval suite with unit tests, LLM-as-judge scoring, and a golden set for regression.
• Guard both input and output; define explicit fallback behavior.
• Log prompt versions and token telemetry on every call — you will need this data sooner than you expect.
• Temperature zero for deterministic tasks; measure cost from the first deploy, not after you get a billing surprise.