AI Chatbot Development: Adding a Smart Assistant to Your Product

---

title: "AI Chatbot Development: Adding a Smart Assistant to Your Product" description: "A practical guide to designing, building, and shipping an LLM-powered chatbot—covering architecture, RAG, function calling, guardrails, cost control, and evaluation." pubDate: 2026-06-24 tags: ["AI", "LLM", "Chatbot", "RAG", "TypeScript", "Product Engineering"]

Most teams that ship an AI assistant in 2026 do it wrong—not because they pick the wrong model, but because they skip the decisions that happen before the first API call. After building and iterating on AI chatbot development across several products at DanixSoft, I can tell you the architecture choices made in week one determine whether your assistant feels magical or broken two months after launch.

Here is the complete picture: use case scoping, architecture, a working code example, guardrails, cost control, and how to know the thing is actually ready to ship.

---

What a Modern AI Chatbot Actually Is

Forget the rule-based bots of 2018 that matched keywords to canned responses. A modern assistant is an LLM with context. It receives a system prompt that defines its role and constraints, a conversation history, optionally retrieved documents from your own data (RAG), and optionally a set of tools it can call—then it generates a response.

The shift matters because:

• The bot can handle phrasing it has never seen before
• It can reason across multi-step queries
• It degrades gracefully when the question is ambiguous, rather than hitting a dead end
• It can take real actions (look up an order, draft a ticket, query a database) via function calling

The cost is non-determinism. You trade a brittle-but-predictable system for a flexible-but-probabilistic one. Every decision after this point is about managing that tradeoff.

---

Deciding the Use Case Before Writing Code

Scope is everything. "Add a chatbot" is not a use case. The three archetypes I see most:

Customer support — deflect tier-1 tickets, answer product questions, escalate edge cases to humans. The assistant needs your docs, your FAQ, your changelog. High volume, latency-sensitive, must never fabricate policy.

Sales / lead qualification — answer pricing questions, surface the right plan, book a call. Needs product data and a hand-off mechanism. Tone matters more here.

Internal knowledge base — let your team query runbooks, SOPs, Confluence pages, Notion docs. Lower volume, higher tolerance for latency, but data privacy becomes a first-class concern.

Pick one. The system prompt, the retrieval strategy, and the acceptable failure modes are different for each.

---

Architecture: Five Layers You Need to Design

1. LLM Provider and Model Tier

In 2026 you have a real choice between providers and tiers. The pattern I use:

• Fast, cheap model for short conversational turns where the user expects <1 s
• Mid-tier model for most domain Q&A
• Frontier model for complex reasoning, tool chains longer than two steps, or when a wrong answer costs money

Pick a model tier per route, not per product. Routing logic is cheap; regret from serving every query through the frontier model is not.

2. System Prompt

The system prompt is your most powerful lever and the one most teams underinvest in. It should define:

• The assistant's persona and what it is allowed to say
• What it should do when it does not know the answer (say so, do not guess)
• How to handle off-topic requests
• Output format expectations (markdown, plain text, structured JSON)

Version-control your system prompt like code. A changed prompt is a changed feature.

3. RAG Over Your Data

Retrieval-augmented generation (RAG) is how you give the model knowledge beyond its training cut-off and your own proprietary content. The pipeline:

1. Chunk your documents (400–800 tokens per chunk, overlap ~10%)
2. Embed each chunk and store in a vector database (Pinecone, pgvector, Weaviate)
3. At query time, embed the user message, retrieve the top-k chunks by cosine similarity
4. Inject retrieved chunks into the prompt as context

A retrieval step adds 50–200 ms. Optimise the chunk size and reranking strategy before reaching for a bigger model—it usually matters more.

4. Tools and Function Calling

Function calling lets the model decide when to invoke an external capability: look up an order, fetch live pricing, create a support ticket. You define the tool schema; the model outputs a structured call; your code executes it and feeds the result back.

Below is a TypeScript streaming endpoint that wires together a system prompt, RAG context, and a tool call using the Anthropic SDK:

import Anthropic from "@anthropic-ai/sdk";
import { retrieveContext } from "./retrieval";
import { lookUpOrder } from "./tools/orders";

const client = new Anthropic();

export async function POST(req: Request) {
  const { messages, sessionId } = await req.json();
  const userMessage = messages[messages.length - 1].content as string;

  // RAG: pull relevant chunks for this query
  const context = await retrieveContext(userMessage, { topK: 5 });

  const systemPrompt = `You are a helpful support assistant for DanixSoft products.
Only answer questions related to our products and docs.
If you do not know the answer, say so clearly—do not guess.
When the user asks about an order, use the look_up_order tool.

Relevant documentation:
${context.map((c) => c.text).join("\n\n")}`;

  const stream = await client.messages.stream({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system: systemPrompt,
    messages,
    tools: [
      {
        name: "look_up_order",
        description: "Fetch the status and details of a customer order by ID.",
        input_schema: {
          type: "object" as const,
          properties: {
            order_id: { type: "string", description: "The order identifier" },
          },
          required: ["order_id"],
        },
      },
    ],
  });

  // Handle streaming + tool use
  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const event of stream) {
        if (
          event.type === "content_block_delta" &&
          event.delta.type === "text_delta"
        ) {
          controller.enqueue(encoder.encode(event.delta.text));
        }

        if (event.type === "content_block_stop") {
          const message = await stream.finalMessage();
          const toolUse = message.content.find((b) => b.type === "tool_use");
          if (toolUse && toolUse.type === "tool_use") {
            const result = await lookUpOrder(
              (toolUse.input as { order_id: string }).order_id
            );
            controller.enqueue(
              encoder.encode(`\n\nOrder status: ${JSON.stringify(result)}`)
            );
          }
        }
      }
      controller.close();
    },
  });

  return new Response(readable, {
    headers: { "Content-Type": "text/plain; charset=utf-8" },
  });
}

Key details: the stream is piped directly to the client so the UI can start rendering immediately. The tool result is appended to the stream after execution. In production you would loop this back into the model for a coherent final response rather than appending raw JSON.

5. Streaming UI

Show tokens as they arrive. Users tolerate 3–4 seconds of perceived wait; they do not tolerate staring at a spinner for 8 seconds before a wall of text appears. Every modern frontend framework has a hook for server-sent events or readable streams.

---

Guardrails, Safety, and Fallback

Three things that must be in place before launch:

Input filtering — detect and refuse prompt injections, jailbreak attempts, and out-of-scope categories before the message reaches the model. A lightweight classifier or a fast cheap model pass is usually enough.

Output validation — for anything the bot says that touches policy, pricing, or legal, add an assertion layer. If the response contains a number that doesn't match your pricing table, suppress it and surface a fallback.

Human escalation path — define the trigger: three failed tool calls, low-confidence classification, user explicitly asking for a human. The assistant must know when to step back. "I'm not sure—let me connect you with the team" is a feature, not a failure.

---

Cost and Latency Control

Token cost compounds fast at scale. The levers I reach for first:

• Prompt caching — if your system prompt and RAG context stay stable across many turns, use cached prompt prefixes. This cuts input token cost by 80–90% on repeated context.
• Model routing — route simple intent-classification or one-sentence answers to a fast small model; escalate to the larger model only when needed.
• Conversation summarisation — after N turns, summarise the history and replace it with the summary. Keeps context windows from exploding on long sessions.
• Streaming — reduces perceived latency without changing actual compute cost.

Set per-session token budgets and monitor them. An uncapped session with a verbose user can cost 10x the average.

---

Evaluation Before Launch

I will not ship a chatbot without a red-team eval run. The checklist:

• Accuracy — create 50–100 golden Q&A pairs from your docs. Score the assistant on retrieval precision and answer correctness.
• Refusals — test every out-of-scope category you care about. The bot should decline gracefully, not hallucinate an answer.
• Latency — p95 end-to-end latency under realistic load, not just a single call.
• Adversarial inputs — prompt injections, gibberish, language switching, extremely long inputs.
• Tool reliability — does the model call the right tool with the right parameters? What happens when the tool returns an error?

Automate the golden Q&A suite as a CI job. Every system prompt change should re-run it.

---

Data Privacy

If users share personal data with your assistant (and they will, even if you tell them not to), you need:

• No-training clauses in your provider agreement
• Data residency constraints if you serve regulated industries
• Conversation retention policy — how long do you store messages, who can access them, is there a deletion path?
• PII scrubbing before logs are written to persistent storage

This is not optional for B2B products. Customers will ask before signing.

---

Key Takeaways

• Scope the use case first; the architecture follows from it, not the other way around.
• Your system prompt is a first-class engineering artifact—version it, test it, review changes to it.
• RAG is usually a better ROI than upgrading the model tier. Fix retrieval before reaching for a bigger LLM.
• Streaming + model tiering handles most cost and latency problems without sacrificing quality.
• You need a human escalation path. No assistant should be the last line of defence.
• Run a structured evaluation suite before launch and again after every meaningful change to prompt or retrieval.
• Data privacy decisions must be made before you write the first line of code, not after the first customer complaint.

AI chatbot development is as much product design as engineering. The teams that ship assistants users trust are the ones who define the failure modes before the success paths.