.streamlit\/secrets.toml<\/code><\/li>\n<\/ul>\n\n\n\nStep 1 Build a minimal chat UI<\/h2>\n\n\n\n
This is the smallest useful Streamlit chat bot. It remembers history and calls an LLM.<\/p>\n\n\n\n
import os\nimport streamlit as st\nfrom openai import OpenAI\n\nst.set_page_config(page_title=\"Streamlit Chat Bot\", page_icon=\"\ud83e\udd16\", layout=\"centered\")\n\n# Initialize LLM client\nclient = OpenAI(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\n# Initialize chat history in session state\nif \"messages\" not in st.session_state:\n st.session_state.messages = [\n {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"}\n ]\n\nst.title(\"Streamlit Chat Bot\")\n\n# Render history (skip system in UI)\nfor msg in st.session_state.messages:\n if msg[\"role\"] == \"system\":\n continue\n with st.chat_message(msg[\"role\"]):\n st.markdown(msg[\"content\"])\n\n# Chat input\nif prompt := st.chat_input(\"Ask me anything\"):\n # Add user message\n st.session_state.messages.append({\"role\": \"user\", \"content\": prompt})\n\n # Display user message immediately\n with st.chat_message(\"user\"):\n st.markdown(prompt)\n\n # Get LLM response\n with st.chat_message(\"assistant\"):\n with st.spinner(\"Thinking...\"):\n completion = client.chat.completions.create(\n model=\"gpt-4o-mini\", # or your preferred model\n messages=st.session_state.messages,\n temperature=0.2,\n )\n reply = completion.choices[0].message.content\n st.markdown(reply)\n\n # Save assistant reply\n st.session_state.messages.append({\"role\": \"assistant\", \"content\": reply})\n<\/code><\/pre>\n\n\n\nRun with streamlit run app.py<\/code> and you\u2019ll have a functional chat bot.<\/p>\n\n\n\nMake it feel fast with streaming<\/h3>\n\n\n\n
Streaming small chunks improves perceived performance.<\/p>\n\n\n\n
with st.chat_message(\"assistant\"):\n placeholder = st.empty()\n collected = []\n for chunk in client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=st.session_state.messages,\n stream=True,\n temperature=0.2,\n ):\n delta = chunk.choices[0].delta.content or \"\"\n collected.append(delta)\n placeholder.markdown(\"\".join(collected))\n reply = \"\".join(collected)\n<\/code><\/pre>\n\n\n\nStep 2 Manage state and prompts responsibly<\/h2>\n\n\n\n\n- Keep a short, clear system prompt that sets persona and constraints.<\/li>\n\n\n\n
- Truncate long histories to control latency and cost.<\/li>\n\n\n\n
- Store only what you need; avoid logging secrets or PII.<\/li>\n<\/ul>\n\n\n\n
def trimmed_history(messages, max_tokens=2000):\n # Simple heuristic: keep last N messages; for production, measure tokens\n keep = []\n for m in reversed(messages):\n keep.append(m)\n if len(keep) > 10: # tune as needed\n break\n return list(reversed(keep))\n<\/code><\/pre>\n\n\n\nStep 3 Add retrieval for accurate answers (RAG)<\/h2>\n\n\n\n
Retrieval Augmented Generation lets the bot cite your documents rather than guessing. Below is a lightweight local approach using sentence embeddings and FAISS.<\/p>\n\n\n\n
\n- Index: compute embeddings for your documents and build a FAISS index.<\/li>\n\n\n\n
- Retrieve: on each question, get top-k chunks and pass them to the model.<\/li>\n<\/ul>\n\n\n\n
# pip install faiss-cpu sentence-transformers\nimport faiss\nfrom sentence_transformers import SentenceTransformer\n\nEMB_MODEL = SentenceTransformer('sentence-transformers\/all-MiniLM-L6-v2')\n\n# Build the index once (cache it for speed)\n@st.cache_resource\ndef build_index(docs: list[str]):\n vectors = EMB_MODEL.encode(docs, normalize_embeddings=True)\n index = faiss.IndexFlatIP(vectors.shape[1])\n index.add(vectors)\n return index, vectors\n\n# Example documents\nDOCS = [\n \"Company policy: Support hours are 9am-5pm AEST, Mon-Fri.\",\n \"Refunds are processed within 7 business days.\",\n \"To reset your password, use the account settings page.\",\n]\nINDEX, DOC_VECS = build_index(DOCS)\n\ndef retrieve(query, k=3):\n qv = EMB_MODEL.encode([query], normalize_embeddings=True)\n D, I = INDEX.search(qv, k)\n return [DOCS[i] for i in I[0]]\n\n# In your chat handler\ncontext = \"\\n\".join(retrieve(prompt))\nrag_system_prompt = (\n \"You are a helpful assistant. Use the provided context to answer. \"\n \"If the answer is not in the context, say you do not know.\\n\\n\"\n f\"Context:\\n{context}\\n\\n\"\n)\nmessages = [{\"role\": \"system\", \"content\": rag_system_prompt}] + trimmed_history(\n st.session_state.messages\n)\ncompletion = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=messages,\n temperature=0.1,\n)\n<\/code><\/pre>\n\n\n\nFor larger corpora, consider a managed vector DB (Pinecone, Weaviate, Qdrant Cloud) and chunking PDFs\/HTML via loaders.<\/p>\n\n\n\n
Use LLM tool calling to let the bot fetch live data or perform actions. Define tool schemas and route model requests to Python functions.<\/p>\n\n\n\n
tools = [\n {\n \"type\": \"function\",\n \"function\": {\n \"name\": \"get_support_hours\",\n \"description\": \"Return current support hours by region\",\n \"parameters\": {\"type\": \"object\", \"properties\": {\"region\": {\"type\": \"string\"}}, \"required\": [\"region\"]},\n },\n }\n]\n\ndef get_support_hours(region: str):\n return {\"AEST\": \"9am-5pm Mon-Fri\", \"PST\": \"9am-5pm Mon-Fri\"}.get(region, \"Unknown\")\n\nresp = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=st.session_state.messages,\n tools=tools,\n)\nchoice = resp.choices[0]\nif choice.finish_reason == \"tool_calls\":\n for call in choice.message.tool_calls:\n if call.function.name == \"get_support_hours\":\n import json\n args = json.loads(call.function.arguments)\n result = get_support_hours(args[\"region\"]) \n st.session_state.messages.append({\n \"role\": \"tool\",\n \"tool_call_id\": call.id,\n \"name\": \"get_support_hours\",\n \"content\": str(result),\n })\n # Send tool results back to the model\n final = client.chat.completions.create(\n model=\"gpt-4o-mini\",\n messages=st.session_state.messages,\n )\n reply = final.choices[0].message.content\n<\/code><\/pre>\n\n\n\nStep 5 Evaluate and observe<\/h2>\n\n\n\n\n- Golden sets: keep a small suite of Q&A pairs that the bot should answer.<\/li>\n\n\n\n
- Telemetry: log prompts, response times, token usage (exclude PII).<\/li>\n\n\n\n
- User feedback: add a thumbs up\/down and capture rationale.<\/li>\n<\/ul>\n\n\n\n
fb = st.radio(\"Was this helpful?\", [\"\ud83d\udc4d\", \"\ud83d\udc4e\"], horizontal=True, key=f\"fb_{len(st.session_state.messages)}\")\nif fb:\n st.session_state.setdefault(\"feedback\", []).append({\"msg\": prompt, \"fb\": fb})\n<\/code><\/pre>\n\n\n\nSecurity, privacy, and governance<\/h2>\n\n\n\n\n- Secrets: store API keys in
.streamlit\/secrets.toml<\/code>, not in code or git.<\/li>\n\n\n\n- PII: mask or avoid sending PII to third-party providers.<\/li>\n\n\n\n
- Rate limits: add retry\/backoff; degrade gracefully on provider outages.<\/li>\n\n\n\n
- Allow-list tools and sanitize tool inputs\/outputs.<\/li>\n\n\n\n
- Model choice: prefer enterprise offerings with data controls (e.g., Azure OpenAI with no training on inputs).<\/li>\n<\/ul>\n\n\n\n
Cost and latency control<\/h2>\n\n\n\n\n- Use small models for routine turns; escalate to larger models only when needed.<\/li>\n\n\n\n
- Trim history and context; use retrieval to provide only relevant chunks.<\/li>\n\n\n\n
- Cache expensive retrieval with
st.cache_data<\/code>.<\/li>\n\n\n\n- Batch background jobs; set
max_tokens<\/code> thoughtfully.<\/li>\n<\/ul>\n\n\n\nPackaging and configuration<\/h2>\n\n\n\n
Add a simple requirements.txt<\/code>:<\/p>\n\n\n\nstreamlit==1.37.0\nopenai>=1.30.0\nfaiss-cpu\nsentence-transformers\n<\/code><\/pre>\n\n\n\nAnd a minimal .streamlit\/secrets.toml<\/code>:<\/p>\n\n\n\nOPENAI_API_KEY = \"<your-key>\"\n<\/code><\/pre>\n\n\n\nDeployment options<\/h2>\n\n\n\n