| Model serving | Ollama · vLLM | Expose behind a model gateway with authentication, model allow-list, rate limits and health checks. Ollama can simplify local development and selected single-node deployments; vLLM is a candidate for higher-throughput GPU serving. Ollama is a runtime, not a model: model families such as Llama or Qwen are selected separately, with compatibility, language quality, hardware fit and weight licenses checked. |
| Agent orchestration | LangGraph · PydanticAI | Represent long-running work as explicit steps with state, retries, checkpoints and human approval. Use typed input/output contracts and validated tool arguments. Select one primary orchestration pattern per agent; avoid adding both without a clear reason. |
| RAG pipeline | Haystack · BGE / FlagEmbedding · Qdrant | Build an ingestion pipeline for approved sources: parse, normalize, chunk, attach ACL/metadata, embed, index and evaluate retrieval. Enforce document permissions at query time; use hybrid/vector retrieval and reranking only when evaluation shows a gain. Keep source citations in answers. |
| Business tools | FastMCP · APIs · webhooks | Expose narrow, schema-validated operations rather than general credentials. Apply per-agent tool allow-lists, least privilege, idempotency, human approval for consequential writes and audit logging for every call. |
| Workflow automation | Kestra | Use deterministic workflows for schedules, event triggers, retries and multi-system jobs. Keep LLM reasoning in scoped steps; use ordinary code or rules for deterministic control and reconciliation. |
| Guardrails | NeMo Guardrails · application policy checks | Apply input/output and tool-use checks as one layer alongside authentication, authorization, validation and network controls. Test prompt injection, unsafe tool arguments, sensitive-data handling and refusal/escalation paths. |
| Observability & evaluation | Langfuse | Trace prompt/model versions, tool calls, latency, token use and errors. Build regression datasets and task-level evaluation; redact or avoid capturing personal data in traces and define retention before production. |
| Persistent memory | Mem0 · scoped application storage | Optional, not default. Persist only useful, justified facts with tenant/user scope, source, expiry, correction and deletion paths. Do not use conversational memory as a substitute for authoritative business records. |