The Agent Governance Stack: Treat Your AI Agent Fleet Like Your Engineering Org
AI Systems Engineering Patterns
Master AI engineering with 30 essential AI systems engineering patterns. From RAG and LLM gateways to multi-agent orchestration and flow engineering, discover the definitive guide to robust LLM design patterns that bridge the gap between traditional software architecture and modern AI development.
Agentic Design Patterns
Agentic Design Patterns A Hands-On Guide to Building Intelligent Systems, Antonio Gulli Table of Contents - total 424 pages = 1+2+1+1+4+9+103+61+34+114+74+5+4 11 Dedication, 1 page Acknowledgment, 2 pages [final, last read done] Foreword, 1 page [final, last read done] A Thought Leader's ...
Building Generative AI Services with FastAPI
Created with Figma
Evaluation Driven Development for Agentic Systems.
My step-by-step approach for building and evolving Agentic Systems that work.
12 Factor Agents: What are the principles we can use to build LLM-powered software that is actually good enough to put in the hands of production customers?
What are the principles we can use to build LLM-powered software that is actually good enough to put in the hands of production customers? - humanlayer/12-factor-agents
Zero to One: Learning Agentic Patterns
Learn common agentic design patterns and workflows for building robust, scalable AI applications, understanding when to use each.
It’s Hard to preserve LLM streaming when using function calls. Don’t Fret, I Got.pdf
Patterns for Building LLM-based Systems & Products
Evals, RAG, fine-tuning, caching, guardrails, defensive UX, and collecting user feedback.
There are seven key patterns.
We can group metrics into two categories: context-dependent or context-free.
First, there’s poor correlation between these metrics and human judgments.
Second, these metrics often have poor adaptability to a wider variety of tasks.
Third, these metrics have poor reproducibility.
Building solid evals should be the starting point for any LLM-based system or product
we can start by collecting a set of task-specific evals
These evals will then guide prompt engineering, model selection, fine-tuning, and so on.
Eval Driven Development (EDD)
Rather than asking an LLM for a direct evaluation (via giving a score), try giving it a reference and asking for a comparison. This helps with reducing noise.
Dense vector retrieval serves as the non-parametric component while a pre-trained LLM acts as the parametric component.
When evaluating an ANN index, some factors to consider include:
Some popular techniques include:
To retrieve documents with low latency at scale, we use approximate nearest neighbors (ANN).
How to Match LLM Patterns to Problems
Distinguishing problems with external vs. internal LLMs, and data vs non-data patterns