GenAI

Using the Foundation Model APIs you can: Query a generalized LLM to verify a project’s validity before investing more resources. Query a generalized LLM in order to create a quick proof-of-concept for an LLM-based application before investing in training and deploying a custom model. Use a foundation model, along with a vector database, to build a chatbot using retrieval augmented generation (RAG). Replace proprietary models with open alternatives to optimize for cost and performance. Efficiently compare LLMs to see which is the best candidate for your use case, or swap a production model with a better performing one. Build an LLM application for development or production on top of a scalable, SLA-backed LLM serving solution that can support your production traffic spikes.

There are seven key patterns.

We can group metrics into two categories: context-dependent or context-free.

First, there’s poor correlation between these metrics and human judgments.

Second, these metrics often have poor adaptability to a wider variety of tasks.

Third, these metrics have poor reproducibility.

Building solid evals should be the starting point for any LLM-based system or product

we can start by collecting a set of task-specific evals

These evals will then guide prompt engineering, model selection, fine-tuning, and so on.

Eval Driven Development (EDD)

Rather than asking an LLM for a direct evaluation (via giving a score), try giving it a reference and asking for a comparison. This helps with reducing noise.

Dense vector retrieval serves as the non-parametric component while a pre-trained LLM acts as the parametric component.

When evaluating an ANN index, some factors to consider include:

Some popular techniques include:

To retrieve documents with low latency at scale, we use approximate nearest neighbors (ANN).

langgraph