Designed for new Braintrust users, this workshop will cover the core foundations of evals and how to get started with Braintrust.
Language: en
Indexed excerpt
WorkshopsEvals foundations7 October 202610:00 AM PTPMsJess WangDeveloper AdvocateRegister for the workshopDesigned for new Braintrust users, this workshop will cover the core foundations of evals and how to get started with Braintrust. Jess will walk through the practical pieces of the eval workflow: moving from a question about your AI system to experiments, results, and feedback you can use to improve it. Bring a system you are building, a quality question you want to answer, or simply your curiosity. You will leave with a clearer picture of how to measure and improve agent quality, plus a practical next step to take. What you’ll learn How Braintrust groups production traces into named failure patterns you can act on How to filter a failure cluster into a labeled dataset How to write an eval that targets a specific failure pattern and validate the fix How to run a repeatable diagnose-to-eval workflow in Braintrust Recent on-demand workshopsEvals foundations9 September 202610:00 AM PTDesigned for new Braintrust users, this workshop will cover the core foundations of evals and how to get started with Braintrust.PMsHow financial services teams ship AI agents to production26 August 2…
Agents or data that involve more than one type of input or output, such as text, images, audio, or PDFs. Multimodal traces and datasets require observability infrastructure that can store and display non-text content alongside structured data.
A change in model behavior or performance over time, often caused by shifts in input distribution or provider updates. It's typically detected via score trends, topic shifts, or regressions on a fixed eval suite.
A dataset containing non-text inputs/outputs (images, audio, PDFs), requiring multimodal evals and storage. Multimodal datasets often need specialized rubrics and display tooling for review.
Evaluating multiple models against the same dataset and scorers to determine which performs best for a given use case. Comparisons are most useful when cost and latency are included, not just quality.
Braintrust's agent that helps teams write scorers, iterate on prompts, generate datasets, run evals, and surface patterns in production data. Loop reduces the friction of going from "something broke" to "we shipped a measured fix."
Capturing structured execution data from both development and production environments. In agent observability, logs are trace-based rather than line-based.
Optional key-value pairs attached to a dataset record or trace, used for filtering, grouping, and analysis. Metadata lets you slice performance by segment (tier, language, intent, etc.).
Adding code/hooks to capture traces, metadata, and events throughout the system. Good instrumentation makes failures reproducible and measurable instead of anecdotal.
A scoring approach where a language model evaluates the quality or correctness of another model's output, typically guided by a rubric. It enables scalable evals of open-ended outputs.
When a model produces output that is not supported by the provided context or by reliable external facts, but presents it as if it were true. Hallucinations are a common form of semantic failure.
A deployment model where the control plane is hosted by the vendor and the data plane runs in the customer's cloud environment. Hybrid deployment keeps customer data in their VPC while reducing operational overhead.
A workflow in which team members manually examine and score traces or dataset records through a dedicated UI. Human review is often reserved for high-risk or high-impact cases.
A unified API layer that routes LLM requests to any supported provider through a single interface. Gateways help provide automatic tracing and consistent telemetry across models.
Braintrust's principles, beliefs, and practical guidance for building high-quality evals.
Language: en
Indexed excerpt
Evals and observability are AI infrastructureThe eval is as important as the model, harness, etc. It should be treated as part of the stack, not an afterthought.If you’re building agents, you need to consider the infrastructure stack the same way you would for any software. What cloud provider, database, what IaC framework? What experiments, which scorers, which rubrics?Agents are now in production, supporting products used by millions. We are past the point of toy demos and neat experiments. Serious products need serious infrastructure. And real infrastructure is observable.
Find gaps in your AI development loop and get an AI quality roadmap.
Language: en
Indexed excerpt
ProductResourcesCustomersPricingContact usSign inSign upObserveTrace everythingEvaluateTest what shipsDiscoverFind behaviorsDocsStart buildingBlogInsights and updates .workshops-icon-col { transition: transform 400ms cubic-bezier(0.65, 0, 0.35, 1); } .workshops-icon-playing .workshops-icon-col { transform: translateY(7.5px); } WorkshopsLive sessions .evals-icon-row { transition: transform 350ms ease-in-out; } .evals-icon-playing .evals-icon-row-down { transform: translateY(5px); } .evals-icon-playing .evals-icon-row-up { transform: translateY(-5px); } Eval libraryOpen source researchFoundationsLearn to evalEncyclopediaEval conceptsProductObserveEvaluateDiscoverResourcesDocsBlogWorkshopsEval libraryFoundationsEncyclopediaCustomersPricingContact usSign inSign upEvaluate your AI development loop.Not all agents are created with the same quality and feedback loops. Find out where your team ranks given today's leading agent development workflows.Start assessment01Describe the productName the agent, workflow, or AI feature you want to improve.02Map the loopMark the quality practices your team already has in motion.03Set confidenceShare how confidently you can catch regressions today.React…
Independent eval research on models, agents, and cost. The work is open source, so you can check the methodology or rerun a study on your own data.
Language: en
Indexed excerpt
ProductResourcesCustomersPricingContact usSign inSign upObserveTrace everythingEvaluateTest what shipsDiscoverFind behaviorsDocsStart buildingBlogInsights and updates .workshops-icon-col { transition: transform 400ms cubic-bezier(0.65, 0, 0.35, 1); } .workshops-icon-playing .workshops-icon-col { transform: translateY(7.5px); } WorkshopsLive sessions .evals-icon-row { transition: transform 350ms ease-in-out; } .evals-icon-playing .evals-icon-row-down { transform: translateY(5px); } .evals-icon-playing .evals-icon-row-up { transform: translateY(-5px); } Eval libraryOpen source researchFoundationsLearn to evalEncyclopediaEval conceptsProductObserveEvaluateDiscoverResourcesDocsBlogWorkshopsEval libraryFoundationsEncyclopediaCustomersPricingContact usSign inSign upIndependent eval research. Open methodology, published datasets, and the tools to run your own studies.ResearchVideosSkillsNewsletterOriginal, open-source studiesModel comparisons, agent setups, cost, and modalities, measured on real tasks. Read the methodology and results, or rerun a study on your own data.9 September 2026Moonshot vs Fireworks for Kimi K3 frontend agentsKimi K3 ran through Moonshot and Fireworks in 30 direct …
Learn the foundations of evals, discover how to measure AI quality, and build a repeatable eval workflow.
Language: en
Indexed excerpt
ProductResourcesCustomersPricingContact usSign inSign upObserveTrace everythingEvaluateTest what shipsDiscoverFind behaviorsDocsStart buildingBlogInsights and updates .workshops-icon-col { transition: transform 400ms cubic-bezier(0.65, 0, 0.35, 1); } .workshops-icon-playing .workshops-icon-col { transform: translateY(7.5px); } WorkshopsLive sessions .evals-icon-row { transition: transform 350ms ease-in-out; } .evals-icon-playing .evals-icon-row-down { transform: translateY(5px); } .evals-icon-playing .evals-icon-row-up { transform: translateY(-5px); } Eval libraryOpen source researchFoundationsLearn to evalEncyclopediaEval conceptsProductObserveEvaluateDiscoverResourcesDocsBlogWorkshopsEval libraryFoundationsEncyclopediaCustomersPricingContact usSign inSign upRegister for live workshops with the Braintrust team, or rewatch past sessions on demand.All audiencesPMsEngineersUpcomingEvals foundations7 October 202610:00 AM PTDesigned for new Braintrust users, this workshop will cover the core foundations of evals and how to get started with Braintrust.PMsKeep up with new workshopsGet notified when new events, live workshops, and on-demand sessions are added.Email addressNotify meOn Dema…
In-depth articles and insights about AI evaluation, development best practices, and technical deep dives from the Braintrust team.
Language: en
Indexed excerpt
ProductResourcesCustomersPricingContact usSign inSign upObserveTrace everythingEvaluateTest what shipsDiscoverFind behaviorsDocsStart buildingBlogInsights and updates .workshops-icon-col { transition: transform 400ms cubic-bezier(0.65, 0, 0.35, 1); } .workshops-icon-playing .workshops-icon-col { transform: translateY(7.5px); } WorkshopsLive sessions .evals-icon-row { transition: transform 350ms ease-in-out; } .evals-icon-playing .evals-icon-row-down { transform: translateY(5px); } .evals-icon-playing .evals-icon-row-up { transform: translateY(-5px); } Eval libraryOpen source researchFoundationsLearn to evalEncyclopediaEval conceptsProductObserveEvaluateDiscoverResourcesDocsBlogWorkshopsEval libraryFoundationsEncyclopediaCustomersPricingContact usSign inSign upGuides for shipping quality AIBraintrust works with customers building AI products at scale.These guides distill the patterns that work. Instrument observability to understand how AI behaves in production. Use evals to improve your AI productsStart buildingLatest articlesRSSBest no-code AI agent builders in 2026Compare the best no-code AI agent builders in 2026. See how Sim AI, Lindy, Relevance AI, Stack AI, Gumloop, and Barde…