Quickstart

Sign in with GitHub and create a project. You'll see your EVALO_API_KEY on the project overview page — save it now, it's only shown once.

npm i @evalo/sdk
# or
pnpm add @evalo/sdk

import { Evalo } from "@evalo/sdk";
import OpenAI from "openai";

const evalo = new Evalo({ apiKey: process.env.EVALO_API_KEY! });
const openai = new OpenAI();

const trace = evalo.trace({
  name: "summarize",
  input: { article },
});

const r = await openai.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content: `Summarize: ${article}` }],
});

await trace.end({
  output: r.choices[0].message.content,
  tokensInput: r.usage?.prompt_tokens,
  tokensOutput: r.usage?.completion_tokens,
});

Or send a raw HTTP request:

curl -X POST https://evalo.dev/api/v1/traces \
  -H "Authorization: Bearer $EVALO_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "summarize",
    "input": { "article": "..." },
    "output": "...",
    "durationMs": 842
  }'

In your project, go to Datasets → New dataset, then paste JSONL — one test case per line:

{"input": "refund my order #1234", "expected_output": "Initiate refund and confirm."}
{"input": "is bitcoin a security?", "expected_output": "Decline; out of scope."}
{"input": "where is my package?", "expected_output": "Ask for order number."}

Go to Eval runs, pick your dataset and eval definition, optionally add a system prompt, and hit Run evals. Or trigger from CI:

await evalo.runEval({
  datasetId: "ds_xxx",
  evalDefinitionId: "ev_xxx",
  systemPrompt: "You are a helpful support agent.",
  model: "gpt-4o-mini",
});

In Settings → GitHub integration, install the Evalo GitHub App. Every PR against the connected repo will trigger your configured eval suite and post a check-run with the pass-rate diff vs main. Set up branch protection to block merges on regression.