Microsoft open-sourced Scope, a self-hosted testing platform for evaluating coding agents like GitHub Copilot and Claude Code under the MIT license. It runs tasks through different agent setups and scores outcomes using an LLM-based Judge against criteria organized as a dependency graph, keeping logs, snapshots, and tool-call traces for debugging. Teams maintaining SDKs, docs, or MCP servers can use it to spot where agents stumble, swap configurations, and rerun tasks for comparison. There's no hosted version; local setup requires Docker, Node 22, pnpm, mkcert, GitHub CLI, a GitHub Copilot entitlement, and model provider credentials, with Kubernetes/Helm available for shared deployments.
Questions this post answers
What is Microsoft's Scope tool for testing AI coding agents?
Scope is an open-source, MIT-licensed platform Microsoft built to evaluate how coding agents like GitHub Copilot and Claude Code perform on tasks. It runs the same task through different agents, scores results with an LLM-based Judge against dependency-graph criteria, and keeps logs, snapshots, and tool-call traces so failed checks can be traced back through each step. Human review of conclusions is still recommended. Teams comparing agent reliability can track tools like Scope on daily.dev as the space evolves.
What do I need installed to self-host Microsoft Scope locally?
A local Scope setup requires Docker with Compose v2, Node 22, pnpm, mkcert, and the GitHub CLI. The default worker also needs an active GitHub Copilot entitlement, and the Judge service needs credentials for a model provider. There is no hosted version, so running it yourself is the only option, though Kubernetes with a Helm chart supports shared deployments. Developers evaluating self-hosted dev tooling can follow setup requirements like these on daily.dev.
How does Scope score whether an AI coding agent succeeded at a task?
Success criteria form a dependency graph where one check can gate another, such as verifying a Node.js server returns JSON from a GET /health route. An LLM-based Judge service scores each run against those criteria, though Microsoft notes it can make mistakes and important conclusions still warrant human review. Versioned profiles let users swap agents, models, or MCP servers and rerun for comparison. Anyone building agent evaluation workflows can keep tabs on tools like this via daily.dev.
1 Comment
Share this post