Grigorii Barakhsin

Researcher, ITMO University AI Center. Saint Petersburg.

barakhsin@gmail.com · Google Scholar · GitHub · LinkedIn

Agent oversight assumes that when a system fails, the record it left is enough to tell you so. I measure whether that holds.

Mostly it does not. Established LLM judges report a failure on clean agent traces between 34% and 78% of the time. And 84% of a trace by volume, the part holding prompts and retrieved documents, contributes nothing at all to locating a failure.

Research

SafeJudgeBench in progress

First author, with Anna Kalyuzhnaya.

A quantitative survey of agent-safety trace corpora, and a judge benchmark built on it. Ten corpora and roughly 24,000 labelled traces, assembled from an exhaustive search of the released literature, with every source's label semantics verified rather than inferred. Results and taxonomy under submission.

Two-dimensional projection of annotator justification embeddings,
coloured by source corpus, showing that the corpora occupy largely
separate regions. — 53,000 annotator justifications from five of the corpora, embedded and projected; colour is the source. They separate almost perfectly, which is a problem before it is a result: what a corpus calls a failure is partly just how its annotators write.

What Must You Log? Minimal Sufficient Trace Signals for Agent Failure Attribution

First author, with Anna Kalyuzhnaya.

What an execution trace has to retain for failure attribution, measured rather than assumed. Holding a benchmark's taxonomy, judge prompt and scoring code fixed, I varied only which trace fields the judge received.

Across ten public agent-failure corpora, median trace size spans a 163-fold range, and individual traces span a factor of 9,180. Only four of the ten separate model input from output, with no published evidence behind either choice.
Localisation depends on one content field, what the agent itself wrote. Withholding it collapses accuracy from 0.39 to 0.06 on one judge and from 0.37 to 0.11 on a second.
Keeping that field with step identifiers and two near-free metadata fields recovers full-trace accuracy at 2.33% of the raw token volume.
The replayed model context never helps localisation, because 85.7% of it repeats text already recorded elsewhere in the same trace.

Paired differences in localisation accuracy for each trace-field
configuration against the minimal configuration, for two judge
backbones, with bootstrap confidence intervals. — What each field is worth. Every row is one configuration's paired difference from the minimal one, on two judge backbones, with bootstrap intervals; the share of raw token volume each configuration costs is given in brackets. Everything left of the line is no better than keeping 2.33%.

TraceJudgeBench: A Unified Benchmark for Validating LLM-as-a-Judge on Agent Execution Traces

Co-first author, equal contribution.

877 agent execution traces unified from ten judge-validation datasets under a single two-level taxonomy, with native trace representations preserved.

Heatmap of ground-truth category counts for ten source benchmarks
across eleven error categories, with TraceJudgeBench as a balanced
final row. — Ground-truth category counts per benchmark, log scale. Every source corpus is sparse and lopsided, and each is missing a different set of categories; the bottom row is the unified corpus.

Clean-trace recognition is a systematic weakness: false-error rates of 34% to 78% across established judges, and 100% for a decomposed ensemble run without abstention.
Length is the hidden variable. Every judge degrades a further 37 to 50 points on long traces, which a length-concentrated corpus could not expose.
Abstention in verdict aggregation, rather than decomposition, recovers most of the gap.

AutoJudge: Automatic Generation of LLM-Based Judges from Execution Traces

Contributing author. Workshop on Failure Modes of Agentic AI at ICML 2026. OpenReview

A MetaAgent synthesises an evaluation pipeline per trace instead of reusing one fixed judge, with full-trace and summary modes for long inputs.

Towards Design of an Automated Judge for Multi-Agent Systems

Contributing author. AAAI 2026 TrustAgent Workshop.

Two-level evaluation methodology, agent and system. F1 0.85 on out-of-distribution TRAIL.

Code

MASeval

Contributor. BSD-3-Clause.

A Python library for automated evaluation of multi-agent systems, built on Pydantic AI. My contributions are judge and verifier implementation, including a no-ground-truth task-completion verifier, plus ablation tooling and reporting over the Who&When corpus.

AutoJudge

Contributor.

The reference implementation of the trace-adaptive judge above, in Python on Pydantic AI, OpenRouter and MCP. My contributions are the ARB/WebArena evaluation and the AEGIS experiments with and without tools, plus the cost analysis of running each judge design across benchmarks.

dogs-track

Developer. GPL-3.0.

Modified YOLOv8 tracking for counting stray dog populations from municipal camera streams — an applied system rather than research, written up afterwards as my BSc thesis. The problem is not detection but counting without double-counting, so the pipeline pairs a YOLOv8 detector with OC-SORT tracking and holds identity across a live stream, with per-camera state so counts attribute to a location. Camera endpoints are third-party infrastructure and are not included.

Background

I came into research from industry rather than through a PhD. Before ITMO I was an AI product manager at a regional bank, where I led the team that shipped computer-vision monitoring on municipal cameras to 92% accuracy against an 85% requirement, and before that the founding ML engineer on an environmental monitoring system I finished alone after the rest of the team left. That is where the interest in this problem comes from: a monitor nobody can act on is not a monitor.