Case study · Agentic ML

Autonomous Data Analyst
an agent that refuses to lie about your data.

Upload a table. It cleans the data, decides what to predict, trains and ranks models, measures what actually drove the predictions — and when the data can't support the question, it says so instead of returning a confident number. A LangGraph agent that uses an LLM only where judgement is needed and keeps every metric deterministic.

Live demo ↗ Repository ↗

0.901

TITANIC ROC-AUC

LEAKAGE DETECTORS

LLM CALLS / ROUND (MAX)

PIPELINE STEPS

01 · The problem

Hand an LLM a dataset and ask it to "analyze this," and it will happily produce a confident, well-formatted, and wrong report — inflated accuracy from a leaked column, a conclusion the sample size can't justify, a chart that doesn't mean what its caption says. The failure mode isn't a lack of capability; it's a lack of restraint.

Autonomous Data Analyst is built around the opposite instinct: an agent that treats "I can't support that claim" as a first-class output. It runs a fixed, auditable pipeline on any CSV and gates every conclusion behind checks it can't talk its way past.

02 · How it runs

A LangGraph state machine. The LLM only makes judgement calls — choosing a target, deciding what to do about a weak result, writing the prose. Profiling, training, explanation and the quality gate are pure Python, so a hallucinated plan can never reach a pipeline or invent a number.

clean & profiledeterministic · schema · dupes · missingness

LEAKAGE AUDIT · MUTUAL INFO + LINEAR RECONSTRUCTION

plannerLLM · picks target & model plan

trainingdeterministic · fit · CV · rank

EXPLAIN · SHAP (TREES) / PERMUTATION

reflectLLM · retry ×≤3 · margin 0.02 · or abandon

summary → reportLLM prose · only evidence-backed claims

A completed run: clean, plan, train, explain, reflect, report steps all green, with permutation feature importances — A finished run on Titanic — every step logged (clean → report), with measured permutation importances. Note the banner: the alive column was removed for restating the target.

03 · Two ways it catches a lie in the data

Most silent ML failures are leakage — a feature that quietly encodes the target. One detector isn't enough, so the agent runs two independent ones and drops a column before training if either fires:

Mutual information caught Titanic's alive column — it determines survived exactly (MI = 1.000). Left in, it would have produced a perfect score and learned nothing. Linear reconstruction caught the bike-share cnt target, which is just casual + registered (R² = 1.000): any model scores perfectly by doing arithmetic, so the run was abandoned rather than reported.

Titanic run banner: columns removed for restating the target — alive, mutual information 1.000 — Mutual information — `alive` flagged (MI 1.000) and removed before a model was trained.

Bike cnt target flagged as derived from casual and registered with R2 1.000, run abandoned — Linear reconstruction — `cnt` = casual + registered (R² 1.000). The agent reflects, marks it weak, and **abandons**.

A real but meaningless score is worse than no score. The whole point is that the agent recognises the difference.

04 · The guardrails

It abstains.

When reflection judges a result unsound — a reconstructed target, too little data, an unstable metric — the run ends in abandon and the report says why. "No supportable finding" is a valid, intended output.

The LLM never touches a number.

Profiling, training, explanation and the quality gate are deterministic Python. The model chooses the target and writes prose, but every metric on the page was measured, not narrated.

Bounded retry.

The reflect → train loop caps at three attempts, and a retry only counts if it beats the incumbent by 0.02 on the gate metric — with no-op detection so it can't churn on noise. No infinite agent spirals, no runaway token spend.

Prompt-injection resistant.

CSV contents are data, never instructions. Directives smuggled through column names or cell values never reach the planning prompt.

05 · The interface

A React + Vite front end over a FastAPI backend, PostgreSQL for durable run history, and LangSmith traces that mirror the graph's real execution including retries. Every project keeps its datasets, runs and reports; you can also just ask the data a question in plain language.

Home dashboard: projects with rows, ROC-AUC or R2, and leakage flags per dataset — Project home — each dataset shows its strong/weak verdict and any leaked column caught.

Per-dataset view: profiler findings, suggested target, and a natural-language Q&A panel — Per-table view — profiler findings, a suggested target, and a grounded natural-language Q&A.

06 · Lessons & where it goes next

Restraint is a feature.

The hard part of an autonomous analyst isn't getting an answer — it's getting it to withhold one. The abstain path is what makes the confident outputs trustworthy.

Two weak checks beat one strong one.

Mutual information and linear reconstruction each miss cases the other catches. Redundant, independent detectors are cheaper insurance than a single clever one.

Draw the LLM boundary hard.

Letting the model judge but never measure kept the whole system debuggable — same CSV, same profile, same numbers, every time.

Try the live demo ↗ Explore the repo ↗ More projects →