WeightsLab by Graybx · on-prem training intelligence
See what your model learns from. Fix it while it trains.
WeightsLab traces what your model does back to the training samples behind it, then lets you drop, retag or re-annotate them while the run is still going, with no restart and no second full training.
$ pip install weightslab
$ weightslab start
- run_039
- run_040
- run_041
- run_042
Today: run after run the loss plateaus and nothing says why. You guess, relabel and retrain from scratch.
- Mid-runObserve, diagnose, intervene
- 22k+PyPI downloads from 20+ countries
- PyTorchImages, video, text, tabular & LiDAR
- On-premRuns on your GPUs, with your data
Problem 1 · In training
Lack of visibility and control of the training process costs you
- 30hper week
- $1Mper month
Dashboards show the loss curve. They do not say which samples, labels or edits caused a bad result, so teams guess, relabel and start another full run.
You see that a run went badly, but rarely why.
Three costs of not seeing inside the run.
- 01
Unknown coverage
You can't tell whether the cases that matter are in the data, or in what quantity.
- 02
Slow diagnosis
A failure doesn't trace back to the samples, labels or model changes behind it.
- 03
Expensive iteration
Every fix means another full run, more labelling and more compute.
Problem 2 · In production
A model that is 99% accurate still fails 1 time in 100.
Most teams get to 99% without much trouble. What is left is the last 1%, then 0.1%, then 0.01%: rare edge cases that barely show up in the training data and are hidden by an aggregate accuracy number, yet each one is a wrong decision in production. Teams track progress in nines: 99%, 99.9%, 99.99%.
The march of nines
Failures left per million decisions at each level of accuracy.
- 99%10,000 failures · 1 in 100
- 99.9%1,000 failures · 1 in 1,000
- 99.99%100 failures · 1 in 10,000
“Every single nine is the same amount of work.”
Going from 99% to 99.99% is mostly a data problem, and WeightsLab is built to speed that up.
Where the last nines matter most.
The same long tail shows up wherever a wrong answer is costly.
-
Physical AI
One rare scene can stop a line or cause a crash.
Robots and vehicles act on what their sensors show them, so a scene they saw too rarely in training is where they go wrong.
See the use case -
Medical imaging
A rare finding the model saw too few times can end up as a missed diagnosis.
A small lesion or an unusual presentation may appear only a handful of times in the training set, and those are the cases you most need the model to get right.
See the use case -
Other sensitive domains
Where mistakes cost a lot and the data has to stay in house.
Safety-critical, regulated and private data often cannot be sent to an outside service, so the review has to happen where the data already is.
See how on-prem works
What a wrong decision costs.
From a failure to a fix
A sample fails in production. Your training workspace helps you find out why.
Monitoring tells you that a prediction was wrong. It does not tell you why. The cause often sits in the training data: a case the model barely saw, a label that is wrong, a near-duplicate that skewed the loss.
With WeightsLab you add the failed samples to the workspace and infer them with your current model, so each one gets the same per-sample signals as your training data. You then project them into the training embedding, see where they land among the training samples, and work out why the model gets them wrong. When the cause is in the data, you fix it and fine-tune your model, then check that its performance on your use case does not drop.
-
01 · In production
A night-shift frame, part not detected
On a production line, a failure like this can stop the line. You have the sample that failed, not the cause.
$2.3Mper hour of unplanned downtime in a large automotive plant
11%of revenue lost to unplanned downtime, estimated for the world's 500 largest companies
-
02 · Add it and infer
Run the failed samples through your current model
Each one gets the same per-sample signals as your training data.
loss 2.9confidence 0.31 -
03 · Project it
See where it lands among the training samples
They land far from every cluster, next to a thin one: the model has barely seen this kind of sample.
They land inside the day cluster: night samples are read as day, so the model applies the wrong rules.
-
04 · Understand and fix
Night frames are 1.8% of the training set, and 38 have labels missing
Night frames are drawn into the day cluster, so they are treated as day
Re-annotate 38 labelsResume from checkpoint ▸Tag them as nightResume from checkpoint ▸
Explore
Where to look next.
-
Product
How WeightsLab works
The workspace, the built-in agent and a few lines of integration, then how you observe, diagnose and intervene inside the run.
See the product -
Use cases
Where it matters
Physical AI, LLM training and fine-tuning, medical imaging and general AI training.
See the use cases -
Sandbox
Try it in your browser
Four setups, from a toy MNIST case to robot manipulation and LLM data.
Open the sandbox page -
About
Who we are
Why we started WeightsLab, and answers to the questions we hear a lot.
Meet the team
See it on one of your runs.
Try the sandbox in your browser, or book 30 minutes with us and bring a real run.