Products and agents
Labs
Test, compare and improve how AI behaves, without becoming a machine-learning engineer.
Labs is where Guth tests, compares and improves how AI behaves. Tilly, the research agent, guides the work so you do not need to be a machine-learning engineer. Parts of Labs are live today; most is in private preview.
What is live, and what is not
| Part of Labs | Status | What it means |
|---|---|---|
| Instrument and field guide docs | Live | Anyone can read the instruments and field guides catalogs. |
| Public experiment records | Live | Write-ups of past checks, such as the instrument verification pass, and the Labs experiments page. |
| Labs app (signed in) | Private preview | Invited preview members can view Labs. It is read-only or limited for now. |
| Experiment builder | Private preview | Building and running your own experiments in the app is part of the preview and still limited. |
| Personal training | In development | Training an agent on your own examples is being built. It is not available yet. |
What Tilly helps with
- Turning a question into an experiment.
- Choosing what to compare and how to score it.
- Collecting evidence and keeping a research journal.
- Comparing models, prompts, tools and settings.
- Reading results and finding where things fail.
- Spotting good training examples.
Experiments
i
A fair test with a written setup: what was tried, on which examples, how it was scored and what happened, so someone else can repeat it.
Common kinds:
- A/B tests: two versions side by side.
- Model comparisons: the same task on different models.
i
The AI system doing the thinking for an agent at a given moment. Models can be swapped; the agent keeps its identity, history and knowledge.
- Prompt sweeps: many small wording changes, to see which matters.
- Robustness tests: tricky or unusual inputs, to see what breaks.
- Did the training help?: checking a trained version against the old one.
Comparisons are evidence, not votes
When several models agree, that is not proof. Labs looks for evidence: scores on examples the model did not train on, failures that can be repeated, and results someone else can check.
How personal training will work In development
i
Teaching a model a specific job or agent from your own examples. Guth collects, checks, trains and tests, and switches it on only if it is better. In development.
- Collect examples.
- Group them by purpose.
- Check quality, consent and where each example came from.
- Train.
- Test on separate examples that were not used for training.
- Compare with how the agent behaved before.
- Turn it on only when the comparison justifies it.
i
Your choice to let your examples help train or improve a model. It is off unless you turn it on, and a bug report does not count as consent.
Local models on every device In development
Also in development: training and running local models across your own devices, such as PCs with graphics cards, Macs and even iPhones, working together as one fleet that shares one memory. You will be able to run chosen agents locally, so they stay with you, stay private and can work offline. This is separate from sharing idle computer power with the community, which is a later idea.
Where to go next
- Agents: meet Tilly and the other agents.
- Instruments: checked calculators and lookups that experiments can rely on.
- Release status: the current label for every product.