Scaling Sliders

Bigger models, more examples, and more practice time all tend to help — but only up to a point, and how much they help depends on the other two.

This is a simplified toy illustration — not a reproduction of real scaling-law experiments. Each curve comes from an actual client-side training run.

Scale knobs

Model size Small

2 layers, 48-dim embeddings

Data size 50%

Fraction of available training sequences used for this run.

Compute 180 steps

Gradient updates per run — more steps can help until loss plateaus.

Change one knob

Hold the other two still so you can see what caused the change.

Loss comparison

Validation loss

What it writes

Start a run to see a sample of what the model writes.

Best config so far

Run a training configuration to start tracking the lowest validation loss.

Things to try

The model learns to predict the next word in this short story. Data size is how much of it each run uses.

the cat sat on the mat
the dog sat on the log
the cat ran to the mat
the dog ran to the log
a cat and a dog played
the mat is flat and soft
the log is long and brown
the cat saw the dog run
the dog saw the cat sit
on the mat the cat sat
by the log the dog ran
the cat and dog rest
the sun is warm today
a dog ran to the log
a cat sat on the mat
the cat sat by the log
the dog ran on the mat
the sun is bright today
the cat and dog sat
the mat and log are brown