Python problems written by Beam, kept only if they pass their own tests.
Beam writes an exercise, a reference solution and a set of unit tests. We run the solution against every test in a sandbox, then try to fool the tests with fake answers. A problem only makes it into the dataset if the solution passes and every fake answer gets caught.
Reference solution vs. its own tests
Fake answers the tests caught
Where problems drop out
Each step removes problems that fail it. The last bar is what ships in the dataset.
Coverage
Verified problems by topic and difficulty. Empty squares are combinations Beam hasn't produced yet.
Rejected problems
Why Beam's output got thrown away.
Browse problems
Everything that passed. Tests shown to the solver appear in the prompt; hidden tests decide the reward.