01Problem sets
There are two sets of 300 problems, each generated from a fixed seed with QubitFlow’s mulberry32 generator. Each problem has 4 to 6 jobs, whole-number costs from 1 to 9, values from 0 to 20, and a budget of 30–70% of the total cost. The development set (seed 987654321) was used once, to choose the CVaR level α = 0.3 and the γ range. The hold-out set (seed 20261009) was not used for any choice, and every figure quoted elsewhere on this site comes from it. The showcase example on the home page is in neither set.
02Metrics
- Optimal: the most frequently sampled plan (seed 12345, 1,024 shots) has the best possible value and is within budget.
- Within budget: the candidate’s cost is at most the budget.
- Greedy optimal: the value-per-cost greedy planner reaches the optimum. It is the same for every search configuration.
- Timing is not shown here; it depends on the device. The lab reports wall-clock time for each run.
03Reading the results
On small problems like these, the greedy planner is a strong baseline: it is optimal about five times in six. The simulated circuit’s candidate is optimal less often, and against greedy it loses more often than it wins. More layers help a little. The textbook plain-mean objective is far worse: with a large penalty it collapses towards cheap plans, which is why QubitFlow searches with CVaR.
None of this says anything about quantum hardware or speed. It measures one carefully specified, simulated algorithm against two classical answers on the same inputs.
04Command line
From the source code, npm run bench prints the same table, and npm test fails if any published hold-out figure changes.