OEIS pretraining ladder: 0.6B run r10b_600m_mlr2e-3_b2p99_s0 (as of 2026-10-16 22:36:41 EDT) model: 28L / 1024d Qwen3-shaped byte-level LM, 441M params, 10B tokens of OEIS (+ synthetic) progress step 47,768 / 105,029 (45.5%) tokens 4.55B / 10B 25,395 tok/s eta training done Mon 2026-10-19 10:14 EDT (59.6 h) train loss 0.8944 (mean of last 500 steps) gap to 50M on identical batches -0.0752 muon lr 1.26e-03 validation (every 2,500 steps; 50M / 5M ladder runs at the same step) step val_loss 50M gap 5M | Neo 50M 5M 2500 1.2263 1.2568 -0.0305 1.3355 | 20.94% 19.91% 16.91% 5000 1.1710 1.1759 -0.0049 1.2802 | 23.62% 22.83% 18.69% 7500 1.1133 1.1442 -0.0309 1.2546 | 25.85% 24.01% 19.54% 10000 1.0894 1.1150 -0.0256 1.2399 | 27.08% 25.21% 19.95% 12500 1.0597 1.0966 -0.0369 1.2295 | 28.01% 25.84% 20.44% 15000 1.0401 1.0844 -0.0443 1.2179 | 29.03% 26.29% 20.83% 17500 1.0267 1.0762 -0.0495 1.2111 | 29.78% 26.60% 20.95% 20000 1.0118 1.0662 -0.0544 1.2033 | 30.29% 27.14% 21.14% 22500 1.0004 1.0580 -0.0576 1.1997 | 30.68% 27.75% 21.50% 25000 0.9915 1.0534 -0.0619 1.1952 | 31.27% 27.70% 21.77% 27500 0.9802 1.0458 -0.0656 1.1899 | 31.81% 28.10% 21.63% 30000 0.9736 1.0406 -0.0670 1.1864 | 32.18% 28.40% 21.97% 32500 0.9667 1.0376 -0.0709 1.1833 | 32.40% 28.47% 22.08% 35000 0.9601 1.0291 -0.0690 1.1793 | 32.50% 28.81% 22.12% 37500 0.9537 1.0257 -0.0720 1.1764 | 32.94% 28.95% 22.20% 40000 0.9498 1.0227 -0.0729 1.1753 | 32.86% 29.03% 22.27% 42500 0.9444 1.0181 -0.0737 1.1723 | 33.35% 29.30% 22.43% 45000 0.9377 1.0132 -0.0755 1.1690 | 33.48% 29.44% 22.71% 47500 0.9348 1.0105 -0.0757 1.1682 | 33.71% 29.65% 22.73% evals (each 2,500-step checkpoint) step M1 median (50M) M1 macro | Ryskina (50M) mean lp median lp 2500 16.07 15.99 20.18 | 19/57 18 -3.208 -2.913 5000 15.13 13.86 18.62 | 26/57 23 -2.761 -2.149 7500 13.61 16.29 18.13 | 23/57 25 -2.598 -2.092 10000 13.72 15.62 18.04 | 27/57 29 -2.440 -1.947 12500 14.19 13.81 17.84 | 30/57 31 -2.395 -1.542 15000 12.64 13.88 19.48 | 30/57 32 -2.279 -1.603 17500 14.42 13.87 18.95 | 32/57 33 -2.264 -1.408 20000 12.83 12.65 19.33 | 33/57 32 -2.109 -1.378 22500 13.14 13.76 18.80 | 35/57 34 -1.975 -1.271 25000 13.12 13.27 17.68 | 33/57 32 -2.011 -1.200 27500 13.90 13.85 17.54 | 37/57 31 -2.037 -1.301 30000 12.70 13.39 18.04 | 36/57 34 -1.955 -1.129 32500 11.65 13.53 18.51 | 39/57 31 -2.026 -1.159 35000 11.35 13.85 16.80 | 36/57 33 -1.984 -1.207 37500 11.92 13.98 16.71 | 36/57 33 -1.980 -1.209 40000 12.67 13.85 17.37 | 36/57 36 -2.058 -1.301 42500 13.71 14.57 17.42 | 34/57 34 -1.991 -1.143 45000 11.83 13.98 17.21 | 36/57 36 -1.842 -1.079 47500 11.19 14.39 16.36 | 34/57 36 -1.941 -1.234 ladder finals at 10B tokens: 50M val 0.9642 Neo 31.44% M1 median 13.69 Ryskina 35/57 5M val 1.1388 Neo 23.46% M1 median 16.93 Ryskina 24/57 reference points: NextTerm-440M (prior model; 14B tokens, nibble tokenizer) OEIS-Eval-Neo: 34.41% Naive2 baseline, M1 median series MAPE: 11.93 more: /help /json /train?last=N&every=K /val /evals /config