Madhuram-v1

Madhuram-v1 has 151 million parameters and was pretrained on one trillion tokens. We compare it with SmolLM2-135M, Gemma 3 270M and LFM2.5-230M, which were trained on two to nineteen times as much data.

Madhuram-v1 151M 1T

SmolLM2 135M 2T

Gemma 3 270M 6T

LFM2.5 230M 19T

As a base model, before any post-training, Madhuram-v1 has the best score on 9 of the 17 benchmarks, including both coding tasks, and the highest commonsense average. LFM2.5, trained on nineteen times as much data, leads on MMLU, CommonSenseQA, GPQA Diamond, TriviaQA, ARC Challenge and MATH-500.

Benchmark	Madhuram-v1	SmolLM2-135M	Gemma 3 270M	LFM2.5-230M
Commonsense reasoning
BoolQ	64.68	60.43	58.47	61.10
ARC Easy	59.89	58.88	56.94	59.64
ARC Challenge	33.19	29.52	28.16	35.41
PIQA	67.14	68.44	68.77	63.71
SIQA	40.12	39.25	39.97	38.02
WinoGrande	55.72	53.35	53.20	54.38
OBQA	32.00	32.80	30.40	32.60
HellaSwag	43.44	43.13	41.33	39.38
Average of the eight	49.52	48.23	47.16	48.03
Knowledge and reasoning
MMLU	35.29	24.49	26.04	40.10
MMLU Pro	13.45	7.12	6.91	11.88
GPQA Diamond	23.23	20.71	21.21	29.29
CommonSenseQA	32.35	20.39	20.39	38.33
SciQ	84.40	78.20	83.90	81.20
TriviaQA	7.47	5.02	9.66	12.88
Coding
MBPP	11.80	3.60	2.00	6.60
HumanEval	12.20	4.88	2.44	9.76
Mathematics
MATH-500	6.20	2.00	2.60	10.20

All benchmarks are 0-shot except MBPP, which is 3-shot, and MATH-500, which is 4-shot and scored with math-verify. Every model ran on the same harness.

We post-trained Madhuram-v1 and evaluated it on the same harness as Gemma 3 270M and LFM2.5-230M. It scores above Gemma 3 on every benchmark except GPQA Diamond, where all three models are at chance. It leads LFM2.5 on BBH, MMLU Pro and GSM8K with chain of thought, and trails it on instruction following, code and GSM8K without chain of thought.

BBH is 3-shot and GSM8K is 5-shot, averaged over flexible and strict answer parsing; all others are 0-shot. Every model ran on the same harness with its default generation settings. GPQA Diamond sits at chance, 25%, for all three.

Pass@k samples k answers per problem and counts the problem as solved if any of them is correct. It shows how much a model can solve when it gets more than one attempt.

• Madhuram-v1 151M
• Gemma 3 270M
• LFM2.5-230M

Show the numbers

Samples	Madhuram-v1	Gemma 3	LFM2.5
k = 2	19.21	15.04	21.12
k = 8	27.46	19.64	31.58
k = 16	31.89	21.56	36.38
k = 32	36.62	23.03	40.79
k = 64	41.76	24.24	46.34

Show the numbers

Samples	Madhuram-v1	Gemma 3	LFM2.5
k = 1	35.8	13.4	30.8
k = 4	39.5	28.1	36.8
k = 8	40.4	36.2	37.9
k = 16	40.7	43.7	40.0

Same harness and prompts for every model, each with its default generation settings.

We think a bigger post-training run can push these numbers further.

With Madhuram v2 we want to go further still: larger models than 151M parameters, and more training data. If you would like to back the next model or work with us, we would like to hear from you.

[email protected] @maruthlabs on X