We're Still Here

It's been a while since our last update, and we know a quiet blog can look like a dead project. It isn't. OpenGCM is still a small team working in the open on language models, and the gap between posts is mostly a gap in writing, not in work. Most of what's happened since GCM MARK II has gone into the Hydrion line, and it's worth catching up on — all of it, not just the current model.

Hydrion yap

Hydrion started as a simple question: how far can you push a language model trained entirely from scratch, on consumer hardware, before you run out of compute rather than ideas. Hydrion v1 was the first real answer to that: 114M parameters, 2.5B tokens, trained on an RTX 3060 with some rented A100 time to fill the gaps. It proved the basic premise, and set the pattern the whole project has followed since — small models, real pretraining, no shortcuts.

What we call Hydrion v3 today isn't really the third model we've built. It's closer to the seventh, once you count every architecture and training run that came before it. We're keeping the v3 name anyway, because the version number was never meant to track attempts — it tracks generations of the project that actually shipped. (If you're wondering where the line is: an attempt is anything we tried and learned from; a generation is a model we finished and stood behind. v1 and v3 cleared that bar. The Mamba hybrid below didn't.) v2 was going to be one of those generations. It wasn't a smaller step than v3; it was a different bet entirely: a Mamba2/Transformer hybrid, aiming to bring state-space efficiency to a model this size. We got far enough into it to find out why that's hard — ABI and torch-pinning issues in `mamba_ssm` that made the training setup fight us at every step — and made the call to scrap it rather than keep fighting the tooling. No shame in that; it's the kind of dead end that's only obvious in hindsight, and it's part of why v3 exists in the form it does: back on a straightforward, well-understood transformer architecture (GQA, SwiGLU, RoPE), where we could put the effort into training quality instead of infrastructure archaeology.

Once v3 was underway on that architecture, it went through repeated rounds of pretraining and evaluation, each one used to diagnose a specific weakness and correct for it in the next. It wasn't a straight line. One round would fix a weak spot and accidentally weaken something else; the next round would try to patch that without giving back the gain. Quantitative reasoning was the clearest example: it started out barely above chance, and it took several rounds of deliberately reshaping the data mix — and, on one occasion, relearning the lesson that you can't leave a source's weight flat while shrinking the pool around it, or its effective share silently drops — before it climbed to a level we were actually satisfied with. Code completion, on the other hand, has been the stubborn one: it's drifted up and down across almost every round without a clean fix yet, and we're still not sure whether that's a real weakness or just noise from a benchmark with too few examples per category. That's the kind of grinding, iterative work that doesn't make for a good headline but is most of what training a model from scratch actually looks like.

One of those rounds surfaced a bug worth telling properly, because it's a good example of the kind of thing that eats weeks and never makes it into a changelog. Early in v3's training, we noticed the model had picked up a strange tic: instead of just answering a question, it would output something like "COMPLETION: ... Explanation: X is X" — a rigid, templated non-answer instead of real content. It took a while to track down, but the cause turned out to be a specific training dataset that had reformatted its source material into a strict question/answer template, and the model had latched onto that template as if it were the shape all answers should take. The fix had two parts: a corrective fine-tuning pass that actively penalized the model for producing that exact pattern, and, longer term, removing the offending datasets from the pretraining mix entirely and replacing them with more natural sources. It worked — a later test with a plain, unscaffolded prompt came back as normal, coherent prose with no trace of the artifact. Small thing in isolation, but it's the kind of bug that, left alone, would have quietly capped how good the model could ever get.

The most significant recent decision was around model size. To be eligible for community benchmark comparisons in the sub-150M parameter class, we first tried compressing v3 down through pruning and healing: trimming parameters out of an already-trained checkpoint and then continuing to train it to recover the lost capability. It worked better than expected in some respects and worse in others — strong on flexible, free-form evaluation, but clearly weaker on stricter benchmarks than a same-sized model trained from scratch would be, even after a substantial amount of recovery training. Digging into why made it clear the shrinking process had done more real damage than our tools predicted it would, and no reasonable amount of additional recovery training was going to fully close that gap. Rather than ship a compromise, we made the call to abandon prune-and-heal entirely and train a dedicated 143M-parameter model from the ground up, with a data mix built specifically to be competitive rather than reused from v3 as-is. That run is underway now, and early checkpoints are already outperforming the pruned attempt at a fraction of the planned training budget.

So what's the actual goal here, beyond "make it better than the last one"? Hydrion v3, in its from-scratch 143M form, is being built to land at or near the top of the Open SLM Leaderboard in its size class — not just to clear the eligibility bar, but to genuinely compete with the best small models out there. That's a real target with real numbers attached, not a vague aspiration, and it's the standard we're training against.

None of this is finished, and we're not going to pretend it is. But active, ongoing training is a good problem to have, and it's the reason posts have been infrequent — the team's attention has gone into the models, not the changelog.

It doesn't have a real name yet, on purpose — we're not slapping a name on it until it's done and the results are in, rather than committing early and having to walk it back. Internally we've been calling it Project Ripple, and that's what we'll keep calling it here until it earns a proper name. "Done" has a concrete shape: finish the training run, run the full leaderboard eval suite, and submit the result. Once that's wrapped, expect a proper benchmark writeup with the real numbers, not just checkpoint snapshots.

What to expect next: continued training and evaluation on the from-scratch 143M model, a proper benchmark writeup once results settle, and more regular updates here going forward, including the smaller milestones we'd normally consider too minor to post about. If you've been checking in and wondering whether OpenGCM was still active, this is your answer. We are, and Hydrion v3 is where the work is.

GCM yap

One more thing worth being upfront about, since this is meant to be an honest update and not just a highlight reel: we don't currently have plans for a GCM Mark III. GCM Mark I and Mark II were learning experiences more than results — in practice that meant fine-tuning on whatever coding datasets we could find, including some fairly rough ones, without a clear enough picture of what was actually driving quality up or down. That's not a knock on the idea behind GCM, just an honest read on where the execution was at the time, and it's part of why the org's name still leads with GCM even though the work has moved elsewhere.

That "elsewhere" is Hydrion, and it's where our attention is staying for the foreseeable future. Part of that is the pretraining work itself, but part of it is genuinely enjoying the small language model community — we've gotten to know people from labs like Axiomic Labs, Bench Labs, and Minima Labs, and being around people who are deep in this space has made the work better and more fun. A GCM Mark III isn't ruled out forever, but for now it's Hydrion, and we're fine with that.