A recurrent model trained decentralized, with no blocking and almost no quality loss

We trained a pure recurrent model across distributed GPUs with fully non-blocking synchronization, and it landed within 0.6% of the centralized baseline at matched steps. As far as we know, that is the first time decentralized non-blocking training has held up on a recurrent architecture.

The tradeoff we set out to break

Training a model across many GPUs in different places forces a familiar choice. Either the GPUs stop and wait to synchronize with each other, which is slow and expensive, or they skip the synchronization and let quality slip. Most decentralized training lives somewhere on that spectrum, trading speed for accuracy or the reverse. We wanted to show the choice is not fixed. In this run there was no blocking and no meaningful quality loss, within 0.6% of a centralized baseline at matched steps.

Why we picked the hardest case on purpose

We ran this on a pure recurrent model, Gated DeltaNet, with no transformer or mixture-of-experts layers to soften the result. That was deliberate. Recurrent models are sequential by nature, every step depends on the one before it, which makes them the hardest architecture to train without tight synchronization. Transformers are far easier to parallelize. If non-blocking decentralized training holds on the hardest case, the easier architectures should follow. Gated DeltaNet is our current target architecture for Parallax, and this is an in-progress research direction rather than a finished system, but the signal is clear.

What this means for Parallax

The point of Parallax is more useful work per watt, and strong training that does not depend on owning a datacenter. A recurrent model also drops the key-value cache that grows with every token, which is part of why the architecture is attractive to us in the first place. Getting it to train decentralized, without blocking and without a quality penalty, is a real step toward training strong models on hardware people already have.

Our low-level kernel work is open too. The MSA attention kernels are on GitHub at github.com/chutesai/MSA. That is the kernel layer, separate from this training run, but part of the same push to do more with less.

This work comes from Parallax's creator, Jon Durbin, who shared it directly: the non-blocking recurrent result and the synchronization approach behind Gated DeltaNet.

We will have more to share as this continues. If you want to follow along or start building on open models today, Chutes is at chutes.ai.