Skip to content Search
11.01.2024
“Our model, MoE-Mamba, outperforms both Mamba and Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in 2.2x less training steps while preserving the inference performance gains of Mamba against the Transformer,” write IDEAS NCBR researchers.

“The preliminary results indicate a very promising research direction that may allow scaling SSMs to tens of billions of parameters.”

Team of IDEAS NCBR researchers has unveiled MoE-Mamba, a combination of Mixture of Experts and State Space Models. This is joint work of Maciej Pióro, Kamil Ciebiera, Krystian Król, Jan Ludziejewski and Sebastian Jaszczur, members of research teams of Piotr Sankowski and Piotr Miłoś.

“By interleaving Mamba with efficient MoE layers we get the best of both worlds – lots of parameters, fast training, and linear time inference,” says Sebastian Jaszczur. “MoE and Mamba seems like a match made in heaven.”

Check it out at arXiv: https://arxiv.org/abs/2401.04081

And blog: https://llm-random.github.io/posts/moe_mamba/

Featured news

21.06.2024
Breakthroughs require freedom and patience
20.06.2024
Krzysztof Walas joins Adra Board of Directors
13.06.2024
I was told that doing computer science is not for women. Alicja Ziarko talks women in AI research