{"id":20855,"date":"2024-01-11T09:51:43","date_gmt":"2024-01-11T09:51:43","guid":{"rendered":"https:\/\/ideas-ncbr.pl\/?p=20855"},"modified":"2024-04-03T13:54:30","modified_gmt":"2024-04-03T13:54:30","slug":"moe-mamba-ideas-ncbr-researchers-combine-two-llm-architectures","status":"publish","type":"post","link":"https:\/\/ideas-ncbr.pl\/en\/moe-mamba-ideas-ncbr-researchers-combine-two-llm-architectures\/","title":{"rendered":"MoE-Mamba. IDEAS NCBR researchers combine two LLM architectures"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">&#8220;The preliminary results indicate a very promising research direction that may allow scaling SSMs to tens of billions of parameters.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Team of IDEAS NCBR researchers has unveiled MoE-Mamba, a combination of Mixture of Experts and State Space Models. This is joint work of Maciej Pi\u00f3ro, Kamil Ciebiera, Krystian Kr\u00f3l, Jan Ludziejewski and Sebastian Jaszczur, members of research teams of Piotr Sankowski and Piotr Mi\u0142o\u015b.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">\u201cBy interleaving Mamba with efficient MoE layers we get the best of both worlds \u2013 lots of parameters, fast training, and linear time inference,\u201d says Sebastian Jaszczur. &#8220;MoE and Mamba seems like a match made in heaven.&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Check it out at arXiv: <a href=\"https:\/\/arxiv.org\/abs\/2401.04081 \" rel=\"nofollow noopener\" target=\"_blank\">https:\/\/arxiv.org\/abs\/2401.04081 <\/a><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">And blog: <a href=\"https:\/\/llm-random.github.io\/posts\/moe_mamba\/\" rel=\"nofollow noopener\" target=\"_blank\">https:\/\/llm-random.github.io\/posts\/moe_mamba\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u201cOur model, MoE-Mamba, outperforms both Mamba and Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in 2.2x less training steps while preserving the inference performance gains of Mamba against the Transformer,\u201d write IDEAS NCBR researchers.<\/p>\n","protected":false},"author":27,"featured_media":20856,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":"","_links_to":"","_links_to_target":""},"categories":[79],"tags":[],"class_list":["post-20855","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-news-en-2"],"acf":[],"_links":{"self":[{"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/posts\/20855","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/users\/27"}],"replies":[{"embeddable":true,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/comments?post=20855"}],"version-history":[{"count":3,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/posts\/20855\/revisions"}],"predecessor-version":[{"id":22115,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/posts\/20855\/revisions\/22115"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/media\/20856"}],"wp:attachment":[{"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/media?parent=20855"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/categories?post=20855"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ideas-ncbr.pl\/en\/wp-json\/wp\/v2\/tags?post=20855"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}