DBRX explained

DBRX is an open-sourced large language model (LLM) developed by Mosaic ML team at Databricks, released on March 27, 2024.^{[1] [2] [3]} It is a mixture-of-experts Transformer model, with 132 billion parameters in total. 36 billion parameters (4 out of 16 experts) are active for each token.^[4] The released model comes in either a base foundation model version or an instruct-tuned variant.^[5]

DRBX outperforms other prominent open-source models such as Meta's LLaMA 2, Mistral AI's Mixtral, and xAI's Grok and close-sourced models such as GPT-3.5 in several benchmarks ranging from language understanding, programming ability and mathematics.^{[6] [7]} As of March 28, 2024, this makes DBRX the world's most powerful open sourced model.^[8]

It was trained in 2.5 months on 3,072 Nvidia H100s connected by 3.2 terabytes per second bandwidth (InfiniBand), for a training cost of $10m USD.

Notes and References

1. Web site: 2024-03-27 . Introducing DBRX: A New State-of-the-Art Open LLM . 2024-03-28 . Databricks . en-US.
2. Web site: New Databricks open source LLM targets custom development TechTarget . 2024-03-28 . Business Analytics . en.
3. Web site: Ghoshal . Anirban . 2024-03-27 . Databricks' open-source DBRX LLM beats Llama 2, Mixtral, and Grok . 2024-03-28 . InfoWorld . en.
4. Web site: Mar 28, 2024 . A New Open Source LLM, DBRX Claims to be the Most Powerful – Here are the Scores . GIZMOCHINA.
5. Web site: Wiggers . Kyle . 2024-03-27 . Databricks spent $10M on new DBRX generative AI model . 2024-03-29 . TechCrunch . en-US.
6. Web site: 2024-03-27 . Databricks releases DBRX: open-source LLM that beats GPT-3.5 and Llama 2 . 2024-03-28 . Techzine Europe . en.
7. Web site: 2024-03-28 . Data and AI company DataBrix has launched a general-purpose large language model (LLM) DBRX that out.. . 2024-03-28 . . en.
8. Knight . Will . Inside the Creation of the World's Most Powerful Open Source AI Model . 2024-03-28 . Wired . en-US . 1059-1028.