Plans multilingual 400 billion parameter version, but starts with more modest fare
Meta has unleashed its latest large language model – named Llama 3 – and claims it will challenge much larger models from the likes of Google, Mistral, and Anthropic.on Thursday, Llama 3 is available in versions ranging from eight billion to over 400 billion parameters. For reference, OpenAI and Google's largest models are nearing two trillion parameters.
Meta claims Llama3-8B and 70B can outperform far larger models including Gemini Pro and Antrhopic's Claude 3 – Click to enlargeOne of the biggest gains, according to Meta, comes from the use of a tokenizer with a vocabulary of 128,000 tokens. In the context of LLMs, tokens can be a few characters, whole words, or even phrases. AIs break down human input into tokens, then use their vocabularies of tokens to generate output.
Llama 3's training dataset is more than seven times larger and contains four times more code than Llama 2, whichjust nine months ago. But, as the saying goes,"garbage in, garbage out" – so Meta claims it developed a series of data-filtering pipelines to ensure Llama 3 was trained on as little bad information as possible.
Training small models on such a large dataset is generally considered a waste of computing time, and even to produce diminishing returns in accuracy. The ideal mix of training data to compute resources is referred to as the"" amount. According to Meta, for an eight billion parameter model like Llama3-8B, this would be about 200 billion tokens.
At 8-bit precision, an eight billion parameter model requires just 8GB of memory. Dropping to 4-bit precision – either using hardware that supports it or using quantization to compress the model – would drop memory requirements by about half.Microsoft aims to triple datacenter capacity to fuel AI boom
Nigeria Latest News, Nigeria Headlines
Similar News:You can also read news stories similar to this one that we have collected from other news sources.
EU opens competition investigations into Apple, Google and MetaThe three tech giants are being investigated over concerns they have failed to comply with the EU’s Digital Markets Act.
Read more »
EU opens competition investigations into Apple, Google and MetaThe three tech giants are being investigated over concerns they have failed to comply with the EU’s Digital Markets Act.
Read more »
96% of US hospital websites share visitor info with Meta, Google, data brokersCould have been worse – last time researchers checked it was 98.6%
Read more »
Naver debuts multilingual HyperCLOVA X LLM it will use to build sovereign AI for AsiaBecause English isn't the only language
Read more »
How this open source LLM chatbot runner hit the gas on x86, Arm CPUsWay to whip that LLaMA's ass
Read more »
Databricks claims its open source foundational LLM outsmarts GPT-3.5In the AI gold rush, analytics outfit wants to provide the shovels
Read more »