https://www.zyphra.com/post/zamba2-7b [66c3951edd1c45b9] * About * Our Work * Get Started Get Started * About * Our Work * Get Started Back to Newsroom [6706d880c339bfc1098f384a_shubham-dhage-l1xJB5sN1ew-unsplash] [670700157cd6aceff5b383ed_zamba27b] October 14, 2024 PALO ALTO, CALIFORNIA Zyphra is excited to release Zamba2-7B, a state-of-the-art small language model. At the 7B scale, we outperform the leading models of Mistral, Google's Gemma and Meta's Llama3 series in both quality and performance. We believe Zamba2-7B is the leading model for running on-device and on consumer GPUs as well as for many enterprise applications which require a powerful but compact and efficient model for natural-language tasks. Authors Zyphra Team Collaborators Daniel A Roberts (Sequoia Capital & MIT), Andrey Gromov (Meta FAIR), Kushal Tirumala (Meta FAIR) and Hassan Shapourian (Cisco) TABLE OF CONTENTS Zamba2-7B HighlightsModel QualityZamba2-7B Inference Performance Model QualityZamba2-7B Inference Performance Zamba2-7B HighlightsModel Quality Zamba2-7B HighlightsModel QualityZamba2-7B Inference Performance Zamba2-7B HighlightsModel QualityZamba2-7B Inference Performance Zamba2-7B HighlightsModel QualityZamba2-7B Inference Performance Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Quality vs. Inference Speed Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance [6707df1fe1] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality [670b31c473] Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance [6707df1fe1] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [6707de16d0] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. Zamba2-7B Highlights Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality [670b31c473] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance [6707df1fe1] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707de16d0] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6708068af8][6707de29d7] [6707de3193] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality [670b31c473] Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [6707df1fe1] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707de16d0] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6708068af8] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. [670b31c473] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. Zamba2-7B Inference Performance Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707df1fe10f98f8b7269326_evals_new] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6707de16d0e11b5bf55867a0_tokens_mmlu] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. [670b31c473f57e3c96e35298_ttft_mmlu] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [6707df1fe10f98f8b7269326_evals_new] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707de16d0e11b5bf55867a0_tokens_mmlu] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). Zamba2-7B Inference Performance Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) [670b31c473f57e3c96e35298_ttft_mmlu] Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Model Quality Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. Zamba2-7B Inference Performance Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance [670b31c473f57e3c96e35298_ttft_mmlu] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6707df1fe10f98f8b7269326_evals_new] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. [6707de16d0e11b5bf55867a0_tokens_mmlu] [6708068af8] [6707de29d70a1d7c77b74e06_ttft] [6707de3193] [6707de35353887430253284d_memory] Link to Cookbook (GitHub) Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [670b31c473] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707df1fe1] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. What is Annealing? [6707de16d0] [6708068af8] [6707de3535] Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality [670b31c473] Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance [6707df1fe1] [6707de16d0][6708068af8][6707de29d7] [6707de31937ef3a27a78df29_throughput] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) [670b31c473f57e3c96e35298_ttft_mmlu] Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Model Quality Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [6707df1fe1] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707de16d0e11b5bf55867a0_tokens_mmlu] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6708068af80b909c9424ad52_Zamba2_7B_DarkMode] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Inference Performance [6707de29d70a1d7c77b74e06_ttft] Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [670b31c473] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6707df1fe1] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707de16d0] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. [670b31c473f57e3c96e35298_ttft_mmlu] [6707df1fe10f98f8b7269326_evals_new] Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [6707de16d0e11b5bf55867a0_tokens_mmlu] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6708068af80b909c9424ad52_Zamba2_7B_DarkMode] Zamba2-7B Inference Performance We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6707de29d70a1d7c77b74e06_ttft][6707de31937ef3a27a78df29_throughput] [6707de35353887430253284d_memory] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. Zamba2-7B Highlights * Zamba2-7B achieves SOTA evaluation benchmark performance and superior inference efficiency compared to currently leading 7B models such as Mistral-7B, Gemma-7B, and Llama3-8B * Zamba2-7B is extremely inference-efficient, achieving 25% faster time to first token, a 20% improvement in tokens per second, and a significant reduction in memory usage compared to models such as Llama3-8B. * Architectural improvements over Zamba1-7B: + Mamba1 blocks have been replaced with Mamba2 blocks + Instead of a single shared attention block, we utilize two shared attention blocks which are interleaved in an ABAB pattern throughout the network. + We apply a LoRA projector to each shared MLP block, which allows the network to specialize the MLPs at each invocation of the shared layer across depth. * We release the model weights open-source (Apache 2.0) Model Quality Zamba2 performs exceptionally well on standard language modeling evaluation sets, especially given its latency and generation speed. Among small language models (<9B), we lead the pack in both quality and performance. Zamba2-7B Inference Performance Our model outperforms existing state-of-the-art models for the following reasons: 1. Our novel shared-attention architecture allows more parameters to be allocated to the Mamba2 backbone. In turn, the shared transformer block preserves the rich cross-sequence dependencies of the attention computation. 2. Our 3 trillion token pre-training dataset, which is composed of a combination of Zyda and openly-available datasets that are aggressively filtered and deduplicated and achieves a state-of-the-art quality in ablations vs the existing top open-source pretraining datasets. 3. We have a separate "annealing" pre-training phase, which rapidly decays the learning rate over 100B high-quality tokens. Our annealing set is carefully curated for quality and collated from varied high-quality sources. Due to the exceptional quality of our pretraining and annealing datasets, Zamba2-7B performs extremely well on a per-training-token basis, sitting comfortably above the curve traced out by competitor models. [670b31c473] Zamba2-7B utilizes and extends our original Zamba hybrid SSM-attention architecture. The core Zamba architecture consists of a backbone of Mamba layers interleaved with one or more shared attention layers (one shared attention in Zamba1, two in Zamba2). This attention has shared weights to minimize the parameter cost of the model. We find that concatenating the original model embeddings of the input to this attention block improves performance, likely due to better maintenance of information across depth. The Zamba2 architecture also applies LoRA projection matrices to the shared MLP to gain some additional expressivity in each block and allow each shared block to specialize slightly to its own unique position while keeping the additional parameter overhead small. [6707df1fe1] We achieve state-of-the-art inference efficiency, including latency, throughput and memory usage because: 1. Mamba2 blocks are extremely efficient, and have roughly 4 times the throughput of an equal-parameter transformer block. 2. Mamba blocks only have small hidden states to store and don't require a KV-cache, so we only need to store KV states for the invocations of the shared attention block. 3. We choose model sizings that are very amenable to parallelization on modern hardware (i.e. multiple streaming multiprocessors on GPUs, multiple cores on CPUs). [6707de16d0] Zamba2-7B was trained on 128 H100 GPUS for approximately 50 days using our internal training framework developed atop Megatron-LM. Zamba2-7B thus demonstrates that at the 7B scale the frontier is still reachable and surpassable with a small team and moderate budget. Zamba2-7B will be released under an open source license, allowing researchers, developers, and companies to leverage its capabilities. We invite the broader AI community to explore Zamba's unique architecture and continue pushing the boundaries of efficient foundation models. A Huggingface integration is available here, and a pure-pytorch implementation is available here. Zyphra's team is committed to democratizing advanced AI systems, exploring novel architectures on the frontier of performance, and advancing the scientific study and understanding of powerful models. We look forward to collaborating with others who share our vision. [6708068af8] [6707de29d7] [6707de3193] [6707de3535] Analysis of Global Duplicates We present histograms depicting distribution of cluster sizes in all the datasets (see Fig. 7-11). Please, note that all the figures are in log-log scale. We see a significant drop in the number of clusters starting from the size of around 100. This drop is present both in DCLM and FineWeb-Edu2 (see Fig. 8 and 9 respectively), and most likely is explained by a combination of the deduplication strategy and quality when creating both datasets: DCLM deduplication was done individually within 10 shards, while FineWeb-Edu2 was deduplicated within every Common Crawl snapshot. We find that large clusters usually contain low quality material (repeated advertisements, license agreements templates, etc), so it's not surprising that such documents were removed. Notably, DCLM still contained one cluster with the size close to 1 million documents, containing low quality documents seemingly coming from the advertisements (see Appendix).We find both Zyda-1and Dolma-CC contain a small amount of duplicates, which is expected, since both datasets were deduplicated globally by their authors. Remaining duplicates are likely false negatives from the initial deduplication procedure. Note, that distribution of duplicates clusters sizes of these two datasets (Fig. 10 and 11) don't contain any sharp drops, but rather hyper exponentially decreases with cluster size. [placeholde] Figure 7: Distribution of cluster sizes of duplicates in global dataset (log-log scale). [670ce32dec] Figure 8: Distribution of cluster sizes of duplicates in DCLM (log-log scale). [670ce39359] Figure 9: Distribution of cluster sizes of duplicates in FineWeb-Edu2 (log-log scale). [670ce40649] Figure 10: Distribution of cluster sizes of duplicates in Zyda-1 (log-log scale). [670ce44b1a] Figure 11: Distribution of cluster sizes of duplicates in Dolma-CC (log-log scale). Largest cluster in DCLM Below is an example of the document from the largest cluster (~1M documents) of duplicates in DCLM (quality score 0.482627): Is safe? Is scam? Is safe for your PC? Is safe or is it scam? Domain is SafeSafe score: 1 The higher the number, the more dangerous the website.Any number higher than 1 means DANGER. Positive votes: Negative votes: Vote Up Vote Down review Have you had bad experience with Warn us, please! Examples of varying quality score in DCLM in a cluster Below one will find a few documents with different quality scores from DCLM coming from the same duplicates cluster. Quality score varies from ~0.2 to ~0.04. (c) 2024 Zyphra Technologies Inc. All rights reserved. AboutFeaturesNewsroomPricing ResourcesTerms Of UsePrivacy Policy