RBS-Attention: How to Speed Up Long-Context LLMs 20.65x Without Training

RBS-Attention: Architecture & Actionable Implementation

RBS-Attention is a training-free sparse prefill method designed to address the challenges of long-context large language model inference. It introduces two complementary selection branches: a centroid base branch and a rescue branch. The centroid base branch captures average relevance, while the rescue branch uses the maximum key-block radius and its prompt-, layer-, and head-dependent distribution to identify blocks at risk of underestimation.

Key Components

RBS-Attention consists of the following key components:

1. Centroid Base Branch: Captures average relevance.
2. Rescue Branch: Identifies blocks at risk of underestimation using the maximum key-block radius and its prompt-, layer-, and head-dependent distribution.
3. Thresholding and Combining Masks: Independently thresholds the two branches and combines their masks to control the contribution of rescue blocks while preserving regular block-sparse FlashAttention execution.

Implementation

To implement RBS-Attention, follow these steps:

1. Install Dependencies: Ensure you have the necessary dependencies installed, including PyTorch and vLLM.
2. Define the Model Architecture: Use the provided PyTorch code snippets to define the RBS-Attention model architecture.
3. Train the Model: Train the model using your dataset, ensuring to use the RBS-Attention sparse prefill method.
4. Evaluate the Model: Evaluate the model using your test dataset to ensure it meets your performance requirements.

import torch
import torch.nn as nn

class RBSAttention(nn.Module):
    def __init__(self, d_model, n_heads, max_radius):
        super().__init__()
        self.d_model = d_model
        self.n_heads = n_heads
        self.max_radius = max_radius
        
        # Centroid base branch
        self.centroid_base = nn.Linear(d_model, n_heads)
        
        # Rescue branch
        self.rescue_branch = nn.Linear(d_model, n_heads)
        
        # Thresholding and combining masks
        self.threshold = nn.Parameter(torch.tensor(0.5))
        
    def forward(self, x):
        # Centroid base branch
        centroid_scores = self.centroid_base(x)
        
        # Rescue branch
        rescue_scores = self.rescue_branch(x)
        
        # Thresholding and combining masks
        centroid_mask = (centroid_scores > self.threshold).float()
        rescue_mask = (rescue_scores > self.threshold).float()
        combined_mask = centroid_mask + rescue_mask
        
        return combined_mask

# Example usage
model = RBSAttention(d_model=512, n_heads=8, max_radius=10)
x = torch.randn(10, 512)  # Example input tensor
output = model(x)
print(output.shape)  # Expected output shape: torch.Size([10, 8])

Hard Production Trade-Offs (The Cost of Scale)

Latency vs. Throughput

RBS-Attention addresses the latency vs. throughput trade-off by reducing the prefill time for long-context LLMs. The method achieves a 20.65x speedup on H100 GPUs, reducing the time to first token (TTFT) and improving the tokens-per-second throughput.

Unit Economics & TCO

The unit economics and total cost of ownership (TCO) of RBS-Attention are favorable. The method reduces the GPU memory footprint and inference serving costs per million tokens, making it a cost-effective solution for long-context LLMs.

Failure Domains

RBS-Attention mitigates several failure domains, including context rot, prompt injection surface area, schema drifting, and tool-call non-determinism. The method’s training-free approach ensures that the model remains stable and reliable in production.

Benchmark Radar Capability Data