Multimodal LLM Architecture: How Vision Gets Aligned With Language

Multimodal Large Language Models (MLLMs) have rapidly evolved, transitioning from unimodal text processing to sophisticated architectures capable of understanding both vision and language. The crux of this evolution lies in Modality Alignment—the mechanism by which continuous visual features are mapped into the discrete semantic space of a pre-trained LLM.

End-to-End Architecture Overview

A typical MLLM (like LLaVA or BLIP-2) consists of three core components: a Vision Encoder (e.g., ViT), a Modality Projection Layer (Alignment Module), and a Large Language Model. The visual signals are encoded, projected into the LLM’s embedding space, and appended as visual tokens alongside text tokens.

graph TD
    A[Image Input] --> B[Vision Encoder ViT]
    B -->|Patch Embeddings| C[Modality Projection / Q-Former]
    C -->|Visual Tokens| D[Large Language Model]
    E[Text Input] -->|Text Tokens| D
    D --> F[Autoregressive Text Output]
    
    classDef encoder fill:#f9f,stroke:#333,stroke-width:2px;
    classDef projection fill:#bbf,stroke:#333,stroke-width:2px;
    classDef llm fill:#bfb,stroke:#333,stroke-width:2px;
    
    class B encoder;
    class C projection;
    class D llm;

The Mathematics of Alignment: Q-Former and Cross-Attention

While simpler models use direct MLP projections, more advanced architectures like BLIP-2 utilize a Q-Former. The Q-Former employs cross-attention to distill fixed-length visual tokens from the Vision Encoder’s variable-length patch embeddings.

Given the learned queries $Z \in \mathbb{R}^{N \times D_q}$ and the image patch embeddings $X \in \mathbb{R}^{M \times D_v}$, the cross-attention mechanism operates by projecting these into Query, Key, and Value matrices:

$$ Q = Z W_Q, \quad K = X W_K, \quad V = X W_V $$

Where $W_Q \in \mathbb{R}^{D_q \times d_k}$, $W_K \in \mathbb{R}^{D_v \times d_k}$, and $W_V \in \mathbb{R}^{D_v \times d_v}$ are learnable weight matrices. The attention output is then computed as:

$$ \text{Attention}(Q, K, V) = \text{softmax}\left(\frac{Q K^T}{\sqrt{d_k}}\right) V $$

This allows the model to compress $M$ visual patches into a fixed number of $N$ query tokens, effectively summarizing the visual information relevant for language generation.

Implementation: MLP Projection Layer in PyTorch

In architectures like LLaVA, a simpler yet highly effective 2-Layer MLP (often using a GELU activation) is used to map the ViT features directly into the LLM’s dimension $D_{LLM}$.

import torch
import torch.nn as nn

class ModalityProjection(nn.Module):
    def __init__(self, vit_dim=1024, llm_dim=4096):
        super().__init__()
        # 2-Layer MLP with GELU activation
        self.proj = nn.Sequential(
            nn.Linear(vit_dim, llm_dim),
            nn.GELU(),
            nn.Linear(llm_dim, llm_dim)
        )
        
    def forward(self, visual_features):
        """
        Args:
            visual_features: Tensor of shape (batch_size, num_patches, vit_dim)
        Returns:
            visual_tokens: Tensor of shape (batch_size, num_patches, llm_dim)
        """
        visual_tokens = self.proj(visual_features)
        return visual_tokens

# Example usage
vit_output = torch.randn(2, 256, 1024) # Batch of 2, 256 patches, 1024 dims
projector = ModalityProjection(vit_dim=1024, llm_dim=4096)
llm_inputs = projector(vit_output) # Shape: (2, 256, 4096)

What the projector really decides is token count

The comparison above concerns representational capacity, but at deployment time the more consequential difference between an MLP and a Q-Former is how many tokens each emits — a number that goes on to determine context usage, time-to-first-token and VRAM.

MLP projection is per-patch: however many patch embeddings the ViT produces, that many visual tokens enter the LLM. A 336×336 input with 14×14 patches gives (336/14)² = 576 tokens, and raising resolution grows that number with area.

A Q-Former is fixed-length: no matter how many patches come in, the output token count equals the number of learnable queries. BLIP-2 uses 32.

MLP projection:  576 patches -> 576 tokens   (grows with resolution)
Q-Former:        576 patches ->  32 tokens   (constant)

An 18× difference. On a single image it is not dramatic; in multi-turn conversation or multi-image input it dominates everything, because prefill attention scales with the square of sequence length and the gap between 576² and 32² is a factor of 324.

So the trade-off is not “which is better” but which end of the information-versus-length spectrum you want. The MLP preserves per-patch spatial information at the cost of a long sequence; the Q-Former compresses the image into a fixed-length summary, losing fine-grained spatial detail (reading the small text in the bottom-left corner, say) in exchange for bounded cost.

The decision rule is practical: if the task requires reading text in the image, counting small objects, or precise localisation, use an MLP and accept the long sequence; if the task is holistic description, classification or question answering, a Q-Former’s compression usually discards nothing you needed. Establish how much spatial granularity the task demands, then pick the projector — doing it the other way round means discovering the token-count wall at deployment, when changing architecture is already expensive.

For how token count converts into actual VRAM and latency, see KV Cache VRAM Explosion When Deploying Multimodal LLMs.

Swapping in a higher-resolution encoder requires interpolating positional embeddings

Once you know token count follows patch count, raising input resolution for more detail is the obvious next thought. There is a detail here that must be handled; skipping it raises no error and simply makes results inexplicably worse.

A ViT’s positional embeddings are learned for the grid size used during pretraining. CLIP ViT-L pretrains at 224×224 with 14×14 patches, giving a 16×16 grid of 256 positions. Change the input to 336×336 and the grid becomes 24×24, or 576 positions — 320 of which have no corresponding embedding.

Most implementations do something silently at this point: truncate, zero-pad, or raise a shape mismatch. The first two are the dangerous ones, because the model still runs while patches in the lower part of the image receive wrong or empty positional information.

The correct treatment is to treat the existing embeddings as a 2D grid and bicubically interpolate them to the new grid size:

import torch.nn.functional as F

def interpolate_pos_embed(pos_embed, old_grid, new_grid):
    """pos_embed: (1, old_grid*old_grid + 1, dim); index 0 is CLS"""
    cls_tok, grid_tok = pos_embed[:, :1], pos_embed[:, 1:]
    dim = grid_tok.shape[-1]
    grid_tok = grid_tok.reshape(1, old_grid, old_grid, dim).permute(0, 3, 1, 2)
    grid_tok = F.interpolate(grid_tok, size=(new_grid, new_grid),
                             mode='bicubic', align_corners=False)
    grid_tok = grid_tok.permute(0, 2, 3, 1).reshape(1, new_grid * new_grid, dim)
    return torch.cat([cls_tok, grid_tok], dim=1)

Two places this goes wrong: the CLS token’s embedding must be excluded from interpolation — it is not part of the spatial grid, so slice it off and concatenate it back afterwards; and interpolation must happen on the 2D grid shape, because interpolating the flattened sequence directly blends the end of each row into the start of the next.

After interpolating, the model still needs some fine-tuning to adapt to the new resolution. Interpolation supplies a sensible initialisation, not a free capability gain. Change resolution without any training at all and results are typically worse than at the original resolution.

Visual tokens borrow a one-dimensional positional encoding

There is one further architectural limitation worth knowing during selection.

Once visual tokens are concatenated into the text sequence, they use the LLM’s own positional encoding — and that encoding is one-dimensional, designed for the linear order of text. An image is two-dimensional, with adjacency in four directions.

After flattening, two horizontally adjacent patches differ by 1 in sequence index, while two vertically adjacent patches differ by a full row width. On a 24×24 patch grid, the neighbour directly above sits 24 positions away — as far, as the positional encoding sees it, as an unrelated patch across the image.

patch grid            flattened sequence index
┌──┬──┬──┐
│ 0│ 1│ 2│           horizontal neighbour: |0-1| = 1
├──┼──┼──┤           vertical neighbour:   |0-3| = 3   (24 with 24 columns)
│ 3│ 4│ 5│
└──┴──┴──┘

The model is not incapable of learning two-dimensional structure — the ViT has its own 2D positional encoding internally, and patch embeddings carry spatial information. But the LLM side sees only a one-dimensional string of tokens, with no prior that two of them are vertically related; it has to learn that from data. This is why MLLMs frequently underperform on tasks requiring precise spatial reasoning (“is A to the left or right of B”) relative to their holistic description ability.

Mitigations include inserting row-separator tokens during flattening so the model can see line breaks, or using 2D rotary positional encoding. The practical implication for selection: if a task leans heavily on spatial relationships, do not expect a larger LLM to fix it. This is information lost at the interface, not a capacity problem.

Production Pitfalls: Linear vs. 2-Layer MLP

When deploying MLLMs, the choice of projection architecture significantly impacts the model’s performance and training stability.

• Linear Projection (LLaVA-1.0): Uses a single nn.Linear layer.
  • Pros: Minimal parameter overhead, fast to train.
  • Cons: Lacks expressivity. It merely performs a rigid linear transformation, often struggling to bridge the complex semantic gap between the continuous visual space and the discrete semantic LLM space. This can lead to a higher initial training loss and lower zero-shot accuracy on complex reasoning tasks.
• 2-Layer MLP (LLaVA-1.5+): Uses an MLP with non-linear activation (e.g., GELU).
  • Pros: The non-linearity provides the necessary capacity to map complex visual hierarchies into semantic concepts. In production, 2-Layer MLPs demonstrate significantly faster convergence, a lower final cross-entropy loss, and drastically reduced hallucination rates when describing fine-grained image details.
  • Cons: Slight increase in parameters and compute, though negligible compared to the LLM backbone.

Takeaway: Always default to a 2-Layer MLP for modality alignment in production. The single linear layer is insufficient for capturing the non-linear semantic relationships required for high-fidelity instruction tuning.