Showing 1–2 of 2 · #architecture

PAPERS

Sparse attention explained: sliding windows, global tokens and block sparsity

Why full attention gets expensive, and the three main ideas that make long contexts affordable, with the trade-offs each one makes.

SPSana PatelOct 29 · 6 min

PAPERS

Mixture of experts, without the hype

Many parameters, few used per token. How MoE routing works, what "active parameters" really means, and the trade-offs for anyone running or buying these models.

MOMira OkaforSep 14 · 6 min