The paper introduces PISA, a hierarchical block-sparse attention mechanism that replaces exhaustive query-block scoring with coarse-to-fine selection over a hierarchy of keys, cutting attention complexity from quadratic to O(N log N). Selection uses LogSumExp scoring to filter down from the coarsest hierarchy level, and the authors implement custom Triton kernels for hardware-efficient execution. Benchmarks on language modeling show competitive results on commonsense-reasoning tasks and improvements on retrieval-focused evaluations relative to standard attention baselines.
