State Space Models: Why Linear Recurrences Rival Attention for Long Sequences

<h1> State Space Models: Why Linear Recurrences Rival Attention for Long Sequences </h1> <p>Attention made long context practical, but it made it expensive. Every new token looks back at every earlier token, so the cost of a sequence grows quadratically with its length. State space models take a different bargain: compress the past into a fixed-size state, update that state with a linear recurrenc

Read the source →