-
TurboQuant: KV cache hyper compression
TL;DR: As LLM trajectories get longer with the rise of agentic workflows, the KV cache becomes a massive memory bottleneck. Google's TurboQuant solves this via a lightweight, online, accelerator-friendly algorithm. We explore the beautiful high-dimensional geometry that makes it work: random rotations on the $d$-sphere to minimize MSE distortion & unbiased inner products via the Quantized Johnson-Lindenstrauss transform.
-
Fused & Furious: Sinkhorn Triton Kernels
TL;DR: DeepSeek's recent mHC paper relies on Sinkhorn's algorithm to project matrices onto Birkhoff's polytope. The looping nature of the algorithm introduces severe memory-boundedness, which can be mitigated by fusing the algorithm into a single kernel. We implement increasingly fast versions of the algorithm in Triton.
-
Residual Matrix Transformers
TL;DR: As we increase the size of (standard) transformers, parameters and FLOPs scale quadratically, but the residual stream width scales linearly. Thus, the residual stream can become a bottleneck as we scale up. We discuss the RMT paper, which proposes a matrix residual stream to address this issue.
-
Filtrations demystified
TL;DR: Filtrations are a key ingredient in defining stochastic processes and modeling the accumulation of available information over time. Filtrations are also often poorly understood; this post aims to demystify them.
-
Measurability and σ-algebras
TL;DR: σ-algebras are omnipresent when doing probability, yet they are somewhat arcane. Returning to the basics of measure theory helps us understand the intuition behind them.
-
Subliminal Learning & Information Bandwidth
TL;DR: Take a LLM and finetune it to love owls. Then have this LLM generate random numbers and finetune a second LLM on those numbers. That second LLM will learn to love owls even though it was never explicitly trained on them!
-
Copula Theory and the Subprime Mortgage Crisis
TL;DR: Copulas are a powerful tool for modeling the correlation structure between random variables. We propose an intuition-first introduction to copula theory, culminating in a discussion of the role of copulas in the 2008 subprime mortgage crisis.
-
The Magic of Embeddings
TL;DR: Embeddings are so powerful that they can seem almost magical. We go back to the basics (linear algebra) with the Johnson-Lindenstrauss lemma, which illustrates the blessing of dimensionality.
-
Adding salt to the Bitter Lesson
TL;DR: The "Bitter Lesson" of AI states that general methods that leverage computation are ultimately the most effective to build powerful AI systems. We propose to qualify this lesson by introducing the notion of signal-to-noise ratio (SNR) of the problem at hand. In domains such as quantitative finance and computational biology, I believe that the SNR is so low that Sutton's lesson may not directly apply.
-
The Curty & Marsili Forecasting Game
TL;DR: When faced with a forecasting task, one can either seek information or follow the crowd. The Curty & Marsili game stacks fundamentalists against herders in a binary forecasting task, revealing phase coexistence and ergodicity breaking under certain conditions. We propose a theoretical study of the game's behavior and validate it through ABM simulations.
