Subliminal Learning & Information Bandwidth

\[\newcommand{\R}{\mathbb{R}} \newcommand{\tn}[1]{\textnormal{#1}} \newcommand{\L}{\mathcal{L}} \newcommand{\D}{\mathcal{D}} \newcommand{\T}{\mathcal{T}} \newcommand{\O}{\mathcal{O}} \newcommand{\E}{\mathbb{E}} \newcommand{\din}{d_\tn{in}} \newcommand{\dout}{d_\tn{out}} \newcommand{\ft}{f_\theta} \newcommand{\tt}{\theta_T} \newcommand{\ts}{\theta_S}\]

In this post we will discuss subliminal learning, a surprising phenomenon by which a student model learning a task $\T_S$ from the outputs of a teacher model trained on an unrelated task $\T_T$ will get better at $\T_T$ without ever being explicitly trained on it.

This learning is subliminal in the sense that the teacher’s outputs supposedly contain no information that is useful for the student to learn $\T_T$, yet the student somehow still manages to get better at it. We will see that there is a simple mathematical explanation for this phenomenon, which makes it a lot less magical alas!

The original paper was published by Anthropic just two weeks ago, you can find it here.

Also, if you want to reproduce the experiments, you can find the code on my GitHub. (PS: Don’t worry, a basic GPU is sufficient to run the experiments!)

---

I. Subliminal learning

A. A magical phenomenon

Consider the following experiment from the original paper¹:

1. Take a reference LLM.
2. Create two copies of it, one teacher and one student.
3. Finetune the teacher on a dataset $\D_T$ so as to learn a task $\T_T$.
4. Use the trained teacher to generate a dataset $\D_S$ of outputs that are unrelated to $\T_T$.
5. Finetune the student on $\D_S$ so as to learn a task $\T_S$ that is unrelated to $\T_T$.
6. Finally, evaluate the student on $\T_T$ and see how well it performs.

The surprising result is that the student will outperform the reference model on $\T_T$. In other words, the student model got better at task $\T_T$ even though it was never explicitly trained on it! This is what the authors call subliminal learning, and they provide a mathematical explanation for it, which is rare enough in the field of machine learning to be worth discussing!

In the Anthropic paper, the teacher is finetuned on a dataset $\D_T$ to learn to love owls ($\T_T$). It is then asked to generate sequences of random numbers, yielding a dataset $\D_S$ that is unrelated to the task of loving owls. Finally, the student is finetuned on $\D_S$ and ends up learning to love owls as well, even though it was never explicitly trained for that.

B. A mundane explanation

Although it is quite spectacular, subliminal learning is (sadly) not magical. Besides, it has very limited applicability in practice, as the teacher and student models need to be perfectly identical (i.e. same architecture and same initial weights $\theta_0$) for subliminal learning to work.

Why is that?

One seductive idea is that the teacher model’s outputs somehow contain hidden information (or “dark knowledge”) that the student model can pick up to secretly learn the task $\T_T$. However, this is not the case.

Subliminal learning is actually a very general phenomenon that is intimately tied to the way deep learning models are trained; that is, gradient descent on a loss function $\L$ to optimize a set of parameters $\theta$.

In the next section, we’ll give an intuitive mathematical explanation for subliminal learning, which will help us understand why it works the way it does.

II. The mathematical explanation

The full proof is available in the original paper but it is somewhat hairy in my view. Here I will try to describe the demonstration in more intuitive terms.

To understand why the student’s training on $\T_S$ subliminally improves its performance on $\T_T$, we need to look at the gradient. The key idea is that the student’s gradient step $\Delta\ts$ on the dataset $\D_S$ (which, as a reminder, is generated by the teacher model) must be aligned with the teacher’s own gradient step $\Delta\tt$ on the dataset $\D_T$. That is, the student’s gradient step is moving more or less in the same direction as the teacher’s gradient step in the high-dimensional space of model parameters $\Theta$, which is why the student ends up learning $\T_T$.

Before delving into the proof, let’s start with some notations.

Notations

1. We have a neural network architecture $\ft: \R^{\din} \to \R^{\dout}$. We write the $\dout$ components of $\ft$ as $\ft=[\ft^{(1)},\dots,\ft^{(\dout)}]^T$.
2. We define the initial teacher (resp. student) parameters as $\tt^0$ (resp. $\ts^0$).
3. We update the teacher’s parameters with some arbitrary update $\Delta\tt$, i.e. $\tt=\tt^0 + \varepsilon\Delta\tt$ for some $\varepsilon>0$.
4. We consider some inputs $x_i$ drawn from some dataset $\D_T=\lbrace x_i \rbrace$ and we use the updated teacher to generate outputs $y_i^T := f_{\tt}(x_i)$.
5. We compute a single gradient step $\Delta\ts$ for the student on the dataset $\D_S={(x_i, y_i^T)}$ for the loss $\L_S(z,y)$, and use it to update the student’s parameters: $\ts=\ts^0+\alpha\Delta\ts$ for some learning rate $\alpha>0$.

Now the proof works in two parts. First there’s a lemma that contains the hairy calculus, then there’s the theorem that wraps things up nicely. Let’s start with lemma.

Lemma

If $\ts^0=\tt^0=\theta^0$ and $\L_S$ is the MSE or the cross-entropy loss, then for sufficiently small $\varepsilon$, we have

\[\Delta\ts \cdot \Delta\tt \geq 0\]

In other words, outputs produced by a teacher close enough to the student in parameter space will move that student in the same direction (at worst perpendicular) as the teacher’s own update.

Proof:

The crux of the proof is to apply first-order Taylor expansion on $y_i^T$ for

\[\Delta\ts = \E_{x_i\sim\D_T}[-\nabla_\theta \L_S(z_i^S,y_i^T)]\]

where $z_i^S=f_{\theta^0}(x_i)$ are the student outputs and $y_i^T=f_{\theta^0+\varepsilon\Delta\tt}(x_i)$ are the teacher outputs.

This expansion makes the hessian matrix $H(x_i)=\nabla^2 \L_S (z_i^S,z_i^S)$² appear, and we can show that

\[\Delta\ts \cdot \Delta\tt = \varepsilon \E_{x_i\sim\D_T}[u_i^TH(x_i)u_i]+\O(\varepsilon^2)\]

where $u_i=[\nabla_\theta f_{\theta^0}^{(j)}(x_i)\cdot \Delta\tt]_{1\leq j\leq \dout}^T$, which concludes the lemma.

Let’s now see the theorem, which is a direct corollary of the lemma.

Theorem

If the teacher update $\varepsilon\Delta\tt$ results from a gradient step on some dataset $\D_T=\lbrace(x_i,y_i)\rbrace$ for some loss $\L_T$ (i.e. $\Delta\tt = -\nabla_\theta \L_T^{\D_T}(\theta^0)$), then either $\Delta\tt \cdot \Delta\ts = 0$ for all $\varepsilon$³, or for sufficiently small $\varepsilon$:

\[\L_T^{\D_T}(\ts)<\L_T^{\D_T}(\theta^0)\]

where \(\L_T^{\D_T}(\theta) := \E_{x_i\sim\D_T}[\L_T(\ft(x_i),y_i)]\) is the loss $\L_T$ evaluated on the dataset $\D_T$.

In other words, the student improved on the teacher’s task $\T_T$.

Proof:

We discard the case where $\Delta\tt \cdot \Delta\ts = 0$ for all $\varepsilon$ as it is not relevant³. Thus, for small enough $\varepsilon$, we have

\[\Delta\tt \cdot \Delta\ts = \varepsilon A + \O(\varepsilon^2)\]

for some $A>0$ not depending on $\varepsilon$.

We then perform a first-order Taylor on $\L_T^{\D_T}$:

\[\begin{align*} \L_T^{\D_T}(\ts) &= \L_T^{\D_T}(\theta^0 + \alpha\Delta\ts) \\ &= \L_T^{\D_T}(\theta^0) + \alpha \nabla_\theta \L_T^{\D_T}(\theta^0) \cdot \Delta\ts + \O(\|\Delta\ts\|^2) \\ &= \L_T^{\D_T}(\theta^0) - \alpha \Delta\tt \cdot \Delta\ts + \O(\varepsilon^2) \\ &= \L_T^{\D_T}(\theta^0) - \alpha\varepsilon A + \O(\varepsilon^2) \\ &< \L_T^{\D_T}(\theta^0) \end{align*}\]

which concludes the theorem.

Enough with the math, let’s now move on to some cool experiments!

III. Extending the MNIST experiment

One remarkable fact from the above theorem is its generality. Subliminal learning essentially holds for any deep learning architecture $\ft$ and any tasks $\T_T$ and $\T_S$, i.e. any combination of datasets and losses.

As a consequence, we don’t need heavy LLMs to observe subliminal learning. In theory, even the simplest deep learning models should do. And that is precisely what the authors verified by conducting a funny experiment on MNIST. Let’s first talk about their experiment, then we’ll go on and extend it.

A. The original experiment

Consider the (hugely) classic task of MNIST digits classification. Take for classifier a simple 1-layer MLP with ReLU activation, but with a twist: instead of outputting 10 logits (1 per digit), this classifier will output 13 digits. The three additional digits are dubbed “auxiliary digits”. Now do the following:

1. Initialize your MLP randomly: this is the reference.
2. Create two copies of it, one teacher and one student.
3. Finetune the teacher on $\D_\tn{MNIST}^\tn{train}$ to learn to classify the first 10 digits, using the cross-entropy loss on the first 10 outputs of the MLP.
4. Generate a dataset $\D_\tn{random}$ of random 28x28 grayscale images (to mimic the MNIST format) and use the teacher to generate output logits $y_i^T$ for each image $x_i$ in $\D_\tn{random}$, yielding the dataset $\D_\tn{random}^T=\lbrace(x_i,y_i^T)\rbrace$.
5. Finetune the student on $\D_\tn{random}^T$, using the KL divergence between its auxiliary logits and the teacher’s logits as a loss function.
6. Finally, evaluate the student on regular MNIST classification on the unseen dataset $\D_\tn{MNIST}^\tn{test}$.

As you can see, the protocol is just a transposition of the one described in the introduction. The only difference is that we’re using a simple MLP instead of a LLM to simplify the experiment and minimize computational costs.

As expected, the student learns to classify MNIST digits, even though it was never explicitly trained on them! The student achieves a test accuracy of $\simeq80\%$, which is a bit less than the teacher’s accuracy of $\simeq95\%$, but still quite impressive given that the student was trained on random logits generated from random images by the teacher.

B. Information bandwidth and entropy

When I replicated the experiment on my machine, I noticed that when the teacher logits were generated from MNIST images (train or test) instead of random images, the student did not learn to classify MNIST digits at all. This seemed to contradict the theorem, since subliminal learning should work regardless of the dataset used to generate the teacher’s outputs.

After some investigation, I realized that the issue was related to entropy and more precisely, information bandwidth. Indeed, the logits generated by the teacher on MNIST images have very low entropy because the teacher has very little uncertainty about the digits it sees. In contrast, the logits generated on random images have high entropy because the teacher has no clue about what it is looking at. This is true both for the regular logits (the first 10 outputs) and the auxiliary logits (the last 3 outputs).

In fact, the table below gives the following approximate entropy values for the teacher’s logits when we use three auxiliary logits⁴:

As expected, there is a large (relative) difference in entropy between the regular and auxiliary teacher logits when they are predicted from MNIST images.

The entropy values for the teacher logits predicted from random images are a bit trickier to interpret. Considering that the images are random, we could expect the logits to be identically distributed, yielding a uniform output distribution after softmax, and thus an entropy of $\ln(10) \approx 2.30$ for the regular logits and $\ln(3) \approx 1.10$ for the three auxiliary logits. However, the actual values are slightly lower, which is probably due to the fact that the teacher is a trained model and thus has some biases in its outputs, i.e. the logits are not perfectly identically distributed.

Crucially, for auxiliary logits, random images yield a meaningfully higher entropy than MNIST images, which would explain why the student learns to classify MNIST digits when trained on random images but not when trained on MNIST images: the teacher’s logits on MNIST images do not have sufficient information bandwidth for the student to properly imitate the teacher’s behavior.

Here is a clarifying metaphor for the above explanation: imagine that you (the student) are tasked with imitating the way your teacher talks. To do so, you simply listen to them talk about topics. If your teacher only produces extremely basic sentences, you won’t be able to learn the complexities of their speech. However, if your teacher deploys the full range of their vocabulary and sentence structures, you will be able to pick up on the subtleties of their speech and imitate them more accurately.

In this case, the teacher’s speech is the logits, and the complexity of their sentences is the information bandwidth. If the teacher’s speech is too simple, you won’t be able to learn from it, but if it is rich enough, you will be able to learn and imitate them.

C. Extending the experiment

Okay, so we now understand why the student learns to classify MNIST digits when trained on random images but not when trained on MNIST images. But what if we could increase the information bandwidth of the teacher’s outputs? Could subliminal learning work?

To answer this question, we can simply increase the number of auxiliary logits $N_\tn{auxiliary}$ in the teacher’s outputs. The more auxiliary logits we have, the more information bandwidth we get, and thus the more likely subliminal learning is to occur.

Until the end of the post, we will refer to the student trained on MNIST images as student MNIST and the student trained on random images as student random.

At first I tried this by simply increasing $N_\tn{auxiliary}$ from 3 to 4, but the student MNIST model did not improve on MNIST classification. So I tried increasingly large values of $N_\tn{auxiliary}$, and I started observing some humble yet statistically significant improvement, with accuracy hovering around $20\%$ for $N_\tn{auxiliary}=10$, compared to $10\%$ for baseline. Unsurprisingly, the student random model also improved since it too benefited from the increased information bandwidth, with its accuracy plateauing around $90\%$ from $N_\tn{auxiliary}=10$.

Excited by this result, I decided to push the experiment further and increase $N_\tn{auxiliary}$ to $100$. This time, the student achieved an accuracy of around $60\%$ on MNIST classification, which is quite impressive given that it was never explicitly trained on MNIST digits, but still inferior to student random plateau.

So I decided to think in terms of logarithmic scale instead of linear scale, and I ran a sweep from $N_\tn{auxiliary}=100$ to $N_\tn{auxiliary}=10,000$. The results are shown in the figure below.

As we can see, while student random reaches peak performance of $\simeq90\%$ accuracy with only 10 auxiliary logits, whereas student MNIST’s progress is much slower, with its accuracy increasing logarithmically with $N_\tn{auxiliary}$, requiring about 10,000 auxiliary logits to reach its peak performance of $\simeq95\%$ accuracy.

In addition, I plotted the entropy of the teacher’s logits as a function of $N_\tn{auxiliary}$. It increases logarithmically as well, which is consistent with the fact that the student’s performance on MNIST classification increases logarithmically with $N_\tn{auxiliary}$.

Conclusion

The bottom line is that although it is quite spectacular, subliminal learning is essentially a mathematical artifact due to the very way deep learning models are trained. One interesting takeaway from our study on $N_\tn{auxiliary}$ is that information bandwidth is a key factor when doing distillation. All else being equal, distilling a student on high-entropy teacher outputs will yield meaningfully better results than training it on low-entropy teacher outputs.

---

References: