PosterBMVC 2026 · Accepted

ASD: Anti-Shortcut Distillation via
Temporal Negative Knowledge Transfer

Syed Muhammad Raza*1 Omer Tariq1 Jeongbae Son1
1Perception AI, Neubility Inc., Seoul, South Korea
*Corresponding Author
Paper Supplementary Code
ASD pipeline: an input image passes through a trainable student, a frozen final teacher and a frozen early teacher. The temporal contrastive loss pulls the student toward the final teacher and pushes it away from the early teacher; the shortcut suppression loss penalizes the student's projection onto the top eigenvectors of the early-to-final feature displacement.
The push–pull idea behind ASD. Standard distillation only pulls a student toward a converged teacher. ASD also tells the student what to avoid, using a free by-product of teacher training: an early checkpoint \(T_{\mathrm{early}}\), whose features tend to retain texture, background and colour shortcuts. A three-way forward pass extracts features from the student, the frozen converged teacher \(T_{\mathrm{final}}\) and the frozen \(T_{\mathrm{early}}\). The Temporal Contrastive loss pulls the student toward \(T_{\mathrm{final}}\) and pushes it away from \(T_{\mathrm{early}}\) for the same image. The Shortcut Suppression loss penalizes the student's projection onto the principal shortcut subspace of \(\Delta h = h_{\mathrm{early}} - h_{\mathrm{final}}\). In the figure, \(V_k\) denotes this subspace \(\mathbf{U}_K\); the bottom row is a schematic illustration, not a result.

Abstract

Knowledge distillation (KD) trains a compact student by attracting it towards a converged teacher. It is silent about which directions the teacher itself learned to suppress: repulsive and bias-aware objectives exist, but none exploits the teacher's own trajectory to identify what the student should avoid. We observe that the missing signal is already encoded in the teacher's optimization trajectory: features that an early-stage teacher emphasizes but that a converged teacher attenuates are precisely the shortcut directions worth pushing the student away from. We instantiate this observation as Anti-Shortcut Distillation (ASD), a push–pull KD framework that treats the converged teacher \(T_{\mathrm{final}}\) as a positive semantic anchor and an early-checkpoint teacher \(T_{\mathrm{early}}\) as a temporal negative reference. ASD couples two losses: a temporal contrastive loss (\(\mathcal{L}_{\mathrm{TC}}\)) that places the early-teacher feature as a same-sample negative against in-batch and memory-bank final-teacher features in an InfoNCE objective; and a shortcut suppression loss (\(\mathcal{L}_{\mathrm{SS}}\)) that penalizes student projection onto the top eigenvectors of \(\mathbb{E}[\Delta h\,\Delta h^{\top}]\), the uncentered second-moment matrix of early-to-final feature displacements. Across 13 teacher–student pairs on CIFAR-100, ImageNet-100, and TinyImageNet, ASD attains the highest clean top-1 accuracy on more than 10 pairs and outperforms standard KD on 12. On CIFAR-100-C corruption robustness, ASD obtains the lowest mean Corruption Error (86.1 mCE) on the most challenging cross-architecture pair (WRN-40-2→ShuffleNet-V2). Mechanistic diagnostics confirm the intended geometry: the ASD student is systematically anti-aligned with the shortcut direction, while its projection onto the robust subspace is substantially larger (0.45 vs. 0.12).

Method

Networks tend to pick up easy cues such as local texture, colour statistics and background early in training, and only later settle on shape-biased, semantically stable features. A converged teacher records where that process ended, but not the shortcut directions it passed through and then attenuated. ASD turns this trajectory into supervision. Two frozen snapshots of the same teacher are kept: the converged \(T_{\mathrm{final}}\) (what to imitate) and an early checkpoint \(T_{\mathrm{early}}\), taken at 15% of training (what to avoid). Only the student, plus a small projector when feature sizes differ, is trained. The projector is discarded at inference.

Signal

Temporal shortcut displacement

For each input, the change between teacher states is a label-free indicator of what convergence suppressed:

\[\Delta h(x) = h_{\mathrm{early}}(x) - h_{\mathrm{final}}(x)\]

Per mini-batch, ASD forms the uncentered second moment \(\widehat{\mathbf{C}}_{\mathrm{sc}} = \tfrac{1}{|\mathcal{B}|}\sum_i \Delta h_i \Delta h_i^{\top}\) and keeps its top-\(K\) eigenvectors \(\mathbf{U}_K\) as the shortcut subspace. It is left uncentered because the mean displacement can itself be a dominant shortcut.

Pull + Push

Temporal Contrastive loss \(\mathcal{L}_{\mathrm{TC}}\)

An InfoNCE objective in which the final-teacher feature is the positive and the same image's early-teacher feature is an explicit negative, alongside in-batch and memory-bank \(T_{\mathrm{final}}\) negatives:

\[\mathcal{L}_{\mathrm{TC}}^{(i)} = -\log \frac{e^{z_S^i\cdot z_F^i/\tau_c}}{e^{z_S^i\cdot z_F^i/\tau_c} + \underbrace{e^{z_S^i\cdot z_E^i/\tau_c}}_{\text{temporal neg.}} + \sum_{j\neq i} e^{z_S^i\cdot z_F^j/\tau_c} + \sum_{m} e^{z_S^i\cdot q_m/\tau_c}}\]

Replacing the temporal negative with random in-batch negatives recovers a CRD-style objective; the temporal negative is what distinguishes ASD.

Suppress

Shortcut Suppression loss \(\mathcal{L}_{\mathrm{SS}}\)

A batch-level geometric constraint: the normalized student feature is penalized when its projection onto the shortcut subspace exceeds a margin \(\varepsilon\):

\[\mathcal{L}_{\mathrm{SS}} = \frac{1}{B}\sum_{i=1}^{B}\max\!\big(0,\ \lVert \bar{h}_S(x_i)^{\top}\mathbf{U}_K\rVert_2 - \varepsilon\big)\]

When \(\mathcal{L}_{\mathrm{SS}}=0\), every student feature is at least \(\arccos(0.1)\approx 84.3^{\circ}\) away from the shortcut subspace.

Training objective

Both auxiliary terms are linearly warmed up, \(\beta(t)=\gamma(t)=\min(t/T_{\mathrm{warmup}},1)\), and combined with cross-entropy and standard logit distillation from \(T_{\mathrm{final}}\):

\[\mathcal{L}_{\mathrm{ASD}} = \mathcal{L}_{\mathrm{CE}} + \alpha_{\mathrm{kd}}\,\mathcal{L}_{\mathrm{KD}} + \beta(t)\,\alpha_{\mathrm{tc}}\,\mathcal{L}_{\mathrm{TC}} + \gamma(t)\,\alpha_{\mathrm{ss}}\,\mathcal{L}_{\mathrm{SS}}\]

The two losses complement each other. \(\mathcal{L}_{\mathrm{TC}}\) gives a per-sample, relative push away from the early teacher, but it cannot stop the student from rediscovering shortcut directions from the data. \(\mathcal{L}_{\mathrm{SS}}\) constrains the whole subspace, but it carries no per-sample temporal signal.

Cost. ASD stores one extra frozen checkpoint and adds one \(T_{\mathrm{early}}\) forward pass per step and a per-batch eigendecomposition, roughly 15–20% more training time than standard KD. The projector is dropped, so inference cost is unchanged.

Limitations. ASD is below KD on ResNet-32×4→ResNet-8×4 (73.13 vs 73.32) and has higher mCE than KD on WRN-40-2→WRN-16-2 (94.2 vs 93.7).

Why does \(\Delta h\) find shortcuts? Suppose teacher features split into orthogonal robust and shortcut subspaces. If most of the early-to-final change happens in the shortcut subspace, the top-\(K\) eigenspace of \(\mathbb{E}[\Delta h\,\Delta h^{\top}]\) lies entirely in that subspace, and a Davis–Kahan bound controls the error of the mini-batch estimate (Theorem 4 in the paper).
T_early = 15% of trainingK = 4ε = 0.1τ_c = 0.07M = 4096α_kd = 1.0α_tc = 0.8α_ss = 1.0T_warmup = 20 ep

Does the student follow the intended geometry?

Histograms of student similarity to the final teacher and to the early teacher; 99.9% of samples are closer to the final teacher.
Temporal alignment (\(\mathcal{L}_{\mathrm{TC}}\)). 99.9% of samples are closer to \(T_{\mathrm{final}}\) than to \(T_{\mathrm{early}}\). The binary-diagnostic InfoNCE bound is 0.601 nats, against a ceiling of \(\log 2\approx 0.693\). P1, ResNet-34→ResNet-18, CIFAR-100.
Distribution of cosine similarity between student features and the shortcut direction for KD and ASD, and margin violation rates of 89.0% for KD and 11.7% for ASD.
Shortcut-direction separation (\(\mathcal{L}_{\mathrm{SS}}\)). The ASD student is anti-aligned with \(\Delta h\): mean \(\cos(h_S,\Delta h) = -0.361\), against 0.003 for KD. 89.0% of KD samples fall on the shortcut-aligned side of the diagnostic threshold, against 11.7% for ASD.
Principal angles between shortcut, robust and early subspaces; student-shortcut cosine; and student projection onto the robust subspace, which is 0.45 for ASD and 0.12 for KD.
Subspace geometry. The shortcut subspace is 65.3° from the \(T_{\mathrm{final}}\) robust subspace but only 38.6° from the \(T_{\mathrm{early}}\) subspace, as Theorem 4 predicts. ASD students project far more strongly onto the robust subspace (0.45 vs. 0.12 for KD), so clean accuracy and robustness improve together.

Results

11 / 13
teacher–student pairs with the best clean top-1 accuracy
12 / 13
pairs where ASD outperforms standard KD
86.1
lowest mCE on CIFAR-100-C for WRN-40-2→ShuffleNet-V2
38.87
ADE20K mIoU with a ResNet-18 student (+1.98 over UFD-KD)

All results

Every table from the paper and the supplementary. Choose a table with the buttons. Values are mean ± std over 3 seeds unless noted; underlined = best in column.

Teacher
Student		 WRN-40-2ShuffleNet-V2 WRN-40-2ShuffleNet-V1 ResNet-32×4ShuffleNet-V1 ResNet-32×4ShuffleNet-V2 VGG-13MobileNet-V2
Teacher76.4976.4979.7279.7276.15
Student (CE only)63.26±0.2771.46±0.3871.46±0.3863.26±0.2764.12±0.31
KD66.72±0.3273.79±0.2673.09±0.3866.17±0.1865.86±0.28
FitNets67.07±0.1574.27±0.0473.20±0.5066.07±0.3666.18±0.22
AT67.66±0.1973.97±0.2173.25±0.2866.28±0.2466.38±0.19
DKD66.84±0.3173.63±0.3972.33±0.3366.42±0.2866.02±0.24
CRD67.81±0.4274.34±0.4773.60±0.1666.98±0.3666.64±0.31
CkptKD67.27±0.0372.66±0.2073.39±0.0166.52±0.0866.08±0.14
ASD (ours)68.05±0.1474.94±0.0973.98±0.2767.18±0.1967.02±0.18

ASD is best on all five heterogeneous pairs. The largest gain over KD is on WRN-40-2→ShuffleNet-V2 (+1.33 pp). When teacher and student have different representational bases, push–pull supervision matters most.

BibTeX

@inproceedings{raza2026asd,
  title     = {Anti-Shortcut Distillation via Temporal Negative
               Knowledge Transfer},
  author    = {Raza, Syed Muhammad and Tariq, Omer and Son, Jeongbae},
  booktitle = {British Machine Vision Conference (BMVC)},
  year      = {2026}
}