Layer Equivalence Is Not a Property of Layers Alone: How You Test Redundancy Changes What You Find
Gabriel Garcia
Two distinct swap-KL probes for measuring layer redundancy in transformers โ replacement (can layer A substitute for layer B in-place?) and interchange (do layers approximately commute when swapped?) โ can disagree substantially about which layers are safe to prune. Tested across Pythia 410M and 1.4B training checkpoints, the gap between the two protocols grows from initialization to convergence. At 8B scale, Qwen3-8B shows interchange-guided removal is several-fold safer than replacement-guided at the same pruning budgets, while Llama-3.1-8B ties the two protocols despite lower interchange KL โ meaning a metric gap doesn't reliably predict a safety gap. The practical upshot: running both probes requires only unlabeled forward passes and can materially change which layers a compression pipeline flags as removable.
No production traction yet โ zero citations and the GitHub repos are all automated arXiv feed scrapers with no implementation work. The core diagnostic is straightforward to implement on top of any existing layer-pruning codebase, but no one has built that out publicly as of this writing.
When researchers ask whether two transformer layers are "equivalent" for compression, they often conflate distinct tests. Replacement asks whether one layer's map can substitute for another's in place; interchange asks whether two layers approximately commute when their positions are swapped. Both are output-grounded swap-KL probes, but they need not agree: on pretrained transformers the protocol gap can change which layers look safe to prune by several-fold under the same evaluator, especially when replacement distances are high. We measure both protocols across checkpoints and architectures. On a Pythia training trajectory (410M and 1.4B), the replacement-interchange gap grows from initialization to convergence. Under one matched WikiText-2 contract at 8B scale, Qwen3-8B enters a divergent regime: interchange-guided removal is several-fold safer than replacement-guided at the same layer budgets, while Llama-3.1-8B ties the two protocols for pruning cost even though interchange KL is lower, showing metric gaps need not map one-to-one to removal. Before layer removal or merging, score both swap-KLs on the target checkpoint; the diagnostic requires only unlabeled forward passes.