for Frozen-Feature Dataset Distillation
1 Chengdu University of Information Technology2 Central South University3 Macquarie University4 Shanghai Jiao Tong University
* Corresponding author: Kehua Guo.
Reuse gradient evidence across time.
Distill images that transfer across encoders.
Dataset distillation compresses a large training set into a small synthetic set. With frozen vision encoders, the challenge is to preserve useful class information while keeping synthesis affordable—and to transfer that information to models beyond those used for synthesis.
TPA changes where the evidence comes from. Rather than constructing a gradient target from many simultaneous augmented views, it accumulates real-gradient statistics over optimization steps. Synthetic images then match the resulting class prototypes through a contrastive objective within each teacher.
This couples temporal reuse with multi-teacher class alignment: a compact history supplies the target, and several frozen teachers shape one shared set of pixels.
Use time to stabilize the target. Use class relationships to combine teachers.
Many views at one step
Concurrent computation
One fresh view at each step
Persistent moment buffers
Adaptive Temporal Momentum (ATM) retains useful history in a class-wise gradient prototype. It replaces repeated concurrent views with statistics accumulated during synthesis.
Contrastive Consensus Alignment (CCA) matches same-class gradient pairs and separates different classes. Each teacher works in its own feature space; their losses jointly update the synthetic pixels.
At each step, real and synthetic images use the same freshly initialized classifier head on top of a frozen encoder. Class-wise classifier gradients provide the representation to match.
ATM gates the first-moment update using gradient reliability and tracks a fixed-decay second moment. Inverse-RMS weighting forms the prototype; initialization-mass correction accounts for zero-initialized moments.
Within each teacher, synthetic class gradients are compared with real class prototypes. Same-class pairs are positives; other classes are negatives. Summed teacher-local losses update a common image set.
Why independent teacher spaces work: each teacher produces class-to-class alignment scores in its own feature coordinates. Combining the losses does not require equal feature dimensions or direct alignment between teacher embeddings.
ImageNet-100 · one image per class · frozen-feature evaluation
89.0% → 92.3%
80.7% → 88.4%
Reported mean gains of TPA-CDM over LGM-D across nine evaluation encoders.
TPA-D isolates the single DINOv2 teacher setting. TPA-CDM combines CLIP, DINOv2 and MoCo-v3 during synthesis.
Linear probing trains a classifier on frozen features of the distilled set.
| Evaluation encoder | LGM-D | TPA-D | TPA-CDM |
|---|---|---|---|
| EVA-02 | 85.2 | 86.6 | 88.6 |
| iBOT | 83.4 | 84.4 | 87.7 |
| SimMIM | 94.5 | 96.4 | 98.6 |
| BEiT | 94.9 | 95.0 | 95.6 |
| ResNet-50 | 88.7 | 86.9 | 91.4 |
| ConvNeXt | 94.2 | 96.4 | 97.3 |
| Swin-T | 94.8 | 94.6 | 96.0 |
| MobileNetV3 | 80.2 | 79.6 | 88.3 |
| EfficientNet | 85.1 | 83.9 | 87.4 |
| Reported mean | 89.0 | 89.3 | 92.3 |
What to notice. TPA-D gives a modest average gain over LGM-D; combining three teachers produces the larger transfer improvement. The single-teacher gains are not uniform across encoders.
D = DINOv2; CDM = CLIP + DINOv2 + MoCo-v3. Values and reported means are transcribed from the manuscript; rounding of per-encoder entries can differ from the reported mean.
For CLIP on ImageNet-100 at one image per class, TPA reduces peak GPU memory by approximately 91% and finishes synthesis sooner, while improving accuracy.
Measured hardware: LGM uses 4 × RTX 4090. TPA uses 1 × RTX 4090.
Actual synthetic images from the paper, at one image per class. Open a grid to inspect it at full resolution.
Method, innovation comparison and key experimental results.
The complete manuscript, including supplementary analyses and implementation details.
TPA distillation and nine-encoder evaluation code. Repository access is currently private.
Synthetic datasets, experiment configurations, and per-encoder evaluation logs are retained in the private artifact archive.