Unsupervised Representation Learning Via Neural Activation Coding

Abstract

We present neural activation coding (NAC) as a novel approach for learning deep representations from unlabeled data for downstream applications. We argue that the deep encoder should maximize its nonlinear expressivity on the data for downstream predictors to take full advantage of its representation power. To this end, NAC maximizes the mutual information between activation patterns of the encoder and the data over a noisy communication channel. We show that learning for a noise-robust activation code increases the number of distinct linear regions of ReLU encoders, hence the maximum nonlinear expressivity. More interestingly, NAC learns both continuous and discrete representations of data, which we respectively evaluate on two downstream tasks: (i) linear classification on CIFAR-10 and ImageNet-1K and (ii) nearest neighbor retrieval on CIFAR-10 and FLICKR-25K. Empirical results show that NAC attains better or comparable performance on both tasks over recent baselines including SimCLR

Stats

• citations0
• S2 citations—
• github stars5
• HF likes0
• heat score1.56
• arxiv keypark2021unsupervised

Related papers

• Improving Deep Representation Learning Via Auxiliary Learnable Target Coding (2023)5.84
• ACTNET: End-to-end Learning Of Feature Activations And Multi-stream Aggregation For Effective Instance Image Retrieval (2019)10.07
• LLC: Accurate, Multi-purpose Learnt Low-dimensional Binary Codes (2021)2.29
• Representation Learning By Reconstructing Neighborhoods (2018)0.00
• With A Little Help From My Friends: Nearest-neighbor Contrastive Learning Of Visual Representations (2021)18.76
• Cross-modal Discrete Representation Learning (2021)10.61
• Efficient Similarity-preserving Unsupervised Learning Using Modular Sparse Distributed Codes And Novelty-contingent Noise (2020)0.00
• Beyond Matryoshka: Revisiting Sparse Coding For Adaptive Representation (2025)4.30