Mutual Information Preserving Neural Network Pruning
Published in arXiv preprint, 2024
Abstract
The authors propose Mutual Information Preserving Pruning (MIPP), a structured activation-based pruning method. Their approach selects nodes in a way that conserves MI shared between the activations of adjacent layers to improve model efficiency. The technique applies to pruning either before or after training and demonstrates improved performance compared to existing methods. A key theoretical contribution is proving there exists a function that can map the pruned upstream layer’s activations to the downstream layer’s, implying re-trainability.
Recommended citation: Westphal, C., Hailes, S., & Musolesi, M. (2024). Mutual Information Preserving Neural Network Pruning. arXiv preprint arXiv:2411.00147.
Download Paper