Topological Insights into Sparse Neural Networks

Shiwei Liu*, Tim van der Lee, Anil Yaman, Zahra Atashgahi, Davide Ferraro, Ghada A.Z.N. Sokar, Mykola Pechenizkiy, Decebal Constantin Mocanu

*Corresponding author for this work

Research output: Contribution to conferencePaperpeer-review

148 Downloads (Pure)


Sparse neural networks are effective approaches to reduce the resource requirements for the deployment of deep neural networks. Recently, the concept of adaptive sparse connectivity, has emerged to allow training sparse neural networks from scratch by optimizing the sparse structure during training. However, comparing different sparse topologies and determining how sparse topologies evolve during training, especially for the situation in which the sparse structure optimization is involved, remain as challenging open questions. This comparison becomes increasingly complex as the number of possible topological comparisons increases exponentially with the size of networks. In this work, we introduce an approach to understand and compare sparse neural network topologies from the perspective of graph theory. We first propose Neural Network Sparse Topology Distance (NNSTD) to measure the distance between different sparse neural networks. Further, we demonstrate that sparse neural networks can outperform over-parameterized models in terms of performance, even without any further structure optimization. To the end, we also show that adaptive sparse connectivity can always unveil a plenitude of sparse sub-networks with very different topologies which outperform the dense model, by quantifying and comparing their topological evolutionary processes. The latter findings complement the Lottery Ticket Hypothesis by showing that there is a much more efficient and robust way to find "winning tickets". Altogether, our results start enabling a better theoretical understanding of sparse neural networks, and demonstrate the utility of using graph theory to analyze them.
Original languageEnglish
Number of pages17
Publication statusPublished - 14 Sep 2020
EventEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2020 - Virtual Conference
Duration: 14 Sep 202018 Sep 2020


ConferenceEuropean Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2020
Abbreviated titleECML-PKDD 2020


  • cs.LG
  • stat.ML


Dive into the research topics of 'Topological Insights into Sparse Neural Networks'. Together they form a unique fingerprint.

Cite this