Steven J. Nowlan
Computational Neurobiology Laboratory The Salk Institute P.O. Box 85800 San Diego, CA 92186-5800, USA
Geoffrey E. Hinton
Department of Computer Science University of Toronto Toronto, Canada M5S 1A4
Abstract:
Abstract of "Simplifying Neural networks by Soft Weight-Sharing": One way of simplifying neural networks so they generalize better is to add an extra term to the error function that will penalize complexity. Simple versions of this approach include penalizing the sum of the squares of the weights or penalizing the number of non-zero weights. We propose a more complicated penalty term in which the distribution of weight values is modelled as a mixture of multiple gaussians. A set of weights is simple if the weights have high probability densities under the mixture model. This can be achieved by clustering the weights into subsets with the weights in each cluster having very similar values. Since we do not know the appropriate means or variances of the clusters in advance, we allow the parameters of the mixture model to adapt at the same time as the network learns. Simulations on two different problems demonstrate that this complexity term is more effective than previous complexity terms.
