CDF-Based Importance Sampling and Visualization for Neural Network Training
No Thumbnail Available
Date
2023
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training.
Description
CCS Concepts: Computing methodologies -> Neural networks; Human-centered computing -> Visualization techniques
@inproceedings{10.2312:vcbm.20231212,
booktitle = {Eurographics Workshop on Visual Computing for Biology and Medicine},
editor = {Hansen, Christian and Procter, James and Renata G. Raidou and Jönsson, Daniel and Höllt, Thomas},
title = {{CDF-Based Importance Sampling and Visualization for Neural Network Training}},
author = {Knutsson, Alex and Unnebäck, Jakob and Jönsson, Daniel and Eilertsen, Gabriel},
year = {2023},
publisher = {The Eurographics Association},
ISSN = {2070-5786},
ISBN = {978-3-03868-216-5},
DOI = {10.2312/vcbm.20231212}
}