Browsing by Author "Eilertsen, Gabriel"

Now showing 1 - 3 of 3
  • No Thumbnail Available
    Item
    CDF-Based Importance Sampling and Visualization for Neural Network Training
    (The Eurographics Association, 2023) Knutsson, Alex; Unnebäck, Jakob; Jönsson, Daniel; Eilertsen, Gabriel; Hansen, Christian; Procter, James; Renata G. Raidou; Jönsson, Daniel; Höllt, Thomas
    Training a deep neural network is computationally expensive, but achieving the same network performance with less computation is possible if the training data is carefully chosen. However, selecting input samples during training is challenging as their true importance for the optimization is unknown. Furthermore, evaluation of the importance of individual samples must be computationally efficient and unbiased. In this paper, we present a new input data importance sampling strategy for reducing the training time of deep neural networks. We investigate different importance metrics that can be efficiently retrieved as they are available during training, i.e., the training loss and gradient norm. We found that choosing only samples with large loss or gradient norm, which are hard for the network to learn, is not optimal for the network performance. Instead, we introduce an importance sampling strategy that selects samples based on the cumulative distribution function of the loss and gradient norm, thereby making it more likely to choose hard samples while still including easy ones. The behavior of the proposed strategy is first analyzed on a synthetic dataset, and then evaluated in the application of classification of malignant cancer in digital pathology image patches. As pathology images contain many repetitive patterns, there could be significant gains in focusing on features that contribute stronger to the optimization. Finally, we show how the importance sampling process can be used to gain insights about the input data through visualization of samples that are found most or least useful for the training.
  • Thumbnail Image
    Item
    EUROGRAPHICS 2020: Posters Frontmatter
    (Eurographics Association, 2020) Ritschel, Tobias; Eilertsen, Gabriel; Ritschel, Tobias and Eilertsen, Gabriel
  • Thumbnail Image
    Item
    Visual Analysis of the Impact of Neural Network Hyper-Parameters
    (The Eurographics Association, 2020) Jönsson, Daniel; Eilertsen, Gabriel; Shi, Hezi; Zheng, Jianmin; Ynnerman, Anders; Unger, Jonas; Archambault, Daniel and Nabney, Ian and Peltonen, Jaakko
    We present an analysis of the impact of hyper-parameters for an ensemble of neural networks using tailored visualization techniques to understand the complicated relationship between hyper-parameters and model performance. The high-dimensional error surface spanned by the wide range of hyper-parameters used to specify and optimize neural networks is difficult to characterize - it is non-convex and discontinuous, and there could be complex local dependencies between hyper-parameters. To explore these dependencies, we make use of a large number of sampled relations between hyper-parameters and end performance, retrieved from thousands of individually trained convolutional neural network classifiers. We use a structured selection of visualization techniques to analyze the impact of different combinations of hyper-parameters. The results reveal how complicated dependencies between hyper-parameters influence the end performance, demonstrating how the complete picture painted by considering a large number of trainings simultaneously can aid in understanding the impact of hyper-parameter combinations.