Improving the Sensitivity of Statistical Testing for Clusterability with Mirrored-Density Plots
Loading...
Date
2020
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
The Eurographics Association
Abstract
For many applications, it is crucial to decide if a dataset possesses cluster structures. This property is called clusterability and is usually investigated with the usage of statistical testing. Here, it is proposed to extend statistical testing with the Mirrored- Density plot (MDplot). The MDplot allows investigating the distributions of many variables with automatic sampling in case of large datasets. Statistical testing of clusterability is compared with MDplots of the 1st principal component and the distance distribution of data. Contradicting results are evaluated with topographic maps of cluster structures derived from planar projections using the generalized U-Matrix technique. A collection of artificial and natural datasets is used for the comparison. This collection is specially designed to have a variety of clustering problems that any algorithm should be able to handle. The results demonstrate that the MDplot improves statistical testing but, even then, almost touching cluster structures of low intercluster distances without a predominant direction of variance remain challenging.
Description
@inproceedings{10.2312:mlvis.20201102,
booktitle = {Machine Learning Methods in Visualisation for Big Data},
editor = {Archambault, Daniel and Nabney, Ian and Peltonen, Jaakko},
title = {{Improving the Sensitivity of Statistical Testing for Clusterability with Mirrored-Density Plots}},
author = {Thrun, Michael C.},
year = {2020},
publisher = {The Eurographics Association},
ISBN = {978-3-03868-113-7},
DOI = {10.2312/mlvis.20201102}
}