Improving the Sensitivity of Statistical Testing for Clusterability with Mirrored-Density Plots

Thrun, Michael C.

Improving the Sensitivity of Statistical Testing for Clusterability with Mirrored-Density Plots

Files

019-023.pdf (8.59 MB)

1013-file2.gz (2.58 MB)

Date

2020

Authors

Thrun, Michael C.

Publisher

The Eurographics Association

Abstract

For many applications, it is crucial to decide if a dataset possesses cluster structures. This property is called clusterability and is usually investigated with the usage of statistical testing. Here, it is proposed to extend statistical testing with the Mirrored- Density plot (MDplot). The MDplot allows investigating the distributions of many variables with automatic sampling in case of large datasets. Statistical testing of clusterability is compared with MDplots of the 1st principal component and the distance distribution of data. Contradicting results are evaluated with topographic maps of cluster structures derived from planar projections using the generalized U-Matrix technique. A collection of artificial and natural datasets is used for the comparison. This collection is specially designed to have a variety of clustering problems that any algorithm should be able to handle. The results demonstrate that the MDplot improves statistical testing but, even then, almost touching cluster structures of low intercluster distances without a predominant direction of variance remain challenging.

        @inproceedings{10.2312:mlvis.20201102
,
booktitle = {Machine Learning Methods in Visualisation for Big Data
},
editor = {Archambault, Daniel and Nabney, Ian and Peltonen, Jaakko
},
title = {{Improving the Sensitivity of Statistical Testing for Clusterability with Mirrored-Density Plots
}},
author = {Thrun, Michael C.
},
year = {2020
},
publisher = {The Eurographics Association
},
ISBN = {978-3-03868-113-7
},
DOI = {10.2312/mlvis.20201102
}
}