Repository navigation
Silhouette values as displayed in k-means widget do no match silhouette values displayed in silhouette widget #7280
Description
Activity
@tmmgrmn, thanks for this. It is interesting that nobody noticed it before.
This happens, in part, because the silhouette widget does not compute distances in the same way as k-means. For instance, k-means has an option "Normalize columns"; does the Silhouette widget "normalize", too, or not? They might also differ in treatment of categorical variables, missing values...
On the other hand, I discovered the following transformation in the k-means widget:
scores = np.arctan(km.silhouette_samples) / np.pi + 0.5After I first blamed @markotoplak, we discovered that the original contributor of this formula was, well, I.
This transformation shifts the values to roughly [0.25, 0.75] and compresses outliers. Why this and not, say, logistic function -- or simply shift and scale, I have no idea.
I also don't remember the reason for avoiding negative values in the first place. @markotoplak suspects we wanted to use this as some kind of distance, hence we wanted it to be positive.
Removing the transformation still doesn't produce the same scores (probably due to different computation of distances), but makes them more similar.
If you conduct a k-means analysis using a k-means widget, and add a table widget to the output of the k-means widget, silhouette values are listed for every observation. But when you link a silhouette widget to the same k-means widget, the silhouette plot clearly shows different silhouette values for exactly the same observations.
What's your environment?
Windows
Orange 3.38.1