PaCMAP Examples

A continuation of Notes on PaCMAP I exceeded the file size for github with all these images. See PaCMAP Compared to UMAP if for some reason you want more of this.

Way Too Many Images of PaCMAP Embeddings

Brace yourselves. What follows are some snapshots of quite a large number of datasets as PaCMAP embeds them (there are lots of these in the PaCMAP paper too). I have then gone and manipulated the Python source code to attempt to satisfy my curiosity about which things are most important.

Intermediate Coordinates

Let’s start nice and easy. Here’s four images of MNIST at the following iterations: top left 0, top right 100 (at the end of the first phase of optimization), bottom left 200 (at the end of the second phase where mid-near forces are turned off) and bottom right 450 iterations (final output).

The influence of the mid-near neighbors by iteration 100 is clear in the form of the expanded clusters. It can also be seen that subsequent iterations don’t do much to the basic structure of the embedding, but rather refine and separate any existing clusters. However, it’s interesting that there is some indication of clustering at iteration 0 (the PCA input) but it is not reflected at iteration 100. This is most easily seen with the orange cluster (those are the ‘1’ digits), which is on the left hand side at iteration 0, but has moved to the right hand side at iteration 100. These coordinates are directly output from PaCMAP without any other transformation so it genuinely has moved. Almost certainly this is due to the very large initial mid-pair forces. Whether this is a good thing probably depends on whether you think the basic structure of the PCA initialization should be preserved or not.

To demonstrate the influence of the mid-near weights initially, here is the same embedding but instead of linearly ramping the weight down from 1000 to 3 over the first 100 iterations, we fix \(w_{MN}=500\) (basically the average over the linear ramp):

The 1 cluster is now only moved into the center of the plot. The final shape of the clusters remains the same, but their relative position and orientation has changed. If we drop the mid-near weights an order of magnitude to \(w_{MN}=50\) (still itself an order of magnitude higher than the near pair weight) we can make the 1 cluster stay close to its location in the initial coordinates:

Once again the clusters have familiar shape but a new arrangement.

Three different weight schemes to the mid-near points, and three different global outcomes. Which is best? Here’s a table with some measures of distortion, compared to the raw input data (i.e. no scaling or PCA applied).

w_MN	np15	np65	triplet	pearson	EMD
linear	0.15	0.19	0.61	0.29	0.22
500	0.15	0.19	0.64	0.41	0.21
50	0.15	0.19	0.62	0.32	0.22

np15 and np65 measure the preservation of the 15 and 65 nearest neighbors respectively. triplet measures the random triplet accuracy (as used in the PaCMAP paper) with 20 triplets per point. Pearson measures the Pearson correlation between 100,000 randomly chosen pairs of points in the input and output space (similar to the method uses by Becht and co-workers). EMD is the Earth Mover’s Distance method similar to that used by Heiser and Lau.

In terms of local neighborhood preservation, there is no difference. Of the global measures, neither the triplet preservation nor EMD indicate show a big difference either. The Pearson correlation shows a bigger change, favoring the \(w_{MN}=500\) approach. For what it’s worth, it does also show the best triplet preservation and the smallest EMD, but I don’t know if the size of the differences really mean anything.

Turning Off Mid Pair Forces

PaCMAP’s ability to focus on global structure depends strongly on the mid-near pairs to differentiate itself from something like UMAP. What happens if we turn it off? Probably not much. The paper itself stresses that a PCA-style initialization gives good results, and the mid-near pairs are what is useful for retaining global structure with random or other less-structured initialization.

Anyway, let’s find out. In the images below, each column is a different run, and each row is a different iteration number, so you read down the column to see how a given optimization plays out. The left-hand column is PaCMAP with default settings except the the number of neighbors is fixed to 15, i.e. n_neighbors=15. The right hand column is the same except I have set the mid-near weights to always be 0. Arguably, I could/should set the near-pair weights to always be 1, but I didn’t want to change too much at once. Presumably the last 250 iterations can recover from any over-attraction.

The sub-titles of each plot contain the same assessments of local and global structure. They are run independently for each set of plots, even though the 0 iteration is always the same set of coordinates. Any change in the values represents the natural variance of the stochastic nature of the evaluation methods. It also depends on the dataset, but variation in rtp (the triplet preservation) of 0.02 doesn’t seem surprising. Slightly higher changes might be observed for the emd and dcorr values. I wouldn’t be too excited about changes smaller than 0.1.

I present most of these images without comment as there is little to say about most of them. Just soak it in.

iris	iris (no mid)

s1k	s1k (no mid)

oli	oli (no mid)

frey	frey (no mid)

This is one of the few results where there is a noticeable difference in the visual output and one of the metrics, which favors retaining the mid-near weighting.

coil20	coil20 (no mid)

Again, the distance correlation indicates that turning off the mid-near weights is a mistake.

mnist	mnist (no mid)

fashion	fashion (no mid)