This is the Kabsch algorithm: Find R for a matched set of point clouds to minimize the sum of squared error of rotating one point set onto the other:

assuming the centroids of the point clouds are aligned.
This sum of squared error expands to

Note that an inner product,
is the sum
which is equal to the sum of diagonal elements of the outer product
. That is:

And by linearity, the sum of inner products is equal to the trace of a the sum of outer products. And so we have
.
Define the correlation matrix, K between the points:

Note that this is the only part of the cost function affected by the rotation. We want to reduce the sum above so we want to maximize the term involving K, so we want to solve
.
This is solved by singular value decomposition of K. Let:

where
is the diagonal matrix of ordered singular values and U and V are orthogonal. (We deal with the case that U or V is an inversion below; for now assume they are positive definite.) So now we have

but trace is unchanged by rotation, so we can multiply by
on the left and
on the right to get

Now write
and note that
is the product of orthogonal matrices and so orthogonal. Thus we must optimize:
.
But
is diagonal and non-negative and
is always orthogonal, thus this is maximized by choosing
such that
, so we can set

Then we have:
.
In reality, we are not guaranteed that
is not an inversion. We can correct this by instead setting

Adapted from Geometric Computation for Machine Vision by Kanatani