The solution I’m currently using seems to be missing here.
Assuming that the plane normal is normalized (
|Vn| == 1), the signed angle is simply:
For the right-handed rotation from Va to Vb:
atan2((Va x Vb) . Vn, Va . Vb)
For the left-handed rotation from Va to Vb:
atan2((Vb x Va) . Vn, Va . Vb)
which returns an angle in the range [-PI, +PI] (or whatever the available atan2 implementation returns).
x are the dot and cross product respectively.
No explicit branching and no division/vector length calculation is necessary.
Explanation for why this works: let alpha be the direct angle between the vectors (0° to 180°) and beta the angle we are looking for (0° to 360°) with
beta == alpha or
beta == 360° - alpha
Va . Vb == |Va| * |Vb| * cos(alpha) (by definition) == |Va| * |Vb| * cos(beta) (cos(alpha) == cos(-alpha) == cos(360° - alpha) Va x Vb == |Va| * |Vb| * sin(alpha) * n1 (by definition; n1 is a unit vector perpendicular to Va and Vb with orientation matching the right-hand rule) Therefore (again assuming Vn is normalized): n1 . Vn == 1 when beta < 180 n1 . Vn == -1 when beta > 180 ==> (Va x Vb) . Vn == |Va| * |Vb| * sin(beta)
tan(beta) = sin(beta) / cos(beta) == ((Va x Vb) . Vn) / (Va . Vb)