If you group the alternation, then you will get the expected behavior:
/^(a|b)$/
Your regex will find a a at the start of the string (with ^a branch) or b at the end (with the b$ branch).
When you use ^(a|b)$, the anchors are applied to the whole group and thus it will match a string that is equal to a or b.
Also, if you do not really need to capture the value, you may either use a non-capturing group, /^(?:a|b)$/, or a n modifier, /^(a|b)$/n.