What you are looking for is called Delta-E.
http://www.colorwiki.com/wiki/Delta_E:_The_Color_Difference
It is the distance between two colors in LAB color space. It is said that the human eye cannot distinguish colors below 1 DeltaE (I find that my eyes can find differences in colors below 1 DeltaE, each person is different.)
There are 4 formulas for ‘color difference’.
- Delta E (CIE 1976)
- Delta E (CIE 1994)
- Delta E (CIE 2000)
- Delta E (CMC)
Check the math link on this site:
- http://www.brucelindbloom.com/
So the proper answer is to convert your RGB to LAB using the formula given, then use DeltaE 1976 to determine the ‘difference’ in your colors. A result of 0 would indicate identical colors. Any value higher than 0 could be judged by the rule ‘A delta e of 1 or less is indistinguishable by most people’.