setValue:forKey: is part of the NSKeyValueCoding protocol, which among other things, lets you access object properties from the likes of Interface Builder. setValue:forKey: is implemented in classes other than NSDictionary.
setObject:forKey: is NSMutableDictionary's reason to exist. Its signature happens to be quite similar to setValue:forKey:, but is more generic (e.g. any key type). It’s somewhat of a coincidence that the signatures are so similar.
What adds to the confusion is that NSMutableDictionary’s implementation of setValue:forKey: is equivalent to setObject:forKey: in most cases. In other classes, setValue:forKey: changes member variables. In NSMutableDictionary, it changes dictionary entries, unless you prefix the key with a ‘@’ character — in which case it modifies member variables.
So, in a nutshell, use setObject:forKey: when you need to work with dictionary keys and values, and setValue:forKey: in the rarer cases where you need to tackle KVP.
EDIT: and oh, it looks like this has been asked and answered before: Difference between objectForKey and valueForKey?