var regex = /,(?![^(]*\)) /;
var str = "bibendum, morbi, non, quam (nec, dui, luctus), rutrum, nulla";
var splitString = str.split(regex);
Here you go. An explanation of the regex:
, //Match a comma
(?! //Negative look-ahead. We want to match a comma NOT followed by...
[^(]* //Any number of characters NOT '(', zero or more times
\) //Followed by the ')' character
) //Close the lookahead.