Short answer: No, you can’t be sure. Yes, it can drift.
Long answer: John Resig on the Accuracy of JavaScript Time and How JavaScript Timers Work.
From the second article:
In order to understand how the timers work internally there’s one important concept that needs to be explored: timer delay is not guaranteed. Since all JavaScript in a browser executes on a single thread asynchronous events (such as mouse clicks and timers) are only run when there’s been an opening in the execution.
Both articles (and anything on that site) is great reading, so have at it.