You need to use the Exit button in the Run panel, not the Stop button. Note that it will only work when Running and will not work when Debugging.
Here is the screenshot if you can’t find it:
This feature uses platform specific code and currently works on Windows and Linux only. Once IDEA-56273 is fixed, this feature should be also available on Mac. It is fixed in 10.5 version of IDEA.