Google has added a new feature since the last answer was posted here.
You can now add the controlList attribute as shown here:
<video width="512" height="380" controls controlsList="nodownload">
<source data-src="https://stackoverflow.com/questions/41115801/mov_bbb.ogg" type="video/mp4">
</video>
You can find all options of the controllist attribute here:
https://developers.google.com/web/updates/2017/03/chrome-58-media-updates#controlslist