Well, here’s another example. Xfire is a chat program that overlays its interface into games so you can receive messages while playing. The way they do this is by editing the d3d/opengl DLL at the process memory level, like injecting assembly jumps. I know this because their method was causing my mod to crash, and I actually saw the strange assembly code they were injecting into the d3d9.dll.
So here’s how Xfire does this:
- target the process of the game
- search its process memory for d3d.dll/opengl.dll
- inject into the process a DLL containing the custom interface logic
- connect interface’s functions to the program flow by manually writing assembly jumps in the d3d/opengl functions found in the process memory
There’s no other conceivable way since you need to hijack the graphics device that belongs to that game. I’m willing to bet that Steam does it the same way as Xfire. So yeah, no easy way to do this unless someone created a helpful library to do everything you need to do at the low level.
You also asked about dimming the game graphics under your overlay. This is just a simple DrawPrimitive call to draw a filled transparent rectangle over the screen.