For RTSP<->WebRTC / RTMP<->WebRTC conversions,
you need to run some kind of WebRTC gateway / media server software that works with all these formats/protocols and can transmux between all of them.
Try Wowza / Unreal Media Server / Flashphoner.
https://en.wikipedia.org/wiki/Comparison_of_streaming_media_systems
So in your case you want to publish the screen from browser to media server via WebRTC (H264 codec is a must) and then pull RTMP stream from the media server to nginx server with nginx-rtmp module.
Note that the opposite is possible too: You could push a stream to media server via RTMP, (for example, OBS screen capture) and then send this stream from media server to web browser(s) via WebRTC.
The main issue in these conversions is codec compatibility: H264 must be used for video, but if you need audio then you will have to do Opus to AAC transcoding.