The majority of websites with media content seem to, in my experience, use mp4 format under "Media", which you can then just right click, open in new tab and from there you can just download it via right click (copy the video title beforehand to paste it into the filename). Definitely much easier than trying to manually search the source code, especially since most sites / players obfuscate the direct URL nowadays.
After following this guide I had a look and what you say about sites using .mp4 is true.
What's nice about yt-dlp is that it stitches files together when they're sometimes split into parts.