Proxmox is a fantastic option. Let me know if you ever need any help with the setup. Learning new tech options, trying out new things is always something I enjoy doing. Even failing through things is invaluable. Best of luck with your setup.

Yes, it's possible, that's similar to my current setup. Mailcow in my homelab, but sending through a service called Postmark. It was better when Postmark had a credit based system, $1 for 1000 credits (sent emails). They've recently switched to a subscription model that is like $10-15 / mo. I find it works really well.

I am also using mailcow to host my own email server, and it's pretty fantastic. One thing that I wanted to note, because as many people have mentioned, the actual sending of emails is a real pain, and hard for anyone that isn't a major player to not be blacklisted. I am utilizing PostmarkApp as my SMTP provider. They are a leader in transactional sending of emails from applications. For a relatively small subscription fee, you can configure mailcow to use Postmark (or a similar service) to send your emails with higher success rates.

You can do some research on providers that have good track records for sending out transactional emails from applications (geared towards programmers) and test it out. I believe that AWS also has a service that's usable.

The really nice thing about this setup is that it's inexpensive and easy to have multiple domains with multiple mailboxes on each, compartmentalized, and not have to worry about your outgoing mail being rejected.