Limiting NGINX Proxy Access on VLAN

So, I have been playing with Nginx (and it's pretty awesome).

My homelab has several VLANs, including a "management" VLAN where my networking equipment lives.

Mine (and my families) personal devices are mainly on our "LAN" VLAN, which can see everything by default, except only certain devices (my laptop and PC) are allowed to see the management network.

The problem I've run into, is that if I setup a proxy for, say my Pi Hole, then all devices on my LAN VLAN can access it, regardless of whether or not they are a part of my mgmt_approved alias (I guess this is a bit expected).

I am wondering what the best way to limit certain devices from accessing specific proxies may be. I see that you can limit them on the Nginx side like so , however I'd prefer to have access rules defined on my firewall (current rules for LAN VLAN).

It seems like once something is allowed to communicate with NGINX, it will be able to access whatever NGINX gives permissions to, and further firewall rules do not matter since traffic will then be going through the proxy.