The wall would have been manned back in the day. Even if you sneaked a small force through, there's no way 10,000+ men are getting through, let alone wagons.
More than preventing invasions, it made it much easier to tax goods.
Every time I see this I wonder how easy it would be for me to get around this. I’m sure it wouldn’t be a problem with a boogie board (depending on how strong the current is), but there must be border guards there to stop you. If you never set foot on land and go back to the other side is it an issue?