It makes sense that left and right is more abstract to our brains than up and down. As humans we're symmetrical from left to right: my right hand is a mirror image of my left. We're asymmetrical up and down: my feet are different from my head. We also have gravity which our whole lives points down. There's no equivalent to that in left vs right.
This is why when learning to write, kids constantly mix up their "b" and "d" and have problems remembering which way "3" should face. Our lizard brains don't really care for left vs. right.
Fun fact... Ancient Greek (and a lot of other early alphabet-based scripts) was often written both left and right. It was called boustrophedon, "as the ox plows."
Basically, write the letters left to write, get to the end of the line, next line goes right to left, etc. We're so used to symmetry that it is actually reasonably natural, to your point.
Whether something is up or down is more meaningful in terms of our average interaction with that object than whether it is left or right. Conceptually the latter is inherently relative to the observer, and in many circumstances the observer could just turn around to change the state of whether it's left or right, so it doesn't matter which way it is. Whereas they can't do as much about many things that are "up" or "down" like the sun, a tree canopy, the earth's mantle. Those things are more constant generally so it's easier to grasp them