Position yourself in front of a mirror and you'll notice it immediately. The text on your sweatshirt is reversed. The part in your hair has switched to the other side of your reflection's head. The mole on your left ear stares back at you from your mirror image's right earlobe. Before you stands a bauplan reversed; what was once left is now right, and vice versa. And yet, up remains up and down is still down — as though the mirror knows to switch left and right, but not top and bottom.
This, of course, is not the case. The mirror doesn't "know" anything about your position; it simply reflects the light that hits it, doing so as objectively as any inanimate object knows how. Why, then, when that reflected light reaches the photoreceptors in your eyes, has your mirror image been reversed from left-to-right?
The short answer is that it hasn't. In fact, the question of what makes the horizontal axis so special in the context of mirrors is itself flawed. That's because a mirror does not reverse images left-to-right or top-to-bottom, but from front-to-back. In other words, your mirror image hasn't been swapped, but inverted along a third dimension, like a glove being turned inside out.
Here's a thought experiment to help illustrate the concept of front-to-back reversal. Assume, for a second, that you are capable of squeezing your body perfectly flat. Imagine, also, that your body is able to pass through itself, without damaging any of its various tissues. When you stand with the tip of your nose pressed gently against a mirror, it's easy to assume that the image you see looking back at you is the result of non-mirror you turning in-place 180 degrees and stepping backwards, through the mirror, into mirror-land. This is not the case.
In actuality, the back half of non-mirror you has been pressed flat in the direction of the mirror. As your form began to pancake, the front half of your body (that is, all parts of your body situated behind the tip of your nose, but still in front of the back half of your body), the back half of your body and the tip of your nose all came to reside within the same plane (i.e., the plane occupied by the mirror). But then your back half kept pushing, continuing on its journey through the plane of the mirror and passing right through your body's front half before re-acquiring its "normal" shape on the other side of the mirror (probably with a satisfying *POP* sound). This new, inverted you is symmetrical to you, but your two bodies cannot be superimposed. In chemistry, such entities are said to be "chiral."
Here's another way to think of it, widely popularized by physicist Richard Feynman (see the interview response featured here). Stand in front of a mirror, and note which direction you're facing. For the sake of this thought experiment, let's assume you're facing North. Point due East with your right hand, and your reflection points East as well. Point due west with your left hand, and your reflection gestures in the same direction. That's because these directions both lie along a plane parallel with the mirror. Similarly, point up or down and your reflection will follow suit, motioning in the same direction.
But deviate from that parallel plane even a little and thinks go wonky. Remember: your image has been reversed along the axis perpendicular to the mirror. Try pointing directly at the mirror, such that your fingertip is now directed due North. Your reflection is now pointing directly at you — not North, like your finger, but South.
For more on the mirror paradox, thinking in three-space, chirality and handedness, see this great explainer, presented in the form of a conversation, by UC Riverside's Eric Schmidt. See also: The Left Hand of the Electron, by Isaac Asimov.