It's because an aeroplane turns by rolling its lift axis - even if the rudder was powerful enough to deliver yaw rates that were comparable to the elevators' pitch rates, the lack of vertical aerofoils around the aircraft's CoG would result in an enormous turning circle while the fuselage tries to act as a lifting body.
In space, since you have no atmosphere, similar pitch and yaw rates would give similar turn rates and circles in both axis.
The real problem, though, would be the g-forces. Humans can tolerate a fair amount of positive g in their vertical axis (positive pitch or vertical accelerations), but not much lateral. So, in real space combat, you'd still need to roll and yaw to avoid breaking the pilot's neck or bruising their brain on the inside of the skull from high yawing and side vectoring forces.
For what it's worth, designs like the Viper are very good in principle - the cockpit is slightly forward of the CoG, so the rotational and linear swings of any attitude change in pitch or yaw will be in the same sense (ie pitch up and the cockpit rises about the CoG, yaw left and the cockpit will rotate and swing left). The Blackbird would be horrible with e cockpit so far aft - the opposite rotational and translational movements (pitch up give up rotation but downward swing of cockpit, and left yaw swings cockpit to right) would be disorientating and vomit inducing, while craft with the cockpit right at the front (like Ralph McQuarrie's original Raider concept) would be subject to very high cockpit g forces just from the swing motion, never mind the change in velocity vector. Then there are fighters like X-wings, whose long, heavy wings would vastly reduce roll rates and long nose reduce pitch rates, and TIE fighters, whose wing inertia would cost them a vast amount of manoeuvrability (as well as visibility). Compare that to the Vipers' stubby wings and relatively short nose, and you'll see the only SW fighter that makes any sense is the A-Wing.
