A-Ha! Found the limits!
There are three defines that govern this:
#define PLAYER_MAX_DIST_WARNING 700000 // distance in KM at which player gets warning to return to battle
#define PLAYER_DISTANCE_MAX_WARNINGS 3 // maximum number of warnings player can receive before mission ends
#define PLAYER_MAX_DIST_END 750000 // distance from starting loc at which we end mission
Basically, if you move 700000 units from the world origin, you'll get warned. Move out beyond 750k units, and you'll get killed. Very, very messily. What happens is that a bomb will go off hitting the player ship with 10 times its current hull value as damage, making sure that nothing remains (Incidentally, this is also why cheating yourself invulnerable will allow you to get past this, as the damage will not be applied to your ship).