I seem to recall something along these lines before, but I'm not certain. Is it possible to have the AI take control of the players shipif he goes outside a certain radius (Achieveable with distance sexps and a waypoint at 0,0,0) and fly him back towards the centre (again, possible with giving the newly AI cotrolled player a waypoints goal). Then, a few seconds later, returning control to the player. This is the way it works in Rogue squadron, and I've never had any trouble accepting it. Agfain like Rogue squadron, the rest of the field could be flat, tile textured earth. Not beautiful I'd admit, but better than nothing.
Anyway, it'd mean the actual landscape model could be a lot smaller, and have therefor a higher pixel/polygon density, and you wouldn;t need to have the player ramming into some invisible shield if they went to far out of bounds.
It's obviously not an ideal solution, but its worked for other games in the past and in my view would be a better choice than not having atmosdpheric combat at all.