Hmm, what about something like this..
When the camera turns its back on an asteroid, its distance is calculated relative to the camera. If it's greater than a certain distance, it's instead flagged as being a "flat" asteroid.
What this means is that the asteroid is noncollideable, and will be represented as one square polygon (or a larger triangle) that always faces the viewer. The image for this is included with the asteroid, and corresponds to a specific orientation of the asteroid.
When the camera turns back around, and gets close to the asteroid, the polygon is replaced by the full asteroid model, rotated relative to the viewer to match the texture (so the change is instantaneous)
To cut down on collision checking, each asteroid could only have collision checks and calculations performed on its radius, or its bounding sphere. (These could be specified in asteroids.tbl, and include full-polygon calculation if the asteroid is some really odd shape.)
As a result, you'd have a helluva lot less polygons on screen even with a massive number of asteroids at a great distance.
Actually, the render-to-texture code could be used to generate asteroid images on the fly to add some variety.
Actually, just allowi8ng a less-accurate physics calculation mode for asteroids would probably help a lot too, assuming asteroids are batch-rendered.