Ok, this is the situation: I can do convincing dense and thin nebulae. It's the in-between that's the problem.
This is the setup: I have some well chosen poofs which interact well one with the other (I may even animate them later, although that may not be necessary). For the background I use a skybox with a
very dark (because it is additive to the distance fog) and blurred image. This just emulates distant lighting variance in the nebula. You don't want a detailed and well defined image here - imagine a Sun behind a very dense cloud on an overcast day. You won't see its shape, just a lighter patch of sky.
So these are the options:
1. Very dense nebulae. Just poofs with no skybox. Works very well. There is no more pop-up than in the original game (since it's the same thing, just different image maps). Asteroids look good.
Problem: thrusters, lasers, beams, explosions and other glowy things don't get obscured by the fog (they don't get obscured in the original game either).
2. Very thin nebulae. Poofs+skybox+very high maximum fog distance (I can do this via objecttypes.tbl, thanks for the advice).
No pop-up for anything except asteroids, since I can't control them in objecttypes.tbl, or don't know how. On the other hand, glowy things look fine, since they don't need to be obscured by the fog.
3. Moderately dense nebulae. These would look the best if they didn't have the most problems. Poofs+skybox with dark and blurry image. This ads another layer to the nebula and gives it more dimension. However, objects
will pop-up, as you've all noticed.
What is annoying here is the fact that I am pretty sure that pop-up is a vestige of the times past when they were trying to preserve framerates in the nebulae, and absolutley unnecessary for modern machines. I'd pull a guess out of my a** that it's just a single switch somewhere in the code, or something like that. As i said, if you look at the table example here:
http://www.hard-light.net/wiki/index.php/Objecttypes.tblyou will notice something like this:
$Fog:
+Start dist: 10.0
+Compl dist: 500.0
+Disappear factor: 1.5
Start dist and
Compl dist control the minimum and maximum fog distance for a ship type, however if I try to use the
disappear factor, the game will report an error for some reason. And it sure looks like something that could be used for controlling the pop-up. So if anyone can help with this...
Also, glowy things are again not obscured by the fog.
I will post some videos and images later when I have the time so you can see what I am talking about.
Concerning the ambient lighting, I already boosted it up in fred, it just looks too dark in the youtube video because compression will do that sometimes. Although ambient lighting is usually evil, in this case it works because most models have baked-in GI so it basically just brings that out, as it should in a diffuse lighting setup like nebula.
I've also seen someone coded crepuscular trails recently. Combine that with the setup I have here and you'll get nebulae which can almost pass for volumetric. Perhaps I'll try to contact the guy later.
tl;dr
I am aware of the pop-up, if anyone can help with that, please step right up. Thanks.