As far as I know: Things that use alpha blending are put into BitmapX: list, while things that use additive blending (black=transparent, file has no alpha channel) will use Bitmap: list. The descriptions in the commentary of stars.tbl are a bit misleading due to volition's hack with the green=transparency on BitmapX PCX files... so planets and other solid objects go to BitmapX: while nebulas, suns and other assorted brightness-related, see-through backhround objects go to Bitmap:.
When you convert a non-alpha channel bitmap into DDS, you need to use either dxt1 or u888 format. When you convert a file with alpha channel, you need to use either dxt3, dxt5 or u8888.
dxt1a and dxt1c will be interpreted similarly by FS2_Open - their alpha channel will not really be interpreted as transparency. u888 (uncompressed, 3x8 bit channels) doesn't have alpha channel at all and it is useful for nebulas since it preserves gradient quality, while dxt compression doesn't deal with gradients well at all without some rather fancy hoops to jump through first. u888 is pretty much the only way to make things like Sun bitmaps look good...
dxt3 is good for textures with sharp transients between transparency and opacity, like Moon, asteroids and other objects. dxt5 is relatively good for gradient alpha transparent textures, but beware of artefacts again. Atmospheric gradient can easily become rather heavily artefacted while the rest of planet looks just fine, which is rather annoying.
A big advantage with uncompressed textures for nebulas and suns (anything wirh gradients basically) is that they upscale much higher without pixelation than dxt compressed versions.
Regardless of what filetype you select for the final texture, I recommend testing things as TGA beforehand. If you get the 32-bit RGBA TGA file to show correctly, the file converted from that should show up properly just as well. Basically what you need is to make sure that
1. Alpha channel shows 0% value on the transparent area
2. The alpha channel is saved along with the image.
Then it should just work when you put it on BitmapX and call it into mission. For comparision, open any planet in mediaVP's to see how the alpha channel should behave.
You could even try by making a white layer, then applying an alpha mask with white center and black edges on it...