1) IMHO, never use JPGs if you can help it. The compression results in icky artifacts.
2) DDS files are horrible for nebula art; they don't handle gradiants very well.
3) If you can, use DDS files over JPGs for ships in general. They're natively supported by video cards, resulting in faster performance and less memory usage all around.
4) TGAs are the best quality for anything.
5) PCXs are good if you're only using 256 colors (You can only use 8-bit -color PCX files) and can't use DDSes.
6) PCX and DDS files are supported by *everything* that can run fs2_open.
I'd say
1) Make all textures as TGAs
2) If they're big, try out other file formats to see what gives the best quality vs filesize, and include that with the TGAs. FS2_Open will automatically use the TGAs if -jpgtga is specified, otherwise, it will use the smaller DDS/PCX files.
Edit: Woo, psychic powers again. *Hides his seeing orb*