Umm.. you are getting the format completely mixed up...
Dxt5nm is the format used. Green + Alpha. Green is green - Red is in Alpha and blue is derived from those two channels.
The grey ones have extra un needed data in the red and blue channels. Aparrently the extra red and blue can cause extra artifacts in compressed normal maps.
@Spoon If they are saved as DXT5nm and they are grey don't worry about it. nm stands as you guessed for normal map, but the channels get switched around and only two are used. DXT5 won't do the job, it has to be DXT5nm.
I think you just reiterated what I just said. But they do not need to be saved specifically as DXT5_NM if the previously mentioned steps are taken.
Green channel data and Alpha channel data are used. We both have that. Red and Blue are irrelevant, we both have that.
So long as both cases are true, it does not need to be saved as NM, _for the purposes of use within the Freespace Open engine_.
As can be exampled by the existence of the normal maps in the mediavp's.
The Advanced VP contains normal maps that are actually saved in the u8888 DDS format to reduce AA compression artifacts along angled lines.
DXT5 and DXT5_NM contain the same compression scheme. Incidentally, as long as the channel management is done, DX1 (with alpha) can be used for a normal map.
Why in the name of Kobol you would want to is beyond me.
It may be called _NM because it does the conversion for you (sometimes well, some times not), but that does not make it the only viable option for creating a $ship$-normal.dds.
Shaders can use different techniques to render tangent-space normal maps, but the normal map directions are usually consistent within a game. Usually the red channel of a tangent-space normal map stores the X axis (pointing the normals predominantly leftwards or rightwards), the green channel stores the Y axis (pointing the normals predominantly upwards or downwards), and the blue channel stores the Z axis (pointing the normals outwards away from the surface).
If you see lighting coming from the wrong angle when you're looking at your normal-mapped model, and the model is using tangent-space normal maps, the normal map shader might be expecting the red or green channel (or both) to point in the opposite direction. To fix this either change the shader, or simply invert the appropriate color channels in an image editor, so that the black pixels become white and the white pixels become black.
Some shaders expect the color channels to be swapped or re-arranged to work with a particular compression format. For example the DXT5_nm format usually expects the X axis to be in the alpha channel, the Y axis to be in the green channel, and the red and blue channels to be empty.
Our shaders are written to only utilize the data in Green and Alpha. It may change at some point for proper tangent space normal mapping, and may even prove a benefit with the -height option, but that will be a bit before that happens. I think getting us a multi-pass capability in the rendering system for dynamic lighting (which can lead to a slew of other things) will make the option of reading the blue channel in the future more viable or simplifying the normal map to reading an RGB with the Alpha serving an entirely different purpose.
On an unrelated note: Layer support will be coming to DDS soon.