It sounds like our starting points are in different places. Maybe just different software. When I use the compressionator it spits out a green texture (DXT5 xGxR) using just green and alpha.
I think you just reiterated what I just said. But they do not need to be saved specifically as DXT5_NM if the previously mentioned steps are taken.
Sounds like you end up with blue+red also using dxt5nm.
For someone new to this, dxt5nm has a better chance of success than channel swaping
It's preferable that people know what they're doing from the beginning, even if they use tools such as selecting dxt5nm as the format... If they just blindly assume they HAVE to select the dxt5nm format as if it were the only thing that worked, they can easily end up in a situation where they save an already functional normal map again into dxt5nm format and, well, since it moves the red channel to alpha and populates red and blue with green channel content, they end up with the same information on all channels as follows:
RGBA -> GGGR -> GGGG
We've had this happen a few times in the past and almost invariably explaining how the system works with the channel information clears up the confusion.
Incidentally at one point it was possible to use TGA files as normal maps, as long as you had the red channel copied to the Alpha channel. I haven't tried this in a while though...
Regarding "green" and "gray" normal maps, the functionality is the same. However, there may or may not be some difference in how well the files compress, which depends largely on the compression utilities used. Using layman's logic it would make sense that loading unnecessary information to Red and Blue channels might affect the compression quality as opposed to full black channels which is just a bunch of zeroes in the file. It doesn't really affect filesizes, but it may have some effect on how the DDS compression algorithms manage to compress the file and how much or how little artefacts are generated in the file. Of course, uncompressed u8888 files aren't affected by this (their filesize is defined only by resolution, amount of channels and bits per channel value), but since we've had good experiences with the green variety of normal maps, we use them for the sake of consistency in the MediaVP's.
Sometimes a small amount of guassian blur can give a smooth bump using a lower intensity.
Experimenting is probably the best way of learning how different gradients work out with different normalmap filters. As a basic rule, sharp change in contrast (from black to white over one pixel change) will result in maximal angle on the normal map. A wider linear gradient will show up as wider linear slope on the normal map. Blurred gradients tend to produce rounded (bell or sine curve) normal map profile.
Also, 4 sample normalmap filter tends to create sharpest transitions, while 3x3, 5x5, 7x7 and 9x9 tend to produce more "rounded" or less detailed normal maps. My experience is that 4 sample and 3x3 work best for small, sharp details, while the 5x5, 7x7 and 9x9 will give better quality of the slope sampling at the expense of resolution. However, if you make the normal map on higher than final resolution, you can use the higher sample filters and the detail will still be preserved adequately...
I would upload sample images but my internet doesn't want to upload files (large packages won't upload, specifically). Perhaps later...
NB: I'm not telling this to you Water, I think you have a pretty good idea of what does what with normalmaps...
* Herra Tohtori takes a look at the Valkyrie
...so... yeah. The tips and tricks section of this post is aimed for Spoon et al.
Here's a few samples of what does what...
Base elevation map:
Here's a few samples about what different things actually do. Hard transitions from black to gray to white, as well as black to white directly; Gaussian 5 pixel blur on the right side; Radial gradient from white to black (linear from center to edge), and 30 pixel gaussian blurred version on the right side of it.
Basically the unblurred radial gradient shows up as a sharp straight-edged cone, whereas blurred one shows up like a rounded missile cover or something like that.
4 sample normalmap:
Worth noticing - change from white to black is equal to change from white to gray or gray to black. 4 sample filter doesn't really bother to take the brightness of the areas into account in this kind of cases. Also, when you zoom into the gradient stuff - especially the radial gradients - you'll notice a lot of artefacting, dithering and stuff like that, which will (if left alone) make the normal map rather fugly in the game itself. Resizing the map to half the size usually tends to smooth out artefacts like that (which is one of the reasons I usually make normalmaps at higher resolution than the final map will be...).
3x3 filtered normalmap:
Smoother on the gradients but slightly thicker on the sharp edges.
Smoother still, thicker still. Same trend continues when the number of samples increases.
Same old, same old...
And this is where GIMP's normal map plugin ends it's offerings pretty much.
In general, blurring tends to lead to "smoothed out" lines and leaving hard contrast changes on the height map usually works better at least for me (unless I actually want smoothed, rounded normal map details... like pipes or dents).