I did some more testing. When I save the -shine map in a format without alpha channel (e.g. DXT1), the model look the same as when using a map with data in the alpha channel.
This used to work differently, shine maps without alpha channel used to result in very glossy ships due to
Available -shine map without alpha channel. The level of environmental reflection is based on the luminance derived from the -shine map RGB values.
So I guess this feature was removed when moving to the PBR render pipeline, with the alpha channel in the reflectance map taking its role?
EDIT: Found it in an old thread from Swifty:
https://www.hard-light.net/forums/index.php?topic=89551.0Legacy specular maps, known as shine maps, are still accepted by appending “-shine” to texture maps. Legacy specular maps do not support gloss channels as the alpha channel is specified to attenuate environment map reflections. The RGB channels are not gamma corrected. A constant gloss factor will be applied to the corresponding model instead of per-pixel gloss.
So everything works as specified, I was just not up to date how the old formats are handled nowadays.