Disable specular disables the -spec maps on ships, which make ships shinier when facing a light source. For maximum graphics keep it on.
Disable glow maps, well, disables glow maps. They're the glowing lights on ships, common on Shivans with their glowy red bits or for docking lights on carriers. Keep this on for best looks.
Disable environment maps - environment maps are pre-baked reflection textures that provide a "computationally cheap" way to get reflections without actually having to calculate them(like you would with raytracing). Keep this on.
Disable normal maps - normal maps try to simulate depth in the texture without actually having extra details on the model(so no extra polygons). They really help ships look less "flat".
Enable emissive light from ships - This makes all ships naturally produce a small amount of light. It has recently been changed to off by default since the game usually looks better when you have a greater difference between "lit" and "unlit" surfaces. This is down to preference and your other lighting settings. For some presets that actually work with modern FSO builds(and not the super old ones that were made for an entirely different renderer in 2014) check the wiki
https://wiki.hard-light.net/index.php/Sample_Lighting_SettingsDisable height/parallax maps - As the name implies height maps store per-pixel height data on the texture presenting more complex shapes without having more polys in the actual model. Very similar to normal maps and should be kept on.
Enable 3D shockwaves - switches the standard 2D sprite shockwave effect with a 3D model. This means shockwaves aren't always facing you and can instead propagate in any plane. The damage and blast effect from a shockwave are always spherical though so this is just a visual change, you can't fly over a visual shockwave to avoid damage. Mostly down to personal preference.
Disable post processing - completely disables all post-process effects. Post-process effects mean they're applied as a filter to parts or the full scene after everything else has been rendered. Keep this on for the HDR tonemapping, anti-aliasing options, and anything coming from post_processing.tbl
Enable soft particles - usually used for smoke effects in other games, FSO uses them for various effects, including some weapons. For best graphics, enable this.
Enable FXAA anti-aliasing. FXAA or Fast approXimate Anti Aliasing is a post-process anti-aliasing method that applies a fullscreen filter to try and smooth out the edges. Its main advantage is that it's very easy to run but the filter can make the game look blurry. For best graphics use SMAA instead.
Enable SMAA anti-aliasing. SMAA or Subpixel Morphological Anti Aliasing is also a post-process anti-aliasing method but it's more complex than FXAA and produces a generally sharper image while still eliminating aliasing(or jaggies). It's slightly slower to run, especially on very high resolutions but if you're aiming for best graphics enable it.
Disable lightshafts - lightshafts are an effect that tries to simulate the way light disperses when hitting surfaces in air. Now, Freespace is set in space but its rather unrealistic in many other ways so you might want the extra eye candy.
Enable Framebuffer shockwaves - these try to make explosions distort the space around them, like they would in air. They're relatively cheap to run, but again rather unrealistic. If this doesn't bother you enable these to get that distortion effect around explosions.
Enable Framebuffer thrusters - similar to the last one except it tries to give a heat haze around thrusters.
Disable Deffered lighting - This essentially bundles and calculates lighting in one pass. On modern systems it should both look better and run faster so don't disable it.
Enable Shadows - pretty self explanitory. It's the single most "computationally expensive" option to run, being able to halve your framerate or sometimes hit even harder. FSO in general is pretty easy to run on any machine that can handle modern AAAs(so a GTX 1070 or better) so if you have something like that feel free to enable them.
Some additional settings from other categories:
Disable motion debris - motion debris is an effect that tries to give you a sense of movement by using small 2D sprites. It doesn't look great but for some it's more difficult to tell what you're doing if this is disabled. The game looks cleaner with it off though.
Enable 3D warp - switches the standard warp-in effect with a 3-dimensional one resembling a wormhole instead of a flat portal. Preference.
Enable flash upon warp - makes warp-in portals a light source that lights up everything around them. Preference.
Stretch interface to fit screen. The assets for the briefings, mainhall, and UI in general are made in a 4:3 aspect ratio. Enable this to stretch the image to your screen resolution. This will result in circles becoming elliptical and everything stretching out but it will get rid of the pillarboxing. Personal preference but keep it off if pillaxboxing doesn't bother you. Missions will always be rendered at your screen or window resolution with no stretching, this flag only affects menus.
So quick rundown for best graphics. If a flag isn't named always keep it unchecked.
Check these flags always:
Enable soft particles
Enable SMAA anti-aliasing
Enable Shadows
Check these flags if you're not super bothered by realism in an arcade game:
Enable Framebuffer shockwaves
Enable Framebuffer thrusters
Down to personal preference:
Enable 3D shockwaves
Disable motion debris
Enable 3D warp
Enable flash upon warp
Non-graphics options that you might want to enable unless you want a very retail-like experience
Enable info next to target
Enable rearm/repair completion timer
Down to preference:
Enable 3D radar - this switches the default forward-facing 2D radar with a 360x360 3D one. It can give you better situational awareness but for some it looks uglier and takes more time to read. Try both and see which one you like better.
Some mods without 2D icons always use these, but if you like them you can use them everywhere:
Use 3D models for ship selection: instead of the standard 2D ship icon you'll see a top-down render of the ship in the selection screen
Use 3D models for weapon selection: Same except instead with weapons
Additional:
Lighting settings. The ones on the wiki are from 2019 so they use -disable_emissive_light but since that has been changed to off by default it will do nothing. Since the rendering engine is the same the settings should still work, just keep in mind that if you see -disable_emissive_light that just means you should uncheck the "enable emissive light from ships" flag.
