Okay, open up regedit, go to HKEY_LOCAL_MACHINE\SOFTWARE\Volition\FreeSpace2
There should be a key called VideocardFS2Open. If it does not exist, right click an empty part of the panel that shows all the keys and create a new string value.
If you want to run, for example, 1024x768x32bit, the value of VideocardFS2Open should be "D3D8-(1024x768)x32 bit" (without the quotes, of course). If you want to run 16 bit, delete the 'x32 bit'. If you want to run 640x480, change 1024x768 to 640x480. That's about it.
The launcher didn't work for me either, so I just peeked at the code to see how it reads the registry (which is in a very untolerant method).