Ahh yes, I was meaning to tell you all last night (my time) that I'd fixed the dialog error - also nailed the black screen around dialogs bug with the same fix.
In your registry, the game needs to be set to run at 640x480, only then will windowed D3D mode run in a 640x480 window at 640x480.
To do this:
1) Goto HKEY_LOCAL_MACHINE\SOFTWARE\Volition\FreeSpace2
2) Goto the string value "Videocard"
3) Ok, mine says Direct 3D - Primary Display Driver (32 bit) (1024x768)
Now if you wanted 640x480 mode then change it to read Direct 3D - Primary Display Driver (32 bit) (640x480)
4) Run the game in windowed mode.
It's for the complexity of the above that FS2 originally had a launcher.