Normally, this should work.
Run the Launcher with admin privileges, make the resolution settings, and let it do its job. And instead of clicking "Run", click on "Apply" before.
Have you tried compatibility options?
-Thaeris
In general, FSOpen doesn't need to be run in compatibility mode.
I think you need to set the game resolution to the resolution that windows vista is running at. Same as what Thaeris said I think.
Actually, no. FS2Open will run at any resolution you specify.
Now, Tantalus, I'm assuming you downloaded 3.6.10 from the release thread recently, as well as the Launcher. If not, please do so again; There were some issues with the first release of the 5.5e Launcher that would cause an error like this.