Well, since everything else about your system seems top notch, you should probably make sure you have the newest video card drivers installed. The development for Vista hasn't come along as fast as it could have, and there's been a lot of bugs that make getting FS2 running in Vista a bugger. So if you haven't gotten new drivers from ATI, I would do that first and see what happens. Your system should be more than capable, I can only imagine it's Vista and ATI's drivers causing the hiccups in OpenGL mode.
You might also try removing the adveffects files if you got those with the installer. Look in your freespace2\mediavps directory, and there may be two files, adveffects and 710_adveffects, or something like that, you want to put those in a new directory in that folder to disable them, so I'd just make a new folder called disabled_vps and put them in that. They have high-memory usage enhancements, and since your video card only has 64 built in MB, the lag of accessing the rest of it in DDR2 could be causing some slowdown as well.