Why not buy a 3$ power splitter cord from Newegg, or even your local computer shop which will probably give you the extra reach you need? You don't need to go modifying cables to use the external power cable on the card. Trust me, I know exactly what you're talking about and yes, it's a pain in the ass, but you can get a splitter that'll reach.
Yes, more framebuffer on the card will smooth things out, but I honestly think your install may be damaged, or you're using the AdvEffects VP and not knowing it or something. If FEAR runs fine with a 9800 Pro (and Yes, I DO know how that runs as well, same card, same game, runs good) and FSO does not... perhaps it's in software. Maybe you should do a fresh install of FS2, then apply the patch, then run whichever version of FSO you'd like, with the necessary VP's. I've had unexplanable slowdowns, crashes, and the like, and what I've done to fix the problem is just move my mods somewhere else on my drive, uninstall FS2, axe registry entries, reinstall and put my mods back in, and it plays much smoother after that.
Here's one other suggestion. Turn on the Show Statistics and Show Memory Usage in the launcher. Watch your memory usage. If your available memory keeps falling to 0, and it skips like mad, I would say it's something in software, for I've had that exact problem for the longest time untill I switched over to OpenGL mode. When I started using OGL, things were a LOT better, and OGL has come a long way in a year's worth of updates from the FSO crew.
I'm not trying to talk you out of buying a card, well.. maybe I am, but I say don't waste money on AGP if you're planning on upgrading your system. If you're going to upgrade your motherboard, get into PCI-X and THEN go nuts with the videocard you need. There's lots of people with cards just like you that can play FSO just fine. If you're perfectly happy with keeping your system forever, and dont care about upgrading to PCI-X, then I'd say get something BETTER than the LE card. I'd reccomend the 6800OC that BFG makes, even though it's $100 more expensive, it's worth the extra bucks. Either that or wait a little longer till the 7800 cards drop below 250, then they'll definetly be worth the purchase. Some are already down to 300 right now.