But a higher quality alternative would be nice, how much work and trouble it would be to let the operating system and any installed decoders handle movie decoding?
A lot. Otherwise I would have added code for this already. Using DirectShow under Windows is a crappy way to do it (as should be obvious by all of the problems everyone has), but it's about the only choice. We can use QuickTime under OSX (a pretty good solution in fact, can run through OpenGL), but QuickTime doesn't support DivX/XviD natively and the plugins for it cost $$ and generally don't work all that well. There is no solution like that for Linux, you have to use the decoder directly, and that would most likely be ffmpeg since it's the best overall solution.
ffmpeg has a problem though, it contains tons of unlicensed IP (mp3, mpeg2/4, etc), and there is absolutely no guarantee that the companies who own that IP would leave us alone forever. Also, ffmpeg contains both LGPL and GPL code, but we can only make use of the LGPL parts since the FS2 source isn't compatible with the GPL. This means that, at least for the builds which are distributed, we have to use a cut-down lib. It's not the end of the world, but it's extra work and extra concern to bother with. And if we stripped out the parts of ffmpeg which give us trouble then it's a lot less useful.
But all of that is really besides the point. As stated already, we should not even allow people to use any video format they want. There should be a well defined list, containing at most 2 video codecs (one being MVE), with specific audio format requirements (not mp3), that can be used with the game. If I was in a pain-in-the-ass mood I'd just rip out DirectShow support, add a mpeg1 decoder that supported all platforms, and be done with it. And if I was really in a strange mood then I might use RoQ instead, just for kicks.