FFmpeg has worked on the Mac for at least 5 years now.
Can't FS2 just do something like include basic support as it stands right now, with a launcher option to load external libraries if they are present on the system? Handbrake, for instance, can't be distributed with a CSS decrypter, so it's distributed as just a plain video encoder *unless* you also have VLC installed, in which case it will happily borrow the decryption libraries.
Just have FS2 try to link against VLC's copy of libavcodec, and if the link fails it falls back to the current playback system.
(I am aware that this is not as simple as I make it sound, but this would be a viable solution, no?)