The OS X version can't play AVI movies, but will play the original MVE versions.
To install mods, create a new folder for them just like you do for Windows. But it gets rather annoying after that... to use them you have to edit the cmdline_fso.cfg file directly. This makes changing mods painful and slow, but you learn not to switch too often.
mod.ini and the launcher6.ini files are completely ignored except under Windows. Those files are both platform and launcher specific. The upcoming cross-platform launcher won't even use them (except for importing old mod.ini files). So on the Mac you can just delete those files since they are of no use to you there.
To use a mod you edit ~/Library/FS2_Open/data/cmdline_fso.cfg (it needs to be a plain text file!) and put in your various cmdline options that you would like. You would also add the -mod options you want to this file, such as:
-mod derelict,mediavps for playing Derelict with the MediaVPs as a secondary. Also, all of the options in cmdline_fso.cfg need to be on one line, the first one, so keep that in mind.
If you have a Windows machine handy this is actually pretty easy. All you need to do is set the various options and mods (assuming the directory names are the same) that you want through the Launcher and then copy the cmdline_fso.cfg file from Windows (data/cmdline_fso.cfg) to the appropriate place in OS X. It should contain all of the options that you need to play the mod with the settings that you like (that's all that the Launcher does anyway, just through a GUI).
You can actually get creative with symbolic links and what not to have each mod as a different install, but not waste more HD space. Then you can make use of different cmdline_fso.cfg files that you only have to edit the once and just use the mod simply by double-clicking on icon for it. I do that for TBP, Derelict, SoL, BtRL, etc. I can walk you through that if you like, but it's a bit more on the technical side.
