AVI files require codecs to play, even on Freepace Open. If you don't have the correct codecs, the videos won't play.
Theora also requires codecs, but as far as I know, there is just one and single Theora codec, unlike avi codecs, which are multiple... to be more exact, avi files can be of many variety. Most avi files are MPEG-3 compressed. And to make things more complex, MPEG-3 includes for example DivX, Xvid and 3ivx compression codec types. There might be other types also, but I don't know about them really.
So, to be able to play all the different avi files you need several different codecs, and Freespace Open also needs those codecs in order to play the videos. Unfortunately, some codecs and/or codec packs sometimes interference each other even in Windows' own media programs such as Media Player or WinAmp. So it's no big wonder that these codecs/codec packs also make things unclick in Freespace Open video playback system, resulting in unwanted phenomena such as picture not visible but sound audible, no videos played at all, or other things like that. DivX toolbar is one of the instances that when enabled, makes the avi playback in FSO not to work. That's the most common problem encountered with avi video playback in FSO, at least according to questions and help queries on these forums.
So, when the files are Theora compressed (*.ogg files, I believe) and FSO knows that and there's just the one and only Theora codec on the PC, things become *much* less complicated. Less complicated code works faster and better, and there are less chances to be considered in the programming work -> less bugs and other problems.
Besides, avi format(s) are widely considered outdated. Theora is better in almost every regards as far as I know.