I'm not familiar with most mods out there, but giving a proper description of problems encountered along with contents of fs2_open.log after running said mod with debug build would go long way in determining what the problem is.
However, my guess is that the problem is not in 3.6.11 exes, but in the mod itself. 3.6.11 is far stricter and comprehensive in its error checking routines and is far more likely to spew out errors, warnings and asserts when there is bad mod data. And this is a good thing as it helps making mods far more stable and bug-free. Of course, it also breaks older mods like you've said. But in the long run, it's definitely worth it.