It's become apparent for quite awhile now that we've needed this, so I decided to just bite the bullet and do it.

This combines the features from difficulty.tbl and the now-defunct UseNewAI command-line and mission flag to make one big übertablefile.
The nice thing about this is that unlike difficulty.tbl or the mission flag, you can have multiple profiles within your campaign. (Thanks to Sesquipedalian for the suggestion.) In addition to the retail profile (stored in the code, not the table file), you can create four additional profiles that you can configure whichever way you want. You can specify which profile the game runs as default, and you can override the default in FRED on a mission to mission basis. This is useful if you started FREDding a campaign using the retail AI but would like to use the new AI in some of the newer missions.
I'm still in the process of testing it, but it looks like it works pretty well and in any case it'll be in the new CVS build. In the meantime, I've attached a sample table file so you can see what it looks like. It's like an INI file in that you have a whole bunch of options that you can specify in any order. Some of them are lists of values and some are AI flags. Each is commented so that you can see what it controls.
There are a few neat settings in here. There are a few bugfixes, but you can also try some other neat things like enabling the FS1 shockwave behavior so that shockwaves damage your subsystems.

EDIT: Oooops. It looks like I uploaded my custom AI table instead of the retail one. I'll upload the retail table when I get back home.
EDIT2: Okay, new version re-uploaded.

[attachment deleted by admin]