Debian's non-free fglrx module works wonderfully with FS2:SCP! No tearing at all in the first level of the official campaign.
To use (run as root):
# init 1
# modprobe -r radeon
# modprobe fglrx
# init 2
Unfortunately, when run with BtRL, the game simply hangs and consumes my entire CPU till killed when hitting "Commit" to start a mission. Looks like I'll be reporting that bug to them. Thanks for helping me solve my generic graphics bug! Now to look into game-specific bugs...
EDIT: This seems to work only when run with the latest (08-06-20) version of 3.6.10-beta. It seems that 3.6.9 always hangs: I forgot I was playing BtRL on 3.6.9 and FS2:Open on 3.6.10... When I tried running FS2:Open on 3.6.9, it hung just like the BtRL demo I have installed, making me think it's an interaction of the video driver and a 3.6.9 bug that's fixed in 3.6.10. Now to figure out if BtRL can be upgraded to 3.6.10, or if I have to wait for the third patch release...
EDIT 2: Copying the 3.6.10 binaries into the BtRL directory and modifying the startup script to use those binaries seems to have done the trick, as far as I can tell, I can now run all FSO:SCP projects without problem! I just needed the fglrx module and the 3.6.10 beta.
I attempted to clear the data/cache folder, but that didn't do anything to solve either problem.