I know next to nothing about FREDing that wasn't in a tutorial on the site, but you might try looking around the Serenity for the root of the issue. I've completed this mission twice - the first time, I failed to save the Serenity, but completed all other objectives save destroying the minelayers prior to calling in the attack, and I called in everything I could and zerged the Carthage. Got the 'light losses' outcome, and I know I lost a few fighters in there as well as the frigate, though I believe all the gunboats made it. I also accepted it's surrender, if that hasn't been looked at as well.
The second time around, I did save the frigate, then played the mission roughly the same, again, calling in the attack without attacking the minelayers beforehand. Once again, I took the Zerg approach, and once again I lost a small amount of fighters and no gunboats, meaning a net gain in tonnage saved in this run. Except the debriefing claimed we'd taken catastrophic losses. So there might be a correlation there.