I'm experiencing pretty major hits towards memory as well, resulting in complete freezups for a second then back to normal gameplay. Sometimes I will CTD when this happens. I have my stats turned on and I see a major hit to Texture memory and my total available memory. I've got a 2 GB swapfile according to FSO, and that usually stays pretty static. I'll start a mission in Derelict, pretty much any mission and I'll start off with 300 MB free texture memory. Any time something new jumps in or anything, the text. mem goes down, which is to be expected. Turns out I hit rock bottom constantly, 0 MB texture memory, and it seems to free up some but it just gets sucked back down again, and when it hits 0, the game freezes.
I'm running an Athlon 64 3000+ on Windows XP Pro 32 bit, 1GB RAM Radeon 9800Pro 128MB onboard VRAM, my flags are similar to those above who have the memory hits too. I have pcx32 turned off, am using spec and glow, 2dpoof, etc. I experience this playing the retail campaign or add-on, specificily BR's updated Derelict mod, and I am using the 3.6.6 media VP zpack, minus the music VP.