Based on my experience with a late-2007 iMac with a slightly better card (ATI Radeon HD 2600 Pro 256 MB VRAM), I'd suggest making the following changes in the list of checkboxes in the launcher's advanced settings tab:
Under Graphics, uncheck:
"Enable post processing"
"Enable soft particles"
"Enable FXAA anti-aliasing"
and then check:
"Disable environment maps"
"Disable normal maps"
"Disable lightshafts"
possibly "Disable specular".
Probably don't check "Disable glow maps", though, since as I remember it's very hard to see inside the Theseus (such as for landing) without them.
Also under Dev Tool, check "Show frames per second on HUD", which'll display a number near the upper-left corner of the screen while you play, indicating what FPS you're getting. For the game to be enjoyable, I'd say that number should be at least 20 or so, and the closer it is to 30, the better, but you can decide for yourself.
You might want to try "Disable vertical sync" under Game Speed to see if that helps.