Okay, here's what I personally think you should do.
1. Create a new directory for the game. Keeping it in .wine will just unnecessarily complicate things. Something like
/games/freespace2/ might be good, it doesn't really matter.
2. Move all the retail VP files from the current FS2 directory into the new one. When you have done this, the directory should contain the following files only:
root_fs2.vp
smarty_fs2.vp
sparky_fs2.vp
sparky_hi_fs2.vp
stu_fs2.vp
tango1_fs2.vp
tango2_fs2.vp
tango3_fs2.vp
warble_fs2.vp
and possibly OggCutscenePack.vp and multiplayer mission pack, but these are not absolutely necessary for playing the game (although the cutscene pack is nice).
3. Move the RC2 builds into this new directory.
At this point, you should be able to run the game and confirm that it works with the retail data.
Yes, it's that simple. Although you
might need to set the permissions of
../freespace2/ directory so that the game can create the
../freespace2/data and subdirectories (like cache, mod data directories et cetera), plus write files in those directories.
After confirming that things work at this stage, you'll likely want to add a new directory called
../freespace2/mediavps/. Into this directory you will want to put the MediaVP files which increase the visual quality of the game quite a bit.
If you chose to download mods through the installer, you might want to move other mod directories from the original directory as well.
You will also want to set the game to use the correct resolution and bit depth; native resolution of your monitor is preferable (FS2_Open supports widescreen resolutions remarkably well) and bit depth should be 32. These options are configured by a file in
~/.fs2_open/fs2_open.ini .
And then we come to interesting bit; enabling features and using mod directories. Or more accurately, setting up the command line.
The command line options of FS2_Open are multiple and various. Most are explained in
here, but for now suffice to say that the Launcher on Windows is basically used as a GUI to manipulate the command line (and video resolution and sound and other options). On Linux there's some kind of launcher I believe, but it's relatively easy to set things up manually.
Basically, fs2_open reads a command line from a file in
../freespace2/data/cmdline_fso.cfg. It's a plain text file and you should save the static command line options (which you will likely use majority of the time) there.
EDIT: typo fixed.
My suggestion on what to put in that file would be as follows (with shameless advertisement of my personal lighting settings, your mileage may vary):
-spec -glow -env -normal -nomotiondebris -missile_lighting -cache_bitmaps -dualscanlines -targetinfo -rearm_timer -ship_choice_3d -weapon_choice_3d -3dwarp -warp_flash -snd_preload -no_emissive_light -ambient_factor 35 -ogl_spec 20 -spec_exp 15 -spec_point 1.2 -spec_static 1.5 -spec_tube 1.5
You can look up what these cmdline flags do in the Wiki. For specific lighting options, look
here.
Mod directories are activated with the -mod command line switch. For example, if you want to enable MediaVP's, you would run
fs2_open_r -mod mediavps. Multiple mods can be run by cumulative mod command line; for example if you want to run Blue Planet which requires MediaVP's, you would run
fs2_open -mod BluePlanet,mediavps. Just separate the directories with comma and it will work.
You may want to create startup scripts for mods; this is described
here. It's basically the equivalent of creating icons with certain command line options on Windows...
Good luck with things.