[The E]

No, we can't use .fs2. Because that extension is already reserved for mission files

What we can use, and what we I think already agreed upon, is .cvp.

[Halleck]

Oh- duh!

It's a good plan, and will prevent some needless confusion of hapless players.

[chief1983]

I think at this point we're just waiting for someone to tackle the implementation.

[Goober5000]

It hasn't been tackled in 2.5 years, so I'm unstickying this.

[Phantom Hoover]

Pity, it was a great idea.

[m!m]

I am very close to completing my cfile system refactoring which includes support for reading 7-zip archives: https://github.com/asarium/fs2open.github.com/tree/feature/vfspp
That branch currently does not have that feature but all I have to do is implement mod support and then enable searching for CVP files.

[The E]

Is that branch compilable currently? Running cmake yields a variety of errors, including the vfspp directory being empty.

[Fury]

I should mention some later insights I've gained into this matter since original activity of this topic.

None of the common compressed archives have true random access. There are some less known alternate versions, such as razip, dictzip and idzip. Something to look out for is also xz container format.
http://tukaani.org/xz/
http://stephane.lesimple.fr/blog/2010-07-20/lzop-vs-compress-vs-gzip-vs-bzip2-vs-lzma-vs-lzma2xz-benchmark-reloaded.html
http://tukaani.org/xz/xz-javadoc/org/tukaani/xz/SeekableXZInputStream.html

Unfortunately xz is similar to gzip in that it only compresses single files. Meaning it needs a container to store all the individually compressed files into one big file. And there really does not seem to be decent alternatives to tar, which does not support indexes. I've read that there are some various methods to add indexes to tar files, but I haven't really checked them out.

So while I did specifically mention random-access in the very first post in this topic, it should have been more like "not needing to decompress entire archive to decompress one file". Decompressing tool still needs to read contents of the entire archive to check what is where, and that can add significant delay before it can start decompressing the files that are needed. This is where indexes would come handy, as then the tool would only need to read indexes and it knows what is where. Zip-files do have indexes albeit they are at end of the zip-file rather than in the beginning, but at least it does have indexes.

So what are the options?
- Use LZMA2 archives and live with the delay that comes from reading contents of the archive prior to decompressing.
- Use Zip-files.
- Use razip, dictzip or idzip.
- Compress individual files into Xz-archives (see the links above) with LZMA2 and then package them into an uncompressed Zip-archive. Should be fairly trivial to create a tool just to handle this process without repeated manual labor.
- Look into making tar viable by adding indexes.
- Look into tar alternatives like xar. https://code.google.com/p/xar/wiki/whyxar
- And now the bomb, think outside of the box and use Matroska as the container. It has random-access and streaming, technically it can handle anything and not just audio and video. http://www.matroska.org/technical/specs/index.html This would require most work from our part though I imagine, so it's probably out.

[niffiwan]

Just to add yet another option, here's something I found that would do most of the heavy lifting for us, it'd just need vanilla VP support added to it:
https://icculus.org/physfs/
https://icculus.org/physfs/docs/html/

(of course if m!m has almost completed his current work on .cvp, I really can't see this being useful right now...)

[m!m]

Is that branch compilable currently? Running cmake yields a variety of errors, including the vfspp directory being empty.

I'm not entirely sure if the version I pushed to github is actually compiling but in any case you need to install and build boost and also initialize and update the submodules to clone the vfspp repository.
Boost: As we only need a few libraries you should be able to build it with the following command:

Code: [Select]

bootstrap && b2 --with-filesystem --with-system --with-iostreams --with-regex variant=debug,release link=static runtime-link=static define=_HAS_ITERATOR_DEBUGGING=0
VFSPP: According to the internet you should be able to clone the submodule with the following commands:

Code: [Select]

git submodule init
git submodule update

<snip>

I don't know much about how the LZMA SDK actually extract the entries but the archive index is only read once when it is initially and then the index of the file will be reused whenever it needs to be extracted. There is an overhead involved but I think that it is worth it.

Just to add yet another option, here's something I found that would do most of the heavy lifting for us, it'd just need vanilla VP support added to it:
https://icculus.org/physfs/
https://icculus.org/physfs/docs/html/

(of course if m!m has almost completed his current work on .cvp, I really can't see this being useful right now...)

I didn't find this when I was searching for an existing solution but now it's sadly too late (if I have serious issues with my library then I can still switch to this one). Thanks for finding this niffiwan

[m!m]

I now implemented 7-zip reading, I'm going to post a test build thread later but here are some test builds: http://www.mediafire.com/download/fjufhkytlv2lt79/CompressedVP.7z
Everything should behave the same but now .cvp files placed in a root directory (either FS root or a mod directory) will be read and used for resource lookup.

[Axem]

My immediate findings by 7z'ing Vassago's Dirge and renaming it .cvp.

• Loading takes a long time and my whole system will lag until the game actually starts Remaking it as a non-solid archive makes it load super fast!
• Its not loading secondary mods, it didn't load the mediavps (edited debug log here http://pastebin.com/gRXTaMKC )
• I have a mod tbm that removes the Freespace2 campaign file and points it to VassagosDirge as the default. The FS2 campaign file gets removed, but the VassagosDirge isn't even visible in the campaign menu, so I get "no valid campaign files found"

That said, this is cool stuff!

[The Dagger]

I tried a simple mod that only adds one ship and uses Mediavps2014:
1- Long loading time confirmed.
2- It does load Mediavps, 'cause I do get the Mediavps splashscreen.
3- It does open cvp files, 'cause debug says I'm mising some subsystem in my ship (ups).
4- It crashes after the loading screen with this message:

Code: [Select]

    Assert: parentDir
    File: cfile.cpp
    Line: 538
     
    ntdll.dll! ZwWaitForSingleObject + 21 bytes
    kernel32.dll! WaitForSingleObjectEx + 67 bytes
    kernel32.dll! WaitForSingleObject + 18 bytes
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    kernel32.dll! BaseThreadInitThunk + 18 bytes
    ntdll.dll! RtlInitializeExceptionChain + 99 bytes
    ntdll.dll! RtlInitializeExceptionChain + 54 bytes

fs2_open.log

Code: [Select]

==========================================================================
DEBUG SPEW: No debug_filter.cfg found, so only general, error, and warning
categories can be shown and no debug_filter.cfg info will be saved.
==========================================================================
FreeSpace 2 Open version: 3.7.1
Passed cmdline options:
  -nosound
  -spec_exp 15
  -ogl_spec 20
  -spec_static 1.5
  -spec_point 1.2
  -spec_tube 1.5
  -ambient_factor 105
  -missile_lighting
  -no_emissive_light
  -3dshockwave
  -soft_particles
  -post_process
  -bloom_intensity 20
  -fxaa
  -fxaa_preset 9
  -cache_bitmaps
  -ballistic_gauge
  -dualscanlines
  -rearm_timer
  -3dwarp
  -ship_choice_3d
  -weapon_choice_3d
  -warp_flash
  -snd_preload
  -mod mymvp,mediavps_2014
Found root C:\Freespace2\mymvp...Found root C:\Freespace2\mediavps_2014...Found root C:\Freespace2...Searching root pack 'C:\Freespace2\mymvp\Hathor.cvp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_A-Glows.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Advanced.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Assets.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_CB_ANI_1.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_CB_ANI_2.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Effects.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Music.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_RadarIcons.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Root.vp' ... Found root pack 'C:\Freespace2\Root_fs2.vp' ... Found root pack 'C:\Freespace2\smarty_fs2.vp' ... Found root pack 'C:\Freespace2\sparky_fs2.vp' ... Found root pack 'C:\Freespace2\sparky_hi_fs2.vp' ... Found root pack 'C:\Freespace2\stu_fs2.vp' ... Found root pack 'C:\Freespace2\tango1_fs2.vp' ... Found root pack 'C:\Freespace2\tango2_fs2.vp' ... Found root pack 'C:\Freespace2\tango3_fs2.vp' ... Found root pack 'C:\Freespace2\warble_fs2.vp' ... ERROR: Unknown Language Checksum: 589986744
Using default language settings...
TBM  =>  Starting parse of 'mv_root-lcl.tbm' ...
Setting language to English
TBM  =>  Starting parse of 'mv_root-lcl.tbm' ...
Game Settings Table: Using Standard Loops For SEXP Arguments
Game Settings Table: Using standard event chaining behavior
Game Settings Table: External shaders are DISABLED
Failed to init speech
Initializing OpenGL graphics device at 1024x768 with 32-bit color...
  Initializing WGL...
  Requested WGL Video values = R: 8, G: 8, B: 8, depth: 24, stencil: 8, double-buffer: 1
  Actual WGL Video values    = R: 8, G: 8, B: 8, depth: 24, stencil: 8, double-buffer: 1
  OpenGL Vendor    : NVIDIA Corporation
  OpenGL Renderer  : GeForce GTX 460M/PCIe/SSE2
  OpenGL Version   : 4.4.0

  Using extension "GL_EXT_fog_coord".
  Using extension "GL_ARB_multitexture".
  Using extension "GL_ARB_texture_env_add".
  Using extension "GL_ARB_texture_compression".
  Using extension "GL_EXT_texture_compression_s3tc".
  Using extension "GL_EXT_texture_filter_anisotropic".
  Using extension "GL_ARB_texture_env_combine".
  Using extension "GL_EXT_compiled_vertex_array".
  Using extension "GL_EXT_draw_range_elements".
  Using extension "GL_ARB_texture_mirrored_repeat".
  Using extension "GL_ARB_texture_non_power_of_two".
  Using extension "GL_ARB_vertex_buffer_object".
  Using extension "GL_ARB_pixel_buffer_object".
  Using extension "GL_SGIS_generate_mipmap".
  Using extension "GL_EXT_framebuffer_object".
  Using extension "GL_ARB_texture_rectangle".
  Using extension "GL_EXT_bgra".
  Using extension "GL_ARB_texture_cube_map".
  Using extension "GL_EXT_texture_lod_bias".
  Using extension "GL_ARB_point_sprite".
  Using extension "GL_ARB_shading_language_100".
  Using extension "GL_ARB_shader_objects".
  Using extension "GL_ARB_vertex_shader".
  Using extension "GL_ARB_fragment_shader".
  Using extension "GL_ARB_shader_texture_lod".
  Using extension "GL_ARB_texture_float".
  Using extension "GL_ARB_draw_elements_base_vertex".
  Found special extension function "wglSwapIntervalEXT".

Compiling new shader:
   Loading built-in default shader for: soft-v.sdr
   Loading built-in default shader for: soft-f.sdr
Shader features:
   Depth-blended Particles
Compiling new shader:
   Loading built-in default shader for: soft-v.sdr
   Loading built-in default shader for: soft-f.sdr
Shader features:
   Distorted Particles

  Compiling post-processing shader 1 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: post-f.sdr
  Compiling post-processing shader 2 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: blur-f.sdr
  Compiling post-processing shader 3 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: blur-f.sdr
  Compiling post-processing shader 4 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: brightpass-f.sdr
  Compiling post-processing shader 5 ...
   Loading built-in default shader for: fxaa-v.sdr
   Loading built-in default shader for: fxaa-f.sdr
  Compiling post-processing shader 6 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: fxaapre-f.sdr
  Compiling post-processing shader 7 ...
   Loading built-in default shader for: post-v.sdr
   Loading built-in default shader for: ls-f.sdr

  Max texture units: 4 (32)
  Max elements vertices: 1048576
  Max elements indices: 1048576
  Max texture size: 16384x16384
  Max render buffer size: 16384x16384
  Can use compressed textures: YES
  Texture compression available: YES
  Post-processing enabled: YES
  Using trilinear texture filter.
  OpenGL Shader Version: 4.40 NVIDIA via Cg compiler
... OpenGL init is complete!
Size of bitmap info = 742 KB
Size of bitmap extra info = 48 bytes
ANI cursorweb with size 24x24 (25.0% wasted)
GRAPHICS: Initializing default colors...
SCRIPTING: Beginning initialization sequence...
SCRIPTING: Beginning Lua initialization...
LUA: Opening LUA state...
LUA: Initializing base Lua libraries...
LUA: Beginning ADE initialization
ADE: Initializing enumeration constants...
ADE: Assigning Lua session...
SCRIPTING: Beginning main hook parse sequence....
Wokka!  Error opening file (scripting.tbl)!
TABLES: Unable to parse 'scripting.tbl'!  Error code = 5.
TBM  =>  Starting parse of 'mv_dbrs-sct.tbm' ...
TBM  =>  Starting parse of 'mv_exp-sct.tbm' ...
TBM  =>  Starting parse of 'mv_flak-sct.tbm' ...
SCRIPTING: Inititialization complete.
SCRIPTING: Splash screen overrides checked
SCRIPTING: Splash hook has been run
SCRIPTING: Splash screen conditional hook has been run
Using high memory settings...
Wokka!  Error opening file (interface.tbl)!
WMCGUI: Unable to parse 'interface.tbl'!  Error code = 5.
TBM  =>  Starting parse of 'mv_effects-sdf.tbm' ...
Dutifully ignoring the extra sound values for retail sound 36, 'l_hit.wav'...
Dutifully ignoring the extra sound values for retail sound 37, 'm_hit.wav'...
Windows reported 16 joysticks, we found 0
TABLES => Unable to find 'colors.tbl'. Initialising colors with default values.
TBM  =>  Starting parse of 'mv_effects-mfl.tbm' ...
Wokka!  Error opening file (armor.tbl)!
TABLES: Unable to parse 'armor.tbl'!  Error code = 5.
TBM  =>  Starting parse of 'mv_effects-amr.tbm' ...
TBM  =>  Starting parse of 'mv_assets-aip.tbm' ...
TBM  =>  Starting parse of 'mv_effects-wxp.tbm' ...
TBM  =>  Starting parse of 'mv_root-wxp.tbm' ...
BMPMAN: Found EFF (exp20.eff) with 75 frames at 20 fps.
BMPMAN: Found EFF (ExpMissileHit1.eff) with 92 frames at 30 fps.
BMPMAN: Found EFF (noeffect.eff) with 1 frames at 1 fps.
BMPMAN: Found EFF (exp04.eff) with 49 frames at 22 fps.
BMPMAN: Found EFF (exp05.eff) with 93 frames at 20 fps.
BMPMAN: Found EFF (exp06.eff) with 92 frames at 22 fps.
BMPMAN: Found EFF (capflash.eff) with 40 frames at 10 fps.
BMPMAN: Found EFF (Maxim_Impact.eff) with 23 frames at 30 fps.
ANI Lamprey_Impact with size 80x80 (37.5% wasted)
TBM  =>  Starting parse of 'mv_assets-wep.tbm' ...
TBM  =>  Starting parse of 'mv_effects-wep.tbm' ...
TBM  =>  Starting parse of 'mv_root-wep.tbm' ...
TBM  =>  Starting parse of 'mv_effects-obt.tbm' ...
TBM  =>  Starting parse of 'Hathor-shp.tbm' ...
TBM  =>  Starting parse of 'mv_assets-shp.tbm' ...
TBM  =>  Starting parse of 'mv_effects-shp.tbm' ...
TBM  =>  Starting parse of 'mv_root-shp.tbm' ...
TBM  =>  Starting parse of 'radar-shp.tbm' ...
TBM  =>  Starting parse of 'mv_root-hdg.tbm' ...
ANI support1 with size 108x24 (25.0% wasted)
ANI damage1 with size 148x25 (21.9% wasted)
ANI wingman1 with size 71x53 (17.2% wasted)
ANI wingman2 with size 35x53 (17.2% wasted)
ANI wingman3 with size 14x53 (17.2% wasted)
ANI toggle1 with size 57x20 (37.5% wasted)
ANI head1 with size 164x132 (48.4% wasted)
ANI weapons1 with size 126x20 (37.5% wasted)
ANI weapons1_b with size 150x20 (37.5% wasted)
ANI objective1 with size 149x21 (34.4% wasted)
ANI netlag1 with size 29x30 (6.3% wasted)
ANI targhit1 with size 31x21 (34.4% wasted)
ANI time1 with size 47x23 (28.1% wasted)
ANI targetview1 with size 137x156 (39.1% wasted)
ANI targetview2 with size 4x96 (25.0% wasted)
ANI targetview3 with size 7x20 (37.5% wasted)
ANI 2_energy2 with size 86x96 (25.0% wasted)
ANI 2_reticle1 with size 40x24 (25.0% wasted)
ANI 2_leftarc with size 103x252 (1.6% wasted)
ANI 2_rightarc1 with size 103x252 (1.6% wasted)
ANI 2_toparc2 with size 35x24 (25.0% wasted)
ANI 2_toparc3 with size 41x29 (9.4% wasted)
ANI 2_lead1 with size 26x26 (18.8% wasted)
ANI 2_lock1 with size 56x53 (17.2% wasted)
ANI 2_lockspin with size 100x100 (21.9% wasted)
ANI energy1 with size 12x41 (35.9% wasted)
ANI 2_radar1 with size 209x170 (33.6% wasted)
TBM  =>  Starting parse of 'mv_effects-str.tbm' ...
loading animated cursor "cursor"
ANI cursor with size 24x24 (25.0% wasted)
MediaVPs: Flaming debris script loaded!
MediaVPs: Explosions script loaded!
ASSERTION: "parentDir" at cfile.cpp:538
Int3(): From h:\code\github\fs2open.github.com\code\globalincs\windebug.cpp at line 966

EDIT: error message.
EDIT2: added log.

[The E]

That message has nothing to do with cvp support, it should appear with normal executables too (since it's a warning about a weapons.tbl entry)

[Fury]

I am under the impression that m|m implemented LZMA SDK, in which case it should also support regular zip files. Can you folks try zips as well? Mostly curious if it reduces or increases loading times.

[The Dagger]

That message has nothing to do with cvp support, it should appear with normal executables too (since it's a warning about a weapons.tbl entry)

Yeah, just saw that after posting. Corrected now.

[Fury]

Slow loading ended up being because people used solid blocks in their 7z-archives. There are two key settings you need to change when compressing 7z-archives for this purpose.
Method: LZMA2 (default is LZMA)
Solid-block: Non-solid (default is 2 GB)

[m!m]

We will need to create a wiki article that covers this stuff.

[m!m]

I tried a simple mod that only adds one ship and uses Mediavps2014:
1- Long loading time confirmed.
2- It does load Mediavps, 'cause I do get the Mediavps splashscreen.
3- It does open cvp files, 'cause debug says I'm mising some subsystem in my ship (ups).
4- It crashes after the loading screen with this message:

Code: [Select]

    Assert: parentDir
    File: cfile.cpp
    Line: 538
     
    ntdll.dll! ZwWaitForSingleObject + 21 bytes
    kernel32.dll! WaitForSingleObjectEx + 67 bytes
    kernel32.dll! WaitForSingleObject + 18 bytes
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
    kernel32.dll! BaseThreadInitThunk + 18 bytes
    ntdll.dll! RtlInitializeExceptionChain + 99 bytes
    ntdll.dll! RtlInitializeExceptionChain + 54 bytes

fs2_open.log

Code: [Select]

<snip>

EDIT: error message.
EDIT2: added log.

Yeah, that assert was not necessary, I removed it and replaced it with a proper if but I'll wait until I upload new builds in case I can resolve more issues.

[m!m]

New builds: http://www.mediafire.com/download/fjufhkytlv2lt79/CompressedVP.7z
This version currently has issues with symlinks so keep that in mind when testing these builds.