Uncompressed 2.54 GB
Deflate Fastest 1.08 GB
Deflate Normal 1.02 GB
Deflate Ultra 1.01 GB
BZip2 is so slow that it's not even worth testing, compression ratio also less than that of PPMd and LZMA but better than deflate.PPMd Fastest 928 MB
PPMd Normal 860 MB
PPMd Ultra 840 MB
LZMA2 Fastest 990 MB
LZMA2 Normal 844 MB
LZMA2 Ultra 837 MB
LZMA2 Fastest 997 MB
LZMA2 Normal 887 MB
LZMA2 Ultra 885 MB
Deflate Fastest 2 MB
Deflate Normal 2 MB
Deflate Ultra 2 MB
PPMd Fastest 3 MB
PPMd Normal 18 MB
PPMd Ultra 130 MB
LZMA2 Fastest 3 MB
LZMA2 Normal 18 MB
LZMA2 Ultra 66 MB
Deflate Fastest 2:16 min
Deflate Normal 2:16 min
Deflate Ultra 2:17 min
PPMd Fastest 8:53 min
PPMd Normal 11:43 min
PPMd Ultra 18:46 min
LZMA2 Fastest 2:08 min
LZMA2 Normal 2:10 min
LZMA2 Ultra 2:16 min
Uncompressed 2:04 min
this is probably a stupid question but how does opening several archives affect performanceI'm not quite sure what you mean. It's not any different from opening multiple files of any kind. Of course application used and file format does play a large role. But in that area 7z and LZMA have been designed to have small overhead. I opened MV_Assets.vp in VPView and then compared to opening MV_Assets.7z (LZMA2) in 7-Zip. VPView used approximately twice as much memory as 7-Zip did.
I only worry with STARTING the extension with .vp and having the engine through some accident not looking at whether there is anything following the p and trying to load it like a regular VP. And failing. Or failing to load regular .vp files properly.Don't worry about that - VPs and 7z have different magic numbers at the top of the file, and even the first byte is different so no chance of confusing them even if they have exactly the same file extension.
Solid archives
A solid archive is a RAR archive packed by a special compression method, which treats all files, within the archive, as one continuous data stream. Solid archives are supported only by the RAR archiving format, ZIP archives are always non-solid. The archiving method for RAR archives is a user selectable option and may be Solid or non-Solid.
Solid archiving significantly increases compression, especially when adding a large number of small, similar files. But it also has a few important disadvantages:Solid archiving is preferable if:
- slower updating of existing solid archives;
- to extract a single file from a solid archive all preceding files must be analyzed. This makes extraction from the middle of a solid archive slower than extraction from a normal archive, but, if all files are to be extracted from a solid archive, the extraction speed will not be affected.
- if any file in a solid archive is damaged, it will be impossible to extract all files which follow the damaged area. Thus if a solid archive is stored to media such as diskette, it is recommended to make use of the recovery record.
- the archive is updated rarely;
- it is not necessary to frequently extract a single file or only part of the files from the archive;
- compression ratio is more important than compression speed.
/me likes this idea.
Wasn't this brought up before? I forget.
/me likes this idea.Taylor, and then later myself, did have a look at this concept. There was some proof of concept code floating around, but it never got to a point at which it was widely circulated on these forums or IRC.
Wasn't this brought up before? I forget.
I'm not sure what the consensus is, but solid archives are no-go.You may want to re-read the first post in this topic, as it states the very same thing and makes it a point that archives need to be non-solid to work for random access.
Great BUMP for Justice.
Another reason why we need this: CF_MAX_FILENAME is maxed at 32. This apparently cannot be bumped because the VP File Format structure was only written to support 32. We can't raise the limitation in the engine, because a VP file still won't store at greater than 32 anyway.
So, the only way to have filenames longer than 32 characters (or any other values longer than 32 characters) will be in the implementation of this "Compressed" VP (or CVP) format.
And of course, we'd still need to make sure that any bumpage only affects when loading data from a CVP to boot.
We might not want users cracking open VP files easily. Right now if someone wants to open a VP file they need to figure out which tool to use (which gives us a chance to warn them not to replace anything in them).
You can easily replace files in a .7z file. They wouldn't come to us until they'd already ****ed their install over. Even if we can make our files usable in 7zip we should still name them .vp just to avoid that.
I'm sure that any user knows that if you touch the game's files, probably they will **** up the game ;). Don't take people to be that stupid. For what is worth, I think going towards more general standards is better, in long term.
I'm sure that any user knows that if you touch the game's files, probably they will **** up the game ;). Don't take people to be that stupid. For what is worth, I think going towards more general standards is better, in long term.
Go look at the support board and tell me if you would still assume that people know what they are doing...
This means deprecation of .vp by all modders, which isn't that much of a trouble.
I'm sure that any user knows that if you touch the game's files, probably they will **** up the game ;). Don't take people to be that stupid.
/me likes this idea.Taylor, and then later myself, did have a look at this concept. There was some proof of concept code floating around, but it never got to a point at which it was widely circulated on these forums or IRC.
Wasn't this brought up before? I forget.
The thinking done ahead of the coding then arrived at largely the same conclusions as here (non-solid, algorithm akin to LZMA) if I remember correctly.
ppl try opening files with compression programs first, so it wont matter.I think you misinterpreted the intent of using a different extension. Its not to prevent people from fiddling with the VPs (the one that Goober mentioned actually hex edited the vp!) but just discourage them.
My best bet is using password protection and no tell anyone about it. So in this way you can only make the CVP with a upgraded vpmage tool, witch it automatically sets the password to the file, and again only open them with upgraded vpview. Game code offcourse will known the password.
And if you want to add security, you can make the game code to have a 10 word dictionary and automatically change the password to another after loaded the .cvp once.
Yes so would we, but in the big picture this is a rather low priority./me likes this idea.Taylor, and then later myself, did have a look at this concept. There was some proof of concept code floating around, but it never got to a point at which it was widely circulated on these forums or IRC.
Wasn't this brought up before? I forget.
The thinking done ahead of the coding then arrived at largely the same conclusions as here (non-solid, algorithm akin to LZMA) if I remember correctly.
I remember Taylor's posts on the matter and, in fact, I have been willing to see the idea pass the planning stage for quite some time. By "compressed VPs", however, I thought Taylor meant a new file format, like "Compressed Volition Package" or something like that. I must have misinterpreted his posts, then.
This means deprecation of .vp by all modders, which isn't that much of a trouble.
It means no such thing. On the contrary, it means that we'd support two different .vp formats. This isn't really a big deal, since we already support multiple image and audio formats.
You would think that, wouldn't you? Unfortunately, experience has shown us that people are that stupid. There have been threads on this very forum where people have edited the VP file directly, despite being told in no uncertain terms not to do so.
I hereby take responsibility for stupidly editing this file even though I was told not to. I will not complain when the game breaks.
I'll point out that while I was for making sure we use a different extension in order to make it less obvious that you just use 7zip or whatever I'm not in favour of making it any harder than that to edit a VP file.Other thing is we don't want to rummage through random zip or 7z files that a user may have left in the Freespace folder.
Some people seem to have taken the ball I threw past the endzone and run out into traffic with it. :p
Wasn't the suggestion for it to essentially be a 7z file but with a .cvp extension? Or is there some reason that isn't an option?No, that is exactly the original suggestion.
Wasn't the suggestion for it to essentially be a 7z file but with a .cvp extension? Or is there some reason that isn't an option?No, that is exactly the original suggestion.
Replace the extencion... i think its mandatory... some users may misunderstand what to do with a file that is linked to its compression program by his registry, thus showing the compress file icon, thus mistakenly uncompressing it.Another very valid reason for doing that.
Additionally i think the launcher should register the .vp and the .cvp extencions to show up a volition icon.What about FSO's icon? FSO's icon in a vice?
BTW, adding support for password protected files is out of the question? personally, i dont care, but it seems some people are worried about. not a lot of protection just support for unprotected files and a fixed password... like "scpfso" or something.Yes, it is out of the question. The point of .vps and .cvps is not to protect the game data but speed game loading by allowing the engine to only have to open a few files rather than thousands. Even on a modern system, a single process (like FSO) holding open thousands of files is a very fast way to cause performance problems.
Sorry, I am not sure why I started with "No" (probably in response to the second question), but nevertheless I meant to agree with you. So,Wasn't the suggestion for it to essentially be a 7z file but with a .cvp extension? Or is there some reason that isn't an option?No, that is exactly the original suggestion.
Sorry, I meant to ask "wasn't the original suggestion..."
.CFP - Compressed FreeSpace Package :)
A word of caution about this,
If this happens... whatever format you would like to replace .vps with... please PLEASE make it easy, even for the less technical among us, to understand and work with.
One of the biggest obstacles to getting into modding (at least for me) is file management. The easier it is, the more people are going to want to play.
So if you think you can shave off a few megs by adding some kind of crazy javascript compression software on everyone's computer (random 'sacrifice of ease for complexity' example) . I advise against it!
Formats like winzip, .7z or .rar are nice ideas since they are more widely used than the freespace community, and it is likely people will have had some familiarity with them before getting into modding here.
Lastly, just to be clear, making things as easy as possible gets my vote even if that means it has less favorable compression.
What the heck is with the 'who can read' argument?!Hey, he's right. If you had even read the first post,
I think you understand my general point. Let's be friends.
I can't say that this isn't going to change a little bit before it actually hits CVS, but having played around with it for months last year, I think it's pretty solid. It's largely the same as the current VP stuff, I wanted to keep it as close as possible to simplify the code.
(All values are little-endian.)
Header:Sint8[4] -- header id, "CPVP" for a CVP
Sint32 -- version #, initial version number for CVP is 1, and it needs to respect this (unlike the current standard VPs)
Sint32 -- offset to the file index (at end of VP/CVP)
Sint32 -- total uncompressed size of archive
Sint32 -- number of files in the archive
Uint32 -- CRC data (NOTE: this isn't a standard CRC32 value, FS2 has a code bug that we can't fix)
File Index (for each directory/file):Sint32 -- archive offset of current file (NOTE: this is the real offset, to compressed data, not to uncompressed data)
Sint32 -- uncompressed size of the current file
Sint32 -- compressed size of the current file
Sint8[32] -- file/directory name
Sint32 -- date/time of last file modification
Each individual file is compressed separately, NOT the entire archive. The file index is also compressed, using level 9 compression. Individual files can use any compression level so long as it's not above 6 (lower levels == faster, but less, compression). Level 6 is recommended as the standard value though as it offers the best compromise between speed, size and memory usage for decompression. The following file types should NOT be compressed: WAV, OGG, MVE. The reason is that those files are often streamed and streaming compressed files is extrememly slow (so much so that I'm not even going to code in support to do it). Also, OGG files don't compress worth a crap anyway.
The only thing guaranteed to be compressed in a CVP is the file index. The exclusion types listed above should never compressed. Beyond that it should be user choice whether to actually compress any other data or not.
Thanks. :)Sint32 -- total uncompressed size of archive
I assume this takes into account the finished size of the archive? (eg not just the files that make it up, but including the header and index as well) Seems like a dumb question but I want to be totally sure we mean the same thing by archive.Uint32 -- CRC data (NOTE: this isn't a standard CRC32 value, FS2 has a code bug that we can't fix)
Dare I ask further? Since this is a new format I don't see why a standard value isn't possible, even if fs2_open has messed up functions right now. I guess if it's being used for network stuff in the future...but that would necessitate a change to the network code, anyway.Sint32 -- uncompressed size of the current file
Sint32 -- compressed size of the current file
Does uncompressed==compressed for all noncompressed files? (I'm looking for a better/more reliable way to determine compression status than file extensions).
I assume this takes into account the finished size of the archive? (eg not just the files that make it up, but including the header and index as well) Seems like a dumb question but I want to be totally sure we mean the same thing by archive.No, it's just the data. Header offset and index size are not included. It's more for informational purposes than a function of use for the game. The primary purpose for this is for utilities, not the game itself, so that you can quickly tell how much HD space would be required if you wanted to extract the entire CVP. It's generally computed by the archiver anyway, for basic info on overall compression success, and simply storing it in the CVP header makes things easier for extraction than manually computing the total size.QuoteDare I ask further? Since this is a new format I don't see why a standard value isn't possible, even if fs2_open has messed up functions right now. I guess if it's being used for network stuff in the future...but that would necessitate a change to the network code, anyway.It's used by the current/retail networking code, so if we change it then it breaks compatibility will all previous versions. It's not really a big deal though and you can quite easily code up your own CRC code which matches what the game uses now (it's basically just missing a XOR on the crc value). Though, now that I think about it, I'm not 100% sure that the CRC table is computed properly either. But, again, it's just minor stuff overall since there are multiple incarnations of CRC32 anyway.QuoteDoes uncompressed==compressed for all noncompressed files? (I'm looking for a better/more reliable way to determine compression status than file extensions).Yes. When writing an uncompressed file you do it just like with a normal VP, and only go through the bzip2 calls for compressed content.
Oh, for anyone else that is interested, here are the docs for the bzip2 API: http://www.bzip.org/1.0.3/html/index.html
Actually, one thing that I think might come in handy would be if the signed integers for position (both TOC and file data position) were changed to unsigned integers or, better yet, 64-bit integers. This would effectively eliminate a possible future problem of VPs growing too big for signed integer positions. The BTRL Demo release is, IIRC, 650-some MB, so technology is starting to get to the point where breaking the 2GB limit is a possibility.
I was actually going to originally move the 64-bit for the size and date issues, but then I decided that I just don't care. OS support wise, it's not really worth the hassle to me. Compression will help keep it under 2gig a bit longer, and it's not like it's that hard to just split a large VP into two smaller ones. I just think that the coding effort required to do 64-bit size support properly greatly outweighs the general ability to avoid requiring it in the first place.
Oh, and it might be this weekend before I get you that test CVP. I got busy doing the BtRL installers, and keeping the server running through the digg hit, and just haven't had a chance to get anything ready for you yet. I'll try and take care of it Friday though, if possible.
My next VP tool release will support 64-bit stuff internally (it will of course check before it sticks it in the C/VP) so it shouldn't be too much of an issue when you do change it. I've just been eyeing that limit warily since I started work on the reading/writing backend. ;)
Crap. I totally forgot about this. :rolleyes:
I should have time to send you the file tomorrow, assuming that I actually get normal mapping 100% by then (and if I haven't killed myself in the process :mad:). I don't think it would be any easier for me to just see the output from you writer though, since I still haven't updated my VP extrator code for the final CVP spec, only the archiver.
7zip is open source. If you had to add something, you could modify .7z behavior. It just wouldn't open with 7zip. Unless you added support with a plugin.
Well, reading the thread should have given you an idea of what we are thinking of in terms of archive support, and that your suggestion was already discarded a few pages back.
Taylor's posts contain a few good points that haven't been addressed yet. For example, he recommends that each file be compressed individually, instead of the whole thing as a solid block, because certain file types don't compress well (often because they're already compressed).Which is what a non-solid 7z archive is...
Another interesting point is that he says streaming compressed data is very slow, which would be a significant problem.So don't compress the archive that you put the streaming media in (.7z or just a .vp).
Have you read the thread? Unlike you or taylor, we are not comfortable with the idea of using a proprietary archive format.
Yeah I'd imagine that was probably one of the reasons it was never implemented - no one had the time to sit down and write/maintain the code for the new format. Of course, no one's sitting down to get LZMA added in yet either, but maybe we can work on that shortly after 3.6.14.
Is that branch compilable currently? Running cmake yields a variety of errors, including the vfspp directory being empty.I'm not entirely sure if the version I pushed to github is actually compiling but in any case you need to install and build boost and also initialize and update the submodules to clone the vfspp repository.
bootstrap && b2 --with-filesystem --with-system --with-iostreams --with-regex variant=debug,release link=static runtime-link=static define=_HAS_ITERATOR_DEBUGGING=0git submodule init
git submodule update<snip>I don't know much about how the LZMA SDK actually extract the entries but the archive index is only read once when it is initially and then the index of the file will be reused whenever it needs to be extracted. There is an overhead involved but I think that it is worth it.
Just to add yet another option, here's something I found that would do most of the heavy lifting for us, it'd just need vanilla VP support added to it::banghead: I didn't find this when I was searching for an existing solution but now it's sadly too late (if I have serious issues with my library then I can still switch to this one). Thanks for finding this niffiwan :)
https://icculus.org/physfs/
https://icculus.org/physfs/docs/html/
(of course if m!m has almost completed his current work on .cvp, I really can't see this being useful right now...)
Assert: parentDir
File: cfile.cpp
Line: 538
ntdll.dll! ZwWaitForSingleObject + 21 bytes
kernel32.dll! WaitForSingleObjectEx + 67 bytes
kernel32.dll! WaitForSingleObject + 18 bytes
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
kernel32.dll! BaseThreadInitThunk + 18 bytes
ntdll.dll! RtlInitializeExceptionChain + 99 bytes
ntdll.dll! RtlInitializeExceptionChain + 54 bytes
==========================================================================
DEBUG SPEW: No debug_filter.cfg found, so only general, error, and warning
categories can be shown and no debug_filter.cfg info will be saved.
==========================================================================
FreeSpace 2 Open version: 3.7.1
Passed cmdline options:
-nosound
-spec_exp 15
-ogl_spec 20
-spec_static 1.5
-spec_point 1.2
-spec_tube 1.5
-ambient_factor 105
-missile_lighting
-no_emissive_light
-3dshockwave
-soft_particles
-post_process
-bloom_intensity 20
-fxaa
-fxaa_preset 9
-cache_bitmaps
-ballistic_gauge
-dualscanlines
-rearm_timer
-3dwarp
-ship_choice_3d
-weapon_choice_3d
-warp_flash
-snd_preload
-mod mymvp,mediavps_2014
Found root C:\Freespace2\mymvp...Found root C:\Freespace2\mediavps_2014...Found root C:\Freespace2...Searching root pack 'C:\Freespace2\mymvp\Hathor.cvp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_A-Glows.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Advanced.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Assets.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_CB_ANI_1.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_CB_ANI_2.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Effects.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Music.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_RadarIcons.vp' ... Found root pack 'C:\Freespace2\mediavps_2014\MV_Root.vp' ... Found root pack 'C:\Freespace2\Root_fs2.vp' ... Found root pack 'C:\Freespace2\smarty_fs2.vp' ... Found root pack 'C:\Freespace2\sparky_fs2.vp' ... Found root pack 'C:\Freespace2\sparky_hi_fs2.vp' ... Found root pack 'C:\Freespace2\stu_fs2.vp' ... Found root pack 'C:\Freespace2\tango1_fs2.vp' ... Found root pack 'C:\Freespace2\tango2_fs2.vp' ... Found root pack 'C:\Freespace2\tango3_fs2.vp' ... Found root pack 'C:\Freespace2\warble_fs2.vp' ... ERROR: Unknown Language Checksum: 589986744
Using default language settings...
TBM => Starting parse of 'mv_root-lcl.tbm' ...
Setting language to English
TBM => Starting parse of 'mv_root-lcl.tbm' ...
Game Settings Table: Using Standard Loops For SEXP Arguments
Game Settings Table: Using standard event chaining behavior
Game Settings Table: External shaders are DISABLED
Failed to init speech
Initializing OpenGL graphics device at 1024x768 with 32-bit color...
Initializing WGL...
Requested WGL Video values = R: 8, G: 8, B: 8, depth: 24, stencil: 8, double-buffer: 1
Actual WGL Video values = R: 8, G: 8, B: 8, depth: 24, stencil: 8, double-buffer: 1
OpenGL Vendor : NVIDIA Corporation
OpenGL Renderer : GeForce GTX 460M/PCIe/SSE2
OpenGL Version : 4.4.0
Using extension "GL_EXT_fog_coord".
Using extension "GL_ARB_multitexture".
Using extension "GL_ARB_texture_env_add".
Using extension "GL_ARB_texture_compression".
Using extension "GL_EXT_texture_compression_s3tc".
Using extension "GL_EXT_texture_filter_anisotropic".
Using extension "GL_ARB_texture_env_combine".
Using extension "GL_EXT_compiled_vertex_array".
Using extension "GL_EXT_draw_range_elements".
Using extension "GL_ARB_texture_mirrored_repeat".
Using extension "GL_ARB_texture_non_power_of_two".
Using extension "GL_ARB_vertex_buffer_object".
Using extension "GL_ARB_pixel_buffer_object".
Using extension "GL_SGIS_generate_mipmap".
Using extension "GL_EXT_framebuffer_object".
Using extension "GL_ARB_texture_rectangle".
Using extension "GL_EXT_bgra".
Using extension "GL_ARB_texture_cube_map".
Using extension "GL_EXT_texture_lod_bias".
Using extension "GL_ARB_point_sprite".
Using extension "GL_ARB_shading_language_100".
Using extension "GL_ARB_shader_objects".
Using extension "GL_ARB_vertex_shader".
Using extension "GL_ARB_fragment_shader".
Using extension "GL_ARB_shader_texture_lod".
Using extension "GL_ARB_texture_float".
Using extension "GL_ARB_draw_elements_base_vertex".
Found special extension function "wglSwapIntervalEXT".
Compiling new shader:
Loading built-in default shader for: soft-v.sdr
Loading built-in default shader for: soft-f.sdr
Shader features:
Depth-blended Particles
Compiling new shader:
Loading built-in default shader for: soft-v.sdr
Loading built-in default shader for: soft-f.sdr
Shader features:
Distorted Particles
Compiling post-processing shader 1 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: post-f.sdr
Compiling post-processing shader 2 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: blur-f.sdr
Compiling post-processing shader 3 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: blur-f.sdr
Compiling post-processing shader 4 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: brightpass-f.sdr
Compiling post-processing shader 5 ...
Loading built-in default shader for: fxaa-v.sdr
Loading built-in default shader for: fxaa-f.sdr
Compiling post-processing shader 6 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: fxaapre-f.sdr
Compiling post-processing shader 7 ...
Loading built-in default shader for: post-v.sdr
Loading built-in default shader for: ls-f.sdr
Max texture units: 4 (32)
Max elements vertices: 1048576
Max elements indices: 1048576
Max texture size: 16384x16384
Max render buffer size: 16384x16384
Can use compressed textures: YES
Texture compression available: YES
Post-processing enabled: YES
Using trilinear texture filter.
OpenGL Shader Version: 4.40 NVIDIA via Cg compiler
... OpenGL init is complete!
Size of bitmap info = 742 KB
Size of bitmap extra info = 48 bytes
ANI cursorweb with size 24x24 (25.0% wasted)
GRAPHICS: Initializing default colors...
SCRIPTING: Beginning initialization sequence...
SCRIPTING: Beginning Lua initialization...
LUA: Opening LUA state...
LUA: Initializing base Lua libraries...
LUA: Beginning ADE initialization
ADE: Initializing enumeration constants...
ADE: Assigning Lua session...
SCRIPTING: Beginning main hook parse sequence....
Wokka! Error opening file (scripting.tbl)!
TABLES: Unable to parse 'scripting.tbl'! Error code = 5.
TBM => Starting parse of 'mv_dbrs-sct.tbm' ...
TBM => Starting parse of 'mv_exp-sct.tbm' ...
TBM => Starting parse of 'mv_flak-sct.tbm' ...
SCRIPTING: Inititialization complete.
SCRIPTING: Splash screen overrides checked
SCRIPTING: Splash hook has been run
SCRIPTING: Splash screen conditional hook has been run
Using high memory settings...
Wokka! Error opening file (interface.tbl)!
WMCGUI: Unable to parse 'interface.tbl'! Error code = 5.
TBM => Starting parse of 'mv_effects-sdf.tbm' ...
Dutifully ignoring the extra sound values for retail sound 36, 'l_hit.wav'...
Dutifully ignoring the extra sound values for retail sound 37, 'm_hit.wav'...
Windows reported 16 joysticks, we found 0
TABLES => Unable to find 'colors.tbl'. Initialising colors with default values.
TBM => Starting parse of 'mv_effects-mfl.tbm' ...
Wokka! Error opening file (armor.tbl)!
TABLES: Unable to parse 'armor.tbl'! Error code = 5.
TBM => Starting parse of 'mv_effects-amr.tbm' ...
TBM => Starting parse of 'mv_assets-aip.tbm' ...
TBM => Starting parse of 'mv_effects-wxp.tbm' ...
TBM => Starting parse of 'mv_root-wxp.tbm' ...
BMPMAN: Found EFF (exp20.eff) with 75 frames at 20 fps.
BMPMAN: Found EFF (ExpMissileHit1.eff) with 92 frames at 30 fps.
BMPMAN: Found EFF (noeffect.eff) with 1 frames at 1 fps.
BMPMAN: Found EFF (exp04.eff) with 49 frames at 22 fps.
BMPMAN: Found EFF (exp05.eff) with 93 frames at 20 fps.
BMPMAN: Found EFF (exp06.eff) with 92 frames at 22 fps.
BMPMAN: Found EFF (capflash.eff) with 40 frames at 10 fps.
BMPMAN: Found EFF (Maxim_Impact.eff) with 23 frames at 30 fps.
ANI Lamprey_Impact with size 80x80 (37.5% wasted)
TBM => Starting parse of 'mv_assets-wep.tbm' ...
TBM => Starting parse of 'mv_effects-wep.tbm' ...
TBM => Starting parse of 'mv_root-wep.tbm' ...
TBM => Starting parse of 'mv_effects-obt.tbm' ...
TBM => Starting parse of 'Hathor-shp.tbm' ...
TBM => Starting parse of 'mv_assets-shp.tbm' ...
TBM => Starting parse of 'mv_effects-shp.tbm' ...
TBM => Starting parse of 'mv_root-shp.tbm' ...
TBM => Starting parse of 'radar-shp.tbm' ...
TBM => Starting parse of 'mv_root-hdg.tbm' ...
ANI support1 with size 108x24 (25.0% wasted)
ANI damage1 with size 148x25 (21.9% wasted)
ANI wingman1 with size 71x53 (17.2% wasted)
ANI wingman2 with size 35x53 (17.2% wasted)
ANI wingman3 with size 14x53 (17.2% wasted)
ANI toggle1 with size 57x20 (37.5% wasted)
ANI head1 with size 164x132 (48.4% wasted)
ANI weapons1 with size 126x20 (37.5% wasted)
ANI weapons1_b with size 150x20 (37.5% wasted)
ANI objective1 with size 149x21 (34.4% wasted)
ANI netlag1 with size 29x30 (6.3% wasted)
ANI targhit1 with size 31x21 (34.4% wasted)
ANI time1 with size 47x23 (28.1% wasted)
ANI targetview1 with size 137x156 (39.1% wasted)
ANI targetview2 with size 4x96 (25.0% wasted)
ANI targetview3 with size 7x20 (37.5% wasted)
ANI 2_energy2 with size 86x96 (25.0% wasted)
ANI 2_reticle1 with size 40x24 (25.0% wasted)
ANI 2_leftarc with size 103x252 (1.6% wasted)
ANI 2_rightarc1 with size 103x252 (1.6% wasted)
ANI 2_toparc2 with size 35x24 (25.0% wasted)
ANI 2_toparc3 with size 41x29 (9.4% wasted)
ANI 2_lead1 with size 26x26 (18.8% wasted)
ANI 2_lock1 with size 56x53 (17.2% wasted)
ANI 2_lockspin with size 100x100 (21.9% wasted)
ANI energy1 with size 12x41 (35.9% wasted)
ANI 2_radar1 with size 209x170 (33.6% wasted)
TBM => Starting parse of 'mv_effects-str.tbm' ...
loading animated cursor "cursor"
ANI cursor with size 24x24 (25.0% wasted)
MediaVPs: Flaming debris script loaded!
MediaVPs: Explosions script loaded!
ASSERTION: "parentDir" at cfile.cpp:538
Int3(): From h:\code\github\fs2open.github.com\code\globalincs\windebug.cpp at line 966
That message has nothing to do with cvp support, it should appear with normal executables too (since it's a warning about a weapons.tbl entry)
I tried a simple mod that only adds one ship and uses Mediavps2014:Yeah, that assert was not necessary, I removed it and replaced it with a proper if but I'll wait until I upload new builds in case I can resolve more issues.
1- Long loading time confirmed.
2- It does load Mediavps, 'cause I do get the Mediavps splashscreen.
3- It does open cvp files, 'cause debug says I'm mising some subsystem in my ship (ups).
4- It crashes after the loading screen with this message:Code: [Select]Assert: parentDir
File: cfile.cpp
Line: 538
ntdll.dll! ZwWaitForSingleObject + 21 bytes
kernel32.dll! WaitForSingleObjectEx + 67 bytes
kernel32.dll! WaitForSingleObject + 18 bytes
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
fs2_open_3_7_1_SSE2-DEBUG_CVP.exe! <no symbol>
kernel32.dll! BaseThreadInitThunk + 18 bytes
ntdll.dll! RtlInitializeExceptionChain + 99 bytes
ntdll.dll! RtlInitializeExceptionChain + 54 bytes
fs2_open.logCode: [Select]<snip>
EDIT: error message.
EDIT2: added log.
It bothered me how it is that FSO loads snappily, yet chokes in mission loads. I decided to look at that a bit more.Thank you very much for doing these tests, I have looked at zip support and I might be able to add support for that.
So I made a 7z-archive with NO compression. Loading of Bearbaiting took 7 seconds. When running debug, it shows FSO spends most of the time loading pofs regardless of compression. This can be already reproduced in tech lab by selecting some of the higher detailed ships.
Compressed 7z-archive without cache or models directories. 32 seconds
Compressed 7z-archive without cache, models or maps directories: 9 seconds.
Compressed 7z-archive without maps only: 11 seconds.
There really was not much difference between fastest and normal compression levels. Only the "store" (uncompressed) was as fast as you'd expect from a vp-file.
I also had weird problems with cache generation, but I suspect it had nothing to do with m|m's builds. Some cache files got regenerated again and again, some were ten times the size they should be and some were 0 kb. Oh well, that got sorted out eventually and no weirdness since.
Uncompressed 7z-archives aren't going to do us much good beyond the fact that there is no need for other tools except archiver that can open 7z-files and more robust file corruption detection (vp's have none). Perhaps it would be prudent to investigate whether zip-files would fare better when compressed?
It should also be noted that 7z-archive with 2 GB solid block resulted in nearly 400 errors and missing campaign. So I guess FSO failed to read the file completely, well it's no wonder considering 7z-archives are roughly 1.5-1.8 GB large depending on compression level.
<Fury`> You want compression ratio of each individual archive, or just compression level? The latter was mentioned in the post.
<m|m> The overall size of the archive vs. the total size of the uncompressed files.
<m|m> I'm just curious if using zip files is actually worth the loading time decrease.
<Fury`> Uncompressed 4 860 299 240 bytes, 1 945 318 001 bytes using normal zip compression
<m|m> Yeah, I think it's worth it :P
<Fury`> vp-files 4 859 635 076 bytes
<Fury`> source data direcory(-ies) 4 859 402 640 bytes