[Fury]

First impressions with mediavps converted to 7z using fastest and normal compression levels, both had LZMA2 and non-solid blocks.

FSO loads very snappily to pilot screen, there really is no difference to uncompressed. But when it comes to mission loading time, the story is very different. I used Bearbaiting to time loading times.
Uncompressed: 5 seconds
Fastest LZMA2: 32 seconds
Normal LZMA2: 27 seconds
I don't know why fastest compression ended up being slower than normal. I only timed these once, so perhaps there is large variance.

[Fury]

It bothered me how it is that FSO loads snappily, yet chokes in mission loads. I decided to look at that a bit more.

So I made a 7z-archive with NO compression. Loading of Bearbaiting took 7 seconds. When running debug, it shows FSO spends most of the time loading pofs regardless of compression. This can be already reproduced in tech lab by selecting some of the higher detailed ships.

Compressed 7z-archive without cache or models directories. 32 seconds
Compressed 7z-archive without cache, models or maps directories: 9 seconds.
Compressed 7z-archive without maps only: 11 seconds.

There really was not much difference between fastest and normal compression levels. Only the "store" (uncompressed) was as fast as you'd expect from a vp-file.

I also had weird problems with cache generation, but I suspect it had nothing to do with m|m's builds. Some cache files got regenerated again and again, some were ten times the size they should be and some were 0 kb. Oh well, that got sorted out eventually and no weirdness since.

Uncompressed 7z-archives aren't going to do us much good beyond the fact that there is no need for other tools except archiver that can open 7z-files and more robust file corruption detection (vp's have none). Perhaps it would be prudent to investigate whether zip-files would fare better when compressed?

It should also be noted that 7z-archive with 2 GB solid block resulted in nearly 400 errors and missing campaign. So I guess FSO failed to read the file completely, well it's no wonder considering 7z-archives are roughly 1.5-1.8 GB large depending on compression level.

[m!m]

I need to fix the location where the cache files are generated as that is probably broken right now.

[m!m]

It bothered me how it is that FSO loads snappily, yet chokes in mission loads. I decided to look at that a bit more.

So I made a 7z-archive with NO compression. Loading of Bearbaiting took 7 seconds. When running debug, it shows FSO spends most of the time loading pofs regardless of compression. This can be already reproduced in tech lab by selecting some of the higher detailed ships.

Compressed 7z-archive without cache or models directories. 32 seconds
Compressed 7z-archive without cache, models or maps directories: 9 seconds.
Compressed 7z-archive without maps only: 11 seconds.

There really was not much difference between fastest and normal compression levels. Only the "store" (uncompressed) was as fast as you'd expect from a vp-file.

I also had weird problems with cache generation, but I suspect it had nothing to do with m|m's builds. Some cache files got regenerated again and again, some were ten times the size they should be and some were 0 kb. Oh well, that got sorted out eventually and no weirdness since.

Uncompressed 7z-archives aren't going to do us much good beyond the fact that there is no need for other tools except archiver that can open 7z-files and more robust file corruption detection (vp's have none). Perhaps it would be prudent to investigate whether zip-files would fare better when compressed?

It should also be noted that 7z-archive with 2 GB solid block resulted in nearly 400 errors and missing campaign. So I guess FSO failed to read the file completely, well it's no wonder considering 7z-archives are roughly 1.5-1.8 GB large depending on compression level.

Thank you very much for doing these tests, I have looked at zip support and I might be able to add support for that.
For now I fixed the issues with symlinks (hopefully) so that should work fine now and I also renamed the file extension to .vp7 so we can add support for other archive formats later. Make sure to rename your .cvp files before testing
Here are the new builds: http://www.mediafire.com/download/jxqycidbt54dxq7/CompressedVP.7z

[Luis Dias]

Wait, these times are absolutely discouraging, if the idea to sacrifice 30 seconds every time we load a mission so that we can get some gigabytes back that we don't even (mostly) need anyway...

[The E]

What these tests tell me is that there is some content that reacts neutrally to being stored in a compressed container, and other content that doesn't. It just means that this is a new tech that we will have to get used to before we can really use it well.

[m!m]

I have added support for using zip files which should have the extension .vpz. The files are currently decompressed entirely when opened to keep the implementation simple. If this causes issues then it might be possible to implement on-demand decompression.
New builds here: http://www.mediafire.com/download/jxqycidbt54dxq7/CompressedVP.7z

EDIT: By files are decompressed completely I mean only the files that are opened by FSO inside the archive are decompressed.

[Fury]

Tested zip, loading Bearbaiting took about 13 seconds with fastest, normal and maximum compression levels. Compression level had no real impact on mission load speed. However, when using uncompressed zip, FSO wouldn't load. Process starts, system becomes very unresponsive for a little while but FSO never loads up to the pilot screen, or I ran out of patience first.

I also ran these tests on an SSD and observed no difference in loading speed compared to HDD. I should however note that I ran 7z and zip tests against source data directory, not a vp-file. That would be because I do not currently have anything that can package a vp-file and too lazy to find and get one. While doing these tests I noted that source data directory would nearly always load up in roughly 5 seconds, but at times it took about as long as zip did. This happened on both SSD and HDD.

Edit: This should be a good opportunity to profile the loading process and optimize it in the hopes of speeding up loading. If we can shave off some seconds off the zip loading, we have pretty decent potential replacement to vp.

[m!m]

I just tested uncompressed zip files and could not reproduce your issues. FSO starts fine so it might be related to your data. Which data did you test it with?
Also if you need a VP editor and you are on Windows Vista or later you can use this: http://www.hard-light.net/forums/index.php?topic=83920.0

[Fury]

I suspect the problem is with file size. Original zip specification limits zip-file size to 4 GB. Support for higher file size limits depends on the library and I guess the library you used doesn't support zip files over 4 GB. The uncompressed zip file I used is 4.52 GB which contains most of the relevant FSU SVN data. Normal zip compression gets it to 1.81 GB.

As for packaging vp-files, I usually prefer command-line cfilearchiver.

[m!m]

With an archive that's bigger than 4GB you should probably break that up into multiple files just like you would do with a VP (which also has this 32-bit limitation).
A recent version of the commandline tools are also available here: http://scp.indiegames.us/builds/cfile.zip

[Fury]

It's not like I plan to distribute 4.5 GB uncompressed zip files.

[niffiwan]

It'd be nice to have FSO spit an error out if it can't process a 4GB+ file, rather than hanging.

(/captain obvious)

[m!m]

I guess it's because I don't use the fopen64 variant to open the files, I'll see what I can do about it, maybe I can even get it to accept wide character strings...

[Fury]

Did further tests on multiple smaller archives. Data directory was split into 8 zip archives using normal compression level. Bearbaiting loaded up in 12 seconds. Next I tried how long it takes when using 8 uncompressed zip archives, Bearbaiting loaded in 8 seconds.

I decided to see how does multiple smaller 7z archives compare, compressed using fastest compression level but with non-solid blocks. Bearbaiting too 36 seconds to load. With multiple uncompressed 7z archives, it took 8 seconds to load Bearbaiting.

And finally, I created vp-files from the same data directory as all the 7z and zip archives so far. Because maps directory was too large for one vp-file to hold (vp-file begins to crap out after it gets past 2GB in size), I split it into three. 10 vp-files in total. Bearbaiting loaded up in 5-6 seconds.

[m!m]

Double the load time seems reasonable, what's the compression ratio for your first test?

[Fury]

This was replied to over at #scp, but copying for reference.

Code: [Select]

<Fury`> You want compression ratio of each individual archive, or just compression level? The latter was mentioned in the post.
<m|m> The overall size of the archive vs. the total size of the uncompressed files.
<m|m> I'm just curious if using zip files is actually worth the loading time decrease.
<Fury`> Uncompressed 4 860 299 240 bytes, 1 945 318 001 bytes using normal zip compression
<m|m> Yeah, I think it's worth it :P
<Fury`> vp-files 4 859 635 076 bytes
<Fury`> source data direcory(-ies) 4 859 402 640 bytes

Which basically means normal zip-compression achieves ~60% compression ratio over vp-files. This is pretty darn good, but it comes at the expense of doubling loading times, in the case of mediavps increase is from 6 seconds to 12 seconds. Even uncompressed archives seem to add 33% increase (2 seconds) in loading speed.

Considering how easily vp-files can become corrupted without end-user being none the wiser about it, I believe either uncompressed or compressed zip-archives should replace vp-files in future mod releases. Unless of course further testing reveals other problems.

Loading times will eventually decrease as coders get around to optimize it. But depending on how big a task that is, it may take a good while. Until then it's a debate between using uncompressed and compressed zip-archives.

[m!m]

Just to keep everyone updated, I just addressed a few issues Fury reported which means that if an archive fails to load it will be a fatal error which is shown to the user and I also added a few checks to the VP reader to make sure it catches VP files which are too big for the loader.

[jr2]

Is .vp crapping out at 2GB a problem with the creator, archive type, or FSO?  If it's FSO, shouldn't that be fixed regardless?

[karajorma]

Given that tools like VPView have problems with a VP larger than 1GB, I've never really thought there was a good excuse for having a VP above that size anyway.