Author Topic: Alternative to VP-files - compressed archives (read before freaking out)  (Read 45110 times)

0 Members and 1 Guest are viewing this topic.

Offline Fury

  • The Curmudgeon
  • 213
Alternative to VP-files - compressed archives (read before freaking out)

EDIT: Here's the TL;DR of the rest of the thread so far (4/25/11).

We don't want to use a new, proprietary format, or a modified existing format, because that would require the addition of tools that we'd have to maintain.  It's basically been decided that non-solid 7z (LZMA) would be the best idea for this format, as 7-zip already exists on all supported platforms for editing archives, and the SDK is open source and compatible with FSO.  No further need to discuss this really, we just need to implement it now.

Chief

--Now here's the rest of the original post--

There was once discussion about allowing support for other file formats than Volition's vp file format for game assets. Obviously the usual suspects of zip, rar, 7z all popped up. They all offer the option to use variety of different compression options or no compression at all. Plus they're easier to manage than vp-files regardless of the vp-managers we have. Would there be other advantages? Certainly, but let's get back to those later.

I decided to perform some tests with data that is very relevant to our use scenario. Compression tests were done with exported MV_Advanced, MV_AnimGlows, MV_Assets, MV_CB_ANI_1, MV_CB_ANI_2, MV_Effects, MV_Intel_ANI, MV_Music, MV_radaricons and MV_Root from FSU SVN. Compression was done using 7-Zip 9.20 utilizing four threads.

For comparison, uncompressed and deflate. Deflate is the standard compression method used in ZIP archives we've all grown to love and hate.
Code: [Select]
Uncompressed 2.54 GB
Deflate Fastest 1.08 GB
Deflate Normal 1.02 GB
Deflate Ultra 1.01 GB
BZip2 is so slow that it's not even worth testing, compression ratio also less than that of PPMd and LZMA but better than deflate.

PPMd is the latest addition to ZIP, RAR and 7-Zip archivers.
Code: [Select]
PPMd Fastest 928 MB
PPMd Normal 860 MB
PPMd Ultra 840 MB

Now, let's test LZMA2 which is successor to LZMA, a popular compression algorithm used in 7-Zip.
Code: [Select]
LZMA2 Fastest 990 MB
LZMA2 Normal 844 MB
LZMA2 Ultra 837 MB

And next, otherwise identical to previous set of LZMA2 archives but these archives are not solid. In solid archives files are compressed as contiguous data blocks. This means that to extract one file from the archive, you also need to extract all other files in same data block. Size of solid data block depends on compression settings, it can be as small as 1MB or encompass the whole archive regardless of its size. Contiguous data blocks increase compression efficiency. Zip archives such as deflate compressed archives are non-solid. A non-solid archive is what you would say is requirement for "stream decompressing", or streaming.
Code: [Select]
LZMA2 Fastest 997 MB
LZMA2 Normal 887 MB
LZMA2 Ultra 885 MB

Now let's review how much RAM is needed to decompress these archives. Whether archive is solid or not has no impact on memory usage.
Code: [Select]
Deflate Fastest 2 MB
Deflate Normal 2 MB
Deflate Ultra 2 MB
PPMd Fastest 3 MB
PPMd Normal 18 MB
PPMd Ultra 130 MB
LZMA2 Fastest 3 MB
LZMA2 Normal 18 MB
LZMA2 Ultra 66 MB

Right, so that covers how small the archives are and how much RAM is needed to decompress. Compression speed is mostly irrelevant in our use scenario, so I'm skipping that. The bigger question is how fast is decompression of compressed archives? Let's try out, both archive and target directory for extracted content is on same physical hard drive which follows our typical use scenario. In this test I'm using non-solid LZMA2 archives as solid archives are unusable in our typical use scenario.
Code: [Select]
Deflate Fastest 2:16 min
Deflate Normal 2:16 min
Deflate Ultra 2:17 min
PPMd Fastest 8:53 min
PPMd Normal 11:43 min
PPMd Ultra 18:46 min
LZMA2 Fastest 2:08 min
LZMA2 Normal 2:10 min
LZMA2 Ultra 2:16 min

From these tests we can immediately rule out PPMd as usable compression algorithm for our use scenario. LZMA2 on the other hand performs admirably, beating even deflate in decompression speed. It is no wonder why some linux distros have adopted LZMA2 compression for use in their installation CD's and repositories as well as slowly gaining popularity in mobile devices for these very reasons.

And so, we have one last question remaining. Just how does LZMA2 compare to uncompressed archive? Again, let's find out!
Code: [Select]
Uncompressed 2:04 min

Are your eyes deceiving you? Certainly not. Fastest LZMA2 decompression is only 4 seconds slower, normal 6 seconds and ultra 12 seconds. And in this test scenario we decompressed 2.54 GB worth of data. The difference in speed should be lower during most mission loads.

Now we can get to the benefits. I can think of following benefits in having compressed LZMA2 archives play role of vp files:
- LZMA2 archives have pretty good checksum checks, so corrupted archives throws up an error. With vp-files you may just end up wondering wtf is going on without really realizing the cause, until JeffVader points out that your checksums in debug log aren't matching.
- Mods and campaigns can be downloaded to mod folders as-is. This is even more convenient with installers such as FSO Installer and Desura.
- Normally you'd keep both extracted vp files as well as downloaded compressed archive around, with support to such compressed archives to work like vp files, you'd only need to keep the compressed archive. This saves a lot of disk space if you have small HDD.

Nobody is forcing anyone to switch from vp files to whatever compressed archives, but it'd be real nice to have the option. Especially since the difference in speed doesn't really seem to be that much.
« Last Edit: April 25, 2011, 10:12:45 am by chief1983 »

 

Offline Turey

  • Installer dude
  • 211
  • The diminutive form of Turambar.
    • FreeSpace Open Installer Homepage
Re: Alternative to VP-files - compressed archives (read before freaking out)
This seems like a great idea, assuming the LZMA2 code integrates cleanly with the existing codebase. I'm all for a test implementation.

Plus the bandwidth savings on installation are really awesome.
« Last Edit: January 05, 2011, 01:32:16 pm by Turey »
Creator of the FreeSpace Open Installer.
"Calm. The ****. Down." -Taristin
why would an SCP error be considered as news? :wtf: *smacks Cobra*It's a feature.

 

Offline Nohiki

  • 28
  • Graf von Kaffeetrinken
    • Minecraft
    • Skype
    • Steam
Re: Alternative to VP-files - compressed archives (read before freaking out)
Can you throw the comparsion to .vp in there? :)
:::ALSO PROUD VASUDAN RIGHTS SUPPORTER:::

  

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
A vp is for all intents and purposes same as uncompressed archive, which was covered.

 

Offline headdie

  • i don't use punctuation lol
  • 212
  • Lawful Neutral with a Chaotic outook
    • Minecraft
    • Skype
    • Twitter
    • Headdie on Deviant Art
Re: Alternative to VP-files - compressed archives (read before freaking out)
Interesting stuff and imo worth investigating,

this is probably a stupid question but how does opening several archives affect performance
Minister of Interstellar Affairs Sol Union - Retired
quote General Battuta - "FRED is canon!"
Contact me at [email protected]
My Release Thread, Old Release Thread, Celestial Objects Thread, My rubbish attempts at art

 

Offline Rodo

  • Custom tittle
  • 212
  • stargazer
    • Minecraft
    • Steam
Re: Alternative to VP-files - compressed archives (read before freaking out)
I always wondered why VP's where still used, I can see a good feature if LMZA2 support gets implemented someday.
el hombre vicio...

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
this is probably a stupid question but how does opening several archives affect performance
I'm not quite sure what you mean. It's not any different from opening multiple files of any kind. Of course application used and file format does play a large role. But in that area 7z and LZMA have been designed to have small overhead. I opened MV_Assets.vp in VPView and then compared to opening MV_Assets.7z (LZMA2) in 7-Zip. VPView used approximately twice as much memory as 7-Zip did.

But in case of FSO opening the archives, I don't know. The tools and file format have been touted to be an excellent choice in embedded applications and mobile devices. So as far as archives go, I believe LZMA to be very efficient in that regard.

 

Offline The E

  • He's Ebeneezer Goode
  • Moderator
  • 213
  • Nothing personal, just tech support.
    • Steam
    • Twitter
Re: Alternative to VP-files - compressed archives (read before freaking out)
The point would be moot, anyway. The engine rarely has more than one file handle open.

The important part for FSO's archive handling (and vp files are archives as well), is that FSO expects random access ability. That means, the ability to seek and retrieve any file from the archive at any time. As long as that is guaranteed and the overhead is reasonably low (overhead being defined as the time between the engine requesting a file, and the archive handler starting to stream the data), any archiving scheme can be worked into FSO. Eventually, anyway; I have no idea at this time how complex this would be.
If I'm just aching this can't go on
I came from chasing dreams to feel alone
There must be changes, miss to feel strong
I really need lifе to touch me
--Evergrey, Where August Mourns

 

Offline Tomo

  • 28
Re: Alternative to VP-files - compressed archives (read before freaking out)
I found it was surprisingly easy to swap out opening individual files in a folder structure for opening ZIP (deflate) archives in an internal application at work.
Hardest bit was swapping out use of file handles all over the codebase for the datastream objects produced by the DLL I used.

- It also made the app run about ten times faster, mostly because it exchanged opening then closing about 4000 individual files to opening just one!

 

Offline Topgun

  • 210
Re: Alternative to VP-files - compressed archives (read before freaking out)
why not add tar support and gzip support? that way we we can release packages as .tar.gz and if the user wants they can decompress it into a .tar for faster loading times without converting it to a vp.

 

Offline Tomo

  • 28
Re: Alternative to VP-files - compressed archives (read before freaking out)
Tar is inappropriate for this usage, as it is optimised for sequential access.

FS2 Open needs random access, as it doesn't extract the complete contents of VP files into memory at any point (and in fact could not do so).
During application start it just makes a list of the files available and where to find them when requested by a campaign or mission.
(Vastly simplified, it does process some data at load eg tables)

VPs start with a header that lists all the files and the folder structure contained within, each with a pointer to the location and size in the binary blob that the file actually occupies.

On mission load, the relevant data is loaded into the game engine.

Any compressed file structure must be equivalent - same information available in a header, and the 'game data' is then decompressed upon request.

There are several compressed packages that would be suitable, most with the bonus of true error-checking.
- VPs don't have this. If you've got a broken VP file, then it silently causes weird stuff to happen. You have to manually determine which VP is broken.

Source code for the library used by 7z is available, so it would be worthwhile taking a closer look at it to see if it is usable.
The 7z format itself looks like it may be suitable, as the folder and file structure is in the header.
« Last Edit: January 14, 2011, 04:27:36 pm by Tomo »

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
Yep, tar is unsuitable for this. Also, gz has ****ty compression ratio when non-text files are involved.

 

Offline Topgun

  • 210
Re: Alternative to VP-files - compressed archives (read before freaking out)
gz might have poor compression ratio but its fast.

 

Offline sigtau

  • 29
  • unfortunate technical art assclown
Re: Alternative to VP-files - compressed archives (read before freaking out)
I'd actually take the 7zip route first, considering it's significantly easier to support from what I understand.
Who uses forum signatures anymore?

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Alternative to VP-files - compressed archives (read before freaking out)
I'm pretty sure it was already shown that LZMA is about the most effective option.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Zacam

  • Magnificent Bastard
  • Administrator
  • 211
  • I go Sledge-O-Matic on Spammers
    • Minecraft
    • Steam
    • Twitter
    • ModDB Feature
Re: Alternative to VP-files - compressed archives (read before freaking out)

LZMA is what 7z uses. As for supporting it, once the big obstacle of implimenting is over come, there won't be much in the way to NEED to support. Though it will be interesting to see if we can replace libpng's need for zlib with this instead instead of having to tack in yet another handler.

I think LZMA2-NS (non-solid) Normal is a good "default" to aim for. Takes a touch more RAM on decompressing than "Fastest" but the size trade off and the mere few seconds time difference strikes a good balance. 887mb space usage vs 997mb is a much better margin than 887 vs 885.
« Last Edit: January 18, 2011, 05:27:44 am by Zacam »
Report MediaVP issues, now on the MediaVP Mantis! Read all about it Here!
Talk with the community on Discord
"If you can keep a level head in all this confusion, you just don't understand the situation"

¤[D+¬>

[08/01 16:53:11] <sigtau> EveningTea: I have decided that I am a 32-bit registerkin.  Pronouns are eax, ebx, ecx, edx.
[08/01 16:53:31] <EveningTea> dhauidahh
[08/01 16:53:32] <EveningTea> sak
[08/01 16:53:40] * EveningTea froths at the mouth
[08/01 16:53:40] <sigtau> i broke him, boys

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
In theory non-solid LZMA2 7z archives are by far the most superior choice in our use scenario that I have tested. But how well it works in PRACTICE is different matter. Unfortunately I don't know any way to test it without actually first implementing it in FSO.

Topgun, gz was practically already tested (see deflate results) and it was all around worse than LZMA2. LZMA2 archives even had faster decompression than deflate, even though they had much superior compression.

 

Offline Topgun

  • 210
Re: Alternative to VP-files - compressed archives (read before freaking out)
ah, I didn't notice that, thanks.

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Alternative to VP-files - compressed archives (read before freaking out)
And if we could just use the existing .7z extension or something else 7-zip can associate, we can just open them up with it.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline MatthTheGeek

  • Captain Obvious
  • 212
  • Frenchie McFrenchface
Re: Alternative to VP-files - compressed archives (read before freaking out)
Extensions don't mean anything. Nothing prevents you to associate any extension to any software you want.
People are stupid, therefore anything popular is at best suspicious.

Mod management tools     -     Wiki stuff!     -     Help us help you

666maslo666: Releasing a finished product is not a good thing! It is a modern fad.

SpardaSon21: it seems like you exist in a permanent state of half-joking misanthropy

Axem: when you put it like that, i sound like an insane person

bigchunk1: it's not retarded it's american!
bigchunk1: ...

batwota: steele's maneuvering for the coup de gras
MatthTheGeek: you mispelled grâce
Awaesaar: grace
batwota: oh right :P
Darius: ah!
Darius: yes, i like that
MatthTheGeek: the way you just spelled it it means fat
Awaesaar: +accent I forgot how to keyboard
MatthTheGeek: or grease
Darius: the killing fat!
Axem: jabba does the coup de gras
MatthTheGeek: XD
Axem: bring me solo and a cookie