Well, first off thanks to ngld who helped me understand how FSO file system worked, and taylor who pointed me in the right way.
How it works:
1) Files are compressed with a compression tool, the writes the correct header, the compressed lz4 data, the offset list, the original file size and block size like this:
This files can be added inside a vp or be just dropped into the data folder, it changes nothing where it is.
2) There is now a "compressed_info" struct inside CFILE in cfilearchive.h, this struct is empty when the file is not compressed, if the file is compressed, at cfopen FSO will first check for the header, if it matches a known header, then it fills the correct compression_info, load the offset list into memory and reserves memory for the decoder cache.
3) cftell, cfeof, cfseek and cfread all have special cases for handling the compressed file, in those cases it just call the equivalent comp_ function, so from the rest of the code everything should be transparent.
From there everything should be the same, the decoder cache is used to store decoded info, this is because LZ4 stores date in blocks and decodes full blocks, the random access function is always able to get X bytes starting from Y offset of the original file, but DDS and pof parsing request multiple 4 bytes info, 512 bytes info, one after the other, this means if the LZ4 block size is 8192 bytes the function would have to read and decode that block multiple times in order to get all the information, the buffer is used in a way that after a block is decoded once, it is stored in memory so if the next requested data is already in buffer it just copies that instead of reading the file and decode again.
at cfclose() the compression_info is cleared and the memory used to store the decoder buffer and offset list is freed.
MVP 4.4.1 - All Models mission load time
----------------------------------------
LZ4-HC L6, 7.01GB, BS: 65536, CP TIME: 5m
0:50 - NVME Uncompressed
0:50 - NVME Compressed
1:01 - SSD Compressed
1:02 - SSD Uncompressed
1:49 - HDD Compressed
2:30 - HDD Uncompressed
Compressed MVP 4.4.1 Block Size: 65536 LZ4-HC L6
7.01GB - Space used on disk
Compression time: 5 minutes
Compressed MVP 4.4.1 Block Size: 65536 LZ4-HC L12 (MAX)
1:45 - Mission load time with all models
4.66GB - Memory used after mission load
6.98GB - Space used on disk
Compression time: 15 minutes
Compressed MVP 4.4.1 Block Size: 65536
1:49 - Mission load time with all models
4.63GB - Memory used after mission load
7.75GB - Space used on disk
Compression time: 4 minutes
Compressed MVP 4.4.1 Block Size: 16384
1:55 - Mission load time with all models
4.53GB - Memory used after mission load
8.02GB - Space used on disk
Uncompressed MVP 4.4.1
2:31 - Mission load time with all models
4.45GB - Memory used after mission load
12.9GB - Space used on Disk
Block Sizes:
--------------
My tests where mostly with 8192 and 16384 block sizes, a higher block size will make the large compressed files smaller, higher block size will also make FSO to use more RAM but not by much and it will also have a effect on loading times, all that needs to be tested.
Formats NOT recomended for compression,everything works but there a few file types that are not recomended to compress:
--------------------------------------------------------------------------------------------------------------------------------------------
-fc2,fs2,tbl,tbm,eff: These are optional, i personally think these formats should stay in a user readable condition. If you want to edit a mission, you dont want having to decompress it just to because someone wanted to save 50kb.
-Movies, mp4,ogg,mve,webm: There is just no much point in trying, the ratio will be negative most of the time, and when is not, you are just saving a few kb, maybe a mb or two.
-Audio, ogg, wav: This is deep into my "NO" list, as with movies, you are almost never going to have a positive ratio, and when you do, is just not worth it, and if some of the audios arent cached before missions, voices, personas, etc, its going to be BAD to do this process in game, the first reading its going to have a lot of overhead.
-pngs: 99% of the time the ratio is negative, there is no point in trying, you are just wasting compression time.
The test compression tool is set to ignore files smaller than 20kb and to skip compressed files that are bigger than the original. But it does not skip files based on extension, that is supported but disabled at in code for the precompiled bin, you can clone, re-enable, and recompile. This is intended as a test tool after all.
Source Code:FSO build:
https://github.com/Shivansps/fs2open.github.com/tree/compressed-filesCompression Tool: https://github.com/Shivansps/VPC_CompressorPrecompiled FSO build + Compressor tool:https://www.dropbox.com/s/0owtxlqp6b6ku9a/cmpTest-1.0.6.zip?dl=0It is also uploaded to Knossos, look for "Compression test build".
Recomended compression is LZ4-HC Level 6.
