Author Topic: Alternative to VP-files - compressed archives (read before freaking out)  (Read 45121 times)

0 Members and 1 Guest are viewing this topic.

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Alternative to VP-files - compressed archives (read before freaking out)
I'm aware of that, but the likelihood of every user being able to figure that out if it's not already handled is a lot lower.
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline MatthTheGeek

  • Captain Obvious
  • 212
  • Frenchie McFrenchface
Re: Alternative to VP-files - compressed archives (read before freaking out)
Your point ? We're talking on a developer point of view here. The average end-user isn't supposed to have to crack-open the archive.
People are stupid, therefore anything popular is at best suspicious.

Mod management tools     -     Wiki stuff!     -     Help us help you

666maslo666: Releasing a finished product is not a good thing! It is a modern fad.

SpardaSon21: it seems like you exist in a permanent state of half-joking misanthropy

Axem: when you put it like that, i sound like an insane person

bigchunk1: it's not retarded it's american!
bigchunk1: ...

batwota: steele's maneuvering for the coup de gras
MatthTheGeek: you mispelled grâce
Awaesaar: grace
batwota: oh right :P
Darius: ah!
Darius: yes, i like that
MatthTheGeek: the way you just spelled it it means fat
Awaesaar: +accent I forgot how to keyboard
MatthTheGeek: or grease
Darius: the killing fat!
Axem: jabba does the coup de gras
MatthTheGeek: XD
Axem: bring me solo and a cookie

 

Offline Zacam

  • Magnificent Bastard
  • Administrator
  • 211
  • I go Sledge-O-Matic on Spammers
    • Minecraft
    • Steam
    • Twitter
    • ModDB Feature
Re: Alternative to VP-files - compressed archives (read before freaking out)

And not all Developers are created equally.

Mod developers come in many different flavors. As do code Developers. And in short, the easiest usage scenario for ALL cases will be the best.

We already HAVE an obscure dedicated format. Why make another?
Report MediaVP issues, now on the MediaVP Mantis! Read all about it Here!
Talk with the community on Discord
"If you can keep a level head in all this confusion, you just don't understand the situation"

¤[D+¬>

[08/01 16:53:11] <sigtau> EveningTea: I have decided that I am a 32-bit registerkin.  Pronouns are eax, ebx, ecx, edx.
[08/01 16:53:31] <EveningTea> dhauidahh
[08/01 16:53:32] <EveningTea> sak
[08/01 16:53:40] * EveningTea froths at the mouth
[08/01 16:53:40] <sigtau> i broke him, boys

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Alternative to VP-files - compressed archives (read before freaking out)
We might not want users cracking open VP files easily. Right now if someone wants to open a VP file they need to figure out which tool to use (which gives us a chance to warn them not to replace anything in them).

You can easily replace files in a .7z file. They wouldn't come to us until they'd already ****ed their install over. Even if we can make our files usable in 7zip we should still name them .vp just to avoid that.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
Err? I'd rather not have same extension as vp-files, otherwise you'd need to manually select app to open them in every damned time. If opening those files is such a concern, then I'd suggest supporting all extensions the LZMA libraries support plus one additional custom extension. That leaves the choice up to whoever creates the archives. Custom extension should be something that is not associated to any common file format. The custom file extension could be something like fso, fs7, fsa.

 

Offline Trivial Psychic

  • 212
  • Snoop Junkie
Re: Alternative to VP-files - compressed archives (read before freaking out)
How about calling them .vpc for VP Compressed?
The Trivial Psychic Strikes Again!

 

Offline karajorma

  • King Louie - Jungle VIP
  • Administrator
  • 214
    • Karajorma's Freespace FAQ
Re: Alternative to VP-files - compressed archives (read before freaking out)
Yeah. That would work for me.
Karajorma's Freespace FAQ. It's almost like asking me yourself.

[ Diaspora ] - [ Seeds Of Rebellion ] - [ Mind Games ]

 
"Neutrality means that you don't really care, cuz the struggle goes on even when you're not there: Blind and unaware."

"We still believe in all the things that we stood by before,
and after everything we've seen here maybe even more.
I know we're not the only ones, and we were not the first,
and unapologetically we'll stand behind each word."

 
Re: Alternative to VP-files - compressed archives (read before freaking out)
How about .VP2 ?
Says what it needs and is not used elsewhere.

 

Offline pecenipicek

  • Roast Chicken
  • 211
  • Powered by copious amounts of coffee and nicotine
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • PeceniPicek's own deviantart page
Re: Alternative to VP-files - compressed archives (read before freaking out)
.vplz mebbe?



(has always preferred extensions with more than 3 letters...)
Skype: vrganjko
Ho, ho, ho, to the bottle I go
to heal my heart and drown my woe!
Rain may fall and wind may blow,
and many miles be still to go,
but under a tall tree I will lie!

The Apocalypse Project needs YOU! - recruiting info thread.

 

Offline Zacam

  • Magnificent Bastard
  • Administrator
  • 211
  • I go Sledge-O-Matic on Spammers
    • Minecraft
    • Steam
    • Twitter
    • ModDB Feature
Re: Alternative to VP-files - compressed archives (read before freaking out)
Call them .CVP for (Compressed Volition Package). Also not an extension that is in use.

I only worry with STARTING the extension with .vp and having the engine through some accident not looking at whether there is anything following the p and trying to load it like a regular VP. And failing. Or failing to load regular .vp files properly.
Report MediaVP issues, now on the MediaVP Mantis! Read all about it Here!
Talk with the community on Discord
"If you can keep a level head in all this confusion, you just don't understand the situation"

¤[D+¬>

[08/01 16:53:11] <sigtau> EveningTea: I have decided that I am a 32-bit registerkin.  Pronouns are eax, ebx, ecx, edx.
[08/01 16:53:31] <EveningTea> dhauidahh
[08/01 16:53:32] <EveningTea> sak
[08/01 16:53:40] * EveningTea froths at the mouth
[08/01 16:53:40] <sigtau> i broke him, boys

 

Offline Tomo

  • 28
Re: Alternative to VP-files - compressed archives (read before freaking out)
I only worry with STARTING the extension with .vp and having the engine through some accident not looking at whether there is anything following the p and trying to load it like a regular VP. And failing. Or failing to load regular .vp files properly.
Don't worry about that - VPs and 7z have different magic numbers at the top of the file, and even the first byte is different so no chance of confusing them even if they have exactly the same file extension.

The only 'need' to make them different is to make it easier for us humans - .CVP is as good as anything else!

 

Offline kkmic

  • 26
Re: Alternative to VP-files - compressed archives (read before freaking out)
I'm not sure what the consensus is, but solid archives are no-go.

This is from WinRAR help file:
Quote
Solid archives
A solid archive is a RAR archive packed by a special compression method, which treats all files, within the archive, as one continuous data stream. Solid archives are supported only by the RAR archiving format, ZIP archives are always non-solid. The archiving method for RAR archives is a user selectable option and may be Solid or non-Solid.

Solid archiving significantly increases compression, especially when adding a large number of small, similar files. But it also has a few important disadvantages:
  • slower updating of existing solid archives;
  • to extract a single file from a solid archive all preceding files must be analyzed. This makes extraction from the middle of a solid archive slower than extraction from a normal archive, but, if all files are to be extracted from a solid archive, the extraction speed will not be affected.
  • if any file in a solid archive is damaged, it will be impossible to extract all files which follow the damaged area. Thus if a solid archive is stored to media such as diskette, it is recommended to make use of the recovery record.
Solid archiving is preferable if:
  • the archive is updated rarely;
  • it is not necessary to frequently extract a single file or only part of the files from the archive;
  • compression ratio is more important than compression speed.

This means that in order to reach a particular file, all files in the archive that are stored before that particular file must be processed.
Doing that for every single file means a very slow loading. Unpacking the archive dynamically kinda defeats the purpose of the archive. Keeping everything from the archive in memory is not viable.

So, no-go for solid archiving, regardless of the method/archiver used
I've seen things you people wouldn't believe. Attack ships on fire off the shoulder of Orion. I've watched C-beams glitter in the dark near the Tannhäuser Gate. All those... moments will be lost in time... like... tears... in rain. (pause) Time... to die.

wxLauncher 0.9.4 Beta | wxLauncher 2.0 Request for Comments

 

Offline Tomo

  • 28
Re: Alternative to VP-files - compressed archives (read before freaking out)
That's not quite true, as Fury mentioned at the start of the thread.

In a solid archive, all the other files in the block containing your required file must be processed. It's not the whole file unless your blocks are massive.
So if you pick a good block size given your expected file size, solid archives will be fine.

(In theory you can get really clever at the packing stage by arranging the files so that the ones you will always use together are in the same block. Not that I think it's likely to be worth bothering.)

Non-solid ones will of course still be faster to extract.

Either way, they'll be much smaller than at present.

 

Offline jr2

  • The Mail Man
  • 212
  • It's prounounced jayartoo 0x6A7232
    • Steam
Re: Alternative to VP-files - compressed archives (read before freaking out)
* jr2 likes this idea.

Wasn't this brought up before?  I forget.

 

Offline Topgun

  • 210
Re: Alternative to VP-files - compressed archives (read before freaking out)
* jr2 likes this idea.

Wasn't this brought up before?  I forget.

It was brought up before but I guess the idea just died.

  

Offline Echelon9

  • 210
Re: Alternative to VP-files - compressed archives (read before freaking out)
* jr2 likes this idea.

Wasn't this brought up before?  I forget.
Taylor, and then later myself, did have a look at this concept. There was some proof of concept code floating around, but it never got to a point at which it was widely circulated on these forums or IRC.

The thinking done ahead of the coding then arrived at largely the same conclusions as here (non-solid, algorithm akin to LZMA) if I remember correctly.

 

Offline Fury

  • The Curmudgeon
  • 213
Re: Alternative to VP-files - compressed archives (read before freaking out)
I'm not sure what the consensus is, but solid archives are no-go.
You may want to re-read the first post in this topic, as it states the very same thing and makes it a point that archives need to be non-solid to work for random access.

 

Offline Zacam

  • Magnificent Bastard
  • Administrator
  • 211
  • I go Sledge-O-Matic on Spammers
    • Minecraft
    • Steam
    • Twitter
    • ModDB Feature
Re: Alternative to VP-files - compressed archives (read before freaking out)

Great BUMP for Justice.

Another reason why we need this: CF_MAX_FILENAME is maxed at 32. This apparently cannot be bumped because the VP File Format structure was only written to support 32. We can't raise the limitation in the engine, because a VP file still won't store at greater than 32 anyway.

So, the only way to have filenames longer than 32 characters (or any other values longer than 32 characters) will be in the implementation of this "Compressed" VP (or CVP) format.

And of course, we'd still need to make sure that any bumpage only affects when loading data from a CVP to boot.
Report MediaVP issues, now on the MediaVP Mantis! Read all about it Here!
Talk with the community on Discord
"If you can keep a level head in all this confusion, you just don't understand the situation"

¤[D+¬>

[08/01 16:53:11] <sigtau> EveningTea: I have decided that I am a 32-bit registerkin.  Pronouns are eax, ebx, ecx, edx.
[08/01 16:53:31] <EveningTea> dhauidahh
[08/01 16:53:32] <EveningTea> sak
[08/01 16:53:40] * EveningTea froths at the mouth
[08/01 16:53:40] <sigtau> i broke him, boys

 

Offline Luis Dias

  • 211
Re: Alternative to VP-files - compressed archives (read before freaking out)

Great BUMP for Justice.

Another reason why we need this: CF_MAX_FILENAME is maxed at 32. This apparently cannot be bumped because the VP File Format structure was only written to support 32. We can't raise the limitation in the engine, because a VP file still won't store at greater than 32 anyway.

So, the only way to have filenames longer than 32 characters (or any other values longer than 32 characters) will be in the implementation of this "Compressed" VP (or CVP) format.

And of course, we'd still need to make sure that any bumpage only affects when loading data from a CVP to boot.


This means deprecation of .vp by all modders, which isn't that much of a trouble. LZMA is open-source, so no problem there too. I say go for it.

Quote
We might not want users cracking open VP files easily. Right now if someone wants to open a VP file they need to figure out which tool to use (which gives us a chance to warn them not to replace anything in them).

You can easily replace files in a .7z file. They wouldn't come to us until they'd already ****ed their install over. Even if we can make our files usable in 7zip we should still name them .vp just to avoid that.

I'm sure that any user knows that if you touch the game's files, probably they will **** up the game ;). Don't take people to be that stupid. For what is worth, I think going towards more general standards is better, in long term.