[WMCoolmon]

Well, I don't mind having compressed VP support in any of my programs. Like you say, many of the filetypes used are already compressed, so it's not like VP files now are 100% dedicated to speedy accessing of files, or else using DDS, JPG, OGG, and all the rest would be disallowed.

And finally, VPCS was a total pain in the ass to use. If it had been a program that was usable to efficiently create VP files, I would never have written VPMage. Not to mention that you have a habit of abandoning your projects to start a new one. And a habit of reacting violently with people who disagree with you. So if you want to hold up progress by flat out refusing to support compressed files, that's fine; I'll just upgrade my library so people don't have to be limited by yours.

Finally, not everyone has or is willing to spend $111, so they can play some campaign or even fs2_open itself.

Edit: And, actually, CVPs would help multi-platform interopability. No more fiddling with platform-specific shareware programs to uncompress a file, or with a platform-specific installer, just drop the VP in a mod dir and go.

Although that does bring up a relevant point, would it be possible (if the VP format is upgraded anyways) to add the mod option I brought up awhile back? (ie the VP is only used if a -mod option agrees with the name stored in the VP file)
This would make installing mods a matter of simply downloading the .cvp to your FS2 directory. Support for some kind of meta-tag (eg parent VP, some kind of meta field) would make mod management via a user-friend GUI very easy.

[Kazan]

Goober5000: prefixing comments with "i don't want to start a flame war" and "i don't want to get too personal" does not make them any less flaming - and I only got pissed because he was flaming me based upon statements I didn't make and positions I didn't hold - I was pissed because he ASSUMED

WMCoolman:
VPCS was "Abandoned" because I went inactive - i don't typically abandon my big apps like that - once VPCS2 is done I will be going back to revisit VPCS2 to finish it off [afterall it's still tagged RC]

[taylor]

Originally posted by WMCoolmon
Although that does bring up a relevant point, would it be possible (if the VP format is upgraded anyways) to add the mod option I brought up awhile back? (ie the VP is only used if a -mod option agrees with the name stored in the VP file)
This would make installing mods a matter of simply downloading the .cvp to your FS2 directory. Support for some kind of meta-tag (eg parent VP, some kind of meta field) would make mod management via a user-friend GUI very easy.

I do want CVP to be a complete upgrade/rewrite to VP and not just a compressed version.  We just need to figure out what we want of a better VP format and starting making a list.  I know that VP in a VP has been brought up before so maybe that's something to consider here as well.

I'm mainly looking to get these features done:
- compression in a reasonable way
- large file support (over 2gig, of the CVP itself, not individual files)
- dedicated (and easily manipulated) structure area for file and directory listings so that files can be more easily added/removed/modified without having the remake the entire CVP

Beyond that it's up in the air.  Like I said though I'm not rushing into this.  I've been thinking about it for a couple of months and it will probably be December before I start coding, even if that early.  I'm looking at this as a from-the-ground-up rewrite so we might as well have it do everything that we want.

[WMCoolmon]

In that case let me add another thing that I think I suggested for Ferrium...usage-based location. That is, fs2_open records the number of times a file is accessed; this info could then be spat out on exit, if the appropriate command line parm exists.

Another program could then make use of it to reorder the file data so that most-often-accessed files are at the front of the VP. More advanced algorithms (ie to detect what order files are loaded for specific missions, and group them appropriately).

It would probably only have a performance increase on computers with slow hard drives, so I never was too adamant about it.


Something else I thought about was symlinks; how these would work would be that, once a given file is specified as being loaded into a VP, additional FAT entries can be created using the first one as a base. Whenever a file is deleted, the calling program checks for any entries that use the same file data, and delete it if no other entries exist.

AFAIK, this would require no changes to the way fs2_open does VP files (ie it could be done right now.)


And, another pipe dream, some kind of subpackage support. I'm mostly thinking of for the mediaVPs, although this could be expanded to mods and such, I suppose.
Basically, the ability to group files into a package, add a title, author, description, site, etc. Then enable/disable them on an individual basis using a GUI (checkboxes, like the flags thing in the Launcher as of now).
This would be under an "Advanced..." button, by default everything would presumeably be enabled.
This one seems a bit too extensive and difficult to code to be worth it, at least as of now, but it might be good in the future.

[dizzy]

Ok, then Taylor I think I have this sugestions for the index structure at least:

struct idxentry {
    uint64 dataOff; /* this can be 2 offsets if we support compression , one for where the compression block starts that includes the beginning of this file and one where in that decompressed block it really starts this file*/
    uint64 child;
    uint64 brother;
}

Which means that the offset to the data content of the file is a 64 bit integer (supporting over 4gb of VP file size), then you got an offset where is the child (if any, this happens to directories only), then where you got the next entry in the directory where you are in too. This structure allows editing of the index rather easily. If you need to add a new file just add it's index entry at the end of index and "link" it by writting the proper offset in it's parent or it's previous brother. The "link" itself can happen by a simply overwrite because you already have those fields of the parent or of the previous brother written in the index so you just overwrite fixed size already allocated fields in the index. Also moving a file (or a whole directory tree) is done just by again overwritting this kind of fields. For example for moving a directory tree you have to put NULL in the offset that was sending to the base directory of the directory tree that you are moving (which can be a previous brother or a parent) and then put the offset to the base directory of the moved directory tree where it is linked in the directory structure (to a previous brother or a parent).  You got a lot of advantages with this aproach.

One disadvantage with this aproach is that after you randomly edit it by moving and adding entries all arround you might get a very "unordered" index if you are to unpack a whole directory tree at once. For this problem we can just support (in the tools and in the library) reordering of the index. In the end I don't see any requirement anywere that the index should be big enough that we should not store it completely in memory. If we store it in memory always then we can work out any solution anyways because we can make sure when the VP file is closed by the library, the index is then written and is written to any possible format (even the current one).

About storing the filename I see 3 posibilities:
1. store as it is now, totally included in the index entry of that file so you allocate a fixed size for it (so you got a max size limitation and waste of space for small names); this has the advantage thet it supports the renaming of a single file extremely easy
2. store it in a index entry but not as a field of maximum size but as a variable C-string (ended with \0) or as a Pascal-like string (store size and then the string) but still variable; this makes renaming of a file more complicated in the case that the new name is bigger than the old; in this case one solution whould be to add a new entry at the end of the index for the new index entry with the new name and  "delink" the old one (but will leave "unused" bytes in the middle of the index)
3. store it as a variable string but at the beginning of the file data area; this makes renaming almost impossible

[Kazan]

WMC: Moving files to the begining isn't going to read things up - a seek is a seek, whether you're seeking a +1gb index or a +10byte index

[Inquisitor]

And Kazan: Everyone else is pissed because you reacted like a jackass. Knock it off.

Talk to me on IM.

[Kazan]

hey taylor - i just realized i'm an idiot - i have KDE/cygwin on both my laptop and dev desktop

I could be compiling on there to make sure it'll compile cleanly in linux

[edit]
can i get that makefile?

[taylor]

Originally posted by Kazan
can i get that makefile?

I still need to get the #include's fixed in CVS but you can do that yourself when you try it first time.  When you work out GCC's hatred I'll go ahead and start with the endianess stuff.

Here's the Makefile (basic copy of what icculus.org freespace2 uses):

Code: [Select]

CPP=g++

PCS2_BIN=pcs2
LDFLAGS=$(shell wx-config --libs)
CPPFLAGS=$(shell wx-config --cflags)
CPPFLAGS+=-D_UNIX


%.o: %.cpp
$(CPP) -c -o $@ $< $(CPPFLAGS)


SOURCES=./BSPDataStructs.cpp \
./BSPHandler.cpp \
./COBHandler.cpp \
./kaz_templates.cpp \
./pcs_file.cpp \
./pcs_file_dstructs.cpp \
./pcs_pmf_cob.cpp \
./pcs_pmf_pof.cpp \
./POFHandler.cpp \
./vector3d.cpp \
./wxCTreeCtrl.cpp

OBJECTS=$(SOURCES:.cpp=.o)


$(PCS2_BIN): $(OBJECTS)
$(CPP) -o $(PCS2_BIN) $(LDFLAGS) $(OBJECTS) $(CPPFLAGS) pcs2.cpp


all: $(PCS2_BIN)

[Kazan]

awesome bro, i'll use that at lunch and commit it to cvs

[WMCoolmon]

Originally posted by Kazan
WMC: Moving files to the begining isn't going to read things up - a seek is a seek, whether you're seeking a +1gb index or a +10byte index

Not quite true. Firstly, searching the index, if the first hit is the file you're looking for then you don't have to look any more. Secondly, seeking, if you put the files in the order that you need them, then any caching the hard drive does will work in your favor, rather than be useless.

I am talking about the actual *order* of files, not the location of them relative to the other components of the file...although if the index were first, the same principle would apply, since the VP header certainly doesn't take up 8MB