Author Topic: Woo, VPMagic 0.2 source release!  (Read 9100 times)

0 Members and 1 Guest are viewing this topic.

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
All nicely packaged and with license/gen. description added to files. Plus, it even includes a README! :p
-C

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
Realized a small buggery with the makefile. This:
Code: [Select]
@echo Install finished; use "sudo make uninstall" if you ever wish to uninstall

should be this:

Code: [Select]
@echo Install finished\; use \"sudo make uninstall\" if you ever wish to uninstall

Not a big problem, unless you have a command named "use" that instantly reformats your hard drive or something. :)
-C

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Woo, VPMagic 0.2 source release!
You really need to make it "run `make install` as root" or something.  As it is the user in question would have to have sudo setup to allow make to run privileged, something it probably won't be by default.  I haven't checked it out yet though so you may actually have included a quick lesson on visudo usage. ;)

 

Offline Admiral LSD

  • 27
  • Shorter of breath and one day closer to death
    • http://adphq.dyndns.org
Woo, VPMagic 0.2 source release!
heh, I can count the number of times I've used sudo (or related tools) on the fingers of one hand justabout. I always either log on as root properly or use plain ol' su to achieve (much) the same thing. Anyone reasonably literate in Linux knows you run this sort of stuff as root and also knows the various ways to get there (and those that don't should shuffle off back to XP Home ;)) so most of the message is redundant.
00:19  * Snail cockslaps BotenAnna
00:19 -!- Snail was kicked from #hard-light by BotenAnna [Don't touch me there! RAPE!!!]

15:36 <@Stealth_T1g4h> MASSIVE PENIS IN YOUR ASS Linux

I normally enjoy your pornographic website... - Stealth
Get Internet Explorer!

 
Woo, VPMagic 0.2 source release!
In your next version of VPMage you should replace rar with tar.bz2 or tar.gz to add the real linux feel to it. ;)

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
Heh.

I just realized that it seems to be saving rather-corrupt files...I suspect this is due to AMD64, I'll be looking into it...
-C

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Woo, VPMagic 0.2 source release!
The time/date in the header for files has to be 32-bit.  time_t is a long so on 64-bit platforms (not including Win64) it's 64-bit.  Just make sure it's 32-bit and you should be ok.

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
Yeah, that would explain it. Actually, there are a whole host of sizeof() calls in a subfunction that have no business being there.

Edit: Updated file will be available later, there are some more tweaks/fixes I've made and I may add some other niceties to the lib.
« Last Edit: July 28, 2005, 07:57:45 am by 374 »
-C

 

Offline dizzy

  • 26
    • http://dizzy.roedu.net
Woo, VPMagic 0.2 source release!
Hi there

First, good job!

Second, I had one hour free time now and I added support to vpmagic for autoconf/automake/libtool. The library (libvpmagic) is still built with "noinst_" (so it's not installed on make install). This was done because I had no time yet to see if those codes are really self contained and so on.

I have posted the tarball resulted from a "make distcheck" (one of the sexy advantages of automake) at: http://dizzy.roedu.net/dizzy.roedu.net/fs/vpmagic-0.1.tar.gz

Next time I will have some free time I will look into really using the autoconf provided #defines by conditionaly including some of the includes (or maybe not, I think the codes can be made very much to just use the provided ANSI C++ standard library and as such have almost no unportable depencies).

The advantages of automake are now:
- automaticallyu can build your library (in static/shared version) and nicely install it along with the headers
- automatically compile test the codes (make distcheck) and create a release tarball
- automatically create and use dependencies (so you need not manage those includes dependencies in the Makefile's as the old ones do)

The one big disadvantage is while automake as been recently ported on Win32 it still largely depends on a powerful command shell on the system so it usually works only in POSIX enviroments and similar (like Mac OS X, BeOS). We could use "scons" (www.scons.org) if you need very good win32 support out of the box too (or without any other build project file that is).

Now about the vplib API. I really don't like it. The thing is, in general, when designing an API always try to use some existent one if there is already one old, used and proved good for the same setup. In the vplib case, vp files are just some virtual filesystems packed in one big file, as such I think a better API is a filesystem API.

So I would make 2 APIs for vp file access  (one in C for C or C++ users and one in C++ for C++ users)

1. the C API I would make it like stdio API, meaning vp_open, vp_close, vp_read, vp_write, vp_seek and everything should be done transparently in the library (the actual positioning in the VP file etc, also the write case is a little bit more complex but can be done too with transparent interface like that); for directory structure scanning and access I would make it like POSIX or Win32 that is: vp_opendir, vp_closedir, vp_readdir (very similar to POSIX ones and pretty close to Win32/DOS that has findfirst, findnext family of functions);

2. the C++ API I would make it based on the existent C++ file API (ie ); so I would extend ifstream and ofstream to support exactly the same API but transparently using a VP file instead of the real filesystem; this means that all of the sudden you can reuse all your codes and learned knowledge on using VP files (you could even use the <<, >> operators), no need to learn YET ANOTHER API, yet another way to use something; for directory scanning (as there is no directory scanning fucntions for ANSI C++ library) I would must make some C++ wrappers on the C directory scanning interface from above

So what do you think ? :)

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Woo, VPMagic 0.2 source release!
In any design, be sure to include future support for CVP.  CVP is basically a standard VP compressed with zlib.  Other than some basic ideas in my head nothing exists for that yet so just keep in mind that it's coming, but I have no idea what it's going to end up looking like.

 

Offline dizzy

  • 26
    • http://dizzy.roedu.net
Woo, VPMagic 0.2 source release!
Oh I see, good idea, I too was thinking yesterday that having some sort of compression in vp should be cool. Well, one thing for sure is that the version will change (instead of version2 will probably be version3 or whatever) the other should be depending on how we decide to compress:
- we may compress per file (the easiest solution to implement, especially with a transparent filesystem API like the above proposed one, but has the drawback that you may NOT get the best compression ratio because compressing per file might not see the best patterns on the whole file)
- compress the whole "data area" (the area of VP where the file contents are one after another); this should provide maximum compression rate but it makes it harder to implement (hmm now that I think that in any case I wouldnt have supporting updating just a file inside the VP but instead create all the VP from the beginning then it wouldnt be that complex at all)

Another issue is that we could also compress the index as with big VPs it has some size and it should compress a lot (this happens because the index format is somewhat not very optimal when it comes to disk usage because it stores the filenames in a fixed size char array instead of doing a variable size format).

Any hints ? :)

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
It sounds like it'd involve a total API redesign. :p

The reason I did it how I did is that I wanted an easy-to-use API for VP management...that is, moving files about. Generally for VP management you don't need lowlevel functions like fopen, and implementing that really would've been more work than how I did it because files are always stored in complete formats.

And actually the way the library has grown is that I've added functions as I've needed them, and in the way that seems the most flexible while still easy to use and being fairly speedy.

For a program like POFCS, vp_write and such would make more sense because it would allow you to abstract file I/O whether a file was simply part of the native filesystem or in a VP archive.

The other thing is that I wrote it to be easily portable between virtually any compiler. Meaning no extensive make or project files...IMHO that just encourages needless complexity in a library of this size. Even if I could write a makefile with a half-dozen configurable options, I wouldn't - it's not worth it.

Compression and encryption with the current lib would need hooks in load_vp, extract_file and (I think) build_vp and add_file, update_file, and load_file (the last three would be for adding a flag to specify if you want the file compressed or not.)

The attention to the library is appreciated, but IMHO you're focusing too much on making it standardized rather than what's most effective for a VP tool of any sort. But for editors like POFCS or, ehm, something else :p what you're saying about coding it C/++ standards style make sense.

What I will say is that I have never really liked the >> operators because they seem too ambiguous, which is why you may notice that I've never really used them in any code I've released. eg, if I hand them a string, will they simply store the string? Will they null-terminate it? Will they add an int that tells the string length? Or will it be a short? I couldn't tell you without experimenting...all I can say is that if you were to take the output, and then use the << operator, you'd get the input to the >> operator. Assuming you used the same type, of course, otherwise all bets are off.

Edit: The most confusing thing is the add_file, update_file, and load_file trio, but that actually isn't too bad. The rest of the names are pretty obvious - extract_file extracts a file and build_vp builds a VP. I doubt 'learning' the API would take much more than 5-10 minutes. Using it would take less time, esp. with an example.
« Last Edit: August 31, 2005, 03:58:27 am by 374 »
-C

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Woo, VPMagic 0.2 source release!
I've got my head wrapped around other things for a while so I don't want to get into this conversation too far just yet.  The basic things to be aware of though are:

 - many files that we use now are already very compressed (DDS, JPEG, OGG, ADPCM) so ship textures, most EFFs and the new ANI replacement and many sounds aren't going to compress well anyway.

 - we have to be able to stream out of a VP to allow for music and, later, movies in a VP/CVP.  we need to avoid decompression overhead as much as possible.


I was going to figure out one of two things to settle all of this:

 - per file, type based compression.  We would compress by file and simply avoid the overhead associated with compressing already compressed file formats.

 - sorted data blocks in the CVP.  A CVP would contain two data sections, one compressed and one uncompressed, sorted by type to be most efficient.

You can probably figure out the details on what I was thinking from that.  I need zlib for something else which is one reason that I has chosen that (plus it works well and is portable and has a BSD type license).

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
When you say stream, though, why not do it like so (presumeably something like this would be in extract_file):
Code: [Select]
char* data_buf = new char[chunk_size];
for(int i = 0; i < 0; i += chunk_size;)
{
fread(file_pos + i, chunk_size, 1, sfp);
decompress_chunk(data_buf, chunk_size);
fwrite(i, chunk_size, 1, dfp);
}


Obviously that exact cod won't work, except with a compression ratio of 1:1 :p, but it should get the general gist across.

CFile, I believe, can 'stream' out of VPs already...I never intended my library to replace the FS2 system though (Possibly the only time that may have happened :nervous:), so the fs2_open code would have to be directly modified in any case.

I need to get to wake up tomorrow so I can't look up zlib now...but the basic code snippet above would work for OGG.
-C

 

Offline dizzy

  • 26
    • http://dizzy.roedu.net
Woo, VPMagic 0.2 source release!
Oh, you will have to excuse my energy, you should know by know is "beginner"s energy when everything seems that it can be improved, reworked, redesigned. All this until the person actually starts doing them and the time and energy consumed takes it's toll, so please bare with me :)

Now, yes, I agree, the current build system is quite simple, but the library might develop into something a little bit more complex as such a more powerful build system comes in handy. Other reasons is that many IDEs (such as kdevelop) have built-in support for automake so importing and working with automake based projects in such IDEs is only fun to do heh. Other advantages are those enlisted in the previous reply (distcheck, automatic release tarballs, etc).

Also (as you have noticed already) I give A LOT of importance on coding style. I do NOT consider conding style more important than the code actuall functionality, I consider them EQUAL. That is I never make something that just works (or at least try to) or make something that just looks nice :) I give them both exactly the same importance so I try to make either both of them or none.

About the API, you are right that because I really dont know much about FS specific VP file needs my sugestions might look a little bit offtopic. I agree that the main usage for the library whould be FS and then this tools, packe/unpacker and the tool you said POFCS.  I will need to investigate more into this (or maybe someone else can say that already knows it) what are the needs of reading/writting to VP files from this programs ?

About ANSI C++ I always compile my programs with "-Wall -std=c++98 -pedantic" :). Now the <<, >> operators are for formatted input/output of some data types into ASCII encoding. For unformatted (writting/reading chunks of bytes) one has other specific methods in iostreams. Speaking exactly about VP, because we don't store ASCII formatted integers or other binary data then the fact that a iostream interface to VP files provides also access with << and >> is kinda useless :) But still there are other advantages (besides that people can reuse their knowledge on iostreams API to use it on VP files). For example one principle I apply in C++ programming is "resource aquisition is initialization", this basically means that if you are aquiring some resource (memory allocation, locking a mutex, opening a file) then you should generally do it by the initialization of an object (make wrappers for your resources if you dont have that). In the iostreams case it means that always open files as ifstream or whatever because in that case when you exit the context of where the ifstream object is defined, then automatically the file is closed and the resource is released. This is actually A LOT MORE important that it sounds at the first glance. To give a specific code example:

in C/C++:

open_vp(filename)
{
FILE *fd = fopen(filename);
if (!fd) return -1;

if (readheader(fd)) {
    fclose(fd);
    return -1;
}

if (readindex(fd)) {
    fclose(fd);
    return -1;
}

return 0;
}

This is code duplication and error prone. The way I would do it in C++ is:

open_vp(filename)
{
    ifstream fd(filename);

   if (!fd || readheader(fd) || readindex(fd)) return -1;

   return 0;
}

That's ALL! Because ifstream destructor will be called when exiting the stack context and the file gets closed. In general this principle is very useful and I consider it one of the most important advantages of C++ (if I couldn't do this then I would probably not program at all in C++ :)).

Anyways, first I have to check out exactly what needs does FS and the tools have on VP files and see if it's really worth it the new API access method.

 

Offline dizzy

  • 26
    • http://dizzy.roedu.net
Woo, VPMagic 0.2 source release!
About streaming, one other solution to reduce decompression latency is to have transparent "readahead decompression" and buffering in the library. You cycle on calling vp_read() but this in turn uses a buffer of some size and decompresses only when that buffer is overread (it will decompress again a chunk for that buffer). Also to further reduce the decompression latency I would also add a block buffer layer in the VP library to cache on a LRU policy decompressed blocks :)

 

Offline WMCoolmon

  • Purveyor of space crack
  • 213
Woo, VPMagic 0.2 source release!
Look in the modding forum for the new POFCS stuff. Kazan has his own VP library (there are at least five different sets of VP functions running around in various places on the web, go figure). However POFCS is apparently online on the sourceforge CVS page.

Also, one idea I was throwing around at one time was 'fragmented' VP files...these would be VP files that weren't necessarily 100% filedata, and might have some empty/unused space as files shrunk or were moved to the end because they grew and there wasn't any empty spaces large enough for their new size. The benefit would be that one could quickly modify VP files. (No need to move all the file data after a fie was modified). However, I ditched the idea because it could spur the development of crappily inefficient mod/campaign releases because nobody bothered to defrag them - and if someone's modifying a VP that much, the data probably shouldn't be in one in the first place.

Might come in handy for something though.
« Last Edit: August 31, 2005, 04:24:07 am by 374 »
-C

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Woo, VPMagic 0.2 source release!
Quote
Originally posted by WMCoolmon
Obviously that exact cod won't work, except with a compression ratio of 1:1 :p, but it should get the general gist across.

I'm not saying that it can't work like that, just that it shouldn't.  I'm trying to avoid the overhead of decompression and wasteful nature of compressing compressed files.  That's one of the reasons that I've said before that I didn't like the idea of a compressed VP format.  With a game like this and with it's memory requirements, anywhere we can save is a good thing.

Quote
CFile, I believe, can 'stream' out of VPs already...I never intended my library to replace the FS2 system though (Possibly the only time that may have happened :nervous:), so the fs2_open code would have to be directly modified in any case.

Yes, but then it's easy since it's really just streaming off of the disk.  The decompression will all be built into CFILE so it will be completely transparent, just like using a file in a VP is now.  I don't want to take the easy road on this one though.  I want something that works the best and is easy to package.  If you've got something like TBP with a 650 meg VP it can take a while to (un)compress that thing.  As more already compressed file formats get stored in there the time vs. ratio doesn't add up anymore.

And I was going to rewrite CFILE to do this.  I just wanted to mention that it was going to be done so that your lib could be ready for the change and not need a complete rewrite to fit this capability in.

 

Offline dizzy

  • 26
    • http://dizzy.roedu.net
Woo, VPMagic 0.2 source release!
If you are going to compress the whole data area (to get the maximum compression ratio) you will have problems with selectively reading some files. That is getting to the point where some specific file data begins. Before thinking of any solutions to make this fast I think we should first check out the algorithm as I think it is "a block compression" algorithm and as such there might be some properties of it that we can use the speed up seeking for a specific byte in the compressed data stream.

For reference I think we should read this: http://www.gzip.org/zlib/rfc-deflate.html

 

Offline Kazan

  • PCS2 Wizard
  • 212
  • Soul lives in the Mountains
    • http://alliance.sourceforge.net
Woo, VPMagic 0.2 source release!
[my comments on this thread]

compressed VPs are a complete and total bad idea: the purpose of the VP format is to save time and space on disk access because you can cache it's FAT (speed++) and it's an agreggation of a bunch of smaller files it ha less FS overhead --- compression would more than completely destroy the speed advantage

replace-changed-file on the fly would be more than slightly dangerous -- i could easily code this: the thing is I don't care to because it the only time you could do this [safely] is when the file is the same

my VPCS 2.x should compile and run on linux just fine - just like PCS 2.x

that was most of my purpose is starting the 2.x branches in both applications.  Once they're both done i'm going to pack'em togeather (i'll probably add some features to VPCS 2.x) in one installer for windows and ship em togeather.

I need someone to prep autoconf/automake/etc for linux
PCS2 2.0.3 | POF CS2 wiki page | Important PCS2 Threads | PCS2 Mantis

"The Mountains are calling, and I must go" - John Muir