[JC Denton]

Since I heard a few of the TBP staffers are worried about someone tweaking the POF code and rendering all their models obsolete, I have an idea to possibly avoid this problem.  It's kinda hard to explain, but in a nutshell, it involves adding a second filetype to the engine.

This second type might be called XOF (eXtended Object Format) (maybe?), and all work in POF recoding should be centralized into this one add-in.  This allows the engine to both understand the original POF format, plus allows modelers to make XOF models to take advantage of any new features the XOF code offers, whatever they may be.  Plus, if it's possible, let the XOF code be backwards compatible with older formats: if advanced features are available in the code but are not present in the model, it won't break anything.

Probably a few fixes for the current POF code (and not needing any funky new file formats) would be allowing textures to be something other than a PCX file.  This can be immediately useful for making animated textures in ANI format and allowing them to be mapped onto the models, as well as making "glowpoints" which are basically the navigation lights we saw in Homeworld.  Simple add-in to the TBL file can specify what texture to use, how big it is, and if it blinks and how fast it does so.  Also either ANI or PCX compatible.

If you're thinking that it might be just as easy to do the backwards-compatibility on the POF files rather than add a new file type, well, you're probably right, but this way allows us to keep a working object format while tweaking the XOF code.  Plus it'll give us a way to differentiate between old and new models, keeping compatibility at a maximum.

[Nico]

would be better to just make the thing backward compatible. So the new pof have added data, and the old data can still be readable. Btw, why not using other format than ani for animated maps? anything would be better, ani are too big. I'm frightened when i think of 512*512 ani maps

[_argv[-1]]

Originally posted by venom2506
would be better to just make the thing backward compatible. So the new pof have added data, and the old data can still be readable. Btw, why not using other format than ani for animated maps? anything would be better, ani are too big. I'm frightened when i think of 512*512 ani maps

How about MNG? Nicely compressible, and code already exists to decode it. Oh, and it's standard, too. Woot! Maybe PNG should also be supported for non-animated textures, since it compresses well. JNG (linked to on the MNG page) would be good, too, since that's JPEG with alpha etc.

Using *NG would also be useful for the user interfaces graphics (briefings, most notably), because of alpha compositing. As you may have noticed, some of the command briefing animations have areas that are supposed to be transparent, but just look exactly like the background picture behind them. This looks strange in 1024x768. With alpha compositing, you have real transparency. You can also antialias the HUD graphics with alpha compositing.

[JC Denton]

Well, it's not like we're going to have the hi-res Lucifer maps from FS1 all animated or anything.  Most animations would be for small areas, and even if they're going to cover a large percentage of the model, it'd be just as easy to make a small hi-res map and tile it.

And as for the *NG files, they look like a promising alternative to ANI files.  Although I didn't know that FS2 already had the code to interpret the formats.

If it does, then great.  If not, I don't know how much of a headache it would be to put it in.  But then, I'm no programmer, and quite clueless when it comes to C++.

[_argv[-1]]

Originally posted by JC Denton
And as for the *NG files, they look like a promising alternative to ANI files.  Although I didn't know that FS2 already had the code to interpret the formats.

It doesn't. I meant that *NG libraries exist, so you just need to use them. Using these instead of PCX/ANI should just be a matter of finding the image loading routines, where the PCX/ANI files get loaded, and bolting in some *NG loading code.

[Shrike]

Interesting.

I don't know what people will be using a lot of animated maps for, really there's not that much where you can use them.  But hey, it's always nice to have.

[mikhael]

Originally posted by _argv[-1]


How about MNG? Nicely compressible, and code already exists to decode it. Oh, and it's standard, too. Woot! Maybe PNG should also be supported for non-animated textures, since it compresses well. JNG (linked to on the MNG page) would be good, too, since that's JPEG with alpha etc.

Using *NG would also be useful for the user interfaces graphics (briefings, most notably), because of alpha compositing. As you may have noticed, some of the command briefing animations have areas that are supposed to be transparent, but just look exactly like the background picture behind them. This looks strange in 1024x768. With alpha compositing, you have real transparency. You can also antialias the HUD graphics with alpha compositing.

BEST. IDEA. EVAR.

[neimad]

Just remember that every compression routine is a two sided sword.

You may decrease your file's by hundreds of kilobytes (or more) each time, but now each and *every* one of those images has to be decompressed prior to use -- this *could* be very costly to overall performance.

[Unknown Target]

How about just getting the pofs to use the standardized .obj format?

[_argv[-1]]

Originally posted by neimad
Just remember that every compression routine is a two sided sword.

You may decrease your file's by hundreds of kilobytes (or more) each time, but now each and *every* one of those images has to be decompressed prior to use -- this *could* be very costly to overall performance.

The biggest problem here is animations. The ANI format is huge! If it's possible to decompress each frame in the number of milliseconds between frames, then only one frame would need to be decompressed at a time. This would make long animations more manageable.

The other reason for compression is not so much to save disk space, but to save download time. Even if the computer's memory, disk, and CPU handle the decompressed image, its Internet connection may not!

[_argv[-1]]

Originally posted by Unkown Target
How about just getting the pofs to use the standardized .obj format?

If we're going to go all out on using standard formats, how about using Ogg Vorbis for music, and ZIP in place of VP?

And yes, both of these have been done in games. Ogg Vorbis is used in Operation Flashpoint and probably others; ZIP is used as the package format in the Quake 3 engine (where its file extension is PK3).

Standards rule.

[neimad]

Originally posted by _argv[-1]
The biggest problem here is animations. The ANI format is huge! If it's possible to decompress each frame in the number of milliseconds between frames, then only one frame would need to be decompressed at a time. This would make long animations more manageable.

Data compression/decompression is almost always a processor intensive act.  So, doing it within a tight loop (as would be required for an animation) would require highly optimised code to avoid bottlenecks -- this in it's self may take many months to do.

The other reason for compression is not so much to save disk space, but to save download time. Even if the computer's memory, disk, and CPU handle the decompressed image, its Internet connection may not![/B]

A litte fact (that you no-doubt already know) about compression is that most routines don't work (very well) on already compressed data.  Now since everyone already compresses there "final product" before making it available on the internet, then you may actually end up with a bigger download if you used compressed images.

[mikhael]

Originally posted by _argv[-1]


If we're going to go all out on using standard formats, how about using Ogg Vorbis for music, and ZIP in place of VP?

And yes, both of these have been done in games. Ogg Vorbis is used in Operation Flashpoint and probably others; ZIP is used as the package format in the Quake 3 engine (where its file extension is PK3).

Standards rule.

Independence War 2 stores the entire resource collection in a standard ZIP archive and pulls things out of it on the fly. The only thing that takes a bit of time is the initial preload, which loads about fifteen or so standard avatars into memory. Very fast.

[neimad]

http://www.descent-freespace.com/ddn/specs/pof/

The above link pretty much says it all as far as I'm concerned.  Each .pof includes a "version number" field which identifies whether it was designed for FS1 or FS2.

Now, just as long as the "file header" is kept intact, there is no reason why you couldn't completely redesign the internal file-format, as long as you changed the version number too -- that way neither FS1, nor FS2 would attempt to parse it.

[_argv[-1]]

Originally posted by neimad
Data compression/decompression is almost always a processor intensive act.  So, doing it within a tight loop (as would be required for an animation) would require highly optimised code to avoid bottlenecks -- this in it's self may take many months to do.

Agreed. A better solution is to decompress the animation at mission load time, as is done for non-animated textures. Because not all computers have enough main/texture memory for an ultra-high-resolution animated texture, it will also be necessary to have lower-resolution animations for such systems. It will have to be decided when this scale-down operation should be performed -- at mission load time, by the texture artist prior to publishing the texture, etc.

A litte fact (that you no-doubt already know) about compression is that most routines don't work (very well) on already compressed data.  Now since everyone already compresses there "final product" before making it available on the internet, then you may actually end up with a bigger download if you used compressed images.

A little fact (that you no-doubt already know) about compression is that specialized compression algorithms (JPEG, MP3, *NG, etc) compress far better than generic compression algorithms. Also, the ZIP format allows certain files in the ZIP file to be stored without the usual ZIP compression, thus preventing the expansion problem you mention.

[neimad]

Originally posted by _argv[-1]
A little fact (that you no-doubt already know) about compression is that specialized compression algorithms (JPEG, MP3, *NG, etc) compress far better than generic compression algorithms. Also, the ZIP format allows certain files in the ZIP file to be stored without the usual ZIP compression, thus preventing the expansion problem you mention.

I was under the impression that *NG used zlib compression, in which case it's just the same as zip.

At the end of the day, I personally don't mind if the files are compressed individually, or as a whole; just as long as I can still play the game on my old Athlon 650MHz/256 MB SDRAM/GF256 32MB SDR.

[_argv[-1]]

Originally posted by neimad
I was under the impression that *NG used zlib compression, in which case it's just the same as zip.

Correct. As you may know, zlib decompression is quite fast on modern computers.

[EdrickV]

One interesting note about transfering data by modems: Modems themselves try to compress the data they are trying to transfer. If that data is already compressed, they won't likely be able to compress it much. If it's not compressed, then the modems may be able to transfer the file faster then a compressed file of the same size. I've actually gotten sustaned download rates of 10k per second with a 56k modem at the end of big DIVX files. Not because the modems suddenly started sending/recieving data faster, but because the data suddenly became very very easy to compress. It always happens at the end of the file, where I assume things like the index (and maybe keyframe data) is. I did a little test. I uploaded a text file and a zip file of about the same size (the copies on my drive were 20 bytes different) and then downloaded them with GetRight. Somehow the text file got smaller on the server, but due to cluster size it still took up the same amount of space on my computer when I downloaded it. Here are the results:

Zip file:
113.9k
Elapsed: 0:00:30 (Ave. 3.8k/sec)

Text file:
112.3k
Elapsed: 0:00:11 (Ave. 10.2k/sec)

[CP5670]

10.2k/sec

You know, that is simply incredible for a 56k. In my 56k days the best I could get was around 4.5k from good servers. I need to try out something like this.

[EdrickV]

Originally posted by CP5670


You know, that is simply incredible for a 56k. In my 56k days the best I could get was around 4.5k from good servers. I need to try out something like this.

It's also extremely abnormal.  Normal downloads range from 3k to 4.5k. Sometimes up to about 5k. Depending on my connection which varies a lot since I use AOL.