[Huggybaby]

I read a post yesterday regarding pofspew.

It got me thinking because I'm at the point in writing my guide to itemize the different ways of optimizing game performance, and it seems that the first time an object is encountered the computer has to slow down to generate the ibx file, which is not an original idea I've learned. I haven't been able to prove that pre-generating ibx's is actually a performance enhancement, though it seems to be on paper.

To get a handle on the subject, I pofspewed every mod to see what I could see. I noticed that the cache folders I was deleting sometimes contained more ibx files than the ones I spewed.

I think I understand how it works, but here's the summary:

NoMod=176 in the FS2 cache
mediavps=94
FSPORT=40
Derelict=1
Homesick=2
Inferno=148
JAD=0
Sol a History=5
Sync=3
Transcend=5

The interesting part is about Deus Ex Machina. If I have all 176 files in FS2 cache, then DEM generates only one  (cargo2t-01). But if I have no files in FS2 cache, then I end up with 176 files in the DEM cache. I wonder where the 173'd went? I'll figure it out, but maybe somebody could save me some time?

It seems that if a mod doesn't find a shared object in the FS2 cache, it puts it in its own cache instead.

Also, the WIKI didn't have an entry for -pofspew, so I added one:

-pofspew
Reads all objects in the pof file of the selected mod, and creates an ibx cache file for each object in the data/cache folder for that mod. Details for each object are output to a pofspew.txt file created in the data folder. Otherwise ibx files are created in game as each object is encountered.

I'm quite sure someone with more experience can do better, so please have pity. There just wasn't one before.

Lastly, running pofspew generates a multi.log file, I'm not sure why.

[taylor]

Actually, the game doesn't slowdown to generate the IBX file, it slows down when generating the index buffer data needed to render models in HTL mode.  That data then is just saved to disk, in the IBX file, so that the game can reuse what it already created rather then having to generate it every time.  The data has to be generated either way, but the IBX files just provide a cache to pull from later rather than having to continuously generate the data.  This is where the performance enhancement comes in, just when the model is loaded.  Since some models can take several minutes (some up to 8-9 minutes) to generate the index buffer data, having it already stored in a file saves you from this wait, and a model that would have taken minutes to load will now only take a couple of seconds.  That is why IBX files were created in the first place, because it sucked having to wait 10-15 minutes to load each and every mission you played.

IBX files are used based on the CRC of the model in question.  When the game loads a POF it will look for an IBX file, and if it finds one it will first verify that it's a valid IBX file and then that the CRC of the POF you are trying to load matches the CRC stored in the IBX file.  If they match then it will use that IBX file, but if they don't match then it will erase the IBX file (if it's not in a VP) and generate a new one.

IBX files are designed to be generated once and then kept in VPs for later use.  Every mod including a POF should also include a corresponding IBX.  Few mods are actually doing this though.  IBX files will always be written to the root[0] cache folder, which means that they are saved in data/cache/ of your primary -mod listing.  The thing to do is just to begin by erasing all IBX files that you find in all folders.  Then, using no mod, run the game with -pofspew.  You will now have all of the retail IBX files in data/cache/ and then you create a VP out of them and store it in the main game directory with the rest of the retail VPs.  Now you will never have to generate the retail IBX files again, and so they won't ever show up in mod folders unless the POF is different.  Again make sure that you don't have any IBX files outside of a VP, then you switch to using a mod, like mediavps, and again run with -pofspew, package the mediavps/data/cache/ IBX files in a VP and keep that VP in the MediaVPs folder.  Do that for every mod and you won't have to worry about the game generating IBX files again. 


And -pofspew doesn't actually create IBX files by design, it's more of a by-product of the process.  What -pofspew does is go through and find every POF that it can, load them up, and dump model stats out to a file (data/pofspew.txt).  It's just that through this action any files which don't have an IBX file will end up getting one, not by design, just as a part of the model loading process.

[karajorma]

I'm so glad I noticed that Taylor was looking at this topic before I started writing my explaination

[Huggybaby]

You got me again taylor. I'm going to have to read your awesome explanation numerous times. To tell the truth I didn't sleep last night and if I had it would still be a challenge. But don't ever talk down to me please, keep giving me the programmer goodness! This forum definitely has a learning curve, so be it.

I don't know how you make such great posts over and over. (I've read your explanations of "Why Theora?" in more than one place, which I've linked in my guide.) Soon I, or someone, will make an FAQ of all your answers which will leave you more time to program, I promise.

But back on topic, are you saying that ibx's are generated at the beginning of each mission, and not in game?

(And maybe my PC is faster than I thought, because doing the entire spew for FS2 took maybe 15 seconds.)

<edit> And now that you've laid it out, maybe other programmer/mod makers will take heed of your advice. I did understand that part!

<edit> So where did cargo2t-01 come from?

[taylor]

But back on topic, are you saying that ibx's are generated at the beginning of each mission, and not in game?

Correct.  They are generated, if needed, whenever a model is loaded.  This happens in one of 5 ways: 1) when you view a model in the Lab, 2) when you view a model in the techroom, 3) when you use -pofspew, 4) during mission load, or 5) when using FRED and you place a model in a mission which isn't already there.

And maybe my PC is faster than I thought, because doing the entire spew for FS2 took maybe 15 seconds.

Yeah, the retail models, and many of the current HTL models, will be pretty quick.  Something like the hi-poly Fenris or Lucifer can take a couple of minutes.  The big issue is when I add IBX² support for 3.6.10 and above.  The current IBX files can work with any model which doesn't exceed 65,000 verts per submodel.  The new IBX files will initially support just over 2 billion verts per submodel, and has the capability to double that without breaking file format.  Models that complex could take over an hour to generate index buffer data for, even on a really fast computer.

The apparent hang when the index buffer data is being generated is something of an issue though, and always has been.  I have tried to create a popup or some other indicator to let you know that the game is actually doing something and isn't locked up.  Unfortunately all attempts at that have failed and the result just wasn't very user friendly, or was such nasty code that I couldn't even look at it anymore.  I do have a new idea though, generating index buffer data in a new thread.  That will make it work great for SMP/dual-core systems, and also allow a crude time-to-complete indicator to be presented to the user.

And before anyone says anything about the new IBX file format, just know that both are fully support.  The format change is dynamic, and the new IBX² files will only be created if the model actually requires it.  Otherwise the normal version 1 IBX files will be used.  All current files will still be support, and the game will still generate those old format IBX files for 99.9% of all models.  It will just be for that 0.1% of models which need more than 65,000 verts per submodel that will end up using the new file format.

<edit> So where did cargo2t-01 come from?

More than likely, it was the only POF which was actually different from what you already had IBX files for.  If the POF is different in any way then the CRC test won't pass, even if only a minor change was made to the POF.  There are several mods which include slightly modified POFs, but they look exactly the same, or just have slightly different textures.  As far as the IBX code is concerned though, they are two very different models and requires a new IBX file.

[karajorma]

<edit> And now that you've laid it out, maybe other programmer/mod makers will take heed of your advice. I did understand that part!

BtRL already were doing this. We didn't much fancy dealing with hundreds of complaints from people who thought the game had crashed when it took 5 minutes to generate the pof for Scar's Asteroid.

I've had a seperate cache VP for over a year now but until recently I've never had anywhere to upload it to and it wasn't worth doing once I did cause several of them will change when the 3.6.9 Official VPs come out.

[Huggybaby]

Models that complex could take over an hour to generate index buffer data for, even on a really fast computer.

I've had a seperate cache VP for over a year now but until recently I've never had anywhere to upload it

Never fear, they can be pre-generated and included with the intaller(s). That's why I started looking into it in the first place, to decide if I should include them as part of my pre-packaged downloads,or go with the ultra clean stripped down version.

Users will easily accept a huge download, believe me. They crave the best at any cost. They do NOT mind downloading/torrenting something even if it takes all day, even 24 hours plus. They DO mind waiting an hour for their new hard won game to start! Now I know I'm on the right track.

You guys don't worry about the ibx's. If you leave them out of your package I'll have to generate them and include them anyway, now I know that.

Rock on! It's the quality that matters. I know this for a fact from other projects on which I've been involved.

The small minority that can't or won't handle it always have the Retail install.

Huggybaby