[chief1983]

Also, it's not just a matter of disk space, but maintaining bx files themselves has always been kind of a pain from a mod management standpoint, etc.

[mjn.mixael]

Also, it's not just a matter of disk space, but maintaining bx files themselves has always been kind of a pain from a mod management standpoint, etc.

Indeed it has... Nothing is more annoying than releasing something only to realize you forgot to include the bx for that last minute update to a complex pof...

[Bobboau]

I remember writing the code that generated those index buffers(don't think I wrote them to file), I remember thinking to myself "this is a ****ty solution, I'll come back to it when it becomes a problem". is this new solution to do a binary search or something? it really should probably be using a hash map

[AdmiralRalwood]

I finally got around to doing a test of this (using -noibx so I didn't have to perform surgery on the MediaVPs' VP files) and found that loading the Arcadia in the F3 lab in a release build takes my system ~22 seconds without this patch, and ~2 seconds with. This is also how much time it took when I removed -noibx, so (anecdotally-speaking) it would seem that cache files may have just become irrelevant.

[Swifty]

Well, looks like this is definitely going into trunk as soon as possible. Followed by a dismantling of cache file reading/writing in the model loading code.

[z64555]

Anybody got an old system to try this on (Goober)? Curious if the affects are the same on different hardware...

[Goober5000]

I'm happy to test this out, but you caught me in the middle of a move.  All of my stuff, including my computer hardware, is piled in the center of the room while the walls get painted. I'm currently restricted to just my laptop.

[niffiwan]

Here's some more stats with a slightly old CPU running Linux; the new code is awesome at cutting down the .bx file generation times!  Using .bx files still seems to provide a small advantage in most situations on my PC.


(Note that times are rounded to the nearest second due to the inherent inaccuracy in my use of a stopwatch for timing)
(Also note that I made a few trivial changes to the patch to let it compile with gcc, and remove some warnings in 2d.h that were being spewed everywhere)

[AdmiralRalwood]

Well, looks like this is definitely going into trunk as soon as possible. Followed by a dismantling of cache file reading/writing in the model loading code.

I'd say a difference in loading times from 10 seconds to 8 seconds is enough to be worth leaving .bx files in the engine (and if some systems get better performance from not reading off the drive, they can always use -noibx).

[The E]

In the above chart, the difference was only two seconds though. I do not think that's significant, or even particukarky noticeable.

[Dragon]

The difference is more pronounced on debug (and when debugging a mod, you would spend quite some time running that), not to mention that between 8 and 10 seconds, it's still a 20% difference. I'd imagine that more complex models would make this noticeable. Since the flag to disable cache files is there, I'd say, the code should be left in. Mods will be free to discontinue distributing .bx files if they want, and users can just disable their generation if they're short on HD space.

[chief1983]

Just out of curiosity, is the difference on debug builds perhaps due to just the extra code solely relating to debugging?  Logging, etc?  If so, it doesn't really mean the algorithm is much faster, just that we've bogged down the old path.

[The E]

4 seconds lost on debug. 2 seconds lost on release. On one of the most complex missions. No, dragon. There really is no good reason why we should keep cache files around.

[Fury]

Or reverse the existing noibx flag to ibx which enables the feature rather than disabling it. That way people have the option to enable it, rather than it defaulting to enabled. You could even hide it under troubleshooting.

[Dragon]

4 seconds lost on debug. 2 seconds lost on release. On one of the most complex missions. No, dragon. There really is no good reason why we should keep cache files around.

Look at it another way. Leaving the cache in increases the performance by 20%-24%. Enough to keep them, IMO, regardless of how the absolute figures. How long do you expect Icarus to remain the most complex mission around? In the future, somebody is almost certainly going to come and top it (maybe even BP itself). Thinking in absolute values is much less future-proof than percentages.

[Swifty]

Time doesn't increase if missions get more complex. It only increases as a function of number of vertices in a model. The time function is logarthmic so a crap load of small models is really no problem compared to one really large model like the Theseus. Most models are expected to be well optimized anyway, most of them should be staying under 65,000 verts in order to keep the index buffers small and fast for the post-transform cache on the GPU. If you need to go over 65,000, there should only be at most like four models like that in a mission or else you're going to suffer in-game performance problems.

So no, average case, the load times will stay the same or be even faster. If your load time is slightly longer by a few seconds, well, you already have bigger problems on your hand in-game.

[The E]

There's a point you're missing, Dragon, which I brought up earlier in my testing: This load time increase is extremely variable. It depends a lot on how fast your HDD is. On slow drives, the time needed to load buffer files from disc is longer than the time needed to generate those same buffers from scratch.

This is why I keep arguing for a removal of buffer files: In absolute terms, we're talking about a couple seconds of loss at maximum. That's barely noticeable; if we had slipped this in without telling you about it, you wouldn't have noticed. You are, in my opinion, making a lot of noise over nothing.

Regarding mission complexity, here's another quick test performed on my system. BP massive battle, using a bunch of rather large models. With cache, it takes 33 seconds to load. Without, 27 seconds. In other words, in a situation where I would load the same mission repeatedly, a combination of read caching by the OS and the speedup here makes the whole thing faster without the cache files.

Please, run your own tests. Make your own experiments. But do NOT talk to me about how big a deal 2 seconds are.

[niffiwan]

Yup, I should note that in my tests I have FSO on a SSD, Diaspora on a 7200rpm SAS HDD, and Linux caches anything on the filesystem into spare RAM after its 1st read (Windows probably does this as well? But I haven't tested/confirmed it). Or in other words, my tests had removed nearly all influence of disk access on the results.

e.g. 1st FSO start after flushing disk cache: ~15 secs; start FSO with disk cache intact, ~10 secs, restart mission without restarting FSO, ~7 secs (IOW disk access times are a bigger factor than the patches changes)

[AdmiralRalwood]

Yup, I should note that in my tests I have FSO on a SSD, Diaspora on a 7200rpm SAS HDD, and Linux caches anything on the filesystem into spare RAM after its 1st read (Windows probably does this as well? But I haven't tested/confirmed it). Or in other words, my tests had removed nearly all influence of disk access on the results.

It's possible my system was doing this as well; I had a longer load on my first read with the cache files than I did on subsequent reads with the cache files; I dismissed it at first as unreproducible behavior, but in hindsight, the OS was probably caching the cache files, causing identical (or near-identical) performance with the -noibx version with the patch.

In that case, the performance increase on my HDD under NTFS is severely in favour of getting rid of cache files; ~45 seconds versus ~19 seconds loading Icarus, and ~5.5 seconds versus ~2 seconds loading the Arcadia through the F3 lab.

[Dragon]

This load time increase is extremely variable. It depends a lot on how fast your HDD is. On slow drives, the time needed to load buffer files from disc is longer than the time needed to generate those same buffers from scratch.

I think that this is actually a good argument for keeping the cache and using the -noibx flag to remove it. That way, the user will be able to optimize performance himself. If you throw out the system, it will increase for some and drop for others. We already have a way of disabling cache files, I don't think this needs to be messed with.