[Axem]

So I noticed these longer than normal loading times recently, which was odd because Swifty had improved on loading times semi-recently. I decided to try and create a mod with some very taxing models to produce as large results as possible.

My mod + mission is a simple one with some BP ships + the colossus + the Sathanas, and it gets me loading times of 2 minutes! But a build from May 9, the day after the speedy bx code got added gets me loading times of 16 seconds! Huh! Did something regress?

MageKing/AdmiralRalwood helped me find that the loading times suddenly increased on May 13, when the deferred rendering code got added. I gave him my mod and he gets much quicker load times (around a minute), which I guess isn't as bad but still higher than May 9 (around 20 seconds).

If interested parties could download this mod: https://drive.google.com/file/d/0B_q0tkX7VcSaVVFTbXZXb1hENnM/view?usp=sharing and report how long it takes you to load the mission "complexmodeltest.fs2" with a nightly after May 13th 2015.

These are my results:
https://docs.google.com/spreadsheets/d/1k4tQq2qRc1GZ7_6ueogVG8ne8L7R3NMXC9WCQxpinrs/edit?usp=sharing

You don't need to run it multiple times, I just did it so I could see if something like the CPU or the game was caching info.

I have an i7 860 @ 2.6GHz, my dev FS install is off of a standard WD drive, but at the end I tried it on my non-dev FreeSpace install on a solid state drive and got an improvement of about 30 seconds.

I gave it to Spoon and he reported loading the mission in 2 minutes on a i7 2600k @ 3.4GHz on a WD Caviar Black HD.

[AdmiralRalwood]

MageKing/AdmiralRalwood helped me find that the loading times suddenly increased on May 13, when the deferred rendering code got added. I gave him my mod and he gets much quicker load times (around a minute), which I guess isn't as bad but still higher than May 9 (around 20 seconds).

To be precise, I said "less than a minute" (it was more like 45 seconds).

Also, for comparison purposes, I do not have FS2 installed on a solid state drive and my CPU is an AMD FX 9590.

[Swifty]

I don't think it's the BX code that increased your load times.

What may be increasing your load times is the processing of index buffers. In order to speed up render times, I merge submodel vertices into one gigantic model so we can draw everything using one draw call. If your model has transparencies, I also look through all triangles to see if they have transparent UVs and add them to a separate transparency index buffer.

I didn't think this would increase load times significantly but go figure. Models render a lot faster at least. I'm sure models will load a lot faster if you disable batched model rendering in the Troubleshoot tab. That just will have the cost of having slower framerates in-game.

[Axem]

I ticked the disable batched model rendering in the Troubleshoot tab and load times didn't change at all. Still in the 2 minute range.

[fightermedic]

i noticed somthing similar, no concrete dates here, but from my feelings the loading times some time ago made a huge jump for large models

[m!m]

@Axem: Can you apply the attached patch to a recent master checkout and run a debug build? That will output some timing information to the debug log which might help narrow down where all that time is spent.

[attachment DELETED!! by Strong Bad]

[Swifty]

I stand corrected. It looks like I process for transparent polies for all models regardless if batched model rendering is enabled or not.

The reason why I do this is to make sure all transparent triangles are rendered last when rendering a scene. Otherwise we'll end up with a bunch of incorrect rendering transparencies with the way stuff gets rendered under the new renderer.

If this is unacceptable, I apologize. This just seems to be the best solution at the moment.

[Spoon]

These long load times are not ideal for fredding and playtesting

[Axem]

Okay, I understand that transparency sorting is pretty important... The effect is present in FRED as well. Anytime I want to save a mission with some large ships, takes minutes for FRED to become responsive again. But if nothing can be done, then I guess we'll just have to live with it...

m|m: Here's the imporant-looking lines from the debug log:

Code: [Select]

LOADING: 0.000: <unknown>
LOADING: 0.230: ** ModelLoad: warp.pof **
LOADING: 1.240: ** starting mission_load() **
LOADING: 0.710: ** ModelLoad: GTD_Erebus.pof **
LOADING: 63.046: ** ModelLoad: GTD_Titan.pof **
LOADING: 30.824: ** ModelLoad: corvette3t-03.pof **
LOADING: 16.567: ** ModelLoad: corvette3t-01.pof **
LOADING: 15.348: ** ModelLoad: Sathanas.pof **
LOADING: 20.427: ** ModelLoad: Colossus.pof **
LOADING: 48.716: ** starting game_post_level_init() **
I'm guessing that the LOADING is paired to the model load from before? So the Erebus took 63s to load?

Also full debug log here: http://pastebin.com/Vb8MDYzT

[Swifty]

I can disable the transparency thing for FRED. There also might be some optimizations we can do to get faster load times but I can't guarantee anything.

[Dragon]

Is that possible (and if so, could it help?) to exclude models that do not have any transparent polygons? Big ships (the ones that cause a problem) don't have them very often, from what I remember.

[Swifty]

It's the act of figuring out if a model has transparent polygons is what's making it take so long. It's not just a matter of checking if a texture is transparent either because that would require the code to check 2048x2048 texels.

[The Dagger]

I thought only submodels or textures with the -trans suffix could be transparent. Are those non-mandatory?

[Dragon]

I think that giving diffuse map an alpha channel also works, though that might not be the case anymore. If it was, it'd make things easier.

It's the act of figuring out if a model has transparent polygons is what's making it take so long. It's not just a matter of checking if a texture is transparent either because that would require the code to check 2048x2048 texels.

Can't you check if the diffuse has an alpha channel instead? This would make using proper DDS formats more important, but I think everyone is doing that anyway. If all diffuse textures are DXT1, JPG or PCX, then skip the model.

[Swifty]

Originally with the old model renderer, a model was assumed that root submodel of detail0 would render last. Models would look right for the most part. Transparencies on the detail0 root would render correctly against their own model since we always knew it would render last. But bring other models into the question, whether or not that transparency will look correct was a bit of a non-deterministic factor. Sure, your transparency would look correct against it's own model, but if other models are rendering in the scene and they get behind the transparency, whether or not they show up behind that glass depends on who rendered first.

The new renderer changes things. It purposely saves all submodel meshes being rendered. After that's done, it does a simple sort to produce the optimal order of draw calls to reduce the number of state changes in the graphics driver. Less state changes means faster render times. So, the render order, even among submodels that belong to a same model, is not guaranteed such that detail0 will render last.

My solution was to basically figure out which submodels have transparent polygons so I could once and for all make them render last. And yes, this was because there were so many models that chose to apply transparencies without suffixing them with -trans. I did not want to get complaints about those models not rendering correctly in the new renderer. I chose to apply a solution that basically made it such that transparencies can render correctly without any artist involvement. I figured that would make everyone happy but it looks like there were unintended consequences. Go figure.

[Swifty]

Can't you check if the diffuse has an alpha channel instead? This would make using proper DDS formats more important, but I think everyone is doing that anyway. If all diffuse textures are DXT1, JPG or PCX, then skip the model.

I already do that.

[Axem]

It would appear this is an asset issue then... Apparently a large number of diffuse maps for the TEI ships are DXT3/DXT5 which have alpha maps (but it doesn't look like anything should be transparent...???). As soon as I replace them with a black DXT1 texture, all of a sudden the loading time is cut in half. Taking out the Colossus and Sath (2manytextures2replace) get me a loading time of 16 seconds.

Well, mystery solved I guess...!

(Also Colossus uses DXT3 diffuse textures too...)

Extra edit: Loading time with all diffuse as a DXT1 texture now 24 seconds... much more acceptable now...

[Swifty]

So this might be a cache issue at large here since there's a lot of redirection going from index -> vert -> UV -> bitmap lookup. I'm going to try some additional tests to see if I can figure out if a particular texture map really does have any alpha values less than 0.99. Checking all texels in a texture map is kind of stupid but we do have MIP maps. Maybe checking the 2nd MIP for transparent texels isn't a bad idea. Don't go on reconverting all assets yet, I might have some magic up my sleeve.

[Axem]

Is there a reason to use DXT3/5 on maps that don't have any transparency? It bloats the filesize up a bit compared to DXT1...

[Swifty]

Yeah if you don't have transparency you should ditch the alpha channel. But still, I don't want any existing assets to needlessly have long load times which is why I'm willing to find a programmatic solution. Unless you guys think this is okay now that we've figured it out.