From my journey into hell, I return with a gift.
Universal truth keeps me up at night. Or at least has made me stay up entirely too late. And no not the one you'd expect from most people, I mean the first Universal Truth in Age of Aquarius. This mission has made me burn weekends on learning new code profiling tools, teasing apart the inner workings of FSO, and clogging up #SCP on discord with my obsessed ramblings and desperate scrounging for framerate.
This big damn battle is just so densely packed with stuff. The battlegroup in close formation, endless the hordes of Shivans pouring in constantly, so many friendly wings... it's packed to the brim. My computer has a little trouble with it to begin with even so many years on.
But then, since it's me, Warmachine of course comes into the picture. I promise that this is about BP core too, but I went down this road because of Warmachine. The mod is a stuff multiplier, and it only gets moreso over time. Every few months Talhydras takes a gun and makes it fire a seven-beam burst instead of a single laser, or I take a Shivan missile and make it a swarm or cluster weapon of at least 8 submissiles.
So after several rounds of this unchecked proliferation I come back to play some Universal Truth and see it in all it's jam-packed glory, and my computer just chokes on it. 14 FPS within a minute of the mission starting, and not coming significantly back from it. It's tragic. I console myself that it's not that much better in standard BPC, but why? Why is this mission so slow, when the WIP missions I've made that are also too full of stuff run fine? Admittedly they're still not quite AS full as this, but I hardly ever notice a blip in them.
So on and off over the last free or four months I poke at the problem. It lies dormant at times, then someone will troubleshoot another problem on discord and remind me of my quest.
And some of you might be about to say "maybe try less stuff", to which I say
At Warmachine we must not allow a laser-spam gap!
In this quest to have everything for free a few opportunities for engine optimization have been surfaced, and a few mysteries remain elusive, but one mystery has been unraveled that most likely has been draining FPS out of universal truth for everyone that played it since it was first released.
At some point I started using the json trace profiling tool in the engine, helped along by some #scp folks and later aided greatly by Rear Admiral Tarsus' very expert guidance. This lets you see roughly how much time the engine takes on each part of the process of making a new frame, and how those fit together. Collision detection is one of the bigger drains on UT performance, which makes some sense anyway. If you double the amount of stuff in a mission, the amount of time needed to check for collisions more than doubles, so the proliferation of STUFF has to catch up sometime. But Tarsus noticed something peculiar. One single pair of objects were eating up hundreds to thousands of times more processing time than any other pair did. That one collision check accounted for the majority of time spent in collision detection.
Something obviously had to be wrong, though it wasn't clear for a while yet what. Eventually I started stepping through to figure out which pair of things this was, and it turned out to be the Orestes and the Fortune. Which makes no sense, right? Look at these two.
They're close enough that it's no shock the engine thinks about the possibility of them colliding, but to actually have to go into the final pass of collision was surprising, and why they would take many times longer than the rest of the mission put together
even when things really started to hit the fan was unfathomable.
At this point I could have stopped, put those ships in a collision group so they were never checked against each other, and walked away. In theory anyway. In practice the question WHY was not one I could stop thinking about.
It was probably Asteroth that pointed out the next piece of the puzzle, one that has caused some long lived and bizarre mod bugs elsewhere before. Because checking for a collision between two full meshes can be quite expensive, Freespace handles every ship-ship collision by checking the larger ship's model against a sphere representing the smaller ship.
This sphere doesn't use the model's radius as one might assume. Instead the sphere's radius is set by the model's eyepoint, or if there isn't an eyepoint it defaults to half the size of the smallest dimension. This works well enough for fighters bouncing off of capital ships, and it's a fairly sane choice for retail freespace, but when big ships start to interact things get weird.
It turns out that the GTL Anemoi does have an eyepoint defined, in a sensible seeming place, right on the nose. That's good if anyone ever wants to make an Anemoi ram something larger than it, but in this case, well, a picture says it best.
So yes, as far as the engine is concerned these two ships are always colliding through the whole mission. It probably always has been since the first version of Blue Planet, unless someone moved the ships in UT around at some point. Since they're moving at such low speed relative to one another, the damage and force of the impact is so low as to be impossible for the player to notice.
That still didn't fully explain why the collision took so damn long to process, but I decided diving into the collision code to figure out what was going on was likely to take much more time and effort than it was worth. Then half a day later I started doing it anyway.
I can't claim to be an expert in collision detection algorithms in general or FSO's collisions in specific, but after a day or so of bashing my head against it here's what I think is going on. FSO uses a fairly standard algorithm that splits the mesh in half, discards any part that can't be colliding, and then splits the remaining portions each in half again, until it reaches the individual polygons. In the best case, this for an Erebus means around sixteen divisions and then one triangle being checked.
This is pretty far from a best case scenario though. If you have enough geometry being intersected then this method probably takes longer than a simpler one would, which is fine since this should never happen. Either way there's somewhere around 30k triangles in this region of the ship, spread across several sub-objects. That's that's more than 100 times greater than the number of collidable objects then engine will typically allow, for reference.
So finally my brain is free of this puzzle. This isn't a bug, at least not in the engine, just something it was never meant to be good at. I put every GTVA capital ship and the two super-jugs in a collision group so they'll never cause this again, on this mission anyway.
My mission file isn't too interchangeable with BP standard, but the event doing this should drop in quite easily.
$Formula: ( when
+Name: cut out the framedeath
+Repeat Count: 1
A more adept fredder could probably make a version that drops every ship into the group as it arrives, but the few seconds of framedeath collision in the intro doesn't bother me enough for me to figure that out myself right now.
In the end, after this change and a number of engine optimization opportunities that were unconvered along the way(only one implemented by myself, the rest being picked up by the lovely people of the SCP), I've roughly doubled my framerate in Warmachine Universal Truth. A few dips aside it's perfectly playable now. I'm looking into using collision LODs to help reduce the performance impact of the Erebus in general, and maybe some other ships too. I even expect it to have smoothed out a little more with changes submitted since my last test of it a few days ago. Some of those should be in 21.2, some will have to wait for the next stable, but the collision group thing can be done immediately and should help a lot for anyone with a weaker processor.
The moral of the story is that filling space with 130 ships, 300 lasers and missiles, and 45 beams at once is perfectly fine and the consequences of my actions can try to catch me some other day. And also be careful about sticking big ships close together I guess.