[wookieejedi]

Happy Star Wars Day everyone! All of us at Fate of the Galaxy want to share an update for May the Fourth. A highlight list includes the following:

• Updated FotG to fully remove the need to use text-based command line flags to set any options, now all options most players will use can be set in-game! This includes being able to set joysticks in-game and without requiring a game restart, which is a big QoL feature that allows players to more easily tune their own custom controls iteratively and quickly. As another example, we updated game resolution to work from any selectable resolution option, all the way down to 640x480 (previously the lower resolutions had UI overlap and other breaking issues). Finally, players can also now set graphics options such as shadows, AA, and more in-game through the options menu (though the advanced graphics options do require a game restart to take effect).
• Updated custom pilot options to now be pilot-specific and work across updates to FotG, so this way you can make several different pilots if you or others are playing on the same computer and account, and these custom changes will also track with your individual pilot flight statistics. To help new players we also cleaned and tuned the ‘Help Tips’ that are shown to new players when the game starts. Finally, related to pilot options, we updated saving and discarding options to work consistently. Before, pressing escape would save some options while discarding others. The now consistent behavior is the result of FSO updates.
• Updated FotG to work more seamlessly with VR. Now, all players have to do is click the ‘Play in VR’ button and the game will auto-set everything needed and easily launch in VR mode. Players will also have the option to tune their VR resolution in-game if they want to customize their screens further.
• Updated the HUD screen configuration with multiple new FSO cleanups and enhancements, including the ability to have the HUD Configuration reflect exactly what will be seen on the HUD, instead of the old FSO way of just using hard coded image files, which is very useful for players who like to customize their HUDs.
• Updated the N-1 model with many quality-of-life enhancements and improvements, including making the ship exterior more detailed for both the model and the texture, which enhances the immersion, especially in pirate/smuggler campaigns where the N-1 is prominently used by the player.
• Updated the first 3 interconnected campaigns with additional cleanup and polish, all in an effort to reach our public released goals.
• Updated Project Alderaan with many fixes, polish, and cleanup updates.
• Updated and fixed many other components with cleanup and fixes, which are too long to list here!

Community Connections
As always, we want to highlight all the terrific work completed by the FSO devs, too. They have continued adding code cleanup, bug fixes, optimizations, and multiple other updates that FotG uses. Just to name a few,  MJN has refined and continues to improve the HUD rendering code and HUD configuration screens, Lafiel has integrated many fixes and enhancements to various FSO systems, and Goober has added multiple enhancements and cleanup with FRED, AI code, and more. In addition, ShivianSPS, Cyborg, and taylor have released a great QoL updated to KnossosNET, which FotG will be releasing on. As always, there’s many other devs adding great fixes, features, and other enhancements, too.

Moving Forward
Finally, thank you all for your sustained enthusiasm and support! We are getting extremely close to releasing FotG publicly--one of the largest remaining milestones is having FSO release 25.0 stable (which is ideally not far away). We simply can’t wait to share the game with you all once those milestones are completed. In the meantime, here is a new screenshot. May the Force be with you!

[Hinchster]

That looks absolutely phenomenal!!

I just cannot wait to play it and see it in all its glory.

Though it might be a bit weird hearing my own voice delivering some of the briefing & debriefings.......

Well done everyone and keep pushing!!

[Sirrush]

Looking forward to this! Been following this mod for years

[wookieejedi]

Thanks! We’re definitely looking forward to FSO 25.0 release and sharing FotG with you all sometime, ideally soon, after that!

[ferlanero]

Hopefully we will see the final version of FotG very soon! It would be the reward for years of eagerly following the development of this game. Thank you for so much effort and dedication!

[wookieejedi]

Bit of behind the scenes, but just wanted to thank all of our testers for all the awesome work they do! Just as an example, their invaluable help has allowed us to hopefully reach an end to the bug hunt saga that begin in January...First, we get reports of FSO flickering during certain conditions, then within days followed by reports of textures getting switched around, the screen sometimes inverting and zooming in, and subsystems not loading properly on ships we know are setup correctly. This deluge occurred in such a relatively short time frame that we first guessed they were perhaps all related, but alas none of them were easy to reproduce and seemingly happened at random times for different testers. Moreover, not all testers were experiencing the same bugs, but almost everyone was getting some kind of bug. The usual culprits were pursued, and debug logs underwent deep scrutiny. Some debug logs showed hundreds of OpenGL Errors, surely this was the cause? On a whim we discovered those error messages were simply the cause of having a third-party screen recorder on, which Discord searching also confirmed to occur on even FS Retail. Perhaps this was the cause all along? Still, to be sure, we developed multiple pull requests to clean-up various FSO code aspects.

Once PRs were merged testing commenced again, yet the bugs still gleeful persisted and mocked our futile attempts to squash them. Efforts to reproduce were in vain, the bugs would disappear into the darkness when restarting FSO and the logs continued to turn up no viable leads. It was time for a more robust approach, we would try bisecting when these bugs appeared, which for FotG was not an easy task as new features were merged continually, and the bugs themselves were so random it would be almost futile to confidently identify when they first started. Still, we embarked on this brute force approach by deploying multiple testing branches built from commits from many moons ago...As expected this effort was ultimately not useful, as even old builds and timeframes revealed some iterations of the bugs. Previously, we had fixed long-standing and hard to reproduce bugs, but this confluence of random manifestations made this bug especially hard to pin down. Equally perplexing was that no other mods were reporting these issues. FotG lives on the cutting edge of FSO builds, but these bugs had been going on for months. At this point we had exhausted our conventional options.

Fortunately, during this time Lafiel had been aware of our plight and was able to coordinate with Limbert, who had been experiencing many of the issue. The plan was to catch one of the issues red-handed while running RenderDoc, though easier said than done given the elusive nature of the bugs. After much collaborative planning and instruction, a bug was finally captured within the program. Digging into the RederDoc results, Lafiel gleaned a root cause was related to FotG's cockpit use of Real-time-texturing. Yet the RenderDoc did not supply the smoking gun we had hoped for. Still, it pointed Lafiel in the right direction, and after many tests he was at least able to reproduce the flickering bug. Not long after a PR followed (https://github.com/scp-fs2open/fs2open.github.com/pull/6767)--but this was not the end. Lafiel correctly deduced there were likely multiple, unrelated bugs at play and the following months proved this hypothesis true.

With the flickering bug now gone, the other graphic errors rose to the occasion, most notably the y-inversion of the screen. This bug proved harder to reproduce than the flickering and Lafiel and other developers were simply never able to have it occur on their own machines. We were once again at a dead end. RenderDoc suggested it was something once again with RTT, but without a reliable reproducible case the chances of fixing it were slim. At this point we had to back up and once again dig through the debug logs and compare systems. Interestingly, Lafiel, who had never managed to hit the y-inversion had nearly the same systems as Kestrellius, who did run into that bug. Ultimately the debug log did reveal a key difference--not in the mission or load events but in the game resolution. It turns out that everyone who experienced this bug had one aspect in common, they were running at a resolution that was not 1920x1080. Out of curiosity, we tested different resolutions and after a bit of exploration we were able to finally reproduce the issue (it had been dependent on resolution all along). With a reproducible case in hand, we brute forced a bisect, and found the error appeared only after updating to scalable fonts. This was the key, as armed with that lead and a way to trigger the bug, Lafiel quickly zeroed in on a cause and a PR to fix it (https://github.com/scp-fs2open/fs2open.github.com/pull/6794).

At this point we were hesitant. Two bugs had been identified and fixed, but two remained, the incorrect model loading error and the random texture replacement bug. We guessed these were unrelated, but both proved again incredibly difficult to reproduce. Learning from our recent experience, we quickly identified that these bugs were occurring even on 1920x1080 resolutions, so at least that avenue was closed. Another aspect we found was that the y-inversion seemed to happen after a string of campaign missions. Pondering this plan, we concocted skeleton test campaigns and dug through old reports to again search for any similarities. We also brute force printed line after line of model loading within the debug logs. After tedious analyzing and testing we finally struck a viable path and found a way to reliably reproduce the bug with a custom campaign. Equipped with this path we again dug into the debug log model prints and found something peculiar. Models that were present in the briefing but not the mission were not being fully loaded. That realization sparked a follow-up recollection of a PR from earlier in the year that was aimed to optimized model loading of briefing ships. On a hunch we dug into that code and found our culprit, a missing argument, and made a fix PR (https://github.com/scp-fs2open/fs2open.github.com/pull/6850).

Perhaps this was it, perhaps the texture replacement bug was just an offshoot manifestation of these earlier bugs. Mobilizing our testers we waited, and right on queue the texture replacement bug reared back into the forefront. Unlike its compatriots it had not yet been squashed. Moreover reports of this bug were the first we had received in the beginning of the year, so it outlasted all the bugs fixed thus far with a timeframe of over 7 months. And, true to fashion, this bug was even more difficult to reproduce reliably than the last. Still our experience fighting the earlier bugs informed our approach, we hypothesized this bug was caused by some kind of texture slot loading mismatch that occurred over the course of loading many campaigns missions within one playthrough. Brute force debug log printing commenced once again, this time specifically targeted at texture slots. Finally, on occasion, we were able to reproduce the bug. This case turned out to be primarily a scripting related issue to texture loading, and we pushed a fix and kindly asked to our testers to begin running missions in earnest once again. This was today, and fingers crossed, reports will come back positive. It is entirely possible this was not the end, and this cliffhanger will result in more hair pulling. Still, we have pushed this far and have no plans to let the bugs win. If you've read this far you certainly deserve a Wookiee Cookie!


Fun fact, during this time Lafiel was developing entirely different PRs for new features, cleanup and optimization including multithreading for collisions. He's a real wizard and we can't thank him enough either for his work on these plus all the fantastic features and enhancements he has made over the years in FSO (VR, deferred shading and shadowing in cockpits, complete animation overhaul, modular POFs, complete particle overhaul and rework, developing an entirely novel way to do multisampling antialiasing within a deferred pipeline then writing a paper about it, plus so much more). 

[Goober5000]

This is an awesome writeup.  Congratulations to the testers and bug-squashers, especially Lafiel and Limbert, for their tenacity, perseverance, and insight!

[Lafiel]

Well, we found the underlying Engine cause of that last "scripting related issue to texture loading"...
And what a bug this was.

Going into debugging, all I knew was that, sometimes, the loading bar animation's second frame was randomly freed and deallocated by the time it was meant to be shown. And that it's likely got to do with scripting somehow.

My first approach was to put a breakpoint at level load, check the handle of the loading bar's second frame, and then add a conditional breakpoint (thank god for those...) into the function that deletes textures, checking for a request to deallocate the handle I just collected from the other breakpoint. A few tries of this (because of course the repro was unreliable, happening just about every fifth time I loaded up the test mission) got me to an unexpected place (well, I probably should have expected it: The deletion request came from within lua's garbage collection pass.

But what could be causing this? All lua texture deletions explicitly check whether the texture is still alive, never get their handle changed after creation, and should have no access at all to the loading bar animation handles... So what else could it be? Were some textures created by lua placed into non-empty slots? Or the loading bar animation on top of scripted textures? It really didn't seem so, and it was also not easily testable, given that the loading bar animation is only loaded just before mission load, and thus we cannot check against its handle in most lua texture handling.

So maybe it was lua texture deletion, even with the safeguards put in place? But how can we even check for that properly? The thing I ended up doing, was to give scripting access to the loading bar animation handle, and every single time that lua would try to delete a texture, it'd try to load up the loading bar animation and see if the second frame was still intact. And lo and behold, after one texture's deletion, this check notified me that, indeed, the loading bar animation was gone.

But the scripted texture that was being deleted didn't seem much problematic. It had a sensible filename, that I knew was used for something being rendered, and that had looked correctly... So to trace the problem further, I added a string to the lua texture data struct, and gave every single call to a constructor for scripted textures their own ID, in order to fingerprint where this texture object comes from, and what might be wrong with it. It turned out to be created by indexing a specific frame from an animation. Wait a minute... A frame from an animation? FSO shouldn't try to even delete that on its own... And if it'd try... it wouldn't be guarded by the "don't try to delete it if its already gone" check, at least not if its original deletion is prompted by deleting the entire animation.

And that is indeed what the problem ended up being. FotG would delete the animation once it was no longer needed, freeing the animation. This left the animation deleted, but both the variable holding the animation handle, as well as a temporary created by indexing into a specific frame of the animation, still existed. When they were Garbage Collected, the former would notice that it was indeed deallocated prior, so it would not try to delete itself again. The latter on the other hand was implicitly deleted by the parent animation, but it itself wasn't deleted before, so it happily tried to deallocate itself. But because lua's Garbage Collection happens sometimes later™, the loading bar animation had been loaded in the meantime, and unfortunately into the very slot that was now being deleted incorrectly.

So, once this became clear, the fix itself was a fairly easy "all subframes of animations should lock and clear their respective parent animation, not the individual frame", but getting to this point, to understanding what actually occured, was surprisingly difficult.

[jg18]

Wow, amazing work!