Hard Light Productions Forums
General FreeSpace => FreeSpace & FreeSpace Open Support => Topic started by: Alan Bolte on June 02, 2021, 03:35:21 am
-
I haven't played FS2 for years, but today I decided to go back through War in Heaven, and it's been... a little unstable. I updated Knossus, updated all my mods, and everything went well for a while...
My first time through Delenda Est, in the middle of a dense furball, the framerate dropped to single digits for a few seconds, and then it crashed to desktop, producing a mdmp file containing an access violation. I'd tell you more, but I couldn't reproduce the crash, and I can't figure out where to get symbol files. I could upload the mdmp file somewhere if you want it.
I then used Knossus to enable fast debug. This resulted in a ton of warning popups that seem to indicate Blue Planet's models and tables need to be updated to modern standards. Clicking through them every time a mission or cutscene loaded was a real drag.
My first time through Nothing is True, I got right up to the part at the end where I ordered my wingmen to attack the federation fighters, and then an assert triggered: "total_ammo" at aicode.cpp:5740
From skimming the code, I think this means that somehow an AI ship with only ballistic weapons (probably my wingman... maybe the Custos?) somehow ended up with its maximum ammo set to 0. Couldn't reproduce on second runthrough, not sure how I would, though of course I could run through the mission a dozen times if you want me to be thorough.
I got through Everthing is Permitted withough errors, but then when I went to the dreamscape afterwards, I experienced another access violation with mdmp during the Custos training sequence. No idea what triggered it, but there was no framerate drop this time.
My log is here: https://fsnebula.org/log/60b73bc1d6a16939c439130d (https://fsnebula.org/log/60b73bc1d6a16939c439130d)
I am running Windows 10 on some seriously old hardware (~2013), and I haven't tried a ton of troubleshooting, but I'm not really sure where to start, with nothing clearly reproducible. I just figured I should log my experience.
-
I can reproduce the crash in the Custos training! All I have to do is hit , (comma) to toggle my weapons.
-
Thanks for going out of your way to test this and figure out how to reproduce it. :yes:
I'm not familiar with BP's modpack but I suspect this isn't a quick fix. In the meantime, you can probably get around the issue by using an older FSO build in Knossos. From the mod.json it looks like the oldest build BP is set to use is 21.1. If you go to Knossos -> Settings -> Show builds in mod list... then go to Knossos -> Explore -> FSO -> Details. There will be a dropdown in the top left to select and install older builds.
@Battuta I know you're tired of maintaining this mod. There are a lot of POF/table warnings in the log here that would be where I would start fixing things. I'm willing to help out with that if you guys need it. My only hesitation is that BP is a complex modpack that I'd be afraid of breaking in some unforseen way.
-
Thanks mjn, u r a frend.
I suspect Darius is the one taking care of all the janitorial duties, unfortunately. My guy may have this under control, I don't know, he is a dad and a doctor!
I really, really do not want to go through the horrorshow of retesting every mission to see what's changed with every creeping unnoticed second-order consequence of some code change. But it probably has to be done. Bluuuuuuuuuuuuuuuuuuuuuuugh
-
Or, of course, you could just pin the modpack to the last known working build. Then you never have to worry about incremental breakages ever again.
-
That is something that can be done after a comprehensive playthrough and retest, yes. But it's not an option before you have a last known working build. Unless you're suggesting we go back to a build from 2015.
-
That is something that can be done after a comprehensive playthrough and retest, yes. But it's not an option before you have a last known working build. Unless you're suggesting we go back to a build from 2015.
why not tho
-
The mod would not work any more and we'd have to throw out our progress on Act 4 missions.
I don't understand why this is hard to grasp! If you need to upgrade to a new build for whatever reason (MVPs compatibility, new features, a vital bugfix like 'turning is no longer framerate dependent') you also have to accept the risk of regressions and the test burden for them.
Also at some point we really need to sit down and see how much the gameplay has changed from 1999 retail and how much of that change is intended and or desired.
-
Or, of course, you could just pin the modpack to the last known working build. Then you never have to worry about incremental breakages ever again.
You scolded me multiple times about not updating BtA for the same reasons here. This is simply to point out that you didn't listen to me then and you aren't listening to Battuta now. It took months of work to update BtA1. Time that I would have rather spent on BtA2.
The mod would not work any more and we'd have to throw out our progress on Act 4 missions.
I don't understand why this is hard to grasp! If you need to upgrade to a new build for whatever reason (MVPs compatibility, new features, a vital bugfix like 'turning is no longer framerate dependent') you also have to accept the risk of regressions and the test burden for them.
Also at some point we really need to sit down and see how much the gameplay has changed from 1999 retail and how much of that change is intended and or desired.
This is a major point that people need to take to heart. It's not easy to keep up with the cycle of features and breakage in SCP while maintaining previously created missions AND trying to create new content. And there is constantly unintentional breakage from SCP's direction. That Mefistofele mission is a prime example. In Shell Game, an event was tied to when the hostiles fire missiles. They always used to fire missiles when they got in range of the friendlies. Now for some reason, they fire missiles the moment they arrive. Why? Dunno. Some SCP change caused it, but the BtA team was forced to deal with it.
Mods like BP are big and complex and whatever one-liner fix is suggested in threads like this are usually easier said than done.
EDIT: Further thoughts. That Shell Game missile timing bug implies an AI change across the board regarding how the AI chooses when to fire missiles. This can easily have balance impacts; to say nothing of other mods that may have sexp timing issues due to checking missile fire via sexp. Is SCP looking into why that changed? I bet not. I also bet that's not the only unintentional breakage in SCP. Mod authors just have to hope the changes are minor enough not to mess with missions too badly. The whack-a-mole game here is only playable when the moles show their heads and sometimes I just don't want to play it.
Using an old build is an easy to write solution. But players want mods updated to use the new MediaVPs or whatever. Sometimes that requires a newer build. We have a complex modding machine here and "use an old build" is not a long term solution.
-
I have so much **** to do and I do almost none of it, I'm good for like 4-5 hours of work a day on anything, the rest of my life is spent being a useless slime emission, I need all the hours in my day to feel bad about not doing the stuff I'm supposed to be doing! I can't find the time to constantly fix mods, I barely have enough time to feel ****ty about not fixing them!
-
I am not requesting anyone spend any time on fixing anything! Presumably at some point it did work, it is vastly easier on you and the users if the build that was used was the one which you last knew it worked on. If you aren't actually using recent features there's no reason to be using recent versions. And there is definitely no reason to always default to the most recent nightly, because that will only ensure these constant incremental breakages will always happen, even if you update it.
All it takes is a few clicks in Knossos to set the build to an earlier one (and don't allow newer versions), at some point a dependency won't let you go earlier back, whatever that build is is a very good candidate to pin it too, as you'll be dealing with way less bugs. If you want to do a full update for recent versions, that's great, but also a lot of work and that isn't actually necessary.
Having a complex mod like this use recent nightlies is a great way to unearth bugs to be sure, so it's very useful for the SCP! But it's absolutely not worth the frustration for the user and the modder.
Using an old build is an easy to write solution. But players want mods updated to use the new MediaVPs or whatever. Sometimes that requires a newer build. We have a complex modding machine here and "use an old build" is not a long term solution.
I disagree completely. By using its own builds, WoD has unintentionally pinned itself to a build it knows is stable, and directly because of that even a complex mod like WoD has enjoyed a very bug free experience in perpetuity! Sure users like new bells and whistles, but I think everyone understands that requesting a mod to update for it can be a big ask, and they would certainly admit that functioning at all is far superior to some new bells and whistles. Sticking to an old build is the only viable long term solution.
-
I am not requesting anyone spend any time on fixing anything! Presumably at some point it did work, it is vastly easier on you and the users if the build that was used was the one which you last knew it worked on.
It's from 2015
-
I thought you said the mod straight up wouldn't work with a build that old?
-
It has had several minor revision bumps over the years, presumably to update to the newer MediaVPs or fix other bugs. So I say again... using an old build is not as easy as you claim it is, Asteroth. It's as much of a dice roll to pick an old one as it is to let it use the most recent stable.
C'mon, man. You made a mod and you code for SCP. You should be able to grasp this.
TLDR: Any mod that has gone this long without actual maintenance needs extensive testing no matter what way you go with it. Please stop acting like it's as simple as a few clicks in Knossos.
-
One option constantly changes over time, and the other does not. It should be very clear which will be easier to maintain. If you're dead set on updating it, then yes obviously, there's no getting around the work that's involved, but the trigger shouldn't be pulled on that until someone has the time and is willing to spend the effort to do so.
-
*epic facepalm*
EDIT: It's clear you aren't listening. The problem is more complex and whether or not you understand that doesn't change the fact.
I've offered assistance to the TC and to the BP team to actually help fix this issues. My job here is done. This other back and forth is pointless.
-
To further drive home just how silly it is to just say "use an old build" for older mods... In theory, the last known stable build for BP is from 2015.
Knossos only works with 3.8 or newer (https://www.hard-light.net/forums/index.php?topic=94068.msg1858961#msg1858961), which came out in 2017. (https://www.hard-light.net/forums/index.php?topic=93812.0)
So please spare the one-liner fixes and let's work as a team, maybe?
-
I'm just trying to help, man. It really pains me to see users and modders frustrated over a constantly reoccurring situation which other mods have never and will never need to worry about.
-
Then offer real solutions. What you suggested here literally will not work for this case.
-
I thought you said the mod straight up wouldn't work with a build that old?
Yes, that's correct. There is no known good build. A full test cycle needs to be performed.
-
I can reproduce the crash in the Custos training! All I have to do is hit , (comma) to toggle my weapons.
If you mean the Dreamscape training, that's not causing the bug for me, I'm afraid. We may need to iterate a bit to get a decent reproduction here. If you're familiar with Visual Studio dev tools, it might help if we can dissect the bug on your machine — you can compile your own version with debug symbols.
-
So, the reproducible issue isn't reproducible in the latest nightly build. Congrats on already fixing the weapon switching bug, lol.
That leaves the other access violation (probably only reproducible if I make my computer struggle to deal with too much at once, I dunno, I'm not worried about it), and the Total Ammo assert, which was just weird. I may or may not bother trying to figure out how to reproduce that one.
-
To try to sum up a long discord discussion about the disconnect between SCP and modders that's... largely orthogonal to the thread topic but is happening here anyway:
SCP doesn't intentionally break things, but sometimes things break, including AI and other hard-to-track balance-affecting things. SCP would like to have better testing, but it's probably impractical, though some ideas are being pursued. SCP doesn't want to tell anyone in overworked, volunteer modding teams that they need to do something, but in practice, if a modder doesn't do all the isolation work required to figure out what SCP changed that caused a problem, SCP usually won't have the resources to do that isolation work themselves, at least not in a timely manner.
It's probably best not to link events to AI behaviors, or to expect AI behavior to be deterministic in general.
Also, SCP is not a hive-mind, so all of the above is probably a misrepresentation of what any individual member actually thinks, and may not actually represent a gestalt.
There was then much discussion of whether more frequent newsletters like this one (http://"https://www.hard-light.net/forums/index.php?topic=97094.0") would reduce the feeling that SCP is an unknowable entity that doesn't care.
I got the impression that Battuta and mjn are correct about mods older than 3.8 needing a full test cycle; no one is happy about this, but there doesn't seem to be a better option. If something broke in between the mod's last update and 3.8, then the modder is either going to have isolate the problem themselves, or convince a specific SCP member that the problem deserves priority over whatever other problems they're trying to fix.
Again, this is just my attempt to summarize, so I might have got something wrong.
-
"SCP would like to have better testing, but it's probably impractical, though some ideas are being pursued."
Yup, and this is also low on the priority list.
qtFRED and PCS3 would be a higher priority.
-
I do not dispute the difficulty of testing SCP bugs and features. I didn't get my SCP badge cause I can write code. I got it because I'm damn good at isolating and reproducing bugs.
SCP does not like it when some mod maker tells them how easy it will be to make some feature or fix some bugs. HLP history is littered with examples of ruffled feathers over that exact issue. So I get a bit annoyed when an SCP dev pops in to more (https://www.hard-light.net/forums/index.php?topic=97599.msg1909892#msg1909892) than one (https://www.hard-light.net/forums/index.php?topic=97606.msg1909943#msg1909943) thread insisting on the same tired fix that does. not. apply. here. and how easy that fix is. In the future? Sure. That's a way to solve the problem. Right now? Won't do it.
BP has almost 60 mission files. I simply ask that people consider just how difficult that's going to be to test thoroughly no matter what FSO build is chosen. It's not like BP was known for simple retail-style missions, either.
-
The thing is, at the end of the day the only practical way to expect your mods to work stably for users for any length of time is to test them against a specific version of FSO and then keep that version fixed in Knossos. You need both parts for stability; the problem BP has here is that it hasn't really gone through a thorough test-and-fix cycle for a very long time, so there's no clear target to pin it on.
-
Next problem: tanks don't show up in Eyes in the Storm. The little rocket trails appear and travel down to the platforms, but there's no models, and therefore no turret fire. That's in the nightly 20210602. I switched back to stable 21.2, and that fixed the problem.
By the way, I tried switching to 3.8 at one point, but the game won't launch at all, with:
Error: game_settings.tbl(line 28):
Error: Missing required token: [#END]. Found [$Movie subtitle font: 1] instead.
File: parselo.cpp
Line: 290
Just for reference...
-
The thing is, at the end of the day the only practical way to expect your mods to work stably for users for any length of time is to test them against a specific version of FSO and then keep that version fixed in Knossos. You need both parts for stability; the problem BP has here is that it hasn't really gone through a thorough test-and-fix cycle for a very long time, so there's no clear target to pin it on.
(Lest anyone think SCP coders are just blaming everyone but themselves for the issues cropping up, I will say that I think there have been a lot of careless breakages pushed into FSO in the past several months which have exacerbated the frustrations felt by modders here. The problem is that while the SCP can certainly be more careful about stuff like that, full backwards compatibility for all engine behaviour is not achievable, and even tiny obvious bugfixes have the potential to break mods. So we still need to get everyone on the same page with their approach to engine stability.)
-
Next problem: tanks don't show up in Eyes in the Storm. The little rocket trails appear and travel down to the platforms, but there's no models, and therefore no turret fire. That's in the nightly 20210602. I switched back to stable 21.2, and that fixed the problem.
By the way, I tried switching to 3.8 at one point, but the game won't launch at all, with:
Error: game_settings.tbl(line 28):
Error: Missing required token: [#END]. Found [$Movie subtitle font: 1] instead.
File: parselo.cpp
Line: 290
https://www.hard-light.net/forums/Smileys/HLP/sigh.gif
Just for reference...
:shaking:
-
Well, duh. From what I understand, BP was at some point updated to work with some pretty recent MVPS... which require a pretty recent build. One far more recent than 3.8.
Also, regarding the compatibility topic, just one comment fished out from the Discord: (https://cdn.discordapp.com/attachments/223511363807346691/850082266746191962/unknown.png)
-
Error: game_settings.tbl(line 28):
Error: Missing required token: [#END]. Found [$Movie subtitle font: 1] instead.
File: parselo.cpp
Line: 290
All that is saying is that game_settings.tbl in the package was at one point updated for FSO 19.0.0, see:
https://wiki.hard-light.net/index.php/Game_settings.tbl#.24Movie_subtitle_font:
To approximate a working FSO version for the tables, it might be good to trace back when the source files for the Knossos build were last updated. If the BP team does not use a software solution that allows for that, it has to be done by hand by someone who is the source for Knossos build or has access to those files.
-
The tables aren't the problem, I am much more worried about Eyes in the Storm tank spawns breaking again
-
Again, what needs to be done here is a full test cycle to find a known good build. The issue is not the ability to launch the game, it is stuff silently breaking due to changes in the engine.
-
On that note can someone nominate a stable build that is recent enough to include the vital fixes we need, like the correct SEXP node flushing behavior and checkpoints working again? Or does such a build not exist outside nightlies?
-
I'm pretty sure 21.2 has the sexp node flushing fix. Not sure about checkpoints.
-
Again, what needs to be done here is a full test cycle to find a known good build. The issue is not the ability to launch the game, it is stuff silently breaking due to changes in the engine.
I've found that with suspected bugs created by the advance of FSO, the reverse process is much more effective.
Set a stable build. Go back and find the non-function element. Recreate it in isolation to narrow down the cause.
EDIT: It might be worth it to point out that the process of recreating a sequence of events in isolation is a prime opportunity to compare it to other possible ways to achieve the same results.
-
Well, just to sum up my experience, at the moment 21.2 isn't stable because switching primaries in dreamscape causes a crash, but the fix for that is in a nightly, but the nightly breaks the tanks for some reason I haven't looked into. So if there is a perfectly-stable build (which there might not be), it's either 19.0, or some version after that but before the dreamscape problem started.
OTOH, the dreamscape problem is minor enough that you could just note it as a 'known issue' and call 21.2 close-enough-to-stable, unless there are other problems I didn't run into. Sure, I got one unexplained CTD and one impossible assertion, but those aren't compatibility issues.
-
Again, what needs to be done here is a full test cycle to find a known good build. The issue is not the ability to launch the game, it is stuff silently breaking due to changes in the engine.
I've found that with suspected bugs created by the advance of FSO, the reverse process is much more effective.
Set a stable build. Go back and find the non-function element. Recreate it in isolation to narrow down the cause.
EDIT: It might be worth it to point out that the process of recreating a sequence of events in isolation is a prime opportunity to compare it to other possible ways to achieve the same results.
I don't really get what you are suggesting here.
-
Hi everyone.
We've started working on a big maintenance update for BP Complete that should bring it back to the latest FSO and MVP version and clear most of the debug issues.
There's a lot of standard debugging maintenance that has been long overdue.
Once this is done and updated on Knossos, and we can confirm all campaign missions run and finish properly under normal playing circumstances, we'll see if we can still reproduce and track down some of those specific issues.
-
I don't really get what you are suggesting here.
Instead of trying to find a new stable build through a long line of tests, you set one build that to serve as your new baseline. The missions will be broken but you'd at least have the firm foundation from which you can plan out the work to make put them back into a working order.
With that you can clean up all the little stuff in Debug Log on start-up - which shortens the list of possible error sources from assets, version mismatches in the tables and the like. This will ensure the mission will "broken" only by what it is in itself.
Once that is cleaned up you go into the mission on a Debug Build and note down each instance in the which the mission is now "broken". As soon as you have the events identified that no longer work, you make a mission that only contains the best approximation of the non-functional events you can - depending on what needs to be done that can take different shapes.
Use the isolated testing mission to narrow down as exact as you can what is not working. Then try to match what is suposed to happen with a similar mechanic that can achieve the same objective - even if it is more complex.
A recent example is the bug described here: https://www.hard-light.net/forums/index.php?topic=97011.msg1907744#msg1907744
When it was first reported, I had the player who encountered send me their Debug Log - it gave me the version information for the replication (21.0.0). As it was a reasonable build for players to use, I made it the target build for the fix, regardless what the build was SerRes was originally released on or patched to. I achieved replication on 21.0.0 and quickly was able to find the offending event by comparing the ingame message and events logs.
Then I created a mission to check the behavior of that event's condition in isolation: A small clean mission that contained nothing but the player fighter and capital ship that was needed for the replication of the event's condition. No other ships and only a few events that let me trigger the arrival and departure of the capital ship with the push of a button.
Using messages and subtitles, I build myself a framework there to check the condition in isolation at first (when-argument with a number-of list using cap-subsys-cargo-known-delay). Then as a second step against a similar event (when using a combination of and & 4xcap-subsys-cargo-known-delay), as a thrid step I compared to a variable based solution (5 events using when and cap-subsys-cargo-known-delay to modifiy-variable +1 each for each subsystem; one to trigger if the variable became >= 4).
Turned out that in between the release build of SerRes on a Nightly and FSO 21.0.0 the behavior of the number-of argument lists was changed, so that the event no longer triggered the way it was originally intended - which is when 4+ subsystem of a certain capital ship were scanned. Instead the event triggered when it was supplied with 2+ valid argument instead when 4+ arguments were valid.
The original event logic would not function under 21.0.0 but an alternative existed and easily achieved. All that it really took was accepting that there was no way to restore the event 1-to-1 but that aiming to achieve the same results through a different approach.
Temporarily I have to suggest to players to get an old Nightly, which still works on the original behavior, for internal development reasons.
Granted, this was an incident where everything alligned perfecty - no scripts were involved and it was pretty simple to isolate what was wrong. For other issues I had to put in additional work to just find what is wrong but it was not hard, it just took the time to prepare the missions with a custom debug overlay that gives the information I require for the situation at hand. Finding an alternative solution to any given event logic is not hard either, most of the time just recreating the original event in the clean mission gives me an idea how to achieve the same results using different SEXP.
And if you are short on time, breaking down the individual steps and keeping a good record of the tasks required helps greatly - I use DIN A3 paper sheets to make my "task boards" so I can pick up and leave a complicated job at will.
-
Yes, that's exactly what the plan is already. Find a build, test on it, fix whatever's broken on it. This is how it's always been done.
The problem is that there is no currently no stable build with all the necessary bugfixes. And if you can't set the mod to run on a stable build then it's going to be running on nightlies. Which could break things again at any moment.
-
Yes, to be clear, the issue is finding a recent build with no really nasty regressions to start from. Things like checkpoints breaking (21.0.0) and hovertanks not deploying (21.2.0, maybe).
-
Yes, that's exactly what the plan is already. Find a build, test on it, fix whatever's broken on it. This is how it's always been done.
The problem is that there is no currently no stable build with all the necessary bugfixes. And if you can't set the mod to run on a stable build then it's going to be running on nightlies. Which could break things again at any moment.
No, it is not what I am proposing. I am proposing a dogmatic stop instead of a search.
Of course that will mean that you have to work from a point were something will be irrevocably broken and has to rebuild from square one. But it preferrable to holding out for an empty promise of the "once and future build".
-
thanks man your input is really helpful to the actual BP and SCP devs who have been talking through this and making different conclusions
-
No, it is not what I am proposing. I am proposing a dogmatic stop instead of a search.
Of course that will mean that you have to work from a point were something will be irrevocably broken and has to rebuild from square one. But it preferrable to holding out for an empty promise of the "once and future build".
What?
-
That's what a stable build is. It's the build you test on and target your release to so you can say 'this is the build the campaign works on, now and until we change something.' We've had stable builds quite regularly for years. They're not empty promises.