Hard Light Productions Forums

Off-Topic Discussion => The Classics => Topic started by: General Battuta on April 16, 2010, 12:31:03 pm

Title: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 16, 2010, 12:31:03 pm
About fifty hours ago we were in the process of wrapping up another mission. This was a pretty FRED-intensive mission using some relatively complex stuff: variables, set-object-positions using relative coordinates, skybox changes and the like.

I was pretty pleased with the outcome. We'd iterated this mission a lot, and this was the third revision of the concept. Finally, after days of work stretched across months of real time, it felt like it had worked out pretty well.

I made a few changes and ran through the mission for the Nth time to make sure everything was working.

About halfway through the mission, right when a critical event fired, the world was devoured. The ships, the skybox, even the player's own vessel all vanished, and the screen was filled with inky blackness. All that was left was the HUD and the sun, and even those weren't working right: you could still target objects, but they were (like everything else) invisible, and the distance and speed monitors were both blank. Even the ambient hum of the engines was gone. The 3D radar, meanwhile, had migrated up into the spot where head .anis are usually displayed.

The game's rendering engine seemed like it had just quit. Given up the ghost. All that was left were the 2D elements. It was as if Ransom's Transcendent had decided to pay a visit and eat reality.

I stared at this in bemusement and horror, posted a brief note on IRC, and restarted the mission.

Same thing. Same point in the mission.

I looked at the event in question. There was a lot of stuff going on right then: skybox changes, coordinate changes, coordinates pulled from variables, engine subsystem repairs, play-sound-from-files, fadeouts. It was a nightmare. Any one of those elements (or a combination thereof) could hypothetically be pushing the engine too far.

But it wasn't worth panicking until it could be reproduced. I posted the mission on SVN and, sure enough, The_E and others got the same result too. But they didn't get it every time. I kept changing things in the mission and testing, and sometimes I'd see it work perfectly, and I'd commit it with a happy note like IT'S FIXED...and then The_E would try the 'fixed' version and hit the same bug.

Now, for a bit of background. We'd met this bug before. Previously, we'd always encountered it in relation to a bad asset: first the Vishnan Keeper, then the Anemoi. In both cases we figured we'd solved it by cleaning up the .pof.

Yet none of the assets in this mission were bad, so far as we could tell. For some reason, the mission was eating itself at this point.

First we tried debug logs. They were useless. That must have eaten up an hour or so.

Next, The_E built a debug filter to get additional information. While he was doing that, I started systematically pulling newer SEXPS away from the event to see what could be causing the quit. Surely the coordinate manipulation or skybox change could explain it. Or perhaps it was the every-time conditional used in a linked event; every-time is notoriously hazardous.

Nothing helped. I pared the event down to two SEXPs: an engine subsystem repair and a hud re-enable. Since the engine repair was a retail SEXP, and the hud problem wasn't causing it (I could move it around with no effect), we were right back where we started.

Now bear in mind I'd been working on the third iteration of this mission for a great many hours already. I was sick, I was exhausted, and somehow I'd managed to break a glass during the trial-and-error debugging process, cut myself, and start bleeding everywhere.

The_E reported in. He'd been backtracking a stream of errors, starting in the sound code and moving all the way back to what appeared to be the original cause. He figured out the cause of the problem, but not the trigger. Somehow the engine was losing track of the player's position, converting a valid vector into NaN (not a number.) There was no error checking here, so as I understand it, the error propagated through the whole engine, smashing things apart like a Shivan hitchhiker on a Thoth.

But he couldn't figure out what caused the NaN error in the first place. We knew it always happened at the time this event fired, but now that it had been stripped down to fundamentally 'safe' levels, how could it cause the problem?

Blinded by RAGE at this bizarre bug crippling my brilliant mission, I started swapping out all the custom assets for MediaVPs ones.

And the bug went away.

I boggled. We had been certain it wasn't caused by bad assets this time. But every time I tried the mission (time-compressing through to the point of the bug) with the Perseus swapped in for the optimized Uhlan, the bug didn't seem to happen.

Except for The_E. He was able to reproduce it with the HTL Perseus. Which meant it couldn't be the Uhlan either.

At this point we were all taking this bug very personally. The_E started doing something very technical which he could probably explain better, but which seemed to involve running the engine forward frame-by-frame, looking for the moment when everything went horribly wrong in the player eyepoint vector. This was an insanely laborious process that, to the best of my knowledge, ate up hours of his time.

Take a moment to think about the timeline here. By this point, more than 24 hours had passed since we'd first posted about this bug on an SCP internal forum and Mantised a test mission reproducing it. Nobody had a bloody clue what was going on.

Finally, almost 48 hours (if I remember right) after first encountering it, I did something nonsensical but ultimately productive: I tried deleting the repair-engine SEXP from the event. This was a retail SEXP, so it should have been safe.

The bug went away. The mission kept on happily rendering. There was no inky void of all-consuming night. Yet there was no way that SEXP could have caused it, because sometimes - maddeningly - the mission worked fine even with it in place.

Truly this was the bug from hell.

Mere minutes later, working independently, IssMneur on #SCP struck gold. "Hey," he said (I paraphrase here.) "Have you been matching speed with something during the mission?"

We all ran through the mission, matching speed with a certain ship. The rendering engine quit when the engine-repair SEXP fired.

We ran through again without matching speed. The mission worked fine.

Matching speed was the source of the problem. When I'd swapped in the Perseus for the Uhlan earlier, I'd simply time-compressed to the failure point without bothering to 'play' the mission, and thus never matched speed. That's why the bug had gone away there.

After some frenzied diagnosis, IssMneur and The_E explained what had happened. Apparently, if the player was matching speed with a target ship at the time that they were disabled, but then later repaired, the game engine would divide zero by zero at the moment of the repair and produce a nonsensical result which worked its way into the player's position. The rendering engine would fumble, choke, and stop working.

As far as I know this could have happened in retail.

They built a patch for the issue and promptly generated a new build that fixed it. I cleaned the blood spatter off my keyboard.

Then the mission started crashing for a totally different reason. But that's a story for another time, and now it's fixed too.

The_E may be able to provide more insight into the hilariously random and difficult-to-reproduce technical side of the bug. But that's my part of the story.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Droid803 on April 16, 2010, 12:44:07 pm
The lesson:

Don't use match speed.
Use the Z key like a real man! :P

(I know I'm missing the point)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Spoon on April 16, 2010, 12:50:03 pm
Oh wow, just wow
Hilarious but at the same time I can completely understand the frustration this must have caused.

I encountered this black void of doom only once my self. I had just build a new ship and was going to test it out ingame, the ship warped in... and then vanished. Then I moved my own ship forward and everything went into a blackhole. That scared the living **** out of me but then it never showed up again.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Sushi on April 16, 2010, 12:54:06 pm
The real lesson:

Don't assume that retail stuff works right. :)


Also, major kudos to those of you who fought and defeated this behemoth.
(http://www.freewebby.com/beer-food-sick-smilies/toast.gif)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: MatthTheGeek on April 16, 2010, 01:09:35 pm
I remember I also managed to reach the black void of doom while playing around with Fury's AI, when I made a type which assigned a ship to an AI class that didn't exist :/

Ah, the delights of modding... :D
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 16, 2010, 01:13:36 pm
The Bug From Hell, or: Is it 6AM again?

Here I am, explaining the technical bits.

And bringing you a shot of the bug in action:
(http://i659.photobucket.com/albums/uu320/FabianW/Untitled-8.png)

The first inkling I had that something was wrong was a small Assertion while running the buggy mission. Now, usually, Assertions are good things to have when debugging, it means that a) there's a clear starting point, and b) another developer had the foresight to check his data and provide a warning in case it was bogus.
So, with the assertion in hand, I dug in. For those of you interested, the assertion was "im > 0.0f" from vecmat.cpp, line 543. This info, by itself, is useless, since the function that the game is processing at that time (vm_vec_normalize()) is called from literally dozens of places.
However, since I happen to have Visual Studio (Microsoft's developer environment), I was able to perform what amounts to a live dissection of the crashed program, yielding the ever-important callstack. This told me the state of the engine at the time of the crash.

What I saw, however, was extremely puzzling. The callstack pointed towards the function that is responsible for generating the three-dimensional sound environment1 , which needs the current camera position as a starting point.
In this case, the global variable that stores the camera's position was apparently corrupted. This is something that should never, ever happen, and I was astonished to see that this was the first visible error message thrown.

So, I did what a developer debugging a large codebase does in these cases. I sprinkled breakpoints2 everywhere I thought they were needed, and ran the mission again.
And this is basically what I did the next few hours. Run the game, inspect the data at the breakpoints, reset breakpoints, run game again, lather, rinse, repeat.
Gradually, through trial and error, it was revealed that we were dealing with a cascading failure.
One that was literally happening in the tiny space between two frames.
As far as I could determine, what happened was that at some point, the player's throttle setting got invalid. When this invalid throttle setting was fed into the physics engine, it tried to calculate the player's velocity based on that, which again yielded invalid results. Then, in the next frame, the physics engine tried to calculate the player's new position using the invalid speed data, which produced coordinates that lay somewhere outside of the known universe.
This data, in turn, was used by the rendering engine as input, which caused it to basically throw up its arms and do as it was told (the mental image I had was of a tired office worker, who basically says "If management wants me to do something stupid, I'll do something stupid").
And on and on, until that assertion in a part of the code that was so far removed from the actual cause as possible.

Which brings me to the cause.
I can't take credit for discovering it, IssMneur was the one who nailed it by stumbling across the strange fact that if the automatic speed matching was turned off, the bug didn't happen.
Coupled with Battutas' discovery that the difference between the bug appearing and it staying hidden was a repair-subsystem sexp, he investigated, tracing through the code until he found the cause of this error.
Now, what happened was:

1. Player's engines get sabotaged via sabotage-subsystem sexp
2. Seeing that the player no longer has engines, the game sets the maximum speed the player can reach to 0.

This is normal so far.
However, if the player happens to be matching speed with another ship, the game tries to calculate the throttle setting necessary to do that by dividing the target's speed by the player's current max speed.

Feel free to headdesk now. I know I did. Because the code never checked to see if the divisor was greater than zero, we had a classical error on our hands.

End result, after countless hours of staring at variables? Two new lines of code that should bury this beast for good.


1: Basically, the game tries to place each sound source that is active in a 3d environment around the currently active camera, and adjust the sound effects' volume accordingly.
2: A Breakpoint is a marker that enables the debugger (NOT the debug build, I'm talking about the Visual Studio debugger here) to automatically halt execution. They are invaluable when trying to analyze a running program.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: mjn.mixael on April 16, 2010, 03:18:34 pm
I'm astonished. Truly. I would have given up hours and hours ago.

Maybe some sort of division error checking should be added to FSO... I'm not a coder so I don't even know if that is possible or plausible.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 16, 2010, 03:24:31 pm
That **** sounds intense, nice job on cracking the problem.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: jr2 on April 16, 2010, 03:24:43 pm
Awesome work, guys!  :nod:
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Droid803 on April 16, 2010, 03:31:01 pm
Oh, on a somewhat unrelated note, is this issue connected to the one where sometimes, upon re-trying a mission, the player ship is...gone?
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 16, 2010, 03:41:34 pm
I'm astonished. Truly. I would have given up hours and hours ago.

Maybe some sort of division error checking should be added to FSO... I'm not a coder so I don't even know if that is possible or plausible.

Yes, well, it was either that, or redoing the mission again from the ground up, hoping that the bug would not trigger, and also hoping that noone else would hit it. Neither of those options was one we wanted to take.
In addition, we had finally gotten a combination of assets that would trigger the bug reliably, that was an opportunity I couldn't let go.

As for the added error checking, I'm kinda astonished that this was not caught before. Modern CPUs are supposed to do just that and start screaming when something as stupid as this happens.

Oh, on a somewhat unrelated note, is this issue connected to the one where sometimes, upon re-trying a mission, the player ship is...gone?

No, probably not. The cause for this, and the resulting error, was very, very specific, and did not seem to be connected to the mission restart part of the code.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 16, 2010, 03:44:25 pm
I'm trying to remember how this worked, but I recall times where I'd be far from the normal location of a mission (Like 100km), if I'm using padlock view and I activate afterburner the screen wigs out and the results are fairly similar, everything vanishes and the HUD locks on the corner, it takes screwing around while playing in the mission for it to happen but it's just something I recall.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Mongoose on April 16, 2010, 04:22:31 pm
I knew I never used speed-matching for a reason...I just didn't realize the reason was that the engine has acquired its own "ghost." :p
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Ravenholme on April 16, 2010, 04:24:56 pm
I knew I never used speed-matching for a reason...I just didn't realize the reason was that the engine has acquired its own "ghost." :p

Likewise, but also because I'm fairly good at just adjusting the throttle on my joystick to match my target myself

>_>

(I admit, I didn't realise there was a match speed function, but I pro'ly wouldn't use it anyway)

Oh, I will say this - you guys are... dedicated. And that's an obscure retail bug, but at least it has now been put to the sword.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 16, 2010, 04:29:40 pm
I knew I never used speed-matching for a reason...I just didn't realize the reason was that the engine has acquired its own "ghost." :p

Likewise, but also because I'm fairly good at just adjusting the throttle on my joystick to match my target myself

>_>

(I admit, I didn't realise there was a match speed function, but I pro'ly wouldn't use it anyway)

Oh, I will say this - you guys are... dedicated. And that's an obscure retail bug, but at least it has now been put to the sword.

Given that the scenario here in question was a non-combat cruise, match speed is a pretty useful utility function no matter what your opinions of its use in combat may be.

In any case this was a combination of speed-matching with a pairing of events. Rather specific.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: High Max on April 16, 2010, 04:48:56 pm
^^
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 16, 2010, 04:50:35 pm
The E, is that red and blue reticle thing part of that mouse script? I've been trying to get that to work and well, nobody's giving me any suggestions on the original thread.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 16, 2010, 04:57:43 pm

As for the added error checking, I'm kinda astonished that this was not caught before. Modern CPUs are supposed to do just that and start screaming when something as stupid as this happens.

Did the modern CPU not do anything because of the fact that the game is still CPU limited? Is that another side effect of it being CPU limited?

No. It's probably a side effect from some optimisation somewhere or other.

See, the problem is, computers in ye olde days used to crash and burn when they encountered a division by zero. Modern Computers and operating systems have built-in checks to prevent this.
In this case, the prevention took the form of giving a special return value called "NaN" (acronym for Not A Number) for the division by zero. The problem was that a) noone checked the divisor before the division took place, thereby ensuring that the data was valid and b) there was no check to make sure the result was valid.

It's a very, very basic programming mistake that, unfortunately, can happen to everyone. It's the equivalent of leaving the house without your keys on the assumption that someone would be there to open the door for you when you return. In most cases, everything will be alright, but in some cases, you'll be locked out of your house.

The E, is that red and blue reticle thing part of that mouse script? I've been trying to get that to work and well, nobody's giving me any suggestions on the original thread.

No, that's the UI for the shield management script (http://www.hard-light.net/forums/index.php?topic=67712.0).
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 16, 2010, 05:00:01 pm
Well that's pretty neat.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Kolgena on April 16, 2010, 08:45:35 pm
Wow. That was a very interesting read.

Something is wrong with me though. I have an urge to take The_E's screenshot, and make a "You divided by 0, didn't you?" demotivational poster of it :/
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: High Max on April 16, 2010, 08:52:49 pm
^^
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Iss Mneur on April 16, 2010, 09:28:30 pm
I knew I never used speed-matching for a reason...I just didn't realize the reason was that the engine has acquired its own "ghost." :p

Likewise, but also because I'm fairly good at just adjusting the throttle on my joystick to match my target myself

>_>

(I admit, I didn't realise there was a match speed function, but I pro'ly wouldn't use it anyway)

Oh, I will say this - you guys are... dedicated. And that's an obscure retail bug, but at least it has now been put to the sword.

Given that the scenario here in question was a non-combat cruise, match speed is a pretty useful utility function no matter what your opinions of its use in combat may be.

In any case this was a combination of speed-matching with a pairing of events. Rather specific.
Non combat cruise alright. :D I assume, that thanks to HerraTohtori's episode in the testing of the last (re)release of BP1, the weapons were locked as well (at least in the problem mission that I received) :D


As for the added error checking, I'm kinda astonished that this was not caught before. Modern CPUs are supposed to do just that and start screaming when something as stupid as this happens.

Did the modern CPU not do anything because of the fact that the game is still CPU limited? Is that another side effect of it being CPU limited?

No. It's probably a side effect from some optimisation somewhere or other.

It is not an optimization thing as we are talking about floating point numbers here. One thing about the IEEE floating point number representation system is it allows for NaN (Not a Number). Because IEEE floating point numbers were designed by a bunch of scientists, it assumes that you know what the hell you are doing and if you can't handle NaN in your data, then you need to check for it or make sure you don't generate it.

Now if we were working with integers and you divided by zero then yes, the OS actually would interrupt the program. (I just tested it, VS2008 will generate a compiler error if you use have a constant expression that evaluates to zero (C2124), and Windows 7 generates the "program has experienced a fatal error" box if you slip the fact that you are dividing by zero past the compiler.)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 16, 2010, 09:32:26 pm
I assume, that thanks to HerraTohtori's episode in the testing of the last (re)release of BP1...
I did have that particular episode on a TBP mission a couple days back.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Qent on April 16, 2010, 09:35:06 pm
Wow you guys really did make it sound like something from Transcend. :P Great job on the fix!

Specifically, it's dividing zero by zero. I'm pretty sure that dividing a non-zero floating-point number by zero should always be +/-infinity.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 16, 2010, 09:48:01 pm
Which I guess also evaluates to NaN....
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Galemp on April 16, 2010, 10:01:43 pm
:jaw: I have... no words...

Consider yourself to have a drink purchased for you on my behalf, sir. Well done.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 16, 2010, 10:08:03 pm
:jaw: I have... no words...

Consider yourself to have a drink purchased for you on my behalf, sir. Well done.

Play Blue Planet!  :nervous:

sorry sorry sorry
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Qent on April 16, 2010, 10:12:14 pm
Infinity and NaN are a bit different. You can do some arithmetic on infinity, e.g. infinity is greater than every number, and negative infinity is less. But NaN just corrupts everything with its NaN-ness.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Iss Mneur on April 16, 2010, 11:19:54 pm
Infinity and NaN are a bit different. You can do some arithmetic on infinity, e.g. infinity is greater than every number, and negative infinity is less. But NaN just corrupts everything with its NaN-ness.

Ya, NaN includes more than just Infinity.  At least in the case of IEEE Floating point, NaN gets used as the catch all error state.  I am too lazy to actually look up all of the possible places to get NaN from, but as far as I know, it includes both positive and negative infinity, positive and negative number overflow (too big), and positive and negative number underflow (too small).  But yes infinity is at least order-able (you can sort it) whereas NaN is not equal to anything (even itself), not smaller than anything, and not bigger than anything

As for the result of dividing something by zero, it depends on the rules of math that you are using.  Non-calculus math completely ignores division by zero.  If your James Anderson (http://en.wikipedia.org/wiki/James_Anderson_%28computer_scientist%29) then it is a nullity.  If you are the IEEE Floating point representation it is NaN.  Others (like Qent) consider division by zero to result in +/- infinity. According to Microsoft Visual C, division by zero results in "-1.#IND00" when represented as a string.


Also, I forgot to do this before.  I figured I should add this, because the way Battuta and The_E tell it, I just pulled the idea for this out of my ass (which I don't mind :P), but I though I would explain my thought process of how I got to the fact that is was caused by speed match, which I hope may be useful to other coders or even mission testers.

At some point I was fiddling with the logging, because I don't have the patience to frame-step through the engine like The_E (props man :yes:).  So, I was adding more logging to the engine and fiddling with debug_filter.cfg because for some reason I could get the bug to happen but not have the Assert that The_E mentioned appear (still not sure why this was happening, there very well could be another bug, but I could not get it to happen repeatably).  In the fiddling with the logging, I noticed that sometimes the bug would be completely gone, then when I would try and verify that this new logging fixed the problem (yes, I know, WTF!) it would happen again. I had a hunch at this point that it was something the user was doing to the engine that was causing the bug to manifest, but still had no idea what the source was.

I fiddled some more, and I started to get frustrated, because I was running out of options short of searching for every time the engine writes to the player position and sprinkling Asserts (which I may still do as the core of the engine just assumes that everything it gets all kosher) or log statements just to see if I could figure out which part of the engine the error was coming from so I could try and narrow it down. As a last resort, I eventually got to just rerunning the mission pressing different buttons and trying different inputs to see if I could get the error to subside, because I noticed that the logging didn't seem to help reliably. At some-point I started ramming the other other ship in the mission and of course, not setting speed match every time because Battuta or The_E would distract me or I was playing the afterburner to make the ramming more fun :D.  Obviously I eventually noticed that it didn't happen if match speed was set.  So I tried enabling match speed then disabling it; the bug happen.  Tried again and kept speed match on the entire time; it bugged out!  I tested this theory several more times, I then asked in the channel if they could confirm if the bug went away when speed match was disabled.

I then started trawling through the code to see if I could find where the engine did calculated the speed match and I seen that that there was a division that was not protected (ie. no range checking on the divisor). Yay a simple fix to a seemingly nonsensical problem.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Herra Tohtori on April 17, 2010, 12:07:16 am
Specifically, it's dividing zero by zero. I'm pretty sure that dividing a non-zero floating-point number by zero should always be +/-infinity.

Technically, no division by zero is allowed in current mathematical rule set, ever.

You can wiggle around the rules by using the concept of limit which is, of course, fundamental basis for things such as derivatives, integrals and differential calculus in general as well as series and especially infinite series, but it still isn't dividing by zero.

The formulation

lim    1 / x = ∞   (x approaches zero asymptotically from positive side)
x→0+

lim    1 / x = -∞   (x approaches zero asymptotically from negative side)
x→0-

means we can handle conditions where a variable approaches zero asymptotically, but it tends to get somewhat confusing rather easily, and a div by zero forgotten in the calculations can really throw a monkey wrench in the end result. Plus the definitions of rules of handling infinities are somewhat illogical due to the nature of such concept.

As an example, we can define that

∞ + ∞ = ∞
-∞ -∞ = (-∞) + (-∞) =  -∞

so multiplication works, but subtraction of infinities

∞ - ∞ = NaN (aka not defined)

Similarly, we can easily define that

∞ * ∞ = ∞

-∞ * -∞ = ∞

∞ * -∞ = -∞

-∞ * ∞ = -∞

so multiplication works as well... however, division is not defined:

∞/∞ = NaN (Not Defined)

There are other oddities in calculating real numbers and infinities together, such as

x / ∞ = 0

is defined and even relatively understandable... but reversing the function by multiplying with infinity doesn't work:

x / ∞ = 0  |  (...) * ∞

x * ∞ / ∞ = 0 * ∞

and because x * ∞ = ∞,

∞/∞ = 0 * ∞

and since ∞/∞ is not defined, therefore 0 * ∞ is not defined either...


 :drevil:

...actually, technically 0*∞ is every real number at the same time which, of course, is the very meaning of "undefined". :p

See what I mean by hairy definitions yet?
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Qent on April 17, 2010, 02:40:03 am
I agree with what I can understand of all that. But I was talking about floating-point (not real number or computer integer) division by zero. I had thought that it was the IEEE that defined 1.0/0.0 as positive infinity (http://en.wikipedia.org/wiki/Division_by_zero#In_computer_arithmetic), but if Visual C gives you that "IND" thing, then I'm not sure. GCC gives me 1.#INF00 for 1.0/0.0 and -1.#IND00 only for 0.0/0.0.

/me notices awkwardly how far he's dragged this off topic. :nervous:
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: pecenipicek on April 17, 2010, 05:07:45 am
As for the added error checking, I'm kinda astonished that this was not caught before. Modern CPUs are supposed to do just that and start screaming when something as stupid as this happens.
We are talking about C and C++ here. Neither of them is going to stop you (reliably) from doing something stupid, thats something you have to stop yourself from doing.



to quote my dear father "With C you can make something either very very fast, or something very veeery stupid. In either case you'll most likely crash the bloody pc you dolt..."

Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 17, 2010, 05:25:39 am
Yes, however, I was more thinking in terms of compiler warnings etcetera.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: pecenipicek on April 17, 2010, 07:56:27 am
Yes, however, I was more thinking in terms of compiler warnings etcetera.
thats true. well, thats what all the warnings are for mostly :p


wanna bet that there was one such warning somewhere during compiling that related to this bugger here?
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Goober5000 on April 17, 2010, 01:31:33 pm
/me applauds

Well done guys.  I shall join in with Galemp in buying y'all a round next time you're in the area. :)

I seem to recall that this bug has been reported in ages past, back before the SCP even.  There were four or five isolated posts about everything being swallowed into inky blackness, but they have been lost to the depths of time.  People usually shrugged and moved on, because they were never able to reproduce it.  It's a testament to your perseverance that you could figure it out here.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: -Norbert- on April 17, 2010, 01:56:36 pm
"Feel free to headdesk"... pretty much yes.
I got really engrossed by the "story" of this bug and then in the end it was caused by something that every student learns (if they don't already know it) in the first or second lesson of programming.

Just proves that thinking about all the complicated stuff makes you forget the simple things sometimes.

My respects to you for finding and smashing this bug and what was necessary to do so.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Shivan Hunter on April 17, 2010, 02:38:02 pm
Holy crap. That bug would literally scare me if I came across it and wasn't expecting it. (I would actually consider simulating the effect in a campaign like Transcend, lol.)

And a heisenbug of that magnitude, coupled with Battman's awesome writing... I just had a nerdgasm. :o

Hats off to you all. That can NOT have been easy to find. I would never have thought of something like match-speed as one of the causes. I usually think that if it doesn't happen under certain conditions, then none of those are the primary cause, which is naturally a horrible way to try to diagnose a FSO bug. Of course, I never use match speed anyway so :\
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: MatthTheGeek on April 17, 2010, 03:22:17 pm
Summed up, the problem was simply that you never considered that a retail sexp could cause such an issue, because after ten years of working and improving on the game engine, one could easily believe that every retail bug would have been fixed...

Ah, there are so many ways to crash a game engine :rolleyes: At least we can learn a lesson from this, young padawan...
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Madcat on April 17, 2010, 03:48:23 pm
Hats off to all of you!

Since I work as an IT Consultant in Support I know quite well how hard it is to find stuff like this, spending endless hours on debugging and in the end, one or two additional lines can fix it.  :pimp:
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 17, 2010, 03:50:09 pm
Summed up, the problem was simply that you never considered that a retail sexp could cause such an issue, because after ten years of working and improving on the game engine, one could easily believe that every retail bug would have been fixed...

Ah, there are so many ways to crash a game engine :rolleyes: At least we can learn a lesson from this, young padawan...

Also because we'd seen the issue occur before for totally different reasons. I imagine there are still ways to trigger it. And it wasn't just the retail SEXP doing it, but the combination of that SEXP with a very unusual set of circumstances that never occurs in retail and a player behavior.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Iss Mneur on April 17, 2010, 06:42:11 pm
Summed up, the problem was simply that you never considered that a retail sexp could cause such an issue, because after ten years of working and improving on the game engine, one could easily believe that every retail bug would have been fixed...

Ah, there are so many ways to crash a game engine :rolleyes: At least we can learn a lesson from this, young padawan...

Also because we'd seen the issue occur before for totally different reasons. I imagine there are still ways to trigger it. And it wasn't just the retail SEXP doing it, but the combination of that SEXP with a very unusual set of circumstances that never occurs in retail and a player behavior.

I imagine it actually has little to to do the sexp other than it was what caused the player's ship max speed to be zero.  Anything that causes max-speed to become zero while speed-match was on would likely cause the bug.  That is, it was the speed-match function that was at fault and not the sexp.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Droid803 on April 17, 2010, 06:55:24 pm
So if you say, tried to pilot a sentry gun and used match speed, the glitch would happen?

EDIT: lol damn that was a lame 5000th post.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 17, 2010, 06:58:08 pm
I'm going to try that.
You got 5,000 posts.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Shivan Hunter on April 17, 2010, 07:29:30 pm
Y'know, I think I'm going to make a (short) FS campaign about this bug. The prospect is just too awesome to pass up. That and I've been playing Transcend all day, which may have had a hand in the decision >.>
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: ChronoReverse on April 17, 2010, 07:54:13 pm
Would be interesting to simulate this effect as part of a campaign.

Great job bug-squashing though :yes:


I would've never seen this bug because I rarely use speed-matching.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: MatthTheGeek on April 18, 2010, 01:51:57 am
Remember they said they had fixed the cause of the bug, so make sure to not use future builds :)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Rhymes on April 18, 2010, 09:41:57 am
  :eek2:

Holy ****.

You guys deserve a medal, or three.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Iss Mneur on April 18, 2010, 11:41:33 am
So if you say, tried to pilot a sentry gun and used match speed, the glitch would happen?

Probably, though I don't know if speed match will enable while max speed is zero.

Remember they said they had fixed the cause of the bug, so make sure to not use future builds :)

At this time the patch for this bug has not being applied to either the 3.6.12 branch or to trunk.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: MatthTheGeek on April 18, 2010, 02:12:59 pm
That's why I said future builds...
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: jr2 on April 18, 2010, 08:22:48 pm
So if you say, tried to pilot a sentry gun and used match speed, the glitch would happen?

Probably, though I don't know if speed match will enable while max speed is zero.


I'm pretty sure it does.  What happens when you match speeds with an enemy cruiser and kill its engine subsystem?  ;)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Commander Zane on April 18, 2010, 08:27:51 pm
You just stop like the Cruiser, I've done that on several occasions before.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: jr2 on April 18, 2010, 08:37:30 pm
Yup and you're match speed stays on.  And if you match speeds with a stationary target, you just drop to 0 speed.  :nod:  FS1 match speeds with the Orff.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 18, 2010, 08:51:21 pm
So if you say, tried to pilot a sentry gun and used match speed, the glitch would happen?

Probably, though I don't know if speed match will enable while max speed is zero.


I'm pretty sure it does.  What happens when you match speeds with an enemy cruiser and kill its engine subsystem?  ;)

That's not what he means. He means your max speed, i.e. the player.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 18, 2010, 08:57:15 pm
Yep. The trigger condition for this particular bug was that the player's max speed was set to 0 and then used as the divisor without anyone checking whether it was 0.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Rodo on April 18, 2010, 09:10:45 pm
And if you match speeds with a stationary target, you just drop to 0 speed.

IIRC you can't match speeds with an object that is at full stop.

I wonder what would happen if you match speeds to a fighter that has negative acc while flying a fighter that can't do that :drevil: (I remember flying some fighters that had negative speeds)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 18, 2010, 09:13:38 pm
It will still have a positive speed (as it is quite impossible for your velocity to become negative). As such, you'll be matching speed with it. Of course, you won't be going in the same direction.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Scotty on April 19, 2010, 06:16:31 am
It will still have a positive speed (as it is quite impossible for your velocity to become negative). As such, you'll be matching speed with it. Of course, you won't be going in the same direction.

Velocity is a vector and can therefore has both direction and magnitude, so it can be negative, depending on where the point of reference is.  Speed cannot be negative, because it has no direction.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: The E on April 19, 2010, 06:29:00 am
I blame the inaccuracies of human language.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Snail on April 19, 2010, 09:20:27 am
Semantics lol
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: kaloonzu on April 19, 2010, 08:22:34 pm
It is human nature to laugh at other peoples's frustration; its a psychological defense mechanism. The amount of frustration is often proportional to the amount of laughter.
Which I am finding out now.
The hard way.
Because I have been doubled over, trying not to vomit from laughter, at the sheer amount of scrap metal I would have created had that happened to me.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: High Max on April 19, 2010, 08:50:57 pm
;-)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Droid803 on April 19, 2010, 08:51:24 pm
It is human nature to laugh at other peoples's frustration; its a psychological defense mechanism. The amount of frustration is often proportional to the amount of laughter.

No, I just think it means that person lacks compassion and empathy. Also, a person who truly cares about you wouldn't laugh at you or at your pain. Maybe the people who do that should get clocked and see how they like it. I don't think it is funny unless I don't like the kind of person they are.

To be fair, this is the internet, where you can't even see the other guy.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: High Max on April 19, 2010, 08:52:50 pm
;-)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 19, 2010, 09:37:38 pm
Ignore High Max, I thought it was a rather cogent analysis.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: High Max on April 19, 2010, 10:11:50 pm
;-)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Aardwolf on April 19, 2010, 10:14:44 pm
And I will try to ignore your self righteous arrogant attitude. Same problem with The E. I refuse to believe it is human nature. It could also be called human nature to slap someone upside the head for laughing at someone's misfortune, of course there's something called self control and people can control laughter whether it is human nature or not, though self control these days is sorely lacking, and the problem seems to increase over time.

[/off-topic rant]
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Herra Tohtori on April 19, 2010, 10:18:06 pm
(...)

[/off-topic rant]

[/metapost]


Nothing further needs to be said on this tangent, I believe.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Aardwolf on April 19, 2010, 10:27:07 pm
So uh... what's this about ", Human Blood" in the thread title?
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 19, 2010, 10:32:41 pm
So uh... what's this about ", Human Blood" in the thread title?

Read the story!
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Dilmah G on April 20, 2010, 06:23:41 am
It was quite a pesky little mother****er, as I understand.  ;)
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Angelus on April 20, 2010, 12:00:56 pm
So uh... what's this about ", Human Blood" in the thread title?

Black Magic usually involves blood rituals, and by performing such a ritual the BP team was able to find and squish this annoying bug.
Since they didn't had the time to find a suitable victim candidate for the ritual, General Battuta volunteered for the sake of a bugfree game. He put his life at risk, but seeing the result, it was worth it.
This dedication demands our respect!




 :P
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: Cobra on April 20, 2010, 03:53:46 pm
Bull****, they poked me with needles in the middle of the night.
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: General Battuta on April 20, 2010, 05:36:30 pm
Did no one administer the Oblivius Curse? NO ONE?
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: mjn.mixael on May 02, 2010, 09:22:49 pm
http://sale.images.woot.com/How_Natural_Selection_Works8tdDetail.png (http://sale.images.woot.com/How_Natural_Selection_Works8tdDetail.png)

This seemed appropriate.. specifically, the third line...  :lol:
Title: Re: Epic Bughunt Claims Sanity, Human Blood
Post by: carbine7 on May 03, 2010, 04:18:53 am
I must say, this story gave me quite a laugh. Who would have thought of a bug that would set the player's location outside of the universe? Great job, btw.

Sorry I can't resist saying this, but you've just been divided by 0.  :pimp: