Author Topic: Let's talk about the sound code... again.  (Read 27771 times)

0 Members and 2 Guests are viewing this topic.

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Re: Let's talk about the sound code... again.
The problem with the full 3d, multi-source audio is that, there is no cutoff on the volume and the number of sources in a certain point.  The old sound code, implicitly had them because the old sound code would only attempt to play so many sounds a time (the "baby feeding") so it didn't overload the sound cards of the day (afaict, the old sound code is from retail except for being ported to openAL), whereas the new sound code sends every single sound to openAL and lets it figure out what to do. The end result is things like a wing warping in results in 4-6 warp sounds played at the same time, which being an already loud sound results in the extreme clipping everyone is complaining about.
There is a volume cutoff, it's just a question of whether it works properly or not.  [V] tried to get around some DS3D issues and I didn't take that code out since I didn't understand exactly what they were trying to do, how DS3D wasn't working the way they needed it too, or whether/how it's different for OpenAL.

There is also a cut off point for the number of sounds being played.  The old code would, unless a certain flag was set, only allow a particular sound to be playing one at the time.  The new code tries to be smarter about allowing things like that when there are plenty of sound sources free for use.  Also the total number of sounds is only marginally higher than it used to be, 48 compared to 32 in the old code.  And the difference there is because in the original DS code, effects and music used different sources of playback, whereas it all goes through the same OpenAL device and people were having voices and music not play due to sound effects taking up all of the available/allocated sound sources.

Quote
Unfortunately, though I think I know what the problem is, I am not familiar enough with OpenAL to know how to fix it, and I haven't had time look into it.
If you have an idea then spit it out and I'll help you figure out if that's the problem and how to go about fixing it.


Also it should be noted that in the test build thread for the new sound code that someone mentioned that the code broke with the go_faster commit.  And before that point, if up to date drivers were used, along with the OpenAL Soft DLL, then the new sound code worked fine.  So it's possible that there is an issue in there which isn't actually directly related to the sound code.

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
That was me.
But I've got a new pc now.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline Iss Mneur

  • 210
  • TODO:
Re: Let's talk about the sound code... again.
The problem with the full 3d, multi-source audio is that, there is no cutoff on the volume and the number of sources in a certain point.  The old sound code, implicitly had them because the old sound code would only attempt to play so many sounds a time (the "baby feeding") so it didn't overload the sound cards of the day (afaict, the old sound code is from retail except for being ported to openAL), whereas the new sound code sends every single sound to openAL and lets it figure out what to do. The end result is things like a wing warping in results in 4-6 warp sounds played at the same time, which being an already loud sound results in the extreme clipping everyone is complaining about.
There is a volume cutoff, it's just a question of whether it works properly or not.  [V] tried to get around some DS3D issues and I didn't take that code out since I didn't understand exactly what they were trying to do, how DS3D wasn't working the way they needed it too, or whether/how it's different for OpenAL.
Okay. I will have to take a closer look to find where it is done then.

There is also a cut off point for the number of sounds being played.  The old code would, unless a certain flag was set, only allow a particular sound to be playing one at the time.  The new code tries to be smarter about allowing things like that when there are plenty of sound sources free for use.  Also the total number of sounds is only marginally higher than it used to be, 48 compared to 32 in the old code.  And the difference there is because in the original DS code, effects and music used different sources of playback, whereas it all goes through the same OpenAL device and people were having voices and music not play due to sound effects taking up all of the available/allocated sound sources.
Okay, I recall seeing what you are referring to, but the issue is still one of, a sound possibly being played as multiple sources (ie. a wing warping in) at the same spot making what is already a loud sound (the warp and the lightning) that much louder.

Indeed, sorry if I was misleading in this regard, I haven't looked at the sound code in quite a while.  Note, I am not trying to be evasive, I am trying to get some other code done up for the fiction viewer and command briefings that I promised to do long ago.

Unfortunately, though I think I know what the problem is, I am not familiar enough with OpenAL to know how to fix it, and I haven't had time look into it.
If you have an idea then spit it out and I'll help you figure out if that's the problem and how to go about fixing it.
I don't have anything that I haven't already stated with regards to what I outlined above about the multiple already loud sounds being played in close proximity, whereas, as far as I recall/could tell, the old code usually ended up only playing one of the loud sounds at a time.

Also it should be noted that in the test build thread for the new sound code that someone mentioned that the code broke with the go_faster commit.  And before that point, if up to date drivers were used, along with the OpenAL Soft DLL, then the new sound code worked fine.  So it's possible that there is an issue in there which isn't actually directly related to the sound code.
That was me.
But I've got a new pc now.
Do you still have the same problem with those builds?  If the builds have been removed, I can rebuild them for you.
"I love deadlines. I like the whooshing sound they make as they fly by." -Douglas Adams
wxLauncher 0.9.4 public beta (now with no config file editing for FRED) | wxLauncher 2.0 Request for Comments

 

Offline Goober5000

  • HLP Loremaster
  • Moderator
  • 214
    • Goober5000 Productions
Re: Let's talk about the sound code... again.
I noticed when playtesting a recent mission that it sounded as if warpin sounds made by a wing very far away were being played with a volume as if they occurred next to the player.

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
Do you still have the same problem with those builds?  If the builds have been removed, I can rebuild them for you.
Not with my current pc I don't. And my old one got salvaged for parts so I can't go back and test it.
What the problem boiled down to was that with the new sound code, it sounded like everything took part in a cave. Then with soft openal installed it was resolved. But then when the go_faster code was commited the sound (and framerate) would become horribly distorted when too many sounds were playing. 
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline taylor

  • Super SCP/Linux Guru
  • Moderator
  • 212
    • http://www.icculus.org/~taylor
Re: Let's talk about the sound code... again.
I noticed when playtesting a recent mission that it sounded as if warpin sounds made by a wing very far away were being played with a volume as if they occurred next to the player.
Depending on the sound, that may very well be what happens.  Not everything gets played in 3D, but all sounds occur in 3D space.  So 2D sounds are played relative to the listener's (you) position in space using stereo panning.  Meaning that 2D sounds are literally played right next to you.

So somebody probably needs to look and see if the warp sound is actually played in 3D or not.  Just find where the sound is played in the code and see if it used snd_play() or snd_play_3d().  I know that the nebula lightning is played in 2D, so if the warp sound is as well then that narrows down where the problem might be.

 
Re: Let's talk about the sound code... again.
Although the lightning issue is partially volume it is mostly frequency.
The nebula was never designed to play the lightning sound every time the lightning showed up, or if it was, it was done with the thought in mind that it wouldn't work.
Playing them allllll is never ever going to work, volume aside.
"Neutrality means that you don't really care, cuz the struggle goes on even when you're not there: Blind and unaware."

"We still believe in all the things that we stood by before,
and after everything we've seen here maybe even more.
I know we're not the only ones, and we were not the first,
and unapologetically we'll stand behind each word."

  

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Let's talk about the sound code... again.
i dont know but i think i always played retail under eax because at the time i had a fairly decent sound blaster and thats what most games were using. but for the last several years ive mostly been satisfied with the onboard sound on whatever mobo i was using at the time. i also got away from the whatever.1 speakers everyone seems to like, last 5 years or so ive just been using full range stereo speaks and whatever analog stereo amp i could find at the time and i say the sound is much better.

anyway sounds like you need some real time gain correction code to level out the levels when you have a lot of sounds playing simultaneously. back in the old days it was one sound, one channel and the sound hardware would do the rest of the mixing for you. if a more modern sound setup tries to cram multiple sounds in the same channel by adding the samples (in the correct phase of course) then the result would be higher amplitude, not an actual mix. you would need to re-normalize the sample so that the amplitude remains constant. of course im not sure if the api works at that low of a level. id hate to roll back to direct sound over openal when the openal implementation can be fixed, given the right skill set.
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN

 

Offline Iss Mneur

  • 210
  • TODO:
Re: Let's talk about the sound code... again.
anyway sounds like you need some real time gain correction code to level out the levels when you have a lot of sounds playing simultaneously. back in the old days it was one sound, one channel and the sound hardware would do the rest of the mixing for you. if a more modern sound setup tries to cram multiple sounds in the same channel by adding the samples (in the correct phase of course) then the result would be higher amplitude, not an actual mix. you would need to re-normalize the sample so that the amplitude remains constant. of course im not sure if the api works at that low of a level. id hate to roll back to direct sound over openal when the openal implementation can be fixed, given the right skill set.
Well that may be, though I think that OpenAL would already do something like that to help manage sounds.

Although the lightning issue is partially volume it is mostly frequency.
The nebula was never designed to play the lightning sound every time the lightning showed up, or if it was, it was done with the thought in mind that it wouldn't work.
Playing them allllll is never ever going to work, volume aside.
Yes.  It seems this was/is an issue.  The code that is in trunk does not enforce the built in concurrent playcount limits for any sound unless there is pressure on available playback channels.  For example fighter sized ships are only supposed to play two warp-in sounds at a time.  The nebula lightning there should only be two of both lightning variation.

So somebody probably needs to look and see if the warp sound is actually played in 3D or not.  Just find where the sound is played in the code and see if it used snd_play() or snd_play_3d().  I know that the nebula lightning is played in 2D, so if the warp sound is as well then that narrows down where the problem might be.
I can confirm that nebula lightning is played in 2D, but the warpin sounds are defiantly played in 3d unless the 3d sound playback is disabled.



As I mentioned in my reply to QuantumDelta, I think I may have found the problem.  It seems that the concurrent play count limits that are in place on a lot of the sounds (in particular the really loud sounds, aka warpin and lightning) were only being enforced once there was pressure on the available playback channels, which means that as I hypothesized previously about the limit on number of sounds at one time was actually true and in my limited testing (warp sounds in one mission) seems to help with the clipping and/or excessive volume.

So, here is a patch (below) and a release and a debug trunk based build below (remember that debug builds do not have lightning enabled) with the patch applied for everyones testing pleasure :P.

Code: (always enforce the concurrent sound playcount.patch) [Select]
Index: code/sound/ds.cpp
===================================================================
--- code/sound/ds.cpp (revision 7535)
+++ code/sound/ds.cpp (working copy)
@@ -1407,25 +1407,26 @@
  }
  }
 
+ // Make sure that we are not going to play more copies of this sound than we should be
+ if ( (instance_count >= limit) && (lowest_instance_vol_index >= 0) ) {
+ // If there is a lower volume duplicate, stop it.... otherwise, don't play the sound
+ if (lowest_instance_vol <= new_volume) {
+ ds_close_channel_fast(lowest_instance_vol_index);
+ first_free_channel = lowest_instance_vol_index;
+ }
+ }
+
  if (first_free_channel < 0) {
- // If we've exceeded the limit, then maybe stop the duplicate if it is lower volume
- if ( (instance_count >= limit) && (lowest_instance_vol_index >= 0) ) {
- // If there is a lower volume duplicate, stop it.... otherwise, don't play the sound
- if (lowest_instance_vol <= new_volume) {
- ds_close_channel_fast(lowest_instance_vol_index);
- first_free_channel = lowest_instance_vol_index;
+ // still don't have a channel and
+ // there is no limit barrier to play the sound, so see if we've ran out of channels
+ // stop the lowest volume instance to play our sound if priority demands it
+ if ( (lowest_vol_index != -1) && (priority == DS_MUST_PLAY) ) {
+ // Check if the lowest volume playing is less than the volume of the requested sound.
+ // If so, then we are going to trash the lowest volume sound.
+ if ( Channels[lowest_vol_index].vol <= new_volume ) {
+ ds_close_channel_fast(lowest_vol_index);
+ first_free_channel = lowest_vol_index;
  }
- } else {
- // there is no limit barrier to play the sound, so see if we've ran out of channels
- // stop the lowest volume instance to play our sound if priority demands it
- if ( (lowest_vol_index != -1) && (priority == DS_MUST_PLAY) ) {
- // Check if the lowest volume playing is less than the volume of the requested sound.
- // If so, then we are going to trash the lowest volume sound.
- if ( Channels[lowest_vol_index].vol <= new_volume ) {
- ds_close_channel_fast(lowest_vol_index);
- first_free_channel = lowest_vol_index;
- }
- }
  }
  }
 
"I love deadlines. I like the whooshing sound they make as they fly by." -Douglas Adams
wxLauncher 0.9.4 public beta (now with no config file editing for FRED) | wxLauncher 2.0 Request for Comments

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
Iss Mneur, you are a king! A King I say!

Using the build you provided, a warp in of 6 fighters no longer makes my ears start bleeding and a turret firing 4 shots at once now sound properly!
Get this **** into the trunk pronto!
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline Iss Mneur

  • 210
  • TODO:
Re: Let's talk about the sound code... again.
How does this change affect nebula lightning?
"I love deadlines. I like the whooshing sound they make as they fly by." -Douglas Adams
wxLauncher 0.9.4 public beta (now with no config file editing for FRED) | wxLauncher 2.0 Request for Comments

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
Nebula lighting doesn't seem to have changed much in frequency, doesn't seem to clip or anything though.
I'm personally content with the current fixes to the warpin and turret sounds. I hardly even recall what the nebula lighting originally sounded like tbh. So I'm going to hand that issue over to QD cause he has a better clue on it.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 
Re: Let's talk about the sound code... again.
This will, I hope, illustrate the differences;

http://www.youtube.com/watch?v=0yIz0O6jQp0

Sorry about the MVP use.

It is clear though that despite the slightly softer MVP effect, the game produced volume is much louder and much more frequent than with the retail sound code, pre-the new mvp (BP) lightning it could be really rather deafening.
Now it's not quite as bad but still something that could really do with proper re-tweaking.

As you can see however, the go_faster and Iss' re-tweak didn't effect the sound codes issues at all on this matter (warp ins seem fixed? you will probably notice however that one of the clips is completely devoid of warpin sound?).


Edit; for anyone really quick back to this reply, the video is about 10 minutes from finish of upload as of this post and will probably be another 5-10 minutes to fully encode. :P
"Neutrality means that you don't really care, cuz the struggle goes on even when you're not there: Blind and unaware."

"We still believe in all the things that we stood by before,
and after everything we've seen here maybe even more.
I know we're not the only ones, and we were not the first,
and unapologetically we'll stand behind each word."

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
Okay noticed just now that it's not working 100% perfectly. If you are at a certain range, the turret sounds will still clip like they used to. Its okay if you are close to the turret firing but after you put a certain amount of distance inbetween you and the turret it reverts to its old heathen ways.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline Iss Mneur

  • 210
  • TODO:
Re: Let's talk about the sound code... again.
That was very helpful QuantumDelta, it illustrates the issue quite clearly. I was able to replicate the issue.  I don't know how you guys could stand the lightning (I never noticed because I normally end up playing with a debug build).

I think I may have been able to figure out what the problem is, I just don't know how to fix if it is actually the problem and/or how to fix it.  What appears to be the problem with the lightning is that we are not allowing the sounds to play through, so we are always hearing the beginning of the rumble now, rather than having 40 playing at a time.

@Spoon, I have noticed that as well, but I am not really sure what the cause is, I need to look into it more. Do you have an example (with retail assets) that will do it?
"I love deadlines. I like the whooshing sound they make as they fly by." -Douglas Adams
wxLauncher 0.9.4 public beta (now with no config file editing for FRED) | wxLauncher 2.0 Request for Comments

 

Offline Kolgena

  • 211
Re: Let's talk about the sound code... again.
We can't stand the lightning :P It was just brought up a bunch of times, and those threads all died from inactivity.

I'm very glad that a fix is being looked into.

 

Offline Spoon

  • 212
  • ヾ(´︶`♡)ノ
Re: Let's talk about the sound code... again.
With retail assets? No, as far as I know there is no ship with a 3 or 4 firing point turret in salvo mode.
I could record you a video of the issue though.
Urutorahappī!!

[02:42] <@Axem> spoon somethings wrong
[02:42] <@Axem> critically wrong
[02:42] <@Axem> im happy with these missions now
[02:44] <@Axem> well
[02:44] <@Axem> with 2 of them

 

Offline chief1983

  • Still lacks a custom title
  • Moderator
  • 212
  • ⬇️⬆️⬅️⬅️🅰➡️⬇️
    • Minecraft
    • Skype
    • Steam
    • Twitter
    • Fate of the Galaxy
Re: Let's talk about the sound code... again.
Wait, so it starts playing a bunch of lightning sounds, but instead of waiting to add another one until the earlier ones are done, it just starts shifting them off starting at the first one played?
Fate of the Galaxy - Now Hiring!  Apply within | Diaspora | SCP Home | Collada Importer for PCS2
Karajorma's 'How to report bugs' | Mantis
#freespace | #scp-swc | #diaspora | #SCP | #hard-light on EsperNet

"You may not sell or otherwise commercially exploit the source or things you created based on the source." -- Excerpt from FSO license, for reference

Nuclear1:  Jesus Christ zack you're a little too hamyurger for HLP right now...
iamzack:  i dont have hamynerge i just want ptatoc hips D:
redsniper:  Platonic hips?!
iamzack:  lays

 

Offline Herra Tohtori

  • The Academic
  • 211
  • Bad command or file name
Re: Let's talk about the sound code... again.
Okay, this might be a silly question but... it'd be sillier not to ask it.


On GIMP, there are two different layer modes for doing almost the same thing: Addition and Screen.

They have a big practical difference, though - Addition simply adds the pixel values of the layer on top to the pixel values of the layer below, which can result to clipping if the values go over 255 for each channel.

Screen mode is more sophisticated, however. It never causes clipping, because it checks how much room there is available to the clipping point, then scales the top layer's pixel values to fill the available scale proportionally to the top layer's pixel values.


Example: I have two layers; bottom layer is a 75% grey (192,192,192 or #c0c0c0) and top layer is 50% grey (128,128,128 or #808080).

Additive blending mode would count the pixel values directly together and end up with clipped white at 192+128=320>255.

Screen mode first determines how much available values there are before clipping occurs:

255-192 = 63



Then it determines how much it wants to increase the value perceptually. The 50% grey would mean a 50% increase of available values. In this case,

63 * 128/255 = 31.6235294 = ~32



...so the final pixel value is

192 + 32 = 224

and the colour ends up as (224,224,224 or #e0e0e0).

Mathematically, it goes as follows:



RGB(out) = RGB(bottom) + ( [ RGB(white) - RGB(bottom) ] * [ RGB(top) / RGB(white) ] )


Now, for image editing, every RGB variable consists of three numerals for RGB, four with RGBA (but that works a bit differently as the alpha channel of top layer is a multiplier for the RGB values in 0,1 range...) and the range for each channel is 0,255.

For audio, we have only one value (sample) but lots of them coming in at high rate of speed (typically 44100 in one second minimum) and the size of individual sample is usually 16bit, 24bit or 32bit integer. Most commonly the sound data is in 44k16bit format; however while processing the sound, they can either be converted to higher sample size or kept at the same 16 bit (at slight sacrifice of rounding errors).


Regardless, with additive mixing of two samples, the sample values are just added together and if they go over the clipping threshold you end up with clipping sound.


With a "screen" type mixing of samples, you will never get clipping; the volume will just asymptotically approach 0 dB. Of course sound is a bit different as waveform contains both positive and negative values, while image data is limited to positive values; however the equation can probably be adjusted to take negative values into account and prevent clipping for negative values as well.


Now, the questions:

Is there some flaw in my thinking as to why this wouldn't work? If yes, what is it?


If this should theoretically work, then:


Does OpenAL support this type of mixing that never clips? If it is, how to use it?

If this is not possible, then I have to ask... why not? Is it too resource-intensive? It wouldn't seem so, as it's all rather simple maths - you just need to do it pretty fast (at least 44100 times per second multiplied by the amount of sounds used).
There are three things that last forever: Abort, Retry, Fail - and the greatest of these is Fail.

 

Offline Nuke

  • Ka-Boom!
  • 212
  • Mutants Worship Me
Re: Let's talk about the sound code... again.
i think the problem with this is the sound api, which should in theory handle all this low level stuff for you and only gives you control of higher level structures, sounds, channels, etc. of course i never used openal before for anything i ever wrote, nor would i. i would probably just use the sound features of sdl, since im somewhat familiar with it.
I can no longer sit back and allow communist infiltration, communist indoctrination, communist subversion, and the international communist conspiracy to sap and impurify all of our precious bodily fluids.

Nuke's Scripting SVN