I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution.
The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'.
I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect.
The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot.
After too much messing around I discovered that at least some of the problem is the mixing of sounds.
Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well.
Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four?
To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard.
So - some workspace code to illustrate
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
"OK, because both sound defualt to quieter" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait.
FMSound brass1 play. 1 second wait. snd stopGracefully.
"nasty glitch using #loudness: plus mixing graunch" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd loudness: 1.0 . 1 second wait. FMSound brass1 play. 1 second wait. snd stopGracefully.
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play. 1 second wait. snd stopGracefully.
Aargh!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A computer program does what you tell it to do, not what you want it to do.
Hi Tim,
tl,dr Let's build a limiter.
I stopped using Squeak for Audio with 3.8 :-) With that in mind:
I believe that all sound goes out through SoundPlayer playing SoundBuffers. The samples get created via mixSsampleCountInto.... which seems to be implemented by many classes and there they all seem to do clipping.
After running some of your examples I found several SoundBuffers containing samples 32767 and -32768 which indicates clipping. I didn't find enough consecutive clipped samples (you shouldn't hear a single one) but that's because I only found a few SoundBuffers. BTW running your example labelled "OK because.." also clipped when run several times.
I didn't find the place where float values are changed to Integers for output.
I did not find any limiter. --> baaad!
So if the above is true and the problem is really the clipping (I had the impression I also heard some wrapping where 32768 uses the top bit as a sign) I suggest: -No clipping during the mixing -Find the place where the SoundBuffers go to the hardware -Implement a limiter there
Basically you need a controlled gain. You look ahead in your SoundBuffer for min and max. In case of overflow you calculate the necessary gain and gradually reduce your actual gain in the time you have until that sample really gets output. (attack phase). If no over/underflow is found you (more) gradually raise your gain back to 1 (release phase). Maybe introduce some delay before raising your gain again (hold phase).
For live real time audio you may not have the luxury to look ahead. You still reduce your gain gradually (let's say in 10 ms) and clip until you reached your final gain. Human's ears tend to tolerate this.
There's tons of compromises to evaluate with a limiter we should go with a bit over TSTTCPW to save CPU. Maybe have another preference to choose the system wide limiter. SCNR :-)))
I can help with rough implementations and audiowise discussions but have no idea where to hook it in and cannot convert it to a primitive or optimize it.
Let's wait until Stef chimes in, I'm sure he has solved that for muO already.
Best regards,
Herbert
Am 16.12.2020 um 23:52 schrieb tim Rowledge:
I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution.
The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'.
I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect.
The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot.
After too much messing around I discovered that at least some of the problem is the mixing of sounds.
Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well.
Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four?
To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard.
So - some workspace code to illustrate
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
"OK, because both sound defualt to quieter" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait.
FMSound brass1 play.
1 second wait. snd stopGracefully.
"nasty glitch using #loudness: plus mixing graunch" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd loudness: 1.0 .
1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
Aargh!
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A computer program does what you tell it to do, not what you want it to do.
Hi all,
just noticed SoundBuffer is 16 bits so my suggestions are not practicable.
Cheers,
Herbert
Am 17.12.2020 um 14:14 schrieb Herbert König:
Hi Tim,
tl,dr Let's build a limiter.
I stopped using Squeak for Audio with 3.8 :-) With that in mind:
I believe that all sound goes out through SoundPlayer playing SoundBuffers. The samples get created via mixSsampleCountInto.... which seems to be implemented by many classes and there they all seem to do clipping.
After running some of your examples I found several SoundBuffers containing samples 32767 and -32768 which indicates clipping. I didn't find enough consecutive clipped samples (you shouldn't hear a single one) but that's because I only found a few SoundBuffers. BTW running your example labelled "OK because.." also clipped when run several times.
I didn't find the place where float values are changed to Integers for output.
I did not find any limiter. --> baaad!
So if the above is true and the problem is really the clipping (I had the impression I also heard some wrapping where 32768 uses the top bit as a sign) I suggest: -No clipping during the mixing -Find the place where the SoundBuffers go to the hardware -Implement a limiter there
Basically you need a controlled gain. You look ahead in your SoundBuffer for min and max. In case of overflow you calculate the necessary gain and gradually reduce your actual gain in the time you have until that sample really gets output. (attack phase). If no over/underflow is found you (more) gradually raise your gain back to 1 (release phase). Maybe introduce some delay before raising your gain again (hold phase).
For live real time audio you may not have the luxury to look ahead. You still reduce your gain gradually (let's say in 10 ms) and clip until you reached your final gain. Human's ears tend to tolerate this.
There's tons of compromises to evaluate with a limiter we should go with a bit over TSTTCPW to save CPU. Maybe have another preference to choose the system wide limiter. SCNR :-)))
I can help with rough implementations and audiowise discussions but have no idea where to hook it in and cannot convert it to a primitive or optimize it.
Let's wait until Stef chimes in, I'm sure he has solved that for muO already.
Best regards,
Herbert
Am 16.12.2020 um 23:52 schrieb tim Rowledge:
I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution.
The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'.
I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect.
The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot.
After too much messing around I discovered that at least some of the problem is the mixing of sounds.
Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well.
Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four?
To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard.
So - some workspace code to illustrate
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
"OK, because both sound defualt to quieter" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait.
FMSound brass1 play.
1 second wait. snd stopGracefully.
"nasty glitch using #loudness: plus mixing graunch" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd loudness: 1.0 .
1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
Aargh!
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A computer program does what you tell it to do, not what you want it to do.
Hi Herbert,
_,,,^..^,,,_ (phone)
On Dec 18, 2020, at 4:46 AM, Herbert König herbertkoenig@gmx.net wrote:
Hi all,
just noticed SoundBuffer is 16 bits so my suggestions are not practicable.
But adding a signed 32-bit, or even a signed 64-bit collection for mixing would be straight forward, except for ensuring it mixes down to 16 bits. A signed 32-bit collection would allow 32k 16-bit sources to be additively mixed without clipping, right? So an underlying 64-bit collection is unnecessary. So a MixedSound that adds into an underlying 32-bit signed sequence, and updates a count of added sounds, and does interpolation when adding sounds of different sample rates, and outputs 16-bit samples derived by dividing the 32-bit sums by the count of added sounds would be a solution right?
Cheers,
Herbert
Am 17.12.2020 um 14:14 schrieb Herbert König: Hi Tim,
tl,dr Let's build a limiter.
I stopped using Squeak for Audio with 3.8 :-) With that in mind:
I believe that all sound goes out through SoundPlayer playing SoundBuffers. The samples get created via mixSsampleCountInto.... which seems to be implemented by many classes and there they all seem to do clipping.
After running some of your examples I found several SoundBuffers containing samples 32767 and -32768 which indicates clipping. I didn't find enough consecutive clipped samples (you shouldn't hear a single one) but that's because I only found a few SoundBuffers. BTW running your example labelled "OK because.." also clipped when run several times.
I didn't find the place where float values are changed to Integers for output.
I did not find any limiter. --> baaad!
So if the above is true and the problem is really the clipping (I had the impression I also heard some wrapping where 32768 uses the top bit as a sign) I suggest: -No clipping during the mixing -Find the place where the SoundBuffers go to the hardware -Implement a limiter there
Basically you need a controlled gain. You look ahead in your SoundBuffer for min and max. In case of overflow you calculate the necessary gain and gradually reduce your actual gain in the time you have until that sample really gets output. (attack phase). If no over/underflow is found you (more) gradually raise your gain back to 1 (release phase). Maybe introduce some delay before raising your gain again (hold phase).
For live real time audio you may not have the luxury to look ahead. You still reduce your gain gradually (let's say in 10 ms) and clip until you reached your final gain. Human's ears tend to tolerate this.
There's tons of compromises to evaluate with a limiter we should go with a bit over TSTTCPW to save CPU. Maybe have another preference to choose the system wide limiter. SCNR :-)))
I can help with rough implementations and audiowise discussions but have no idea where to hook it in and cannot convert it to a primitive or optimize it.
Let's wait until Stef chimes in, I'm sure he has solved that for muO already.
Best regards,
Herbert
Am 16.12.2020 um 23:52 schrieb tim Rowledge: I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution.
The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'.
I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect.
The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot.
After too much messing around I discovered that at least some of the problem is the mixing of sounds.
Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well.
Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four?
To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard.
So - some workspace code to illustrate
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
"OK, because both sound defualt to quieter" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait.
FMSound brass1 play.
1 second wait. snd stopGracefully.
"nasty glitch using #loudness: plus mixing graunch" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd loudness: 1.0 .
1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
Aargh!
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim A computer program does what you tell it to do, not what you want it to do.
I've made a tiny bit of progress on fighting this and I've discovered some points that might help with understanding why a VolumeEnvelope stops the volume adjustment working.
So first, using a simple FMSound with the base volume set to 40% solves a lot of problems in practice when it comes to the note-chooser in Scratch - simply keeping the volume lower avoids any clamping. The simple FMSound has not volume envelope and so the volume control actually works.
Looking at how the volume envelope works it seems like it shouldn't cause any issues; after all the operation of the envelope uses the #adjustVolume... method. I think that we're looking at the attempt to set the volume with it is clashing with the VolumeEnvelope>>#updateTargetAt: stomping on the related variables. That happens every time the #doControl method runs, which is nominally 100Hz. Somewhere there is a way to composite the two and get the desired result. I just don't see it right now because I'm trying too many things at once.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "How many Motie Warriors does it take to change a lightbulb?” "None. One of the dead ones will do it."
Well nothing to do with sound ever seems as simple as one would like.
Remember, the waggling keyboard thing makes horrific scratchy noises BUT it doesn't do that on a Mac running exactly the same Smalltalk code, so I got suspicious.
I now have three Pi setups that make completely different sounds.
1) an old Pi-top CEED with a Pi 3 and a May 2017 OS, a)with the as-provided NuScratch. This makes perfectly ok sounds under the piano-waggle regime, event though the volume is 100%. b) same Pi-top but using a newer (but still 2017 vintage) Squeak system, where the sound seems fine. Both cases are using the old release of alsa.
2) my main daily use Pi4 with the last non-pulse version of OS. a) using the supplied NuScratch it makes barely audible noises that are terrible when actually audible. Even the cat meow is terrible! b) using an early-this-year Squeak VM there is just no sound at all
3) a fresh Pi4 with the release candidate OS Simon pointed me to. a) in as-supplied NuScratch with a pulse audio sound lib added the sound is bad unless the internal volume is turned way down, when there are still nasty scratchy-glitches b) in a new system the sound is still bad but much nearer to ok with the internal volume reduced.
As mentioned, the Mac runs the exact same Smalltalk image just fine. I don't have Windows so I can't opine on that option. Other than "Eww, Windows?"
The obvious thing to dig into now is what changes have been made to the lowest-level unix sound stuff since 2016. Anybody got ideas? Could it even be changes to the unix libraries in the pulse/alsa world?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: MET: Misread and Eat Tape
There's always a chance that the OS meddles with it too. Notably on Windows if you simultaneously play and record it looses buffers sometimes. And often it is the combination of application and OS.
I didn't find the place where Squeak audio goes out to the OS but writing this to a (preferably .wav) file would give a chance to analyse what happens inside Squeak. If anybody can post the necessary changes I can do that and check.
I will set up a Pi and record the audio (hardware) output with another computer to see the sum of the effects.
Are the snippets in your original post those which produce the problem on a Pi? Would be more comparable then waggling the keyboard.
Cheers,
Herbert
Am 24.12.2020 um 04:32 schrieb tim Rowledge:
Well nothing to do with sound ever seems as simple as one would like.
Remember, the waggling keyboard thing makes horrific scratchy noises BUT it doesn't do that on a Mac running exactly the same Smalltalk code, so I got suspicious.
I now have three Pi setups that make completely different sounds.
- an old Pi-top CEED with a Pi 3 and a May 2017 OS,
a)with the as-provided NuScratch. This makes perfectly ok sounds under the piano-waggle regime, event though the volume is 100%. b) same Pi-top but using a newer (but still 2017 vintage) Squeak system, where the sound seems fine. Both cases are using the old release of alsa.
- my main daily use Pi4 with the last non-pulse version of OS.
a) using the supplied NuScratch it makes barely audible noises that are terrible when actually audible. Even the cat meow is terrible! b) using an early-this-year Squeak VM there is just no sound at all
- a fresh Pi4 with the release candidate OS Simon pointed me to.
a) in as-supplied NuScratch with a pulse audio sound lib added the sound is bad unless the internal volume is turned way down, when there are still nasty scratchy-glitches b) in a new system the sound is still bad but much nearer to ok with the internal volume reduced.
As mentioned, the Mac runs the exact same Smalltalk image just fine. I don't have Windows so I can't opine on that option. Other than "Eww, Windows?"
The obvious thing to dig into now is what changes have been made to the lowest-level unix sound stuff since 2016. Anybody got ideas? Could it even be changes to the unix libraries in the pulse/alsa world?
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: MET: Misread and Eat Tape
On 2020-12-24, at 2:35 AM, Herbert König herbertkoenig@gmx.net wrote:
There's always a chance that the OS meddles with it too. Notably on Windows if you simultaneously play and record it looses buffers sometimes. And often it is the combination of application and OS.
Yah. It certainly isn't impossible that changes to the underlying libraries are causing issues; after, my original work on this was done six years ago and the basic sound related stuff may be 20 years old.
I didn't find the place where Squeak audio goes out to the OS but writing this to a (preferably .wav) file would give a chance to analyse what happens inside Squeak. If anybody can post the necessary changes I can do that and check.
I can see three plausible places where it might be smart to record things a) copy the sound buffer samples as sent to the prim in the #playLoop method b) copy the samples actually sent to the OS by the VM c) copy the sound output by the OS
I suspect that somebody can point to an application to record the final output cleanly.
I will set up a Pi and record the audio (hardware) output with another computer to see the sum of the effects.
I was almost at the point of just aiming my iPhone at the screen.
Are the snippets in your original post those which produce the problem on a Pi? Would be more comparable then waggling the keyboard.
Yes and no; that code shows off how the volume level can drive things into a screeching mess.
Here is another snippet that seems to illustrate things a bit better -
|snd volume | volume:= 40. 10 timesRepeat: [ snd := FMSound new. "250 milliSeconds wait." snd setPitch: (AbstractSound pitchForMIDIKey: 60) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully. "250 milliSeconds wait." snd := FMSound new. snd setPitch: (AbstractSound pitchForMIDIKey: 58) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully]
You might experiment with - the volume setting - the variety of FMSound (organ1 includes an envelope or two to mix things up a bit) - the commented out between-sound delay, which I used to try to avoid two sounds overlapping in order to test the hypothesis that mixing was messing up
And depending on the update level of your image, the "X milliSeconds wait" might need to be changed back to "(Delay forMIlliSeconds: X) wait" etc.
I get a strangely interesting effect that is probably suggesting something important (and that I'm not yet getting) with the between-sound delay used and set to 150 mS. It's as if there is some mangling of buffers going on. At 100mS it goes berserk; at 200 it sounds perfectly regular.
I'm going to indulge in some code scanning to try to spot changes made that might affect this. I'll be interested how you see, or rather hear, things.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother!" said Pooh, as Piglet pressed <START> on the Microwave...
Hi Tim,
ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS but it still doesn't say "Raspberry Pi OS"
With some googling I got Audio working from Audacity (simple Audio Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439.
Recording with a phone will show some glitches but the bandwidth of phone (and monitor speakers) mellows down (err filters) the waveform. Surely only second choice.
I would not mind downloading and using the proper Squeak to get sound going.
Any quick help?
Cheers,
Herbert
Am 24.12.2020 um 20:58 schrieb tim Rowledge:
On 2020-12-24, at 2:35 AM, Herbert König herbertkoenig@gmx.net wrote:
There's always a chance that the OS meddles with it too. Notably on Windows if you simultaneously play and record it looses buffers sometimes. And often it is the combination of application and OS.
Yah. It certainly isn't impossible that changes to the underlying libraries are causing issues; after, my original work on this was done six years ago and the basic sound related stuff may be 20 years old.
I didn't find the place where Squeak audio goes out to the OS but writing this to a (preferably .wav) file would give a chance to analyse what happens inside Squeak. If anybody can post the necessary changes I can do that and check.
I can see three plausible places where it might be smart to record things a) copy the sound buffer samples as sent to the prim in the #playLoop method b) copy the samples actually sent to the OS by the VM c) copy the sound output by the OS
I suspect that somebody can point to an application to record the final output cleanly.
I will set up a Pi and record the audio (hardware) output with another computer to see the sum of the effects.
I was almost at the point of just aiming my iPhone at the screen.
Are the snippets in your original post those which produce the problem on a Pi? Would be more comparable then waggling the keyboard.
Yes and no; that code shows off how the volume level can drive things into a screeching mess.
Here is another snippet that seems to illustrate things a bit better -
|snd volume | volume:= 40. 10 timesRepeat: [ snd := FMSound new. "250 milliSeconds wait." snd setPitch: (AbstractSound pitchForMIDIKey: 60) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully. "250 milliSeconds wait." snd := FMSound new. snd setPitch: (AbstractSound pitchForMIDIKey: 58) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully]
You might experiment with
- the volume setting
- the variety of FMSound (organ1 includes an envelope or two to mix things up a bit)
- the commented out between-sound delay, which I used to try to avoid two sounds overlapping in order to test the hypothesis that mixing was messing up
And depending on the update level of your image, the "X milliSeconds wait" might need to be changed back to "(Delay forMIlliSeconds: X) wait" etc.
I get a strangely interesting effect that is probably suggesting something important (and that I'm not yet getting) with the between-sound delay used and set to 150 mS. It's as if there is some mangling of buffers going on. At 100mS it goes berserk; at 200 it sounds perfectly regular.
I'm going to indulge in some code scanning to try to spot changes made that might affect this. I'll be interested how you see, or rather hear, things.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "Bother!" said Pooh, as Piglet pressed <START> on the Microwave...
On 2020-12-24, at 12:18 PM, Herbert König herbertkoenig@gmx.net wrote:
Hi Tim,
ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS but it still doesn't say "Raspberry Pi OS"
Hmm, updating a Pi can be confusing at times.
If you do `uname -a` in a terminal, what do you see?
On my still-alsa machine I get Linux Goldskin 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
On my test-pulse I get Linux Pi-4-1-pulse 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux
Remember that you need to include the vm sound flag when you start up squeak! I keep forgetting that. Though.. if you're start scratch with the standard menu entry, that should do it. Let's see - there is also the possibility you are not directing sound to the right output. The Pi sound-volume control widget on the menubar can set where the sound goes - right-click on it and you should see a menu with at least Analogue HDMI entries. I use the analogue and headphones since none of my Pi have a direct HDMI display attached but go via VNC. If you have an HDMI monitor attached it may not have speakers or you might not have them turned on! DAMHIKT.
With some googling I got Audio working from Audacity (simple Audio Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439.
Recording with a phone will show some glitches but the bandwidth of phone (and monitor speakers) mellows down (err filters) the waveform. Surely only second choice.
I would not mind downloading and using the proper Squeak to get sound going.
The 5.3 armlinux package from squeak.org ought to do ok for this. Remembering of course to include that '-vm-sound-alsa' (or pulse) in your commandline
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: PRM: PRint Money
Am 24.12.2020 um 21:36 schrieb tim Rowledge:
On 2020-12-24, at 12:18 PM, Herbert König herbertkoenig@gmx.net wrote:
Hi Tim,
ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS but it still doesn't say "Raspberry Pi OS"
Hmm, updating a Pi can be confusing at times.
If you do `uname -a` in a terminal, what do you see?
On my still-alsa machine I get Linux Goldskin 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
Linux herpi2 5.4.79-V7+ .. Rest same as yours above.
On my test-pulse I get Linux Pi-4-1-pulse 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux
Remember that you need to include the vm sound flag when you start up squeak! I keep forgetting that. Though.. if you're start scratch with the standard menu entry, that should do it. Let's see - there is also the possibility you are not directing sound to the right output. The Pi sound-volume control widget on the menubar can set where the sound goes - right-click on it and you should see a menu with at least Analogue HDMI entries. I use the analogue and headphones since none of my Pi have a direct HDMI display attached but go via VNC. If you have an HDMI monitor attached it may not have speakers or you might not have them turned on! DAMHIKT.
I use the 3.5mm headphone output and verified that via Audacity and VLC. I have a cheap active speaker connected. Just checked: Scratch 2 plays sound Scratch1 started via the menu doesn't. Now downloading a fresh Squeak.
I tried the -vmsound alsa and pulse but I'm not sure If I used the right place in the script. I tried editing the last line to: exec "${VM}" "-vm-sound-alsa" "${IMAGE}" (and pulse and w/o the "" but my Linux fu is not enough to know if that's ok.
Cheers,
Herbert
With some googling I got Audio working from Audacity (simple Audio Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439.
Recording with a phone will show some glitches but the bandwidth of phone (and monitor speakers) mellows down (err filters) the waveform. Surely only second choice.
I would not mind downloading and using the proper Squeak to get sound going.
The 5.3 armlinux package from squeak.org ought to do ok for this. Remembering of course to include that '-vm-sound-alsa' (or pulse) in your commandline
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: PRM: PRint Money
Ha, ran the script in a terminal and on each piano keypress I get: sound_Start(default) soundStart: snd_add_pcm_handler: Function not implemented.
... Handling two mouses and two keyboards is quite confusing ... Have to run to pick up my wife
Hoping for some enlightenment tomorrow :-)
Cheers,
Herbert
Am 24.12.2020 um 22:05 schrieb Herbert König:
Am 24.12.2020 um 21:36 schrieb tim Rowledge:
On 2020-12-24, at 12:18 PM, Herbert König herbertkoenig@gmx.net wrote:
Hi Tim,
ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS but it still doesn't say "Raspberry Pi OS"
Hmm, updating a Pi can be confusing at times.
If you do `uname -a` in a terminal, what do you see?
On my still-alsa machine I get Linux Goldskin 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux
Linux herpi2 5.4.79-V7+ .. Rest same as yours above.
On my test-pulse I get Linux Pi-4-1-pulse 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux
Remember that you need to include the vm sound flag when you start up squeak! I keep forgetting that. Though.. if you're start scratch with the standard menu entry, that should do it. Let's see - there is also the possibility you are not directing sound to the right output. The Pi sound-volume control widget on the menubar can set where the sound goes - right-click on it and you should see a menu with at least Analogue HDMI entries. I use the analogue and headphones since none of my Pi have a direct HDMI display attached but go via VNC. If you have an HDMI monitor attached it may not have speakers or you might not have them turned on! DAMHIKT.
I use the 3.5mm headphone output and verified that via Audacity and VLC. I have a cheap active speaker connected. Just checked: Scratch 2 plays sound Scratch1 started via the menu doesn't. Now downloading a fresh Squeak.
I tried the -vmsound alsa and pulse but I'm not sure If I used the right place in the script. I tried editing the last line to: exec "${VM}" "-vm-sound-alsa" "${IMAGE}" (and pulse and w/o the "" but my Linux fu is not enough to know if that's ok.
Cheers,
Herbert
With some googling I got Audio working from Audacity (simple Audio Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439.
Recording with a phone will show some glitches but the bandwidth of phone (and monitor speakers) mellows down (err filters) the waveform. Surely only second choice.
I would not mind downloading and using the proper Squeak to get sound going.
The 5.3 armlinux package from squeak.org ought to do ok for this. Remembering of course to include that '-vm-sound-alsa' (or pulse) in your commandline
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: PRM: PRint Money
On 2020-12-24, at 1:21 PM, Herbert König herbertkoenig@gmx.net wrote:
Ha, ran the script in a terminal and on each piano keypress I get: sound_Start(default) soundStart: snd_add_pcm_handler: Function not implemented.
That's an error to do with some asynchronous handler stuff in alsa. I have no idea why it would fail.
All in all this sound stuff is getting very, very, frustrating
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: NOP: Randomize the PSW and then branch
So another data point.
Linux Mint 20 (so basically Ubuntu) on x86-64. Happens to be a MacBook Air but I don't think that's important.
My normal install only build vm-sound-OSS. And with that there is no sound and a complaint in the terminal about /dev/dsp not existing. Which is true, /dev/dsp does not exist.
But if install libpluse-dev before I build the VM then I also get vm-sound-pluse.
tmp/opensmalltalk-vm/build.linux64x64/[squeak.cog.spur/build/squeak](http://squeak.cog.spur/build/squeak) -vm-sound-pulse myimage
Good news - I get sound!
Bad news - Well it seems a bit drunk in that it pauses about every second to two. It acts like a buffering issue. It is slightly worse without doing the RT priority limits.conf file.
I'll also poke at this a bit.
cheers
bruce
On 2020-12-24, at 1:21 PM, Herbert König wrote:
Ha, ran the script in a terminal and on each piano keypress I get: sound_Start(default) soundStart: snd_add_pcm_handler: Function not implemented.
That's an error to do with some asynchronous handler stuff in alsa. I have no idea why it would fail.
All in all this sound stuff is getting very, very, frustrating
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: NOP: Randomize the PSW and then branch
I just downloaded Squeak on my up to date Linux Mint computer (a Core I 3 from 2009) and right at the start (before the message you report) I get the message copied below.
No sound with vm-sound-alsa but sound with vm-sound-pulse and always the same message. But it sounds like high latency and not as bad as on the Pi (with the keyboard). When waggling the keyboard Squeak goes from 5% to 56% CPU. That seems a lot to me.
Cheers,
Herbert
"Using /home/herbert/Schreibtisch/Squeak5.3alpha-19256-64bit-201911012148-Linux/bin/squeak... pthread_setschedparam failed: Operation not permitted This VM uses a separate heartbeat thread to update its internal clock and handle events. For best operation, this thread should run at a higher priority, however the VM was unable to change the priority. The effect is that heavily loaded systems may experience some latency issues. If this occurs, please create the appropriate configuration file in /etc/security/limits.d/ as shown below:
cat <<END | sudo tee /etc/security/limits.d/squeak.conf * hard rtprio 2 * soft rtprio 2 END
and report to the squeak mailing list whether this improves behaviour.
You will need to log out and log back in for the limits to take effect. For more information please see https://github.com/OpenSmalltalk/opensmalltalk-vm/releases/tag/r3732#linux "
Am 25.12.20 um 13:55 schrieb Bruce O'Neel:
So another data point.
Linux Mint 20 (so basically Ubuntu) on x86-64. Happens to be a MacBook Air but I don't think that's important.
My normal install only build vm-sound-OSS. And with that there is no sound and a complaint in the terminal about /dev/dsp not existing. Which is true, /dev/dsp does not exist.
But if install libpluse-dev before I build the VM then I also get vm-sound-pluse.
tmp/opensmalltalk-vm/build.linux64x64/squeak.cog.spur/build/squeak http://squeak.cog.spur/build/squeak -vm-sound-pulse myimage
Good news - I get sound!
Bad news - Well it seems a bit drunk in that it pauses about every second to two. It acts like a buffering issue. It is slightly worse without doing the RT priority limits.conf file.
I'll also poke at this a bit.
cheers
bruce
/25 December 2020 05:41 tim Rowledge tim@rowledge.org wrote:/
> On 2020-12-24, at 1:21 PM, Herbert König wrote: > > Ha, ran the script in a terminal and on each piano keypress I get: > sound_Start(default) > soundStart: snd_add_pcm_handler: Function not implemented. That's an error to do with some asynchronous handler stuff in alsa. I have no idea why it would fail. All in all this sound stuff is getting very, very, frustrating tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: NOP: Randomize the PSW and then branch
Hi,
Just for reference
padsp squeak image -vm-sound-OSS
produces for me the same pauses every second or two.
cheers
bruce
Am 25.12.2020 um 05:41 schrieb tim Rowledge:
On 2020-12-24, at 1:21 PM, Herbert König herbertkoenig@gmx.net wrote:
Ha, ran the script in a terminal and on each piano keypress I get: sound_Start(default) soundStart: snd_add_pcm_handler: Function not implemented.
That's an error to do with some asynchronous handler stuff in alsa. I have no idea why it would fail.
All in all this sound stuff is getting very, very, frustrating
Same on my RPi4. Audio is working except Squeak and Scratch1, same error. So I tried my Pi2 which I haven't updated in times but mounted on a RPi touch display -->no sound at all and no Volume control in the taskbar.
Since the RPI2 I use them for Audio recording and playing and audio measurements. They all saw an IQaudio DAC + and Digiamp so maybe that's part of the problem. Everything always worked out of the box, just I never used Squeak.
Maybe it has to do with the recent switch from Alsa to Pulse Audio (just names for me) but the Pi2 should not have seen this.
Anyway I got some time so I will check the messages from Subbu and Bruce and then just do the reverse. Use Squeak audio on Windows (Just checked, 5.3 has the nasty effects) and record with my soundcard on a Pi.
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
Cheers,
Herbert
On 25/12/20 6:49 pm, Herbert König wrote:
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
You may use parec (sound capture to file) and paplay (file to sound output) to test the audio subsystem of the OS to rule out any audio driver or mixer configuration problems [1]. The corresponding ALSA utilities are arecord and aplay. The mixer is alsamixer.
padsp acts as a relay between the older /dev/dsp (audio device under OSS) and the newer /dev/snd/* (audio i/o devices).
[1] https://manpages.debian.org/testing/pulseaudio-utils/pacat.1.en.html
HTH .. Subbu
Hey you seem to know Linux's audio stuff.
Thanks!
Herbert
Am 25.12.2020 um 14:54 schrieb K K Subbu:
On 25/12/20 6:49 pm, Herbert König wrote:
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
You may use parec (sound capture to file) and paplay (file to sound output) to test the audio subsystem of the OS to rule out any audio driver or mixer configuration problems [1]. The corresponding ALSA utilities are arecord and aplay. The mixer is alsamixer.
padsp acts as a relay between the older /dev/dsp (audio device under OSS) and the newer /dev/snd/* (audio i/o devices).
[1] https://manpages.debian.org/testing/pulseaudio-utils/pacat.1.en.html
HTH .. Subbu
I'm glad a few people around here do - there seems to be a bewildering mix of things out there.
Looking at the assorted C code for soundplugin, sqUnixSound, sqUnixSoundALSA & SsqUnixSoundPulseAudio I see a bunch of places where pointer types don't agree and just maybe there are some problems there. The worst potential for pointer issues is avoided in practice because we never actually pass any value for startIndex other than 0 (the '1' used in all the Smalltalk is pre-decremented in SoundPlugin>primitiveSoundPlaySamples)
Both the ALSA & pulseaudio files appear to connect to the semaphore in a reasonable manner. Ther pulseaudio code is clearly long untouched and in need of a lot of tidying up and the addition of a number of new routines like sound_GetDefaultSoundPlayer - try using that prim and it just dumps you VM with extreme prejudice!
I think one of the really puzzling things right now is how does my older Pi setup *not* make such terrible noises? The scratch image & vm involved are *identical* to the ones on the newer-but-still-alsa Pi.
On 2020-12-25, at 6:03 AM, Herbert König herbertkoenig@gmx.net wrote:
Hey you seem to know Linux's audio stuff.
Thanks!
Herbert
Am 25.12.2020 um 14:54 schrieb K K Subbu:
On 25/12/20 6:49 pm, Herbert König wrote:
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
You may use parec (sound capture to file) and paplay (file to sound output) to test the audio subsystem of the OS to rule out any audio driver or mixer configuration problems [1]. The corresponding ALSA utilities are arecord and aplay. The mixer is alsamixer.
padsp acts as a relay between the older /dev/dsp (audio device under OSS) and the newer /dev/snd/* (audio i/o devices).
[1] https://manpages.debian.org/testing/pulseaudio-utils/pacat.1.en.html
HTH .. Subbu
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim When flying inverted, remember that down is up and up is expensive
I've managed to record some samples with parecord on the pulseaudio Pi. As usual, the man page for parecord etc was about as helpful as a chocolate teapot in a sauna, but some flailing around on google eventually found a page with something that worked.
parecord --channels=1 -d alsa_output.platform-bcm2835_audio.analog-mono.2.monitor testsound.wav
records the output of the headphone socket channel until you ctrl-C. I also tried with the HDMI output in case this is all down to some bug in the headphpone output - which has happened in the past. When playing the two samples I couldn't hear any difference, so I think that saves one level of nightmare.
I'm loathe to dump a meg-and-a-half of horrible noise file in list mail, or indeed on the swiki - so you ought to be able to download from https://mega.nz/folder/iN1wWajL#SkljQIWhHP8c6McE9HDwDg
Currently there are two files, the Pi4-pulse-1.wav is the noise made by my code snippet plus some waggling of mouse over the normal pianokeyboardmorph, and -2 is waggling the mouse over the scratch note chooser keyboard morph.
I'd add files for the alsa Pi but I can't make arecord work at all. As usual, making sense of a man page is ridiculous.
I've tried a variety -
arecord -d 5 -f cd -D -t wav output Pi4-alsa-1.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM -t arecord: main:828: audio open error: No such file or directory
arecord -d 5 -f cd -D output -t wav Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
arecord -d 5 -f cd -t wav Pi4-alsa-1.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
arecord -d 5 -f cd -t wav foo ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
arecord -d 5 -D output -t wav Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
arecord -d 5 -D output -t wav ./Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
arecord -d 5 foo.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
I even found the arecord.c sources and looked for the error message - of course, there is no reasonable way to work out what file is not there...
Advice on a suitable command most welcome.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: BBL: Branch on Burned out Light
On 26/12/20 4:09 am, tim Rowledge wrote:
I'd add files for the alsa Pi but I can't make arecord work at all. As usual, making sense of a man page is ridiculous.
;-). A man(ual) page is detailed spec. Reading all options can be quite intimidating. I just skip to the examples section in the beginning. I only use options to override defaults.
arecord -d 5 -f cd -D -t wav output Pi4-alsa-1.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM -t arecord: main:828: audio open error: No such file or directory
-D requires an argument.
$ arecord -d 5 -f cd Pi4-alsa-1.wav
will sample mic at 44khz (16-bit little endian) for 5 seconds and store it in wav format file (-t wav is the default anyway).
arecord -d 5 -f cd -D output -t wav Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
The "output" argument to -D is not a valid audio input device. What valid audio input devices does "arecord -l" show? You could also use alsamixer TUI to see all valid input/output devices, check their gain/vol settings and test recording and playback.
arecord -d 5 -f cd -t wav Pi4-alsa-1.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
arecord -d 5 -f cd -t wav foo ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
arecord -d 5 -D output -t wav Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
arecord -d 5 -D output -t wav ./Pi4-alsa-1.wav arecord: main:828: audio open error: No such file or directory
arecord -d 5 foo.wav ALSA lib pcm.c:2565:(snd_pcm_open_noupdate) Unknown PCM input arecord: main:828: audio open error: No such file or directory
The default device is not available (bad driver?) or misconfigured. Try listing audio devices with arecord -l and then pick the appropriate name to use with -D.
HTH .. Subbu
On 2020-12-25, at 10:28 PM, K K Subbu kksubbu.ml@gmail.com wrote:
On 26/12/20 4:09 am, tim Rowledge wrote:
I'd add files for the alsa Pi but I can't make arecord work at all. As usual, making sense of a man page is ridiculous.
;-). A man(ual) page is detailed spec. Reading all options can be quite intimidating. I just skip to the examples section in the beginning. I only use options to override defaults.
Me too - except it always seems that the examples section has been deleted when things are really confusing. I claim that this is all part of a Vast Global Conspiracy To Confuse Us. It's working.
$ arecord -d 5 -f cd Pi4-alsa-1.wav
will sample mic at 44khz (16-bit little endian) for 5 seconds and store it in wav format file (-t wav is the default anyway).
Yeah, I saw that example somewhere but it returns the same old "arecord: main:828: audio open error: No such file or directory" error as the others.
All the other cases I posted were as recommended on assorted pages, which is terribly frustrating. I think maybe my alsa-Pi has got some wider alsa setup issue - which is strange since I haven't ever deliberately done any sound stuff on this particular unit and so it should be completely default. None of the examples to test the sound seem to work - 'sudo aplay /usr/share/sounds/alsa/Front_Center.wav' makes no sound at all. And yet Squeak does make faint and feeble (scratchy) noises ... totally nuts.
I guess I can try a totally fresh load-up on yet another Pi to compare.
The "output" argument to -D is not a valid audio input device. What valid audio input devices does "arecord -l" show?
Well, 'arecord -l' shows no devices, which isn't surprising on a bare Pi. But this is supposed to be recording what gets sent to the headphone output, so all the examples I've seen have referred to things like 'output' and 'default' and 'hw:1,0' etc etc. The parecord on the pulse-Pi had no problem.
OK, I'll put this email aside for a moment while I fire up the 'very old alsa Pi' to see if it can do better.
Later - the old alsa-Pi system (that 2017 version OS) plays sound from Scratch perfectly well, and even when playing a looped sample guitar I can thrash up and down the pianokeyboard without glitches. I can bang on the 'normal' pianokeyboardmorph without scratchy-glitches. My code snippet plays fine even with volume set to 100% with either a plan FMSound or an organ1 sound (that adds the envelope stuff to the mix). The test samples stuff ' aplay /usr/share/sounds/alsa/Front_Center.wav' works fine and clear.
However, I can't record the headphone output with arecord which makes it difficult to share. I bet there's some magic incantation to make the headphone out channel appear as an input.
So - 4 y.o. Pi system makes nice sounds. iMac makes nice sounds. Newer alsa Pi makes nasty sounds but may be borked in other ways. New pulse-Pi makes nasty sounds. Herbert has shown convincing evidence of bad clipping occurring. Dave's chord example makes graunchy noises on all machines *except iMac*, which is interesting, and if the volume is increased even the mac makes graunch.
I guess this implies we need to check the clipping code at the very least. And I've already mentioned that the pulse sound driver needs a bunch of additional code for new stuff. Such fun!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim The downside of being better than everyone else is that people tend to assume you're pretentious
Tim, thanks for the recording.
No idea if that happens inside Squeak but basically clipping is not working. In the image instead of going from positive full-scale to negative full-scale the signal should just stay at the resp. full-scale. Instead of clipping 32767 + 1 to 32767 the code uses 32768 as being - full-scale (wraparound). See pulse1PosWrap.png. For negative full-scale the symmetric thing happens. The signal should stay at full-scale there.
mixSampleCount: n into: aSoundBuffer startingAt: startIndex leftVol: leftVol rightVol: rightVol seems to do the right thing: leftVol > 0 ifTrue: [ i := (2 * outIndex) - 1. s := (aSoundBuffer at: i) + ((sample * leftVol) // ScaleFactor). s > 32767 ifTrue: [s := 32767]. "clipping!" s < -32767 ifTrue: [s := -32767]. "clipping!" aSoundBuffer at: i put: s]. except we can go to 32767 and -32768 IIRC but that's no problem.
If you look at Pi4-pulse-1.wav second 5.2352 it seems there's multiple wraparounds going on. See pulse1MultipleWrap. For this I have no explanation because the signal if it were unclipped looks like going to about 1.5 * full-scale.
pulse2ClipAndWrap shows that somewhere (red mark) the normal clipping is going on (sounds bad if this strong, that's why I suggested a limiter) and close to this the wrapping is going on. Maybe there are places where the clipping is taking something >32767 as max.
Or (a wild idea): SoundBuffer stores 16 bit unsigned quantities. So if a signal is first put into a sound buffer w/o clipping and only the mixing does the clipping and that happens in several places (several sounds generated independently) that could explain the wrapping. That's why I gave up when I saw that SoundBuffer only uses 16 bits.
Here Eliot's suggestion from Dec 18:
"But adding a signed 32-bit, or even a signed 64-bit collection for mixing would be straight forward, except for ensuring it mixes down to 16 bits. A signed 32-bit collection would allow 32k 16-bit sources to be additively mixed without clipping, right? So an underlying 64-bit collection is unnecessary. So a MixedSound that adds into an underlying 32-bit signed sequence, and updates a count of added sounds, and does interpolation when adding sounds of different sample rates, and outputs 16-bit samples derived by dividing the 32-bit sums by the count of added sounds would be a solution right? "
could make sense. I'd suggest using 32 Bit Float. Seeing loudness being 0.0 to 1.0 and scale in Envelope having Float values > 1 (I see even 25) I hope that this already happens. But looking at e.g. FMSound's class var SineTable I see it is a SoundBuffer, on which we operate with float operations and then we must go back to 16 Bit unsigned to store the result. Lots of room for human error in this.
All audio precessing (incl. mixing) should be done in Float (usually normalized to +-1.0) and then use a limiter and then go to 16 bit sound buffers. Or better use 32 bit Float for output to the OS too. Usually AD and DA conversion is done with 24 bit by moderately cheap audio cards so I guess most OSses use 32 bit float. But that's just a guess.
Cheers,
Herbert
Am 25.12.2020 um 23:39 schrieb tim Rowledge:
I've managed to record some samples with parecord on the pulseaudio Pi. As usual, the man page for parecord etc was about as helpful as a chocolate teapot in a sauna, but some flailing around on google eventually found a page with something that worked.
......
On 2020-12-26, at 4:57 AM, Herbert König herbertkoenig@gmx.net wrote:
Tim, thanks for the recording.
No idea if that happens inside Squeak but basically clipping is not working. In the image instead of going from positive full-scale to negative full-scale the signal should just stay at the resp. full-scale. Instead of clipping 32767 + 1 to 32767 the code uses 32768 as being - full-scale (wraparound). See pulse1PosWrap.png.
Staring at the generated C code it looks possibly suspicious that we have
short int *aSoundBuffer; sqInt s; if (leftVol > 0) { i = (2 * sliceIndex) - 1; s = (aSoundBuffer[i]) + ((sample * leftVol) / ScaleFactor); if (s > 0x7FFF) { s = 0x7FFF; } if (s < -32767) { s = -32767; } aSoundBuffer[i] = s; } But surely no C compiler screws that up these days?
I added printfs to show when over/underflow happened and indeed it happens - duh.
I've compared latest and ancient (as in ~2016 vintage) copies of SoundPlugin.c, SoundGenerationPlugin.c and sqUnixSoundALSA.c and found nothing that looks anything other than trivial changes - dates, some macros, etc.
There doesn't seem to be any place other than primitiveMixFMSound() that is involved in clipping sample values. None of the lowest level send-to-OS code looks to do any sort of casting or limiting.
I've even just for grins tried limiting the 's' value with a s & 0x3FFF just to see what happens. It sounds a bit strange but also seems to remove the clicks, so maybe there is something in it.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: BYEBYE: Store in Write-Only Storage
I searched all method sources for 3276 and only found one suspicious place in : adjustVolumeTo: vol overMSecs: mSecs "Adjust the volume of this sound to the given volume, a number in the range [0.0..1.0], over the given number of milliseconds. The volume will be changed a little bit on each sample until the desired volume is reached."
| newScaledVol |
self flag: #bob. "I removed the upper limit to allow making sounds louder. hmm..."
newScaledVol := (32768.0 * vol) truncated. "<------------------------------- might exceed 32767 if vol = 1.0 -> wrap not clip" newScaledVol = scaledVol ifTrue: [^ self].
but changing that to 32767 didn't help. Also I didn't find the version where Bob had not yet removed the limit (my oldest image being 3.6) so that is most probably not the culprit. Also I looked at a lot of in image code but didn't find anything.
Next step is to try to record the output before it goes to the OS and see if the image or some primitive creates the problem.
Cheers,
Herbert
Am 27.12.2020 um 01:57 schrieb tim Rowledge:
On 2020-12-26, at 4:57 AM, Herbert König herbertkoenig@gmx.net wrote:
Tim, thanks for the recording.
No idea if that happens inside Squeak but basically clipping is not working. In the image instead of going from positive full-scale to negative full-scale the signal should just stay at the resp. full-scale. Instead of clipping 32767 + 1 to 32767 the code uses 32768 as being - full-scale (wraparound). See pulse1PosWrap.png.
Staring at the generated C code it looks possibly suspicious that we have
short int *aSoundBuffer; sqInt s; if (leftVol > 0) { i = (2 * sliceIndex) - 1; s = (aSoundBuffer[i]) + ((sample * leftVol) / ScaleFactor); if (s > 0x7FFF) { s = 0x7FFF; } if (s < -32767) { s = -32767; } aSoundBuffer[i] = s; } But surely no C compiler screws that up these days?
I added printfs to show when over/underflow happened and indeed it happens - duh.
I've compared latest and ancient (as in ~2016 vintage) copies of SoundPlugin.c, SoundGenerationPlugin.c and sqUnixSoundALSA.c and found nothing that looks anything other than trivial changes - dates, some macros, etc.
There doesn't seem to be any place other than primitiveMixFMSound() that is involved in clipping sample values. None of the lowest level send-to-OS code looks to do any sort of casting or limiting.
I've even just for grins tried limiting the 's' value with a s & 0x3FFF just to see what happens. It sounds a bit strange but also seems to remove the clicks, so maybe there is something in it.
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: BYEBYE: Store in Write-Only Storage
On 2020-12-27, at 10:49 AM, Herbert König herbertkoenig@gmx.net wrote:
I searched all method sources for 3276 and only found one suspicious place in : adjustVolumeTo: vol overMSecs: mSecs
I even tried not using the adjustVol... at all in case that was a cause - no change whatsoever.
Next stop, using GDB. I hate using GDB.
I suppose one option is also faking a nice clean sound buffer and finding a way to feed it as directly as possible to various stages. Maybe it's even a problem in the audio libraries! Not like we haven't discovered similar issues in the past.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Hipatitis (n): Terminal coolness
I changed SoundPlayer class playLoop which I figured is what passes the samples to the OS like attached, writing 1 mega samples to an aiff file. TSTTCPW. I must have made a mistake because I suddenly get lost buffers but frankly I had some wine :-).
I used Tim's first snippet:
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 // 2 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
It's repeatable the sound seems to have the right length but the discontinuity marked red usually happens if the sound system of the OS looses a buffer. I had my laptop run at 4 GHZ, the audio output sounded normally bad and the loss is in the first second, where the audio is still ok. Playing the recorded file reveals the additional distortions.
I just want to save another trip through the time zones so I put it out buggy as it is. Maybe I just used the wrong place to do my recording, maybe I recorded mono from a stereo buffer, whatever.
Be mild :-)
Herbert
Am 27.12.2020 um 19:49 schrieb Herbert König:
Next step is to try to record the output before it goes to the OS and see if the image or some primitive creates the problem.
Cheers,
Herbert
I tested Tim's snippet (modified to use Delay forSeconds: ) on Squeak 3.6 (Windows) and it has the same problem. But it's only clipping and no wrapping on Windows! Still very ugly.
Tomorrow I'll switch to a Pi to see if my recording to file also shows wrapping. Then we'll know if it's the primitive handling audio to the OS in Linux or Rapbian or if it happens inside Squeak's sound architecture.
Cheers,
Herbert
Am 27.12.2020 um 23:03 schrieb Herbert König:
I changed SoundPlayer class playLoop which I figured is what passes the samples to the OS like attached, writing 1 mega samples to an aiff file. TSTTCPW. I must have made a mistake because I suddenly get lost buffers but frankly I had some wine :-).
I used Tim's first snippet:
"awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait.
FMSound brass1 setPitch: 470 // 2 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully
It's repeatable the sound seems to have the right length but the discontinuity marked red usually happens if the sound system of the OS looses a buffer. I had my laptop run at 4 GHZ, the audio output sounded normally bad and the loss is in the first second, where the audio is still ok. Playing the recorded file reveals the additional distortions.
I just want to save another trip through the time zones so I put it out buggy as it is. Maybe I just used the wrong place to do my recording, maybe I recorded mono from a stereo buffer, whatever.
Be mild :-)
Herbert
Am 27.12.2020 um 19:49 schrieb Herbert König:
Next step is to try to record the output before it goes to the OS and see if the image or some primitive creates the problem.
Cheers,
Herbert
Thanks for banging on this Herbert - I suspect a few drinks might actually help with this one...
GDB is such fun, not.
So far as I can see the algorithm is doing correct things - the 16bit values get read & written without whacking other values, we get suitably matching left & right channel values, when the value over/under flows the max value is properly written, etc etc. Of course, it's extra fun to mentally deal with the 'short' addressing.
I've double checked the 'FMSound new' wavetable values in case they changed across some release but they appear to be the same.
Anybody got any more ideas? In particular, a recipe for recording the output with alsa would be interesting so at least I could add the record of the old-alsa-pi getting it right!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Strange OpCodes: KFP: Kindle Fire in Printer
On 26/12/20 1:02 am, tim Rowledge wrote:
I think one of the really puzzling things right now is how does my older Pi setup*not* make such terrible noises? The scratch image & vm involved are*identical* to the ones on the newer-but-still-alsa Pi.
Audio drivers and servers can make a difference between distro versions.
Didn't Raspbian switch to pulse audio server to pulse in their recent release [1]?
[1] https://www.raspberrypi.org/blog/new-raspberry-pi-os-release-december-2020/
Regards .. Subbu
Thanks for the pointer to Raspbian switching to Pulse Audio.
I thought the opening paragraph was a nice bit of understatement.
First, a bit of background. Audio on Linux is really quite complicated. There are multiple different standards for handling audio input and output, and it does sometimes seem that what has happened, historically, is that whenever anyone wanted to use audio in Linux, they looked at the existing libraries and programs and went “Hmmm… I don’t like that, I’ll write something new and better.” This has resulted in a confused mass of competing and conflicting software, none of which *quite* works the way anyone wants it to!
On Fri, Dec 25, 2020 at 11:32:52AM -0800, tim Rowledge wrote:
On 2020-12-25, at 6:03 AM, Herbert K??nig herbertkoenig@gmx.net wrote:
Am 25.12.2020 um 14:54 schrieb K K Subbu:
On 25/12/20 6:49 pm, Herbert K??nig wrote:
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
You may use parec (sound capture to file) and paplay (file to sound output) to test the audio subsystem of the OS to rule out any audio driver or mixer configuration problems [1]. The corresponding ALSA utilities are arecord and aplay. The mixer is alsamixer.
padsp acts as a relay between the older /dev/dsp (audio device under OSS) and the newer /dev/snd/* (audio i/o devices).
[1] https://manpages.debian.org/testing/pulseaudio-utils/pacat.1.en.html
HTH .. Subbu
Hey you seem to know Linux's audio stuff.
Thanks!
Herbert
I'm glad a few people around here do - there seems to be a bewildering mix of things out there.
Looking at the assorted C code for soundplugin, sqUnixSound, sqUnixSoundALSA & SsqUnixSoundPulseAudio I see a bunch of places where pointer types don't agree and just maybe there are some problems there. The worst potential for pointer issues is avoided in practice because we never actually pass any value for startIndex other than 0 (the '1' used in all the Smalltalk is pre-decremented in SoundPlugin>primitiveSoundPlaySamples)
Both the ALSA & pulseaudio files appear to connect to the semaphore in a reasonable manner. Ther pulseaudio code is clearly long untouched and in need of a lot of tidying up and the addition of a number of new routines like sound_GetDefaultSoundPlayer - try using that prim and it just dumps you VM with extreme prejudice!
I think one of the really puzzling things right now is how does my older Pi setup *not* make such terrible noises? The scratch image & vm involved are *identical* to the ones on the newer-but-still-alsa Pi.
Tim, I adapted your earlier code example as follows:
volume := 0.30. { 55 . 60 . 64 } collect: [ :e | FMSound new setPitch: (AbstractSound pitchForMIDIKey: e) dur: 5 loudness: volume] thenDo: [:s | s play. (Delay forSeconds: 1) wait ].
This plays a major triad, which makes it very easy to hear the clipping distortion. On my Linux PC with pulse audio, I get audible clipping distortion when volume in the snippet above is set to anything higher than 0.30. Results are very similar regardless of the VM and the image version. I tried Squeak 3.8, Squeak trunk, interpreter VM, Spur64 VM, and SqueakJS on Chrome.
Maybe you can compare this on Raspbery Pi and see if you get the same result.
Dave
Hi,
after a lot of trying I managed to have
parec --file-format=wav --device=alsa_output.pci-0000_00_1b.0.analog-stereo.monitor test.wav
record the audio output of Squeak. Device is found via pactl list sinks and Squeak is started via
exec padsp "${VM}" "${IMAGE}"
From there on I used my slightly improved but still not working playLoop to record inside Squeak and the above to record outside Squeak.
Inside Squeak I saw a lot of clipping as expected, outside I saw wrapping whenever it clipped inside. I saw no buffer loss in the recording outside of Squeak up to the 6 voices I checked. My recording inside playLoop still looses most of the buffers.
I used:
SoundPlayer startReverb. FMSound organ1 setPitch: 261.6 dur: 10 loudness: 0.3 ; play. FMSound organ1 setPitch: 329.6 dur: 10 loudness: 0.3 ; play. FMSound organ1 setPitch: 392 dur: 10 loudness: 0.3 ; play. FMSound organ1 setPitch: 523.2 dur: 10 loudness: 0.3 ; play. FMSound organ1 setPitch: 659.3 dur: 10 loudness: 0.3 ; play. FMSound organ1 setPitch: 784 dur: 10 loudness: 0.3 ; play.
C major chord plus an octave above.
With loudness 0.2 nearly no clipping occurred up to 0.3 with constant clipping. As playing even one voice sets the CPU to 100% load on the Pi (as per top). I checked up to 6 voices to see if it was a matter of CPU load which it seems to be not on a Pi3.
Conclusion: It seems to happen somewhere between
primSoundPlaySamples: count from: Buffer startingAt: 1 (Before that I recorded inside Squeak to a file)
and the output to the os. primSoundPlay... is sent in SoundPlayer class playLoop.
quoting Tim:
Staring at the generated C code it looks possibly suspicious that we have
short int *aSoundBuffer; sqInt s; if (leftVol > 0) { i = (2 * sliceIndex) - 1; s = (aSoundBuffer[i]) + ((sample * leftVol) / ScaleFactor); if (s > 0x7FFF) { s = 0x7FFF; } if (s < -32767) { s = -32767; } aSoundBuffer[i] = s; } But surely no C compiler screws that up these days?
if that's from the primitive, why do we clip here? Clipping is done inside Squeak.
Cheers,
Herbert
Am 25.12.20 um 14:54 schrieb K K Subbu:
On 25/12/20 6:49 pm, Herbert König wrote:
It would be cool if someone can implement Your suggestion on how to write Squeak's audio output buffers to a file because then I could discern if the whole mess gets created by Squeak or if the OS plays a role too.
You may use parec (sound capture to file) and paplay (file to sound output) to test the audio subsystem of the OS to rule out any audio driver or mixer configuration problems [1]. The corresponding ALSA utilities are arecord and aplay. The mixer is alsamixer.
padsp acts as a relay between the older /dev/dsp (audio device under OSS) and the newer /dev/snd/* (audio i/o devices).
[1] https://manpages.debian.org/testing/pulseaudio-utils/pacat.1.en.html
HTH .. Subbu
On 2020-12-28, at 10:36 AM, Herbert herbertkoenig@gmx.net wrote:
Inside Squeak I saw a lot of clipping as expected, outside I saw wrapping whenever it clipped inside.
That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library. Just ofr a moment I got excited looking at the pulse driver code when I spotted it is using audioIn.pa_spec.format = PA_SAMPLE_S16LE; ... because I could have sworn that squeak is generating 16 bit BIG endian samples. But no...
Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim MIPS: Meaningless Indicator of Processor Speed.
Am 28.12.2020 um 22:48 schrieb tim Rowledge:
On 2020-12-28, at 10:36 AM, Herbert herbertkoenig@gmx.net wrote:
Inside Squeak I saw a lot of clipping as expected, outside I saw wrapping whenever it clipped inside.
That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library.
I'd suspect the Squeak primitive that hands the buffers to the OS before looking into OS code.
Would it be conclusive to record the outside for a minute or so, read that file and count wrapping (sample(n) = +fullscale and sample(n+1) = -fullscale) or (sample(n) = -fullscale and sample(n+1) = +fullscale) and do the same for all SoundBuffers we get hold of inside Squeak during the recording? (repeatedly checking SoundBuffer allInstances)
Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.
AbstractSound or the like has a class var Sine with that but I feel that's directed at finding a bug in the OS and discards the possibility of Squeak messing it up. And the organ sounds used are predictable enough re clipping but I hope the individual envelopes add some randomness to cover more cases.
Cheers,
Herbert
I changed playLoop to just copy the current SoundBuffer into a collection. I got some 11 seconds sound (2 below) Each buffer I differentiated and took the abs value. (see attached st) Of those I took the max. In case of wraparound that would be close to 64k. The zeroes in these buffers are the clippings. (3 below) Again I took the max. (4 below).
I can't email screenshots from the Pi but I did it on a Pi too. I used Tim's first snippet. I found a lot of clipping inside Squeak and not a single wraparound in several attempts.
Same observation id waggling the Keyboard or playing my 6 voiced C major while waggling the keyboard morph.
On the Pi3 I still lost buffers (getting 7 seconds worth of buffers from 10+ seconds of Audio). I would not search in the Squeak Smalltalk code for the problem but would suspect the primitive where Squeak hands the output to Raspbian.
That said clipping still makes for horrible sound (SCNR :-).
Cheers,
Herbert
Am 29.12.2020 um 12:38 schrieb Herbert König:
Am 28.12.2020 um 22:48 schrieb tim Rowledge:
On 2020-12-28, at 10:36 AM, Herbert herbertkoenig@gmx.net wrote:
Inside Squeak I saw a lot of clipping as expected, outside I saw wrapping whenever it clipped inside.
That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library.
I'd suspect the Squeak primitive that hands the buffers to the OS before looking into OS code.
Would it be conclusive to record the outside for a minute or so, read that file and count wrapping (sample(n) = +fullscale and sample(n+1) = -fullscale) or (sample(n) = -fullscale and sample(n+1) = +fullscale) and do the same for all SoundBuffers we get hold of inside Squeak during the recording? (repeatedly checking SoundBuffer allInstances)
Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.
AbstractSound or the like has a class var Sine with that but I feel that's directed at finding a bug in the OS and discards the possibility of Squeak messing it up. And the organ sounds used are predictable enough re clipping but I hope the individual envelopes add some randomness to cover more cases.
Cheers,
Herbert
Hi Herbert, Hi Tim,
I suspect I've made a mistake somewhere. This used to play smoothly and now I can hear some distortion and it's unlilely to do with my speakers, as the same is heard either through my laptop speakers or my apple display speakers:
| samples sineTable sound | "1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring" sineTable := SoundPlayer sineTable: 73. sineTable doWithIndex: "And let's not deafen anyone..." [:sample :index| sineTable at: index put: sample // 4]. samples := SoundBuffer new: 16000. 1 to: samples size by: sineTable size do: [:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1]. 1 to: 146 do: [:i| samples at: i put: ((samples at: i) * i / 146) asInteger. samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger]. sound := SampledSound samples: samples samplingRate: 16000. sound := MixedSound new add: sound pan: 0.25; add: sound pan: 0.75; yourself. sound play
So I'll try and do a regression test soon.
On Tue, Dec 29, 2020 at 10:53 AM Herbert König herbertkoenig@gmx.net wrote:
I changed playLoop to just copy the current SoundBuffer into a collection. I got some 11 seconds sound (2 below) Each buffer I differentiated and took the abs value. (see attached st) Of those I took the max. In case of wraparound that would be close to 64k. The zeroes in these buffers are the clippings. (3 below) Again I took the max. (4 below).
I can't email screenshots from the Pi but I did it on a Pi too. I used Tim's first snippet. I found a lot of clipping inside Squeak and not a single wraparound in several attempts.
Same observation id waggling the Keyboard or playing my 6 voiced C major while waggling the keyboard morph.
On the Pi3 I still lost buffers (getting 7 seconds worth of buffers from 10+ seconds of Audio). I would not search in the Squeak Smalltalk code for the problem but would suspect the primitive where Squeak hands the output to Raspbian.
That said clipping still makes for horrible sound (SCNR :-).
Cheers,
Herbert
Am 29.12.2020 um 12:38 schrieb Herbert König:
Am 28.12.2020 um 22:48 schrieb tim Rowledge:
On 2020-12-28, at 10:36 AM, Herbert herbertkoenig@gmx.net herbertkoenig@gmx.net wrote:
Inside Squeak I saw a lot of clipping as expected, outside I saw wrapping whenever it clipped inside.
That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library.
I'd suspect the Squeak primitive that hands the buffers to the OS before looking into OS code.
Would it be conclusive to record the outside for a minute or so, read that file and count wrapping (sample(n) = +fullscale and sample(n+1) = -fullscale) or (sample(n) = -fullscale and sample(n+1) = +fullscale) and do the same for all SoundBuffers we get hold of inside Squeak during the recording? (repeatedly checking SoundBuffer allInstances)
Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.
AbstractSound or the like has a class var Sine with that but I feel that's directed at finding a bug in the OS and discards the possibility of Squeak messing it up. And the organ sounds used are predictable enough re clipping but I hope the individual envelopes add some randomness to cover more cases.
Cheers,
Herbert
On 2020-12-29, at 11:03 AM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Herbert, Hi Tim,
I suspect I've made a mistake somewhere.
Well, I guess that happens occasionally - but I should note that my experiments are using a 5.3 release image (though I have tried the fairly up to date 6alpha stream as well). and so don't have your change involved.
Herbert - the prim to play the sound simply passes pointers around and I can't find any place where values might get changed. Given that both ALSA and pulseaudio allow setting of the sample format in order to handle 8, 16, 32 bt, integer and float, compressed, spindled, folded and mutilated, it is certainly possible there is an issue, though finding it will be 'fun'.
One thing I should do is dig out another Pi and set it up with a clean 'last before pulse' copy of the OS and make sure we know what happens in that case.
So many things to break, so little time...
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim The world is not magic
Hi Tim, Hi Eliot,
Eliot our problem is on the Pi only, not on Windows10 not on Linux Mint and I think Tim uses Mac Os w/o these problems. 5.3 release Squeak here too.
I created a fullscale sine and a 1.01 * fullscale sine in Audacity (cross platform audio editor) on the Pi, saved them as 16 Bit Wav and read them again -> all as expected. But Audacity was a bit unclear if it used Pulse or Alsa.
Next I'll create these sines in Python where I hope to have more control over the audio drivers.
Cheers,
Herbert
Am 29.12.2020 um 20:49 schrieb tim Rowledge:
On 2020-12-29, at 11:03 AM, Eliot Miranda eliot.miranda@gmail.com wrote:
Hi Herbert, Hi Tim,
I suspect I've made a mistake somewhere.
Well, I guess that happens occasionally - but I should note that my experiments are using a 5.3 release image (though I have tried the fairly up to date 6alpha stream as well). and so don't have your change involved.
Herbert - the prim to play the sound simply passes pointers around and I can't find any place where values might get changed. Given that both ALSA and pulseaudio allow setting of the sample format in order to handle 8, 16, 32 bt, integer and float, compressed, spindled, folded and mutilated, it is certainly possible there is an issue, though finding it will be 'fun'.
One thing I should do is dig out another Pi and set it up with a clean 'last before pulse' copy of the OS and make sure we know what happens in that case.
So many things to break, so little time...
tim
tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim The world is not magic
The Great Saga marches onward...
I've hacked the playLoop to also save the samples sent to SoundPlayer class>>#primSoundPlaySamples:from:startingAt: in a SamplesSound with a big buffer. Running sample sound snippets results in a saveable sample that I can then play in various ways on other machines.
Right now it appears to strongly suggest that the samples being created by Squeak are just fine; make horrible noises on the pulse-Pi, copy the generated .wav to an alsa-Pi and it sounds fine. Play the .wav on my iMac via iTunes and it sounds fine. try to play the .wav via VLC media player on the pulse-Pi and if anything it sounds worse. VLC player on the alsa Pi sounds very similar to the iTunes/iMac.
More testing to try but I'm leaning towards the pulse system on the Pi being borked.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim To err is human; to really foul things up requires a computer.
On 2021-01-01, at 9:38 PM, tim Rowledge tim@rowledge.org wrote:
The Great Saga marches onward...
Attached is a changeset that has the hacked playLoop and a rather long preamble that *should* explain it.
Basically, make a global BigBuffer, change playLoop to write samples to BigBuffer if it is not nil and not full. Inspect the SampledSound that owns a big buffer and use the included code snippets (that magically appear in the inspector, woo-hoo!) to point BigBuffer at a .. buffer... make some noises and the set BigBuffer to nil to stop recording.
Save the processed samples as a WAV file and try them on other machines and setups. You can load WAV files 'as sound' from the Squeak FileList, or play them with iTunes, VLC, whatever.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "How many Motie Mediators does it take to chage a lightbulb?” "Are you insane? Only Crazy Eddie would want to change *anything*!"
On 2021-01-02, at 2:14 PM, tim Rowledge tim@rowledge.org wrote:
On 2021-01-01, at 9:38 PM, tim Rowledge tim@rowledge.org wrote:
The Great Saga marches onward...
The recently update pulse libraries for Pi seem to have removed a lot of the especially ugly graunching, which is nice to hear. It's still quite nasty when using full volume range sounds and playing multiple sounds simultaneously though; maybe some use of MixedSound might help but it remains true that the alsa setup doesn't do this.
I've also realised that there is a definite buffer-size aliasing problem regardless of the alsa/pulse setup.
Given that the normal buffer size is set to 120mS worth of samples, consider this snippet
|snd volume dur| volume:= 100. dur := 120. 10 timesRepeat: [ snd := FMSound new. snd setPitch: (AbstractSound pitchForMIDIKey: 60) dur: 1000 loudness: ((volume / 100.0) within: 0.0 and: 1.0). snd play. dur milliSeconds wait. snd stopGracefully. snd := FMSound new. snd setPitch: (AbstractSound pitchForMIDIKey: 58) dur: 1000 loudness: ((volume / 100.0) within: 0.0 and: 1.0). snd play. dur milliSeconds wait. snd stopGracefully]
This makes a nice dee-dah-dee-dah... sequence, very cleanly.
Change 'dur' to 150 and try again...on either sound library the second version is incorrect and more dee-dah-de-da-dee-daaah-d-dah.. sort of choppy. 300 is less awful but still wrong. 240 is fine, as one might imagine.
Just whose idea was generating sounds with computers? I want a word...
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Flabbergasted (adj.), appalled over how much weight you have gained.
On 03/01/21 6:23 am, tim Rowledge wrote:
I've also realised that there is a definite buffer-size aliasing problem regardless of the alsa/pulse setup.
Given that the normal buffer size is set to 120mS worth of samples, consider this snippet
On Pi, the cpu gets throttled based on temperature and power rail voltage levels. This may cause variations in sound latency.
You can monitor actual latency on your sink/playback/output device with: ---- ~$ pactl list sinks | grep -i 'sink|latency' Sink #0 Latency: 75611 usec, configured 76000 usec Flags: HARDWARE HW_MUTE_CTRL HW_VOLUME_CTRL DECIBEL_VOLUME LATENCY ----
HTH .. Subbu
Hi, so yes, a Pi does indeed throttle when it gets hot - but hot is > 60C core temperature and believe me, none of my Pi ever get anywhere near that. Nor are they suffering low-voltage problems right now; I had a nasty issue with a failing PSU some time last year, or last century, whenever we think 2020 was. All fixed now with bigger margins on the supply.
It really does appear to be an issue with the basic design of the unix sound handling prims. Without digging too deep (because I'm knackered this weekend) it looks to me as if the unix sound prims are all single buffer stuff, whereas the Mac one appears double-buffered. For Windows I can't tell because none of the code makes any sense to me. For RISC OS there is a curious OS level buffering system that I vaguely understood for about 10 minutes 10 years ago.
Currently we by default create 120mSecs worth of buffer, fill it up and send it out to be played. Supposedly we can work out how many slots are free to refilled at any time but what does that even mean? It depends upon the details of the OS sound handling; if the OS takes a copy of the squeak buffer then in a sense said squeak buffer is now completely empty and ready to be filled completely. But what if the next attempt to pass a buffer finds that the OS cannot accept any more samples just yet? How complicated can we make this?
Even on a Mac you can trivially see the effect of some level of aliasing in the combination of the size of the buffer and the lengths of the 'notes' played by that snippet. Just try playing it with 'dur' set to 120 and then 150. Then edit the SoundPlayer class>>#initialize method to change the BufferMSecs value, then try again. It's really audible. But be careful - it appears to be possible to break the sound player completely by changing the BufferMSecs by much.
The problem seems to me that the interface is really set up for streaming not short chunk work. The Squeak side of it was designed 20 years ago on the basis of machines barely 1% as fast as the slowest machines we generally use these days (I'd guess a Raspberry Pi Zero would count for that) and has barely changed. We probably ought to do a lot better now.
On 2021-01-02, at 11:02 PM, K K Subbu kksubbu.ml@gmail.com wrote:
You can monitor actual latency on your sink/playback/output device with:
~$ pactl list sinks | grep -i 'sink|latency'
That's seriously cool, thanks.
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim Useful Latin Phrases:- Ne auderis delere orbem rigidum meum! = Don't you dare erase my hard disk!
On 04/01/21 7:01 am, tim Rowledge wrote:
It depends upon the details of the OS sound handling; if the OS takes a copy of the squeak buffer then in a sense said squeak buffer is now completely empty and ready to be filled completely. But what if the next attempt to pass a buffer finds that the OS cannot accept any more samples just yet? How complicated can we make this?
Latency could be introduced in any layer - driver, kernel, library, app.
* aplay plays local files well that rules out any latency issues in lib/kernel/driver layers. * PulseAudio introduces another buffering layer with its audio server app. pactl should reveal this. * Squeak also uses its own buffers. As an app, it could be subject to fair-share scheduling. As an interpreter, Squeak would be CPU intensive workload. If there are other CPU intensive loads, then its scheduling latency may impact audio play.
In addition to pactl, I would also have vmstat running in a separate terminal to reveal CPU/RAM/IO spikes. $ vmstat 5
HTH .. Subbu
On Mon, Jan 04, 2021 at 10:46:45AM +0530, K K Subbu wrote:
On 04/01/21 7:01 am, tim Rowledge wrote:
It depends upon the details of the OS sound handling; if the OS takes a copy of the squeak buffer then in a sense said squeak buffer is now completely empty and ready to be filled completely. But what if the next attempt to pass a buffer finds that the OS cannot accept any more samples just yet? How complicated can we make this?
Latency could be introduced in any layer - driver, kernel, library, app.
- aplay plays local files well that rules out any latency issues in
lib/kernel/driver layers.
- PulseAudio introduces another buffering layer with its audio server
app. pactl should reveal this.
- Squeak also uses its own buffers. As an app, it could be subject to
fair-share scheduling. As an interpreter, Squeak would be CPU intensive workload. If there are other CPU intensive loads, then its scheduling latency may impact audio play.
In addition to pactl, I would also have vmstat running in a separate terminal to reveal CPU/RAM/IO spikes. $ vmstat 5
HTH .. Subbu
<OT> Hi Subbu,
I have been using Linux for many years, but I am always learning new things from the tips and explanations that you provide here.
So thank you :-)
Dave </OT>
On 25/12/20 2:35 am, Herbert König wrote:
I tried the -vmsound alsa and pulse but I'm not sure If I used the right place in the script. I tried editing the last line to: exec "${VM}" "-vm-sound-alsa" "${IMAGE}" (and pulse and w/o the "" but my Linux fu is not enough to know if that's ok.
ALSA is a bit long in the tooth. Pulse is the preferred audio server these days.
You could try running Squeak with padsp which mixes the OSS API calls from squeak vm through Pulse audio service:
exec padsp "${VM}" ...
The GUI mixer panel is pavucontrol (from pavucontrol package).
HTH .. Subbu
Hi Subbu,
thanks, that did it. Squeak playing Audio with the problems described, Scratch one through the menu still w/o audio (as expected) but both showing up in the pavucontrol. Except that now I have a crackling noise when using the volume control (Crossfade still being a secret to some audio implementors, really :-)
So on to electrical recording and then try to implement Tim's suggestion on recording the buffers from Squeak.
Cheers,
Herbert
Am 25.12.2020 um 13:02 schrieb K K Subbu:
On 25/12/20 2:35 am, Herbert König wrote:
I tried the -vmsound alsa and pulse but I'm not sure If I used the right place in the script. I tried editing the last line to: exec "${VM}" "-vm-sound-alsa" "${IMAGE}" (and pulse and w/o the "" but my Linux fu is not enough to know if that's ok.
ALSA is a bit long in the tooth. Pulse is the preferred audio server these days.
You could try running Squeak with padsp which mixes the OSS API calls from squeak vm through Pulse audio service:
exec padsp "${VM}" ...
The GUI mixer panel is pavucontrol (from pavucontrol package).
HTH .. Subbu
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
There is no effect because amplitude change is supposed to be driven by an AmplitudeEnvelope (these are the ones that send #adjustVolumeTo:overMSecs:).
Unfortunately, there is already one such envelope active in FMSound organ1, which uses amplitude modulation, so a second one would be ignored anyway.
What we need is a way to combine several layers of amplitude control; we could start by supporting several AmplitudeEnvelopes in a single AbstractSound.
Stef
I'd think if nobody clips early we can just add all simultaneous sounds (each one after using their own envelope), do that all in Float32 and then add a limiter like I described. It just must be put at the end of the audio chain. Either a clever one with lookahed and no clipping at all or (for live recording and playback with little latency) a conventional one which changes the gain with a certain attack rate and has some clipping until the attack is over. Music industry lived with the latter for decades.
No idea if Squeak's sound architecture is suitable for that, it seems the conversion to Integer happens too early. But it was done with 16 Bit DSPs so some search or thinking should uncover that.
So first question: Where is the place, where the audio samples are handed to the OS? Second question: Are there more places than the diverse mixSampleCount:into:startingAt:leftVol:rightVol: which are responsible for clipping.
Cheers,
Herbert
Am 17.12.2020 um 20:00 schrieb Stéphane Rollandin:
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
There is no effect because amplitude change is supposed to be driven by an AmplitudeEnvelope (these are the ones that send #adjustVolumeTo:overMSecs:).
Unfortunately, there is already one such envelope active in FMSound organ1, which uses amplitude modulation, so a second one would be ignored anyway.
What we need is a way to combine several layers of amplitude control; we could start by supporting several AmplitudeEnvelopes in a single AbstractSound.
Stef
Thanks for chiming in (see what I did there?).
On 2020-12-17, at 11:00 AM, Stéphane Rollandin lecteur@zogotounga.net wrote:
"no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play.
1 second wait. snd stopGracefully.
There is no effect because amplitude change is supposed to be driven by an AmplitudeEnvelope (these are the ones that send #adjustVolumeTo:overMSecs:).
That's extra-puzzling because it is the mechanism used in the original Scratch code to control volume and I'd swear it used to work. The other mechanism #loudness: is rather more brutal and makes glitchy noises if used whilst playing.
Unfortunately, there is already one such envelope active in FMSound organ1, which uses amplitude modulation, so a second one would be ignored anyway.
What we need is a way to combine several layers of amplitude control; we could start by supporting several AmplitudeEnvelopes in a single AbstractSound.
I'm not very familiar with the workings of the sound system and happy to keep it that way if possible. There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work?
What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them? Wouldn't that mean that starting a new sound would make the others go quieter in an odd sounding manner? Do you just clip the final total for every sample?
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim "How many Slavers does it take to change a lightbulb?” "Dunno. How susceptible are lightbulbs to telepathy?"
There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work?
Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop.
What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them?
Yes, and whether or not their volumes are varying, the sounds must have been scaled beforehand to ensure there is enough headroom for their superposition. This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence).
Stef
Well...
On 2020-12-17, at 5:12 PM, Stéphane Rollandin lecteur@zogotounga.net wrote:
There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work?
Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop.
I *think* that the FMSound>>#primitiveMixFMSound code handles this; at least it looks like it pays attention to the waveTable etc. I've just taken a quick look and the code that changes the scaleVol if the scaledVolIncr value is non-0 is there in the prim too, so there hasn't been a major regression in the code. Probably.
What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them?
Yes, and whether or not their volumes are varying, the sounds must have been scaled beforehand to ensure there is enough headroom for their superposition.
Yikes. That smacks of needing a lot of advance information. How on earth do we handle sometihng like a true ten-finger keyboard? Make each sound only allow volume 0.1 ?
This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence).
Oh yes. Wave interference maths is great fun. I was just reading a couple of nights ago about using the positive Grassmanian and Kodama-Williams equations to model soliton waves. I hope we don't need to worry about that to solve this tiny problem!
tim -- tim Rowledge; tim@rowledge.org; http://www.rowledge.org/tim If at first you don't succeed, destroy all evidence that you tried.
Am 18.12.2020 um 02:52 schrieb tim Rowledge:
Yikes. That smacks of needing a lot of advance information. How on earth do we handle sometihng like a true ten-finger keyboard? Make each sound only allow volume 0.1 ?
This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence).
That is just very unlikely, all tones you press on your keyboard must start with that exact delay that they reach their maxima at the same time. So the higher tone must start later than the lower one because it reaches its max quicker when both start from zero. But yes, if you look at the waveform of some classical music you see a lot of headroom for the few peaks. Just not a factor 10. Different story for current popular music. Or music to be listened to while driving.
Cheers,
Herbert
By the way, it is possible in muO to have that kind of waveform display, along with the spectrogram.
Try for example
(AbstractSound bachFugueVoice1On: PluckedSound default) viewSamples
This will yield a zoommable-field morph representing the waveform, where click-dragging smoothly scrolls the display, a shift+click-drag smoothly scales it (images attached). The right-click menu gives access to the FFT settings.
Best,
Stef
Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop.
I *think* that the FMSound>>#primitiveMixFMSound code handles this; at least it looks like it pays attention to the waveTable etc. I've just taken a quick look and the code that changes the scaleVol if the scaledVolIncr value is non-0 is there in the prim too, so there hasn't been a major regression in the code. Probably.
That's what I am talking about. Because there is only one scaledVol instance, its evolution in time can be (and actually *is*) described by a *single* envelope.
Now see the code in FMSound>>#organ1 There, that envelope is defined in
snd addEnvelope: (VolumeEnvelope points: p loopStart: 2 loopEnd: 4).
So that is what drives the time evolution of scaledVol. It is a short envelope constantly looping over break points 2 to 4.
If you now want to control the overall loudness of the sound, let's say for a crescendo, you have to surimpose to this scaledVol evolution a gradual increase of its max value - this cannot anymore be defined by a looping envelope.
Stef
squeak-dev@lists.squeakfoundation.org