Psychoacoustics, bias, and the problems with listening tests

[Phobic]

Thought it might be worth a thread talking about Psychoacoustics, bais & how hard it actually is to perform proper listening tests.

I suspect that the TLDR of this thread will be that it's all very complicated, your ears are easily fooled, and it's very hard to setup truly unbiased testing. Anecdotal corroboration of tests/tweaks/changes is just that, it's anecdotal. preference plays a much bigger part in all this than people would like to admit. Don't worry so much about it, it's not the end of the world, just be aware of it, ultimately it's only you who can fool yourself.

some pre-reading

https://en.wikipedia.org/wiki/Psychoacoustics

https://blog.landr.com/psychoacoustics/

https://www.******************.com/forum/index.php?threads/psychoacoustics-fundamentals.16018/

THE RATIONAL AUDIOPHILE on listening tests more articles in the blog which are worth a read

no one can demonstrate that the knowledge you are taking part in an experiment doesn’t impede your ability to hear differences

the act of listening for sound quality in scientific trials may kill our ability to discern sound quality. Can this be proved either way? No.

 Our hearing begins to change the moment we start listening. It becomes desensitised to repeated exposure to a sound – one of the cornerstones of many types of listening-based testing.

I would be only too keen to hear people’s subjective experiences if they are referring to something measurably out of the ordinary and arguably good; less so if they are describing differences between digital cables that, rationally, cannot be affecting the sound. 

ABX test is based on a completely flawed premise: it is not comparing audio systems using human hearing, but human memory.

Blind audio testing

Blind testing isn’t so much about the differences in the equipment as much as it is a test about the listener.  Moreover, just because a listener is able to perceive a difference between Product A and Product B, you can only talk about the listener’s preference for a particular product.  If you want to take the variable of the listener out of the equation, then your listener panel needs training.

without trained listeners the only thing you’ve proven in the blind test is something about the listeners, not the equipment being tested

How to Listen: A Course on How to Critically Evaluate the Quality of Recorded and Reproduced Sound

Harman's how to listen test software - teach yourself how to listen better

 

[Johannes]

Is it the case that when you imagine something, your feelings about it are the same as when it is real?

 

[Phobic]

Is it the case that when you imagine something, your feelings about it are the same as when it is real?

Just because you feel differently it doesn't mean it's imagined or real...

 

[petecallaghan]

On the positive, real progress can be made and real improvements can be heard. Selection between relatively similar but otherwise inferior set ups are still possible via listening tests. 

What techniques do people use that have proved successful? 

 

[Johannes]

No blind test. See here: https://www.linn.co.uk/tunedem
I think Linn can be called successful.

 

[Paulssurround]

No blind test. See here: https://www.linn.co.uk/tunedem
I think Linn can be called successful.

Thank you Phobic for starting this thread and the information you have linked, however.....

I agree with Linn regarding Tune Dem. In the Linn world and on the Lejonklou forum, Tune Dem is considered the gold standard at the Linn factory and encouraged by Linn for their owners for comparing improvements, setting up Space optimisation, setting speaker positioning and so on.

Linn and Lejonklou, are talking about active listening and following the tune using Tune Dem. They don't discount listening sessions with Tune Dem to expectation bias,  nor do they advocate using ABX testing, or measurements with a microphone, to compare sound quality between different components or changes to the system. 

Can the same people who talk expectation bias also be advocates of Tune Dem? 

Are people who are comparing the torque values of chassis screws doing measurements, or are they actually listening using Tune Dem, using their ears to evaluate. They are certainly not using double blind studies to evaluate a 0.01 difference in torque values? I don't recall any of them using a microphone to measure the differences in torque values?

Can the same people who do changes in torque values, or change a setting in SO have all of their listening discounted because it can all be written off to expectation bias, and they have imagined the changes? 

I give our ears and ability to listen more credit than some would have you believe.  

 

[StingRay]

"The approach we suggest when doing an A/B comparison is to listen to component A, then listen to B. If one sounds better, buy it. We have always said, “If it sounds better, then it is better.”"

Some systems may sound better initially, but can be tiring and irritating after a while. 

So it does not sound like a good test to me. 

 

[akamatsu]

The problem with buying a component after reading positive reviews, and getting recommendations from friends is that this creates a bias, and that bias cannot be escaped as hearing is trumped by visual cues. This has been proven. One can very easily prove this to themselves, but only if they believe in science. I have no tolerance for science deniers. If you deny science, you are a science denier. I don't mean favouring one scientific theory over another. I mean denying science in favour of unproven lore and the like.

 

[Newton John]

Can the same people who talk expectation bias also be advocates of Tune Dem? 

Expectation bias is a real thing to be guarded against in rigorous research. However, I question whether it has much relevance to comparisons made using Tune Dem. In hifi forums, it's just an empty phrase trotted out without any care over its true meaning as a lazy way of rubbishing someone else's findings.

Some systems may sound better initially, but can be tiring and irritating after a while. 

So it does not sound like a good test to me. 

The purpose of the Tune Dem is to avoid situations like that.

You are putting forward a circular argument - you don't think it works so therefore it is not a good test.

 

[Johannes]

In hifi forums, it's just an empty phrase trotted out without any care over its true meaning as a lazy way of rubbishing someone else's findings.

Well said.

I also see it as a danger for the exchange of experiences. Everywhere there are only deadly arguments against which one can no longer argue objectively and therefore leaves it alone. Everyone's experience is their own kingdom and often other arguments are simply no longer allowed. Even the demand for proof goes in the direction of if you can't prove it then it's not true and the madness is that they don't even try "it can't be true". I mean even Einstein's theories have only recently been proven, where would we be if science did what I just described. Even Linn has brought some things onto the market that were not scientifically proven in the beginning. At least not for us users but we did it and were happy and excited about the improvement.

 

[akamatsu]

Expectation bias is a real thing to be guarded against in rigorous research. However, I question whether it has much relevance to comparisons made using Tune Dem.

I did a little tunedem exercise yesterday. I have a Majik DSM system in my house temporarily. I set it up and listened. I was immediately drawn into the music, I was dancing about, sorry, no video.

Then I sat down and did some more critical listening. If I followed the tune, I could really enjoy the music and have it move me. but when I started comparing to my main system by listening for the hifi attributes, thoughts like "The vocal isn't as smooth" crept in. Immediately, the enjoyment went away. Then I just allowed myself to be with the music, not the hifi system, but the music, the joy returned. When I listened to the music, the lack of smoothness disappeared (was no longer noticed).

Doing this really allowed me to distinguish my own expectation bias, and remove it, somewhat, from the equation. Of course, it can't be removed completely but only distinguished.

I think I finally really get tunedem. Here is how I experienced it.

Listening to the music and not the hifi provided the joy that the music brings. Which brings more joy? That's the one to buy.

 

[Paulssurround]

Expectation bias is a real thing to be guarded against in rigorous research. However, I question whether it has much relevance to comparisons made using Tune Dem. In hifi forums, it's just an empty phrase trotted out without any care over its true meaning as a lazy way of rubbishing someone else's findings.

The purpose of the Tune Dem is to avoid situations like that.

You are putting forward a circular argument - you don't think it works so therefore it is not a good test.

I agree that expectation bias is a real thing, and clearly influences much of what we do.

‘I have a concern when someone rolls out the term “expectation bias” when it is something they don’t agree with someone, because the listener used their ears and find it is subjective. Then in another post, they rally behind the benefits of Tune Dem, which is also a subjective way to evaluate music.  

 

[StingRay]

You are putting forward a circular argument - you don't think it works so therefore it is not a good test.

Not at all, it is from experience of people buying a hifi product after a  demo but then finding they can’t live with it after longer listening periods. You also need to try a wide range of music, as some music may sound better but other genres maybe worse. 

The Tune Dem sounds too simple. 

 

[Metatron]

On the positive, real progress can be made and real improvements can be heard. Selection between relatively similar but otherwise inferior set ups are still possible via listening tests. 

What techniques do people use that have proved successful? 

Firstly, I listen subjectively like anybody else, hoping to find kit that sounds promising, has the features I am looking for, perceived VFM etc.

Then it's important I get to hear it at home in the room my system is in, given room acoustics have the largest effect (even if running some form of correction, whether DSP, DEQ, SO or other). I need to like it in that scenario.

Then after I am starting to think it has merit, I have to listen to it blind if the cost is significant.

I get somebody to randomly switch it in and out the system and compile a list of whether it was in the system or not, and I listen blind and report which I prefer. This gets noted.

I will happily do this for a few hours and do it with as wide a variety of music as possible.

At the end, I wish to discover if I could select the supposedly better sounding component more than about 75% of the time. If it's marginal (60-74%) I will run more tests and see if that edges in one direction or the other. If not, I will contemplate if being in this 'grey' area where the outcome is a bit better than chance is enough for me to consider the item as a worthwhile purchase, since clearly I wouldn't always be able to discern it as 'better'. However, if the things I believe made it 'better' when listening subjectively are still there in the blind tests and I definitely seem to be able to tell when the item is in the hifi chain or not, then it would seem that a difference is real and repeatedly identifiable. In such as case, one can conclude it does indeed sound better and the improvement is identifiable on hearing alone (no multi-modal perception issues). That basically would result in a purchase, so long as there aren't other comparative equipment to also compare against.

If there is comparative equipment to compare against, then ultimately, the one that I can most easily discern makes improvements under controlled conditions becomes the one I will buy - all other things being equal (such as features, ballpark cost etc).

But I've always experienced differences with anything - cables, PSUs, all DACs and so forth. I think the point is I am aware science says my own brain can fool me and concoct a difference for me to experience, such that I believe I heard differences. So I have to ensure that those differences are real. Since any cognitive biases cannot be turned off, by anybody, when only listening subjectively, it mean a difference is always heard, and therefore, paying for whatever reportedly 'better' thing is on demo, would always result in the need to buy it - when the reality is, it might be making about a gnats fart of difference.

I also stay away from any marketing claim that makes unsubstantiable claims that conflict with established proven science. For instance, quantum resonance filters.... it's just BS. But sure, whack it in the system and listen subjectively only... and you might just end up buying them. Then do a controlled test and find you cannot tell when they are in the system, so basically don't have any merit and have spent your hard earned cash for snake oil.

There's only one way to determine snake oil from anecdotal claims. Scrutinize them with appropriate tests that remove cognitive biases. And I don't care if I say that term and people read "expectation bias" because that is just one of the cognitive biases at work, and disregarding the fact that our brains are fundamentally concocting our experience of the world most of the time, and our perceived rational thought is an after process, just helps prove the old saying that:

an audiophile and his money are soon parted.

 

[petecallaghan]

Listening to the music and not the hifi provided the joy that the music brings. Which brings more joy? That's the one to buy.

The best system is the one that brings most joy. And the system is the music, the hifi, the room and the listeners.

 

[akamatsu]

The best system is the one that brings most joy. And the system is the music, the hifi, the room and the listeners.

And it's possible to have the music, the room, and the listeners be the same, thus isolating the effect of the hifi. Then hifis can be compared.

Edit; I just realized that the listener(s) can vary in that mood, and other psychological factors can vary moment to moment. So the listening should be more valid if your mood /state of mind is consistent.

 

[Metatron]

https://www.audioholics.com/room-acoustics/mind-over-music

(^^ article there ^^ / vid from article below)

 

[Metatron]

And the major paper referenced in the above is here:

http://www.keithstanovich.com/Site/Research_on_Reasoning_files/West_Stanovich_JPSP2012.pdf

 

[Blackmetalboon]

The best system is the one that brings most joy. And the system is the music, the hifi, the room and the listeners.

Unfortunately, three of the four are often ignored.