Measurement standardization (split from MQA discussions), and headphone measurement complexity...

Continuing from the discussions on measurement standardization and headphones in particular; @botrytis...

Quote

 

As a person, who has been involved in Analytical Chemistry for years, standardization of methods used for measurement is the key. Even if the initial setup if off, at least all the measurements are done the same way. 

 

I can give an example, from the industry I am consulting in, currently. In Canada and the US, where recreational Cannabis is legal, many producers are shopping for the lab that gives them the highest number for THC, not the most accurate number. This is due to the fact, the regulatory agencies have not really put out any recommendations as to the accepted methods and no blind testing from standard samples. Until standardization of testing methods is done, which measurements does one believe?

 

I would argue potentially not any one of them in the case of headphone testing and correlating to subjective preferences. More important I think is seeing if one can find similarities between multiple methods or relative differences using the same method. Even if we all used a standardized system like Head-Fi's B&K 5128, would that ensure each of us can look at the graph and immediately agree that those lines prove we'll enjoy the headphone from which the measurement was obtained? I don't think it'll ever be that simple... Unless we already have a reference headphone, know the B&K 5128 measurement for that (recognizing that even the same brand and model headphone could have significant variations - look at that Westone W60 samples in the link), and then compare with the one from the new headphone, then we *might* be able to consider that maybe the enhanced mid-range could be more enjoyable, or the extra +3dB bass would be more "fun", etc...

Some things are more straight forward and demand accuracy because there is a linear correlation with effect. For example, knowing exactly how many "honest" milligrams of THC (not just a marketing number like "1000W PMPO!") there is in a gummy might mean that if it's the same price, maybe one could split the gummy and be able to use a box longer, it becomes a meaningful value comparison. But this is more difficult with non-linear correlations (like to subjective preferences). We of course still care that the measurement method is accurate and reliable, but beyond a threshold, there's no longer a subjective importance.

I might be unhappy if my expensive DAC scores a measly 1kHz THD+N of -65dB at best (which likely suggests bad noise floor, but even then unless we look at the detailed FFT, cannot be sure) using a reasonably precise measurement system say +/- 1dB. But once we can measure -100dB, does it really matter to my ears if it's -110dB? I don't think so because the threshold for "clean", "high enough fidelity" sound has likely been achieved (again, I would still like to see the FFT, not just a "one number objectivism"!). For things like SINAD, this is easily done with great accuracy for little cost these days. I doubt headphone testing will ever be as simple, and reductionistic as this; nor should it be!

2 minutes ago, Archimago said:

 

Continuing from the discussions on measurement standardization and headphones in particular; @botrytis...

 

 

I would argue potentially not any one of them in the case of headphone testing and correlating to subjective preferences. More important I think is seeing if one can find similarities between multiple methods or relative differences using the same method. Even if we all used a standardized system like Head-Fi's B&K 5128, would that ensure each of us can look at the graph and immediately agree that those lines prove we'll enjoy the headphone from which the measurement was obtained? I don't think it'll ever be that simple... Unless we already have a reference headphone, know the B&K 5128 measurement for that (recognizing that even the same brand and model headphone could have significant variations - look at that Westone W60 samples in the link), and then compare with the one from the new headphone, then we *might* be able to consider that maybe the enhanced mid-range could be more enjoyable, or the extra +3dB bass would be more "fun", etc...

 

Some things are more straight forward and demand accuracy because there is a linear correlation with effect. For example, knowing exactly how many "honest" milligrams of THC (not just a marketing number like "1000W PMPO!") there is in a gummy might mean that if it's the same price, maybe one could split the gummy and be able to use a box longer, it becomes a meaningful value comparison. But this is more difficult with non-linear correlations (like to subjective preferences). We of course still care that the measurement method is accurate and reliable, but beyond a threshold, there's no longer a subjective importance.

 

I might be unhappy if my expensive DAC scores a measly 1kHz THD+N of -65dB at best (which likely suggests bad noise floor, but even then unless we look at the detailed FFT, cannot be sure) using a reasonably precise measurement system say +/- 1dB. But once we can measure -100dB, does it really matter to my ears if it's -110dB? I don't think so because the threshold for "clean", "high enough fidelity" sound has likely been achieved (again, I would still like to see the FFT, not just a single number!). For things like SINAD, this is easily done with great accuracy for little cost these days. I doubt headphone testing will ever be as simple, and reductionistic as this; nor should it be!

I am not saying they are or are not. All I am saying is unless there is some standardization to the measurement methods or to report all the conditions the measurements were made at, This is done is science papers all the time, so that people can repeat the measurements/experiments.

Cheers 

8 minutes ago, botrytis said:

 

I am not saying they are or are not. All I am saying is unless there is some standardization to the measurement methods or to report all the conditions the measurements were made at, This is done is science papers all the time, so that people can repeat the measurements/experiments.

 

Cheers 

Yes, agreed, so long as there is transparency on the methods and results for verification and critical evaluation... No problem.

1 minute ago, Archimago said:

 

Yes, agreed, so long as there is transparency on the methods and results for verification and critical evaluation... No problem.

That is why I have issues with some measurements over others. 

Totally agree. Dang two audiophiles agreeing.

25 minutes ago, botrytis said:

 

That is why I have issues with some measurements over others. 

 

Totally agree. Dang two audiophiles agreeing.

 

One has to have a self-deprecating sense of humor also 

On 4/13/2023 at 12:06 PM, botrytis said:

 

One has to have a self-deprecating sense of humor also 

Which one is more fun, "self-deprecating" or "other-denigrating"? 😎

Prefer self-depricating humor.

1 hour ago, botrytis said:

Prefer self-depricating humor.

How boring! ;)

Back to repeatable measurements — something I’ve been taking seriously lately, at least in electronic measurements with Multitone. The latest feature is the creation of a complete test plan containing multiple, fully repeatable and preconfigured sets of measurements that others can execute on their own.  This produces results that can be directly compared to each other, both graphically and in table format. Still work on progress but so far a number of testers have been able to repeat a battery of tests I created and results are encouraging.

From the other thread… @Archimago I’m not sure if you saw, but a Headphones.com writer had several non-audiophile friends try a variety of IEMs with a wide range in price, and all three picked the Cadenza as their favorite. 

On 4/11/2023 at 11:50 AM, Archimago said:

Agree...

 

Which I think is a nice reflection of the fact that there is no one standard head/pinna/auditory canal/perceptual mechanism. Lots to potentially explore but I suspect that ultimately it will remain a "soup sandwich" reflective of Floyd Toole's "circle of confusion" as applied to the idiosyncrasies and complexities of human physique and perceptual preferences even without the room in the way. One can spend insane amounts of money on all kinds of standardized headphone test equipment; not sure to what end unless one is in the research/academic world developing devices...

 

Even if one spent $$$$$ on test equipment, I don't think results will ever translate to universal approval nor (more importantly) the sound recommendations directly applicable to everyone aiming to enjoy music. That's why IMO, work by @JoshM recently using his EARS and SBAF curve is good enough to give us a sense of relative sound signatures (and why I also have my EARS for testing and not particularly keen to spend money to upgrade). The rest is up to us to try listening for ourselves ;-).

1 hour ago, AcousticTheory said:

All the measurements in the world don't correlate to aural perception or whether you will like the sound. Linearity doesn't care about sounding "good". This is why I quickly tire of the "audio omniskeptic" persona - they're only trying to convince me that what I hear doesn't exist, and isn't meaningful, while bench-racing components by what they can quantify. I mean, that's cool and all, but it's not the entire story unless somebody puts ears on something and calls it good. That's what will motivate my interest in the product.

 

If there's consensus between good measurements and listener reports of good sound, then that seems to be the best of both worlds, but they are separate worlds.

Linearity is about transparency and accuracy of the system. It is just one of the characteristics that can be measured and correlated to audibility. It is certainly not meant as a predictor of what someone will like.

Preferences can be studied (and have been studied) using properly controlled scientific experiments to eliminate confounding variables to achieve repeatable and statistically valid results. But yes, to properly study preferences between two different devices, one must first demonstrate that the difference is actually audible and can be consistently detected. That's a necessary step before declaring any sort of preference as valid. How you measure the detection (ABX or other blind test, galvanic skin response or an MRI machine) is really irrelevant, as long as the experiment is done properly and the result is reproducible and statistically significant.

On 7/29/2024 at 6:17 AM, AcousticTheory said:

All the measurements in the world don't correlate to aural perception

Well, some do

On 7/29/2024 at 6:17 AM, AcousticTheory said:

or whether you will like the sound.

Whether you like the sound or not is a preference.

 If you choose by preference A vs B 50% of the time, similar to a coin toss, it doesn't mean there is no discernible difference. What if A and B were flavors of ice cream. There can be strong difference but no particular preference either way.

On 7/29/2024 at 6:17 AM, AcousticTheory said:

but it's not the entire story unless somebody puts ears on something and calls it good. That's what will motivate my interest in the product.

Absolutely!...and those ears would be mine!😅

On 7/28/2024 at 10:17 PM, AcousticTheory said:

If there's consensus between good measurements and listener reports of good sound, then that seems to be the best of both worlds, but they are separate worlds.

Although I fully agree with your statement about omniskeptics, I do not think it makes sense to see the approach of either measuring or judging by ear as separate worlds. For the simple reason many people perceive it as a statement claiming that listening and judging sound quality is beyond physics, something mystical or something you cannot match at all.

It is not, of course. In theory a sound field could be comprehensively described by measurements predicting both if listeners could distinguish differences and how they will perceive them. In practice this will fail due to the enormous reduction in complexity when doing measurements, the enormous complexity of both acoustical and psychoacoustical phenomena and the overly simplified models measurements are presented and interpreted.

In many discussions over the years I have come to the personal conclusion that total subjectivists rejecting measurements per se and self-declared objectivists claiming that a bunch of graphs and specs says everything are equally wrong in their judgement and the inherent flaws of how they evolve their theories are pretty much the same. 

On 7/29/2024 at 12:13 AM, pkane2001 said:

to properly study preferences between two different devices, one must first demonstrate that the difference is actually audible and can be consistently detected. That's a necessary step before declaring any sort of preference as valid.

If it would be a scientific process I would agree. But it is not. For many people it is just fun. 

If you declare all differences not passing the threshold of an ABX group test with significance to be negligible and non-existent, you deny people their personal experience and make it more likely they will reject the idea of a more rational take on sound quality in general. And yes I am aware this might lead to a lot of errors in judgement and people might spend more money than necessary compared to a ´no-frills approach´, but if they enjoy it, what is the problem?

4 hours ago, Arindal said:

If it would be a scientific process I would agree. But it is not. For many people it is just fun. 

Well, of course! Who am I to deny some fun if nobody gets hurt? In fact, here are some more things to have fun with that aren't real [spoiler alert!]:

• Magic
• Santa Claus
• Tripping on recreational drugs
• Online dating profiles
• The Flat Earth Society
• ...

😎

8 minutes ago, pkane2001 said:

 

Well, of course! Who am I to deny some fun if nobody gets hurt? In fact, here are some more things to have fun with that aren't real [spoiler alert!]:

• Magic
• Santa Claus
• Tripping on recreational drugs
• Online dating profiles
• The Flat Earth Society
• ...

😎

Measurements of audibility

Valhalla 

Self validating gadgets

Casper the friendly ghost

My sporting achievements 

Fishing stories

Big foot

..

🤔

9 minutes ago, Audiophile Neuroscience said:

 

Measurements of audibility

Valhalla 

Self validating gadgets

Casper the friendly ghost

My sporting achievements 

Fishing stories

Big foot

..

🤔

 

At least two of these are real (my fishing stories is one of them!)

6 hours ago, Arindal said:

If you declare all differences not passing the threshold of an ABX group test with significance to be negligible and non-existent,

I'd rather not declare anything for others. I'm much more for each person learning their own thresholds and ability to hear, with their own equipment and music. I've shared some tools for this over the years, although these were built primarily for my own testing and learning.

5 hours ago, Arindal said:

And I also noticed many people who are actually pretty good in hearing aspects of sound reproduction and expressing what they like and what not, but seemingly lost when it comes to finding the root of that within all the acoustical parameters and gear. That is in many cases resulting in endless gear flipping, tuning and tweaking, instead of addressing the flaws of their system which they themselves have detected with appropriate measures including room treatment, choosing the right speakers for their environment and applying digital EQ.

I agree that flipping and upgraditis may be a symptom of some as you describe as not being able, for whatever reason, to track down why a particular system annoys them.

Then there are some (okay one !) that feel that with their special magical tweaking powers that they can make just about any system shine with just about any recording

I think for some tweaking can become a hobby in itself yielding differences that may not be better or worse (in their assessment). The exploration part is the fun

10 hours ago, pkane2001 said:

I'd rather not declare anything for others. I'm much more for each person learning their own thresholds and ability to hear, with their own equipment and music.

Who are you and what have you done with Paul?! Have you issued a ransom note - I am willing to chip in for his safe return ! 😁

1 hour ago, Audiophile Neuroscience said:

 

Who are you and what have you done with Paul?! Have you issued a ransom note - I am willing to chip in for his safe return ! 😁

6 minutes ago, pkane2001 said:

Not too much as a measure or worth of a man but it will put a dent in my new cables fund !😅

13 hours ago, Audiophile Neuroscience said:

Not too much as a measure or worth of a man but it will put a dent in my new cables fund !😅

$1M may make a small dent in the new AC cord fund, agreed. But an uninterrupted and an unending debate about measurements vs. subjective evaluation— that’s priceless!

On 4/14/2023 at 1:22 AM, Archimago said:

Until standardization of testing methods is done, which measurements does one believe?

Totally agree.

Until there is standardization of testing methods and measurement tools used, people doing the testing and measuring are not even necessarily talking the same 'language' amongst themselves and making interpretation of the results impossible.

From a validity point of view, you don’t know who to believe i.e which one is accurate.

From a reliability point of view even if there is some consistency in results (if the results are repeatable within the same test situation) i.e. there is good precision ,if one cannot reproduce the results beyond that test situation, you cannot generalize the results outside of the test situation.

On 4/13/2023 at 11:22 AM, Archimago said:

 

Continuing from the discussions on measurement standardization and headphones in particular; @botrytis...

 

...

 

Some things are more straight forward and demand accuracy because there is a linear correlation with effect. For example, knowing exactly how many "honest" milligrams of THC (not just a marketing number like "1000W PMPO!") there is in a gummy might mean that if it's the same price, maybe one could split the gummy and be able to use a box longer, it becomes a meaningful value comparison. But this is more difficult with non-linear correlations (like to subjective preferences). We of course still care that the measurement method is accurate and reliable, but beyond a threshold, there's no longer a subjective importance.

This is the crux of the endless debate. *If* there were a linear correlation then measurements would predict enjoyability. Unfortunately humans are anything but linear.

On 4/13/2023 at 11:22 AM, Archimago said:

 

I might be unhappy if my expensive DAC scores a measly 1kHz THD+N of -65dB at best (which likely suggests bad noise floor, but even then unless we look at the detailed FFT, cannot be sure) using a reasonably precise measurement system say +/- 1dB. But once we can measure -100dB, does it really matter to my ears if it's -110dB? I don't think so because the threshold for "clean", "high enough fidelity" sound has likely been achieved (again, I would still like to see the FFT, not just a "one number objectivism"!). For things like SINAD, this is easily done with great accuracy for little cost these days. I doubt headphone testing will ever be as simple, and reductionistic as this; nor should it be!

Speaking about THC and measurements, oh were we able to measure the ability to induce the hallucination of really being there, live ...