Why you can't trust measurements

It's curious how some people don't trust their ears, believe they are of such poor quality as a means of registering what's going on that they constantly run to a piece of equipment, to justify some stance on the audio equipment game.

Personally, what I have found intensely satisfying over the decades, is seeing how a particular setup can steadily be pushed, evolved, to slowly eliminate its unrefined personality to the point where that of the recording starts to take over. The music event that was captured, which is now effectively eternal, just bides its time, patiently, until its "true colours" can properly emerge - each configuration is better able to do such in some areas than others; the 'truth' of the recording is the combination of the very best that one has ever heard from a particular track, over the years of experiencing it on many setups and situations. This builds into a memory of the musical performance - and when one gets very close to the essence of what's in the source, the jolt of recognising all the little giveaways that you picked up over the years is similar to meeting someone you knew extremely well some years ago, but hadn't come across for a long time in between. The familiarity with what's on the recording is one of the pleasures of getting the SQ right - and this where conventional measurements are nowhere near good enough to do the job of determining the, 'accuracy' ...

4 minutes ago, fas42 said:

It's curious how some people don't trust their ears, believe they are of such poor quality as a means of registering what's going on that they constantly run to a piece of equipment, to justify some stance on the audio equipment game.

 

Personally, what I have found intensely satisfying over the decades, is seeing how a particular setup can steadily be pushed, evolved, to slowly eliminate its unrefined personality to the point where that of the recording starts to take over. The music event that was captured, which is now effectively eternal, just bides its time, patiently, until its "true colours" can properly emerge - each configuration is better able to do such in some areas than others; the 'truth' of the recording is the combination of the very best that one has ever heard from a particular track, over the years of experiencing it on many setups and situations. This builds into a memory of the musical performance - and when one gets very close to the essence of what's in the source, the jolt of recognising all the little giveaways that you picked up over the years is similar to meeting someone you knew extremely well some years ago, but hadn't come across for a long time in between. The familiarity with what's on the recording is one of the pleasures of getting the SQ right - and this where conventional measurements are nowhere near good enough to do the job of determining the, 'accuracy' ...

 

 

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

20 minutes ago, botrytis said:

 

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

Yet a conductor in the storm of an orchestral crescendo drowning him in sound, from a flotilla of instruments feet away, all giving their all, can then point to a particular player in a section - and say, "You did that wrong!" ...

I have no idea what 'crazy uncle Frank' is talking about.

It is not possible to rely on my ears fully to test audio equipment or audio app because I cannot hear some noise.

Younger people's ears have better frequency range (they hear mouse repellent, elevator power converter noise, fluorescent lamp noise, car brake pad squeal: it squeals every time when brakes and its frequency lowered when pad worn, SMPS acoustic noise, NTSC flyback transformer noise of old recordings) and better dynamic range (they hear mosquito sound in 3 meter distance, -10dB SPL and complains constant hiss or intermittent hiss (it seems the noise stops when amplifier power saving is activated) from loudspeakers)

24 minutes ago, fas42 said:

52 minutes ago, botrytis said:

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

 

Yet a conductor in the storm of an orchestral crescendo drowning him in sound, from a flotilla of instruments feet away, all giving their all, can then point to a particular player in a section - and say, "You did that wrong!" ...

And no dog I know can do that! That proves it: human ears are better.

29 minutes ago, pkane2001 said:

 

And no dog I know can do that! That proves it: human ears are better.

 

You would be amazed. Some dogs yeowl or bark when hearing noise they don't like. Others don't care.

1 hour ago, botrytis said:

I have no idea what 'crazy uncle Frank' is talking about.

What I'm talking about, is that human hearing can be as precise as needed for the situation at hand - if the owner of the ears cares about he's listening to, .

How precise ... oooh, about 120dB worth - what's that mean? How about, you get in a car, drive like crazy for just under a km; and as you get to the km point, slow down; and finally stop - to an accuracy of 1 mm, over the whole distance of 1km .. translating that to human hearing, not too bad for a "poor quality" sense  ...

10 hours ago, botrytis said:

 

Human ears are poor quality, compared to dogs and other species. Humans are a vision and tactile centric species.

And frankly, our vision isn’t all that great either.

I watched the video 🙂 Thank you for sharing.

From the observation, it seems AP analyzers are less susceptible to ground loop problem and its DAC measurement result is relatively consistent and reliable. On the gaming PC review sites, RMAA is still mainly used to measure onboard DAC quality. RMAA itself is excellent app but people use it with too carelessly and their measurement result is totally unreliable, there is a 15dB to 20dB SNR discrepancy on the same onboard DAC, almost always contaminated with ground loop problems, sometimes it seems measurement is simply failed with accidental click noise or other issues.

From 16:45 of your video, there is some solution to measurement setup problem presented. With RMAA and off the shelf audio interface, additional care is needed to get more consistent and meaningful results. I learned inexpensive but adequate measurement method from an local DIY amp builder magazine articles, but it seems the knowledge is not shared enough and/or gaming PC reviewers do not have enough curiosity for audio measurement.

About intersample overs, I read somewhere, some oversampler of earlier DAC has no tolerance and waveform is flipped by integer arithmetic overflow!

Here is my RMAA result of Asus Z370i onboard Realtek S1220A DAC. I think it is excellent DAC but on the internet, there are 15dB to 25dB worse RMAA results circulated 😱

On 4/20/2022 at 7:13 PM, Allan F said:

 

Don't trust audiophiles who believe that measurements will tell them what audio gear sounds like. Measurements have their place, but they are no substitute for listening.

Very few audiophiles do ears only audio auditioning.

But if products do sound different there is a measurable testable reason for that sound difference.

2 hours ago, Speedskater said:

Very few audiophiles do ears only audio auditioning.

But if products do sound different there is a measurable testable reason for that sound difference.

Not necessarily. You, incorrectly IMO, assume that there is a direct relationship between measurements and sound quality. Measurements may or may not explain the difference.

Very few audiophiles base their purchasing decisions on measurements. They consider them, but the final evaluation is based on listening.

21 minutes ago, Allan F said:

 

Not necessarily. You, incorrectly IMO, assume that there is a direct relationship between measurements and sound quality. Measurements may or may not explain the difference.

There is a direct relationship between measurements and transparency. Sound quality is a subjective metric and what I prefer may differ significantly from you or from anyone else. Using measurements designed to test for transparency to judge sound quality assumes that you prefer transparent playback. Not everyone does.

The interesting thing for me is that I started my digital journey with one of the best DACs you could buy at the time, which had very good specs, and capabilities - which delivered 'transparent' SQ, when the whole system was sufficiently optimised. And this made the lack of transparency of nominally extremely good measuring, expensive gear I came across elsewhere painfully obvious. This was one side of the equation. The other side was then trying modding of cheap 'n' cheerful stuff, which used parts that had only mediocre measured performance - and extracting subjectively transparent SQ from them.

This demonstrated that the measurements of equipment had close to zero correlation with their ability to deliver satisfying listening. As of now, things have barely improved - not many would accept having to live indefinitely with some combo of gear, bought purely on the basis of numbers - because, "they were right!" ...

The video states a variation of measurement test vector is insufficient and big SQ deterioration or noise is sometimes overlooked. I agree about it. Personally, I'd like to see coil/MLCC whine acoustic high-pitched noise spectrum test of laptop PC and computer monitors to find suitable gears for high resolution recording. Also it is nice to have LPS power transformer rumble noise test caused by DC offset added AC mains power (some household in Japan has this problem). DAC click noise level test when sample rate is changed or mute is activated/deactivated or PCM/DSD mode change.

18 hours ago, yamamoto2002 said:

RMAA itself is excellent app but people use it with too carelessly and their measurement result is totally unreliable, there is a 15dB to 20dB SNR discrepancy on the same onboard DAC, almost always contaminated with ground loop problems, sometimes it seems measurement is simply failed with accidental click noise or other issues.

I don't think the issue is with RMAA itself.
RMAA as software is fine and very useful, but you can use any ADC you want with it, so the results are going to be heavily dependent on what ADC someone is using and how they have things set up.

If you use RMAA with an ADI-2 Pro FS R for example like Archimago (or myself prior to getting the APx555) then you can get very good results.
If you just use a basic focusrite interface then you're not gonna have such a good time.

Additionally, when it comes to motherboard audio there are also so many other factors. The PSU you're using, what other hardware such as GPU/CPU are in that particular PC etc, is it under load or idle.
How noisy a PC is can vary massively.

 

18 hours ago, yamamoto2002 said:

About intersample overs, I read somewhere, some oversampler of earlier DAC has no tolerance and waveform is flipped by integer arithmetic overflow!

I saw this actually but can't remember for the life of me what DAC it was. I think it might have been an earlier denafrips DAC? I had some interesting and similar behaviour when testing the Ares 2, I didn't do an intersample overs test but when using the slow filter, just doing an impulse response caused the waveform to wrap around as shown below.

I'm really not sure how this would happen but it seems it did :P
It didn't happen with Sharp OS though

This would most definitely be MUCH more problematic than how intersample overs are handled on modern DACs (which is that they simply clip/max out). But luckily I don't think this is something that happens with most DACs.
 

1 hour ago, GoldenOne said:

If you use RMAA with an ADI-2 Pro FS R for example like Archimago (or myself prior to getting the APx555) then you can get very good results.
If you just use a basic focusrite interface then you're not gonna have such a good time.

From my experience, self powered ADC is more problematic than bus powered USB/firewire ADC. For example, Babyface Pro or bus powered Fireface 400 is easier to use than Fireface UC or ADI-2 Pro

1 hour ago, GoldenOne said:

I saw this actually but can't remember for the life of me what DAC it was. I think it might have been an earlier denafrips DAC?

 I don't remember correctly but some early portable CD player have this problem, and content with intersample overs sounds very differently with such device. IMO it is better for every DAC to produce loud noise and bright red lamp is lit when intersample over happens, because it is defect of music content that should be found and fixed before delivery. Intersample overs resemble to bugged software that cause undefined behavior.