Why you can't trust measurements

On 4/25/2022 at 10:48 AM, botrytis said:

I was happy to walk away from that forum.

Alive. 😂

On 4/25/2022 at 3:26 AM, botrytis said:

Well, the way Amir dismissed and banned ELAC designer. Andrew Jones, speaks volume about his arrogance.

Ahoi, i just joined to tell you that is a false statement. Andrew Jones wasn't banned.

23 minutes ago, TheBatsEar said:

 

Ahoi, i just joined to tell you that is a false statement. Andrew Jones wasn't banned.

Stop it with posting facts! This is the Objective sub-forum, after all. We don't like facts here, only speculation ;)

12 hours ago, vortecjr said:

Welcome to the art of measurements.

 

We are still waiting for you to publish yours.

https://forums.stevehoffman.tv/threads/just-got-banned-by-asr-for-pointing-out-the-inconvenience-truth.1144772/

The OP is not me btw!

45 minutes ago, Iving said:

https://forums.stevehoffman.tv/threads/just-got-banned-by-asr-for-pointing-out-the-inconvenience-truth.1144772/

 

The OP is not me btw!

Didn't realize he got banned! Too bad, he isn't a bad guy, just a bit annoying, posting the same things in nearly every thread, including the latest one he started, where he demanded that Amir prove him wrong by inviting him to his house for a blind test :)

I'm guessing that Amir didn't take to the idea, based on the above..

3 minutes ago, pkane2001 said:

 

Didn't realize he got banned! Too bad, he isn't a bad guy, just a bit annoying, posting the same things in nearly every thread, including the latest one he started, where he demanded that Amir prove him wrong by inviting him to his house for a blind test :)

 

I'm guessing that Amir didn't take to the idea, based on the above..

 

Yeah - I imagined the human factor hardly absent!

7 hours ago, Iving said:

https://forums.stevehoffman.tv/threads/just-got-banned-by-asr-for-pointing-out-the-inconvenience-truth.1144772/

 

The OP is not me btw!

What's going on ... that thread is no longer there?

14 minutes ago, JoshM said:

Of course, when it comes to disfavored brands, like Schiit

Recently he's found several Schiit models that measure well, and has praised Schiit.

35 minutes ago, firedog said:

Recently he's found several Schiit models that measure well, and has praised Schiit.

Which for me raises the interesting question of to what extent veteran designers are able to engage in specmanship that doesn't have an audible effect.  I too have noticed the earlier slagging of Schiit and the later appearance of a couple of their units relatively high up on the Holy SINAD List.

42 minutes ago, JoshM said:

I didn’t see the SHF thread before it got removed, but I’m guessing it’s the guy who recently posted on ASR that six of nine of the budget components recommended by Amir that the poster bought failed or measured differently from what Amir found. The poster raised the possibility that some companies were sending cherry-picked units, and requested to send his units to Amir to verify (since, as usual, people didn’t believe the poster, even though he did his own measurements, blind tests, etc.). In response, Amir said (contrary to his “we measure anything for free” claim), that the poster would have to pay him $200 per piece of equipment to confirm the issues, then locked the thread.  

 

What that all illustrated for me (and aligns with my own experiences with these brands) is that certain companies lauded by ASR have major failure rates if you search relevant threads (on Head-Fi, any retailer with user reviews, and sometimes ASR itself) for words like “fail,” “issue,” “problem,” or “died.” But anyone overtly raising the possibility that this is a systematic issue on ASR is shut down. 
 

Of course, when it comes to disfavored brands, like Schiit, Amir will measure multiple units to find faults, conduct teardowns to inspect for minor solder splatter, etc. Favored companies are, of course, spared this critical eye. It’s science, you know. 

Umm, no, not what happened. None of his components failed. Wayne did have one unit with a bad display from the start. He tried to prove that they all sounded different. His claim was that since they all measured well (by Amir) he wanted others to acknowledge that his hearing was detecting things that were beyond measurements. Meanwhile, he had problems with channel imbalances, equalizing levels, and at one point told me he wasn't interested in learning or understanding technology or measurements. This was after about a month of me trying to help him use DeltaWave properly (he failed). And no, I've not seen a single successful measurement from him, and he gave up trying after a while.

In the last thread he started, he insisted that Amir should organize a blind test for him to come over to Amir's house to prove to him that he can hear these differences. His one theory was that this may be due to "golden samples" sent to Amir by manufacturers. He had no measurements, and no real evidence to back up his claims, but he was very vocal and insistent about what he heard in other threads. Don't know what got him banned in the end.

22 minutes ago, Jud said:

 

Which for me raises the interesting question of to what extent veteran designers are able to engage in specmanship that doesn't have an audible effect.  I too have noticed the earlier slagging of Schiit and the later appearance of a couple of their units relatively high up on the Holy SINAD List.

Schiit invested in AP measurement equipment and started doing actual automated QA testing on their equipment before shipping, and also publishing these test reports. While some of their equipment still doesn't measure well, they are one of only a few manufacturers that publish a complete set of measurements. Good for them, regardless of where they wind up on the SINAD chart :)

1 hour ago, firedog said:

Recently he's found several Schiit models that measure well, and has praised Schiit.

It would be impossible to get around them measuring well. But, as I said, he’s done teardowns on some of those Schiit products and nitpicked solder spatter, claiming without evidence that it could cause failures. Meanwhile, products from other companies not only aren’t subjected to teardowns, but well-documented issues with failures and other issues are dismissed. 

43 minutes ago, pkane2001 said:

 

Umm, no, not what happened. None of his components failed. Wayne did have one unit with a bad display from the start. He tried to prove that they all sounded different. His claim was that since they all measured well (by Amir) he wanted others to acknowledge that his hearing was detecting things that were beyond measurements. Meanwhile, he had problems with channel imbalances, equalizing levels, and at one point told me he wasn't interested in learning or understanding technology or measurements. This was after about a month of me trying to help him use DeltaWave properly (he failed). And no, I've not seen a single successful measurement from him, and he gave up trying after a while.

 

In the last thread he started, he insisted that Amir should organize a blind test for him to come over to Amir's house to prove to him that he can hear these differences. His one theory was that this may be due to "golden samples" sent to Amir by manufacturers. He had no measurements, and no real evidence to back up his claims, but he was very vocal and insistent about what he heard in other threads. Don't know what got him banned in the end.

 

I can only go by what he said, where he referenced measuring his units and conducting blind tests. I see you were trying to help him some in another thread.
 

My read of the thread is that he assumed (correctly!) that no one would believe him about his measurements or blind tests. (Here we’re back to blind tests by others being dismissed. A “no true Scotsman” issue.) So he asked if Amir would assist with blind tests or measure his units.

Amir then requested $200 per unit for measurements. Then a mod locked the thread and (apparently) banned him. From what I read, nothing the original poster said was mean or offensive.

I have no reason to doubt the original poster’s claims. Nor do I know them to be true. But shutting him down for daring to raise the possibility that certain brands might have reliability issues that don’t get caught by ASR’s reviews seems to be what raised hackles. 
 

The overarching issue in my view is that not all products over there are subjected to the same measurements (as @GoldenOne noted) or the same critical analysis when it comes to construction and reliability. 
 

I’ll say that personally (starting before ASR even existed), I’ve owned five products by the two most-praised brands at ASR. Three failed. One literally sparked when plugged in. Threads and reviews elsewhere indicate such experiences aren’t uncommon. Meanwhile, solder spatter on other brands is labeled a reliability or safety risk. It’s just not ethical reviewing practices, which requires that all products are subjected to roughly the same analysis. I realize that lots of journalistic ethics are out the window in the internet age, but they still matter to me. 

52 minutes ago, Jud said:

 

Which for me raises the interesting question of to what extent veteran designers are able to engage in specmanship that doesn't have an audible effect.  I too have noticed the earlier slagging of Schiit and the later appearance of a couple of their units relatively high up on the Holy SINAD List.

I think this comes down to “incentives matter.” ASR helped cultivate a market for budget SINAD-chasing DACs and headamps. That wasn’t Schiit’s initial goal, but they realized they could easily make both that would retail for ~$100 each and have better warranties than similarly-priced competitors. It was a smart business decision, and I tend to tell people who are measurements-focused to just buy the budget Schiit stack and be done with their audio journey. 

4 minutes ago, JoshM said:

I can only go by what he said, where he referenced measuring his units and conducting blind tests. I see you were trying to help him some in another thread.
 

My read of the thread is that he assumed (correctly!) that no one would believe him about his measurements or blind tests. (Here we’re back to blind tests by others being dismissed. A “no true Scotsman” issue.) So he asked if Amir would assist with blind tests or measure his units.

 

Amir then requested $200 per unit for measurements. Then a mod locked the thread and (apparently) banned him. From what I read, nothing the original poster said was mean or offensive.

I have no reason to double the original poster’s claims. Nor do I know them to be true. But shutting him down for daring to raise the possibility that certain brands might have reliability issues that don’t get caught by ASR’s reviews seems to be what raised hackles. 
 

The overarching issue in my view is that not all products over there are subjected to the same measurements (as @GoldenOne noted) or the same critical analysis when it comes to construction and reliability. 
 

I’ll say that personally (starting before ASR even existed), I’ve owned five products by the two most-praised brands at ASR. Three failed. One literally sparked when plugged in. Threads and reviews elsewhere indicate such experiences aren’t uncommon. Meanwhile, solder spatter on other brands is labeled a reliability or safety risk. It’s just not ethical reviewing practices, which requires that all products are subjected to roughly the same analysis. I realize that lots of journalistic ethics are out the window in the internet age, but they still matter to me. 

It's important to know what measurements are and what they are good for. Certainly, poor measurements can point to poor design and/or poor engineering. Or, they may demonstrate intentional deviation from transparency.

Good measurements are no guarantee of reliability and hour-long test of a DAC or other component isn't going to reveal longer-term issues with the design. GoldenOne or Amir are not testing for this. Reliability testing is a much more complicated and involved process and is better done by test labs with dedicated people and equipment over a much longer period of time, with the ability to vary temperature, humidity, mechanical vibration, simulating various source and downstream component, and accounting for many other factors. Or you can just read the posts and complaints by existing owners :)

5 minutes ago, pkane2001 said:

 

It's important to know what measurements are and what they are good for. Certainly, poor measurements can point to poor design and/or poor engineering. Or, they may demonstrate intentional deviation from transparency.

 

Good measurements are no guarantee of reliability and hour-long test of a DAC or other component isn't going to reveal longer-term issues with the design. GoldenOne or Amir are not testing for this. Reliability testing is a much more complicated and involved process and is better done by test labs with dedicated people and equipment over a much longer period of time, with the ability to vary temperature, humidity, mechanical vibration, simulating various source and downstream component, and accounting for many other factors. Or you can just read the posts and complaints by existing owners :)

I agree. I think it would be perfectly fine and ethical to say, “I’m not mentioning warranties, or reliability, or conducting teardowns. Just the same suite of measurements. Caveat emptor.” But I don’t think it’s ethical to conduct teardowns and speculate about reliability for some reviews, but not others. 

5 minutes ago, GoldenOne said:

I've had a couple products measure differently from the measurements amir showed. Not necessarily by a huge degree but enough that if amir were to put the unit I'd received on his ranking list it'd be several spots lower.

 

Unfortunately with Amir's position it would be silly to assume that manufacturers WERENT sending cherry picked units.

 

Realistically a fairer way to do things would be for Amir to ask the manufacturer to reimburse him for the purchase of a retail unit from a distributor. This would ensure that he's testing something that will be in consumer hands. Not a cherry picked unit.

 

In regards to failure of devices though. Unfortunately reliability is measured best by user reports. It's not really something Amir can test so that's not really fair to put on him. 

A product can measure well but die 2 months later.

 

Also, on the note of having to pay to get stuff measured, if anyone would like anything measured I'll do a full set for free on my APx555. Just cover shipping.

Totally agree. We really need some comprehensive surveys of owners about reliability by brand, product, or ideally both.

10 minutes ago, JoshM said:

I agree. I think it would be perfectly fine and ethical to say, “I’m not mentioning warranties, or reliability, or conducting teardowns. Just the same suite of measurements. Caveat emptor.” But I don’t think it’s ethical to conduct teardowns and speculate about reliability for some reviews, but not others. 

It would be unethical for him to conduct a teardown of a product that the owner doesn't want torn down :) Where permission is given, Amir usually does and posts a separate thread for these. 

17 minutes ago, GoldenOne said:

I've had a couple products measure differently from the measurements amir showed. Not necessarily by a huge degree but enough that if amir were to put the unit I'd received on his ranking list it'd be several spots lower.

This was brought up here before, but I'm not sure what the answer is. Are your measurements done with the same AP project settings as those that Amir is using? If not, there could be differences in settings (as per your video) that could account for some of the differences.