HomePod listening tests reveal the grimy truth about audio reviews


As Apple’s HomePod has been out for a week, most of the news, reviews, and early problems have already been discussed ad nauseum. But you might have missed two unusual articles from Yahoo’s David Pogue, whose real world HomePod listening test last Saturday created enough controversy to merit a “readers weigh in” follow-up today. After having praised HomePod’s audio in his initial review, Pogue discovered that a panel of listeners preferred the Sonos One or Google Home Max instead. When he published that finding, some of his readers freaked out, leading him to dive deeper into his testing methodology — and other testers’ results, as well.

My review of the HomePod was pretty straightforward on audio quality: It’s an overly bass-heavy, monaural speaker with recessed mids, ultimately not worthy of its steep asking price. Some real-world reviewers agree with my take, and some don’t. That’s fine — like pocketbooks, audio preferences are somewhat personal. Even before the review was published, my advice was to audition HomePod for yourself.

But it’s worth exploring why there’s been so much difference between the early, universally positive HomePod sonic reviews and the less enthusiastic ones that have followed from members of the public. Specifically, I’d like to fill you in on some details that will help explain why something as simple as a speaker can generate such polarized opinions between “expert” reviewers, “audiophiles,” and regular people.

If you don’t like knowing how review sausage is made, stop reading here, because this is going to get a little messy.

How they tested, or should have tested

Over the course of his articles, Pogue first hinted and then fully explained that Apple required early reviewers to attend a special listening session comparing the HomePod against three competing speakers. Apple selected the tracks, and in an important detail most reviewers probably didn’t realize, played them over different audio connections ranging from Wi-Fi to Bluetooth, Ethernet, and audio line-in. As Pogue notes, reviewers typically left Apple’s listening sessions feeling that HomePod had outperformed its rivals — no shock, given that two were around half or less of HomePod’s price.

But after Pogue disclosed Apple’s speaker shootout, readers questioned it, leading him to run a more controlled comparison of his own. He took the four speakers, picked five revealing songs, assembled a fairly representative panel of listeners, and made sure the input source and volume were kept constant across all the speakers. Contrary to Pogue’s initial assumptions, and his early review’s conclusions, the panel rated HomePod’s overall sonic performance below the Sonos One and Google Home Max.

Because the internet has its fair share of self-proclaimed audiophiles, the second and more neutral round of test results didn’t sit well with some of Pogue’s readers, either. There were questions over the thin curtain that masked the speakers’ identities, suggestions of bias due to listeners’ physical positions relative to the speakers, and complaints that Spotify was the audio source for all the speakers. Pogue was called “grandpaw,” asked to use more randomized methodologies, and told to retest in either sonically silent or visually neutral rooms. The list of reader comments reads much like a series of high school excuses for failing to turn in homework — except nastier.

Here’s how review sausage is made

Although I don’t agree with Pogue’s original claims that “HomePod sounds really, really great” or “amazing,” I applaud him for the courage it took to publicly re-examine the comparison process that led to his initial conclusions. His all-important first impressions were created using a skewed test, and a more neutral methodology would have yielded different results, as his panel’s experiences demonstrated.

Why was there such a difference between the “expert” reviewers’ opinions and Pogue’s real world panel? I’m going to fill you in on a critically important piece of background information that will help you understand the reality behind speaker reviews:

Most journalists know very little about speakers, or audio in general.

Here’s one more key point:

Many journalists, including even legitimate audio experts, have damaged their hearing, or aged past their ears’ sonic prime.

Apple knew this, and it knew that (some of) the people it picked to review the HomePod might feel compelled to disclose their lack of audio experience. That’s why it set up a four-speaker testing room — a step it literally never took with competitors to the Apple TV, Apple Watch, iPad, or iPhone — and walked reviewers through the tests. Apple spared writers the need to do their own comparisons and helped them reach a positive conclusion about HomePod’s audio, presenting it in the best possible light.

These details should help you understand why some of the first HomePod reviewers said that they could have stopped writing about the speaker after a paragraph or so, but felt obliged to write more. Normally, most of these people don’t write about the nitty-gritty of audio products, and they don’t feel comfortable discussing the technical nuances of “great sound.” But in this case, they were being given early access to a new Apple product, knew they had to write full reviews, and did the best jobs they could under the circumstances. That might seem like a somewhat charitable interpretation given how over-the-top some early reviewers were in praising HomePod, but in my experience, it’s largely accurate.

An alternative: Don’t trust (most) audio reviews

If you care about audio — and you certainly should, for $ 349 per speaker — one option would be to seek out opinions from people who have actual audio expertise. Unfortunately, there’s a lot of noise on the internet these days, and true expert opinions are surprisingly hard to find. Assuming you can’t decide for yourself, the best alternative is to find someone with a public track record of audio reviews that match your preferences and experiences, or alternately, someone whose opinions are reliably the opposite of yours.

One thing you shouldn’t do is put faith in people who appear out of nowhere armed with audio graphs purporting to show that a given speaker has “perfect, audiophile-quality sound,” or reviewers who write effusively about the magical melodies produced by every piece of audio gear they test. Reviews that focus heavily on specs or poetic descriptions of sound are usually full of crap — regrettably, many readers confuse pages of numbers and words with genuine expertise. It takes a lot more courage for a writer to produce brief, blunt, and accurate conclusions about a speaker than to convince you it’s going to deliver sonic heaven every time you turn it on.

Unlike his first HomePod review, Pogue’s last article doesn’t attempt to be authoritative on the speaker’s sound. While raising an eyebrow at certain conclusions, it acknowledges a diversity of opinions from everyone, from his panel of listeners to various readers to professional testers at Consumer Reports. That’s a reasonable approach: If you can’t find a great single reviewer whose tastes mirror yours, considering a variety of different opinions is probably the second healthiest way to approach audio purchases.

The best option? Try the product for yourself, and return it if you don’t like it. You’re the best judge of whether something sounds (and works, and looks) great enough to be worthy of your hard-earned cash.

Happy listening!

Apple – VentureBeat

AppleInsider podcast interviews comedian Will Abeles, reviews Apple’s HomePod

Article Image

This week on the AppleInsider podcast, Victor interviews Will Abeles, whose comedy album debuted at No. 1 on iTunes. In addition, Victor and Neil talk about Neil’s HomePod review, as well as Apple’s growing share of the smartphone market’s revenue.
AppleInsider – Frontpage News

Huawei responds to allegations about soliciting fake user reviews for the Mate 10 Pro

Yesterday a report came out alleging that Huawei solicited fake user reviews of the Mate 10 Pro in its private Facebook group, which has 60,000 members. The company told the members of that group to go and leave positive reviews on the Best Buy page listing the Mate 10 Pro (which is still on pre-order in the US), in exchange for a chance to be one of a few “beta testers” of the phone. It was strange wording for sure – how can a positive review be anything but fake if you haven’t used the device you’re talking about? Android Authority has reached out to Huawei about this baffling situation,…

GSMArena.com – Latest articles

Huawei Asking People To Write Fake Positive Reviews In Exchange For A Chance To Test Mate 10 Pro For Free

It seems that Huawei is lowering itself to less than scrupulous practices in order to try and promote its new Mate 10 Pro smartphone. In a private Facebook group, which has approximately 60,000 members, Huawei has been soliciting positive Best Buy reviews in exchange for the possibility of being selected as an early hardware beta tester.

[ Continue reading this over at RedmondPie.com ]

Redmond Pie

Huawei loads its Best Buy Mate 10 Pro listing with over 100 fake reviews, inevitably gets caught

Almost three months ago, we reviewed Huawei’s Mate 10 Pro. We concluded that it’d a good buy when it hit the US market, so long as it wasn’t priced at $ 1000. Well, Huawei hasn’t made that mistake (the phone is currently $ 799.99 with a $ 150 gift card at Best Buy), but it isn’t doing itself any favors today. The company has overloaded the Mate 10 Pro’s Best Buy listing with over 100 fake reviews from aspiring “beta testers,” giving the phone – one still in the pre-order phase, mind you – a 4.9-star overall rating.

Read More

Huawei loads its Best Buy Mate 10 Pro listing with over 100 fake reviews, inevitably gets caught was written by the awesome team at Android Police.

Android Police – Android news, reviews, apps, games, phones, tablets