TANNER CAMPBELL

Register for upcoming cohort

Listen Notes Has A Measurement Problem

A sheep sitting along side a chalkboard in a classroom. The chalkboard says 2+2=5

Share This Post

Listen Notes is a podcast directory and podcast search engine that, a couple of years ago, starting giving ratings to podcasts in the form of “Podcast X is in the top Y% of all podcasts.” I never much trusted the way they were calculating these scores because, unless they can gain direct access to the platforms hosting the audio and serving the RSS feeds of these podcasts, all they can do is guess. That said, it seemed harmless, and while I made a comment about how useless I thought the score was then, I thought it was mostly harmless. I was wrong.

Coaches, Gurus, and Experts

People who are good at marketing know how to say things that are technically true but which, if their target market were curious enough to dig, is easily discovered to be either untrue or a non-statement.

Before I continue, I’ll state the obvious: I label myself as an expert. I am one, so I’m not going to apologize for that. I just wanted to ensure that you know that I know that, to some degree, I’m including myself in what is to follow.

Sometime in the last 8-months I’ve begun to see statements in a lot of bios on various social platforms that include the statement: “Top X% Podcaster.” Below is an example.

The first thing I notice about these accounts is their low follower counts.

Almost all of them are under a thousand followers and every single one I’ve listened to is poorly produced, with crappy audio. Now, neither of those things necessarily mean the podcast can’t be a top 2% podcast, but I was absolutely perplexed that these podcasts, in particular, were. At first I thought people were just lying, but that’s a big risk. I mean sure you could *say* you’re a top 2% podcaster, but why say top 2% instead of top 5%? Or Top 10%? Those are both less risky more believable claims… right?

Then I had a podcaster submit a #PodAudit to Podcasting Sucks, and in her bio she claimed to be a top 5% podcaster. I listened to the episode and I said, out loud, “There’s no fucking way. That’s impossible.” I expressed this to a couple of friends and they said “It’s probably ListenNotes.com.”

Now, at this point I had completely forgotten about Listen Notes but, the second I heard it, I thought, “Oh. Of course. That’s exactly what it is.”

And sure enough, here’s Sonja Martinovic’s Mindset Mastery & Manifestation podcast on Listen Notes:

Okay, so certainly a top 2% podcast must rank in the charts, right?

I mean, it’s the top 2% globally! So, where is The Vault in the charts?

It’s not there. It doesn’t rank anywhere, in any country, on Apple Podcasts. Not in its category or in any other. Don’t believe me? Click here: source.

What it means to be in the top 2% of podcasts

This data is a little old, about a 18-month or so, but here’s what the top 2% of podcasts see in terms of downloads according to Omny Studio which is owned by Triton Digital (a data collection company that does a lot of work in the podcasting space):

Source: Omny Studio blog, get the full context here.

To be in the top 2% you need 13,000 downloads per episode, in the first 30-days of that episode’s release. That means if you release an episode every week, on Friday, you would expect that episode to have somewhere in the ballpark of 13,000 non-unique downloads before the end of that month.

Does The Vault podcast do that?

I cannot know, of course, but by claiming to be in the top 2% of podcasts, it sure as hell is suggesting that it does.

So let’s look at a podcast for which I can know…

This is one of my own podcasts, Practical Stoicism. Here are the stats for the last few weeks for Practical Stoicism:

Since its initial release on Megaphone, just a few episodes ago, Practical Stoicism has accrued 622 downloads. That’s a good amount of downloads for a podcast that isn’t very old, and for the sake of math let’s divide that total by the total number of episodes (excluding the trailer, just to inflate the number a little), and see how many downloads per episode we’ve gotten in these first 30-days.

124 downloads per episode in the first 30-days.

Okay, well, where do I rank according to Omny?

Source: Omny Studio blog, get the full context here.

I think a fair guess is in the top ~40%.

How about according to Listen Notes?

Top 10%!

A podcast that’s been out for a month and gets 124 downloads per episode is ranked globally in the top 10% of podcasts?

Clearly there’s a disconnect because, if I were really ranked in the top 10%, at least according to Triton Digital, I’d be receiving 1800 downloads per episode per month, not 124.

But do I rank in any charts? Surely if The Vault didn’t, I shouldn’t either, right?

Wrong.

Source: https://rephonic.com/podcasts/practical-stoicism

Here are some other examples from other podcasters who were willing to share their numbers with me for this article

Podcast Name30-Day ~DLsApple Podcasts RankGlobal Rank LNTriton Rank
Cinematic Sound Radio760#70 USA / MusicTop 1.5%Top 20%
A Nefarious Nightmare160NoneTop 5%Top 40%
Beer in Front80#77 USA / Leisure / CraftsTop 5%Bottom 50%
The Greatest Song Ever Sung (Poorly)70NoneTop 10%Bottom 50%
Dad’s Kitchen200NoneTop 5%Top 40%
Nomads of Fantasy120NoneTop 10%Top 50%
Confirmed with ListenNotes.com and Rephonic.com

So what the hell is Listen Notes measuring, exactly?

In their own words, Listen Notes ranks podcasts with an LS (Listen Score) and a GR (Global Rank, represented as a %):

Generally speaking, the higher the Listen Score of a podcast, the better. But it’s important not to judge Listen Scores in absolute terms. That’s because Listen Score is a relative metric by definition. It’s not possible to say that a Listen Score of 30 is good, or 50, or 60, or 70. It’s all relative. Here’s a general rule of thumb: A Listen Score is good if it’s higher than or comparable to similar podcasts.

https://www.listennotes.com/listen-score/

I’m willing to be happy with relative Listen Scores because it seems like Listen Scores are calculations of listener engagement, and that’s fine. It may even be useful. Rephonic does the same thing.

But I’m surely not going to be okay with a relative Global rank because that’d be like telling the guy that came in last place in the 40-meter that he didn’t finish dead last and that he’s actually, technically 3rd. Your global rank, in anything, isn’t relative – though it could be handicapped or compartmentalized like in Golf or Boxing (Featherweight vs. Welterweight, for example).

What do they say about Global Rank?

Global Rank is the ranking of 2,772,805 podcasts based on Listen Score. For example: Let’s assume there are only 5 podcasts in the world: A, B, C, D, and E. Podcast A’s Listen Score (LS) is 25, B’s is 30, C’s is 50, D’s is 70, and E’s is 95. As we can see, E has the highest Listen Score (i.e., LS 95), meaning E is the most popular podcast. E’s Global Rank is Top 20%. D (LS 70) is the second most popular podcast, and its Global Rank is Top 40%. C (LS 50) is the third most popular podcast, and its Global Rank is Top 60%… You get the idea. Global Rank is for podcasts of all regions, languages, and genres. In the future, we may consider ranking podcasts in specific regions, languages, and genres, e.g., Top 1% podcasts in Japan.

https://www.listennotes.com/listen-score/#faq9

This is, to resurrect an old timey phrase: poppycock.

If what they’re saying is that their global rank is based on their calculations from their relative Listen Scores, then they aren’t ranking based on downloads (which is what ad buyers, sponsors, and people looking to trust someone are most likely looking for), they’re ranking based on… something like engagement.

If gurus, experts, and coaches were saying they were among the top 5% in audience or social engagement, that would be a far different claim, suggesting a far different thing, than “I get a lot of downloads.” And that’s what Listen Notes is, inadvertently or otherwise, allowing clever marketers to do, they’re allowing people who get shit downloads to parade around as if they get a lot and that’s making the misrepresentation of skill and impact easier to get away with.

This might hurt your feelings

Ya’ll, I get it. When I ran my first serious podcast in 2013-2015, Webalizer promised me I had hundreds of thousands of subscribers. And when podcast hosting providers became more mainstream and I found that that number was more like 10,000, I was sad. When the IAB started introducing even better measurement practices and that number went down to 5,000, I was even more sad.

But you know what?

It was 5,000 the whole time, and I was never at top .01% podcaster as Listen Notes probably would have told me had it been around then.

The truth is what’s valuable, and if you don’t agree with that, you’re trying to mislead people… and probably for the sake of making money. That’s wrong, and you should be exposed for doing it.

Most of you, though, are likely to have truly believed your podcast was a top 5% podcast, and that probably felt pretty good, and being told it’s an illusion probably feels the opposite of that.

I’m sorry to do that to you, but this is part of my job.


Header photo by Michal Matlon on Unsplash

More To Explore

I Minted An NFT, Let’s Talk About It

I minted an NFT, it’s a membership card to an exclusive community. It’s a one-time purchase, it costs $10, and I think it’s very cool.

How Search Works in Apple Podcasts

In today’s episode we’ll be reviewing Apple Podcasts’ recent publication of a page on apple.com that explains how search works within the Apple Podcasts ecosystem.