Does Reddit have an astroturfing problem? What the data suggests
One chef's-knife brand gets 31% of its "what should I buy" mentions from 5% of the accounts, four times what chance predicts. So I pulled those accounts' full Reddit histories.
Spots 
I like to cook, and cooking turned into
I like to cook, and cooking turned into an obsession with high-end Japanese chef's knives. When I want to buy a knife, or anything else, I type the product name into Google and add the word "reddit". A lot of people do this. Mike Riggs wrote it up in Reason in 2022 as "the Reddit hack", after Dmitri Brereton's essay on Google search made the same point: a query with "reddit" on the end returns humans instead of affiliate pages. The bet behind the habit is that a stranger in r/chefknives has no reason to lie to you about a knife.
That bet has an obvious weak point. If
That bet has an obvious weak point. If Reddit is where buyers go for unpaid opinions, Reddit is where a brand would want to plant paid ones. I wanted to know whether the knife subreddits I read, and scrape for New Knife Day, show any sign of that. Not a hunch about one suspicious comment, but something I could count and someone else could recount. Why the question is testable at all
Last year I fine-tuned a small named-entity model
Last year I fine-tuned a small named-entity model, GLiNER, to pull brands, models and steels out of knife comments. From "picked up a Mazaki in white #2, way better than my old Fibrox" it returns Mazaki as a brand, Fibrox as a model and white #2 as a steel. That model runs over every comment the New Knife Day scraper collects from six subreddits: r/knives, r/knifeclub, r/chefknives, r/japaneseknives, r/FixedBladeEdc and r/KnifeSteels. The write-up on that model is here.
So for every comment I already have who
So for every comment I already have who wrote it, which brands it names, and whether the thread it sits in is someone asking what to buy. That is enough to ask a narrow question: in the threads where a recommendation changes a purchase, who is doing the recommending?
New Knife Day is my site for knife
New Knife Day is my site for knife collectors. It catalogs knives and steels and tracks which knives people on Reddit are buying and arguing about, so I have a stake in knife Reddit being worth reading. New Knife Day is at new.knife.day. What astroturfing would look like in the data
Nobody publishes their shill accounts, so I had
Nobody publishes their shill accounts, so I had to decide in advance what paid posting would leave behind. The market for it is not hidden. REDCmts sells one Reddit comment for $9.99 and 100 for $699.99, from what it calls "real, aged accounts", and shows a gallery of brand mentions it says it delivered. Soar says its accounts are "aged and manually warmed" for weeks before a single brand mention. Bazzly advertises automated replies to every post that looks like someone shopping.
Taking those sales pages as the description of
Taking those sales pages as the description of the product, a paid campaign for one brand, delivered through a handful of prepared accounts, should show up as: A small tail of accounts writing a disproportionate share of the brand mentions in "what should I buy" threads. Those accounts naming one brand almost every time they name any. Thin accounts: few comments, low scores, no real standing in the subreddit. Young accounts, or accounts with histories that are hidden or wiped. Accounts that post mostly in knife subreddits, since the knife comments are what is being paid for. Links to a store or an affiliate page.
Every one of those is also what a
Every one of those is also what a devoted fan, a maker's employee posting on their own time, or a brand's own subreddit regulars wandering into a buying thread would produce. Public Reddit data can show that recommendations are concentrated, and cannot show why. Everything below is about the first half. The corpus, and the refresh that changed the answer
The scraper stores each new post and its
The scraper stores each new post and its comments shortly after posting. That turned out to be the wrong moment for this question: a buying thread collects its recommendations over the following day or two, and r/knives posts had 1.5 stored comments each. A refresh pass went back to 3,607 posts older than 48 hours and refetched their comment trees, which took the corpus from 21,673 comments to 51,129. Before the refresh the test below found nothing, because about 800 buying-thread mentions were too few to tell 7.6% from the 7.1% chance gives.
News
Does Reddit have an astroturfing problem? What the data suggests
One chef's-knife brand gets 31% of its "what should I buy" mentions from 5% of the accounts, four times what chance predicts.
@spots
Source: Hacker News