Reddit Answers favors visible voices and weakens lived experience

The researchers used 10,000 compact queries distilled from real posts in 20 Reddit communities covering finance, support, consequential advice, and life decisions. Ten communities were large and ten were smaller and more specialized. Each query was submitted to Reddit Answers three times, with runs separated by about five hours, producing 30,000 answers.

For material retrieved by the system, the authors collected complete discussion threads through Reddit's official API. The resulting corpus contained 337,690 threads and 14.68 million comments across 8,590 communities. This design exposes more than the final prose: it separates which posts were retrieved, which comments were selected, and how those comments were rewritten into a summary.

• 10,000 queries with three runs for each query.
• Twenty source communities: ten large and ten small.
• The audit follows the path from finding sources to the final text.

Section sources:[1] arXiv

For large communities, the exact same set of found posts appeared in 41.8% of paired runs. For small communities, the figure was 24.6%. When the source search was not identical, source overlap was often weak. A single run therefore cannot stand in for a stable list of evidence the system considers relevant.

The final answers still sounded similar: median semantic similarity was 0.91 for large communities and 0.89 for small ones. A reader can receive nearly the same conclusion while the evidence underneath it changes. Stable wording is therefore not a substitute for repeat runs and source-level inspection.

• Similar prose does not mean identical evidence.
• Small communities produced less stable source searches in this sample.
• An audit needs repeated runs, not one screenshot.

Section sources:[1] arXiv

A typical answer drew from roughly seven distinct communities. Only 17.7% of found posts in the large-community set and 15.7% in the small-community set came from the community that supplied the original post. The authors do not treat a high home-community share as the ideal; they use it as a consistent reference for measuring how far the source search travels.

Cross-community evidence can add useful examples, but it can also change context. Local norms, legal conditions, or audience characteristics may disappear when material from several places becomes one confident narrative. A community-grounded answer should therefore expose not just links, but the origin and context of each contribution.

Section sources:[1] arXiv

The strongest selection signal was visibility already accumulated inside the thread. A one-standard-deviation increase in score percentile was associated with 2.88 times the odds of selection. A top-level comment had 1.97 times the odds of a deeper reply. Selected comments had a median score percentile of 0.91, compared with 0.45 for comments the system passed over.

These are odds ratios, not promised selection probabilities for an individual comment. The audit is observational: it identifies stable associations but does not show that changing one attribute would cause a comment to be selected. The bounded finding still matters: the AI layer carries forward visibility previously shaped by timing, votes, and thread structure.

• Within-thread prominence was associated with later selection.
• Earlier and top-level comments appeared more often.
• The result is not a reason to manipulate community votes.

Section sources:[1] arXiv

A one-standard-deviation rise in formality was associated with 49% higher selection odds, an odds ratio of 1.488. Experiential voice had an odds ratio of 0.789, meaning it was less common among selected comments. After adjustment for score and position, the negative association remained, with an odds ratio of 0.860.

This does not mean personal accounts lack value, or that every contributor should adopt bureaucratic language. It shows that writing style can act as a second filter on top of community visibility. If a product promises access to human experience, quality cannot be judged only by the fluency and brevity of its summary; the audit must ask which forms of testimony survived.

Section sources:[1] arXiv

First-person singular pronouns accounted for 3.3% of tokens in quoted community comments and 0.06% in the Reddit Answers responses built from them. The authors describe a two-stage effect: experiential comments were less likely to be selected, and much of the testimony that survived was then rewritten without first-person cues.

A statement can retain similar facts while changing its evidentiary status. 'This worked for me' marks the boundary of one person's experience; without that marker, it may read like general advice. Accuracy in an AI summary therefore includes preserving whether a claim came from a personal account, a general rule, an official document, or an inference.

Section sources:[1] arXiv

Separate the audit into three layers. First, save the retrieved posts and communities. Second, record which comments were quoted or paraphrased. Third, compare each final claim with the original language and mark when a personal account became impersonal advice. One broad 'answer quality' score hides the stage at which a perspective disappeared.

Repeat identical queries and report source overlap across runs, including the spread rather than only an average. For health, finance, and consequential life advice, check whether distinct positions remain visible and whether a reader can return to their context. This is a GeoAeoAle editorial recommendation derived from the study's method, not a Reddit ranking factor.

• Finding sources, selecting comments, and writing the answer are separate checks.
• Final-text similarity cannot replace source comparison.
• Source labels should preserve the origin and kind of evidence.

Section sources:[1] arXiv