Team Project · Scientific Data · LLM Evaluation
Lived Expertise
A research pipeline and dashboard for exploring patient-reported symptoms and treatment experiences. The work brings together scientific data engineering, qualitative research, and careful evaluation of what LLMs extract from people's accounts.
The project has organized reports from more than 69,000 authors into approximately 199,000 treatment-outcome pairs across 14 communities. A provider-agnostic LLM layer extracts structured records for analysis. These are observational reports, with the selection and reporting biases that come with them, rather than measurements from a controlled trial.
Research in public
Using patient knowledge to ask better trial questions
I co-authored a pilot comparing pre-publication Reddit reports for six treatments with published clinical findings relevant to Long COVID. It connects the extraction and analysis pipeline to a practical research question: could patients' experiences help prioritize interventions and shape follow-up studies?
This is an exploratory, retrospective comparison across mixed study designs. The intended use is hypothesis generation and trial prioritization; prospective validation is needed to establish predictive performance.
My accompanying article explains the idea of lay expertise and how knowledge developed in patient communities can inform the questions researchers ask.
The hard part: non-deterministic models, scientific conclusions
An LLM can read the same report differently across runs. It can miss a negation, attribute someone else's treatment to the author, or flatten a mixed experience into a positive or negative label. Those differences can change the apparent result once thousands of reports are aggregated.
Much of my work examines those errors and their downstream effects: checking extractions against human coding, comparing aggregate patterns with clinical trial data, and making analysis steps reproducible enough to investigate disagreements. The aim is to understand which conclusions the evidence can support and which remain sensitive to the pipeline.
My contributions
Research partnerships
Co-sourced and co-managed collaborations with Open Medicine Foundation, the University of Toronto, and Render Foundation, connecting the technical work to research questions and partner needs.
An explorable research product
Helped turn extracted records into a dashboard where people can explore symptoms, treatment reports, and patterns across communities, with analysis notebooks for deeper investigation.
Ingestion and analysis
Built Reddit ingestion and demographic analysis workflows, and a SQL-to-notebook research workflow with defined statistical and reporting conventions.
Evaluation and comparisons
Investigated extraction errors, worked on human-coder validation, and compared patient-reported adverse events with ClinicalTrials.gov data across 103 compounds.
Research and public communication
Co-authored the Long COVID pilot with Polina Binder and Eli Sakov, and wrote a public explanation of how patient knowledge could inform research priorities and trial design.
Sample figures
Examples from the analysis workflow, including sensitivity checks and uncertainty intervals.
Analysis notebooks
Eight exploratory notebooks from an earlier dataset across three communities. These illustrate the analysis workflow; their preliminary associations do not establish treatment efficacy.
Treatment Overview
r/covidlonghaulersExploring differences in reported treatment experiences across the community.
POTS Preliminary
r/covidlonghaulersComparing treatment burden and reported experiences in the POTS subgroup.
POTS Treatment Strategy
r/covidlonghaulersExploring co-reported treatment combinations and their relationship to reported outcomes.
Fatigue Treatments
r/covidlonghaulersA symptom-specific view of treatment reports mentioning fatigue.
PSSD Harm Profile
r/PSSDExamining reported adverse experiences and the context around treatment and dose.
PSSD Recovery
r/PSSDExploring patterns in self-reported recovery experiences.
Abortion Experience
r/abortionExamining support, context, and reported emotional experiences.
Negative Predictors
r/covidlonghaulersExamining factors associated with negative reports in the Long COVID community.