Research Engineering · Open-Source Contributor
Naturalv2
My contributions to an open-source research pipeline: adapting it to Long COVID, evaluating trial eligibility with token log probabilities, and improving extraction and statistical analysis.
View upstream repository ↗Adapting the pipeline to Long COVID
I adapted the pipeline to Long COVID reports and trial-specific questions, including fatigue and brain fog, to compare extracted treatment and outcome information with clinical trial results.
That meant working through eligibility criteria, treatment names, outcome definitions, and evidence cutoffs. I also assembled a reproducible lithium-trial analysis with frozen inputs and model configurations so changes in extraction and weighting could be compared on the same reports.
The question is how much the conclusion depends on the decisions inside the pipeline: which reports count, whose experience is being described, and how those observations are weighted.
Inside the model
Working with token-level log probabilities
I worked with the pipeline's logprob-based scoring to examine whether a patient report matches a trial's inclusion criteria. Using Qwen through vLLM, this approach scores candidate answers from their token log probabilities and normalizes them into relative probabilities.
In the Long COVID analysis, I compared this conditional scoring with repeated sampled eligibility decisions, then examined how inclusion weighting affected the outcome estimates. Keeping the inputs fixed made it possible to investigate the effect of the scoring method itself.
I use these model-derived scores to test the stability of the analysis: how much does the scientific result change when the scoring and inclusion decisions change? Their clinical interpretation requires separate calibration and validation.
- 01Patient report + trial criteria
- 02Score candidate eligibility answers
- 03Compare weighted outcome estimates
Engineering improvements merged upstream
Attribute a treatment to the person who actually used it
A comment can discuss someone else's treatment or quote a parent post. I added an ownership check that keeps first-person treatment experiences and rejects ambiguous attribution.
Merged PR #55 ↗Use the author as the unit of analysis
I aggregated curated reports by author, removed duplicate permalinks, and separated an author's text from conversation context. Multiple posts from one person should not silently become multiple independent patients.
Merged PR #61 ↗Correct how treatment arms are weighted
I fixed inverse-propensity weight normalization in NaturalMC so weights are normalized within each observed treatment arm. Pooling that denominator across arms had diluted the estimates.
Merged PR #60 ↗Make unsupported outcome types explicit
I added a guard against passing continuous outcomes through a conditional extraction path intended for discrete values, making an invalid configuration fail clearly.
Merged PR #57 ↗