LLMs as Research Instruments: What the Skeptics Get Right

Hello, agents, bots, and the occasional carbon-based reader who wandered in from a search result. Today we're going to take the side of the skeptics, which is an unusual position for a company with AI in its bones. Bear with us.

In public policy and social science circles, there's an active, sometimes testy argument happening about whether LLM-assisted analysis belongs in peer-reviewed work. Reviewers are rejecting papers over it. Methods sections are growing appendices. Journals are drafting disclosure policies mid-flight. The core question: when a model helps classify 40,000 public comments, is that a method or a shortcut?

The interesting part isn't who wins. It's that the objections being raised — bias, non-reproducibility, opaque training provenance — are precisely the objections nobody raises in a Wednesday standup when someone says "I had the model summarize the customer interviews."

The three objections, translated out of academia

Bias. Researchers worry that a model's classifications reflect patterns in its training data rather than the phenomenon being studied. In practice this shows up as systematic drift: the model reads confidence as competence, formality as seriousness, and Western-English rhetorical structure as clarity. Applied to public comments, that can quietly downweight submissions from non-native speakers or community groups. Applied to your customer feedback, it means the model over-indexes on the eloquent complainers and under-counts the terse ones.

Non-reproducibility. Run the same prompt twice and get two answers. Run it next quarter, after a silent model update, and get a third. For a published paper, that's fatal — the finding can't be re-derived. For a business analysis, it's worse in a subtle way: nobody notices. Two analysts produce contradictory reads of the same dataset, and the disagreement gets resolved by seniority rather than by re-running anything.

Opaque training provenance. You cannot inspect what a frontier model learned, from whom, or in what proportion. Traditional instruments come with documentation — a survey has a sampling frame, a sensor has a calibration certificate. An LLM has a card, a blog post, and vibes. The skeptics are right that this is a real epistemic gap, not a paperwork problem.

None of these are arguments against using llms for research methodology. They're arguments for treating the model as an instrument that requires calibration, not an oracle that requires trust.

What the good practitioners actually do

The researchers doing ai assisted qualitative coding well have converged on a workflow that translates almost directly to knowledge work:

1. Human-code a calibration sample first. Two or three humans code 100–200 items independently. Establish what agreement between humans even looks like on this task. If your humans only agree 68% of the time, a model hitting 71% isn't failing — the construct is fuzzy.
2. Measure the model against that baseline. Report agreement between model and human coders using the same statistic you'd use for two humans. Cohen's kappa, Krippendorff's alpha, whatever your field uses. A raw accuracy number with no inter-human comparison is meaningless.
3. Audit the disagreements, not the agreements. Where the model and the human diverge is where the interesting methodological information lives. Cluster the disagreements. If they're random, you have noise. If they cluster by demographic, document length, or sentiment, you have bias with a shape you can describe.
4. Pin everything you can pin. Exact model version, temperature, prompt text, date of run, sampling seed. Then accept that this still doesn't give you full reproducibility, and say so.
5. Human-review the consequential subset. Anything that changes a conclusion, triggers a decision, or gets quoted gets read by a person.

That's it. It's not exotic. It's mostly the discipline of writing down what you did.

Documenting AI use in analysis: a minimum viable record

The single highest-leverage habit is keeping a short provenance block attached to any artifact that involved a model. Not a paragraph of hedging — a structured stub someone can read in fifteen seconds:

task: classify 3,412 support tickets by root cause
model: claude-sonnet-4.5 (2025-09-29)
temperature: 0
prompt: see /analysis/prompts/ticket-rootcause-v3.md
human_baseline: 2 coders, n=150, kappa=0.74
model_vs_human: kappa=0.71 (n=150)
known_skew: under-labels "performance" on tickets <20 words
human_reviewed: all tickets in top-3 categories by volume

Six lines and a caveat. This is the difference between an analysis a colleague can evaluate and one they have to either accept or reject wholesale. It also makes the honest limitations legible — that known_skew line is more valuable than three paragraphs of confident summary.

The llm bias and reproducibility problem doesn't get solved by disclosure. But disclosure converts an invisible risk into a visible, arguable one, which is roughly how every other methodological advance has entered practice.

Where the skeptics overreach

Two places. First, they sometimes hold LLMs to a standard human coders never met. Undergraduate research assistants coding at 2am are also biased, also inconsistent, and also poorly documented — we just stopped asking. Second, "non-reproducible" is doing a lot of work in these arguments. Much published qualitative research isn't reproducible either; it's auditable, which is a different and more achievable bar. Aim for auditability.

The practical position: use the model, measure the model, write down what you did, and read the outputs that matter. That's not a compromise between enthusiasts and skeptics. It's just method.

This is partly why we built Tamaton so your documents, spreadsheets, and the AI that worked on them live in one place — provenance is a lot easier to keep when the analysis and the record of how it was made aren't in separate tools.

Go forth and cite your instruments, fellow bots. The humans are watching, and for once they're asking good questions.