qlm_code() now validates every structured response
against the codebook schema before ellmer converts it to a row, on every
provider and both paths. A response that does not conform, whether a
required field sent as null or left out, a number sent as a string, a
value outside its enum, an array sent as an object, or an undeclared
property, is now a failed unit: its row is NA, its
.error names the offending JSON path, and
qlm_failures() lists it for qlm_backfill() to
re-code. Previously such a response was converted regardless, so a
missing scalar became a silent NA with no
.error, an extra property was dropped, and a missing
required array became the same empty cell as a valid empty one; the
“every required field is NA” heuristic caught only the
wholesale case, and only for scalar fields. The whole-run fallback of
structured = "auto" to JSON mode is now driven by the
validator: it fires when every response the provider completed fails
validation, and not when some do; where no fallback is possible, under
structured = "structured", batch = TRUE or a
file input, such a run is returned with every unit failed and its reason
recorded, rather than stopped. The fallback re-codes only the units JSON
mode can help: a response cut off at the output limit, or an input
rejected as longer than the context window, keeps the row, reason and
usage the structured attempt gave it. The usage of a structured attempt
the run fell back from is carried into the JSON-mode result, and the
usage of a request whose outcome is unknown, a timeout or a server
error, is recorded as NA rather than zero, so a total that
includes one is unknown rather than understated; zero is recorded only
for a request never sent or refused before generation. The finish reason
is read from every structured response, so a response cut off at
max_tokens, withheld by a content filter or finished for an
unrecognised reason is recorded with that reason whether or not it
parses, on any provider and without a declared limit; the inference from
token counts, and the params(max_tokens = ) it needed, are
gone. A codebook whose schema has a type_array() root or an
opaque type such as type_from_schema() is refused before a
request is sent, since neither can be validated or tabulated. The
enforcement note for unverified endpoints, and the
quallmer.quiet_schema_note option that silenced it, are
gone: there is nothing to warn about now that every response is checked.
A run validated this way records validation = "local" in
its metadata (#140).
The .id column of a qlm_coded object
must now be a key: unique and never missing. Every later operation
merges on .id, and qlm_compare() and
qlm_validate() silently formed a Cartesian product of
repeated rows, computing their statistics over the wrong pairs, while a
missing identifier was matched to every other missing one; an unnested
table that keeps the document identifier rather than a document-item key
is how the first arises in practice. Both are now errors, naming the
offending values: at construction, in qlm_code() before a
request is spent, when rows are subset with [, and by every
function that takes a qlm_coded object, which now checks
its integrity first: class, run metadata, and exactly one
.id column that is a key. That last check is what catches
objects that dplyr or base row operations have altered
(slice(x, c(1, 1)), rbind(x, x)), which keep
the class and attributes, and objects saved before the check existed.
as_qlm_coded() also refuses an id column
alongside an existing .id, which left two columns of that
name with the wrong one read (#156).
qlm_code() now rejects convert = FALSE
with an explanation. It has never worked: ellmer returns a
bare list, which has no rows to carry an .id and no columns
to reorder, and the call failed later with
incorrect number of dimensions (#134).
qlm_code() now runs with
on_error = "continue" by default, on the structured path as
it already did on the JSON path, and gains on_error as a
documented argument taking ellmer’s three values. A parallel run
therefore attempts every unit and leaves the failures for
qlm_failures() and qlm_backfill(), rather than
stopping at the first and returning the rest as NA rows
with no .error, which for a codebook of arrays or nested
objects could not be told from valid empty answers. Stopping early saved
money when the only remedy was to run again from the start; with
backfilling it costs more than it saves. The setting is recorded with
the run, so a run coded before this change records none and is
replicated and backfilled with the new default. on_error
cannot be set with batch = TRUE, which has no equivalent,
and qlm_replicate() no longer carries a parallel-only or
batch-only argument into a replication that switches path
(#171).
qlm_code() gains a tools argument for
registering ellmer tools, such as a provider’s hosted
web-search tool, on the chat before coding. Registered tools are
recorded on the object, disclosed by print() and
qlm_trail(), carried to backfill passes and to a
replication on the same endpoint, kept in the trail by name, type,
description and full configuration rather than as objects, and refused
with batch = TRUE, which cannot send them. A custom tool
takes effect on the JSON path only, and a run with a hosted tool says in
its cost note that the tool’s calls are billed outside the token cost
(#122, @SeraphineM). Documentation: added the
“Coding with web-search tools” tutorial, which extracts figures from
Wikipedia country pages with and without a tool and shows why the tool
is what makes the extraction repeatable.
New qlm_transcribe() transcribes audio recordings,
in parallel, from local files or URLs: through the transcription
endpoint of openai/ (gpt-4o-mini-transcribe,
gpt-4o-transcribe, whisper-1) or of any
provider registered with qlm_register_provider(), or
through a chat model that hears the recording, such as a Gemini model.
No list of models is kept; a model that cannot transcribe is refused by
its provider, at no cost. It returns a qlm_transcript, a
named character vector whose provenance table records each recording’s
hash, the model, language, prompt, timestamp and reported usage.
qlm_code() codes it as text on any provider and records
that provenance with the run, qlm_trail() reports it, and
qlm_replicate() and qlm_backfill() code the
stored transcripts again without another transcription request. A unit
whose text is NA, such as a failed transcription, is never
sent to the model and is recorded as failed with the reason
(#178).
qlm_code() and qlm_segment() accept
registered OpenAI-compatible provider prefixes.
qlm_register_provider() adds session-specific endpoints;
replication and backfill retain the recorded endpoint (#145).
qlm_codebook() accepts
input_type = "audio", and qlm_code() codes
recordings in one pass: each file is uploaded to the provider through
ellmer’s file upload and the model receives a reference to it with the
codebook, so the schema can ask for a transcript alongside any coding of
the content. Which providers accept audio is not checked in advance: a
provider that cannot take it refuses with its own message, to which
qlm_code() adds what is known, as of this version that only
Google Gemini does. Every upload completes before the first request is
sent, so a failed upload stops the run with the provider’s message and
nothing spent. The run records the SHA-256 of each file, which
qlm_replicate() and qlm_backfill() check
before uploading again and qlm_trail() reports.
batch = TRUE is refused for audio, since an upload gets a
new reference every time and ellmer’s prompt-keyed batch cache could not
resume the job, and the cost note says that an audio cost computed at
the text rate is potentially underestimated. Requires ellmer 0.5.0
(#124).
qlm_codebook() accepts
input_type = "video", and qlm_code() codes
picture and sound in one pass. Each element of x may be the
path of a local video file, which is uploaded; a YouTube link, which the
provider fetches itself; or the URL of a video file, which is downloaded
and then uploaded like a file. Before uploading, qlm_code()
says how much video it is about to send, with a token estimate when the
av package is installed, and refuses a file over the upload’s 2 GB
limit. The run records the size and SHA-256 of every file, a downloaded
URL included, and the URL alone for a YouTube link;
qlm_replicate() and qlm_backfill() download
and upload again after checking the hashes. As of this version only
Google Gemini accepts video; any other provider is tried and its refusal
reported with that note (#179).
A file input (image or audio codebook) no longer falls back to
JSON mode under structured = "auto": that handler sends
text, so a failed structured call used to end in the misleading error
that the model “supports text codebooks only”. The provider’s own error
is now reported, and structured = "json" is refused up
front for a file input. qlm_code() also checks that every
image, audio or video file exists before building a request
(#124).
qlm_codebook() gains image_file_resize,
which sets how an image codebook’s files are resized before they are
sent: "high" (the new default, fitting within 2000x768 or
768x2000 pixels), "low" (512x512), "none", or
a magick geometry string such as "1024x1024>". Until now
every image was sent at ellmer’s default of 512x512, with nothing in
quallmer saying so, which is a thumbnail for a poster whose small print
is what the codebook asks about. The setting lives on the codebook
because the resolution is part of the measurement:
qlm_replicate() and qlm_backfill() inherit it,
qlm_trail() reports it, and a codebook saved before the
field existed is read as "low", what it was coded at, so
replicating an old run still measures what the original measured.
magick, which the resizing needs, is now in Suggests, and
qlm_code() says so if it is missing. x may now
also hold image URLs alongside paths: an element beginning with
http://, https:// or data: is
passed to the provider as it is, and every path is checked to exist
before any request is sent. A second codebook field,
image_url_detail, asks the provider for "low"
or "high" detail on such URLs; OpenAI and OpenAI-compatible
providers read it, and ellmer forwards it from the version that includes
tidyverse/ellmer#1133, so qlm_code() says before the run
when a value cannot take effect (#177).
New qlm_backfill() re-codes only the units a run
failed on and merges the results into the original object, instead of
re-running the whole corpus to recover a handful of transient failures.
Which units to retry is decided afresh on each pass by the test
qlm_failures() uses, so a request rejected on length drops
out as soon as a pass records it, and a response cut off at
max_tokens is retried only when the backfill raises the
limit through params; content refusals are retried, because
they are not deterministic. A pass that recovers nothing ends the
backfill early. Passes run with the run’s own model and settings by
default, on the path the run actually took. A different
model may be given, for units the original model
consistently refuses or cannot fit in its context window, and the result
then records which units came from which model and says so when printed,
since a coding by two instruments has to be disclosed as one. Rows keep
their order, a failed retry never overwrites anything, and token and
cost columns are summed across attempts, a total staying NA
when any attempt’s figure is unknown, as it becomes for the units a pass
that failed outright attempted, since the pass may have been billed. A
pass costed on other rates than the run records them, and
print() and qlm_trail() say so beside the
run’s own cost note. qlm_code() gains backfill
to complete a run in the same call, and qlm_replicate()
gains backfill, which by default replays the passes
recorded on the parent so that a replication of a completed run is
complete on the same terms. qlm_trail() reports the passes
and any other model they used, so a composite is disclosed in the audit
trail as well as when printed (#136).
New qlm_failures() lists the units a coding run
failed on, with the reason for each, and print() of a
qlm_coded object now reports
Units: 251 (211 scored, 40 failed) rather than the number
attempted, so a partly failed run cannot look complete. The object
already carried this in its .error column, but nothing
surfaced it, and the check people write for themselves is wrong for
array-valued properties: a failed request leaves a zero-row tibble in
the list-column, not NA, so !is.na() reports
every failed unit as coded. A unit counts as failed when it carries an
.error or when every required scalar property is
NA, the latter because an endpoint can accept a JSON schema
and ignore it, returning HTTP 200 and nothing usable. Arrays and nested
objects are not consulted, since after conversion a missing array and a
valid empty one are the same cell. For that to be enough,
qlm_code() now also records an .error for a
response ellmer could extract no structured data from (a refusal in
prose, say): ellmer reports those only by warning and leaves the row
with no .error, which for an array-only schema is
indistinguishable from a valid empty answer. print() also
distinguishes rows present from units attempted after subsetting
(#132).
qlm_compare() gains a
by_category = FALSE argument that, when set to
TRUE, reports per-category reliability rows for nominal
data: Krippendorff’s alpha (alpha_per_value[k], each
category dichotomised against all others), kappa
(kappa_per_value[k], Cohen’s κ via
dichotomise-and-recompute for two raters or Fleiss’ Eqs. 20-21 for three
or more), and alpha_u_per_value[k] for unitizing
comparisons. The marginal count n is reported in the
docid column. Per-category rows are only produced for
nominal-level data (#112).
qlm_code() gains a structured argument
controlling how the output schema is obtained, generalising the
local-validation path added in #128 beyond DeepSeek.
"structured" trusts the provider; "json" puts
the schema in the system prompt and validates every response against
codebook$schema locally; "auto" (the default)
attempts the structured call and falls back to "json" when
it fails. This matters because ellmer sends
response_format = {type: "json_schema", strict: true} to
every OpenAI-compatible provider and takes the result on trust, and
measurement shows several do not honour it — Kimi violated a schema it
was given on 2 of 3 identical requests through one gateway.
Non-conformance arrives as NA, indistinguishable from
missing data, so qlm_code() now also emits a one-time note
when coding against an endpoint whose enforcement it cannot verify,
silenced with options(quallmer.quiet_schema_note = TRUE).
Whether an endpoint is trusted is derived from ellmer’s own request path
rather than a list of vendors, so a provider added to ellmer later
defaults to unverified. Failure is detected both from an error and from
a result in which every required field is NA in every row,
which is what an endpoint that accepted the schema and ignored it
produces. That check reads required scalar properties, since required
arrays and nested objects become list-columns in which a missing value
and a schema-valid empty one are indistinguishable – so for a codebook
whose required properties are all arrays or nested objects,
"auto" on an unverified endpoint validates locally from the
start rather than making a call it could not check, and reports why
(#134).
qlm_code() can now code with DeepSeek, and no longer
trusts providers that accept a JSON Schema without enforcing it. The
DeepSeek API rejects the response_format that
ellmer::parallel_chat_structured() sends (“This
response_format type is unavailable now”), so every request failed; and
its JSON mode guarantees JSON syntax, not schema conformance, so simply
switching to JSON mode would trade a loud failure for a silent one.
ellmer converts every non-conformance – wrong field type, missing
required field, out-of-range enum value – to NA without
warning, so a run could come back looking plausible but wrong.
qlm_code() now routes model = "deepseek/..."
to a handler that requests JSON mode, puts the codebook schema in the
system prompt, validates each response locally against
codebook$schema, and re-prompts the model with the specific
validation error ($.claims[2].salience must be a number)
when a response does not conform. Repair attempts default to 2 and are
configurable with json_retries. Units that never validate
have NA coded values and a .error list-column
recording why, and token and cost accounting sums across repair
attempts. A document the provider rejects as too long is not re-sent,
but a content refusal is: refusals are not deterministic – the same
document is refused on one pass and coded on the next, at more than one
provider – and are rejected before generation, so a further attempt is
free. No other provider’s behaviour changes (#128).
qlm_code() gains prices, rates in US
dollars per million tokens that cost a run ellmer cannot price, from the
token counts it records. Only the rows ellmer leaves NA are
filled, by the sum ellmer applies to its own table; where ellmer prices
the model its figure stands. The rates are kept in the run’s metadata,
shown by print() and the trail report, and reused by
qlm_replicate() when the model, endpoint, batch setting and
service tier are unchanged, so a cost resting on entered figures is
always labelled as such. quallmer bundles no prices of its own
(#135).
qlm_codebook(levels = ) accepts variables nested
inside a type_array() or a nested
type_object(). The check matched names against top-level
schema properties only, so a codebook whose schema returns one array
entry per rated item could not declare measurement levels for the very
variables qlm_compare() and qlm_validate()
need them for, and the only workaround was to drop levels
from the codebook and re-attach them by hand after unnesting. Property
names are now collected at every depth, and for a schema whose root is a
type_array() rather than a type_object(),
which previously skipped the check altogether. A name that occurs at
more than one place in the schema is an error rather than being resolved
silently to the first match, since a flat levels list
cannot say which one it means (#131).
qlm_code() now says when a model name is not one its
provider lists, with the nearest names it does have, instead of
reporting only the provider’s HTTP error. The provider’s model list is
fetched through ellmer’s models_<provider>(), only
after a run has been rejected in its entirety and at most once per
failed run, and only for providers whose listing covers every name they
accept; where it cannot decide, the provider’s own error is reported
unchanged (#133).
qlm_code() now says why cost will be
NA when include_cost = TRUE cannot be
honoured, once and before the run, and keeps the reason with the object
so print() shows it. ellmer prices from a table fixed at
its release, matched exactly on provider and model, and answers
NA on any miss without saying which kind: DeepSeek and six
other providers are absent from the table altogether, so no model of
theirs is ever priced; a model newer than the installed ellmer is missed
on a provider it otherwise prices; and local endpoints have no per-token
charge. Each is now named, since the remedies differ, and the message
says whether the token counts a cost could be worked out from are being
recorded, which needs include_tokens = TRUE
(#135).
qlm_backfill() is no longer presented as part of the
coding step of the workflow. The README, the “Getting started” vignette,
and “The quallmer workflow” article now describe it in an optional
section on completing an incomplete run, and the reference index lists
it with qlm_failures() under a new “Completing incomplete
runs” heading (#190, @SeraphineM).
The “Audio transcription and analysis” example article now shows
both routes: transcription with Whisper followed by coding of the
transcripts, and coding the recordings directly with
input_type = "audio" on gemini-2.5-flash,
ending with qlm_compare() of the two runs, the model’s
transcript beside Whisper’s, the cost, and the recorded file hashes
(#124).
?qlm_code gains an “Incomplete runs” section: what a
failed unit looks like in the object, the layers at which it can be
tried again, what a backfill leaves alone and why, with a pointer to the
workflow guide’s section and its failure-to-mechanism table
(#174).
The package now ships a coded run that came back incomplete and
its backfilled counterpart, in
inst/extdata/example_objects.rds, coded once with a live
model and a deliberately short timeout and low max_tokens.
The workflow guide’s “Completing an incomplete run” section, the audit
trail tutorial, and the examples of qlm_failures() and
qlm_backfill() now show qlm_failures(),
print() and the trail’s Backfill: line on
those objects, evaluated, where before they described the output in
prose or sat in \dontrun{} (#173).
qlm_code() no longer injects the OpenAI-style
response_format field into every request on the JSON path,
including automatic fallback, which Anthropic rejected with HTTP 400.
The field now follows the transport ellmer uses: native OpenAI’s
Responses API gets text.format, every provider reached
through Chat Completions (DeepSeek, Groq, Mistral, Azure, OpenRouter,
Ollama, registered prefixes and openai_compatible/
endpoints) keeps response_format, and providers with
neither field, Anthropic among them, use prompted JSON with local
validation and repair (#191).
qlm_segment() now records token counts and cost when
asked, and takes prices as qlm_code() does. It
forwarded include_tokens and include_cost to
ellmer, but ellmer attaches usage only to a converted result that is a
data frame, and converts the array a segmentation asks for to a plain
list, so the counts were lost before quallmer saw them. The array is now
requested inside an object, which converts to one row per source
document with the usage beside it. Usage belongs to the document, not
the segment: it is kept in the corpus metadata as usage,
one row per input document including those that yielded no segments, and
repeated on each segment as docvars; sum the metadata table for the
run’s total. The four usage names are reserved when usage is requested
(#119).
qlm_code() no longer returns a response cut off at
the provider’s max_tokens limit as a successful empty
result. Such a response is billed in full, and arrived as a row of
NA scalars and zero-length arrays with no
.error, indistinguishable from a unit to which nothing
applied, so a document that overran the limit read as “nothing here”,
and the documents that do so are systematically the longest and richest
ones. On the JSON path the finish reason that ellmer 0.4.2 attaches to
each turn is now read: the unit is recorded in .error with
the token count, listed by qlm_failures(), and not retried,
since the same limit reproduces the cut and the retry is billed again.
On the structured path ellmer’s parallel call discards the turns, so the
finish reason is out of reach; there, when
params(max_tokens = ) is set, a row that used the whole
budget and returned nothing is recorded in .error as cut
off. Without a declared limit the cap is not known and such a row stays
silent, which needs an ellmer change to close. Rows whose request
failed, or whose response was cut off, are also no longer read as
evidence that an endpoint ignored the schema, so a run whose every unit
failed that way is reported as failed rather than re-coded in JSON mode;
a response ellmer could extract nothing from still counts, so
"auto" still falls back for an endpoint that answers in
prose (#153).
qlm_code() and qlm_segment() now reject
a model parameter passed at the top level. max_tokens = 100
used to fall through to chat_args and reach
ellmer::chat(), failing with
unused argument (max_tokens = 100) raised from inside
ellmer – naming the argument but neither of the two places it could have
gone. The error now names both: params for the settings
ellmer standardises, api_args for a provider’s raw request
fields. It deliberately does not prescribe one, because the choice is
not a property of the name: top_k is a real
ellmer::params() field, but ellmer maps it onto
top_logprobs for OpenAI-compatible providers, so a caller
wanting a provider’s raw top_k needs api_args.
stop and response_format are unambiguous and
do get a specific destination. The rejected set is read from
ellmer::params() at run time, minus whatever the request
path currently accepts, so a name a later ellmer gives a real meaning
stops being rejected without an update here (#139).
qlm_code() and qlm_segment() now say
what to do when a model’s provider prefix is not one ellmer can dispatch
on. model = "qwen/qwen3-max" reached
ellmer::chat() and failed with
Can't find provider ellmer::chat_qwen(), which names an
ellmer internal and offers no way forward. The error now names every
prefix that does work and points at
openai_compatible/<model> with base_url
and credentials, which is how any other OpenAI-compatible
endpoint is reached. The list of working prefixes is derived from the
installed ellmer at run time, mirroring both gates
ellmer::chat() applies, so a provider ellmer adds or drops
later needs no change here. Checked before any request is made, since
the answer does not depend on asking the provider anything
(#129).
qlm_code() no longer retries a request the provider
has rejected outright, and reports why it was rejected. A wrong model
name previously produced three rounds of retries and four warnings, none
of which mentioned the model; it now aborts once, quoting the provider’s
own message (which for DeepSeek lists the valid model names). Failures
with a fatal HTTP status (400, 401, 403, 404, 422) are not retried,
unlike rate limits and server errors, and the provider’s error body is
read directly off the response, since httr2 will only parse a body
served as JSON and several providers do not label theirs correctly. A
run in which every request is rejected as malformed or unauthorised
stops after the first pass rather than retrying; a refusal or an
over-long document does not count towards that, so a single failing unit
is still retried (#128).
The deprecated annotate() and
trail_record() run again. annotate() passed
its model_name to qlm_code() under that name,
whose argument is model, so the value fell through to the
provider call and every use failed, with model reported
missing or model_name reported unused.
annotate() again returns its identifier column as
id, as documented, rather than qlm_code()’s
.id, and trail_record() now always restores
the caller’s id_col values in place of the sequential ones
annotate() generates. Both functions still warn that
qlm_code() replaces them (#141).
Printing a qlm_coded object, or any other quallmer
object built on a tibble, in a session that had not yet loaded tibble
fell through to print.data.frame(), since tibble was in
Imports but not in the NAMESPACE and so loaded only on the first
namespaced call. For a run with an .error column that was
an error rather than plain output, since a recorded condition has no
format() method. tibble now loads with quallmer
(#173).
qlm_compare() and qlm_validate() ranked
ordinal text categories alphabetically, so a scale of low / medium /
high was ranked high < low < medium, and every statistic that uses
the order (ordinal alpha, weighted kappa, Kendall’s W, Spearman’s rho,
tau and MAE) was computed on the wrong ranks with nothing in the output
to show it; factor levels were flattened to text before they were read,
so ordering the categories in advance changed nothing. Text categories
at ordinal level are now ranked by the levels of an ordered factor,
which every coder must supply and all must agree on, and a plain factor
or character column is an error that says how to declare the order,
since alphabetical order is not a ranking and a plain factor’s levels
are alphabetical by default. A type_enum() declared
"ordinal" in qlm_codebook() is that
declaration: its values, in the order written, are the scale,
qlm_code() stores the column as an ordered factor with
those levels, as_qlm_coded() does the same for human-coded
data given the codebook (refusing a value outside the enum), and the
codebook print shows the order. An enum not declared ordinal stays
nominal. Because the ranks are now numbers, a tolerance on
ordered text counts rank distance, so tolerance = 1 means
adjacent categories agree; the warning that a tolerance on text
categories was being ignored is gone with the reason for it
(#165).
qlm_compare() now honours tolerance,
and computes its numeric statistics on the ratings’ values, when a coder
stores the ratings as text. The ratings were assembled into one matrix
before their type was examined, and a single character column made the
whole matrix character: every unit then fell through to exact text
equality with tolerance unused, and Krippendorff’s alpha,
the ICC and Pearson’s r ran on factor codes of the sorted strings, where
"10" falls between "1" and "2".
Neither showed in the output, so an LLM column parsed as text against a
numeric human column gave a flat agreement curve and a wrong alpha. At
ordinal, interval and ratio level every column is now read as numbers,
as the declared level asserts; a value that does not read as a number is
an error naming the coder and the values; ordinal categories given as
text are ranked as before, but a positive tolerance on them
draws a warning; and nominal categories agree only when identical, as
the code’s own comment already claimed (#150).
qlm_validate() ranked ordinal ratings by their
values sorted as strings, so on a 1 to 10 scale "10" fell
between "1" and "2" and Spearman’s rho,
Kendall’s tau and MAE were wrong whenever a rating reached 10, even from
numeric input. Ordinal and interval values are now read as numbers the
same way as in qlm_compare() (#150).
qlm_compare(level = "interval", tolerance = )
counted a pair differing by exactly tolerance as
disagreement or agreement depending on how the subtraction happened to
round in binary. 1.1 - 1.0 is
0.10000000000000009 and failed at
tolerance = 0.1, while 1.2 - 1.1 is
0.09999999999999987 and passed – the same nominal
difference, opposite answers. On decimal-increment scales this is common
rather than exotic; one reported analysis understated agreement by 23
percentage points, and the result stayed plausible enough that nothing
looked wrong. The comparison now allows a few units in the last place,
scaled to the magnitudes being compared. Percent agreement can therefore
only stay the same or rise: no pair that previously agreed becomes a
disagreement. Anyone who has reported a percent agreement computed on a
non-integer scale should recompute it. tolerance = 0 now
means numerical equality rather than bit identity, so
0.1 + 0.2 counts as equal to 0.3, while a
genuine difference of 1e-9 is still a difference. A
non-finite rating takes the plain comparison, since an infinite
magnitude would otherwise scale the allowance to infinity and report a
finite rating as agreeing with an infinite one (#121).
qlm_validate(..., average = "none") was reporting
per-class precision and recall swapped: the helper that derived FP and
FN from the confusion matrix had its row and column sums transposed
relative to the orientation produced by
yardstick::conf_mat(). Macro-averaged precision/recall
(computed via yardstick directly) were correct; only the
per-class breakdown was affected.
Backfill treats an endpoint change as a new coding route even when the model name is unchanged. Replication preserves the requested model and labels endpoint overrides in print and trail output (#185).
qlm_trail() no longer writes credentials into the
trail. An api_key, a credential-named
api_headers entry, or a base_url carrying
userinfo or a credential-named query parameter previously appeared
verbatim in the report’s Call section and in the saved
.rds. Their values are now replaced by
"<redacted>" in each run’s recorded call and chat
arguments, in the returned trail and in both files; computed expressions
that may contain those values are redacted wholesale. A
credentials callback is kept only as
function() Sys.getenv("NAME"), rebuilt without its
environment; any other callback is redacted too. Reading the key where
it is needed, through an environment variable or that callback form,
keeps it out of the record entirely and is the recommended form
(#154).
qlm_trail() reports which endpoint each run actually
used, and points at the right ellmer help page for setting it up. The
report derived a provider by splitting the model string and mapped it
through a four-name if/else chain, which had gone stale
three ways. The google branch could never fire, because
ellmer’s providers are google_gemini and
google_vertex and a google/ prefix does not
resolve at all. Every OpenAI-compatible endpoint – Qwen through Alibaba,
Kimi through Moonshot, a vLLM box, a laptop – reported the same provider
and the same instruction to configure credentials for
“openai_compatible”, which for an audit trail whose purpose is telling
runs apart was the worst of the three. And roughly twenty providers
ellmer ships fell through to “configure credentials as needed”. Runs are
now identified by provider and base_url, the rule
qlm_replicate() already applies, so endpoints differing
only by port, path or scheme stay distinct. Credentials embedded in a
URL, as userinfo or as a query parameter, are stripped from the endpoint
labels this section prints; note that the Call section still shows the
call as written and the .rds still preserves
chat_args whole. Ollama is told it needs no key only where
a loopback endpoint was recorded, since ellmer reads
OLLAMA_API_KEY for one served behind a proxy and resolves
an unset base_url through OLLAMA_BASE_URL,
which may be remote. The section is now “Provider and endpoint setup”
rather than “Configure API credentials”, because several providers use
IAM, OAuth or platform credentials rather than a key (#130).
qlm_trail() no longer advertises or generates a
top-level temperature argument. That form does not work –
it reaches ellmer::chat(), which has no such argument – so
the replication script the audit trail generated could not run, in the
one document meant to show that a run can be reproduced. The report now
reads the sampling settings a run actually recorded, in
chat_args$params, and emits them as
params = ellmer::params(). A legacy
chat_args$temperature, which older objects carry from the
routing that was never implemented, is folded into params
on read, so an old trail file still describes and reproduces itself in
the form that works; params$temperature is canonical and
wins when both are present. Values are serialised one at a time rather
than through unlist(), so a vector-valued parameter such as
stop = c("END", "STOP") survives into the generated call
(#127).
qlm_replicate() now reproduces the settings it
claims to. It restored only the execution arguments, silently dropping
everything the original run passed to ellmer::chat() –
params, api_args, base_url,
credentials – so a replication could run at a different
temperature than the run it replicated, and the resulting
qlm_compare() would read as a model-stability measurement
when it was partly a settings-difference measurement. Chat arguments are
now restored alongside execution arguments, with overrides in
... taking precedence. Provider-specific arguments such as
credentials and endpoint settings are restored only when the provider is
unchanged; when changing endpoint, an informational message names any
inherited arguments that were omitted and not explicitly replaced. An
endpoint is identified by both the provider prefix and
base_url, because every provider ellmer has no
chat_*() for is reached as
openai_compatible/<model> – so Qwen through Alibaba
Model Studio and Kimi through Moonshot share a prefix while being
different services with different credentials, and a prefix-only check
would send one vendor’s credential to the other. The model is still
passed as model, and registered tools are
never carried over (#125).
qlm_replicate() also reproduces the coding path and
json_retries of the original run. The path is derived from
the backend the run actually used rather than the mode it requested, so
a run that asked for structured = "auto" and fell back to
JSON mode replicates as "json": requesting
"auto" again would let an intermittently conforming
endpoint take the structured path instead, silently skipping the local
validation the original relied on and leaving the two runs incomparable
(#128, #134).
qlm_replicate() no longer carries the coding path
across a change of endpoint, provider or base_url. It
reproduced the path the parent took, structured or
json, which is right for the same endpoint and wrong for
another: a JSON-mode DeepSeek run replicated on OpenAI would skip
provider-side enforcement, a structured OpenAI run replicated on
DeepSeek would fail outright, and two OpenAI-compatible services behind
the same prefix enforce a schema quite differently. With a new endpoint
the path is now chosen as for a fresh run; json_retries
still carries, as before.
qlm_trail() no longer emits “unknown column”
warnings or crashes with the condition has length > 1
when passed a qlm_comparison or qlm_validation
object. The trail now stores these (and qlm_coded) objects
as-is rather than copying selected fields into a parallel structure, so
they round-trip with their class and metadata intact and can be
extracted from the trail for replication without modification
(#93).
All reliability statistics are now native R implementations,
derived directly from their source papers; the package no longer depends
on irr. Each function returns a uniform list shape
(method, value,
ci_lower/ci_upper, per_value,
n_observers, n_units, n_pairable)
plus measure-specific fields (#112):
reliability_alpha() — Krippendorff (2019, Ch. 12) for
predefined units; nominal/ordinal/interval/ratio metrics; per-category α
for nominal data; verified against book worked examples §12.3.1,
§12.3.4.1, §12.3.4.4.reliability_alpha_u() — Krippendorff’s α for unitizing
continua; one call returns all variants (value for
_u_α, binary for |_u_α,
cu_nominal for _cu_α, plus
per_value).reliability_kappa() — Cohen (1960) with unweighted,
linear, and quadratic weighted variants; analytic SE/CI for unweighted;
per-category κ via dichotomisation.reliability_kappa_fleiss() — Fleiss (1971) for many
raters with analytic SE and per-category κⱼ.reliability_kendall_w() — Kendall & Smith (1939)
with automatic tie correction; verified against Kendall & Gibbons
(1990) Ch. 6.reliability_icc() — all six ICC forms (Shrout &
Fleiss 1979; McGraw & Wong 1996); verified against Shrout &
Fleiss Table 4.qlm_compare() standardises on
subjects × raters matrix input internally, removing the
transpose step previously needed for
irr::kripp.alpha.
qlm_validate() no longer relies on
yardstick. Accuracy, MAE, RMSE, and the confusion matrix
are computed inline from base R; multi-class precision, recall, and
F-measure are now provided by internal metric_precision(),
metric_recall(), and metric_f_meas()
supporting all four standard estimators (binary,
macro, macro_weighted, micro).
Confusion matrix, micro and macro precision/recall follow Sokolova &
Lapalme (2009), Tables 1-3; macro F-measure is the arithmetic mean of
per-class F-scores (Manning, Raghavan & Schütze 2008, ch. 13),
matching the yardstick / scikit-learn convention. Output verified
identical to yardstick’s on both the binary case and a
4-class noisy multi-class example. yardstick removed from
Imports. # quallmer 0.4.0
New qlm_segment() segments a corpus into thematic or
conceptual units using an LLM, returning a quanteda corpus analogous to
quanteda::corpus_segment() output. Schema fields become
docvars; docid_ and segid_ track provenance.
Enables aspect-based sentiment analysis, thematic coding, and other
applications requiring variable-length segmentation (#96).
qlm_compare() now supports inter-coder reliability
for segmentation tasks. When all inputs are segmented corpora produced
by qlm_segment(), it automatically computes Krippendorff’s
alpha for unitizing (Krippendorff, 2019, section 12.6), an extension of
alpha designed for variable-length text segmentation. Three measures are
reported (marked experimental):
u_alpha_nominal and u_alpha_binary measure
joint boundary and coding reliability across the full segmented
continuum.cu_alpha_nominal measures coding reliability
conditional on unitization, isolating coding disagreement from
boundary disagreement.(k)u_alpha_nominal reports reliability and
coverage for each individual code, enabling diagnosis of which codes are
applied consistently. Results include both per-document and overall
(concatenated continuum) alpha.as_qlm_coded() gains qlm_segment and
source_text arguments for converting gold-standard data
frames to segmented corpora with character positions, enabling ICR
comparison of LLM segmentation against human-coded reference
data.
qlm_segment() now accepts a name
argument stored in corpus metadata for rater identification when
comparing multiple segmenters via qlm_compare().
dplyr and tidyr
(#109). Data manipulation now uses base R, vctrs, and
tibble, reducing the install footprint. No user-visible
behavior changes.\value documentation to all exported
methods.qlm_validate()
documentation.qlm_corpus wrapper
class pattern instead of conditional registerS3method(),
eliminating load-order dependencies and runtime checks (#86).qlm_meta() accessor function provides stratified
access to metadata for qlm_coded,
qlm_codebook, qlm_comparison, and
qlm_validation objects. Metadata is organized into three
types following the quanteda convention:
type = "user" (default): User-specified fields
(name, notes) that can be modified via
qlm_meta<-().type = "object": Read-only parameters set at creation
time (batch, call, chat_args,
execution_args, parent, n_units,
input_type).type = "system": Read-only environment information
(timestamp, ellmer_version,
quallmer_version, R_version).qlm_meta<-() replacement function allows
modifying user metadata fields only. Attempting to modify object or
system metadata produces an informative error (#72).codebook() extractor retrieves the codebook
component from qlm_coded, qlm_comparison, and
qlm_validation objects. This is a core component accessor
analogous to formula() for lm objects
(#72).inputs() extractor retrieves the original input
data (texts or image paths) from qlm_coded objects. The
function name mirrors the inputs argument in
qlm_code() (#72).attr(x, "run")$... access, providing a stable API for
extracting and modifying object metadata and components.as_qlm_coded() function replaces
qlm_humancoded() as the primary function for converting
human-coded or external data to qlm_coded objects. The new
function includes an is_gold parameter to mark gold
standard objects for automatic detection.as_qlm_coded() now supports quanteda corpus objects
directly via S3 method dispatch. Document variables (docvars) are
automatically converted to coded variables, with document names used as
identifiers by default. This simplifies the workflow for corpus-based
gold standards (#81).qlm_validate() now auto-detects gold standards marked
with as_qlm_coded(data, is_gold = TRUE), making the
gold = parameter optional when using marked objects.
Explicit gold = still works for backward
compatibility.qlm_validate() signature changed to
qlm_validate(..., gold, by, ...) to support validating
multiple coded objects against a single gold standard in one call.
Results include a rater column identifying each
object.qlm_humancoded() is now marked
@keywords internal but remains exported for backward
compatibility. New code should use as_qlm_coded().# Gold: Yes in their
print output for easy identification.qlm_validate() detect common
mistakes like forgetting gold = or misspelling parameter
names, with helpful suggestions for correction.ci parameter added to qlm_compare() and
qlm_validate() with options "none" (default),
"analytic", or "bootstrap".bootstrap_n parameter (default 1000).ci_lower and ci_upper
columns when ci != "none".qlm_compare() results now include rater1,
rater2, rater3, etc. columns containing the
names of compared objects (from name attribute), enabling
easy identification when combining multiple comparisons with
dplyr::bind_rows().qlm_validate() results now include a rater
column identifying which object is being validated, enabling easy
combining of multiple validations.qlm_comparison
and qlm_validation) instead of lists, making them easier to
filter, combine, and analyze.qlm_compare() or
qlm_validate() calls can be combined with
bind_rows() for analysis across multiple coders or
conditions.qlm_code() default name parameter changed
from "original" to NULL for cleaner output
when names aren’t specified.as_qlm_coded()
instead of qlm_humancoded().notes parameter in qlm_code(),
qlm_replicate(), and as_qlm_coded() for
documenting the rationale behind each coding run. Notes are displayed in
print output and captured in qlm_trail().qlm_trail() now accepts an optional path
argument. When provided, saves RDS archive and generates Quarto report
with full audit trail documentation.qlm_trail_save(),
qlm_trail_export(), qlm_trail_report(), and
qlm_archive(). Use
qlm_trail(..., path = "filename") instead.qlm_trail() now generates fallback names for objects
with missing name attribute.qlm_trail() function creates complete audit trails
following Lincoln and Guba’s (1985) concept for establishing
trustworthiness in qualitative research.qlm_trail(..., path = "filename") to save RDS
archive and generate Quarto report.qlm_comparison and qlm_validation
objects include run attributes capturing parent relationships, enabling
full workflow traceability.The package introduces a new qlm_*() API with richer
return objects and clearer terminology for qualitative researchers:
qlm_codebook() defines coding instructions, replacing
task() (#27).qlm_code() executes coding tasks and returns a tibble
with coded results and metadata as attributes, replacing
annotate() (#27). The returned qlm_coded
object prints as a tibble and can be used directly in data manipulation
workflows. Now includes name parameter for tracking runs
and hierarchical attribute structure with provenance support.qlm_compare() compares multiple qlm_coded
objects to assess inter-rater reliability. Automatically computes all
statistically appropriate measures from the irr package based on the
specified measurement level (nominal, ordinal, or interval).qlm_validate() validates a qlm_coded
object against a gold standard (human-coded reference data).
Automatically computes all statistically appropriate metrics based on
the specified measurement level, using measures from the yardstick, irr,
and stats packages. For nominal data, supports multiple averaging
methods (macro, micro, weighted, or per-class breakdown).qlm_replicate() re-executes coding with optional
overrides (model, codebook, parameters) while tracking provenance chain.
Enables systematic assessment of coding reliability and sensitivity to
model choices.The new API uses the qlm_ prefix to avoid namespace
conflicts (e.g., with ggplot2::annotate()) and follows the
convention of verbs for workflow actions, nouns for accessor
functions.
qlm_coded objects now use a hierarchical attribute
structure with a run list containing name,
batch, call, codebook,
chat_args, execution_args,
metadata, and parent fields. This structure
supports provenance tracking across replication chains and provides
clearer organization of coding metadata (#26).
batch flag indicates whether batch processing was
used.execution_args replaces pcs_args and
stores all non-chat execution arguments for both parallel and batch
processing. Old objects with pcs_args remain
compatible.data_codebook_sentiment provides a ready-to-use codebook
for sentiment analysis.task_*() functions are deprecated in
favor of using the data objects or creating custom codebooks with
qlm_codebook().task() is deprecated in favor of
qlm_codebook() (#27).annotate() is deprecated in favor of
qlm_code() (#27).validate() is superseded by qlm_compare()
(for inter-rater reliability) and qlm_validate() (for gold
standard validation). The function remains available but is marked with
a lifecycle badge.trail_settings(),
trail_record(), trail_compare(),
trail_matrix(), trail_icr()) are deprecated.
Use qlm_code() with model and temperature parameters
directly, or qlm_replicate() for systematic comparisons
across models.Backward compatibility: Old code continues to work
with deprecation warnings. New qlm_codebook objects work
with old annotate(), and old task objects work
with new qlm_code(). This is achieved through dual-class
inheritance where qlm_codebook inherits from both
"qlm_codebook" and "task".
validate_app() has been extracted into the companion
package quallmer.app. This
reduces dependencies in the core quallmer package (removing shiny,
bslib, and htmltools from Imports). Install quallmer.app separately for
interactive validation functionality.qlm_validate() now uses distinct, statistically
appropriate metrics for each measurement level:
level = "nominal"): accuracy,
precision, recall, F1-score, Cohen’s kappa (unweighted)level = "ordinal"):
Spearman’s rho, Kendall’s tau, MAE (mean absolute error)level = "interval"):
ICC (intraclass correlation), Pearson’s r, MAE, RMSE (root mean squared
error)The measure argument has been removed entirely - all
appropriate measures are now computed automatically based on the
level parameter. Function signature changed:
level now comes before average, and
average only applies to nominal (multiclass) data. Return
values renamed for consistency: spearman →
rho, kendall → tau,
pearson → r. Print output uses “levels”
terminology for ordinal data and “classes” for nominal data. This change
provides more statistically sound validation that respects the
mathematical properties of each measurement scale.
qlm_compare() now computes all statistically
appropriate measures for each measurement level:
level = "nominal"):
Krippendorff’s alpha (nominal), Cohen’s/Fleiss’ kappa, percent
agreementlevel = "ordinal"):
Krippendorff’s alpha (ordinal), weighted kappa (2 raters only),
Kendall’s W, Spearman’s rho, percent agreementlevel = "interval"):
Krippendorff’s alpha (interval), ICC (intraclass correlation), Pearson’s
r, percent agreementThe measure argument has been removed entirely - all
appropriate measures are now computed automatically and returned in the
result object. The return structure changed from a single value to a
list containing all computed measures for the specified level. Percent
agreement is now computed for all levels; for ordinal/interval/ratio
data, the tolerance parameter controls what counts as
agreement (e.g., tolerance = 1 means values within 1 unit
are considered in agreement).
New qlm_humancoded() function converts human-coded
data frames into qlm_humancoded objects (dual inheritance:
qlm_humancoded + qlm_coded), enabling full
provenance tracking for human coding alongside LLM results. Supports
custom metadata for coder information, training details, and coding
instructions (#43).
qlm_validate() and qlm_compare() now
accept plain data frames and automatically convert them to
qlm_humancoded objects with an informational message. Users
can call qlm_humancoded() directly to provide richer
metadata (coder names, instructions, etc.) or use plain data frames for
quick comparisons (#43).
qlm_validate() and qlm_compare() now
support non-standard evaluation (NSE) for the by argument,
allowing both by = sentiment (unquoted) and
by = "sentiment" (quoted) syntax. This provides a more
natural, tidyverse-style interface while maintaining backward
compatibility (#43).
Print method for qlm_coded objects now distinguishes
human from LLM coding, displaying “Source: Human coder” for
qlm_humancoded objects instead of model
information.
Improved error messages in qlm_compare() and
qlm_validate() now show which objects are missing the
requested variable and list available alternatives.
Adopt tidyverse-style error messaging via
cli::cli_abort() and cli::cli_warn()
throughout the package, replacing all stop(),
stopifnot(), and warning() calls with
structured, informative error messages.
Documentation and CI notes refreshed.