The drift report now surfaces per-column numeric spread
changes: a new “Numeric spread changes (min / max / sd)” table
shows each stat’s previous and current value with its shift, and
compare_snapshots() returns them as
spread_changes. (The values were already computed and
stored in column_snapshots every run; they were never read
back.) Three new comparison rules —
max_numeric_sd_shift_pct,
max_numeric_min_shift_pct,
max_numeric_max_shift_pct — breach-flag a shift, settable
at rules level or per column like their mean-shift sibling. They have
no default: unset, the shifts are reported but never
judged, so existing configs and reports gain information without gaining
new failure modes. Generated global configs list them commented out with
an example value. Internally, the drift side’s per-column threshold
resolution was generalised from a mean-shift-only helper to one
rule-keyed resolver (.column_comparison_overrides())
covering all four per-column comparison rules.
validate_config()’s unknown-key warning now
distinguishes a likely typo from a possibly-newer vocabulary key: a
near-miss still gets “Did you mean…”, while an unknown key resembling
nothing known states the installed dqcheckr version and that a newer
release may define it — so a config written against a newer dqcheckr
explains itself on an older one instead of reading as a
mistake.
A snapshot_db pointing at a directory (or any path
SQLite cannot open) now raises the typed dqcheckr_db_error
instead of leaking a raw RSQLite error. file.exists() is
TRUE for a directory, so such a path slipped past the
missing-file guard and failed opaquely at connect; connect-time failures
are now typed at the single connection helper, covering every database
path (read and write), with a directory-specific message.
compare_snapshots() now checks that the database has
a column_snapshots table alongside the existing
snapshots-table check, raising the same typed
dqcheckr_schema_error. A database carrying only
snapshots (a partial restore or export of run history)
passed every guard and then failed deep in the drift computation with a
raw untyped “no such table” error.
The database initialiser now refuses a foreign
column_snapshots table with the same typed
dqcheckr_schema_error it already raised for a foreign
snapshots table. A snapshot_db pointing at a
database that held another application’s column_snapshots
used to be adopted silently: dqcheckr’s snapshots table was
planted inside the foreign file and every subsequent run then failed to
record its snapshot (“Snapshot not recorded”), so the dataset could
never accumulate history and a database dqcheckr did not create had been
modified. The refusal rolls back inside the initialiser’s transaction,
leaving the foreign file exactly as it was.
col_threshold() no longer aborts with an untyped
subscript out of bounds error when a
column_rules entry is a bare value where a rule map belongs
(e.g. column_rules: {mycol: 5}). The same guard now also
covers malformed threshold values inside a valid rule map — a
quoted number ("0.5"), a vector, or a word crashed the
exported check functions with untyped base-R errors
(threshold * 100 on a character; if on a
length-2 condition) while the drift side silently ignored the identical
override. A value that is not a single finite numeric now simply does
not resolve at its level and the descent continues to the rules-level
threshold, then the default — so the run and drift surfaces agree. The
run path already rejected such configs via
validate_config(); this guards the exported check functions
when they are called directly. The table-level twin
table_threshold() (min_row_count,
max_row_count, max_file_size_mb) now applies
the same single-finite-numeric discipline: a malformed value falls
through to the default instead of crashing
check_min_row_count() untyped — where the pre-crash status
comparison was lexicographic (50 < "100" is FALSE), so
the crash was the only thing preventing a silently wrong
verdict.
sniff_dataset() no longer reports a single-quote
quote_char as “detected” on the strength of the file merely
containing no apostrophes. On an unquoted file with stray double-quotes
(inch marks: 5" nail), the default-quote pass fails by line
merge and the ' candidate used to win vacuously — writing a
config that asserts quoting the file does not use, which a later
delivery with an unclosed leading apostrophe would repay by row-merging
the rest of the file into one field. A non-default quote is now accepted
only when some sampled field actually starts and ends with it; otherwise
detection re-runs under the default quote with readr’s tokenizer
(field-start-only quote semantics, the same as the runtime
reader).
sniff_dataset() now detects the delimiter of an
all-fields-quoted CSV whose quoted free-text fields carry embedded
newlines (so a logical record spans several physical lines) – e.g. a
ServiceNow export. The per-physical-line field counter cannot see across
a multi-line quoted record and used to reject every delimiter,
mis-classifying the file as single-column; when the strict pass finds
nothing, a readr-based fallback re-detects the delimiter,
guarded so a lone stray delimiter in one row cannot fake a multi-column
file.
generate_dataset_config() can no longer emit a
config its own validate_config() cannot parse: a mapping
key derived from a column name is emitted in explicit-key block form
(? key / : value) once it would exceed
libyaml’s 1024-character inline-key limit, and .y_quote()
now escapes embedded newlines/tabs plus every other C0 control
character, DEL, and the C1 range as \xNN
(YAML’s printable set excludes them and libyaml refuses a document
carrying one raw — a stray vertical tab from an Excel cell break was
enough), so even a pathological (or mis-sniffed) column name yields
parseable YAML that round-trips to the delivery’s exact spelling. As a
structural backstop, both generators now parse the emitted YAML back
before renaming it into place: any future emitter gap aborts
typed (dqcheckr_generate_error) with no file written,
instead of leaving a created-but-unreadable config that the create-only
guard would then defend from regeneration. A failure of the write itself
is likewise typed (dqcheckr_write_error, the class already
used when the rename into place fails) rather than escaping as a raw
writeLines error.
infer_col_type() now recognises ISO-style date-time
strings ("%Y-%m-%d %H:%M:%S" and
"%Y-%m-%dT%H:%M:%S") as the date type, so a
timestamp column (e.g. a ServiceNow opened_at export)
classifies as date rather than falling through to
character. The new shapes are fully anchored, so they match
only a genuine <date><sep>HH:MM:SS value and
never overlap the date-only shapes. The GUI’s wizard-preview type
inference is kept in sync.
New vignette “The dqcheckr workflow: generate, edit,
validate, run” — a fully executed walk-through of the whole loop on
a bundled example: bootstrap a deployment with the two generators, read
the generated self-documenting YAML (including the duplicate-header
rename and the positional-list warning), edit a rule, validate
(including the did-you-mean typo path), run, list the history, compare
two deliveries, and the packed-FWF TODO and never-overwrite
special cases. Because it executes, R CMD check runs the
entire chain as a living test.
New validate_config(dataset_name, config_dir):
validation of the global and dataset YAML against the config vocabulary,
reporting every finding in one pass (severities error/warning/note)
instead of aborting on the first. The severity rule: config
mistakes are errors — wrong types/ranges, misplaced rule keys
(rules-level vs per-column), fwf_col_names/
fwf_widths length mismatches, duplicate output column
names, unresolved TODO width placeholders, a missing file
source — while delivery-facing findings are warnings,
because they can equally mean the supplier changed the delivery, and
drift must be recorded, not crash the run. Unknown keys get a
did-you-mean suggestion and warn, so hand-kept extra keys round-trip.
The validator never crashes on malformed values (any internal error
becomes an error-severity finding). An unreadable
dataset config aborts with distinct classes —
dqcheckr_missing_file, dqcheckr_empty_config,
dqcheckr_config_parse_error,
dqcheckr_invalid_config — while an empty or comments-only
global config is tolerated as all-defaults with a
warning (matching how the package has always run).
When the delivery file the config points at is resolvable, validation
additionally cross-checks the config against the file’s header
only (never the body — cheap even for multi-GB files on a network
share): col_names length vs the physical column count,
key_columns/
expected_columns/column_types/column_rules
naming columns that exist, and fwf_widths summing to the
record length — all warnings, per the severity rule. The CSV header is
parsed with readr (honouring the configured delimiter,
quote, and encoding), so a legal quoted field carrying an embedded
newline is counted as one column rather than mis-split by a line-based
read into a spurious column-count warning. When no delivery is
resolvable the header tier is skipped with the reason stated in the
result and by print() — a verdict always says which tier it
reached.
run_dq_check() validates first, before any other work:
error findings abort with a dqcheckr_validation_error whose
message lists every finding, and no snapshot row is
written — an unrunnable config can no longer leave a
pending orphan in the database. Warning findings are
printed and persisted into the run’s results as
VC-01 (“Config validation”) WARN checks — they land in the
report, the warn counts, and the overall status, so delivery drift the
runtime checks cannot see (e.g. a surplus column readr silently
auto-names) still leaves a recorded trace. Note-severity findings
(e.g. both current_file and folder set, where
current_file wins) are now surfaced in the run log too,
rather than being silently dropped on the run path. A corrupt dataset
config aborts the run with the typed classes above instead of the YAML
parser’s raw error. The description key the GUI wizard
writes is part of the config vocabulary (never read by the checks). The
two change-thresholds (max_numeric_mean_shift_pct,
max_row_count_change_pct) accept any value >= 0 — they
bound an unbounded relative change, and deployed configs legally carry
values above 1 — with a warning when a value looks like a raw
percentage.
CP-04 (numeric mean shift) now honours a per-column
max_numeric_mean_shift_pct set in
column_rules, which the GUI has always written but the
check silently ignored — the read goes through the same column >
rules > default resolution as every other per-column threshold. The
drift report applies the same per-column values, so the run report and
compare_snapshots() cannot contradict each other on one
snapshot pair. A malformed per-column value is now rejected up front by
validation — both run_dq_check() and
compare_snapshots() validate the config before doing any
work (see below) — rather than silently tolerated; the internal drift
resolver still warns and falls back to the rules level as a last-resort
safety net if ever reached with a raw value.
compare_snapshots() now validates the dataset
configuration before reading any thresholds — the same gate
run_dq_check() applies — so a malformed threshold (or any
error-severity config finding) is rejected with a
dqcheckr_validation_error on the drift path too, not
tolerated there while the run path aborts. Run and drift can no longer
disagree on what counts as a runnable config. A snapshot-only comparison
may still omit current_file/ folder: a drift
diffs snapshots already in the database and reads no delivery file, so
that run-only requirement is exempted and does not block it. Relatedly,
a corrupt config now surfaces the same typed
dqcheckr_config_parse_error on the
compare_snapshots() and list_runs() paths (and
load_config() generally) as run_dq_check()
already did, rather than leaking the YAML parser’s raw error.
Two more config mistakes now fail with a clear, typed message
instead of a confusing downstream error. An explicit
expected_columns: [] is an error-severity validation
finding that names the fix — an empty list is never a way to disable the
schema-contract check; omit the key instead — rather than being read as
“expect zero columns” and failing every column. And a directory given
where a current_file/previous_file is expected
aborts with dqcheckr_invalid_config pointing at
folder: mode, instead of failing deep in the CSV
reader.
New list_runs(dataset_name, config_dir, n):
name-based run history. Resolves the snapshot database from the
dataset’s merged configuration exactly as run_dq_check()
does and returns the recent-runs data frame from
read_recent_snapshots(), so the workflow can be driven end
to end by dataset names — the returned id column is what
compare_snapshots() takes to compare a specific pair of
runs.
Internal: a config vocabulary (R/vocabulary.R) now
records every YAML key the package consumes — scope, type, default,
positional flag, description — as the single source of truth the
upcoming validator checks against and the config generators emit from.
In support, the per-site default literals were consolidated into shared
constants (.default_read, .default_qc_rules,
joining the existing
.default_paths/.default_comparison_rules):
max_missing_rate’s 0.05 previously lived independently in
two files. No behaviour change.
New sniff_dataset(path): pure structure inference
over a delivery file — format (CSV vs fixed-width), encoding (the same
full-file streamed UTF-8 scan and single-byte fallback as
read_dataset()), delimiter/quote, header presence, column
names (duplicate header names renamed positionally with the originals
recorded), per-column types via infer_col_type() (one
implementation — the sniff cannot disagree with run-time
classification), and key-column candidates. Fixed-width boundaries come
from readr::fwf_empty(), with the widths anchored at
position 0 so a leading blank margin (or a right-aligned first column)
folds into column 1 instead of shifting the whole grid; a packed file
(no blank gutters) is reported with an explicit fwf_packed
marker rather than a wrong confident guess. The same explicit marker is
used for any fixed-width file that is not single-byte-per-character —
non-UTF-8 multibyte, UTF-16/32, and also valid multibyte UTF-8 (accented
text): readr::fwf_empty() returns byte offsets
while the sample is sliced by characters, and
readr::read_fwf() itself slices by bytes at run time, so a
confident guess for such a file could silently truncate its last column
(and no single width vector fits a file whose accented and plain rows
differ in byte length). Widths are left as a TODO for
review instead of being sniffed wrongly. The fixed-width gate itself
tests byte lengths as well as character lengths, so a genuine byte-fixed
multibyte file (uniform bytes, varying characters — the on-disk norm,
since readr::read_fwf() slices by bytes) reaches that
explicit TODO path instead of falling through to a
confidently wrong single-column CSV whose “header” is the first data
record; the minimum packed-record length is likewise measured in bytes.
No side effects; each headline field’s origin is recorded as
detected/default/generated. This is the detection half of the config
generator.
New
generate_dataset_config(path, config_dir, dataset_name):
sniffs a delivery and writes a fully-optioned, self-documenting YAML
config. Every config key appears exactly once — detected values live,
optional settings commented out with their defaults — and each key
carries the same description the validator’s vocabulary uses. Duplicate
and empty header names (a trailing delimiter) are renamed
positionally with # was "..." annotations and
csv_skip: 1, with rename suffixes bumped past any name
already in the header; positional lists are always emitted complete
under a do-not-comment warning; fixed-width files get a
character-position ruler comment, and a packed file gets explicit
TODO widths that validate_config() flags as an
error, so run_dq_check() refuses to run until they are
filled in. Emitted values are YAML-safe (backslashes escaped, map keys
quoted, paths in forward-slash form with relative paths kept relative);
columns whose sampled type is unknown stay unpinned. Create-only: an
existing config aborts with dqcheckr_config_exists and is
never touched (the never-overwrite rule). Both generators write
crash-safely — the YAML is emitted to a temp file and renamed into place
only once complete, so an interrupted generation can never leave a
truncated config that the create-only guard would then treat as “already
generated” (the accepted residual concurrent-generation window is
documented in the specification vignette, Accepted design
decisions).
New generate_global_config(config_dir): writes a
fully-optioned dqcheckr.yml — the shared infrastructure
paths set live to their built-in defaults, and a commented
default_rules block listing every rule key with its default
value and description (rules with no default show an example and a note
that unset keeps the check off). Change a threshold by uncommenting one
line. Create-only, like the dataset generator. A brand-new deployment
now bootstraps with exactly two generator calls and one
run_dq_check().
The per-column check loops (QC-07/08/09/10/11/13/15, SC-01/02,
and the comparison checks CP-04/05/06/07) no longer grow their result
list with c(results, list(...)) inside a for
loop, which reallocated the whole list on every column and made a run
O(columns^2). They now build results with lapply() and
compact once, so wide deliveries (100+ columns) build their results in
linear time. Check output is unchanged.
The missing/empty predicate (is.na(x) | x == "") and
the four comparison-check default thresholds are each now defined once
and shared by the QC checks, the comparison checks, the drift report,
and the snapshot writer, instead of being reimplemented at each call
site. This removes the risk of, for example, the QC report and the drift
report applying different default thresholds to the same rule after only
one copy was edited.
The shared Quarto render pipeline (render into a temp dir, verify
an output file was produced, abort with
dqcheckr_render_error if not, then move the result into
place) is now a single internal helper
(.quarto_render_to_file()) called by both the main report
writer and the drift report writer, instead of being duplicated
near-verbatim in each. A future change to the render pipeline now lands
in one place rather than needing to be mirrored by hand.
fwf_widths (must abort with
dqcheckr_invalid_config), and render_report()
when Quarto returns without writing an output file (must abort with
dqcheckr_render_error rather than name a report that does
not exist).QC-07 (check_numeric_stats()) no longer reports
Inf/NaN summary statistics.
as.numeric("Inf") is Inf, not NA,
so a literal infinity in a delivery classified the column as numeric
and survived the check’s is.na() filter — min/max
reported ±Inf, and with both signs present the mean came
back NaN. The aggregates are now taken over the finite
subset, and the number of excluded values is appended to
observed, so an infinity in a delivery is still visible
rather than silently dropped. (Snapshot statistics were never affected:
compute_col_stats() already filtered non-finite inputs and
serialised its aggregates through
.finite_or_na().)
Duplicate per-column statistics no longer multiply a column’s
rows in the drift report. column_snapshots carries no
uniqueness constraint (a constraint would turn a duplicate into a lost
snapshot row through the non-fatal write path), and custom checks may
emit several results for one column under one check_id — so
two rows for the same (column, statistic) are reachable on a clean run.
Both consumers join on the column name, and a join row-multiplies: the
column appeared twice in every per-column drift table (four times if
duplicated in both snapshots), breach flags included.
compare_snapshots() now collapses duplicates to the first
value and warns, naming what was collapsed. See Accepted design
decisions (AD-03) in the specification vignette.
A custom-check result carrying a non-character field (e.g. a
plain list built without dq_result(), with
observed = 42) no longer costs the run its entire snapshot
row. Such a result passed run_custom_checks()’s
field-presence validation but failed the snapshot writer’s typed
extraction much later, dropping the whole history row and all column
statistics under a misleading “SQLite write failed” warning — while the
report still rendered, so the loss was easy to miss.
run_custom_checks() now normalises every string field at
the boundary (length-1 atomic values are coerced with
as.character(), matching what dq_result()
produces; anything else aborts with
dqcheckr_invalid_custom_checks, naming the element and
field), and dq_result() itself now coerces
check_id, check_name, and column
the same way it already coerced observed and
message.
The CP-04 numeric-mean-shift comparison no longer aborts the
whole run when a numeric column contains a literal
Inf/-Inf value. as.numeric("Inf")
is Inf (not NA), so such a column still
classifies as numeric and reached the mean comparison; a non-finite mean
slipped past the guard and made the shift NaN, which
crashed the pipeline at if (shift_pct > threshold). The
check now filters to finite values first (as
check_outliers() and compute_col_stats()
already do), so an Inf-bearing column is handled as a
warning rather than a crash. R’s own write.csv() emits the
literal Inf, so this is a realistic corrupted-delivery
input.
compare_snapshots() now rejects a NULL
or empty dataset_name with a clear
dqcheckr_invalid_argument error instead of aborting with an
empty message. A NULL bound as SQL NULL
(matching no rows) and the “need 2 snapshots” abort then collapsed to a
zero-length, blank-text condition.
read_recent_snapshots() now clamps a negative
n to 0 rather than returning the entire
history. SQLite reads LIMIT -1 as “no limit”, so a negative
n silently returned every row instead of capping the result
as documented.
compare_snapshots() now returns the rendered drift
report’s path as a report_path element on its result
(NULL when no report was written), so a caller can link to
the report directly instead of reconstructing the filename from a slug
pattern – the reconstruction approach is fragile and broke when the
drift filename gained its snapshot ids.
A snapshot’s render_status now carries a
"pending" state while its report is still rendering. The
row is written as "pending" and only flipped to
"success" (with report_file) once the report
is confirmed on disk, or to "failed" if the render is
skipped or errors. Previously the row was written as
"success" with a NULL report_file before the
render finished, which a concurrent reader could not tell apart from a
completed pre-0.2.3 row and would turn into a broken report link.
Consumers linking to a report should treat a "pending" row
as not-yet-available rather than reconstructing a filename.
The drift report no longer silently reports a report that was
never written. When Quarto returned without error but produced no output
file, compare_snapshots() still announced (and, with
open_report = TRUE, tried to open) the intended path. The
drift writer now raises the same typed error the main report already
did, and compare_snapshots() downgrades it to a warning
with no report link – the computed drift is still returned.
Drift report filenames now include both compared snapshot ids
(drift_dataset_20260718_010203_1_2.html), so two
comparisons of one dataset started in the same wall-clock second (a loop
over id pairs, two GUI users) no longer collide on a single filename and
silently overwrite each other – the same protection
report_filename() already gives the main report.
Column statistics can no longer store a non-finite value even
when the aggregate itself overflows. An earlier fix excluded non-finite
inputs (Inf/-Inf from a corrupted
delivery), but sd() of finite-but-very-large values still
overflows the double range internally and returned Inf,
which was stored as the literal string "Inf" and poisoned
drift arithmetic. Every numeric aggregate is now serialised through a
finiteness guard (non-finite -> NA), and the drift
reader maps any non-finite value parsed from an older database to
NA as well, so a snapshot written before this fix cannot
re-poison a comparison.
Report filenames now include the snapshot id
(dataset_20260718_010203_47.html), so two runs of one
dataset that start in the same wall-clock second no longer collide on a
single filename. Previously the second run silently overwrote the first
run’s report while the first run’s snapshot kept pointing at it. The
filename is now written to the snapshot’s report_file
column by an update after the report is confirmed rendered, rather than
optimistically at insert time, so the column can never name a report
that was not written.
Configuration lookups no longer partial-match. R’s $
operator falls back to a prefix match when the exact element is absent,
so a config with no column_rules: key but a parked or
mistyped one (column_rules_disabled,
column_rules_old, …) had that section silently drive
per-column thresholds – producing wrong PASS/FAIL verdicts from a
section the config did not contain. Every config-key access now uses
exact [[ ]] indexing, so a parked or renamed section is
correctly treated as absent.
QC-16 no longer reports a spurious clean pass for a delivery
whose encoding it did not actually verify. A declared multi-byte or
unknown encoding (UTF-16LE, UTF-32,
Shift-JIS, GB18030, …) used to be reported as
“a single-byte encoding; every byte is valid by construction” – which is
false, and which also silently disabled the crash guard that scanning
gives UTF-8 deliveries. Such an encoding is now read as declared but
reported as a WARN stating it was not validity-checked. Genuine
single-byte encodings (ISO-8859-x, Windows-125x) still PASS.
The UTF-8 validity scan now streams the file in bounded chunks instead of reading it into one in-memory vector, so an arbitrarily large delivery is verified in flat memory rather than exhausting it on the network-share hosts dqcheckr deploys to.
Non-finite values in a numeric column
(Inf/-Inf, e.g. from a corrupted upstream
delivery whose CSV contains the literal text “Inf”) are no longer
mishandled. compute_col_stats() excludes them from the
mean, standard deviation, minimum, and maximum, so the snapshot database
can no longer store the literal string “NaN”/“Inf” and poison later
drift comparisons; the finite mean shift a contaminated delivery causes
is still reported as drift. check_outliers() (QC-15), which
previously aborted with an uninformative “missing value where TRUE/FALSE
needed” on such a column, now runs cleanly.
A zero-column delivery (for example an empty file) is now
recorded. Previously compute_col_stats() returned
NULL, and the snapshot write then failed and lost the
entire run; the run is now snapshotted with a column count of
zero.
Failures that were previously swallowed into silent, benign-looking results are now surfaced.
read_recent_snapshots() and
list_snapshots() no longer report a corrupt, locked, or
unreadable database as an empty history. They emit a warning naming the
cause before returning the empty frame, so “the read failed” is
distinguishable from “no runs yet”.run_dq_check() now
appends “[snapshot NOT recorded]” to its result line instead of printing
an unqualified success while the history row was lost.Concurrent runs sharing one snapshot database no longer lose a
snapshot or die during migration. Connections now set
PRAGMA busy_timeout, so a run that meets a database another
run is writing waits for the lock instead of failing instantly (its
snapshot was previously swallowed into a warning). Schema creation and
the auto-migration of older databases now run inside a single
BEGIN IMMEDIATE transaction, so two first-runs after an
upgrade can no longer both add the same column (the loser used to die
with “duplicate column name”). WAL journalling is intentionally not
used, as it is unsafe on the network file systems dqcheckr is deployed
on.
init_snapshot_db() no longer modifies a
snapshots table it did not create. If the target database
already contains an unrelated table of that name, it now aborts with a
typed dqcheckr_schema_error instead of altering the user’s
table. Column-existence checks during migration are now
case-insensitive, matching SQLite, so a column stored as
e.g. Report_File is recognised rather than
re-added.
The drift report now flags a missing-rate or non-numeric-rate
change in the same direction as the corresponding comparison check. Both
.compute_drift() columns used abs(), so a
column that improved between deliveries (fewer missing values,
or less non-numeric junk) was flagged as breaching, while the CP-03 and
CP-07 checks – reading the same max_missing_rate_change_pp
/ max_non_numeric_rate_change_pp thresholds – passed it.
Drift now breaches only on an increase, matching the checks. (The
numeric-mean-shift drift keeps its two-directional test, matching the
two-directional CP-04 check.)
A run whose HTML report is not produced no longer records a
successful-looking snapshot. When the Quarto CLI is absent the report is
skipped with a warning, but the snapshot row previously kept
render_status = "success" and a report_file
naming a report that was never written, so history readers (and the
GUI’s “Open report” link) pointed at a file that 404s. The snapshot is
now reconciled against what actually happened: if no report file exists
the row is marked render_status = "failed" and its
report_file is cleared. Rendering that returns without
producing a file now raises instead of reporting success, and
report_file is guaranteed to name a report that
exists.
Column type inference no longer misclassifies a value whose
prefix happens to be a date. as.Date() matches a
prefix and silently ignores trailing characters, so
"2024-01-15x" and the 9-digit id "202401159"
(whose first eight digits parse under %Y%m%d) were both
classified as "date" — which made corrupt dates pass QC-06
and stripped the numeric checks (QC-07/08/11) from id columns of eight
or more digits. Each date format is now gated on an anchored shape
before calendar validation. The documented caveat is unchanged: a
genuine eight-digit value that is also a valid %Y%m%d date
still classifies as a date; a nine-digit id now correctly classifies as
numeric.
read_recent_snapshots() now returns the full set of
columns even for a snapshot database created before 0.2.3. Previously it
ran SELECT * without migrating, so an older database
returned rows with the newer columns (such as report_file)
absent rather than NA, and callers relying on
those columns errored instead of degrading. The missing columns are now
filled in after the read with the same defaults a migration would apply;
the database file itself is not modified, so reads remain safe on
read-only or network-shared databases.
starwars_csv demonstration data
(inst/demonstrations/data/) is now shipped. An unanchored
.gitignore rule had excluded it, so a fresh clone was
missing the file that 27 example blocks and two tests depend on, and
R CMD check failed on a clean checkout.starwars_fwf) now has
its data file. A new inst/demonstrations/makedata.R derives
starwars.fwf from the CSV, so the FWF half of
demo.R runs end-to-end instead of aborting on a missing
file.encoding of ASCII (or a formal alias such as
US-ASCII) is now read as UTF-8. ASCII is a strict subset of
UTF-8, so this is lossless — and it removes a hard R session crash
(“Invalid multibyte sequence” inside vroom/iconv, not a catchable R
error) when a delivery declared ASCII contains a byte above 127 beyond
whatever sample an encoding sniffer originally looked at.read_dataset() now validity-scans the entire file
before parsing. A delivery that is not valid UTF-8 no longer risks
crashing or silently producing mojibake: it is read with a single-byte
fallback encoding, the run completes, and QC-16 reports a FAIL naming
the detector’s best guess at the actual encoding (suppliers can change
export encodings between deliveries, so this is checked per delivery).
Valid files and declared single-byte encodings (which have no invalid
byte sequences) report PASS. New dependency: stringi.run_dq_check() with an untyped “missing value where
TRUE/FALSE needed” error. Missing rates are defined as 0 for empty
inputs, and QC-14 gains an unconditional “Empty file” FAIL sub-check so
an empty delivery always fails the run — with a snapshot and report —
instead of crashing it.detect_files() no longer considers subdirectories of
folder when picking the current/previous file by
modification time.dq_result(threshold = NULL) now yields an
NA threshold instead of failing with “argument is of length
zero”; invalid vector status values produce a clear typed
error.column_rules pattern
now produces a QC-13 FAIL result naming the pattern instead of aborting
the run; an invalid column_order_severity is rejected at
load_config() time with a typed
dqcheckr_invalid_config error instead of aborting
mid-check.run_custom_checks() now validates each returned element
(required fields and a valid status) and aborts with
dqcheckr_invalid_custom_checks naming the offending
element, instead of failing later with misleading errors.snapshots row without its
column_snapshots stats; column-level custom results are
batch-inserted.run_timestamp and the report filename are
now derived from one timestamp taken at the start of the run. Previously
they came from two separate clock reads, so links that reconstruct the
report filename from the snapshot timestamp (as the GUI does) could
intermittently point at a file with a name one second off.compare_snapshots() no longer announces (and offers to
open) a drift report path when Quarto is unavailable and no report was
written.file.rename() fails across
filesystems (e.g. a network-share report directory); previously the move
failed silently."2.10" matches an
allowed value of 2.1.read_recent_snapshots()’s empty-database fallback now
has the same columns as the live query (it was missing
comparison_mode, render_status, and
type_changed_cols_vs_previous).report_file column in the snapshots
table stores the rendered report’s filename outright (auto-migrated into
existing databases), so consumers no longer need to reconstruct it from
the run timestamp. read_recent_snapshots() returns it;
NA for pre-0.2.3 rows.run_dq_check() and compare_snapshots() now
document that relative snapshot_db /
report_output_dir paths resolve against the working
directory, not config_dir.infer_col_type() documents its date-format precedence,
the all-must-parse rule, and the all-8-digit-identifier caveat, with a
pointer to column_types overrides.run_qc_checks() and the
individual check functions gain an optional types argument;
previously each of QC-06/07/08/11/15, the column statistics, and five
comparison checks re-ran full-column type inference (five date parses
plus a numeric parse) independently — the dominant cost on large
files.infer_col_type() rejects non-matching date formats from
a 100-value head sample before scanning the full column (results are
identical; only the rejection path gets cheaper).compute_col_stats() builds one data frame per column
instead of one per statistic row, cutting allocations on wide
files.csv_skip (parallel to
fwf_skip): read_dataset() now forwards
skip = config$csv_skip %||% 0L to
readr::read_delim(). This lets a config supply an explicit
col_names list and drop the file’s original header
row — required for delivery files whose header repeats column names
(e.g. a header that repeats Name/Amount per
bundled record), where the real header is unusable and must be replaced
positionally. Defaults to 0L, so existing configs are
byte-for-byte unaffected.?dqcheckr package help now resolves (the
"_PACKAGE" doc block no longer carries
@noRd).csv_skip value (skip
exceeding the row count yields a zero-row frame).expect_error().@importFrom declarations for
template-only packages.read_pass_rate_trend() function from
drift.R.CP-07 type-change comparison now uses a single guard so
it only runs the non-numeric rate comparison when both snapshots
classify the column as numeric — eliminates spurious WARNs
on type-changed columns.read_dataset() now forwards the col_names
config key to readr::read_delim(), so headerless CSVs are
read correctly.read_dataset() now forwards the quote_char
config key to readr::read_delim() as
quote =.rlang::abort() calls (27 sites) now carry a typed,
two-level class hierarchy
(c("<specific>", "dqcheckr_error")), letting callers
catch errors broadly or precisely.helper.R.compare_snapshots() and
list_snapshots() functions for drift analysis. Compares any
two historical snapshots and optionally renders an HTML drift report
with per-column statistical drift, schema changes, and trend
charts.resolve_col_type() function: returns the effective
type for a column, respecting per-column type overrides set in the
column_types config key.QC-15 outlier detection check
(check_outliers()). Configured via max_z_score
and/or iqr_fence_multiplier; skipped silently when neither
is set.check_key_uniqueness() (QC-12) now supports composite
keys: set key_columns to a character vector in the dataset
YAML.check_min_row_count() (QC-14) gains
max_row_count and max_file_size_mb
thresholds.col_threshold() and table_threshold()
added as internal helpers; column_rules per-column
threshold overrides are now correctly stored in
column_snapshots.threshold (G-01).render_report() uses quarto::quarto_render()
and returns NULL with a warning when Quarto CLI is not
installed.compare_schema() (CP-02) split into three separate
result objects: CP-02a (new columns), CP-02b (dropped columns), CP-02c
(type changes).compare_non_numeric_rate() (CP-07) now always emits a
result for every eligible column, including PASS for columns where the
rate did not increase (G-02).run_timestamp is now stored in ISO-8601 UTC format
(2026-01-01T12:00:00Z) rather than local time (RC-07).snapshots table gains three columns on first use:
comparison_mode, render_status, and
type_changed_cols_vs_previous. Existing 0.1.x databases are
auto-migrated on the first 0.2.0 run.PRAGMA foreign_keys = ON).detect_files() uses filename as a secondary sort when
two files share the same modification time, making folder-mode ordering
deterministic (RC-01).check_allowed_values() (QC-09),
compare_new_values() (CP-05), and
compare_dropped_values() (CP-06) cap observed
at 20 values with an "... and N more" suffix (RC-04,
RC-05).check_non_numeric() (QC-11) gains a
warn_non_numeric_rate config key for a separate WARN
threshold (C-01).compare_missing_rate() (CP-03) gains a
missing_rate_change_severity config key (warn
/ fail) (B-07).compare_column_order() (CP-08) gains a
column_order_severity config key that overrides the
format-based default (C-02).compute_col_stats() stores the numeric mean under the
key numeric_parseable_mean (renamed from
numeric_mean) to clarify that non-parseable values are
excluded from the calculation (C-04).compare_snapshots() uses the full merged per-dataset
config for drift threshold comparisons, so *** markers
match the original check run for datasets with
rule_overrides (G-05/G-06).compute_col_stats() unused qc_results
parameter removed (D-01).flag_new_columns, flag_dropped_columns,
flag_type_changes (in CP-02) and
flag_column_order_change (CP-08) are now honoured. Setting
any flag to false suppresses the corresponding check from
the report. Schema changes are still tracked in the SQLite snapshot
regardless of flags.type_inference_threshold is now configurable per
dataset via rule_overrides in the dataset YAML (or
default_rules in the global config). Previously fixed at
90%, it now defaults to 90% if not set. Affects QC-06, QC-07, QC-08,
QC-11, CP-02, CP-04, CP-05, CP-06, and CP-07.Initial release.
dqcheckr.yml and per-dataset
YAML files.
Need a high-speed mirror for your open-source project?
Contact our mirror admin team at info@clientvps.com.
This archive is provided as a free public service to the community.
Proudly supported by infrastructure from VPSPulse , RxServers , BuyNumber , UnitVPS , OffshoreName and secure payment technology by ArionPay.