The dqcheckr workflow: generate, edit, validate, run

dqcheckr checks recurring data deliveries: each time a file arrives, one function call verifies it against a per-dataset configuration, records a snapshot, and renders an HTML report. The workflow is deliberately plain:

This vignette walks the whole loop on a small example: generate the configs, read what was generated, edit a rule, validate, run, list the history, and compare two deliveries.

A delivery arrives

Our example delivery has a wrinkle that real files often have: the header repeats a column name (Amount appears twice).

writeLines(c(
  "Date,Amount,Currency,Amount,Status",
  "2026-07-01,100.00,AUD,15.00,settled",
  "2026-07-02,250.50,AUD,0.00,settled",
  "2026-07-03,80.25,NZD,8.10,pending"
), "orders.csv")

Bootstrap: two generator calls

A brand-new deployment needs exactly two calls: one for the shared global options, one per dataset. Both are create-only — they will never overwrite a config that already exists.

library(dqcheckr)
generate_global_config(config_dir = "config")
#> [dqcheckr] Global config written: config/dqcheckr.yml
#>   Relative snapshot_db/report_output_dir resolve against the working directory -- run from the deployment root.
generate_dataset_config("orders.csv", config_dir = "config")
#> [dqcheckr] Config written: config/orders.yml
#>   Review the commented options, then check with validate_config("orders", config_dir = "config").

(The detection itself is also available standalone: sniff_dataset("orders.csv") returns everything the generator is about to write — format, delimiter, encoding, column names and types, key-column candidates — without touching disk, if you want to inspect a delivery before committing to a config.)

Read what was generated

The dataset config is fully-optioned and self-documenting: every key the package understands appears exactly once — detected values live, optional settings commented out with their defaults — each with the same description the validator uses. Note three things as you read it:

the duplicated Amount arrived renamed positionally (Amount_2, with a # was "Amount" note), and csv_skip: 1 drops the file’s own header so the renamed list replaces it;
the col_names list sits under a warning: it is positional, so you must never comment out a single entry — every entry below it would shift onto the wrong column;
the sniffer proposes key_columns candidates commented out — columns unique in the sample, for you to confirm, never enforced silently.

writeLines(readLines(file.path("config", "orders.yml")))
#> # dqcheckr dataset config -- generated by generate_dataset_config()
#> # from: orders.csv
#> # Detected values are set live; optional settings are shown commented out
#> # with their defaults. Uncomment and edit only what you want to change.
#> # Check your edits any time with: validate_config("orders")
#> 
#> ## Identity of the dataset. Written into every config by convention;
#> ## dqcheckr itself takes the name as a function argument and matches it to
#> ## the config filename.
#> dataset_name: "orders"
#> 
#> ## Free-text description of the dataset, for humans reading the config. Not
#> ## read by the checks; the GUI wizard has always written it.
#> # description: "describe this dataset"
#> 
#> ## Explicit path to the current delivery. Required unless folder is set;
#> ## takes precedence over folder.
#> current_file: "orders.csv"
#> 
#> ## Directory holding the deliveries; the two most recently modified files
#> ## become current and previous. Required unless current_file is set.
#> # folder: "path/to/deliveries/"
#> 
#> ## Explicit path to the previous delivery for comparison checks. Only
#> ## honoured alongside current_file.
#> # previous_file: "path/to/previous.csv"
#> 
#> ## File format of the delivery. Defaults to CSV; "fwf" switches to the
#> ## fixed-width reader and makes fwf_widths required.
#> format: "csv"
#> 
#> ## Text encoding of the delivery. ASCII and its aliases are read as UTF-8
#> ## (lossless superset); a declared UTF-8 is validity-scanned before parsing.
#> encoding: "UTF-8"
#> 
#> ## Field separator for CSV files.
#> delimiter: ","
#> 
#> ## Quote character for CSV files, passed through to the reader.
#> # quote_char: "\""
#> 
#> ## POSITIONAL LIST -- always keep one entry per physical column, in file
#> ## order. NEVER comment out a single entry: every entry below it would
#> ## shift onto the wrong column. To exclude a column from checks, leave it
#> ## here and omit it from expected_columns / rules instead.
#> ## Explicit column names replacing the file's own header, in physical column
#> ## order. Omit to use the file's header. POSITIONAL: always list every
#> ## column; pair with csv_skip: 1 when the file has a header row being
#> ## replaced.
#> col_names:
#>   - "Date"
#>   - "Amount"
#>   - "Currency"
#>   - "Amount_2"   # was "Amount"
#>   - "Status"
#> 
#> ## Leading lines to drop from a CSV before reading, e.g. the original header
#> ## row when col_names supplies replacement names.
#> csv_skip: 1
#> 
#> 
#> ## Column widths for fixed-width files, in physical order. Required when
#> ## format is "fwf". POSITIONAL: always list every column; never comment out
#> ## an entry.
#> # fwf_widths: [5, 10]
#> 
#> ## Column names for fixed-width files, matching fwf_widths entry-for-entry.
#> ## POSITIONAL.
#> # fwf_col_names:
#> #   - "col_1"
#> #   - "col_2"
#> 
#> ## Leading lines to drop from a fixed-width file before reading.
#> # fwf_skip: 0
#> 
#> ## Columns the delivery is expected to contain; missing or unexpected
#> ## columns are flagged. Name-keyed: commenting an entry out safely drops
#> ## that column from the expectation.
#> expected_columns: ["Date", "Amount", "Currency", "Amount_2", "Status"]
#> 
#> ## Columns forming the row identity, checked for uniqueness and missingness.
#> # key_columns: ["Date", "Amount", "Amount_2"]   # candidates unique in the sample -- confirm before uncommenting
#> 
#> ## Per-column type pins (character, numeric, or date) overriding type
#> ## inference.
#> column_types:
#>   "Date": date
#>   "Amount": numeric
#>   "Currency": character
#>   "Amount_2": numeric
#>   "Status": character
#> 
#> ## Per-column quality rules; see the rule vocabulary for the keys valid
#> ## inside each column's map.
#> # column_rules:
#> #   some_column:
#> #     max_missing_rate: 0.05
#> #     min_value: 0
#> #     pattern: "^[A-Z0-9]+$"
#> 
#> ## Dataset-level overrides merged over the global default_rules; see the
#> ## rule vocabulary.
#> # rule_overrides:
#> #   max_missing_rate: 0.05
#> #   min_row_count: 1
#> 
#> ## Path to a user-supplied R file of custom check functions run after the
#> ## built-in checks.
#> # custom_checks_file: "custom_checks.R"
#> 
#> ## SQLite snapshot database recording every run. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # snapshot_db: "data/snapshots.sqlite"
#> 
#> ## Directory receiving rendered HTML reports. Dataset value overrides the
#> ## global one; relative paths resolve against the working directory.
#> # report_output_dir: "reports/"

Edit a rule

Editing a config is editing a text file. Here we do with code what you would do in an editor: uncomment the rule_overrides block and require at least one data row.

cfg_path <- file.path("config", "orders.yml")
lines <- readLines(cfg_path)
lines <- sub("^# rule_overrides:", "rule_overrides:", lines)
lines <- sub("^#   min_row_count: 1", "  min_row_count: 1", lines)
writeLines(lines, cfg_path)

Validate after every edit

validate_config() reports all findings in one pass — vocabulary, types and ranges, positional-list consistency — and, when the delivery file is reachable, cross-checks the config against the file’s header (never its body, so this is cheap even for very large files).

validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#> No findings.

Typos get a suggestion rather than a silent no-op. Watch what happens if we misspell a key:

writeLines(c(lines, 'delimitter: ";"'), cfg_path)
validate_config("orders", config_dir = "config")
#> dqcheckr config validation: orders -- VALID (tier: config+header)
#>   [warning] orders.yml: Unknown key 'delimitter' in the dataset config. Did you mean 'delimiter'?
writeLines(lines, cfg_path)   # undo

Findings follow one rule worth understanding, because it decides what can stop a run:

Config mistakes are errors, and errors block. A wrong value type, a broken positional list, a missing file source — things only an edit can cause — abort run_dq_check() before any snapshot is written. You cannot accidentally run an unrunnable config.
Delivery-facing findings are warnings, and warnings are recorded. A key column absent from the file, a col_names count that no longer matches — these can mean a config typo or the supplier changed the delivery, and drift must be recorded, not crash the run. The run proceeds, and each warning is persisted into the run’s results as a VC-01 check — visible in the report, the WARN counts, and the history — as well as printed to the console. Unknown keys (your own annotations round-trip safely) are warnings too.

Run the check

result <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: PASS - 0 warning(s), 0 failure(s). Report: /private/var/folders/ys/99f86vb54ys1_r7kg9lhfw780000gn/T/RtmpkHmFAn/dq-demo/reports/orders_20260725_211016_1.html
result$status
#> [1] "PASS"

Each run appends a snapshot to the SQLite database and renders a self-contained HTML report into reports/ (when the Quarto CLI is available; open_report = TRUE, the default, opens it for you in interactive use). The report is a plain file — open it from the file manager, mail it, archive it.

The run history

The history is one call away, newest first. The id column is what the drift comparison takes.

runs <- list_runs("orders", config_dir = "config")
runs[, c("id", "run_timestamp", "overall_status",
         "check_pass_count", "check_warn_count", "check_fail_count")]
#>   id        run_timestamp overall_status check_pass_count check_warn_count
#> 1  1 2026-07-25T21:10:16Z           PASS               19                0
#>   check_fail_count
#> 1                0

list_runs() is per-dataset and resolves the right database from the configs (honouring a per-dataset snapshot_db override). For the whole deployment at once — every dataset sharing a database — drop to the primitive and name the file yourself:

list_snapshots(db_path = file.path("data", "snapshots.sqlite"))[
  , c("id", "dataset_name", "run_timestamp", "overall_status")]
#>   id dataset_name        run_timestamp overall_status
#> 1  1       orders 2026-07-25T21:10:16Z           PASS

A second delivery, and drift

The next delivery arrives — with a suspiciously changed Amount profile and a missing status. Run the check again, then compare the two snapshots.

writeLines(c(
  "Date,Amount,Currency,Amount,Status",
  "2026-08-01,900.00,AUD,15.00,settled",
  "2026-08-02,1250.50,AUD,0.00,",
  "2026-08-03,880.25,NZD,8.10,pending"
), "orders.csv")
result2 <- run_dq_check("orders", config_dir = "config", open_report = FALSE)
#> [dqcheckr] orders: FAIL - 0 warning(s), 1 failure(s). Report: /private/var/folders/ys/99f86vb54ys1_r7kg9lhfw780000gn/T/RtmpkHmFAn/dq-demo/reports/orders_20260725_211018_2.html

compare_snapshots() defaults to the two most recent runs; pass two ids from list_runs() to compare any historical pair. With report = TRUE (the default) it renders a drift report HTML like the run report; here we just compute.

cmp <- compare_snapshots("orders", config_dir = "config",
                         report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
names(cmp)
#>  [1] "dataset_name"         "snap_prev"            "snap_curr"           
#>  [4] "table_drift"          "schema_changes"       "missing_rate_changes"
#>  [7] "non_numeric_changes"  "mean_shifts"          "spread_changes"      
#> [10] "distinct_changes"     "report_path"

Judging spread changes

The drift result also carries each numeric column’s spread (min, max, standard deviation). These are always reported — but never judged until you set a threshold, so a config that predates the rule gains information without gaining new failure modes. The Amount column’s spread more than doubled with the second delivery, yet exceeds stays FALSE:

cmp$spread_changes[, c("Column", "numeric_sd_prev", "numeric_sd_curr",
                       "numeric_sd_shift_pct", "numeric_sd_exceeds")]
#>     Column numeric_sd_prev numeric_sd_curr numeric_sd_shift_pct
#> 1   Amount       93.117645      208.296813             1.236921
#> 2 Amount_2        7.507996        7.507996             0.000000
#>   numeric_sd_exceeds
#> 1              FALSE
#> 2              FALSE

Activating the judgement is the usual edit — uncomment the rule in the global config (one default for every dataset; a per-column value in column_rules would win for its column):

gcfg_path <- file.path("config", "dqcheckr.yml")
glines <- readLines(gcfg_path)
glines <- sub("^# default_rules:", "default_rules:", glines)
glines <- sub("^#   max_numeric_sd_shift_pct:.*",
              "  max_numeric_sd_shift_pct: 0.2", glines)
writeLines(glines, gcfg_path)

cmp2 <- compare_snapshots("orders", config_dir = "config",
                          report = FALSE, open_report = FALSE)
#> [dqcheckr] drift: orders snapshot #1 vs #2
cmp2$spread_changes[, c("Column", "numeric_sd_shift_pct", "numeric_sd_exceeds")]
#>     Column numeric_sd_shift_pct numeric_sd_exceeds
#> 1   Amount             1.236921               TRUE
#> 2 Amount_2             0.000000              FALSE

Special cases worth knowing

Fixed-width files. The sniffer guesses column boundaries from blank gutters and always emits a character-position ruler comment above fwf_widths so you can correct a mis-split by counting. A packed file — columns touching, no gutters — gets explicit TODO widths that validate_config() flags as an error, so a run refuses to start until you fill them in against the ruler:

writeLines(c("AB12XY0099QQWW2026",
             "CD34ZW0100RRTT2026"), "ledger.txt")
generate_dataset_config("ledger.txt", config_dir = "config")
#> [dqcheckr] Config written: config/ledger.yml
#>   Review the commented options, then check with validate_config("ledger", config_dir = "config").
validate_config("ledger", config_dir = "config")
#> dqcheckr config validation: ledger -- INVALID (tier: config+header)
#>   [error] ledger.yml: 'fwf_widths' still contains the generator's TODO placeholder. Fill in the column widths (see the ruler comment in the config) before running.

The never-overwrite rule. Generators create; humans edit. Re-running a generator against an existing config is refused, and the file is untouched — your hand-tuning can never be silently clobbered:

generate_dataset_config("orders.csv", config_dir = "config")
#> Error in `generate_dataset_config()`:
#> ! Config already exists and will not be overwritten: config/orders.yml
#> Generated configs are created once and then owned by hand edits (the never-overwrite rule). To re-sniff, pass a different dataset_name or config_dir and diff the result.

To re-sniff a structurally changed delivery, generate under another name (or into a scratch directory) and diff.

Where everything lives

deployment-root/
├── config/            # one YAML per dataset + dqcheckr.yml -- the dataset list
├── data/
│   └── snapshots.sqlite
├── reports/           # one self-contained HTML per run
└── orders.csv         # deliveries (or point configs at their real location)

The config/ folder is the dataset list: add a dataset by generating a config, remove one by deleting its file, inspect one by opening it. The whole workflow is these calls:

Action	Call
New deployment	`generate_global_config("config")`
New dataset	`generate_dataset_config("file.csv", config_dir = "config")`
Check an edit	`validate_config("name", config_dir = "config")`
Run	`run_dq_check("name", config_dir = "config")`
History	`list_runs("name", config_dir = "config")`
Drift	`compare_snapshots("name", config_dir = "config")`

Welcome to ClientVPS Mirrors