Phase overview

cleangrowth() runs as a sequence of numbered phases. Phases 6–14 execute once per outer batch; all other phases run once per call. The numbering is not consecutive: Phase 2 no longer exists — it was a partial-run filter that has been removed (incremental re-cleans now live in the standalone gc_partial_run() wrapper; see Partial runs and preloaded references below). The remaining numbers are kept as-is so cross-references stay stable.

Most phases are pre- and post-processing bookkeeping. Phases 9 and 10 (z-scores and GA correction) are the only wrapper-level phases that directly affect child algorithm behavior.

Phase-by-phase walkthrough of cleangrowth()

Returning the user’s input

One rule runs through the whole wrapper: a column with the user’s name holds only the user’s own values, exactly as entered. growthcleanr works on interpreted copies of the input, but what it returns under the user’s names is what the user submitted. The column-by-column contract for users is in vignette("output") (§1, “Output fidelity”, and the carry_through row); this section lists where the code applies the rule, so that a new output column or a new code path can follow it everywhere.

• Values come back as entered. sex, param and subjid keep verbatim copies (sex_input, param_input, subjid_input) beside their working versions (Phase 3); measurement keeps v_adult (and meas_input for a text or factor column, so the submitted type is kept); and agedays keeps agedays_input beside the whole-day working value. The verbatim copies are restored just before the Phase 15 trim. growthcleanr’s interpreted values are returned only under growthcleanr names (gc_internal_*), never under the user’s.
• Columns come back under the user’s names. After the trim, the six role columns are renamed from the canonical names to the names the user supplied (data-frame style) or that were derived from the call (vector style).
• A user column is renamed only when it would collide. A user column (a carried column, or the user’s id in long output from semi-wide/wide input) is returned as userinput_<name>, with a warning, only when its name matches a column growthcleanr returns in this run. The list of returned columns comes from .gc_added_output_cols(), the same list that builds the Phase 15 trim, and .gc_userinput_names() applies the rename; a name growthcleanr does not return in this run is left alone.
• Carried columns are reattached by input row position. With carry_through = TRUE, non-role columns are parked at entry in input row order and added back after the rename, by row position rather than by a join key, so no internal column (such as line) can collide with a user’s column name. The single-run path uses the input row positions captured before the trim; the checkpointed path relies on the combined result being in input row order.
• Long output from semi-wide/wide input. .gc_crossformat_id_cols() returns the user’s id under the user’s own name (repeated on each measurement from its input row), growthcleanr’s per-measurement number as gc_internal_id, and the user’s other columns repeated from their row, with the same userinput_<name> rule.
• Checkpointed runs follow the same steps. The checkpoint path (see Checkpointed runs) repeats the rename, carry reattachment, cross-format id handling and re-widening on the combined result, in the same order as a single run, so a checkpointed run returns the same columns and values as a single run.

Purpose

The outer batching wrapper (added by Chris Penney) divides subjects into groups for memory management. Without batching, preprocessing (z-score calculation, recentering, reference table merges) for the full dataset must fit in memory at once. For large datasets (100s of thousands or millions of subjects), this can cause memory pressure. The wrapper processes one batch at a time, then combines results.

Subject-level batching

Batching occurs at the subject level, not the row level. All of a given subject’s rows — across all parameters (WEIGHTKG, HEIGHTCM, HEADCM) and all ages — are assigned to the same batch. This is a logistical choice: batching by subject before the child/adult age split keeps both halves of a subject’s data within the same outer-batch iteration, so output reassembly can combine child and adult results for that subject without cross-batch bookkeeping. The child and adult algorithms themselves are independent and do not share trajectory data across the age split.

Interaction with the child/adult age split

After batching by subject, cleangrowth() splits each batch’s data by adult_cutpoint (default 20 years, range 18–20) into pediatric (data.all) and adult (data.adult) subsets. A subject whose rows span the cutpoint will have some rows sent to cleanchild() and others to cleanadult(). This is intentional: the child and adult algorithms are independent pipelines with no interaction between them. A subject’s child measurements are cleaned using only their other child measurements, and their adult measurements are cleaned using only their other adult measurements.

Both sets of rows for such a subject are guaranteed to be in the same batch because batching is done on subjid before the age split. The result assembly at the end of each batch iteration recombines the child and adult results for the batch’s subjects, and rbindlist() combines across batches.

Two batching systems in the code

The code contains two layered batching systems with different purposes:

System 1 — Outer wrapper (for memory management):

• In child_clean.R, inside cleangrowth().
• Creates batches of up to batch_size subjects (default 5000; configurable via the batch_size parameter): patients[, batch := (seq_len(.N) - 1L) %/% batch_size + 1L].
• Iterates the batches with a for loop. Each iteration filters the input to the current batch’s subjid values (both the pediatric data.all and the adult data.adult are limited to those subjects), runs preprocessing and the child/adult algorithms on just that batch, and pushes the per-batch result onto results_list.
• After the loop, rbindlist(results_list) combines per-batch results and the merge against data.batch restores user input columns.

System 2 — Inner batching (for parallelism):

• Also in child_clean.R, inside each outer-batch iteration.
• Assigns round-robin batch numbers to subjects in the already-filtered data.all: (seq_along(subjid.unique) - 1L) %% cores + 1L.
• When parallel = FALSE (or a single-core auto-detect), cores = 1, making this a no-op: all subjects in the outer batch go to a single cleanchild() call.
• When parallel = TRUE, dispatches via foreach::foreach(...) %dopar% to process inner batches in parallel across CPU cores.

The inner batching subdivides whatever data the outer wrapper provides into parallel work units. With parallel = FALSE, it has no effect. The adult algorithm has its own inner parallel batching system, structured identically.

Parallel processing

parallel = TRUE runs the inner batches concurrently. The worker processes are separate R sessions that resolve libraries from .libPaths() and need system.file() access to the inst/extdata/ reference tables. This means growthcleanr must be installed (via devtools::install_github(), devtools::install_local(), or CRAN), not merely source-loaded via devtools::load_all().

Practical note for developers: There is a limitation that applies only to package developers. Most uses of devtools — including devtools::install_github("growth-data-tools/growthcleanr", ...) and devtools::install_local(".") — actually install the package into your R library, and parallel = TRUE works normally after those calls. The issue only appears when a developer uses devtools::load_all() to load the source tree into the current R session without installing (a workflow used for rapid iteration on package code). The workaround in that case is to run devtools::install_local(".") once, which takes only a few seconds.

Results are bit-identical to sequential processing (verified on the 77,721-row syngrowth dataset). Typical observed speedups on 2,697 subjects with many errors (which results in longer run-time): ~114 s at parallel = FALSE, ~65 s at cores = 2 (1.7×), ~30 s at cores = 4 (3.9×).

Batch-invariant setup (the exclude.levels factor and the Tanner velocity reference) is performed once outside the outer loop so the cost is not repeated per batch; the WHO velocity tables are loaded per batch inside cleanchild().

Partial runs and preloaded references

Two mechanisms support workflows that call growthcleanr many times on overlapping data (simulation loops, error-injection pipelines, secure-environment workflows with updated data extracts): preloaded reference tables, and the gc_partial_run() wrapper for incremental re-cleans.

ref_tables + gc_preload_refs(). gc_preload_refs() reads the CDC and WHO growth-chart reference files once and returns a named list of closures (mtz_cdc_prelim and mtz_who_prelim). Passing that list as the ref_tables argument to subsequent cleangrowth() calls skips the per-call read_anthro() disk reads (≈0.9 seconds per call). The return value is a session-lifetime object; rebuild only if the reference files themselves change.

refs <- gc_preload_refs()
res  <- cleangrowth(..., ref_tables = refs)

gc_partial_run() — incremental re-cleans. When you re-clean a dataset that changed for only a subset of subjects, gc_partial_run() re-cleans just those subjects and reuses the earlier cleaned result for everyone else. This is correct because growthcleanr cleans each subject independently after reference recentering, so re-cleaning only the changed subjects yields exactly the rows those subjects would get in a full run; the combined result equals cleangrowth(new_data, ...).

baseline <- cleangrowth(old_data)
updated  <- gc_partial_run(new_data, baseline, changed_subjids = who_changed)

gc_partial_run() is a thin wrapper around cleangrowth(), not a mode of it: it slices new_data to the changed subjects (plus any subjects newly added relative to the cache; removed subjects are dropped), hands that subset to a single ordinary cleangrowth() call, and row-binds the retained cached rows to the fresh rows, ordered by id. Everything passed in ... — including checkpoint_dir, checkpoint_size, batch_size, ref_tables, and the output switches — is forwarded verbatim to that inner call, so a large changed subset can still be checkpointed. Because the cache never enters cleangrowth(), it cannot interact with the cleaning core.

Requirements: input must be long format; both new_data and cached_output must carry a per-row unique id (the stable key that links retained and fresh rows), and id values must be stable across runs; and ... must specify the same output options used to build cached_output — if the resulting columns differ, gc_partial_run() stops rather than silently produce NA-filled rows. See ?gc_partial_run for the full contract.

Because only the changed subjects are cleaned, the per-call cost scales with the fraction of subjects that changed: re-cleaning 5 % of subjects does roughly 5 % of the cleaning work of a full run (plus a small fixed cost to slice and merge). The savings are largest in loops where each iteration perturbs only a few subjects.

Checkpointed runs (the checkpoint / checkpoint_dir switch)

How the switch resolves. The pair checkpoint / checkpoint_dir is resolved once, before the checkpoint loop, into a concrete target. In priority order: a nested per-checkpoint call (.checkpoint_driven) never re-checkpoints; checkpoint = FALSE disables checkpointing and, if a checkpoint_dir was also passed, ignores it with a warning (the “nothing on disk” escape hatch is absolute); an explicit checkpoint_dir is honored as a persistent, resumable location (keep_checkpoints as given); and the default checkpoint = TRUE with no directory writes to an ephemeral file.path(tempdir(), "growthcleanr_checkpoints") with keep_checkpoints = FALSE and checkpoint_resume = FALSE, engaging only when the data exceeds one checkpoint (uniqueN(subjid) > checkpoint_size). A single checkpoint is the whole dataset cleaned in one in-memory pass and gives no benefit, so at or below one checkpoint’s worth of subjects the run stays fully in memory and writes nothing — which keeps the on-by-default behavior free for small and typical data. On a system whose temp dir is RAM-backed (e.g. Linux tmpfs), a very large dataset should pass a checkpoint_dir on a real disk. User-facing guidance is in vignette("large-datasets").

Whichever way it engages, the mechanics are the same. cleangrowth() splits subjects into checkpoints of checkpoint_size (default 5000, capped at 9999 checkpoints — a checkpoint_size that would exceed the cap is increased to stay within it, with a warning when you chose the size), cleans each checkpoint, and writes that checkpoint’s result to its own checkpoint_NNNN.rds file in the checkpoint directory before moving to the next. After the last checkpoint it combines all checkpoint files into a single result in the original input order. Reference tables are preloaded once (via gc_preload_refs()) and reused across checkpoints. Because subjects are independent, the checkpointed result is identical to cleaning the whole dataset at once — this includes wide / semi-wide input and output, which reshape once around the checkpoint loop (input is reshaped to long before checkpointing; the combined long result is re-widened once at the end).

Internally each checkpoint is cleaned by a nested cleangrowth(checkpoint_dir = NULL) call in long mode: the input funnel, date parsing, and id generation happen once in the outer call, so a checkpoint receives canonical long data with a valid unique id and every subject’s rows land in one checkpoint. Carry-through of extra non-growthcleanr columns works under checkpointing: the parked columns are reattached once to the combined result at the end, so a checkpointed run returns the same columns as a non-checkpointed one. Every subject is cleaned independently, so checkpointing never changes a result. The adult_cutpoint (clamped to [18, 20]) is finalized once in the outer call and forwarded to each checkpoint so every checkpoint uses the same value; growthcleanr does not change the cutpoint based on the data, so this is a dataset-independent constant. The all-adult advisory (which suggests adult_cutpoint = 18 when the data looks adult-only) and the consolidated input warning are each emitted once from the outer call over the whole dataset rather than per checkpoint.

checkpoint_size is the persist unit (one .rds per checkpoint) and is independent of batch_size, the in-memory unit within a checkpoint; batch_size < checkpoint_size is allowed, so a checkpoint sized for a reasonable number of resumable files can still be processed in smaller memory batches.

The per-checkpoint files exist for durability and resumability:

• By default (checkpoint_resume = FALSE) a run starts fresh: any checkpoint files already in checkpoint_dir are deleted at the start, so the combined result contains only the current run’s checkpoints and a re-run never silently mixes in stale results from a different prior dataset.
• With checkpoint_resume = TRUE, checkpoints that already have a checkpoint file are skipped, so an interrupted run resumes on only the unfinished checkpoints. Checkpoint membership is determined by the sorted set of subjects and checkpoint_size, so resuming requires the same dataset and checkpoint_size. It also requires the same other cleangrowth() arguments; these are not checked, and a mismatch silently mixes results from different settings.
• keep_checkpoints (default TRUE) governs only whether the files are retained after a successful run; it is independent of checkpoint_resume, which governs start-of-run reuse.

cleangrowth_checkpoint() is a deprecated wrapper that forwards to cleangrowth(checkpoint_dir = ...) — its use_existing_checkpoint_results argument maps to checkpoint_resume — and warns on use; it will be removed in a future version. cleangrowth_checkpoint_combine() re-combines the saved checkpoint files without re-running the algorithm — useful if all checkpoints completed but the final combine step failed. It reads every checkpoint_NNNN.rds file in the directory. User-facing usage is in vignette("large-datasets").

Two unrelated uses of “checkpoint.” The checkpoint_data capture in Phase 12 of cleangrowth() is an internal z-score snapshot merged back onto the output for diagnostics; it is unrelated to checkpoint_dir and the checkpointed-run machinery that saves per-checkpoint result files to disk. The shared word is incidental.

Progress output

cleangrowth() reports progress at one of three levels, set by the quietly and verbose flags (both plain and checkpointed checkpoint_dir modes):

• Silent (quietly = TRUE) — nothing is printed: no progress lines and no cleaning summary.
• Minimal (the default) — a one-line cleaning in progress header (subject / measurement / unit counts), an X/Y counter as each memory batch or checkpoint finishes, and the end-of-run cleaning summary. A single-batch, non-checkpointed run therefore prints just the header and the summary.
• Verbose (verbose = TRUE, which implies quietly = FALSE) — the detailed view: the milestone lines (preprocessing, cleaning child/cleaning adult, the per-batch BATCH i/n banner) plus granular per-step diagnostics (pre-filter and iteration counts for CF, EWMA, SDE, Evil twins; adult step headers). Intended for development and debugging.

The three nested processing units keep distinct, fixed names: checkpoint (checkpoint_size; used whenever a run has more than checkpoint_size subjects, unless checkpoint = FALSE), batch (batch_size, the in-memory split inside one cleangrowth() call), and parallel group (verbose only).

A minimal (default) checkpointed run — header, a per-checkpoint X/Y counter, then the summary:

[10:22:58] growthcleanr: cleaning in progress: 435,000 subjects, 1,740,000 measurements, 87 checkpoints of up to 5,000 subjects | checkpoint_dir: results/gc_ckpt

==================================================
  CHECKPOINT 1 / 87  |  5,000 subjects
==================================================
[10:23:36] checkpoint 1/87: done | 32s | elapsed 32s
...
[11:09:30] all 87 checkpoints complete | 435,000 subjects, 1,740,000 rows | total 46m32s
************************************************************
growthcleanr cleaning summary  (1,740,000 measurements)
  ...
************************************************************

At the verbose level each checkpoint additionally prints its preprocessing / cleaning child / cleaning adult milestone lines and the per-step diagnostics. A plain cleangrowth() call (no checkpoint_dir) prints the same header and summary; when batch_size forces more than one memory batch it adds a one-line batch i/n -- done counter at the minimal level (and the fuller BATCH i/n banner at verbose). Under a checkpointed run, each per-checkpoint cleangrowth() call is told its framing is owned by the checkpoint driver, so it stays quiet while the driver supplies the header, the per-checkpoint progress, and the total. When log_file is set, the full log is written there even under quietly = TRUE, so a colleague can monitor a run remotely.

Calling styles

cleangrowth() supports two calling styles:

1. Data-frame style (primary). Pass a data.frame or data.table as the first argument (data). The six per-row arguments (subjid, param, agedays, sex, measurement, id) are then column-name strings, defaulting to the canonical names. Examples:

   cleangrowth(df)                                    # all defaults
   cleangrowth(df, subjid = "patid", id = "row_id")   # remap two columns

2. Vector style (legacy). Omit data; pass each field as a vector via named args, e.g. cleangrowth(subjid = df$subjid, param = df$param, ...). Retained for backward compatibility.

The data-frame path extracts vectors from data and merges into the vector path; both produce identical results.

Flexible sex and parameter coding

cleangrowth() accepts a range of sex and param codings so data from varied sources needs less pre-formatting, while guarding against silent miscoding. Both interpretations preserve the user’s verbatim submission (echoed back in the output sex / param columns); the interpreted values gc worked with are available as gc_internal_sex / gc_internal_param under internal_detail (or full_detail).

Sex. By default 0, "m", "b", "male", "boy" are read as male and 1, "f", "g", "female", "girl" as female — matched case-insensitively and whitespace-trimmed. Supply sex_coding to use your own codes; it is a named list whose male and female elements are the code vectors, and it replaces the defaults wholesale:

cleangrowth(df, sex_coding = list(male = c(0, "M", "boy"), female = c(1, "F", "girl")))

Handling of the interpreted sex:

• Recognized codes clean normally.
• Missing (NA/blank) and present-but-unrecognized codes on pediatric rows are set aside as Exclude-Missing-Info, contributing an “unrecognized sex” bullet (which names the offending values and leads with the sex_coding fix) to the consolidated input warning, so a coding gc did not expect is visible, not silent. Pediatric measurements are cleaned only with a recognized sex value (the child z-scores are sex-specific). If no pediatric row has a usable value in the sex column and there are no adult rows, the run stops, since nothing could be cleaned. If there are adult rows, the run continues: the pediatric rows are set aside and the bullet says that this is every pediatric measurement and asks you to check the sex column mapping. In either case, when every pediatric row is at least 18 years old, the message also suggests adult_cutpoint = 18 so all values are cleaned with the adult algorithm.
• Adult rows. A sex column is always required, but the sex column value may be blank or NA for adults (ages at or above adult_cutpoint): the adult algorithm does not require a valid value in the sex column, but when available, it is used in the Evil Twins step and in calculating whether a value falls in the unit error range.
• The 1=male/2=female trap. growthcleanr reads 1 as female. A column coded 1=male/2=female would therefore misclassify every male as female while only the 2s drop out — a plausible-looking wrong answer. So when the default coding is in use and every present sex value is 1 or 2 (with at least one 2), cleangrowth() hard-stops and tells the user to recode or to declare the coding with sex_coding = list(male = 1, female = 2). Passing an explicit sex_coding suppresses the guard. The guard is checked once on the whole dataset, so under checkpointing it stops before the first checkpoint is cleaned, however the data are sorted. A column with pediatric rows and more than one subject in which every value is 1 has no 0 or 2 to reveal the coding and could be single-sex data under either scheme, so it is not stopped: it is read as female with a prominent warning, which an explicit sex_coding also removes.

Parameter. growthcleanr recognizes only the spelled-out canonical tokens — WEIGHTKG, WEIGHTLBS, HEIGHTCM, HEIGHTIN, LENGTHCM, LENGTHIN, HEADCM, HEADIN — matched case-insensitively and whitespace-trimmed. There are deliberately no built-in abbreviations: a short label such as WT, HT, or HC is not assumed to be a growthcleanr parameter, because it just as easily names something that is not a body measurement (a statistical sample weight, for instance). To have a non-canonical label recognized, declare it with param_coding. The imperial tokens convert at 2.54 cm/in (HEIGHTIN → HEIGHTCM, LENGTHIN → LENGTHCM, HEADIN → HEADCM) or 2.2046226 lb/kg (WEIGHTLBS → WEIGHTKG).

Supply param_coding to map your own labels. A remapped label defaults to its metric target; to declare that a column is imperial, route it to an imperial parameter:

# route two custom labels; treat a "WEIGHT" column as pounds and "HT"/"STATURE" as cm
cleangrowth(df, param_coding = list(WEIGHTLBS = "WEIGHT", HEIGHTCM = c("HT", "STATURE")))

It is a named list keyed by canonical parameter. A name that is not a canonical parameter, or one label mapped to two parameters, is an error. Params matching nothing are set aside as Exclude-Not-GC-Param (a bullet in the consolidated input warning, never a stop — extracts legitimately carry labs and vitals).

Systematic unit-error warning. After cleaning, if more than 20% of a parameter’s cleaned values fall in the unit-error range (unit_error_range; magnitudes consistent with a metric/imperial mix-up), cleangrowth() emits a soft warning() — a large share suggests the whole column is in the wrong unit (e.g. weight recorded in pounds but labeled WEIGHTKG, or a column mapped to a metric target that was actually imperial). Fix by labeling the column with the imperial parameter or mapping it via param_coding.

Input format: long, semi-wide, and wide

The input_format argument selects the shape of data:

• "long" (default) — one row per measurement, the classic format described above.
• "semi_wide" — one row per visit: a subjid, an age source, a sex, and one column per parameter (at most one value per parameter per visit).
• "wide" — one row per subject: a subjid, a sex, and repeated observation blocks marked by a _N column suffix (agedays_1, wt_1, ht_1, agedays_2, wt_2, …).

Semi-wide and wide both require the data-frame style, and both are reshaped to canonical long form before any cleaning, so results are identical to submitting the same measurements in long format. Results come back in the same shape by default (output_format mirrors input_format); see Output format: semi-wide and Output format: wide below. Set output_format = "long" for the full per-measurement result table, which merges back onto the source data by subjid/param/agedays.

The format is never auto-detected: input_format defaults to "long", so semi-wide or wide data submitted without setting it is not silently mis-cleaned — it fails, because long format requires param/measurement columns (and a single agedays column) that reshaped data lack. When the columns look reshaped, the error names input_format as the likely fix.

Parameter value columns are mapped with semiwide_cols, a named list of gc-parameter → column name. If omitted, columns literally named as recognized parameters (WEIGHTKG, HEIGHTCM, LENGTHCM, HEADCM, WEIGHTLBS, HEIGHTIN, LENGTHIN, HEADIN) are used:

# columns already named WEIGHTKG / HEIGHTCM / HEADCM
cleangrowth(visits, input_format = "semi_wide")

# custom value-column names
cleangrowth(visits, input_format = "semi_wide",
            semiwide_cols = list(WEIGHTKG = "wt_kg", HEIGHTCM = "ht_cm", HEADCM = "head_cm"))

Structural columns reuse the usual remap arguments (subjid, sex, and either an agedays column or dob + visit_date columns, from which agedays is computed). Behavior details:

• Empty cells vs. zeros. An NA cell means “no measurement of that parameter at this visit” and is dropped during reshape (no output row). A literal 0 is not an empty cell: exactly as in long format, its value is returned unchanged in measurement and the row is flagged Exclude-Missing-Info (growthcleanr treats 0 as missing for cleaning purposes but never alters the value it hands back). Do not pad absent measurements with 0 — use an empty/NA cell.
• At least one recognized parameter column must be present (otherwise an error). A missing weight or height/length column warns; a missing head-circumference column warns only when the data include any visit under 2 years of age.
• Length vs height is a long-format-only feature. Semi-wide input always uses default handling of linear measurements (values under 2 years are treated as recumbent length, values 2 years and over as standing height, using estimated corrected age for children whose data are consistent with prematurity or small-for-gestational age (SGA) and who receive an assigned gestational age for cleaning purposes), so length.adjust = TRUE is rejected. A LENGTHCM/LENGTHIN column is accepted at any age and gets default handling, giving the same result as mapping it to HEIGHTCM/HEIGHTIN; the adult algorithm ignores the length/height label entirely. To use the two-directional length.adjust reconciliation, submit the data in long format.
• Duplicate column names are an error. If a subjid, sex, age, or parameter column name appears more than once, growthcleanr stops. R’s column indexing silently keeps only the first such column, so cleaning would proceed on a dataset with a shadowed, dropped column — rename or drop the duplicate instead. (An unmapped extra column is simply ignored: parameters come from the column you map, not from any value.)
• Same-day duplicates. Because a visit row holds at most one value per parameter, two measurements of the same parameter on the same day must appear as two rows sharing the same (subjid, agedays). growthcleanr keeps one and flags the other as a same-day extraneous value, exactly as for long input. Each parameter is handled independently (weight/height pairing across same-day rows is not preserved).
• Per-measurement ids / same-day tiebreaking. The reshape mints deterministic per-measurement ids in a stable, documented emission order (subject in first-appearance order; within a subject by visit, then by parameter in weight → height/length → head-circumference order). If you supply an id (the id argument, or a column named id), a subject’s visit rows are ordered by your id first (sorted by value, so character ids sort alphabetically) and by input row second; the id need not be unique. The same input always produces the same result, but which of two same-day duplicates is flagged can depend on this emission order. To control same-day tiebreaking, give same-day visit rows ids in the order you want (a higher id is preferred, except at birth), or reformat to long format and set a per-measurement id yourself.
• Ids in long output. With output_format = "long", each input row becomes one output row per measurement, so a row id cannot identify a single measurement. growthcleanr returns its own per-measurement number as gc_internal_id, and your id (the id argument, or a column named id) under your own column name, repeated on each measurement from that input row. With no id column there is no id column in the output. Your other columns are returned too (carry_through), repeated on each measurement from their row. A column with your name always holds only your own values: if your id or another column has the name of a column growthcleanr returns (e.g. exclude or param), it comes back as userinput_<name>, with a warning.
• Column names in long output. The subject, sex and age columns come back under your own names (e.g. patid, male, age_d; for wide input, the age prefix such as age), as in long input. param and measurement have no column of yours to be named after, so they use those names; on the date route the computed ages are agedays.
• Factor value columns are taken by their labels (never their level codes), with a warning naming the columns.

Wide input: the block layout

Wide data has one row per subject and repeats a whole observation across numbered columns. You choose the prefix; growthcleanr requires the _N suffix and discovers the blocks from it — there is no need to declare how many observations your widest subject has:

# subjid sex agedays_1 wt_1 ht_1 agedays_2 wt_2 ht_2 ...
cleangrowth(subjects, input_format = "wide", output_format = "long",
            wide_cols = list(WEIGHTKG = "wt", HEIGHTCM = "ht", HEADCM = "hc"))

wide_cols maps each gc parameter to its column-name prefix. If omitted, prefixes literally named as recognized parameters are used (WEIGHTKG_1, HEIGHTCM_1, …). A mapped prefix with no <prefix>_N columns in the data is an error, so map only the parameters your dataset actually carries.

• Levels. subjid and sex are patient-level (one unsuffixed column each). The age source is per block: either agedays_1, agedays_2, … (rename the prefix with agedays = "age" if your columns are age_N), or — on the date route — a single patient-level dob column plus visit_date_1, visit_date_2, ….
• Ragged subjects need no special handling. A subject with 3 observations in a 6-block table simply leaves blocks 4–6 empty; empty cells are dropped during reshape.
• Block numbers are any non-negative integers. A _0 block (some exports number a baseline observation _0) is accepted and, like every block, is ordered by its number — so _0 sorts ahead of _1. Block numbering may start at 0 or 1; a missing intermediate block warns (see Strictness) but does not fail.
• Empty cells vs. zeros, missing-parameter warnings, and same-day duplicates behave exactly as for semi-wide input (above). Two blocks carrying the same agedays are resolved as same-day extraneous values.
• A value with no age — a populated wt_3 where agedays_3 is missing — keeps its row and is flagged Exclude-Missing-Info, the same as a long-format row with a missing agedays.
• Per-measurement ids are minted in a stable, documented emission order: subject in first-appearance order; within a subject by input row, then block number ascending; within a block by parameter in weight → height/length → head-circumference order. The same-day tiebreaking caveat above applies. A user id column (the id argument, or a column named id) identifies a row, not a measurement: it does not affect cleaning, comes back unchanged in wide output, and in long output follows the semi-wide rule above (your id under your column name, growthcleanr’s number in gc_internal_id).
• Strictness — these warn rather than fail, and the data are processed as submitted:
  • a gap in the block numbering (ht_1, ht_3, no ht_2);
  • more blocks than max_blocks (default 25) — raise it, or reshape to long or semi-wide for data this wide. An implausible block count errors instead, since it usually means the _N suffixes parsed wrong;
  • ages that do not increase across ascending blocks;
  • a subjid appearing on more than one row (each row’s blocks become additional observations for that subject).
• Errors: two parameters mapped to the same prefix; a duplicate structural column name (two subjid columns); two columns landing on the same block — an exact duplicate name, or leading-zero padding (ht_1 and ht_01), including duplicated age or visit-date columns; no age source; length.adjust = TRUE (length vs height is a long-format-only feature — see the semi-wide behavior notes above).

Wide input returns wide output by default (see Output format: wide below); set output_format = "long" for the per-measurement table.

Output format: semi-wide

With output_format = "semi_wide" (the default for semi-wide input), growthcleanr returns your table: every column you submitted, unchanged, one row per input row, with the requested growthcleanr columns inserted immediately to the right of the value column they describe.

cleangrowth(visits, input_format = "semi_wide")

subjid agedays sex  wt_kg  exclude_wt_kg  ht_cm  exclude_ht_cm  site
A      100     0    6.0    Include        60.0   Include        clinic1

• Which columns are added is controlled by widened_emit, default "exclude". Any per-measurement output column may be named — "bin_exclude", "exclude_detailed", "unit_error_range", "measurement", "agedays", and so on. Naming a column whose own switch was not enabled (e.g. "exclude_detailed" without exclude_detail = TRUE) is an error. Note that semi-wide output does not inherit the long-format defaults: bin_exclude and unit_error_range appear only if you ask for them.
• Naming. growthcleanr’s own output columns take the value-column suffix only (exclude_wt_kg, bin_exclude_wt_kg). growthcleanr’s internal version of a value you supplied takes a gc_internal_ prefix (gc_internal_meas_wt_lbs, gc_internal_param_wt_lbs, gc_internal_agedays), because those report what growthcleanr used rather than what you submitted.
• Your values are never rewritten. Submit weights in pounds and the wt_lbs column comes back in pounds. widened_emit = "measurement" adds gc_internal_meas_<column> with the value growthcleanr worked from (converted to metric; length.adjust does not apply to semi-wide or wide input), and automatically adds gc_internal_param_<column> alongside it so the units of that value are visible in the data rather than only in this vignette.
• Empty cells get an NA verdict: a visit with no head-circumference measurement has NA in exclude_head_cm. A visit row where every parameter is empty is still returned.
• Same-day duplicates stay on their own visit rows, each with its own verdict — the round-trip is keyed by input row, not by (subjid, agedays).
• Other columns. With carry_through = TRUE (default) every column you submitted returns. carry_through = FALSE returns only the structural columns (subjid, sex, the age source), the parameter value columns, and the growthcleanr columns.
• Not available for: long input (growthcleanr cannot infer visit rows from long data — same-day measurements are ambiguous; reshape the long output yourself with data.table::dcast()). Checkpointed runs are supported (the re-widen happens once around the checkpoint loop).

Output format: wide

With output_format = "wide" (the default for wide input), growthcleanr returns your subject rows the same way — every submitted column unchanged, one row per subject, with each verdict column slotted immediately right of the value block column it describes:

cleangrowth(subjects, input_format = "wide",
            wide_cols = list(WEIGHTKG = "wt", HEIGHTCM = "ht"))

subjid sex agedays_1 wt_1 exclude_wt_1 ht_1 exclude_ht_1 agedays_2 wt_2 exclude_wt_2 ...
A      0   100       6.0  Include      60.0 Include      200       8.0  Include      ...

Everything from Output format: semi-wide carries over — the same widened_emit control, the same naming (exclude_wt_1, gc_internal_meas_wt_1), values never rewritten, empty ragged blocks getting NA verdicts, carry_through, and the direct echo of your input table (so rows can’t be lost or reordered and ragged blocks come back exactly as submitted). Two wide-specific points:

• agedays widens per block, not per row. widened_emit = "agedays" adds gc_internal_agedays_1, gc_internal_agedays_2, … — one per block, because the age varies within a subject row. subjid and sex stay one unsuffixed column each. Each carries the whole-day age growthcleanr cleaned on; on the date route this is the only way to see the ages growthcleanr computed.
• Not available for: long or semi-wide input (growthcleanr will not invent observation blocks from data that is not already wide). Checkpointed runs are supported. Wide input with output_format = "long" gives the full per-measurement table instead.

Required input fields

Input handling

• measurement == 0 is replaced with NaN (treated as missing) in the internal variable v
• Imperial measurements are converted to metric before processing: HEIGHTIN multiplied by 2.54 and relabeled HEIGHTCM; WEIGHTLBS divided by 2.2046226 and relabeled WEIGHTKG
• LENGTHCM is relabeled to HEIGHTCM. By default (length.adjust = FALSE) no length-height correction is applied (except for gestational-age-corrected children between chronological age 2 and corrected age 2; Phase 8, fact 6), because of variability in measurement and recording practices across health systems; see Phase 8: Length vs. height handling for the optional reconciliation.
• sex is recoded to integer: 0 = male, 1 = female
• Cleaning uses the whole-day value of agedays (floor(agedays), as on the date route, so a negative fractional age such as -0.5 is day -1 and is set aside like other negative ages); the submitted value, fractional or not, is what the output agedays column returns
• Data are split by adult_cutpoint (default 20 years): rows with agedays < cutpoint * 365.25 go to the child algorithm; rows at or above go to the adult algorithm

Sorting

The primary sort order is:

setkey(data.df, subjid, param, agedays, internal_id)

This sort is critical for deterministic results. internal_id (assigned 1:N in id order at entry) breaks ties when multiple measurements share the same subjid, param, and agedays (same-day duplicates). If internal_id is not included in the sort key, SDE resolution order is undefined and results may differ between sequential and parallel runs.

Internal identifiers

• line — row number in original input order; used to restore output order
• index — sequential row number after sorting; used as internal key within cleanchild()
• id — user-provided or auto-generated; returned in output only. internal_id is the tiebreaker in sorting and SDE resolution
• v — working measurement value (metric, with 0 → NaN)
• v.orig — copy of v before any transformations

Trying a sample first (gc_runtrial())

Before committing to a full run, gc_runtrial() lets you confirm your data are formatted correctly and that growthcleanr is reading your columns, sex coding, and parameters the way you intend. It draws a small sample of patients, runs cleangrowth() on them with the exact configuration you pass through (...), and prints a short diagnostic report: how the input was interpreted (format, column mapping, age range), dedicated SEX and PARAMETERS sections (the two highest-risk settings) that each show what values map to each category, any present-but-unrecognized values, and the resulting counts, then anything flagged while reading and a summary of the cleaning result. The cleaned sample is returned invisibly, with the structured diagnostic attached as attr(x, "gc_trial").

gc_runtrial(mydata)                                  # 100 random patients, default long
gc_runtrial(mydata, n = 50, seed = 1,                # reproducible 50-patient sample
            input_format = "semi_wide")

• Sampling is by patient — all of a sampled subject’s rows are kept, so trajectories stay intact. sampling = "random" (default) with an optional seed for reproducibility, or sampling = "first" for the first n subjects. Random is effectively free: the cost is running growthcleanr on the sampled patients, the same either way.
• Data-frame style only, and the trial always analyses/returns results in long format so the diagnostic is well defined regardless of output_format.
• A configuration hard-stop is reported, not re-thrown. If the setup makes cleangrowth() stop (a forgotten input_format, a sex coding it will not guess), the trial surfaces that stop as the headline finding with the same guidance the error carries — turning a hard error into setup help. This makes it a natural engine for a “preview my setup” step in a wrapping app.

Calculation method: CSD (Conditional Standard Deviation)

All z-scores in the cleaning algorithm use the CSD method, the same approach the CDC uses for its modified z-scores (Kuczmarski et al., 2002; full citation in vignette("algorithm-functions")):

If measurement < M:  sd = (measurement - M) / csd_neg
If measurement >= M: sd = (measurement - M) / csd_pos

Where:

• M = age/sex-specific median from reference table
• csd_pos = half the distance from M to the value at z = +2, precomputed in reference tables
• csd_neg = half the distance from M to the value at z = -2, precomputed in reference tables

This is NOT the standard LMS z-score used in clinical practice. CSD z-scores are intentionally more sensitive to extreme high values for weight, which helps the algorithm detect implausible measurements.

Reference tables

Z-score variants

Age blending — main z-score (sd.orig)

Note: The blending window is 2–5 years with divisor 3. At exactly 2 years, the formula yields 100% WHO. At exactly 5 years, it yields 100% CDC.

Code-verified detail: The code assigns WHO for ageyears < 2 and CDC for ageyears > 5 (strict inequality). For ages exactly 2.0 or 5.0, the smooth_val condition (>= 2 & <= 5) applies, so they go through the blending formula. This means ages exactly at the boundaries use the blending formula, which gives 100% WHO at 2.0 and 100% CDC at 5.0.

Age blending — corrected z-score (sd.corr)

For potcorr subjects, sd.corr is calculated in two stages:

Stage 1 — Compute sd.c (the corrected z-score before smoothing into original):

First, corrected WHO and CDC z-scores (sd.c_who, sd.c_cdc) are calculated using corrected age (cagedays) with an adjustment for standing vs. recumbent position when chronological age is 730 days or more but corrected age is under 730 days (+0.7 cm for both WHO and CDC; see Phase 8, fact 6).

Then sd.c is assigned using WHO/CDC blending with the same 2–5 year window as sd.orig:

• HEADCM: WHO at all ages
• HT/WT age ≤ 2: WHO
• HT/WT age 2–5: weighted blend (who × (5-age) + cdc × (age-2)) / 3
• HT/WT age ≥ 5: CDC

Then the gestational-age reference z-score is preferred over corrected WHO for ages ≤ 2: if it (unmod_zscore) is available, it overwrites sd.c; corrected WHO is used only as fallback.

Stage 2 — Smoothing corrected into original:

Note: This smoothing window is 2–4 years with divisor 2, which is narrower than the main WHO/CDC blending window (2–5 years, divisor 3).

Age blending — calc_and_recenter_z_scores() (Child Step 15/16 p_plus / p_minus)

This helper function, used in the Child Step 15/16 pre-loop (EWMA2 / moderate trajectory outliers), recalculates recentered z-scores for each row’s perturbed p_plus and p_minus measurement values. It uses the same WHO/CDC blending formula as the main z-score calculation (2–5 year window) so that the perturbed z-scores are on the same footing as tbc.sd.

Recentering

After z-score calculation, the algorithm subtracts age-, sex-, and parameter-specific median z-scores to produce tbc.sd:

tbc.sd = sd.orig - rcmedian.sd
ctbc.sd = sd.corr - rcmedian.sd

Recentering uses a precomputed reference file (inst/extdata/rcfile-resmoothed.csv.gz) whose medians were built offline by smoothing per-day median SD scores from large EHR samples (see vignette("recentering") for the derivation). Callers can override by passing a sd.recenter data.table or data.frame to cleangrowth().

The recentering median file is indexed by param, sex, and agedays. It is merged into the data by an exact join on these keys.

Required schema for a custom sd.recenter table. A user-supplied override must be a data.table or data.frame (converted to a data.table) with these columns:

A median column named sd.median is also accepted. growthcleanr works on a copy of the supplied table (the caller’s object is not keyed or renamed) and sets the key itself. The table must have exactly one row per param, sex, and agedays (duplicates stop with an error), and a median for every param, sex, and agedays of the child measurements to be cleaned, meaning every child row that is not already set aside (missing or unusable values) and is not head circumference at 5 years or older. Adult rows are not recentered. If any such row has no median (including an NA median), cleangrowth() stops before cleaning, reports how many measurements are uncovered, and lists up to five example param/sex/agedays combinations. Anything other than NA or a table (for example a character string) also stops with an error.

Recentering limitation. Recentering has a small but sometimes meaningful impact on cleaning results. The provided recentering data are based on growth from a single large health system. Some datasets may have different growth patterns. Users may wish to use their own data to create a recentering reference. If doing so, we would suggest creating smoothed medians using data from your own study population or a similar population, and would caution about bias related to small sample sizes and overfitting.

Input and identification

Z-score variables

Prematurity correction variables

Overview

Very preterm subjects whose anthropometric references should be preterm growth standards are identified as “potentially correctable” (potcorr) and have their z-scores recomputed against gestational-age reference values using postmenstrual age. The correction is only kept if true prematurity is corroborated, by EITHER a consistent early weight trajectory (the corrected early-weight spread is no worse than uncorrected — the abssumdiff check) OR a same-day height or head circumference whose own uncorrected z is <= -1.5 and no more than 3.5 z above the first weight’s uncorrected z. A potcorr subject corroborated by neither is reverted (uncorr := 1, sd.corr := sd.orig) — this is what prevents a misplaced-decimal birth weight (low weight, normal same-day length, no early follow-up) from being hidden by a false correction. All three columns (sd.corr, uncorr, potcorr) are written back to data.all for every row; downstream steps read them through ctbc.sd and related recentered fields. Full mechanics are in Phase 4 below.

Gestational-age reference values

The gestational-age correction uses two tables of approximated reference values derived from the printed Fenton (2025) charts:

Fenton TR, Elmrayed S, Alshaikh BN. Fenton Third-Generation Growth Charts of Preterm Infants Without Abnormal Fetal Growth: A Systematic Review and Meta-Analysis. Paediatr Perinat Epidemiol. 2025 Aug;39(6):543-555. doi: 10.1111/ppe.70035. Epub 2025 Jun 19. PMID: 40534585; PMCID: PMC12391854.

These values are approximations derived from the charts, not values published by the Fenton authors. growthcleanr does not use LMS values for the Fenton reference.

• Approximated Fenton CSD values (inst/extdata/fenton2025_ms_lookup_smoothed.csv): for each sex, measurement (weight, length, head circumference), and gestational-age day from 157 to 350 days (22+3 to 50 weeks), the median (M), S_upper, and S_lower. S_upper × M is half the distance between the median and +2 SD, and S_lower × M is half the distance between the median and −2 SD. These give the Fenton CSD z-score in Phase 3, step 6. Length values below 23+3 weeks (157–163 days) were extended by linear extrapolation from the 23+3 to 24+3 week values, so length covers the same range as weight and head circumference (data-raw/extend_fenton_length.R, in the source repository).
• Approximated Fenton median weights by gestational age (inst/extdata/fent_foraga.csv.gz): a table of approximated Fenton median weights by gestational age, derived from the printed Fenton (2025) charts. It gives, for each sex and weight (grams, rounded to 10 g; 500–5,850 g for males and 500–5,370 g for females), the gestational age in days at which that weight is the median (Phase 3, step 2).

Phase 1: potcorr identification

Subjects are flagged by a two-level rule.

• Sort. Sort by (subjid, param, agedays, id_sort) where id_sort = internal_id at agedays == 0 and -internal_id otherwise. At birth this keeps the earliest-recorded row as the “first” weight; at later ages it keeps the most recent internal_id — the age-dependent tiebreaker used throughout the algorithm.
• potcorr_wt (row-level): a weight row is potcorr_wt = TRUE iff (a) it is the first weight row for that subject under the sort above (seq_len(.N) == 1L with by = subjid), (b) sd.orig < -2, and (c) age is less than 10 months. NA-safe: sd.orig is NA for Exclude-Missing rows so they never flag.
• potcorr (subject-level): any(potcorr_wt, na.rm = TRUE) per subject. Propagated to all rows of that subject.

Phase 2: fast path when no potcorr

has_potcorr <- any(data.all$potcorr, na.rm = TRUE). If FALSE, Child Step 2b short-circuits: sd.corr := sd.orig, uncorr := 0. Gestational-age reference files are not read. This typical-case fast path matters because potcorr subjects are often not present or comprise < 1 % of a dataset.

Phase 3: Gestational-age correction (potcorr subjects only)

All correction work happens on a copy pc <- data.all[subjid %in% pc_ids].

1. Integer weight. intwt := trunc(v * 100) * 10 (grams rounded to 10 g); values in [100, 500) are floored to 500 to hit the reference table’s minimum.
2. Reference merge #1 — weight → GA. Merge pc with fent_foraga (from inst/extdata/fent_foraga.csv.gz) on (sex, intwt) to get fengadays (the gestational age in days for which that weight (rounded to 10g) is the median).
3. Propagate. Use the qualifying first-weight row’s fengadays for every row of the subject (fengadays_subj := min(fengadays[potcorr_wt], ...), by subjid).
4. Ages. pmagedays := agedays + fengadays (postmenstrual age); cagedays := pmagedays - 280 (corrected age). Both computed only when fengadays < 259 (term cutoff).
5. Reference merge #2 — GA → M/S. Map gc param names (WEIGHTKG/HEIGHTCM/HEADCM) to the reference table’s param names (weight/length/headcirc), then merge with fenton2025_ms_lookup_smoothed.csv on (sex, fengadays, param) to pull M, S_upper, S_lower.
6. Gestational-age CSD z-score. v_fenton is v * 1000 for weight and v otherwise. unmod_zscore := (v_fenton − M) / (S_upper * M) if v_fenton >= M, else / (S_lower * M). This uses the same CSD method as WHO/CDC elsewhere in the algorithm.
7. Correction failure. If the reference merge failed (no M), reset potcorr_wt to FALSE for that row and recompute the subject-level potcorr. A subject whose weight is outside the reference table’s range silently falls out of the correction.
8. Initial sd.corr. Rows with unmod_zscore non-NA get sd.corr := unmod_zscore. Rows still NA fall back to sd.orig.
9. Corrected WHO/CDC z-scores. Compute sd.c_who and sd.c_cdc via the in-memory measurement.to.z_who / measurement.to.z closures at cagedays. For HEIGHTCM with chronological age at or past day 730 and corrected age before day 730, apply the recumbent/standing offset (+0.7 cm for both WHO and CDC, the same offset length.adjust uses) before the z-score call. This is done because the formal recommendation is for linear growth to be measured as length <2y and height >=2y age. Since we are converting chronologic to corrected age, if recommendations are followed we need to account for the length-height discrepancy (length is higher than height). We need a consistent way of handling this issue, but we recognize that practices vary widely regarding length-height measurement and documentation. Most health systems do not provide metadata re length vs. height in extracted EHR data, and metadata that are provided are not necessarily reliable. Under the default, window values are presumed recumbent, so cleangrowth() scores them 0.7 cm lower for the uncorrected z (and passes that same lowered value here); adding the offset back means the corrected z uses the entered value. Full rules: Phase 8, fact 6.
10. sd.c (corrected blend). HEADCM uses WHO at all ages. Other params: WHO for ageyears_2b ≤ 2, smoothed WHO→CDC blend for 2 < ageyears_2b < 5, CDC for ageyears_2b ≥ 5.
11. GA reference takes precedence under 2. For ageyears_2b ≤ 2 when available, sd.c := unmod_zscore.
12. Final sd.corr. Age-dependent:
    • ageyears_2b ≤ 2 with potcorr: sd.c
    • 2 < ageyears_2b ≤ 4: smoothed blend (sd.orig * (4 − ageyears_2b) + sd.c * (ageyears_2b − 2)) / 2
    • otherwise: sd.orig
    • Note that smoothing age intervals are intentionally different: CDC/WHO: 2-5 years; corrected/uncorrected: 2-4 years

Phase 4: uncorr check (revert when correction worsens trajectory)

Only weight rows of potcorr subjects participate. A temporary tmp copy is built:

• First, all but one copy of identical values for the same subject/parameter/ageday (aka SDE-Identicals) are filtered via the age-dependent keep_id rule (match Early Child Step 13 behavior).
• seq_win := sequence(.N) per subject; keep rows with seq_win ≤ 4 AND ageyears_2b < 2.
• Drop subjects that have only one value or no non-NA z-scores.
• Compute sd.corr_abssumdiff := |sum(sd.corr[1] − sd.corr)| and sd.orig_abssumdiff := |sum(sd.orig[1] − sd.orig)| per subject.

Corroboration gate. A potcorr subject keeps its correction only if true prematurity is corroborated, by EITHER:

• form 1 — a consistent early weight trajectory: it has ≥ 2 weights < 2y and sd.corr_abssumdiff <= sd.orig_abssumdiff (correction does not scatter the early points more than no-correction), OR
• form 2 — a same-day height or head circumference (a HEIGHTCM or HEADCM row at the qualifying first weight’s agedays) whose own uncorrected z (sd.orig) is <= -1.5 and no more than 3.5 z above the first weight’s uncorrected z: an infant who is short (or small-headed) as well as light, consistent with real prematurity. The 3.5 cap prevents a merely-somewhat-low height/HC from corroborating a far-lower (e.g. misplaced-decimal) birth weight. Birth values are not recentered, so sd.orig here is effectively the same as the recentered tbc.sd would be.

A potcorr subject corroborated by neither is reverted (sd.corr := sd.orig, uncorr := 1). This reverts a subject whose corrected early trajectory is more scattered than uncorrected (form-1 failure) and a subject with no early follow-up to corroborate (e.g. a misplaced-decimal birth weight: low weight, normal same-day length, next weight only > 2y), while form 2 keeps real short/small-headed preterms corrected. Withdrawing a false correction leaves the decimal-error birth uncorrected (very low z), so the trajectory steps exclude it instead of being fooled by a normalized ctbc. Because form 1, height, and HC are ORed, HC only decides the outcome when there is no early-weight corroboration and no confirming height.

Phase 4b: GA-window recumbent presumption (default only)

After the uncorr check, and only when length.adjust = FALSE, height rows of a still-corrected subject with chronological age at or past day 730 and corrected age before day 730 are presumed recumbent: their lin_off, p_plus and p_minus are lowered by 0.7 cm, their uncorrected z-scores (sd.orig_who, sd.orig_cdc, sd.orig) are rescored, and their corrected z-scores are recomputed (which adds the 0.7 cm back, so the corrected z uses the entered value). Subjects reverted in Phase 4 are not adjusted. v is not changed. See Phase 8, fact 6.

Phase 5: merge results back

data.all[, sd.corr := sd.orig] and uncorr := 0L baseline. For GA-window rows (Phase 4b), lin_off, p_plus, p_minus and the rescored sd.orig_who/sd.orig_cdc/sd.orig are written back first; then the pc results are merged back. Both joins key on internal_id (the user’s id is output-only):

data.all[pc_result, on = "internal_id", `:=`(
  sd.corr = i.sd.corr,
  uncorr = i.uncorr,
  potcorr = i.potcorr,
  sd.corr_plus = i.sd.corr_plus,
  sd.corr_minus = i.sd.corr_minus
)]

Temporary columns (agemonths, ageyears_2b, potcorr_wt, etc.) are dropped before the data flows into recentering.

read_anthro(path = "", cdc.only = FALSE)

Reads CDC and WHO growth-chart reference tables from inst/extdata/ (growthfile_cdc_ext_infants.csv.gz and growthfile_who.csv.gz) and returns a closure that, given (param, agedays, sex, measurement, csd), computes the appropriate CSD z-score for that observation. Two flavors are loaded based on cdc.only: TRUE returns a CDC-only mtz_cdc_prelim closure; FALSE returns a WHO-augmented mtz_who_prelim closure that uses WHO under 5 years and CDC at 5 years and above. path is an optional override for the reference directory (default "" → installed package’s inst/extdata/).

The closure approach lets the reference tables be resolved at load time (one disk read) and reused across many calls (one function call per row). It’s the foundation of gc_preload_refs().

gc_preload_refs(path = "")

Preloads both reference closures via two read_anthro() calls and returns them as list(mtz_cdc_prelim, mtz_who_prelim). Pass the returned list as cleangrowth(..., ref_tables = refs) to skip the per-call disk reads (~0.9 sec saved per call). The closures are session-lifetime — rebuild only if the reference files themselves change. See Batching and Dispatch → Partial runs and preloaded references above.

Note that velocity reference tables (tanner_ht_vel, who_ht_vel_for_age, who_hc_vel_for_age) are NOT included in gc_preload_refs() and are loaded separately (the Tanner table once per cleangrowth() call, the WHO velocity tables once per batch inside cleanchild()). All three are tiny so the loads are fast.

Procedure:

1. Assign each input row its integer year-of-age (ageyears = floor(agedays / 365.25)).
2. Cap ageyears at 19 so all observations at age 19 or older are pooled into a single terminal year.
3. For each (param, ageyears) cell, compute median(sd.orig) pooled across sexes (rows with is.na(sd.orig) are dropped first).
4. Treat each per-year median as anchored at midyear-age in days — floor((ageyears + 0.5) * 365.25), i.e. year 0 → day 182, year 1 → day 547, …, year 19 → day 7122.
5. Linearly interpolate rcmedian.sd by day of age between midyear anchors (approx()) onto the full day range from floor(min(ageyears) * 365.25) through floor(max(ageyears + 1) * 365.25).
6. For days outside the interpolation range, clamp to the nearest midyear median (approx(..., rule = 2) — constant extrapolation to both endpoints).
7. Replicate the resulting (param, agedays) → rcmedian.sd map for both sex = 0 and sex = 1.

Sex is carried as a column on the output for downstream join compatibility with the main algorithm’s setkey(sd.recenter, param, sex, agedays) exact join, but the underlying medians are not sex-stratified. A caller who wants sex-specific recentering can run the same procedure manually per sex.

get_dop(param_name)

Returns the designated other parameter (DOP) for a given param. WEIGHTKG → HEIGHTCM, HEIGHTCM → WEIGHTKG, HEADCM → HEIGHTCM. Used by the child algorithm in same-day duplicate resolution and cross-parameter plausibility checks. Cheap; called inline wherever a DOP is needed.

Sorting and identifiers

• Sort-key contract. setkey(data.df, subjid, param, agedays, internal_id) is assumed throughout the pediatric branch (and the analogous adult sort key inside cleanadult()). If you add or modify rows in preprocessing, call setkey() again or downstream tiebreakers will be undefined.
• internal_id is the algorithm identifier; user id is not. Algorithm logic must use internal_id for sorting and tiebreaking. The user’s id (snapshotted as user_id_orig at entry) is restored to the output id column at the Phase 15 trim when a regen replaced it; otherwise it is the same vector and the restore is skipped (the positional restore is correct because all_results is back in input row order at that point). Length mismatch between id and measurement still raises a hard stop() (no sensible recovery). Absent / NA / non-unique id does not error — it warns and auto-regens; see Phase 3 + the display_gc_id output switch.
• gc_partial_run() orders its output by user id. Because it merges freshly cleaned rows with rows carried over from a cache, it sorts the combined result by id (which must be unique and stable across runs) rather than by input row position — so a partial-run result may be in a different row order than the same data through a single cleangrowth() call, though the per-subject rows are identical.

Parallel execution

• parallel = TRUE requires installed package. Will fail with load_all() only — workers need system.file() access to inst/extdata/. Install with devtools::install_local(".") first.
• Cluster export list must include all algorithm-internal helpers. When adding a new helper that’s called from inside cleanchild() or cleanadult(), add it to the clusterExport() list in Phase 5 — otherwise parallel runs will fail with “object not found” inside workers but serial runs will pass.
• Batch-invariant setup belongs outside the outer loop. Loading exclude.levels, the Tanner table, and any other per-run-but-not-per-batch constants once before the loop saves real time on multi-batch runs.

Factor and code conventions

• Factor pitfall. Assigning an unlisted string to the exclude factor silently produces NA. Always verify new exclusion codes exist in exclude.levels first. The exclude.levels factor is built from exclude.levels.peds + exclude.levels.adult in Phase 4.
• Duplicate function definitions. R loads all .R files in R/ alphabetically. If two files define the same function, the later one wins silently. Do not duplicate .child_valid() (defined in child_clean.R) elsewhere. Only one adult support file should exist in R/ for the same reason — duplicate files have caused silent overwrites of check_between / round_pt in the past.
• Adult result vs child exclude column names. cleanadult() uses result as its exclusion column; cleangrowth() maps it to exclude in the combined output (Phase 14). Do not rename result inside cleanadult().
• Adult rounding tolerance (0.12 cm/kg) is intentional and should not be removed. It allows for rounding to nearest half-cm and floating point precision.

Cross-algorithm assembly

• Exclude-Missing is set at different times for child vs adult. Child: pre-dispatch in cleangrowth() (because .child_valid() needs it pre-set). Adult: post-dispatch. Don’t move either side without understanding why.
• Exclude-C-Temp-Same-Day must be resolved before output. The post-loop guard (Phase 15) issues a warning() and reclassifies any survivor to Exclude-C-Extraneous rather than halting. If a new step introduces or preserves temp-SDE codes, ensure Child Step 13 (or an explicit cleanup) resolves them.
• Cross-algorithm columns are NA-filled. Adult rows have NA for tbc.sd, ctbc.sd, cf_rescued, etc.; child rows have NA for mean_ht, bin_result. Downstream consumers must handle this.

Reference data

• Reference-table loading is gated by ref_tables. When set, disk reads are skipped — but the closures must still be valid for the current input. Rebuild gc_preload_refs() if the reference files change.
• HC cleaning extends to 5 years. Phase 11 sets Exclude-Not-Cleaned (reported as Exclude-HC-Out-of-Range) for HC at agedays >= 1827, because the WHO HC z-score reference ends at 5 years. The recentering file provides HC rcmedian.sd through 5 years. HC measurements through 5 years are cleaned by all standard steps; the velocity check (Child Step 17) applies only mindiff = -1.5 with no upper bound for HC pairs spanning ages 24 months to 5 years, since the WHO HC velocity reference ends at 24 months.