vectra 0.5.1

CRAN resubmission fixes (0.5.0 incoming pretest feedback)

• configure / configure.win: rewritten as POSIX /bin/sh (previously #!/usr/bin/env bash with set -o pipefail and [[ ... ]]). Bash is not guaranteed on all CRAN build hosts.
• src/window.c: the OpenMP task-parallel merge sort helper was defined unconditionally but called only from #ifdef _OPENMP branches, producing a clang -Wunneeded-internal-declaration warning under Debian’s no-OpenMP build. The definition now shares the guard.
• Vendored tdc: all fprintf(stderr, ...) debug/timing prints are routed through a TDC_LOG(...) macro that is a no-op unless TDC_ENABLE_STDERR_LOG is defined at build time, so the released .so contains no stderr / fprintf symbols. Addresses the WRE §1.6.4 policy forbidding compiled code from writing to stdout/stderr.

Fixes

• collect(): fix use-after-free in the cross-batch CHARSXP dedup cache. Each slot stored a raw pointer into the decoder’s heap buffer, which is freed when the batch is consumed; the next batch’s hash-collision memcmp then dereferenced freed memory. Manifested as segfaults on the second consecutive collect() of a large multi-rowgroup string-heavy .vtr (register, backbones), more likely under the parallel reader where batches accumulate before the serial consumer loop. Now verifies cache hits against CHAR(sexp), which points into the still-alive interned CHARSXP body.

vectra 0.5.0

Compression backend rewire

• Replaced the bespoke v4 codec with tdc, a standalone typed-dimensional compression library vendored into src/tdc/. Encode and decode go through a self-describing block record (model + transform chain + entropy) rather than per-column tag constants. Deleted vtr_codec.c, vtr_encodings.c, vtr_compress.c, vtr1.c, and vtr_codec_internal.h.
• The .vtr on-disk format is a deliberate breaking change: pre-0.5 files are not readable. write_vtr() and append_vtr() write the new container; tbl() reads only the new container.
• Per-row-group column statistics (min/max) are carried in the container index so the scan layer can still prune unreachable row groups.
• Parallel row-group reads are preserved.
• Custom vendoring via tools/vendor_tdc.sh and configure / configure.win pull the latest upstream tdc on every install when the source checkout is present; the pre-vendored copy is used otherwise.

Known regression

• The v4 dict-defer CHARSXP fast path is gone — duplicate strings now hit R’s CHARSXP hash per row. Will be re-implemented on top of tdc’s dictionary-encoded varlen output when it becomes a hot spot.

Fixes

• man/write_vtr.Rd: replaced a literal percent sign in the compress argument description that produced malformed Rd output on build.
• Windows: write_vtr(), append_vtr() and delete_vtr() now use MoveFileEx with a short retry loop for the final temp-to-target swap. Previously, a preceding tbl() read could leave the target file mmap’d pending GC, and the swap would fail with a sharing violation.

vectra 0.4.1

Star schema and lookup

• New vtr_schema(), link(), and lookup() functions for star-schema workflows. Register a fact table with named dimension links once, then pull columns from any dimension without writing explicit joins. Only referenced dimensions are scanned.
• lookup() reports unmatched keys per dimension by default, catching referential integrity issues before they propagate NAs silently.
• Supports both "left" (default) and "inner" join modes, named keys for differing column names, and reusable schema objects across multiple queries.

vectra 0.3.2

• Fix misaligned int64_t memory access in vtr_codec.c (UBSAN). Dictionary encoding wrote and read 8-byte offsets through an unaligned pointer; delta decoding had the same issue. All fixed with memcpy.

vectra 0.3.1

• CRAN submission fixes: title case, quoted technical terms in DESCRIPTION, corrected documentation URLs.

vectra 0.3.0

File operations

• append_vtr(df, path): append a data.frame as a new row group to an existing .vtr file. Existing row groups are never rewritten.
• delete_vtr(path, row_ids): logically delete rows by 0-based physical index. Writes a tombstone side file (<path>.del); the .vtr file is never modified. Deletions are cumulative and excluded automatically on the next tbl() call.
• diff_vtr(old_path, new_path, key_col): key-based logical diff between two .vtr files. Returns a list with added (a lazy vectra_node) and deleted (a vector of key values). Implemented as a single-pass C streaming engine with O(n_unique_keys) memory.

Expressions

• tolower(), toupper(), trimws(): case conversion and whitespace trimming for string columns in filter() and mutate().
• levenshtein(x, y) / levenshtein_norm(x, y): Levenshtein edit distance and normalised variant (0–1). Supports column-vs-column and column-vs-literal comparisons. Optional max_dist argument for early termination.
• dl_dist(x, y) / dl_dist_norm(x, y): Damerau-Levenshtein distance (counts transpositions as cost 1) and normalised variant.
• jaro_winkler(x, y): Jaro-Winkler similarity (0–1, higher = more similar). All string-similarity functions propagate NA and work in filter() and mutate().
• resolve(fk, pk, value): scalar self-join — looks up value where pk == fk within the same batch. Useful for denormalising parent-child tables without a join.
• propagate(parent_id, id, seed): tree-traversal aggregation — propagates non-NA seed values down a parent-child hierarchy until all reachable nodes are filled. Converges in O(depth) passes.

Format

• .vtr format version 4 with a two-layer codec (no external dependencies):
  • Encoding: PLAIN (default), DICTIONARY (string columns with < 50% unique values), DELTA (monotonically increasing int64 columns).
  • Compression: custom LZ77 byte compressor (LZ_VTR, ~120 lines of C). Applied after encoding; skipped for buffers < 64 bytes or when compression does not reduce size. Files written with v4 are typically 30–60% smaller than v3. tbl() reads v1–v4 files; write_vtr() always writes v4.

vectra 0.2.2

Query optimizer

• Column pruning: scan nodes only read columns needed by the query plan.
• Predicate pushdown: filter predicates are attached to scan nodes and use .vtr v3 per-rowgroup min/max statistics to skip entire row groups.

Engine

• .vtr format version 3 with per-column per-rowgroup statistics (min/max).
• O(n log n) rank() and dense_rank() (replaces O(n²) comparison-based).
• Nested expressions in summarise(): summarise(m = mean(x + y)) auto-inserts a hidden mutate.

Expressions

• year(), month(), day(), hour(), minute(), second(): date/time component extraction for Date and POSIXct columns.
• as.Date() and as.POSIXct() literals in filter expressions (e.g. filter(date > as.Date("2020-01-01"))).
• as.Date(string_col): convert ISO-format date strings to Date values.
• nchar(): returns string length as integer.
• substr(x, start, stop): substring extraction (1-based, like R).
• grepl(pattern, x): fixed string matching (no regex).
• paste0(a, b): two-argument string concatenation.
• gsub(pattern, replacement, x) / sub(): fixed-string replacement.
• startsWith() / endsWith(): string prefix/suffix matching.
• pmin() / pmax(): element-wise minimum/maximum.
• log2(), log10(), sign(), trunc(): additional math functions.

Aggregation

• sd() and var(): sample standard deviation and variance via Welford’s online algorithm. Returns NA for groups with fewer than 2 values (R semantics).
• first() and last(): first and last non-NA value per group. Both support na.rm = TRUE.

Verbs

• slice_min() and slice_max() gain a working with_ties parameter (default TRUE). Ties at the boundary are now included by default; use with_ties = FALSE for exactly n rows.
• count() and tally() gain a working sort parameter. sort = TRUE returns results in descending order of the count column.
• transmute() and reframe() now support across().
• distinct(.keep_all = TRUE) with a column subset now emits a message when falling back to R.

Utilities

• glimpse(): preview column names, types, and first few values without collecting the full result.
• collect() now works on data.frames (no-op), so slice_min(...) |> collect() works regardless of the with_ties path.

Documentation

• New quickstart vignette: vignette("quickstart").
• @details sections added to filter(), mutate(), summarise(), arrange(), distinct(), count(), and join functions.

vectra 0.2.1

Engine

• External merge sort with 1 GB memory budget and automatic spill-to-disk.
• Sort-based group_by() |> summarise() path for spill-safe aggregation.
• Chunked FULL join finalize (65,536 rows per batch).
• Automatic type coercion (int64 <-> double) in join keys and bind_rows().
• rank() and dense_rank() window functions.

Type system

• .vtr format version 2 with per-column annotations.
• Date, POSIXct, and factor columns roundtrip through write_vtr() / collect().
• where() predicates work in select(), rename(), relocate(), and across().

Infrastructure

• Engine reference vignette (vignette("engine")).
• 17-scenario benchmark suite with baseline snapshots and regression thresholds.
• ASAN/UBSAN CI job on Linux.
• Benchmark smoke job on PRs.

vectra 0.1.0

• Initial release.
• Custom columnar on-disk format (.vtr) with multi-row-group support.
• dplyr-compatible verbs: filter(), select(), mutate(), transmute(), rename(), relocate(), group_by(), summarise(), count(), tally(), distinct(), reframe(), arrange(), slice_head(), slice_tail(), slice_min(), slice_max(), pull().
• Hash joins: left_join(), inner_join(), right_join(), full_join(), semi_join(), anti_join().
• bind_rows() and bind_cols() for combining queries.
• Window functions: row_number(), lag(), lead(), cumsum(), cummean(), cummin(), cummax().
• across() support in mutate() and summarise().
• explain() for inspecting the execution plan.
• tidyselect integration for column selection helpers.
• Data sources: .vtr, CSV, SQLite, GeoTIFF.
• Data sinks: write_csv(), write_sqlite(), write_tiff().