Package {sidrar}

Get a SIDRA table

Description

Retrieves aggregate data from the Brazilian Institute of Geography and Statistics (IBGE) SIDRA API.

Usage

get_sidra(
  x,
  variable = "allxp",
  period = "last",
  geo = "Brazil",
  geo.filter = NULL,
  classific = "all",
  category = "all",
  header = TRUE,
  format = 4,
  digits = "default",
  api = NULL,
  value_type = c("numeric", "character", "both"),
  geo_view = NULL,
  include_extinct = FALSE
)

Arguments

Details

Supported values of geo are "Brazil", "Region", "State", "IntermediaryRegion", "ImmediateRegion", "MesoRegion", "MicroRegion", "MetroRegion", "MetroRegionDiv", "IRD", "UrbAglo", "PopArrang", "City", "District", "subdistrict", and "Neighborhood". Their corresponding nNN codes and all aliases are accepted without regard to letter case.

format = 1 returns codes, format = 2 returns names, format = 3 returns codes and names for geographic units plus names for other descriptors, and format = 4 returns codes and names for all descriptors.

Requests use HTTPS, UTF-8 decoding, a timeout, and limited retries for transient failures. Set options(sidrar.timeout = 120) or options(sidrar.retries = 4) to override their defaults. Responses are requested live and are not cached by the package. HTTP 429 and 503 responses honor a valid Retry-After delay (seconds or HTTP date). Set options(sidrar.retry_after_max = 120) to change the maximum server-requested delay accepted for another attempt (default: 60 seconds). The option must be one finite positive number; invalid settings use 60. If the requested delay exceeds that limit, a sidrar_retry_after_error (also a sidrar_http_error) carries retry_after and retry_after_max, rather than retrying before the server permits it. This limit does not change the per-attempt timeout or the total number of attempts.

HTTP conditions inherit from sidrar_http_error and carry status_code, response_body, and url. Transport failures may additionally inherit from sidrar_timeout_error, sidrar_tls_error, sidrar_dns_error, sidrar_connection_error, or sidrar_transient_error when the underlying failure can be identified conservatively.

Cloudflare browser challenges raise sidrar_challenge_error, which inherits from sidrar_http_error and also carries cf_ray when available. For compatible values queries, the package retries through IBGE's official aggregate API v3 with view=flat. This fallback supports multiple geographic levels, explicit periods and ranges, all, first, and last selections, standard variable/category selections, and the default descriptor format. Dimension columns follow the original URL, including when variable precedes period; observation order remains that returned by the alternative service. Explicit decimal precision is accepted only when numeric values already have the requested decimal places. Otherwise sidrar_fallback_precision_error (also a sidrar_parse_error) is raised: the alternative cannot reconstruct unavailable precision, and values are not rounded or padded. The default precision preserves values as received; maximum precision is unsupported. Unsupported selections retain the original challenge error with a fallback_reason field. Set options(sidrar.fallback = FALSE) to disable this alternative route. If the alternative request fails, its error carries primary_error with the original challenge. Availability still depends on IBGE; increasing retries does not solve a browser challenge. Automatic classification discovery (classific = "all") can use official aggregate metadata if SIDRA's table descriptor returns a browser challenge.

Alternative responses are checked for complete dimension fields, textual identifiers, duplicate observation keys, and membership in explicit filters. Violations raise sidrar_parse_error subclasses and retain primary_error. Missing explicitly selected members produce sidrar_incomplete_warning, whose missing field identifies them, without adding or removing rows. This warning does not prove truncation: sparse tables can legitimately omit observations. No Cartesian product is required. Full coverage of all, the exact membership of first/latest selections, and containing-level geographic filters require catalog or territorial metadata comparisons and are not inferred by these local checks.

When SIDRA rejects a query for exceeding its per-request value limit, get_sidra() raises a sidrar_limit_error, which also inherits from sidrar_http_error. The condition records requested_values, limit_values, and minimum_batches. Use sidra_split() to split a URL or structured query across disjoint calls, or sidra_collect() with batch_size for opt-in period batching and checkpoint for resumable downloads. get_sidra() itself does not split requests or save values.

The SIDRA API uses special value symbols. With the default value_type = "numeric", non-numeric symbols such as "-", "X", "..", and "..." become NA, as in earlier versions. Use value_type = "character" or "both" when those distinctions matter.

Geographic identifiers returned by SIDRA should be kept as character strings. In particular, "Neighborhood" (n102) identifiers belong to SIDRA's territorial level and are not census tract identifiers; do not join them directly without an official correspondence.

Value

A base data.frame.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

info_sidra(), search_sidra(), sidra_query(), and sidra_collect()

Examples

## Not run: 
get_sidra(
  x = 7060,
  variable = 63,
  period = c(last = 12),
  geo = "City",
  geo.filter = list(State = 50),
  classific = "c315",
  category = list(7169)
)

get_sidra(
  api = "/t/7060/n1/all/v/63/p/last/c315/7169/h/n"
)

## End(Not run)

List the parameters of a SIDRA table

Description

Uses the official JSON table descriptor to return variables, periods, classifications, categories, and geographic levels available in a table.

Usage

info_sidra(x, wb = FALSE)

Arguments

Details

Metadata is requested live from the official descriptor endpoint and is not cached by the package. The timeout and retry options described in get_sidra() also apply. If the descriptor returns a recognized Cloudflare challenge, the official aggregate API v3 metadata and periods are used unless options(sidrar.fallback = FALSE) is set. This alternative preserves the five legacy components but cannot supply descriptor-specific geographic names, active-unit counts, or variable availability exceptions. Geographic descriptions label the unavailable counts explicitly; no zero counts or availability ranges are inferred. The sidrar_metadata attribute records the alternative source URLs and unavailable fields. Periods are listed as codes, without assuming continuous coverage. Missing variable units are recorded in this attribute and omitted from their descriptions.

Value

When wb = FALSE, a list with components table, period, variable, classific_category, and geo. When wb = TRUE, the descriptor URL is returned invisibly after the browser is opened.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

get_sidra() and sidra_metadata()

Examples

## Not run: 
info_sidra(7060)
info_sidra(7060, wb = TRUE)

## End(Not run)

Search SIDRA tables

Description

Searches table titles in the official IBGE aggregate catalog.

Usage

search_sidra(x)

Arguments

Details

Matching is case- and accent-insensitive. When x contains multiple terms, every term must occur in the title, but the terms need not be adjacent. The catalog is requested live and is not cached by the package. The timeout and retry options described in get_sidra() also apply.

Value

A named character vector with matching SIDRA table titles. Names are the table codes. Returns character(0) when there are no matches.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

get_sidra(), info_sidra(), and sidra_catalog()

Examples

## Not run: 
search_sidra("contas nacionais")
search_sidra("IPCA")

## End(Not run)

Clear the sidrar metadata cache

Description

Removes only cache entries created by sidrar; the cache directory itself and unrelated files are preserved.

Usage

sidra_cache_clear(cache_dir = NULL)

Arguments

Value

The number of removed entries, invisibly.

See Also

sidra_cache_info()

Inspect the sidrar metadata cache

Description

Lists entries created explicitly by the discovery functions when their cache argument is enabled. Value responses downloaded by get_sidra() are never cached by these helpers.

Usage

sidra_cache_info(cache_dir = NULL)

Arguments

Value

A data frame with one row per sidrar cache file: normalized key, absolute path, byte size, fetched_at, expires_at, and valid. valid is TRUE only for a structurally readable, unexpired entry whose stored key matches its file name.

See Also

sidra_cache_clear()

Discover SIDRA tables

Description

Returns the official IBGE aggregate catalog as a rectangular table while retaining the research group to which each SIDRA table belongs.

Usage

sidra_catalog(refresh = FALSE, cache = FALSE, ttl = 1800, cache_dir = NULL)

Arguments

Details

The catalog is obtained from the official version 3 aggregate API. Unknown fields added by the API are ignored. Missing fields are returned as NA without changing the documented column types.

Value

A base data.frame() with columns research_id, research_name, table_id, and table_name. Identifier columns are character vectors. Research groups without tables are retained with missing table fields.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

search_sidra(), sidra_metadata()

Other SIDRA discovery functions: sidra_locations(), sidra_metadata(), sidra_periods()

Examples

## Not run: 
catalog <- sidra_catalog()
subset(catalog, grepl("prices", table_name, ignore.case = TRUE))

## End(Not run)

Collect one or more planned SIDRA queries

Description

Executes queries sequentially, checks that every batch has the same column names and types, and combines the rows without sorting or deduplicating them. Parallel requests are intentionally not used.

Usage

sidra_collect(
  x,
  value_type = NULL,
  provenance = FALSE,
  batch_size = NULL,
  checkpoint = NULL,
  resume = TRUE
)

Arguments

Details

Checkpoints are separate from the metadata cache. They preserve successful responses, including their original access times; they are not refreshed automatically and can span upstream revisions. Use a new directory for a fresh collection. Relative periods are resolved once and reused on resume, even if the current catalog has changed. Corruption, incompatible schemas, mismatched settings, and concurrent writers fail explicitly. An interrupted process may leave a .sidrar-lock directory; remove that lock only after confirming that no collector is still running. Results are still combined in memory; checkpoints are not an out-of-core database. Row order is not a key: join different extracts by identifiers. Warnings from completed batches are saved and signaled again on resume, so reuse does not hide incomplete-coverage diagnostics.

Value

A base data.frame(). When provenance = TRUE, its sidrar_provenance attribute can be read with sidra_provenance().

See Also

sidra_query(), sidra_split(), sidra_provenance()

Examples

## Not run: 
query <- sidra_query(
  7060,
  variable = 63,
  period = as.character(202401:202406),
  geo = "Brazil",
  classific = "c315",
  category = list(7169)
)
sidra_collect(sidra_split(query, "period", size = 3))

## End(Not run)

List the locations available for a SIDRA table

Description

List the locations available for a SIDRA table

Usage

sidra_locations(
  table,
  level,
  refresh = FALSE,
  cache = FALSE,
  ttl = 1800,
  cache_dir = NULL
)

Arguments

Details

Locations come from the official table- and level-specific version 3 endpoint. Location identifiers are SIDRA territorial identifiers; they must not be coerced to numeric values or assumed to be census tract codes.

Value

A base data.frame() with columns table_id, level_id, level_name, location_id, and location_name. All identifiers are character vectors.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

sidra_metadata()

Other SIDRA discovery functions: sidra_catalog(), sidra_metadata(), sidra_periods()

Examples

## Not run: 
sidra_locations(7060, "N6")

## End(Not run)

Retrieve structured SIDRA table metadata

Description

Normalizes the official version 3 metadata and period responses into a stable collection of base data frames.

Usage

sidra_metadata(
  table,
  refresh = FALSE,
  cache = FALSE,
  ttl = 1800,
  cache_dir = NULL
)

Arguments

Details

sidra_metadata() requests both the table metadata endpoint and the table-specific period endpoint. This makes periods equivalent to sidra_periods() and keeps the table's complete period inventory separate from its frequency and first/last-period summary. Within each classification, category_order is the one-based order in the official response. Together with level, it preserves the API's hierarchical preorder; categories are deliberately not sorted by identifier.

Value

A named list with six data frames. table contains table_id, table_name, research_name, subject, url, frequency, period_start, and period_end. periods has the schema documented by sidra_periods(). variables contains table_id, variable_id, variable_name, unit, and the character list-column summarized_by. classifications contains table_id, classification_id, classification_name, summarizable, and the character list-column summarization_exceptions. categories contains table_id, classification_id, category_id, category_name, unit, level, and category_order. geographies contains table_id, group, and level_id. All API identifiers are character vectors; no raw JSON object is exposed.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

sidra_catalog(), sidra_periods(), sidra_locations()

Other SIDRA discovery functions: sidra_catalog(), sidra_locations(), sidra_periods()

Examples

## Not run: 
metadata <- sidra_metadata(7060)
metadata$variables
metadata$geographies

## End(Not run)

List the periods available for a SIDRA table

Description

List the periods available for a SIDRA table

Usage

sidra_periods(
  table,
  refresh = FALSE,
  cache = FALSE,
  ttl = 1800,
  cache_dir = NULL
)

Arguments

Details

Data come from the official table-specific version 3 period endpoint. Period identifiers are kept as character strings so leading zeros and non-calendar codes cannot be lost.

Value

A base data.frame() with columns table_id, period_id, period_name, alternative_names, and modified. The alternative_names column is a base list-column of character vectors.

Author(s)

Renato Prado Siqueira rpradosiqueira@gmail.com

See Also

sidra_metadata()

Other SIDRA discovery functions: sidra_catalog(), sidra_locations(), sidra_metadata()

Examples

## Not run: 
sidra_periods(7060)

## End(Not run)

Estimate the size of a SIDRA query

Description

Creates an offline plan from a sidra_query() object or official SIDRA values URL. Exact cardinalities are calculated only when they follow from explicit selections in the URL. Special selections such as all, allxp, first, and last, territorial views and containment filters remain unknown until metadata is available.

Usage

sidra_plan(query, limit = NULL)

Arguments

Value

A list of class sidra_plan containing table, url, a stable dimensions data frame, total_estimated, limit, exceeds_limit, and risk. The originating query is retained internally for use by sidra_collect(). total_estimated is NA whenever any cardinality is unknown.

See Also

sidra_query()

Examples

query <- sidra_query(
  1612,
  variable = c(214, 215),
  period = c("2020", "2021"),
  geo = "n1",
  classific = "c81",
  category = list(c(2702, 2703))
)
sidra_plan(query, limit = 20)

Extract SIDRA result provenance

Description

Extract SIDRA result provenance

Usage

sidra_provenance(x)

Arguments

Value

The provenance list, with accessed_at, package_version, batch_count, urls, and value_type; or NULL when none is attached. When a fallback was used, urls records the actual endpoints and requested_urls records the original queries. Checkpointed collections also include batch_accessed_at (original download times) and resumed (logical flags for reused batches).

See Also

sidra_collect()

Build a SIDRA query without downloading values

Description

Constructs and validates a SIDRA values URL. This is useful for inspecting a request or passing it to sidra_plan() before any values are downloaded.

Usage

sidra_query(
  x,
  variable = "allxp",
  period = "last",
  geo = "Brazil",
  geo.filter = NULL,
  classific = "all",
  category = "all",
  header = TRUE,
  format = 4,
  digits = "default",
  api = NULL,
  value_type = c("numeric", "character", "both"),
  geo_view = NULL,
  include_extinct = FALSE
)

Arguments

Value

A list of class sidra_query with stable url, header, and parameters components. No values are downloaded.

See Also

get_sidra() and sidra_plan()

Examples

query <- sidra_query(
  1612,
  variable = 214,
  period = "2021",
  geo = "n1",
  classific = "c81",
  category = list(2702)
)
query

Split a SIDRA query into disjoint batches

Description

Divides one explicit query dimension without downloading values. The resulting batches can be executed with sidra_collect(). Period selectors all, first, last, and ranges are resolved through sidra_periods() into a fixed inventory before splitting. Other special selections must be expanded explicitly. No values are downloaded during splitting.

Usage

sidra_split(
  query,
  by = c("period", "variable", "geo.filter", "category"),
  size,
  index = 1L
)

Arguments

Details

Geographic filters can be split only when the query requests one non-Brazil geographic level. Queries with multiple territorial levels are rejected because unchanged levels would overlap across batches. Classifications already resolved in the original URL are reused without another metadata request.

Value

An object of class sidra_batch containing disjoint queries in their original order.

See Also

sidra_query(), sidra_plan(), sidra_collect()

Examples

query <- sidra_query(
  1612,
  variable = 214,
  period = as.character(2018:2022),
  geo = "Brazil",
  classific = "c81",
  category = list(2702)
)
batches <- sidra_split(query, by = "period", size = 2)
length(batches$queries)