The catalogue is now Unicode’s own, current to Emoji 18.0,
and the package says exactly what a glyph is and what it could not
measure. Until 0.5.0 tidyEmoji detected against the table
inside the emoji package, which has stayed at Emoji 16.0
through two Unicode releases, so the 182 emoji added in 17.0 and 18.0
were not detected at all. The catalogue is now Unicode’s
emoji-test.txt, bundled with the package and rebuilt by
data-raw/crosswalks.R when Unicode ships;
emoji still supplies the shortcodes, the keywords and the
detection regex. On top of it come ten new verbs, two detection repairs,
a presentation argument on every verb that reads text, and
association and polarisation measures on three shipped verbs.
Counts move, in both directions. A corpus scored under 0.5.0 and again under 0.6.0 can report more emoji, because the 17.0 and 18.0 additions are now detected, and it can report fewer, because two kinds of emoji material are no longer counted as emoji:
U+1F600 U+1F3FB, a grinning face followed by a skin
tone it cannot take, was two emoji in 0.5.0 (the second named “light
skin tone”, in group "Component") and is one emoji in
0.6.0.U+1F1FD U+1F1FD, two regional indicators spelling “XX”,
was one emoji with no name in 0.5.0 and is no emoji in 0.6.0, as is a
lone indicator.U+1FAEA, distorted face (Emoji 17.0), was nothing in
0.5.0 and is one emoji in 0.6.0.emoji_summary() counts the first two kinds, so a diff
between the two releases can be explained from the output, and
emoji_provenance() reports the catalogue version and the
date of its data file. The corpus behind the introduction vignette holds
none of the three, so every figure the vignettes quote for it is
unchanged.
Detection reads Emoji 18.0.
emoji_unicode_version() is "18.0", read from
the data file’s own # Version: header when the data is
built, because Unicode serves the current file at an unversioned address
and the per-version directories stopped at 16.0. The catalogue has 5244
spellings of 3972 emoji. All 3963 fully-qualified and 1029
minimally-qualified spellings are detected as exactly one emoji, and the
182 additions are named, though emoji has no shortcodes or
keywords for them yet.
The detector is still emoji’s regex, behind a short
prefix for the 16 code points newer than it (the two 18.0 thumb signs,
which also take skin tones, among them). The prefix is derived from the
catalogue and the installed regex together, so it can only match where
the upstream regex cannot, and an emoji release that
catches up empties it on its own. Checked: on all 5042 Emoji 16.0
spellings, alone and in context, and on the bundled corpus, the raw
matches are byte-identical to 0.5.0’s.
The switch was proven before it was made. Built
from Unicode’s Emoji 16.0 file with emoji 16.0.0’s aliases,
the new data-raw/crosswalks.R reproduces both shipped
crosswalks with identical() TRUE. Built from the 18.0 file,
emoji_unicode_crosswalk keeps the same 5761 rows (seven of
them in a new order) and category_unicode_crosswalk gains
the 202 new spellings in their groups and loses none. The input file is
committed as data-raw/emoji-test.txt, so the build repeats
without the network, and a test does exactly that.
Three flag names follow CLDR. “flag: Heard & McDonald Islands”, “flag: St. Helena” and “flag: French Southern Territories” are now “flag: Heard Island & McDonald Islands”, “flag: St. Helena, Ascension & Tristan da Cunha” and “flag: French Southern and Antarctic Lands”. Join on the glyph or the code, not the name.
emoji_unicode_releases() gains Emoji
18.0, released 2026-09-16.
emoji_provenance() gains
unicode_emoji_date, the date in the catalogue
file’s header, beside unicode_emoji.
A skin-tone or hair component standing alone is no longer
an emoji occurrence. It was counted by every verb, and listed
by emoji_frequency() and top_n_emojis() as an
emoji in its own right. The rule is applied once, where detection
decides what a match is, so no verb can bring it back. A component
inside a sequence, as in a toned thumb or a red-haired person, is part
of that emoji as before.
Regional indicators that spell no flag are no longer an
emoji. Any well-formed pair used to be read as one, so a stray
“XX” turned up in emoji_frequency() with every name column
NA, in emoji_dfm() as a column and in
emoji_pairs() as a node. The 259 country flags of the
catalogue are the valid set. Indicators pair from the start of a run, as
Unicode says and as a phone displays them.
Neither is silently discarded.
emoji_summary() gains n_orphan_components and
n_invalid_flags. The verbs that rewrite text treat both
exactly as before, because what changed is what is counted:
emoji_sanitize(policy = "strip") still removes a stray
swatch, emoji_to_text() still names it,
emoji_context() masks it out of its windows,
emoji_ratio() does not let it stop a row being emoji-only,
and emoji_incongruity(where = "final") still finds the
emoji before it.
presentation = "any"presentation. The default, "emoji",
is the detection every earlier release used and it stays the default.
"any" also reads the text-presentation spellings Unicode
lists, such as the bare heart U+2764 that several keyboards
emit, U+263A and a keycap without its selector: 240 of the
catalogue’s 243 unqualified spellings. The other three are the
copyright, registered and trade-mark signs, which are left out so that
the copyright line of a legal footer is not emoji use.
"any" only ever adds occurrences to what
"emoji" finds. The argument is the last one of every
signature, so no positional call written for 0.5.0 changes meaning.emoji_ratio() gains .emoji_n,
.emoji_codepoints and .emoji_n_char,
the counts .emoji_ratio is the quotient of. It returned
neither before.emoji_density() gains
.emoji_n_char and .emoji_n_token, the
two denominators of its two densities.emoji_trend() gains
n_period, every emoji token in the period, so
share = n / n_period keeps its base when top_n
keeps only some emoji.?tidyEmoji: every rate the package returns ships
with its numerator and denominator, so a count model can be fitted
instead of a linear model on a ratio. The new columns sit before the
rates they explain.emoji_coverage() reports, for every
lexicon at once, how much of a corpus it can score: as a share of the
emoji occurrences, as a share of the distinct emoji, and as the number
of emoji-bearing rows left with no score at all. On the corpus behind
the introduction vignette the sentiment lexicon scores 62.8% of the
occurrences and 50.8% of the distinct emoji, and 187 of the 560 rows
holding an emoji get no .emoji_sentiment. It also reports
what presentation = "any" would add, and how much of that
the lexicon could score.emoji_properties() says what the
catalogue records about a glyph: its code points, its qualification
status, its name, group, subgroup and emoji version, and whether it
takes a skin tone. With no argument it returns the whole catalogue.as_emoji_canonical() maps every
spelling of an emoji to its fully-qualified one, the normalisation the
verbs use internally, so a join of your own cannot miss rows over a
U+FE0F.emoji_identical() says whether two
glyphs are the same emoji and, with details = TRUE, how
they differ: not at all, in the variation selector, in skin tone, or in
the emoji itself. Its help page states the limit a forensic user most
needs: code points are not what anyone saw.emoji_find() returns the emoji whose
CLDR keywords carry any (or all) of a set of terms, and
emoji_keywords() returns those keywords in
long form, with NA, never "", for an emoji
that has none.emoji_country() maps the country flags
in a text column to ISO 3166-1 codes and region names, keeping a stray
pair with its glyph and NA for the code.
emoji_subdivision() does the same for the
tag-sequence flags (GB-SCT), and
emoji_flag() converts codes to flags and
back.emoji_from_counts() turns a tally of
(item, emoji, count) rows, such as GitHub reactions, into text every
other verb reads, so reaction data can be scored, counted and turned
into a document-by-emoji matrix. A tally and its expanded text give
identical answers, and that identity is tested.emoji_ambiguity(), emoji_risk()
and emoji_flag_ambiguous() gain
measure = "polarisation". The annotation classes
are ordered, negative to neutral to positive, and the shipped measures
are nominal, so entropy and Gini gave a 50/0/50 split exactly the score
of a 50/50/0 one although the first is a glyph readers take in opposite
directions. Polarisation is the expected distance between two
annotations on a -1, 0, 1 scale, the statistic Miller et al. (2016)
used, rescaled. Over the whole lexicon it agrees with entropy closely
(Spearman 0.96) and parts from it where it matters: the face with open
mouth and cold sweat, with 302 annotations, is at rank 6 of 969 by
polarisation and at rank 156 by entropy. The default stays
"entropy", so no published ranking moves.emoji_collocations() gains the association
measures "ppmi", "dice",
"logdice", "llr" and
"tscore" beside "pmi" and
"count", labelled on its help page as effect sizes or
significance measures. PMI ties on 40.6% of the rows of the bundled
corpus and favours rare pairs. The default output is unchanged; another
measure adds its column and reorders the rows, and never changes the
pairs or their n.emoji_pairs() gains
measure = "ppmi", "log_odds" or
"dice", scored over the documents that hold an
emoji, so adding emoji-free rows changes nothing. On the bundled corpus
93.9% of the raw edges are seen once, which is why a network needs one
of these and a frequency threshold; the new
vignette("emoji-networks") builds one.emoji_tweets() now warns
(lifecycle::deprecate_warn()), after four releases of soft
deprecation, and it will be removed in 0.7.0. Use
emoji_filter().\concept{}), so
help.search("accessibility", package = "tidyEmoji") finds
the verbs for a question, and the introduction vignette opens with an
index from questions to verbs.emoji-networks, on
association-weighted edges, a frequency threshold and the checks a real
network should pass.?register_emoji_lexicon names three sets of arousal
norms worth registering (Emoji-SP, Emoji-Dis and the face-emoji norms),
all CC BY 4.0 on OSF, since neither bundled lexicon has arousal.?emoji_sanitize cites the evidence that an unexamined
emoji reaching a language model is a measurable risk, and says that the
shortcode policy leaves the 182 newest emoji in place, since they have
no shortcode yet.inst/LICENSE.note carries the Unicode licence for the
bundled catalogue.catalogue job now asks Unicode whether a
newer emoji version has shipped, through a canary test that stays out of
every other run, and checks the vocabulary columns read from
emoji.stringi, already installed with emoji, is
now a direct Import: the detector calls it with the extended regex.
igraph joins Suggests, for the networks vignette.No new verbs. emoji_context() returns its answers a
great deal faster, and emoji_collocations(), which answered
a non-Latin-script corpus wrongly in a non-UTF-8 locale, now answers it
the same in every locale. A source reading before submission then found
eleven smaller defects, each a wrong, order-dependent or crashing answer
on an input the documentation already covered: a user-supplied lexicon,
a one-row input, a count past integer range,
where = "final" in the incongruity profile, trailing
Unicode whitespace under policy = "strip", and the C-locale
order under one dplyr option. They are listed below, each
with a test. The rest of the release is documentation and the
infrastructure that checks it.
The bundled crosswalks track Unicode emoji 16.0, via
emoji 16.0.0. Re-generating them from
data-raw/crosswalks.R against that release reproduces the
shipped data exactly, so the catalogue is current.
emoji_context() was still quadratic in the
emoji per row, and emoji_collocations() inherited
it. 0.4.0 bounded the tokenising to the window, which
was half the cost. The other half is that substr() on a
multi-byte string rescans from its first byte to reach a character
offset, so cutting one window still cost as much as the text before it;
and on a row whose masked text is all spaces – which is what a row of
nothing but emoji becomes – no bounded slice ever yields a token. That
is the shape a chat or reaction corpus is full of.
Each row now gets one index of its code points and token boundaries,
and every window is read off it. 3200 emoji in one row went from
7.1s to about a tenth of a second, and the verb is flat in the
emoji per row: the same 3200 spread over 320 rows costs the same tenth,
where the two were more than twenty times apart before. A realistic
mixed row and unit = "char" are each around three times
faster. A row holding a single emoji keeps the old path, which is still
cheaper there, so nothing regresses on short texts.
The output is unchanged, and checked to be: byte-identical on 13977
occurrences across eight window/unit
combinations, punctuation, no-break and ideographic spaces, ZWJ
sequences and keycaps included.
emoji_collocations() reported no
collocations at all for a non-Latin-script corpus in a non-UTF-8
locale. It trimmed leading and trailing punctuation off each
context word with [[:alnum:]], which resolves through the C
library: under LC_CTYPE=C nothing outside ASCII is
alphanumeric, so a Hebrew, Arabic, Thai, Devanagari or CJK word was
trimmed away to nothing and then dropped for being empty, and “cafe”
spelled with an acute came back as caf. The same corpus
reported its collocations normally in a UTF-8 session, which is what
kept this out of sight: the answer moved with the session rather than
with the data.
The trim now reads Unicode’s own tables through PCRE properties, and
a token holding no letter and no digit is not a word. Byte-identical to
0.4.0 on the bundled corpus in a UTF-8 locale, and identical between
en_US.UTF-8 and LC_ALL=C. One deliberate
difference: a superscript digit is now kept, where
[[:alnum:]] cut it.
The same verb counted two spellings of a Cyrillic or
Greek word as two words in a non-UTF-8 locale. Its case fold
used an explicit chartr() table precisely so
tolower()’s locale rules could not reach it, but the table
covered only ASCII and Latin – what the catalogue needs, where
this verb folds the user’s text. The table now covers every script in
the Basic Multilingual Plane with a 1:1 lower-case mapping, 1158 pairs
in all, derived from tolower() rather than written by hand
and reproducing it exactly in a UTF-8 session. The fold is also faster
than before, because it now tests for a non-ASCII byte and uses a
26-pair table when there is none.
It stops at the Basic Multilingual Plane on purpose:
wchar_t is 16 bits on Windows, so chartr() and
tolower() work on UTF-16 code units there and mangle a
character outside it. Astral scripts are left to the trailing
tolower(), exactly as before this table existed.
Multi-character mappings stay absent: the German sharp s and the
Greek iota-subscript capitals lower-case to more than one code point,
which chartr() cannot express and tolower()
does not do either.
That is three bugs of one shape, after tolower() in the
fold and iswspace() in the whitespace class, so a test now
scans the package for every locale-dependent regex class rather than
waiting for a fourth.
A data frame with duplicate column names failed with
tibble’s message rather than one of ours. A base
data.frame may carry two columns of the same name; a tibble
may not, and the verbs that add columns return a tibble. The conversion
failed with “Column name a must not be duplicated. Use
.name_repair to specify repair”, which names an argument no
verb in this package has and an internal, repaired_names(),
the user cannot reach, and never says which verb raised it. It was the
one error here that did not tell the user what to do. 0.4.0 had already
fixed the dangerous half of this, where the duplicated column is the
text column and [[ would silently read whichever
came first; what was left was the benign half, which matters only
because the verbs that add columns return a tibble. The check now
happens where the frame becomes a tibble, so one message serves every
affected verb, and it is worded like its sibling: same problem, same
remedy. The verbs that build their own output
(emoji_summary(), emoji_frequency(),
emoji_dfm() and the rest) never had the problem and still
accept such a frame.
emoji_incongruity_profile(where = "final")
credited glyphs a row was never scored on. With
where = "final" a row is scored on the run of emoji that
ends it, but the profile gave the row’s gap to every emoji in the row
and counted all of them in n, its “scored occurrences”. A
glyph that opened messages and never closed one could top the table on
gaps computed without it. Only the glyphs a row was scored on are
credited now, and where = "all" is unchanged.
A registered lexicon could be read through the wrong
column. The scoring verbs keyed a registered table on their own
by, which defaults to "emoji", so a lexicon
registered with by = "glyph" that also carried a column
named emoji (a label, say) was keyed on the label and
scored nothing, silently, in emoji_score(),
emoji_sentiment() and emoji_emotion(). A
registered lexicon is now always read through the column it was
registered with, which is what ?emoji_score already
promised. The same page said score is ignored for a
registered lexicon; it is honoured, as the page’s own error advice
assumed, and now says so.
Two rows for one emoji, one scored and one
NA, gave an answer that depended on their order. A
lexicon may list both spellings of an emoji when they agree, and the
duplicate check ignores NA while comparing, but the lookup
read whichever row came first: the qualified heart scored 0.5 or nothing
depending on which spelling sat on top. Each emoji now takes its
non-NA value (per dimension, for an emotion lexicon), so
row order cannot matter. A real disagreement is still refused.
emoji_emotion() counted an emoji as scored
when it had nothing to be scored on. A custom lexicon row whose
every dimension was NA, or whose only value was infinite
and dropped with a warning that the emoji “count as unscored”, still
counted towards .emoji_n_scored while every score in the
row stayed NA. It no longer does, which matches what
emoji_score() and emoji_sentiment() report for
the same table.
emoji_emotion_label() could label a row from
a column it never computed. It picked the
.emoji_<emotion> columns out of its result by name,
which also caught any the input still carried from an earlier call, so
emoji_emotion(df, text) |> emoji_emotion_label(text, lexicon = own)
with own scoring only joy came back “fear”
from the bundled profile. The label is now taken from exactly the
dimensions the call scored.
emoji_emotion() named its columns on a
one-row input. Each of the eight
.emoji_<emotion> columns came back as a named vector
(joy = 0.69) on a single row and as a plain one on any
other number of rows, because R keeps a dimension name when a one-row
matrix drops to a vector. The test that checks every verb for named
columns only ever used four rows; it now runs on zero, one, two and
four.
A count past integer range crashed three verbs.
emoji_ngrams(n = 1e10),
emoji_context(window = 1e10) and
emoji_collocations(window = 1e10) passed validation and
then failed on R’s “NAs introduced by coercion to integer range”, and a
window of 1e9, which stays in range, still
overflowed the window arithmetic. A large count now means “all of it”:
no n-gram is longer than any row, and the window covers the whole
text.
emoji_sanitize(policy = "strip") trimmed
only ASCII whitespace off the ends. It trimmed with
trimws(), whose default set is ASCII, so a row ending in an
emoji and an ideographic space came back with a trailing space and the
ideographic one. The ends are now trimmed with the package’s own
whitespace set, the one ?emoji_ratio describes and every
other verb splits and tests text with.
The C-locale order the help pages promise now holds under
options(dplyr.legacy_locale = TRUE). Eleven verbs
sort their glyphs or words through dplyr::arrange(), which
follows that global option into the session’s collation; with it set,
under an English locale, every one of them returned its ties in a
different order. They now ask for the C locale explicitly.
emoji_score() did not validate
score. A vector reached an internal
%in% and failed as R’s “the condition has length > 1”,
naming neither the argument nor the verb. It is now refused like every
other string argument.
emoji_lexicons() listed a registered
lexicon’s text columns among its dimensions,
columns emoji_score(score = ) then refuses. Only the
numeric and logical columns are listed now.
emoji_categorize() and emoji_ngrams() were
the last two verbs the pre-release audit listed as pooling grouped data
silently. Neither does. emoji_categorize() answers per row
and carries the grouping through untouched, so a grouping cannot change
what it returns; it drops the rows with no emoji, as its
@return says, which is not pooling because dropping a row
cannot merge two groups. emoji_ngrams() returns one row per
n-gram occurrence keyed by .row_number and carries none of
the user’s columns, so there is no grouping for it to ignore. Both are
pinned by tests now.emoji_position() already states that
positions are logical rather than visual order. A test now holds it
there, because it is the sentence a right-to-left corpus depends
on.vignette("reversible-preprocessing"). It runs the
emoji_sanitize() loss ladder rather than describing it,
shows the shortcode round trip surviving skin tones, flags, ZWJ
sequences and keycaps, demonstrates that wrap is part of
the reversibility contract (a wrap without colons restores nothing,
silently), and shows how to keep the emoji signal as feature columns
while the text you send to a model holds no emoji. It also makes the
detection limitation concrete: a bare U+2764 carries text
presentation, so it is not detected and survives a policy meant to
remove it.text_score recipe at eval = FALSE,
for tidytext with AFINN and for sentimentr, next to the word-list scorer
it uses to stay runnable.?emoji_sanitize and ?emoji_token_cost all
point at the new article.reviews does not have. Both blocks are self-contained now,
and a test evaluates every README chunk in document order so an
unevaluated one can only lean on an evaluated one above it.emoji_seasonality()’s denominator is, and that
emoji_context()’s character windows are code points and can
cut a grapheme.emoji_collocations() cannot find a collocation
in an unsegmented script, and said nothing about it. 0.5.0
fixed the character class that dropped non-Latin context words, which
makes a Chinese, Japanese, Thai, Khmer, Lao or Burmese corpus a
supported input. What the verb does not do is segment one: its
tokens are runs of non-whitespace, so a clause in such a script is a
single token. Measured: three rows of Chinese meaning “the weather is
good today”, “my mood is good today” and “going to Beijing tomorrow”
yield three tokens, each a whole clause, each with n = 1,
so nothing clears the default min_n and the result is
empty. The same three sentences in English yield good,
is and today at n = 2. Both help
pages now say so, and say that the segmentation has to happen before the
text reaches the verb, since the result cannot be passed to a tokeniser
to repair it. emoji_context()’s unit = "char"
sidesteps the question and is the better default on such text.?emoji_sanitize enumerates every way the round trip can
fail through wrap, and said nothing about the input text:
text_to_emoji() cannot tell a :shortcode:
token the sanitiser wrote from one the text always held, so a Slack,
Discord or GitHub export where people type :thumbsup: comes
back with an emoji it never contained. The count moves with it, from 0
to 1 on "nice work :thumbsup:", and it compounds with real
emoji rather than replacing them. The damage is bounded at one pass, and
the colon shapes that are not shortcodes ("meet at 10:30",
"ratio 3:4") were already safe and stay so. Both help pages
now state it from their own side, and four tests pin the inflation and
the sentences.?emoji_trend rendered its definition of
share as part of a list item. The Markdown that
roxygen renders folds an unindented line directly after a list item into
the item, so the paragraph defining share appeared as the
tail of the emoji_seasonality() bullet. It is its own
paragraph again, and a test scans every roxygen block for the same
shape.?tidyEmoji said 262 of the regional-indicator pairs are
real flags, and the figure is 259. The page had counted in the three
subdivision flags (England, Scotland, Wales), which are tag sequences
rather than pairs, and the test pinning it summed the two subgroups
without asking which rows are pairs.?emoji_unicode_releases described the emoji series as
starting at 1.0, while its own table begins at 0.6 and 0.7, the labels
Unicode gives the emoji of Unicode 6.0 and 7.0.?emoji_lexicons gave type as sentiment or
emotion, where a registered lexicon reports "custom", and
now says which columns dimensions lists.citation("tidyEmoji") asked for the sentiment
lexicon to be cited by users of three of the nine verbs that read
it, and the emotion lexicon by users of one of three. The
header now names every one, and a test derives the list from the
package’s own call graph."shortcode" can leave an emoji behind, when the reason is a
glyph too new to name. The introduction said the four datasets are
regenerated “from the current Unicode emoji list”; only the two
crosswalks are, and the scripts live in the source repository rather
than in the installed package.man/ is now verified to be what roxygen2
generates. 0.4.0’s help pages were produced by a stand-in
converter because no R was available in the authoring environment, which
left “re-run the real thing and commit the difference” outstanding for
two releases. Running roxygen2 7.3.2 over the tree reproduces 49
of the 51 .Rd files byte for byte and
NAMESPACE exactly; the only two that differ are
the two edited in this release. The debt is discharged rather than
carried again.inst/CITATION hardcoded a version fallback that
had already slipped a release. The file derives the publication
year rather than writing it out, with a comment explaining that a fixed
year “goes stale silently the first time the release slips past New
Year”. The same reasoning applies to the version, which was written out
as "0.4.0" while the package was 0.5.0. It is now derived
from the installed package, with no version literal left in the file,
and two tests hold it there: one that the cited version matches
packageVersion(), one that the file still renders when
meta is unavailable. citation("tidyEmoji") was
already correct in normal use, since the fallback only fires when
meta is missing; this removes the trap rather than a
visible defect.cran-comments.md had gone stale on one of its
own figures, claiming 170 entries in inst/WORDLIST
where the file holds 166. It is the one file in the repo whose entire
purpose is claims a reviewer can reproduce, so the count is now derived
in test-invariants.R alongside the marked-UTF-8 breakdown
and the skip inventory, and cannot drift again unnoticed.R/,
man/ and tests/. One comment in
test-regression-0.2.1.R held the last non-ASCII byte in the
suite, in a cross-reference to a planning document that is not shipped
and no longer has the section it named. The comment now describes the
defect each block pins.None of this changes the package, but all of it was promised in an earlier release and kept slipping.
data-raw/benchmark.R makes the timing baseline
repeatable instead of a number measured once in a session, and it grew a
second axis that holds the emoji count constant while varying how
densely they are packed – the axis the row-count axis cannot see, and
the one the fix above is about. Its ratio now compares the densest
corpus against the cheapest, which is the direction the defect moves it;
the previous max / min was dominated by per-row overhead
and scored every verb near its threshold on a healthy build.emoji-translate.R at 97.2%, which is why the job’s floor is
on the total rather than per file. The floor is enforced inside the job
because the codecov upload needs a CODECOV_TOKEN secret and
is allowed to fail without one: a job whose only output is an upload
nobody accepted reports success for nothing. Reading the uncovered
expressions rather than counting them found three pieces of dead code,
now gone.LC_COLLATE=C and
en_US.UTF-8, and a job running it under
LC_ALL=C. Both halves of the locale have cost this package
bugs and every one of them was green on every other flavour. Each job
refuses to run if the locale it asked for did not take effect.knitr tangles out of the introduction vignette, so a
checking machine that merely lacked a Suggests package failed. It
recognises both now.spelling is wired into tests/spelling.R
(non-failing, skipped on CRAN) and into the weekly job, where it does
fail. The check reports 0 unknown words across the help pages, both
vignettes, README.md and NEWS.md; inst/WORDLIST is 167
entries.testthat floor was wrong.
DESCRIPTION asked for >= 3.0.0 while the suite has
called expect_no_error() and
expect_no_warning() since 0.4.0, both of which arrived in
3.1.5. It now declares >= 3.1.5, and the table in
test-invariants.R that couples each declared floor to the
argument needing it covers testthat too.This release delivers the first wave of the feature roadmap filed as issue #5: the items that are cheap, research-grounded, and need no new dataset and no new dependency. Most of them are recombinations of machinery the package already had – the Novak lexicon’s annotation counts, the reference table’s Unicode version column, the grapheme-aware locator – read out in a way no R package exposed before.
Miller et al. (2016) found that readers of the same
rendering disagree about whether an emoji is positive, neutral or
negative roughly a quarter of the time. The bundled Emoji Sentiment
Ranking keeps the raw
negative/neutral/positive
annotation counts behind its collapsed score, so that disagreement was
already inside the package as an empirical distribution. It is now a
number.
emoji_ambiguity() reports per-glyph annotation shares
and one of four disagreement statistics – Shannon entropy
(the default), gini, neutral_share or
ci_width – with a rank over the whole lexicon.emoji_risk() is the per-row version:
.emoji_ambiguity_mean, .emoji_ambiguity_max
and .emoji_n_ambiguous.emoji_flag_ambiguous() is the content-QA shortlist: the
emoji in your corpus most likely to be misread.emoji_sentiment(se = TRUE) adds
.emoji_sentiment_se, so a glyph annotated eight times no
longer carries the same authority as one annotated eight thousand
times.emoji_context() returns one row per emoji occurrence
with a window of the surrounding text, in words or characters. All other
emoji are blanked out of the window, and character offsets stay
exact.emoji_collocations() aggregates those windows into an
emoji-word table scored by pointwise mutual information, shaped like
widyr::pairwise_count() output. Corpus-derived senses have
neither the licence problem nor the staleness problem of an imported
sense inventory.emoji_trend() counts emoji per period
("day", "week", "month",
"quarter", "year") and returns a
complete period-by-emoji grid, so a trend line does not
silently skip its zeros.emoji_turnover() reports vocabulary churn between
consecutive periods: jaccard, n_new,
n_lost, n_core.emoji_version_profile() breaks a corpus down by the
Unicode emoji version that introduced each glyph, and
emoji_adoption_lag() compares first use in the corpus with
the release date. Both come almost free from the version
column the reference table already carries.emoji_seasonality() aggregates by month, weekday or
hour, returning every level of the cycle including the empty ones, with
fixed English labels so a script’s output does not change with the
machine that runs it.emoji_unicode_releases() is the version-to-release-date
lookup behind the two verbs above. It is a function rather than a
bundled dataset: at a few dozen rows it belongs beside the code that
uses it.emoji_incongruity() measures the signed gap between a
row’s text sentiment and its emoji sentiment – the sarcasm feature in
NLP, the (in)congruence variable in marketing research.
emoji_congruence() is the same engine under the marketing
framing; emoji_incongruity_profile() reports which glyphs
go against the grain of their host text.text_score from tidytext, sentimentr, vader or a model.
Because those live on incompatible scales, scale has
no default – you have to say how the two sides were
made comparable.NA, never
0, in every new column.as_emoji_type(), emoji_type() and
emoji_faceness() recode the Unicode group and subgroup into
face, gesture, person,
nature, food, place,
activity, object, symbol,
flag and component. The emotional (face)
versus semantic (object) contrast is the key variable in the
consumer-behaviour literature and is now a one-liner.emoji_sanitize() applies one named policy –
"keep", "strip", "name",
"shortcode" or "placeholder" – to a text
column. The capability mostly existed; the value is an argument that
shows up in a script diff and in a methods section.emoji_token_cost() reports exact
.emoji_bytes, .emoji_codepoints and
.emoji_graphemes plus a clearly-labelled
.emoji_token_estimate, or the real count if you pass your
own tokenizer.emoji_provenance() puts every version an emoji result
depends on in one row: tidyEmoji, the emoji package, the Unicode emoji
version, the size of the detectable emoji set, and the lexicons.emoji_unicode_version() reports that Unicode version on
its own.inst/CITATION now credits the package and the two
lexicon papers users have to cite anyway.This section is long because the release was audited exhaustively rather than spot-checked, and both kinds of finding are recorded: changes you can observe, and verifications that confirmed existing behaviour was already right. The second kind is kept deliberately – knowing that a figure was re-derived from the data, or that a formula was checked against an independent implementation, is worth as much to the next maintainer as knowing what moved.
Entries whose first sentence is bold are the ones where something was actually wrong and got fixed – in the package, in its documentation, or in a test that was passing for the wrong reason. There are one hundred and eighteen of them, and reading just those leads gives the release without the verification detail. Not all one hundred and eighteen changed observable behaviour: several record a test that could not have failed, or a figure the documentation quoted incorrectly, which are worth the same prominence because both meant something was unverified.
The whole of this release’s polish was audited against the
version it started from, by installing both side by side and comparing
57 verb calls over the bundled 2000-row corpus. 54 are
byte-identical, and the three that differ are exactly the three
intended behaviour changes, in exactly the places they should be:
emoji_emotion() |> emoji_emotion_label() gains the eight
emotion columns and nothing else (every shared column identical, and the
scores equal emoji_emotion()’s own); and
emoji_incongruity(scale = "rank") and "zscore"
change on precisely the 373 scored rows, with NA placement
untouched, scale = "none" and
emoji_congruence() byte-identical, and the rank-scale mean
gap now 0 (-2.1e-17) where it had been -0.0042. Every guard, message and
validation change in this release leaves valid input alone.
The same comparison over 29 adversarial inputs – list, matrix and
data-frame columns, duplicated and missing column names, a wide frame, a
non-finite score, dd/mm/yyyy dates, an unknown
measure, an emotion-shaped lexicon,
NA search queries – shows no guard was
loosened: nothing that was rejected before is accepted now.
Five inputs that used to produce a plausible-looking result are refused
(the list and data-frame text columns, a bare list to
emoji_tokens(), a whole dd/mm/yyyy column, an
unrecognised measure); two now warn and drop instead of
answering from bad data; eight kept erroring but say something useful
instead of surfacing dplyr’s, tibble’s or match.arg()’s
internals; and twelve are unchanged in value, message and warning
alike.
A duplicated column name silently resolved to the first
column. The bare-name fast path added earlier this release
tested nm %in% names(data), which is true even when the
name appears twice, and [[ then returns the first – so
emoji_summary() and emoji_frequency() answered
from whichever column came first while the verbs that convert
data to a tibble failed with tibble’s own message instead.
dplyr::select(), which that path replaced, had rejected the
ambiguity for all of them. A data frame really can carry the name twice
– read.csv(check.names = FALSE) on a sheet with repeated
headers does it – and picking one of two columns unasked is a wrong
answer that looks right. All ten verbs tried now report the ambiguity,
naming the argument and the count, for every column argument.
Checked that this release’s type guards have not caught anything
a caller might reasonably hold text in: difftime,
complex, AsIs, noquote, a factor
with an NA level and a custom-classed double all still work
as a text column, and difftime is still
refused as time or text_score with the type
message rather than a coercion.
Verified emoji_context()’s window content,
which no test had checked. The earlier round proved the bounded-slice
optimisation agrees with the package’s own naive version – but two
implementations that cut the window in the same wrong place would agree
just as well. It is now compared against a reference sharing no code
with the package: every glyph span is masked to spaces, the two sides
are split by a character-at-a-time whitespace splitter, and the nearest
window tokens are taken. 5160 windows over
window values 0, 1, 2, 4 and 9 match exactly, and
the documented promise holds – with three emoji in a row, the middle
one’s window reaches past its neighbours to real words and never quotes
the neighbours. emoji_collocations()’s word list was
checked the same way against an independent tokenisation (400
comparisons, no difference), along with each rule its documentation
states: lower-cased, leading and trailing punctuation stripped but
internal punctuation and digits kept, and a word counted once per
occurrence however often it repeats in the window. No defect was found
in either; both are now pinned.
Pinned the version cluster’s behaviour against a future
emoji release, the one way this package can break
without anything here changing. emoji_version_profile() and
emoji_adoption_lag() read the introducing
version out of the installed reference table and join it to
a release-date table kept in tidyEmoji’s own source; the day Unicode 18
ships, that label will not be in the table. R CMD check
cannot reveal this, because it runs against today’s emoji.
Injecting 18.0, the data-file spelling E18.0,
an absurd 99.9 and a missing version now confirms all four
degrade rather than fail: the glyph keeps its row, its
release_date and lag_days are NA
instead of invented, share_tokens still sums to 1, the
other glyphs are untouched, and emoji_unicode_version()
follows the new label. Also pinned the release table itself –
version unique across both numbering series,
version_num equal to the parsed label (string ordering
would put 10.0 before 9.0), no missing dates,
and within each series a later version never released earlier.
Closed the gaps in the grouping contract. Grouping is what makes
group_by() |> verb() |> summarise() work, and it is
upheld by two separate mechanisms: .emoji_as_tibble()
passing a grouped_df through untouched, and
.emoji_regroup() re-deriving the indices for the three
verbs that rewrite the text column. Only 13 row verbs were pinned; the
seven omitted ones – emoji_emotion(),
emoji_emotion_label(), emoji_incongruity(),
emoji_congruence(), emoji_sanitize(),
emoji_to_text() and text_to_emoji(), which is
all three of the rewriting verbs – are now covered too, so a
stray ungroup() cannot change semantics silently. The
mirror case is pinned as well: emoji_summary(),
emoji_extract_unnest() and
emoji_flag_ambiguous() must not be grouped,
because they do not return the caller’s rows and so have no group column
to carry. And .emoji_regroup() is now tested in the case it
was written for, in its sharpest form – grouped by the text column where
policy = "strip" maps two distinct texts onto one,
collapsing three groups into two and shifting every index. All four
rewrites leave each group’s indices pointing at rows that really carry
its key, and are indistinguishable from regrouping from scratch. No
defect was found; the behaviour was already right in all 22
verbs.
A "bytes"-encoded text column failed with
R’s message, not the package’s. A character vector whose
Encoding() is "bytes" is a bag of bytes R
refuses to read as characters: nchar(type = "chars"),
gsub(), tolower() and substr()
each stop on one. Emoji detection is built out of precisely those, so 22
verbs already failed on such a column – but they failed with
"bytes encoding is not supported by this function", thrown
from inside R, naming no argument, no column and no remedy. Strings
arrive marked this way from
readBin()/rawToChar() and from text decoded
with the wrong encoding upstream. The shared text resolver now catches
it, so all 22 give one message that names the argument, the column, how
many values carry the mark, and the fix. Coercing was rejected
deliberately: a string is usually marked "bytes"
because it is not valid UTF-8, so enc2utf8()
cannot repair it and would substitute replacement characters. Only the
caller knows what the bytes meant, which is why the message says
iconv() and says why enc2utf8() will not
do.
The same mark on a string argument is now refused as
well – sep, placeholder and
query, which reached gsub() and
tolower() with R’s message. wrap was the worst
of them and the only one that did not error at all:
emoji_to_text(format = "shortcode") pasted the template
into the rewritten text, which marked the output column
"bytes" too. The verb returned a perfectly ordinary-looking
tibble whose text column would then stop the next thing that touched it,
arbitrarily far from the cause. That emoji_to_text() never
returns a bytes-encoded column is now a test in its own right.
latin1- and unknown-marked text, and factors, are
unaffected – they are readable, and the guard is only about the encoding
R will not read.
The collation test had a field that asserted
nothing. The snapshot guarding “no ordered output depends on
LC_COLLATE” compared emoji_pairs()’s output
across locales – but over a fixture whose every row held a single emoji,
and both emoji_pairs() and
emoji_cooccurrence() pair within a row. The field
was character(0), which compares equal to itself in every
locale, so the ordering of those two verbs had never been tested.
collocs was nearly as empty: its fixture was
paste("good", ...), so the word column was three copies of
"good" and no permutation of it could differ either. Both
now use fixtures that produce real output, and the test refuses any
field that is empty or constant – the two shapes that make a
comparison vacuous.
The same test only tried C and
en_US.UTF-8, which order the fixture identically, so it
could not have caught a collation bug even in a populated field. It now
also runs under Danish, Czech and Estonian collation, which disagree
with C about where a-ring, ch and o-tilde
sort, and it asserts that the fixture words re-order under at
least one locale it reached – so a fixture that stops being
collation-sensitive fails instead of passing quietly. Coverage went from
16 outputs to 31, adding emoji_trend(),
emoji_turnover(), emoji_seasonality(),
emoji_cooccurrence(), emoji_tokens(),
emoji_extract_unnest(), emoji_type(),
emoji_version_profile() and
emoji_adoption_lag(). It runs faster than before despite
that (8.9s to 3.9s): the old shape called
emoji_search("hand") three times per locale and six other
verbs twice.
Documented why the head of emoji_ambiguity()’s
ranking can show something that is not an emoji. The docs already
explained the tie at rank = 1 by annotation count, which is
right as far as it goes – but three of those five rows are box-drawing
and dingbat characters, not emoji. The lexicon was built from 2015
tweets and 233 of its 969 rows are absent from the reference table, so
no corpus this package analyses can yield them. They carry 6% of the
lexicon’s annotations, and 86% have fewer than 50, so the
n_annotations filter the page already recommends removes
200 of the 233 as a side effect. Every figure is pinned twice, from the
data and from the rendered Rd; the coverage figures on
[emoji_sentiment_lexicon] (736 in the reference table, 233 not, 3790
distinct keys, “about 19%”) were re-derived and all hold.
A non-numeric score column registered happily and then
scored nothing. register_emoji_lexicon() resolves
the score column at registration precisely so a bad table fails there
rather than later – but it checked only that the column
existed, not that it held numbers. A score column
of text (which is what a stray "NA" or a decimal comma
makes of a whole column) therefore registered without complaint, and at
scoring time reached mean(), which returns NA
with R’s own “argument is not numeric or logical” warning. Meanwhile
.emoji_n_scored still counted the emoji as scored, so the
row claimed a score it did not have – contradicting that column’s
documented meaning, where 0 is exactly “had emoji the
lexicon could not score”. Both the registration path and the
bare-data-frame path now refuse it and say what to look for. A logical
column still works, and a genuinely NA numeric score is
still counted as unscored.
Two lexicon rows for one emoji let the row order pick the
score. Spellings that differ only by a variation selector share
one code-point key, so a table listing both U+2764 and
U+2764 U+FE0F holds one emoji twice. The lookup took
whichever came first: swapping two rows of the caller’s own table
changed .emoji_score from 0.5 to
-0.5, silently – the same failure as the duplicated column
name fixed earlier this release. Identical scores are the normal case
and still collapse quietly, and an NA beside a value is not
a disagreement; only genuinely conflicting scores are refused, naming
the key. Neither bundled lexicon has a duplicated key at all, which is
now asserted so the guard cannot start reaching them.
by is validated as a single string. It reaches
%in%, so by = c("emoji", "other") surfaced R’s
own “the condition has length > 1” from inside
register_emoji_lexicon() and emoji_score(),
naming neither the argument nor the verb.
Checked the suite the way a suite should be checked: by breaking
the package and seeing whether it notices. Nine mutations were applied
one at a time to the numeric and ordering core – weakening
emoji_position()’s short-text guard, making whitespace-only
text score NA per token, dropping
emoji_dfm()’s glyph tiebreaker, shifting its tf-idf
denominator, moving the Monday the week buckets to, moving the month a
quarter starts on, ordering emoji_unicode_releases() by
version string instead of number, turning off
fixed = TRUE in emoji_search(), and removing
the "bytes"-encoding guard. All nine were
caught, by 1 to 92 failing tests each; none survived. Three
were caught by a single test apiece, which is the thin margin worth
knowing about rather than a defect.
Verified emoji_context(unit = "char"), the half of
that verb an earlier round left alone. The word unit was checked against
a reference sharing no code with the package; the character unit has its
own bounded-slice optimisation and its own “enough characters after
trimming” fallback, and neither had been checked against anything but
itself. 4135 windows over window values 0, 1, 3, 7
and 25 match exactly, including the doubling fallback that
whitespace-heavy input triggers. Also pinned that a masked neighbour can
pad a character window but can never appear in it.
Tied together top_n_emojis()’s two naming branches,
which had nothing connecting them. emoji_name comes from
emoji_frequency()’s shortcode when
duplicated = FALSE and from a many-to-many join onto
emoji_unicode_crosswalk when TRUE, so a change
to either could leave them disagreeing about what an emoji is called –
and only the expanded branch was exercised anywhere. Over a corpus
holding every catalogued glyph, the single name is the first of
the expanded names for all 4830 of them, and neither branch leaves a
glyph nameless: 189 reference rows carry no alias, but canonicalisation
matches on the codepoint key, so such a glyph borrows the alias of its
other spelling. Also pinned that n cuts on distinct emoji
rather than rows, so a glyph with several aliases cannot eat another
glyph’s slot, and that n = 0 gives a typed zero-row tibble
in both branches.
Verified the declared R (>= 4.1.0) floor exactly
rather than by inspection: every base-package function the code calls –
collected by walking the syntax tree of R/ and
tests/, 443 distinct call names – exists in a real R 4.1.0
installation. The only name that resolves on 4.4 but not 4.1 is
%||%, which entered base R in 4.4.0 and which this package
has always defined for itself.
emoji_turnover() could not tell
new from lost. Mutation testing found
this one: swapping n_new to compute
setdiff(a, b) instead of setdiff(b, a) – so
that “new emoji” reports the lost count – passed the entire
suite. Two tests looked like they covered it and neither could. One
asserts n_new == 1 and n_lost == 1; the other
checks each against set arithmetic done by hand, but over a fixture
whose first period pair gains exactly one glyph and loses exactly one.
Both numbers are 1, so no permutation of them is observable. The
assertions were right; the fixtures had no teeth – the same failure as
the empty snapshot field fixed above, in a subtler form. There is now a
deliberately asymmetric case (three glyphs arrive, none leaves), the
reverse of it, an assertion that the fixture can distinguish
the two, and the identity that fixes the direction:
n_new + n_core is the later period’s vocabulary and
n_lost + n_core the earlier one’s. The implementation was
correct throughout; only the test was blind. Re-running the mutation now
fails four assertions.
emoji_provenance() mis-described its own
headline number. n_emoji was documented as “the
size of the detectable emoji set” but reports
nrow(emoji_reference()) – rows of the reference table,
which are spellings. With emoji 16.0.0 that is
5042 rows carrying only 3790 distinct code-point keys, because an emoji
whose presentation can be selected appears both with and without
U+FE0F, and 212 of the 5042 are not detectable in text as
written. This is a function built to be pasted into a methods section,
so “5042 detectable emoji” is a number that would have gone into papers
overstating the vocabulary by 1252. The value is unchanged – it is the
table’s size, which is the right thing for a provenance row to record –
and the documentation now names the quantity, points at
length(unique(emoji_reference()$key)) for the count of
distinct emoji, and notes that the lexicon strings count rows the same
way. Also established the fact that makes the overstatement harmless
rather than a defect: no emoji is lost to an undetectable spelling,
because every one of the 3790 keys is reachable through at least one
spelling that is detected.
emoji_version_profile() could not tell type
shares from token shares. Found by extending round 79’s search
from one column pair to every pair a verb could swap: exchanging the
expressions behind share_types and
share_tokens passed the whole suite. The only assertion
either column ever had was sum(share_tokens) == 1 – and
both columns sum to 1, so the swap was invisible. There is now
a fixture where the two genuinely differ (one glyph three times, another
once, so share_types is 0.5 where share_tokens
is 0.75), each column checked against its own count, and an assertion
that the fixture separates them. Re-running the swap fails four
assertions. As with n_new/n_lost, the
implementation was right and only the test was blind.
The same sweep exchanged nine other pairs –
.emoji_first/.emoji_last,
.emoji_per_char/.emoji_per_token,
.emoji_only’s polarity,
.emoji_n_typed/.emoji_n_face,
n_types/n_tokens,
release_date/first_seen,
n_texts/n_with_emoji,
p_neg/p_pos and emoji_summary()’s
two counts – and every one was caught. The seasonality pair was the
closest call: its own fixture reads 2 for both columns, and the month
where they differ is asserted for neither, but two unrelated tests fail
on the swap anyway. It now has a fixture that separates them directly
rather than relying on that.
Audited every guard in the package by shadowing stop
inside its own namespace, which makes one suite run log every error the
tests actually trigger. 53 of the 54 guards fired; the one that never
did was emoji_sentiment()’s fallback for a lexicon it
cannot use, and reaching it exposed two message defects rather than
one.
emoji_sentiment(lexicon = "emotag1200") said
the name was invalid. The bundled emotion lexicon is the only
input that reaches that guard, and the message was “lexicon
must be ‘novak2015’, a registered lexicon, or a data frame” – telling a
user whose name is a perfectly valid bundled lexicon that it is not, and
pointing nowhere. It now says the lexicon is an emotion lexicon, that
this verb needs one score per emoji, and names both [emoji_emotion()]
for the per-emotion profile and emoji_score(lexicon = ...)
for the mean over its dimensions – the answer emoji_score()
already gave its own callers. emoji_emotion()’s reciprocal
case now mirrors it instead of only listing what it wants.
A guard offered an option it rejects.
.emoji_lexicon_lookup()’s type message read
“lexicon must be a name (string), a data frame, or NULL for
the default” – and NULL fails that very guard, since
is.character(NULL) is FALSE. No verb accepts
or documents lexicon = NULL. The message now describes what
is taken and names what was passed. Tightening it to a single
name also fixed two messages that were R’s rather than the package’s: a
length-2 name reached %in% inside an if and
produced “the condition has length > 1”, character(0)
produced “argument is of length zero”, and NA_character_
fell through to a misleading “Unknown lexicon NA”.
Two guards remain that no test can reach, and deliberately so:
the final else in each of emoji_sentiment()
and emoji_emotion(). .emoji_lexicon_lookup()
returns one of four shapes – a data frame, or a record of type
sentiment, emotion or custom – and both verbs now handle all four, so
those branches are unreachable defence against a fifth rather than
untested behaviour.
The deprecated duplicated_unicode was more
permissive than its replacement, and silently so. It converted
with isTRUE(x) || identical(x, "yes"), which collapses
everything it does not recognise to FALSE. So
duplicated_unicode = "TRUE" and
duplicated_unicode = 1 – both entirely plausible for an
argument that once took a string – returned the opposite of
what was asked, without a word, while NA,
"Yes", character(0) and a length-2 vector all
passed unnoticed too. The current duplicated rejects every
one of them through .emoji_check_flag(), so migrating
to the deprecated spelling was a downgrade in safety. It now
accepts exactly the four values it ever meant – TRUE,
FALSE, "yes", "no" – each still
giving what the modern spelling gives, and errors on anything else
naming the replacement. Found by extending the guard audit to warnings:
both bare warning() sites fire in the suite, and probing
the deprecation surface instead turned this up.
Fifteen enum arguments answered a typo by naming a
variable the caller never wrote. match.arg()
reports its own formal, so emoji_context(unit = "words")
said *“‘arg’ should be one of”word”, “char”“* and
emoji_sanitize(policy = c("keep", "strip")) said ”‘arg’
must be of length 1”. emoji_turnover() had already
been given a hand-rolled check for precisely this, with a comment saying
it exists to name measure”rather than match.arg()’s own
'arg'” – and the fix was never carried to the other fifteen
call sites, the same written-where-noticed-and-never-grepped pattern
that left seven aggregators without a grouped-input guard in 0.3.0. They
now share .emoji_match_arg(): unit,
policy, format, weighting,
period, by (twice), measure (five
verbs), method, where and scale
all name themselves and list their options.
That conversion changes no accepted value.
match.arg()’s behaviours are preserved deliberately and
pinned: a value identical to the whole choice vector still means “no
value supplied” and takes the first (which is why
emoji_context(unit = c("word", "char")) is accepted and
emoji_sanitize(policy = c("keep", "strip")) is not), exact
matches win, and unambiguous prefixes still resolve, so
by = "mon" keeps working while an ambiguous one is refused
rather than guessed. The accept/reject matrix over full values,
prefixes, wrong case, unknown values and length-2 vectors is unchanged
across all fifteen.
Converting them meant writing each choice set at the call site,
since match.arg(x) reads it from the formal and a helper
cannot. That is the one risk the change introduces, so a test now
derives every choice set from the function’s own formals and requires
the error message to list exactly it – no more, no fewer.
(emoji_incongruity()’s scale is deliberately
defaultless, so it has no formal to derive from and is covered by the
message test instead.)
Closed the last gap in the grouping story.
.emoji_warn_grouped() exists because “silently ignoring a
grouping turns a per-group question into a global one”, and an earlier
round pinned which verbs keep a grouping and which drop it – but not
whether the droppers say so. Three do not:
emoji_ngrams(), emoji_extract_unnest() and
emoji_context() discard the grouping, and the grouping
column with it, without a word. All three are defensible, and for a
reason worth stating: each returns .row_number instead of
the caller’s columns, and none of them pools rows, so there is no
per-group answer being quietly globalised – unlike the thirteen verbs
that do pool and therefore warn. That distinction was recorded nowhere.
Of 37 help pages exactly two mentioned grouping, both promising it is
kept. All four reshapers now say which side they are on, and point at
.row_number as the way to join your columns back.
The three-way classification is now a contract rather than an
accident: every data-first verb must either keep the grouping (and the
column it names), or warn and ignore it, or be one of the three named
reshapers that return .row_number and none of the caller’s
columns. The silent set is asserted to be exactly those three, so a new
verb cannot join it by accident; both other groups are asserted
non-empty so the classification cannot go vacuous; and a verb that warns
is required not to return the grouping, which would be the
confusing middle case of warning about something it went on to
keep.
Verified the statistical core against the raw annotation counts
rather than against itself, since these are numbers that reach papers.
All four ambiguity measures recompute exactly over all 969 rows –
Shannon entropy in nats with 0 log 0 taken as 0, Gini
impurity, the neutral share, and the standard error from
Var(X) = E[X^2] - E[X]^2 for X in
{-1, 0, 1} – and every documented figure on the page holds:
the median glyph has 18 annotations, 69.0% have fewer than 50, 11 of the
top 20 do, the five rows tied at rank = 1 carry 3, 3, 3, 9
and 15, and the entropy/count Spearman correlation is 0.556, which is
the “0.56” the page quotes. gini’s documented 2/3 ceiling
is now asserted alongside entropy’s log(3)
one, which already was.
Pinned the invariant that makes
emoji_sentiment(se = TRUE) meaningful: the score map and
the standard-error map cover exactly the same emoji, with no
NA in either. The row mean drops glyphs the score
map cannot score and the standard error drops glyphs the se map
cannot, so if those two sets ever diverged the reported error would be
the error of a mean nobody computed. .emoji_n_scored is now
checked to equal the standard error’s own denominator over a many-glyph
corpus, and the propagation (sqrt(sum(se^2)) / n_scored)
re-derived independently. Nothing was wrong; nothing had been
checking.
emoji_type()’s .emoji_type is
NA for two different reasons – a row with no emoji, and a
row whose every emoji the recode cannot type – and the page documented
only the first. The recode maps the ten Unicode groups the catalogue
currently uses and starts from NA, so a glyph in a group
added to Unicode after your emoji package was built has no
type. It is the same conflation [emoji_categorize()] already describes
for .emoji_category, and the fix is the same: say both, and
say how to tell them apart – [emoji_faceness()]’s
.emoji_n_typed is NA when the row had no emoji
and 0 when it had emoji that could not be typed.
Pinned the recode’s coverage and its behaviour on the untypable
path. Nothing falls through today: all 5042 catalogue rows type, every
type produced is one of the eleven declared levels, and all eleven are
reachable so none is dead. An unknown group yields NA
rather than erroring or landing silently in a catch-all. Forcing a glyph
untyped confirms the rest of the contract holds around it: the row keeps
its other emoji’s type instead of being dropped, .emoji_n
still counts the glyph because it is still an emoji,
.emoji_n_typed falls to match, and
.emoji_faceness is the share among typable emoji –
NA rather than a division by zero when none of them
is.
Swept the “NA for two reasons” class across every
verb instead of meeting it one at a time. Over a fixture holding all
three causes – a row with no emoji, a row whose emoji the lexicon or the
recode cannot use, and an NA text – the design turns out
uniform and sound: every answer column is NA for
all three, and every count column
(.emoji_n_scored, .emoji_n_typed,
.emoji_n_ambiguous) is 0 for “had emoji I
could not use” and NA for “had no emoji”. That is the
contract [emoji_sentiment()] states, and [emoji_score()],
[emoji_emotion()] and [emoji_risk()] each restate it. Two verbs carrying
the same count column never said it: [emoji_faceness()] and
[emoji_incongruity()]. Both do now, and the whole pattern is pinned
rather than left as a coincidence of six separate
implementations.
A documentation-pinning helper reported gaps that did not
exist. Rd prose is line-wrapped, so a sentence written as one
line in the roxygen source arrives from tools::Rd_db() with
a newline in the middle – and a fixed = TRUE match for that
sentence fails. Three of this round’s own assertions failed exactly that
way, on pages whose wording was already correct. The test helper gained
rd_flat(), which collapses whitespace first, and the
thirteen existing assertions that matched a multi-word phrase against
unflattened text were converted to it. None of them was failing – each
passed only because its phrase happened not to straddle a break, so any
edit to the surrounding prose could have turned a correct page into a
red test. The pass count is unchanged by the conversion, which is the
point.
The suite’s dependency on one emoji release
was undeclared and unexplained. About seventy counts derived
from the catalogue are pinned here – 5042 rows, 3790 distinct keys, 969
lexicon rows, 736/233, 212 undetectable spellings, 1252 carrying
U+FE0F, 216, 200 – deliberately, because the documentation
quotes them and the pinning is what catches doc-vs-data drift. But
nothing recorded which release they came from and nothing checked it, so
a new Unicode version would have surfaced as dozens of unrelated-looking
failures with no statement of the cause, plus four help pages quietly
naming a release the user does not have. There is now a single constant,
a canary test that fails loudly and says exactly which figures need
re-deriving, and an assertion that no help page names a
different release. Simulating the update confirms three tests
fail, all three named so the cause is legible, rather than seventy that
are not.
Declared emoji (>= 16.0.0), which had no version
floor at all. The other floors in DESCRIPTION are keyed to
arguments the code calls, which is why emoji was
not among them – tidyEmoji reads its data, not its functions. The reason
it needs one is different and just as real: every catalogue figure in
the documentation describes a single release, and an older
emoji has fewer rows, so those figures would simply be
wrong. The floor, the version the help pages name and the pinned counts
are now governed by one constant and asserted to agree.
cran-comments.md asserted something a
reviewer could test and find false. Its explanation of the one
“possibly invalid URL” note said “The site itself is up:
curlGetHeaders(url, verify = FALSE) returns
200”. That was true when written and is not now: the
redirect target refuses TCP connections outright, with or without TLS
verification, so the claim would not reproduce for anyone checking it.
The passage now separates what the package can vouch for from what it
cannot. The documented URL is healthy and that is the part that matters
– https://hdl.handle.net/11356/1048 returns
HTTP/2 302 to
https://www.clarin.si/repository/xmlui/handle/11356/1048,
verified again for this release – and the note describes a third-party
CLARIN.SI host reached only by following that redirect. Both observed
failure modes are recorded, the original certificate-chain one and the
present timeout, precisely because they differ: the point is that the
failure is not the package’s and not stable enough to characterise. No
claim is made about the target’s availability.
Re-checked every other URL in the package over the network: the
two citations, the EmoTag repository, the pkgdown site and its
introduction article, the issue tracker, the CRAN page and the three
badges all return 200, and only doi.org and
the CRAN page redirect (both to their expected canonical targets). The
t.co links the URL sweep turns up are rows of
inst/extdata/ata_tweets.csv, not documentation, and are not
fetched.
Added the offline half as a test, since the network half cannot
run on CRAN: every URL in the help pages and DESCRIPTION
must be well formed – https, no whitespace, no stray
backtick or quote swept in from the surrounding markup, no trailing
punctuation, and a parseable host and path. That is the class of defect
that turns a working link into a reported one.
A second stale claim in
cran-comments.md. It said “Two tests skip: one
gated on readr (>= 2.0.0) … and one
skip_on_cran() maintenance check” – written when the suite
was much smaller, and describing a suite that no longer exists: there
are now seven skip_on_cran() tests. The passage reports the
re-measured inventory and names all seven, marks the R 4.1.0 figure as
history rather than a current measurement (that tree cannot be rebuilt
here, so asserting a fresh number would be inventing one), and explains
why a tarball check skips three more than a source check – they read
README.Rmd, the R/ tree and the vignette
corpus, none of which a tarball carries.
The rest of the file was re-verified rather than trusted. The
marked-UTF-8 breakdown is exact –
emoji_unicode_crosswalk$unicode 5761,
emoji_sentiment_lexicon$emoji 969,
emoji_emotion_lexicon$emoji 150,
category_unicode_crosswalk$unicodes 10, summing to the 6890
it quotes – and no other column in any bundled dataset carries a marked
string, as it claims. Against a live CRAN index: still no reverse
dependencies, the published version is still 0.3.0, and
emoji is still 16.0.0, which confirms the version floor
added above is satisfiable and that the documented catalogue figures
describe the release CRAN currently ships. The package count was
refreshed from 24,744 to 24,748.
Those figures are now derived in a test rather than asserted in
prose, so they cannot drift again: the skip count, each component of the
UTF-8 breakdown and their total, the claim that no non-glyph column is
marked, and the licence. Writing it caught its own bug first – counting
occurrences of the string skip_on_cran() in the test
sources returns nine, because the literal appears in the counting code
and in comments. It counts lines that are the call, and the
runtime skip count confirms the figure independently.
README.md is knitted from README.Rmd,
so every #> line in it is output the package produced at
some past moment. An existing test compares the two files’
prose, which catches an edit made to README.md
instead of to its source – but not the likelier rot: prose and code both
untouched while the package’s output moves underneath them. This release
changed several messages and added columns, so it was worth checking.
It had not gone stale – a fresh
github_document render is byte-identical to the shipped
file – but nothing was checking, and now something does.
That test compares the #> lines rather than the
whole render, deliberately. The prose is re-wrapped by pandoc, whose
line breaking differs between versions, so diffing whole files would
fail on a different pandoc even when the README is perfectly current.
Three cosmetic markers are normalised for the same reason: testthat
forces cli.unicode = FALSE, so pillar decorates its tibbles
in ASCII (x, ~, i) where a
maintainer’s render uses Unicode (\u00d7,
\u2026, \u2139). Each was found by running the
comparison and reading what differed rather than by guessing. Everything
that can actually go stale – row and column counts, column names, values
– is compared byte for byte.
Fixed a loop in last round’s own new test that iterated zero
times. Its “no non-glyph column carries a marked string” check ran over
a hand-written list of datasets that included emoji_tweets
– which is the deprecated synonym for [emoji_filter()], a
function, whose names() is empty. That entry
therefore asserted nothing, the same vacuity that left
emoji_pairs()’s collation untested for a release. The list
is now derived from utils::data(), so it is exactly the
four datasets the package ships and a fifth would be covered
automatically, and the number of character columns visited is pinned at
13 rather than bounded below – a new one fires the test and has to be
classified as a glyph column or not.
Measured where the check’s time goes, since this release’s suite
has grown by roughly 2,100 assertions.
R CMD check --as-cran is 158s of CPU: 100s tests, 12s
re-building the vignette, 7s incoming feasibility (its 143s of wall time
is network, not compute). Inside the suite no single test dominates –
the slowest is 4.5s and 21 tests over a second account for 64% of the
total – so there is nothing pathological to trim, and the whole check
sits comfortably inside CRAN’s per-flavour budget.
Audited the suite for assertions that cannot fail, since the last
two rounds each found a defect in a test rather than in the package. Two
sweeps, both clean. Comparing every block’s expect_* count
in its source against the number it actually executes at runtime found
one shortfall across all 478 blocks, and that one is a
genuine either/or: the tolower() audit asserts against the
R/ sources when they are present and against the installed
namespace when they are not, so exactly one of its two assertions runs.
Scanning for assertions whose two operands are syntactically identical –
expect_equal(x, x), expect_true(TRUE) – found
none.
The second sweep is cheap and the test sources ship, so it is now
a test rather than a one-off: no assertion in the suite may compare
something to itself, and every test_that block must contain
at least one expectation (testthat warns about an empty block but does
not fail, which is easy to miss in a suite this size). Both were
verified by injecting the defects they look for and confirming each
fires.
Added metamorphic tests – properties that must hold
between related inputs. They catch a class unit tests cannot,
because they need no known answer: a verb can be wrong in a way no
fixture reveals and still be caught by “these two corpora must agree”.
Four families, all of which the package already satisfied:
additivity (every glyph’s count over a concatenated
corpus is its count in the parts, and emoji_summary()’s two
counters add); scale invariance (doubling every row
doubles emoji_frequency(), emoji_pairs() and
n_tokens exactly, while share_types,
share_tokens and emoji_collocations()’s PMI –
all ratios – do not move at all); padding invariance
(adding emoji-free rows leaves emoji_frequency(),
emoji_pairs(), top_n_emojis() and the version
profile identical, and moves only the n_total
those rows belong to); and row independence.
Row independence is the sharpest of the four, and it is now a
contract: for thirteen row verbs, verb(data)[i, ] must
equal verb(data[i, ]). Exactly one row verb is allowed to
break it – emoji_incongruity(scale = "rank"), whose whole
purpose is to map each score to its percentile within the
corpus – and the test requires it to differ, so that if rank
scaling ever quietly became scale = "none" the suite would
say so. Verified by injecting a corpus coupling into
emoji_ratio() (dividing by the corpus maximum) and
confirming the test fails.
Extended the same four families to the calendar arithmetic and to
emoji_dfm(). Translation invariance is a strong property
for a cycle, and a bucketing bug would show up as a shift leaking into
the counts: a whole number of years cannot move a month cycle, a whole
number of weeks cannot move a weekday cycle, and in a zone without DST
whole days of seconds cannot move an hour cycle.
emoji_trend() keeps its counts and shifts every period by
one constant offset; emoji_turnover(), which compares
periods to each other, is unmoved entirely. None of the four time verbs
depends on the order of the rows.
Pinned the DST case as behaviour rather than leaving it implicit. Adding 86400 seconds twice across spring forward is not adding two calendar days, so the displayed hours move by one and the cycle moves with them – correct, because the verb buckets by the timestamp’s own zone as documented. The test asserts the buckets are the displayed hours on both sides of the shift, and that the emoji total is conserved across it. (That area turned out to be well defended already: injecting a UTC-instead-of-local hour read fails six tests.)
emoji_dfm() conserves its cells: aggregating rows
into documents with doc_id redistributes them without
creating any, the total equals emoji_frequency()’s,
permuting rows permutes the matrix rows identically, and duplicating the
corpus leaves the first copy’s rows untouched – tf-idf included, since
N and df both double and
log(N / df) cancels.
Checked the session cache, which nothing had examined. The package memoises six things – the reference table, its key set, the sentiment and emotion maps, the ambiguity table and the type map – plus the user-writable lexicon registry. A memoised accessor has a failure mode no other test could see: computing one value, storing another, and so answering differently the first time than afterwards. All six now assert that the first call equals the second and third with the slot cleared beforehand, and that the call actually populates its slot, so the memo is real rather than decorative. Clearing a derived slot is safe because each repopulates from a source that cannot change within a session.
The registry is the one slot a caller can rewrite, so it gets a
coherence check: registering a second table under a name already used
must replace the first rather than be shadowed by it,
emoji_lexicons() must list the name once and report the new
row count, and re-registering with a different glyph column
must still resolve – which it does, because
register_emoji_lexicon() stores a computed key
column. Verified the whole guard by making
emoji_reference() memoise a table one row shorter than the
one it returns; the new test fails.
The submission notes now cite cross-platform evidence instead of
asserting it. cran-comments.md said “0 errors | 0 warnings
| 0 notes” and then explained that the local 1 WARNING and 3 NOTEs are
artefacts of this host. Every flavour of the GitHub Actions matrix now
reports Status: OK for the submitted commit – macOS and
Windows on R 4.6.1, Ubuntu on R-release, R-devel and R-oldrel-1 – and
all five re-build the vignette and run the full suite. None of the four
local findings reproduces anywhere the host is not ours, which is the
evidence for that reading rather than our word for it. The note also
records what CI does not cover: it passes
--no-manual, so the PDF manual is checked only locally.
Each of those claims was read out of the run logs rather than
inferred.
Confirmed the package survives a check without its Suggests,
which some CRAN flavours run. Hiding the Suggests from the library path
turned out not to prove anything here – the R system library carries
readr and stringr and they cannot be masked –
so the property is established by construction instead, and it holds:
neither the package’s own code nor a single example references a
Suggests package. Both are now asserted, reading the namespace
and the help database so the check works in a tarball too.
Every Suggests use in the vignette is properly conditional. Two
chunks look ungated and are not: the setup chunk uses
knitr, which is the VignetteBuilder and
therefore always present, and its library(ggplot2) sits
inside if (has_plot_pkgs). The six plotting chunks that use
forcats and stringr all carry
eval = has_plot_pkgs. One apparent third case was a comment
explaining why readr::read_csv() is not used –
worth noting because a naive scan counts that as a dependency.
Verified inst/CITATION, which nothing had checked
and which is what a user pastes into a paper. It renders three entries –
the package and the two lexicon papers whose data ships here – and every
field agrees with DESCRIPTION: version, URL, author, and
the title, which differs only by the conventional package-name prefix.
The paper details hold against the live records too: the EmoTag page
range 8957-8967 matches the ACL Anthology exactly, and the PLoS DOI
resolves to the right article.
The offline half is now a test, including one assertion aimed at
a specific trap: the file falls back to a hardcoded version string when
meta is unavailable, so the test requires the rendered
version to equal DESCRIPTION’s. At the next release that
fallback goes stale, and this is what will say so.
Measured line coverage for the first time:
99.23% across R/, with fourteen
expressions uncovered. Reading them was more useful than the number.
Thirteen are defensive early-returns and the two unreachable lexicon
fallbacks – if (!nrow(m)) return(s) in the gap slicer, the
non-data-frame guard in the lexicon resolver, the non-character guard in
the encoding check, and so on. Every one guards a condition no caller
can produce, and they are kept: an internal helper that is safe only
because of what its callers happen to do is a helper waiting to be
reused wrongly.
One uncovered line was genuinely reachable and simply untested:
emoji_trend()’s early return when the top_n
cut leaves no emoji to follow. top_n = 0 is a legitimate
request, and the answer is correct – a typed zero-row tibble
whose five columns have the classes a populated call gives, so the two
bind. That is now asserted, together with
emoji_flag_ambiguous(top_n = 0).
Confirmed the generated files are not stale, which nothing had
checked since this loop began editing roxygen blocks: regenerating
man/ and NAMESPACE from the current sources
produces no diff, and RoxygenNote matches the roxygen2 that
would do it.
This release’s prose broke the package’s spell check, and
nothing said so. inst/WORDLIST and
Language: en-GB exist because the maintainer runs
spelling::spell_check_package(); against the commit this
loop started from it reported zero flagged words.
Against the current tree it reported 53 – every one
introduced by the documentation and release notes written here. None is
a misspelling: they are package names (readr,
tidyr, forcats, pandoc), R
vocabulary (formals, AST,
tidyselect, memoise), domain terms
(keycap, dingbats, faceness,
which is already an exported concept) and hyphenation artefacts
(mis from “mis-described”). Only four reach a help page.
All 53 are now in inst/WORDLIST, which is back to reporting
zero.
The word list keeps the ordering it had.
spelling::update_wordlist() re-sorts case-sensitively,
which would have rewritten the whole file and buried 53 real additions
in an 86-line diff; restoring the original case-insensitive order makes
the change exactly 53 added lines and nothing removed.
Corrected two references left stale by moving the vignette corpus
to inst/extdata/ last round – one of them in
NEWS.md, which ships. Re-read the whole tarball manifest
afterwards rather than assuming the move was self-contained: the root
carries only DESCRIPTION, NAMESPACE,
NEWS.md and README.md, and inst/
carries CITATION, WORDLIST, the three vignette
artefacts and the corpus.
Re-examined why the remaining tests skip on CRAN, having just
found that one long-standing skip was hiding a real defect. Four of the
eight were not gated on an unshipped file, so their skip was a choice;
timing them showed cost was not the reason either – 4.7s between them,
out of 123s. Two have a real reason and keep it: the declared-R-minimum
check reads the installed dependencies’ own R floors, and the
select()-avoidance check compares two wall-clock timings.
Both interrogate the checking machine rather than the package, and
neither is a fair question to ask a CRAN flavour.
The other two had no reason beyond caution and now run
everywhere: emoji_dfm() at its widest, which builds a
3791-column table and is precisely the shape most likely to behave
differently on a platform that cannot be tested here, and the audit that
no example except ?register_emoji_lexicon registers a
lexicon – a hazard created by R CMD check running every
example in one session, so a CRAN flavour is where it most needs to
hold. Twelve more assertions now run on all thirteen CRAN flavours, and
the tarball check’s skips fall from ten to eight.
The vignette’s code, as R ships it to users, could not be
run. R installs the tangled introduction.R
precisely so a reader can execute the vignette, and
browseVignettes() offers it. It read its corpus as
read.csv("ata_tweets.csv"), a bare relative path that
resolves while the vignette is being built in vignettes/
and nowhere afterwards: the corpus shipped in the tarball but was never
installed, so the script failed on line 28 with “cannot open the
connection”. The corpus now lives in inst/extdata/ and both
the vignette and the shipped script reach it with
system.file(). The tarball grew by 3.4 KB, because the file
moved rather than being duplicated; the installed package grows by the
file’s 150 KB, which is the cost of the script working.
Two tests stop skipping as a result. They were gated on the
corpus being reachable, which inside R CMD check it never
was – tests run from .Rcheck/tests/, where
../../vignettes does not exist. Now that the corpus is
installed they run everywhere, including on CRAN, and the tarball
check’s skip count falls from twelve to ten.
Swept every number of two or more digits on a help page against
every number appearing anywhere in the test suite. Seventeen were
unaccounted for; twelve are code points, citation years or the CLARIN
handle. The other five were real claims about this package’s data that
nothing checked, and all five are exact: ?emoji_search’s
“189 of the catalogue’s 5042 rows carry no GitHub-style alias” and “7 of
the 198 rows emoji_search("face") returns”, and
?emoji_density’s “115 of the 560 emoji-bearing rows in the
corpus behind the introduction vignette contain a multi-code-point
emoji”. All five are now derived in tests rather than asserted in prose
– the last of the documented figures to be unpinned.
Measured how the verbs scale, which no test could show: every
fixture in the suite is a handful of rows, so an accidentally quadratic
verb would be invisible. All nineteen are linear – at 1k, 4k and 16k
rows the 16-fold input gives 16-fold time across the board, the slowest
being emoji_ngrams() at 6.7s for 16k rows.
Verified the one cost model the documentation actually states.
?emoji_cooccurrence warns that the result “grows with the
square of the distinct emoji in a document” and quotes 319,600 pairs for
800 and 7.2 million for 3790. Both are exact, 3790 really is the
catalogue’s distinct-emoji count rather than an invented ceiling, and
the measured timings are quadratic and land where the page says – 0.09s,
0.24s, 0.81s and 3.03s for 100, 200, 400 and 800, so 800 is the promised
“few seconds”. The arithmetic and the one-row-per-pair property are now
tests; the timing cannot be, so it is recorded here.
A related detail the measurement turned up, in the model’s
favour: cost depends on distinct emoji, not on occurrences.
4000 rows pooled into two documents is faster than 4000 one-row
documents, because a document holding 6000 glyphs drawn from a 10-emoji
vocabulary still has only 45 pairs. 800 distinct spellings
likewise collapse to 726 emoji and 263,175 pairs – exactly
C(726, 2), which is the model confirming itself.
The deployed pkgdown site was checked against this release rather than assumed current: its home, reference index, introduction article and news pages all serve, the index links all 50 topics (51 help pages less the package doc, which pkgdown excludes by design), and the reference pages carry the wording added in this release.
Verified the pkgdown reference index, since a new exported verb
that nobody adds to _pkgdown.yml breaks
pkgdown::build_site() with “topics missing from index” –
after the release, on someone else’s machine.
pkgdown::check_pkgdown() reports no problems: all 51 help
topics are reachable. A hand-rolled version of that check flagged two
entries the real one accepts, because an alias counts as covering its
topic (emoji_tweets lives on emoji_filter’s
page) and the package-level doc is excluded by design – asking the tool
rather than reimplementing its rules was the difference between two
false alarms and a clean answer.
That check is deliberately not in the test suite. Adding
it raised a second R CMD check WARNING – “‘::’ or ‘:::’
import not declared from ‘pkgdown’” – and the only way to clear it is to
declare pkgdown in Suggests, which would make
every CRAN check machine install a heavy dependency for a test that can
never run there: _pkgdown.yml is build-ignored, so the test
would skip in the tarball every time. The site index is verified here
instead, and build_site() catches a regression on the first
run after one.
The cran-comments.md skip-count assertion no longer
hardcodes the number in two places. It counts the
skip_on_cran() call sites and derives the word the file
must contain, so adding a skipped test forces the submission notes to be
updated – which is the point of the coupling – without also forcing an
edit to the test.
The missing-column message added earlier this release listed
every column in data, which is the caller’s data and can be
wide. A 500-column frame produced a 5074-character
error and a 40-column survey export 665 – burying the one name that is
wrong behind the hundreds that are not – and a frame with no columns at
all trailed off as Available: .. Five names are now listed
and the rest counted (and 495 more), so the message is
84-118 characters whatever the data looks like, and a frame with no
columns says so.
A test pins that ?register_emoji_lexicon‘s example
is the only one that registers a lexicon. R CMD check runs
every example in one session, so a registration is visible to every
topic sorting after it, and emoji_lexicons() reports what
is registered – today that verb sorts before
register_emoji_lexicon() and the printed output is
unaffected, which is alphabetical luck rather than design. Running all
46 topics’ examples forward and in reverse also errors nowhere, so
nothing else depends on the order.
?emoji_pairs now says what doc_id
costs. The result has a row per pair, so it grows with the
square of the distinct emoji in a document – a conversation or
a day is cheap, pooling a whole corpus under one id is not. Measured:
800 distinct emoji in one document is 319,600 pairs and 3.5s, and the
bundled corpus pooled into a single document (187 distinct emoji) is
17,391 pairs in 0.27s. The behaviour is inherent to the question rather
than a defect, but the shape of the cost was not written down.
emoji_incongruity_profile()’s statistics are now
derived independently in the tests – the last formula in the package
that no test computed for itself. n,
mean_incongruity, sd_incongruity and
n_flips all match crediting every glyph in a row with that
row’s gap, across all three scale values, along with
flip_rate, the documented sort order, the NA
standard deviation for a single occurrence, and min_n
selecting rows without changing a statistic. No defect found.
Two scale limits nothing had reached are now tested:
emoji_dfm() at its widest – one document holding every
distinct emoji, 3791 columns, all names distinct and non-empty, tf-idf
zero everywhere as log(N/df) requires for a single document
– and a wide multi-document table keeping its documented column ordering
and row sums. Also that a missing column is reported identically for all
four column arguments, naming the argument and the column and never a
same-named variable from the caller’s session.
A misspelled column name could be reported as the
contents of one of your variables. tidyselect falls back to an
external vector when a bare name is not a column, and
text is a common variable name – so
emoji_sentiment(df, text), on a data frame whose column had
been renamed, in a session where text was also a character
vector, gave
Can't select columns that don't exist. Columns `from the global ...` and ...
– naming the contents of the caller’s variable as though they were
column names – plus a tidyselect deprecation warning advising
all_of(), which is not what the caller meant. That is the
same failure mode this resolver was written to replace, arriving by
another route. A bare name is now checked against
names(data) directly: the message names the argument and
the missing column and lists the ones that exist, with no spurious
deprecation warning. Where the column does exist it still wins
over a variable of the same name, and every other tidyselect form – a
string, a position, all_of(), starts_with() –
still resolves through dplyr::select(), two-column
selections included.
That also makes the commonest call much cheaper. Resolving a bare
column name no longer runs dplyr::select(): 2000
resolutions went from 2.0s to 0.058s, and a realistic loop of
500 per-group emoji_sentiment() calls from 1.32s to 0.93s.
Large-corpus timings are unchanged, being detection bound. It also
retires the note added earlier this release about five verbs resolving
the name twice – at 0.03ms a resolution there is nothing left to
restructure.
The time verbs’ buckets follow the session timezone when
the column has none, and nothing said so.
?emoji_trend promised that a date-time is bucketed by the
day it “displays as in its own timezone” – but a POSIXct
built by as.POSIXct("2024-01-01 23:30"), which is what most
CSV readers give you, has no tzone of its own, so R
displays it in the session’s. The same column reports hour 23 in one
session and hour 4 in another; measured across UTC, America/New_York,
Asia/Tokyo and Pacific/Kiritimati, the hour-of-day buckets came out
0/23, 0/1, 13/14 and 18/19. That is correct POSIXct
behaviour rather than a defect – a column with no timezone has none to
display in – but it is the same reproducibility trap as the collation
and ctype dependence this release fixed, and it was undocumented.
?emoji_trend’s time now says to tag the column
(as.POSIXct(x, tz = "UTC")) when the result has to be
reproducible.
Three tests pin the timezone axis, which nothing had varied: a
tagged POSIXct gives byte-identical days, hours
and months across four timezones spanning a date line and a DST
boundary; a Date column is immune across all five bucket
widths; and an untagged POSIXct demonstrably follows the
session, so the documented warning cannot quietly stop being
true.
A time column written
dd/mm/yyyy was read as dates in the year 1, and nothing
said so. as.Date()’s %Y accepts a
one- or two-digit year, so
as.Date("01/02/2024", format = "%Y/%m/%d") is
0001-02-20 – year 1, month 2, day 20. The character branch
tried "%Y/%m/%d" as a fallback, so a whole column in
dd/mm/yyyy or mm/dd/yyyy form – most CSVs
written outside ISO-land – parsed without complaint, and every time verb
bucketed on years 1, 3, 12 and so on. The four-digit year is now
required before parsing, which sends those values to the paths that
already existed for unreadable dates: mixed with real dates they warn
and are dropped, and a column where nothing reads as a date errors.
Every form that worked still works, including "2024-1-1"
and a trailing time.
The error for a wholly unreadable time column now
diagnoses what it found instead of restating the argument’s type, quotes
the value it choked on, and points at as.Date(format = ) –
so the dd/mm/yyyy case above arrives with a route out.
?emoji_trend’s time documents the accepted
forms, that a column with nothing readable is an error rather than
all-NA, and names "01/02/2024" as the case to
convert first.
A bad lexicon was reported against
tbl, an argument the scoring verbs do not have.
emoji_score(), emoji_sentiment() and
emoji_emotion() share their lexicon validation with
register_emoji_lexicon(), whose own formal is called
tbl – so passing a data frame as lexicon
produced No score column found in `tbl`; supply `score`,
telling the user to fix an argument that function does not take. That is
the same failure mode the column resolver’s arg parameter
was added for, and the helpers now take one too: all six messages
reachable through lexicon = name lexicon, and
the three reachable through tbl = still name
tbl. Two of them also read better for it –
Lexicon has no score column `nope` is now
`lexicon` has no column `nope` to take the score from.
Found by inspecting all 43 stop() messages in the
package for whether each names something a caller can actually type. The
lexicon helpers were the only exception; every other message either
names a real argument or fills a %s with one at run time. A
test now enforces it, allowing the argument-with-value form
(period = "hour") that two messages use.
.emoji_text_col() resolved the text column’s name
twice. The one-value-per-row check added this release routed the read
back through .emoji_col(), which resolves the name again –
and resolving it runs dplyr::select(), about a millisecond
a call. Invisible next to detection on a real corpus (20,000 rows is
unchanged at 0.26-0.86s a verb), but the whole cost of a verb called
once per group in a loop over a split data frame. The check now takes
the name it already has, so the read is back to one resolution.
emoji_turnover(measure = ) swallowed a value
it did not recognise. It is the only argument in the package
that takes several values at once, and
match.arg(several.ok = TRUE) returns the ones it matches
without a word about the rest – so
measure = c("jaccard", "flip") produced the
jaccard column and no hint that the second name was wrong.
That is the absorbed-argument failure mode this release swept everywhere
else. An unrecognised value is now an error that names
measure and the value, and an empty, NULL,
NA or non-character measure gets one message
instead of match.arg()’s internal
'arg' must be of length >= 1. What had to survive does:
abbreviations ("jac") still match, duplicates are still
ignored, the four-measure default is unchanged, and the columns still
come back in the fixed statistic order rather than the order asked for.
?emoji_turnover now says all four of those things.
A matrix column passed as time,
doc_id or text_score failed four different
ways – including not failing. A matrix holds one element per
cell, so a 2x2 column carries four values for two rows, and
nothing downstream noticed: a character time gave
emoji_trend() and emoji_adoption_lag()
invalid 'times' argument, emoji_seasonality()
missing value where TRUE/FALSE needed, and
emoji_turnover() a result, computed over
periods that were not in the data. A doc_id made
emoji_dfm() report four documents for a two-row frame, with
the id column filled from rows that do not exist. A
text_score passed the type guard outright – a matrix
is numeric – and reached tibble as
Assigned data `gap` must be compatible with existing data.
The one-value-per-row check added for text last round now
lives in the shared column reader, so all four arguments get it and all
ten verbs give the same message. emoji_dfm() was the one
column read that bypassed that helper; it no longer does, and a test
asserts no other does either.
The type and length checks now report in the informative order. A
data-frame column’s length() is its column count,
so the length check fired first and said nothing about the real problem;
the type check runs before it. A POSIXlt column is
unaffected – R’s length() method counts times rather than
list components – so emoji_seasonality(period = "hour")
still accepts one.
A list column passed as text was scored from
its own deparsed source code. Every verb reads the text column
through one helper that calls as.character() – which is
exactly what lets a factor column work, and is harmless on
a numeric, Date or logical one. On a list column it
deparses: a column holding list(c("a", "<U+1F600>"))
was read as the source text
c("a", "<U+1F600>"), the emoji inside that was
detected, and the row came back with a real-looking
.emoji_sentiment of 0.5718 for data that never contained an
emoji. A data-frame column deparsed the same way. Both are now refused,
with a message that says why; every atomic column still works exactly as
before.
A matrix column errored opaquely in two verbs and
silently over-counted in a third. A matrix holds one element
per cell, not per row, so a 2x2 matrix column gave
emoji_sentiment() and emoji_tokens() a tibble
error naming a variable from this package’s own source
(Assigned data `glyphs` must be compatible with existing data)
while emoji_frequency() happily counted all four cells
against two rows. The shared reader now requires one value per row, so
all six verbs tried give the same message; a single-column matrix has
one element per row and still works.
emoji_score() sent an emotion-lexicon user
down a dead end. It averages the eight emotion dimensions into
one number for the bundled "emotag1200", but that averaging
is special-cased to the bundled table: a registered or inline
lexicon of exactly the same shape has no score column, so the call
failed with “No score column found in tbl; supply
score” – advice that cannot help, since score
names a single column and the user wants the mean. The message now names
both real routes (emoji_emotion() for the profile, or
score = "joy" to score on one dimension) and says the
averaging is specific to the bundled lexicon; ?emoji_score
says the same. A table with neither a score nor an emotion column keeps
the original message. No behaviour changed beyond the message.
emoji_emotion() |> emoji_emotion_label()
silently threw away all eight emotion scores. The label verb
computes the scores internally and then drops every
.emoji_<emotion> column by name – which cannot tell a
column it just added from one that was already in data. So
the obvious way to get the profile and the label kept only the
label, and a caller’s own column of one of those names was deleted
rather than overwritten. Both contradict a @return that
says the label is added. It now drops only the columns that
call introduced: emoji_emotion_label() on plain data
returns exactly what it always did, and the chain now returns all
thirteen columns with the label and the scores agreeing with the
standalone calls. ?emoji_emotion_label says so.
Found by sweeping the pipelines a user would actually write,
which nothing had checked – every verb had only ever been tested on its
own, against a package whose headline claim is that the verbs “compose
naturally with the pipe”. Fourteen chains are now tests, along with the
grouped contract end to end (group_by() through a row verb
into summarise(), grouping surviving a chain, and a
text-rewriting verb re-deriving the groups when the user grouped by that
very column), re-running ten row verbs on their own output, and feeding
four aggregators a row verb’s output column.
?emoji_cooccurrence said it “is
emoji_pairs() under another name, with one addition”, and
it also silently drops an argument. There is no
directed on emoji_cooccurrence(), so a reader
taking that sentence literally gets
unused argument (directed = TRUE) with no explanation. The
omission is right – a co-occurrence matrix is symmetric, so an ordered
pair has no meaning on it, and neither would the diagonal the verb
exists to add – but only the addition was documented. The description
now states both, with the reason and where to go for
directed, and a test pins the two signatures as differing
by exactly directed and diagonal.
Found by a cross-product sweep of every enum and flag argument –
182 combinations across seventeen verbs, now a test. method
x scale x where for the three mismatch verbs,
weighting x doc_id for
emoji_dfm(), directed x sort x
diagonal x doc_id for the relational pair,
by x measure x top_n for
emoji_trend(), all four measure subsets of
emoji_turnover(), and the rest. Every combination that
exists returns a data frame with nameable columns; the one that did not
is the item above.
emoji_tokens() was the one verb that did not
validate data, and a bare list or a character matrix
silently succeeded. It called .emoji_as_tibble()
before resolving the text column, and
tibble::as_tibble() converts a list or a matrix quite
happily – so the is.data.frame(data) guard every other verb
enforces never saw the original object.
emoji_tokens(list(text = c("a", "b")), text) returned a
result where the other twenty-five verbs error, and NULL, a
number, a function and an environment reached dplyr’s or R’s own
internals (cannot coerce class "function" to a data.frame)
rather than the package’s message. The column is now resolved first, so
all twenty-six verbs give `data` must be a data frame for
all seven wrong shapes. A data frame that simply lacks the column stays
a different error, delegated to dplyr::select() so the
message names the column.
The detection figures this release quotes were wrong, in
NEWS.md and in cran-comments.md both. They read
“exact detection of the reference table goes from 80.1% to 95.8%” and
“spellings that lose a joiner from 793 to 2”, which pairs a no-repair
baseline with an after-rules-1-and-2 one. Staging the rules and
measuring each: no repair gives 63.2% (3189 of 5042
spellings read as exactly one emoji equal to the whole spelling) and
1643 orphaned joiners; rule 1 alone, which is UAX #29’s GB11, gives
75.5% and 1025; rule 2 leaves 793; rule 3 takes those to 2. No staging
produces 80.1%, and 793 is where rule 3 starts, not where the
three rules together do. Both documents now give 63.2% to 95.8% and 1643
to 2, with the intermediate figures stated so each is attributable, and
a test derives all six from the rules.
Four help topics had no @seealso at
all – and they were the four other pages point at:
emoji_categorize(), emoji_emotion_label(),
emoji_extract_unnest() and emoji_filter(). A
reader arriving at one from the reference index or a search had no route
onward, while the other 44 exports did. Each now points at its pair and
its neighbours, and a test requires every exported topic to carry
one.
?emoji_sanitize’s reversibility table
promised more than wrap allows. It says
policy = "shortcode" restores the original text, calls it
“the only policy that permits it”, and quotes a 100% byte-exact figure –
but the section never mentioned wrap, an argument of the
same function that voids the promise. text_to_emoji() looks
for exactly :shortcode:, so measured across eleven
templates: the default ":{x}:" restores the text exactly; a
decorating wrap ("[:{x}:]", ":{x}:!") brings
the emoji back but leaves the decoration; a wrap with no
:token: ("{x}", "<{x}>",
"@{x}@", ":{x}", "{x}:",
"a{x}b") restores nothing at all and does so
silently, leaving the shortcode in the text as an ordinary
word; and "::{x}::" brings the emoji back wrapped in
leftover colons, so it is not even stable under a second pass. All four
cases are now in the section, with the precondition stated on
?emoji_to_text’s wrap and in
?text_to_emoji’s opening sentence, and pinned by a
test.
The grouped-input warning is now asserted for all fourteen cross-row aggregators at once: each warns exactly once, names itself rather than the helper or the verb it delegates to, and does not tell the user to report a bug in tidyEmoji. Previously one verb was spot-checked.
The introduction vignette could not be built without its
Suggests packages, so R CMD build failed outright on a
library tree lacking one. It read its example corpus with
readr::read_csv() and drew its charts with
ggplot2, forcats and stringr, all
unconditionally – optional dependencies used as if they were required.
Verified by building on an R 4.6.0 tree with neither ‘readr’ nor
‘forcats’: Error: Vignette re-building failed. The corpus
is now read with utils::read.csv() wrapped in
tibble::as_tibble() (byte-identical full_text,
same encodings, same 900 emoji tokens, identical verb output), and the
nine plotting chunks are gated on a requireNamespace()
flag. The vignette builds either way, and the readr Suggests entry now
exists only for one guarded test.
Five documentation-pinning tests were silently inert on
every CRAN flavour. They read ../../man/*.Rd,
which exists only when the suite runs from the source tree; inside
R CMD check the tests execute from
<pkg>.Rcheck/tests/testthat/ and skipped with “man/
not available”. So every assertion that a figure or a column name really
appears in the rendered help page – the mechanism several of this
release’s fixes rely on – was never actually checked where it mattered.
They now read the installed help database via
tools::Rd_db(), whose as.character()
reconstructs the markup, so \code{...} matches still work.
Two more tests were reaching for NEWS.md,
DESCRIPTION and the vignette source the same way; all three
are installed with the package, so they read the installed copy when the
source tree is absent. Skips inside R CMD check drop from
14 to 6 – two deliberate skip_on_cran() maintenance checks,
three that genuinely need R/*.R, and one that needs the
vignette’s CSV – and 160 more assertions run there than before.
cran-comments.md contradicted itself about this
host’s check result, reporting both “1 WARNING and 3 NOTEs” and, three
paragraphs later, “one WARNING and two NOTEs” for the same run. The
duplicate paragraph is gone. Its other stale figures are corrected or
removed: R 4.6.0 gives Status: 2 NOTEs rather than
Status: OK, the package has 16 URLs rather than the 12
claimed (so the count is no longer asserted – only the one URL that is
reported), and the CRAN-index size is dated as a snapshot. The three new
dependency floors and the vignette change above are now described there
too.
?emoji_density said “emoji per character”
without saying that a character is a code point, which reads as
per-grapheme and is not. A multi-code-point emoji inflates the
denominator by all of its code points, so the same visible text gives
different answers: "hi <emoji>" is four graphemes
either way, but .emoji_per_char is 0.25 for a
single-code-point smiley, 0.200 for a two-code-point flag
and 0.100 for a seven-code-point ZWJ family. 115 of the 560
emoji-bearing rows in the vignette’s corpus contain a multi-code-point
emoji, so it is not an edge case. emoji_ratio() had always
stated this basis and emoji_position() states the opposite
one (each emoji counts as one position, because a proportion of a
message has to); emoji_density() sat between them saying
neither. It now states the basis, gives those three figures, and points
at .emoji_per_token, which does not move with an emoji’s
internal length. This closes the last open action from the audit that
fixed .emoji_rel_position: “grep for other
nchar() uses on user text”. All thirteen are now accounted
for – four feed a documented user-facing figure, nine are internal
offsets.
cran-comments.md also undercounted its own
skips. It said “seven tests skip, all seven
skip_on_cran()”, and a CRAN run prints ten: the seven, plus
three that stand down because a tarball does not carry the file they
read (README.Rmd, the package sources, and the
build-ignored .github/). A reviewer running the check would
have seen the letter contradicted by its own output. The test that
couples the count only ever counted skip_on_cran() call
lines, which is why it never noticed, so it now also requires the file
to name each stand-down message that exists in the suite.
cran-comments.md said five CI flavours when
the matrix defines six. The workflow gained an
r: '4.1' job precisely so the declared floor of R 4.1.0 is
built against, oldrel-1 sitting far above it, and the file
a reviewer reads to learn what was tested went on saying “five” in three
places and listing the other five. It now says six, names the 4.1 job (R
4.1.3 on that runner) and says why it is there. A test couples the
spelled-out count and every runner name to the workflow itself, so the
two cannot drift again; it reads a build-ignored file, so it runs from
the source tree and stands down inside R CMD check. The
stale claim inside the test that checks the declared minimum is
corrected too: it said CI cannot reach the floor, which stopped being
true when that job was added.
Nothing guarded the figure references, and this package
has already shipped a broken one.
man/figures/lifecycle-deprecated.svg was missing while
?top_n_emojis pointed at it, and no check caught it: a
\figure{} sits inside the html branch of an
\ifelse, so the PDF manual takes the other branch and never
looks. The figure was restored earlier in this release and then left
unprotected. A test now reads every figure name out of the installed
help database and requires the file to exist and be non-empty, in
man/figures from the source tree or
help/figures from an installed package, which is where
R CMD check reads them. It also pins the two references
that exist today, so a reference vanishing is as visible as a file
vanishing, and checks that the README points at the logo by its
repository path, which is what makes it render on the CRAN package
page.
The vignette builds without its optional packages; the
script knitr tangles out of it does not, and nothing said
so. A chunk’s eval option is a build-time
instruction and does not survive into the extracted code, so
inst/doc/introduction.R – the file
edit(vignette("introduction", package = "tidyEmoji")) opens
– calls ggplot() unconditionally, while
library(ggplot2) above it stays behind the
requireNamespace() gate. Sourcing it on an installation
without ggplot2, forcats and
stringr therefore fails partway through, which is exactly
the situation the gate was written to survive. The vignette now says
which half of the script runs on the declared dependencies alone and
what to install for the rest, and a test holds that claim to the
installed file. Nothing changed in the code: the chunks stay ungated so
the rendered vignette shows the plotting code as a reader wants to read
it, and dropping it from the tangled script would take working code away
from everyone who does have the three packages. Verified separately that
the script runs to completion, with zero warnings, when they are
present.
The other four companion arguments are inert on the
branches they do not belong to, and now say so.
emoji_sanitize()’s placeholder and
wrap each serve one policy, and
emoji_score()’s by and score only
apply when lexicon is a table rather than a name. All four
were quietly ignored elsewhere, changing nothing but leaving the reader
to infer it. Each @param now says which branch it belongs
to and that it is ignored off it, as
?emoji_to_text already did for wrap under
format = "name". They stay silent deliberately, and the
page says why: policy and lexicon are meant to
be variables, so a script sweeping all five policies with one
placeholder = set would be warned at four fifths of its
calls. That is the line between these four and the
threshold above, which is a number the caller chose in
order to move a result. emoji_sentiment(se =) is the one
companion that cannot be ignored, needing annotation counts only the
bundled lexicon carries, and it already errored. Tests pin all of it:
silent, and byte-identical to the call without the inert
argument.
threshold did nothing under
method = "sign_flip", quietly. That method flags a
polarity flip rather than measuring a gap, so there is nothing for a
threshold to cut, and a call giving both had half of it absorbed without
a word. It is the same shape the package refuses elsewhere:
emoji_turnover() errors on a measure value it
cannot honour, and emoji_to_text() on a wrap
with no placeholder. Here the answer is right rather than wrong, so the
proportionate signal is a warning, and it fires only when the caller
actually typed the argument, after match.arg() so an
abbreviated method = "sign" is caught too. All three verbs
of the family do it. The page also records that a threshold at or below
0 flags every scored row, abs(gap) never being
negative.
Three verbs take a top_n and only one said
how it cuts. ?top_n_emojis records that a tie
straddling position n is settled by the glyph, and that a
corpus with fewer emoji than n returns all of them rather
than padding. emoji_trend() and
emoji_flag_ambiguous() do the same thing and said neither,
so a reader could not tell whether their cut was arbitrary. Both now say
it, and a test drives all three with four glyphs tied on every key so
that only the glyph order can decide, and checks top_n
above the available count and top_n = 0 on each.
emoji_emotion_label() labelled a row with no
dominant emotion "anger".
max.col(ties.method = "first") returns the first column
when nothing wins, and the first of the eight in Plutchik order is
anger, so a custom lexicon scoring an emoji the same across all eight
dimensions – zero across the board, most obviously – came back asserting
anger. The row now gets NA, which
.emoji_n_scored still distinguishes from a row with nothing
to score, since that count stays at its value rather than becoming
NA too. A genuine winner is untouched, and so is an
ordinary two-way tie, which the documented Plutchik order still settles.
Only reachable through lexicon =: none of the bundled
lexicon’s 150 glyphs scores flat across the eight.
?emoji_emotion_label said ties are broken in
Plutchik order without saying it happens. It does: 3 of
the bundled lexicon’s 150 glyphs tie for their top emotion, and
because Plutchik order is alphabetical the tie-break quietly favours the
early names. U+1F3A4 scores anticipation and joy at 0.39
and is labelled anticipation; U+1F619 scores joy and trust
at 0.83 and is labelled joy. The page now names both, so a reader knows
the rule is load-bearing rather than theoretical, and a test holds all
three ties to it.
emoji_seasonality()’s
emoji_per_text had no definition, and the table hands you
both candidate denominators. It is
n_emoji / n_texts, an average over every text in
the level rather than over the ones carrying an emoji, and the two
readings are far apart on a mixed corpus: four emoji spread over three
texts, one of which carries all four, is 1.33 one way and
4 the other. The @return listed the column
name and stopped, while printing n_texts and
n_with_emoji side by side, so a reader had no way to tell
which had been used. It now gives the formula, the worked contrast and
the one-line recipe for the other reading, and says the column is
NA rather than 0 where a level holds no text.
emoji_trend()’s share was already precise
about its own denominator; a test now also pins the consequence that a
top_n cut does not renormalise it, so a followed glyph
keeps the share it had in the full call.
?tidyEmoji’s contract glossed
.position once, and it means two different things.
The page lists five structural index names and describes them together,
.position among them as “the character offset of an
occurrence”. That is right for emoji_context(), where it is
a code-point offset into the text and
substr(text, .position, .position + nchar(.emoji) - 1)
returns the glyph. It is wrong for emoji_ngrams(), where
.position is the index within the row’s emoji sequence, so
the first n-gram of a row is 1 whatever the text looks like. Both verb
pages were already right about their own column; the contract that a
reader meets first was not. It now singles .position out as
the one of the five whose unit depends on the verb, and
?emoji_context names the unit and the round trip rather
than saying “character position”, which is the vagueness
?emoji_position had already avoided for
.emoji_first and .emoji_last.
emoji_context(unit = "char") can return half
a grapheme, and did not say so. “Character” there means code
point, the unit nchar() and substr() count, so
a window can begin or end part-way through a cluster: where an
e carries a combining acute (U+0065 U+0301),
window = 1 hands back the bare U+0301, a
diacritic with nothing to sit on. ?emoji_density and
?emoji_ratio both spell the code-point basis out already;
this is the verb where it shows up in the output rather than
only in a denominator, and it was the one that did not mention it. The
unit argument now says it, names that case, and points at
the two ways round it (a word window, or a couple of characters more
than you need). Emoji are not affected, being masked out whole before
the window is cut.
The README comparison would have failed on the declared R
minimum, for a reason outside the package. It re-renders
README.Rmd and compares every #> line byte
for byte, normalising the three ASCII markers testthat
forces. Column padding was not normalised, and it moves without
the package moving: pillar pads to a glyph’s display width, and
utf8 has changed its mind about the East-Asian-ambiguous
ones. U+263A is width 2 under utf8 1.2.1,
which is what R 4.1.0 ships, and width 1 under 1.2.6. One column shifted
by one space and the whole comparison failed. Runs of spaces are now
collapsed before comparing, so every value, column name and row count
still has to match while the padding cannot decide it. Found by
re-running the suite on R 4.1.0, which is the only failure it produced:
14664 pass, 1 skip (the readr (>= 2.0.0) gate, that tree
carrying 1.4.0).
Nothing said which verbs merge two spellings of one
emoji, and they do not all agree. Fourteen verbs report a
glyph. Six hand back the spelling they found
(emoji_frequency(), top_n_emojis(),
emoji_tokens(), the two extraction verbs and
emoji_context()), because an occurrence is the text you
had. Eight collapse both spellings onto the catalogue’s
(emoji_pairs(), emoji_cooccurrence(),
emoji_ngrams(), emoji_dfm(),
emoji_collocations(), emoji_trend(),
emoji_adoption_lag() and
emoji_incongruity_profile()), because an item, a node, a
feature or a series is one thing per emoji; and
emoji_turnover() and emoji_version_profile()
count that way in n_types without reporting a glyph at all.
On a corpus holding both spellings of U+1F642 U+200D U+2195
the split is one line: emoji_frequency() returns two rows
of n = 1 where emoji_cooccurrence() returns
one node of n = 2. ?tidyEmoji gains a
Which spelling comes back section naming both lists and the
remedy (the name is the same for both spellings, and
as_emoji_name() supplies it for a column of your own), and
the four verb pages that were silent point at it. Only eight of the
fourteen had said anything, and the sentence ?tidyEmoji
already carried – that the key “resolves identically in every verb” – is
about the metadata join and invites exactly the wrong generalisation. A
test derives the split from behaviour, so a verb that changes sides
fails rather than quietly contradicting the page.
Two of the four time verbs never said an undated row is
dropped. emoji_trend() and
emoji_adoption_lag() both state it;
emoji_seasonality() and emoji_turnover() did
not, and seasonality is where the omission bites, because it returns
every level of the cycle whether the data reaches it or not. The table
therefore looks complete while every count in it is over the
dated rows: on a four-row corpus with one undated row,
sum(n_texts) is 3 rather than 4, and an emoji sitting in
that row reaches neither n_emoji nor share.
Both pages say it now, seasonality with the consequence for its own
columns and a pointer to [emoji_summary()] for counting the corpus
itself. A test holds all four pages to the same sentence and checks the
arithmetic behind it.
?emoji_lexicons called n the
“number of emoji”, and it is a row count. For the two bundled
lexicons those coincide, 969 and 150, because each has one row per
code-point key, and the page now says so rather than leaving it to luck.
A registered lexicon need not: two spellings of one emoji are two rows
and score one glyph, and a row whose glyph yields no key at all is
counted here and matched never. It is the distinction
?emoji_provenance already spells out for
n_emoji, missing from the one other place the package
reports a lexicon’s size.
?register_emoji_lexicon did not say that a
repeat registration replaces. It does, silently and completely,
and since there is no public counterpart that removes a registration,
re-registering is the only way to change a lexicon. The page says that
now, and that a key column already in the caller’s table is
replaced rather than trusted, the registry looking a lexicon up by that
column.
emoji_token_cost(tokenizer = ) took its
answer on trust. The function is the widest surface the verb
has, and three wrong answers from it were silent. A negative count went
straight into .emoji_token_estimate, which is the nonsense
top_n_emojis(n = ) already refuses. An infinite or
out-of-integer-range one became NA through R’s own
NAs introduced by coercion to integer range, a warning
naming neither this argument nor this verb. And a data frame is a list,
so lengths() counted its columns and the shape
check passed whenever the caller happened to have as many columns as the
data has rows. All three now error, naming the values that came back.
NA is still accepted and passed through, for a tokeniser
that cannot answer for a row, and a fractional count still rounds
up.
A user lexicon’s Inf was counted as a
score. The score column is refused if it is not numeric, on the
stated reasoning that a value mean() cannot use must not be
reported as scored, and a non-finite text_score warns and
is treated as missing in emoji_incongruity() for the same
reason. Inf fell between the two: numeric, so the type
guard passed, and not NA, so .emoji_n_scored
counted the emoji and .emoji_score came back
Inf, while NaN and NA in the same
column were already counted as unscored. One column, two answers for the
same kind of unusable value. A non-finite score now warns and is treated
as missing wherever a lexicon is read, so emoji_score(),
emoji_sentiment(), emoji_emotion() and a
registered lexicon all agree.
A custom emotion lexicon skipped the type check its
sentiment counterpart makes. A character or factor emotion
column reached as.matrix() and failed later with R’s own
'x' must be numeric, naming neither the argument, the
column nor the verb: the failure mode the score-column guard exists to
replace. emoji_emotion() now names the offending columns
and their classes. Logical and integer dimensions are numbers and stay
accepted.
An all-missing emotion dimension reported
NaN, not NA.
colMeans(na.rm = TRUE) over a dimension no emoji in the row
has is 0/0, and the NaN reached the output column. Every
other verb reports an unknown value as NA, and
emoji_sentiment() returns NA_real_ for exactly
this case. Only reachable through lexicon =, since the
bundled emotion lexicon has no missing cells.
?emoji_incongruity’s invariance claim reads
as more than it holds. The page says the two scalings are
computed over the rows the comparison is defined on, so rows with no
scorable emoji cannot move the answer, and subsetting to the scored rows
before calling gives the same numbers. Both are true, and both are easy
to read as “the gap is a property of the two scores”. It is not:
"rank" and "zscore" are relative, so a gap’s
size depends on how many scored rows it was computed over. The sharpest
case is duplicating a scored corpus exactly, which adds no information
whatsoever and still multiplies every rank gap by
2 * (n - 1) / (2 * n - 1) (0.933 at n = 8,
0.995 at n = 100, 0.999 at n = 500) and every
z-scored one by sqrt((2 * n - 1) / (2 * (n - 1))), because
sd() divides by n - 1. The page now says where
the invariance stops, and what to reach for when a number has to travel
between corpora. Found by a sweep that runs every verb on a corpus and
on the same corpus with every row repeated: counts must double, per-row
measures and every documented share must not move, and
emoji_collocations()’s PMI must not either, being a ratio
of probabilities. Everything else in that sweep already held, and all of
it is now pinned, the scaling factors included.
Six help pages named a sort key but not enough of them to
order the output. Every verb here sorts deterministically, and
had been pinned as doing so, but a reader following the
@return could not predict the result.
?top_n_emojis is the clearest case: it promises descending
n with ties broken by the glyph, “so the order is
deterministic”, while duplicated = TRUE leaves several rows
sharing both. The answer, now written down and pinned over 200 catalogue
glyphs, is emoji_unicode_crosswalk’s order for the glyph’s
codepoint key, with the first row carrying the same name
duplicated = FALSE reports; the same paragraph records a
consequence of joining on the key rather than the spelling that the page
had never mentioned, namely that a glyph collects the aliases of every
spelling of itself, so the unqualified U+26F9 U+200D U+2640
is listed under woman_bouncing_ball and under the two
aliases its fully-qualified spelling carries. The other five:
emoji_collocations() sorts by measure
descending, then the other of the two, then the glyph and the word,
where ties in pmi are the common case rather than the
exception; emoji_trend() by period, then
measure, then the glyph, which is what separates the zeros
it fills in; emoji_incongruity_profile() by
flip_rate, then n, then the glyph, where the
first key is 0 for every glyph that never flips;
emoji_adoption_lag() by n then the glyph; and
emoji_version_profile() oldest first, with the emoji whose
version the reference table cannot give pooled into one last row rather
than a first one. A test now sorts each result independently by the
documented keys and requires the two to agree, and requires each to
survive a permutation of the input rows.
R CMD check --as-cran now passes under
LC_ALL=C as well. Examples, the vignette build and
the full test suite all run there, and the result is the usual host
artefacts and nothing else. It had never been tried, and the four test
failures the previous entry describes are what it found.
Four tests and six warnings were a latent red on a
non-UTF-8 checking flavour. Nothing had ever run the suite
under LC_ALL=C: the earlier locale sweeps compared verb
output, which is locale-proof, and R CMD check
runs in whatever locale the machine has. Doing it gives 4 failures and 6
warnings, and not one of them is the package. R transliterates a string
it cannot represent natively on the way into a symbol and on the way out
of deparse(), so rlang::sym("<glyph>")
becomes the symbol <U+0001F602> and
as.character(list(c("a", "<glyph>"))) deparses to the
escape; three tests were measuring that rather than anything here,
including the README render, since pillar prints an ASCII tibble there
too. The three are now gated on l10n_info()$"UTF-8" with
the reason given, the two AST scans that parse() every test
file no longer report R’s “unable to translate” warning against
whichever test is running, and the three marker characters the README
comparison needs are built with intToUtf8() instead of a
\uXXXX escape, which is what emitted those warnings at
parse time before any test had started. Under LC_ALL=C the
suite is now 14074 pass, 0 fail, 0 warn, 3 skip.
?emoji_position never documented the
convention its own tests called documented. A text that
collapses to a single position, meaning one emoji and nothing else at
all, cannot tell its start from its end, and
.emoji_rel_position is 0 there. The test
asserting it said “documented as 0” and the help page said so nowhere.
It matters for the one filter the column exists for:
.emoji_rel_position > 0.8 finds sentence-final emoji and
skips every emoji-only message, which is the commonest shape in a chat
or reaction corpus. The page now states the convention, points at
emoji_ratio()’s .emoji_only for finding that
family, and records that one character either side settles the ambiguity
(a trailing space scores the emoji 0, a leading one
1). Pinned for a one-code-point glyph and a ZWJ family,
since the convention counts positions rather than code points.
The reproducibility claim behind the bundled data is now
verified. Every dataset’s @source names the
data-raw/ script that builds it and the vignette says they
“are regenerated from the current Unicode emoji list by the scripts in
data-raw/”, but nothing had ever re-run them. All four
reproduce their .rda byte-for-byte against
emoji 16.0.0: emoji_unicode_crosswalk (5761
rows), category_unicode_crosswalk (10),
emoji_sentiment_lexicon (969) and
emoji_emotion_lexicon (150). The two crosswalks derive
purely from emoji::emojis, so a skip_on_cran()
test now rebuilds them and requires an exact match – which will also
flag the datasets as stale the next time emoji ships new
glyphs.
data-raw/emoji_emotion_lexicon.R cited
next_release.md §4.1 for why the lexicon is keyed on a
selector-stripped code point. That file is a planning document rewritten
each release, and §4.1 is now about skin-tone modifiers, so the pointer
had gone stale. The reason is stated in the script instead.
Two dependency arguments the package relies on had no declared version floor, and one of them would have stopped the vignette building. Found by running the suite on R 4.1.0 – the version DESCRIPTION declares as the minimum – and on R 4.6.0.
lifecycle::deprecate_warn(env=, user_env=) does not
exist in lifecycle 1.0.0; it arrived in 1.0.3, whose release notes are
exactly the caller-environment work those arguments were added for.
Every grouped-input call in the package goes through
.emoji_warn_grouped(), so on an older lifecycle a dozen
verbs would have failed with “unused argument”. Now
lifecycle (>= 1.0.3).readr::read_csv(show_col_types=) arrived in readr
2.0.0. R 4.1.0 here ships readr 1.4.0, where the introduction vignette
fails to build and the corpus regression test errored instead
of skipping – its guard checked that readr was installed, not that it
was new enough. Now readr (>= 2.0.0) in Suggests, and
the test gates on minimum_version = "2.0.0". The suite now
passes on R 4.1.0, 4.4.1 and 4.6.0.Two tests guard the class. One walks the AST of every
R/ file and checks that each named argument passed to a
pkg::fun() call is a formal of that function, so an “unused
argument” cannot hide until someone runs against the version that lacks
it. The other holds a table of the five arguments the package would
break without – lifecycle’s env/user_env,
dplyr’s relationship and .locale, tidyr’s
delim, readr’s show_col_types – and requires
DESCRIPTION to declare a floor at least as high as each one
needs.
?emoji_unicode_crosswalk and the vignette
both called it “one row per emoji name”, and a join by that column
silently duplicates rows. The mapping is many-to-many in
both directions: 5761 rows cover 4698 distinct names and 4853
distinct glyphs. A glyph repeats for every alias it has (the documented
direction), and a name repeats for every spelling of the emoji
it names – 973 do, because the qualified and unqualified forms are
separate rows sharing one alias, so A_button_blood_type_
names both U+1F170 U+FE0F and the bare
U+1F170. Both places now give the row unit as a (name,
glyph) pair and say to join on key, which collapses the
spellings. A test pins all three counts and that a repeated name never
spans two different emoji.
The introduction vignette said two time verbs need no
timestamp; only one does. emoji_version_profile()
runs without a time column;
emoji_adoption_lag() uses the same version lookup but still
has to find each glyph’s first appearance, so it requires one – as the
vignette’s own next sentence says two lines later. The claim came from
misreading emoji-time.R‘s “two of these are almost free”,
which meant needing no new dataset. Both the sentence and the comment
that caused it are fixed, and a test derives the count from the verbs’
formals.
The vignette no longer teaches a superseded function. It reached
for tidyr::separate_rows(), which tidyr marks
[Superseded] and says will “only receive critical bug
fixes”, pointing at separate_longer_delim() instead; since
the vignette is the package’s primary teaching document, that is the
call a reader copies. Switched, with tidyr (>= 1.3.0)
now declared to match – the same release vintage as the existing
dplyr (>= 1.1.0) floor. A test scans the vignette, the
README and every R/ file for fourteen superseded tidyverse
calls so none creeps back, and the vignette’s overview table is asserted
to name every export bar the soft-deprecated
emoji_tweets().
emoji_ambiguity()’s entropy column held
negative zero on 166 of its 969 rows.
-sum(1 * log(1)) is -0 for a glyph its
annotators were unanimous about. It compares equal to zero and prints as
0, so nothing downstream was wrong – but
sprintf("%.3f", x) renders it -0.000, and
formatting a table for a paper is the commonest thing anyone does with
that column. An entropy of -0.000 reads as a bug in the
package.
?emoji_ambiguity never warned that the head
of its ranking is an artefact of thin evidence. The lexicon’s
annotation counts run from 1 to 14622 – median 18, and 69% below 50 –
and entropy, Gini and neutral share are shape statistics that do not
care how many annotations produced the shape. Three annotators splitting
one-one-one score the maximum entropy of log(3), which is
why five of the rows tied at rank = 1 have 3, 3, 3, 9 and
15 annotations, and why 11 of the top 20 have fewer than 50. So
head(emoji_ambiguity()) – the help page’s own first example
– returned box-drawing characters and arrows, which reads as the
function being broken. The details now say to filter on
n_annotations first (as the introduction vignette already
did), and the example demonstrates the filter. They also say where the
bias is not: entropy is positively correlated with the
annotation count overall (Spearman 0.56), because a thinly annotated
glyph is usually unanimous. What three annotators can do that thousands
cannot is hit the exact maximum. "ci_width" is named as the
measure that accounts for thin evidence by construction, being a Wald
interval that scales as 1 / sqrt(n) at a given spread – its
marginal correlation with the count is only -0.19, so the formula is the
claim, not the correlation. Behaviour is unchanged.
?tidyEmoji told readers to supply “the
emoji-presentation (qualified) form” to fix an undetected glyph, and for
12 emoji that advice does not work. The variation selector does
not always go at the end. For 200 of the 212 catalogue spellings
detection skips, appending U+FE0F repairs them; the
exceptions are the keycap sequences – #, * and
0 to 9 followed by the enclosing keycap mark
U+20E3 – where the selector belongs between the
two. U+0031 U+FE0F U+20E3 is detected;
U+0031 U+20E3 U+FE0F is not, so a reader who appended got
the same zero count and no hint why. The page now gives the rule that
repairs all 212 (insert U+FE0F after the first code point)
and names the exception. It also now states the 212 figure alongside the
existing 216, since those count different things – spellings that are
themselves undetectable, versus spellings that break when the selector
is dropped – and an unequal pair of numbers about the same subject
invites one being “corrected” into the other.
Detection is now tested in context, not only on bare glyphs. Every one of the 5042 catalogue spellings is checked beside words, punctuation, brackets, a bare letter, an orphan joiner, a stray selector and another emoji on either side: the glyph count never changes, and next to a second emoji the split is exactly the two of them (bar the 212 undetectable spellings, which contribute nothing). Span well-formedness – ordered, non-overlapping, inside the string, and slicing back to the glyph – is asserted over five string shapes. No defect found; the invariant every verb rests on simply had no test outside the bare case.
One infinite text_score made every row’s
mismatch result infinite. scale = "zscore" takes
sd() of the column, which is NaN as soon as
any value is infinite; the degenerate branch then subtracted an infinite
mean, so the whole column came back Inf or
NaN. Measured on 100 rows with a single Inf:
0 of the 60 scorable rows had a finite gap. A
non-finite text_score is now treated as missing – which is
what NaN already was, since na.rm = TRUE drops
it – and reported, in the style of the unreadable-date warning. The
damage stays on its own row: the same 100 rows now give 59 finite gaps
under all three scales.
?emoji_emotion’s @return said the long
form carries .emoji_emotion and .emoji_score
“instead of the eight columns”, which reads as a promise that
.emoji_n and .emoji_n_scored survive. They do
not – long = TRUE returns neither. That matters because the
same help page opens by telling you to read .emoji_n_scored
alongside .emoji_n before concluding a corpus carries no
emotion, advice a long-form user could not follow on a lexicon covering
4% of emoji. The @return now says the counts are dropped
and where to get them. Behaviour is unchanged: the long form has had
this shape since 0.3.0.
Three claimed equivalences now have the differential tests they
never had. .emoji_window_at() slices a bounded piece of the
masked text rather than the whole side, on the argument that only the
nearest window tokens can matter;
.emoji_final_glyphs() walks back over blank gaps rather
than searching for the longest qualifying suffix directly. Both are now
compared against a naive implementation over randomly generated text
(thousands of comparisons each, across window 0 to 13 and
both unit values), and the eight worked examples in
?emoji_incongruity’s “ends the text” paragraph are asserted
one by one. No defect found in either – but neither was tested, and an
optimisation that is only nearly equivalent is the kind that survives a
release.
scale = "rank" and
scale = "zscore" in the text-emoji mismatch verbs were
scaled over the wrong population, and reported mismatch that was not
there. Each side was ranked (or standardised) over its own
non-missing rows. text_score is normally present on every
row, while the emoji score is missing wherever a row has no scorable
emoji, so the two percentiles referred to different populations and the
emoji-free rows – which contribute nothing to the output, their own gap
being NA – shifted the answer for the rows that do. On 100
rows whose emoji sentiment equals their text score exactly, so the true
gap is 0 throughout, adding 100 emoji-free rows with higher
text scores moved the mean .emoji_incongruity to
+0.50 on the rank scale and +0.92 on the
z-score scale, and made .emoji_incongruent flag 54
of the 100 as mismatched. Both scales are now computed over the
rows the comparison is defined on, so subsetting to the scored rows
gives the same numbers as calling on everything.
emoji_congruence() and
emoji_incongruity_profile() share the engine and are fixed
with it. scale = "none" was never affected and is unchanged
– which is why this survived: 34 of the 40 scale =
arguments in the test suite were "none", and the six that
were not asserted only bounds, NA placement and the
degenerate one-row branch, never a value against a known truth. That is
now tested.
?tidyEmoji’s naming contract described two
classes of output column and the package has three. Five dotted
names are not .emoji_* – .row_number,
.position, .period, .period_prev
and .period_label – and they are the ones a reader is most
likely to meet first, since emoji_extract_unnest(),
emoji_context(), emoji_dfm() and the three
time verbs lead with them. The page now names all three shapes, says the
structural-index list is complete, notes that emoji_dfm()’s
column names are glyphs, and says that the reserved-prefix rule applies
to the text column too if you happen to have named it
.emoji_n. A test asserts no verb invents a fourth dotted
name and that the page lists the five.
The same contract said the overwrite was always harmless,
and twice it is not. It illustrates the reserved-prefix rule
with emoji_sentiment() then emoji_position(),
where both write .emoji_n and “both mean the same thing”.
Two shared dotted names do not. .emoji_n_scored counts the
emoji that verb’s lexicon could score, and the lexicons cover
different emoji: U+203C U+FE0F scores 0 under
emoji_sentiment(), emoji_score(),
emoji_risk() and emoji_incongruity() and
1 under emoji_emotion(), so the natural chain
of the first and the last leaves a count describing one lexicon beside a
sentiment from the other. .emoji_sentiment averages every
emoji in the row from emoji_sentiment() but only the
trailing run from emoji_incongruity(where = "final"). Both
are now named in the contract, with the remedy, and a test pins the
divergence in both directions as well as the fact that
.emoji_n really is the same measurement in all eight verbs
that write it.
emoji_search(NA_character_) failed with R’s
own “missing value where TRUE/FALSE needed” instead of the
verb’s own message. nzchar() defaults to
keepNA = FALSE, which reads NA_character_ as
the two-character string "NA", so the non-empty guard
passed and the NA propagated to an internal
if (!any(hit)). The guard now tests is.na()
first, as the other string-argument checks in the package already
did.
?text_to_emoji gave the right round-trip
figure with the wrong reason. It said the emojize/demojize
round trip returns “identical bytes for the 79% that were already fully
qualified”. The 79% is correct (4002 of 5042 entries), but those are not
the fully-qualified entries, and stating it that way invites the number
being “corrected” to the fully-qualified share (3790/5042 = 75%). Which
spelling comes back is decided by the shortcode table, not by the
input’s selectors, so the two sets differ in both directions: 212
unqualified entries keep their bytes – the bare heart
U+2764 among them, because that is the spelling
:heart: maps to – and 92 already-qualified ones do not,
such as the man detective U+1F575 U+FE0F U+200D U+2642,
which comes back with a second selector on the gender sign. The details
now say which question is which, add the guarantee that matters – the
1040 differ by U+FE0F alone, never by more – and point at
?emoji_sanitize, which already tabulated both denominators
correctly. Every figure and both worked examples are pinned to the
data.
?emoji_search listed 11 of the 17 strings
that resolve differently through as_emoji() than
through text_to_emoji(), punctuated as if that were the
whole set. It now names all 17 and says so; camel,
satellite and train were missing from every
list in the package. The set is derived from the data in the tests, so a
new emoji release cannot leave it stale.
?emoji_search promised that its
shortcode column “recovers every row exactly” through
text_to_emoji(), and for some rows there is nothing to
recover it from. 189 of the catalogue’s 5042 rows carry no
GitHub-style alias, so emoji_search() reports
shortcode = NA for them – 7 of the 198 rows
emoji_search("face") returns, for one. Neither the details
nor the @return said the column could be NA;
the @return had gone out of its way to say that
keyword is the empty string rather than NA,
which made the omission read as a guarantee. Both now say so, and point
at the emoji column or as_emoji_shortcode() –
which does answer for all 189, borrowing the alias of the glyph’s other
spelling – as the way through.
as_emoji_shortcode() was documented as
returning the glyph’s “first shortcode”, and for 175 of the catalogue’s
5042 rows it does not. It returns one shortcode per
emoji, resolved through the codepoint key – the first alias of
the fully-qualified (RGI) spelling – which is what makes it agree with
emoji_to_text() and survive a round trip. 344 keys carry a
different first alias on each of their two spellings, so
as_emoji_shortcode("\u2764") is "heart" where
the bare U+2764 row’s own first alias is
"red_heart". emoji_search() reports the other
one, the matched row’s own alias, since a search result is a row – so
the two verbs can disagree about the same glyph. Both resolve back to
the same emoji through text_to_emoji(), so nothing was
broken, but neither help page said which it gave. Both now do, and the
344/175 split is derived from the data in the tests. No behaviour
changed.
?emoji_score and ?emoji_emotion now
state the .emoji_n_scored convention that
?emoji_sentiment and ?emoji_risk already
spelled out: 0 means the row had emoji the lexicon could
not score, NA that it had no emoji to score. All five verbs
behaved this way; two of them did not say so.
?emoji_ratio now covers empty text, which is neither
of the two cases it described: .emoji_ratio is
NA (no characters to take a share of) while
.emoji_only is FALSE (an empty string is not a
row of emoji). Whitespace-only text, by contrast, has a real ratio of
0.
?emoji_pairs now says what sort = FALSE
does. It sorts by item1 then item2 in the C
locale – a fixed order, not the order the pairs happened to be counted
in – which the parameter name alone did not suggest.
Grouped data frames no longer break six verbs, and no
longer lose their grouping in the rest. Two independent bugs,
found by calling every verb on a grouped input:
emoji_to_text(), text_to_emoji(),
emoji_sanitize(), emoji_context() and
emoji_collocations() resolved their text column with
dplyr::select(), which silently re-adds the grouping
columns, so a grouped data frame made the selection return two names and
the call failed with `text` must select exactly one column
– an error blaming an argument the user had got right. And the verbs
that work a row at a time returned tibble::as_tibble(data),
which strips the grouped_df class, so
group_by(author) |> emoji_sentiment(text) |> summarise(...)
quietly collapsed to one corpus-wide row instead of one row per author.
Those verbs now carry the grouping through, as
dplyr::mutate() and dplyr::filter() do; the
three that rewrite the text column in place re-derive the group indices
if the user grouped by that column. See ?tidyEmoji for the
contract.
Seven cross-row aggregators were silently pooling grouped
input. emoji_version_profile(),
emoji_trend(), emoji_turnover(),
emoji_seasonality(), emoji_adoption_lag(),
emoji_collocations() and
emoji_incongruity_profile() had no grouped-input guard, so
a user grouping by author, platform or date got a corpus-wide answer
that looked like a per-group one. All seven now warn.
emoji_cooccurrence() and
emoji_flag_ambiguous() warned under the name of the verb
they delegate to (emoji_pairs(),
emoji_frequency()) and now warn under their own. The guard
lives in one helper rather than being copy-pasted into each verb – that
copy-paste is why the seven were missed – and the helper reports the
warning against the caller’s frame, so it no longer appends “Please
report the issue” to a warning about the user’s own data.
A missing, ambiguous or misspelled column argument now
names the argument the user actually wrote. Every verb resolved
its column with dplyr::pull(), whose errors are phrased in
terms of var – a formal of pull() that appears
in no tidyEmoji signature. emoji_sentiment(df) said
`var` is absent but must be supplied,
emoji_trend(df, text) said the same about the missing
time, a two-column selection said
`!!enquo(var)` must select exactly one column, and a typo
was reported as object 'txet' not found, as if the user’s
own code had a free variable in it. All 38 call sites now go through one
resolver: the messages name text, time,
text_score or doc_id, and a misspelling is
reported as Column \txet` doesn’t exist`.
emoji_extract_nest() accepted a missing text
column. It resolved {{ text }} in the data mask
rather than as a column selection, so
emoji_extract_nest(df) returned a bogus empty list-column
instead of erroring, and a misspelled column was not caught either. It
now uses the same resolver as every other verb, and still returns
data with its class and grouping intact.
The package’s own help page was stale relative to
DESCRIPTION: the paragraph describing interpretation risk,
context, time, incongruity and the sanitiser was missing from
?tidyEmoji. The build no longer ships the
.claude directory, which R CMD check --as-cran
flagged as a hidden file included in error.
Zero-row input now really does return a typed
zero-row tibble. Five verbs built their output columns with
ifelse(), which takes the result’s type from its arguments,
so emoji_density(), emoji_ratio(),
emoji_faceness() and emoji_risk() returned
logical columns where a populated call returns
double or integer, and
emoji_tokens() returned an unspecified .emoji
instead of character. Splitting a corpus, mapping a verb
and binding the pieces back therefore produced a different schema
depending on whether any chunk was empty. The values are
unchanged.
A fractional count errors instead of being silently
truncated. top_n_emojis(n = 2.5) returned two
rows, emoji_context(window = 2.7) used a window of two,
emoji_ngrams(n = 2.9) built bigrams: in each case the
number the user wrote was not the number that was used – the same
failure mode as the head(n = -1) this release already
caught, in the other direction. n, top_n,
window and min_n now require a whole number;
NULL and Inf keep their “all of them” meanings
where they had one. emoji_ngrams(sep = ) is checked too: it
reached paste(collapse = ), which failed with
invalid 'collapse' argument and an internal call stack for
NA or a number, and silently used only the first element of
a longer vector.
emoji_extract_nest() now returns a tibble, like
every other verb. ?tidyEmoji promises “every verb … returns
a tibble”, but this was the one row verb that did not route its output
through the shared helper, so a plain data.frame in gave a
plain data.frame back while the other seventeen returned a
tibble – which also meant a list-column printing badly instead of as
<list>. A grouped input still stays grouped. The
whole output contract is now asserted across every verb at once rather
than verb by verb.
Test line coverage went from 96.6% to 99.4%, and closing the gap
found four documented features that no test exercised:
emoji_score() with the "emotag1200" lexicon
(the mean over its eight emotion dimensions),
emoji_sentiment(lexicon = ) given a data frame or a
registered lexicon, emoji_trend(by = "quarter"), and
sort = FALSE on the relational verbs. Also now covered: the
lexicon aliases "sentiment" and
"emoji_sentiment_lexicon", a factor time
column, the degenerate branches of the rank and z-score rescalings,
emoji_incongruity_profile()’s zero-row return, and every
argument-validation error on the lexicon surface – each of which had
been checked by hand in an earlier round and never written down. The
nine lines still uncovered are guards whose callers validate
first.
commonmark and xml2 join
Suggests. The test that checks NEWS.md parses
calls utils::news(), and R’s Markdown news reader calls
both of them unguarded – packages that were present in the development
library only because roxygen2 and testthat pull them in. Declaring them
means CI installs them and the test runs, rather than skipping
everywhere but the maintainer’s machine. NEWS.md’s version
headings are now also checked directly, without parsing
Markdown at all, so that invariant holds even where the reader is
unavailable.
DESCRIPTION declares Language: en-GB.
The field was missing, so spelling::spell_check_package()
defaulted to en-US and flagged 55 correct British spellings
– licence, normalised, analysed,
summarise, behaviour,
neighbouring – as errors, which made the check unusable as
a gate. The prose was already consistently British: across ten
British/American word pairs, the R sources, help pages, vignette,
README, NEWS and cran-comments.md contain
zero American spellings. The only American tokens are
the exported names emoji_categorize() and
emoji_sanitize(), which follow R convention and
stay.
A new inst/WORDLIST records the 84 remaining terms
the dictionary cannot know – author surnames, package names, and
vocabulary like codepoint, grapheme,
shortcode, keycaps, ZWJ,
Plutchik, idf. With the language declared and
the wordlist in place, spell_check_package() now reports
zero, so a future typo is visible instead of buried in 139 lines of
false positives.
The test suite now passes in a non-UTF-8 locale. Ten test files
carried 114 literal non-ASCII characters inside string literals –
zero-width joiners, gender signs, hearts, the no-break and ideographic
spaces – and R parses a source literal byte-wise under
LC_ALL=C, so each became a run of replacement characters
and every fixture built from one silently tested the wrong string. All
114 are now \u / \U escapes, and the suite
gives identical results under LC_ALL=C and a UTF-8 locale.
Two new tests keep it that way: one asserts R/ is pure
ASCII (the invariant 0.4.0 introduced so the PDF manual builds, checked
by hand until now), the other that test string literals stay escaped.
The package’s own detection was never affected – verified by re-running
the whole-catalogue sweep under LC_ALL=C, which gives
exactly the same 4830 of 5042 as a UTF-8 locale.
The test suite no longer attaches dplyr. One test
file called library(dplyr), which in a single
testthat session leaks into every alphabetically-later file
and masks filter(), lag(),
intersect(), setdiff(),
setequal(), union() and
testthat::matches() for all of them – so nine set-operation
call sites were resolving to dplyr’s generics purely because of filename
order. They worked, but by accident. The attach is gone, those calls are
base::-qualified, and the suite’s behaviour no longer
depends on the order its files happen to sort in.
as_emoji() and text_to_emoji() accept
every GitHub alias, not just the primary one each emoji
is listed under, and that is now tested. The reference table keeps only
an emoji’s first alias as its shortcode, so 751 of the 4698
aliases – "grinning_face", "satisfied",
"face_with_tears_of_joy" – resolve solely through
as_emoji()’s fallback to emoji::emoji_name.
Nothing exercised that path.
emoji_incongruity(threshold =) is documented as the
gap “at or above which” .emoji_incongruent is
TRUE, and the boundary is now tested. A row sitting exactly
on the threshold is classified as incongruent; nothing had pinned that,
so the comparison could have drifted to a strict one and silently
reclassified every borderline row.
emoji_search()’s three search fields are now tested
separately. It is documented to match against keywords, name
and shortcodes, and some queries match on one field only –
"grinning_face" and "thumbsup" appear in no
name (names use spaces, not underscores) and in no keyword – so a
regression that dropped the shortcode field would have gone unnoticed.
Also pinned: the search is case-insensitive, a query matching nothing
returns a typed zero-row tibble, and the +1 shortcode’s
regex metacharacter is safe.
emoji_dfm()’s column order is now guaranteed
independent of the order the rows arrive in. Columns are sorted by
descending total count with the glyph as tiebreak; without that
tiebreak, tied columns fell back to the order the glyphs happened to
appear in the data, so the same corpus sorted differently produced a
differently-ordered feature matrix. The behaviour was already correct –
this release pins it, along with row-order independence for
emoji_frequency(), top_n_emojis(),
emoji_version_profile(), emoji_pairs(),
emoji_cooccurrence() and
emoji_collocations().
?emoji_emotion_label now documents that ties are
broken in Plutchik order, so the winning emotion is deterministic and
does not depend on a row’s position in the data – it was a code comment
only. The help page also points at emoji_emotion() for the
profile the label collapses, since a near-tie is invisible in a single
winning name.
The introduction vignette’s “design choices” list now mentions
how grouping composes, which became a design choice in this release and
was documented only on ?tidyEmoji.
?emoji_incongruity now says what “ends the text”
means for where = "final". Only whitespace may follow the
last glyph, so "great \U0001f602" has a final run and
"great \U0001f602." does not – a trailing full stop,
bracket or quote mark disqualifies it, which is easy to meet unaware in
a punctuated corpus. The behaviour is unchanged; it was simply not
stated.
A POSIXct time column was bucketed by its
UTC day, not its own. as.Date() on a date-time
converts in UTC whatever the object’s tzone says, so an
emoji posted at 23:30 New York time was counted on the next
calendar day – and for an evening-heavy corpus, systematically so. It
also made the package disagree with itself:
emoji_seasonality(period = "hour") reads
format(x, "%H") and had always used the timestamp’s own
zone, so the same row could be hour 23 and the following day at once.
emoji_trend(), emoji_turnover(),
emoji_adoption_lag() and
emoji_seasonality(period = "month" / "weekday") now all
take the calendar day the timestamp displays as. The session’s
TZ does not affect any of them, before or after.
A time column of strings silently shrank the
corpus. A value that would not parse as a date became
NA, and the time verbs then dropped the row –
indistinguishable, in the result, from a row whose date was genuinely
missing. emoji_trend(), emoji_turnover(),
emoji_seasonality() and emoji_adoption_lag()
now say how many values were unreadable and show the first one; a real
NA still passes without comment, and a column where nothing
parses still errors.
?emoji_sanitize now documents which policies
can be undone. The five policies were presented as five
parallel options; they are a ladder of information loss, and how far
down it you stepped only became visible when you tried to restore the
emoji after a model call. The help page gains the table:
"shortcode" is the only rewriting policy that round-trips
(losslessly, including skin tones, flags, ZWJ sequences, keycaps and
U+FE0F forms), "placeholder" keeps
where but not which, and "strip" keeps
neither.
232 of the catalogue’s 2501 zero-width-joiner sequences
were detected as their component emoji rather than as one
emoji. The repair that rejoins sequences the upstream regex
does not know required the gap between two matches to be exactly one
joiner. In a sequence whose middle component is a text-presentation code
point, that component is not matched either, so the gap is
ZWJ + component + ZWJ and the rule declined. The damage was
a wrong count, not a missing one – 🚶
U+200D U+2640 U+200D
U+27A1 U+FE0F, “woman walking facing right”,
arrived as two emoji, “person walking” and “right arrow”, neither of
which the text contains, and most of the 232 are 2023-2024 additions.
The repair now also merges when the gap holds a joiner and the union of
the two spans is itself a catalogued emoji: exact, so it can only join
code points that really do spell one emoji, and the original rule still
covers everything the catalogue has not heard of. 2499 of the 2501 now
detect as one glyph; the two that do not are unqualified spellings whose
every component needs U+FE0F, and their qualified forms are
found. This changes .emoji_n, frequency tables and every
count derived from them for corpora containing these sequences.
A second, subtler form of the same defect needed a second rule. Both
merge rules need two matches to work with, and a sequence whose
only detectable component is one of its parts yields a single match – so
there was no pair to merge and the sequence arrived as that part.
U+2764 U+200D U+1F525 (“heart on fire”) with its selectors
omitted read as U+1F525 (“fire”), and
U+1F9D4 U+200D U+2642 (“man: beard”) as “person with
beard”. Counting glyphs cannot see this – one match is still one glyph –
so the test for it looks for a joiner left outside every
detected span. A lone match beside such a joiner is now grown outwards
while the span stays a catalogued emoji, bounded by the longest
catalogued emoji (10 code points), never crossing a neighbouring match,
and skipped entirely unless the string actually has an orphaned
joiner.
Together the three rules take exact detection of the reference table from 63.2% to 95.8% (3189 of 5042 spellings read as exactly one emoji equal to the whole spelling, against 4830 now) and orphaned joiners from 1643 to 2 – the two being spellings with no detectable component at all, both of which have a canonical form that is found. Staged, so each figure is attributable: rule 1 alone, which is UAX #29’s GB11, gives 75.5% and leaves 1025 orphaned joiners; rule 2 leaves 793; rule 3 takes those to 2. Well-formed text costs about 6% more; text that was broken now costs more and is right.
emoji_dfm() names one column per emoji glyph, so a
doc_id column named with one of those glyphs was
overwritten by the count column and the document identifiers vanished
without a word. It is now an error that says which column to
rename.
emoji_categorize() silently dropped rows
that do contain emoji. It filtered on
.emoji_category being non-NA, and that column
is NA for two different reasons: the row has no emoji, or
the row’s emoji are not in the reference table. The second case is real
and grows with every Unicode release – detection is grapheme-aware, so a
zero-width-joiner sequence newer than your installed emoji
is found as one emoji but cannot be categorised – and those rows
vanished from the result. 0.2.1 fixed one instance of this (a
U+FE0F-qualified heart went missing) by repairing that
particular join; the conflation behind it survived. The filter is now on
“contains at least one emoji”, so such a row is kept with
.emoji_category = NA, and
nrow(emoji_categorize()) now always equals
nrow(emoji_filter()).
emoji_risk() treated the same row two
ways. A row holding emoji the ambiguity lexicon cannot score –
anything added to Unicode after 2015 – correctly got
.emoji_n_scored = 0, but
.emoji_n_ambiguous = NA, where the count of ambiguous
glyphs found is genuinely zero. The @return had promised
NA only for rows with no emoji at all, so the documentation
was right and the code was wrong. .emoji_ambiguity_mean and
.emoji_ambiguity_max stay NA there, because
there is nothing to average.
emoji_key(), the codepoint key every
glyph-to-metadata join goes through, had two “no key here” values:
NA for empty or missing input, but "" for a
string made of nothing but variation selectors. Every consumer had to
remember to filter both. It is now NA in all three cases; a
lexicon row whose glyph is a stray U+FE0F is ignored rather
than keyed on "".
?emoji_sanitize’s reversibility claim is now
measured rather than asserted, and the measurement is stronger than the
old wording: for all 3790 emoji in their canonical spelling – the
spelling a keyboard emits – the "shortcode" round trip
returns the original text byte for byte, 100% of the
time. Feed it one of Unicode’s shorter spellings, with the
U+FE0F selectors omitted, and it returns the canonical one
instead; across all 4853 catalogued spellings that is 79.5%
byte-identical, with every difference being U+FE0F alone
and never more. Tests assert the exact claim rather than a byte-identity
threshold, because that rate falls each time detection improves – a
spelling that used to fragment now merges and normalises.
The lexicon coverage ceilings are now on the help pages,
with the right denominator. emoji_emotion() can
score 150 glyphs – about 4% of the 3790 distinct emoji
tidyEmoji can detect – and emoji_sentiment() about
19%, with nothing added to Unicode after 2015. A user
who does not know that reads a column of NA as “no
emotional content” rather than “not in the lexicon”. Both verbs and both
dataset pages now state the figure, name the {emoji}
version it is computed against, and point at
.emoji_n_scored as the per-row answer. (The internal
roadmap had quoted 3.0% and 19.2% against the reference table’s 5042
rows; the table stores the qualified and unqualified forms of
an emoji separately, so the denominator is 3790 distinct codepoint
keys.)
?emoji_emotion and ?emoji_emotion_label
named fewer columns than they return: the eight emotion columns and the
.emoji_n / .emoji_n_scored counts were
missing, and the completely different shape of long = TRUE
(one row per row per emotion) was not in the
@return at all.
README.md is now byte-for-byte what
README.Rmd renders to, so it cannot drift from its source
unnoticed.
The lexicon registry accepted two registrations it could
never honour.
register_emoji_lexicon("novak2015", ...) succeeded, showed
up in emoji_lexicons() as a second row with the same
name, and was then unreachable, because a bundled name
resolves before the registry is consulted – so
lexicon = "novak2015" still got the bundled table. All six
bundled names are now refused with the list of them. And a lexicon with
no usable score column registered happily and only failed at first use,
from inside emoji_score(), in a message naming
tbl – an argument of a call that had long since returned;
the score column is resolved at registration. An NA name is
rejected too.
The detection contract is now on the help pages.
Code points that are emoji only when they carry U+FE0F –
the bare heart U+2764 being the one people meet – are not
detected, which was documented on ?emoji_sentiment_lexicon
but nowhere a user counting emoji would look. ?tidyEmoji
gains a Detection section with the measured size of the
exclusion (1252 of 5042 catalogue emoji carry U+FE0F; 216
of those are undetectable without it), the reason the default does not
change (U+00A9, U+00AE and U+2122
are in the same set), and the fact that it affects detection only and
never the U+FE0F-stripped join. The .emoji_*
prefix is documented as reserved: a verb overwrites a column of its own
output name without warning, which is what makes the verbs
chainable.
emoji_dfm(doc_id = ) no longer orders its
rows by the session’s collation. Documents were grouped with
factor(), whose levels are sorted with the locale’s
collation, so the row order of the result could differ between machines
– the same class of bug 0.3.0 fixed for glyph ordering in
emoji_pairs() and the dfm’s columns. Documents now appear
in the order their id is first seen in the data.
emoji_pairs(), emoji_cooccurrence() and
emoji_dfm() share one grouping helper, so they cannot drift
apart again.
emoji_sanitize(policy = "strip") tidies only the
whitespace left behind by a removed glyph, and only on rows that
actually contained one.
emoji_sanitize(policy = "strip") could leave
an emoji behind. Deleting a span makes the characters on either
side of it adjacent, and on malformed input those two spell an emoji the
original text never contained: a bare U+2603 beside an
orphan U+FE0F becomes the qualified snowman, #
beside a stray U+FE0F U+20E3 becomes a keycap, and two
regional indicators separated by an emoji become a flag. Found by
stripping 1500 randomly assembled strings and re-detecting: 1 came back
still carrying an emoji, from the one policy whose whole promise is that
they are gone. strip now repeats until there is nothing
left to remove, which terminates because every pass shortens the row.
The rescan is gated on the row still holding a non-ASCII code point, so
ordinary text pays nothing and a corpus of accented words pays about
10%. "placeholder" was not affected, because a non-empty
token keeps the neighbours apart, and "name" and
"shortcode" are documented to leave an unnameable glyph in
place, which ?emoji_sanitize now says in its own words
rather than by reference. The page is careful about which policies clear
the column and on what terms: "strip" does so whatever it
is handed, "placeholder" does unless the token you supply
is itself emoji-forming (one that is an emoji survives, and a
lone U+FE0F binds to whatever the removed glyph stood next
to), and "name" and "shortcode" do not when
the glyph has no name. All four cases are pinned.
Arguments given nonsense now error instead of quietly
returning a different answer. An audit of every argument that
reaches a base R function without validation found one shape of bug
repeated across the package: a value the function cannot honour was
absorbed rather than rejected. emoji_to_text(wrap = ) now
requires the {x} placeholder, since a template without it
replaces every emoji with the same literal string;
top_n_emojis(n = ) rejects a negative n, which
head() had silently read as “drop the last row”; and every
TRUE/FALSE argument
(emoji_pairs(directed = , sort = ),
emoji_cooccurrence(diagonal = ),
emoji_emotion(long = ),
emoji_context(keep_text = ),
top_n_emojis(duplicated = ),
emoji_sentiment(se = )) is now checked, because
isTRUE() reads every non-TRUE value as
FALSE – so long = "yes" used to return the
wide form without complaint. The deprecated
top_n_emojis(duplicated_unicode = "yes") still works: the
check runs after the lifecycle conversion.
emoji_score() gains the
lexicon = "novak2015" default that
emoji_sentiment() always had, so calling it without a
lexicon works instead of raising a missing-argument error.
The reference-manual sources are ASCII throughout, so the PDF manual builds everywhere.
next_release.md, the repo’s release ledger, gains a
section 13 recording what wave 1 shipped, the third-audit defect, and
the design decisions locked at implementation.
Three test assertions that were weaker than the behaviour they
described are now exact. Two counted the residual of the ZWJ
segmentation work with <= 2 – the catalogued spellings
that go undetected, and those that leave a joiner orphaned – where the
true number is exactly two, and one bounded a row count that is exactly
two. An inequality in that position accepts a regression back up to the
bound without reporting it, and a count cannot see the residual
set change while its size stays the same. The two residual
spellings (U+1F441 U+200D U+1F5E8 and
U+1F3F3 U+200D U+26A7) are now named, with the reason they
are irreparable asserted alongside them: neither carries
U+FE0F, and each one’s fully-qualified sibling is detected
as a single glyph, so the residual is confined to the spelling and never
reaches the emoji. No package code changed.
emoji_search() and emoji_collocations()
no longer change answer with the session’s locale. Both fold case before
matching, and tolower() honours LC_CTYPE:
under a Turkish or Azerbaijani locale it maps "I" to a
dotless i (U+0131) rather than "i", so any
query containing a capital I stopped matching the catalogue’s ASCII
text. Measured under tr_TR.utf8,
emoji_search("FIRE") returned 0 rows instead of 27,
"SMILING" 0 instead of 23, "INDIA" 0 instead
of 2, and "I" 37 instead of 4640;
emoji_collocations() counted "BIG" and
"big" as two different words. Matching now folds
A-Z with chartr() before
tolower(), which settles ASCII deterministically while
still folding non-ASCII, so a query for "VICUÑA" or
"O’CLOCK" keeps working. This is not a behaviour change in
an ASCII or Western locale: the new fold agrees with
tolower() on all 5042 catalogue names, 10701 keywords and
5761 aliases. emoji_type()’s group-to-type recode was
checked and was never affected – its subgroups are already lower-case
and none of the literals it matches contains an i.
Case-insensitive search missed accented names outside a
UTF-8 locale. The fold above settled A-Z with
chartr() and left the rest to tolower(), on
the reasoning that tolower() “still folds non-ASCII”. It
does – but only where the locale is UTF-8. Under LC_ALL=C
tolower() returns every non-ASCII letter untouched, and 24
catalogue names carry an uppercase accented letter: the Åland Islands,
Curaçao, Côte d’Ivoire, Réunion, São Tomé & Príncipe,
St. Barthélemy, Türkiye and the seventeen Japanese button glyphs, whose
names are quoted with typographic quotation marks. So
emoji_search("åland") found the flag in a UTF-8 session and
0 rows in a C one, and no error said why. The fold is
now driven from an explicit table throughout: A-Z plus the
200 cased Latin letters of the Latin-1 Supplement and Latin
Extended-A/-B, folded with chartr(), which is a plain
code-point substitution and so is subject to neither the Turkish rule
nor the locale’s character set. tolower() still runs
afterwards for anything the table does not name. Only 1:1 pairs are
listed – an uppercase code point whose lowercase form is a single,
different code point – which is what keeps the result byte-identical to
tolower() in a UTF-8 locale rather than a new folding of
its own; it was re-verified against tolower() over all 5042
names, 10701 keywords and 5761 aliases, and the C locale merely catches
up. The invariant test that claimed the fold does not depend on
LC_CTYPE only ever exercised the Turkish half, so it passed
throughout; it is now joined by one that pins the accented answers by
hand, because tolower() cannot produce them in the locale
where it matters.
A factor or numeric glyph column in a lexicon failed with
R’s message. emoji_key() never coerced, so a
by = column that was a factor reached nzchar()
– whose message is only “requires a character vector” – and a numeric
one reached utf8ToInt(). Neither named the argument, the
column or the verb, and the second is the likelier mistake: pointing
by = at the wrong column. as_emoji_name() and
emoji_ambiguity() already coerced; the lexicon path was the
one that forgot.
A code point that is not valid UTF-8 produced the literal
key "NA". utf8ToInt() answers
NA_integer_, and NA != 0xFE0F is
NA, so the variation-selector filter kept it and
sprintf("%X", NA) made the string "NA" – a
real key that every undecodable value collided on, and that consumers
read as present rather than missing. It now returns
NA_character_ like every other unkeyable input.
A data-frame lexicon carrying a column named
type was silently ignored.
emoji_sentiment() tested lex$type before
is.data.frame(lex), and the lookup returns a user’s data
frame unchanged – so a lexicon with a type column reading
"sentiment" was discarded and the bundled Novak scores used
in its place, with no message. $ partial-matches, so
types, type_of and type_label did
it too. emoji_score() and emoji_emotion()
already branched in the safe order.
emoji_emotion() accepted a lexicon that
emoji_score() refuses. The duplicate-key check
lived inside the sentiment path only, so a custom emotion lexicon
listing both U+2764 and U+2764 U+FE0F with
different scores gave an answer that changed when the caller reordered
their own table. One table now gets one answer whichever verb reads
it.
emoji_emotion_label() returned
NA when any one emotion was NA.
max.col() answers NA_integer_ for a row
holding a missing value, so a row that scored on seven emotions and was
NA on the eighth came back unlabelled even though its
maximum was unambiguous.
A long ZWJ chain could lose its tail.
.emoji_extend_zwj() wrote
for (e in (b + 1L):min(...)) without establishing that the
sequence counts upwards. Rule 1 merges arbitrarily long chains, so a
span longer than ten code points is reachable, and there the loop ran
backwards over spans inside the current match and shrank it. A
twelve-code-point kiss sequence joined to a grinning face came back as
ten, with the trailing glyph counted nowhere.
.emoji_category was ordered by first
appearance. So "Flags|Smileys & Emotion" and
"Smileys & Emotion|Flags" were two values for one
combination, and count(.emoji_category) split every
multi-category row across as many groups as the corpus had orderings –
which is what the introduction vignette does, one chart of which was
wrong for this reason. Sorted into the crosswalk’s fixed order now, as
emoji_type() already was.
emoji_sanitize(policy = "keep") skipped
every content check. It resolved the column’s name, so
a typo was caught, but never read it: the "bytes"-encoding
refusal, the list-column refusal and the one-value-per-row check all
belonged to the other policies. A guard one policy skips is a guard the
caller cannot rely on.
emoji_sentiment(se = TRUE) shrank the error
when a glyph repeated. The standard error divided by the
occurrence count, so one emoji pasted twice reported the same
uncertainty as two independent glyphs and
strrep(emoji, 100) claimed a tenfold reduction – for a row
carrying exactly as many annotations as the single-glyph row. Repeating
a glyph adds no annotations. It is now weighted over distinct glyphs,
which leaves the answer for genuinely distinct emoji unchanged to the
last digit. The test that should have caught it asserted the property
over a fixture of one glyph twice, the single case where the property is
false.
emoji_seasonality() reported a structural
zero as a measured zero. A cycle level the data never reaches
had emoji_per_text = NA and share = 0 in the
same row – two columns disagreeing about whether there was data.
share is NA there now, n_texts
remains the discriminator. Every level is returned so a bar chart has no
invisible gaps, and a zero-height bar for “no data” was exactly the gap
that was meant to be visible.
emoji_score() returned its three columns in
the reverse order of every other scorer. It is the generic
emoji_sentiment(), emoji_emotion(),
emoji_risk(), emoji_incongruity() and
emoji_faceness() sit on, and the only one putting the
measurement before the counts, so selecting by position gave different
columns from the generic and the specific verb. Now
.emoji_n, .emoji_n_scored,
.emoji_score, like the rest.
emoji_token_cost() truncated a fractional
tokeniser result. A tokeniser answering 2.6 was recorded as 2.
It rounds up, matching the built-in heuristic.
A doc_id list-column was deparsed instead of
read. .emoji_col() had no atomic guard, so
emoji_dfm() and emoji_pairs() grouped
documents by their deparsed R source and two different ids that deparse
alike merged into one. This is the same hazard the text column was given
a guard for in this release; the other three column arguments now share
it.
A future emoji release could have silently
undone the version fix. .emoji_fill_by_key()
parsed the upstream version with a bare
as.numeric(), while every consumer strips a leading
E first – E17.0 being the spelling Unicode’s
own data files use. Had a release carried it, every entry would have
been NA, the fill would have done nothing, and the 1252
rows without a version this release exists to repair would have come
back, with nothing failing.
?emoji_ambiguity recommended
ci_width as the remedy for thin evidence, and it is not
one. It is a Wald interval, so it is exactly zero wherever the
spread is zero: 166 of the lexicon’s 969 rows report
ci_width = 0, on annotation counts running from 1 to 68.
Ranked ascending it puts the thinnest unanimous glyphs first, as the
most certain rows in the table. The page now says so, and still points
at n_annotations.
emoji_score(lexicon = "emotag1200") returns
an intensity, not a valence. EmoTag1200’s eight dimensions are
each on [0, 1] and four are negatively valenced, so their
mean is non-negative and puts a maximally angry emoji near a maximally
joyful one – while "novak2015" is a signed valence on
[-1, 1]. Both arrive as .emoji_score. The
@return now names both scales and says they must not be
pooled.
?emoji_trend claimed a completeness it does
not deliver. The grid is complete over the observed
periods, and “observed” means periods holding at least one emoji: a
period whose rows carry no emoji, and any calendar gap, is absent. The
three time verbs answer this differently on purpose –
emoji_turnover() keeps emoji-free periods,
emoji_seasonality() returns every level unconditionally –
and that was documented nowhere, so joining two of them on
.period quietly lost rows. All three policies are now
stated together on ?emoji_trend.
The Detection section only ever said what detection
misses. It quantifies that to four significant figures and said
nothing about what it falsely includes. Two things: an invalid
regional-indicator pair such as U+1F1FD U+1F1FD is one
well-formed grapheme cluster and is counted as an emoji with
name = NA, and an orphan skin-tone modifier is counted as
an emoji and named, because the Component group is in the
reference table. Both are defensible as raw detection and misleading as
a corpus statistic, so both are now named, with
as_emoji_type() != "component" given as the
filter.
The missing lifecycle badge.
?top_n_emojis expands
lifecycle::badge("deprecated") into a reference to
man/figures/lifecycle-deprecated.svg, which was not in the
package: the HTML help page and the pkgdown reference rendered a broken
image. The PDF manual takes the \ifelse else-branch, which
is why the PDF-focused checks never caught it.
The suite would have gone red on every CRAN flavour the
day emoji 17.0.0 ships. Roughly seventy catalogue
figures are pinned deliberately, because the pinning is what catches
documentation drifting from data – but emoji’s version
tracks the Unicode emoji version, DESCRIPTION sets no upper
bound, and Unicode Emoji 17.0 is already out. Those assertions are now
gated on the installed release matching the documented one, so the
mismatch is reported once, by one test, where the maintainer can act on
it. Nothing skips while emoji is 16.0.0.
The caches derived from the emoji reference table are now dropped
whenever that table is rebuilt. ref_keys and the type
recode are computed from emoji_reference(), each
guarded only on is.null(), so nothing connected the three:
a rebuilt reference left the other two pointing at the previous
catalogue. ref_keys is the one that would have mattered,
because the zero-width-joiner repair tests membership in it, so a stale
copy changes detection rather than only metadata. One function now names
the derived slots, so a cache added later is invalidated by
construction.
The bundled sentiment lexicon’s two structural invariants are
asserted, in data-raw/ and again over the shipped data so a
check runs them: occurrences equals the three class counts
summed, and no two rows canonicalise to one code-point key.
emoji_sentiment() reads occurrences while
emoji_ambiguity() recomputes the sum, and both resolve a
duplicate key first-wins and silently, so either violation would have
made two verbs disagree about the same glyph without raising anything.
Both hold today; the emotion script already asserted its
analogues.
data-raw/ no longer keeps its own copy of the
code-point key function. Two verbatim copies had drifted from the real
one – neither had the guard that keys a selector-only string to
NA rather than "" – so the bundled
key columns were generated by a function that no longer
matched the one used to look them up. The crosswalks still rebuild
byte-for-byte.
Tests that register a lexicon now restore the registry when they finish. Eleven registrations were still live when the suite ended, which made results depend on file order, and one test coped by wiping the registry outright – discarding every registration made before it and so depending on line order too. A source guard now requires every registering test to restore, which is order-independent in a way that inspecting the registry at the end cannot be.
The two bundled lexicons’ licences were named in the
documentation and their notices never shipped.
inst/LICENSE.note now carries the MIT text and copyright
line for EmoTag1200, the CC BY-SA 4.0 URI for the Emoji Sentiment
Ranking, and the Creative Commons declaration that makes CC BY-SA 4.0
one-way compatible with the GNU GPL version 3.
CI runs weekly as well as on push, and the matrix gained a job on
R 4.1.0, the declared minimum, which no job previously exercised.
cran-comments.md reports its check result per environment
rather than as one headline its own next paragraph
contradicted.
“Nothing but whitespace” meant different things in
different sessions. The same defect as the fold above, in a
second place. emoji_ratio()’s .emoji_only is
documented as “the text contains emoji and nothing else but whitespace”,
and the test for it, the token split behind
.emoji_per_token, emoji_context()’s word split
and emoji_incongruity(where = "final")’s “only whitespace
may follow the last glyph” all asked that question with \s
or [[:space:]]. Both resolve through the C library’s
iswspace(), which depends on the locale and on the
platform. So an emoji followed by an ideographic space
(U+3000) was emoji-only in a UTF-8 session and not
emoji-only under LC_ALL=C – and CRAN checks on flavours
whose iswspace() need not agree with glibc’s. All six sites
now use one explicit character class: Unicode’s White_Space
property, written out. That also settles an inconsistency glibc had left
behind, since U+00A0 and U+202F – the two
no-break spaces – were the only White_Space characters it
rejected, so U+3000 counted and U+00A0 did
not; both count now. U+200B remains excluded, because
despite the name Unicode does not give the zero-width space
White_Space. The bundled corpus has one row containing
U+00A0 and no value in it changes, so no figure quoted
anywhere moves. The test asserting that emoji_context() and
emoji_density() “tokenise identically” had reimplemented
the density side inline, so it compared the package against a stale copy
of itself and failed on the corrected code; it now reads the count back
out of the verb.
emoji_version_profile() and
emoji_adoption_lag() now resolve a Unicode version for
every emoji they detect. The upstream emoji::emojis table
records the introducing version on the unqualified member of a
variation pair and leaves it NA on the fully-qualified one,
so 1252 of 5042 reference rows carried no version – among them everyday
glyphs like the red heart (U+2764 U+FE0F), the smiling face
(U+263A U+FE0F) and the skull and crossbones
(U+2620 U+FE0F). Those all fell into
emoji_version_profile()’s version = NA bucket,
which is precisely what that verb exists to report, and
emoji_adoption_lag() returned NA for them. A
version describes the emoji, not one spelling of it, so the reference
table now fills version across every glyph sharing a
codepoint key – the same normalisation the rest of the package already
applied to names, shortcodes, types and lexicon joins. No key carries
two different versions, so this only ever fills gaps; every one of the
1252 is recovered and the unknown-version row disappears for the full
catalogue. Profile counts shift accordingly (Emoji 0.6 goes from 656 to
719 types), and lag_days is now computed where it
previously could not be. A glyph with a genuinely unknown version is
still reported rather than dropped.
The ordering verbs now document how they settle ties, which
several of them enforced in code without promising.
emoji_frequency() already said “descending n
with ties broken by the glyph so the order is deterministic”;
top_n_emojis(), emoji_dfm(),
emoji_ambiguity() and emoji_flag_ambiguous()
applied the same discipline but said nothing.
top_n_emojis()’s @return now records the sort
order, that the glyph order decides which side of the cut a straddling
tie falls on, and that a corpus with fewer than n distinct
emoji returns all of them rather than padding.
emoji_dfm()’s records that emoji columns run in descending
corpus total with ties broken by the glyph, computed in the C locale so
the column order is safe to index by position and does not follow the
session’s collation. Both ambiguity verbs now state that
rank uses ties.method = "min": tied glyphs
share the lowest rank of their group and the next distinct value skips
ahead, so ranks are deliberately not consecutive. No behaviour
changed.
?emoji_sentiment_lexicon stated its coverage
inconsistently. It read “969 rows, covering about 19% of the distinct
emoji tidyEmoji can detect (3790 distinct codepoint keys)”, but 969/3790
is 25.6%. The 19% is the right figure and the attribution was wrong:
only 736 of the 969 rows resolve to a detectable emoji, and 736/3790 is
19.4% – which is also exactly how many of the 3790 detectable emoji
emoji_sentiment() scores. The paragraph now says so, and
its arithmetic is checkable against the sentence that follows it (736
resolving + 233 absent = 969). The ?emoji_emotion_lexicon
figure needed no change: all 150 of its rows resolve, so 150/3790 = 4.0%
was already exact.
emoji_search()’s documentation now says how to get
back from its shortcode column to a glyph. The column is
the emoji’s first alias, and passing it to text_to_emoji()
recovers every row exactly, while as_emoji() resolves a
bare string by Unicode name first and so returns a different emoji for
the eleven strings that name one emoji and alias another
(dog, cat, cow, pig,
tiger, mouse, rabbit,
horse, whale, kiss,
sunglasses). That is the documented precedence rather than
a defect, but emoji_search() advertised its result as ready
for “piping into other verbs” without saying which path is safe. Its
@return also now records that keyword is the
empty string, not NA, when the query matched the name or a
shortcode rather than a keyword.
emoji_lexicons() no longer returns columns carrying
stray element names after register_emoji_lexicon(). The
registry is a named list, so the
lapply()/vapply() that built the
dimensions and n columns returned named
results and dplyr::bind_rows() then padded the bundled rows
with "". The tibble printed normally, which is why this
went unnoticed, but extracting a column showed it:
emoji_lexicons()$n printed a stray name header above the
values. Both columns are now unnamed, and a test asserts that no verb
returns a column with element names – checked across the 42 exported
functions that return a data frame, with a custom lexicon
registered.
The declared R dependency is raised from
R (>= 3.5.0) to R (>= 4.1.0), which is
the oldest R the package can actually be installed on. The old value was
a promise it could not keep: the current dplyr and
tidyr both require R >= 4.1.0, and
install.packages() serves only current versions, so a user
on R 3.5 or 4.0 got an opaque dependency-resolution failure instead of a
clear message about their R version. Nothing in the package’s own code
needed more than R 3.5 – the floor comes entirely from the dependency
chain – and no behaviour changed. A test now checks the declared minimum
against the installed hard dependencies’ own floors, since the CI matrix
only reaches oldrel-1 and so cannot detect this class of drift.
The introduction vignette described 68% of emoji-bearing entries as “the overwhelming majority” (and “the vast majority” in a figure’s alt text) where the measured share carrying exactly one emoji is 68.2%. Both now say “about two-thirds”.
text_to_emoji() no longer claims flatly to be the
inverse of emoji_to_text(). It is the inverse up to the
presentation selector: because both directions resolve through the
U+FE0F-stripped code-point key, an unqualified glyph and
its fully-qualified form share one shortcode and the fully-qualified
form is what comes back. Feeding the whole catalogue through and back
preserves the key for all 5042 entries and the exact bytes for 79% of
them; a second pass changes nothing. The details now say so, and say to
compare with emoji_key() rather than string equality. They
no longer attribute that 79% to the entries “that were already fully
qualified”: the two sets differ – 212 unqualified entries keep their
bytes and 92 qualified ones do not – because which spelling comes back
is decided by the shortcode table, not by the input’s
selectors.
as_emoji()’s resolution order is now documented. It
accepts names and shortcodes in one argument, and 464 strings belong to
both namespaces – the exact Unicode name of one emoji and a shortcode
alias of another. It has always tried the exact Unicode name first, so
as_emoji("dog") is the emoji named “dog”
(U+1F415) rather than the one aliased :dog: (a
dog face, U+1F436). For 17 of the 464 the namespaces
disagree and as_emoji() differs from
text_to_emoji() by design, since a :dog: token
is explicitly a shortcode; both places now say which is which and how to
ask for one namespace specifically. No behaviour changed.
Emoji-dense rows no longer cost quadratic time. Five hot paths
reached a character offset with
substr()/substring(), which rescans a
multi-byte string from its first byte on every call, so the work grew
with the square of the emoji in a row: glyph slicing (used by nearly
every verb), the splice in the translation verbs, the residual test in
emoji_ratio(), the trailing-run walk-back behind
emoji_incongruity(), and the per-occurrence window loop in
emoji_context(), which handed the whole prefix and suffix
to a windowing function that only ever needed the few tokens next to the
glyph. The four that cut text around glyph spans now share one helper
that converts the string once and slices code points past a threshold,
keeping ordinary rows on the existing path; emoji_context()
reads a bounded slice anchored at the glyph. Measured on a row holding
6400 emoji: emoji_ratio() about thirteen times faster,
emoji_summary() and emoji_sentiment() about
ten, the trailing-run walk-back about seven,
emoji_context() about four, and every stage of the engine
now scales linearly. Results are unchanged – the fast paths are pinned
to the slow ones by tests that compare them directly, and fall back
whenever utf8ToInt() cannot represent the string.
README.md’s emoji_ambiguity() line
gained the annotation-count caveat that ?emoji_ambiguity
and the vignette already carried.
Test coverage for this release, in one place rather than
scattered through the notes above. The suite now also pins: every one of
the 49 exports called as the first thing in a fresh session
(several populate a shared cache lazily, which a suite running them in
order cannot expose); the functional type recode as a clean partition of
all 32 Unicode subgroups in Smileys & Emotion and
People & Body; the pluggable-lexicon API through every
consumer that documents it, registered and inline, sentiment- and
emotion-shaped, full and partial; non-syntactic column names
("my text", "2024", "if",
"x-y") and every tidyselect form that resolves to one
column; 182 combinations of the enum and flag arguments; fourteen verb
pipelines and the grouped contract end to end; the two internal
optimisations that claim exact equivalence, against naive
implementations; and README.md against
README.Rmd’s prose, so it cannot drift unnoticed. Three
tests that scanned R/*.R now walk the namespace instead, so
they run inside R CMD check rather than skipping: skips
there are down from 14 to 5, 262 more assertions than at the start of
the pass, and the five that remain cannot be reached from an installed
package.
emoji_emotion() scores each row’s emoji across the
eight Plutchik emotions (anger, anticipation, disgust, fear, joy,
sadness, surprise, trust), using the new bundled
emoji_emotion_lexicon (EmoTag1200, Shoeb & de Melo
2020, MIT). Supports a long form (long = TRUE) with one row
per (row, emotion).emoji_emotion_label() adds the dominant emotion per
row.emoji_lexicons() lists bundled
and registered lexicons, register_emoji_lexicon() adds your
own, and emoji_score() is the generic scorer all the verbs
share. emoji_sentiment() gains a lexicon
argument (default "novak2015", unchanged behaviour).emoji_pairs() returns a tidy,
graph-ready edge list (item1, item2,
n) of the emoji that co-occur in the same document – each
row is a document, or supply doc_id to pool rows – with
directed = TRUE to order pairs by first appearance.
emoji_cooccurrence() is the same with an optional
diagonal (each emoji’s document frequency).
emoji_ngrams() slides a window over each row’s emoji in
reading order and returns one row per consecutive n-gram.emoji_position() reports where
emoji sit in each text (first/last character position and mean relative
position in [0, 1]), emoji_density() reports
emoji per character and per token, and emoji_ratio()
reports the share of the text’s characters that are emoji plus an
.emoji_only flag.emoji_dfm() builds a document-by-emoji feature table
(weightings: counts, binary, tf-idf), keeping every document – including
emoji-free ones – so the result binds row-for-row to outcome columns in
modelling workflows.emoji_pairs(),
emoji_cooccurrence(), emoji_ngrams(),
emoji_dfm()) canonicalise glyphs through the package’s
codepoint key, so qualified and unqualified forms of the same emoji (for
example the victory hand with and without U+FE0F) count as
one node/feature. emoji_frequency() intentionally still
reports the exact extracted glyph.emoji_to_text() replaces emoji in a text column with
their Unicode names or shortcodes (demojize – useful for accessibility
and NLP preprocessing), and text_to_emoji() is the inverse
(emojize).as_emoji_name(),
as_emoji_shortcode() and as_emoji() for ad-hoc
conversion.emoji_search() finds emoji by keyword, name or
shortcode and returns a tidy tibble of matches.emoji_emotion_lexicon.man/figures/logo.svg is the vector master,
logo.png the raster copy).emoji_ratio() recognising an emoji-only row; all of that is
fixed. Emoji separated by anything other than a ZWJ are unaffected.text_to_emoji() no longer misses a
:shortcode: that follows an unrelated colon.
"meet at 10:30 :grinning:" and
"https://example.org :grinning:" previously came back
unchanged, because the permissive :...: pattern consumed
the shortcode’s opening colon; shortcode tokens are now matched on the
character set GitHub-style aliases actually use.emoji_pairs() and emoji_dfm() order glyphs
in the C locale, so which glyph lands in item1 and the
order of a dfm’s tied columns no longer depend on the session’s
collation. Results are now reproducible across machines.emoji_lexicons() no longer reports the glyph column as
a score dimension for a lexicon registered with a by other
than "emoji".emoji_emotion(long = TRUE) no longer drops a user
column named .row_number.emoji_ngrams(n = Inf) gives the documented error
instead of a coercion warning followed by “missing value where
TRUE/FALSE needed”.emoji_to_text() is several times faster: it locates
emoji once for the whole column rather than once per row.emoji_density() returns
.emoji_per_token = 0 (not NA) for
whitespace-only text, matching .emoji_per_char and the
documented “no emoji -> 0” contract (#1).as_emoji() accepts the spaced Unicode names produced by
as_emoji_name() (routing through the reference table), so
as_emoji(as_emoji_name(x)) round-trips instead of returning
NA (#2).?emoji_sentiment_lexicon now explains which lexicon
entries are stored as unqualified, text-presentation code points (the
bare heart U+2764 without U+FE0F, the white
smiling face, the heavy check mark, …) and are therefore not detected in
text, and notes that the qualified form resolves to the same entry
(#3).?register_emoji_lexicon no longer
points at an internal development file.emoji_search() matches literally, so queries containing
regex metacharacters (for example the +1 alias) are safe
and cannot error.emoji_to_text(format = "shortcode") now always emits
the emoji’s canonical (first) GitHub-style alias – the same one reported
by emoji_frequency() and as_emoji_shortcode()
– and the wrap template is honoured. Emoji with no known
name/shortcode are left in place rather than dropped from the text.emoji_to_text() and text_to_emoji() keep
NA text entries as NA.emoji_emotion() and emoji_emotion_label()
accept registered or data-frame emotion lexicons (any subset of the
eight Plutchik dimensions), not just the bundled
"emotag1200".register_emoji_lexicon(by = ) works with any glyph column
name in emoji_sentiment() and
emoji_emotion().emoji_frequency() (and therefore
top_n_emojis()) breaks count ties by the glyph, making the
output order deterministic.emoji_lexicons() no longer lists a custom lexicon’s
glyph/key columns among its score dimensions.?tidyEmoji) documents the output
and naming contract shared by all verbs.doc_id argument); it was already a hard transitive
dependency, so the installed footprint is unchanged.emoji_pairs(),
emoji_cooccurrence() or emoji_dfm() warn that
grouping is ignored – use doc_id to express per-group
structure.U+FE0F variation selector no longer get NA
metadata, are no longer dropped by emoji_categorize(), and
no longer disappear from
top_n_emojis(duplicated = TRUE).emoji_summary() and emoji_filter() use the
same detection as the extraction verbs.emoji_sentiment() gains .emoji_n_scored
(emoji actually found in the lexicon), distinct from
.emoji_n.top_n_emojis(n =) counts distinct emoji rather than
rows, breaks ties deterministically, keeps emoji that have no
GitHub-style alias, and preserves the exact extracted glyph in
duplicated mode (one row per distinct alias;
left_join instead of inner_join).emoji_extract_unnest() now uses
.row_number (dotted) to avoid collision with user columns
and dplyr::row_number.emoji_summary() column names renamed from
emoji_tweets/total_tweets to
n_with_emoji/n_total. The old names are no
longer available in this release.emoji_tweets() is soft-deprecated in favour of
emoji_filter().emoji_sentiment()
and emoji_categorize().emoji_summary(),
emoji_frequency() and top_n_emojis() now warn
that grouping is ignored (per-group results land in 1.0).ata_tweets.rda (a CSV
misnamed .rda) to ata_tweets.csv and
downsampled from 10k to 2k rows. Vignette language updated to be less
Twitter-specific.key column for
normalised joins.tidyEmoji is now positioned as a general toolkit for emoji in any text column (social-media posts, reviews, chat logs, survey responses, …), not just tweets.
emoji_sentiment() scores the emoji in each row using
the bundled emoji_sentiment_lexicon (the Emoji Sentiment
Ranking of Kralj Novak et al., 2015), returning a mean sentiment in
[-1, 1].emoji_frequency() returns the count of every
emoji in a text column, with name, shortcode and category.
top_n_emojis() is now a thin wrapper over it.emoji_tokens() expands data to one row per emoji
occurrence with its name, category and sentiment score – a tidy,
“one-token-per-row” shape.emoji_filter() is a clearer, text-agnostic name for
emoji_tweets() (which is kept as a synonym).emoji_sentiment_lexicon.top_n_emojis() in particular is
dramatically faster on large inputs.top_n_emojis() no longer emits a many-to-many join
warning, and reports the emoji’s canonical shortcode
(e.g. mask) by default.emoji_tweets() previously returned a plain data frame),
and emoji_extract_unnest() no longer prints a grouping
message.data-raw/).tweet_tbl -> data
and tweet_text -> text. Code that passed
these positionally (e.g. df %>% emoji_summary(text_col))
is unaffected; update any calls that named the old arguments.top_n_emojis(duplicated_unicode = "yes"/"no") is
deprecated in favour of the logical
duplicated = TRUE/FALSE. The old argument still works with
a warning.