Package 'stringi'

Description

Binary operators for joining (concatenating) two character vectors, with a typical R look-and-feel.

Usage

e1 %s+% e2

e1 %stri+% e2

Arguments

Details

Vectorized over e1 and e2.

These operators act like a call to stri_join(e1, e2, sep=''). However, note that joining 3 vectors, e.g., e1 %s+% e2 %s+% e3 is slower than stri_join(e1, e2, e3, sep=''), because it creates a new (temporary) result vector each time the operator is applied.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other join: stri_dup(), stri_flatten(), stri_join_list(), stri_join()

Examples

c('abc', '123', 'xy') %s+% letters[1:6]
'ID_' %s+% 1:5

Description

Relational operators for comparing corresponding strings in two character vectors, with a typical R look-and-feel.

Usage

e1 %s<% e2

e1 %s<=% e2

e1 %s>% e2

e1 %s>=% e2

e1 %s==% e2

e1 %s!=% e2

e1 %s===% e2

e1 %s!==% e2

e1 %stri<% e2

e1 %stri<=% e2

e1 %stri>% e2

e1 %stri>=% e2

e1 %stri==% e2

e1 %stri!=% e2

e1 %stri===% e2

e1 %stri!==% e2

Arguments

Details

These functions call stri_cmp_le or its friends, using the default collator options. As a consequence, they are vectorized over e1 and e2.

%stri==% tests for canonical equivalence of strings (see stri_cmp_equiv) and is a locale-dependent operation.

%stri===% performs a locale-independent, code point-based comparison.

Value

All the functions return a logical vector indicating the result of a pairwise comparison. As usual, the elements of shorter vectors are recycled if necessary.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_sensitive: about_locale, about_search_boundaries, about_search_coll, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Examples

'a' %stri<% 'b'
c('a', 'b', 'c') %stri>=% 'b'

C-Style Formatting with stri_sprintf as a Binary Operator

Description

Provides access to stri_sprintf in form of a binary operator in a way similar to Python's % overloaded for strings.

Missing values and empty vectors are propagated as usual.

Usage

e1 %s$% e2

e1 %stri$% e2

Arguments

Details

Vectorized over e1 and e2.

e1 %s$% atomic_vector is equivalent to e1 %s$% list(atomic_vector).

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other length: stri_isempty(), stri_length(), stri_numbytes(), stri_pad_both(), stri_sprintf(), stri_width()

Examples

"value='%d'" %s$% 3
"value='%d'" %s$% 1:3
"%s='%d'" %s$% list("value", 3)
"%s='%d'" %s$% list("value", 1:3)
"%s='%d'" %s$% list(c("a", "b", "c"), 1)
"%s='%d'" %s$% list(c("a", "b", "c"), 1:3)

x <- c("abcd", "\u00DF\u00B5\U0001F970", "abcdef")
cat("[%6s]" %s$% x, sep="\n")  # width used, not the number of bytes

Description

Below we explain how stringi deals with its functions' arguments.

If some function violates one of the following rules (for a very important reason), this is clearly indicated in its documentation (with discussion).

Coercion of Arguments

When a character vector argument is expected, factors and other vectors coercible to characters vectors are silently converted with as.character, otherwise an error is generated. Coercion from a list which does not consist of length-1 atomic vectors issues a warning.

When a logical, numeric, or integer vector argument is expected, factors are converted with as.*(as.character(...)), and other coercible vectors are converted with as.*, otherwise an error is generated.

Vectorization

Almost all functions are vectorized with respect to all their arguments and the recycling rule is applied whenever necessary. Due to this property you may, for instance, search for one pattern in each given string, search for each pattern in one given string, and search for the i-th pattern within the i-th string.

We of course took great care of performance issues: e.g., in regular expression searching, regex matchers are reused from iteration to iteration, as long as it is possible.

Functions with some non-vectorized arguments are rare: e.g., regular expression matcher's settings are established once per each call.

Some functions assume that a vector with one element is given as an argument (like collapse in stri_join). In such cases, if an empty vector is given you will get an error and for vectors with more than 1 elements - a warning will be generated (only the first element will be used).

You may find details on vectorization behavior in the man pages on each particular function of your interest.

Handling Missing Values (NAs)

stringi handles missing values consistently. For any vectorized operation, if at least one vector element is missing, then the corresponding resulting value is also set to NA.

Preserving Object Attributes

Generally, all our functions drop input objects' attributes (e.g., names, dim, etc.). This is due to deep vectorization as well as for efficiency reasons. If the preservation of attributes is needed, important attributes can be manually copied. Alternatively, the notation x[] <- stri_...(x, ...) can sometimes be used too.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other stringi_general_topics: about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_search, about_stringi

Description

This manual page explains how stringi deals with character strings in various encodings.

In particular we should note that:

• R lets strings in ASCII, UTF-8, and your platform's native encoding coexist. A character vector printed on the console by calling print or cat is silently re-encoded to the native encoding.

• Functions in stringi process each string internally in Unicode, the most universal character encoding ever. Even if a string is given in the native encoding, i.e., your platform's default one, it will be converted to Unicode (precisely: UTF-8 or UTF-16).

• Most stringi functions always return UTF-8 encoded strings, regardless of the input encoding. What is more, the functions have been optimized for UTF-8/ASCII input (they have competitive, if not better performance, especially when performing more complex operations like string comparison, sorting, and even concatenation). Thus, it is best to rely on cascading calls to stringi operations solely.

Details

Quoting the ICU User Guide, 'Hundreds of encodings have been developed over the years, each for small groups of languages and for special purposes. As a result, the interpretation of text, input, sorting, display, and storage depends on the knowledge of all the different types of character sets and their encodings. Programs have been written to handle either one single encoding at a time and switch between them, or to convert between external and internal encodings.'

'Unicode provides a single character set that covers the major languages of the world, and a small number of machine-friendly encoding forms and schemes to fit the needs of existing applications and protocols. It is designed for best interoperability with both ASCII and ISO-8859-1 (the most widely used character sets) to make it easier for Unicode to be used in almost all applications and protocols' (see the ICU User Guide).

The Unicode Standard determines the way to map any possible character to a numeric value – a so-called code point. Such code points, however, have to be stored somehow in computer's memory. The Unicode Standard encodes characters in the range U+0000..U+10FFFF, which amounts to a 21-bit code space. Depending on the encoding form (UTF-8, UTF-16, or UTF-32), each character will then be represented either as a sequence of one to four 8-bit bytes, one or two 16-bit code units, or a single 32-bit integer (compare the ICU FAQ).

Unicode can be thought of as a superset of the spectrum of characters supported by any given code page.

UTF-8 and UTF-16

For portability reasons, the UTF-8 encoding is the most natural choice for representing Unicode character strings in R. UTF-8 has ASCII as its subset (code points 1–127 represent the same characters in both of them). Code points larger than 127 are represented by multi-byte sequences (from 2 to 4 bytes: Please note that not all sequences of bytes are valid UTF-8, compare stri_enc_isutf8).

Most of the computations in stringi are performed internally using either UTF-8 or UTF-16 encodings (this depends on type of service you request: some ICU services are designed only to work with UTF-16). Due to such a choice, with stringi you get the same result on each platform, which is – unfortunately – not the case of base R's functions (for instance, it is known that performing a regular expression search under Linux on some texts may give you a different result to those obtained under Windows). We really had portability in our minds while developing our package!

We have observed that R correctly handles UTF-8 strings regardless of your platform's native encoding (see below). Therefore, we decided that most functions in stringi will output its results in UTF-8 – this speeds ups computations on cascading calls to our functions: the strings does not have to be re-encoded each time.

Note that some Unicode characters may have an ambiguous representation. For example, “a with ogonek” (one character) and “a”+“ogonek” (two graphemes) are semantically the same. stringi provides functions to normalize character sequences, see stri_trans_nfc for discussion. However, it is observed that denormalized strings do appear very rarely in typical string processing activities.

Additionally, do note that stringi silently removes byte order marks (BOMs - they may incidentally appear in a string read from a text file) from UTF8-encoded strings, see stri_enc_toutf8.

Character Encodings in R

Data in memory are just bytes (small integer values) – an encoding is a way to represent characters with such numbers, it is a semantic 'key' to understand a given byte sequence. For example, in ISO-8859-2 (Central European), the value 177 represents Polish “a with ogonek”, and in ISO-8859-1 (Western European), the same value denotes the “plus-minus” sign. Thus, a character encoding is a translation scheme: we need to communicate with R somehow, relying on how it represents strings.

Overall, R has a very simple encoding marking mechanism, see stri_enc_mark. There is an implicit assumption that your platform's default (native) encoding always extends ASCII – stringi checks that whenever your native encoding is being detected automatically on ICU's initialization and each time when you change it manually by calling stri_enc_set.

Character strings in R (internally) can be declared to be in:

• UTF-8;

• latin1, i.e., either ISO-8859-1 (Western European on Linux, OS X, and other Unixes) or WINDOWS-1252 (Windows);

• bytes – for strings that should be manipulated as sequences of bytes.

Moreover, there are two other cases:

• ASCII – for strings consisting only of byte codes not greater than 127;

• native (a.k.a. unknown in Encoding; quite a misleading name: no explicit encoding mark) – for strings that are assumed to be in your platform's native (default) encoding. This can represent UTF-8 if you are an OS X user, or some 8-bit Windows code page, for example. The native encoding used by R may be determined by examining the LC_CTYPE category, see Sys.getlocale.

Intuitively, “native” strings result from reading a string from stdin (e.g., keyboard input). This makes sense: your operating system works in some encoding and provides R with some data.

Each time when a stringi function encounters a string declared in native encoding, it assumes that the input data should be translated from the default encoding, i.e., the one returned by stri_enc_get (unless you know what you are doing, the default encoding should only be changed if the automatic encoding detection process fails on stringi load).

Functions which allow 'bytes' encoding markings are very rare in stringi, and were carefully selected. These are: stri_enc_toutf8 (with argument is_unknown_8bit=TRUE), stri_enc_toascii, and stri_encode.

Finally, note that R lets strings in ASCII, UTF-8, and your platform's native encoding coexist. A character vector printed with print, cat, etc., is silently re-encoded so that it can be properly shown, e.g., on the console.

Encoding Conversion

Apart from automatic conversion from the native encoding, you may re-encode a string manually, for example when you read it from a file created on a different platform. Call stri_enc_list for the list of encodings supported by ICU. Note that converter names are case-insensitive and ICU tries to normalize the encoding specifiers. Leading zeroes are ignored in sequences of digits (if further digits follow), and all non-alphanumeric characters are ignored. Thus the strings 'UTF-8', 'utf_8', 'u*Tf08' and 'Utf 8' are equivalent.

The stri_encode function allows you to convert between any given encodings (in some cases you will obtain bytes-marked strings, or even lists of raw vectors (i.e., for UTF-16). There are also some useful more specialized functions, like stri_enc_toutf32 (converts a character vector to a list of integers, where one code point is exactly one numeric value) or stri_enc_toascii (substitutes all non-ASCII bytes with the SUBSTITUTE CHARACTER, which plays a similar role as R's NA value).

There are also some routines for automated encoding detection, see, e.g., stri_enc_detect.

Encoding Detection

Given a text file, one has to know how to interpret (encode) raw data in order to obtain meaningful information.

Encoding detection is always an imprecise operation and needs a considerable amount of data. However, in case of some encodings (like UTF-8, ASCII, or UTF-32) a “false positive” byte sequence is quite rare (statistically speaking).

Check out stri_enc_detect (among others) for a useful function in this category.

Author(s)

Marek Gagolewski and other contributors

References

Unicode Basics – ICU User Guide, https://unicode-org.github.io/icu/userguide/icu/unicode.html

Conversion – ICU User Guide, https://unicode-org.github.io/icu/userguide/conversion/

Converters – ICU User Guide, https://unicode-org.github.io/icu/userguide/conversion/converters.html (technical details)

UTF-8, UTF-16, UTF-32 & BOM – ICU FAQ, https://www.unicode.org/faq/utf_bom.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other stringi_general_topics: about_arguments, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_search, about_stringi

Other encoding_management: stri_enc_info(), stri_enc_list(), stri_enc_mark(), stri_enc_set()

Other encoding_detection: stri_enc_detect2(), stri_enc_detect(), stri_enc_isascii(), stri_enc_isutf16be(), stri_enc_isutf8()

Other encoding_conversion: stri_enc_fromutf32(), stri_enc_toascii(), stri_enc_tonative(), stri_enc_toutf32(), stri_enc_toutf8(), stri_encode()

Description

In this section we explain how we specify locales in stringi. Locale is a fundamental concept in ICU. It identifies a specific user community, i.e., a group of users who have similar culture and language expectations for human-computer interaction.

Details

Because a locale is just an identifier of a region, no validity check is performed when you specify a Locale. ICU is implemented as a set of services. If you want to verify whether particular resources are available in the locale you asked for, you must query those resources. Note: when you ask for a resource for a particular locale, you get back the best available match, not necessarily precisely the one you requested.

Locale Identifiers

ICU services are parametrized by locale, to deliver culturally correct results. Locales are identified by character strings of the form Language code, Language_Country code, or Language_Country_Variant code, e.g., 'en_US'.

The two-letter Language code uses the ISO-639-1 standard, e.g., 'en' stands for English, 'pl' – Polish, 'fr' – French, and 'de' for German.

Country is a two-letter code following the ISO-3166 standard. This is to reflect different language conventions within the same language, for example in US-English ('en_US') and Australian-English ('en_AU').

Differences may also appear in language conventions used within the same country. For example, the Euro currency may be used in several European countries while the individual country's currency is still in circulation. In such a case, ICU Variant '_EURO' could be used for selecting locales that support the Euro currency.

The final (optional) element of a locale is a list of keywords together with their values. Keywords must be unique. Their order is not significant. Unknown keywords are ignored. The handling of keywords depends on the specific services that utilize them. Currently, the following keywords are recognized: calendar, collation, currency, and numbers, e.g., fr@collation=phonebook;calendar=islamic-civil is a valid French locale specifier together with keyword arguments. For more information, refer to the ICU user guide.

For a list of locales that are recognized by ICU, call stri_locale_list.

Note that in stringi, 'C' is a synonym of 'en_US_POSIX'.

A Note on Default Locales

Each locale-sensitive function in stringi selects the current default locale if an empty string or NULL is provided as its locale argument. Default locales are available to all the functions; initially, the system locale on that platform is used, but it may be changed by calling stri_locale_set.

Your program should avoid changing the default locale. All locale-sensitive functions may request any desired locale per-call (by specifying the locale argument), i.e., without referencing to the default locale. During many tests, however, we did not observe any improper behavior of stringi while using a modified default locale.

Locale-Sensitive Functions in stringi

One of many examples of locale-dependent services is the Collator, which performs a locale-aware string comparison. It is used for string comparing, ordering, sorting, and searching. See stri_opts_collator for the description on how to tune its settings, and its locale argument in particular.

When choosing a resource bundle that is not available in the explicitly requested locale (but not when using the default locale) nor in its more general variants (e.g., 'es_ES' vs 'es'), a warning is emitted.

Other locale-sensitive functions include, e.g., stri_trans_tolower (that does character case mapping).

Author(s)

Marek Gagolewski and other contributors

References

Locale – ICU User Guide, https://unicode-org.github.io/icu/userguide/locale/

ISO 639: Language Codes, https://www.iso.org/iso-639-language-codes.html

ISO 3166: Country Codes, https://www.iso.org/iso-3166-country-codes.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_management: stri_locale_info(), stri_locale_list(), stri_locale_set()

Other locale_sensitive: %s<%(), about_search_boundaries, about_search_coll, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other stringi_general_topics: about_arguments, about_encoding, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_search, about_stringi

Description

This man page explains how to perform string search-based operations in stringi.

Details

The following independent string searching engines are available in stringi.

• stri_*_regex – ICU's regular expressions (regexes), see about_search_regex,

• stri_*_fixed – locale-independent byte-wise pattern matching, see about_search_fixed,

• stri_*_coll – ICU's StringSearch, locale-sensitive, Collator-based pattern search, useful for natural language processing tasks, see about_search_coll,

• stri_*_charclass – character classes search, e.g., Unicode General Categories or Binary Properties, see about_search_charclass,

• stri_*_boundaries – text boundary analysis, see about_search_boundaries

Each search engine is able to perform many search-based operations. These may include:

• stri_detect_* - detect if a pattern occurs in a string, see, e.g., stri_detect,

• stri_count_* - count the number of pattern occurrences, see, e.g., stri_count,

• stri_locate_* - locate all, first, or last occurrences of a pattern, see, e.g., stri_locate,

• stri_extract_* - extract all, first, or last occurrences of a pattern, see, e.g., stri_extract and, in case of regexes, stri_match,

• stri_replace_* - replace all, first, or last occurrences of a pattern, see, e.g., stri_replace and also stri_trim,

• stri_split_* - split a string into chunks indicated by occurrences of a pattern, see, e.g., stri_split,

• stri_startswith_* and stri_endswith_* detect if a string starts or ends with a pattern match, see, e.g., stri_startswith,

• stri_subset_* - return a subset of a character vector with strings that match a given pattern, see, e.g., stri_subset.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other text_boundaries: about_search_boundaries, stri_count_boundaries(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_brkiter(), stri_split_boundaries(), stri_split_lines(), stri_trans_tolower(), stri_wrap()

Other search_regex: about_search_regex, stri_opts_regex()

Other search_fixed: about_search_fixed, stri_opts_fixed()

Other search_coll: about_search_coll, stri_opts_collator()

Other search_charclass: about_search_charclass, stri_trim_both()

Other search_detect: stri_detect(), stri_startswith()

Other search_count: stri_count_boundaries(), stri_count()

Other search_locate: stri_locate_all_boundaries(), stri_locate_all()

Other search_replace: stri_replace_all(), stri_replace_rstr(), stri_trim_both()

Other search_split: stri_split_boundaries(), stri_split_lines(), stri_split()

Other search_subset: stri_subset()

Other search_extract: stri_extract_all_boundaries(), stri_extract_all(), stri_match_all()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_stringi

Description

Text boundary analysis is the process of locating linguistic boundaries while formatting and handling text.

Details

Examples of the boundary analysis process include:

• Locating positions to word-wrap text to fit within specific margins while displaying or printing, see stri_wrap and stri_split_boundaries.

• Counting characters, words, sentences, or paragraphs, see stri_count_boundaries.

• Making a list of the unique words in a document, see stri_extract_all_words and then stri_unique.

• Capitalizing the first letter of each word or sentence, see also stri_trans_totitle.

• Locating a particular unit of the text (for example, finding the third word in the document), see stri_locate_all_boundaries.

Generally, text boundary analysis is a locale-dependent operation. For example, in Japanese and Chinese one does not separate words with spaces - a line break can occur even in the middle of a word. These languages have punctuation and diacritical marks that cannot start or end a line, so this must also be taken into account.

stringi uses ICU's BreakIterator to locate specific text boundaries. Note that the BreakIterator's behavior may be controlled in come cases, see stri_opts_brkiter.

• The character boundary iterator tries to match what a user would think of as a “character” – a basic unit of a writing system for a language – which may be more than just a single Unicode code point.

• The word boundary iterator locates the boundaries of words, for purposes such as “Find whole words” operations.

• The line_break iterator locates positions that would be appropriate to wrap lines when displaying the text.

• The break iterator of type sentence locates sentence boundaries.

For technical details on different classes of text boundaries refer to the ICU User Guide, see below.

Author(s)

Marek Gagolewski and other contributors

References

Boundary Analysis – ICU User Guide, https://unicode-org.github.io/icu/userguide/boundaryanalysis/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_sensitive: %s<%(), about_locale, about_search_coll, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other text_boundaries: about_search, stri_count_boundaries(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_brkiter(), stri_split_boundaries(), stri_split_lines(), stri_trans_tolower(), stri_wrap()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_search, about_stringi

Description

Here we describe how character classes (sets) can be specified in the stringi package. These are useful for defining search patterns (note that the ICU regex engine uses the same scheme for denoting character classes) or, e.g., generating random code points with stri_rand_strings.

Details

All stri_*_charclass functions in stringi perform a single character (i.e., Unicode code point) search-based operations. You may obtain the same results using about_search_regex. However, these very functions aim to be faster.

Character classes are defined using ICU's UnicodeSet patterns. Below we briefly summarize their syntax. For more details refer to the bibliographic References below.

UnicodeSet patterns

A UnicodeSet represents a subset of Unicode code points (recall that stringi converts strings in your native encoding to Unicode automatically). Legal code points are U+0000 to U+10FFFF, inclusive.

Patterns either consist of series of characters bounded by square brackets (such patterns follow a syntax similar to that employed by regular expression character classes) or of Perl-like Unicode property set specifiers.

[] denotes an empty set, [a] – a set consisting of character “a”, [\u0105] – a set with character U+0105, and [abc] – a set with “a”, “b”, and “c”.

[a-z] denotes a set consisting of characters “a” through “z” inclusively, in Unicode code point order.

Some set-theoretic operations are available. ^ denotes the complement, e.g., [^a-z] contains all characters but “a” through “z”. Moreover, [[pat1][pat2]], [[pat1]\&[pat2]], and [[pat1]-[pat2]] denote union, intersection, and asymmetric difference of sets specified by pat1 and pat2, respectively.

Note that all white-spaces are ignored unless they are quoted or back-slashed (white spaces can be freely used for clarity, as [a c d-f m] means the same as [acd-fm]). stringi does not allow including multi-character strings (see UnicodeSet API documentation). Also, empty string patterns are disallowed.

Any character may be preceded by a backslash in order to remove its special meaning.

A malformed pattern always results in an error.

Set expressions at a glance (according to https://unicode-org.github.io/icu/userguide/strings/regexp.html):

Some examples:

[abc]

Match any of the characters a, b or c.

[^abc]

Negation – match any character except a, b or c.

[A-M]

Range – match any character from A to M. The characters to include are determined by Unicode code point ordering.

[\u0000-\U0010ffff]

Range – match all characters.

[\p{Letter}] or [\p{General_Category=Letter}] or [\p{L}]

Characters with Unicode Category = Letter. All forms shown are equivalent.

[\P{Letter}]

Negated property (Note the upper case \P) – match everything except Letters.

[\p{numeric_value=9}]

Match all numbers with a numeric value of 9. Any Unicode Property may be used in set expressions.

[\p{Letter}&\p{script=cyrillic}]

Set intersection – match the set of all Cyrillic letters.

[\p{Letter}-\p{script=latin}]

Set difference – match all non-Latin letters.

[[a-z][A-Z][0-9]] or [a-zA-Z0-9]

Implicit union of sets – match ASCII letters and digits (the two forms are equivalent).

[:script=Greek:]

Alternative POSIX-like syntax for properties – equivalent to \p{script=Greek}.

Unicode properties

Unicode property sets are specified with a POSIX-like syntax, e.g., [:Letter:], or with a (extended) Perl-style syntax, e.g., \p{L}. The complements of the above sets are [:^Letter:] and \P{L}, respectively.

The names are normalized before matching (for example, the match is case-insensitive). Moreover, many names have short aliases.

Among predefined Unicode properties we find, e.g.:

• Unicode General Categories, e.g., Lu for uppercase letters,

• Unicode Binary Properties, e.g., WHITE_SPACE,

and many more (including Unicode scripts).

Each property provides access to the large and comprehensive Unicode Character Database. Generally, the list of properties available in ICU is not well-documented. Please refer to the References section for some links.

Please note that some classes might overlap. However, e.g., General Category Z (some space) and Binary Property WHITE_SPACE matches different character sets.

Unicode General Categories

The Unicode General Category property of a code point provides the most general classification of that code point. Each code point falls into one and only one Category.

Cc

a C0 or C1 control code.

Cf

a format control character.

Cn

a reserved unassigned code point or a non-character.

Co

a private-use character.

Cs

a surrogate code point.

Lc

the union of Lu, Ll, Lt.

Ll

a lowercase letter.

Lm

a modifier letter.

Lo

other letters, including syllables and ideographs.

Lt

a digraphic character, with the first part uppercase.

Lu

an uppercase letter.

Mc

a spacing combining mark (positive advance width).

Me

an enclosing combining mark.

Mn

a non-spacing combining mark (zero advance width).

Nd

a decimal digit.

Nl

a letter-like numeric character.

No

a numeric character of other type.

Pd

a dash or hyphen punctuation mark.

Ps

an opening punctuation mark (of a pair).

Pe

a closing punctuation mark (of a pair).

Pc

a connecting punctuation mark, like a tie.

Po

a punctuation mark of other type.

Pi

an initial quotation mark.

Pf

a final quotation mark.

Sm

a symbol of mathematical use.

Sc

a currency sign.

Sk

a non-letter-like modifier symbol.

So

a symbol of other type.

Zs

a space character (of non-zero width).

Zl

U+2028 LINE SEPARATOR only.

Zp

U+2029 PARAGRAPH SEPARATOR only.

C

the union of Cc, Cf, Cs, Co, Cn.

L

the union of Lu, Ll, Lt, Lm, Lo.

M

the union of Mn, Mc, Me.

N

the union of Nd, Nl, No.

P

the union of Pc, Pd, Ps, Pe, Pi, Pf, Po.

S

the union of Sm, Sc, Sk, So.

Z

the union of Zs, Zl, Zp

Unicode Binary Properties

Each character may follow many Binary Properties at a time.

Here is a comprehensive list of supported Binary Properties:

ALPHABETIC

alphabetic character.

ASCII_HEX_DIGIT

a character matching the [0-9A-Fa-f] charclass.

BIDI_CONTROL

a format control which have specific functions in the Bidi (bidirectional text) Algorithm.

BIDI_MIRRORED

a character that may change display in right-to-left text.

DASH

a kind of a dash character.

DEFAULT_IGNORABLE_CODE_POINT

characters that are ignorable in most text processing activities, e.g., <2060..206F, FFF0..FFFB, E0000..E0FFF>.

DEPRECATED

a deprecated character according to the current Unicode standard (the usage of deprecated characters is strongly discouraged).

DIACRITIC

a character that linguistically modifies the meaning of another character to which it applies.

EXTENDER

a character that extends the value or shape of a preceding alphabetic character, e.g., a length and iteration mark.

HEX_DIGIT

a character commonly used for hexadecimal numbers, see also ASCII_HEX_DIGIT.

HYPHEN

a dash used to mark connections between pieces of words, plus the Katakana middle dot.

ID_CONTINUE

a character that can continue an identifier, ID_START+Mn+Mc+Nd+Pc.

ID_START

a character that can start an identifier, Lu+Ll+Lt+Lm+Lo+Nl.

IDEOGRAPHIC

a CJKV (Chinese-Japanese-Korean-Vietnamese) ideograph.

LOWERCASE

...

MATH

...

NONCHARACTER_CODE_POINT

...

QUOTATION_MARK

...

SOFT_DOTTED

a character with a “soft dot”, like i or j, such that an accent placed on this character causes the dot to disappear.

TERMINAL_PUNCTUATION

a punctuation character that generally marks the end of textual units.

UPPERCASE

...

WHITE_SPACE

a space character or TAB or CR or LF or ZWSP or ZWNBSP.

CASE_SENSITIVE

...

POSIX_ALNUM

...

POSIX_BLANK

...

POSIX_GRAPH

...

POSIX_PRINT

...

POSIX_XDIGIT

...

CASED

...

CASE_IGNORABLE

...

CHANGES_WHEN_LOWERCASED

...

CHANGES_WHEN_UPPERCASED

...

CHANGES_WHEN_TITLECASED

...

CHANGES_WHEN_CASEFOLDED

...

CHANGES_WHEN_CASEMAPPED

...

CHANGES_WHEN_NFKC_CASEFOLDED

...

EMOJI

Since ICU 57

EMOJI_PRESENTATION

Since ICU 57

EMOJI_MODIFIER

Since ICU 57

EMOJI_MODIFIER_BASE

Since ICU 57

POSIX Character Classes

Avoid using POSIX character classes, e.g., [:punct:]. The ICU User Guide (see below) states that in general they are not well-defined, so you may end up with something different than you expect.

In particular, in POSIX-like regex engines, [:punct:] stands for the character class corresponding to the ispunct() classification function (check out man 3 ispunct on UNIX-like systems). According to ISO/IEC 9899:1990 (ISO C90), the ispunct() function tests for any printing character except for space or a character for which isalnum() is true. However, in a POSIX setting, the details of what characters belong into which class depend on the current locale. So the [:punct:] class does not lead to a portable code (again, in POSIX-like regex engines).

Therefore, a POSIX flavor of [:punct:] is more like [\p{P}\p{S}] in ICU. You have been warned.

Author(s)

Marek Gagolewski and other contributors

References

The Unicode Character Database – Unicode Standard Annex #44, https://www.unicode.org/reports/tr44/

UnicodeSet – ICU User Guide, https://unicode-org.github.io/icu/userguide/strings/unicodeset.html

Properties – ICU User Guide, https://unicode-org.github.io/icu/userguide/strings/properties.html

C/POSIX Migration – ICU User Guide, https://unicode-org.github.io/icu/userguide/icu/posix.html

Unicode Script Data, https://www.unicode.org/Public/UNIDATA/Scripts.txt

icu::Unicodeset Class Reference – ICU4C API Documentation, https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/classicu_1_1UnicodeSet.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_charclass: about_search, stri_trim_both()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_coll, about_search_fixed, about_search_regex, about_search, about_stringi

Description

String searching facilities described here provide a way to locate a specific piece of text. Interestingly, locale-sensitive searching, especially on a non-English text, is a much more complex process than it seems at first glance.

Locale-Aware String Search Engine

All stri_*_coll functions in stringi use ICU's StringSearch engine, which implements a locale-sensitive string search algorithm. The matches are defined by using the notion of “canonical equivalence” between strings.

Tuning the Collator's parameters allows you to perform correct matching that properly takes into account accented letters, conjoined letters, ignorable punctuation and letter case.

For more information on ICU's Collator and the search engine and how to tune it up in stringi, refer to stri_opts_collator.

Please note that ICU's StringSearch-based functions are often much slower that those to perform fixed pattern searches.

Author(s)

Marek Gagolewski and other contributors

References

ICU String Search Service – ICU User Guide, https://unicode-org.github.io/icu/userguide/collation/string-search.html

L. Werner, Efficient Text Searching in Java, 1999, https://icu-project.org/docs/papers/efficient_text_searching_in_java.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_coll: about_search, stri_opts_collator()

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_fixed, about_search_regex, about_search, about_stringi

Description

String searching facilities described here provide a way to locate a specific sequence of bytes in a string. The search engine's settings may be tuned up (for example to perform case-insensitive search) via a call to the stri_opts_fixed function.

Byte Compare

The fast Knuth-Morris-Pratt search algorithm, with worst time complexity of O(n+p) (n == length(str), p == length(pattern)) is implemented (with some tweaks for very short search patterns).

Be aware that, for natural language processing, fixed pattern searching might not be what you actually require. It is because a bitwise match will not give correct results in cases of:

1. accented letters;

2. conjoined letters;

3. ignorable punctuation;

4. ignorable case,

see also about_search_coll.

Note that the conversion of input data to Unicode is done as usual.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_fixed: about_search, stri_opts_fixed()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_regex, about_search, about_stringi

Description

A regular expression is a pattern describing, possibly in a very abstract way, a text fragment. With so many regex functions in stringi, regular expressions may be a very powerful tool to perform string searching, substring extraction, string splitting, etc., tasks.

Details

All stri_*_regex functions in stringi use the ICU regex engine. Its settings may be tuned up (for example to perform case-insensitive search) via the stri_opts_regex function.

Regular expression patterns in ICU are quite similar in form and behavior to Perl's regexes. Their implementation is loosely inspired by JDK 1.4 java.util.regex. ICU Regular Expressions conform to the Unicode Technical Standard #18 (see References section) and its features are summarized in the ICU User Guide (see below). A good general introduction to regexes is (Friedl, 2002). Some general topics are also covered in the R manual, see regex.

ICU Regex Operators at a Glance

Here is a list of operators provided by the ICU User Guide on regexes.

|

Alternation. A|B matches either A or B.

*

Match 0 or more times. Match as many times as possible.

+

Match 1 or more times. Match as many times as possible.

?

Match zero or one times. Prefer one.

{n}

Match exactly n times.

{n,}

Match at least n times. Match as many times as possible.

{n,m}

Match between n and m times. Match as many times as possible, but not more than m.

*?

Match 0 or more times. Match as few times as possible.

+?

Match 1 or more times. Match as few times as possible.

??

Match zero or one times. Prefer zero.

{n}?

Match exactly n times.

{n,}?

Match at least n times, but no more than required for an overall pattern match.

{n,m}?

Match between n and m times. Match as few times as possible, but not less than n.

*+

Match 0 or more times. Match as many times as possible when first encountered, do not retry with fewer even if overall match fails (Possessive Match).

++

Match 1 or more times. Possessive match.

?+

Match zero or one times. Possessive match.

{n}+

Match exactly n times.

{n,}+

Match at least n times. Possessive Match.

{n,m}+

Match between n and m times. Possessive Match.

(...)

Capturing parentheses. Range of input that matched the parenthesized sub-expression is available after the match, see stri_match.

(?:...)

Non-capturing parentheses. Groups the included pattern, but does not provide capturing of matching text. Somewhat more efficient than capturing parentheses.

(?>...)

Atomic-match parentheses. The first match of the parenthesized sub-expression is the only one tried; if it does not lead to an overall pattern match, back up the search for a match to a position before the (?>.

(?#...)

Free-format comment (?# comment ).

(?=...)

Look-ahead assertion. True if the parenthesized pattern matches at the current input position, but does not advance the input position.

(?!...)

Negative look-ahead assertion. True if the parenthesized pattern does not match at the current input position. Does not advance the input position.

(?<=...)

Look-behind assertion. True if the parenthesized pattern matches text preceding the current input position, with the last character of the match being the input character just before the current position. Does not alter the input position. The length of possible strings matched by the look-behind pattern must not be unbounded (no * or + operators.)

(?<!...)

Negative Look-behind assertion. True if the parenthesized pattern does not match text preceding the current input position, with the last character of the match being the input character just before the current position. Does not alter the input position. The length of possible strings matched by the look-behind pattern must not be unbounded (no * or + operators.)

(?<name>...)

Named capture group, where name (enclosed within the angle brackets) is a sequence like [A-Za-z][A-Za-z0-9]*

(?ismwx-ismwx:...)

Flag settings. Evaluate the parenthesized expression with the specified flags enabled or -disabled, see also stri_opts_regex.

(?ismwx-ismwx)

Flag settings. Change the flag settings. Changes apply to the portion of the pattern following the setting. For example, (?i) changes to a case insensitive match, see also stri_opts_regex.

ICU Regex Meta-characters at a Glance

Here is a list of meta-characters provided by the ICU User Guide on regexes.

\a

Match a BELL, \u0007.

\A

Match at the beginning of the input. Differs from ^. in that \A will not match after a new line within the input.

\b

Match if the current position is a word boundary. Boundaries occur at the transitions between word (\w) and non-word (\W) characters, with combining marks ignored. For better word boundaries, see ICU Boundary Analysis, e.g., stri_extract_all_words.

\B

Match if the current position is not a word boundary.

\cX

Match a control-X character.

\d

Match any character with the Unicode General Category of Nd (Number, Decimal Digit.).

\D

Match any character that is not a decimal digit.

\e

Match an ESCAPE, \u001B.

\E

Terminates a \Q ... \E quoted sequence.

\f

Match a FORM FEED, \u000C.

\G

Match if the current position is at the end of the previous match.

\h

Match a Horizontal White Space character. They are characters with Unicode General Category of Space_Separator plus the ASCII tab, \u0009. [Since ICU 55]

\H

Match a non-Horizontal White Space character. [Since ICU 55]

\k<name>

Named Capture Back Reference. [Since ICU 55]

\n

Match a LINE FEED, \u000A.

\N{UNICODE CHARACTER NAME}

Match the named character.

\p{UNICODE PROPERTY NAME}

Match any character with the specified Unicode Property.

\P{UNICODE PROPERTY NAME}

Match any character not having the specified Unicode Property.

\Q

Quotes all following characters until \E.

\r

Match a CARRIAGE RETURN, \u000D.

\s

Match a white space character. White space is defined as [\t\n\f\r\p{Z}].

\S

Match a non-white space character.

\t

Match a HORIZONTAL TABULATION, \u0009.

\uhhhh

Match the character with the hex value hhhh.

\Uhhhhhhhh

Match the character with the hex value hhhhhhhh. Exactly eight hex digits must be provided, even though the largest Unicode code point is \U0010ffff.

\w

Match a word character. Word characters are [\p{Alphabetic}\p{Mark}\p{Decimal_Number}\p{Connector_Punctuation}\u200c\u200d].

\W

Match a non-word character.

\x{hhhh}

Match the character with hex value hhhh. From one to six hex digits may be supplied.

\xhh

Match the character with two digit hex value hh

\X

Match a Grapheme Cluster.

\Z

Match if the current position is at the end of input, but before the final line terminator, if one exists.

\z

Match if the current position is at the end of input.

\n

Back Reference. Match whatever the nth capturing group matched. n must be a number > 1 and < total number of capture groups in the pattern.

\0ooo

Match an Octal character. 'ooo' is from one to three octal digits. 0377 is the largest allowed Octal character. The leading zero is required; it distinguishes Octal constants from back references.

[pattern]

Match any one character from the set.

.

Match any character except for - by default - newline, compare stri_opts_regex.

^

Match at the beginning of a line.

$

Match at the end of a line.

\

[outside of sets] Quotes the following character. Characters that must be quoted to be treated as literals are * ? + [ ( ) { } ^ $ | \ ..

\

[inside sets] Quotes the following character. Characters that must be quoted to be treated as literals are [ ] \; Characters that may need to be quoted, depending on the context are - &.

Character Classes

The syntax is similar, but not 100% compatible with the one described in about_search_charclass. In particular, whitespaces are not ignored and set-theoretic operations are denoted slightly differently. However, other than this about_search_charclass is a good reference on the capabilities offered.

The ICU User Guide on regexes lists what follows.

[abc]

Match any of the characters a, b, or c

[^abc]

Negation – match any character except a, b, or c

[A-M]

Range – match any character from A to M (based on Unicode code point ordering)

[\p{L}], [\p{Letter}], [\p{General_Category=Letter}], [:letter:]

Characters with Unicode Category = Letter (4 equivalent forms)

[\P{Letter}]

Negated property – natch everything except Letters

[\p{numeric_value=9}]

Match all numbers with a numeric value of 9

[\p{Letter}&&\p{script=cyrillic}]

Intersection; match the set of all Cyrillic letters

[\p{Letter}--\p{script=latin}]

Set difference; match all non-Latin letters

[[a-z][A-Z][0-9]], [a-zA-Z0-9]

Union; match ASCII letters and digits (2 equivalent forms)

Regex Functions in stringi

Note that if a given regex pattern is empty, then all the functions in stringi give NA in result and generate a warning. On a syntax error, a quite informative failure message is shown.

If you wish to search for a fixed pattern, refer to about_search_coll or about_search_fixed. They allow to perform a locale-aware text lookup, or a very fast exact-byte search, respectively.

Author(s)

Marek Gagolewski and other contributors

References

Regular expressions – ICU User Guide, https://unicode-org.github.io/icu/userguide/strings/regexp.html

J.E.F. Friedl, Mastering Regular Expressions, O'Reilly, 2002

Unicode Regular Expressions – Unicode Technical Standard #18, https://www.unicode.org/reports/tr18/

Unicode Regular Expressions – Regex tutorial, https://www.regular-expressions.info/unicode.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_regex: about_search, stri_opts_regex()

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search, about_stringi

Description

stringi is THE R package for fast, correct, consistent, and convenient string/text manipulation. It gives predictable results on every platform, in each locale, and under any native character encoding.

Keywords: R, text processing, character strings, internationalization, localization, ICU, ICU4C, i18n, l10n, Unicode.

Homepage: https://stringi.gagolewski.com/

License: The BSD-3-clause license for the package code, the ICU license for the accompanying ICU4C distribution, and the UCD license for the Unicode Character Database. See the COPYRIGHTS and LICENSE file for more details.

Details

Manual pages on general topics:

• about_encoding – character encoding issues, including information on encoding management in stringi, as well as on encoding detection and conversion.

• about_locale – locale issues, including locale management and specification in stringi, and the list of locale-sensitive operations. In particular, see stri_opts_collator for a description of the string collation algorithm, which is used for string comparing, ordering, ranking, sorting, case-folding, and searching.

• about_arguments – information on how stringi handles the arguments passed to its function.

Facilities available

Refer to the following:

• about_search for string searching facilities; these include pattern searching, matching, string splitting, and so on. The following independent search engines are provided:

  • about_search_regex – with ICU (Java-like) regular expressions,

  • about_search_fixed – fast, locale-independent, byte-wise pattern matching,

  • about_search_coll – locale-aware pattern matching for natural language processing tasks,

  • about_search_charclass – seeking elements of particular character classes, like “all whites-paces” or “all digits”,

  • about_search_boundaries – text boundary analysis.

• stri_datetime_format for date/time formatting and parsing. Also refer to the links therein for other date/time/time zone- related operations.

• stri_stats_general and stri_stats_latex for gathering some fancy statistics on a character vector's contents.

• stri_join, stri_dup, %s+%, and stri_flatten for concatenation-based operations.

• stri_sub for extracting and replacing substrings, and stri_reverse for a joyful function to reverse all code points in a string.

• stri_length (among others) for determining the number of code points in a string. See also stri_count_boundaries for counting the number of Unicode characters and stri_width for approximating the width of a string.

• stri_trim (among others) for trimming characters from the beginning or/and end of a string, see also about_search_charclass, and stri_pad for padding strings so that they are of the same width. Additionally, stri_wrap wraps text into lines.

• stri_trans_tolower (among others) for case mapping, i.e., conversion to lower, UPPER, or Title Case, stri_trans_nfc (among others) for Unicode normalization, stri_trans_char for translating individual code points, and stri_trans_general for other universal text transforms, including transliteration.

• stri_cmp, %s<%, stri_order, stri_sort, stri_rank, stri_unique, and stri_duplicated for collation-based, locale-aware operations, see also about_locale.

• stri_split_lines (among others) to split a string into text lines.

• stri_escape_unicode (among others) for escaping some code points.

• stri_rand_strings, stri_rand_shuffle, and stri_rand_lipsum for generating (pseudo)random strings.

• stri_read_raw, stri_read_lines, and stri_write_lines for reading and writing text files.

Note that each man page provides many further links to other interesting facilities and topics.

Author(s)

Marek Gagolewski, with contributions from Bartek Tartanus and many others. ICU4C was developed by IBM, Unicode, Inc., and others.

References

stringi Package Homepage, https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

ICU – International Components for Unicode, https://icu.unicode.org/

ICU4C API Documentation, https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/

The Unicode Consortium, https://home.unicode.org/

UTF-8, A Transformation Format of ISO 10646 – RFC 3629, https://www.rfc-editor.org/rfc/rfc3629

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other stringi_general_topics: about_arguments, about_encoding, about_locale, about_search_boundaries, about_search_charclass, about_search_coll, about_search_fixed, about_search_regex, about_search

Description

These functions may be used to determine if two strings are equal, canonically equivalent (this is performed in a much more clever fashion than when testing for equality), or to check whether they are in a specific lexicographic order.

Usage

stri_compare(e1, e2, ..., opts_collator = NULL)

stri_cmp(e1, e2, ..., opts_collator = NULL)

stri_cmp_eq(e1, e2)

stri_cmp_neq(e1, e2)

stri_cmp_equiv(e1, e2, ..., opts_collator = NULL)

stri_cmp_nequiv(e1, e2, ..., opts_collator = NULL)

stri_cmp_lt(e1, e2, ..., opts_collator = NULL)

stri_cmp_gt(e1, e2, ..., opts_collator = NULL)

stri_cmp_le(e1, e2, ..., opts_collator = NULL)

stri_cmp_ge(e1, e2, ..., opts_collator = NULL)

Arguments

Details

All the functions listed here are vectorized over e1 and e2.

stri_cmp_eq tests whether two corresponding strings consist of exactly the same code points, while stri_cmp_neq allows to check whether there is any difference between them. These are locale-independent operations: for natural language processing, where the notion of canonical equivalence is more valid, this might not be exactly what you are looking for, see Examples. Please note that stringi always silently removes UTF-8 BOMs from input strings, therefore, e.g., stri_cmp_eq does not take BOMs into account while comparing strings.

stri_cmp_equiv tests for canonical equivalence of two strings and is locale-dependent. Additionally, the ICU's Collator may be tuned up so that, e.g., the comparison is case-insensitive. To test whether two strings are not canonically equivalent, call stri_cmp_nequiv.

stri_cmp_le tests whether the elements in the first vector are less than or equal to the corresponding elements in the second vector, stri_cmp_ge tests whether they are greater or equal, stri_cmp_lt if less, and stri_cmp_gt if greater, see also, e.g., %s<%.

stri_compare is an alias to stri_cmp. They both perform exactly the same locale-dependent operation. Both functions provide a C library's strcmp() look-and-feel, see Value for details.

For more information on ICU's Collator and how to tune its settings refer to stri_opts_collator. Note that different locale settings may lead to different results (see the examples below).

Value

The stri_cmp and stri_compare functions return an integer vector representing the comparison results: -1 if e1[...] < e2[...], 0 if they are canonically equivalent, and 1 if greater.

All the other functions return a logical vector that indicates whether a given relation holds between two corresponding elements in e1 and e2.

Author(s)

Marek Gagolewski and other contributors

References

Collation – ICU User Guide, https://unicode-org.github.io/icu/userguide/collation/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, about_search_coll, stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Examples

# in Polish, ch < h:
stri_cmp_lt('hladny', 'chladny', locale='pl_PL')

# in Slovak, ch > h:
stri_cmp_lt('hladny', 'chladny', locale='sk_SK')

# < or > (depends on locale):
stri_cmp('hladny', 'chladny')

# ignore case differences:
stri_cmp_equiv('hladny', 'HLADNY', strength=2)

# also ignore diacritical differences:
stri_cmp_equiv('hladn\u00FD', 'hladny', strength=1, locale='sk_SK')

marios <- c('Mario', 'mario', 'M\\u00e1rio', 'm\\u00e1rio')
stri_cmp_equiv(marios, 'mario', case_level=TRUE, strength=2L)
stri_cmp_equiv(marios, 'mario', case_level=TRUE, strength=1L)
stri_cmp_equiv(marios, 'mario', strength=1L)
stri_cmp_equiv(marios, 'mario', strength=2L)

# non-Unicode-normalized vs normalized string:
stri_cmp_equiv(stri_trans_nfkd('\u0105'), '\u105')

# note the difference:
stri_cmp_eq(stri_trans_nfkd('\u0105'), '\u105')

# ligatures:
stri_cmp_equiv('\ufb00', 'ff', strength=2)

# phonebook collation
stri_cmp_equiv('G\u00e4rtner', 'Gaertner', locale='de_DE@collation=phonebook', strength=1L)
stri_cmp_equiv('G\u00e4rtner', 'Gaertner', locale='de_DE', strength=1L)

Description

These functions count the number of occurrences of a pattern in a string.

Usage

stri_count(str, ..., regex, fixed, coll, charclass)

stri_count_charclass(str, pattern)

stri_count_coll(str, pattern, ..., opts_collator = NULL)

stri_count_fixed(str, pattern, ..., opts_fixed = NULL)

stri_count_regex(str, pattern, ..., opts_regex = NULL)

Arguments

Details

Vectorized over str and pattern (with recycling of the elements in the shorter vector if necessary). This allows to, for instance, search for one pattern in each given string, search for each pattern in one given string, and search for the i-th pattern within the i-th string.

If pattern is empty, then the result is NA and a warning is generated.

stri_count is a convenience function. It calls either stri_count_regex, stri_count_fixed, stri_count_coll, or stri_count_charclass, depending on the argument used.

Value

All the functions return an integer vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_count: about_search, stri_count_boundaries()

Examples

s <- 'Lorem ipsum dolor sit amet, consectetur adipisicing elit.'
stri_count(s, fixed='dolor')
stri_count(s, regex='\\p{L}+')

stri_count_fixed(s, ' ')
stri_count_fixed(s, 'o')
stri_count_fixed(s, 'it')
stri_count_fixed(s, letters)
stri_count_fixed('babab', 'b')
stri_count_fixed(c('stringi', '123'), 'string')

stri_count_charclass(c('stRRRingi', 'STrrrINGI', '123'),
   c('\\p{Ll}', '\\p{Lu}', '\\p{Zs}'))
stri_count_charclass(' \t\n', '\\p{WHITE_SPACE}') # white space - binary property
stri_count_charclass(' \t\n', '\\p{Z}') # white-space - general category (note the difference)

stri_count_regex(s, '(s|el)it')
stri_count_regex(s, 'i.i')
stri_count_regex(s, '.it')
stri_count_regex('bab baab baaab', c('b.*?b', 'b.b'))
stri_count_regex(c('stringi', '123'), '^(s|1)')

Description

These functions determine the number of text boundaries (like character, word, line, or sentence boundaries) in a string.

Usage

stri_count_boundaries(str, ..., opts_brkiter = NULL)

stri_count_words(str, locale = NULL)

Arguments

Details

Vectorized over str.

For more information on text boundary analysis performed by ICU's BreakIterator, see stringi-search-boundaries.

In case of stri_count_words, just like in stri_extract_all_words and stri_locate_all_words, ICU's word BreakIterator iterator is used to locate the word boundaries, and all non-word characters (UBRK_WORD_NONE rule status) are ignored. This function is equivalent to a call to stri_count_boundaries(str, type='word', skip_word_none=TRUE, locale=locale).

Note that a BreakIterator of type character may be used to count the number of Unicode characters in a string. The stri_length function, which aims to count the number of Unicode code points, might report different results.

Moreover, a BreakIterator of type sentence may be used to count the number of sentences in a text piece.

Value

Both functions return an integer vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_count: about_search, stri_count()

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, about_search_coll, stri_compare(), stri_duplicated(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other text_boundaries: about_search_boundaries, about_search, stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_brkiter(), stri_split_boundaries(), stri_split_lines(), stri_trans_tolower(), stri_wrap()

Examples

test <- 'The\u00a0above-mentioned    features are very useful. Spam, spam, eggs, bacon, and spam.'
stri_count_boundaries(test, type='word')
stri_count_boundaries(test, type='sentence')
stri_count_boundaries(test, type='character')
stri_count_words(test)

test2 <- stri_trans_nfkd('\u03c0\u0153\u0119\u00a9\u00df\u2190\u2193\u2192')
stri_count_boundaries(test2, type='character')
stri_length(test2)
stri_numbytes(test2)

Description

Modifies a date-time object by adding a specific amount of time units.

Usage

stri_datetime_add(
  time,
  value = 1L,
  units = "seconds",
  tz = NULL,
  locale = NULL
)

stri_datetime_add(time, units = "seconds", tz = NULL, locale = NULL) <- value

Arguments

Details

Vectorized over time and value.

Note that, e.g., January, 31 + 1 month = February, 28 or 29.

Value

Both functions return an object of class POSIXct.

The replacement version of stri_datetime_add modifies the state of the time object.

Author(s)

Marek Gagolewski and other contributors

References

Calendar Classes - ICU User Guide, https://unicode-org.github.io/icu/userguide/datetime/calendar/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_create(), stri_datetime_fields(), stri_datetime_format(), stri_datetime_fstr(), stri_datetime_now(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

x <- stri_datetime_now()
stri_datetime_add(x, units='months') <- 2
print(x)
stri_datetime_add(x, -2, units='months')
stri_datetime_add(stri_datetime_create(2014, 4, 20), 1, units='years')
stri_datetime_add(stri_datetime_create(2014, 4, 20), 1, units='years', locale='@calendar=hebrew')

stri_datetime_add(stri_datetime_create(2016, 1, 31), 1, units='months')

Description

Constructs date-time objects from numeric representations.

Usage

stri_datetime_create(
  year = NULL,
  month = NULL,
  day = NULL,
  hour = 0L,
  minute = 0L,
  second = 0,
  lenient = FALSE,
  tz = NULL,
  locale = NULL
)

Arguments

Details

Vectorized over year, month, day, hour, hour, minute, and second.

Value

Returns an object of class POSIXct.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_fields(), stri_datetime_format(), stri_datetime_fstr(), stri_datetime_now(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

stri_datetime_create(2015, 12, 31, 23, 59, 59.999)
stri_datetime_create(5775, 8, 1, locale='@calendar=hebrew')  # 1 Nisan 5775 -> 2015-03-21
stri_datetime_create(2015, 02, 29)
stri_datetime_create(2015, 02, 29, lenient=TRUE)
stri_datetime_create(hour=15, minute=59)

Description

Computes and returns values for all date and time fields.

Usage

stri_datetime_fields(time, tz = attr(time, "tzone"), locale = NULL)

Arguments

Details

Vectorized over time.

Value

Returns a data frame with the following columns:

1. Year (0 is 1BC, -1 is 2BC, etc.)

2. Month (1-based, i.e., 1 stands for the first month, e.g., January; note that the number of months depends on the selected calendar, see stri_datetime_symbols)

3. Day

4. Hour (24-h clock)

5. Minute

6. Second

7. Millisecond

8. WeekOfYear (this is locale-dependent)

9. WeekOfMonth (this is locale-dependent)

10. DayOfYear

11. DayOfWeek (1-based, 1 denotes Sunday; see stri_datetime_symbols)

12. Hour12 (12-h clock)

13. AmPm (see stri_datetime_symbols)

14. Era (see stri_datetime_symbols)

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_create(), stri_datetime_format(), stri_datetime_fstr(), stri_datetime_now(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

stri_datetime_fields(stri_datetime_now())
stri_datetime_fields(stri_datetime_now(), locale='@calendar=hebrew')
stri_datetime_symbols(locale='@calendar=hebrew')$Month[
   stri_datetime_fields(stri_datetime_now(), locale='@calendar=hebrew')$Month
]

Description

These functions convert a given date/time object to a character vector, or vice versa.

Usage

stri_datetime_format(
  time,
  format = "uuuu-MM-dd HH:mm:ss",
  tz = NULL,
  locale = NULL
)

stri_datetime_parse(
  str,
  format = "uuuu-MM-dd HH:mm:ss",
  lenient = FALSE,
  tz = NULL,
  locale = NULL
)

Arguments

Details

Vectorized over format and time or str.

When parsing strings, unspecified date-time fields (e.g., seconds where only hours and minutes are given) are based on today's midnight in the local time zone (for compatibility with strptime).

By default, stri_datetime_format (for compatibility with the strftime function) formats a date/time object using the current default time zone.

format may be one of DT_STYLE or DT_relative_STYLE, where DT is equal to date, time, or datetime, and STYLE is equal to full, long, medium, or short. This gives a locale-dependent date and/or time format. Note that currently ICU does not support relative time formats, thus this flag is currently ignored in such a context.

Otherwise, format is a pattern: a string where specific sequences of characters are replaced with date/time data from a calendar when formatting or used to generate data for a calendar when parsing. For example, y stands for 'year'. Characters may be used multiple times: yy might produce 99, whereas yyyy yields 1999. For most numerical fields, the number of characters specifies the field width. For example, if h is the hour, h might produce 5, but hh yields 05. For some characters, the count specifies whether an abbreviated or full form should be used.

Two single quotes represent a literal single quote, either inside or outside single quotes. Text within single quotes is not interpreted in any way (except for two adjacent single quotes). Otherwise, all ASCII letters from a to z and A to Z are reserved as syntax characters, and require quoting if they are to represent literal characters. In addition, certain ASCII punctuation characters may become available in the future (e.g., : being interpreted as the time separator and / as a date separator, and replaced by respective locale-sensitive characters in display).

Note that any characters in the pattern that are not in the ranges of [a-z] and [A-Z] will be treated as quoted text. For instance, characters like :, ., (a space), # and @ will appear in the resulting time text even if they are not enclosed within single quotes. The single quote is used to “escape” the letters. Two single quotes in a row, inside or outside a quoted sequence, represent a “real” single quote.

A few examples:

Value

stri_datetime_format returns a character vector.

stri_datetime_parse returns an object of class POSIXct.

Author(s)

Marek Gagolewski and other contributors

References

Formatting Dates and Times – ICU User Guide, https://unicode-org.github.io/icu/userguide/format_parse/datetime/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_create(), stri_datetime_fields(), stri_datetime_fstr(), stri_datetime_now(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

x <- c('2015-02-28', '2015-02-29')
stri_datetime_parse(x, 'yyyy-MM-dd')
stri_datetime_parse(x, 'yyyy-MM-dd', lenient=TRUE)
stri_datetime_parse(x %s+% " 17:13", "yyyy-MM-dd HH:mm")
stri_datetime_parse('19 lipca 2015', 'date_long', locale='pl_PL')
stri_datetime_format(stri_datetime_now(), 'datetime_relative_medium')

Description

This function converts strptime or strftime-style format strings to ICU format strings that may be used in stri_datetime_parse and stri_datetime_format functions.

Usage

stri_datetime_fstr(x, ignore_special = TRUE)

Arguments

Details

For more details on conversion specifiers please refer to the manual page of strptime. Most of the formatters of the form %x, where x is a letter, are supported. Moreover, each %% is replaced with %.

Warnings are given in the case of %x, %X, %u, %w, %g, %G, %c, %U, and %W as in such circumstances either ICU does not support the functionality requested using the string format API or there are some inconsistencies between base R and ICU.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_create(), stri_datetime_fields(), stri_datetime_format(), stri_datetime_now(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

stri_datetime_fstr('%Y-%m-%d %H:%M:%S')

Description

Returns the current date and time.

Usage

stri_datetime_now()

Details

The current date and time in stringi is represented as the (signed) number of seconds since 1970-01-01 00:00:00 UTC. UTC leap seconds are ignored.

Value

Returns an object of class POSIXct.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_create(), stri_datetime_fields(), stri_datetime_format(), stri_datetime_fstr(), stri_datetime_symbols(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Description

Returns a list of all localizable date-time formatting data, including month and weekday names, localized AM/PM strings, etc.

Usage

stri_datetime_symbols(locale = NULL, context = "standalone", width = "wide")

Arguments

Details

context stands for a selector for date formatting context and width - for date formatting width.

Value

Returns a list with the following named components:

1. Month - month names,

2. Weekday - weekday names,

3. Quarter - quarter names,

4. AmPm - AM/PM names,

5. Era - era names.

Author(s)

Marek Gagolewski and other contributors

References

Calendar - ICU User Guide, https://unicode-org.github.io/icu/userguide/datetime/calendar/

DateFormatSymbols class – ICU API Documentation, https://unicode-org.github.io/icu-docs/apidoc/dev/icu4c/classicu_1_1DateFormatSymbols.html

Formatting Dates and Times – ICU User Guide, https://unicode-org.github.io/icu/userguide/format_parse/datetime/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other datetime: stri_datetime_add(), stri_datetime_create(), stri_datetime_fields(), stri_datetime_format(), stri_datetime_fstr(), stri_datetime_now(), stri_timezone_get(), stri_timezone_info(), stri_timezone_list()

Examples

stri_datetime_symbols() # uses the Gregorian calendar in most locales
stri_datetime_symbols('@calendar=hebrew')
stri_datetime_symbols('he_IL@calendar=hebrew')
stri_datetime_symbols('@calendar=islamic')
stri_datetime_symbols('@calendar=persian')
stri_datetime_symbols('@calendar=indian')
stri_datetime_symbols('@calendar=coptic')
stri_datetime_symbols('@calendar=japanese')

stri_datetime_symbols('ja_JP_TRADITIONAL') # uses the Japanese calendar by default
stri_datetime_symbols('th_TH_TRADITIONAL') # uses the Buddhist calendar

stri_datetime_symbols('pl_PL', context='format')
stri_datetime_symbols('pl_PL', context='standalone')

stri_datetime_symbols(width='wide')
stri_datetime_symbols(width='abbreviated')
stri_datetime_symbols(width='narrow')

Description

These functions determine, for each string in str, if there is at least one match to a corresponding pattern.

Usage

stri_detect(str, ..., regex, fixed, coll, charclass)

stri_detect_fixed(
  str,
  pattern,
  negate = FALSE,
  max_count = -1,
  ...,
  opts_fixed = NULL
)

stri_detect_charclass(str, pattern, negate = FALSE, max_count = -1)

stri_detect_coll(
  str,
  pattern,
  negate = FALSE,
  max_count = -1,
  ...,
  opts_collator = NULL
)

stri_detect_regex(
  str,
  pattern,
  negate = FALSE,
  max_count = -1,
  ...,
  opts_regex = NULL
)

Arguments

Details

Vectorized over str and pattern (with recycling of the elements in the shorter vector if necessary). This allows to, for instance, search for one pattern in each given string, search for each pattern in one given string, and search for the i-th pattern within the i-th string.

If pattern is empty, then the result is NA and a warning is generated.

stri_detect is a convenience function. It calls either stri_detect_regex, stri_detect_fixed, stri_detect_coll, or stri_detect_charclass, depending on the argument used.

See also stri_startswith and stri_endswith for testing whether a string starts or ends with a match to a given pattern. Moreover, see stri_subset for a character vector subsetting.

If max_count is negative, then all stings are examined. Otherwise, searching terminates once max_count matches (or, if negate is TRUE, no-matches) are detected. The uninspected cases are marked as missing in the return vector. Be aware that, unless pattern is a singleton, the elements in str might be inspected in a non-consecutive order.

Value

Each function returns a logical vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_detect: about_search, stri_startswith()

Examples

stri_detect_fixed(c('stringi R', 'R STRINGI', '123'), c('i', 'R', '0'))
stri_detect_fixed(c('stringi R', 'R STRINGI', '123'), 'R')

stri_detect_charclass(c('stRRRingi','R STRINGI', '123'),
   c('\\p{Ll}', '\\p{Lu}', '\\p{Zs}'))

stri_detect_regex(c('stringi R', 'R STRINGI', '123'), 'R.')
stri_detect_regex(c('stringi R', 'R STRINGI', '123'), '[[:alpha:]]*?')
stri_detect_regex(c('stringi R', 'R STRINGI', '123'), '[a-zC1]')
stri_detect_regex(c('stringi R', 'R STRINGI', '123'), '( R|RE)')
stri_detect_regex('stringi', 'STRING.', case_insensitive=TRUE)

stri_detect_regex(c('abc', 'def', '123', 'ghi', '456', '789', 'jkl'),
   '^[0-9]+$', max_count=1)
stri_detect_regex(c('abc', 'def', '123', 'ghi', '456', '789', 'jkl'),
   '^[0-9]+$', max_count=2)
stri_detect_regex(c('abc', 'def', '123', 'ghi', '456', '789', 'jkl'),
   '^[0-9]+$', negate=TRUE, max_count=3)

Description

Duplicates each str(e1) string times(e2) times and concatenates the results.

Usage

stri_dup(str, times)

e1 %s*% e2

e1 %stri*% e2

Arguments

Details

Vectorized over all arguments.

e1 %s*% e2 and e1 %stri*% e2 are synonyms for stri_dup(e1, e2)

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other join: %s+%(), stri_flatten(), stri_join_list(), stri_join()

Examples

stri_dup('a', 1:5)
stri_dup(c('a', NA, 'ba'), 4)
stri_dup(c('abc', 'pqrst'), c(4, 2))
"a" %s*% 5

Description

stri_duplicated() determines which strings in a character vector are duplicates of other elements.

stri_duplicated_any() determines if there are any duplicated strings in a character vector.

Usage

stri_duplicated(
  str,
  from_last = FALSE,
  fromLast = from_last,
  ...,
  opts_collator = NULL
)

stri_duplicated_any(
  str,
  from_last = FALSE,
  fromLast = from_last,
  ...,
  opts_collator = NULL
)

Arguments

Details

Missing values are regarded as equal.

Unlike duplicated and anyDuplicated, these functions test for canonical equivalence of strings (and not whether the strings are just bytewise equal) Such operations are locale-dependent. Hence, stri_duplicated and stri_duplicated_any are significantly slower (but much better suited for natural language processing) than their base R counterparts.

See also stri_unique for extracting unique elements.

Value

stri_duplicated() returns a logical vector of the same length as str. Each of its elements indicates whether a canonically equivalent string was already found in str.

stri_duplicated_any() returns a single non-negative integer. Value of 0 indicates that all the elements in str are unique. Otherwise, it gives the index of the first non-unique element.

Author(s)

Marek Gagolewski and other contributors

References

Collation - ICU User Guide, https://unicode-org.github.io/icu/userguide/collation/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, about_search_coll, stri_compare(), stri_count_boundaries(), stri_enc_detect2(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Examples

# In the following examples, we have 3 duplicated values,
# 'a' - 2 times, NA - 1 time
stri_duplicated(c('a', 'b', 'a', NA, 'a', NA))
stri_duplicated(c('a', 'b', 'a', NA, 'a', NA), from_last=TRUE)
stri_duplicated_any(c('a', 'b', 'a', NA, 'a', NA))

# compare the results:
stri_duplicated(c('\u0105', stri_trans_nfkd('\u0105')))
duplicated(c('\u0105', stri_trans_nfkd('\u0105')))

stri_duplicated(c('gro\u00df', 'GROSS', 'Gro\u00df', 'Gross'), strength=1)
duplicated(c('gro\u00df', 'GROSS', 'Gro\u00df', 'Gross'))

Description

This function uses the ICU engine to determine the character set, or encoding, of character data in an unknown format.

Usage

stri_enc_detect(str, filter_angle_brackets = FALSE)

Arguments

Details

Vectorized over str and filter_angle_brackets.

For a character vector input, merging all text lines via stri_flatten(str, collapse='\n') might be needed if str has been obtained via a call to readLines and in fact represents an image of a single text file.

This is, at best, an imprecise operation using statistics and heuristics. Because of this, detection works best if you supply at least a few hundred bytes of character data that is mostly in a single language. However, because the detection only looks at a limited amount of the input data, some of the returned character sets may fail to handle all of the input data. Note that in some cases, the language can be determined along with the encoding.

Several different techniques are used for character set detection. For multi-byte encodings, the sequence of bytes is checked for legible patterns. The detected characters are also checked against a list of frequently used characters in that encoding. For single byte encodings, the data is checked against a list of the most commonly occurring three letter groups for each language that can be written using that encoding.

The detection process can be configured to optionally ignore HTML or XML style markup (using ICU's internal facilities), which can interfere with the detection process by changing the statistics.

This function should most often be used for byte-marked input strings, especially after loading them from text files and before the main conversion with stri_encode. The input encoding is of course not taken into account here, even if marked.

The following table shows all the encodings that can be detected:

Value

Returns a list of length equal to the length of str. Each list element is a data frame with the following three named vectors representing all the guesses:

• Encoding – string; guessed encodings; NA on failure,

• Language – string; guessed languages; NA if the language could not be determined (e.g., in case of UTF-8),

• Confidence – numeric in [0,1]; the higher the value, the more confidence there is in the match; NA on failure.

The guesses are ordered by decreasing confidence.

Author(s)

Marek Gagolewski and other contributors

References

Character Set Detection – ICU User Guide, https://unicode-org.github.io/icu/userguide/conversion/detection.html

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_detection: about_encoding, stri_enc_detect2(), stri_enc_isascii(), stri_enc_isutf16be(), stri_enc_isutf8()

Examples

## Not run:
## f <- rawToChar(readBin('test.txt', 'raw', 100000))
## stri_enc_detect(f)

Description

This function tries to detect character encoding in case the language of text is known.

Usage

stri_enc_detect2(str, locale = NULL)

Arguments

Details

Vectorized over str.

First, the text is checked whether it is valid UTF-32BE, UTF-32LE, UTF-16BE, UTF-16LE, UTF-8 (as in stri_enc_detect, this is roughly inspired by ICU's i18n/csrucode.cpp) or ASCII.

If locale is not NA and the above fails, the text is checked for the number of occurrences of language-specific code points (data provided by the ICU library) converted to all possible 8-bit encodings that fully cover the indicated language. The encoding is selected based on the greatest number of total byte hits.

The guess is of course imprecise, as it is obtained using statistics and heuristics. Because of this, detection works best if you supply at least a few hundred bytes of character data that is in a single language.

If you have no initial guess on the language and encoding, try with stri_enc_detect (uses ICU facilities).

Value

Just like stri_enc_detect, this function returns a list of length equal to the length of str. Each list element is a data frame with the following three named components:

• Encoding – string; guessed encodings; NA on failure (if and only if encodings is empty),

• Language – always NA,

• Confidence – numeric in [0,1]; the higher the value, the more confidence there is in the match; NA on failure.

The guesses are ordered by decreasing confidence.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, about_search_coll, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_extract_all_boundaries(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other encoding_detection: about_encoding, stri_enc_detect(), stri_enc_isascii(), stri_enc_isutf16be(), stri_enc_isutf8()

Description

This function converts integer vectors, representing sequences of UTF-32 code points, to UTF-8 strings.

Usage

stri_enc_fromutf32(vec)

Arguments

Details

UTF-32 is a 32-bit encoding where each Unicode code point corresponds to exactly one integer value.

This function is a vectorized version of intToUtf8. As usual in stringi, it returns character strings in UTF-8. See stri_enc_toutf32 for a dual operation.

If an ill-defined code point is given, a warning is generated and the corresponding string is set to NA. Note that 0s are not allowed in vec, as they are used internally to mark the end of a string (in the C API).

See also stri_encode for decoding arbitrary byte sequences from any given encoding.

Value

Returns a character vector (in UTF-8). NULLs in the input list are converted to NA_character_.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_toascii(), stri_enc_tonative(), stri_enc_toutf32(), stri_enc_toutf8(), stri_encode()

Description

Gets basic information on a character encoding.

Usage

stri_enc_info(enc = NULL)

Arguments

Details

An error is raised if the provided encoding is unknown to ICU (see stri_enc_list for more details).

Value

Returns a list with the following components:

• Name.friendly – friendly encoding name: MIME Name or JAVA Name or ICU Canonical Name (the first of provided ones is selected, see below);

• Name.ICU – encoding name as identified by ICU;

• Name.* – other standardized encoding names, e.g., Name.UTR22, Name.IBM, Name.WINDOWS, Name.JAVA, Name.IANA, Name.MIME (some of them may be unavailable for all the encodings);

• ASCII.subset – is ASCII a subset of the given encoding?;

• Unicode.1to1 – for 8-bit encodings only: are all characters translated to exactly one Unicode code point and is the translation scheme reversible?;

• CharSize.8bit – is this an 8-bit encoding, i.e., do we have CharSize.min == CharSize.max and CharSize.min == 1?;

• CharSize.min – minimal number of bytes used to represent a UChar (in UTF-16, this is not the same as UChar32)

• CharSize.max – maximal number of bytes used to represent a UChar (in UTF-16, this is not the same as UChar32, i.e., does not reflect the maximal code point representation size)

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_management: about_encoding, stri_enc_list(), stri_enc_mark(), stri_enc_set()

Description

The function checks whether all bytes in a string are <= 127.

Usage

stri_enc_isascii(str)

Arguments

Details

This function is independent of the way R marks encodings in character strings (see Encoding and stringi-encoding).

Value

Returns a logical vector. The i-th element indicates whether the i-th string corresponds to a valid ASCII byte sequence.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_detection: about_encoding, stri_enc_detect2(), stri_enc_detect(), stri_enc_isutf16be(), stri_enc_isutf8()

Examples

stri_enc_isascii(letters[1:3])
stri_enc_isascii('\u0105\u0104')

Description

These functions detect whether a given byte stream is valid UTF-16LE, UTF-16BE, UTF-32LE, or UTF-32BE.

Usage

stri_enc_isutf16be(str)

stri_enc_isutf16le(str)

stri_enc_isutf32be(str)

stri_enc_isutf32le(str)

Arguments

Details

These functions are independent of the way R marks encodings in character strings (see Encoding and stringi-encoding). Most often, these functions act on raw vectors.

A result of FALSE means that a string is surely not valid UTF-16 or UTF-32. However, false positives are possible.

Also note that a data stream may be sometimes classified as both valid UTF-16LE and UTF-16BE.

Value

Returns a logical vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_detection: about_encoding, stri_enc_detect2(), stri_enc_detect(), stri_enc_isascii(), stri_enc_isutf8()

Description

The function checks whether given sequences of bytes forms a proper UTF-8 string.

Usage

stri_enc_isutf8(str)

Arguments

Details

FALSE means that a string is certainly not valid UTF-8. However, false positives are possible. For instance, (c4,85) represents ('a with ogonek') in UTF-8 as well as ('A umlaut', 'Ellipsis') in WINDOWS-1250. Also note that UTF-8, as well as most 8-bit encodings, extend ASCII (note that stri_enc_isascii implies that stri_enc_isutf8).

However, the longer the sequence, the greater the possibility that the result is indeed in UTF-8 – this is because not all sequences of bytes are valid UTF-8.

This function is independent of the way R marks encodings in character strings (see Encoding and stringi-encoding).

Value

Returns a logical vector. Its i-th element indicates whether the i-th string corresponds to a valid UTF-8 byte sequence.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_detection: about_encoding, stri_enc_detect2(), stri_enc_detect(), stri_enc_isascii(), stri_enc_isutf16be()

Examples

stri_enc_isutf8(letters[1:3])
stri_enc_isutf8('\u0105\u0104')
stri_enc_isutf8('\u1234\u0222')

Description

Gives the list of encodings that are supported by ICU.

Usage

stri_enc_list(simplify = TRUE)

Arguments

Details

Apart from given encoding identifiers and their aliases, some other specifiers might additionally be available. This is due to the fact that ICU tries to normalize converter names. For instance, 'UTF8' is also valid, see stringi-encoding for more information.

Value

If simplify is FALSE, a list of character vectors is returned. Each list element represents a unique character encoding. The name attribute gives the ICU Canonical Name of an encoding family. The elements (character vectors) are its aliases.

If simplify is TRUE (the default), then the resulting list is coerced to a character vector and sorted, and returned with removed duplicated entries.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_management: about_encoding, stri_enc_info(), stri_enc_mark(), stri_enc_set()

Examples

stri_enc_list()
stri_enc_list(FALSE)

Description

Reads declared encodings for each string in a character vector as seen by stringi.

Usage

stri_enc_mark(str)

Arguments

Details

According to Encoding, R has a simple encoding marking mechanism: strings can be declared to be in latin1, UTF-8 or bytes.

Moreover, we may check (via the R/C API) whether a string is in ASCII (R assumes that this holds if and only if all bytes in a string are not greater than 127, so there is an implicit assumption that your platform uses an encoding that extends ASCII) or in the system's default (a.k.a. unknown in Encoding) encoding.

Intuitively, the default encoding should be equivalent to the one you use on stdin (e.g., your 'keyboard'). In stringi we assume that such an encoding is equivalent to the one returned by stri_enc_get. It is automatically detected by ICU to match – by default – the encoding part of the LC_CTYPE category as given by Sys.getlocale.

Value

Returns a character vector of the same length as str. Unlike in the Encoding function, here the possible encodings are: ASCII, latin1, bytes, native, and UTF-8. Additionally, missing values are handled properly.

This gives exactly the same data that is used by all the functions in stringi to re-encode their inputs.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_management: about_encoding, stri_enc_info(), stri_enc_list(), stri_enc_set()

Description

stri_enc_set sets the encoding used to re-encode strings internally (i.e., by R) declared to be in native encoding, see stringi-encoding and stri_enc_mark. stri_enc_get returns the currently used default encoding.

Usage

stri_enc_set(enc)

stri_enc_get()

Arguments

Details

stri_enc_get is the same as stri_enc_info(NULL)$Name.friendly.

Note that changing the default encoding may have undesired consequences. Unless you are an expert user and you know what you are doing, stri_enc_set should only be used if ICU fails to detect your system's encoding correctly (while testing stringi we only encountered such a situation on a very old Solaris machine). Note that ICU tries to match the encoding part of the LC_CTYPE category as given by Sys.getlocale.

If you set a default encoding that is neither a superset of ASCII, nor an 8-bit encoding, a warning will be generated, see stringi-encoding for discussion.

stri_enc_set has no effect if the system ICU assumes that the default charset is always UTF-8 (i.e., where the internal U_CHARSET_IS_UTF8 is defined and set to 1), see stri_info.

Value

stri_enc_set returns a string with previously used character encoding, invisibly.

stri_enc_get returns a string with current default character encoding.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_management: about_encoding, stri_enc_info(), stri_enc_list(), stri_enc_mark()

Description

This function converts input strings to ASCII, i.e., to character strings consisting of bytes not greater than 127.

Usage

stri_enc_toascii(str)

Arguments

Details

All code points greater than 127 are replaced with the ASCII SUBSTITUTE CHARACTER (0x1A). R encoding declarations are always used to determine which encoding is assumed for each input, see stri_enc_mark. If ill-formed byte sequences are found in UTF-8 byte streams, a warning is generated.

A bytes-marked string is assumed to be in an 8-bit encoding extending the ASCII map (a common assumption in R itself).

Note that the SUBSTITUTE CHARACTER (\x1a == \032) may be interpreted as the ASCII missing value for single characters.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_fromutf32(), stri_enc_tonative(), stri_enc_toutf32(), stri_enc_toutf8(), stri_encode()

Description

Converts character strings with declared encodings to the current native encoding.

Usage

stri_enc_tonative(str)

Arguments

Details

This function just calls stri_encode(str, NULL, NULL). The current native encoding can be read with stri_enc_get. Character strings declared to be in bytes encoding will fail here.

Note that if working in a UTF-8 environment, resulting strings will be marked with UTF-8 and not native, see stri_enc_mark.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_fromutf32(), stri_enc_toascii(), stri_enc_toutf32(), stri_enc_toutf8(), stri_encode()

Description

UTF-32 is a 32-bit encoding where each Unicode code point corresponds to exactly one integer value. This function converts a character vector to a list of integer vectors so that, e.g., individual code points may be easily accessed, changed, etc.

Usage

stri_enc_toutf32(str)

Arguments

Details

See stri_enc_fromutf32 for a dual operation.

This function is roughly equivalent to a vectorized call to utf8ToInt(enc2utf8(str)). If you want a list of raw vectors on output, use stri_encode.

Unlike utf8ToInt, if ill-formed UTF-8 byte sequences are detected, a corresponding element is set to NULL and a warning is generated. To deal with such issues, use, e.g., stri_enc_toutf8.

Value

Returns a list of integer vectors. Missing values are converted to NULLs.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_fromutf32(), stri_enc_toascii(), stri_enc_tonative(), stri_enc_toutf8(), stri_encode()

Description

Converts character strings with declared marked encodings to UTF-8 strings.

Usage

stri_enc_toutf8(str, is_unknown_8bit = FALSE, validate = FALSE)

Arguments

Details

If is_unknown_8bit is set to FALSE (the default), then R encoding marks are used, see stri_enc_mark. Bytes-marked strings will cause the function to fail.

If a string is in UTF-8 and has a byte order mark (BOM), then the BOM will be silently removed from the output string.

If the default encoding is UTF-8, see stri_enc_get, then strings marked with native are – for efficiency reasons – returned as-is, i.e., with unchanged markings. A similar behavior is observed when calling enc2utf8.

For is_unknown_8bit=TRUE, if a string is declared to be neither in ASCII nor in UTF-8, then all byte codes > 127 are replaced with the Unicode REPLACEMENT CHARACTER (\Ufffd). Note that the REPLACEMENT CHARACTER may be interpreted as Unicode missing value for single characters. Here a bytes-marked string is assumed to use an 8-bit encoding that extends the ASCII map.

What is more, setting validate to TRUE or NA in both cases validates the resulting UTF-8 byte stream. If validate=TRUE, then in case of any incorrect byte sequences, they will be replaced with the REPLACEMENT CHARACTER. This option may be used in a case where you want to fix an invalid UTF-8 byte sequence. For NA, a bogus string will be replaced with a missing value.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_fromutf32(), stri_enc_toascii(), stri_enc_tonative(), stri_enc_toutf32(), stri_encode()

Description

These functions convert strings between encodings. They aim to serve as a more portable and faster replacement for R's own iconv.

Usage

stri_encode(str, from = NULL, to = NULL, to_raw = FALSE)

stri_conv(str, from = NULL, to = NULL, to_raw = FALSE)

Arguments

Details

stri_conv is an alias for stri_encode.

Refer to stri_enc_list for the list of supported encodings and stringi-encoding for a general discussion.

If from is either missing, '', or NULL, and if str is a character vector then the marked encodings are used (see stri_enc_mark) – in such a case bytes-declared strings are disallowed. Otherwise, i.e., if str is a raw-type vector or a list of raw vectors, we assume that the input encoding is the current default encoding as given by stri_enc_get.

However, if from is given explicitly, the internal encoding declarations are always ignored.

For to_raw=FALSE, the output strings always have the encodings marked according to the target converter used (as specified by to) and the current default Encoding (ASCII, latin1, UTF-8, native, or bytes in all other cases).

Note that some issues might occur if to indicates, e.g, UTF-16 or UTF-32, as the output strings may have embedded NULs. In such cases, please use to_raw=TRUE and consider specifying a byte order marker (BOM) for portability reasons (e.g., set UTF-16 or UTF-32 which automatically adds the BOMs).

Note that stri_encode(as.raw(data), 'encodingname') is a clever substitute for rawToChar.

In the current version of stringi, if an incorrect code point is found on input, it is replaced with the default (for that target encoding) 'missing/erroneous' character (with a warning), e.g., the SUBSTITUTE character (U+001A) or the REPLACEMENT one (U+FFFD). Occurrences thereof can be located in the output string to diagnose the problematic sequences, e.g., by calling: stri_locate_all_regex(converted_string, '[\ufffd\u001a]'.

Because of the way this function is currently implemented, maximal size of a single string to be converted cannot exceed ~0.67 GB.

Value

If to_raw is FALSE, then a character vector with encoded strings (and appropriate encoding marks) is returned. Otherwise, a list of vectors of type raw is produced.

Author(s)

Marek Gagolewski and other contributors

References

Conversion – ICU User Guide, https://unicode-org.github.io/icu/userguide/conversion/

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other encoding_conversion: about_encoding, stri_enc_fromutf32(), stri_enc_toascii(), stri_enc_tonative(), stri_enc_toutf32(), stri_enc_toutf8()

Description

Generates an ASCII string where all non-printable characters and non-ASCII characters are converted to escape sequences.

Usage

stri_escape_unicode(str)

Arguments

Details

For non-printable and certain special (well-known, see also the R man page Quotes) ASCII characters, the following (also recognized in R) convention is used. We get \a, \b, \t, \n, \v, \f, \r, \", \', \\ or either \uXXXX (4 hex digits) or \UXXXXXXXX (8 hex digits) otherwise.

As usual in stringi, any input string is converted to Unicode before executing the escape process.

Value

Returns a character vector.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other escape: stri_unescape_unicode()

Examples

stri_escape_unicode('a\u0105!')

Description

These functions extract all substrings matching a given pattern.

stri_extract_all_* extracts all the matches. stri_extract_first_* and stri_extract_last_* yield the first or the last matches, respectively.

Usage

stri_extract_all(str, ..., regex, fixed, coll, charclass)

stri_extract_first(str, ..., regex, fixed, coll, charclass)

stri_extract_last(str, ..., regex, fixed, coll, charclass)

stri_extract(
  str,
  ...,
  regex,
  fixed,
  coll,
  charclass,
  mode = c("first", "all", "last")
)

stri_extract_all_charclass(
  str,
  pattern,
  merge = TRUE,
  simplify = FALSE,
  omit_no_match = FALSE
)

stri_extract_first_charclass(str, pattern)

stri_extract_last_charclass(str, pattern)

stri_extract_all_coll(
  str,
  pattern,
  simplify = FALSE,
  omit_no_match = FALSE,
  ...,
  opts_collator = NULL
)

stri_extract_first_coll(str, pattern, ..., opts_collator = NULL)

stri_extract_last_coll(str, pattern, ..., opts_collator = NULL)

stri_extract_all_regex(
  str,
  pattern,
  simplify = FALSE,
  omit_no_match = FALSE,
  ...,
  opts_regex = NULL
)

stri_extract_first_regex(str, pattern, ..., opts_regex = NULL)

stri_extract_last_regex(str, pattern, ..., opts_regex = NULL)

stri_extract_all_fixed(
  str,
  pattern,
  simplify = FALSE,
  omit_no_match = FALSE,
  ...,
  opts_fixed = NULL
)

stri_extract_first_fixed(str, pattern, ..., opts_fixed = NULL)

stri_extract_last_fixed(str, pattern, ..., opts_fixed = NULL)

Arguments

Details

Vectorized over str and pattern (with recycling of the elements in the shorter vector if necessary). This allows to, for instance, search for one pattern in each given string, search for each pattern in one given string, and search for the i-th pattern within the i-th string.

Check out stri_match for the extraction of matches to individual regex capture groups.

stri_extract, stri_extract_all, stri_extract_first, and stri_extract_last are convenience functions. They merely call stri_extract_*_*, depending on the arguments used.

Value

For stri_extract_all*, if simplify=FALSE (the default), then a list of character vectors is returned. Each list element represents the results of a different search scenario. If a pattern is not found and omit_no_match=FALSE, then a character vector of length 1 with single NA value will be generated.

Otherwise, i.e., if simplify is not FALSE, then stri_list2matrix with byrow=TRUE argument is called on the resulting object. In such a case, the function yields a character matrix with an appropriate number of rows (according to the length of str, pattern, etc.). Note that stri_list2matrix's fill argument is set either to an empty string or NA, depending on whether simplify is TRUE or NA, respectively.

stri_extract_first* and stri_extract_last* return a character vector. A NA element indicates a no-match.

Note that stri_extract_last_regex searches from start to end, but skips overlapping matches, see the example below.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_extract: about_search, stri_extract_all_boundaries(), stri_match_all()

Examples

stri_extract_all('XaaaaX', regex=c('\\p{Ll}', '\\p{Ll}+', '\\p{Ll}{2,3}', '\\p{Ll}{2,3}?'))
stri_extract_all('Bartolini', coll='i')
stri_extract_all('stringi is so good!', charclass='\\p{Zs}') # all white-spaces

stri_extract_all_charclass(c('AbcdeFgHijK', 'abc', 'ABC'), '\\p{Ll}')
stri_extract_all_charclass(c('AbcdeFgHijK', 'abc', 'ABC'), '\\p{Ll}', merge=FALSE)
stri_extract_first_charclass('AaBbCc', '\\p{Ll}')
stri_extract_last_charclass('AaBbCc', '\\p{Ll}')

## Not run: 
# emoji support available since ICU 57
stri_extract_all_charclass(stri_enc_fromutf32(32:55200), '\\p{EMOJI}')

## End(Not run)

stri_extract_all_coll(c('AaaaaaaA', 'AAAA'), 'a')
stri_extract_first_coll(c('Yy\u00FD', 'AAA'), 'y', strength=2, locale='sk_SK')
stri_extract_last_coll(c('Yy\u00FD', 'AAA'), 'y',  strength=1, locale='sk_SK')

stri_extract_all_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+', '\\p{Ll}{2,3}', '\\p{Ll}{2,3}?'))
stri_extract_first_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+', '\\p{Ll}{2,3}', '\\p{Ll}{2,3}?'))
stri_extract_last_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+', '\\p{Ll}{2,3}', '\\p{Ll}{2,3}?'))

stri_list2matrix(stri_extract_all_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+')))
stri_extract_all_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+'), simplify=TRUE)
stri_extract_all_regex('XaaaaX', c('\\p{Ll}', '\\p{Ll}+'), simplify=NA)

stri_extract_all_fixed('abaBAba', 'Aba', case_insensitive=TRUE)
stri_extract_all_fixed('abaBAba', 'Aba', case_insensitive=TRUE, overlap=TRUE)

# Searching for the last occurrence:
# Note the difference - regex searches left to right, with no overlaps.
stri_extract_last_fixed("agAGA", "aga", case_insensitive=TRUE)
stri_extract_last_regex("agAGA", "aga", case_insensitive=TRUE)

Description

These functions extract data between text boundaries.

Usage

stri_extract_all_boundaries(
  str,
  simplify = FALSE,
  omit_no_match = FALSE,
  ...,
  opts_brkiter = NULL
)

stri_extract_last_boundaries(str, ..., opts_brkiter = NULL)

stri_extract_first_boundaries(str, ..., opts_brkiter = NULL)

stri_extract_all_words(
  str,
  simplify = FALSE,
  omit_no_match = FALSE,
  locale = NULL
)

stri_extract_first_words(str, locale = NULL)

stri_extract_last_words(str, locale = NULL)

Arguments

Details

Vectorized over str.

For more information on text boundary analysis performed by ICU's BreakIterator, see stringi-search-boundaries.

In case of stri_extract_*_words, just like in stri_count_words, ICU's word BreakIterator iterator is used to locate the word boundaries, and all non-word characters (UBRK_WORD_NONE rule status) are ignored.

Value

For stri_extract_all_*, if simplify=FALSE (the default), then a list of character vectors is returned. Each string consists of a separate word. In case of omit_no_match=FALSE and if there are no words or if a string is missing, a single NA is provided on output.

Otherwise, stri_list2matrix with byrow=TRUE argument is called on the resulting object. In such a case, a character matrix with length(str) rows is returned. Note that stri_list2matrix's fill argument is set to an empty string and NA, for simplify TRUE and NA, respectively.

For stri_extract_first_* and stri_extract_last_*, a character vector is returned. A NA element indicates a no-match.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other search_extract: about_search, stri_extract_all(), stri_match_all()

Other locale_sensitive: %s<%(), about_locale, about_search_boundaries, about_search_coll, stri_compare(), stri_count_boundaries(), stri_duplicated(), stri_enc_detect2(), stri_locate_all_boundaries(), stri_opts_collator(), stri_order(), stri_rank(), stri_sort_key(), stri_sort(), stri_split_boundaries(), stri_trans_tolower(), stri_unique(), stri_wrap()

Other text_boundaries: about_search_boundaries, about_search, stri_count_boundaries(), stri_locate_all_boundaries(), stri_opts_brkiter(), stri_split_boundaries(), stri_split_lines(), stri_trans_tolower(), stri_wrap()

Examples

stri_extract_all_words('stringi: THE string processing package 123.48...')

Description

Joins the elements of a character vector into one string.

Usage

stri_flatten(str, collapse = "", na_empty = FALSE, omit_empty = FALSE)

Arguments

Details

The stri_flatten(str, collapse='XXX') call is equivalent to paste(str, collapse='XXX', sep='').

If you wish to use some more fancy (e.g., differing) separators between flattened strings, call stri_join(str, separators, collapse='').

If str is not empty, then a single string is returned. If collapse has length > 1, then only the first string will be used.

Value

Returns a single string, i.e., a character vector of length 1.

Author(s)

Marek Gagolewski and other contributors

See Also

The official online manual of stringi at https://stringi.gagolewski.com/

Gagolewski M., stringi: Fast and portable character string processing in R, Journal of Statistical Software 103(2), 2022, 1-59, doi:10.18637/jss.v103.i02

Other join: %s+%(), stri_dup(), stri_join_list(), stri_join()

Examples

stri_flatten(LETTERS)
stri_flatten(LETTERS, collapse=',')
stri_flatten(stri_dup(letters[1:6], 1:3))
stri_flatten(c(NA, '', 'A', '', 'B', NA, 'C'), collapse=',', na_empty=TRUE, omit_empty=TRUE)
stri_flatten(c(NA, '', 'A', '', 'B', NA, 'C'), collapse=',', na_empty=NA)

Description

Gives the current default settings used by the ICU library.

Usage

stri_info(short = FALSE)

Arguments