Skip to content

std::collation

Collation: ordering text as a reader of a language orders it. < on String compares UTF-8 bytes, and collation is never implicit, so "Zorn" < "apple" and "Ärlig" sorts after every ASCII word. A Collator orders strings by the Unicode Collation Algorithm with the CLDR rules of a locale: in German ä sorts with a, in Swedish after z, and in Polish ł is a letter after l.

Make a collator once and reuse it: sort with Iter.sort_with(names, &Collator.compare(sv, &1, &2)), or keep each string’s Collator.sort_key and compare those. A program carries the collation data of the locales its nomi.toml lists; a collator for a locale it does not carry orders as that locale’s nearest listed parent, or as the root order every language starts from. The Chinese and Korean tailorings, 160 to 250 KB each, are carried only when the program lists those locales. Case mapping for a language and caseless matching are String.uppercase_in, String.lowercase_in, String.capitalize_in and String.fold_case (std::strings).

Import with import std::collation.

The order is the Unicode Collation Algorithm (UTS #10) over CLDR’s root collation, with the locale’s CLDR tailoring laid over it (UTS #35 part 5): German keeps ä with a, Swedish puts å, ä and ö after z, Polish puts ł after l, Czech ch after h, Lithuanian y with i, Chinese sorts Han by pinyin before Latin, and Traditional Chinese by stroke count. Strings are compared in NFD, so canonically equivalent strings are equal at every strength below Identical. A Han character a tailoring does not list sorts by its code point, the core ideographs before the rest, and an unassigned code point after every assigned one.

The locale’s tag chooses among a language’s collations and sets the options: -u-co- names a CLDR collation type (de-u-co-phonebk sorts ü as ue; es-u-co-trad sorts ch and ll as letters; zh-u-co-stroke), and -u-ks- (level1 to level4, identic), -u-kn-true, -u-kf-upper or -u-kf-lower, and -u-ka-shifted set what strength, numeric, case_first and ignore_punctuation set. The settings apply in order: the tailoring’s own (Danish puts upper case first), then the tag’s keywords, then each argument of Collator.new that is not its default.

A locale whose collation the program does not carry orders as its nearest carried parent along the collation chain (the collations component of CLDR’s parent locales, else the main ones short of root, else the tag less its last subtag: zh-Hant takes zh’s stroke order), else in the root order.

type Collator String

opaque — construction surface is private to its defining module

Compares strings as readers of one language order them. Make it with Collator.new. A Collator is Equatable and Hashable, and its Debug is the call that makes it.

The locale’s tag can ask for a different order of the language: a -u-co- keyword picks a CLDR collation type (de-u-co-phonebk sorts ü as ue, es-u-co-trad sorts ch after c and ll after l, as letters of their own, zh-Hant sorts by stroke), and -u-kn-true orders digits by value, -u-kf-upper puts upper case first and -u-ks-level1 compares base letters only, as the arguments to Collator.new do.

sv = Collator.new(Locale"sv")
assert Iter.sort_with(["ö", "z", "a", "å"], &Collator.compare(sv, &1, &2)) == ["a", "z", "å", "ö"]
fn new(locale: Locale, strength: Strength = Strength.Tertiary, numeric: Bool = False, case_first: CaseFirst = CaseFirst.Default, ignore_punctuation: Bool = False) -> Collator

type function on Collator

A collator for locale. strength is how fine a difference counts; numeric orders a run of decimal digits, of any script, by its value (file9 before file10); case_first moves upper case before or after lower case; ignore_punctuation is UTS #10’s shifted alternate: spaces and punctuation are ignored at the first three levels and compared at the quaternary, so they count only when the strings are otherwise equal (de luge with deluge). An argument left at its default keeps what the locale’s tag and its collation say.

de = Collator.new(Locale"de")
assert Collator.compare(de, "Ärger", "Arzt") == Less

loose = Collator.new(Locale"en", strength: Strength.Primary)
assert Collator.compare(loose, "résumé", "RESUME") == Equal

files = Collator.new(Locale"en", numeric: True)
assert Collator.compare(files, "file9", "file10") == Less
fn compare(collator: Collator, a: String, b: String) -> Ordering

type function on Collator

Orders a and b as the collator’s locale does.

sv = Collator.new(Locale"sv")
assert Collator.compare(sv, "Zorn", "Åberg") == Less
assert Collator.compare(Collator.new(Locale"de"), "Zorn", "Åberg") == Greater
fn sort_key(collator: Collator, text: String) -> Bytes

type function on Collator

The bytes that order text under the collator: two strings’ keys compare, under Bytes’ Comparable order, as Collator.compare compares the strings. Compute each string’s key once to sort or index many strings, or to store the order. A key is only meaningful beside keys from the same collator and the same version of Nomi.

pl = Collator.new(Locale"pl")
assert Collator.sort_key(pl, "łoś") > Collator.sort_key(pl, "lis")
assert Collator.sort_key(pl, "łoś") < Collator.sort_key(pl, "mysz")

impl Debug for Collator

fn inspect(collator: Collator) -> String

impl Debug.inspect

The call that makes the collator, its arguments that are not defaults included.

assert Debug.inspect(Collator.new(Locale"sv")) == "Collator.new(Locale\"sv\")"

assert Debug.inspect(Collator.new(Locale"en", strength: Strength.Primary, numeric: True))
    == "Collator.new(Locale\"en\", strength: Strength.Primary, numeric: True)"

impl Equatable for Collator

fn equal(a: Collator, b: Collator) -> Bool

impl Equatable.equal

impl Hashable for Collator

fn hash(value: Collator) -> Int

impl Hashable.hash

enum Strength
    Primary
    Secondary
    Tertiary
    Quaternary
    Identical
end

How fine a difference a collator tells apart. Each level breaks the ties of the one before it.

Variants

  • Primary

    Base letters only: a, á and A are equal.

  • Secondary

    Base letters and accents: a and A are equal, á is not.

  • Tertiary

    Letters, accents and case: the default.

  • Quaternary

    Also what ignore_punctuation ignores, and Japanese hiragana from katakana.

  • Identical

    Every difference: strings equal at every level compare by their code points.

impl Debug for Strength

fn inspect(value: Strength) -> String

impl Debug.inspect

impl Equatable for Strength

fn equal(a: Strength, b: Strength) -> Bool

impl Equatable.equal

enum CaseFirst
    Default
    Upper
    Lower
    Off
end

Where upper case sorts beside lower case.

Variants

  • Default

    As the locale’s collation puts it: Danish and Maltese upper case first, most languages lower case first.

  • Upper

    Upper case before lower case.

  • Lower

    Lower case before upper case.

  • Off

    As the tertiary weights fall, with no rule for case: lower case first for most letters, even in Danish.

impl Debug for CaseFirst

fn inspect(value: CaseFirst) -> String

impl Debug.inspect

impl Equatable for CaseFirst

fn equal(a: CaseFirst, b: CaseFirst) -> Bool

impl Equatable.equal