unicode/utf8 - tau doc

utf8 - text as code points.

A tau string is a run of bytes, and indexing or slicing one counts bytes, the way Go does it. The text inside is UTF-8, so a letter outside ASCII takes more than one byte and len counts them all:

utf8 = import("unicode/utf8")
len("città")             # 6, the bytes
utf8.RuneCount("città")  # 5, the letters

A code point is a number here, what Go calls a rune. Decoding one at a time over a string is faster on bytes than on the string, because a string has to be converted first:

b = bytes(s)
i = 0
for i < len(b) {
	r = utf8.DecodeRune(b, i)
	println(r[0])
	i = i + r[1]
}

¶RuneError = 0xfffdvaluesource

RuneError is what a byte that isn't valid UTF-8 decodes to, the replacement character U+FFFD.

¶RuneSelf = 0x80valuesource

RuneSelf is the first code point that doesn't fit in a single byte.

¶MaxRune = 0x10ffffvaluesource

MaxRune is the largest code point there is.

¶UTFMax = 4valuesource

UTFMax is the most bytes one code point takes.

¶ValidRune = fn(r)source

ValidRune reports whether r is a code point that can be encoded.

¶RuneLen = fn(r)source

RuneLen returns how many bytes the code point r takes, or -1 when it cannot be encoded.

¶EncodeRune = fn(r)source

EncodeRune returns the string of the code point r. An r that cannot be encoded gives RuneError, as it does in Go.

¶DecodeRune = fn(s, i)source

DecodeRune returns [code point, bytes read] for the character starting at offset i of s, which is a string or bytes. Bytes that are not valid UTF-8 decode to [RuneError, 1].

¶Runes = fn(s)source

Runes returns the code points of s, one number per character.

¶FromRunes = fn(rs)source

FromRunes returns the string of the code points in rs.

¶RuneCount = fn(s)source

RuneCount returns how many characters s holds, which is len(s) only while the text stays inside ASCII.

¶Valid = fn(s)source

Valid reports whether s is UTF-8 from end to end.

¶RuneIndex = fn(s, n)source

RuneIndex returns the byte offset where character n starts, or -1 when the string is shorter than that. It is what to feed slice when a position is counted in characters rather than bytes.

¶Slice = fn(s, start, end)source

Slice returns the characters of s from start to end, counted in characters rather than bytes.