Streamly.Internal.Unicode.Stream

Decode a stream of bytes to Unicode characters by mapping each byte to a corresponding Unicode Char in 0-255 range.

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. The function throws an error if an invalid codepoint is encountered.

Decode a UTF-8 encoded bytestream to a stream of Unicode characters. Any invalid codepoint encountered is dropped.

Similar to decodeUtf16le but throws an error if an invalid codepoint is encountered.

Decode a UTF-16 encoded stream to a stream of Unicode characters. Any invalid codepoint encountered is replaced with the unicode replacement character.

The Word16s are expected to be in the little-endian byte order.

Pre-release

Pre-release

Like decodeUtf8 but for a chunked stream. It may be slightly faster than flattening the stream and then decoding with decodeUtf8.

Like 'decodeUtf8'' but for a chunked stream. It may be slightly faster than flattening the stream and then decoding with 'decodeUtf8''.

Like decodeUtf8_ but for a chunked stream. It may be slightly faster than flattening the stream and then decoding with decodeUtf8_.

Like encodeLatin1' but silently maps input codepoints beyond 255 to arbitrary Latin1 chars in 0-255 range. No error or exception is thrown when such mapping occurs.

Encode a stream of Unicode characters to bytes by mapping each character to a byte in 0-255 range. Throws an error if the input stream contains characters beyond 255.

Like encodeLatin1 but drops the input characters beyond 255.

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. Any Invalid characters (U+D800-U+D8FF) in the input stream are replaced by the Unicode replacement character U+FFFD.

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. When any invalid character (U+D800-U+D8FF) is encountered in the input stream the function errors out.

Encode a stream of Unicode characters to a UTF-8 encoded bytestream. Any Invalid characters (U+D800-U+D8FF) in the input stream are dropped.

Encode a stream of String using the supplied encoding scheme. Each string is encoded as an Array Word8.

Similar to encodeUtf16le but throws an error if any invalid character is encountered.

Encode a stream of Unicode characters to a UTF-16 encoded stream. Any invalid characters in the input stream are replaced by the Unicode replacement character U+FFFD.

The resulting Word16s are encoded in little-endian byte order.

Remove leading whitespace from a string.

>>> stripHead = Stream.dropWhile Char.isSpace

Pre-release

Fold each line of the stream using the supplied Fold and stream the result.

Definition:

>>> lines f = Stream.foldMany (Fold.takeEndBy_ (== '\n') f)

Usage:

>>> Stream.toList $ Unicode.lines Fold.toList (Stream.fromList "line1\nline2\nline3\n\n\n")
["line1","line2","line3","",""]

Pre-release

Fold each word of the stream using the supplied Fold.

Definition:

>>> words = Stream.wordsBy Char.isSpace

Usage:

>>> Stream.toList $ Unicode.words Fold.toList (Stream.fromList " ab  cd   ef ")
["ab","cd","ef"]

Pre-release

Unfold a stream to character streams using the supplied Unfold and concat the results suffixing a newline character \n to each stream.

Definition:

>>> unlines = Stream.unfoldEachEndBy '\n'
>>> unlines = Stream.unfoldEachEndBySeq "\n" Unfold.fromList

Pre-release

Unfold the elements of a stream to character streams using the supplied Unfold and concat the results with a whitespace character infixed between the streams.

>>> unwords = Stream.unfoldEachSepBy ' '
>>> unwords = Stream.unfoldEachSepBySeq " " Unfold.fromList

Pre-release

See section "3.9 Unicode Encoding Forms" in https://www.unicode.org/versions/Unicode13.0.0/UnicodeStandard-13.0.pdf

Read UTF-8 encoded bytes as chars from an Addr# until a 0 byte is encountered, the 0 byte is not included in the stream.

Unsafe: The caller is responsible for safe addressing.

Note that this is completely safe when reading from Haskell string literals because they are guaranteed to be NULL terminated:

>>> Stream.fold Fold.toList (Unicode.fromStr# "Haskell"#)
"Haskell"

Ensure chunks of even length. This can be used before casting the arrays to Word16. Use this API when interacting with external data.

The chunks are split and merged accordingly to create arrays of even length. If the sum of length of all the arrays in the stream is odd then the trailing byte of the last array is dropped.

Swap the byte order of Word16

swapByteOrder 0xABCD == 0xCDAB
swapByteOrder . swapByteOrder == id

Same as decodeUtf8

Same as encodeLatin1

Same as encodeUtf8