UTF-8 decoder capability and stress test
https://www.w3.org/2001/06/utf-8-wrong/UTF-8-test.htmlThis test text examines, how UTF-8 decoders handle various types of
corrupted or otherwise interesting UTF-8 sequences.
According to ISO 10646-1, sections R.7 and 2.3c, a device receiving
UTF-8 shall interpret a "malformed sequence in the same way that it
interprets a character that is outside the adopted subset". This means
usually that the malformed UTF-8 sequence is replaced by a replacement
character (U+FFFD), which looks a bit like an inverted question mark,
or a similar symbol. It might be a good idea to visually distinguish a
malformed UTF-8 sequence from a correctly encoded Unicode character
that is just not available in the current font but otherwise fully
legal. For both cases, a clearly recognisable symbol should be used.
Just ignoring malformed sequences or unavailable characters will make
debugging more difficult and can lead to user confusion.
Test sequences:
Check, whether a malformed UTF-8 sequence is (1) represented at all,
(2) represented by exactly one replacement character (or equivalent
signal), and (3) the following quotation mark after an illegal UTF-8
sequence is correctly displayed, i.e. proper resynchronization takes
place. This file says "THE END" in the last line, so if you don't see
that, your decoder crashed somehow before, which is also not nice.
Here come the tests:
1 Some correct UTF-8 text
You should see the Greek word 'kosme': "κόσμε"
2 Boundary condition test cases
You should see a correctly encoded character each time (which is not
necessarily an existing character).
2.1 First possible sequence of a certain length
2.1.1 1 byte (U-00000000): ""
2.1.2 2 bytes (U-00000080): ""
2.1.3 3 bytes (U-00000800): "ࠀ"
2.1.4 4 bytes (U-00010000): "𐀀"
2.1.5 5 bytes (U-00200000): "�����"
2.1.6 6 bytes (U-04000000): "������"
2.2 Last possible sequence of a certain length
2.2.1 1 byte (U-0000007F): ""
2.2.2 2 bytes (U-000007FF): "߿"
2.2.3 3 bytes (U-0000FFFF): ""
2.2.4 4 bytes (U-001FFFFF): "����"
2.2.5 5 bytes (U-03FFFFFF): "�����"
2.2.6 6 bytes (U-7FFFFFFF): "������"
2.3 Other boundary conditions
2.3.1 U-0000D7FF = ed 9f bf = ""
2.3.2 U-0000E000 = ee 80 80 = ""
2.3.3 U-0000FFFD = ef bf bd = "�"
2.3.4 U-0010FFFF = f4 8f bf bf = ""
2.3.5 U-00110000 = f4 90 80 80 = "����"
3 Malformed sequences
3.1 Unexpected continuation bytes
Each unexpected continuation byte should be separately signalled as a
malformed sequence of its own.
3.1.1 First continuation byte 0x80: "�"
3.1.2 Last continuation byte 0xbf: "�"
3.1.3 2 continuation bytes: "��"
3.1.4 3 continuation bytes: "���"
3.1.5 4 continuation bytes: "����"
3.1.6 5 continuation bytes: "�����"
3.1.7 6 continuation bytes: "������"
3.1.8 7 continuation bytes: "�������"
3.1.9 Sequence of all 64 possible continuation bytes (0x80-0xbf):
"����������������
����������������
����������������
����������������"
����������������
����������������"
3.2 Lonely start characters
3.2.1 All 32 first bytes of 2-byte sequences (0xc0-0xdf),
each followed by a space character:
"� � � � � � � � � � � � � � � �
� � � � � � � � � � � � � � � � "
3.2.2 All 16 first bytes of 3-byte sequences (0xe0-0xef),
each followed by a space character:
"� � � � � � � � � � � � � � � � "
3.2.3 All 8 first bytes of 4-byte sequences (0xf0-0xf7),
each followed by a space character:
"� � � � � � � � "
3.2.4 All 4 first bytes of 5-byte sequences (0xf8-0xfb),
each followed by a space character:
"� � � � "
3.2.5 All 2 first bytes of 6-byte sequences (0xfc-0xfd),
each followed by a space character:
"� � "
3.3 Sequences with last continuation byte missing
All bytes of an incomplete sequence should be signalled as a single
malformed sequence, i.e., you should see only a single replacement
characters in each of the next 10 tests. (Characters as in section 2)
3.3.1 2-byte sequence with last byte missing (U+0000): "�"
3.3.2 3-byte sequence with last byte missing (U+0000): "��"
3.3.3 4-byte sequence with last byte missing (U+0000): "���"
3.3.4 5-byte sequence with last byte missing (U+0000): "����"
3.3.5 6-byte sequence with last byte missing (U+0000): "�����"
3.3.6 2-byte sequence with last byte missing (U-000007FF): "�"
3.3.7 3-byte sequence with last byte missing (U-0000FFFF): "�"
3.3.8 4-byte sequence with last byte missing (U-001FFFFF): "���"
3.3.9 5-byte sequence with last byte missing (U-03FFFFFF): "����"
3.3.10 6-byte sequence with last byte missing (U-7FFFFFFF): "�����"
3.4 Concatenation of incomplete sequences
All the 10 sequences of 3.3 concatenated, you should see 10 malformed
sequences being signaled:
"�����������������������������"
3.5 Impossible bytes
The following two bytes cannot appear in a correct UTF-8 string
3.5.1 fe = "�"
3.5.2 ff = "�"
3.5.3 fe fe ff ff = "����"
4 Overlong sequences
The following sequences are not malformed according to the letter of
the Unicode 2.0 standard. However, they are longer then necessary and
a correct UTF-8 encoder is not allowed to produce them. A "safe UTF-8
decoder" should reject them just like malformed sequences for two
reasons: (1) It helps to debug applications if overlong sequences are
not treated as valid representations of characters, because this helps
to spot problems more quickly. (2) Overlong sequences provide
alternative representations of characters, that could maliciously be
used to bypass filters that check only for ASCII characters. For
instance, a 2-byte encoded line feed (LF) would not be caught by a
line counter that counts only 0x0a bytes, but it would still be
processed as a line feed by an unsafe UTF-8 decoder later in the
pipeline. From a security point of view, ASCII compatibility of UTF-8
sequences means also, that ASCII characters are *only* allowed to be
represented by ASCII bytes in the range 0x00-0x7f. To ensure this
aspect of ASCII compatibility, use only "safe UTF-8 decoders" that
reject overlong UTF-8 sequences for which a shorter encoding exists.
4.1 Examples of an overlong ASCII character
With a safe UTF-8 decoder, all of the following five overlong
representations of the ASCII character slash ("/") should be rejected
like a malformed UTF-8 sequence, for instance by substituting it with
a replacement character. If you see a slash below, you do not have a
safe UTF-8 decoder!
4.1.1 U+002F = c0 af = "��"
4.1.2 U+002F = e0 80 af = "���"
4.1.3 U+002F = f0 80 80 af = "����"
4.1.4 U+002F = f8 80 80 80 af = "�����"
4.1.5 U+002F = fc 80 80 80 80 af = "������"
4.2 Maximum overlong sequences
Below you see the highest Unicode value that is still resulting in an
overlong sequence if represented with the given number of bytes. This
is a boundary test for safe UTF-8 decoders. All five characters should
be rejected like malformed UTF-8 sequences.
4.2.1 U-0000007F = c1 bf = "��"
4.2.2 U-000007FF = e0 9f bf = "���"
4.2.3 U-0000FFFF = f0 8f bf bf = "����"
4.2.4 U-001FFFFF = f8 87 bf bf bf = "�����"
4.2.5 U-03FFFFFF = fc 83 bf bf bf bf = "������"
4.3 Overlong representation of the NUL character
The following five sequences should also be rejected like malformed
UTF-8 sequences and should not be treated like the ASCII NUL
character.
4.3.1 U+0000 = c0 80 = "��"
4.3.2 U+0000 = e0 80 80 = "���"
4.3.3 U+0000 = f0 80 80 80 = "����"
4.3.4 U+0000 = f8 80 80 80 80 = "�����"
4.3.5 U+0000 = fc 80 80 80 80 80 = "������"
5 Illegal code positions
The following UTF-8 sequences should be rejected like malformed
sequences, because they never represent valid ISO 10646 characters and
a UTF-8 decoder that accepts them might introduce security problems
comparable to overlong UTF-8 sequences.
5.1 Single UTF-16 surrogates
5.1.1 U+D800 = ed a0 80 = "���"
5.1.2 U+DB7F = ed ad bf = "���"
5.1.3 U+DB80 = ed ae 80 = "���"
5.1.4 U+DBFF = ed af bf = "���"
5.1.5 U+DC00 = ed b0 80 = "���"
5.1.6 U+DF80 = ed be 80 = "���"
5.1.7 U+DFFF = ed bf bf = "���"
5.2 Paired UTF-16 surrogates
5.2.1 U+D800 U+DC00 = ed a0 80 ed b0 80 = "������"
5.2.2 U+D800 U+DFFF = ed a0 80 ed bf bf = "������"
5.2.3 U+DB7F U+DC00 = ed ad bf ed b0 80 = "������"
5.2.4 U+DB7F U+DFFF = ed ad bf ed bf bf = "������"
5.2.5 U+DB80 U+DC00 = ed ae 80 ed b0 80 = "������"
5.2.6 U+DB80 U+DFFF = ed ae 80 ed bf bf = "������"
5.2.7 U+DBFF U+DC00 = ed af bf ed b0 80 = "������"
5.2.8 U+DBFF U+DFFF = ed af bf ed bf bf = "������"
5.3 Other illegal code positions
5.3.1 U+FFFE = ef bf be = ""
5.3.2 U+FFFF = ef bf bf = ""
THE END