Repository navigation
Description
Summary
encode() writes a JavaScript string with a lone UTF-16 surrogate in two different ways, and the length of the string selects the way:
- In a string of 50 characters or fewer,
utf8EncodeJs()writes the surrogate as its 3-byte form. For example, U+D800 becomesED A0 80. These bytes are not valid UTF-8. - In a longer string,
TextEncoder.encodeInto()writes U+FFFD (EF BF BD), as the WHATWG Encoding Standard tells.
The MessagePack spec says that the str format holds UTF-8. Thus a short string with a lone surrogate gives a str that a strict decoder in another language can refuse. The same value also gives different bytes when it has padding.
Version: 3.1.3. src/utils/utf8.ts on main (6d4b666) is the same.
Repro
npm install @msgpack/msgpack@3.1.3, then node repro.ts with Node.js 22.18 or later:
// F0137: @msgpack/msgpack 3.1.3 encodes a string of 50 characters or fewer with utf8EncodeJs, and a longer string with
// TextEncoder.encodeInto. utf8EncodeJs writes a lone UTF-16 surrogate as the 3-byte form of the surrogate (ED A0 80 for
// U+D800), which is not valid UTF-8. TextEncoder writes U+FFFD (EF BF BD). The MessagePack str type holds UTF-8.
import { encode } from "@msgpack/msgpack";
const hex = (bytes: Uint8Array): string => Buffer.from(bytes).toString("hex").replace(/(..)/g, "$1 ").trim();
const strPayload = (bytes: Uint8Array): Uint8Array => {
const head = bytes[0]!;
if (head >= 0xa0 && head <= 0xbf) return bytes.subarray(1);
if (head === 0xd9) return bytes.subarray(2);
throw new Error(`not a fixstr or a str 8: ${head.toString(16)}`);
};
const isUtf8 = (bytes: Uint8Array): boolean => {
try {
new TextDecoder("utf-8", { fatal: true }).decode(bytes);
return true;
} catch {
return false;
}
};
const textEncoder = new TextEncoder();
const padding = "x".repeat(60);
const cases: Array<[string, string]> = [
["lone high surrogate at the end", "a\ud800"],
["lone low surrogate", "a\udc00b"],
["low surrogate before high surrogate", "\udc00\ud800"],
];
let faults = 0;
for (const [label, text] of cases) {
for (const [size, value] of [["short", text], ["long", padding + text]] as const) {
const payload = strPayload(encode(value));
const expected = textEncoder.encode(value);
const same = hex(payload) === hex(expected);
if (!same) faults++;
const tail = (bytes: Uint8Array): string => hex(bytes.subarray(Math.max(0, bytes.length - 6)));
console.log(
`${same ? "ok " : "FAULT"} ${label}, ${size} (${value.length} characters): last bytes ${tail(payload)}, valid UTF-8: ${isUtf8(payload)}, TextEncoder: ${tail(expected)}`,
);
}
}
if (faults > 0) {
console.log(`BUG PRESENT: ${faults} of ${cases.length * 2} strings encode a lone surrogate as bytes that are not valid UTF-8, but only when the string has 50 characters or fewer.`);
process.exit(1);
}
console.log("OK: each string encodes as TextEncoder encodes it.");Output
FAULT lone high surrogate at the end, short (2 characters): last bytes 61 ed a0 80, valid UTF-8: false, TextEncoder: 61 ef bf bd
ok lone high surrogate at the end, long (62 characters): last bytes 78 78 61 ef bf bd, valid UTF-8: true, TextEncoder: 78 78 61 ef bf bd
FAULT lone low surrogate, short (3 characters): last bytes 61 ed b0 80 62, valid UTF-8: false, TextEncoder: 61 ef bf bd 62
ok lone low surrogate, long (63 characters): last bytes 78 61 ef bf bd 62, valid UTF-8: true, TextEncoder: 78 61 ef bf bd 62
FAULT low surrogate before high surrogate, short (2 characters): last bytes ed b0 80 ed a0 80, valid UTF-8: false, TextEncoder: ef bf bd ef bf bd
ok low surrogate before high surrogate, long (62 characters): last bytes ef bf bd ef bf bd, valid UTF-8: true, TextEncoder: ef bf bd ef bf bd
BUG PRESENT: 3 of 6 strings encode a lone surrogate as bytes that are not valid UTF-8, but only when the string has 50 characters or fewer.
The CI run of the repro (Node.js on Ubuntu): https://github.com/mark1russell7/wirelab/actions/runs/44229138782
Expected
Each string encodes as TextEncoder encodes it: a lone surrogate becomes EF BF BD, at each string length.
Possible fix
In utf8EncodeJs(), after the check for a surrogate pair, a value that is still in the range 0xD800 to 0xDFFF is a lone surrogate. The function can write U+FFFD for it. U+FFFD also has 3 bytes, so utf8Count() stays correct. I can send a pull request with a test if you want one.
An AI coding assistant (Claude Code) found this fault, wrote the repro and wrote this text.
Activity