Why Tree-Sitter Uses Byte Offsets

When writing our AST-based context pruner context-pilot-mcp, we used tree-sitter to locate function signatures and block bodies. Tree-sitter's syntax tree nodes return coordinates as row/column numbers and raw byte offsets.

The UTF-8 Multi-Byte Trap

In languages like Python or JavaScript, string characters (like emojis or non-ASCII characters) can occupy up to 4 bytes in UTF-8. If your source code contains:

# 😊 emoji comment
def hello():
    pass

The emoji 😊 is 1 character, but is encoded as 4 bytes in UTF-8.

If you slice a Python string using tree-sitter's byte offsets (e.g. source_str[start_byte:end_byte]), you are indexing a character array with a byte coordinate. In JavaScript or Python, this leads to string corruption or character-boundary slicing errors (splitting a 4-byte emoji in half, causing decoding exceptions).

The Solution: Byte-Space Manipulation

To resolve this safely, we encode the character string into raw bytes, perform all interval slicing and offset lookups on the byte array, and decode the final sliced byte array back to a UTF-8 character string.