260808.5 · 9dfdba41f5dc1b5f712529fa923bdcf67eb09c30

Prefilter rune-mode input for non-ASCII patterns

The scan only ran for ASCII patterns, so searching CJK text with a CJK query still built the full score matrix. Scan for one byte of the pattern rune and verify all four.

Which byte matters. Every ASCII rune contributes three zero bytes, so U+AE00 scanned by its zero low byte hits on nearly every character of an ASCII-heavy line. Pick a byte that cannot occur in an ASCII rune, else any non-zero one.

A non-ASCII pattern rune is safe only when no other rune lowercases onto it. Uncased is not sufficient: U+00DF has no simple uppercase, yet U+1E9E lowercases to it, so the foldable set is excluded too.

Measured on 1.4M-line corpora, with a non-ASCII query:

- Every line CJK: 5.1x to 10.2x
- Mostly-ASCII paths behind a Hangul prefix: 5.6x to 6.0x, and 1.2x
where every line matches so nothing can be rejected
- ASCII queries unchanged, kept off the non-inlinable guard

@@ -13,3 +13,11 @@ func indexAsciiRune(runes []rune, caseSensitive bool, b byte, from int) int { func lastIndexAsciiRune(runes []rune, caseSensitive bool, b byte, from int) int { 	return lastIndexAsciiRuneRef(runes, caseSensitive, b, from) }++func indexRune(runes []rune, r rune, from int) int {+	return indexRuneRef(runes, r, from)+}++func lastIndexRune(runes []rune, r rune, from int) int {+	return lastIndexRuneRef(runes, r, from)+}