Makefile

@@ -1,4 +1,4 @@-.PHONY: all build run test bench clean install uninstall deb install-deb uninstall-deb+.PHONY: all build syntax run test bench clean install uninstall deb install-deb uninstall-deb  # WORKING_NAME holds the program's name, provisional until a final one is # chosen. The binary, the package and the paths below take it.@@ -9,6 +9,12 @@ all: build build: 	CGO_ENABLED=0 go build -o build/$(NAME) . +# syntax builds the tree-sitter helper beside the binary, which finds it+# there. It is a module of its own, in syntax/, since tree-sitter needs cgo+# and a C compiler; the binary runs without it and says so on a tree page.+syntax:+	cd syntax && CGO_ENABLED=1 go build -o ../build/$(NAME)-syntax .+ # run serves this directory from build/ with --stamp, on port 8080 beside # an installed server on port 80. A server already there is replaced by # this build, serving the same trees, so its open pages go on under the@@ -16,7 +22,7 @@ build: # open, and www says which happened. RUNADDR := 127.31.$(shell expr $$(id -u) - 1000).1:8080 -run: build+run: build syntax 	@build/$(NAME) --restart --stamp -a $(RUNADDR) .  test:

README.md

@@ -81,7 +81,7 @@ www needs git. `make deb` builds a Debian package in `build/`, such as `www_0~git20261015223930.20f02fd_amd64.deb`, versioned by the last commit's UTC time and hash. It depends on git. For another architecture, pass `ARCH`, eg `make deb ARCH=arm64`. `make install-deb` builds the package while sudo asks for the password, then installs it with `sudo apt install`, which pulls in git. It first removes the `~/.local/bin` link from `make install`, which would otherwise shadow the package's binary. Run it again to replace an installed build, even of the same commit. `make uninstall-deb` removes the package. -To try a change without installing it, `make run` builds `build/www` and serves this directory on port 8080 with `--stamp`, beside an installed server on port 80. A later run replaces the server on 8080 with the new build, serving the same trees, and open pages go on under it, so edit and `make run` again. It opens the browser only when no page of this directory was open on the old server, and says which happened. A hidden tab holds no connection, so it does not count and gets a second tab. `www --kill -a 127.31.N.1:8080` stops it.+To try a change without installing it, `make run` builds `build/www` and the syntax tree helper and serves this directory on port 8080 with `--stamp`, beside an installed server on port 80. A later run replaces the server on 8080 with the new build, serving the same trees, and open pages go on under it, so edit and `make run` again. It opens the browser only when no page of this directory was open on the old server, and says which happened. A hidden tab holds no connection, so it does not count and gets a second tab. `www --kill -a 127.31.N.1:8080` stops it.  `make bench` times what a reader waits for: a search across a generated tree of 4,000 files and 24 MB, and a code page's first view per language: lexing alone, the whole page, and the page with scope boxes. `BENCH` picks benchmarks by pattern and `COUNT` repeats them, eg `make bench BENCH=Search COUNT=10` for benchstat. Python, JavaScript and shell are timed on argparse.py, jquery.js and bash_completion from Debian's packages and skipped where those are not installed. Warm pages have no benchmark, since `TestPageBudget` in `make test` holds them under 5 ms. @@ -108,7 +108,7 @@ In a large repo git can take a second for a history, because it reads every comm  Open pages reload when what they show changes. A doc takes its new version in place, without a reload, and each block that changed fills with the highlight colour and fades over three seconds, so the eye finds the change after a save. A code page reloads when its file is written, a listing when a name in it changes, any page in a repo when its HEAD moves, and the index when a tree joins or leaves. A page whose file or tree is gone waits two seconds for it to return, then reloads to say so. A page outlives its server. After three seconds without it, a line over the page says the server is gone and since when, and the page keeps trying each second. When a new server answers, eg after `make run` or `--restart`, the page takes its version from it and a line says new server, fading over three seconds. A restart takes about 60 ms, so its new server answers before the gone line shows. Each page holds a server-sent events stream while its tab is visible; a hidden tab lets go, since a browser opens at most six connections to one host, and checks again when shown. The server watches the directory of each open page and its repo's git directory through inotify, since editors save by renaming a new file over the old one, and polls every second where inotify fails. -Docs follow CommonMark with GitHub's extensions, rendered by goldmark: tables, strikethrough, task lists, bare URLs as links, footnotes and definition lists. Wiki links work as relative links do: `[[name]]` opens `name.md` in the doc's directory, `[[sub/name]]` opens `sub/name.md`, and a name with an extension, eg `[[main.go]]`, keeps it. `[[name|label]]` shows the label and `[[name#heading]]` opens at the heading. Lines join into a paragraph, and a blank line starts the next. A line that starts with a dash begins a list even right after text. Straight quotes and dashes turn typographic. A fenced block marked `dot` or `graphviz` becomes a diagram, drawn by Graphviz's `dot` when it is installed and kept by the hash of its source; without `dot`, or when `dot` rejects the graph, the block shows as code under a line that says why. `dot` may not read image files the graph names. Code in a fenced block with a known language is highlighted by chroma, as are code pages, whose language follows the file name. Go is lexed by Go's own scanner from the standard library instead, about 150 times faster, with the same colours. Python, JavaScript and shell have lexers of www's own, which follow chroma's rules and give every byte the class chroma gives it; chroma took 180 ms to lex argparse.py, 2,633 lines, and they take 2 ms. A whole page of it renders in about 5 ms, and jquery.js, 10,907 lines, in 28 ms instead of 400. Other languages go through chroma, at about 70 µs a line. The server keeps each coloured page by its content's hash, within 16 MB, so a later view takes a few milliseconds. A code page's links, scopes and raw, sit over the code's top right corner and follow the page down as it scrolls. Add `?scopes` to a code page, or follow scopes, and it draws each block of code that spans lines as a sunken box behind its text, nested as deep as the code goes, with deeper boxes fainter. A block is a bracket pair whose ends sit on different lines, or in Python the lines indented under a line that ends with a colon. Brackets and strings within a line, around something, are smaller boxes inside it, down to the innermost, much as vim's `ci(` and `ci"` see them; a string that spans lines is a block. Fills alternate inward between a light shade and the page's colour, so depth shows without the page darkening. The boxes lie on a layer of their own, so no character moves and copied text is unchanged. A box is a link to its lines. The box under the pointer is tinted, and a click marks it and the boxes in it and puts its lines in the address, eg `#L482-L520`; a click on the marked box clears both. When the file is committed and unchanged since, the address also names that commit's version, eg `@261024.22/server.go?scopes#L3-L26`, so the link keeps pointing at the same lines after the file changes. A live reload goes back to the current file. In the same corner, flat goes back to the plain page. Every gap between blocks is the same, whether the block is a paragraph, list, table or code. Headings sit close to their text.+Docs follow CommonMark with GitHub's extensions, rendered by goldmark: tables, strikethrough, task lists, bare URLs as links, footnotes and definition lists. Wiki links work as relative links do: `[[name]]` opens `name.md` in the doc's directory, `[[sub/name]]` opens `sub/name.md`, and a name with an extension, eg `[[main.go]]`, keeps it. `[[name|label]]` shows the label and `[[name#heading]]` opens at the heading. Lines join into a paragraph, and a blank line starts the next. A line that starts with a dash begins a list even right after text. Straight quotes and dashes turn typographic. A fenced block marked `dot` or `graphviz` becomes a diagram, drawn by Graphviz's `dot` when it is installed and kept by the hash of its source; without `dot`, or when `dot` rejects the graph, the block shows as code under a line that says why. `dot` may not read image files the graph names. Code in a fenced block with a known language is highlighted by chroma, as are code pages, whose language follows the file name. Go is lexed by Go's own scanner from the standard library instead, about 150 times faster, with the same colours. Python, JavaScript and shell have lexers of www's own, which follow chroma's rules and give every byte the class chroma gives it; chroma took 180 ms to lex argparse.py, 2,633 lines, and they take 2 ms. A whole page of it renders in about 5 ms, and jquery.js, 10,907 lines, in 28 ms instead of 400. Other languages go through chroma, at about 70 µs a line. The server keeps each coloured page by its content's hash, within 16 MB, so a later view takes a few milliseconds. A code page's links, scopes and raw, sit over the code's top right corner and follow the page down as it scrolls. Add `?scopes` to a code page, or follow scopes, and it draws each block of code that spans lines as a sunken box behind its text, nested as deep as the code goes, with deeper boxes fainter. A block is a bracket pair whose ends sit on different lines, or in Python the lines indented under a line that ends with a colon. Brackets and strings within a line, around something, are smaller boxes inside it, down to the innermost, much as vim's `ci(` and `ci"` see them; a string that spans lines is a block. Fills alternate inward between a light shade and the page's colour, so depth shows without the page darkening. The boxes lie on a layer of their own, so no character moves and copied text is unchanged. A box is a link to its lines. The box under the pointer is tinted, and a click marks it and the boxes in it and puts its lines in the address, eg `#L482-L520`; a click on the marked box clears both. When the file is committed and unchanged since, the address also names that commit's version, eg `@261024.22/server.go?scopes#L3-L26`, so the link keeps pointing at the same lines after the file changes. A live reload goes back to the current file. In the same corner, flat goes back to the plain page. Add `?tree`, or follow tree, for the syntax tree: one row per named node, with its field in its parent, its kind, its text on one line and a link to its lines, folded so the file's top-level declarations show first. A helper parses it with tree-sitter: `www-syntax`, which `make syntax` builds beside www and which needs a C compiler. It knows Go, Python, JavaScript, TypeScript, Rust, C, C++, shell, Java, C#, JSON, HTML, CSS and Ruby. Without the helper, or for another language, the page says so above the plain code. A first view of a 1,000-line file takes about 60 ms; the server keeps the tree, so a later view takes a few. Every gap between blocks is the same, whether the block is a paragraph, list, table or code. Headings sit close to their text.  www refuses to serve the home directory and every directory above it, such as `/home` and `/`. A tree exposes everything below it, and home holds `~/.ssh`, `~/.gnupg` and every token file. Symlinks inside the tree are followed, absolute ones included. A link that leads out of the tree is followed only into a directory that holds a file named `.www-follow`, or below one, and only when other users could read the target. A file needs the others-read bit and a directory others-read and others-execute, checked from the marker's directory down. So nobody reads through www what `cat` would not show them.

assets/style.css

@@ -461,6 +461,18 @@ pre.scopes i.scope.leaf {   height: 1.35em; } pre.filtering i.scope { display: none; }+/* A syntax tree is nested folds, one row per named node: its field in+   its parent, its kind, its text on one line and a link to its lines.+   Each level indents by two cells beside a rule, so depth reads as on a+   code page with scopes, and the text is code's size. */+div.syntax { font-family: ui-monospace, "Iosevka", monospace; font-size: max(0.5em, 11px); line-height: 1.35; padding-top: 1.35em; }+div.syntax details > :not(summary) { margin-left: 2ch; padding-left: 1ch; border-left: 1px solid var(--rule); }+div.syntax > div, div.syntax details > div { padding-left: 2ch; }+div.syntax summary { cursor: pointer; }+div.syntax .field { color: var(--muted); }+div.syntax code { color: var(--muted); font-size: 1em; }+div.syntax a { color: var(--gutter); text-decoration-color: var(--rule); margin-left: 0.5ch; }+div.syntax a:hover { text-decoration-color: currentColor; } /* A box is a link to its lines. The box under the pointer is tinted with    the target colour and its edges show at full strength, so it shows    what a click marks; a marked box and the boxes in it fill with the

server.go

@@ -789,8 +789,8 @@ func (t *tree) missing(name string) func(string) bool { }  // serveCode shows a text file as a highlighted code page with numbered,-// linkable lines; ?scopes draws each block of code as a box. Binary files-// go raw.+// linkable lines; ?scopes draws each block of code as a box, and ?tree+// shows the syntax tree. Binary files go raw. func (t *tree) serveCode(w http.ResponseWriter, r *http.Request, name string, info fs.FileInfo) { 	b, err := fs.ReadFile(t.fsys, name) 	if err != nil {@@ -803,15 +803,26 @@ func (t *tree) serveCode(w http.ResponseWriter, r *http.Request, name string, in 	} 	var body strings.Builder 	body.WriteString(t.logLine(r.Context(), name))-	// The links to the other view and to the raw file float over the+	// The links to the other views and to the raw file float over the 	// code's top right corner; div.code bounds how far they follow the-	// page down.-	_, scoped := r.URL.Query()["scopes"]+	// page down. A tree that cannot be had says why above the flat code.+	q := r.URL.Query()+	_, scoped := q["scopes"]+	if _, ok := q["tree"]; ok {+		tree, err := syntaxPage(r.Context(), name, pageHref(r.URL.Path), b)+		if err == nil {+			body.WriteString(`<div class="code">` + codeViews("tree") + tree + `</div>`)+			t.render(w, r, http.StatusOK, page{Title: name, Crumbs: t.crumbs(name), Kind: "tree", Body: template.HTML(body.String())})+			return+		}+		logErr("no syntax tree for %s: %v", name, err)+		body.WriteString(treeNote(name, err))+	}+	view := "flat" 	if scoped {-		body.WriteString(`<div class="code"><div class="view"><a href="?">flat</a><a href="?raw">raw</a></div>`)-	} else {-		body.WriteString(`<div class="code"><div class="view"><a href="?scopes">scopes</a><a href="?raw">raw</a></div>`)+		view = "scopes" 	}+	body.WriteString(`<div class="code">` + codeViews(view)) 	body.WriteString(highlightPage(codeLexer(name), b, scoped)) 	body.WriteString(`</div>`) 	t.render(w, r, http.StatusOK, page{Title: name, Crumbs: t.crumbs(name), Kind: "code", Body: template.HTML(body.String())})

syntax.go

new file mode 100644@@ -0,0 +1,223 @@+package main++import (+	"bytes"+	"context"+	"crypto/sha256"+	"errors"+	"html/template"+	"net/url"+	"os"+	"os/exec"+	"path/filepath"+	"strconv"+	"strings"+	"syscall"+	"time"+	"unicode/utf8"+)++// A code page's syntax tree comes from a helper, <name>-syntax, built+// from syntax/ with tree-sitter. Tree-sitter is C, so the helper keeps cgo+// out of the main binary, which runs without it and says so. The helper+// lies beside the binary, as make builds it, or on PATH.++// synNode is one named node of a syntax tree, as the helper prints it.+// Rows and columns count from 0, columns in bytes.+type synNode struct {+	depth          int+	kind, field    string+	sr, sc, er, ec int+}++// errNoHelper says the helper is not installed; errNoGrammar that it has+// no grammar for the file.+var (+	errNoHelper  = errors.New("no helper")+	errNoGrammar = errors.New("no grammar")+)++// syntaxWait bounds a parse. The helper parses a large file in tens of+// milliseconds.+const syntaxWait = 5 * time.Second++func syntaxHelper() (string, error) {+	name := progName + "-syntax"+	if exe, err := os.Executable(); err == nil {+		p := filepath.Join(filepath.Dir(exe), name)+		if _, err := os.Stat(p); err == nil {+			return p, nil+		}+	}+	p, err := exec.LookPath(name)+	if err != nil {+		return "", errNoHelper+	}+	return p, nil+}++// parseSyntax runs the helper on src, which name's extension picks the+// grammar for.+func parseSyntax(ctx context.Context, name string, src []byte) ([]synNode, error) {+	helper, err := syntaxHelper()+	if err != nil {+		return nil, err+	}+	ctx, cancel := context.WithTimeout(ctx, syntaxWait)+	defer cancel()+	cmd := exec.CommandContext(ctx, helper, filepath.Base(name))+	cmd.Stdin = bytes.NewReader(src)+	var out, stderr bytes.Buffer+	cmd.Stdout, cmd.Stderr = &out, &stderr+	if err := cmd.Run(); err != nil {+		var ee *exec.ExitError+		if errors.As(err, &ee) && syscall.Errno(ee.ExitCode()) == syscall.EOPNOTSUPP {+			return nil, errNoGrammar+		}+		if msg := strings.TrimSpace(stderr.String()); msg != "" {+			return nil, errors.New(msg)+		}+		return nil, err+	}+	var nodes []synNode+	for line := range strings.Lines(out.String()) {+		f := strings.Split(strings.TrimSuffix(line, "\n"), "\t")+		if len(f) != 7 {+			return nil, errors.New("the helper printed a line of " + strconv.Itoa(len(f)) + " fields")+		}+		var v [5]int+		for i, s := range []string{f[0], f[3], f[4], f[5], f[6]} {+			if v[i], err = strconv.Atoi(s); err != nil {+				return nil, errors.New("the helper printed a bad number: " + s)+			}+		}+		nodes = append(nodes, synNode{depth: v[0], kind: f[1], field: f[2], sr: v[1], sc: v[2], er: v[3], ec: v[4]})+	}+	return nodes, nil+}++// syntaxCache keeps rendered trees by file name and content hash, as+// codeCache keeps code pages, since a parse costs a process start.+var syntaxCache = newCache(8 << 20)++// syntaxPage renders the syntax tree of src as nested folds: a node with+// children folds them, and the root starts open, so the top-level+// declarations show. Each row names the node's kind, its field in its+// parent, its text, cut to one line, and links its lines on the code page+// at href.+func syntaxPage(ctx context.Context, name, href string, src []byte) (string, error) {+	sum := sha256.Sum256(src)+	key := filepath.Ext(name) + "\x00" + href + "\x00" + string(sum[:])+	if h, ok := syntaxCache.get(key); ok {+		return h.(string), nil+	}+	nodes, err := parseSyntax(ctx, name, src)+	if err != nil {+		return "", err+	}+	lines := bytes.Split(src, []byte("\n"))+	esc := template.HTMLEscapeString+	var b strings.Builder+	b.WriteString("<div class=\"syntax\">\n")+	open := 0 // folds left open+	for i, n := range nodes {+		for ; open > n.depth; open-- {+			b.WriteString("</details>\n")+		}+		er := n.er+		// A node that ends at the start of a row ends on the row before.+		if n.ec == 0 && er > n.sr {+			er--+		}+		span := "L" + strconv.Itoa(n.sr+1)+		if er > n.sr {+			span += "-L" + strconv.Itoa(er+1)+		}+		var row strings.Builder+		if n.field != "" {+			row.WriteString("<span class=\"field\">" + esc(n.field) + ":</span> ")+		}+		row.WriteString("<span class=\"kind\">" + esc(n.kind) + "</span>")+		if text := excerpt(lines, n); text != "" {+			row.WriteString(" <code>" + esc(text) + "</code>")+		}+		row.WriteString(" <a href=\"" + esc(href+"#"+span) + "\">" + esc(strings.ReplaceAll(span[1:], "-L", "–")) + "</a>")+		if i+1 < len(nodes) && nodes[i+1].depth > n.depth {+			if n.depth == 0 {+				b.WriteString("<details open>")+			} else {+				b.WriteString("<details>")+			}+			b.WriteString("<summary>" + row.String() + "</summary>\n")+			open+++		} else {+			b.WriteString("<div>" + row.String() + "</div>\n")+		}+	}+	for ; open > 0; open-- {+		b.WriteString("</details>\n")+	}+	b.WriteString("</div>\n")+	h := b.String()+	syntaxCache.put(key, h, int64(len(h)))+	return h, nil+}++// excerptLen is how many characters of a node's text a row shows.+const excerptLen = 60++// excerpt is n's text on its first row, with runs of spaces and tabs as+// one space, cut to excerptLen characters; a node that goes on past its+// first row ends in an ellipsis.+func excerpt(lines [][]byte, n synNode) string {+	if n.sr >= len(lines) {+		return ""+	}+	line := lines[n.sr]+	start, end := min(n.sc, len(line)), len(line)+	more := n.er > n.sr && !(n.er == n.sr+1 && n.ec == 0)+	if n.er == n.sr {+		end = min(n.ec, len(line))+	}+	text := strings.Join(strings.Fields(string(line[start:end])), " ")+	if utf8.RuneCountInString(text) > excerptLen {+		text = string([]rune(text)[:excerptLen])+		more = true+	}+	if more && text != "" {+		text += " …"+	}+	return text+}++// codeViews are the links over a code page's top right corner: the views+// other than the one shown, then the raw file.+func codeViews(shown string) string {+	var b strings.Builder+	b.WriteString(`<div class="view">`)+	for _, v := range []struct{ name, href string }{{"flat", "?"}, {"scopes", "?scopes"}, {"tree", "?tree"}} {+		if v.name != shown {+			b.WriteString(`<a href="` + v.href + `">` + v.name + `</a>`)+		}+	}+	b.WriteString(`<a href="?raw">raw</a></div>`)+	return b.String()+}++// treeNote says why a tree page shows the flat code instead.+func treeNote(name string, err error) string {+	esc := template.HTMLEscapeString+	switch {+	case errors.Is(err, errNoHelper):+		return "<p class=\"own warn\">The syntax tree needs " + esc(progName) + "-syntax, which is not installed. <code>make syntax</code> builds it beside " + esc(progName) + ".</p>\n"+	case errors.Is(err, errNoGrammar):+		return "<p class=\"own\">No grammar for " + esc(filepath.Base(name)) + ", so no syntax tree.</p>\n"+	}+	return "<p class=\"own warn\">No syntax tree: " + esc(err.Error()) + "</p>\n"+}++// pageHref is the code page of urlPath, with no query, for a tree's line+// links.+func pageHref(urlPath string) string {+	return (&url.URL{Path: urlPath}).String()+}

syntax/go.mod

new file mode 100644@@ -0,0 +1,23 @@+module syntax++go 1.25++require (+	github.com/tree-sitter/go-tree-sitter v0.25.0+	github.com/tree-sitter/tree-sitter-bash v0.25.1+	github.com/tree-sitter/tree-sitter-c v0.24.2+	github.com/tree-sitter/tree-sitter-c-sharp v0.23.5+	github.com/tree-sitter/tree-sitter-cpp v0.23.4+	github.com/tree-sitter/tree-sitter-css v0.25.0+	github.com/tree-sitter/tree-sitter-go v0.25.0+	github.com/tree-sitter/tree-sitter-html v0.23.2+	github.com/tree-sitter/tree-sitter-java v0.23.5+	github.com/tree-sitter/tree-sitter-javascript v0.25.0+	github.com/tree-sitter/tree-sitter-json v0.24.8+	github.com/tree-sitter/tree-sitter-python v0.25.0+	github.com/tree-sitter/tree-sitter-ruby v0.23.1+	github.com/tree-sitter/tree-sitter-rust v0.24.2+	github.com/tree-sitter/tree-sitter-typescript v0.23.2+)++require github.com/mattn/go-pointer v0.0.1 // indirect

syntax/go.sum

new file mode 100644@@ -0,0 +1,44 @@+github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=+github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=+github.com/mattn/go-pointer v0.0.1 h1:n+XhsuGeVO6MEAp7xyEukFINEa+Quek5psIR/ylA6o0=+github.com/mattn/go-pointer v0.0.1/go.mod h1:2zXcozF6qYGgmsG+SeTZz3oAbFLdD3OWqnUbNvJZAlc=+github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=+github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=+github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=+github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=+github.com/tree-sitter/go-tree-sitter v0.25.0 h1:sx6kcg8raRFCvc9BnXglke6axya12krCJF5xJ2sftRU=+github.com/tree-sitter/go-tree-sitter v0.25.0/go.mod h1:r77ig7BikoZhHrrsjAnv8RqGti5rtSyvDHPzgTPsUuU=+github.com/tree-sitter/tree-sitter-bash v0.25.1 h1:ZD3MK4oDB5lAsFztqbdcyYEd24pxDtx3g9UOWA062rE=+github.com/tree-sitter/tree-sitter-bash v0.25.1/go.mod h1:AksQ6zE+sP9hnp7mKTMT7Q+CwpthV7VGQLXvweVXz9U=+github.com/tree-sitter/tree-sitter-c v0.24.2 h1:nW+M6BnPUa/fBwks8nqf1NiVvu7nltaC+5bR/lTtJCs=+github.com/tree-sitter/tree-sitter-c v0.24.2/go.mod h1:/SpJlv2BuiCgFA5xvtgukFGi51WxctByPUGDxPl60fc=+github.com/tree-sitter/tree-sitter-c-sharp v0.23.5 h1:EeUc2WJE5G1pD6YEqera2yVHYeroeR+/MakKX2a+0xQ=+github.com/tree-sitter/tree-sitter-c-sharp v0.23.5/go.mod h1:H7/aFm5vR1A8Yn5VIOfLWPdlKuJsMgZ5eDmaJdv8bY0=+github.com/tree-sitter/tree-sitter-cpp v0.23.4 h1:LaWZsiqQKvR65yHgKmnaqA+uz6tlDJTJFCyFIeZU/8w=+github.com/tree-sitter/tree-sitter-cpp v0.23.4/go.mod h1:doqNW64BriC7WBCQ1klf0KmJpdEvfxyXtoEybnBo6v8=+github.com/tree-sitter/tree-sitter-css v0.25.0 h1:S5NbzhdZ5LE5V474wmdg+7NthmLjIg5v4wbyewMpziw=+github.com/tree-sitter/tree-sitter-css v0.25.0/go.mod h1:0Z46XCb3L16nVOVw0Lhb43pzloUG/4T6E/pAOE62fEw=+github.com/tree-sitter/tree-sitter-embedded-template v0.23.2 h1:nFkkH6Sbe56EXLmZBqHHcamTpmz3TId97I16EnGy4rg=+github.com/tree-sitter/tree-sitter-embedded-template v0.23.2/go.mod h1:HNPOhN0qF3hWluYLdxWs5WbzP/iE4aaRVPMsdxuzIaQ=+github.com/tree-sitter/tree-sitter-go v0.25.0 h1:cEB0Q3LHgZtS+ECHx9wcP7AwzoOddJFQCVmytX42cVU=+github.com/tree-sitter/tree-sitter-go v0.25.0/go.mod h1:Jrx8QqYN0v7npv1fJRH1AznddllYiCMUChtVjxPK040=+github.com/tree-sitter/tree-sitter-html v0.23.2 h1:1UYDV+Yd05GGRhVnTcbP58GkKLSHHZwVaN+lBZV11Lc=+github.com/tree-sitter/tree-sitter-html v0.23.2/go.mod h1:gpUv/dG3Xl/eebqgeYeFMt+JLOY9cgFinb/Nw08a9og=+github.com/tree-sitter/tree-sitter-java v0.23.5 h1:J9YeMGMwXYlKSP3K4Us8CitC6hjtMjqpeOf2GGo6tig=+github.com/tree-sitter/tree-sitter-java v0.23.5/go.mod h1:NRKlI8+EznxA7t1Yt3xtraPk1Wzqh3GAIC46wxvc320=+github.com/tree-sitter/tree-sitter-javascript v0.25.0 h1:ZkWETb66/w8cc13yhfnNuHOLDQWl3BnKlH6f9AdR88c=+github.com/tree-sitter/tree-sitter-javascript v0.25.0/go.mod h1:lmGD1EJdCA+v0S1u2fFgepMg/opzSg/4pgFym2FPGAs=+github.com/tree-sitter/tree-sitter-json v0.24.8 h1:tV5rMkihgtiOe14a9LHfDY5kzTl5GNUYe6carZBn0fQ=+github.com/tree-sitter/tree-sitter-json v0.24.8/go.mod h1:F351KK0KGvCaYbZ5zxwx/gWWvZhIDl0eMtn+1r+gQbo=+github.com/tree-sitter/tree-sitter-php v0.23.11 h1:iHewsLNDmznh8kgGyfWfujsZxIz1YGbSd2ZTEM0ZiP8=+github.com/tree-sitter/tree-sitter-php v0.23.11/go.mod h1:T/kbfi+UcCywQfUNAJnGTN/fMSUjnwPXA8k4yoIks74=+github.com/tree-sitter/tree-sitter-python v0.25.0 h1:O6XD9v8U1LOcRc3cNj9nM7XufrtEBezE6VrpRrHZDf0=+github.com/tree-sitter/tree-sitter-python v0.25.0/go.mod h1:cpdthSy/Yoa28aJFBscFHlGiU+cnSiSh1kuDVtI8YeM=+github.com/tree-sitter/tree-sitter-ruby v0.23.1 h1:T/NKHUA+iVbHM440hFx+lzVOzS4dV6z8Qw8ai+72bYo=+github.com/tree-sitter/tree-sitter-ruby v0.23.1/go.mod h1:kUS4kCCQloFcdX6sdpr8p6r2rogbM6ZjTox5ZOQy8cA=+github.com/tree-sitter/tree-sitter-rust v0.24.2 h1:NL4nF67ib21RMzzfvkmXlVwe45vvhW10DVyO+D0z/W0=+github.com/tree-sitter/tree-sitter-rust v0.24.2/go.mod h1:hfeGWic9BAfgTrc7Xf6FaOAguCFJRo3RBbs7QJ6D7MI=+github.com/tree-sitter/tree-sitter-typescript v0.23.2 h1:/Odvphn18PniVixb9e97X0DbNVsU6Qocv9mfkyzdXwU=+github.com/tree-sitter/tree-sitter-typescript v0.23.2/go.mod h1:zjzMXT/Ulffel2xfOcAkQQkiAkmgnbtPGlFQw/5X4xA=+gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=+gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

syntax/main.go

new file mode 100644@@ -0,0 +1,158 @@+// Command syntax is the program's tree-sitter helper. It parses the source+// on stdin by the grammar its file name picks and prints the named nodes+// of the syntax tree in preorder, one per line, tab-separated: depth,+// kind, field name, start row, start column, end row, end column. Rows+// and columns count from 0, columns in bytes. A missing node, which the+// parser inserted to recover from an error, prints its kind after+// "MISSING ".+//+// It is a program of its own because tree-sitter is C, and cgo would make+// the main binary need a C compiler to build and a C library to run. The+// main binary runs it for each tree page and does without it when it is+// absent. The binary takes its name from the Makefile, eg www-syntax.+//+// Usage: <name>-syntax FILE < source+package main++import (+	"bufio"+	"fmt"+	"io"+	"os"+	"path/filepath"+	"strconv"+	"strings"+	"syscall"+	"unsafe"++	sitter "github.com/tree-sitter/go-tree-sitter"+	bash "github.com/tree-sitter/tree-sitter-bash/bindings/go"+	csharp "github.com/tree-sitter/tree-sitter-c-sharp/bindings/go"+	c "github.com/tree-sitter/tree-sitter-c/bindings/go"+	cpp "github.com/tree-sitter/tree-sitter-cpp/bindings/go"+	css "github.com/tree-sitter/tree-sitter-css/bindings/go"+	golang "github.com/tree-sitter/tree-sitter-go/bindings/go"+	html "github.com/tree-sitter/tree-sitter-html/bindings/go"+	java "github.com/tree-sitter/tree-sitter-java/bindings/go"+	javascript "github.com/tree-sitter/tree-sitter-javascript/bindings/go"+	json "github.com/tree-sitter/tree-sitter-json/bindings/go"+	python "github.com/tree-sitter/tree-sitter-python/bindings/go"+	ruby "github.com/tree-sitter/tree-sitter-ruby/bindings/go"+	rust "github.com/tree-sitter/tree-sitter-rust/bindings/go"+	typescript "github.com/tree-sitter/tree-sitter-typescript/bindings/go"+)++// grammars maps a file extension to its grammar. A .h file is read as C,+// since C++ headers mostly parse as C and C headers often fail as C++.+var grammars = map[string]func() unsafe.Pointer{+	".go":   golang.Language,+	".py":   python.Language,+	".pyi":  python.Language,+	".js":   javascript.Language,+	".mjs":  javascript.Language,+	".cjs":  javascript.Language,+	".jsx":  javascript.Language,+	".ts":   typescript.LanguageTypescript,+	".mts":  typescript.LanguageTypescript,+	".cts":  typescript.LanguageTypescript,+	".tsx":  typescript.LanguageTSX,+	".rs":   rust.Language,+	".c":    c.Language,+	".h":    c.Language,+	".cc":   cpp.Language,+	".cpp":  cpp.Language,+	".cxx":  cpp.Language,+	".hh":   cpp.Language,+	".hpp":  cpp.Language,+	".hxx":  cpp.Language,+	".sh":   bash.Language,+	".bash": bash.Language,+	".java": java.Language,+	".cs":   csharp.Language,+	".json": json.Language,+	".html": html.Language,+	".htm":  html.Language,+	".css":  css.Language,+	".rb":   ruby.Language,+}++func main() {+	if len(os.Args) != 2 {+		os.Exit(fail(syscall.EINVAL, "usage: "+filepath.Base(os.Args[0])+" FILE < source"))+	}+	lang, ok := grammars[strings.ToLower(filepath.Ext(os.Args[1]))]+	if !ok {+		os.Exit(fail(syscall.EOPNOTSUPP, "no grammar for "+filepath.Base(os.Args[1])))+	}+	src, err := io.ReadAll(os.Stdin)+	if err != nil {+		os.Exit(fail(syscall.EIO, "cannot read the source: "+err.Error()))+	}+	p := sitter.NewParser()+	defer p.Close()+	if err := p.SetLanguage(sitter.NewLanguage(lang())); err != nil {+		os.Exit(fail(syscall.EINVAL, "cannot load the grammar: "+err.Error()))+	}+	tree := p.Parse(src, nil)+	if tree == nil {+		os.Exit(fail(syscall.EIO, "the parser gave no tree"))+	}+	defer tree.Close()+	w := bufio.NewWriter(os.Stdout)+	write(w, tree)+	if err := w.Flush(); err != nil {+		os.Exit(fail(syscall.EIO, "cannot write the tree: "+err.Error()))+	}+}++// write prints the named nodes in preorder. Anonymous nodes, eg+// punctuation and keywords, are left out, but their named descendants+// keep their place under the nearest named ancestor.+func write(w *bufio.Writer, tree *sitter.Tree) {+	cur := tree.Walk()+	defer cur.Close()+	depth := 0+	named := []int{} // the depths of the named nodes above the cursor+	for {+		n := cur.Node()+		if n.IsNamed() || n.IsMissing() {+			kind := n.Kind()+			if n.IsMissing() {+				kind = "MISSING " + kind+			}+			s, e := n.StartPosition(), n.EndPosition()+			w.WriteString(strconv.Itoa(len(named)))+			for _, f := range []string{kind, cur.FieldName()} {+				w.WriteByte('\t')+				w.WriteString(f)+			}+			for _, v := range []uint{s.Row, s.Column, e.Row, e.Column} {+				w.WriteByte('\t')+				w.WriteString(strconv.FormatUint(uint64(v), 10))+			}+			w.WriteByte('\n')+		}+		if cur.GotoFirstChild() {+			if n.IsNamed() || n.IsMissing() {+				named = append(named, depth)+			}+			depth+++			continue+		}+		for !cur.GotoNextSibling() {+			if !cur.GotoParent() {+				return+			}+			depth--+			if len(named) > 0 && named[len(named)-1] == depth {+				named = named[:len(named)-1]+			}+		}+	}+}++// fail prints an error as E{code}: message and returns the code.+func fail(code syscall.Errno, msg string) int {+	fmt.Fprintf(os.Stderr, "E%d: %s\n", int(code), msg)+	return int(code)+}

syntax_test.go

new file mode 100644@@ -0,0 +1,77 @@+package main++import (+	"context"+	"errors"+	"os"+	"path/filepath"+	"strings"+	"testing"+)++// fakeSyntax puts a helper on PATH that prints a fixed tree for .go files+// and refuses the rest, as the real one does without a grammar.+func fakeSyntax(t *testing.T) {+	dir := t.TempDir()+	script := "#!/bin/sh\ncase \"$1\" in *.go) ;; *) echo \"E95: no grammar for $1\" >&2; exit 95;; esac\ncat >/dev/null\n" ++		"printf '0\\tsource_file\\t\\t0\\t0\\t3\\t0\\n1\\tfunction_declaration\\t\\t0\\t0\\t2\\t1\\n2\\tidentifier\\tname\\t0\\t5\\t0\\t6\\n'\n"+	if err := os.WriteFile(filepath.Join(dir, progName+"-syntax"), []byte(script), 0o755); err != nil {+		t.Fatal(err)+	}+	t.Setenv("PATH", dir)+}++// A tree page folds each node with children, opens the root, and gives+// each row its field, kind, text and lines.+func TestSyntaxPage(t *testing.T) {+	fakeSyntax(t)+	src := []byte("func f() {\n\treturn\n}\n")+	h, err := syntaxPage(context.Background(), "a.go", "/t/a.go", src)+	if err != nil {+		t.Fatal(err)+	}+	for _, want := range []string{+		`<details open><summary><span class="kind">source_file</span> <code>func f() { …</code> <a href="/t/a.go#L1-L3">1–3</a></summary>`,+		`<details><summary><span class="kind">function_declaration</span> <code>func f() { …</code> <a href="/t/a.go#L1-L3">1–3</a></summary>`,+		`<div><span class="field">name:</span> <span class="kind">identifier</span> <code>f</code> <a href="/t/a.go#L1">1</a></div>`,+	} {+		if !strings.Contains(h, want) {+			t.Errorf("missing %s in\n%s", want, h)+		}+	}+	if strings.Count(h, "<details") != strings.Count(h, "</details>") {+		t.Errorf("folds not closed:\n%s", h)+	}+}++// Without the helper, or without a grammar, a tree page says so.+func TestSyntaxMissing(t *testing.T) {+	fakeSyntax(t)+	if _, err := parseSyntax(context.Background(), "x.txt", nil); !errors.Is(err, errNoGrammar) {+		t.Errorf("no grammar: %v", err)+	}+	t.Setenv("PATH", t.TempDir())+	if _, err := parseSyntax(context.Background(), "b.go", nil); !errors.Is(err, errNoHelper) {+		t.Errorf("no helper: %v", err)+	}+}++// An excerpt keeps a node's first row, its spaces folded, and marks text+// cut or continued.+func TestExcerpt(t *testing.T) {+	lines := [][]byte{[]byte("x := f(a,\t  b)"), []byte("y"), []byte(strings.Repeat("z", 80))}+	for _, tc := range []struct {+		n    synNode+		want string+	}{+		{synNode{sr: 0, sc: 5, er: 0, ec: 14}, "f(a, b)"},+		{synNode{sr: 0, sc: 0, er: 1, ec: 1}, "x := f(a, b) …"},+		{synNode{sr: 0, sc: 0, er: 1, ec: 0}, "x := f(a, b)"},+		{synNode{sr: 2, sc: 0, er: 2, ec: 80}, strings.Repeat("z", excerptLen) + " …"},+		{synNode{sr: 5}, ""},+	} {+		if got := excerpt(lines, tc.n); got != tc.want {+			t.Errorf("%+v: %q, want %q", tc.n, got, tc.want)+		}+	}+}