Characters that killed a render, what may sit next to a klammer, and ref/

A character written in the ^...^ mnemonic form -- ^oe^, ^zz^, ^hello^ --
used to kill ktext outright: no message, no output, and an exit status
that looked like a crash because it was one.  Only the forms that happen
to spell hexadecimal digits escaped, and they were right by luck rather
than by reading.  Any caret form that names no character is now a located
error that quotes what was written.

A klammer may now sit inside square brackets and braces as it already
could inside parentheses, and may be followed by the punctuation that
ends a sentence: [@i word@], {@b word@}, @i word@'s.  The rule is a fixed
list of characters, because none of them can occur inside a klammer name
and so none of them is ambiguous next to a delimiter.  A letter still
cannot: "x@" is the named closing delimiter, so a letter before "@" has
two readings and no rule can separate them.

The abbreviations an author writes -- the hyphen form @i-word, a klammer
glued to its neighbouring punctuation -- are now recognised as author
conveniences only.  They are not applied to @@ definitions or @@@ system
commands, which are written by someone who needs no such shorthand and
whose forms are already whole.

Also fixed, each of them silent: an exported shell function in the
environment broke every @eval in the document; a line break written with
@nl was destroyed when it passed through a klammer that generates a
table, printing as \textbackslash{}newline in the finished PDF; and
"kdesc --coverage" shortened its file column by comparing the spelling of
a path rather than the path, so a klammer set reached by a symbolic link
showed a full path where its neighbours showed a short one.

New in the distribution: ref/, a small klammerset that describes the
Klammermachine's own tables in Klammertext.  It currently provides the
character tables -- the diacritics and the extended Latin letters -- as
klammers that render to any target, so the documentation of what can be
written cannot drift from what the machine accepts.  It is early, and
will grow.

The Klammertext taxonomy now has a section on targets, describes the
@@@klammerset system command alongside the other three, and states the
precedence of "^" -- what a caret means before a letter, a digit, a
punctuation character, and a quote, in one ordered list.

(from dev 6594c7ff8c1f)

mac/katomize.h

@@ -0,0 +1,94 @@
+#pragma once
+
+#include "katom.h"
+#include "katom_rewrite_rules.h"
+
+// KATOMIZATION: text becomes katoms, by dividing it with the katom rewrite
+// rules in katom_rewrite_rules.{h,cpp}.  Live since 2026-09-02, when it
+// replaced make_katoms_from_word(), split_into_katoms() and katomize() in
+// mac/katom.cpp -- dbg/katomize_compare.sh had shown the two identical over
+// every tracked .k and .kt first, and the katoms of all 186 files are
+// byte-identical across the change.
+//
+// THE SEPARATION.  mac/katom.cpp used to hold two unrelated subjects.  One is the KATOM: what it is, what its type means,
+// how a list of them is trimmed, split on bars, turned back into a string.
+// The other is KATOMIZATION: how a stream of characters becomes that list.
+// A katom is a data structure the whole engine uses; katomization is one
+// algorithm, run once per file, that nothing else calls into.  Splitting
+// them means the rest of the engine can include katom.h without also
+// getting the katomizer, and the katomizer can be replaced -- as this file is doing
+// -- without touching the type everything depends on.
+//
+// Done 2026-09-02: make_katoms_from_word, split_into_katoms, katomize,
+// expand_compound_katom and warn_unparsed_katoms left katom.cpp for this
+// file, and katom.cpp keeps Katom itself, the predicates, and
+// trim/bar_split/as_string.  warn_unparsed_katoms came too, because it is
+// the only reader of m_unparsed, which only katomization writes -- what
+// makes it LOOK like a later pass is its call site, and timing is not
+// ownership.  line_split() followed on 2026-09-02, to file.{h,cpp}: it
+// produces no katom and one overload opens a file, so it failed this test
+// in both directions.
+
+// Make a katom from matched source text.  The one place that knows how a
+// construct is WRITTEN differs from what it MEANS: a quoted special is
+// written "^@" and means "@", so the caret comes off here.  Both katomizers
+// go through this, and nothing else constructs a katom whose type was
+// decided by matching source.
+//
+// Note what is NOT lost: m_src keeps the text exactly as written, so the
+// caret is recoverable.  hide_special_katoms() replaces m_text with a KTESC
+// marker shortly afterwards, precisely so the bare character cannot be
+// re-read as syntax; where that pass is skipped -- inside a definition or
+// an @eval/@read/@cond span -- the bare character is what the code or the
+// declaration should receive.
+Katom source_katom(const std::string& text, katom_t type, const Locator& loc);
+
+// Katomize one piece of text containing no whitespace.  The replacement for
+// make_katoms_from_word().
+//
+// The recursion needs no name for what it divides: at every level the thing
+// being katomized is TEXT, and it stops when the text IS a katom.  (The
+// original called it a "word", which collides with katom_t::word, a katom
+// TYPE meaning letters only -- two referents, one term, in one file.)
+//
+// Same shape as the original: text matching a katom type IS that katom;
+// otherwise the first katom rewrite rule that usefully rewrites it decides,
+// and the pieces of that rewrite are katomized in turn; text no rule can
+// divide becomes an unparsed katom, which warn_unparsed_katoms() reports
+// later.
+katom_list katomize_by_rewriting(const std::string& text, const std::string& source, int line, int chr,
+                                 rewrite_order order = rewrite_order::by_category);
+
+// One line (or any string) to katoms.  The replacement for
+// split_into_katoms(), including its \r removal and tab expansion.
+katom_list katomize_text(const std::string& text, const std::string& source, int line,
+                         rewrite_order order = rewrite_order::by_category);
+
+// The same, katomizing by MATCHING instead of by rewriting -- the second
+// entry point, so that naming a katomization method stays this file's job.
+// Its component lives in katom_matching.{h,cpp}; the desugaring of the
+// abbreviated application is a separate pass there, because matching
+// produces the abbreviated form as one katom and something must expand it.
+katom_list katomize_text_by_matching(const std::string& text, const std::string& source, int line);
+
+// A file's lines to katoms.  The replacement for katomize(): each line is
+// katomized with its newline restored, and the final newline is dropped.
+katom_list katomize_lines(const strings_t& lines, const std::string& source,
+                          rewrite_order order = rewrite_order::by_category);
+
+// "kdiag --rewrite": report each rule as it fires, with its category.  It
+// now names the rule that ACTUALLY fired, which the old path could not do
+// for the abbreviated application -- that was applied inside
+// expand_compound_katom() and attributed to whichever rule the loop
+// happened to be holding, so it always read as "bar precedence".
+inline bool show_rewrite_rules = false;
+
+// Warn about text that matched no katom type and that no rule could divide
+// -- but only where it survives processing: text removed by #, ##, or
+// #[...]#, replaced spans, and literal content never warn.  Called at the
+// end of Machine::process_katoms(), after those passes have marked the
+// katoms.  Clears the flag afterwards so reprocessing the same list does
+// not warn twice.  With warn=false (the @eval read-back sub-Machine, whose
+// katoms hold machine-generated result text) nothing is printed and every
+// flag is cleared, so spliced result katoms stay silent.
+void warn_unparsed_katoms(katom_list& katoms, bool warn = true);