Characters that killed a render, what may sit next to a klammer, and ref/

A character written in the ^...^ mnemonic form -- ^oe^, ^zz^, ^hello^ --
used to kill ktext outright: no message, no output, and an exit status
that looked like a crash because it was one.  Only the forms that happen
to spell hexadecimal digits escaped, and they were right by luck rather
than by reading.  Any caret form that names no character is now a located
error that quotes what was written.

A klammer may now sit inside square brackets and braces as it already
could inside parentheses, and may be followed by the punctuation that
ends a sentence: [@i word@], {@b word@}, @i word@'s.  The rule is a fixed
list of characters, because none of them can occur inside a klammer name
and so none of them is ambiguous next to a delimiter.  A letter still
cannot: "x@" is the named closing delimiter, so a letter before "@" has
two readings and no rule can separate them.

The abbreviations an author writes -- the hyphen form @i-word, a klammer
glued to its neighbouring punctuation -- are now recognised as author
conveniences only.  They are not applied to @@ definitions or @@@ system
commands, which are written by someone who needs no such shorthand and
whose forms are already whole.

Also fixed, each of them silent: an exported shell function in the
environment broke every @eval in the document; a line break written with
@nl was destroyed when it passed through a klammer that generates a
table, printing as \textbackslash{}newline in the finished PDF; and
"kdesc --coverage" shortened its file column by comparing the spelling of
a path rather than the path, so a klammer set reached by a symbolic link
showed a full path where its neighbours showed a short one.

New in the distribution: ref/, a small klammerset that describes the
Klammermachine's own tables in Klammertext.  It currently provides the
character tables -- the diacritics and the extended Latin letters -- as
klammers that render to any target, so the documentation of what can be
written cannot drift from what the machine accepts.  It is early, and
will grow.

The Klammertext taxonomy now has a section on targets, describes the
@@@klammerset system command alongside the other three, and states the
precedence of "^" -- what a caret means before a letter, a digit, a
punctuation character, and a quote, in one ordered list.

(from dev 6594c7ff8c1f)

sks/target/phases.py

@@ -14,6 +14,50 @@ def txt_nl_marker():
 def txt_vspace_marker():
     return "__KTEXTVSPACE__"
 
+# A space that survives to the output.  "~" used to serve, and could not:
+# it is also a character a writer may want, and the substitution at the end
+# of txt_justify_blocks turned EVERY tilde into a space -- so a literal
+# tilde could not be written in the txt target at all (a "^~" resolves
+# before the phases run, and by then is indistinguishable).  A marker of
+# its own frees the character.  SYNC: the txt target's "~ ^0020^" transform
+# in sks/target/target.k is the WRITER-facing convention (a tilde means a
+# non-breaking space, as in LaTeX) and is deliberately unchanged.
+def txt_nbsp_marker():
+    return "__KTEXTNBSP__"
+
+# The tex line break, carried as a marker.
+#
+# @nl.tex used to emit "\\newline" directly, and that is destroyed the moment
+# it passes through a GENERATOR.  A klammer whose @eval result still holds
+# klammers is generator output, so the machine escapes its text before
+# applying them -- which is what lets a writer put a "%" in generated data
+# without knowing LaTeX.  A backslash meets the same rule and becomes
+# "\\textbackslash{}".  So a row written
+#
+#     @offer_list  work done @nl@ (delivery date) | 10 ||  ...
+#
+# resolved @nl to "\\newline" while binding the rest argument, handed that to
+# Python as ordinary text, and had it neutralised on the way back:
+# "\\textbackslash{}newline" reached the .tex and printed as literal text.
+#
+# The marker has no backslash, so the escape pass has nothing to act on, and
+# tex_resolve_markers puts the real command back at the very end.
+#
+# LETTERS AND DIGITS ONLY -- deliberately unlike the txt markers above.  The
+# tex target escapes "_" to "\\_", so a "__KTEXT...__" marker would arrive as
+# "\\_\\_KTEXT...\\_\\_" and never match.  Nothing in this marker is in tex's
+# :escape list.
+def tex_nl_marker():
+    return "KTEXNEWLINEMARK"
+
+# Declared as an :after_apply phase on BOTH tex and pdf.  ":includes" carries
+# escapes, transforms and resolves from tex into pdf but NOT phases
+# (target_registry.cpp), and pdf's own phase compiles the document, so this
+# has to run before it -- hence the explicit entry in each rather than
+# inheritance.
+def tex_resolve_markers(text, K):
+    return text.replace(tex_nl_marker(), "\\newline")
+
 def txt_justify_blocks(text, K):
     delim = '__DIVIDE__'
     text = re.sub("\n\n+", delim, text)
@@ -33,27 +77,28 @@ def txt_justify_blocks(text, K):
         # and the one a klammer relies on to emit lines that must not
         # re-flow -- a @table's rows, a listing.
         #
-        # "~" counts as that whitespace, and MUST: a klammer's indent cannot
-        # be a real space, because Eval::eval TRIMS the result katoms, so
-        # the leading spaces a klammer emits never reach this pass at all
-        # (the "~" is the non-breaking space marker, resolved to a space at
-        # the end of this function -- the only space that survives).  A
-        # writer's own "~" has become a real space long before this, in the
-        # target's transform table, so there is no ambiguity here.
+        # The nbsp MARKER counts as that whitespace, and must: a klammer's
+        # indent cannot be a real space, because Eval::eval TRIMS the result
+        # katoms, so the leading spaces a klammer emits never reach this
+        # pass at all.  The marker is the only space that survives, and it
+        # becomes one at the end of this function.  (It was the character
+        # "~" until 2026-08-27, which meant every tilde in the document
+        # became a space and a literal tilde could not be written at all.)
         #
         # The emission below was INSIDE this "if" until 2026-08-26, so a
         # verbatim paragraph was not left alone, it was DELETED: the one
         # branch that had nothing to do also did not copy the text through.
         # An indented paragraph vanished from the output with no error.
-        if par and par[0] not in ' ~':
-            label = re.match(r"~*\[\d+\] ", par)
+        nbsp = txt_nbsp_marker()
+        if par and par[0] != ' ' and not par.startswith(nbsp):
+            label = re.match("(?:" + re.escape(nbsp) + r")*\[\d+\] ", par)
             par = "\n".join(textwrap.wrap(
                 par, width=int(K.Target_txt_width), break_long_words=False,
                 subsequent_indent=" " * len(label.group(0)) if label else ""))
         result += par + "\n\n"
     result = re.sub(" *" + txt_vspace_marker() + " *", "\n", result)
     result = re.sub(r"\n? *" + txt_nl_marker() + r" *\n?", "\n", result)
-    result = re.sub("~", " ", result)
+    result = result.replace(txt_nbsp_marker(), " ")
     return result
 
 def txt_insert_captions(text, K):
@@ -81,11 +126,11 @@ def txt_insert_captions(text, K):
             caption = textwrap.wrap(
                 caption, indented_width,
                 #initial_indent ="~"*indent,
-                subsequent_indent="~"*indent)
+                subsequent_indent=txt_nbsp_marker()*indent)
             caption = "\n".join(caption)
             caption_width = indented_width
         line_length = max([len(e) for e in caption.split("\n")])
-        line = "~" * indent + "-" * line_length
+        line = txt_nbsp_marker() * indent + "-" * line_length
         return caption + "\n" + line
     result = caption_rgx.sub(replace, text)
     return result
@@ -112,7 +157,8 @@ def txt_format_footnotes(text, K):
         width = len(f"[{len(footnotes)}]") 
         for n, footnote in enumerate(footnotes, 1):
             label = f"[{n}]"
-            result += "~" * (width - len(label)) + label + " " + footnote + "\n\n"
+            result += (txt_nbsp_marker() * (width - len(label))
+                       + label + " " + footnote + "\n\n")
     return result
 
 def html_footnote_marker():

sks/target/target.k

@@ -42,6 +42,18 @@ of this: neither the transforms nor the quoting reach it.
 @@@
 
 # The txt target is not fully implemented in the SKS yet.
+#
+# A writer's "~" is a non-breaking space (the LaTeX convention), and it
+# becomes the nbsp MARKER here rather than a space directly, for two
+# reasons.  A transform whose replacement is whitespace is silently
+# DISCARDED -- parse_transforms takes the pair with word_split, and a
+# space is not a word -- so "~ ^0020^" never did anything, and the
+# substitution that made a tilde look like a space lived in the
+# justification phase and caught EVERY tilde, so a literal one could not
+# be written at all.  The marker survives to the phase and becomes a space
+# there; a quoted "^~" is a KTESC marker by then, so it is not matched
+# here and reaches the reader as a tilde.
+# SYNC: phases.txt_nbsp_marker() in sks/target/phases.py.
 @@@target txt | Plain text with formatting |  
   --  -  |
   --- -- |
@@ -49,7 +61,7 @@ of this: neither the transforms nor the quoting reach it.
   ''  "  |
   `   `  |
   '   '  |
-  ~   ^0020^
+  ~   __KTEXTNBSP__
   :after_apply
     phases.txt_format_footnotes ;
     phases.txt_insert_captions ;
@@ -81,11 +93,16 @@ of this: neither the transforms nor the quoting reach it.
 @@@target tex | LaTeX
   :escape \ \textbackslash{} & \& { \{ } \} $ \$ % \% _ \_ ^# \^# ^^ \textasciicircum{}
   :resolve - {-} ~ \textasciitilde{}
+  :after_apply phases.tex_resolve_markers
 @@@
 
 @@@target pdf | PDF from LaTeX
   :includes tex
+  # Phases are NOT inherited through :includes -- escapes, transforms and
+  # resolves are (target_registry.cpp) -- and tex_to_pdf compiles the
+  # document, so the marker resolution is named here too, and first.
   :after_apply
+     phases.tex_resolve_markers ;
      ^:cpp *KLAMMERTEXT_HOME*/sks/document/document tex_to_pdf
 @@@
 

sks/target/txt_util.py

@@ -6,11 +6,11 @@ hang anything on: a block is set off from the prose around it by being
 INDENTED, and by having laid itself out before the target sees it.  Two
 consequences run through this module:
 
-  * The indent is written with "~", the non-breaking space, never with a
-    real space.  Eval::eval TRIMS a klammer's result, so leading spaces
-    never survive to the output; and the txt target's justification pass
-    treats a paragraph beginning with " " or "~" as already laid out and
-    leaves it alone (phases.txt_justify_blocks).  So the same character
+  * The indent is written with the NBSP MARKER, never with a real space.
+    Eval::eval TRIMS a klammer's result, so leading spaces never survive
+    to the output; and the txt target's justification pass treats a
+    paragraph beginning with a space or that marker as already laid out
+    and leaves it alone (phases.txt_justify_blocks).  So the same token
     both survives and declares "do not re-flow this".
 
   * A block takes the indent out of its own measure.  The page has
@@ -25,11 +25,26 @@ import sys
 import textwrap
 sys.path.append(f'{os.environ.get("KLAMMERTEXT_HOME")}/sks/kutil')
 import kutil
+import klammer_base
+import phases
 
 # The non-breaking space marker, resolved to a real space at the end of
-# phases.txt_justify_blocks.  Named here rather than written as "~" at
-# each use, since it is the whole mechanism of this module.
-NBSP = "~"
+# phases.txt_justify_blocks.  Stated once there, with the other txt
+# markers, and referred to rather than spelled out here.
+NBSP = phases.txt_nbsp_marker()
+
+def visible_length(s):
+    """How many columns s will occupy once the markers in it resolve.
+
+    len() is the wrong measure for anything laid out: a KTESC marker is
+    fourteen characters and one glyph, and an nbsp marker is thirteen and
+    one space.  Measuring the unresolved text sizes a column to a width
+    the reader never sees -- and then pads to it, so the padding is wrong
+    twice.  (The same fault as abbreviate() in mac/util.cpp, which counted
+    escape sequences as visible width, and the reason escape resolution
+    runs BEFORE the txt phases.)
+    """
+    return len(klammer_base.unescape_ktesc(s).replace(NBSP, " "))
 
 def indent_text(text, indent):
     """Every line of text prefixed with `indent` columns of NBSP."""