Literal @c, @source_listing with :marker, and a large-directory speedup

Three changes.

@c now takes its content literally, like @code -- it is the inline form
and @code the block form of the same thing.  The named close "c@" is
required, and characters that are special in a target no longer break
the file: @c a_b c@ renders correctly everywhere.  The Markdown
converter stops quoting inline code, since nothing needs protecting.

@source_file is renamed @source_listing.  Code read from a file is its
own klammer; @code is only for a block written inline (its never-
implemented :filename and :pattern options are removed).  The new
:marker P option lists the region between two lines that are exactly
//P, so the source file declares its own extractable regions.  A marker
missing or not appearing exactly twice is an error, never a fallback.

Rendering a document that sits in a large directory was paying a
recursive walk of that directory's whole tree on every @eval -- 27
seconds for a document that renders in a third of one.  The walk is now
a non-recursive look decided once per directory.

Assembled from dev commit 071b1b183de4.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

README.md

@@ -51,7 +51,7 @@ are regenerated on each release — patches cannot be merged directly.
 Report problems (or send patches) to the author; accepted changes are
 applied to the development tree and appear in a following snapshot.
 
-This snapshot was assembled from development commit `6c2ff7b8fce3`.
+This snapshot was assembled from development commit `071b1b183de4`.
 
 ## License
 

doc/klammertext_for_llm.md

@@ -92,7 +92,7 @@ klammer's name immediately before it: `@i text i@.
 
 ```
 @i just text i@                      ← unnecessary; @i just text @ is better
-@s1 A @c literal c@ heading s1@      ← name it: the body contains a klammer call
+@s1 A @c fragment c@ heading s1@     ← name it: the body contains a klammer call
 ```
 
 **Arguments.**
@@ -110,17 +110,42 @@ klammer's name immediately before it: `@i text i@.
 **Abbreviated form.** `@name-arg1-arg2` is `@name arg1 | arg2 @`, valid only
 while every argument is letters and digits: `@i-italic`, `@sup-2-3`, `@date@`.
 
-**Literal klammers** take their content uninterpreted (`@c`, `@code`). Their
+**Literal klammers** take their content uninterpreted. It is not scanned for
-content is not scanned for `@`, `|`, `#`, or `^`, so the machine cannot find the
+`@`, `|`, `#`, or `^`, so the machine cannot find the end by counting and **the
-end by counting — **these must be closed with the named form**:
+named close is required**:
 
 ```
-@c ImfStandardAttributes.h c@
 @code
   int x = a[0];   // # and @ are safe in here
 code@
 ```
 
+```
+@c a | b c@      ← inline: same rule, and the close must be "c@"
+```
+
+`@c` and `@code` are the in-line and block forms of the same thing, and both are
+literal: **nothing inside either is interpreted**, so `#`, `@`, `|`, `^` and `*`
+all pass through as written and none of them needs quoting.
+
+Code that lives in a **file** is `@source_listing`, never `@code` — `@code` is
+only for a block written out inline, and has no filename argument:
+
+```
+@source_listing src/derivations.cpp @              ← the whole file
+@source_listing src/derivations.cpp :marker fromRdd55 @   ← one region of it
+```
+
+`:marker P` lists what lies between two lines that are exactly `//P`, each
+beginning in the first column. The source file thus declares its own extractable
+regions and the document asks for one by name, so editing the code cannot
+silently change what the document shows. A marker that is missing, or that does
+not appear exactly twice, is an **error** — never a silent fallback to the whole
+file.
+
+`kdesc -k <name>` is how you tell: a parameter shown as `text.literal` is a
+literal parameter. Do not infer it from a klammer's purpose — check.
+
 ---
 
 ## 4. A complete document
@@ -137,7 +162,7 @@ code@
 @s1 Principles s1@
 
 Ordinary paragraph text. A blank line starts a new paragraph.
-Use @i emphasis i@ and @b bold @ inline, and @c literal_text c@ for code.
+Use @i emphasis i@ and @b bold @ inline, and @c code_fragment c@ for code.
 
 @link https://openexr.com/ :text the OpenEXR site link@
 
@@ -172,9 +197,9 @@ Confirm parameters with `kdesc -k <name>` — this list is names and intent only
 
 | | |
 |---|---|
-| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline literal, `@sup`, `@sub` |
+| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline code (literal; close with `c@`), `@sup`, `@sub` |
 | structure | `@s1`…`@s7` numbered headings, `@h1`…`@h7` unnumbered, `@part`, `@preface`, `@par` |
-| blocks | `@code`, `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` |
+| blocks | `@code` (a listing written inline), `@source_listing` (a listing read from a file), `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` |
 | lists | `@ul` unordered, `@ol` ordered, `@define` term/definition — items separated by `\|` |
 | tables | `@table` — cells by `\|`, rows by `\|\|`; `@tbl`, `@rowcolor` |
 | figures | `@image`, `@image_grid`, `@fig`, `@reference` |

mac/machine.cpp

@@ -322,9 +322,46 @@ void Machine::mark_literal_klammer_content(katom_list& katoms)
     if (literal_names.empty()) return;
 
     // Scan for matching @name ... name@ spans.
-    // Stop at ## (ignore-rest) since everything after it will be removed.
+    //
+    // REMOVED TEXT IS SKIPPED.  This pass runs first in process_katoms(), before
+    // mark_ignored_katoms() takes out the "#" forms, and that ordering is not
+    // accidental: a literal klammer's content must be marked before anything
+    // else can interpret what is inside it, or removal would take a "#" that
+    // belongs to the literal body.  The cost is that this scan sees text the
+    // writer has removed, so a literal klammer merely NAMED in a comment --
+    //
+    //     # @image and @code share the same arguments.
+    //
+    // -- was found as an opening delimiter, went unclosed, and failed the whole
+    // file (TODO #40, diagnosed 2026-08-05).  "##" was already handled, which is
+    // half the case; the fix is to teach the scan the other three removal forms
+    // rather than to reorder the passes.
+    //
+    // The converse still holds, structurally: this skipping happens only while
+    // looking for an OPENING delimiter, and finding one jumps k past the whole
+    // span, so a "#" inside literal content is never examined here and stays
+    // content.
     for (auto k = katoms.begin(); k != katoms.end(); ++k) {
-        if (k->m_type == katom_t::ignore_rest) break;
+        if (k->m_type == katom_t::ignore_rest) break;   // "##": rest of the file
+        if (k->m_type == katom_t::ignore_line) {        // "#": rest of the line
+            while (k + 1 != katoms.end() && (k + 1)->m_type != katom_t::newline) {
+                ++k;
+            }
+            continue;
+        }
+        if (k->m_type == katom_t::ignore_begin) {       // "#[ ... ]#", nestable
+            int removed = 1;
+            while (++k != katoms.end() && removed > 0) {
+                if (k->m_type == katom_t::ignore_begin) {
+                    ++removed;
+                } else if (k->m_type == katom_t::ignore_end) {
+                    --removed;
+                }
+            }
+            if (k == katoms.end()) break;   // unterminated: the rest is removed
+            --k;                            // the loop's ++k steps past the "]#"
+            continue;
+        }
         if (k->m_type != katom_t::apply_begin) continue;
         std::string name = trim_char(k->m_text, '@');
         if (literal_names.count(name) == 0) continue;

mac/state.cpp

@@ -1,5 +1,6 @@
 #include <regex>
 #include <iostream>
+#include <map>
 
 #include "show.h"
 #include "file.h"
@@ -10,6 +11,35 @@
 
 int State::class_id = 0;
 
+namespace {
+
+// A sys.path entry reaches only the directory's own files, so the decision
+// must not recurse; and the answer is fixed for the run (the search-dir set
+// only grows, and a directory's answer does not change), so it is decided
+// once per directory per process. The former recursive walk here ran on
+// every @eval and took ~200 ms per eval on a document in a large home
+// directory (27 s for one render).
+bool directory_has_python_file(const std::string& dir)
+{
+    static std::map<std::string, bool> cache {};
+    auto it = cache.find(dir);
+    if (it != cache.end()) {
+        return it->second;
+    }
+    bool found = false;
+    std::error_code ec {};
+    for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) {
+        if (entry.is_regular_file(ec) && entry.path().extension() == ".py") {
+            found = true;
+            break;
+        }
+    }
+    cache[dir] = found;
+    return found;
+}
+
+} // namespace
+
 bool Var::defined() const
 {
     return !m_name.empty();
@@ -297,8 +327,7 @@ std::string State::python_code()
     python_dirs.insert(python_dirs.end(),
                        m_search_dirs.begin(), m_search_dirs.end());
     for (const auto& d : python_dirs) {
-        auto python_files = pathnames_with_extension(d, "py");
+        if (directory_has_python_file(d)) {
-        if (!python_files.empty()) {
             ss << "sys.path.append('" << d << "')\n";
         }
     }

sks/code/code.k

@@ -1,15 +1,27 @@
 
-@@code.k :filename :pattern
+@@code.k
   @hpos_args :hpos left @
   @caption_args :caption_side top @
   | text.literal :
-A source file displayed verbatim
+A block of code, given here and displayed uninterpreted.  Code that lives in a
+file is ^@source_listing instead; ^@code is only for a block written inline.
+
+It carried ":filename" and ":pattern" until 2026-08-16.  Neither was ever
+implemented -- "filename" is read only by ^@source_listing and "pattern" by
+nothing at all -- and because a literal parameter's content begins at the first
+katom when no bar is written, "^@code :filename f :pattern p" DISPLAYED THOSE
+WORDS as the listing rather than acting on them.
 @@
 
 @@code.html,tex :: @eval code_block.Code(K) @ @@
 
-@@c.k code_text :
+@@c.k code_text.literal :
-A word or phrase displayed verbatim in a line
+A word or phrase displayed verbatim in a line.  Like ^@code, its content is
+LITERAL -- nothing in it is interpreted as Klammertext -- so it must be closed
+with the named delimiter "c^@".  ^@c is the in-line form and ^@code the block
+form of the same thing; before 2026-08-16 this parameter was an ordinary
+string, so a "^#" or a bare "^@" inside it was read as Klammertext and usually
+failed the file, which is not what "verbatim" can mean.
 @@
 
 @@c.html,tex :: @eval code_block.Code_fragment(K) eval@
@@ -17,5 +29,19 @@ A word or phrase displayed verbatim in a line
 
 # :cwd makes the filename resolve against the DOCUMENT's directory, not
 # the directory ktext happens to run in.
-@@source_file.k filename : Display the contents of the file verbatim. @@
+@@source_listing.k filename :marker
-@@source_file.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@
+  @hpos_args :hpos left @
+  @caption_args :caption_side top @
+  : Code read from a file and displayed uninterpreted.  *filename* is the file;
+without ":marker" the whole file is listed.
+
+":marker P" lists only the region BETWEEN two lines that consist solely of
+"//P" and begin in the first column.  The source file therefore declares its
+own extractable regions and the document asks for one by name, so the two
+cannot drift apart silently: renaming or reformatting the code does not change
+what is extracted, and a region that disappears is an error rather than a
+quietly different listing.
+
+Named "^@source_file" until 2026-08-16.  # retired-ok
+@@
+@@source_listing.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@

sks/code/code_block.py

@@ -324,35 +324,89 @@ class Code_fragment(klammer_base.Klammer_base):
     def __init__(self, K):
         super().__init__(K)
 
+    # code_text is a LITERAL parameter (sks/code/code.k), so its content
+    # reaches here exactly as written and NOTHING has escaped it -- the
+    # machine's target-character pass does not touch literal content.  Both
+    # methods must therefore escape it themselves, with the same helpers the
+    # block form uses on its lines.  Before 2026-08-16 the parameter was an
+    # ordinary string and the machine did the escaping; html() got away with
+    # handling "<" by hand and tex() with nothing at all.
     def html(self):
-        #print(f"code: |{self.code_text}|")
+        return f'<span class="code">{html_line(undash(self.code_text.strip()))}</span>'
-        result = self.code_text.strip()
-        result = undash(result)
-        #result = re.escape(result)
-        result = re.sub("<", "&lt;", result)
-        result = re.sub(" ", "&nbsp;", result)
-        #print(f"code: |{self.code_text}| -> |{result}|")
-        return f'<span class="code">{result}</span>'
 
     def tex(self):
-        return f"{{\\tt {self.code_text.strip()}}}"
+        return f"{{\\tt {tex_line(self.code_text.strip())}}}"
 
 
-class Source(klammer_base.Klammer_base):
+def extract_marked_region(src, marker, filename):
+    """The region of `src` between two lines that are exactly "//<marker>".
+
+    The delimiter lines must consist SOLELY of "//" + marker and start in the
+    first column, so a marker cannot be matched inside indented code or in a
+    trailing comment.  Both delimiters are the same text: the source brackets a
+    region rather than naming a start and a separate end.
+
+    A marker that is missing, or that appears only once, is an ERROR -- the
+    document asked for a region the file does not offer, and silently listing
+    the whole file (or nothing) would let the document drift from the code it
+    claims to quote, which is the one thing this option exists to prevent.
+    """
+    delimiter = "//" + marker
+    lines = src.split("\n")
+    at = [i for i, line in enumerate(lines) if line == delimiter]
+    if len(at) < 2:
+        found = "once" if len(at) == 1 else "not at all"
+        raise Exception(
+            f'The marker "{marker}" appears {found} in "{filename}".\n'
+            f'  A marked region is bracketed by TWO lines that are exactly\n'
+            f'  "{delimiter}", each beginning in the first column.')
+    if len(at) > 2:
+        raise Exception(
+            f'The marker "{marker}" appears {len(at)} times in "{filename}"\n'
+            f'  (lines {", ".join(str(i + 1) for i in at)}); a region needs exactly two.')
+    region = lines[at[0] + 1:at[1]]
+    while region and not region[0].strip():
+        region.pop(0)
+    while region and not region[-1].strip():
+        region.pop()
+    return "\n".join(region)
+
+
+class Source(Code):
+    """@source_listing -- a Code listing whose text comes from a FILE.
+
+    It IS a Code: @source_listing and @code differ only in where the text
+    comes from, so they must render identically, and subclassing is what
+    guarantees that rather than a second implementation that drifts.
+
+    It rendered separately until 2026-08-16, and was wrong in a way nothing
+    caught: html() quoted only "@" and tex() wrapped the raw text in a
+    verbatim environment.  An @eval result is RE-READ as Klammertext, so an
+    unquoted "#" starts a text removal -- and since the html path had already
+    joined the source into one line, a file beginning "#include" produced an
+    EMPTY LISTING and exited 0.  quote_specials() in this module documents
+    exactly that hazard, and this was the one place not using it.  Inheriting
+    Code's rendering fixes both targets at once: tex_line()/html_line() quote
+    the Klammertext specials AND escape the target's own, so the author of the
+    source file needs to know about neither.
+
+    A verbatim environment could not have been made correct here, incidentally:
+    a quoted "^#" resolves to "\#" through the tex target's escape list, which
+    inside verbatim would print as "\#" rather than "#".
+    """
     def __init__(self, K):
-        super().__init__(K)
+        # Klammer_base, NOT Code: Code's constructor expands whitespace markers
+        # in self.text, and there is no "text" parameter here -- the text does
+        # not exist until the file has been read.  File content carries no
+        # whitespace markers anyway, since those come from the katomizer.
+        klammer_base.Klammer_base.__init__(self, K)
+        try:
             with open(self.filename) as fp:
-            self.src = fp.read()
+                text = fp.read()
-
+        except OSError as e:
-    def tex(self):
+            raise Exception(
-        result = self.src
+                f'Cannot read the source listing "{self.filename}": {e.strerror}.\n'
-        # result = re.sub("#", "^#", result)
+                f'  A relative name resolves against the DOCUMENT\'s directory.')
-        # result = re.sub("\\^", "\\^", result)
+        if self.marker:
-        result = f"\\begin{{verbatim}}\n{result}\n\\end{{verbatim}}\n"
+            text = extract_marked_region(text, self.marker, self.filename)
-        return result
+        self.text = text
-
-    def html(self):
-        result = escape_newlines(self.src.strip()) + "\n"
-        result = re.sub("@", "^@", result)
-        result = E("div").body(result).cls("code_text").str()
-        return result

sks/tns/md_to_sks.py

@@ -128,8 +128,17 @@ def inline(text):
     text = "".join(p if re.match(r'^[\x00\x01]', p) else quote(p) for p in parts)
 
     text = re.sub(r'\x01(\d+)\x01', lambda m: verbatim[int(m.group(1))], text)
+    # Inline code goes in RAW, and is closed with the named delimiter.  "@c"
+    # took an ordinary string parameter until 2026-08-16, so every Klammertext
+    # special inside a code span had to be quoted -- and about 40% of the
+    # inline code in the measured corpus contains one (see
+    # doc/markdown_to_klammertext.md).  It is now a LITERAL parameter, like
+    # "@code": nothing inside is interpreted, so nothing needs quoting, and the
+    # source reads as the author wrote it.  A literal klammer cannot be closed
+    # with a bare "@" -- the machine cannot find the end by counting -- so the
+    # close is "c@".
     text = re.sub(r'\x00(\d+)\x00',
-                  lambda m: OPEN + "@c " + quote(spans[int(m.group(1))]) + " @" + CLOSE, text)
+                  lambda m: OPEN + "@c " + spans[int(m.group(1))] + " c@" + CLOSE, text)
     # Klammertext delimiters need whitespace around them, but Markdown
     # emphasis abuts its neighbours: "un**bold**ed" and "*name*_rectified"
     # both put a word character hard against a delimiter, which the

tst/escape_test.sh

@@ -124,6 +124,63 @@ check_eq "17. @eval result ^@ survives read-back as @"      '@'
 # Klammertext -- a resolved @ in the result must not be re-parsed.
 check_eq "18. :after_apply phase: raw result, @ intact"     'A @ B'              --klammersets none -t u -s '@@@target u | up :after_apply string.capwords @@@ a ^@ b'
 
+# --- Literal klammers and REMOVED text (TODO #40, fixed 2026-08-16) ---
+#
+# mark_literal_klammer_content() runs FIRST in process_katoms(), before
+# mark_ignored_katoms() takes out the "#" forms.  That ordering is deliberate --
+# a literal klammer's content must be marked before anything can interpret what
+# is inside it, or removal would take a "#" belonging to the literal body -- but
+# it meant the scan saw text the writer had removed, so a literal klammer merely
+# NAMED in a comment was read as an opening delimiter, went unclosed, and failed
+# the whole file.  Andy's minimal case was `ktext -s "# @code" -d`.
+#
+# Both directions need a case, and the second is the one a careless fix breaks.
+
+# A literal klammer with a literal parameter, defined inline: the engine tier
+# loads no klammer set, so @code is not available here.
+LIT='@@lit.k t.literal : A literal klammer @@ @@lit.t :: [*t*] @@'
+
+echo
+echo "-- literal klammers named inside removed text --"
+
+check_eq "19. named in a # comment: removed, not an opening delimiter" \
+    'kept' --klammersets none -t t -s "$T $LIT # @lit
+kept"
+check_eq "20. ... including the realistic case that raised it" \
+    'kept' --klammersets none -t t -s "$T $LIT # @image and @lit share the same arguments.
+kept"
+check_eq "21. named inside #[ ... ]#" \
+    'kept' --klammersets none -t t -s "$T $LIT #[ @lit ]# kept"
+check_eq "22. ... and inside NESTED #[ #[ ]# ]#" \
+    'kept' --klammersets none -t t -s "$T $LIT #[ a #[ @lit ]# b ]# kept"
+check_eq "23. after ## the rest of the file is gone" \
+    'kept' --klammersets none -t t -s "$T $LIT kept ## @lit"
+
+echo
+echo "-- and the converse: a # INSIDE literal content is CONTENT --"
+# The direction a careless fix breaks.  Skipping removed text must happen only
+# while looking for an OPENING delimiter; once one is found the scan jumps past
+# the whole span, so a "#" in the content is never examined.
+# A BARE "#" -- not a quoted "^#".  The quoted form would pass whether or not
+# the content was treated as literal, and so would prove nothing.
+#
+# The use must be in a FILE with the definition in "-s", not both in one "-s":
+# mark_literal_klammer_content() only knows the klammers REGISTERED WHEN IT
+# RUNS, and "-s" is processed as one unit, so a literal klammer defined and used
+# in the same string is not yet literal while that string is scanned.  "-s" is
+# processed before the input files, so this ordering is what a real document
+# has -- the klammer set is loaded first.
+LITUSE=$(mktemp /tmp/escape_lit.XXXXXX).kt
+printf '@lit a # b lit@\n' > "$LITUSE"
+# "-d" is required here and nowhere else in this suite: with a FILE input ktext
+# writes a file instead of displaying, so stdout would be empty.
+check_eq "24. a bare # inside literal content survives as content" \
+    '[a # b]' "$LITUSE" --klammersets none -t t -d -s "$T $LIT"
+rm -f "$LITUSE"
+check_eq "25. a real literal klammer still works after a comment naming it" \
+    '[x]' --klammersets none -t t -s "$T $LIT # mentions @lit here
+@lit x lit@"
+
 rm -f "$ERR"
 
 echo