From bd39d9a3698a925fa50e7ff3e0850272edbe0862 Mon Sep 17 00:00:00 2001 From: Andy Kopra Date: Sun, 16 Aug 2026 16:58:50 +0200 Subject: [PATCH] Literal @c, @source_listing with :marker, and a large-directory speedup Three changes. @c now takes its content literally, like @code -- it is the inline form and @code the block form of the same thing. The named close "c@" is required, and characters that are special in a target no longer break the file: @c a_b c@ renders correctly everywhere. The Markdown converter stops quoting inline code, since nothing needs protecting. @source_file is renamed @source_listing. Code read from a file is its own klammer; @code is only for a block written inline (its never- implemented :filename and :pattern options are removed). The new :marker P option lists the region between two lines that are exactly //P, so the source file declares its own extractable regions. A marker missing or not appearing exactly twice is an error, never a fallback. Rendering a document that sits in a large directory was paying a recursive walk of that directory's whole tree on every @eval -- 27 seconds for a document that renders in a third of one. The walk is now a non-recursive look decided once per directory. Assembled from dev commit 071b1b183de4. Co-Authored-By: Claude Fable 5 --- README.md | 2 +- doc/klammertext_for_llm.md | 41 +++++++++++--- mac/machine.cpp | 41 +++++++++++++- mac/state.cpp | 33 +++++++++++- sks/code/code.k | 38 ++++++++++--- sks/code/code_block.py | 108 +++++++++++++++++++++++++++---------- sks/tns/md_to_sks.py | 11 +++- tst/escape_test.sh | 57 ++++++++++++++++++++ 8 files changed, 284 insertions(+), 47 deletions(-) diff --git a/README.md b/README.md index 9e45d29..3fc5995 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ are regenerated on each release — patches cannot be merged directly. Report problems (or send patches) to the author; accepted changes are applied to the development tree and appear in a following snapshot. -This snapshot was assembled from development commit `6c2ff7b8fce3`. +This snapshot was assembled from development commit `071b1b183de4`. ## License diff --git a/doc/klammertext_for_llm.md b/doc/klammertext_for_llm.md index bde2da4..ab77cfa 100644 --- a/doc/klammertext_for_llm.md +++ b/doc/klammertext_for_llm.md @@ -92,7 +92,7 @@ klammer's name immediately before it: `@i text i@. ``` @i just text i@ ← unnecessary; @i just text @ is better -@s1 A @c literal c@ heading s1@ ← name it: the body contains a klammer call +@s1 A @c fragment c@ heading s1@ ← name it: the body contains a klammer call ``` **Arguments.** @@ -110,17 +110,42 @@ klammer's name immediately before it: `@i text i@. **Abbreviated form.** `@name-arg1-arg2` is `@name arg1 | arg2 @`, valid only while every argument is letters and digits: `@i-italic`, `@sup-2-3`, `@date@`. -**Literal klammers** take their content uninterpreted (`@c`, `@code`). Their -content is not scanned for `@`, `|`, `#`, or `^`, so the machine cannot find the -end by counting — **these must be closed with the named form**: +**Literal klammers** take their content uninterpreted. It is not scanned for +`@`, `|`, `#`, or `^`, so the machine cannot find the end by counting and **the +named close is required**: ``` -@c ImfStandardAttributes.h c@ @code int x = a[0]; // # and @ are safe in here code@ ``` +``` +@c a | b c@ ← inline: same rule, and the close must be "c@" +``` + +`@c` and `@code` are the in-line and block forms of the same thing, and both are +literal: **nothing inside either is interpreted**, so `#`, `@`, `|`, `^` and `*` +all pass through as written and none of them needs quoting. + +Code that lives in a **file** is `@source_listing`, never `@code` — `@code` is +only for a block written out inline, and has no filename argument: + +``` +@source_listing src/derivations.cpp @ ← the whole file +@source_listing src/derivations.cpp :marker fromRdd55 @ ← one region of it +``` + +`:marker P` lists what lies between two lines that are exactly `//P`, each +beginning in the first column. The source file thus declares its own extractable +regions and the document asks for one by name, so editing the code cannot +silently change what the document shows. A marker that is missing, or that does +not appear exactly twice, is an **error** — never a silent fallback to the whole +file. + +`kdesc -k ` is how you tell: a parameter shown as `text.literal` is a +literal parameter. Do not infer it from a klammer's purpose — check. + --- ## 4. A complete document @@ -137,7 +162,7 @@ code@ @s1 Principles s1@ Ordinary paragraph text. A blank line starts a new paragraph. -Use @i emphasis i@ and @b bold @ inline, and @c literal_text c@ for code. +Use @i emphasis i@ and @b bold @ inline, and @c code_fragment c@ for code. @link https://openexr.com/ :text the OpenEXR site link@ @@ -172,9 +197,9 @@ Confirm parameters with `kdesc -k ` — this list is names and intent only | | | |---|---| -| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline literal, `@sup`, `@sub` | +| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline code (literal; close with `c@`), `@sup`, `@sub` | | structure | `@s1`…`@s7` numbered headings, `@h1`…`@h7` unnumbered, `@part`, `@preface`, `@par` | -| blocks | `@code`, `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` | +| blocks | `@code` (a listing written inline), `@source_listing` (a listing read from a file), `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` | | lists | `@ul` unordered, `@ol` ordered, `@define` term/definition — items separated by `\|` | | tables | `@table` — cells by `\|`, rows by `\|\|`; `@tbl`, `@rowcolor` | | figures | `@image`, `@image_grid`, `@fig`, `@reference` | diff --git a/mac/machine.cpp b/mac/machine.cpp index 01a3f5d..49cc787 100644 --- a/mac/machine.cpp +++ b/mac/machine.cpp @@ -322,9 +322,46 @@ void Machine::mark_literal_klammer_content(katom_list& katoms) if (literal_names.empty()) return; // Scan for matching @name ... name@ spans. - // Stop at ## (ignore-rest) since everything after it will be removed. + // + // REMOVED TEXT IS SKIPPED. This pass runs first in process_katoms(), before + // mark_ignored_katoms() takes out the "#" forms, and that ordering is not + // accidental: a literal klammer's content must be marked before anything + // else can interpret what is inside it, or removal would take a "#" that + // belongs to the literal body. The cost is that this scan sees text the + // writer has removed, so a literal klammer merely NAMED in a comment -- + // + // # @image and @code share the same arguments. + // + // -- was found as an opening delimiter, went unclosed, and failed the whole + // file (TODO #40, diagnosed 2026-08-05). "##" was already handled, which is + // half the case; the fix is to teach the scan the other three removal forms + // rather than to reorder the passes. + // + // The converse still holds, structurally: this skipping happens only while + // looking for an OPENING delimiter, and finding one jumps k past the whole + // span, so a "#" inside literal content is never examined here and stays + // content. for (auto k = katoms.begin(); k != katoms.end(); ++k) { - if (k->m_type == katom_t::ignore_rest) break; + if (k->m_type == katom_t::ignore_rest) break; // "##": rest of the file + if (k->m_type == katom_t::ignore_line) { // "#": rest of the line + while (k + 1 != katoms.end() && (k + 1)->m_type != katom_t::newline) { + ++k; + } + continue; + } + if (k->m_type == katom_t::ignore_begin) { // "#[ ... ]#", nestable + int removed = 1; + while (++k != katoms.end() && removed > 0) { + if (k->m_type == katom_t::ignore_begin) { + ++removed; + } else if (k->m_type == katom_t::ignore_end) { + --removed; + } + } + if (k == katoms.end()) break; // unterminated: the rest is removed + --k; // the loop's ++k steps past the "]#" + continue; + } if (k->m_type != katom_t::apply_begin) continue; std::string name = trim_char(k->m_text, '@'); if (literal_names.count(name) == 0) continue; diff --git a/mac/state.cpp b/mac/state.cpp index b096920..2243eb7 100644 --- a/mac/state.cpp +++ b/mac/state.cpp @@ -1,5 +1,6 @@ #include #include +#include #include "show.h" #include "file.h" @@ -10,6 +11,35 @@ int State::class_id = 0; +namespace { + +// A sys.path entry reaches only the directory's own files, so the decision +// must not recurse; and the answer is fixed for the run (the search-dir set +// only grows, and a directory's answer does not change), so it is decided +// once per directory per process. The former recursive walk here ran on +// every @eval and took ~200 ms per eval on a document in a large home +// directory (27 s for one render). +bool directory_has_python_file(const std::string& dir) +{ + static std::map cache {}; + auto it = cache.find(dir); + if (it != cache.end()) { + return it->second; + } + bool found = false; + std::error_code ec {}; + for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) { + if (entry.is_regular_file(ec) && entry.path().extension() == ".py") { + found = true; + break; + } + } + cache[dir] = found; + return found; +} + +} // namespace + bool Var::defined() const { return !m_name.empty(); @@ -297,8 +327,7 @@ std::string State::python_code() python_dirs.insert(python_dirs.end(), m_search_dirs.begin(), m_search_dirs.end()); for (const auto& d : python_dirs) { - auto python_files = pathnames_with_extension(d, "py"); - if (!python_files.empty()) { + if (directory_has_python_file(d)) { ss << "sys.path.append('" << d << "')\n"; } } diff --git a/sks/code/code.k b/sks/code/code.k index 1535b1d..17565e6 100644 --- a/sks/code/code.k +++ b/sks/code/code.k @@ -1,15 +1,27 @@ -@@code.k :filename :pattern +@@code.k @hpos_args :hpos left @ @caption_args :caption_side top @ | text.literal : -A source file displayed verbatim +A block of code, given here and displayed uninterpreted. Code that lives in a +file is ^@source_listing instead; ^@code is only for a block written inline. + +It carried ":filename" and ":pattern" until 2026-08-16. Neither was ever +implemented -- "filename" is read only by ^@source_listing and "pattern" by +nothing at all -- and because a literal parameter's content begins at the first +katom when no bar is written, "^@code :filename f :pattern p" DISPLAYED THOSE +WORDS as the listing rather than acting on them. @@ @@code.html,tex :: @eval code_block.Code(K) @ @@ -@@c.k code_text : -A word or phrase displayed verbatim in a line +@@c.k code_text.literal : +A word or phrase displayed verbatim in a line. Like ^@code, its content is +LITERAL -- nothing in it is interpreted as Klammertext -- so it must be closed +with the named delimiter "c^@". ^@c is the in-line form and ^@code the block +form of the same thing; before 2026-08-16 this parameter was an ordinary +string, so a "^#" or a bare "^@" inside it was read as Klammertext and usually +failed the file, which is not what "verbatim" can mean. @@ @@c.html,tex :: @eval code_block.Code_fragment(K) eval@ @@ -17,5 +29,19 @@ A word or phrase displayed verbatim in a line # :cwd makes the filename resolve against the DOCUMENT's directory, not # the directory ktext happens to run in. -@@source_file.k filename : Display the contents of the file verbatim. @@ -@@source_file.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@ +@@source_listing.k filename :marker + @hpos_args :hpos left @ + @caption_args :caption_side top @ + : Code read from a file and displayed uninterpreted. *filename* is the file; +without ":marker" the whole file is listed. + +":marker P" lists only the region BETWEEN two lines that consist solely of +"//P" and begin in the first column. The source file therefore declares its +own extractable regions and the document asks for one by name, so the two +cannot drift apart silently: renaming or reformatting the code does not change +what is extracted, and a region that disappears is an error rather than a +quietly different listing. + +Named "^@source_file" until 2026-08-16. # retired-ok +@@ +@@source_listing.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@ diff --git a/sks/code/code_block.py b/sks/code/code_block.py index 850e3e7..045083a 100644 --- a/sks/code/code_block.py +++ b/sks/code/code_block.py @@ -324,35 +324,89 @@ class Code_fragment(klammer_base.Klammer_base): def __init__(self, K): super().__init__(K) + # code_text is a LITERAL parameter (sks/code/code.k), so its content + # reaches here exactly as written and NOTHING has escaped it -- the + # machine's target-character pass does not touch literal content. Both + # methods must therefore escape it themselves, with the same helpers the + # block form uses on its lines. Before 2026-08-16 the parameter was an + # ordinary string and the machine did the escaping; html() got away with + # handling "<" by hand and tex() with nothing at all. def html(self): - #print(f"code: |{self.code_text}|") - result = self.code_text.strip() - result = undash(result) - #result = re.escape(result) - result = re.sub("<", "<", result) - result = re.sub(" ", " ", result) - #print(f"code: |{self.code_text}| -> |{result}|") - return f'{result}' - + return f'{html_line(undash(self.code_text.strip()))}' + def tex(self): - return f"{{\\tt {self.code_text.strip()}}}" + return f"{{\\tt {tex_line(self.code_text.strip())}}}" -class Source(klammer_base.Klammer_base): +def extract_marked_region(src, marker, filename): + """The region of `src` between two lines that are exactly "//". + + The delimiter lines must consist SOLELY of "//" + marker and start in the + first column, so a marker cannot be matched inside indented code or in a + trailing comment. Both delimiters are the same text: the source brackets a + region rather than naming a start and a separate end. + + A marker that is missing, or that appears only once, is an ERROR -- the + document asked for a region the file does not offer, and silently listing + the whole file (or nothing) would let the document drift from the code it + claims to quote, which is the one thing this option exists to prevent. + """ + delimiter = "//" + marker + lines = src.split("\n") + at = [i for i, line in enumerate(lines) if line == delimiter] + if len(at) < 2: + found = "once" if len(at) == 1 else "not at all" + raise Exception( + f'The marker "{marker}" appears {found} in "{filename}".\n' + f' A marked region is bracketed by TWO lines that are exactly\n' + f' "{delimiter}", each beginning in the first column.') + if len(at) > 2: + raise Exception( + f'The marker "{marker}" appears {len(at)} times in "{filename}"\n' + f' (lines {", ".join(str(i + 1) for i in at)}); a region needs exactly two.') + region = lines[at[0] + 1:at[1]] + while region and not region[0].strip(): + region.pop(0) + while region and not region[-1].strip(): + region.pop() + return "\n".join(region) + + +class Source(Code): + """@source_listing -- a Code listing whose text comes from a FILE. + + It IS a Code: @source_listing and @code differ only in where the text + comes from, so they must render identically, and subclassing is what + guarantees that rather than a second implementation that drifts. + + It rendered separately until 2026-08-16, and was wrong in a way nothing + caught: html() quoted only "@" and tex() wrapped the raw text in a + verbatim environment. An @eval result is RE-READ as Klammertext, so an + unquoted "#" starts a text removal -- and since the html path had already + joined the source into one line, a file beginning "#include" produced an + EMPTY LISTING and exited 0. quote_specials() in this module documents + exactly that hazard, and this was the one place not using it. Inheriting + Code's rendering fixes both targets at once: tex_line()/html_line() quote + the Klammertext specials AND escape the target's own, so the author of the + source file needs to know about neither. + + A verbatim environment could not have been made correct here, incidentally: + a quoted "^#" resolves to "\#" through the tex target's escape list, which + inside verbatim would print as "\#" rather than "#". + """ def __init__(self, K): - super().__init__(K) - with open(self.filename) as fp: - self.src = fp.read() - - def tex(self): - result = self.src - # result = re.sub("#", "^#", result) - # result = re.sub("\\^", "\\^", result) - result = f"\\begin{{verbatim}}\n{result}\n\\end{{verbatim}}\n" - return result - - def html(self): - result = escape_newlines(self.src.strip()) + "\n" - result = re.sub("@", "^@", result) - result = E("div").body(result).cls("code_text").str() - return result + # Klammer_base, NOT Code: Code's constructor expands whitespace markers + # in self.text, and there is no "text" parameter here -- the text does + # not exist until the file has been read. File content carries no + # whitespace markers anyway, since those come from the katomizer. + klammer_base.Klammer_base.__init__(self, K) + try: + with open(self.filename) as fp: + text = fp.read() + except OSError as e: + raise Exception( + f'Cannot read the source listing "{self.filename}": {e.strerror}.\n' + f' A relative name resolves against the DOCUMENT\'s directory.') + if self.marker: + text = extract_marked_region(text, self.marker, self.filename) + self.text = text diff --git a/sks/tns/md_to_sks.py b/sks/tns/md_to_sks.py index 1e3eac6..43c2680 100644 --- a/sks/tns/md_to_sks.py +++ b/sks/tns/md_to_sks.py @@ -128,8 +128,17 @@ def inline(text): text = "".join(p if re.match(r'^[\x00\x01]', p) else quote(p) for p in parts) text = re.sub(r'\x01(\d+)\x01', lambda m: verbatim[int(m.group(1))], text) + # Inline code goes in RAW, and is closed with the named delimiter. "@c" + # took an ordinary string parameter until 2026-08-16, so every Klammertext + # special inside a code span had to be quoted -- and about 40% of the + # inline code in the measured corpus contains one (see + # doc/markdown_to_klammertext.md). It is now a LITERAL parameter, like + # "@code": nothing inside is interpreted, so nothing needs quoting, and the + # source reads as the author wrote it. A literal klammer cannot be closed + # with a bare "@" -- the machine cannot find the end by counting -- so the + # close is "c@". text = re.sub(r'\x00(\d+)\x00', - lambda m: OPEN + "@c " + quote(spans[int(m.group(1))]) + " @" + CLOSE, text) + lambda m: OPEN + "@c " + spans[int(m.group(1))] + " c@" + CLOSE, text) # Klammertext delimiters need whitespace around them, but Markdown # emphasis abuts its neighbours: "un**bold**ed" and "*name*_rectified" # both put a word character hard against a delimiter, which the diff --git a/tst/escape_test.sh b/tst/escape_test.sh index 0c282fb..1270a03 100755 --- a/tst/escape_test.sh +++ b/tst/escape_test.sh @@ -124,6 +124,63 @@ check_eq "17. @eval result ^@ survives read-back as @" '@' # Klammertext -- a resolved @ in the result must not be re-parsed. check_eq "18. :after_apply phase: raw result, @ intact" 'A @ B' --klammersets none -t u -s '@@@target u | up :after_apply string.capwords @@@ a ^@ b' +# --- Literal klammers and REMOVED text (TODO #40, fixed 2026-08-16) --- +# +# mark_literal_klammer_content() runs FIRST in process_katoms(), before +# mark_ignored_katoms() takes out the "#" forms. That ordering is deliberate -- +# a literal klammer's content must be marked before anything can interpret what +# is inside it, or removal would take a "#" belonging to the literal body -- but +# it meant the scan saw text the writer had removed, so a literal klammer merely +# NAMED in a comment was read as an opening delimiter, went unclosed, and failed +# the whole file. Andy's minimal case was `ktext -s "# @code" -d`. +# +# Both directions need a case, and the second is the one a careless fix breaks. + +# A literal klammer with a literal parameter, defined inline: the engine tier +# loads no klammer set, so @code is not available here. +LIT='@@lit.k t.literal : A literal klammer @@ @@lit.t :: [*t*] @@' + +echo +echo "-- literal klammers named inside removed text --" + +check_eq "19. named in a # comment: removed, not an opening delimiter" \ + 'kept' --klammersets none -t t -s "$T $LIT # @lit +kept" +check_eq "20. ... including the realistic case that raised it" \ + 'kept' --klammersets none -t t -s "$T $LIT # @image and @lit share the same arguments. +kept" +check_eq "21. named inside #[ ... ]#" \ + 'kept' --klammersets none -t t -s "$T $LIT #[ @lit ]# kept" +check_eq "22. ... and inside NESTED #[ #[ ]# ]#" \ + 'kept' --klammersets none -t t -s "$T $LIT #[ a #[ @lit ]# b ]# kept" +check_eq "23. after ## the rest of the file is gone" \ + 'kept' --klammersets none -t t -s "$T $LIT kept ## @lit" + +echo +echo "-- and the converse: a # INSIDE literal content is CONTENT --" +# The direction a careless fix breaks. Skipping removed text must happen only +# while looking for an OPENING delimiter; once one is found the scan jumps past +# the whole span, so a "#" in the content is never examined. +# A BARE "#" -- not a quoted "^#". The quoted form would pass whether or not +# the content was treated as literal, and so would prove nothing. +# +# The use must be in a FILE with the definition in "-s", not both in one "-s": +# mark_literal_klammer_content() only knows the klammers REGISTERED WHEN IT +# RUNS, and "-s" is processed as one unit, so a literal klammer defined and used +# in the same string is not yet literal while that string is scanned. "-s" is +# processed before the input files, so this ordering is what a real document +# has -- the klammer set is loaded first. +LITUSE=$(mktemp /tmp/escape_lit.XXXXXX).kt +printf '@lit a # b lit@\n' > "$LITUSE" +# "-d" is required here and nowhere else in this suite: with a FILE input ktext +# writes a file instead of displaying, so stdout would be empty. +check_eq "24. a bare # inside literal content survives as content" \ + '[a # b]' "$LITUSE" --klammersets none -t t -d -s "$T $LIT" +rm -f "$LITUSE" +check_eq "25. a real literal klammer still works after a comment naming it" \ + '[x]' --klammersets none -t t -s "$T $LIT # mentions @lit here +@lit x lit@" + rm -f "$ERR" echo