ack/klammertext
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Literal @c, @source_listing with :marker, and a large-directory speedup

Browse Source 
Three changes.

@c now takes its content literally, like @code -- it is the inline form
and @code the block form of the same thing.  The named close "c@" is
required, and characters that are special in a target no longer break
the file: @c a_b c@ renders correctly everywhere.  The Markdown
converter stops quoting inline code, since nothing needs protecting.

@source_file is renamed @source_listing.  Code read from a file is its
own klammer; @code is only for a block written inline (its never-
implemented :filename and :pattern options are removed).  The new
:marker P option lists the region between two lines that are exactly
//P, so the source file declares its own extractable regions.  A marker
missing or not appearing exactly twice is an error, never a fallback.

Rendering a document that sits in a large directory was paying a
recursive walk of that directory's whole tree on every @eval -- 27
seconds for a document that renders in a third of one.  The walk is now
a non-recursive look decided once per directory.

Assembled from dev commit 071b1b183de4.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Andy Kopra
2026-08-16 16:58:50 +02:00
parent 8b38a34841
commit bd39d9a369
8 changed files with 284 additions and 47 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
README.md
View File

@@ -51,7 +51,7 @@ are regenerated on each release — patches cannot be merged directly.
Report problems (or send patches) to the author; accepted changes are
applied to the development tree and appear in a following snapshot.
This snapshot was assembled from development commit `6c2ff7b8fce3`.
This snapshot was assembled from development commit `071b1b183de4`.
## License

41
doc/klammertext_for_llm.md
View File

@@ -92,7 +92,7 @@ klammer's name immediately before it: `@i text i@.
```
@i just text i@ ← unnecessary; @i just text @ is better
@s1 A @c literal c@ heading s1@ ← name it: the body contains a klammer call
@s1 A @c fragment c@ heading s1@ ← name it: the body contains a klammer call
```
**Arguments.**
@@ -110,17 +110,42 @@ klammer's name immediately before it: `@i text i@.
**Abbreviated form.** `@name-arg1-arg2` is `@name arg1 | arg2 @`, valid only
while every argument is letters and digits: `@i-italic`, `@sup-2-3`, `@date@`.
**Literal klammers** take their content uninterpreted (`@c`, `@code`). Their
content is not scanned for `@`, `|`, `#`, or `^`, so the machine cannot find the
end by counting — **these must be closed with the named form**:
**Literal klammers** take their content uninterpreted. It is not scanned for
`@`, `|`, `#`, or `^`, so the machine cannot find the end by counting and **the
named close is required**:
```
@c ImfStandardAttributes.h c@
@code
int x = a[0]; // # and @ are safe in here
code@
```
```
@c a | b c@ ← inline: same rule, and the close must be "c@"
```
`@c` and `@code` are the in-line and block forms of the same thing, and both are
literal: **nothing inside either is interpreted**, so `#`, `@`, `|`, `^` and `*`
all pass through as written and none of them needs quoting.
Code that lives in a **file** is `@source_listing`, never `@code` — `@code` is
only for a block written out inline, and has no filename argument:
```
@source_listing src/derivations.cpp @ ← the whole file
@source_listing src/derivations.cpp :marker fromRdd55 @ ← one region of it
```
`:marker P` lists what lies between two lines that are exactly `//P`, each
beginning in the first column. The source file thus declares its own extractable
regions and the document asks for one by name, so editing the code cannot
silently change what the document shows. A marker that is missing, or that does
not appear exactly twice, is an **error** — never a silent fallback to the whole
file.
`kdesc -k <name>` is how you tell: a parameter shown as `text.literal` is a
literal parameter. Do not infer it from a klammer's purpose — check.
---
## 4. A complete document
@@ -137,7 +162,7 @@ code@
@s1 Principles s1@
Ordinary paragraph text. A blank line starts a new paragraph.
Use @i emphasis i@ and @b bold @ inline, and @c literal_text c@ for code.
Use @i emphasis i@ and @b bold @ inline, and @c code_fragment c@ for code.
@link https://openexr.com/ :text the OpenEXR site link@
@@ -172,9 +197,9 @@ Confirm parameters with `kdesc -k <name>` — this list is names and intent only
| | |
|---|---|
| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline literal, `@sup`, `@sub` |
| inline | `@i` italic, `@b` bold, `@t` typewriter, `@u` underline, `@c` inline code (literal; close with `c@`), `@sup`, `@sub` |
| structure | `@s1`…`@s7` numbered headings, `@h1`…`@h7` unnumbered, `@part`, `@preface`, `@par` |
| blocks | `@code`, `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` |
| blocks | `@code` (a listing written inline), `@source_listing` (a listing read from a file), `@quote`, `@note`, `@indent`, `@block`, `@center`, `@right`, `@columns` |
| lists | `@ul` unordered, `@ol` ordered, `@define` term/definition — items separated by `\|` |
| tables | `@table` — cells by `\|`, rows by `\|\|`; `@tbl`, `@rowcolor` |
| figures | `@image`, `@image_grid`, `@fig`, `@reference` |

41
mac/machine.cpp
View File

@@ -322,9 +322,46 @@ void Machine::mark_literal_klammer_content(katom_list& katoms)
if (literal_names.empty()) return;
// Scan for matching @name ... name@ spans.
// Stop at ## (ignore-rest) since everything after it will be removed.
//
// REMOVED TEXT IS SKIPPED. This pass runs first in process_katoms(), before
// mark_ignored_katoms() takes out the "#" forms, and that ordering is not
// accidental: a literal klammer's content must be marked before anything
// else can interpret what is inside it, or removal would take a "#" that
// belongs to the literal body. The cost is that this scan sees text the
// writer has removed, so a literal klammer merely NAMED in a comment --
//
// # @image and @code share the same arguments.
//
// -- was found as an opening delimiter, went unclosed, and failed the whole
// file (TODO #40, diagnosed 2026-08-05). "##" was already handled, which is
// half the case; the fix is to teach the scan the other three removal forms
// rather than to reorder the passes.
//
// The converse still holds, structurally: this skipping happens only while
// looking for an OPENING delimiter, and finding one jumps k past the whole
// span, so a "#" inside literal content is never examined here and stays
// content.
for (auto k = katoms.begin(); k != katoms.end(); ++k) {
if (k->m_type == katom_t::ignore_rest) break;
if (k->m_type == katom_t::ignore_rest) break; // "##": rest of the file
if (k->m_type == katom_t::ignore_line) { // "#": rest of the line
while (k + 1 != katoms.end() && (k + 1)->m_type != katom_t::newline) {
++k;
}
continue;
}
if (k->m_type == katom_t::ignore_begin) { // "#[ ... ]#", nestable
int removed = 1;
while (++k != katoms.end() && removed > 0) {
if (k->m_type == katom_t::ignore_begin) {
++removed;
} else if (k->m_type == katom_t::ignore_end) {
--removed;
}
}
if (k == katoms.end()) break; // unterminated: the rest is removed
--k; // the loop's ++k steps past the "]#"
continue;
}
if (k->m_type != katom_t::apply_begin) continue;
std::string name = trim_char(k->m_text, '@');
if (literal_names.count(name) == 0) continue;

33
mac/state.cpp
View File

@@ -1,5 +1,6 @@
#include <regex>
#include <iostream>
#include <map>
#include "show.h"
#include "file.h"
@@ -10,6 +11,35 @@
int State::class_id = 0;
namespace {
// A sys.path entry reaches only the directory's own files, so the decision
// must not recurse; and the answer is fixed for the run (the search-dir set
// only grows, and a directory's answer does not change), so it is decided
// once per directory per process. The former recursive walk here ran on
// every @eval and took ~200 ms per eval on a document in a large home
// directory (27 s for one render).
bool directory_has_python_file(const std::string& dir)
{
static std::map<std::string, bool> cache {};
auto it = cache.find(dir);
if (it != cache.end()) {
return it->second;
}
bool found = false;
std::error_code ec {};
for (const auto& entry : std::filesystem::directory_iterator(dir, ec)) {
if (entry.is_regular_file(ec) && entry.path().extension() == ".py") {
found = true;
break;
}
}
cache[dir] = found;
return found;
}
} // namespace
bool Var::defined() const
{
return !m_name.empty();
@@ -297,8 +327,7 @@ std::string State::python_code()
python_dirs.insert(python_dirs.end(),
m_search_dirs.begin(), m_search_dirs.end());
for (const auto& d : python_dirs) {
auto python_files = pathnames_with_extension(d, "py");
if (!python_files.empty()) {
if (directory_has_python_file(d)) {
ss << "sys.path.append('" << d << "')\n";
}
}

38
sks/code/code.k
View File

@@ -1,15 +1,27 @@
@@code.k :filename :pattern
@@code.k
@hpos_args :hpos left @
@caption_args :caption_side top @
| text.literal :
A source file displayed verbatim
A block of code, given here and displayed uninterpreted. Code that lives in a
file is ^@source_listing instead; ^@code is only for a block written inline.
It carried ":filename" and ":pattern" until 2026-08-16. Neither was ever
implemented -- "filename" is read only by ^@source_listing and "pattern" by
nothing at all -- and because a literal parameter's content begins at the first
katom when no bar is written, "^@code :filename f :pattern p" DISPLAYED THOSE
WORDS as the listing rather than acting on them.
@@
@@code.html,tex :: @eval code_block.Code(K) @ @@
@@c.k code_text :
A word or phrase displayed verbatim in a line
@@c.k code_text.literal :
A word or phrase displayed verbatim in a line. Like ^@code, its content is
LITERAL -- nothing in it is interpreted as Klammertext -- so it must be closed
with the named delimiter "c^@". ^@c is the in-line form and ^@code the block
form of the same thing; before 2026-08-16 this parameter was an ordinary
string, so a "^#" or a bare "^@" inside it was read as Klammertext and usually
failed the file, which is not what "verbatim" can mean.
@@
@@c.html,tex :: @eval code_block.Code_fragment(K) eval@
@@ -17,5 +29,19 @@ A word or phrase displayed verbatim in a line
# :cwd makes the filename resolve against the DOCUMENT's directory, not
# the directory ktext happens to run in.
@@source_file.k filename : Display the contents of the file verbatim. @@
@@source_file.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@
@@source_listing.k filename :marker
@hpos_args :hpos left @
@caption_args :caption_side top @
: Code read from a file and displayed uninterpreted. *filename* is the file;
without ":marker" the whole file is listed.
":marker P" lists only the region BETWEEN two lines that consist solely of
"//P" and begin in the first column. The source file therefore declares its
own extractable regions and the document asks for one by name, so the two
cannot drift apart silently: renaming or reformatting the code does not change
what is extracted, and a region that disappears is an error rather than a
quietly different listing.
Named "^@source_file" until 2026-08-16. # retired-ok
@@
@@source_listing.html,tex :: @eval :cwd *K_input_dir* code_block.Source(K) @ @@

106
sks/code/code_block.py
View File

@@ -324,35 +324,89 @@ class Code_fragment(klammer_base.Klammer_base):
def __init__(self, K):
super().__init__(K)
# code_text is a LITERAL parameter (sks/code/code.k), so its content
# reaches here exactly as written and NOTHING has escaped it -- the
# machine's target-character pass does not touch literal content. Both
# methods must therefore escape it themselves, with the same helpers the
# block form uses on its lines. Before 2026-08-16 the parameter was an
# ordinary string and the machine did the escaping; html() got away with
# handling "<" by hand and tex() with nothing at all.
def html(self):
#print(f"code: |{self.code_text}|")
result = self.code_text.strip()
result = undash(result)
#result = re.escape(result)
result = re.sub("<", "&lt;", result)
result = re.sub(" ", "&nbsp;", result)
#print(f"code: |{self.code_text}| -> |{result}|")
return f'<span class="code">{result}</span>'
return f'<span class="code">{html_line(undash(self.code_text.strip()))}</span>'
def tex(self):
return f"{{\\tt {self.code_text.strip()}}}"
return f"{{\\tt {tex_line(self.code_text.strip())}}}"
class Source(klammer_base.Klammer_base):
def extract_marked_region(src, marker, filename):
"""The region of `src` between two lines that are exactly "//<marker>".
The delimiter lines must consist SOLELY of "//" + marker and start in the
first column, so a marker cannot be matched inside indented code or in a
trailing comment. Both delimiters are the same text: the source brackets a
region rather than naming a start and a separate end.
A marker that is missing, or that appears only once, is an ERROR -- the
document asked for a region the file does not offer, and silently listing
the whole file (or nothing) would let the document drift from the code it
claims to quote, which is the one thing this option exists to prevent.
"""
delimiter = "//" + marker
lines = src.split("\n")
at = [i for i, line in enumerate(lines) if line == delimiter]
if len(at) < 2:
found = "once" if len(at) == 1 else "not at all"
raise Exception(
f'The marker "{marker}" appears {found} in "{filename}".\n'
f' A marked region is bracketed by TWO lines that are exactly\n'
f' "{delimiter}", each beginning in the first column.')
if len(at) > 2:
raise Exception(
f'The marker "{marker}" appears {len(at)} times in "{filename}"\n'
f' (lines {", ".join(str(i + 1) for i in at)}); a region needs exactly two.')
region = lines[at[0] + 1:at[1]]
while region and not region[0].strip():
region.pop(0)
while region and not region[-1].strip():
region.pop()
return "\n".join(region)
class Source(Code):
"""@source_listing -- a Code listing whose text comes from a FILE.
It IS a Code: @source_listing and @code differ only in where the text
comes from, so they must render identically, and subclassing is what
guarantees that rather than a second implementation that drifts.
It rendered separately until 2026-08-16, and was wrong in a way nothing
caught: html() quoted only "@" and tex() wrapped the raw text in a
verbatim environment. An @eval result is RE-READ as Klammertext, so an
unquoted "#" starts a text removal -- and since the html path had already
joined the source into one line, a file beginning "#include" produced an
EMPTY LISTING and exited 0. quote_specials() in this module documents
exactly that hazard, and this was the one place not using it. Inheriting
Code's rendering fixes both targets at once: tex_line()/html_line() quote
the Klammertext specials AND escape the target's own, so the author of the
source file needs to know about neither.
A verbatim environment could not have been made correct here, incidentally:
a quoted "^#" resolves to "\#" through the tex target's escape list, which
inside verbatim would print as "\#" rather than "#".
"""
def __init__(self, K):
super().__init__(K)
with open(self.filename) as fp:
self.src = fp.read()
def tex(self):
result = self.src
# result = re.sub("#", "^#", result)
# result = re.sub("\\^", "\\^", result)
result = f"\\begin{{verbatim}}\n{result}\n\\end{{verbatim}}\n"
return result
def html(self):
result = escape_newlines(self.src.strip()) + "\n"
result = re.sub("@", "^@", result)
result = E("div").body(result).cls("code_text").str()
return result
# Klammer_base, NOT Code: Code's constructor expands whitespace markers
# in self.text, and there is no "text" parameter here -- the text does
# not exist until the file has been read. File content carries no
# whitespace markers anyway, since those come from the katomizer.
klammer_base.Klammer_base.__init__(self, K)
try:
with open(self.filename) as fp:
text = fp.read()
except OSError as e:
raise Exception(
f'Cannot read the source listing "{self.filename}": {e.strerror}.\n'
f' A relative name resolves against the DOCUMENT\'s directory.')
if self.marker:
text = extract_marked_region(text, self.marker, self.filename)
self.text = text

11
sks/tns/md_to_sks.py
View File

@@ -128,8 +128,17 @@ def inline(text):
text = "".join(p if re.match(r'^[\x00\x01]', p) else quote(p) for p in parts)
text = re.sub(r'\x01(\d+)\x01', lambda m: verbatim[int(m.group(1))], text)
# Inline code goes in RAW, and is closed with the named delimiter. "@c"
# took an ordinary string parameter until 2026-08-16, so every Klammertext
# special inside a code span had to be quoted -- and about 40% of the
# inline code in the measured corpus contains one (see
# doc/markdown_to_klammertext.md). It is now a LITERAL parameter, like
# "@code": nothing inside is interpreted, so nothing needs quoting, and the
# source reads as the author wrote it. A literal klammer cannot be closed
# with a bare "@" -- the machine cannot find the end by counting -- so the
# close is "c@".
text = re.sub(r'\x00(\d+)\x00',
lambda m: OPEN + "@c " + quote(spans[int(m.group(1))]) + " @" + CLOSE, text)
lambda m: OPEN + "@c " + spans[int(m.group(1))] + " c@" + CLOSE, text)
# Klammertext delimiters need whitespace around them, but Markdown
# emphasis abuts its neighbours: "un**bold**ed" and "*name*_rectified"
# both put a word character hard against a delimiter, which the

57
tst/escape_test.sh
View File

@@ -124,6 +124,63 @@ check_eq "17. @eval result ^@ survives read-back as @" '@'
# Klammertext -- a resolved @ in the result must not be re-parsed.
check_eq "18. :after_apply phase: raw result, @ intact" 'A @ B' --klammersets none -t u -s '@@@target u | up :after_apply string.capwords @@@ a ^@ b'
# --- Literal klammers and REMOVED text (TODO #40, fixed 2026-08-16) ---
#
# mark_literal_klammer_content() runs FIRST in process_katoms(), before
# mark_ignored_katoms() takes out the "#" forms. That ordering is deliberate --
# a literal klammer's content must be marked before anything can interpret what
# is inside it, or removal would take a "#" belonging to the literal body -- but
# it meant the scan saw text the writer had removed, so a literal klammer merely
# NAMED in a comment was read as an opening delimiter, went unclosed, and failed
# the whole file. Andy's minimal case was `ktext -s "# @code" -d`.
#
# Both directions need a case, and the second is the one a careless fix breaks.
# A literal klammer with a literal parameter, defined inline: the engine tier
# loads no klammer set, so @code is not available here.
LIT='@@lit.k t.literal : A literal klammer @@ @@lit.t :: [*t*] @@'
echo
echo "-- literal klammers named inside removed text --"
check_eq "19. named in a # comment: removed, not an opening delimiter" \
'kept' --klammersets none -t t -s "$T $LIT # @lit
kept"
check_eq "20. ... including the realistic case that raised it" \
'kept' --klammersets none -t t -s "$T $LIT # @image and @lit share the same arguments.
kept"
check_eq "21. named inside #[ ... ]#" \
'kept' --klammersets none -t t -s "$T $LIT #[ @lit ]# kept"
check_eq "22. ... and inside NESTED #[ #[ ]# ]#" \
'kept' --klammersets none -t t -s "$T $LIT #[ a #[ @lit ]# b ]# kept"
check_eq "23. after ## the rest of the file is gone" \
'kept' --klammersets none -t t -s "$T $LIT kept ## @lit"
echo
echo "-- and the converse: a # INSIDE literal content is CONTENT --"
# The direction a careless fix breaks. Skipping removed text must happen only
# while looking for an OPENING delimiter; once one is found the scan jumps past
# the whole span, so a "#" in the content is never examined.
# A BARE "#" -- not a quoted "^#". The quoted form would pass whether or not
# the content was treated as literal, and so would prove nothing.
#
# The use must be in a FILE with the definition in "-s", not both in one "-s":
# mark_literal_klammer_content() only knows the klammers REGISTERED WHEN IT
# RUNS, and "-s" is processed as one unit, so a literal klammer defined and used
# in the same string is not yet literal while that string is scanned. "-s" is
# processed before the input files, so this ordering is what a real document
# has -- the klammer set is loaded first.
LITUSE=$(mktemp /tmp/escape_lit.XXXXXX).kt
printf '@lit a # b lit@\n' > "$LITUSE"
# "-d" is required here and nowhere else in this suite: with a FILE input ktext
# writes a file instead of displaying, so stdout would be empty.
check_eq "24. a bare # inside literal content survives as content" \
'[a # b]' "$LITUSE" --klammersets none -t t -d -s "$T $LIT"
rm -f "$LITUSE"
check_eq "25. a real literal klammer still works after a comment naming it" \
'[x]' --klammersets none -t t -s "$T $LIT # mentions @lit here
@lit x lit@"
rm -f "$ERR"
echo