klammertext/mac/katomize.h

#pragma once

#include "katom.h"
#include "katom_rewrite_rules.h"

// KATOMIZATION: text becomes katoms, by dividing it with the katom rewrite
// rules in katom_rewrite_rules.{h,cpp}.  Live since 2026-09-02, when it
// replaced make_katoms_from_word(), split_into_katoms() and katomize() in
// mac/katom.cpp -- dbg/katomize_compare.sh had shown the two identical over
// every tracked .k and .kt first, and the katoms of all 186 files are
// byte-identical across the change.
//
// THE SEPARATION.  mac/katom.cpp used to hold two unrelated subjects.  One is the KATOM: what it is, what its type means,
// how a list of them is trimmed, split on bars, turned back into a string.
// The other is KATOMIZATION: how a stream of characters becomes that list.
// A katom is a data structure the whole engine uses; katomization is one
// algorithm, run once per file, that nothing else calls into.  Splitting
// them means the rest of the engine can include katom.h without also
// getting the katomizer, and the katomizer can be replaced -- as this file is doing
// -- without touching the type everything depends on.
//
// Done 2026-09-02: make_katoms_from_word, split_into_katoms, katomize,
// expand_compound_katom and warn_unparsed_katoms left katom.cpp for this
// file, and katom.cpp keeps Katom itself, the predicates, and
// trim/bar_split/as_string.  warn_unparsed_katoms came too, because it is
// the only reader of m_unparsed, which only katomization writes -- what
// makes it LOOK like a later pass is its call site, and timing is not
// ownership.  line_split() followed on 2026-09-02, to file.{h,cpp}: it
// produces no katom and one overload opens a file, so it failed this test
// in both directions.

// Make a katom from matched source text.  The one place that knows how a
// construct is WRITTEN differs from what it MEANS: a quoted special is
// written "^@" and means "@", so the caret comes off here.  Both katomizers
// go through this, and nothing else constructs a katom whose type was
// decided by matching source.
//
// Note what is NOT lost: m_src keeps the text exactly as written, so the
// caret is recoverable.  hide_special_katoms() replaces m_text with a KTESC
// marker shortly afterwards, precisely so the bare character cannot be
// re-read as syntax; where that pass is skipped -- inside a definition or
// an @eval/@read/@cond span -- the bare character is what the code or the
// declaration should receive.
Katom source_katom(const std::string& text, katom_t type, const Locator& loc);

// Katomize one piece of text containing no whitespace.  The replacement for
// make_katoms_from_word().
//
// The recursion needs no name for what it divides: at every level the thing
// being katomized is TEXT, and it stops when the text IS a katom.  (The
// original called it a "word", which collides with katom_t::word, a katom
// TYPE meaning letters only -- two referents, one term, in one file.)
//
// Same shape as the original: text matching a katom type IS that katom;
// otherwise the first katom rewrite rule that usefully rewrites it decides,
// and the pieces of that rewrite are katomized in turn; text no rule can
// divide becomes an unparsed katom, which warn_unparsed_katoms() reports
// later.
katom_list katomize_by_rewriting(const std::string& text, const std::string& source, int line, int chr,
                                 rewrite_order order = rewrite_order::by_category);

// One line (or any string) to katoms.  The replacement for
// split_into_katoms(), including its \r removal and tab expansion.
katom_list katomize_text(const std::string& text, const std::string& source, int line,
                         rewrite_order order = rewrite_order::by_category);

// The same, katomizing by MATCHING instead of by rewriting -- the second
// entry point, so that naming a katomization method stays this file's job.
// Its component lives in katom_matching.{h,cpp}; the desugaring of the
// abbreviated application is a separate pass there, because matching
// produces the abbreviated form as one katom and something must expand it.
katom_list katomize_text_by_matching(const std::string& text, const std::string& source, int line);

// A file's lines to katoms.  The replacement for katomize(): each line is
// katomized with its newline restored, and the final newline is dropped.
katom_list katomize_lines(const strings_t& lines, const std::string& source,
                          rewrite_order order = rewrite_order::by_category);

// "kdiag --rewrite": report each rule as it fires, with its category.  It
// now names the rule that ACTUALLY fired, which the old path could not do
// for the abbreviated application -- that was applied inside
// expand_compound_katom() and attributed to whichever rule the loop
// happened to be holding, so it always read as "bar precedence".
inline bool show_rewrite_rules = false;

// Warn about text that matched no katom type and that no rule could divide
// -- but only where it survives processing: text removed by #, ##, or
// #[...]#, replaced spans, and literal content never warn.  Called at the
// end of Machine::process_katoms(), after those passes have marked the
// katoms.  Clears the flag afterwards so reprocessing the same list does
// not warn twice.  With warn=false (the @eval read-back sub-Machine, whose
// katoms hold machine-generated result text) nothing is printed and every
// flag is cleared, so spliced result katoms stay silent.
void warn_unparsed_katoms(katom_list& katoms, bool warn = true);