51 lines
2.7 KiB
C++
51 lines
2.7 KiB
C++
|
#pragma once
|
||
|
|
|
||
|
|
#include "katom.h"
|
||
|
|
|
||
|
|
// Finding katoms in text by MATCHING: at each
|
||
|
|
// position, the katom type that matches the most characters wins, and the
|
||
|
|
// position advances past it. No text is divided and no katom rewrite rule
|
||
|
|
// is consulted.
|
||
|
|
//
|
||
|
|
// A COMPONENT, not a katomization method in itself -- which is what the
|
||
|
|
// name says. The composition that makes a method of it,
|
||
|
|
// katomize_text_by_matching(), lives in katomize.{h,cpp} beside the entry
|
||
|
|
// point for katomization by rewriting, because naming the process is that
|
||
|
|
// file's job and this one only supplies a way of doing part of it.
|
||
|
|
//
|
||
|
|
// It is built into libklammertext.so but NOTHING IN THE ENGINE CALLS IT:
|
||
|
|
// katomization goes through katomize.{h,cpp}, which divides text by the
|
||
|
|
// rules. This is the alternative, kept so the two stay comparable
|
||
|
|
// (dbg/katomize_compare.sh --which category,matching) and so the question
|
||
|
|
// stays open rather than being settled by deletion.
|
||
|
|
//
|
||
|
|
// The claim under test is not "this is faster" or "this is more correct".
|
||
|
|
// It is that matching over the EXISTING katom_types table is easier for one
|
||
|
|
// person to hold in their head than 30 ordered katom rewrite rules. It
|
||
|
|
// diverges on 153 of 9,694 corpus lines, every one of them prose that
|
||
|
|
// MENTIONS syntax rather than using it -- so the choice is not between two
|
||
|
|
// readings of the language but between two behaviours on text that is not
|
||
|
|
// quite in it.
|
||
|
|
|
||
|
|
// The character stream to katoms, longest match at each position. No
|
||
|
|
// rewriting, no recursion.
|
||
|
|
katom_list katomize_by_matching(const std::string& s, const std::string& source, int source_line);
|
||
|
|
|
||
|
|
// Desugaring only: the abbreviated form @name-arg-arg becomes the katoms
|
||
|
|
// @name arg | arg @. A separate pass over the katom list, because it is a
|
||
|
|
// separate job -- it INSERTS a delimiter that was not in the source, which
|
||
|
|
// is a source-to-source transform and not a question about where katoms
|
||
|
|
// begin and end. In katomization by rewriting these two jobs share one loop.
|
||
|
|
katom_list expand_abbreviations(const katom_list& katoms);
|
||
|
|
|
||
|
|
|
||
|
|
// Whether a katom type is something the SOURCE can contain. Five of the
|
||
|
|
// katom_t values are processing STATES a katom is later moved into
|
||
|
|
// (replaced, ignored, literal, eval_result, ws_added), not source
|
||
|
|
// categories, and their entries in katom_types carry no usable pattern --
|
||
|
|
// four are the empty string, which as a regex matches everywhere and would
|
||
|
|
// stall any katomizer. See notes/Klammertext_improvements.md §1: "katom_t
|
||
|
|
// mixes source categories with processing states". The rewrite katomizer
|
||
|
|
// never trips over this because it only ever asks whole-text "does this
|
||
|
|
// match", and an empty pattern matches only the empty string there.
|
||
|
|
bool appears_in_source(katom_t type);
|