ack/klammertext
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unicode sequences, @typeface, and the font/typeface distinction

Browse Source 
A run of Unicode code points now needs one pair of carets, not one pair
each: ^9053 53EF 9053^ renders the characters adjacently, and a single
character inside such a run stands for itself.  An invalid code point is
an error naming the position that wrote it, rather than an apology
rendered into the finished document; noncharacters are refused alongside
surrogates; and a code point that lands on a Klammertext special is safe
through the re-reads that assemble a document.  The character tables now
produce the precomposed form, so a diacritic written as ^a" and one typed
directly are the same text to a search or a copy-paste.

"kdesc -u" describes how to write a Unicode character, with tables of
glyphs beside the codes that produce them.  It is new and not yet
complete: it predates the sequence form and the typeface klammer below.

A typeface and a style are no longer both called "font".  A typeface is a
family resolved by name; a style code is the one letter naming a type
style.  So @document takes :serif_typeface and its siblings, a table takes
:style, captions take :caption_style and :caption_size, the uniform scale
is :text_scale, and the command is "kdesc --typeface".  This renames
arguments a document may already use; an old name is a located error
rather than a silent omission.

@typeface sets a passage in another typeface for the length of that
passage:

    @typeface Noto Sans Egyptian Hieroglyphs | ^132AA 133CF^ :scale 1.5 @

Writing the klammer is the whole declaration -- the document resolves the
typeface, embeds it in what it produces, and carries its license along.
Two typefaces for scripts no text font covers now ship with Klammertext:
Noto Sans Egyptian Hieroglyphs and Noto Sans Cuneiform.  Every typeface in
the distribution now carries its license file.

(from dev 61e598698891)
This commit is contained in:
ack committed 2026-08-25 23:30:55 +02:00
1 parent 37b6ba1c4f
commit af29bbf7ac
47 files changed
+1921 -135
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

mac/character.cpp
+97 -10
View File
@@ -2,9 +2,26 @@
#include "util.h"
#include "character.h"
#include "precomposed.h"
#include "error.h"
#include "target.h"
#include "log.h"
#include "show.h"
// A base letter composed with a combining mark. The PRECOMPOSED (NFC)
// code point is emitted where one exists -- that is the byte sequence
// editors and input methods produce for direct entry, so ^a" and a typed
// ä compare equal (search, copy-paste); the tables formerly emitted
// base + combining mark, which renders identically but is a different
// string. Pairs with no precomposed form keep the combining fallback.
// The table is GENERATED: doc/make_precomposed_table.py -> precomposed.h.
static std::string compose(const std::string& base, const std::string& mark_cp)
{
auto it = precomposed.find(base + mark_cp);
if (it != precomposed.end()) return it->second;
return base + utf8char(std::stoi(mark_cp, nullptr, 16));
}
inline
std::string klammertext_special_characters { "@|*^#" };
@@ -44,6 +61,24 @@ std::string utf8char(int cp)
}
bool valid_codepoint(int cp)
{
if (cp < 0 || cp > 0x10FFFF) return false;
if (0xD800 <= cp && cp <= 0xDFFF) return false; // surrogates
if (0xFDD0 <= cp && cp <= 0xFDEF) return false; // noncharacters
if ((cp & 0xFFFE) == 0xFFFE) return false; // U+xFFFE / U+xFFFF
return true;
}
std::string invalid_codepoint_reason(int cp)
{
if (cp < 0 || cp > 0x10FFFF)
return "beyond U+10FFFF, the top of the Unicode range";
if (0xD800 <= cp && cp <= 0xDFFF)
return "a surrogate (U+D800-U+DFFF), not a character";
return "a noncharacter, reserved by Unicode for internal use";
}
std::string unicode_hex_to_char(std::string s, int width=4) //, std::string marker)
{
(void)K::log(4, s);
@@ -82,7 +117,7 @@ std::string process_diacritics(std::string s)
std::regex hit { regex_escape((*p)[0].str()) };
std::string ch = (*p)[1].str();
std::string d = (*p)[2].str();
result = std::regex_replace(result, hit, ch + unicode_hex_to_char(diacritics[d].first));
result = std::regex_replace(result, hit, compose(ch, diacritics[d].first));
}
return result;
}
@@ -124,15 +159,14 @@ std::string process_pinyin(std::string s)
std::string vowel = (*p)[1].str();
std::string tone = (*p)[2].str();
result = std::regex_replace(result, hit,
vowel + unicode_hex_to_char(pinyin_tones[tone].first));
compose(vowel, pinyin_tones[tone].first));
}
return result;
}
std::string process_unicode_codepoint(std::string s)
std::string process_unicode_codepoint(std::string s, const Locator& loc = Locator::none())
{
(void)K::log(4);
//return std::regex_replace(s, unicode_re, hidehat + "$1" + hidehat);
std::string result {s};
std::sregex_iterator end {};
for (std::sregex_iterator p { s.begin(), s.end(), unicode_re }; p!= end; ++p) {
@@ -141,8 +175,22 @@ std::string process_unicode_codepoint(std::string s)
// its argument at a fixed width, and its 4-digit default truncated a
// 5-digit code point to its first four digits (^13000^ rendered as
// U+1300 followed by a literal "0").
int cp = std::stoi((*p)[1].str(), nullptr, 16);
// An invalid code point is a located ERROR, not text in the document
// (the no-warn-and-continue policy): a generator's typo must not
// produce a finished-looking document with an apology embedded
// where a character should be.
if (!valid_codepoint(cp)) {
throw Parsing_error("Invalid Unicode code point \"" + (*p)[0].str()
+ "\": " + invalid_codepoint_reason(cp), loc);
}
// A code point that lands on a Klammertext structural character
// (@ | # ^ : *) is hidden as its KTESC marker, exactly as the
// quoted form of that character would be -- a bare "@" from
// ^0040^ would otherwise be re-interpreted when the text is
// re-katomized (@document :text, the @eval read-back).
result = std::regex_replace(result, hit_re,
utf8char(std::stoi((*p)[1].str(), nullptr, 16)));
hide_structural_characters(utf8char(cp)));
}
return result;
@@ -154,7 +202,7 @@ std::string process_unicode_codepoint(std::string s)
std::string encode(const std::string& s)
std::string encode(const std::string& s, const Locator& loc)
{
if (s.find("^") == std::string::npos)
return s;
@@ -177,7 +225,7 @@ std::string encode(const std::string& s)
result = process_extended_latin(result);
if (dbg) std::cout << "extended_latin: " << result << "\n";
result = process_unicode_codepoint(result);
result = process_unicode_codepoint(result, loc);
if (dbg) std::cout << "unicode: " << result << "\n";
result = process_diacritics(result);
@@ -225,7 +273,7 @@ void diacritics_examples(const std::string& kt_filename)
<< " @i-Displayed | @i-Written | @i-Name\n";
else
std::cout << boldblack
<< "\nDiacritics (with typical base characters)\n" << black;
<< "\nDiacritics (with typical base characters)\n\n" << black;
std::string line_sep = "|| @line@ ";
for (char symbol : diacritic_symbols_order) {
@@ -260,7 +308,7 @@ void extended_latin_examples(const std::string& kt_filename)
kt << "@table :caption " << title << " |\n"
<< " @i-Displayed | @i-Written | @i-Name\n";
else
std::cout << "\n" << boldblack << title << black << "\n";
std::cout << "\n" << boldblack << title << black << "\n\n";
std::string line_sep = "|| @line@ ";
for (const std::string& symbol : extended_latin_symbols) {
@@ -291,7 +339,7 @@ void pinyin_examples(const std::string& kt_filename)
<< " @i-Displayed | @i-Written | @i-Name\n";
else
std::cout << boldblack
<< "\nMandarin pinyin tones (using vowel \"a\")\n" << black;
<< "\nMandarin pinyin tones (using vowel \"a\")\n\n" << black;
std::string line_sep = "|| @line@ ";
for (auto [symbol, codename] : pinyin_tones) {
@@ -309,6 +357,45 @@ void pinyin_examples(const std::string& kt_filename)
write_kt_example_file(kt, kt_filename);
}
void unicode_examples(const std::string& kt_filename)
{
std::stringstream kt {};
bool write_kt_file = kt_filename.size() != 0;
std::cout << boldblack << "\nIncluding Unicode in Klammertext files\n\n" << black
<< "To specify a wide variety of different letter forms, the Unicode Standard\n"
<< "defines a large set of \"code points\" organized by groups. To include a\n"
<< "Unicode character in Klammertext, you surround its hexadecimal number with\n"
<< "the \"^\" character. For example, the Cyrillic captial letter \"I\" is \"И\",\n"
<< "included in Klammertext as ^0418^.\n\n"
<< "To search for Unicode characters by their description, see:\n"
<< " https://www.fileformat.info/info/unicode/char/search.htm\n";
std::cout << "\n⟐ Examples of Chinese characters\n\n"
<< " Pinyin\n";
for (auto chr : dao) {
std::string code = "^" + chr[1] + "^";
std::cout << " " << process_unicode_codepoint(code) << " "
<< code << " "
<< chr[0] << "\n";
}
std::cout << "\n⟐ Examples of Egyptian hieroglyphs\n\n";
std::cout << std::setw(16) << " " << "Sound Gardiner\n";
for (auto glyph : ptolemy) {
std::cout << " " << std::setw(4) << process_unicode_codepoint("^" + glyph[2] + "^") << " "
<< "^" + glyph[2] + "^ "
<< std::setw(4)
<< glyph[1] << " "
<< std::setw(4)
<< glyph[0] << "\n";
}
std::cout << "\nSee: https://en.wikipedia.org/wiki/List_of_Egyptian_hieroglyphs\n"
<< "The Unicode codepoint in the second column of the \"List of hieroglyphs\"\n"
<< "table is the number in the ^...^ code used by Klammertext.\n\n";
}
void show_special_characters()
{
std::cout << std::setfill(' ');