What a Windows editor saves, and a standard user's Windows install (from dev 9d6f66456a3d)
- CRLF line endings are read as newlines on every system (the reader used to delete the pair, collapsing a Notepad++ file into one line on Linux and macOS); a leading UTF-8 byte-order mark is ignored; a UTF-16 or Windows-1252 source is refused with the encoding, the place and the remedy named. .gitattributes pins LF in every clone. - Windows source install: the repository path must hold no space (GNU Make), accented letters are fine (every linker input is relative now); python.org's Python is selected ahead of an MSYS2 one that poppler pulls in; TeX Live goes to its Windows default C:\texlive\2026. - The Python binding needs an ASCII installation path on Windows, and its suite says so there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This commit is contained in:
1 parent
fc2e848e91
commit
715d939727
15 files changed
+236
-41
No files matched your search
@@ -0,0 +1,9 @@
|
|||||||
|
# Every text file is LF in the working tree on every system, whatever
|
||||||
|
# core.autocrlf says: .kt, .k and the shell suites must be LF (2026-09-26).
|
||||||
|
* text=auto eol=lf
|
||||||
|
# Shipped verbatim, as the foundries wrote them (CRLF): never normalized.
|
||||||
|
fnt/**/*.txt -text
|
||||||
|
sks/document/fonts/**/*.txt -text
|
||||||
|
# Not shipped; kept byte-stable.
|
||||||
|
dbg/** -text
|
||||||
|
sks.alpha/** -text
|
||||||
@@ -53,7 +53,7 @@ are regenerated on each release — patches cannot be merged directly.
|
|||||||
Report problems (or send patches) to the author; accepted changes are
|
Report problems (or send patches) to the author; accepted changes are
|
||||||
applied to the development tree and appear in a following snapshot.
|
applied to the development tree and appear in a following snapshot.
|
||||||
|
|
||||||
This snapshot was assembled from development commit `d5f9c20b94b3`.
|
This snapshot was assembled from development commit `9d6f66456a3d`.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,9 @@ current file, and this package uses exactly those two things.
|
|||||||
version qualifies).
|
version qualifies).
|
||||||
- For the Run commands: `python` on the `Path`, which the Klammertext
|
- For the Run commands: `python` on the `Path`, which the Klammertext
|
||||||
installation already needs.
|
installation already needs.
|
||||||
|
- Files saved as UTF-8 (Notepad++'s default for new files; **Encoding →
|
||||||
|
Convert to UTF-8** for an older one). Line endings do not matter, and
|
||||||
|
a file in another encoding is refused by `ktext` with the remedy named.
|
||||||
|
|
||||||
## Install the highlighting
|
## Install the highlighting
|
||||||
|
|
||||||
|
|||||||
@@ -97,11 +97,17 @@ git clone https://git.andykopra.com/ack/klammertext.git
|
|||||||
cd klammertext
|
cd klammertext
|
||||||
```
|
```
|
||||||
|
|
||||||
Git for Windows must not translate line endings: Klammertext's `.kt`
|
The repository's path must contain no space: GNU Make cannot build under
|
||||||
files, `.k` files and shell scripts are LF files. The clone above respects
|
one. If your user name has a space in it (`C:\Users\Anna Maria`), clone
|
||||||
that as long as `core.autocrlf` is not `true`; check with `git config
|
somewhere else, for example into `C:\klammertext`, and write that path
|
||||||
--get core.autocrlf` and, if it prints `true`, set it to `false` and clone
|
wherever `C:\Users\<you>\klammertext` appears below. Accented letters in
|
||||||
again.
|
the path are fine for the commands; only the Python binding (`kpy\`) needs
|
||||||
|
a path of plain ASCII letters.
|
||||||
|
|
||||||
|
Klammertext's `.kt` files, `.k` files and shell scripts are LF files, and
|
||||||
|
the repository says so itself (`.gitattributes`), so the clone is correct
|
||||||
|
whatever your Git's `core.autocrlf` setting. A `.kt` you write with
|
||||||
|
Windows line endings is fine too: ktext reads them as newlines.
|
||||||
|
|
||||||
|
|
||||||
## Environment variables
|
## Environment variables
|
||||||
@@ -110,14 +116,14 @@ Klammertext's runtime environment is one self-configuring file, the same
|
|||||||
one Linux and macOS use. Source it from the UCRT64 shell's profile:
|
one Linux and macOS use. Source it from the UCRT64 shell's profile:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
echo 'source /c/Users/<you>/klammertext/env/runtime.env' >> ~/.bashrc
|
echo 'source "/c/Users/<you>/klammertext/env/runtime.env"' >> ~/.bashrc
|
||||||
source ~/.bashrc
|
source ~/.bashrc
|
||||||
```
|
```
|
||||||
|
|
||||||
On Windows it sets `KLAMMERTEXT_HOME` and `TMPDIR` in the form Windows
|
On Windows it sets `KLAMMERTEXT_HOME` and `TMPDIR` in the form Windows
|
||||||
programs understand (`C:/Users/<you>/klammertext`), adds `bin/` to `PATH`,
|
programs understand (`C:/Users/<you>/klammertext`), adds `bin/` to `PATH`,
|
||||||
finds a TeX Live installed under `C:\Users\<you>\external\texlive\<year>`
|
finds a TeX Live installed under `C:\texlive\<year>` (see below), and
|
||||||
(see below), and sets `PYTHON=python`. Nothing is edited by hand.
|
sets `PYTHON=python`. Nothing is edited by hand.
|
||||||
|
|
||||||
|
|
||||||
## Build
|
## Build
|
||||||
@@ -171,26 +177,28 @@ ktext hello.kt -t html # writes hello\index.html
|
|||||||
## TeX Live (for PDF output)
|
## TeX Live (for PDF output)
|
||||||
|
|
||||||
The Standard Klammer Set renders PDF with **XeLaTeX**. Install TeX Live's
|
The Standard Klammer Set renders PDF with **XeLaTeX**. Install TeX Live's
|
||||||
`scheme-small` plus the packages the SKS needs into
|
`scheme-small` plus the packages the SKS needs into `C:\texlive\2026`,
|
||||||
`C:\Users\<you>\external\texlive\2026`, where `runtime.env` finds it. This
|
TeX Live's own default location on Windows, where `runtime.env` finds it.
|
||||||
is a download of about 600 MB and takes twenty to thirty minutes.
|
(It also looks under `C:\Users\<you>\external\texlive\<year>`, the place
|
||||||
|
the Linux and macOS pages use, but TeX Live does not work under a path
|
||||||
|
with a space or an accented letter, and a user name may have either.)
|
||||||
|
This is a download of about 600 MB and takes twenty to thirty minutes.
|
||||||
|
|
||||||
1. Download `install-tl.zip` from a CTAN mirror, for example
|
1. Download `install-tl.zip` from a CTAN mirror, for example
|
||||||
`https://ctan.math.illinois.edu/systems/texlive/tlnet/install-tl.zip`,
|
`https://ctan.math.illinois.edu/systems/texlive/tlnet/install-tl.zip`,
|
||||||
and unzip it. It unpacks into a folder named `install-tl-<date>`.
|
and unzip it. It unpacks into a folder named `install-tl-<date>`.
|
||||||
|
|
||||||
2. In that folder, create a file `texlive.profile` with this content,
|
2. In that folder, create a file `texlive.profile` with this content:
|
||||||
replacing `<you>` with your Windows user name:
|
|
||||||
|
|
||||||
```
|
```
|
||||||
selected_scheme scheme-small
|
selected_scheme scheme-small
|
||||||
TEXDIR C:/Users/<you>/external/texlive/2026
|
TEXDIR C:/texlive/2026
|
||||||
TEXMFLOCAL C:/Users/<you>/external/texlive/2026/texmf-local
|
TEXMFLOCAL C:/texlive/2026/texmf-local
|
||||||
TEXMFSYSCONFIG C:/Users/<you>/external/texlive/2026/texmf-config
|
TEXMFSYSCONFIG C:/texlive/2026/texmf-config
|
||||||
TEXMFSYSVAR C:/Users/<you>/external/texlive/2026/texmf-var
|
TEXMFSYSVAR C:/texlive/2026/texmf-var
|
||||||
TEXMFHOME C:/Users/<you>/texmf
|
TEXMFHOME C:/texlive/2026/texmf-home
|
||||||
TEXMFVAR C:/Users/<you>/.texlive2026/texmf-var
|
TEXMFVAR C:/texlive/2026/texmf-uvar
|
||||||
TEXMFCONFIG C:/Users/<you>/.texlive2026/texmf-config
|
TEXMFCONFIG C:/texlive/2026/texmf-uconfig
|
||||||
tlpdbopt_install_docfiles 0
|
tlpdbopt_install_docfiles 0
|
||||||
tlpdbopt_install_srcfiles 0
|
tlpdbopt_install_srcfiles 0
|
||||||
tlpdbopt_autobackup 0
|
tlpdbopt_autobackup 0
|
||||||
@@ -214,7 +222,7 @@ is a download of about 600 MB and takes twenty to thirty minutes.
|
|||||||
installs on Linux and macOS — and rebuild the formats:
|
installs on Linux and macOS — and rebuild the formats:
|
||||||
|
|
||||||
```powershell
|
```powershell
|
||||||
$tl = "$env:USERPROFILE\external\texlive\2026\bin\windows"
|
$tl = "C:\texlive\2026\bin\windows"
|
||||||
$env:Path = "$tl;$env:Path"
|
$env:Path = "$tl;$env:Path"
|
||||||
& "$tl\tlmgr.bat" option repository https://ctan.math.illinois.edu/systems/texlive/tlnet
|
& "$tl\tlmgr.bat" option repository https://ctan.math.illinois.edu/systems/texlive/tlnet
|
||||||
& "$tl\tlmgr.bat" install adjustbox collectbox collection-fontsrecommended enumitem fontaxes footmisc inconsolata layouts mdframed multirow needspace opensans pict2e textpos titlesec upquote zref
|
& "$tl\tlmgr.bat" install adjustbox collectbox collection-fontsrecommended enumitem fontaxes footmisc inconsolata layouts mdframed multirow needspace opensans pict2e textpos titlesec upquote zref
|
||||||
@@ -244,7 +252,7 @@ from PowerShell):
|
|||||||
```powershell
|
```powershell
|
||||||
[Environment]::SetEnvironmentVariable("KLAMMERTEXT_HOME", "C:\Users\<you>\klammertext", "User")
|
[Environment]::SetEnvironmentVariable("KLAMMERTEXT_HOME", "C:\Users\<you>\klammertext", "User")
|
||||||
$p = [Environment]::GetEnvironmentVariable("Path", "User")
|
$p = [Environment]::GetEnvironmentVariable("Path", "User")
|
||||||
[Environment]::SetEnvironmentVariable("Path", "$p;C:\Users\<you>\klammertext\bin;C:\Users\<you>\external\texlive\2026\bin\windows", "User")
|
[Environment]::SetEnvironmentVariable("Path", "$p;C:\Users\<you>\klammertext\bin;C:\texlive\2026\bin\windows", "User")
|
||||||
```
|
```
|
||||||
|
|
||||||
(Not `setx Path "$env:Path;…"`: `$env:Path` is the machine and user
|
(Not `setx Path "$env:Path;…"`: `$env:Path` is the machine and user
|
||||||
@@ -374,8 +382,15 @@ incomplete or `KLAMMERTEXT_HOME` points elsewhere.
|
|||||||
**"The 'xelatex' command required to make PDF files is not installed".**
|
**"The 'xelatex' command required to make PDF files is not installed".**
|
||||||
TeX Live's `bin\windows` directory is not on the `Path` of the shell you
|
TeX Live's `bin\windows` directory is not on the `Path` of the shell you
|
||||||
are using. In the UCRT64 shell `runtime.env` adds it when the tree is under
|
are using. In the UCRT64 shell `runtime.env` adds it when the tree is under
|
||||||
`C:\Users\<you>\external\texlive\<year>`; in PowerShell add it as shown
|
`C:\texlive\<year>` or `C:\Users\<you>\external\texlive\<year>`; in
|
||||||
above.
|
PowerShell add it as shown above.
|
||||||
|
|
||||||
|
**`make` stops at once with `"python" is C:/msys64/ucrt64/python, not
|
||||||
|
python.org's interpreter`.** MSYS2's own Python was installed as a
|
||||||
|
dependency (poppler brings it), and in the UCRT64 shell it precedes
|
||||||
|
python.org's on the `Path`. `runtime.env` puts python.org's first: source
|
||||||
|
it (a new UCRT64 shell does, once the line is in `~/.bashrc`) and run
|
||||||
|
`make` again.
|
||||||
|
|
||||||
**`git` or `python` are "command not found" in the UCRT64 shell.**
|
**`git` or `python` are "command not found" in the UCRT64 shell.**
|
||||||
`MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before
|
`MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before
|
||||||
|
|||||||
Vendored
+13
@@ -34,6 +34,19 @@ endif
|
|||||||
ifneq (,$(filter MINGW% MSYS%,$(UNAME_S)))
|
ifneq (,$(filter MINGW% MSYS%,$(UNAME_S)))
|
||||||
PYTHON := python
|
PYTHON := python
|
||||||
PYTHON_PREFIX := $(shell cygpath -m "$$($(PYTHON) -c 'import sys; print(sys.base_prefix)')")
|
PYTHON_PREFIX := $(shell cygpath -m "$$($(PYTHON) -c 'import sys; print(sys.base_prefix)')")
|
||||||
|
# The interpreter "python" resolves to must be python.org's, which has an
|
||||||
|
# import library under libs/. MSYS2's own python (installed as a dependency
|
||||||
|
# of poppler, through glib2) sits in /ucrt64/bin ahead of it and has none:
|
||||||
|
# a build against it compiles nothing (no Include/Python.h) or embeds an
|
||||||
|
# interpreter without the user's pip packages. env/runtime.env puts
|
||||||
|
# python.org's directory first; refuse rather than build the wrong thing.
|
||||||
|
# (Asked of the interpreter, not $(wildcard): the prefix "C:/Program Files"
|
||||||
|
# has a space, on which wildcard splits.)
|
||||||
|
ifeq (,$(NOPYTHON))
|
||||||
|
ifneq (yes,$(shell $(PYTHON) -c 'import os, sys; print("yes" if os.path.isdir(os.path.join(sys.base_prefix, "libs")) else "no")'))
|
||||||
|
$(error "python" is $(PYTHON_PREFIX)/python, not python.org's interpreter (no libs/ there); source env/runtime.env, which puts python.org's Python first on PATH)
|
||||||
|
endif
|
||||||
|
endif
|
||||||
PYTHON_INC := -I"$(PYTHON_PREFIX)/Include"
|
PYTHON_INC := -I"$(PYTHON_PREFIX)/Include"
|
||||||
PYTHON_LIB := -L"$(PYTHON_PREFIX)/libs" -l$(shell $(PYTHON) -c 'import sys; print("python%d%d" % sys.version_info[:2])')
|
PYTHON_LIB := -L"$(PYTHON_PREFIX)/libs" -l$(shell $(PYTHON) -c 'import sys; print("python%d%d" % sys.version_info[:2])')
|
||||||
else
|
else
|
||||||
|
|||||||
Vendored
+27
-6
@@ -31,6 +31,21 @@ case "$_kt_uname" in
|
|||||||
export KLAMMERTEXT_HOME="$(cygpath -m "$KLAMMERTEXT_HOME")"
|
export KLAMMERTEXT_HOME="$(cygpath -m "$KLAMMERTEXT_HOME")"
|
||||||
export TMPDIR="$(cygpath -m "${TMPDIR:-/tmp}")"
|
export TMPDIR="$(cygpath -m "${TMPDIR:-/tmp}")"
|
||||||
export PYTHON="${PYTHON:-python}" # python.org installs no python3
|
export PYTHON="${PYTHON:-python}" # python.org installs no python3
|
||||||
|
# "python" must be python.org's interpreter: the build embeds it and
|
||||||
|
# pip installs OpenImageIO into it. But the UCRT64 shell puts
|
||||||
|
# /ucrt64/bin first on PATH, and an MSYS2 package installed as a
|
||||||
|
# dependency (poppler -> glib2 -> python) puts ITS python there, which
|
||||||
|
# then shadows python.org's for make, the suites and the tooling.
|
||||||
|
# python.org's launcher knows only python.org installs: put the
|
||||||
|
# directory of the interpreter it selects first. (Its output ends
|
||||||
|
# in CR LF, hence the tr.)
|
||||||
|
if command -v py >/dev/null 2>&1; then
|
||||||
|
_kt_py="$(py -3 -c 'import sys; print(sys.executable)' 2>/dev/null | tr -d '\r')"
|
||||||
|
if [ -n "$_kt_py" ]; then
|
||||||
|
export PATH="$(dirname "$(cygpath -u "$_kt_py")"):$PATH"
|
||||||
|
fi
|
||||||
|
unset _kt_py
|
||||||
|
fi
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
|
|
||||||
@@ -50,13 +65,19 @@ fi
|
|||||||
# just returns nothing. The -path pattern keeps the 4-digit-year restriction.
|
# just returns nothing. The -path pattern keeps the 4-digit-year restriction.
|
||||||
# Windows: a user installs under the Windows home, %USERPROFILE%\external
|
# Windows: a user installs under the Windows home, %USERPROFILE%\external
|
||||||
# (MSYS2's $HOME is C:\msys64\home\<user>, which no Windows guide would name).
|
# (MSYS2's $HOME is C:\msys64\home\<user>, which no Windows guide would name).
|
||||||
_kt_tlroot="$HOME/external/texlive"
|
# Windows also: TeX Live's own default, C:\texlive\<year> -- the place for a
|
||||||
case "$_kt_uname" in MINGW*|MSYS*) _kt_tlroot="$(cygpath -u "$USERPROFILE")/external/texlive";; esac
|
# user whose profile path holds a space or an accented letter, which TeX
|
||||||
_kt_tl="$(find "$_kt_tlroot" -maxdepth 3 -type d \
|
# Live's installer and kpathsea do not take (2026-09-26). The newest YEAR
|
||||||
-path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null \
|
# wins across both roots (the year is the sort key, not the path).
|
||||||
| sort -V | tail -1)"
|
_kt_tlroot="$HOME/external/texlive"; _kt_tlroot2=""
|
||||||
|
case "$_kt_uname" in MINGW*|MSYS*) _kt_tlroot="$(cygpath -u "$USERPROFILE")/external/texlive"; _kt_tlroot2="/c/texlive";; esac
|
||||||
|
_kt_tl="$( { find "$_kt_tlroot" -maxdepth 3 -type d \
|
||||||
|
-path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null
|
||||||
|
[ -n "$_kt_tlroot2" ] && find "$_kt_tlroot2" -maxdepth 3 -type d \
|
||||||
|
-path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null; } \
|
||||||
|
| awk -F/ '{ print $(NF-2) " " $0 }' | sort -V | tail -1 | cut -d' ' -f2-)"
|
||||||
[ -n "$_kt_tl" ] && export KLAMMERTEXT_TEXLIVE_BIN="$_kt_tl"
|
[ -n "$_kt_tl" ] && export KLAMMERTEXT_TEXLIVE_BIN="$_kt_tl"
|
||||||
unset _kt_tl _kt_arch _kt_tlroot
|
unset _kt_tl _kt_arch _kt_tlroot _kt_tlroot2
|
||||||
|
|
||||||
# --- PATH (TeX Live appended only if one was found) ---------------------------
|
# --- PATH (TeX Live appended only if one was found) ---------------------------
|
||||||
export PATH="$_kt_home_posix/bin:$_kt_home_posix/tst${KLAMMERTEXT_TEXLIVE_BIN:+:$KLAMMERTEXT_TEXLIVE_BIN}:$PATH"
|
export PATH="$_kt_home_posix/bin:$_kt_home_posix/tst${KLAMMERTEXT_TEXLIVE_BIN:+:$KLAMMERTEXT_TEXLIVE_BIN}:$PATH"
|
||||||
|
|||||||
+3
-1
@@ -18,7 +18,9 @@ K := $(KLAMMERTEXT_HOME)
|
|||||||
|
|
||||||
include $(K)/env/makefile.env
|
include $(K)/env/makefile.env
|
||||||
|
|
||||||
LIBDIR := $(K)/lib
|
# Relative, never $(K)/...: the mingw linker cannot open a path holding a
|
||||||
|
# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26).
|
||||||
|
LIBDIR := ../lib
|
||||||
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
||||||
|
|
||||||
EXT_SUFFIX := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))')
|
EXT_SUFFIX := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))')
|
||||||
|
|||||||
+75
-1
@@ -325,7 +325,20 @@ std::string string_from_file(const std::string& pathname, bool strip_surrounding
|
|||||||
} else {
|
} else {
|
||||||
result = buffer.str();
|
result = buffer.str();
|
||||||
|
|
||||||
result = string_replace(result, "\r\n", ""); // Urgh.
|
// A leading UTF-8 byte-order mark is not text: Windows tools
|
||||||
|
// (PowerShell's Set-Content, Notepad's "UTF-8 with BOM") put
|
||||||
|
// one at the start of a file, and without this a .kt beginning
|
||||||
|
// "@document" was reported as "Word not parsed" at 1:1, the
|
||||||
|
// offending word invisible (decided 2026-09-26).
|
||||||
|
if (result.compare(0, 3, "\xEF\xBB\xBF") == 0)
|
||||||
|
result.erase(0, 3);
|
||||||
|
|
||||||
|
// Windows line endings become newlines. (This line DELETED the
|
||||||
|
// pair until 2026-09-26, so on Linux and macOS a file written by
|
||||||
|
// Notepad++ or a DOS-mode Emacs collapsed into one line and its
|
||||||
|
// spans no longer matched; on Windows the C runtime's text mode
|
||||||
|
// had already translated the pair, which hid the bug there.)
|
||||||
|
result = string_replace(result, "\r\n", "\n");
|
||||||
|
|
||||||
|
|
||||||
if (std::regex_match(pathname, klammertext_filename_re)) {
|
if (std::regex_match(pathname, klammertext_filename_re)) {
|
||||||
@@ -575,6 +588,67 @@ void write_to_cache(const std::string& cache_dir, const std::string& basename, c
|
|||||||
string_to_file(output_filename, text);
|
string_to_file(output_filename, text);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The offset of the first byte that is not part of well-formed UTF-8, or
|
||||||
|
// npos. Overlong forms, surrogates and code points above 10FFFF are rejected
|
||||||
|
// like a decoder would (RFC 3629, table 3-7 of the Unicode standard).
|
||||||
|
static std::size_t first_invalid_utf8(const std::string& s)
|
||||||
|
{
|
||||||
|
const auto n = s.size();
|
||||||
|
for (std::size_t i = 0; i < n;) {
|
||||||
|
const unsigned char b = static_cast<unsigned char>(s[i]);
|
||||||
|
std::size_t len; unsigned char lo = 0x80, hi = 0xBF;
|
||||||
|
if (b < 0x80) { i += 1; continue; }
|
||||||
|
else if (b >= 0xC2 && b <= 0xDF) len = 2;
|
||||||
|
else if (b == 0xE0) { len = 3; lo = 0xA0; }
|
||||||
|
else if (b >= 0xE1 && b <= 0xEC) len = 3;
|
||||||
|
else if (b == 0xED) { len = 3; hi = 0x9F; }
|
||||||
|
else if (b >= 0xEE && b <= 0xEF) len = 3;
|
||||||
|
else if (b == 0xF0) { len = 4; lo = 0x90; }
|
||||||
|
else if (b >= 0xF1 && b <= 0xF3) len = 4;
|
||||||
|
else if (b == 0xF4) { len = 4; hi = 0x8F; }
|
||||||
|
else return i;
|
||||||
|
if (i + len > n) return i;
|
||||||
|
for (std::size_t k = 1; k < len; ++k) {
|
||||||
|
const unsigned char c = static_cast<unsigned char>(s[i + k]);
|
||||||
|
const unsigned char l = (k == 1) ? lo : 0x80, h = (k == 1) ? hi : 0xBF;
|
||||||
|
if (c < l || c > h) return i;
|
||||||
|
}
|
||||||
|
i += len;
|
||||||
|
}
|
||||||
|
return std::string::npos;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string klammertext_from_file(const std::string& pathname)
|
||||||
|
{
|
||||||
|
std::string text = string_from_file(pathname);
|
||||||
|
// A UTF-16 file (Notepad++ offers "UCS-2 LE BOM") begins with FF FE or
|
||||||
|
// FE FF; read as bytes it is NUL-interleaved and every word is "not
|
||||||
|
// parsed". Name the encoding and the remedy instead.
|
||||||
|
if (text.size() >= 2 && ((static_cast<unsigned char>(text[0]) == 0xFF && static_cast<unsigned char>(text[1]) == 0xFE)
|
||||||
|
|| (static_cast<unsigned char>(text[0]) == 0xFE && static_cast<unsigned char>(text[1]) == 0xFF))) {
|
||||||
|
throw File_error("File \"" + pathname + "\" is UTF-16 (it begins with a byte-order mark), and Klammertext reads UTF-8. "
|
||||||
|
"Save it as UTF-8 in your editor (Notepad++: Encoding, Convert to UTF-8) and run the command again.",
|
||||||
|
Locator(fs::path(pathname), 1, 0)); // the Locator's character counts from 0
|
||||||
|
}
|
||||||
|
// A file saved as ANSI (Windows-1252, the default of some editors) is
|
||||||
|
// not UTF-8 from its first accented letter on; the bytes otherwise
|
||||||
|
// reach the embedded Python and fail there with a message about
|
||||||
|
// something else entirely.
|
||||||
|
if (const std::size_t bad = first_invalid_utf8(text); bad != std::string::npos) {
|
||||||
|
int line = 1, chr = 1;
|
||||||
|
for (std::size_t i = 0; i < bad; ++i) {
|
||||||
|
if (text[i] == '\n') { ++line; chr = 1; }
|
||||||
|
else if ((static_cast<unsigned char>(text[i]) & 0xC0) != 0x80) ++chr; // count characters, not bytes
|
||||||
|
}
|
||||||
|
char hex[8]; std::snprintf(hex, sizeof hex, "%02X", static_cast<unsigned char>(text[bad]));
|
||||||
|
throw File_error("File \"" + pathname + "\" is not UTF-8 at line " + std::to_string(line) + ", character "
|
||||||
|
+ std::to_string(chr) + " (byte " + hex + "), and Klammertext reads UTF-8. "
|
||||||
|
"Save it as UTF-8 in your editor (Notepad++: Encoding, Convert to UTF-8) and run the command again.",
|
||||||
|
Locator(fs::path(pathname), line, chr - 1));
|
||||||
|
}
|
||||||
|
return text;
|
||||||
|
}
|
||||||
|
|
||||||
std::string read_from_cache(const std::string& cache_dir, const std::string& basename)
|
std::string read_from_cache(const std::string& cache_dir, const std::string& basename)
|
||||||
{
|
{
|
||||||
std::string input_filename = cache_dir + "/" + basename;
|
std::string input_filename = cache_dir + "/" + basename;
|
||||||
|
|||||||
@@ -70,6 +70,12 @@ std::vector<std::string> group_filename_tokens(
|
|||||||
std::vector<std::string> resolve_filename_list(
|
std::vector<std::string> resolve_filename_list(
|
||||||
const std::string& text, const std::string& base_dir="");
|
const std::string& text, const std::string& base_dir="");
|
||||||
std::string string_from_file(const std::string& pathname, bool strip_surrounding_whitespace=false);
|
std::string string_from_file(const std::string& pathname, bool strip_surrounding_whitespace=false);
|
||||||
|
// A Klammertext source (a .kt, a .k, a @read file): string_from_file() plus
|
||||||
|
// the check that it IS text Klammertext can read -- a File_error names the
|
||||||
|
// first byte that is not UTF-8 (line and character), or a UTF-16 file by its
|
||||||
|
// byte-order mark, with what to do in an editor. string_from_file() itself
|
||||||
|
// stays permissive: it also reads XeTeX logs and a shell command's stderr.
|
||||||
|
std::string klammertext_from_file(const std::string& pathname);
|
||||||
void string_to_file(const std::string& pathname, std::string contents);
|
void string_to_file(const std::string& pathname, std::string contents);
|
||||||
std::vector<std::string> get_files_in_directory(const std::string& dir);
|
std::vector<std::string> get_files_in_directory(const std::string& dir);
|
||||||
std::string find_file(const std::string& basename, std::vector<std::string> search_path,
|
std::string find_file(const std::string& basename, std::vector<std::string> search_path,
|
||||||
|
|||||||
@@ -221,7 +221,7 @@ static std::string klammerset_description(const fs::path& declaration)
|
|||||||
{
|
{
|
||||||
std::string text {};
|
std::string text {};
|
||||||
try {
|
try {
|
||||||
text = string_from_file(declaration.string());
|
text = klammertext_from_file(declaration.string());
|
||||||
} catch (...) {
|
} catch (...) {
|
||||||
return "";
|
return "";
|
||||||
}
|
}
|
||||||
|
|||||||
+3
-3
@@ -491,7 +491,7 @@ void Machine::read(const fs::path& pathname)
|
|||||||
// :cpp libraries live next to the file that uses them).
|
// :cpp libraries live next to the file that uses them).
|
||||||
m_state.add_search_dir(fs::absolute(pathname).parent_path().string());
|
m_state.add_search_dir(fs::absolute(pathname).parent_path().string());
|
||||||
m_state.open_frame("Machine state: " + pathname.string());
|
m_state.open_frame("Machine state: " + pathname.string());
|
||||||
input(string_from_file(pathname.string()), pathname.generic_string());
|
input(klammertext_from_file(pathname.string()), pathname.generic_string());
|
||||||
m_sources.push_back(pathname);
|
m_sources.push_back(pathname);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -552,7 +552,7 @@ void Machine::expand_read_katoms(
|
|||||||
|
|
||||||
input_filename = fs::canonical(input_filename);
|
input_filename = fs::canonical(input_filename);
|
||||||
m_state.add_search_dir(input_filename.parent_path().string());
|
m_state.add_search_dir(input_filename.parent_path().string());
|
||||||
std::string text = trim_right(string_from_file(input_filename.string()));
|
std::string text = trim_right(klammertext_from_file(input_filename.string()));
|
||||||
katom_list ks = katomize_lines(line_split(text), input_filename.generic_string());
|
katom_list ks = katomize_lines(line_split(text), input_filename.generic_string());
|
||||||
// ks =
|
// ks =
|
||||||
process_katoms(
|
process_katoms(
|
||||||
@@ -684,7 +684,7 @@ void Machine::load_klammerset_files(const Klammerset& klammerset, katom_iter ins
|
|||||||
pathname = fs::canonical(pathname);
|
pathname = fs::canonical(pathname);
|
||||||
loaded_files.push_back(pathname.generic_string());
|
loaded_files.push_back(pathname.generic_string());
|
||||||
m_state.add_search_dir(pathname.parent_path().string());
|
m_state.add_search_dir(pathname.parent_path().string());
|
||||||
std::string text = trim_right(string_from_file(pathname.string()));
|
std::string text = trim_right(klammertext_from_file(pathname.string()));
|
||||||
katom_list ks = katomize_lines(line_split(text), pathname.generic_string());
|
katom_list ks = katomize_lines(line_split(text), pathname.generic_string());
|
||||||
process_katoms(ks, pathname.generic_string());
|
process_katoms(ks, pathname.generic_string());
|
||||||
loaded.insert(loaded.end(), ks.begin(), ks.end());
|
loaded.insert(loaded.end(), ks.begin(), ks.end());
|
||||||
|
|||||||
+3
-1
@@ -19,7 +19,9 @@ include $(K)/env/makefile.env
|
|||||||
# CPPFLAGS already carries -I$(KLAMMERTEXT_HOME)/mac, which is every header
|
# CPPFLAGS already carries -I$(KLAMMERTEXT_HOME)/mac, which is every header
|
||||||
# reflect.cpp includes (machine, character, error, show). No local -I.
|
# reflect.cpp includes (machine, character, error, show). No local -I.
|
||||||
|
|
||||||
LIBDIR := $(K)/lib
|
# Relative, never $(K)/...: the mingw linker cannot open a path holding a
|
||||||
|
# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26).
|
||||||
|
LIBDIR := ../lib
|
||||||
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
||||||
REFLECT := reflect$(DSO)
|
REFLECT := reflect$(DSO)
|
||||||
|
|
||||||
|
|||||||
@@ -11,7 +11,9 @@ include $(K)/env/makefile.env
|
|||||||
LOCAL_CPPFLAGS := -I$(KM) -I$(KS)/kutil -I$(KS)/target
|
LOCAL_CPPFLAGS := -I$(KM) -I$(KS)/kutil -I$(KS)/target
|
||||||
|
|
||||||
# Shared library location
|
# Shared library location
|
||||||
LIBDIR := $(K)/lib
|
# Relative, never $(K)/...: the mingw linker cannot open a path holding a
|
||||||
|
# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26).
|
||||||
|
LIBDIR := ../../lib
|
||||||
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE)
|
||||||
|
|
||||||
# The product: document.so, on Windows document.dll (makefile.env's DSO)
|
# The product: document.so, on Windows document.dll (makefile.env's DSO)
|
||||||
@@ -25,8 +27,8 @@ LOCAL_OBJECTS := document_class.o document_html.o document_latex.o heading.o ref
|
|||||||
LOCAL_DEPFILES := document_class.d document_html.d document_latex.d heading.d reference.d document.d
|
LOCAL_DEPFILES := document_class.d document_html.d document_latex.d heading.d reference.d document.d
|
||||||
|
|
||||||
# External object files from sks/ (mac/*.o now in the shared library)
|
# External object files from sks/ (mac/*.o now in the shared library)
|
||||||
SKS_OBJECTS := $(KS)/kutil/kutil.o $(KS)/kutil/klammer_base.o \
|
SKS_OBJECTS := ../kutil/kutil.o ../kutil/klammer_base.o \
|
||||||
$(KS)/target/html_util.o $(KS)/target/latex_util.o
|
../target/html_util.o ../target/latex_util.o
|
||||||
|
|
||||||
ALL_OBJECTS := $(LOCAL_OBJECTS) $(SKS_OBJECTS)
|
ALL_OBJECTS := $(LOCAL_OBJECTS) $(SKS_OBJECTS)
|
||||||
|
|
||||||
|
|||||||
@@ -36,6 +36,10 @@ mkdir "$DIR/my dir"
|
|||||||
printf 'ALPHA\n' > "$DIR/my file.kt"
|
printf 'ALPHA\n' > "$DIR/my file.kt"
|
||||||
printf 'BETA\n' > "$DIR/b.kt"
|
printf 'BETA\n' > "$DIR/b.kt"
|
||||||
printf 'GAMMA\n' > "$DIR/my dir/c.kt"
|
printf 'GAMMA\n' > "$DIR/my dir/c.kt"
|
||||||
|
printf '\357\273\277DELTA\n' > "$DIR/bom.kt" # a UTF-8 byte-order mark, then the text
|
||||||
|
printf 'EPSILON\r\nZETA\r\n' > "$DIR/crlf.kt" # Windows line endings
|
||||||
|
printf '\377\376h\000i\000\n\000' > "$DIR/utf16.kt" # UTF-16LE with its byte-order mark
|
||||||
|
printf 'ETA\nTHETA M\374ller\n' > "$DIR/cp1252.kt" # Windows-1252: an umlaut that is not UTF-8
|
||||||
cd "$DIR" || exit 1
|
cd "$DIR" || exit 1
|
||||||
|
|
||||||
# check TEST_NAME EXPECTED_STDOUT EXPECTED_STDERR_SUBSTRING KTEXT_ARGS...
|
# check TEST_NAME EXPECTED_STDOUT EXPECTED_STDERR_SUBSTRING KTEXT_ARGS...
|
||||||
@@ -151,6 +155,37 @@ check_error "8. unrescuable name is reported as written" \
|
|||||||
"no such.kt" \
|
"no such.kt" \
|
||||||
"no such.kt" -d --klammersets none
|
"no such.kt" -d --klammersets none
|
||||||
|
|
||||||
|
# A leading UTF-8 byte-order mark is ignored (2026-09-26): Windows tools write
|
||||||
|
# one, and it was reported as "Word not parsed" at 1:1 with an invisible word.
|
||||||
|
check "9. a leading UTF-8 BOM in the input file is ignored" \
|
||||||
|
"DELTA" "" \
|
||||||
|
bom.kt -d --klammersets none
|
||||||
|
|
||||||
|
check "10. a leading UTF-8 BOM in a @read file is ignored" \
|
||||||
|
"DELTA" "" \
|
||||||
|
-s '@read bom.kt @' -d --klammersets none
|
||||||
|
|
||||||
|
# Windows line endings are newlines (2026-09-26): the reader DELETED the pair,
|
||||||
|
# so on Linux and macOS a CRLF file collapsed into one line.
|
||||||
|
check "11. CRLF line endings are read as newlines" \
|
||||||
|
$'EPSILON\nZETA' "" \
|
||||||
|
crlf.kt -d --klammersets none
|
||||||
|
|
||||||
|
# What a Windows editor can also save (2026-09-26): a UTF-16 file and an
|
||||||
|
# ANSI (Windows-1252) file are named as such, with the remedy, instead of
|
||||||
|
# "Word not parsed" or a failure inside the embedded Python.
|
||||||
|
check_error "12. a UTF-16 file is named as such" \
|
||||||
|
"UTF-16" \
|
||||||
|
utf16.kt -d --klammersets none
|
||||||
|
|
||||||
|
check_error "13. a Windows-1252 file is located at its first non-UTF-8 byte" \
|
||||||
|
"line 2, character 8" \
|
||||||
|
cp1252.kt -d --klammersets none
|
||||||
|
|
||||||
|
check_error "14. ... through @read as well" \
|
||||||
|
"not UTF-8" \
|
||||||
|
-s '@read cp1252.kt @' -d --klammersets none
|
||||||
|
|
||||||
echo
|
echo
|
||||||
echo "======================="
|
echo "======================="
|
||||||
echo "Results: ${green}$PASS passed${reset}, ${red}$FAIL failed${reset}"
|
echo "Results: ${green}$PASS passed${reset}, ${red}$FAIL failed${reset}"
|
||||||
|
|||||||
@@ -49,6 +49,19 @@ trap 'rm -rf "$OUTF" "$ERRF" "$WORK"' EXIT
|
|||||||
if command -v cygpath >/dev/null 2>&1; then sep=';'; else sep=':'; fi
|
if command -v cygpath >/dev/null 2>&1; then sep=';'; else sep=':'; fi
|
||||||
export PYTHONPATH="$K/kpy${PYTHONPATH:+$sep$PYTHONPATH}"
|
export PYTHONPATH="$K/kpy${PYTHONPATH:+$sep$PYTHONPATH}"
|
||||||
|
|
||||||
|
# On Windows the binding cannot load under a KLAMMERTEXT_HOME that holds a
|
||||||
|
# non-ASCII character: inside python.exe, a process without Klammertext's
|
||||||
|
# UTF-8 manifest, the narrow getenv() is the ANSI code page, and the engine
|
||||||
|
# takes every path as UTF-8. A documented limitation, not a defect to fix
|
||||||
|
# (TODO 70 (f), Andy 2026-09-26): the binding needs an ASCII installation
|
||||||
|
# path there. Say so and skip, rather than fail every case with a decode
|
||||||
|
# error that names no cause.
|
||||||
|
if command -v cygpath >/dev/null 2>&1 && LC_ALL=C grep -q '[^ -~]' <<<"$K"; then
|
||||||
|
echo "SKIP kpy_test: KLAMMERTEXT_HOME ($K) holds a non-ASCII character, and the Python binding needs an ASCII installation path on Windows (TODO 70 (f))"
|
||||||
|
echo "Passed: 0, Failed: 0"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
# A DEBUG=1 build instruments the module with AddressSanitizer, and an
|
# A DEBUG=1 build instruments the module with AddressSanitizer, and an
|
||||||
# instrumented extension imported into an uninstrumented python needs the
|
# instrumented extension imported into an uninstrumented python needs the
|
||||||
# ASan runtime loaded FIRST -- the import otherwise fails with "ASan runtime
|
# ASan runtime loaded FIRST -- the import otherwise fails with "ASan runtime
|
||||||
|
|||||||
Reference in new issue
Block a user