diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..0d7b5d8 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,9 @@ +# Every text file is LF in the working tree on every system, whatever +# core.autocrlf says: .kt, .k and the shell suites must be LF (2026-09-26). +* text=auto eol=lf +# Shipped verbatim, as the foundries wrote them (CRLF): never normalized. +fnt/**/*.txt -text +sks/document/fonts/**/*.txt -text +# Not shipped; kept byte-stable. +dbg/** -text +sks.alpha/** -text diff --git a/README.md b/README.md index 657cd11..faec703 100644 --- a/README.md +++ b/README.md @@ -53,7 +53,7 @@ are regenerated on each release — patches cannot be merged directly. Report problems (or send patches) to the author; accepted changes are applied to the development tree and appear in a following snapshot. -This snapshot was assembled from development commit `d5f9c20b94b3`. +This snapshot was assembled from development commit `9d6f66456a3d`. ## License diff --git a/doc/edit/notepadpp/README.md b/doc/edit/notepadpp/README.md index d7cdbb3..c2f8fd8 100644 --- a/doc/edit/notepadpp/README.md +++ b/doc/edit/notepadpp/README.md @@ -26,6 +26,9 @@ current file, and this package uses exactly those two things. version qualifies). - For the Run commands: `python` on the `Path`, which the Klammertext installation already needs. +- Files saved as UTF-8 (Notepad++'s default for new files; **Encoding ￫ + Convert to UTF-8** for an older one). Line endings do not matter, and + a file in another encoding is refused by `ktext` with the remedy named. ## Install the highlighting diff --git a/doc/install/windows_source_install.md b/doc/install/windows_source_install.md index ea61715..c164194 100644 --- a/doc/install/windows_source_install.md +++ b/doc/install/windows_source_install.md @@ -97,11 +97,17 @@ git clone https://git.andykopra.com/ack/klammertext.git cd klammertext ``` -Git for Windows must not translate line endings: Klammertext's `.kt` -files, `.k` files and shell scripts are LF files. The clone above respects -that as long as `core.autocrlf` is not `true`; check with `git config ---get core.autocrlf` and, if it prints `true`, set it to `false` and clone -again. +The repository's path must contain no space: GNU Make cannot build under +one. If your user name has a space in it (`C:\Users\Anna Maria`), clone +somewhere else, for example into `C:\klammertext`, and write that path +wherever `C:\Users\\klammertext` appears below. Accented letters in +the path are fine for the commands; only the Python binding (`kpy\`) needs +a path of plain ASCII letters. + +Klammertext's `.kt` files, `.k` files and shell scripts are LF files, and +the repository says so itself (`.gitattributes`), so the clone is correct +whatever your Git's `core.autocrlf` setting. A `.kt` you write with +Windows line endings is fine too: ktext reads them as newlines. ## Environment variables @@ -110,14 +116,14 @@ Klammertext's runtime environment is one self-configuring file, the same one Linux and macOS use. Source it from the UCRT64 shell's profile: ```bash -echo 'source /c/Users//klammertext/env/runtime.env' >> ~/.bashrc +echo 'source "/c/Users//klammertext/env/runtime.env"' >> ~/.bashrc source ~/.bashrc ``` On Windows it sets `KLAMMERTEXT_HOME` and `TMPDIR` in the form Windows programs understand (`C:/Users//klammertext`), adds `bin/` to `PATH`, -finds a TeX Live installed under `C:\Users\\external\texlive\` -(see below), and sets `PYTHON=python`. Nothing is edited by hand. +finds a TeX Live installed under `C:\texlive\` (see below), and +sets `PYTHON=python`. Nothing is edited by hand. ## Build @@ -171,26 +177,28 @@ ktext hello.kt -t html # writes hello\index.html ## TeX Live (for PDF output) The Standard Klammer Set renders PDF with **XeLaTeX**. Install TeX Live's -`scheme-small` plus the packages the SKS needs into -`C:\Users\\external\texlive\2026`, where `runtime.env` finds it. This -is a download of about 600 MB and takes twenty to thirty minutes. +`scheme-small` plus the packages the SKS needs into `C:\texlive\2026`, +TeX Live's own default location on Windows, where `runtime.env` finds it. +(It also looks under `C:\Users\\external\texlive\`, the place +the Linux and macOS pages use, but TeX Live does not work under a path +with a space or an accented letter, and a user name may have either.) +This is a download of about 600 MB and takes twenty to thirty minutes. 1. Download `install-tl.zip` from a CTAN mirror, for example `https://ctan.math.illinois.edu/systems/texlive/tlnet/install-tl.zip`, and unzip it. It unpacks into a folder named `install-tl-`. -2. In that folder, create a file `texlive.profile` with this content, - replacing `` with your Windows user name: +2. In that folder, create a file `texlive.profile` with this content: ``` selected_scheme scheme-small - TEXDIR C:/Users//external/texlive/2026 - TEXMFLOCAL C:/Users//external/texlive/2026/texmf-local - TEXMFSYSCONFIG C:/Users//external/texlive/2026/texmf-config - TEXMFSYSVAR C:/Users//external/texlive/2026/texmf-var - TEXMFHOME C:/Users//texmf - TEXMFVAR C:/Users//.texlive2026/texmf-var - TEXMFCONFIG C:/Users//.texlive2026/texmf-config + TEXDIR C:/texlive/2026 + TEXMFLOCAL C:/texlive/2026/texmf-local + TEXMFSYSCONFIG C:/texlive/2026/texmf-config + TEXMFSYSVAR C:/texlive/2026/texmf-var + TEXMFHOME C:/texlive/2026/texmf-home + TEXMFVAR C:/texlive/2026/texmf-uvar + TEXMFCONFIG C:/texlive/2026/texmf-uconfig tlpdbopt_install_docfiles 0 tlpdbopt_install_srcfiles 0 tlpdbopt_autobackup 0 @@ -214,7 +222,7 @@ is a download of about 600 MB and takes twenty to thirty minutes. installs on Linux and macOS — and rebuild the formats: ```powershell - $tl = "$env:USERPROFILE\external\texlive\2026\bin\windows" + $tl = "C:\texlive\2026\bin\windows" $env:Path = "$tl;$env:Path" & "$tl\tlmgr.bat" option repository https://ctan.math.illinois.edu/systems/texlive/tlnet & "$tl\tlmgr.bat" install adjustbox collectbox collection-fontsrecommended enumitem fontaxes footmisc inconsolata layouts mdframed multirow needspace opensans pict2e textpos titlesec upquote zref @@ -244,7 +252,7 @@ from PowerShell): ```powershell [Environment]::SetEnvironmentVariable("KLAMMERTEXT_HOME", "C:\Users\\klammertext", "User") $p = [Environment]::GetEnvironmentVariable("Path", "User") -[Environment]::SetEnvironmentVariable("Path", "$p;C:\Users\\klammertext\bin;C:\Users\\external\texlive\2026\bin\windows", "User") +[Environment]::SetEnvironmentVariable("Path", "$p;C:\Users\\klammertext\bin;C:\texlive\2026\bin\windows", "User") ``` (Not `setx Path "$env:Path;…"`: `$env:Path` is the machine and user @@ -374,8 +382,15 @@ incomplete or `KLAMMERTEXT_HOME` points elsewhere. **"The 'xelatex' command required to make PDF files is not installed".** TeX Live's `bin\windows` directory is not on the `Path` of the shell you are using. In the UCRT64 shell `runtime.env` adds it when the tree is under -`C:\Users\\external\texlive\`; in PowerShell add it as shown -above. +`C:\texlive\` or `C:\Users\\external\texlive\`; in +PowerShell add it as shown above. + +**`make` stops at once with `"python" is C:/msys64/ucrt64/python, not +python.org's interpreter`.** MSYS2's own Python was installed as a +dependency (poppler brings it), and in the UCRT64 shell it precedes +python.org's on the `Path`. `runtime.env` puts python.org's first: source +it (a new UCRT64 shell does, once the line is in `~/.bashrc`) and run +`make` again. **`git` or `python` are "command not found" in the UCRT64 shell.** `MSYS2_PATH_TYPE` is not set to `inherit`, or the shell was opened before diff --git a/env/makefile.env b/env/makefile.env index 8c847b3..4019985 100644 --- a/env/makefile.env +++ b/env/makefile.env @@ -34,6 +34,19 @@ endif ifneq (,$(filter MINGW% MSYS%,$(UNAME_S))) PYTHON := python PYTHON_PREFIX := $(shell cygpath -m "$$($(PYTHON) -c 'import sys; print(sys.base_prefix)')") +# The interpreter "python" resolves to must be python.org's, which has an +# import library under libs/. MSYS2's own python (installed as a dependency +# of poppler, through glib2) sits in /ucrt64/bin ahead of it and has none: +# a build against it compiles nothing (no Include/Python.h) or embeds an +# interpreter without the user's pip packages. env/runtime.env puts +# python.org's directory first; refuse rather than build the wrong thing. +# (Asked of the interpreter, not $(wildcard): the prefix "C:/Program Files" +# has a space, on which wildcard splits.) +ifeq (,$(NOPYTHON)) +ifneq (yes,$(shell $(PYTHON) -c 'import os, sys; print("yes" if os.path.isdir(os.path.join(sys.base_prefix, "libs")) else "no")')) +$(error "python" is $(PYTHON_PREFIX)/python, not python.org's interpreter (no libs/ there); source env/runtime.env, which puts python.org's Python first on PATH) +endif +endif PYTHON_INC := -I"$(PYTHON_PREFIX)/Include" PYTHON_LIB := -L"$(PYTHON_PREFIX)/libs" -l$(shell $(PYTHON) -c 'import sys; print("python%d%d" % sys.version_info[:2])') else diff --git a/env/runtime.env b/env/runtime.env index 3752a22..8323bca 100644 --- a/env/runtime.env +++ b/env/runtime.env @@ -31,6 +31,21 @@ case "$_kt_uname" in export KLAMMERTEXT_HOME="$(cygpath -m "$KLAMMERTEXT_HOME")" export TMPDIR="$(cygpath -m "${TMPDIR:-/tmp}")" export PYTHON="${PYTHON:-python}" # python.org installs no python3 + # "python" must be python.org's interpreter: the build embeds it and + # pip installs OpenImageIO into it. But the UCRT64 shell puts + # /ucrt64/bin first on PATH, and an MSYS2 package installed as a + # dependency (poppler -> glib2 -> python) puts ITS python there, which + # then shadows python.org's for make, the suites and the tooling. + # python.org's launcher knows only python.org installs: put the + # directory of the interpreter it selects first. (Its output ends + # in CR LF, hence the tr.) + if command -v py >/dev/null 2>&1; then + _kt_py="$(py -3 -c 'import sys; print(sys.executable)' 2>/dev/null | tr -d '\r')" + if [ -n "$_kt_py" ]; then + export PATH="$(dirname "$(cygpath -u "$_kt_py")"):$PATH" + fi + unset _kt_py + fi ;; esac @@ -50,13 +65,19 @@ fi # just returns nothing. The -path pattern keeps the 4-digit-year restriction. # Windows: a user installs under the Windows home, %USERPROFILE%\external # (MSYS2's $HOME is C:\msys64\home\, which no Windows guide would name). -_kt_tlroot="$HOME/external/texlive" -case "$_kt_uname" in MINGW*|MSYS*) _kt_tlroot="$(cygpath -u "$USERPROFILE")/external/texlive";; esac -_kt_tl="$(find "$_kt_tlroot" -maxdepth 3 -type d \ - -path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null \ - | sort -V | tail -1)" +# Windows also: TeX Live's own default, C:\texlive\ -- the place for a +# user whose profile path holds a space or an accented letter, which TeX +# Live's installer and kpathsea do not take (2026-09-26). The newest YEAR +# wins across both roots (the year is the sort key, not the path). +_kt_tlroot="$HOME/external/texlive"; _kt_tlroot2="" +case "$_kt_uname" in MINGW*|MSYS*) _kt_tlroot="$(cygpath -u "$USERPROFILE")/external/texlive"; _kt_tlroot2="/c/texlive";; esac +_kt_tl="$( { find "$_kt_tlroot" -maxdepth 3 -type d \ + -path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null + [ -n "$_kt_tlroot2" ] && find "$_kt_tlroot2" -maxdepth 3 -type d \ + -path "*/[0-9][0-9][0-9][0-9]/bin/$_kt_arch" 2>/dev/null; } \ + | awk -F/ '{ print $(NF-2) " " $0 }' | sort -V | tail -1 | cut -d' ' -f2-)" [ -n "$_kt_tl" ] && export KLAMMERTEXT_TEXLIVE_BIN="$_kt_tl" -unset _kt_tl _kt_arch _kt_tlroot +unset _kt_tl _kt_arch _kt_tlroot _kt_tlroot2 # --- PATH (TeX Live appended only if one was found) --------------------------- export PATH="$_kt_home_posix/bin:$_kt_home_posix/tst${KLAMMERTEXT_TEXLIVE_BIN:+:$KLAMMERTEXT_TEXLIVE_BIN}:$PATH" diff --git a/kpy/Makefile b/kpy/Makefile index 2a6ba59..809138f 100644 --- a/kpy/Makefile +++ b/kpy/Makefile @@ -18,7 +18,9 @@ K := $(KLAMMERTEXT_HOME) include $(K)/env/makefile.env -LIBDIR := $(K)/lib +# Relative, never $(K)/...: the mingw linker cannot open a path holding a +# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26). +LIBDIR := ../lib LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE) EXT_SUFFIX := $(shell $(PYTHON) -c 'import sysconfig; print(sysconfig.get_config_var("EXT_SUFFIX"))') diff --git a/mac/file.cpp b/mac/file.cpp index d82fd7c..a6889fe 100644 --- a/mac/file.cpp +++ b/mac/file.cpp @@ -325,7 +325,20 @@ std::string string_from_file(const std::string& pathname, bool strip_surrounding } else { result = buffer.str(); - result = string_replace(result, "\r\n", ""); // Urgh. + // A leading UTF-8 byte-order mark is not text: Windows tools + // (PowerShell's Set-Content, Notepad's "UTF-8 with BOM") put + // one at the start of a file, and without this a .kt beginning + // "@document" was reported as "Word not parsed" at 1:1, the + // offending word invisible (decided 2026-09-26). + if (result.compare(0, 3, "\xEF\xBB\xBF") == 0) + result.erase(0, 3); + + // Windows line endings become newlines. (This line DELETED the + // pair until 2026-09-26, so on Linux and macOS a file written by + // Notepad++ or a DOS-mode Emacs collapsed into one line and its + // spans no longer matched; on Windows the C runtime's text mode + // had already translated the pair, which hid the bug there.) + result = string_replace(result, "\r\n", "\n"); if (std::regex_match(pathname, klammertext_filename_re)) { @@ -575,6 +588,67 @@ void write_to_cache(const std::string& cache_dir, const std::string& basename, c string_to_file(output_filename, text); } +// The offset of the first byte that is not part of well-formed UTF-8, or +// npos. Overlong forms, surrogates and code points above 10FFFF are rejected +// like a decoder would (RFC 3629, table 3-7 of the Unicode standard). +static std::size_t first_invalid_utf8(const std::string& s) +{ + const auto n = s.size(); + for (std::size_t i = 0; i < n;) { + const unsigned char b = static_cast(s[i]); + std::size_t len; unsigned char lo = 0x80, hi = 0xBF; + if (b < 0x80) { i += 1; continue; } + else if (b >= 0xC2 && b <= 0xDF) len = 2; + else if (b == 0xE0) { len = 3; lo = 0xA0; } + else if (b >= 0xE1 && b <= 0xEC) len = 3; + else if (b == 0xED) { len = 3; hi = 0x9F; } + else if (b >= 0xEE && b <= 0xEF) len = 3; + else if (b == 0xF0) { len = 4; lo = 0x90; } + else if (b >= 0xF1 && b <= 0xF3) len = 4; + else if (b == 0xF4) { len = 4; hi = 0x8F; } + else return i; + if (i + len > n) return i; + for (std::size_t k = 1; k < len; ++k) { + const unsigned char c = static_cast(s[i + k]); + const unsigned char l = (k == 1) ? lo : 0x80, h = (k == 1) ? hi : 0xBF; + if (c < l || c > h) return i; + } + i += len; + } + return std::string::npos; +} + +std::string klammertext_from_file(const std::string& pathname) +{ + std::string text = string_from_file(pathname); + // A UTF-16 file (Notepad++ offers "UCS-2 LE BOM") begins with FF FE or + // FE FF; read as bytes it is NUL-interleaved and every word is "not + // parsed". Name the encoding and the remedy instead. + if (text.size() >= 2 && ((static_cast(text[0]) == 0xFF && static_cast(text[1]) == 0xFE) + || (static_cast(text[0]) == 0xFE && static_cast(text[1]) == 0xFF))) { + throw File_error("File \"" + pathname + "\" is UTF-16 (it begins with a byte-order mark), and Klammertext reads UTF-8. " + "Save it as UTF-8 in your editor (Notepad++: Encoding, Convert to UTF-8) and run the command again.", + Locator(fs::path(pathname), 1, 0)); // the Locator's character counts from 0 + } + // A file saved as ANSI (Windows-1252, the default of some editors) is + // not UTF-8 from its first accented letter on; the bytes otherwise + // reach the embedded Python and fail there with a message about + // something else entirely. + if (const std::size_t bad = first_invalid_utf8(text); bad != std::string::npos) { + int line = 1, chr = 1; + for (std::size_t i = 0; i < bad; ++i) { + if (text[i] == '\n') { ++line; chr = 1; } + else if ((static_cast(text[i]) & 0xC0) != 0x80) ++chr; // count characters, not bytes + } + char hex[8]; std::snprintf(hex, sizeof hex, "%02X", static_cast(text[bad])); + throw File_error("File \"" + pathname + "\" is not UTF-8 at line " + std::to_string(line) + ", character " + + std::to_string(chr) + " (byte " + hex + "), and Klammertext reads UTF-8. " + "Save it as UTF-8 in your editor (Notepad++: Encoding, Convert to UTF-8) and run the command again.", + Locator(fs::path(pathname), line, chr - 1)); + } + return text; +} + std::string read_from_cache(const std::string& cache_dir, const std::string& basename) { std::string input_filename = cache_dir + "/" + basename; diff --git a/mac/file.h b/mac/file.h index 7554ea1..3b649b5 100644 --- a/mac/file.h +++ b/mac/file.h @@ -70,6 +70,12 @@ std::vector group_filename_tokens( std::vector resolve_filename_list( const std::string& text, const std::string& base_dir=""); std::string string_from_file(const std::string& pathname, bool strip_surrounding_whitespace=false); +// A Klammertext source (a .kt, a .k, a @read file): string_from_file() plus +// the check that it IS text Klammertext can read -- a File_error names the +// first byte that is not UTF-8 (line and character), or a UTF-16 file by its +// byte-order mark, with what to do in an editor. string_from_file() itself +// stays permissive: it also reads XeTeX logs and a shell command's stderr. +std::string klammertext_from_file(const std::string& pathname); void string_to_file(const std::string& pathname, std::string contents); std::vector get_files_in_directory(const std::string& dir); std::string find_file(const std::string& basename, std::vector search_path, diff --git a/mac/klammerset_registry.cpp b/mac/klammerset_registry.cpp index a65ac88..7398f88 100644 --- a/mac/klammerset_registry.cpp +++ b/mac/klammerset_registry.cpp @@ -221,7 +221,7 @@ static std::string klammerset_description(const fs::path& declaration) { std::string text {}; try { - text = string_from_file(declaration.string()); + text = klammertext_from_file(declaration.string()); } catch (...) { return ""; } diff --git a/mac/machine.cpp b/mac/machine.cpp index 40305d9..c406d27 100644 --- a/mac/machine.cpp +++ b/mac/machine.cpp @@ -491,7 +491,7 @@ void Machine::read(const fs::path& pathname) // :cpp libraries live next to the file that uses them). m_state.add_search_dir(fs::absolute(pathname).parent_path().string()); m_state.open_frame("Machine state: " + pathname.string()); - input(string_from_file(pathname.string()), pathname.generic_string()); + input(klammertext_from_file(pathname.string()), pathname.generic_string()); m_sources.push_back(pathname); } @@ -552,7 +552,7 @@ void Machine::expand_read_katoms( input_filename = fs::canonical(input_filename); m_state.add_search_dir(input_filename.parent_path().string()); - std::string text = trim_right(string_from_file(input_filename.string())); + std::string text = trim_right(klammertext_from_file(input_filename.string())); katom_list ks = katomize_lines(line_split(text), input_filename.generic_string()); // ks = process_katoms( @@ -684,7 +684,7 @@ void Machine::load_klammerset_files(const Klammerset& klammerset, katom_iter ins pathname = fs::canonical(pathname); loaded_files.push_back(pathname.generic_string()); m_state.add_search_dir(pathname.parent_path().string()); - std::string text = trim_right(string_from_file(pathname.string())); + std::string text = trim_right(klammertext_from_file(pathname.string())); katom_list ks = katomize_lines(line_split(text), pathname.generic_string()); process_katoms(ks, pathname.generic_string()); loaded.insert(loaded.end(), ks.begin(), ks.end()); diff --git a/ref/Makefile b/ref/Makefile index 000d6fc..2475592 100644 --- a/ref/Makefile +++ b/ref/Makefile @@ -19,7 +19,9 @@ include $(K)/env/makefile.env # CPPFLAGS already carries -I$(KLAMMERTEXT_HOME)/mac, which is every header # reflect.cpp includes (machine, character, error, show). No local -I. -LIBDIR := $(K)/lib +# Relative, never $(K)/...: the mingw linker cannot open a path holding a +# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26). +LIBDIR := ../lib LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE) REFLECT := reflect$(DSO) diff --git a/sks/document/Makefile b/sks/document/Makefile index 9865905..dd71b99 100644 --- a/sks/document/Makefile +++ b/sks/document/Makefile @@ -11,7 +11,9 @@ include $(K)/env/makefile.env LOCAL_CPPFLAGS := -I$(KM) -I$(KS)/kutil -I$(KS)/target # Shared library location -LIBDIR := $(K)/lib +# Relative, never $(K)/...: the mingw linker cannot open a path holding a +# non-ASCII letter (C:/Übungen/...), while the compiler can (2026-09-26). +LIBDIR := ../../lib LIBRARY := $(LIBDIR)/$(KLAMMERTEXT_LIBFILE) # The product: document.so, on Windows document.dll (makefile.env's DSO) @@ -25,8 +27,8 @@ LOCAL_OBJECTS := document_class.o document_html.o document_latex.o heading.o ref LOCAL_DEPFILES := document_class.d document_html.d document_latex.d heading.d reference.d document.d # External object files from sks/ (mac/*.o now in the shared library) -SKS_OBJECTS := $(KS)/kutil/kutil.o $(KS)/kutil/klammer_base.o \ - $(KS)/target/html_util.o $(KS)/target/latex_util.o +SKS_OBJECTS := ../kutil/kutil.o ../kutil/klammer_base.o \ + ../target/html_util.o ../target/latex_util.o ALL_OBJECTS := $(LOCAL_OBJECTS) $(SKS_OBJECTS) diff --git a/tst/filename_test.sh b/tst/filename_test.sh index 1313ad1..dd77f54 100755 --- a/tst/filename_test.sh +++ b/tst/filename_test.sh @@ -36,6 +36,10 @@ mkdir "$DIR/my dir" printf 'ALPHA\n' > "$DIR/my file.kt" printf 'BETA\n' > "$DIR/b.kt" printf 'GAMMA\n' > "$DIR/my dir/c.kt" +printf '\357\273\277DELTA\n' > "$DIR/bom.kt" # a UTF-8 byte-order mark, then the text +printf 'EPSILON\r\nZETA\r\n' > "$DIR/crlf.kt" # Windows line endings +printf '\377\376h\000i\000\n\000' > "$DIR/utf16.kt" # UTF-16LE with its byte-order mark +printf 'ETA\nTHETA M\374ller\n' > "$DIR/cp1252.kt" # Windows-1252: an umlaut that is not UTF-8 cd "$DIR" || exit 1 # check TEST_NAME EXPECTED_STDOUT EXPECTED_STDERR_SUBSTRING KTEXT_ARGS... @@ -151,6 +155,37 @@ check_error "8. unrescuable name is reported as written" \ "no such.kt" \ "no such.kt" -d --klammersets none +# A leading UTF-8 byte-order mark is ignored (2026-09-26): Windows tools write +# one, and it was reported as "Word not parsed" at 1:1 with an invisible word. +check "9. a leading UTF-8 BOM in the input file is ignored" \ + "DELTA" "" \ + bom.kt -d --klammersets none + +check "10. a leading UTF-8 BOM in a @read file is ignored" \ + "DELTA" "" \ + -s '@read bom.kt @' -d --klammersets none + +# Windows line endings are newlines (2026-09-26): the reader DELETED the pair, +# so on Linux and macOS a CRLF file collapsed into one line. +check "11. CRLF line endings are read as newlines" \ + $'EPSILON\nZETA' "" \ + crlf.kt -d --klammersets none + +# What a Windows editor can also save (2026-09-26): a UTF-16 file and an +# ANSI (Windows-1252) file are named as such, with the remedy, instead of +# "Word not parsed" or a failure inside the embedded Python. +check_error "12. a UTF-16 file is named as such" \ + "UTF-16" \ + utf16.kt -d --klammersets none + +check_error "13. a Windows-1252 file is located at its first non-UTF-8 byte" \ + "line 2, character 8" \ + cp1252.kt -d --klammersets none + +check_error "14. ... through @read as well" \ + "not UTF-8" \ + -s '@read cp1252.kt @' -d --klammersets none + echo echo "=======================" echo "Results: ${green}$PASS passed${reset}, ${red}$FAIL failed${reset}" diff --git a/tst/kpy_test.sh b/tst/kpy_test.sh index 35e1287..6a02f80 100755 --- a/tst/kpy_test.sh +++ b/tst/kpy_test.sh @@ -49,6 +49,19 @@ trap 'rm -rf "$OUTF" "$ERRF" "$WORK"' EXIT if command -v cygpath >/dev/null 2>&1; then sep=';'; else sep=':'; fi export PYTHONPATH="$K/kpy${PYTHONPATH:+$sep$PYTHONPATH}" +# On Windows the binding cannot load under a KLAMMERTEXT_HOME that holds a +# non-ASCII character: inside python.exe, a process without Klammertext's +# UTF-8 manifest, the narrow getenv() is the ANSI code page, and the engine +# takes every path as UTF-8. A documented limitation, not a defect to fix +# (TODO 70 (f), Andy 2026-09-26): the binding needs an ASCII installation +# path there. Say so and skip, rather than fail every case with a decode +# error that names no cause. +if command -v cygpath >/dev/null 2>&1 && LC_ALL=C grep -q '[^ -~]' <<<"$K"; then + echo "SKIP kpy_test: KLAMMERTEXT_HOME ($K) holds a non-ASCII character, and the Python binding needs an ASCII installation path on Windows (TODO 70 (f))" + echo "Passed: 0, Failed: 0" + exit 0 +fi + # A DEBUG=1 build instruments the module with AddressSanitizer, and an # instrumented extension imported into an uninstrumented python needs the # ASan runtime loaded FIRST -- the import otherwise fails with "ASan runtime