chatter/doc/Chatter_implementation.md

Chatter — Implementation Plan & Method

Status: Living document — revised as we learn. Read the Changelog at the bottom for what changed. Version: 0.7 (2026-06-26) Audience: The Chatter development team (Andreas, Matt, and Claude as development assistant).

This document is the shared source of truth for how we build Chatter. It is deliberately incremental: early phases will answer questions that later phases depend on, and this document will be updated as those answers come in. If something here is marked (unknown / to confirm on-device), treat it as an open question, not a decision.

---

1. What Chatter is

Chatter is a minimal, purpose-built interface for a reMarkable tablet used as an assistive-speech device by a user who has difficulty speaking — she writes, and her conversation partner reads. Chatter augments the tablet; it does not replace the standard reMarkable GUI.

The design priority is intuitiveness under intermittent recall: the user should not have to remember technical steps. See the full motivation and the user-experience reasoning in Chatter_application_proposal.md.

Chatter's core actions, by expected frequency:

1. Erase part of the text — a simple, intuitive gesture (e.g. wiping a fingertip back and forth over an area).
2. Erase the whole page — a single gesture (e.g. a diagonal finger stroke, upper-right → lower-left).
3. Toggle to/from the standard GUI — necessary; Chatter must be able to hand off to the stock interface and be returned to.
4. Save the current page — importance to the user is not yet confirmed; a single button writes a timestamped transcript into a "Chatter" folder.

---

2. Target device and key constraints

Device: reMarkable Paper Pro Move — model RM03A. (Identified from the official user-guide PDF linked in the proposal.)

Property	Value	Why it matters
Display	E Ink Gallery 3, color, 7.3", 954×1696 (264 PPI), backlit	Color refresh pipeline differs from older grayscale models
Touch	Multi-point capacitive	Finger gestures (erase) are viable
Pen	Active Marker / Marker Plus (flip-to-erase on Plus)	Pen and touch are separate input streams → we can tell finger from stylus natively
SoC / RAM	ARM 1.7 GHz dual A55, 2 GB RAM, 64 GB	Standard ARM-Linux cross-compile target
OS	"Codex" (custom Linux)	Stock note app is Xochitl (Qt/C++, proprietary)
Dev access	Enable developer mode, which requires a factory reset	Back up the device first

Key risk — this is the newest model. The mature community frameworks (rmkit, libreMarkable, Plato, KOReader ports) were built against the older RM1/RM2 grayscale framebuffer and refresh ioctls. They likely do not support the Paper Pro Move yet, or only partially. We should expect to do first-principles work — discovering this model's framebuffer format, color refresh path, and input device nodes by probing the hardware over SSH — rather than relying on an existing framework. This raises the value of hands-on device access early and argues for a small, well-proven first milestone.

---

3. Roles

• Andreas — development, with Claude's assistance, on his own Paper Pro Move over SSH.
• Matt — installation and QA, including on-site testing with the end user on her tablet. Matt has the same model (Paper Pro Move). Matt may also take on development; the split will be worked out as the project progresses.
• Claude — development assistant: writes/reviews code, drives the build-deploy loop over SSH where possible, and maintains this document.

This document is how Matt stays in sync. When the design changes, the change lands here first.

---

4. Development phases

The phases are ordered so each de-risks the next. Toggle (necessary) precedes Save (provisional).

Phase 0 — Safety & access

• Back up first (critical on the user's device; skippable on a new tablet with nothing saved). Enabling developer mode performs a factory reset — on the Paper Pro family, SSH is unavailable until developer mode is on, and turning it on wipes the device. Sync to the reMarkable cloud account before enabling. Document this backup procedure so Matt can repeat it.
• Enable developer mode: Hamburger menu (upper left) → Settings → Software → Advanced → Developer mode → Accept. This factory-resets the device (deletes all local files; user data cleared on reboot). Re-pair with the reMarkable account afterward to access cloud content. (Disabling later requires reMarkable's recovery application.)
• Reconnect wifi after the reset (the wifi config is wiped).
• Get SSH credentials: Menu → Settings → Help → Copyright and licenses; the root username (root), password, and device IP addresses are under the "GPLv3 Compliance" header. USB exposes the tablet at 10.11.99.1.
• Enable SSH over wifi (it is OFF by default): connect over USB, SSH in, and run rm-ssh-over-wlan on. Only then is the listed 192.168.x.x address reachable; after that the USB cable is optional.
• Install your workstation's SSH public key into the tablet's ~/.ssh/authorized_keys (user root) for passwordless access.
• Confirm SSH login, scp of a binary, and remote execution.
• Stand up the x86_64-hosted cross-toolchain for Codex OS; verify a trivial ARM binary runs on-device.

Dev environment note: development host is Jatke (on the wired LAN); we connect to the tablet over wifi (USB not in use).

Phase 1 — On-device investigation (no app code; produces a findings writeup)

Answer the questions that can only be answered with the tablet over SSH. See the checklist in §5. Output: a short findings document added to doc/. Status: substantially complete (2026-06-25) — see Chatter_phase1_findings.md. Key results: pen (event2) and finger (event3) are separate evdev devices; the display is DRM/KMS only (no fbdev) driven by imx-drm; and the stock app is a Qt 6.8.2 app rendering through a reusable epaper QPA plugin — so Chatter should be a Qt 6 app run with -platform epaper. Remaining sub-items (verifying -platform epaper from a third-party binary, input delivery, toggle hand-off) roll into the Phase 2 spike.

Phase 2 — Minimal canvas spike

Read the pen, render strokes to the (color) display, and clear the page — nothing more. This proves the two hardest pieces (input + color refresh) end to end and de-risks everything after it.

Status (2026-06-25): display/toolchain half PROVEN on hardware. A third-party Qt Quick app (build/chatter) cross-built with the Chiappa SDK renders crisp black-on-white on the e-paper via -platform epaper (QT_QUICK_BACKEND=epaper).

Status (2026-06-26): inking WORKS, with one known quality issue. Custom evdev pen reader (src/PenDevice.cpp, reads event2) drives a QQuickPaintedItem canvas (src/InkCanvas.cpp): ink tracks the pen, orientation correct, eraser (BTN_TOOL_RUBBER) erases, finger-tap "Clear" works, latency good.

• Known issue: strokes render as a "dashed line that fills in" (two-pass), unlike the solid stock pen. Root cause: we use the panel's default grayscale mode. Fix path: xochitl wraps its writing area in a QML ScreenModeItem set to Mono/FAST with a dedicated pen waveform (/usr/share/remarkable/ct33_pen.bin). The control is the private EPScreenModeItem class in libqsgepaper.so (setMode(), modes Mono/FAST/FastGrayscale) — not a public QML type, no SDK header — so Chatter must integrate it directly (instantiate, link the plugin, drive mode via the meta-object, wrap the canvas). Antialiasing on/off does not affect it.
• Pen-style decision: Chatter uses the pen style set in the standard GUI (xochitl.conf LastPen/LastPenSize/LastPenColor, readable via QSettings), re-read so it tracks changes the user makes after toggling — no style menu in Chatter.

Lessons learned (encoded in scripts/):

• The e-paper framebuffer is a singleton guarded by an flock (/tmp/epframebuffer.lock, from libqsgepaper's EPFramebufferAcep2). Only ONE process may hold it; a stranded instance shows Failed to lock epframebuffer / Failed to initialize SWTCON and the panel stays blank. Always stop the previous instance first. Journal success marker: SWTCON initialized \o/.
• Run as a transient systemd service (systemd-run --unit=chatter …), not an ssh background job (which holds the SSH channel open and can strand the process). Manage with systemctl stop chatter / journalctl -u chatter.
• A full-screen Window needs an explicit size (width: Screen.width; height: Screen.height); visibility: FullScreen alone left it unsized/blank.
• xochitl runs with no special Qt env — -platform epaper + QT_QUICK_BACKEND=epaper suffices.

Run recipe: scripts/build.sh && scripts/deploy-and-run.sh; scripts/restore-xochitl.sh to return to the standard GUI.

Phase 3 — Erase gestures

• Region erase: finger wipe removes ink near the wiped area ("near" tuned experimentally with the user).
• Whole-page erase: single diagonal finger stroke (upper-right → lower-left).
• Provide stylus fallbacks if finger/stylus separation proves unreliable.

Phase 4 — Toggle to/from the standard GUI (necessary)

Status (2026-06-26): BIDIRECTIONAL toggle works on-device (verified).

• Chatter → standard: top-left "Back" button → AppControl::returnToStandard() → transient unit stops chatter, starts xochitl.

• standard → Chatter: a persistent launcher daemon (tools/chatter_launcher.c, installed as a rootfs systemd service via scripts/install-launcher.sh) watches the touch device and, on a 4-finger hold (~700 ms) while xochitl is in front, runs to-chatter.sh to switch to Chatter. Gesture is tunable; finalize the memorable one with the user.

• Deployment note: /etc and /run are volatile overlays (wiped on reboot); persistent installs go to /home (app/binaries/scripts) or the rootfs (mount -o remount,rw /, lost on OS update). Disable OS auto-updates on a delivered device.

• A gesture/button in Chatter hands off to the standard reMarkable GUI.

• A memorable mechanism returns from the standard GUI to Chatter. The proposal asks that the user help choose this; we will prototype options and let her pick.

• Phase 1 insight: because the display is DRM/KMS (single master) and xochitl owns it while running, the toggle is a hand-off, not an overlay — stop/pause xochitl to give Chatter the display, and reverse to return. Mechanism (systemctl stop/start vs. SIGSTOP/SIGCONT) to be validated.

Phase 5 — Save the page (provisional — pending confirmation it matters to the user)

• A single visible button saves the current page.
• Filename fully derived from date/time (year, month, day, hour, minute, second).
• Written into a folder named "Chatter", using the same on-disk document format the stock app uses, so transcripts appear as normal notebooks and can be renamed/deleted/synced via the standard GUI (see §6).

Phase 6 — Field test & refine

Matt installs and observes the user. Capture especially the "Why does this…" and "Why doesn't this…" questions; feed them back into this document and the phase plan.

---

5. On-device investigation checklist (Phase 1)

These are the open questions the hands-on phase exists to answer.

• Input devices: which /dev/input/event* is the pen vs. the capacitive touch; event format, coordinate range and orientation, pressure; the flip-to-erase signal on Marker Plus.
• Display: how the color Gallery 3 panel is driven — framebuffer device and pixel format, and the refresh mechanism (ioctls or this model's equivalent). (Highest-risk unknown.)
• Document store: where Xochitl stores notebooks; the .metadata / .content / per-page .rm layout for this model; the .rm format version (newer, color-capable); how a collection (folder) is represented; and whether a hand-built document appears in the library after a restart/sync.
• Coexistence & toggle: how to launch our binary alongside / over Xochitl, and what a reliable toggle and return-to-Chatter mechanism looks like.

---

6. The "save" approach (clarification)

reMarkable exposes no public callable on-device API for creating notebooks. The stock app simply writes files to an on-disk document store in a specific format (roughly: a UUID per document, a .metadata JSON, a .content JSON, and one binary .rm line-file per page, plus the color/folder data and a sync layer). A "folder" such as the Chatter folder is itself a document of type collection.

So "save using the same method the environment uses" means: write our canvas into the exact on-disk format Xochitl reads, for this model's format version. The open question for Phase 5 is confirming/replicating the Paper Pro Move's current .rm format. This is investigated in Phase 1.

---

6a. Confirmed device facts (from SSH, 2026-06-25)

• Internal codename: "Chiappa" (/proc/device-tree/model = "reMarkable Chiappa"; hostname imx93-chiappa). This is the Paper Pro Move (RM03A).
• SoC: NXP i.MX93 (Arm Cortex-A55). Relevant to the toolchain target and to how the display is driven.
• OS: Codex Linux 5.7.121 (Yocto scarthgap), image version 3.27.1.0.
• SSH server: dropbear (not OpenSSH). Root login; ~/.ssh/authorized_keys honored. Wifi SSH enabled via rm-ssh-over-wlan on.
• Access from Jatke: passwordless via ed25519 key as root over wifi (the tablet's IP) or root@10.11.99.1 (USB, fixed on every reMarkable). The tablet's actual IP/MAC and credentials are in tablet_access.md (SENSITIVE, gitignored).

Phase 0 status: access established (dev mode, SSH over USB + wifi, key-based login, scp verified). Remaining for Phase 0: stand up the cross-toolchain and run a trivial ARM binary on-device.

6b. Toolchain / build & deploy (researched 2026-06-25)

Official SDK (cross-toolchain + Qt 6.8.2 sysroot), self-extracting .sh, x86_64 host only. For this device (codename chiappa):

https://storage.googleapis.com/remarkable-codex-toolchain/3.27.0.97/chiappa/remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh

• ~486 MB. Device image is 3.27.1.0 (Codex 5.7.121); newest published Chiappa SDK is 3.27.0.97 (5.7.119) — same 3.27.x family, so binaries built against it run on the device. (An aarch64-host variant also exists; we use x86_64 for Jatke.) Download links live on developer.remarkable.com/links; the GCS bucket itself is not directly listable (403).
• Build host Jatke (x86_64, ample disk). SDKs expect a Yocto-supported Linux host.

Install & activate:

chmod u+x remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
./remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh -d <install-dir>
source <install-dir>/environment-setup-cortexa55-remarkable-linux   # exact filename confirmed post-install (i.MX93 = Cortex-A55)

Build a Qt Quick app (per developer.remarkable.com/documentation/qt_epaper): CMake with find_package(Qt6 REQUIRED COMPONENTS Quick) + qt_add_executable / qt_add_qml_module; cmake . -B build && cmake --build build after sourcing the SDK env.

Deploy & run on device:

scp build/<app> chatter:/home/root/chatter/      # deploy under /home, not rootfs
ssh chatter 'systemctl stop xochitl'             # release the display (pre-authorized)
ssh chatter 'cd /home/root/chatter && QT_QUICK_BACKEND=epaper ./<app> -platform epaper'

Constraints (official + Phase 1): Qt Quick/QML only, no Qt Widgets; xochitl must be stopped (DRM single-master); touch is automatic, pen is custom (read event2). Some SDK/OS combos require copying libqsgepaper.so to the device — verify during the spike.

7. Decisions & open questions

Decided (from Phase 1):

• Implementation stack: C++ / Qt 6.8.2, rendered via the device's epaper QPA plugin (-platform epaper). Chosen because the stock app proves this path works and gives us e-ink display + refresh for free; the older fbdev-mmap approach is not available (DRM/KMS only). Cross-build with a Yocto scarthgap aarch64 SDK matching Qt 6.8.2. (Matt — flag any objection.)
• Deploy location: under /home (45 GB free), never the rootfs (~88 MB free, overlay, reset by updates).
• Toggle is a display hand-off with xochitl (see Phase 4), not an overlay.

Still open:

• Save mechanism — leaning PDF-backed document over authoring native v6 .rm; decide in Phase 5 (and gated by whether save matters to the user).
• Whether "save" matters to the user — learned from observation.
• Toggle gesture + return-to-Chatter mechanism — prototype and choose with the user (Phase 4).
• Erase "closeness" thresholds — tuned experimentally with the user (Phase 3).

---

Changelog

• 0.6 (2026-06-26) — Inking engine working on hardware: custom evdev pen reader (event2) + QQuickPaintedItem canvas — pen draws, eraser erases, finger-tap Clear, good latency. Added top control bar (Back left, Clear right); Back hands off to the standard GUI (verified). Bound the private EPScreenModeItem (key-function trick) to control panel refresh mode — but the dashed two-pass stroke refresh is the render loop's behavior, independent of mode, so it's deferred to a framebuffer-direct effort (needs disassembly of swapBuffers for EPContentType/UpdateFlag). Implemented pen-style matching: Chatter reads the standard GUI's selected pen (type/size/color) from xochitl.conf LastWritingTool and applies width + color — verified (Calligraphy, size 3, black).
• 0.7 (2026-06-26) — ⭐ Solved stroke quality with a direct-framebuffer ink pipeline. After proving Qt Quick can't render stock-quality e-ink strokes (geometry nodes unsupported by the software backend; painted-textures always get the dashed two-pass; screen-mode and antialiasing irrelevant), switched architecture: FbCapture interposes EPFramebuffer::setBuffers in-process (-Wl,--export-dynamic) to capture the real framebuffer image; InkEngine draws strokes straight into it and refreshes each segment with an explicit swapBuffers — solid, crisp, low-latency, correctly-aligned strokes on hardware (verified), plus working Clear. Qt Quick now only renders the static Back/Clear UI. Key RE findings: swapBuffers renders solid (not dashed) on a single explicit call; intra-DSO calls aren't LD_PRELOAD-interposable but the cross-DSO setBuffers is. Remaining: calligraphic nib (uniform round width vs the angled direction-dependent Calligraphy nib), return-to-Chatter trigger, Save, field testing.
• 0.5 (2026-06-25) — Phase 2 spike: cross-built a Qt Quick hello-app with the Chiappa SDK and rendered it on the e-paper via -platform epaper (proven on hardware). Recorded lessons (panel flock singleton, run-as-systemd-service, explicit Window size) and added scripts/build.sh, deploy-and-run.sh, restore-xochitl.sh. Seeded the source tree (CMakeLists.txt, src/main.cpp, qml/Main.qml).
• 0.4 (2026-06-25) — Researched the toolchain (§6b): identified the official Chiappa SDK (3.27.0.97 / Codex 5.7.119, x86_64 host, ~486 MB), install/ activate steps, the Qt Quick build + -platform epaper run commands, and the deploy-and-stop-xochitl flow. Confirmed Qt-Quick-only + custom-pen constraints.
• 0.3 (2026-06-25) — Phase 1 on-device investigation substantially complete (see Chatter_phase1_findings.md). Resolved the stack decision to C++/Qt 6 + epaper QPA plugin; recorded display = DRM/KMS only, pen/finger as separate evdev devices, toggle = display hand-off, deploy under /home, and the document-store format for the Save phase.
• 0.2 (2026-06-25) — Phase 0 access established. Added confirmed device facts (§6a): codename "Chiappa", NXP i.MX93 SoC, Codex Linux 5.7.121 (scarthgap), dropbear SSH, passwordless key access from Jatke. Corrected the dev-mode menu path and documented that wifi SSH is off by default (rm-ssh-over-wlan on).
• 0.1 (2026-06-25) — Initial version: scope, target device (RM03A) and its constraints, roles, phase plan (with Toggle before Save), Phase 1 investigation checklist, save-format clarification, and open decisions.