chatter/doc/Chatter_implementation.md

# Chatter — Implementation Plan & Method

**Status:** Living document — revised as we learn. Read the Changelog at the bottom for what changed.
**Version:** 0.7 (2026-06-26)
**Audience:** The Chatter development team (Andreas, Matt, and Claude as development assistant).

> This document is the shared source of truth for *how* we build Chatter. It is
> deliberately incremental: early phases will answer questions that later phases
> depend on, and this document will be updated as those answers come in. If
> something here is marked **(unknown / to confirm on-device)**, treat it as an
> open question, not a decision.

---

## 1. What Chatter is

Chatter is a minimal, purpose-built interface for a reMarkable tablet used as an
**assistive-speech device** by a user who has difficulty speaking — she writes,
and her conversation partner reads. Chatter **augments** the tablet; it does not
replace the standard reMarkable GUI.

The design priority is **intuitiveness under intermittent recall**: the user
should not have to remember technical steps. See the full motivation and the
user-experience reasoning in [`Chatter_application_proposal.md`](Chatter_application_proposal.md).

Chatter's core actions, by expected frequency:

1. **Erase part of the text** — a simple, intuitive gesture (e.g. wiping a
   fingertip back and forth over an area).
2. **Erase the whole page** — a single gesture (e.g. a diagonal finger stroke,
   upper-right → lower-left).
3. **Toggle to/from the standard GUI** — *necessary*; Chatter must be able to
   hand off to the stock interface and be returned to.
4. **Save the current page** — *importance to the user is not yet confirmed*; a
   single button writes a timestamped transcript into a "Chatter" folder.

---

## 2. Target device and key constraints

**Device: reMarkable Paper Pro Move — model RM03A.** (Identified from the
official user-guide PDF linked in the proposal.)

| Property | Value | Why it matters |
|---|---|---|
| Display | E Ink Gallery 3, **color**, 7.3", 954×1696 (264 PPI), backlit | Color refresh pipeline differs from older grayscale models |
| Touch | Multi-point capacitive | Finger gestures (erase) are viable |
| Pen | Active Marker / Marker Plus (flip-to-erase on Plus) | Pen and touch are **separate input streams** → we can tell finger from stylus natively |
| SoC / RAM | ARM 1.7 GHz dual A55, 2 GB RAM, 64 GB | Standard ARM-Linux cross-compile target |
| OS | "Codex" (custom Linux) | Stock note app is Xochitl (Qt/C++, proprietary) |
| Dev access | Enable **developer mode**, which requires a **factory reset** | **Back up the device first** |

**Key risk — this is the newest model.** The mature community frameworks
(rmkit, libreMarkable, Plato, KOReader ports) were built against the older
RM1/RM2 grayscale framebuffer and refresh ioctls. They likely **do not support
the Paper Pro Move yet**, or only partially. We should expect to do
**first-principles work** — discovering this model's framebuffer format, color
refresh path, and input device nodes by probing the hardware over SSH — rather
than relying on an existing framework. This raises the value of hands-on device
access early and argues for a small, well-proven first milestone.

---

## 3. Roles

- **Andreas** — development, with Claude's assistance, on his own Paper Pro Move
  over SSH.
- **Matt** — installation and QA, including on-site testing with the end user on
  her tablet. Matt has the same model (Paper Pro Move). Matt may also take on
  development; the split will be worked out as the project progresses.
- **Claude** — development assistant: writes/reviews code, drives the
  build-deploy loop over SSH where possible, and maintains this document.

This document is how Matt stays in sync. When the design changes, the change
lands here first.

---

## 4. Development phases

The phases are ordered so each de-risks the next. **Toggle (necessary) precedes
Save (provisional).**

### Phase 0 — Safety & access
- **Back up first** (critical on the user's device; skippable on a new tablet
  with nothing saved). Enabling developer mode **performs a factory reset** — on
  the Paper Pro family, SSH is unavailable until developer mode is on, and
  turning it on wipes the device. Sync to the reMarkable cloud account before
  enabling. Document this backup procedure so Matt can repeat it.
- **Enable developer mode:** Hamburger menu (upper left) → Settings → Software →
  Advanced → Developer mode → Accept. This factory-resets the device (deletes
  all local files; user data cleared on reboot). Re-pair with the reMarkable
  account afterward to access cloud content. (Disabling later requires
  reMarkable's recovery application.)
- **Reconnect wifi** after the reset (the wifi config is wiped).
- **Get SSH credentials:** Menu → Settings → Help → Copyright and licenses;
  the root username (`root`), password, and device IP addresses are under the
  "GPLv3 Compliance" header. USB exposes the tablet at `10.11.99.1`.
- **Enable SSH over wifi (it is OFF by default):** connect over USB, SSH in,
  and run `rm-ssh-over-wlan on`. Only then is the listed `192.168.x.x` address
  reachable; after that the USB cable is optional.
- Install your workstation's SSH public key into the tablet's
  `~/.ssh/authorized_keys` (user `root`) for passwordless access.
- Confirm SSH login, `scp` of a binary, and remote execution.
- Stand up the x86_64-hosted cross-toolchain for Codex OS; verify a trivial
  ARM binary runs on-device.

> Dev environment note: development host is **Jatke** (on the wired LAN); we
> connect to the tablet **over wifi** (USB not in use).

### Phase 1 — On-device investigation (no app code; produces a findings writeup)
Answer the questions that can only be answered with the tablet over SSH. See the
checklist in §5. Output: a short findings document added to `doc/`.
**Status: substantially complete (2026-06-25)** — see
[`Chatter_phase1_findings.md`](Chatter_phase1_findings.md). Key results: pen
(`event2`) and finger (`event3`) are separate evdev devices; the display is
DRM/KMS only (no fbdev) driven by `imx-drm`; and the stock app is a Qt 6.8.2 app
rendering through a reusable **`epaper` QPA plugin** — so Chatter should be a
Qt 6 app run with `-platform epaper`. Remaining sub-items (verifying `-platform
epaper` from a third-party binary, input delivery, toggle hand-off) roll into
the Phase 2 spike.

### Phase 2 — Minimal canvas spike
Read the pen, render strokes to the (color) display, and clear the page —
nothing more. This proves the two hardest pieces (input + color refresh) end to
end and de-risks everything after it.

**Status (2026-06-25): display/toolchain half PROVEN on hardware.** A
third-party Qt Quick app (`build/chatter`) cross-built with the Chiappa SDK
renders crisp black-on-white on the e-paper via `-platform epaper`
(`QT_QUICK_BACKEND=epaper`).

**Status (2026-06-26): inking WORKS, with one known quality issue.** Custom
evdev pen reader (`src/PenDevice.cpp`, reads `event2`) drives a
`QQuickPaintedItem` canvas (`src/InkCanvas.cpp`): ink tracks the pen, orientation
correct, eraser (BTN_TOOL_RUBBER) erases, finger-tap "Clear" works, latency good.
- **Known issue:** strokes render as a "dashed line that fills in" (two-pass),
  unlike the solid stock pen. **Root cause:** we use the panel's default
  *grayscale* mode. **Fix path:** xochitl wraps its writing area in a QML
  `ScreenModeItem` set to **Mono/FAST** with a dedicated **pen waveform**
  (`/usr/share/remarkable/ct33_pen.bin`). The control is the private
  `EPScreenModeItem` class in `libqsgepaper.so` (`setMode()`, modes
  Mono/FAST/FastGrayscale) — not a public QML type, no SDK header — so Chatter
  must integrate it directly (instantiate, link the plugin, drive `mode` via the
  meta-object, wrap the canvas). Antialiasing on/off does not affect it.
- **Pen-style decision:** Chatter uses the pen style set in the standard GUI
  (xochitl.conf `LastPen`/`LastPenSize`/`LastPenColor`, readable via QSettings),
  re-read so it tracks changes the user makes after toggling — no style menu in
  Chatter.

**Lessons learned (encoded in `scripts/`):**
- The e-paper framebuffer is a **singleton guarded by an flock**
  (`/tmp/epframebuffer.lock`, from `libqsgepaper`'s `EPFramebufferAcep2`). Only
  ONE process may hold it; a stranded instance shows `Failed to lock
  epframebuffer` / `Failed to initialize SWTCON` and the panel stays blank.
  Always stop the previous instance first. Journal success marker:
  `SWTCON initialized \o/`.
- **Run as a transient systemd service** (`systemd-run --unit=chatter …`), not an
  ssh background job (which holds the SSH channel open and can strand the
  process). Manage with `systemctl stop chatter` / `journalctl -u chatter`.
- A full-screen `Window` needs an **explicit size** (`width: Screen.width;
  height: Screen.height`); `visibility: FullScreen` alone left it unsized/blank.
- xochitl runs with **no special Qt env** — `-platform epaper` +
  `QT_QUICK_BACKEND=epaper` suffices.

**Run recipe:** `scripts/build.sh && scripts/deploy-and-run.sh`;
`scripts/restore-xochitl.sh` to return to the standard GUI.

### Phase 3 — Erase gestures
- Region erase: finger wipe removes ink near the wiped area ("near" tuned
  experimentally with the user).
- Whole-page erase: single diagonal finger stroke (upper-right → lower-left).
- Provide stylus fallbacks if finger/stylus separation proves unreliable.

### Phase 4 — Toggle to/from the standard GUI  *(necessary)*

**Status (2026-06-26): BIDIRECTIONAL toggle works on-device (verified).**
- **Chatter → standard:** top-left **"Back"** button → `AppControl::returnToStandard()`
  → transient unit stops chatter, starts xochitl.
- **standard → Chatter:** a persistent **launcher daemon** (`tools/chatter_launcher.c`,
  installed as a rootfs systemd service via `scripts/install-launcher.sh`) watches
  the touch device and, on a **4-finger hold (~700 ms)** while xochitl is in front,
  runs `to-chatter.sh` to switch to Chatter. Gesture is tunable; finalize the
  memorable one with the user.
- **Deployment note:** `/etc` and `/run` are volatile overlays (wiped on reboot);
  persistent installs go to `/home` (app/binaries/scripts) or the rootfs
  (`mount -o remount,rw /`, lost on OS update). Disable OS auto-updates on a
  delivered device.

- A gesture/button in Chatter hands off to the standard reMarkable GUI.
- A **memorable** mechanism returns from the standard GUI to Chatter. The
  proposal asks that the user help choose this; we will prototype options and
  let her pick.
- **Phase 1 insight:** because the display is DRM/KMS (single master) and
  xochitl owns it while running, the toggle is a **hand-off**, not an overlay —
  stop/pause `xochitl` to give Chatter the display, and reverse to return.
  Mechanism (`systemctl stop/start` vs. `SIGSTOP/SIGCONT`) to be validated.

### Phase 5 — Save the page  *(provisional — pending confirmation it matters to the user)*
- A single visible button saves the current page.
- Filename fully derived from date/time (year, month, day, hour, minute,
  second).
- Written into a folder named **"Chatter"**, using the same on-disk document
  format the stock app uses, so transcripts appear as normal notebooks and can
  be renamed/deleted/synced via the standard GUI (see §6).

### Phase 6 — Field test & refine
Matt installs and observes the user. Capture especially the "Why does this…" and
"Why doesn't this…" questions; feed them back into this document and the phase
plan.

---

## 5. On-device investigation checklist (Phase 1)

These are the open questions the hands-on phase exists to answer.

- **Input devices:** which `/dev/input/event*` is the pen vs. the capacitive
  touch; event format, coordinate range and orientation, pressure; the
  flip-to-erase signal on Marker Plus.
- **Display:** how the color Gallery 3 panel is driven — framebuffer device and
  pixel format, and the refresh mechanism (ioctls or this model's equivalent).
  *(Highest-risk unknown.)*
- **Document store:** where Xochitl stores notebooks; the
  `.metadata` / `.content` / per-page `.rm` layout for this model; the `.rm`
  format version (newer, color-capable); how a *collection* (folder) is
  represented; and whether a hand-built document appears in the library after a
  restart/sync.
- **Coexistence & toggle:** how to launch our binary alongside / over Xochitl,
  and what a reliable toggle and return-to-Chatter mechanism looks like.

---

## 6. The "save" approach (clarification)

reMarkable exposes **no public callable on-device API** for creating notebooks.
The stock app simply **writes files to an on-disk document store** in a specific
format (roughly: a UUID per document, a `.metadata` JSON, a `.content` JSON, and
one binary `.rm` line-file per page, plus the color/folder data and a sync
layer). A "folder" such as the **Chatter** folder is itself a document of type
*collection*.

So "save using the same method the environment uses" means: **write our canvas
into the exact on-disk format Xochitl reads, for this model's format version.**
The open question for Phase 5 is confirming/replicating the Paper Pro Move's
current `.rm` format. This is investigated in Phase 1.

---

## 6a. Confirmed device facts (from SSH, 2026-06-25)

- **Internal codename:** "Chiappa" (`/proc/device-tree/model` = "reMarkable
  Chiappa"; hostname `imx93-chiappa`). This is the Paper Pro Move (RM03A).
- **SoC:** NXP **i.MX93** (Arm Cortex-A55). Relevant to the toolchain target and
  to how the display is driven.
- **OS:** Codex Linux **5.7.121** (Yocto **scarthgap**), image version
  `3.27.1.0`.
- **SSH server:** **dropbear** (not OpenSSH). Root login; `~/.ssh/authorized_keys`
  honored. Wifi SSH enabled via `rm-ssh-over-wlan on`.
- **Access from Jatke:** passwordless via ed25519 key as `root` over wifi (the
  tablet's IP) or `root@10.11.99.1` (USB, fixed on every reMarkable). The tablet's
  actual IP/MAC and credentials are in `tablet_access.md` (SENSITIVE, gitignored).

**Phase 0 status:** access established (dev mode, SSH over USB + wifi, key-based
login, scp verified). Remaining for Phase 0: stand up the cross-toolchain and
run a trivial ARM binary on-device.

## 6b. Toolchain / build & deploy (researched 2026-06-25)

**Official SDK (cross-toolchain + Qt 6.8.2 sysroot), self-extracting `.sh`,
x86_64 host only.** For this device (codename **chiappa**):

```
https://storage.googleapis.com/remarkable-codex-toolchain/3.27.0.97/chiappa/remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
```

- ~486 MB. Device image is `3.27.1.0` (Codex 5.7.121); newest published Chiappa
  SDK is `3.27.0.97` (5.7.119) — same `3.27.x` family, so binaries built against
  it run on the device. (An aarch64-host variant also exists; we use x86_64 for
  Jatke.) Download links live on developer.remarkable.com/links; the GCS bucket
  itself is not directly listable (403).
- Build host **Jatke** (x86_64, ample disk). SDKs expect a Yocto-supported Linux
  host.

**Install & activate:**
```
chmod u+x remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
./remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh -d <install-dir>
source <install-dir>/environment-setup-cortexa55-remarkable-linux   # exact filename confirmed post-install (i.MX93 = Cortex-A55)
```

**Build a Qt Quick app** (per developer.remarkable.com/documentation/qt_epaper):
CMake with `find_package(Qt6 REQUIRED COMPONENTS Quick)` +
`qt_add_executable` / `qt_add_qml_module`; `cmake . -B build && cmake --build
build` after sourcing the SDK env.

**Deploy & run on device:**
```
scp build/<app> chatter:/home/root/chatter/      # deploy under /home, not rootfs
ssh chatter 'systemctl stop xochitl'             # release the display (pre-authorized)
ssh chatter 'cd /home/root/chatter && QT_QUICK_BACKEND=epaper ./<app> -platform epaper'
```

Constraints (official + Phase 1): **Qt Quick/QML only, no Qt Widgets**; **xochitl
must be stopped** (DRM single-master); **touch is automatic, pen is custom**
(read `event2`). Some SDK/OS combos require copying `libqsgepaper.so` to the
device — verify during the spike.

## 7. Decisions & open questions

**Decided (from Phase 1):**
- **Implementation stack: C++ / Qt 6.8.2**, rendered via the device's **`epaper`
  QPA plugin** (`-platform epaper`). Chosen because the stock app proves this
  path works and gives us e-ink display + refresh for free; the older
  fbdev-mmap approach is not available (DRM/KMS only). Cross-build with a Yocto
  scarthgap aarch64 SDK matching Qt 6.8.2. *(Matt — flag any objection.)*
- **Deploy location: under `/home`** (45 GB free), never the rootfs (~88 MB
  free, overlay, reset by updates).
- **Toggle is a display hand-off** with xochitl (see Phase 4), not an overlay.

**Still open:**
- **Save mechanism** — leaning **PDF-backed document** over authoring native v6
  `.rm`; decide in Phase 5 (and gated by whether save matters to the user).
- **Whether "save" matters to the user** — learned from observation.
- **Toggle gesture + return-to-Chatter mechanism** — prototype and choose with
  the user (Phase 4).
- **Erase "closeness" thresholds** — tuned experimentally with the user
  (Phase 3).

---

## Changelog

- **0.6 (2026-06-26)** — Inking engine working on hardware: custom evdev pen
  reader (`event2`) + `QQuickPaintedItem` canvas — pen draws, eraser erases,
  finger-tap **Clear**, good latency. Added top control bar (**Back** left,
  **Clear** right); Back hands off to the standard GUI (verified). Bound the
  private `EPScreenModeItem` (key-function trick) to control panel refresh mode —
  but the **dashed two-pass stroke refresh is the render loop's behavior,
  independent of mode**, so it's deferred to a framebuffer-direct effort
  (needs disassembly of `swapBuffers` for `EPContentType`/`UpdateFlag`).
  Implemented **pen-style matching**: Chatter reads the standard GUI's selected
  pen (type/size/color) from `xochitl.conf` `LastWritingTool` and applies width +
  color — verified (Calligraphy, size 3, black).
- **0.7 (2026-06-26)** — ⭐ **Solved stroke quality with a direct-framebuffer ink
  pipeline.** After proving Qt Quick can't render stock-quality e-ink strokes
  (geometry nodes unsupported by the software backend; painted-textures always
  get the dashed two-pass; screen-mode and antialiasing irrelevant), switched
  architecture: `FbCapture` interposes `EPFramebuffer::setBuffers` in-process
  (`-Wl,--export-dynamic`) to capture the real framebuffer image; `InkEngine`
  draws strokes straight into it and refreshes each segment with an explicit
  `swapBuffers` — **solid, crisp, low-latency, correctly-aligned strokes on
  hardware (verified), plus working Clear.** Qt Quick now only renders the static
  Back/Clear UI. Key RE findings: `swapBuffers` renders solid (not dashed) on a
  single explicit call; intra-DSO calls aren't LD_PRELOAD-interposable but the
  cross-DSO `setBuffers` is. Remaining: calligraphic nib (uniform round width vs
  the angled direction-dependent Calligraphy nib), return-to-Chatter trigger,
  Save, field testing.
- **0.5 (2026-06-25)** — Phase 2 spike: cross-built a Qt Quick hello-app with the
  Chiappa SDK and rendered it on the e-paper via `-platform epaper` (proven on
  hardware). Recorded lessons (panel flock singleton, run-as-systemd-service,
  explicit Window size) and added `scripts/build.sh`, `deploy-and-run.sh`,
  `restore-xochitl.sh`. Seeded the source tree (`CMakeLists.txt`, `src/main.cpp`,
  `qml/Main.qml`).
- **0.4 (2026-06-25)** — Researched the toolchain (§6b): identified the official
  Chiappa SDK (`3.27.0.97` / Codex 5.7.119, x86_64 host, ~486 MB), install/
  activate steps, the Qt Quick build + `-platform epaper` run commands, and the
  deploy-and-stop-xochitl flow. Confirmed Qt-Quick-only + custom-pen constraints.
- **0.3 (2026-06-25)** — Phase 1 on-device investigation substantially complete
  (see `Chatter_phase1_findings.md`). Resolved the stack decision to C++/Qt 6 +
  `epaper` QPA plugin; recorded display = DRM/KMS only, pen/finger as separate
  evdev devices, toggle = display hand-off, deploy under `/home`, and the
  document-store format for the Save phase.
- **0.2 (2026-06-25)** — Phase 0 access established. Added confirmed device
  facts (§6a): codename "Chiappa", NXP i.MX93 SoC, Codex Linux 5.7.121
  (scarthgap), dropbear SSH, passwordless key access from Jatke. Corrected the
  dev-mode menu path and documented that wifi SSH is off by default
  (`rm-ssh-over-wlan on`).
- **0.1 (2026-06-25)** — Initial version: scope, target device (RM03A) and its
  constraints, roles, phase plan (with Toggle before Save), Phase 1 investigation
  checklist, save-format clarification, and open decisions.