chatter/doc/Chatter_technical_reference.md

# Chatter — Technical Reference (master document)

**Audience:** a programmer who needs to understand, build, modify, or maintain
Chatter — including how the reMarkable standard software works and how Chatter
fits alongside it.

This is the **index + complete architecture**. Several topics have their own
deep-dive documents; this file summarizes each and links to it, then fills in
everything not covered elsewhere. Read this first; follow the links for detail.

## Document map

| Document | What it covers |
|---|---|
| [`Chatter_application_proposal.md`](Chatter_application_proposal.md) | The why: the assistive-speech use case, user-experience goals. |
| [`Chatter_implementation.md`](Chatter_implementation.md) | The phased plan and its running changelog; device facts, toolchain, decisions. |
| [`Chatter_phase1_findings.md`](Chatter_phase1_findings.md) | On-device investigation results (input devices, display, doc store). |
| [`Chatter_stylus_research.md`](Chatter_stylus_research.md) | Pen rendering, the Calligraphy model, width calibration, tooling. |
| [`Chatter_user_guide.md`](Chatter_user_guide.md) | The end-user-facing instructions. |
| [`developer_mode_screen.md`](developer_mode_screen.md) | Enabling developer mode on the device. |
| `tablet_access.md` | **SENSITIVE** (root password + IPs) — gitignored, not in the repo. |

---

## 1. The device

**reMarkable Paper Pro Move — model RM03A, codename "Chiappa".**

- **SoC:** NXP i.MX93 (Arm Cortex-A55, aarch64), 2 GB RAM, 64 GB storage.
- **OS:** "Codex" Linux 5.7.121 (Yocto *scarthgap*), image `3.27.1.0`. SSH server
  is **dropbear** (not OpenSSH).
- **Display:** E Ink **Gallery 3 — color** (ACeP, "acep2"), 7.3", **954×1696
  logical / 960×1696 framebuffer, 264 PPI** (≈10.4 px/mm). Driven by **DRM/KMS
  only** (`imx-drm`); there is **no `/dev/fb*`**.
- **Input:** pen = `/dev/input/event2` (Elan marker: `ABS_X` 0–6760, `ABS_Y`
  0–11960, `ABS_PRESSURE` 0–4096, `ABS_TILT_X/Y` ±9000, `BTN_TOOL_PEN`/
  `BTN_TOOL_RUBBER` for flip-to-erase). Finger = `/dev/input/event3` (10-point
  capacitive, `ABS_MT_*`, grid 1248×2208). Power = `event0`, hall/folio = `event1`.
  Pen and finger are **separate evdev devices** — Chatter tells them apart natively.

Full investigation: [`Chatter_phase1_findings.md`](Chatter_phase1_findings.md).

### 1.1 Filesystem / persistence model (critical)

- **`/` (rootfs, `/dev/mmcblk0p3`)** — ext4, mounted **read-only**, but
  **persistent**. Remount rw (`mount -o remount,rw /`) to write; survives reboots,
  **lost on an OS update** (A/B partition swap).
- **`/etc`, `/run`, `/var/volatile`** — **VOLATILE overlays** (upperdir on tmpfs).
  Anything written here is **lost on reboot**. (This is why a systemd unit dropped
  in `/etc/systemd/system` vanished after a reboot.)
- **`/home`** — encrypted, **fully persistent**, ~45 GB free. Everything Chatter
  installs lives here (`/home/root/chatter/`). SSH keys persist because they are
  under `/home/root/.ssh`.

**Consequence:** the launcher's systemd unit is installed onto the **rootfs**
(`/usr/lib/systemd/system`) so it survives reboots; the binary and scripts live
under `/home`. **An OS update wipes rootfs changes — disable OS auto-updates on a
delivered device, and keep `scripts/install-launcher.sh` to reinstall.**

---

## 2. The standard reMarkable software stack

Understanding the stock stack is necessary because Chatter reuses its display
plumbing and hands off to it.

- **`xochitl`** — the stock note app. Qt **6.8.2** / Qt Quick, proprietary. Owns
  the display while running. Started/stopped as the `xochitl` systemd service.
- **The `epaper` QPA platform plugin** (`libqsgepaper.so`, in
  `/usr/lib/plugins/`). A Qt platform + **software** scene-graph renderer for the
  e-paper. A Qt app runs on it with `-platform epaper` (+ `QT_QUICK_BACKEND=epaper`).
  Key facts learned by reverse engineering:
  - It is a **software** renderer: custom `QSGGeometryNode`s are silently dropped;
    only textures/rects/glyphs render. (This is why Chatter cannot draw ink via
    the scene graph — see §3.)
  - **`EPFramebuffer`** is the panel singleton (actually `EPFramebufferAcep2` on
    this color device), guarded by an **flock** at `/tmp/epframebuffer.lock` —
    only one process may hold the panel. Success marker in the journal:
    `SWTCON initialized \o/`. A stranded holder → `Failed to lock epframebuffer` /
    `Failed to initialize SWTCON` and a blank panel. **Always stop the previous
    holder first.**
  - Relevant `EPFramebuffer` methods (mangled symbols bound in `src/epfb.h`):
    - `instance()` → the singleton.
    - `setBuffers(std::tuple<QImage,QImage>, QImage*)` — sets the front/back
      buffers; **cross-DSO and interposable** (see §3).
    - `swapBuffers(QRect, EPContentType, EPScreenMode, QFlags<UpdateFlag>)` —
      pushes a region to the panel. A single explicit call renders **solid**
      (no dashed two-pass). Screen modes (from `EPScreenModeItem::Mode`):
      `Pen=0, Mono=1, Animation=2, UI=3, Content=4, Sleep=5`. Chatter uses
      `Pen` (0) for fast ink and `Content` (4, "full update, STD") for full
      refreshes.
    - `ghostControl(GhostControlMode)` — the panel's anti-ghosting API. Modes 0/3
      do an immediate full-screen de-ghost via the *region* `swapBuffers`; mode 1
      schedules one. **Chatter does NOT call it** — its region-swap path uses
      internal `EPContentMap`/`EPScreenModeMap` members the stock app maintains and
      we do not, so calling it corrupts state and hangs after a few calls. Chatter
      de-ghosts manually instead (§6.3).
  - The panel uses **ACeP color waveforms** (`acep2_lut`, `get_waveform_data`,
    software TCON "SWTCON"). Color **ghosting** is real and only cleared by a
    full-update waveform driven to the **dark** extreme (§6.3).
- **Pen styles** live in `~/.config/remarkable/xochitl.conf`, key
  **`LastWritingTool`** (a `@Variant` `QVariantMap`, readable via `QSettings`):
  `LastPen` (tool id, e.g. 21 = Calligraphy, 16 = a fineliner), `LastPenSize`
  (category 1/2/3 = thin/thicker/thickest), `LastPenColorCode` (`0xAARRGGBB`).
  **xochitl writes this to disk only when you leave a document** (return to the
  document list) — not on toolbar taps, and not reliably on an abrupt stop.

---

## 3. Chatter architecture (the core idea)

**Problem:** the make-or-break requirement is that ink look like the stock pen —
solid, crisp, low-latency. The `epaper` software scene graph **cannot** do this:
geometry nodes don't render, and `QQuickPaintedItem` textures always come out as
the e-ink **dashed two-pass** refresh, regardless of screen mode or antialiasing.

**Solution — a direct-framebuffer ink pipeline that bypasses the Qt scene:**

1. **`FbCapture`** (`src/FbCapture.{h,cpp}`) interposes
   `EPFramebuffer::setBuffers` **in-process**. The executable is linked
   `-Wl,--export-dynamic`, so its definition of that symbol wins the cross-DSO
   call from the plugin; we capture the real framebuffer's pixel memory and wrap
   it as a `QImage` (`FbCapture::framebuffer()`) with **no copy** (`constBits`).
   (Intra-DSO calls like `swapBuffers` are *not* LD_PRELOAD-interposable due to
   direct binding — but in-process `--export-dynamic` on `setBuffers` works.)
2. **`InkEngine`** (`src/InkEngine.{h,cpp}`) draws strokes with `QPainter`
   **straight into that framebuffer image**, then calls
   `EPFramebuffer::swapBuffers` (via `src/epfb.h`) on just the dirty rectangle.
   One explicit swap → **solid, single-pass** ink.
3. **Qt Quick renders only the static UI** (the two buttons). The scene never
   recomposites over the ink because nothing in it animates.

This is what gives stock-quality strokes. The reverse-engineering trail and the
dead-ends (geometry nodes, screen modes, antialiasing) are in
[`Chatter_stylus_research.md`](Chatter_stylus_research.md) and the
`Chatter_implementation.md` changelog (v0.7).

### 3.1 Process / run model

- Chatter is a single Qt 6 executable run as a **transient systemd service**
  (`systemd-run --unit=chatter …`), with `QT_QUICK_BACKEND=epaper`,
  `LD_LIBRARY_PATH=/usr/lib/plugins/scenegraph`, `-platform epaper`.
- It **requires xochitl to be stopped** (single DRM master + the panel flock).
- Managed via `systemctl {stop,status} chatter` and `journalctl -u chatter`.

---

## 4. Source components

```
src/
  main.cpp          Entry point: reads pen style, wires PenDevice → InkEngine,
                    exposes `ink`/`appControl` to QML, loads the QML UI.
  FbCapture.{h,cpp} Interposes EPFramebuffer::setBuffers; exposes the live
                    framebuffer QImage + FbCapture::ready().
  InkEngine.{h,cpp} THE core. Virtual canvas, ink drawing, finger-erase, buttons
                    (paint/hit-test/actions), two-finger scroll, de-ghosting.
  PenDevice.{h,cpp} evdev reader for the pen (event2); maps to screen coords;
                    emits strokeStart / strokeMove(pos,pressure,tiltX,tiltY,eraser)
                    / strokeEnd.
  AppControl.{h,cpp} returnToStandard(): transient unit stops chatter, starts xochitl.
  epfb.h            asm-label bindings to the private EPFramebuffer symbols
                    (instance, swapBuffers, ghostControl).
  EPScreenModeItem.{h,cpp}  Legacy/unused: binding to the private screen-mode item
                    from the scene-graph era. Kept for reference.
  InkCanvas.*, Experiment.*  Legacy from the Qt-Quick-canvas spike; NOT built.

qml/Main.qml        White Window; the Back/Clear buttons (TopButton); a
                    MultiPointTouchArea for finger erase (1) and scroll (2).

tools/
  chatter_launcher.c  The return-to-Chatter daemon (4-finger watcher on event3).
  grabtest.c          Probe whether an input device is EVIOCGRAB-exclusive.
  fbdump.cpp          LD_PRELOAD framebuffer snapshot (BMP; device has no PNG plugin).
  swapshim.cpp, setbufshim.cpp  Feasibility shims used during RE.

scripts/
  build.sh            Source the SDK env, cmake build.
  deploy-and-run.sh   Stop xochitl+chatter, scp, relaunch as a transient unit.
  to-chatter.sh       Switch standard → Chatter (run by the launcher).
  install-launcher.sh Persistently install the launcher unit on the rootfs.
  chatter-launcher.service  The systemd unit (installed to rootfs).
  restore-xochitl.sh  Return to the standard GUI.
```

---

## 5. Input & gesture model

Two independent input streams, never confused:

- **Pen (`event2`)** — read by `PenDevice` and delivered to `InkEngine`. The
  `epaper` QPA does **not** deliver the pen to Qt, so the pen never triggers QML.
- **Finger (`event3`)** — delivered by the `epaper` QPA to Qt as touch, handled in
  QML (`MultiPointTouchArea`, and the buttons' `MouseArea`s).

Gesture map:

| Input | Action | Where handled |
|---|---|---|
| Stylus draw | Ink (flip = erase via `BTN_TOOL_RUBBER`) | PenDevice → InkEngine |
| Stylus tap on a button | Button action | InkEngine hit-tests (`m_buttons`) |
| **1 finger** drag | **Erase wipe** (~12 mm; a tap erases nothing) | QML MultiPointTouchArea → `InkEngine::erase*` |
| **2 fingers** drag | **Vertical scroll** | QML → `InkEngine::panBy/panEnd` |
| Finger tap on a button | Button action (gray feedback) | QML MouseArea → `InkEngine::flashButton/activateButton` |
| **4 fingers** hold (~700 ms) | **Return to Chatter** (only while xochitl is front) | `tools/chatter_launcher.c` |

The MultiPointTouchArea **latches** the gesture type until all fingers lift, so a
two-finger scroll never degrades into an erase when one finger is raised. Erase
requires movement past a ~1.5 mm threshold (tap-safe).

Buttons (`Back`, `Clear`) are a **single source of truth** in `InkEngine`: QML
registers their geometry (`registerButton`) so the engine can exclude ink, redraw
them after a blit, and hit-test stylus taps. Press feedback (`flashButton`) and
actions (`activateButton`) are drawn directly to the framebuffer (instant, no
scene flashing); both finger and stylus route through them.

---

## 6. The virtual canvas, scrolling, and de-ghosting

### 6.1 Growable raster canvas
`InkEngine` holds a `QImage` **canvas** larger than the screen (starts 2× tall,
**grows downward** as you write near the bottom). The screen is a **viewport**
into it at vertical offset `m_panY`. Drawing maps screen→canvas (`+m_panY`); a
dirty canvas rect is blitted back to the framebuffer (`blitRegion`) and the
buttons are repainted on top. (Storage model chosen: raster, to preserve the
exact ink quality; trade-off is no crisp zoom-in. Horizontal/zoom are future work.)

### 6.2 Scrolling
Two-finger drag → `panBy(dy)` (natural: content follows fingers), clamped to the
canvas, fast-blitted per step. `panEnd()` debounces a de-ghost (§6.3).

### 6.3 De-ghosting (color ACeP ghosting)
Fast `Pen`-waveform swaps leave **color residue** ("faint red duplicate") that
accumulates while scrolling and is **not** cleared by a white redraw — it is
panel retention, cleared only by a full-update waveform driven to **black**
(white/gray do not clear it; this was tested). Chatter's `fullRefresh()`:
fills the screen **black** + full-update swap (`screenMode=Content`), waits
~220 ms, then full-updates the real content. `ghostControl()` would be the
"proper" API but corrupts state (§2), so this manual flash is used.

To keep the flash from being intrusive:
- It runs **only after scrolling**, **debounced** ~1 s after the last scroll
  (not on every finger-lift).
- **Clear has two methods:** if there was **no scrolling** since the last clear
  (the common fill-one-screen case) it uses the **gentle fast clear**; if there
  **was** scrolling, it does the **black de-ghost** clear. (`m_scrolled` flag.)
- Tunables via env: `CHATTER_FULL_SM` (full-update screen mode, default 4),
  `CHATTER_FLASH_GRAY` (flash level 0=black..255; black is what actually clears).

A black flash is intrinsic to clearing color ghosting (the stock UI flashes on
its full refreshes too); we minimized *when* it happens rather than eliminating it.

---

## 7. Pen-style matching, width calibration, calligraphy

- Chatter reads `xochitl.conf` `LastWritingTool` at startup (`readPenStyle` in
  `main.cpp`) and applies tool type, size, and color — **no style menu in
  Chatter**. Because each switch-to-Chatter restarts the process, it re-reads the
  current pen. **The flush sequence matters** (§2): set the pen, *use it*, **leave
  the document**, then switch to Chatter.
- **Width** is calibrated to measured widths on the 264-PPI panel:
  size 3 ≈ 2 mm, size 2 ≈ 1 mm, thinnest ≈ 2 px; pressure ≈ doubles width.
- **Calligraphy** (tool 21) is approximated with a dynamic
  direction/pressure/speed/tilt width model. Full detail, measurements, and the
  formula: [`Chatter_stylus_research.md`](Chatter_stylus_research.md).

---

## 8. Toggle & the return launcher

- **Chatter → standard:** `Back` → `InkEngine::backRequested` →
  `AppControl::returnToStandard()` → a transient unit stops chatter and starts
  xochitl (sequenced so the panel lock is released first).
- **standard → Chatter:** `tools/chatter_launcher.c` runs always as a rootfs
  systemd service. It reads `event3` **without grabbing it** (verified possible
  via `tools/grabtest.c` — xochitl does not hold an exclusive grab), counts
  multitouch slots, and on a **4-finger hold ≥700 ms** while xochitl is the front
  app runs `to-chatter.sh`. It reopens the device on any read interruption (sleep/
  wake) so it never dies.
- **Install persistently:** `scripts/install-launcher.sh` (remounts the rootfs rw,
  places the unit + its `multi-user.target.wants` symlink under
  `/usr/lib/systemd/system`). See the persistence model in §1.1.

---

## 9. Build & deploy

**Toolchain:** official reMarkable Chiappa SDK `3.27.0.97` (Qt 6.8.2 sysroot,
x86_64 host). Install, then `source environment-setup-cortexa55-remarkable-linux`.

```
scripts/build.sh            # source SDK env + cmake build  -> build/chatter
scripts/deploy-and-run.sh   # stop xochitl+chatter, scp, relaunch as transient unit
scripts/restore-xochitl.sh  # back to the standard GUI
scripts/install-launcher.sh # one-time (and after each OS update): persist the launcher
```

`CMakeLists.txt`: Qt6 Quick app (`qt_add_executable` + `qt_add_qml_module`), links
`libqsgepaper.so`, and crucially `target_link_options(... -Wl,--export-dynamic)`
so the `setBuffers` interposition wins.

**Device housekeeping:** only one process may hold the panel — always stop the
previous holder. Deploy under `/home/root/chatter`, never the rootfs (except the
launcher unit). The device sleeps and DHCP may reassign its IP on wake; if you
script a reconnect, rescan for the tablet's MAC address (shown on the device's
GPLv3-compliance screen, alongside its IPs).

---

## 10. Status & open items

**Working on hardware:** solid stock-quality ink matching the selected pen;
finger-wipe erase; whole-page Clear (two methods); stylus eraser; bidirectional
toggle (Back + 4-finger launcher, persistent); vertical scroll on a growable
canvas with debounced de-ghosting; instant button feedback (finger + stylus).

**Open / future:**
- **Save** (Phase 5) — write a timestamped transcript into a "Chatter" folder in
  xochitl's on-disk document format; importance to the user still unconfirmed.
  See `Chatter_implementation.md` §6 / §6a.
- **Horizontal scroll & zoom-out overview** — deferred extensions of the canvas.
- **Calligraphy fidelity** — "close enough" approximation; exact nib unknown.
- **Erase / gesture thresholds** — to be tuned with the end user (Matt's field test).
- **OS auto-update** would wipe the rootfs launcher unit (and could break paths) —
  disable it on a delivered device.

---

*This document is the technical entry point. When the design changes, update this
file and the relevant component doc; keep `Chatter_implementation.md`'s changelog
as the chronological record.*