ack/chatter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: Chatter — assistive-writing app for reMarkable Paper Pro Move

Browse Source 
Direct-framebuffer ink pipeline (stock-quality strokes), finger-wipe erase,
growable scrolling canvas with color-ghost cleanup, bidirectional toggle with a
persistent 4-finger return launcher, instant button feedback. Includes prebuilt
aarch64 binaries (dist/), build/deploy/install scripts, a user guide, and a
complete technical reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Andy Kopra
2026-06-27 16:34:28 +02:00
commit 5f21d9099c
43 changed files with 2878 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/.gitignore vendored Normal file
View File

@@ -0,0 +1,2 @@
tablet_access.md

96
doc/Chatter_application_proposal.md Normal file
View File

@@ -0,0 +1,96 @@
# Chatter — A custom application for the reMarkable tablet
## Background
I am helping someone who has a hard time speaking and who is using a
[reMarkable tablet](https://image.email.remarkable.com/lib/fe3511737364047c771479/m/1/e39f7595-fc99-4f8c-b7e3-dde042451668.pdf)
in conversations to write what she cannot say. I believe that the
reMarkable table is the best choice for a digital handwriting interface.
However, the tablet is designed for taking notes that can be preserved or even
transformed into digital text. It is not intentionally designed to be used as an
*assistive-speech device*. There are also a wide variety of customizable options
that are typically unnecessary if using the table in that way.
## Project goal
I would like to create a replacement for the reMarkable graphical user interface
(GUI) that would be tailored for using the table as an assistive-speech device.
This interface would not replace the standard GUI; choosing tablet settings for
the writing style or other interface parameters would still be done in the
standard GUI. This assumes, however, that the user can change settings as needed
in the normal way (or that technical help is available when necessary).
The goal here is to make the interaction with the tablet during conversation as
simple as possible.
The custom application's name is "Chatter", with the implication that
conversational interaction can be accelerated. The word also can refer to the
speaker as well as to the result of speaking.
## User-interface elements
In watching the user with the tablet's standard GUI, I have noticed a confusion
about whether writing or erasing is the active mode. A simple action like
clearing the page requires several user-interface gestures. The user's
inability to easily use the table is often not wholly due to a lack of technical
understanding but a failure to remember technical details when they are not
intuitive.
Chatter implements four actions, ordered here by their potential frequency of
use:
- *Erase part of the text* — This should be simple and intuitive gesture. For
example, I have seen the user wiping her fingers back and forth across some
text to erase it.
- *Erase the page* — Erasing the entire page should be a single gesture. The
blank page that results continues to use the current pen styling and other
parameters set in the standard GUI.
- *Save current text to a file named by time* — The current page is saved
with an automatically generated name that contains a human-readable text of the year,
month, date, hour, minutes and seconds. The seconds may not be necessary, but
I would rather have an elaborate name than risk losing a previously saved
file. These Chatter transcripts should be saved to a separate directory to
avoid cluttering the main screen. Deleting and renaming these files are
possible through the standard GUI; a tutorial about that should be provided if
the user expresses interesting in managing the transcripts in that way.
- *Toggle between the standard GUI and Chatter* — The standard GUI should still
be available through a single gesture; Chatter augments the tablet's interface
and does not replace it. However, returning to Chatter will require an
addition to the standard GUI that will be memorable for the user.
### Implementation considerations
If the table can differentiate between a fingertip and the stylus, then erasing
part of the text as well as the entire page could be implemented by a finger
gesture. Rubbing the finger back and forth of an area of text could remove any
drawing close to that area (where "close" might be determined experimentally
working with the user). A diagonal stroke with a finger from upper-right to
lower-left would be interpreted as a request to erase the entire page.
If differentiating between fingertip and stylus is not possible, then Chatter
should define some shortcut for the stylus that easily creates the standard
region-based erase gesture. For example, if the user circles an area twice,
that could signify erasure, not drawing. A stylus stroke from upper-right to
lower-left could specify whole-page erasure, even if the finger stroke
method is also possible.
The toggling the standard GUI and saving a file are not actions that the user
would frequently perform. I think that small icons at the bottom of the page,
activated by stylus or finger, would provide an ongoing reminder of how to do
those actions. The standard GUI will need to provide some way of switching to
Chatter; the user should suggest a method that she would find memorable.
## Deployment and refinement
A developer working on this project will install the custom GUI on the user's
tablet. After verifying that the installation is successful, the developer will
watch the user with the tablet to evaluate the current interface efficiency of
Chatter, talking with the user about what is missing and what could be improved.
Paying attention to the user's questions that begin, "Why does this..." and "Why
doesn't this..." will be very important in refining the design. I believe that
using a custom interface for the tablet might also inspire the user to suggest
other features that would not be obvious to the developers.

381
doc/Chatter_implementation.md Normal file
View File

@@ -0,0 +1,381 @@
# Chatter — Implementation Plan & Method
**Status:** Living document — revised as we learn. Read the Changelog at the bottom for what changed.
**Version:** 0.7 (2026-06-26)
**Audience:** The Chatter development team (Andreas, Matt, and Claude as development assistant).
> This document is the shared source of truth for *how* we build Chatter. It is
> deliberately incremental: early phases will answer questions that later phases
> depend on, and this document will be updated as those answers come in. If
> something here is marked **(unknown / to confirm on-device)**, treat it as an
> open question, not a decision.
---
## 1. What Chatter is
Chatter is a minimal, purpose-built interface for a reMarkable tablet used as an
**assistive-speech device** by a user who has difficulty speaking — she writes,
and her conversation partner reads. Chatter **augments** the tablet; it does not
replace the standard reMarkable GUI.
The design priority is **intuitiveness under intermittent recall**: the user
should not have to remember technical steps. See the full motivation and the
user-experience reasoning in [`Chatter_application_proposal.md`](Chatter_application_proposal.md).
Chatter's core actions, by expected frequency:
1. **Erase part of the text** — a simple, intuitive gesture (e.g. wiping a
fingertip back and forth over an area).
2. **Erase the whole page** — a single gesture (e.g. a diagonal finger stroke,
upper-right → lower-left).
3. **Toggle to/from the standard GUI** — *necessary*; Chatter must be able to
hand off to the stock interface and be returned to.
4. **Save the current page** — *importance to the user is not yet confirmed*; a
single button writes a timestamped transcript into a "Chatter" folder.
---
## 2. Target device and key constraints
**Device: reMarkable Paper Pro Move — model RM03A.** (Identified from the
official user-guide PDF linked in the proposal.)
| Property | Value | Why it matters |
|---|---|---|
| Display | E Ink Gallery 3, **color**, 7.3", 954×1696 (264 PPI), backlit | Color refresh pipeline differs from older grayscale models |
| Touch | Multi-point capacitive | Finger gestures (erase) are viable |
| Pen | Active Marker / Marker Plus (flip-to-erase on Plus) | Pen and touch are **separate input streams** → we can tell finger from stylus natively |
| SoC / RAM | ARM 1.7 GHz dual A55, 2 GB RAM, 64 GB | Standard ARM-Linux cross-compile target |
| OS | "Codex" (custom Linux) | Stock note app is Xochitl (Qt/C++, proprietary) |
| Dev access | Enable **developer mode**, which requires a **factory reset** | **Back up the device first** |
**Key risk — this is the newest model.** The mature community frameworks
(rmkit, libreMarkable, Plato, KOReader ports) were built against the older
RM1/RM2 grayscale framebuffer and refresh ioctls. They likely **do not support
the Paper Pro Move yet**, or only partially. We should expect to do
**first-principles work** — discovering this model's framebuffer format, color
refresh path, and input device nodes by probing the hardware over SSH — rather
than relying on an existing framework. This raises the value of hands-on device
access early and argues for a small, well-proven first milestone.
---
## 3. Roles
- **Andreas** — development, with Claude's assistance, on his own Paper Pro Move
over SSH.
- **Matt** — installation and QA, including on-site testing with the end user on
her tablet. Matt has the same model (Paper Pro Move). Matt may also take on
development; the split will be worked out as the project progresses.
- **Claude** — development assistant: writes/reviews code, drives the
build-deploy loop over SSH where possible, and maintains this document.
This document is how Matt stays in sync. When the design changes, the change
lands here first.
---
## 4. Development phases
The phases are ordered so each de-risks the next. **Toggle (necessary) precedes
Save (provisional).**
### Phase 0 — Safety & access
- **Back up first** (critical on the user's device; skippable on a new tablet
with nothing saved). Enabling developer mode **performs a factory reset** — on
the Paper Pro family, SSH is unavailable until developer mode is on, and
turning it on wipes the device. Sync to the reMarkable cloud account before
enabling. Document this backup procedure so Matt can repeat it.
- **Enable developer mode:** Hamburger menu (upper left) → Settings → Software →
Advanced → Developer mode → Accept. This factory-resets the device (deletes
all local files; user data cleared on reboot). Re-pair with the reMarkable
account afterward to access cloud content. (Disabling later requires
reMarkable's recovery application.)
- **Reconnect wifi** after the reset (the wifi config is wiped).
- **Get SSH credentials:** Menu → Settings → Help → Copyright and licenses;
the root username (`root`), password, and device IP addresses are under the
"GPLv3 Compliance" header. USB exposes the tablet at `10.11.99.1`.
- **Enable SSH over wifi (it is OFF by default):** connect over USB, SSH in,
and run `rm-ssh-over-wlan on`. Only then is the listed `192.168.x.x` address
reachable; after that the USB cable is optional.
- Install your workstation's SSH public key into the tablet's
`~/.ssh/authorized_keys` (user `root`) for passwordless access.
- Confirm SSH login, `scp` of a binary, and remote execution.
- Stand up the x86_64-hosted cross-toolchain for Codex OS; verify a trivial
ARM binary runs on-device.
> Dev environment note: development host is **Jatke** (on the wired LAN); we
> connect to the tablet **over wifi** (USB not in use).
### Phase 1 — On-device investigation (no app code; produces a findings writeup)
Answer the questions that can only be answered with the tablet over SSH. See the
checklist in §5. Output: a short findings document added to `doc/`.
**Status: substantially complete (2026-06-25)** — see
[`Chatter_phase1_findings.md`](Chatter_phase1_findings.md). Key results: pen
(`event2`) and finger (`event3`) are separate evdev devices; the display is
DRM/KMS only (no fbdev) driven by `imx-drm`; and the stock app is a Qt 6.8.2 app
rendering through a reusable **`epaper` QPA plugin** — so Chatter should be a
Qt 6 app run with `-platform epaper`. Remaining sub-items (verifying `-platform
epaper` from a third-party binary, input delivery, toggle hand-off) roll into
the Phase 2 spike.
### Phase 2 — Minimal canvas spike
Read the pen, render strokes to the (color) display, and clear the page —
nothing more. This proves the two hardest pieces (input + color refresh) end to
end and de-risks everything after it.
**Status (2026-06-25): display/toolchain half PROVEN on hardware.** A
third-party Qt Quick app (`build/chatter`) cross-built with the Chiappa SDK
renders crisp black-on-white on the e-paper via `-platform epaper`
(`QT_QUICK_BACKEND=epaper`).
**Status (2026-06-26): inking WORKS, with one known quality issue.** Custom
evdev pen reader (`src/PenDevice.cpp`, reads `event2`) drives a
`QQuickPaintedItem` canvas (`src/InkCanvas.cpp`): ink tracks the pen, orientation
correct, eraser (BTN_TOOL_RUBBER) erases, finger-tap "Clear" works, latency good.
- **Known issue:** strokes render as a "dashed line that fills in" (two-pass),
unlike the solid stock pen. **Root cause:** we use the panel's default
*grayscale* mode. **Fix path:** xochitl wraps its writing area in a QML
`ScreenModeItem` set to **Mono/FAST** with a dedicated **pen waveform**
(`/usr/share/remarkable/ct33_pen.bin`). The control is the private
`EPScreenModeItem` class in `libqsgepaper.so` (`setMode()`, modes
Mono/FAST/FastGrayscale) — not a public QML type, no SDK header — so Chatter
must integrate it directly (instantiate, link the plugin, drive `mode` via the
meta-object, wrap the canvas). Antialiasing on/off does not affect it.
- **Pen-style decision:** Chatter uses the pen style set in the standard GUI
(xochitl.conf `LastPen`/`LastPenSize`/`LastPenColor`, readable via QSettings),
re-read so it tracks changes the user makes after toggling — no style menu in
Chatter.
**Lessons learned (encoded in `scripts/`):**
- The e-paper framebuffer is a **singleton guarded by an flock**
(`/tmp/epframebuffer.lock`, from `libqsgepaper`'s `EPFramebufferAcep2`). Only
ONE process may hold it; a stranded instance shows `Failed to lock
epframebuffer` / `Failed to initialize SWTCON` and the panel stays blank.
Always stop the previous instance first. Journal success marker:
`SWTCON initialized \o/`.
- **Run as a transient systemd service** (`systemd-run --unit=chatter …`), not an
ssh background job (which holds the SSH channel open and can strand the
process). Manage with `systemctl stop chatter` / `journalctl -u chatter`.
- A full-screen `Window` needs an **explicit size** (`width: Screen.width;
height: Screen.height`); `visibility: FullScreen` alone left it unsized/blank.
- xochitl runs with **no special Qt env** — `-platform epaper` +
`QT_QUICK_BACKEND=epaper` suffices.
**Run recipe:** `scripts/build.sh && scripts/deploy-and-run.sh`;
`scripts/restore-xochitl.sh` to return to the standard GUI.
### Phase 3 — Erase gestures
- Region erase: finger wipe removes ink near the wiped area ("near" tuned
experimentally with the user).
- Whole-page erase: single diagonal finger stroke (upper-right → lower-left).
- Provide stylus fallbacks if finger/stylus separation proves unreliable.
### Phase 4 — Toggle to/from the standard GUI *(necessary)*
**Status (2026-06-26): BIDIRECTIONAL toggle works on-device (verified).**
- **Chatter → standard:** top-left **"Back"** button → `AppControl::returnToStandard()`
→ transient unit stops chatter, starts xochitl.
- **standard → Chatter:** a persistent **launcher daemon** (`tools/chatter_launcher.c`,
installed as a rootfs systemd service via `scripts/install-launcher.sh`) watches
the touch device and, on a **4-finger hold (~700 ms)** while xochitl is in front,
runs `to-chatter.sh` to switch to Chatter. Gesture is tunable; finalize the
memorable one with the user.
- **Deployment note:** `/etc` and `/run` are volatile overlays (wiped on reboot);
persistent installs go to `/home` (app/binaries/scripts) or the rootfs
(`mount -o remount,rw /`, lost on OS update). Disable OS auto-updates on a
delivered device.
- A gesture/button in Chatter hands off to the standard reMarkable GUI.
- A **memorable** mechanism returns from the standard GUI to Chatter. The
proposal asks that the user help choose this; we will prototype options and
let her pick.
- **Phase 1 insight:** because the display is DRM/KMS (single master) and
xochitl owns it while running, the toggle is a **hand-off**, not an overlay —
stop/pause `xochitl` to give Chatter the display, and reverse to return.
Mechanism (`systemctl stop/start` vs. `SIGSTOP/SIGCONT`) to be validated.
### Phase 5 — Save the page *(provisional — pending confirmation it matters to the user)*
- A single visible button saves the current page.
- Filename fully derived from date/time (year, month, day, hour, minute,
second).
- Written into a folder named **"Chatter"**, using the same on-disk document
format the stock app uses, so transcripts appear as normal notebooks and can
be renamed/deleted/synced via the standard GUI (see §6).
### Phase 6 — Field test & refine
Matt installs and observes the user. Capture especially the "Why does this…" and
"Why doesn't this…" questions; feed them back into this document and the phase
plan.
---
## 5. On-device investigation checklist (Phase 1)
These are the open questions the hands-on phase exists to answer.
- **Input devices:** which `/dev/input/event*` is the pen vs. the capacitive
touch; event format, coordinate range and orientation, pressure; the
flip-to-erase signal on Marker Plus.
- **Display:** how the color Gallery 3 panel is driven — framebuffer device and
pixel format, and the refresh mechanism (ioctls or this model's equivalent).
*(Highest-risk unknown.)*
- **Document store:** where Xochitl stores notebooks; the
`.metadata` / `.content` / per-page `.rm` layout for this model; the `.rm`
format version (newer, color-capable); how a *collection* (folder) is
represented; and whether a hand-built document appears in the library after a
restart/sync.
- **Coexistence & toggle:** how to launch our binary alongside / over Xochitl,
and what a reliable toggle and return-to-Chatter mechanism looks like.
---
## 6. The "save" approach (clarification)
reMarkable exposes **no public callable on-device API** for creating notebooks.
The stock app simply **writes files to an on-disk document store** in a specific
format (roughly: a UUID per document, a `.metadata` JSON, a `.content` JSON, and
one binary `.rm` line-file per page, plus the color/folder data and a sync
layer). A "folder" such as the **Chatter** folder is itself a document of type
*collection*.
So "save using the same method the environment uses" means: **write our canvas
into the exact on-disk format Xochitl reads, for this model's format version.**
The open question for Phase 5 is confirming/replicating the Paper Pro Move's
current `.rm` format. This is investigated in Phase 1.
---
## 6a. Confirmed device facts (from SSH, 2026-06-25)
- **Internal codename:** "Chiappa" (`/proc/device-tree/model` = "reMarkable
Chiappa"; hostname `imx93-chiappa`). This is the Paper Pro Move (RM03A).
- **SoC:** NXP **i.MX93** (Arm Cortex-A55). Relevant to the toolchain target and
to how the display is driven.
- **OS:** Codex Linux **5.7.121** (Yocto **scarthgap**), image version
`3.27.1.0`.
- **SSH server:** **dropbear** (not OpenSSH). Root login; `~/.ssh/authorized_keys`
honored. Wifi SSH enabled via `rm-ssh-over-wlan on`.
- **Access from Jatke:** passwordless via ed25519 key as `root` over wifi (the
tablet's IP) or `root@10.11.99.1` (USB, fixed on every reMarkable). The tablet's
actual IP/MAC and credentials are in `tablet_access.md` (SENSITIVE, gitignored).
**Phase 0 status:** access established (dev mode, SSH over USB + wifi, key-based
login, scp verified). Remaining for Phase 0: stand up the cross-toolchain and
run a trivial ARM binary on-device.
## 6b. Toolchain / build & deploy (researched 2026-06-25)
**Official SDK (cross-toolchain + Qt 6.8.2 sysroot), self-extracting `.sh`,
x86_64 host only.** For this device (codename **chiappa**):
```
https://storage.googleapis.com/remarkable-codex-toolchain/3.27.0.97/chiappa/remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
```
- ~486 MB. Device image is `3.27.1.0` (Codex 5.7.121); newest published Chiappa
SDK is `3.27.0.97` (5.7.119) — same `3.27.x` family, so binaries built against
it run on the device. (An aarch64-host variant also exists; we use x86_64 for
Jatke.) Download links live on developer.remarkable.com/links; the GCS bucket
itself is not directly listable (403).
- Build host **Jatke** (x86_64, ample disk). SDKs expect a Yocto-supported Linux
host.
**Install & activate:**
```
chmod u+x remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh
./remarkable-production-image-5.7.119-chiappa-public-x86_64-toolchain.sh -d <install-dir>
source <install-dir>/environment-setup-cortexa55-remarkable-linux # exact filename confirmed post-install (i.MX93 = Cortex-A55)
```
**Build a Qt Quick app** (per developer.remarkable.com/documentation/qt_epaper):
CMake with `find_package(Qt6 REQUIRED COMPONENTS Quick)` +
`qt_add_executable` / `qt_add_qml_module`; `cmake . -B build && cmake --build
build` after sourcing the SDK env.
**Deploy & run on device:**
```
scp build/<app> chatter:/home/root/chatter/ # deploy under /home, not rootfs
ssh chatter 'systemctl stop xochitl' # release the display (pre-authorized)
ssh chatter 'cd /home/root/chatter && QT_QUICK_BACKEND=epaper ./<app> -platform epaper'
```
Constraints (official + Phase 1): **Qt Quick/QML only, no Qt Widgets**; **xochitl
must be stopped** (DRM single-master); **touch is automatic, pen is custom**
(read `event2`). Some SDK/OS combos require copying `libqsgepaper.so` to the
device — verify during the spike.
## 7. Decisions & open questions
**Decided (from Phase 1):**
- **Implementation stack: C++ / Qt 6.8.2**, rendered via the device's **`epaper`
QPA plugin** (`-platform epaper`). Chosen because the stock app proves this
path works and gives us e-ink display + refresh for free; the older
fbdev-mmap approach is not available (DRM/KMS only). Cross-build with a Yocto
scarthgap aarch64 SDK matching Qt 6.8.2. *(Matt — flag any objection.)*
- **Deploy location: under `/home`** (45 GB free), never the rootfs (~88 MB
free, overlay, reset by updates).
- **Toggle is a display hand-off** with xochitl (see Phase 4), not an overlay.
**Still open:**
- **Save mechanism** — leaning **PDF-backed document** over authoring native v6
`.rm`; decide in Phase 5 (and gated by whether save matters to the user).
- **Whether "save" matters to the user** — learned from observation.
- **Toggle gesture + return-to-Chatter mechanism** — prototype and choose with
the user (Phase 4).
- **Erase "closeness" thresholds** — tuned experimentally with the user
(Phase 3).
---
## Changelog
- **0.6 (2026-06-26)** — Inking engine working on hardware: custom evdev pen
reader (`event2`) + `QQuickPaintedItem` canvas — pen draws, eraser erases,
finger-tap **Clear**, good latency. Added top control bar (**Back** left,
**Clear** right); Back hands off to the standard GUI (verified). Bound the
private `EPScreenModeItem` (key-function trick) to control panel refresh mode —
but the **dashed two-pass stroke refresh is the render loop's behavior,
independent of mode**, so it's deferred to a framebuffer-direct effort
(needs disassembly of `swapBuffers` for `EPContentType`/`UpdateFlag`).
Implemented **pen-style matching**: Chatter reads the standard GUI's selected
pen (type/size/color) from `xochitl.conf` `LastWritingTool` and applies width +
color — verified (Calligraphy, size 3, black).
- **0.7 (2026-06-26)** — ⭐ **Solved stroke quality with a direct-framebuffer ink
pipeline.** After proving Qt Quick can't render stock-quality e-ink strokes
(geometry nodes unsupported by the software backend; painted-textures always
get the dashed two-pass; screen-mode and antialiasing irrelevant), switched
architecture: `FbCapture` interposes `EPFramebuffer::setBuffers` in-process
(`-Wl,--export-dynamic`) to capture the real framebuffer image; `InkEngine`
draws strokes straight into it and refreshes each segment with an explicit
`swapBuffers` — **solid, crisp, low-latency, correctly-aligned strokes on
hardware (verified), plus working Clear.** Qt Quick now only renders the static
Back/Clear UI. Key RE findings: `swapBuffers` renders solid (not dashed) on a
single explicit call; intra-DSO calls aren't LD_PRELOAD-interposable but the
cross-DSO `setBuffers` is. Remaining: calligraphic nib (uniform round width vs
the angled direction-dependent Calligraphy nib), return-to-Chatter trigger,
Save, field testing.
- **0.5 (2026-06-25)** — Phase 2 spike: cross-built a Qt Quick hello-app with the
Chiappa SDK and rendered it on the e-paper via `-platform epaper` (proven on
hardware). Recorded lessons (panel flock singleton, run-as-systemd-service,
explicit Window size) and added `scripts/build.sh`, `deploy-and-run.sh`,
`restore-xochitl.sh`. Seeded the source tree (`CMakeLists.txt`, `src/main.cpp`,
`qml/Main.qml`).
- **0.4 (2026-06-25)** — Researched the toolchain (§6b): identified the official
Chiappa SDK (`3.27.0.97` / Codex 5.7.119, x86_64 host, ~486 MB), install/
activate steps, the Qt Quick build + `-platform epaper` run commands, and the
deploy-and-stop-xochitl flow. Confirmed Qt-Quick-only + custom-pen constraints.
- **0.3 (2026-06-25)** — Phase 1 on-device investigation substantially complete
(see `Chatter_phase1_findings.md`). Resolved the stack decision to C++/Qt 6 +
`epaper` QPA plugin; recorded display = DRM/KMS only, pen/finger as separate
evdev devices, toggle = display hand-off, deploy under `/home`, and the
document-store format for the Save phase.
- **0.2 (2026-06-25)** — Phase 0 access established. Added confirmed device
facts (§6a): codename "Chiappa", NXP i.MX93 SoC, Codex Linux 5.7.121
(scarthgap), dropbear SSH, passwordless key access from Jatke. Corrected the
dev-mode menu path and documented that wifi SSH is off by default
(`rm-ssh-over-wlan on`).
- **0.1 (2026-06-25)** — Initial version: scope, target device (RM03A) and its
constraints, roles, phase plan (with Toggle before Save), Phase 1 investigation
checklist, save-format clarification, and open decisions.

140
doc/Chatter_phase1_findings.md Normal file
View File

@@ -0,0 +1,140 @@
# Chatter — Phase 1 On-Device Findings
**Date:** 2026-06-25
**Device:** reMarkable Paper Pro Move (RM03A, codename "Chiappa")
**Method:** Read-only SSH probing from Jatke (`ssh chatter`). No changes made to
the device except enabling developer mode / SSH (Phase 0) and adding our SSH key.
This document records what we learned and the implementation decisions those
findings support. It feeds the phase plan in
[`Chatter_implementation.md`](Chatter_implementation.md).
---
## 1. Hardware & OS
- **SoC / arch:** NXP **i.MX93**, **aarch64** (Arm Cortex-A55). Kernel
`6.12.49+git-imx93-chiappa`.
- **OS:** Codex Linux 5.7.121 (Yocto **scarthgap**), image `3.27.1.0`. glibc
(`/lib/ld-linux-aarch64.so.1`, `libc.so.6`).
- **RAM:** 2 GB (~1.4 GB free) — ample for our app.
- **Storage (matters for deployment):**
- `/` (rootfs): ~435 MB, **only ~88 MB free (78% used)** — also an
OS-managed overlay, likely reset by firmware updates. **Do not install here.**
- `/home`: **46 GB encrypted volume, ~45.8 GB free.** Install Chatter under
`/home` (e.g. `/home/root/chatter`).
## 2. Input devices
Pen and finger are **separate evdev devices** — finger-vs-stylus discrimination
is free, exactly what Chatter's gesture design needs.
| Node | Name | Role | Notes |
|---|---|---|---|
| `event0` | `bbnsm:pwrkey` | Power button | |
| `event1` | Hall effect sensors | Folio open/close | switch events |
| **`event2`** | **Elan marker input** | **Stylus / Marker** | on SPI |
| **`event3`** | **Elan touch input** | **Capacitive multitouch (finger)** | `INPUT_PROP_DIRECT` |
**Pen (`event2`)** — digitizer space **6760 × 11960**:
- Buttons: `BTN_TOOL_PEN`, **`BTN_TOOL_RUBBER`** (Marker Plus eraser end —
flip-to-erase is detectable), `BTN_TOUCH`, `BTN_STYLUS`, `BTN_STYLUS2`.
- `ABS_PRESSURE` 0–4096, `ABS_DISTANCE` (hover) 0–65535, `ABS_TILT_X/Y` ±9000.
**Touch (`event3`)** — grid **1248 × 2208**:
- Up to **10 contacts** (`ABS_MT_SLOT` 0–9) with `ABS_MT_POSITION_X/Y`,
`ABS_MT_PRESSURE` (0–255), `ABS_MT_TRACKING_ID`, `ABS_MT_TOOL_TYPE`.
All three coordinate spaces share the screen's portrait aspect (~0.5625):
panel 954×1696, pen 6760×11960, touch 1248×2208.
## 3. Display pipeline
- **DRM/KMS only — there is NO `/dev/fb*`** and `/sys/class/graphics` is empty.
Driver is **`imx-drm`**; device `/dev/dri/card0`; connector
**`card0-LVDS-1`** ("connected", "disabled" at rest — normal for e-ink, which
holds its image without continuous scanout).
- The connector advertises a **packed `365×1700` mode** (hardware buffer for the
E Ink Gallery 3 color subpixel layout); the logical screen is 954×1696. The
mapping is handled by the epaper plugin (below) — we don't touch it directly.
- **Implication:** the `mmap /dev/fb0` approach used by rmkit / libreMarkable on
older grayscale models **does not apply** to this device.
## 4. Graphics / UI stack — the key finding
- The stock app **`xochitl`** is a **Qt 6.8.2 / Qt Quick (QML)** application.
- It renders through a **custom Qt platform (QPA) plugin: `epaper`**
(`/usr/lib/plugins/platforms/libepaper.so`, links `libdrm`). Available QPA
plugins on-device: **`epaper`**, `minimal`, `offscreen`, `vnc`.
- Qt 6.8.2 runtime libraries and QML modules (`QtQuick`, `QtQuickControls2`,
etc.) are all present under `/usr/lib` and `/usr/lib/qml`. No dev headers
on-device (expected — we cross-build).
**This means the native path is wide open:** build Chatter as a **Qt 6 app** and
run it with **`-platform epaper`** to inherit working e-ink display + refresh,
instead of reverse-engineering DRM/KMS and waveform handling. This resolves the
project's highest-risk unknown in our favor.
## 5. Stock app & services
- `xochitl.service` — the main UI app (`/usr/bin/xochitl`, pid varies).
- `marker-manager.service` — "Remarkable CSL Marker Manager" (pen support).
- `rm-sync.service` — document sync to the reMarkable cloud.
- Plus metrics / MDM / crash-uploader services.
**Coexistence / toggle implication:** on a DRM/KMS device only one process owns
the display (DRM master) at a time, and xochitl holds it (and likely grabs the
input devices) while running. So the Chatter ↔ standard-GUI **toggle is a
hand-off** (stop or pause `xochitl` ↔ run Chatter, and a return path), **not an
overlay**. The exact mechanism (systemctl stop/start vs. SIGSTOP/SIGCONT) is to
be validated in the Phase 2 spike.
## 6. Document store (for the Save phase)
Location: **`/home/root/.local/share/remarkable/xochitl/`**. Per document:
- `UUID.metadata` — JSON: `visibleName`, `parent` (empty = top level; a folder
is a `CollectionType` document whose UUID is used as children's `parent`),
`type` (`DocumentType`), timestamps, `pinned`, etc.
- `UUID.content` — JSON: page list under `cPages.pages[]` (each page has an `id`,
ordering `idx`, `template`, scroll position, CRDT-style `timestamp` fields).
- `UUID/` — directory of per-page **`<pageUUID>.rm`** files.
- `UUID.thumbnails/` — page thumbnails.
- Store-root extras (Codex additions): `.tree` (binary "rM sync tree" index) and
`rm-search-index.db` (SQLite **search index**, not the canonical store).
**`.rm` page format:** header confirmed as **`reMarkable .lines file,
version=6`** — the v6 binary scene-tree format (documented by the community,
e.g. `rmscene`). Authoring valid v6 is possible but non-trivial.
**Save options (decide in Phase 5):**
1. **PDF/PNG-backed document** — render Chatter's canvas to PDF/PNG and create a
document that references it (xochitl already supports PDF documents). Far
simpler and robust; still a real, renamable/syncable library item.
2. **Native v6 `.rm`** — author the binary page format so the transcript is an
editable notebook. More work; revisit only if option 1 proves insufficient.
Either way, transcripts go under a **`Chatter`** `CollectionType` folder, with
filenames derived from date/time.
---
## 7. Decisions supported by Phase 1
1. **Stack: C++ / Qt 6.8.2**, rendered via the on-device **`epaper` QPA plugin**.
Cross-build with a Yocto **scarthgap** aarch64 SDK matching Qt 6.8.2 / the
device glibc.
2. **Input:** pen from `event2` (including `BTN_TOOL_RUBBER` for flip-to-erase),
finger from `event3` (10-pt). Confirm in Phase 2 whether Qt+epaper already
delivers these or we read evdev directly for custom gestures.
3. **Toggle = display/input hand-off** with xochitl, not an overlay.
4. **Save = PDF-backed document** (leaning), in a `Chatter` folder.
5. **Deploy under `/home`,** never the rootfs.
## 8. Open items for the Phase 2 spike
- Stand up the cross-SDK and build a trivial Qt app; run it with
`-platform epaper` **after stopping xochitl**; confirm it draws and refreshes.
- Verify how input arrives in a Qt app under epaper (Qt event stream vs. raw
evdev) and how reliably finger and pen separate at the Qt layer.
- Validate the toggle hand-off (stop/start vs. stop/cont) and a return path.

79
doc/Chatter_stylus_research.md Normal file
View File

@@ -0,0 +1,79 @@
# Chatter — Stylus & Pen-Rendering Research
A record of what we learned trying to reproduce the reMarkable stock pen
behavior in Chatter, so we can resume this later. (Status 2026-06-26: the solid
ink pipeline is done and shipping; calligraphy is "close enough for now" per the
user and is **not essential** for the conversational-assistance use case.)
## 1. The render pipeline (solved)
The reMarkable epaper Qt backend is a **software scene-graph renderer**:
- Custom `QSGGeometryNode`s are silently dropped — they don't render.
- `QQuickPaintedItem` textures render but always via the e-ink **dashed
two-pass** refresh, with latency. Screen mode (Pen/Mono) and antialiasing do
not change this.
- **Solution:** bypass Qt for ink. Capture the framebuffer `QImage` by
interposing `EPFramebuffer::setBuffers` (cross-DSO, interposable; the
executable is linked `-Wl,--export-dynamic` so its symbol wins). Draw strokes
straight into that buffer (`InkEngine`) and refresh each segment with an
explicit `EPFramebuffer::swapBuffers(rect, …)` — which renders **solid,
single-pass**. Qt Quick is used only for the static UI.
- Buffer A is `960×1696 RGB32` (logical screen 954×1696, padded to 960);
`bytesPerLine 3840`. There is **no PNG image-format plugin** on the device
(only gif/ico/jpeg/svg) — save snapshots as **BMP**.
## 2. The Calligraphy pen model
reMarkable's Calligraphy is **not a fixed geometric/flat nib**. Confirmed by
experiment + reMarkable docs: it's a **dynamic** model combining:
- **Stroke direction** — downstrokes thick, upstrokes thin. (User's stock-pen
circles: thick on the descending side — right for clockwise, left for CCW. A
4-direction asterisk showed *no* variation, because quick uniform straight
strokes don't trigger it.)
- **Pressure** — heavy pressure ≈ doubles the width.
- **Speed** — faster = thinner.
- **Tilt/orientation** — simulates an angled nib.
The exact angle/algorithm is compiled into xochitl's proprietary brush engine
(assets `LS_Calligraphy_*` / `P_Calligraphy_*`, `rm-brushgfx`) — **not** in a
readable config, so exact duplication would need deep reverse-engineering.
## 3. Width calibration
- `LastPenSize` (from `xochitl.conf` `LastWritingTool`) is only a **category**:
1/2/3 = thin/thicker/thickest. Not a pixel width.
- Measured on the physical tablet: **thicker(2) ≈ 1 mm, thickest(3) ≈ 2 mm,
thinnest ≈ 2 px**. Pressure can **almost double** the width.
- Display is **264 PPI → 264/25.4 ≈ 10.4 px/mm.**
- Chatter mapping (`main.cpp`): `maxMm = max(0.2, (size-1)·1.0)` →
size3 = 2 mm, size2 = 1 mm; `setWidthRange(2 px, maxMm·pxPerMm)`.
## 4. Chatter's current approximation (`InkEngine`, pen type 21)
```
f = 0.25·dirF + 0.50·pressure + 0.12·speedF + 0.13·tiltMag // clamp 0..1
width = minWidth + (maxWidth − minWidth)·f
dirF = 0.5 + 0.5·(Δy/len) // downstroke → 1, upstroke → 0
speedF = 1 − clamp(speed/3, 0, 1) // speed = len/dt (px/ms), slow → 1
tiltMag = clamp(hypot(tiltX, tiltY), 0, 1) // from event2 ABS_TILT_X/Y (±9000)
```
Uniform (non-calligraphy) pens: `f = 0.5 + 0.5·pressure`.
`PenDevice` emits `strokeMove(pos, pressure, tiltX, tiltY, eraser)`.
## 5. Tooling (in `tools/`)
- `setbufshim.cpp` — LD_PRELOAD shim; proved `setBuffers` capture (feasibility).
- `fbdump.cpp` — LD_PRELOAD into xochitl; snapshots the framebuffer to
`/home/root/fbdump.{png,bmp}` when `/tmp/fbdump` is touched (use BMP — no PNG
plugin). Note: `setBuffers` interposition into *xochitl* via LD_PRELOAD was
flaky/unconfirmed in one attempt; the in-process `--export-dynamic` capture in
Chatter itself is reliable.
## 6. Open questions / ways to go deeper later
- Recover exact reMarkable widths-per-size and the nib/pressure curves by
snapshotting stock strokes (fix the xochitl fbdump path) and measuring, or by
parsing the v6 `.rm` per-point width/direction data.
- Add a proper speed estimate (smoothed) and tilt-direction (not just magnitude).
- Confirm `LastPenSize` values for the thin/medium categories (only size 3 = 3.0
is confirmed).

334
doc/Chatter_technical_reference.md Normal file
View File

@@ -0,0 +1,334 @@
# Chatter — Technical Reference (master document)
**Audience:** a programmer who needs to understand, build, modify, or maintain
Chatter — including how the reMarkable standard software works and how Chatter
fits alongside it.
This is the **index + complete architecture**. Several topics have their own
deep-dive documents; this file summarizes each and links to it, then fills in
everything not covered elsewhere. Read this first; follow the links for detail.
## Document map
| Document | What it covers |
|---|---|
| [`Chatter_application_proposal.md`](Chatter_application_proposal.md) | The why: the assistive-speech use case, user-experience goals. |
| [`Chatter_implementation.md`](Chatter_implementation.md) | The phased plan and its running changelog; device facts, toolchain, decisions. |
| [`Chatter_phase1_findings.md`](Chatter_phase1_findings.md) | On-device investigation results (input devices, display, doc store). |
| [`Chatter_stylus_research.md`](Chatter_stylus_research.md) | Pen rendering, the Calligraphy model, width calibration, tooling. |
| [`Chatter_user_guide.md`](Chatter_user_guide.md) | The end-user-facing instructions. |
| [`developer_mode_screen.md`](developer_mode_screen.md) | Enabling developer mode on the device. |
| `tablet_access.md` | **SENSITIVE** (root password + IPs) — gitignored, not in the repo. |
---
## 1. The device
**reMarkable Paper Pro Move — model RM03A, codename "Chiappa".**
- **SoC:** NXP i.MX93 (Arm Cortex-A55, aarch64), 2 GB RAM, 64 GB storage.
- **OS:** "Codex" Linux 5.7.121 (Yocto *scarthgap*), image `3.27.1.0`. SSH server
is **dropbear** (not OpenSSH).
- **Display:** E Ink **Gallery 3 — color** (ACeP, "acep2"), 7.3", **954×1696
logical / 960×1696 framebuffer, 264 PPI** (≈10.4 px/mm). Driven by **DRM/KMS
only** (`imx-drm`); there is **no `/dev/fb*`**.
- **Input:** pen = `/dev/input/event2` (Elan marker: `ABS_X` 0–6760, `ABS_Y`
0–11960, `ABS_PRESSURE` 0–4096, `ABS_TILT_X/Y` ±9000, `BTN_TOOL_PEN`/
`BTN_TOOL_RUBBER` for flip-to-erase). Finger = `/dev/input/event3` (10-point
capacitive, `ABS_MT_*`, grid 1248×2208). Power = `event0`, hall/folio = `event1`.
Pen and finger are **separate evdev devices** — Chatter tells them apart natively.
Full investigation: [`Chatter_phase1_findings.md`](Chatter_phase1_findings.md).
### 1.1 Filesystem / persistence model (critical)
- **`/` (rootfs, `/dev/mmcblk0p3`)** — ext4, mounted **read-only**, but
**persistent**. Remount rw (`mount -o remount,rw /`) to write; survives reboots,
**lost on an OS update** (A/B partition swap).
- **`/etc`, `/run`, `/var/volatile`** — **VOLATILE overlays** (upperdir on tmpfs).
Anything written here is **lost on reboot**. (This is why a systemd unit dropped
in `/etc/systemd/system` vanished after a reboot.)
- **`/home`** — encrypted, **fully persistent**, ~45 GB free. Everything Chatter
installs lives here (`/home/root/chatter/`). SSH keys persist because they are
under `/home/root/.ssh`.
**Consequence:** the launcher's systemd unit is installed onto the **rootfs**
(`/usr/lib/systemd/system`) so it survives reboots; the binary and scripts live
under `/home`. **An OS update wipes rootfs changes — disable OS auto-updates on a
delivered device, and keep `scripts/install-launcher.sh` to reinstall.**
---
## 2. The standard reMarkable software stack
Understanding the stock stack is necessary because Chatter reuses its display
plumbing and hands off to it.
- **`xochitl`** — the stock note app. Qt **6.8.2** / Qt Quick, proprietary. Owns
the display while running. Started/stopped as the `xochitl` systemd service.
- **The `epaper` QPA platform plugin** (`libqsgepaper.so`, in
`/usr/lib/plugins/`). A Qt platform + **software** scene-graph renderer for the
e-paper. A Qt app runs on it with `-platform epaper` (+ `QT_QUICK_BACKEND=epaper`).
Key facts learned by reverse engineering:
- It is a **software** renderer: custom `QSGGeometryNode`s are silently dropped;
only textures/rects/glyphs render. (This is why Chatter cannot draw ink via
the scene graph — see §3.)
- **`EPFramebuffer`** is the panel singleton (actually `EPFramebufferAcep2` on
this color device), guarded by an **flock** at `/tmp/epframebuffer.lock` —
only one process may hold the panel. Success marker in the journal:
`SWTCON initialized \o/`. A stranded holder → `Failed to lock epframebuffer` /
`Failed to initialize SWTCON` and a blank panel. **Always stop the previous
holder first.**
- Relevant `EPFramebuffer` methods (mangled symbols bound in `src/epfb.h`):
- `instance()` → the singleton.
- `setBuffers(std::tuple<QImage,QImage>, QImage*)` — sets the front/back
buffers; **cross-DSO and interposable** (see §3).
- `swapBuffers(QRect, EPContentType, EPScreenMode, QFlags<UpdateFlag>)` —
pushes a region to the panel. A single explicit call renders **solid**
(no dashed two-pass). Screen modes (from `EPScreenModeItem::Mode`):
`Pen=0, Mono=1, Animation=2, UI=3, Content=4, Sleep=5`. Chatter uses
`Pen` (0) for fast ink and `Content` (4, "full update, STD") for full
refreshes.
- `ghostControl(GhostControlMode)` — the panel's anti-ghosting API. Modes 0/3
do an immediate full-screen de-ghost via the *region* `swapBuffers`; mode 1
schedules one. **Chatter does NOT call it** — its region-swap path uses
internal `EPContentMap`/`EPScreenModeMap` members the stock app maintains and
we do not, so calling it corrupts state and hangs after a few calls. Chatter
de-ghosts manually instead (§6.3).
- The panel uses **ACeP color waveforms** (`acep2_lut`, `get_waveform_data`,
software TCON "SWTCON"). Color **ghosting** is real and only cleared by a
full-update waveform driven to the **dark** extreme (§6.3).
- **Pen styles** live in `~/.config/remarkable/xochitl.conf`, key
**`LastWritingTool`** (a `@Variant` `QVariantMap`, readable via `QSettings`):
`LastPen` (tool id, e.g. 21 = Calligraphy, 16 = a fineliner), `LastPenSize`
(category 1/2/3 = thin/thicker/thickest), `LastPenColorCode` (`0xAARRGGBB`).
**xochitl writes this to disk only when you leave a document** (return to the
document list) — not on toolbar taps, and not reliably on an abrupt stop.
---
## 3. Chatter architecture (the core idea)
**Problem:** the make-or-break requirement is that ink look like the stock pen —
solid, crisp, low-latency. The `epaper` software scene graph **cannot** do this:
geometry nodes don't render, and `QQuickPaintedItem` textures always come out as
the e-ink **dashed two-pass** refresh, regardless of screen mode or antialiasing.
**Solution — a direct-framebuffer ink pipeline that bypasses the Qt scene:**
1. **`FbCapture`** (`src/FbCapture.{h,cpp}`) interposes
`EPFramebuffer::setBuffers` **in-process**. The executable is linked
`-Wl,--export-dynamic`, so its definition of that symbol wins the cross-DSO
call from the plugin; we capture the real framebuffer's pixel memory and wrap
it as a `QImage` (`FbCapture::framebuffer()`) with **no copy** (`constBits`).
(Intra-DSO calls like `swapBuffers` are *not* LD_PRELOAD-interposable due to
direct binding — but in-process `--export-dynamic` on `setBuffers` works.)
2. **`InkEngine`** (`src/InkEngine.{h,cpp}`) draws strokes with `QPainter`
**straight into that framebuffer image**, then calls
`EPFramebuffer::swapBuffers` (via `src/epfb.h`) on just the dirty rectangle.
One explicit swap → **solid, single-pass** ink.
3. **Qt Quick renders only the static UI** (the two buttons). The scene never
recomposites over the ink because nothing in it animates.
This is what gives stock-quality strokes. The reverse-engineering trail and the
dead-ends (geometry nodes, screen modes, antialiasing) are in
[`Chatter_stylus_research.md`](Chatter_stylus_research.md) and the
`Chatter_implementation.md` changelog (v0.7).
### 3.1 Process / run model
- Chatter is a single Qt 6 executable run as a **transient systemd service**
(`systemd-run --unit=chatter …`), with `QT_QUICK_BACKEND=epaper`,
`LD_LIBRARY_PATH=/usr/lib/plugins/scenegraph`, `-platform epaper`.
- It **requires xochitl to be stopped** (single DRM master + the panel flock).
- Managed via `systemctl {stop,status} chatter` and `journalctl -u chatter`.
---
## 4. Source components
```
src/
main.cpp Entry point: reads pen style, wires PenDevice → InkEngine,
exposes `ink`/`appControl` to QML, loads the QML UI.
FbCapture.{h,cpp} Interposes EPFramebuffer::setBuffers; exposes the live
framebuffer QImage + FbCapture::ready().
InkEngine.{h,cpp} THE core. Virtual canvas, ink drawing, finger-erase, buttons
(paint/hit-test/actions), two-finger scroll, de-ghosting.
PenDevice.{h,cpp} evdev reader for the pen (event2); maps to screen coords;
emits strokeStart / strokeMove(pos,pressure,tiltX,tiltY,eraser)
/ strokeEnd.
AppControl.{h,cpp} returnToStandard(): transient unit stops chatter, starts xochitl.
epfb.h asm-label bindings to the private EPFramebuffer symbols
(instance, swapBuffers, ghostControl).
EPScreenModeItem.{h,cpp} Legacy/unused: binding to the private screen-mode item
from the scene-graph era. Kept for reference.
InkCanvas.*, Experiment.* Legacy from the Qt-Quick-canvas spike; NOT built.
qml/Main.qml White Window; the Back/Clear buttons (TopButton); a
MultiPointTouchArea for finger erase (1) and scroll (2).
tools/
chatter_launcher.c The return-to-Chatter daemon (4-finger watcher on event3).
grabtest.c Probe whether an input device is EVIOCGRAB-exclusive.
fbdump.cpp LD_PRELOAD framebuffer snapshot (BMP; device has no PNG plugin).
swapshim.cpp, setbufshim.cpp Feasibility shims used during RE.
scripts/
build.sh Source the SDK env, cmake build.
deploy-and-run.sh Stop xochitl+chatter, scp, relaunch as a transient unit.
to-chatter.sh Switch standard → Chatter (run by the launcher).
install-launcher.sh Persistently install the launcher unit on the rootfs.
chatter-launcher.service The systemd unit (installed to rootfs).
restore-xochitl.sh Return to the standard GUI.
```
---
## 5. Input & gesture model
Two independent input streams, never confused:
- **Pen (`event2`)** — read by `PenDevice` and delivered to `InkEngine`. The
`epaper` QPA does **not** deliver the pen to Qt, so the pen never triggers QML.
- **Finger (`event3`)** — delivered by the `epaper` QPA to Qt as touch, handled in
QML (`MultiPointTouchArea`, and the buttons' `MouseArea`s).
Gesture map:
| Input | Action | Where handled |
|---|---|---|
| Stylus draw | Ink (flip = erase via `BTN_TOOL_RUBBER`) | PenDevice → InkEngine |
| Stylus tap on a button | Button action | InkEngine hit-tests (`m_buttons`) |
| **1 finger** drag | **Erase wipe** (~12 mm; a tap erases nothing) | QML MultiPointTouchArea → `InkEngine::erase*` |
| **2 fingers** drag | **Vertical scroll** | QML → `InkEngine::panBy/panEnd` |
| Finger tap on a button | Button action (gray feedback) | QML MouseArea → `InkEngine::flashButton/activateButton` |
| **4 fingers** hold (~700 ms) | **Return to Chatter** (only while xochitl is front) | `tools/chatter_launcher.c` |
The MultiPointTouchArea **latches** the gesture type until all fingers lift, so a
two-finger scroll never degrades into an erase when one finger is raised. Erase
requires movement past a ~1.5 mm threshold (tap-safe).
Buttons (`Back`, `Clear`) are a **single source of truth** in `InkEngine`: QML
registers their geometry (`registerButton`) so the engine can exclude ink, redraw
them after a blit, and hit-test stylus taps. Press feedback (`flashButton`) and
actions (`activateButton`) are drawn directly to the framebuffer (instant, no
scene flashing); both finger and stylus route through them.
---
## 6. The virtual canvas, scrolling, and de-ghosting
### 6.1 Growable raster canvas
`InkEngine` holds a `QImage` **canvas** larger than the screen (starts 2× tall,
**grows downward** as you write near the bottom). The screen is a **viewport**
into it at vertical offset `m_panY`. Drawing maps screen→canvas (`+m_panY`); a
dirty canvas rect is blitted back to the framebuffer (`blitRegion`) and the
buttons are repainted on top. (Storage model chosen: raster, to preserve the
exact ink quality; trade-off is no crisp zoom-in. Horizontal/zoom are future work.)
### 6.2 Scrolling
Two-finger drag → `panBy(dy)` (natural: content follows fingers), clamped to the
canvas, fast-blitted per step. `panEnd()` debounces a de-ghost (§6.3).
### 6.3 De-ghosting (color ACeP ghosting)
Fast `Pen`-waveform swaps leave **color residue** ("faint red duplicate") that
accumulates while scrolling and is **not** cleared by a white redraw — it is
panel retention, cleared only by a full-update waveform driven to **black**
(white/gray do not clear it; this was tested). Chatter's `fullRefresh()`:
fills the screen **black** + full-update swap (`screenMode=Content`), waits
~220 ms, then full-updates the real content. `ghostControl()` would be the
"proper" API but corrupts state (§2), so this manual flash is used.
To keep the flash from being intrusive:
- It runs **only after scrolling**, **debounced** ~1 s after the last scroll
(not on every finger-lift).
- **Clear has two methods:** if there was **no scrolling** since the last clear
(the common fill-one-screen case) it uses the **gentle fast clear**; if there
**was** scrolling, it does the **black de-ghost** clear. (`m_scrolled` flag.)
- Tunables via env: `CHATTER_FULL_SM` (full-update screen mode, default 4),
`CHATTER_FLASH_GRAY` (flash level 0=black..255; black is what actually clears).
A black flash is intrinsic to clearing color ghosting (the stock UI flashes on
its full refreshes too); we minimized *when* it happens rather than eliminating it.
---
## 7. Pen-style matching, width calibration, calligraphy
- Chatter reads `xochitl.conf` `LastWritingTool` at startup (`readPenStyle` in
`main.cpp`) and applies tool type, size, and color — **no style menu in
Chatter**. Because each switch-to-Chatter restarts the process, it re-reads the
current pen. **The flush sequence matters** (§2): set the pen, *use it*, **leave
the document**, then switch to Chatter.
- **Width** is calibrated to measured widths on the 264-PPI panel:
size 3 ≈ 2 mm, size 2 ≈ 1 mm, thinnest ≈ 2 px; pressure ≈ doubles width.
- **Calligraphy** (tool 21) is approximated with a dynamic
direction/pressure/speed/tilt width model. Full detail, measurements, and the
formula: [`Chatter_stylus_research.md`](Chatter_stylus_research.md).
---
## 8. Toggle & the return launcher
- **Chatter → standard:** `Back` → `InkEngine::backRequested` →
`AppControl::returnToStandard()` → a transient unit stops chatter and starts
xochitl (sequenced so the panel lock is released first).
- **standard → Chatter:** `tools/chatter_launcher.c` runs always as a rootfs
systemd service. It reads `event3` **without grabbing it** (verified possible
via `tools/grabtest.c` — xochitl does not hold an exclusive grab), counts
multitouch slots, and on a **4-finger hold ≥700 ms** while xochitl is the front
app runs `to-chatter.sh`. It reopens the device on any read interruption (sleep/
wake) so it never dies.
- **Install persistently:** `scripts/install-launcher.sh` (remounts the rootfs rw,
places the unit + its `multi-user.target.wants` symlink under
`/usr/lib/systemd/system`). See the persistence model in §1.1.
---
## 9. Build & deploy
**Toolchain:** official reMarkable Chiappa SDK `3.27.0.97` (Qt 6.8.2 sysroot,
x86_64 host). Install, then `source environment-setup-cortexa55-remarkable-linux`.
```
scripts/build.sh # source SDK env + cmake build -> build/chatter
scripts/deploy-and-run.sh # stop xochitl+chatter, scp, relaunch as transient unit
scripts/restore-xochitl.sh # back to the standard GUI
scripts/install-launcher.sh # one-time (and after each OS update): persist the launcher
```
`CMakeLists.txt`: Qt6 Quick app (`qt_add_executable` + `qt_add_qml_module`), links
`libqsgepaper.so`, and crucially `target_link_options(... -Wl,--export-dynamic)`
so the `setBuffers` interposition wins.
**Device housekeeping:** only one process may hold the panel — always stop the
previous holder. Deploy under `/home/root/chatter`, never the rootfs (except the
launcher unit). The device sleeps and DHCP may reassign its IP on wake; if you
script a reconnect, rescan for the tablet's MAC address (shown on the device's
GPLv3-compliance screen, alongside its IPs).
---
## 10. Status & open items
**Working on hardware:** solid stock-quality ink matching the selected pen;
finger-wipe erase; whole-page Clear (two methods); stylus eraser; bidirectional
toggle (Back + 4-finger launcher, persistent); vertical scroll on a growable
canvas with debounced de-ghosting; instant button feedback (finger + stylus).
**Open / future:**
- **Save** (Phase 5) — write a timestamped transcript into a "Chatter" folder in
xochitl's on-disk document format; importance to the user still unconfirmed.
See `Chatter_implementation.md` §6 / §6a.
- **Horizontal scroll & zoom-out overview** — deferred extensions of the canvas.
- **Calligraphy fidelity** — "close enough" approximation; exact nib unknown.
- **Erase / gesture thresholds** — to be tuned with the end user (Matt's field test).
- **OS auto-update** would wipe the rootfs launcher unit (and could break paths) —
disable it on a delivered device.
---
*This document is the technical entry point. When the design changes, update this
file and the relevant component doc; keep `Chatter_implementation.md`'s changelog
as the chronological record.*

58
doc/Chatter_user_guide.md Normal file
View File

@@ -0,0 +1,58 @@
# Chatter — How to Use It
Chatter is a simple writing screen. You write with the pen; the person you are
talking with reads what you wrote. That's it.
## Writing
- **Write with the pen**, just like on paper.
- The pen's color and thickness are whatever is set in the normal reMarkable
screen. (To change them, see "Changing the pen" below.)
## The two buttons (top of the screen)
```
┌──────┐ ┌───────┐
│ Back │ │ Clear │
└──────┘ └───────┘
```
- **Clear** (top right) — erases everything and gives you a fresh page.
- **Back** (top left) — leaves Chatter and goes to the normal reMarkable screen.
A button turns **gray** the moment you touch it, so you know it heard you. You
can press a button with **either the pen or your finger**.
## Erasing
- **Erase a little:** rub a **finger** back and forth over what you want to
remove, like erasing pencil with your fingertip.
- **Erase everything:** tap **Clear**.
- You can also flip the pen over and use the **eraser end**, like a pencil.
## More room to write
- **Two fingers, drag up or down** to scroll, when you want more space than one
screen. The page follows your fingers.
- (After scrolling, the screen may blink once to clean itself up. That's normal.)
## Going back and forth
- **Leave Chatter:** tap **Back**.
- **Return to Chatter** from the normal reMarkable screen: place **four fingers
on the screen and hold** for about a second.
## Changing the pen (color / thickness)
Chatter copies the pen you last used in the normal reMarkable screen. To change
it:
1. Tap **Back** to go to the normal screen.
2. Open a page, pick the pen, color, and thickness you want, and **write a little
with it**.
3. **Go back to the document list** (this is what saves your choice).
4. Return to Chatter (four-finger hold). Your new pen is now used.
---
*Questions or problems during setup go to Matt.*

17
doc/developer_mode_screen.md Normal file
View File

@@ -0,0 +1,17 @@
Developer mode
This mode is designed for experienced developers to make local software changes via SH access to the system.
Please note that any changes to software puts the device security at increased risk.
- Access to root shell is offered in compliance with GPL V3 license.
- Running custom or modified software may cause the device to stop working or adversely affect performance.
- A factory reset is required to enter.
- This will delete all files stored on your paper tablet.
- User data will be cleared once the device is rebooted.
- The device will no longer verify the authenticity of the software, putting your system stability and data security at risk.
- Access any content stored in the cloud by re-pairing with your reMarkable account.
For instructions on how to exit developer mode, visit support.remarkable.com
(Accept)

BIN
doc/initial_Claude_discussion.pdf Normal file
View File

Binary file not shown.