How It Works

How It Works

Escribano is a local evidence layer for your work. It captures your screen, understands what's happening, and makes it queryable — all on your Mac, nothing in the cloud.


Overview

flowchart LR
    A[Screen capture] --> B[Vision model<br/>on-device]
    B --> C[Description<br/>+ entities]
    C --> D[(SQLite<br/>+ FTS5)]
    D --> E[escribano-query]
    E --> F[Human]
    E --> G[Agent]

Every step runs on-device. No cloud upload, no telemetry on your activity.

The Pipeline

1. Capture

A small menu-bar app watches your screen in the background, sampling a frame every few seconds and skipping visual duplicates.

  • Runs entirely on your Mac

  • Energy-efficient — adaptive sampling, dedupe before any heavy processing

  • Pause and resume from the menu bar at any time


2. Understand

Each frame is processed by a vision-language model running entirely on your Apple Silicon GPU:

Screenshot → [VLM, on-device] → "User is debugging JWT flow in VS Code…"

The model extracts:

  • A plain-language description of what's happening

  • The applications visible on screen

  • Entities — code files, URLs, error messages, git branches, tools, languages

A second NLP pass normalizes entities and applies PII filtering before anything is written to disk.

3. Store

Descriptions and entities are written to a local SQLite database with FTS5 full-text indexing:

Table

Contents

Observations

Timestamped descriptions plus metadata (apps, file paths, URLs)

Entities

Tools, languages, frameworks, errors, etc., linked to their observations

Frames

Optional — retained only if image access is enabled

Everything lives on your machine. There is no sync, no cloud backup, no remote index.

4. Query

You query history through escribano-query, the CLI built for both humans and agents:

escribano-query today                                  # timeline of today's activity
escribano-query search "bug"                           # find mentions of "bug"
escribano-query entities --kind programming_language   # which languages you used

The CLI is JSON-first, versioned (api_version: 2), and self-describing via --help --json. See the CLI Reference for the full command surface.

Privacy by Design

Layer

What happens

Capture

Screenshots never leave your machine

Understanding

VLM runs on the local Apple Silicon GPU

Storage

SQLite database on local disk

Query

Text output is PII-filtered by default

Images

--images flag required to surface screenshots

For Agents

The CLI is designed for programmatic access from coding assistants and other agents:

  • JSON output — structured, versioned (api_version: 2)

  • Self-discovery — escribano-query --help --json returns the command surface

  • Exit codes — 0 on success (even empty data), non-zero on failure

  • Progressive disclosure — text by default, --images for visual verification

Quick install for popular agents: