# contexts.live — Capabilities

contexts.live is a live performance accessibility and enhancement platform.
It delivers real-time and recorded content to audience devices and production
screens, with a focus on the performing arts.

---

## 1. Live Subtitles & Captions

Timed text delivered to audience devices during live performance or playback.

- **Multi-language** — simultaneous display in any number of languages;
  audience selects their preferred language
- **Multiple display styles** — full-screen, positioned, scrolling, braille,
  low vision
- **Offset lines** — show previous and next lines alongside the current cue
- **Scrolling word list** — all lines visible, current line highlighted
- **Language links** — foreign words highlighted and linked to definitions
  (currently implemented for te reo Māori via maoriDictionary.js)
- **Speaker / character** — character name displayed alongside lines
- **Stage descriptions** — separate stream for scene and stage information

---

## 2. Audio Description (AD)

Timed audio files played to audience devices during performance.

- AD audio plays automatically on cue, overlapping or separate from subtitle display
- AD text displayed alongside or instead of audio
- Countdown timer showing remaining AD audio duration
- Multiple simultaneous audio streams (one per source)
- Audio cleanup on page navigation — no runaway audio

---

## 3. SmaTTer — Speech to Text

Live transcription using AI speech recognition (whisper.cpp / OpenAI Whisper).

**Two modes:**

**Live streaming** (`listen.html`) — microphone input transcribed in real time:
- Configurable chunk duration (2–15 seconds)
- Output formats: JSON, VTT, SRT, plain text
- Audio level monitoring and statistics

**File upload** — transcribe pre-recorded audio/video files:
- Supports MP3, WAV, M4A, OGG, FLAC, MP4, MOV, AVI, up to ~1GB
- Progress bar and download transcript

**SmaTTer mobile display** (`smatter-display.html`) — audience-facing:
- Animated splash screen
- Transcription displayed as scrolling text on black background
- Māori word detection — recognised words highlighted, definition shown below
- Auto-switches from splash to transcription when text arrives

**Languages supported:** auto-detect, Arabic, Czech, German, English, Spanish,
French, Hebrew, Hindi, Italian, Japanese, Korean, Māori, Dutch, Portuguese,
Russian, Chinese

---

## 4. Sign Language Video

Timed video played in a page element, synced to performance cues.

- Video fades in on `canplaythrough`, fades out on `ended`
- `playsinline` for iOS compatibility
- Source addressable as `sl.video.en` or any language variant

---

## 5. Music Score Display

Timed score page images synced to cues.

- Individual score pages displayed on cue
- Scrolling score list with click-to-advance
- Score offset — show adjacent pages (±N)
- Fade in/out option

---

## 6. Programme Navigation

A list of items (works, movements, songs) that operators can navigate between.

- Programme panel showing all items
- Click to load a different pack
- Used in multi-work concerts and programmes

---

## 7. Controller / Cueing

Production tools for operators driving the system.

**Auto controller (`cx-control-auto.html`):**
- Scrolling line list with click-to-cue
- **Client-side autoplay** — clicking Play schedules all cues automatically
  from `cueIn` timecodes; Pause halts without sending a blank
- **Nudge** — visual track + buttons to shift timing ±2 seconds in real time;
  keyboard shortcuts ←/→ (±0.1s), ⇧←/⇧→ (±1s), 0 to zero
- Play/Pause button reflects actual auto state in real time

**Manual controller (`cx-control-list.html`):**
- Click-to-cue, keyboard shortcuts

**Full controller (`cx-control-full.html`):**
- Toolbar buttons, keyboard shortcuts

---

## 8. Autoplay / Timecode-driven Playback

The client schedules cues automatically from `cueIn` values in the pack JSON,
without an operator advancing each cue manually.

- Runs entirely in the browser — no server-side timers
- `startAuto(n)` schedules all subsequent lines relative to line n's `cueIn`
- `nudge` offset applied at schedule time
- Any manual line click cancels pending timers
- Page unload (`beforeunload`) cancels all pending timers — no runaway cues

---

## 9. Braille / Low Vision Display

Optimised display for braille readers and low vision users.

- Lexend font, large text, full black background
- `aria-live="polite"` — VoiceOver sends text to braille device
- Focus management — after each update, focus moves to the text element
- English Précis (`ens`) language support — simplified subtitle text

---

## 10. Monitor & Statistics

Tools for monitoring live sessions and reviewing logs.

- **Statistics viewer** — engagement chart, IP/language/unit/page data
- **Log viewers** — harness log, cueing log
- **Client monitor events** — display clients send telemetry back to server

---

## 11. Media Management

Infrastructure for managing content files.

- **Subtitle Editor** — browser-based editor for creating and editing pack JSON; supports TSV, CSV, VTT, SRT, PPTX import; save directly to server with version history; at `https://contexts.live/org/{slug}/tools/editor/`
- **Pack Migrator** — convert v1 unit JSON to v2 pack format
- **Pack Merge** — merge two or more v2 packs interleaved by cueIn timecode
- **Media folder** — audio, video, score images organised by pack and triple (`org/{slug}/media/{packId}/{triple}/`)

---

## 12. Multi-server / Multi-org

- Multiple organisations hosted on the same server, each with their own slug
- ANZ (New Zealand) and UK servers, each independently configured
- Each org has its own packs, media, config, and page styles
- Socket namespaces per org — no cross-org bleed
- Org index page — per-org launcher for operators

---

## 13. Per-org CSS

Each org can have its own CSS overrides in `/org/{slug}/css/`. These are
injected automatically by `cx-connect.js` without changing any HTML pages.

---

## 14. QR Code Generation

QR codes generated for any page/pack combination. Available from the org dashboard (`/org/{slug}/`) for distributing display page URLs to audience members, and as a printable QR sheet (`cx-qr-sheet.html`) for front-of-house use.

---

## 15. Live Meta Display

Any page element can display live runtime state using `data-cx="meta {field}"`:

- `meta status` — socket connection state
- `meta pack` — currently loaded pack name
- `meta line` — current line number
- `meta org` — org slug
- `meta server` — server URL

These update in place without a cue, driven by `state.liveMeta` in `cx-render.js`.

---

## Technology Stack

| Layer | Technology |
|---|---|
| Server | Node.js, HTTPS, socket.io |
| Transport | WebSocket (socket.io namespaces per org) |
| Client | Vanilla JS ES modules (cx-connect.js, cx-render.js, cx-keys.js) |
| AI transcription | whisper.cpp / OpenAI Whisper (SmaTTer) |
| Media | nginx serving static files |
| Editing | Browser-based subtitle editor (editor.js) |
| CSS | cx-base.css + cx-theme-dark/light.css |
| Fonts | Google Fonts (EB Garamond, IM Fell Great Primer, Lexend, Roboto Mono, Noto Serif SC, Voces, Bubblegum Sans) |