Monitors the Situation

On-device, configurable visual monitoring with Gemma 3n

1) TL;DR

Gemma Monitors the Situation is a privacy-first mobile app that turns any spare phone into a flexible, on-device watcher: point the camera, describe what to look for, and get audible/visual/webhook alerts when conditions match. It runs Gemma 3n locally via Google AI Edge (MediaPipe/LiteRT-LM), so it works offline and keeps data on the device. The UX focuses on quick setup, transparent logs, and “trigger → action” automations.

---

2) Why this matters

When you’re busy (or offline), small but important checks get missed: is the toddler still in the yard? is the pizza boiling over? did the pet’s water bowl run empty? do I keep slouching? A general, user-programmable, private monitor lowers the cost of vigilance for home, health, accessibility, and lightweight safety scenarios—without sending your video to the cloud.

---

3) What the app does

Core loop

1. Capture frame(s) from the device camera.
2. Run a short, AX-inspired prompt against Gemma 3n (image → structured fields).
3. Evaluate Triggers against the output.
4. Execute Actions (beep, flash, cURL webhook, etc.).
5. Log everything (inputs, outputs, trigger decisions), and let users export.

Current features

• On-device inference with MediaPipe GenAI Tasks / LiteRT-LM.

• Custom prompt schema & output fields (AX-inspired formatting).

• Trigger system

  • Numeric / scale thresholds, always-on, and conditional triggers.
  • Actions: cURL (webhook/IFTTT/Home Assistant/etc.), Flash, Beep.
  • Variable templating in cURL: $output_<fieldName>, $image_base64, $image_url (local path).

• Robust logging

  • Cycle log with structured entries; limit long-run logs for performance.
  • Export to clipboard or file; image thumbnails clickable.

• Tools drawer (opt-in on the start page):

  • Chat with image, STT, image describe/analysis, “debug” tools.

• Performance controls

  • Adjustable camera resolution with true pixelated preview.
  • Wake-lock while running; sensible defaults for CPU/GPU.

• Privacy by default

  • Everything on device; networking only if you choose (e.g., cURL).

---

4) Notable use cases

• Peace of mind: “Are the kids not climbing on the tree?”
• Health & habits: posture checks; enforce micro-breaks; detect “phone in hand while working”.
• Accessibility: for hearing-impaired users, beep/flash/tts only when something relevant is seen.
• Pets & small farms: “Is the dog inside?” “Is the hen in the nesting box?” “Food/water present?”
• Kitchen safety: “Is the pot boiling over?” “Is the pizza browning too much?”
• Crisis/low-connectivity: offline watch for people/signals/water level in the basement.

---

5) Implementation overview

Tech stack

• App: React Native (Expo) + Kotlin bridge.
• Camera: react-native-vision-camera.
• AI runtime: Google AI Edge stack (MediaPipe GenAI Tasks / LiteRT-LM).
• State & UI: Zustand, structured cycle logs, export utilities.

Design choices

• MVP first, UX second, polish third. We shipped a working POC early, then layered in logging, triggers, and export.
• AX-inspired prompt format to keep outputs short, parsable, and robust.
• Templated cURL so any output field can parametrize notifications and downstream tools.

Performance work

• Adjustable, very low-res capture to enable ~1 fps monitoring on modest devices.
• Planned benchmark step to predict per-pass latency (“don’t use the phone or battery saver during this step”).
• Question under study: smallest image size vs. inference throughput on device.

---

6) UX highlights & upcoming

• Clear estimated time per pass and (optional) estimated “cost” for settings.

• Sensible defaults based on hardware & inference type; opt-in “show tools on start page”.

• Reference-compare mode (planned): render a 1024×512 panel per iteration

  • Top: reference image (user provided); Bottom: current frame.
  • Prompt: “What’s missing/different vs. reference (lower half)?” Useful for “is anything out of place?” checks.

---

7) Future work

• AX/DSPy auto-tuning: feed labeled reference images and let the system search for the shortest reliable prompt.
• Audio input for richer triggers (“uhm” counter, glass break, etc.).
• Long-horizon conditions (e.g., “if posture < 2/5 for >2 minutes within any 60-minute window, alert”).
• iOS port: evaluate LiteRT-LM path for a single cross-platform API.
• More actions: system notifications, SMS relays, Home Assistant helpers.

Source: https://github.com/choltha/gemma-monitors-the-situation