Project · Wed, 24 Jun, 2026 · 12:13 pm UTC

AURA (Adaptive Unified Response Assistant)

Shubham Dubey

Knowledge workers lose hours daily context-switching between tools and manually re-explaining background context to traditional AI chatbots.

Source

Idea

AURA is a local-first, system-wide AI productivity layer for Windows that sits beside your existing applications. Activated instantly via a global hotkey or push-to-talk voice trigger, AURA automatically captures real-time screen and browser context (focused windows, URLs, page text, selections) to execute tasks like drafting contextual replies, summarizing documents, and extracting task lists immediately—without forcing the user to switch apps or copy-paste prompts.

How to use

AURA is organized as a monorepo containing the React desktop shell, Python AI sidecar, and Chrome/Edge extension. Prerequisites: Python 3.12, Node.js, and an API key (Gemini, OpenAI, or Anthropic). Clone the Repo & Install Dependencies: Navigate to apps/desktop and run npm install. Navigate to services/sidecar and install Python requirements (pip install -r requirements.txt). Launch the Sidecar Service: Run the FastAPI sidecar server: python main.py (defaults to localhost port 29301). Launch the Desktop UI: Start the frontend application using npm run dev (or launch via Tauri using npm run tauri dev). Load the Browser Extension: Open Chrome/Edge -> Extensions -> Enable "Developer Mode". Click "Load unpacked" and select the apps/browser-extension folder. How to Test: Open your browser or a document application, use the hotkey (Win + Space), and watch AURA instantly ingest your active page context to answer prompts or draft replies.

Notes & next steps

Current Status: Monorepo architecture configured; desktop prototype UI shell, browser context extension capture, and multi-provider LLM routing engine successfully scaffolded. Future Plans: Integration of full local STT using faster-whisper, setting up automated OAuth loops for Google Workspace (Gmail, Calendar, Docs), implementing local AES-256 database encryption (SQLCipher), and compiling the self-contained Windows application installer. Blockers: Managing dynamic cross-app Windows UI Automation (UIA) consistency across varying third-party legacy desktop applications—currently mitigated via our robust Chromium browser extension and vision fallback layers.

README

View on GitHub

AURA

AURA is an AI desktop companion prototype focused on Google-first workflows, rich browser context, intelligent memory, multi-provider LLM routing, and local-first privacy.

What Exists Now

React desktop prototype shell in apps/desktop.
FastAPI sidecar in services/sidecar.
Chrome/Edge MV3 browser extension in apps/browser-extension.
Product implementation plan in docs/IMPLEMENTATION_PLAN.md.

Requirements

Node.js 22+
npm 10+
Python 3.12+
Rust 1.96+
Microsoft C++ Build Tools with the Windows SDK
WebView2 Runtime

Setup

npm install
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r services/sidecar/requirements.txt
Copy-Item .env.example .env

Add provider keys to .env as needed:

GEMINI_API_KEY=your_key_here
OPENAI_API_KEY=
ANTHROPIC_API_KEY=
DEEPGRAM_API_KEY=

For local LLM mode, install Ollama and pull a model:

ollama pull llama3.1:8b

Run

Start the sidecar:

npm run sidecar

Start the desktop prototype in another terminal:

npm run dev

Start the native Tauri overlay:

npm run tauri:dev

If you run an installed build before the Python sidecar is bundled, point AURA at the sidecar source:

$env:AURA_SIDECAR_APP_DIR = "D:\Projects\AURA\services\sidecar\src"

Build the Python sidecar executable:

npm run sidecar:build

Then point an installed build at it if needed:

$env:AURA_SIDECAR_EXE = "D:\Projects\AURA\services\sidecar\dist\aura-sidecar.exe"

Build the browser extension:

npm run extension:build

Then open Chrome or Edge extension management, enable developer mode, and load apps/browser-extension/dist as an unpacked extension.

Current Prototype Behavior

The UI defaults to a realistic local demo context if the sidecar is not running.
The Tauri shell provides a transparent always-on-top Windows panel, tray menu, global shortcut, native foreground-window title capture, privacy-mode capture exclusion, and sidecar start/stop commands.
The native launcher prefers a bundled aura-sidecar.exe, then falls back to the Python source sidecar for local development.
When the sidecar runs, /api/v1/context/ingest accepts browser snapshots from the extension.
The sidecar keeps a short context cache and provides suggestions based on Gmail, Calendar, Docs, WhatsApp Web, and general page text.
The LLM router supports live Gemini, Ollama, OpenAI, and Anthropic calls when the selected provider is configured.
Memory and settings persist locally in SQLite under %APPDATA%\AURA\aura.db.
If no provider key or local model is ready, the assistant returns a clear fallback response while preserving the full context/memory flow.

Recommended Commit Slices

Scaffold monorepo and architecture docs.
Add desktop prototype shell.
Add FastAPI sidecar API.
Add browser extension context bridge.
Add setup docs and verification.

Next Engineering Slice

Add Google OAuth skeleton for Gmail, Calendar, and Docs.
Replace foreground-window title capture with full Windows UI Automation text extraction and screenshot fallback.
Add streaming responses from sidecar to UI.
Package Python sidecar as a bundled executable instead of relying on system Python.

README

Loading README…

Hackers

Builders behind this 503 Day project.

Shubham Dubey

Solution Engineer

Passionate about building and shipping meaningful products. Enjoys travelling.

Demo Hour · 18:00 IST

Wed, Nov 18, 2026

8th503 Day

Join with Google Meetmeet.google.com/duj-wvmx-veq

Careers

Join Team
COPIOUS