Feat notification system (#2472)

* feat: SSE notification system

Adds a per-user SSE pipe (GET /api/events) plus a per-message
chat-stream reconnect endpoint (GET /api/messages/<id>/events).

Backend substrate:
- application/events/ — durable journal (Redis Streams) + live
  pub/sub for user-scoped events, with publish_user_event() as
  the worker-side entrypoint.
- application/streaming/ — broadcast_channel for pub/sub fanout
  and event_replay for the per-message snapshot+tail path.
- application/storage/db/repositories/message_events.py +
  alembic 0007 — Postgres journal for chat-stream events.
- application/worker.py — ingest/reingest/remote/connector/
  attachment/mcp_oauth tasks publish queued/progress/completed/
  failed envelopes alongside their existing status updates.

Frontend client:
- frontend/src/events/ — connect/reconnect, Last-Event-ID cursor,
  backoff with jitter. Each tab runs its own connection; no
  cross-tab dedup (future work).
- frontend/src/notifications/ — recentEvents ring, cursor
  tracking, tool-approval toast.
- frontend/src/upload/uploadSlice.ts — extraReducers for
  source.ingest.* and attachment.* events.

Coverage: 132 SSE tests across events substrate, replay, journal,
routes, and worker publishes.

* refactor(attachments): remove polling, SSE-only

frontend/src/components/MessageInput.tsx no longer runs a 2s
setInterval against getTaskStatus for every processing
attachment. The attachment.* SSE reducers in uploadSlice.ts are
now the sole driver of attachment state transitions.

* feat(connector): consume source.ingest.* SSE, remove polling

frontend/src/components/ConnectorTree.tsx now mirrors FileTree's
slice-walking pattern: it watches notifications.recentEvents
for source.ingest.{completed,failed} envelopes matching the
sync's source id, and no longer polls /task_status every 2s.

* refactor(source-ingest): remove polling, SSE-only

frontend/src/upload/Upload.tsx and
frontend/src/components/FileTree.tsx no longer run getTaskStatus
polling fallbacks. The source.ingest.* SSE reducers in
uploadSlice.ts and FileTree's slice walk are now the sole
drivers of upload/reingest state transitions.

* refactor(mcp-oauth): carry authorization_url in SSE, remove polling

application/worker.py::mcp_oauth now publishes
authorization_url on the mcp.oauth.awaiting_redirect envelope.
frontend/src/modals/MCPServerModal.tsx consumes it from SSE
instead of polling /oauth_status/<task_id> every 1s.

The URL is generated inside DocsGPTOAuth.redirect_handler when
the FastMCP client triggers OAuth. The worker now plumbs a
publish callback through tool_config -> MCPTool -> DocsGPTOAuth
so the awaiting_redirect publish fires from inside the handler
at the exact point the URL becomes known. The legacy Redis
mcp_oauth_status setex writes and the GET
/api/mcp_server/oauth_status/<task_id> endpoint are kept as
belt-and-suspenders; nothing in the frontend reads them now.

* feat(source-ingest): plumb limited flag through SSE for token-cap UX

application/worker.py::ingest_worker and remote_worker now publish
``limited: bool`` on the source.ingest.completed envelope.
uploadSlice routes ``payload.limited === true`` to a failed status
with a ``tokenLimitReached`` flag, and UploadToast surfaces the
translated tokenLimit i18n string. No worker code path sets
limited=true today; this is a forward-looking contract so when
token-cap detection lands, the UX is already wired.

* refactor(mcp-oauth): read status from SSE journal, drop polling endpoint

MCPOAuthManager.get_oauth_status now walks the per-user SSE Streams
journal (user:{user_id}:stream) for the latest mcp.oauth.* envelope
matching the task id, returning the status string derived from the
event type suffix and the payload fields. The worker is the single
source of truth — its publish_user_event calls write the same
record the SSE client receives live.

Removed:
- /api/mcp_server/oauth_status/<task_id> route in
  application/api/user/tools/mcp.py
- mcp_oauth_status worker function and mcp_oauth_status_task Celery
  wrapper
- All mcp_oauth_status:{task_id} Redis setex writes (4 in mcp_oauth,
  2 in DocsGPTOAuth.redirect_handler / callback_handler)
- The update_status closure in mcp_oauth that wrote the polling
  payload

Tests updated:
- get_oauth_status now takes (task_id, user_id); new coverage walks
  a fake xrevrange response for the completed envelope, the no-match
  case, and a Redis-down case
- Removed TestMCPOAuthStatus route tests and TestMcpOauthStatusTask
  celery-wrapper test
- Removed the two oauth_status methods from the integration runner

mcp_oauth:auth_url/state/code/error Redis keys remain — they are
the OAuth flow's own state (not the dropped polling payload).

* chore(mcp-oauth): delete orphaned getMCPOAuthStatus client

The /api/mcp_server/oauth_status/<task_id> endpoint was removed in
the prior commit; the corresponding userService method and the
MCP_OAUTH_STATUS endpoint constant had no remaining callers in the
frontend, so they're deleted along with it.

* fix(events): drop live publish when journal write fails

application/events/publisher.py returned an envelope to live
pubsub subscribers even when the XADD to the durable journal
failed. The envelope had no ``id`` field, which bypassed the SSE
route's dedup floor and broke ``Last-Event-ID`` semantics for any
reconnecting client.

Best-effort delivery means dropping consistently, not delivering
inconsistent state. Now: if the journal write fails the publisher
returns None and skips the live publish entirely.

* fix(notifications): dedupe sseEventReceived against immediate dupes

Snapshot replay + live tail can both deliver the same id when the
live pubsub frame and the replay XRANGE overlap. The route's own
dedup floor catches the common case, but consumers walking
``recentEvents`` (FileTree, ConnectorTree, MCPServerModal,
ToolApprovalToast) would otherwise act on the same envelope
twice when a duplicate slipped through.

Belt-and-suspenders: short-circuit when the most recent id in
the ring matches the incoming one.

* fix(events): skip replay budget INCR when no snapshot work possible

_allow_replay incremented the per-user counter on every
/api/events GET, including no-op connects from a fresh client
with no cursor against an empty backlog. React StrictMode dev
double-mounts plus a few tabs trivially tripped the default
30-per-60s budget on idle reconnects.

XLEN pre-check: when last_event_id is None and the user stream
is empty, the connect can't do snapshot work — return True
without INCR. Cursor-bearing connects still INCR unconditionally
(probing the cursor's relationship to stream contents would
require a redundant XRANGE).

* fix(streaming): tighten journal contract + recover from seq collisions

Two related fixes to application/streaming/message_journal.py.

1. record_event now rejects non-dict payloads at the gate. The
   live path (base.py::_emit) wrapped non-dicts as
   {"value": payload}; the replay path in event_replay synthesized
   {"type": event_type}. A reconnecting client would receive a
   different envelope than the one originally streamed. Now both
   paths see byte-identical envelopes because non-dicts can't be
   journaled at all. The corresponding event_replay fallback is
   replaced with a warn-and-skip for any legacy rows.

2. record_event handles IntegrityError on (message_id, sequence_no)
   collisions by reading latest_sequence_no and retrying once with
   latest+1. The most likely cause is a stale seq seed on a
   continuation retry where the route read MAX(seq) from a
   separate connection before another writer committed past it.
   Previously the error was swallowed and the event silently
   dropped from the journal; now it lands at the next available
   seq. The live pubsub publish uses the materialised seq so the
   journal row and the live frame agree.

* perf(streaming): batch message_events INSERTs per stream

complete_stream previously opened a fresh db_session() per yielded
event, doing one Postgres INSERT + commit per chunk on the WSGI
thread. Streaming answers emit ~100s of answer chunks per response,
so the route was paying ~100 PG roundtrips per stream serialized on
commit latency.

New BatchedJournalWriter in application/streaming/message_journal.py
accumulates rows per stream and flushes on three triggers:
- size: buffer reaches 16 entries
- time: 100ms elapsed since the last flush
- lifecycle: close() at end-of-stream

Live pubsub publishes still fire synchronously per record(), so
subscribers see events in real time — only the durable journal write
is amortized. On bulk INSERT IntegrityError the writer falls back to
per-row record() with the existing seq+1 retry so a single colliding
seq doesn't drop the rest of the batch.

complete_stream wires journal_writer.close() into every exit path
(happy end, tool-approval-paused end, GeneratorExit, error handler)
so the terminal event is committed before the generator returns —
otherwise a reconnecting client could snapshot up to the last flush
boundary and live-tail waiting for an end that's still in memory.

Repository gets bulk_record() — one SQLAlchemy executemany INSERT
for the bulk path. All-or-nothing on collision (Postgres aborts the
whole batch); the writer's per-row fallback handles recovery.

* chore(upload): drop dead UploadTask.lastEventAt field

The lastEventAt field on UploadTask had no remaining consumers — the
matching Attachment.lastEventAt was cleaned up earlier. Remove the
field declaration and the slice write site.

* chore(frontend): drop orphaned getTaskStatus client

After the polling-removal sweep no caller in frontend/src/ references
userService.getTaskStatus or endpoints.USER.TASK_STATUS. The backend
route /api/task_status itself stays — agents, webhooks, e2e specs,
and the public docs still depend on it.

* docs(repo): remove stale planning docs from repo root

notification-channel-design.md, plan.md, and reminder-tool-design.md
were leftover Claude planning artifacts from the SSE substrate work
that landed accidentally. CLAUDE.md prohibits creating planning docs
unless asked — delete them.

* docs(message-events): clarify repo vs wrapper payload contract

MessageEventsRepository.record accepts any JSONB-compatible value; the
streaming wrapper record_event tightens this to dicts only because the
live and replay paths reconstruct non-dict payloads differently. Spell
the split out so the next reader of the repo method doesn't assume the
wrapper's contract applies here.

* refactor(events): raise on malformed stream id instead of lex fallback

stream_id_compare's lex-fallback branch was a footgun: a malformed id
that sorts lex-greater than a real one would pin live-tail dedup
forever, dropping every subsequent legitimate event silently. Both
current callers in application/api/events/routes.py pre-validate
inputs against _STREAM_ID_RE before calling, so changing the function
to raise ValueError is a no-op on the happy path and turns the future-
caller footgun into a loud failure.

* test(tasks): cover cleanup_message_events task body

Adds skipped-when-no-POSTGRES_URI and happy-path coverage for the
Celery janitor. The skipped path returns the documented short-circuit
shape without touching the repo. The happy path seeds a backdated
row, runs the task against the pg_conn fixture, and asserts the
retention window's row is deleted while in-window rows survive.
Mirrors the TestCleanupPendingToolState pattern.

* fix(notifications): treat /c/new as no current conversation

useMatch('/c/:conversationId') treats the literal URL /c/new as a
real conversation id, so the toast suppression check confused
'user is on /c/new' with 'user is on the conversation needing
approval'. Explicit guard: when the matched id is 'new', fall
through to the no-match case so approval toasts still surface.

* docs(events): enumerate publish_user_event None-return paths

The function returns Optional[str] today, with None conflating five
distinct outcomes (missing args / push disabled / unserialisable /
Redis down / XADD failed). Every current call site is fire-and-
forget and ignores the return, so the right move is to document the
five cases rather than promote to an enum return — keeps the API
small while making the diagnostic surface (logs) obvious. If a
future caller needs to react differently per reason, promote then.

* refactor(sources): move source-id derivation out of worker module

application/api/user/sources/upload.py imported _derive_source_id
from application.worker — pulling the entire Celery worker module
into the API process at import time just for a two-line helper.

Move DOCSGPT_INGEST_NAMESPACE and the derivation function to a
new application/storage/db/source_ids.py module that both layers
can import without that dependency edge. worker.py re-exports the
old names (_derive_source_id, DOCSGPT_INGEST_NAMESPACE) for
backward-compatible imports from tests and any other in-tree
callers; new code should import from the new module directly.

* fix(cache): enable Redis health_check_interval to surface half-open TCP

Without health_check_interval, a half-open TCP socket (NAT silently
dropped state, ELB idle-close) can leave pubsub.get_message hanging
past the SSE generator's keepalive cadence — the kernel never
surfaces the dead socket because no payload is in flight. Setting
health_check_interval=10 makes redis-py ping every 10s when
otherwise idle, so the next get_message after the dead window
raises and the SSE loop falls into its reconnect path instead of
silently freezing on the user.

* chore(events): rename attachment.processing.progress to attachment.progress

The event-type taxonomy was inconsistent: source ingest emits
source.ingest.progress (three segments) while attachments emitted
attachment.processing.progress (four segments). Drops the
.processing. infix for parity. Worker publish sites, the slice
reducer's match, and the worker tests all flip together.

No external consumers — the event type is purely internal between
the publisher and the in-tab slice; safe to rename in one commit.

* feat: events cleanup

* fix: better docs

* fix: e2e tests
This commit is contained in:
Alex authored and GitHub committed 2026-05-15 12:23:31 +01:00
1 parent 4d6f360e3a
commit e351f45d88
135 files changed
+14394 -1253

No files matched your search

+385
View File
@@ -0,0 +1,385 @@
# SSE Notifications Runbook
> Operations guide for "user says they didn't get a notification" — and
> the related "the bell never lights up" / "my upload toast hangs" /
> "the chat answer doesn't reconnect" symptoms.
The user-facing notifications channel is the SSE pipe at
`/api/events` plus per-message reconnects at
`/api/messages/<id>/events`. This document maps a user complaint to
the diagnostic that surfaces the cause.
---
## TL;DR — first 60 seconds
Run these three commands in parallel before anything else:
```bash
# 1) Is Redis up and serving the pipe? Should print PONG instantly.
redis-cli -n 2 PING
# 2) Anyone subscribed to the channel right now? Numbers per channel.
redis-cli -n 2 PUBSUB NUMSUB user:<user_id>
# 3) Is the user's backlog populated? Returns the count of journaled events.
redis-cli -n 2 XLEN user:<user_id>:stream
```
- `PING` failing → Redis is the problem. Skip to "Redis-down".
- `NUMSUB user:<user_id>` returns 0 → no client connected. Skip to "Client never connects".
- `XLEN user:<user_id>:stream` returns 0 or low → publisher isn't writing. Skip to "Publisher silent".
- All three look healthy → the events are flowing on the wire; the issue is downstream of the slice (UI rendering, toast suppression, etc.). Skip to "Events flowing but UI silent".
---
## Architecture cheat-sheet
```
Worker (publish_user_event) Frontend tab
│ ▲
▼ │ GET /api/events SSE
Redis Streams: XADD Flask route
user:<id>:stream ──────────────► replay_backlog (snapshot)
│ +
▼ Topic.subscribe (live tail)
Redis pub/sub: PUBLISH │
user:<id> ────────────────────────────────┘
```
**Source of truth:**
- Persistent journal: Redis Stream `user:<user_id>:stream`, capped at
`EVENTS_STREAM_MAXLEN` (default 1000) entries via `MAXLEN ~`. ~24h
at typical event rates.
- Live fan-out: Redis pub/sub channel `user:<user_id>`. No durability;
subscribers must be attached at publish time.
The chat-stream pipe is separate, parallel infrastructure:
- Journal: Postgres `message_events` table.
- Live fan-out: Redis pub/sub `channel:<message_id>`.
Same patterns, different durability layer. This doc covers both;
they share most diagnostic commands.
---
## Symptom → diagnostic map
### A. "I uploaded a source and the toast never appeared"
User flow: chat → upload → expect toast.
| Step | Command | Expect |
| ------------------------------------------------- | ------------------------------------------------------------- | ----------------------------------------------- |
| Worker received the task | `tail -f celery.log` filtered by user | `ingest_worker` start log line |
| Worker published the queued event | `redis-cli -n 2 XREVRANGE user:<id>:stream + - COUNT 5` | A `source.ingest.queued` entry within seconds |
| Frontend got it | DevTools → Network → `/api/events` → EventStream tab | `data: {"type":"source.ingest.queued",...}` |
| Slice updated | Redux DevTools → state.upload.tasks | Task with matching `sourceId`, `status:'training'` |
If the worker's queued log line is there but the XADD didn't land →
look for a `publish_user_event payload not JSON-serializable` warning
in the worker log (the publisher swallows `TypeError`).
If the XADD landed but the frontend never received it → check
`PUBSUB NUMSUB user:<id>` while the user is on the page. If 0, the
SSE connection isn't subscribed; skip to "Client never connects".
If the frontend received it but the toast didn't render → the
`uploadSlice` extraReducer requires `task.sourceId` to match the
event's `scope.id`. Check the upload route returned `source_id` in
its POST response (the upload, connector, and reingest paths all
include it). Idempotent / cached responses must also include
`source_id` (`_claim_task_or_get_cached`).
### B. "The bell badge never goes up"
There is no bell — the global notifications surface is per-event
toasts, not an aggregated counter. If the user is on an old build,
`Cmd-Shift-R` to bypass cache. The surfaces they're looking for are
`UploadToast` for source uploads and `ToolApprovalToast` for
tool-approval events.
### C. "My chat answer froze mid-stream and never recovered"
User flow: ask question → answer streaming → network blip → answer
stops; should reconnect.
```bash
# Was the original message reserved in PG?
psql -c "SELECT id, status, prompt FROM conversation_messages \
WHERE user_id = '<user>' ORDER BY timestamp DESC LIMIT 5;"
# Did the journal capture events past the user's last-seen seq?
psql -c "SELECT sequence_no, event_type FROM message_events \
WHERE message_id = '<id>' ORDER BY sequence_no;"
# Is the live tail still producing? (subscribe and watch)
redis-cli -n 2 SUBSCRIBE channel:<message_id>
```
The frontend should reconnect via `GET /api/messages/<id>/events`
when the original POST stream closes without a typed `end` or
`error` event. If it's not reconnecting, `console.warn('Stream
reconnect failed', ...)` will be in the browser console — the
reconnect HTTP errored. Common cases:
- The user's JWT rotated mid-stream → 401 on the GET. Frontend
doesn't auto-refresh; the user reloads.
- The user is on a different host than the API and CORS is rejecting
the GET → check `application/asgi.py` allow-headers.
### D. "The dev install never delivers any notifications at all"
Default `AUTH_TYPE` unset means `decoded_token = {"sub": "local"}`
for every request. The SSE client connects without the
`Authorization` header in this case, and `user:local:stream` is
the shared channel everything goes to. If the user has multiple dev
machines pointing at the same Redis, they will see each other's
events. Confirm with:
```bash
redis-cli -n 2 KEYS 'user:local:*'
```
If multiple deployments share the Redis, document that as a known
multi-user-on-local-channel limitation. Set `AUTH_TYPE=simple_jwt`
to scope per-user.
### E. "The notifications channel was working, then suddenly stopped after the user reloaded the page"
Likely path: `backlog.truncated` event fired, the slice cleared
`lastEventId` to null, the closure was carrying the same stale id and
re-tripped the same truncation on every reconnect. **Verify the user
is on a current build — `eventStreamClient.ts` must re-read
`lastEventId = opts.getLastEventId();` without a truthy guard so the
null clear propagates into the next reconnect.**
### F. "I keep getting 429 on /api/events"
The per-user concurrent-connection cap (`SSE_MAX_CONCURRENT_PER_USER`,
default 8) refused the connection. User has too many tabs open or a
runaway reconnect loop. `redis-cli -n 2 GET user:<id>:sse_count`
shows the live counter; the TTL is 1h from the last connection
attempt (rolling — every INCR re-seeds it), so the key only ages
out after the user stops reconnecting for a full hour.
If the count is wedged high without explanation, the
counter-DECR-in-finally path didn't run (worker SIGKILL, OOM). Wait
for the TTL or `redis-cli -n 2 DEL user:<id>:sse_count` to reset.
### G. "Replay snapshot stops at 200 events"
The route caps each replay at `EVENTS_REPLAY_MAX_PER_REQUEST`
(default 200). The cap is intentionally **silent** — the route does NOT
emit a `backlog.truncated` notice for cap-hit. The 200 entries each
carry their own `id:` header, so the frontend's slice cursor
advances to the most-recent delivered id. Next reconnect sends
`last_event_id=<max_replayed>` and the snapshot resumes from there.
A user that was 1000 entries behind catches up over ~5 reconnects.
If the user reports getting HTTP 429 on `/api/events` despite being
well under `SSE_MAX_CONCURRENT_PER_USER`, they hit the windowed
replay budget (`EVENTS_REPLAY_BUDGET_REQUESTS_PER_WINDOW`, default
30 / `EVENTS_REPLAY_BUDGET_WINDOW_SECONDS` 60s). The route refuses
the connection so the slice cursor stays pinned at whatever value
it had; the frontend backs off and the next reconnect (after the
window rolls) gets the proper snapshot. Serving the live tail
without a snapshot used to be the behavior here, but that let the
client advance `lastEventId` past entries it never received,
permanently stranding the un-replayed window — so the route now
429s instead. `redis-cli -n 2 GET user:<id>:replay_count` shows the
current counter; TTL is the window size.
`backlog.truncated` is emitted ONLY when the client's
`Last-Event-ID` has slid off the MAXLEN'd window — i.e. the journal
is genuinely gone past the cursor and the frontend should clear the
slice cursor and refetch state. Treating cap-hit or
budget-exhaustion the same way would lock the user into re-receiving
the oldest 200 entries on every reconnect (the cursor would clear,
the snapshot would re-serve from the start, the cap would re-trip).
### H. "User says push notifications stopped after a deploy"
- Pull `event.published topic=user:<id> type=...` from the worker
logs to confirm the publisher is still firing.
- Pull `event.connect user=<id>` from the API logs to confirm the
client is reconnecting.
- Check the gunicorn worker count and `WSGIMiddleware(workers=32)` —
if the deploy reduced worker count, the per-user cap is still 8
but total concurrent SSE connections are bounded by `gunicorn
workers × 32`. A capacity miss looks like users randomly getting
429'd.
---
## Common failure modes
### Redis-down
Symptoms: `/api/events` returns 200 but emits only `: connected`
then the body closes. `XLEN` and `PUBLISH` both fail. The publisher's
`record_event` swallows the failure and returns False; the live tail
publish also drops on the floor. Frontend retries forever with
exponential backoff.
Resolution: bring Redis back. The journal is gone (was in-memory
only — Streams persist within a single Redis instance, no replication
configured). New events flow as soon as Redis comes back.
### `AUTH_TYPE` misconfigured = sub:"local" cross-stream
Symptoms: every user shares `user:local:stream`. Any user sees
everyone else's notifications.
Resolution: set `AUTH_TYPE=simple_jwt` (or `session_jwt`) in `.env`.
The events route logs a one-time WARNING per process when
`sub == "local"` is observed. A repeat WARNING after a restart
confirms the misconfiguration.
### MAXLEN trimmed past Last-Event-ID
Symptoms: client reconnects with `last_event_id=X`, snapshot returns
the entire MAXLEN'd backlog (because X is older than the oldest
retained entry). Old events appear duplicated.
Detection: the route's `_oldest_retained_id` check emits
`backlog.truncated` when this case fires. Frontend's
`dispatchSSEEvent` clears `lastEventId` so the next reconnect starts
fresh.
If the WARNING isn't firing but symptoms match: the user's client
may have a corrupt cached `lastEventId`. `localStorage` doesn't
store this state; check Redux state via DevTools.
### Stale event-stream client
Symptoms: events visible in `XRANGE` but the frontend slice doesn't
update.
```bash
# Is the client subscribed?
redis-cli -n 2 PUBSUB NUMSUB user:<id>
# When did its connection start?
grep "event.connect user=<id>" /var/log/docsgpt.log | tail -3
```
If `NUMSUB` is 0 and no recent `event.connect`, the user's tab is
closed or the connection died and never reconnected. Push them to
reload.
### Publisher silent
Symptoms: worker is processing the task (Celery says SUCCESS), but
no XADD and no PUBLISH. User sees no events.
```bash
# Was the publisher import error suppressed?
grep "publish_user_event" /var/log/celery.log | grep -i "warn\|error" | tail -20
# Is push disabled?
grep "ENABLE_SSE_PUSH" /var/log/docsgpt.log | tail -5
```
`ENABLE_SSE_PUSH=False` in `.env` would silence the publisher
globally. Useful for incident response if a runaway publisher is
DoS'ing Redis; toggle off, fix root cause, toggle on.
---
## Useful one-liners
```bash
# Watch a user's live event stream in real time (all events, all types)
redis-cli -n 2 PSUBSCRIBE 'user:*' | grep "user:<id>"
# Last 10 events the user would see on reconnect
redis-cli -n 2 XREVRANGE user:<id>:stream + - COUNT 10
# Live count of subscribed clients per user
redis-cli -n 2 PUBSUB NUMSUB $(redis-cli -n 2 PUBSUB CHANNELS 'user:*')
# Trim a runaway stream (CAREFUL — destroys backlog for all current
# subscribers; OK after explaining to the user)
redis-cli -n 2 XTRIM user:<id>:stream MAXLEN 0
# Clear a wedged concurrent-connection counter
redis-cli -n 2 DEL user:<id>:sse_count
# Force-flip every client to re-snapshot (drop the stream key entirely
# — destroys the backlog; clients reconnect with their last id and
# get a backlog.truncated)
redis-cli -n 2 DEL user:<id>:stream
```
---
## Settings reference
Everything in `application/core/settings.py`:
| Setting | Default | Purpose |
| --------------------------------------------- | ------- | --------------------------------------------- |
| `ENABLE_SSE_PUSH` | `True` | Master switch. False = publisher no-ops, route serves "push_disabled" comment. |
| `EVENTS_STREAM_MAXLEN` | `1000` | Per-user backlog cap. Approximate via `XADD MAXLEN ~`. |
| `SSE_KEEPALIVE_SECONDS` | `15` | Comment-frame cadence. Must sit under reverse-proxy idle close. |
| `SSE_MAX_CONCURRENT_PER_USER` | `8` | Cap on simultaneous SSE connections per user. 0 = disabled. |
| `EVENTS_REPLAY_MAX_PER_REQUEST` | `200` | Hard cap on snapshot rows per request. |
| `EVENTS_REPLAY_BUDGET_REQUESTS_PER_WINDOW` | `30` | Per-user replays per window. 0 = disabled. |
| `EVENTS_REPLAY_BUDGET_WINDOW_SECONDS` | `60` | Window length. |
| `MESSAGE_EVENTS_RETENTION_DAYS` | `14` | Retention for the `message_events` journal; `cleanup_message_events` beat task deletes older rows. |
---
## Known limitations
### Each tab runs its own SSE connection
There is no cross-tab dedup. Every tab open to the app holds its
own SSE connection and dispatches every received event into its
own Redux store, so a user with N tabs open will see N copies of
each toast. With `SSE_MAX_CONCURRENT_PER_USER=8` (the default) a
heavy multi-tab user can also hit the connection cap and start
seeing 429s. Cross-tab dedup via a `BroadcastChannel` ring +
`navigator.locks`-based leader election is tracked as future work.
### `/c/<unknown-id>` normalises to `/c/new`
If a user navigates to a conversation id that isn't in their
loaded list, the conversation route rewrites the URL to `/c/new`.
`ToolApprovalToast`'s gate uses `useMatch('/c/:conversationId')`,
so for the brief window after the rewrite the toast may surface
for a conversation the user *thought* they were already viewing.
Pre-existing route behaviour; not a notifications regression.
### Terminal events un-dismiss running uploads
`frontend/src/upload/uploadSlice.ts` sets `dismissed: false` when
an upload reaches `completed` or `failed`. If the user dismissed a
running task and the terminal SSE arrives later, the toast pops
back. Intentional ("notify the user it's done"); revisit if the
re-surface UX is too aggressive for v2.
### Werkzeug doesn't auto-reload route files
The dev server (`flask run`) doesn't watch
`application/api/events/routes.py` for changes by default.
After editing the route, restart Flask manually — `--reload`
isn't on. (Production gunicorn reloads via deploy.)
### MCP OAuth completion can fall outside the user stream's MAXLEN window
`get_oauth_status` scans up to `EVENTS_STREAM_MAXLEN` (~1000) entries via `XREVRANGE`. If the user has a high-rate ingest running concurrent with the OAuth handshake, the `mcp.oauth.completed` envelope can be trimmed off the back before they click Save. Symptom: backend returns "OAuth failed or not completed" even though the popup completed successfully.
Mitigation today: bump `EVENTS_STREAM_MAXLEN` per-deployment if your users routinely flood the channel during OAuth flows. A dedicated short-TTL Redis key for OAuth task results is tracked as a follow-up.
### React StrictMode double-mounts SSE
In dev, React 18 StrictMode mounts → unmounts → remounts every
component, briefly opening two SSE connections per tab before the
first is aborted. With `SSE_MAX_CONCURRENT_PER_USER=8` and 4–5
tabs open concurrently you can transiently hit the cap and see
HTTP 429 on cold-load. The first connection's counter increment
fires before the AbortController-induced disconnect can decrement
it. Production (single mount, no StrictMode) is unaffected; raise
the cap in dev or accept transient 429s.