fix(admin): second review pass

Correctness
- /api/remote never recorded source.created, so URL, GitHub and connector
  sources had a source.deleted with no matching creation. All three creation
  paths now go through one _audit_source_created helper.
- The prompt-cache rate divided cached tokens by a whole bucket's prompt
  tokens. A bucket is a day and mixes calls whose provider reports a cache
  breakdown with calls whose provider does not, so filtering buckets in the
  client could not separate them and the rate was understated by however much
  traffic ran on a non-reporting provider. The denominator is now computed in
  SQL over the reporting rows.
- The outcome pill matched values nothing writes. Guardrails emit triggered /
  not_evaluated and the device feed emits dispatched; the map had blocked /
  denied / allowed, so a guardrail that fired rendered neutral grey -- the one
  signal the merged feed exists to surface. Fixtures were seeding the
  fictional values, so the tests passed on it too.
- Stream duration_ms timed the consumer. stream_token_usage is a generator,
  so start-to-exhaustion includes the agent loop's tool handling and the SSE
  client's pace; a slow browser recorded ~30s for a sub-second call. It now
  accumulates only the time spent inside next().

Safety
- Activity filters failed open: an unknown facet or unparseable timestamp was
  dropped, and no filter means every row, so a typo widened an audit view and
  on the export streamed the full history. Both are now a 400.
- The search term was interpolated into an ILIKE pattern, so "100%" matched
  everything and "q1_report" matched more than it should. Escaped.
- 0034 set actor_id NOT NULL with no default. A previous-release process
  inserting mid-rollout would raise, and in admin/routes.py that insert shares
  the request transaction, so a role grant beside it would roll back too.

Noise and dead code
- The per-user panel is a security panel: data-plane events file under the
  actor, so an active account's routine deletes pushed a denied login out of
  the 20-row window. It now excludes them; the Activity tab shows everything.
- device_audit_log had no created_at-leading index, so the merged feed
  sequentially scanned that branch every page (migration 0036).
- conversation.deleted_all no longer records when nothing was deleted, and
  agent.updated no longer records an empty field list.
- Dropped by_model from /admin/usage (no consumer; an extra aggregate per page
  load), the duplicate filter surface on AuthEventsRepository that nothing
  called, and the unreachable FLOW_LABELS.schedule entry.
- Type hints on record_event's conn and the remaining unannotated helpers.

docsgpt/api/admin/activity.py

@@ -43,23 +43,38 @@ _EXPORT_CHUNK = 1_000
 _MAX_SEARCH_LENGTH = 200
 
 
+class _InvalidFilter(ValueError):
+    """A filter argument the caller got wrong. Surfaces as a 400."""
+
+
 def _facet_list(name: str, allowed: tuple[str, ...]) -> Optional[list[str]]:
-    """Parse a repeated/comma-separated query arg, dropping unknown values.
+    """Parse a repeated/comma-separated query arg into known facet values.
+
+    Rejects unknown values rather than dropping them. Dropping would leave the
+    request with no filter on that facet, and "no filter" means *everything* --
+    so a typo or a stale chip would silently widen an audit view instead of
+    narrowing it, and on the export would stream the whole history.
 
     Args:
         name: Query parameter name.
-        allowed: The permitted values; anything else is ignored rather than
-            rejected, so an old bookmark degrades instead of erroring.
+        allowed: The permitted values.
 
     Returns:
-        The distinct valid values in the order given, or None when absent.
+        The distinct values in the order given, or None when absent.
+
+    Raises:
+        _InvalidFilter: If any supplied value is not in ``allowed``.
     """
     raw: list[str] = []
     for value in request.args.getlist(name):
-        raw.extend(part.strip() for part in value.split(","))
+        raw.extend(part.strip() for part in value.split(",") if part.strip())
     seen: list[str] = []
     for value in raw:
-        if value in allowed and value not in seen:
+        if value not in allowed:
+            raise _InvalidFilter(
+                f"Unknown {name}: {value!r}. Expected one of {', '.join(allowed)}."
+            )
+        if value not in seen:
             seen.append(value)
     return seen or None
 
@@ -76,14 +91,21 @@ def _event_list() -> Optional[list[str]]:
 
 
 def _time_arg(name: str) -> Optional[datetime]:
-    """Parse an ISO-8601 timestamp query arg, or None when absent/unparseable."""
+    """Parse an ISO-8601 timestamp query arg, or None when absent.
+
+    Raises:
+        _InvalidFilter: If present but unparseable. Silently ignoring it would
+            drop the time bound and return the full history.
+    """
     raw = (request.args.get(name) or "").strip()
     if not raw:
         return None
     try:
         parsed = datetime.fromisoformat(raw.replace("Z", "+00:00"))
-    except ValueError:
-        return None
+    except ValueError as exc:
+        raise _InvalidFilter(
+            f"{name} must be an ISO-8601 timestamp, got {raw!r}."
+        ) from exc
     # A naive bound is read as UTC, matching how the feed stores timestamps.
     return parsed if parsed.tzinfo else parsed.replace(tzinfo=timezone.utc)
 
@@ -122,7 +144,10 @@ class AdminActivityResource(Resource):
     def get(self):
         """Merged audit feed, newest first, with filters and a total."""
         page, page_size, offset = _page()
-        filters = _filters()
+        try:
+            filters = _filters()
+        except _InvalidFilter as exc:
+            return make_response(jsonify({"success": False, "message": str(exc)}), 400)
         with db_readonly() as conn:
             repo = ActivityRepository(conn)
             total = repo.count(**filters)
@@ -183,7 +208,10 @@ class AdminActivityExportResource(Resource):
                 jsonify({"success": False, "message": "Invalid format"}), 400
             )
         limit = max(1, min(_EXPORT_MAX_ROWS, _int_arg("max_rows", _EXPORT_MAX_ROWS)))
-        filters = _filters()
+        try:
+            filters = _filters()
+        except _InvalidFilter as exc:
+            return make_response(jsonify({"success": False, "message": str(exc)}), 400)
         stamp = datetime.now(timezone.utc).strftime("%Y%m%d-%H%M%S")
         generator = _csv_rows if fmt == "csv" else _ndjson_rows
         return Response(

docsgpt/api/admin/routes.py

@@ -38,6 +38,9 @@ admin_ns = Namespace("admin", description="Admin-only management endpoints", pat
 _DEFAULT_PAGE_SIZE = 25
 _MAX_PAGE_SIZE = 100
 
+# Event namespaces the per-user security panel leaves out; see the drill-down.
+_DATA_PLANE_PREFIXES = ("source.", "agent.", "conversation.")
+
 
 def _int_arg(name: str, default: int) -> int:
     try:
@@ -135,8 +138,12 @@ class AdminUserResource(Resource):
                 },
                 "roles": sorted({"user", *roles_repo.role_names_for(user_id)}),
                 "grants": roles_repo.list_for(user_id),
+                # Identity and access only. Data-plane events file under the
+                # actor, so an active user's routine deletes would push a
+                # denied login or a role grant out of a 20-row window.
+                # The Activity tab, filtered by user, shows everything.
                 "recent_events": AuthEventsRepository(conn).list_recent(
-                    user_id, limit=20
+                    user_id, limit=20, exclude_events_like=_DATA_PLANE_PREFIXES
                 ),
                 "counts": AdminStatsRepository(conn).user_counts(user_id),
             }
@@ -347,7 +354,6 @@ class AdminUsageResource(Resource):
             total = usage_repo.sum_tokens_in_range(
                 start=start, end=datetime.now(timezone.utc)
             )
-            by_model = usage_repo.tokens_by_model(start=start)
             latency = stats_repo.latency_summary(since=start)
             top_users = stats_repo.top_token_users(since=start, limit=10)
         return make_response(
@@ -362,7 +368,6 @@ class AdminUsageResource(Resource):
                     # Summed from the series, not from ``by_model``: the latter
                     # drops rows with no model_id, so it would undercount.
                     "total_cost": round(sum(row["cost"] for row in series), 4),
-                    "by_model": by_model,
                     "latency": latency,
                     "top_users": top_users,
                 }

docsgpt/api/audit.py

@@ -25,6 +25,7 @@ import logging
 from typing import Any, Optional
 
 from flask import has_request_context, request
+from sqlalchemy import Connection
 
 from docsgpt.storage.db.repositories.auth_events import AuthEventsRepository
 
@@ -32,7 +33,7 @@ logger = logging.getLogger(__name__)
 
 
 def record_event(
-    conn,
+    conn: Connection,
     event: str,
     *,
     actor: Optional[str],
@@ -42,8 +43,9 @@ def record_event(
     """Append one audit event, best-effort, inside the caller's transaction.
 
     Args:
-        conn: Open database connection, inside the action's transaction so the
-            audit row commits atomically with the change it describes.
+        conn: Open SQLAlchemy ``Connection`` -- not a Session -- inside the
+            action's transaction, so the audit row commits atomically with the
+            change it describes and ``begin_nested`` gives it a real SAVEPOINT.
         event: Dotted event name (``source.deleted``, ``agent.created``).
         actor: Who performed the action; ``"unknown"`` when unauthenticated.
         target: The user acted upon, when the event is about a user. Data-plane

docsgpt/api/user/agents/routes.py

@@ -1294,7 +1294,8 @@ class UpdateAgent(Resource):
                     updated = bool(result)
                 else:
                     updated = agents_repo.update(pg_agent_id, user, update_fields)
-                if updated:
+                # An update that changed no field is not an event.
+                if updated and update_fields:
                     record_event(
                         conn,
                         "agent.updated",

docsgpt/api/user/conversations/routes.py

@@ -73,12 +73,15 @@ class DeleteAllConversations(Resource):
         try:
             with db_session() as conn:
                 deleted = ConversationsRepository(conn).delete_all_for_user(user_id)
-                record_event(
-                    conn,
-                    "conversation.deleted_all",
-                    actor=user_id,
-                    deleted=deleted,
-                )
+                # Nothing deleted is not an event; the endpoint is idempotent
+                # and a row here would render as a destructive action.
+                if deleted:
+                    record_event(
+                        conn,
+                        "conversation.deleted_all",
+                        actor=user_id,
+                        deleted=deleted,
+                    )
         except Exception as err:
             current_app.logger.error(
                 f"Error deleting all conversations: {err}", exc_info=True

docsgpt/api/user/sources/upload.py

@@ -5,6 +5,7 @@ import os
 import tempfile
 import uuid
 import zipfile
+from typing import Optional
 
 from flask import current_app, jsonify, make_response, request
 from flask_restx import fields, Namespace, Resource
@@ -179,6 +180,40 @@ def _source_archive_limits() -> ZipExtractionLimits:
     )
 
 
+def _audit_source_created(
+    *,
+    source_id: str,
+    user: Optional[str],
+    name: Optional[str],
+    source_type: str,
+    task_id: Optional[str],
+) -> None:
+    """Record ``source.created`` for an accepted ingestion job.
+
+    Audited at the request, not in the worker: enqueuing is the user's action.
+    The ingest may still fail, which the source's own status records.
+
+    Every path that mints a ``source_id`` and dispatches must call this --
+    ``source.deleted`` is recorded for remote sources too, and a trail showing
+    a deletion with no matching creation is worse than no trail.
+    """
+    try:
+        with db_session() as conn:
+            record_event(
+                conn,
+                "source.created",
+                actor=user,
+                source_id=source_id,
+                name=name,
+                type=source_type,
+                task_id=task_id,
+            )
+    except Exception as err:
+        current_app.logger.warning(
+            "Could not audit source.created for %s: %s", source_id, err, exc_info=True
+        )
+
+
 @sources_upload_ns.route("/upload")
 class UploadFile(Resource):
     @api.expect(
@@ -397,26 +432,13 @@ class UploadFile(Resource):
             return make_response(jsonify({"success": False}), 400)
         # Predetermined id matches the dedup-claim row; loser GET sees same.
         response_task_id = predetermined_task_id or task.id
-        # Audited here, not in the worker: this is the user action. The
-        # ingest may still fail, which the source's own status records.
-        try:
-            with db_session() as conn:
-                record_event(
-                    conn,
-                    "source.created",
-                    actor=user,
-                    source_id=str(source_uuid),
-                    name=job_name,
-                    type="local",
-                    task_id=response_task_id,
-                )
-        except Exception as err:
-            current_app.logger.warning(
-                "Could not audit upload for source %s: %s",
-                source_uuid,
-                err,
-                exc_info=True,
-            )
+        _audit_source_created(
+            source_id=str(source_uuid),
+            user=user,
+            name=job_name,
+            source_type="local",
+            task_id=response_task_id,
+        )
         # ``source_uuid`` was minted above and passed to the worker as
         # ``source_id``; the worker uses it verbatim for every SSE event,
         # so the frontend can correlate inbound ``source.ingest.*`` to
@@ -568,6 +590,13 @@ class UploadRemote(Resource):
                     connector_kwargs["task_id"] = predetermined_task_id
                 task = ingest_connector_task.apply_async(**connector_kwargs)
                 response_task_id = predetermined_task_id or task.id
+                _audit_source_created(
+                    source_id=str(source_uuid),
+                    user=user,
+                    name=data["name"],
+                    source_type=data["source"],
+                    task_id=response_task_id,
+                )
                 # ``source_uuid`` was minted above and passed to the
                 # worker as ``source_id``; the worker uses it verbatim
                 # for every SSE event, so the frontend can correlate
@@ -601,6 +630,13 @@ class UploadRemote(Resource):
                 _release_claim(scoped_key)
             return make_response(jsonify({"success": False}), 400)
         response_task_id = predetermined_task_id or task.id
+        _audit_source_created(
+            source_id=str(source_uuid),
+            user=user,
+            name=data["name"],
+            source_type=data["source"],
+            task_id=response_task_id,
+        )
         response_payload = {
             "success": True,
             "task_id": response_task_id,