Introduction

The AI Tender Opportunity Scanner has now reached an advanced stage of development.

The platform can:

Collect tender opportunities from external sources
Normalize and store tender data
Analyze tender documents with AI
Match opportunities to organization profiles
Rank tenders by relevance
Manage saved opportunities
Track bid pipelines
Generate analytics
Send email notifications
Deliver Slack and Microsoft Teams alerts
Authenticate users
Manage organizations and permissions
Isolate tenant data
Process subscriptions and billing
Track AI usage and infrastructure costs
Route AI requests across models and providers

These capabilities create a useful and commercially viable SaaS platform.

However, a production system cannot be operated safely by simply deploying the application and waiting for users to report problems.

We need to know:

			
Is the API responding?
Are background workers running?
Are tender ingestion jobs failing?
Are AI providers timing out?
Are webhook events being processed?
Are emails being delivered?
Are database queries becoming slower?
Is one organization generating abnormal traffic?
Is the error rate increasing?
Are subscription events delayed?
Is the platform approaching capacity?

		

Traditional application monitoring often focuses on whether servers are running.

Observability goes further.

It helps us understand the internal state of the entire system by examining its outputs:

Logs
Metrics
Traces
Events
Errors
Alerts

In this article, we will build the operational foundation required to run the Tender Opportunity Scanner reliably in production.

We will design:

Structured application logging
Request correlation IDs
Centralized log collection
Application metrics
Infrastructure metrics
Distributed tracing
Error tracking
Health and readiness endpoints
Background-job monitoring
AI-provider monitoring
Billing webhook monitoring
Alert policies
Incident severity levels
Operational dashboards
Service-level indicators
Service-level objectives
Audit-safe logging
Data-retention policies
Testing and incident response workflows

The goal is not merely to collect more data.

The goal is to make production behavior understandable and actionable.

Development Progress

			
Project Scope and Roadmap        ████████████████████ 100%
MVP Definition                   ████████████████████ 100%
Technology Stack                 ████████████████████ 100%
System Architecture              ████████████████████ 100%
Database Design                  ████████████████████ 100%
Backend Project Structure        ████████████████████ 100%
SQLAlchemy Models                ████████████████████ 100%
Alembic Migration                ████████████████████ 100%
Repository Layer                 ████████████████████ 100%
CRUD Service Layer               ████████████████████ 100%
Tender Ingestion Pipeline        ████████████████████ 100%
AI Analysis Pipeline             ████████████████████ 100%
AI Matching Engine               ████████████████████ 100%
React Dashboard                  ████████████████████ 100%
Tender Detail Page               ████████████████████ 100%
Saved Opportunities Workspace    ████████████████████ 100%
Pipeline Analytics Dashboard     ████████████████████ 100%
Notifications & Monitoring       ████████████████████ 100%
Email Notifications              ████████████████████ 100%
Slack & Teams Integrations       ████████████████████ 100%
Authentication & Organizations   ████████████████████ 100%
Multi-Tenant Data Isolation      ████████████████████ 100%
Billing & Subscription System    ████████████████████ 100%
AI Usage & Cost Optimization     ████████████████████ 100%
Production Observability         ████████████████████ 100%
Deployment & Release Automation  ░░░░░░░░░░░░░░░░░░░░   0%

		

What Is Observability?

Observability is the ability to understand what is happening inside a system by examining the information it produces.

A production platform emits several forms of operational data.

			
Application
    |
    +---- Logs
    +---- Metrics
    +---- Traces
    +---- Errors
    +---- Events

		

Each signal answers different questions.

Logs

Logs record individual events.

Examples:

			
User logged in
Tender ingestion started
AI analysis completed
Webhook signature rejected
Email delivery failed
Subscription updated

		

Logs are useful for investigating specific events and failures.

Metrics

Metrics record numerical values over time.

Examples:

			
API requests per second
Error rate
Average response time
Queue depth
Active users
AI request cost
Database connection usage

		

Metrics are useful for dashboards, trends, and alerts.

Traces

Traces follow a request or job across multiple services.

Example:

			
Upload Tender
      |
      v
FastAPI Endpoint
      |
      v
Storage Service
      |
      v
Background Queue
      |
      v
Document Parser
      |
      v
AI Provider
      |
      v
Database

		

A trace helps identify which component caused a slow or failed workflow.

Errors

Error-tracking systems group exceptions and provide:

Stack traces
Frequency
Affected users
Environment
Release version
Request context
First and latest occurrence

This is more useful than searching raw log files for exception messages.

Observability Architecture

			
FastAPI API
     |
     +---- Structured Logs
     +---- Metrics
     +---- Traces
     +---- Errors
     |
     v
Observability Collector
     |
     +---- Log Platform
     +---- Metrics Platform
     +---- Trace Platform
     +---- Error Tracker
     |
     v
Dashboards and Alerts
     |
     v
Operations Team

		

The platform may use separate tools or a combined observability service.

Core Observability Principles

The implementation should follow several principles.

			
Every request receives a correlation ID
Every background job receives a job ID
Every log is structured
Every critical workflow emits metrics
Every external dependency is monitored
Every alert has an owner
Every alert should require action
Sensitive customer data is excluded

		

Observability data must help operators solve problems rather than create noise.

Monitoring Versus Observability

Monitoring tells us that a known problem has occurred.

Example:

API error rate exceeds 5%

Observability helps us investigate unfamiliar problems.

Example:

			
Why are Professional-plan customers experiencing slow tender analysis only when PDF files exceed 100 pages?

Both capabilities are required.

Recommended Project Structure

Create an observability module.

			
app/
├── observability/
│   ├── __init__.py
│   ├── logging.py
│   ├── context.py
│   ├── metrics.py
│   ├── tracing.py
│   ├── errors.py
│   ├── middleware.py
│   ├── health.py
│   └── redaction.py
│
├── api/
│   └── routes/
│       └── health.py
│
└── services/
    └── operational_event_service.py

		

This centralizes operational concerns and prevents inconsistent logging across services.

Structured Logging

Plain-text logs are difficult to query reliably.

Bad:

Tender 551 failed for Northstar because provider timed out

Better:

			
{
  "event": "tender_analysis_failed",
  "tender_id": 551,
  "organization_id": 10,
  "provider": "primary_ai",
  "error_type": "timeout",
  "correlation_id": "req_7e3b...",
  "severity": "error"
}

		

Structured logs let the log platform index individual fields.

Logging Configuration

Create:

app/observability/logging.py

The configuration should define:

JSON formatting
Log level
Timestamp format
Application name
Environment
Release version
Correlation context
Redaction filters

Example:

			
import logging
import sys
from pythonjsonlogger import jsonlogger
def configure_logging(
    *,
    service_name: str,
    environment: str,
    release_version: str,
    log_level: str = "INFO",
) -> None:
    handler = logging.StreamHandler(sys.stdout)
    formatter = jsonlogger.JsonFormatter(
        "%(asctime)s %(levelname)s %(name)s "
        "%(message)s %(correlation_id)s "
        "%(organization_id)s %(user_id)s"
    )
    handler.setFormatter(formatter)
    root_logger = logging.getLogger()
    root_logger.handlers.clear()
    root_logger.addHandler(handler)
    root_logger.setLevel(log_level.upper())
    logging.getLogger(__name__).info(
        "logging_configured",
        extra={
            "service_name": service_name,
            "environment": environment,
            "release_version": release_version,
        },
    )

		

In a production implementation, a custom formatter can inject standard fields automatically.

Standard Log Fields

Every structured log should use consistent field names.

			
timestamp
severity
service_name
environment
release_version
event
message
correlation_id
trace_id
span_id
organization_id
user_id
request_method
request_path
status_code
duration_ms

		

Workflow-specific fields can then be added.

Example:

			
tender_id
job_id
provider
model
subscription_id
webhook_event_id

		

Consistency is more important than logging every possible value.

Event Names

Use stable machine-readable event names.

Good:

			
tender_ingestion_started
tender_ingestion_completed
tender_ingestion_failed
ai_analysis_started
ai_analysis_completed
ai_analysis_failed

		

Avoid vague messages such as:

Something went wrong

Stable event names improve queries, dashboards, and alert rules.

Correlation IDs

A correlation ID connects all logs generated by one request or workflow.

Example:

			
Browser Request
      |
      v
correlation_id = req_abc123
      |
      +---- API log
      +---- Database log
      +---- Queue message
      +---- Worker log
      +---- AI-provider log

		

Without a shared identifier, investigating a distributed workflow becomes much harder.

Correlation Context

Create:

app/observability/context.py

Use contextvars so request information remains available throughout asynchronous execution.

			
from contextvars import ContextVar
correlation_id_context: ContextVar[
    str | None
] = ContextVar(
    "correlation_id",
    default=None,
)
organization_id_context: ContextVar[
    int | None
] = ContextVar(
    "organization_id",
    default=None,
)
user_id_context: ContextVar[
    int | None
] = ContextVar(
    "user_id",
    default=None,
)

		

Correlation Middleware

Create:

app/observability/middleware.py

			
from uuid import uuid4
from fastapi import Request
from starlette.middleware.base import (
    BaseHTTPMiddleware,
)
class CorrelationIdMiddleware(
    BaseHTTPMiddleware
):
    async def dispatch(
        self,
        request: Request,
        call_next,
    ):
        incoming_id = request.headers.get(
            "X-Correlation-ID"
        )
        correlation_id = (
            incoming_id
            if incoming_id
            else f"req_{uuid4().hex}"
        )
        token = correlation_id_context.set(
            correlation_id
        )
        try:
            response = await call_next(request)
            response.headers[
                "X-Correlation-ID"
            ] = correlation_id
            return response
        finally:
            correlation_id_context.reset(token)

		

The response returns the identifier so support teams can ask customers for it.

Validating Incoming Correlation IDs

Do not trust arbitrary, excessively long values from external clients.

Validate:

			
Maximum length
Allowed characters
Expected prefix
No line breaks
No control characters

		

Invalid values should be replaced with a server-generated identifier.

Request Logging Middleware

Create middleware that records:

HTTP method
Path
Status code
Duration
Correlation ID
User and organization when available

			
import time
import logging
from fastapi import Request
from starlette.middleware.base import (
    BaseHTTPMiddleware,
)
logger = logging.getLogger(__name__)
class RequestLoggingMiddleware(
    BaseHTTPMiddleware
):
    async def dispatch(
        self,
        request: Request,
        call_next,
    ):
        started_at = time.perf_counter()
        try:
            response = await call_next(request)
        except Exception:
            duration_ms = (
                time.perf_counter() - started_at
            ) * 1000
            logger.exception(
                "http_request_failed",
                extra={
                    "request_method":
                        request.method,
                    "request_path":
                        request.url.path,
                    "duration_ms":
                        round(duration_ms, 2),
                },
            )
            raise
        duration_ms = (
            time.perf_counter() - started_at
        ) * 1000
        logger.info(
            "http_request_completed",
            extra={
                "request_method":
                    request.method,
                "request_path":
                    request.url.path,
                "status_code":
                    response.status_code,
                "duration_ms":
                    round(duration_ms, 2),
            },
        )
        return response

		

Do not log query parameters or request bodies by default because they may contain sensitive information.

Logging Authenticated Context

After authentication succeeds, add:

			
organization_id
user_id
membership_id

to the current logging context.

Do not place:

Access tokens
Refresh tokens
Passwords
Invitation tokens
Raw authorization headers

in logs.

Sensitive Data Redaction

Create:

app/observability/redaction.py

Sensitive keys may include:

			
password
access_token
refresh_token
authorization
api_key
secret
webhook_secret
payment_method
tax_id

		

Example:

			
SENSITIVE_KEYS = {
    "password",
    "access_token",
    "refresh_token",
    "authorization",
    "api_key",
    "secret",
    "webhook_secret",
}
def redact_mapping(
    value: dict,
) -> dict:
    redacted = {}
    for key, item in value.items():
        normalized_key = key.lower()
        if normalized_key in SENSITIVE_KEYS:
            redacted[key] = "[REDACTED]"
        elif isinstance(item, dict):
            redacted[key] = redact_mapping(item)
        else:
            redacted[key] = item
    return redacted

		

Redaction should happen before data reaches the log formatter.

Avoid Logging Tender Content

Tender documents may contain:

Confidential commercial information
Personal data
Supplier details
Pricing data
Internal notes
Uploaded attachments

Log identifiers and metadata instead.

Bad:

			
Prompt:
Analyze this confidential 180-page tender...

Better:

			
{
  "event": "ai_analysis_started",
  "organization_id": 10,
  "tender_id": 551,
  "document_hash": "sha256:...",
  "document_pages": 180,
  "prompt_version": "risk-analysis-v3"
}

		

Log Levels

Use log levels consistently.

			
DEBUG
Detailed development diagnostics
INFO
Normal business and operational events
WARNING
Unexpected condition that did not stop processing
ERROR
Operation failed
CRITICAL
Major platform or security failure

		

Do not use ERROR for ordinary validation failures.

Example:

			
User entered an invalid email:
INFO or no log
Database unavailable:
ERROR
All database replicas unavailable:
CRITICAL

		

Development and Production Logging

Development logs can be more readable.

Production logs should be machine-readable.

			
Development:
Colored human-readable output
Production:
JSON to standard output

Container platforms can collect standard output centrally.

Avoid writing logs only to local container files because containers may be replaced at any time.

Centralized Log Collection

A centralized platform allows operators to query logs from:

API instances
Background workers
Schedulers
Ingestion services
AI processing workers
Webhook processors

Architecture:

			
API Containers
Worker Containers
Scheduler
Webhook Service
       |
       v
Log Collector
       |
       v
Central Log Storage
       |
       v
Search and Dashboards

		

Possible technology categories include:

			
Managed cloud logging
OpenSearch or Elasticsearch
Grafana Loki
Observability SaaS platform

The architecture should not depend heavily on a single vendor.

Log Retention

Not all logs need permanent storage.

Example policy:

			
Debug logs:
7 days
Application logs:
30–90 days
Security logs:
12 months
Audit events:
According to contractual and regulatory requirements

		

Longer retention increases cost and privacy exposure.

Retention should match actual operational and compliance needs.

Metrics Architecture

			
Application
      |
      v
Metrics Endpoint or Collector
      |
      v
Time-Series Database
      |
      v
Dashboards and Alerts

		

Metrics should have:

Stable names
Clear units
Low-cardinality labels
Documented meanings

The Cardinality Problem

Metric labels create separate time series.

Bad label:

tender_id

A unique tender ID may create millions of series.

Good labels:

			
environment
service
endpoint_group
status_class
provider
model_family
job_type

		

High-cardinality identifiers belong in logs and traces, not metric labels.

Core API Metrics

Track:

			
HTTP request count
HTTP error count
Request duration
Active requests
Request body size
Response body size
Rate-limit events

		

Example metric names:

			
http_requests_total
http_request_duration_seconds
http_requests_in_progress
http_response_errors_total

Metrics Module

Create:

app/observability/metrics.py

Example:

			
from prometheus_client import (
    Counter,
    Histogram,
)
HTTP_REQUESTS_TOTAL = Counter(
    "http_requests_total",
    "Total HTTP requests",
    [
        "method",
        "route",
        "status_class",
    ],
)
HTTP_REQUEST_DURATION = Histogram(
    "http_request_duration_seconds",
    "HTTP request duration",
    [
        "method",
        "route",
    ],
)

		

Use route templates such as:

/saved-opportunities/{opportunity_id}

rather than actual paths containing unique identifiers.

Metrics Middleware

			
class MetricsMiddleware(
    BaseHTTPMiddleware
):
    async def dispatch(
        self,
        request: Request,
        call_next,
    ):
        started_at = time.perf_counter()
        response = await call_next(request)
        route = request.scope.get("route")
        route_path = (
            route.path
            if route
            else "unmatched"
        )
        status_class = (
            f"{response.status_code // 100}xx"
        )
        HTTP_REQUESTS_TOTAL.labels(
            method=request.method,
            route=route_path,
            status_class=status_class,
        ).inc()
        HTTP_REQUEST_DURATION.labels(
            method=request.method,
            route=route_path,
        ).observe(
            time.perf_counter() - started_at
        )
        return response

		

Database Metrics

Monitor:

			
Connection-pool usage
Active connections
Waiting connections
Query duration
Transaction duration
Deadlocks
Rollback count
Database size
Replication delay

		

A healthy API can still fail when the database pool is exhausted.

Slow Query Logging

Define a threshold.

Example:

			
Queries slower than 500 ms:
Warning
Queries slower than 2 seconds:
Error or performance event

Log:

			
{
  "event": "slow_database_query",
  "duration_ms": 1842,
  "query_name": "list_ranked_opportunities",
  "organization_id": 10
}

		

Do not log full SQL statements containing sensitive parameter values by default.

Background-Job Metrics

Track:

			
Jobs queued
Jobs started
Jobs completed
Jobs failed
Job duration
Queue depth
Queue waiting time
Retry count
Dead-letter count

		

Example metrics:

			
background_jobs_total
background_job_duration_seconds
background_job_retries_total
background_queue_depth
background_job_wait_seconds

		

Job Context

Every background job should include:

			
job_id
correlation_id
organization_id
job_type
attempt_number
queued_at

		

Example payload:

			
{
  "job_id": "job_384a...",
  "correlation_id": "req_7e3b...",
  "organization_id": 10,
  "job_type": "tender_analysis",
  "tender_id": 551,
  "attempt_number": 1
}

		

This context should appear in logs and traces.

Queue-Latency Monitoring

Job processing time is not the same as total waiting time.

Track:

			
queued_at
started_at
completed_at

Then calculate:

			
Queue wait:
started_at - queued_at
Processing duration:
completed_at - started_at
End-to-end duration:
completed_at - queued_at

		

A worker can be fast while users still wait because the queue is overloaded.

Tender Ingestion Metrics

Track:

			
Sources polled
Source requests
Source failures
Tenders discovered
New tenders stored
Duplicate tenders skipped
Documents downloaded
Parsing failures
Last successful source run

		

Example:

			
tender_source_poll_total
tender_source_errors_total
tenders_discovered_total
tenders_ingested_total
tender_duplicates_total

		

Source Freshness

A tender source can fail silently while the rest of the application remains healthy.

Track:

Last successful ingestion timestamp

Alert when:

No successful ingestion for expected interval

Example:

			
Source normally updates hourly
Alert if no success for three hours

The threshold should match each source’s normal schedule.

AI Monitoring

AI processing requires dedicated metrics.

Track:

			
Requests per provider
Requests per model family
Success rate
Failure rate
Timeouts
Rate-limit responses
Input tokens
Output tokens
Latency
Estimated cost
Cache hit rate
Fallback usage

		

AI Metrics

Example names:

			
ai_requests_total
ai_request_duration_seconds
ai_request_failures_total
ai_input_tokens_total
ai_output_tokens_total
ai_estimated_cost_minor_total
ai_cache_hits_total
ai_provider_fallback_total

		

Use controlled labels:

			
operation
provider
model_family
status

Avoid organization_id as a general metric label if there may be thousands of organizations.

Organization-specific analysis can use database records or logs.

AI Reliability Dashboard

A dashboard could show:

			
AI Requests per Minute
Success Rate
P50 Latency
P95 Latency
P99 Latency
Provider Error Rate
Fallback Rate
Cache Hit Rate
Estimated Hourly Cost

		

This helps identify whether AI-provider problems are affecting users.

Detecting AI Quality Problems

Operational success does not guarantee useful output.

An AI request may return 200 OK but produce:

Invalid JSON
Missing fields
Unsupported claims
Low-confidence analysis
Empty summaries
Hallucinated deadlines

Track application-level quality metrics:

			
Schema-validation failures
Output-repair attempts
Low-confidence results
User correction rate
Regeneration requests
Human-review rejection rate

		

Billing Monitoring

Billing workflows require strong visibility.

Track:

			
Checkout sessions created
Checkout completion rate
Webhook events received
Webhook failures
Duplicate webhook events
Subscription sync duration
Payment failures
Past-due subscriptions
Trial conversions

		

A delayed webhook can incorrectly restrict a paying customer.

Webhook Processing Metrics

Example:

			
billing_webhook_events_total
billing_webhook_processing_seconds
billing_webhook_failures_total
billing_webhook_retries_total
billing_webhook_backlog

		

Alert when unprocessed billing events remain above a safe threshold.

Email Delivery Monitoring

Track:

			
Emails queued
Emails sent
Emails failed
Retry count
Delivery-provider latency
Bounce events
Complaint events

		

Separate:

			
Technical delivery success
Customer inbox delivery
User engagement

Sending an email successfully does not guarantee that it reached the inbox.

Slack and Teams Monitoring

Track:

			
Messages queued
Messages delivered
Webhook failures
Rate limits
Disabled integrations
Invalid credentials
Average delivery latency

		

Repeated authentication failures may indicate that a customer revoked an integration.

Authentication Metrics

Track:

			
Login attempts
Login successes
Login failures
Refresh failures
Session revocations
Invitation acceptance failures
Password reset requests
Rate-limit events

		

Security-sensitive metrics should be monitored without storing passwords or token values.

Tenant-Aware Operational Analysis

Operators may need to understand whether one organization is causing unusual load.

Useful database-backed reports include:

			
Requests per organization
AI cost per organization
Failed jobs per organization
Storage usage per organization
Notification volume per organization

		

Avoid exposing other tenants’ information to customer-facing dashboards.

Internal operational access should be tightly restricted and audited.

Distributed Tracing

Distributed tracing follows one operation through multiple components.

Example:

			
POST /tenders/{id}/analyze
        |
        +---- Authenticate user
        |
        +---- Check entitlement
        |
        +---- Reserve usage
        |
        +---- Queue AI job
                     |
                     +---- Load document
                     |
                     +---- Check cache
                     |
                     +---- Call AI provider
                     |
                     +---- Validate output
                     |
                     +---- Save analysis
                     |
                     +---- Confirm usage

		

Each step becomes a span.

Trace Structure

			
Trace
├── HTTP request span
├── Authentication span
├── Database query span
├── Entitlement check span
├── Queue publish span
├── Worker processing span
├── AI-provider span
└── Database write span

		

A trace exposes which span consumed the most time.

Tracing Module

Create:

app/observability/tracing.py

A provider-neutral implementation can use an open telemetry standard.

Example initialization:

			
def configure_tracing(
    *,
    service_name: str,
    environment: str,
    exporter_endpoint: str,
):
    resource = Resource.create(
        {
            "service.name": service_name,
            "deployment.environment":
                environment,
        }
    )
    provider = TracerProvider(
        resource=resource
    )
    processor = BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=exporter_endpoint
        )
    )
    provider.add_span_processor(
        processor
    )
    trace.set_tracer_provider(provider)

		

The exact instrumentation packages depend on the selected observability stack.

Manual Spans

Automatic instrumentation covers many framework operations.

Business workflows still need custom spans.

			
tracer = trace.get_tracer(__name__)
def analyze_tender(
    tender_id: int,
    organization_id: int,
):
    with tracer.start_as_current_span(
        "tender_analysis"
    ) as span:
        span.set_attribute(
            "tender.id",
            tender_id,
        )
        span.set_attribute(
            "organization.id",
            organization_id,
        )
        return orchestrator.analyze(
            tender_id=tender_id,
            organization_id=organization_id,
        )

		

Be cautious when placing high-cardinality attributes in systems that charge based on indexed fields.

Trace Sampling

Recording every trace may be expensive at high traffic volumes.

Use sampling.

Example strategy:

			
100% of errors
100% of slow requests
100% of billing workflows
100% of security-sensitive workflows
10% of ordinary successful requests

		

Adaptive sampling can preserve important traces without excessive cost.

Error Tracking

Error tracking should capture unhandled exceptions automatically.

Useful context:

			
Exception type
Stack trace
Release version
Environment
Request path
Correlation ID
Trace ID
Organization ID
User ID
Job type

		

Sensitive fields must be removed before transmission.

Expected Versus Unexpected Errors

Not every error response belongs in an error tracker.

Expected:

			
Invalid password
Usage limit reached
Permission denied
Tender not found

Unexpected:

			
Database connection dropped
Attribute error
Unexpected provider response
Transaction corruption
Uncaught serialization failure

		

Capture unexpected exceptions automatically.

Record expected business failures as structured events or metrics.

Global Exception Handler

Create a FastAPI exception handler.

			
from fastapi import Request
from fastapi.responses import JSONResponse
@app.exception_handler(Exception)
async def unhandled_exception_handler(
    request: Request,
    exception: Exception,
):
    logger.exception(
        "unhandled_application_exception",
        extra={
            "request_method":
                request.method,
            "request_path":
                request.url.path,
        },
    )
    error_tracker.capture_exception(
        exception
    )
    return JSONResponse(
        status_code=500,
        content={
            "detail": {
                "code":
                    "internal_server_error",
                "message":
                    "An unexpected error occurred.",
                "correlation_id":
                    correlation_id_context.get(),
            }
        },
    )

		

Never expose stack traces to production clients.

Release Tracking

Every log, error, metric deployment marker, and trace should identify the release.

Example:

release_version = git commit SHA

This allows operators to answer:

Did the error rate increase after deployment 8f6b2c1?

Release markers should appear on dashboards.

Health Endpoints

The application needs separate endpoints for different purposes.

Recommended:

			
GET /health/live
GET /health/ready
GET /health/details

These endpoints should not all expose the same information.

Liveness Endpoint

Liveness answers:

Is the application process alive?

Example:

			
@router.get("/health/live")
def liveness():
    return {
        "status": "alive",
    }

		

It should be lightweight and should not depend on external services.

A temporary database failure should not necessarily cause the process to restart repeatedly.

Readiness Endpoint

Readiness answers:

Can this instance safely receive traffic?

Check critical dependencies:

Database connectivity
Required configuration
Migration compatibility
Queue availability, if mandatory

Example:

			
@router.get("/health/ready")
def readiness(
    db: Session = Depends(get_db),
):
    checks = {
        "database": check_database(db),
        "configuration":
            check_required_configuration(),
    }
    ready = all(
        item["status"] == "ok"
        for item in checks.values()
    )
    return JSONResponse(
        status_code=200 if ready else 503,
        content={
            "status":
                "ready"
                if ready
                else "not_ready",
            "checks":
                checks,
        },
    )

		

Detailed Health Endpoint

A detailed endpoint can show:

			
Database
Queue
Object storage
AI provider
Email provider
Billing provider
Tender sources

		

This endpoint should be restricted.

It may reveal infrastructure details that should not be publicly available.

Dependency Health

Classify dependencies.

			
Critical:
Database
Authentication configuration
Primary queue
Degraded-mode:
Secondary AI provider
Email provider
Slack delivery
Optional tender source

		

Failure of an optional dependency should not always mark the entire API as unavailable.

Readiness and Database Migrations

A new application version may require a newer database schema.

During startup, verify that the expected migration version is present.

			
Application release
expects
database revision X

A mismatch should prevent the incompatible instance from accepting traffic.

Synthetic Monitoring

Synthetic monitoring runs automated checks from outside the application.

Examples:

			
Open login page
Call public API endpoint
Perform test authentication
Run a safe tender-search request
Verify response time
Verify expected page content

		

This confirms that the platform works from a customer’s perspective.

Server metrics alone cannot detect every frontend, DNS, certificate, or routing failure.

Uptime Checks

Monitor:

			
Homepage
API liveness
API readiness
Authentication endpoint
Critical customer workflow

		

Checks should run from more than one geographic region where practical.

This helps distinguish local network failures from platform-wide outages.

Frontend Observability

The React frontend also needs monitoring.

Track:

			
Page-load performance
JavaScript errors
Failed API requests
Route-transition failures
Authentication-refresh failures
User-visible rendering errors

		

Frontend problems may not appear in backend logs.

React Error Boundary

Create:

src/observability/AppErrorBoundary.jsx

Example:

			
import React from "react";
export class AppErrorBoundary
  extends React.Component {
  constructor(props) {
    super(props);
    this.state = {
      hasError: false,
    };
  }
  static getDerivedStateFromError() {
    return {
      hasError: true,
    };
  }
  componentDidCatch(error, errorInfo) {
    errorTracker.captureException(
      error,
      {
        extra: errorInfo,
      }
    );
  }
  render() {
    if (this.state.hasError) {
      return (
        <div>
          <h1>
            Something went wrong
          </h1>
          <p>
            Refresh the page or contact
            support if the problem continues.
          </p>
        </div>
      );
    }
    return this.props.children;
  }
}

		

Do not display technical stack traces to users.

Frontend Request Correlation

The browser can generate or reuse correlation IDs for API requests.

			
api.interceptors.request.use(config => {
  config.headers[
    "X-Correlation-ID"
  ] = createCorrelationId();
  return config;
});

		

For workflows involving several requests, reuse one workflow identifier where appropriate.

Browser Performance Metrics

Useful measurements include:

			
Largest Contentful Paint
Interaction latency
Cumulative layout shift
Initial bundle load time
API request duration
Frontend route load time

		

Performance targets should reflect actual user workflows rather than vanity metrics.

Service-Level Indicators

A Service-Level Indicator, or SLI, measures service behavior.

Examples:

			
Availability
Latency
Error rate
Data freshness
Job completion rate
Notification delivery rate

		

An SLI must be measurable.

Service-Level Objectives

A Service-Level Objective, or SLO, defines a target for an SLI.

Examples:

			
API availability:
99.9% per month
API latency:
95% of requests below 500 ms
Tender ingestion:
95% of scheduled source runs succeed
AI analysis:
95% complete within 5 minutes
Email notifications:
99% queued within 2 minutes

		

These targets should be realistic and aligned with customer expectations.

Example API Availability SLO

Define successful requests:

			
HTTP responses
excluding
expected 4xx client errors

Availability:

			
Successful eligible requests
divided by
all eligible requests

Do not count customer validation mistakes as platform outages.

Error Budgets

An error budget is the amount of unreliability allowed by an SLO.

Example:

			
Monthly availability target:
99.9%
Allowed unavailability:
Approximately 43 minutes in a 30-day month

When the error budget is exhausted:

Slow feature releases
Prioritize reliability
Investigate recurring failures
Improve tests and architecture

Error budgets help balance product speed and operational stability.

Alerting Philosophy

An alert should indicate that human action is needed.

Bad alert:

One request returned 500

Better alert:

API 5xx error rate exceeded 5% for 10 minutes

Individual failures belong in logs and error tracking.

Alerts should focus on sustained or severe conditions.

Alert Severity Levels

Define a consistent severity model.

			
SEV-1
Platform unavailable or major security incident
SEV-2
Critical customer workflow seriously degraded
SEV-3
Limited functionality or rising operational risk
SEV-4
Non-urgent issue requiring planned investigation

		

Each severity level should define response expectations.

SEV-1 Examples

			
All API traffic failing
Database unavailable
Cross-tenant data exposure suspected
Authentication system unavailable
Billing incorrectly charging many customers

		

These require immediate response.

SEV-2 Examples

			
AI analysis failing for most customers
Tender ingestion stopped across major sources
Large webhook backlog
Email delivery unavailable
Subscription access incorrectly restricted

		

SEV-3 Examples

			
One provider degraded but failover works
A non-critical source has stopped updating
Queue latency is increasing
Cache hit rate has dropped significantly

Alert Routing

Different alerts should reach different owners.

			
API and database alerts
Engineering operations
AI-provider alerts
AI platform owner
Billing alerts
Finance and engineering
Security alerts
Security owner
Tender-source alerts
Data ingestion owner

		

Every alert needs:

Owner
Severity
Runbook
Escalation path

Alert Channels

Possible channels:

			
On-call notification platform
SMS
Phone
Email
Slack or Teams incident channel

		

Critical alerts should not rely solely on email.

Alert Deduplication

A single database outage may cause:

API errors
Worker failures
Billing failures
Notification failures

Without grouping, operators may receive hundreds of alerts.

Use:

Deduplication keys
Alert grouping
Dependency-aware suppression
Cooldown windows

The primary root-cause alert should remain visible.

Example Alert Rules

API Availability

			
Condition:
5xx rate exceeds 5%
Duration:
10 minutes
Severity:
SEV-2

		

Database Connections

			
Condition:
Connection pool usage above 90%
Duration:
5 minutes
Severity:
SEV-2

		

Queue Backlog

			
Condition:
Oldest critical job waiting over 10 minutes
Duration:
5 minutes
Severity:
SEV-2

		

AI Provider

			
Condition:
Primary-provider error rate above 20%
Duration:
5 minutes
Severity:
SEV-3 if failover works
SEV-2 if failover fails

		

Dead-Letter Queue Monitoring

Jobs that exceed retry limits should move to a dead-letter queue.

Track:

			
Dead-letter job count
Job type
Oldest job age
Failure reason

Alert when critical jobs enter the queue.

Do not silently discard failed jobs.

Operational Event Model

Some events deserve durable application storage in addition to logs.

Create:

app/models/operational_event.py

Fields:

			
id
organization_id
event_type
severity
service_name
resource_type
resource_id
correlation_id
event_metadata
occurred_at
resolved_at

		

Use this selectively for:

Persistent processing failures
Customer-visible degraded operations
Manual intervention requirements
Reconciliation failures

Do not duplicate every log entry in the database.

Operational Event Status

			
class OperationalEventStatus(
    str,
    Enum,
):
    OPEN = "open"
    ACKNOWLEDGED = "acknowledged"
    RESOLVED = "resolved"
    IGNORED = "ignored"

		

This supports an internal operations dashboard.

Internal Operations Dashboard

The internal dashboard can display:

			
Platform Status
Open Incidents
Failed Background Jobs
Dead-Letter Queue
Tender Source Freshness
AI Provider Health
Billing Webhook Backlog
Email Delivery Status
Recent Deployments

		

This dashboard must not be accessible to ordinary organization users.

Customer-Facing Status Information

Customers may benefit from a limited status view.

Examples:

			
Tender analysis delayed
Email notifications degraded
Scheduled maintenance
Resolved incident

Avoid exposing:

Internal hostnames
Provider credentials
Customer identities
Detailed attack information
Sensitive architecture details

Incident Model

Create:

incidents

Recommended fields:

			
id
title
status
severity
summary
started_at
detected_at
acknowledged_at
resolved_at
incident_commander_user_id
customer_impact
root_cause
resolution
created_at
updated_at

		

Incident Lifecycle

			
Detected
    |
    v
Acknowledged
    |
    v
Investigating
    |
    v
Mitigated
    |
    v
Resolved
    |
    v
Post-Incident Review

		

The exact states can be stored in an enum.

Incident Response Workflow

A basic workflow:

			
Confirm the alert
Assign severity
Identify incident owner
Assess customer impact
Mitigate immediate damage
Communicate status
Resolve root cause
Verify recovery
Write post-incident review
Track follow-up actions

		

The priority during an incident is restoring safe service, not proving the root cause immediately.

Runbooks

Every actionable alert should link to a runbook.

A runbook should contain:

			
Alert meaning
Possible causes
Diagnostic queries
Relevant dashboards
Safe mitigation steps
Rollback instructions
Escalation contacts
Verification steps

		

Example runbook:

			
Alert:
AI provider error rate high
Check:
Provider status
Rate-limit logs
Credential validity
Fallback status
Mitigate:
Route traffic to secondary provider
Reduce concurrency
Pause non-critical batch jobs

		

Deployment Monitoring

Every deployment should create an observability event.

Track:

			
Release version
Deployment start
Deployment end
Environment
Changed services
Migration version
Deployment result

		

Dashboard annotations help correlate changes with incidents.

Canary Releases

For high-risk changes:

			
Deploy to small percentage
↓
Observe metrics
↓
Continue or rollback

		

Monitor:

Error rate
Latency
AI-output validation
Database load
Queue failures

Do not promote the release automatically when key indicators regress.

Rollback Signals

Automatic or manual rollback may be triggered by:

			
Error rate increase
Readiness failures
Severe latency regression
Database migration incompatibility
Authentication failure spike
Cross-tenant security test failure

		

Some database migrations cannot be rolled back safely and require forward fixes.

Capacity Monitoring

Track resource saturation.

			
CPU usage
Memory usage
Disk usage
Database storage
Database connections
Queue depth
Worker concurrency
Object-storage growth
Network throughput

		

Utilization alone is insufficient.

Track whether resource limits are approaching customer-impact thresholds.

Capacity Forecasting

Use historical trends to estimate:

			
When database storage will reach 80%
When queue workers need scaling
When AI budgets need adjustment
When log storage costs will increase
When object storage requires lifecycle cleanup

		

Capacity planning should happen before an outage.

Autoscaling Signals

Possible scaling signals:

			
API CPU
Active requests
Request latency
Queue depth
Oldest job age
Worker utilization

		

For background workers, queue waiting time may be more useful than CPU alone.

Business Metrics and Operational Metrics

Do not confuse business analytics with system observability.

Business metrics:

			
New organizations
Trial conversions
Tenders saved
Subscriptions upgraded

Operational metrics:

			
API errors
Job failures
Queue latency
Database saturation

They can appear on related dashboards but serve different purposes.

Domain-Specific Success Metrics

For the Tender Opportunity Scanner, track whether the product is functioning end-to-end.

Examples:

			
Tenders discovered per source
Successful tender analyses
Matching jobs completed
High-score opportunities generated
Deadline notifications sent
Saved opportunities created

		

A technically healthy platform that discovers zero tenders is not delivering value.

Data Freshness Dashboard

Display:

			
Source
Last successful poll
Last tender discovered
Expected frequency
Current status
Failure count

		

Example:

			
TED Europe
Last poll:
12 minutes ago
Expected:
Hourly
Status:
Healthy

		

End-to-End Workflow Metrics

Track complete workflows rather than only individual services.

Example tender-processing latency:

			
Tender discovered
↓
Document downloaded
↓
Content parsed
↓
AI analysis completed
↓
Match generated
↓
Customer notified

		

Measure the total time from discovery to customer notification.

This is often more meaningful than isolated API latency.

Trace and Log Linking

Logs should include:

			
trace_id
span_id

The log platform can then link an error log to the related trace.

The trace can link to:

Database spans
Provider calls
Queue processing
Worker retries

This reduces investigation time.

Metrics Exemplars

Where supported, a metric spike can link to a representative trace.

Example:

			
P99 latency increased
↓
Open example trace
↓
AI-provider call took 18 seconds

		

This connects high-level monitoring to detailed diagnostics.

Audit Logs Versus Application Logs

Audit logs and operational logs serve different purposes.

Application logs:

Help debug and operate the platform

Audit logs:

Record who performed important actions

Audit events should include:

Actor
Organization
Action
Target
Timestamp
Result

They often require stronger retention and tamper protection.

Security Monitoring

Monitor for:

			
Repeated login failures
Suspicious token refreshes
Invitation abuse
Permission-denied spikes
Cross-tenant access attempts
Unusual exports
Unexpected billing changes
API-key misuse
Webhook signature failures

		

A single denied request may be harmless.

Patterns over time may indicate abuse.

Cross-Tenant Access Alerts

Tenant-isolation failures are critical.

Log every attempted cross-tenant access only when it can be distinguished from an ordinary missing record.

Possible event:

			
{
  "event": "cross_tenant_access_blocked",
  "severity": "critical",
  "actor_organization_id": 10,
  "target_organization_id": 25,
  "resource_type": "saved_opportunity",
  "resource_id": 551,
  "correlation_id": "req_..."
}

		

Be careful not to reveal the target organization to the requester.

Internally, this may require security investigation.

Privacy and Observability

Observability systems process potentially sensitive metadata.

Apply:

			
Data minimization
Purpose limitation
Access control
Retention limits
Encryption
Audit logging
Regional storage requirements

		

Observability platforms should not become uncontrolled copies of customer data.

Access Control for Operational Tools

Operational dashboards may contain:

Organization identifiers
User identifiers
System architecture
Billing status
Failure details

Restrict access using:

			
Single sign-on
Multi-factor authentication
Least privilege
Role separation
Access reviews
Audit logging

		

Do not share one administrator account across the team.

Environment Separation

Keep observability data separated by environment.

			
Development
Testing
Staging
Production

Every signal should include:

deployment.environment

A staging error must not trigger a production incident.

Local Development Observability

Developers should be able to test observability locally.

Possible local stack:

			
FastAPI
Background Worker
Metrics Collector
Dashboard
Trace Viewer
Log Viewer

		

A minimal local environment can be started through Docker Compose.

Docker Compose Example

			
services:
  api:
    build: ./backend
    environment:
      ENVIRONMENT: development
      LOG_FORMAT: json
      METRICS_ENABLED: "true"
      TRACING_ENABLED: "true"
  metrics:
    image: prom/prometheus
    volumes:
      - ./observability/prometheus.yml:
          /etc/prometheus/prometheus.yml
  dashboard:
    image: grafana/grafana
    ports:
      - "3000:3000"

		

Exact services depend on the selected observability tools.

Application Configuration

Add:

			
ENVIRONMENT=development
SERVICE_NAME=tender-api
RELEASE_VERSION=local
LOG_LEVEL=INFO
LOG_FORMAT=json
METRICS_ENABLED=true
METRICS_PATH=/internal/metrics
TRACING_ENABLED=true
OTEL_EXPORTER_ENDPOINT=http://collector:4317
ERROR_TRACKING_ENABLED=true
ERROR_TRACKING_DSN=...

		

Never commit production credentials.

Configuration Model

			
class ObservabilitySettings(
    BaseSettings
):
    environment: str = "development"
    service_name: str = "tender-api"
    release_version: str = "local"
    log_level: str = "INFO"
    log_format: str = "json"
    metrics_enabled: bool = True
    metrics_path: str = (
        "/internal/metrics"
    )
    tracing_enabled: bool = False
    otel_exporter_endpoint: (
        str | None
    ) = None
    error_tracking_enabled: bool = False
    error_tracking_dsn: str | None = None

		

Validate required settings at startup.

Protecting the Metrics Endpoint

The metrics endpoint can expose:

Route names
Service behavior
Infrastructure state
Error rates

Do not expose it publicly without protection.

Possible strategies:

			
Private network only
Internal authentication
Collector scraping from trusted subnet
Service mesh policy

Application Startup

Initialize observability before most application services.

			
settings = load_settings()
configure_logging(
    service_name=settings.service_name,
    environment=settings.environment,
    release_version=(
        settings.release_version
    ),
    log_level=settings.log_level,
)
configure_tracing(
    service_name=settings.service_name,
    environment=settings.environment,
    exporter_endpoint=(
        settings.otel_exporter_endpoint
    ),
)
configure_error_tracking(
    dsn=settings.error_tracking_dsn,
    environment=settings.environment,
    release_version=(
        settings.release_version
    ),
)

		

Startup failures should be logged clearly before the application exits.

Operational Dashboards

Create several focused dashboards instead of one overloaded screen.

Recommended dashboards:

			
Platform Overview
API Performance
Database Health
Background Jobs
Tender Ingestion
AI Operations
Notifications
Billing and Webhooks
Security
Business Health

		

Platform Overview Dashboard

Include:

			
Current availability
Request volume
Error rate
P95 latency
Open incidents
Queue backlog
Database health
AI-provider status
Last deployment

		

This should answer:

Is the platform healthy right now?

API Performance Dashboard

Display:

			
Requests per second
Response status distribution
P50 latency
P95 latency
P99 latency
Slowest routes
Rate-limit events
Authentication failures

		

Use route templates rather than raw URLs.

Background Jobs Dashboard

Display:

			
Queue depth
Oldest waiting job
Jobs completed per minute
Failure rate
Retry rate
Dead-letter count
Processing duration by job type
Worker availability

		

AI Operations Dashboard

Display:

			
Requests by operation
Requests by provider
Model-family distribution
Success rate
P95 latency
Input and output tokens
Estimated cost
Cache hit rate
Fallback rate
Output-validation failures

		

Billing Operations Dashboard

Display:

			
Webhook events received
Webhook processing failures
Unprocessed event backlog
Checkout completion rate
Payment failures
Subscription synchronization delay
Trial conversions
Past-due organizations

		

Dashboard Time Windows

Support:

			
Last 15 minutes
Last hour
Last 24 hours
Last 7 days
Custom range

		

Operational dashboards should default to a recent window.

Business dashboards may use longer periods.

Alert Testing

Alerts should be tested intentionally.

Examples:

			
Generate controlled API errors
Pause one worker
Block a test dependency
Insert a dead-letter test job
Send an invalid webhook signature
Trigger a staging payment failure

		

Do not wait for a production incident to discover that alerts are misconfigured.

Preventing Alert Fatigue

Alert fatigue occurs when too many low-value notifications are generated.

Reduce it by:

Raising thresholds appropriately
Requiring sustained conditions
Grouping related alerts
Removing non-actionable alerts
Assigning clear ownership
Reviewing noisy alerts after incidents

An ignored alert is effectively no alert.

Operational Reviews

Run a regular operational review covering:

			
Incidents
SLO performance
Error-budget consumption
Noisy alerts
Slow queries
Capacity trends
AI-provider reliability
Billing failures
Security anomalies
Upcoming risks

		

This turns observability data into engineering improvements.

Post-Incident Review

A post-incident review should include:

			
Incident summary
Customer impact
Timeline
Detection method
Root cause
Contributing factors
What worked
What failed
Corrective actions
Owners and deadlines

		

The purpose is system improvement, not blame.

Operational Data Retention

Example retention matrix:

Data Type	Example Retention
High-volume metrics	30–90 days
Aggregated metrics	12–24 months
Application logs	30–90 days
Security logs	12 months
Traces	7–30 days
Error groups	6–12 months
Audit events	Contractual or legal requirement
Incident reports	Long-term

Actual retention depends on cost, contracts, and legal requirements.

Testing the Observability Layer

Create:

			
tests/
└── observability/
    ├── test_logging_context.py
    ├── test_redaction.py
    ├── test_correlation_middleware.py
    ├── test_metrics_middleware.py
    ├── test_health_endpoints.py
    ├── test_error_handler.py
    ├── test_job_instrumentation.py
    └── test_sensitive_data_exclusion.py

		

Correlation Middleware Test

			
def test_response_contains_correlation_id(
    client,
):
    response = client.get(
        "/health/live"
    )
    assert response.status_code == 200
    assert (
        "X-Correlation-ID"
        in response.headers
    )

		

Incoming Correlation ID Test

			
def test_valid_incoming_correlation_id_is_reused(
    client,
):
    response = client.get(
        "/health/live",
        headers={
            "X-Correlation-ID":
                "req_test123"
        },
    )
    assert (
        response.headers[
            "X-Correlation-ID"
        ]
        == "req_test123"
    )

		

Redaction Test

			
def test_sensitive_fields_are_redacted():
    payload = {
        "email": "user@example.com",
        "password": "secret-value",
        "access_token": "token-value",
    }
    result = redact_mapping(payload)
    assert result["email"] == (
        "user@example.com"
    )
    assert result["password"] == (
        "[REDACTED]"
    )
    assert result["access_token"] == (
        "[REDACTED]"
    )

		

Health Endpoint Tests

Test:

			
Liveness returns 200
Readiness returns 200 when dependencies work
Readiness returns 503 when database is unavailable
Detailed health requires authorization
Sensitive configuration is not returned

		

Metrics Tests

Test:

Request counter increases
Status classes are labeled correctly
Route templates are used
Duration histogram records requests
Unmatched routes are handled
Organization IDs are not added as high-cardinality labels

Background Job Instrumentation Test

			
def test_failed_job_records_metrics_and_logs(
    job_runner,
    metrics_registry,
    captured_logs,
):
    with pytest.raises(
        TemporaryProcessingError
    ):
        job_runner.run(
            job_type="tender_analysis",
            job_id="job_test",
            organization_id=10,
        )
    assert metrics_registry.counter_value(
        "background_jobs_total",
        labels={
            "job_type":
                "tender_analysis",
            "status":
                "failed",
        },
    ) == 1
    assert any(
        log["event"]
        == "background_job_failed"
        for log in captured_logs
    )

		

Sensitive Data Exclusion Tests

Test that logs and error events never contain:

			
Passwords
Authentication tokens
API keys
Billing secrets
Raw tender content
Private internal notes
Full request bodies

		

This should be part of automated security testing.

Load Testing

Observability must remain useful under load.

Run tests that generate:

High API traffic
Many concurrent AI jobs
Large queue backlogs
Database saturation
Provider timeouts

Verify:

			
Metrics remain accurate
Logs are not dropped unexpectedly
Tracing overhead is acceptable
Alerts trigger correctly
Application latency remains within target

		

Observability Overhead

Instrumentation consumes resources.

Measure:

			
CPU overhead
Memory overhead
Network traffic
Log volume
Trace volume
Metric cardinality
Storage cost

		

The observability system should not become the cause of an outage.

Failure of the Observability Platform

The application should continue operating when the external observability service is temporarily unavailable.

Use:

			
Buffered exporters
Batch sending
Short timeouts
Non-blocking log delivery
Graceful failure

		

Do not make ordinary user requests wait for remote telemetry delivery.

End-to-End Operational Scenario

A production workflow might look like this:

			
Tender source poll begins
Correlation and job IDs are created
Source request succeeds
Twelve new tenders are discovered
Tender-analysis jobs enter the queue
Queue depth metric increases
Workers start processing
AI provider latency rises
Primary provider returns rate limits
Model router uses secondary provider
Fallback metric increases
One job exceeds retry limit
Job moves to dead-letter queue
Alert opens
Operator opens the runbook
Trace shows repeated provider timeouts
Provider routing is adjusted
Failed job is replayed
Metrics return to normal
Incident is resolved

		

Without observability, operators would only know that one tender analysis was missing.

With observability, the complete chain is visible.

Production Readiness Checklist

Before launch, verify:

			
✔ Structured JSON logging is enabled
✔ Correlation IDs connect requests and jobs
✔ Sensitive values are redacted
✔ Logs are collected centrally
✔ API metrics are available
✔ Database metrics are monitored
✔ Background queues are monitored
✔ Tender-source freshness is monitored
✔ AI providers are monitored
✔ Billing webhooks are monitored
✔ Email and collaboration delivery are monitored
✔ Distributed tracing is configured
✔ Unexpected exceptions reach error tracking
✔ Release versions are attached to telemetry
✔ Liveness and readiness checks are separate
✔ Detailed health information is restricted
✔ SLOs are defined
✔ Critical alerts have runbooks
✔ Alert ownership is documented
✔ Dead-letter queues are visible
✔ Incident response procedures exist
✔ Retention policies are configured
✔ Operational access is protected
✔ Observability failure does not block requests
✔ Alert rules have been tested

		

Current Architecture

			
Tender Sources
      |
      v
Ingestion Services
      |
      v
Tender Database
      |
      v
AI Orchestration
      |
      v
Matching and Opportunity Workflows
      |
      v
Customer Notifications
      |
      v
Billing and Usage Management
All Components
      |
      +---- Structured Logs
      +---- Metrics
      +---- Traces
      +---- Error Events
      |
      v
Observability Platform
      |
      +---- Dashboards
      +---- Alerts
      +---- Incidents
      +---- Runbooks

		

The application is no longer a black box.

Operators can understand its health, performance, cost, and customer impact.

Current Deliverables

At this stage, we have designed:

			
✔ Structured application logging
✔ Standard event naming
✔ Request correlation IDs
✔ Background-job context
✔ Sensitive-data redaction
✔ Centralized log collection
✔ Log-level guidelines
✔ Log-retention policies
✔ API performance metrics
✔ Database metrics
✔ Background-job metrics
✔ Queue-latency monitoring
✔ Tender-source freshness monitoring
✔ AI-provider monitoring
✔ Billing-webhook monitoring
✔ Email-delivery monitoring
✔ Slack and Teams monitoring
✔ Authentication metrics
✔ Distributed tracing
✔ Custom business spans
✔ Error tracking
✔ Release tracking
✔ Liveness endpoints
✔ Readiness endpoints
✔ Restricted detailed health checks
✔ Synthetic monitoring
✔ Frontend error monitoring
✔ Service-level indicators
✔ Service-level objectives
✔ Error budgets
✔ Alert severity levels
✔ Alert routing
✔ Incident response workflows
✔ Operational dashboards
✔ Security monitoring
✔ Capacity monitoring
✔ Runbooks
✔ Post-incident reviews
✔ Observability tests
✔ Production-readiness checklist

		

The Tender Opportunity Scanner now has the operational visibility required for production use.

Recommended Git Commit

			
git add .
git commit -m "Add production monitoring logging alerting and observability"

Push the changes:

git push origin main

What We Will Build Next

The platform is now observable and operationally manageable.

The next major step is making deployments repeatable, safe, and automated.

We will introduce:

Production Docker images
Environment-specific configuration
Continuous integration
Automated testing
Container security scanning
Database migration workflows
Continuous deployment
Staging environments
Deployment approvals
Canary releases
Rollback procedures
Infrastructure provisioning
Release versioning

The next article in the development series is:

“Building Docker-Based Deployment, CI/CD Pipelines, and Safe Release Automation for the Tender Opportunity Scanner SaaS.”

Conclusion

Production observability transforms the Tender Opportunity Scanner from a feature-complete application into an operable SaaS platform.

Structured logging gives every important event a consistent and searchable representation. Correlation IDs connect browser requests, API calls, queues, background workers, database operations, and AI-provider requests into one understandable workflow.

Metrics expose availability, latency, throughput, failures, resource saturation, ingestion freshness, AI performance, billing reliability, and notification delivery. Distributed tracing reveals where time is spent across complex workflows, while error tracking groups unexpected failures by environment and release.

Health checks, service-level objectives, error budgets, alert policies, dashboards, runbooks, and incident procedures ensure that operational data leads to effective action. At the same time, redaction, retention controls, and access restrictions prevent the observability platform from becoming a source of customer-data exposure.

With monitoring, logging, alerting, and observability in place, the platform can detect failures before customers report them, identify root causes faster, measure reliability objectively, and scale with greater confidence.

The next step is to automate how each new version of the application is tested, packaged, deployed, verified, and rolled back.

Building Production Monitoring, Logging, Alerting, and Observability for the Tender Opportunity Scanner SaaS

Introduction

Development Progress

What Is Observability?

Logs

Metrics

Traces

Errors

Observability Architecture

Core Observability Principles

Monitoring Versus Observability

Recommended Project Structure

Structured Logging

Logging Configuration

Standard Log Fields

Event Names

Correlation IDs

Correlation Context

Correlation Middleware

Validating Incoming Correlation IDs

Request Logging Middleware

Logging Authenticated Context

Sensitive Data Redaction

Avoid Logging Tender Content

Log Levels

Development and Production Logging

Centralized Log Collection

Log Retention

Metrics Architecture

The Cardinality Problem

Core API Metrics

Metrics Module

Metrics Middleware

Database Metrics

Slow Query Logging

Background-Job Metrics

Job Context

Queue-Latency Monitoring

Tender Ingestion Metrics

Source Freshness

AI Monitoring

AI Metrics

AI Reliability Dashboard

Detecting AI Quality Problems

Billing Monitoring

Webhook Processing Metrics

Email Delivery Monitoring

Slack and Teams Monitoring

Authentication Metrics

Tenant-Aware Operational Analysis

Distributed Tracing

Trace Structure

Tracing Module

Manual Spans

Trace Sampling

Error Tracking

Expected Versus Unexpected Errors

Global Exception Handler

Release Tracking

Health Endpoints

Liveness Endpoint

Readiness Endpoint

Detailed Health Endpoint

Dependency Health

Readiness and Database Migrations

Synthetic Monitoring

Uptime Checks

Frontend Observability

React Error Boundary

Frontend Request Correlation

Browser Performance Metrics

Service-Level Indicators

Service-Level Objectives

Example API Availability SLO

Error Budgets

Alerting Philosophy

Alert Severity Levels

SEV-1 Examples

SEV-2 Examples

SEV-3 Examples