PacketSafari

Audit Logging

How PacketSafari on-prem emits security audit logs, which settings control them, and how customers can ingest them.

PacketSafari on-prem emits two different log streams:

  • operational logs for normal application/runtime troubleshooting
  • security audit logs for login, logout, MFA, password reset, PCAP access, downloads, and onboarding changes

Default behavior

Fresh on-prem installs now seed these runtime env keys:

  • AUDIT_LOG_ENABLED=true
  • AUDIT_LOG_PERSIST=true
  • AUDIT_RETENTION_DAYS=365
  • AUDIT_FORWARDING_MODE=stdout_json
  • AUDIT_FORWARDER_TYPE=none

The default enterprise contract is stdout_json, which keeps ingestion simple for customer SIEM pipelines.

Forwarding modes

PacketSafari currently distinguishes between two audit delivery modes:

  • stdout_json
  • forwarder_profile

stdout_json

This is the default and the recommended baseline.

Behavior:

  • the backend emits audit events into container stdout
  • operators collect them with normal container log tooling such as docker compose logs, journald integration, or a host/container log agent
  • PacketSafari does not need to open a new outbound path just to deliver audit logs in this mode

Operationally, this means stdout_json does not require a new external destination allowlist entry in the PacketSafari egress proxy. You are reading logs locally from the deployment host or from the container runtime logging path.

forwarder_profile

This mode is intended for deployments where the on-prem bundle manages shipping audit events directly to a customer sink such as Splunk, Loki, syslog, or OpenSearch-compatible infrastructure.

Behavior:

  • PacketSafari is configured with a forwarder type and sink settings
  • the deployment is expected to ship audit logs to that sink instead of relying only on host-side collection
  • if that sink is outside the PacketSafari container network, it becomes an outbound destination and must be allowed by the on-prem egress controls

Egress allowlisting for managed forwarding

If you keep AUDIT_FORWARDING_MODE=stdout_json, no additional PacketSafari egress approval is required for audit delivery.

If you use AUDIT_FORWARDING_MODE=forwarder_profile and the sink is external to the PacketSafari container network, you must treat that sink like any other approved outbound destination.

Examples:

  • a Splunk HEC endpoint on another host
  • a Loki endpoint on another VM
  • a syslog collector on another subnet
  • an OpenSearch endpoint outside the PacketSafari service network

In those cases, the sink host must be present in the PacketSafari egress allowlist and proxy configuration, otherwise the runtime egress controls can block delivery.

Important distinction:

  • PacketSafari already ships a host approval helper for external AI hosts
  • this repo does not currently expose a dedicated one-command helper specifically for audit sink approval

So today the practical recommendation is:

  1. prefer stdout_json unless you explicitly need managed shipping from the bundle
  2. if you do need managed shipping, ensure the target sink host is included in your on-prem egress allowlist/proxy configuration as part of deployment setup
  3. verify delivery after rollout using both container logs and the destination system

For fixed external sinks, the clean operator flow is:

  1. add the sink hostname and port to configuration/egress-allowlist.yaml
  2. run python3 scripts/packetsafari_egress.py sync-config
  3. restart the proxy or stack if needed
  4. verify delivery in both PacketSafari logs and the destination sink

This is different from external AI host approval, which has its own approve-ai-host helper because those destinations are admin-configured at runtime.

For the underlying egress model, see Local Models and External AI Approval and the broader network control notes in documentation/security/network-egress-and-container-hardening.md.

Operator checklist: when you need one step vs both

Use this rule from the container's point of view:

  • if the audit sink is outside the PacketSafari container network, you need egress approval / allowlisting
  • if the audit sink uses a self-signed certificate or a private/internal CA, you need custom CA trust
  • many internal HTTPS audit sinks require both

Examples:

  • internal HTTP syslog collector on another host:
    • egress approval / allowlisting: yes
    • custom CA trust: no
  • internal HTTPS Splunk HEC with a private enterprise CA:
    • egress approval / allowlisting: yes
    • custom CA trust: yes
  • localhost or PacketSafari-internal service name with public-trust TLS:
    • egress approval / allowlisting: no
    • custom CA trust: usually no
  • external HTTPS audit sink with a public CA:
    • egress approval / allowlisting: yes
    • custom CA trust: usually no

Where to configure it

PacketSafari uses the same app-side onboarding model and runtime env for audit logging:

  • the installer seeds initial defaults into runtime.env
  • the web onboarding flow is the canonical place to review and finalize deployment config
  • the admin application settings page shows the effective audit posture after deployment

What is audited

Current security audit coverage includes:

  • login success and failure
  • account lockout
  • logout
  • MFA setup, enable, disable, and failed challenges
  • password reset requests and completions
  • PCAP upload start, completion, and failure
  • PCAP access grants and denials
  • completed PCAP downloads
  • on-prem onboarding completion

Event Catalog

The following event names are currently emitted by the backend audit pipeline for user/admin security and access flows.

Authentication and session events

  • auth.registration.completed
  • auth.login.succeeded
  • auth.login.failed
  • auth.login.locked_out
  • auth.login.blocked_concurrent_session
  • auth.logout
  • auth.session.revoked
  • auth.sessions.revoked_others
  • auth.password.changed
  • auth.password.reset_requested
  • auth.password.reset_completed
  • auth.mfa.setup_started
  • auth.mfa.enabled
  • auth.mfa.disabled
  • auth.mfa.challenge.failed

Admin user-management and security events

  • admin.user.created
  • admin.user.deleted
  • admin.user.enabled
  • admin.user.disabled
  • admin.user.subscription.changed
  • admin.user.entitlements.changed
  • admin.user.password_reset
  • admin.user.sessions.revoked
  • admin.user.session.revoked
  • admin.role.changed
  • admin.security_setting.changed

Capture access and deployment events

  • pcap.upload.started
  • pcap.upload.completed
  • pcap.upload.failed
  • pcap.access.viewed
  • pcap.access.denied
  • pcap.download.completed
  • deployment.onboarding.completed

Notes on deployment mode

  • Audit payloads include deployment_mode with onprem or saas.
  • Upload audit events include the actor username, source IP, filename, and the upload session ID and/or PCAP ID in the audit context.
  • Some events may be feature-gated by deployment mode.
  • auth.login.blocked_concurrent_session is primarily relevant for SaaS because concurrent-session enforcement is disabled on on-prem deployments by default.
  • Admin password resets, entitlement overrides, MFA changes, session revocation, and PCAP access/download events are relevant to both SaaS and on-prem deployments.

Customer ingestion

For enterprise customers, the recommended first step is to ingest container stdout:

docker compose logs -f backend

You can then route those JSON lines into Splunk, an Elasticsearch/OpenSearch audit sink, Loki, syslog, or another customer-managed collector. These are optional customer SIEM destinations; they are not PacketSafari application datastores.

If your deployment later adopts a managed forwarder profile, keep stdout_json as the baseline compatibility path.

For proxy and network-control observability, best practice is:

  • keep proxy access logs structured and metadata-only
  • keep firewall bypass logs structured and separate
  • ingest both into the same SIEM or correlation pipeline
  • avoid logging authorization headers, rewritten secrets, or request/response bodies

Forwarder profile requirements

If AUDIT_FORWARDING_MODE=forwarder_profile, PacketSafari renders a Vector config and starts the optional audit-forwarder service.

Required environment variables depend on the selected sink:

Splunk

  • AUDIT_FORWARDER_TYPE=splunk
  • SPLUNK_HEC_URL
  • SPLUNK_HEC_TOKEN
  • optional: SPLUNK_HEC_INDEX

Elasticsearch / OpenSearch audit sink

  • AUDIT_FORWARDER_TYPE=elasticsearch or AUDIT_FORWARDER_TYPE=opensearch
  • AUDIT_ELASTICSEARCH_ENDPOINT
  • optional: AUDIT_ELASTICSEARCH_INDEX

Loki

  • AUDIT_FORWARDER_TYPE=loki
  • AUDIT_LOKI_ENDPOINT

Syslog

  • AUDIT_FORWARDER_TYPE=syslog
  • AUDIT_SYSLOG_ADDRESS

Allowlist planning for external sinks

If a managed audit sink lives outside the PacketSafari container network, plan the same way you would for any other outbound-controlled destination:

  • identify the sink hostname and port
  • decide whether the sink is internal to the deployment network or truly external from the container perspective
  • ensure the sink is represented in the deployment's egress allowlist/proxy policy
  • validate that the destination is reachable only through the approved egress path

Examples:

  • SPLUNK_HEC_URL=https://splunk.example.com:8088 means splunk.example.com:8088 must be reachable through the approved outbound path
  • AUDIT_LOKI_ENDPOINT=https://loki.internal.example/loki/api/v1/push means loki.internal.example:443 must be approved if it is outside the PacketSafari container network
  • AUDIT_SYSLOG_ADDRESS=syslog.branch.example:6514 means syslog.branch.example:6514 must be approved if it is outside the PacketSafari container network

If you are unsure whether a sink requires approval, use the backend container's point of view:

  • localhost, PacketSafari internal service names, and internal .svc.packetsafari.internal names do not require external approval
  • another VM, appliance, reverse proxy, or collector on the customer LAN usually does require approval because it is still external to the PacketSafari service network

Internal TLS certificates and private CAs

Allowlisting only answers the reachability question. It does not solve TLS trust.

If your audit sink uses:

  • a self-signed certificate
  • a private enterprise root CA
  • an internal PKI chain that is not part of the public system trust bundle

then the egress proxy must also trust that CA chain, otherwise forwarding can still fail after the host is allowlisted.

This commonly applies to:

  • internal Splunk HEC endpoints
  • internal Loki endpoints
  • internal OpenSearch-compatible audit sinks
  • internal HTTPS gateways or reverse proxies

Durable custom CA trust for the egress proxy

PacketSafari now supports a durable custom CA directory for the egress proxy:

  • host path: configuration/custom-ca
  • container path: /usr/local/share/ca-certificates

Any .crt file placed there is loaded into the egress-ironproxy trust store when the container starts.

That means:

  • trust survives container restarts
  • trust survives long-running on-prem deployments
  • operators do not need to re-run manual trust commands after every restart

Recommended host workflow

Install a customer CA with the helper script:

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_custom_ca.py add /path/to/company-root-ca.crt --restart

If you use the dev stack locally:

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_custom_ca.py add /path/to/company-root-ca.crt \
  --compose-file docker-compose-dev.yml \
  --restart

List installed customer CAs:

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_custom_ca.py list

Remove a previously installed CA and restart the proxy:

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_custom_ca.py remove company-root-ca.crt --restart

If you prefer to restart manually, the explicit Docker command is:

docker compose --env-file .env.production --profile production -f docker-compose-production.yml restart egress-ironproxy

Verification

After restart:

  1. verify the proxy is healthy
  2. verify the sink host is allowlisted if required
  3. trigger a real audit event such as a login
  4. confirm the sink receives it

Basic health check:

docker compose --env-file .env.production --profile production -f docker-compose-production.yml ps egress-ironproxy

Important distinction

  • the proxy CA used by PacketSafari itself is generated automatically if missing and persisted in the proxy state volume
  • that proxy CA is long-lived, but it is not a rotating short-lived certificate authority
  • customer/private upstream CAs are separate and must be supplied by the operator when internal HTTPS sinks use private PKI

So the practical model is:

  • PacketSafari proxy CA: auto-created and persisted
  • customer internal CA trust: operator-managed through configuration/custom-ca

Proxy CA lifetime and rotation

The PacketSafari egress proxy CA is:

  • generated automatically if missing
  • stored in the persistent proxy state volume
  • reused across container restarts

That means long-lived on-prem installations do not regenerate it on every restart.

Current proxy CA defaults:

  • validity: 3650 days
  • rotation warning window: 30 days

Optional startup settings:

  • PACKETSAFARI_EGRESS_PROXY_CA_VALID_DAYS
  • PACKETSAFARI_EGRESS_PROXY_CA_ROTATE_BEFORE_DAYS
  • PACKETSAFARI_EGRESS_PROXY_CA_AUTO_ROTATE

If PACKETSAFARI_EGRESS_PROXY_CA_AUTO_ROTATE=true, the proxy rotates its CA on startup when the current CA is inside the configured warning window.

For deterministic operations, PacketSafari also provides a host-side helper script that can inspect and rotate the proxy CA and restart dependent services.

Show current proxy CA status

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_proxy_ca.py status

Check expiry window

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_proxy_ca.py check --rotate-before-days 30

Rotate manually and restart dependent services

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_proxy_ca.py rotate

Rotate automatically when needed

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_proxy_ca.py check --rotate-if-needed --rotate-before-days 30

That helper restarts:

  • egress-ironproxy
  • backend
  • worker

so trust changes are applied consistently for new TLS connections.

Scheduling for long-lived on-prem installs

For long-running customer environments, schedule a periodic check from the host, for example daily:

cd /Users/otr/code/packetsafari

python3 scripts/packetsafari_proxy_ca.py check --rotate-if-needed --rotate-before-days 30

This gives a simple future-proof host-side maintenance path without requiring operators to inspect certificate expiry manually.

Non-interactive install examples

Default stdout JSON:

curl -fsSL https://raw.githubusercontent.com/oripka/packetsafari-onprem/main/bootstrap.sh | bash -s -- install \
  --license /path/to/license-token.json \
  --manifest ./release-manifest.json \
  --non-interactive \
  --audit-forwarding-mode stdout_json

Splunk forwarder profile:

AUDIT_FORWARDER_TYPE=splunk \
SPLUNK_HEC_URL=https://splunk.example.com:8088 \
SPLUNK_HEC_TOKEN=replace-me \
curl -fsSL https://raw.githubusercontent.com/oripka/packetsafari-onprem/main/bootstrap.sh | bash -s -- install \
  --license /path/to/license-token.json \
  --manifest ./release-manifest.json \
  --non-interactive \
  --audit-forwarding-mode forwarder_profile \
  --audit-forwarder-type splunk

Upgrade with an existing forwarder-aware runtime env:

curl -fsSL https://raw.githubusercontent.com/oripka/packetsafari-onprem/main/bootstrap.sh | bash -s -- upgrade \
  --manifest ./release-manifest.json

Troubleshooting

Use the backend container logs first for audit delivery problems and the egress proxy logs for trust or upstream transport failures.

Symptom: HTTPS sink or model endpoint is allowlisted but still fails

Typical messages:

  • certificate verify failed
  • self signed certificate
  • unable to get local issuer certificate
  • x509: certificate signed by unknown authority

Likely cause:

  • the destination host is reachable, but the proxy does not trust the upstream CA chain

What to do:

  1. install the customer CA with scripts/packetsafari_custom_ca.py
  2. restart egress-ironproxy
  3. retry the connection

Symptom: HTTPS sink still does not receive logs after CA install

Typical messages:

  • connection timeout
  • connection refused
  • host blocked / denied by proxy policy

Likely cause:

  • the destination is not correctly approved in the egress allowlist, or the target host/port is wrong

What to do:

  1. verify the sink hostname and port
  2. confirm the destination is present in the egress allowlist/proxy policy
  3. confirm the sink is reachable from the deployment network

Symptom: proxy CA nearing expiry

Typical message from egress-ironproxy startup logs:

  • proxy CA expires within <N> day(s); set PACKETSAFARI_EGRESS_PROXY_CA_AUTO_ROTATE=true or rotate manually

What to do:

  • run python3 scripts/packetsafari_proxy_ca.py rotate
  • or schedule python3 scripts/packetsafari_proxy_ca.py check --rotate-if-needed --rotate-before-days 30

Useful log locations

Backend logs:

docker compose --env-file .env.production --profile production -f docker-compose-production.yml logs -f backend

Proxy logs:

docker compose --env-file .env.production --profile production -f docker-compose-production.yml logs -f egress-ironproxy