VoIP over IPsec and ESP Packet Analysis

VoIP troubleshooting becomes harder when the signaling or media is protected by IPsec Encapsulating Security Payload (ESP). A packet capture may contain a clearly visible call setup, followed by UDP traffic that cannot be decoded as RTP because the payload is encrypted.

Encryption does not make the capture useless. It changes which questions the evidence can answer.

PacketSafari identifies encrypted media coverage, summarizes the IPsec security associations visible in the capture, correlates IKE behavior, and reports what remains unavailable. If an authorized operator supplies a compatible Wireshark esp_sa file, PacketSafari applies it only to that capture and reprocesses the decrypted traffic through the ordinary VoIP analysis workflow.

The important rule is simple: never turn missing visibility into a quality score.

What remains visible when VoIP media is encrypted

An ESP packet exposes its outer IP endpoints, Security Parameters Index (SPI), sequence number, length, timing, and direction. The protected transport headers and media payload normally remain hidden until the correct security association is available.

These visible fields still support useful troubleshooting:

PacketSafari therefore gives an encrypted call an explicit coverage state:

Media encrypted—audio quality unavailable

This is not an analysis failure. It is a truthful boundary around the evidence. The investigation can still explain the IPsec behavior while withholding RTP loss, jitter, delay, R-factor, and audio-quality conclusions that require decrypted media.

Reading ESP sequence behavior correctly

Each ESP security association is directional and has its own SPI and sequence space. PacketSafari tracks those streams independently and reports sequence gaps and regressions without automatically labeling every irregularity as packet loss.

A forward gap may indicate:

• packets lost before the capture point
• packets omitted by the capture process
• asymmetric visibility or capture filtering
• an incomplete capture that begins mid-association

A regression or repeated sequence number may indicate:

• packet duplication
• reordering at the observation point
• merged captures with overlapping traffic
• replayed traffic that the receiver's anti-replay window may reject

The capture location matters. A trace recorded after the receiver cannot prove what arrived before an upstream loss point, and a capture with dropped packets cannot distinguish capture loss from network loss by ESP sequence numbers alone. PacketSafari preserves the exact SPI, endpoints, direction, sequence range, and representative frames so an operator can evaluate the observation in context.

IKE errors and re-key churn

IKE establishes and maintains the security associations used by ESP. Repeated negotiation does not automatically mean failure: healthy deployments re-key when time or byte lifetimes expire. The useful question is whether the observed rate and outcome match the expected policy.

PacketSafari summarizes IKE errors and SA turnover alongside the ESP streams. Useful indicators include:

• repeated CHILD_SA creation over a short interval
• rapid SPI turnover
• delete payloads immediately after creation
• error or failure notifications
• incomplete request-response exchanges
• bursts of informational exchanges
• protected traffic stopping near a failed re-key

This helps distinguish a media-quality complaint from a tunnel-lifecycle problem. A call that becomes silent at the same time as a failed CHILD_SA replacement deserves a different investigation from a decrypted RTP stream with rising jitter.

VoIP quality after successful decryption

When PacketSafari can decode the media—either because it was cleartext or because the operator supplied an authorized esp_sa file—the same bounded VoIP reducer processes RTP and RTCP.

The call-quality dossier includes:

• RTP packet counts and duration
• expected and observed sequence ranges
• missing, duplicate, and illegally formed RTP packet counters
• jitter and inter-arrival variation
• skew, delta, and bounded delay statistics
• RTCP packet and report counts
• sender and receiver report evidence
• round-trip timing when the required RTCP fields and timestamps are present
• codec context, estimated R-factor, and derived quality interpretation when inputs are sufficient

R-factor and MOS-style estimates are models, not direct listening tests. They depend on codec assumptions, loss behavior, delay, and the completeness of the observation. PacketSafari labels the inputs and coverage so an estimate derived from partial evidence does not appear more certain than it is.

Delay also needs careful language. A single passive capture usually cannot measure true one-way mouth-to-ear delay because the endpoint clocks and processing stages are unavailable. RTCP can support round-trip calculations when matching sender-report timing is present; its delay-since-last-sender-report field is not itself an end-to-end one-way-delay measurement.

Per-subnet statistics without analyzing every audio stream

Large captures may contain thousands of calls and millions of media packets. Decoding every payload into audio, generating waveforms, or retaining per-packet model input would be expensive and unnecessary for first-pass triage.

PacketSafari uses compact online aggregation. As packets are processed, bounded stream summaries feed per-subnet statistics such as:

• call and RTP stream counts
• encrypted versus quality-measurable media coverage
• packets, bytes, and active duration
• loss, duplicate, illegal-packet, and RTCP totals
• distributions of jitter, delay, and R-factor where available
• counts of affected calls and representative evidence pivots

The retained state grows with meaningful streams, security associations, and subnet groups—not with an unbounded copy of every packet observation. This keeps capture-wide triage useful on multi-gigabyte files while detailed packet retrieval and audio-oriented work remain on demand for selected calls.

Supplying a Wireshark esp_sa file

PacketSafari accepts a Wireshark-compatible esp_sa file as an authorized decryption-secret type. The file describes ESP security associations, including the addressing, SPI, algorithms, and keys needed by the dissector.

The workflow is deliberately explicit:

1. An authorized operator selects the capture.
2. The operator uploads the matching esp_sa file.
3. PacketSafari scopes the secret to that capture and analysis generation.
4. Sharkd applies the security associations while decoding the capture.
5. PacketSafari reprocesses newly visible SIP, RTP, and RTCP evidence.
6. Derived findings are labelled as decrypted evidence.
7. Secret material follows the configured expiry and deletion lifecycle.

An esp_sa file must be handled like a password or private key. Anyone who has the capture and its working security associations may be able to decrypt protected traffic. It should not be pasted into prompts, logs, findings, support tickets, or ordinary report exports.

For offline Wireshark workflows, compatible secrets can also be embedded in a new PCAPNG file with editcap:

editcap --inject-secrets esp,./esp_sa ./input.pcapng ./output-with-esp-secrets.pcapng

The resulting PCAPNG is more sensitive than the original capture because it carries decryption material. Store, share, and delete it accordingly.

Why PacketSafari does not harvest IMS-AKA keys automatically

Some IMS environments expose enough related material across separate Mw and Gm captures to derive ESP security associations. Doing this safely would require extracting CK and IK, correlating SIP authentication and IPsec negotiation, joining exact captures by Call-ID, deriving candidate keys, and trial-matching them against media SAs.

PacketSafari does not start that workflow merely because ESP is present. Automatic cross-capture key recovery would cross an important authorization boundary and could expose credentials unrelated to the operator's intended investigation.

The implemented workflow instead makes the immediate evidence useful: it identifies encrypted media, explains the ESP and IKE state, states that audio quality is unavailable, and accepts an operator-supplied esp_sa file when decryption is explicitly authorized.

Any future IMS-AKA recovery workflow should be on-premises and operator initiated, with exact capture selection, bounded candidate matching, encrypted temporary storage, automatic expiry, a complete audit trail, and strict guarantees that CK, IK, and derived keys never enter model prompts, logs, or findings.

A scalable investigation sequence

For a large VoIP capture, the most efficient order is:

1. Run capture-wide protocol, endpoint, call, ESP, and IKE triage.
2. Separate cleartext, successfully decrypted, and encrypted-unavailable media coverage.
3. Compare compact per-subnet and per-call summaries.
4. Prioritize calls with signaling failures, ESP anomalies, re-key churn, or measurable RTP degradation.
5. Retrieve exact packet evidence only for the selected calls and time windows.
6. Add an authorized esp_sa secret only when the unresolved question requires protected content.
7. Re-run ordinary VoIP analysis on the newly decrypted evidence.

This avoids a special heavyweight analysis path for every call. VoIP remains a protocol-gated troubleshooting capability: capture-wide triage finds the relevant calls and coverage gaps, then detailed work happens on demand.

How PacketSafari helps

PacketSafari combines deterministic protocol decoding with compact investigation facts and exact packet pivots. For VoIP over IPsec, that means the report can connect call symptoms to:

• ESP SPIs, directions, endpoints, and sequence anomalies
• IKE negotiation errors and re-key behavior
• explicit encrypted-media coverage
• RTP and RTCP quality measurements after decryption
• subnet-level impact without storing per-packet working state indefinitely
• operator-authorized, capture-scoped esp_sa handling

The AI PCAP analyzer turns those facts into a reviewable troubleshooting workflow, while the network root-cause analysis workflow keeps observations, interpretations, evidence, and coverage limitations separate.

Bottom line

Encrypted VoIP does not justify guessing, and it does not make packet analysis pointless. ESP and IKE provide meaningful tunnel evidence before decryption. PacketSafari reports that evidence, marks hidden media honestly, scales the first pass with compact summaries, and unlocks full RTP and RTCP analysis only when an authorized operator supplies the required security associations.