VideoToolbox Background Deadlocks and -12903 on iOS

Apple's VideoToolbox framework gives iOS apps direct access to hardware video encoding and decoding. It works well — until the app goes to the background. Lock screens, incoming calls, or app switches during a long-running export can trigger three distinct failures in FFmpeg-based pipelines:

1. The encoding thread deadlocks immediately upon entering the background;
2. Returning to the foreground floods the console with -12903 errors (kVTInvalidSessionErr);
3. Even after resetting the decoder, the output video shows dropped frames, stutter, and visual artifacts.

This post traces these failures through production logs, explains why iOS reclaims hardware sessions so aggressively, and presents a complete recovery architecture.

---

1. The Symptoms: Error -12903 and Pipeline Deadlocks

In standard lifecycle testing, transitioning an active hardware transcoding pipeline to the background surfaces two failure modes:

Decoder Failure: Invalid Session

[hevc @ 0x11e165c00] Failed to decode frame (invalid session, -12903)
[hevc @ 0x11e165c00] hardware accelerator failed to decode picture
Demuxer::avcodec send video packet error, but will continue, Error: Unknown error occurred

Encoder Failure: Hang and Deadlock

[hevc_videotoolbox @ 0x11e166300] Error: cannot encode frame: -12903
avcodec_send_frame, error: Generic error in an external library
[hevc_videotoolbox @ 0x11e166300] Error flushing frames: -12903
avcodec_send_frame, error: Generic error in an external library

While the decoder generally drops packets and returns error codes, the encoder often deadlocks when attempting to feed frames or flush stale data in the background, locking up the calling thread completely.

---

2. Root Cause Analysis: Why Does -12903 Occur?

2.1 Understanding -12903 (kVTInvalidSessionErr)

In Apple's VideoToolbox/VTErrors.h, the error is explicitly defined:

kVTInvalidSessionErr = -12903,

This error indicates that the VTCompressionSession or VTDecompressionSession has been invalidated. The hardware acceleration context bound to the session is no longer recognized by the kernel driver.

2.2 Strict iOS Hardware Resource Reclamation

iOS enforces aggressive power management policies. When an app moves to the background (without special audio/camera streaming entitlements), the operating system aggressively reclaims hardware:

1. Hardware VPU/GPU Reclamation: The dedicated video processing units and GPU contexts are revoked. The system marks active VideoToolbox sessions as invalid;
2. Background GPU Submissions Prohibited: If Metal or OpenGL pipelines attempt to touch framebuffers in the background, command buffers are aborted immediately:

   Insufficient Permission (to submit GPU work from background)
   (00000006:kIOGPUCommandBufferCallbackErrorBackgroundExecutionNotPermitted)

3. Sessions Cannot Be Resumed: Once invalidated, internal hardware registers, shared memory bindings, and decoded picture buffers (DPB) are cleared. Any subsequent calls return -12903.

---

3. Decoder Recovery: The Missing IDR Frame Trap

When encountering -12903, the first instinct is to discard the old decoder context and create a fresh one:

int Demuxer::reset_video_decoder() {
    // Release stale contexts
    if (m_video_dec_ctx) {
        avcodec_free_context(&m_video_dec_ctx);
        m_video_dec_ctx = nullptr;
    }
    if (m_hw_device_ctx) {
        av_buffer_unref(&m_hw_device_ctx);
        m_hw_device_ctx = nullptr;
    }

    // Reinitialize hardware decoder; fall back to software on failure
    int ret = init_hw_decoder(&m_video_dec_ctx, AVMEDIA_TYPE_VIDEO);
    m_video_software_decoding = false;
    if (ret < 0) {
        ret = init_decoder(&m_video_dec_ctx, AVMEDIA_TYPE_VIDEO);
        m_video_software_decoding = true;
    }
    return ret;
}

3.1 Dropped Frames: Missing Keyframe Reference

If the pipeline simply feeds subsequent AVPacket objects from the current file offset into the new decoder, decoding fails continuously:

retry: -1313558101
retry skip
retry skip
... (Skipped 23 consecutive non-keyframe packets)

Why This Happens: H.264 and HEVC rely heavily on inter-frame prediction. A newly initialized decoder must encounter an IDR (Instantaneous Decoder Refresh) keyframe to establish its reference picture list. Feeding P or B frames without an IDR reference forces the decoder to discard all packets until the next GOP keyframe.

In a video with 1- to 2-second GOP intervals, this results in noticeable frame loss and stutter upon returning to the foreground.

3.2 The Solution: Backward Demuxer Seek + PTS Deduplication

To recover the frames between the last decoded timestamp and the nearest keyframe, the demuxer must seek backward to the preceding keyframe:

void Demuxer::read_frame() {
    AVPacket pkt;
    while (m_state != Stopped) {
        if (!m_can_read_frames) {
            m_read_frame_mutex.wait();
        }

        // Rewind to the previous keyframe
        if (m_seek_keyframe_pts != AV_NOPTS_VALUE) {
            int ret = av_seek_frame(m_fmt_ctx,
                                  m_video_info.m_stream_index,
                                  m_seek_keyframe_pts,
                                  AVSEEK_FLAG_BACKWARD);
            if (ret >= 0) {
                clear_video_pkt_list();
            }
            m_seek_keyframe_pts = AV_NOPTS_VALUE;
        }

        int ret = av_read_frame(m_fmt_ctx, &pkt);
        // ... Dispatch packets
    }
}

Handling Duplicate Frames

Rewinding the file pointer causes the decoder to re-emit previously processed frames (e.g., from PTS 13881 up to the crash point 14341).

Feeding duplicates to the encoder would cause timestamps to jump backward, corrupting playback. We filter these with a strictly monotonic PTS gate:

/// Last successfully dispatched video PTS
private var lastOfferVideoPst: CMTime = .zero

public func getDecodeVideoData(_ sampleBuffer: CMSampleBuffer?) {
    guard let sampleBuffer = sampleBuffer else {
        videoDecodeFinished = true
        return
    }

    if cacheType != .all && cacheType != .video { return }

    // Discard frames at or before the last recorded timestamp
    if sampleBuffer.presentationTimeStamp > lastOfferVideoPst {
        offer(sampleBuffer, type: .video)
        lastOfferVideoPst = sampleBuffer.presentationTimeStamp
    } else {
        CMSampleBufferInvalidate(sampleBuffer)
    }
}

This ensures the encoder receives a continuous sequence of frames without timestamp regressions.

---

4. Encoder Deadlock: The Stale Flush Trap

With decoding stabilized, the remaining hurdle was resolving encoder deadlocks during background transitions.

4.1 The Flawed Assumption: "Always Flush Before Backgrounding"

In standard encoding flows, shutting down an encoder involves passing a null frame (avcodec_send_frame(ctx, NULL)) to flush buffered B-frames. Many developers hook into applicationWillResignActive or applicationDidEnterBackground to trigger this flush.

The reality on mobile: Thread dumps revealed that the flush itself is often the direct cause of the deadlock:

1. When applicationDidEnterBackground fires, an encoding worker thread may be mid-stride inside a mutex-protected call;
2. By the time the flush acquires the lock, iOS has already revoked hardware permissions;
3. Submitting a flush to an invalidated VideoToolbox session causes Apple's driver to block indefinitely waiting for hardware that no longer exists — a deadlock.

4.2 The Solution: Fast Cutoff and Foreground Rebuilds

The reliable approach is a circuit breaker:

Never attempt to flush an invalidated or expiring VideoToolbox encoder during background transitions.

The robust strategy:

1. Immediate Pause: The moment the app resigns active status, set an atomic pipeline state to Paused and block all calls to avcodec_send_frame;
2. Snapshot State: Record the last successfully muxed DTS and PTS;
3. Foreground Rebuild: Upon returning to the foreground (didBecomeActive), discard the invalidated session via avcodec_free_context and create a fresh hevc_videotoolbox / h264_videotoolbox context;
4. Resume: The demuxer rewinds to the last keyframe, and the PTS deduplication gate ensures seamless continuation.

---

5. Edge Cases: Pixel Formats and Lifecycle Nuances

Over months of profiling across iOS devices, two additional subtleties emerged:

5.1 Pixel Format Impact on Background Teardown

In FFmpeg VideoToolbox pipelines, two primary input pixel formats are used:

• AV_PIX_FMT_VIDEOTOOLBOX: Backed directly by zero-copy CVPixelBuffer / GPU memory allocations;
• AV_PIX_FMT_YUV420P: Standard host CPU memory buffers marshaled to the hardware encoder.

Key Difference:

• Releasing an invalidated AV_PIX_FMT_VIDEOTOOLBOX encoder while in the background frequently triggers internal deadlocks;
• With AV_PIX_FMT_YUV420P, background calls still return -12903, but teardown completes without locking the thread.

Rule: For zero-copy AV_PIX_FMT_VIDEOTOOLBOX pipelines, defer all session teardown and recreation until the app is confirmed back in the foreground.

5.2 Lifecycle Timing: willResignActive vs didEnterBackground

In UIKit lifecycle notifications:

• willResignActiveNotification: Fired on interruptions (notification center, app switcher, incoming call);
• didEnterBackgroundNotification: Fired when the app is actually suspended.

Creating or reconfiguring VideoToolbox sessions during willResignActive is risky: If a hardware encoder is initialized during willResignActive and the app subsequently enters didEnterBackground, the session is almost guaranteed to deadlock on subsequent access.

Best practice:

• Anchor teardown and pause gates to didEnterBackgroundNotification;
• Treat any encoder that existed during the transition as invalid until didBecomeActiveNotification confirms foreground state.

NotificationCenter.default.addObserver(
    self,
    selector: #selector(didBecomeActive),
    name: UIApplication.didBecomeActiveNotification,
    object: nil
)

NotificationCenter.default.addObserver(
    self,
    selector: #selector(didEnterBackground),
    name: UIApplication.didEnterBackgroundNotification,
    object: nil
)

---

6. End-to-End Recovery Flow

The production-tested recovery architecture is:

[App Enters Background]
      ↓
Set PipelineState = Paused; block all avcodec_send_frame calls
(Do NOT flush — flushing causes driver deadlocks)
      ↓
[App Returns to Foreground (didBecomeActive)]
      ↓
1. Decoder: Free stale context, create fresh VideoToolbox decoder
      ↓
2. Keyframe Rewind: Demuxer calls av_seek_frame(BACKWARD) to the nearest IDR
      ↓
3. Deduplication: Discard re-decoded frames where PTS <= lastDispatchedPTS
      ↓
4. Encoder: If -12903, destroy context and create fresh VideoToolbox session
      ↓
Pipeline resumes with clean, continuous timestamps

Key Takeaways

1. Never flush in the background: Flushing an invalidated hardware encoder deadlocks the driver;
2. Seek backward on recovery: A fresh decoder needs an IDR keyframe — rewind the demuxer and deduplicate;
3. Defer teardown to the foreground: Destroying hardware sessions in the background risks deadlocks, especially with zero-copy pixel formats.