End-to-End Mobile Video Encoding Optimization: From 60s to Sub-2s Zero-Copy Acceleration

In mobile video editing, social media cameras, and real-time effects applications, the "render pipeline + offline export" stage is often the most computationally demanding and latency-critical path of the entire architecture.

In an early iteration of our mobile export pipeline, historical technical debt led to using FFmpeg integrated with the libx264 software encoder. Under the naive baseline implementation, effect textures rendered by the GPU had to be repeatedly copied across buses and converted across pixel formats before reaching the encoder. For a 20-second, 1308 × 1530 resolution high-framerate video, total export time exceeded one full minute (60s+). CPU utilization remained pinned at peak capacity, creating severe thermal throttling and battery drain.

This post documents the step-by-step optimization journey: from basic encoder parameter tuning and GPU shader color conversion, through the pitfalls and architectural realities of mobile PBO readbacks, to Apple's native CVPixelBuffer shared memory mapping, and finally achieving a true zero-copy pipeline with VideoToolbox hardware encoding.

---

Optimization Timeline Overview

Each stage systematically tackled the primary bottleneck exposed by profiling the previous iteration:

---

1.0 Tuning x264 Parameters for Immediate Gains

In our initial implementation, video frames passed through an OpenGL filter graph, yielded RGBA buffers, were converted to YUV420P on the CPU, and were finally submitted to libx264. Because no encoder flags were explicitly set, x264 defaulted to preset = medium.

On desktop workstations, medium offers a balanced tradeoff between compression ratio and encoding time. On mobile devices, however, extensive motion vector searches, sub-pixel estimations, and multi-reference frames overwhelm the mobile CPU.

By configuring FFmpeg's private codec options, we traded compression efficiency for encoding throughput:

// 1. Locate the x264 software encoder
*codec = avcodec_find_encoder_by_name("libx264");

// 2. Adjust preset to dramatically reduce motion search and intra-prediction complexity
// Presets: ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow, placebo
av_opt_set(codec_ctx->priv_data, "preset", "ultrafast", 0);

// 3. Set tune to zerolatency: disables B-frames and frame reordering buffers
// Tunes: film, animation, grain, stillimage, psnr, ssim, fastdecode, zerolatency
av_opt_set(codec_ctx->priv_data, "tune", "zerolatency", 0);

// 4. Bound the Profile level
// Profiles: baseline, main, high, high10, high422, high444
av_opt_set(codec_ctx->priv_data, "profile", "main", 0);

Results:

• preset = ultrafast simplified motion estimation to low-complexity diamond searches and disabled heavy in-loop deblocking filters;
• tune = zerolatency eliminated lookahead delays, enabling instantaneous NALU delivery per submitted frame;
• Export latency dropped from 60s+ down to ~20s.

---

2.0 Offloading Color Conversion to the GPU (RGBA -> I420)

With x264 tuned, the 20-second clip exported in ~20s (roughly a 1:1 real-time ratio). Yet flame graphs revealed that color conversion and data marshaling consumed nearly 40% of the remaining time.

Because the OpenGL effect stage output RGBA textures while x264 required planar YUV (such as AV_PIX_FMT_YUV420P), the pipeline had to bridge the gap:

let videoInfo = VideoInfo()
videoInfo.size = CGSize(width: 1308, height: 1530)
videoInfo.fps = demuxer.videoInfo.fps
videoInfo.pixelFmt = AV_PIX_FMT_BGRA.rawValue

The CPU previously performed this conversion using FFmpeg's libswscale:

// Initialize conversion context: RGBA -> YUV420p via bilinear sampling
ost->m_sws_ctx = sws_getContext(c->width, c->height, AV_PIX_FMT_BGRA,
                                c->width, c->height, AV_PIX_FMT_YUV420p,
                                SWS_BILINEAR, NULL, NULL, NULL);
if (!ost->m_sws_ctx) {
    std::cout << "Could not initialize the conversion context" << std::endl;
    return NULL;
}

// Software conversion executing purely on CPU threads
sws_scale(ost->m_sws_ctx, (const uint8_t * const *) ost->m_frame->data,
          ost->m_frame->linesize, 0, c->height, ost->m_tmp_frame->data,
          ost->m_tmp_frame->linesize);

On mobile CPUs, sws_scale burns substantial clock cycles performing pixel-by-pixel matrix multiplications. Because the image data already resided in GPU memory as a texture, we could perform this color space transformation in parallel via a GLSL fragment shader.

Converting RGBA to I420 in a Single Shader Pass

I420 comprises three discrete planar buffers:

• Y Plane: Full resolution height $H$;
• U Plane: Subsampled horizontally and vertically by half, spanning $H/4$ rows in the FBO layout;
• V Plane: Subsampled similarly, spanning $H/4$ rows in the FBO layout.

By configuring an off-screen FBO whose total byte footprint equals $W \times H \times 1.5$, the fragment shader splits the output surface into three normalized sampling regions:

• Region [0, 2/3]: Writes the Y plane;
• Region (2/3, 5/6]: Writes the U plane;
• Region (5/6, 1.0]: Writes the V plane.

#version 300 es
precision mediump float;
layout(location = 0) out vec4 outColor;

in vec2 v_texCoord;

uniform sampler2D inputImageTexture;
uniform vec2 u_ImgSize; // Actual image dimensions

const vec3 COEF_Y = vec3( 0.299,  0.587,  0.114);
const vec3 COEF_U = vec3(-0.147, -0.289,  0.436);
const vec3 COEF_V = vec3( 0.615, -0.515, -0.100);
const float U_DIVIDE_LINE = 2.0 / 3.0;
const float V_DIVIDE_LINE = 5.0 / 6.0;

void main() {
    float offsetX = 1.0 / u_ImgSize.x;
    vec2 texelOffset = vec2(offsetX, 0.0);

    if (v_texCoord.y <= U_DIVIDE_LINE) {
        // [Y Plane] Every pixel sampled
        // 1 sample + 3 horizontal offsets packs 4 luminance values into one RGBA output
        vec2 texCoord = vec2(v_texCoord.x, v_texCoord.y * 3.0 / 2.0);
        vec4 color0 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texelOffset);
        vec4 color2 = texture(inputImageTexture, texCoord + texelOffset * 2.0);
        vec4 color3 = texture(inputImageTexture, texCoord + texelOffset * 3.0);

        float y0 = dot(color0.rgb, COEF_Y);
        float y1 = dot(color1.rgb, COEF_Y);
        float y2 = dot(color2.rgb, COEF_Y);
        float y3 = dot(color3.rgb, COEF_Y);
        outColor = vec4(y0, y1, y2, y3);
    } else if (v_texCoord.y <= V_DIVIDE_LINE) {
        // [U Plane] 2x subsampled horizontally and vertically
        float offsetY = 1.0 / 3.0 / u_ImgSize.y;
        vec2 texCoord;
        if (v_texCoord.x <= 0.5) {
            texCoord = vec2(v_texCoord.x * 2.0, (v_texCoord.y - U_DIVIDE_LINE) * 2.0 * 3.0);
        } else {
            texCoord = vec2((v_texCoord.x - 0.5) * 2.0, ((v_texCoord.y - U_DIVIDE_LINE) * 2.0 + offsetY) * 3.0);
        }

        vec4 color0 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texelOffset * 2.0);
        vec4 color2 = texture(inputImageTexture, texCoord + texelOffset * 4.0);
        vec4 color3 = texture(inputImageTexture, texCoord + texelOffset * 6.0);

        float u0 = dot(color0.rgb, COEF_U) + 0.5;
        float u1 = dot(color1.rgb, COEF_U) + 0.5;
        float u2 = dot(color2.rgb, COEF_U) + 0.5;
        float u3 = dot(color3.rgb, COEF_U) + 0.5;
        outColor = vec4(u0, u1, u2, u3);
    } else {
        // [V Plane] Similar subsampling logic
        float offsetY = 1.0 / 3.0 / u_ImgSize.y;
        vec2 texCoord;
        if (v_texCoord.x <= 0.5) {
            texCoord = vec2(v_texCoord.x * 2.0, (v_texCoord.y - V_DIVIDE_LINE) * 2.0 * 3.0);
        } else {
            texCoord = vec2((v_texCoord.x - 0.5) * 2.0, ((v_texCoord.y - V_DIVIDE_LINE) * 2.0 + offsetY) * 3.0);
        }

        vec4 color0 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texelOffset * 2.0);
        vec4 color2 = texture(inputImageTexture, texCoord + texelOffset * 4.0);
        vec4 color3 = texture(inputImageTexture, texCoord + texelOffset * 6.0);

        float v0 = dot(color0.rgb, COEF_V) + 0.5;
        float v1 = dot(color1.rgb, COEF_V) + 0.5;
        float v2 = dot(color2.rgb, COEF_V) + 0.5;
        float v3 = dot(color3.rgb, COEF_V) + 0.5;
        outColor = vec4(v0, v1, v2, v3);
    }
}

Once rendered, the CPU allocates a single contiguous buffer to read back the GPU texture and slices it into AVFrame:

// Allocate contiguous buffer for I420 (Width * Height * 1.5)
imgByteSize = Int(bufferSize.width * bufferSize.height * 3 / 2)
let address = malloc(imgByteSize)
pixelBuffer = unsafeBitCast(address, to: UnsafeMutablePointer<UInt8>.self)

// Synchronously read back rendered texture
glReadPixels(0, 0, renderFramebuffer.size.width, renderFramebuffer.size.height,
             GLenum(GL_RGBA), GLenum(GL_UNSIGNED_BYTE), pixelBuffer)

Forwarding slices to the FFmpeg frame structure:

- (void)writeVideoData:(uint8_t *)data {
    AVFrame *frame = _muxer->get_video_buffer();
    int ySize = _videoInfo.size.width * _videoInfo.size.height;
    int uSize = ySize / 4;

    memcpy(frame->data[0], data, ySize);                 // Y Plane
    memcpy(frame->data[1], data + ySize, uSize);         // U Plane
    memcpy(frame->data[2], data + ySize + uSize, uSize); // V Plane
    _muxer->write_video_frame(frame);
}

Results: Eliminating libswscale from the CPU hot loop reduced overall export latency from 20s down to 15s.

---

3.0 Pitfall Analysis: Why Dual PBO Async Readback Backfired on Mobile

Profiling after Phase 2 isolated the next bottleneck: glReadPixels() was blocking synchronously, freezing the calling CPU thread for 15ms to 24ms per frame:

glReadPixels latency: 24.58 ms
glReadPixels latency: 16.06 ms
glReadPixels latency: 14.82 ms
glReadPixels latency: 14.55 ms
glReadPixels latency: 15.38 ms
glReadPixels latency: 16.02 ms

The Desktop Pattern: Dual PBO Ping-Pong Buffers

In desktop OpenGL programming, the textbook solution to synchronous readback stalls is PBO (Pixel Buffer Object) streaming:

1. Allocate two PBO buffers (PBO[0] and PBO[1]);
2. Trigger an asynchronous transfer from the FBO to PBO[index] via glReadPixels (which theoretically returns immediately);
3. Map and read the previous frame from PBO[nextIndex] on the CPU via glMapBufferRange.

// Initialize dual PBOs
glGenBuffers(2, &downloadPboId[0])
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), downloadPboId[0])
glBufferData(GLenum(GL_PIXEL_PACK_BUFFER), imgByteSize, nil, GLenum(GL_STREAM_READ))
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), downloadPboId[1])
glBufferData(GLenum(GL_PIXEL_PACK_BUFFER), imgByteSize, nil, GLenum(GL_STREAM_READ))
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), 0)

// Ping-pong pinging during render loop
let index = frameIndex % 2
let nextIndex = (index + 1) % 2
frameIndex += 1

// Signal GPU to copy current frame into PBO[index]
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), downloadPboId[index])
glReadPixels(0, 0, yuvBufferSize.width, yuvBufferSize.height, GLenum(GL_RGBA), GLenum(GL_UNSIGNED_BYTE), nil)

// CPU maps and reads previous frame from PBO[nextIndex]
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), downloadPboId[nextIndex])
let bufPtr = glMapBufferRange(GLenum(GL_PIXEL_PACK_BUFFER), 0, imgByteSize, GLenum(GL_MAP_READ_BIT))
if bufPtr != nil {
    pixelBuffer = unsafeBitCast(bufPtr, to: UnsafeMutablePointer<UInt8>.self)
    glUnmapBuffer(GLenum(GL_PIXEL_PACK_BUFFER))
}
glBindBuffer(GLenum(GL_PIXEL_PACK_BUFFER), 0)

Benchmarks: Unexpected Performance Regression

When we ran this implementation on actual hardware, the numbers regressed substantially:

glReadPixels latency: 36.96 ms | glMapBufferRange: 0.06 ms
glReadPixels latency: 36.18 ms | glMapBufferRange: 0.06 ms
glReadPixels latency: 36.53 ms | glMapBufferRange: 0.06 ms
glReadPixels latency: 38.46 ms | glMapBufferRange: 0.06 ms
glReadPixels latency: 36.79 ms | glMapBufferRange: 0.06 ms

• While glMapBufferRange executed in a mere 0.06ms, glReadPixels latency jumped from 15ms to over 36ms;
• Total export time degraded from 15s back up to 22s.

Architectural Post-Mortem: Why Did PBO Fail on Mobile?

This experiment exposed key architectural differences between discrete desktop GPUs and mobile systems:

1. TBDR (Tile-Based Deferred Rendering) Architecture: Mobile GPUs (Apple Silicon, PowerVR) do not rasterize immediately upon receiving DrawCalls. Instead, primitive geometry is binned into tile memory and executed in deferral for maximum memory efficiency.
2. Forced Pipeline Flushes: Calling glReadPixels against a PBO forces the OpenGL ES driver to maintain immediate buffer state consistency. The driver must prematurely flush and resolve all active tile buffers into system memory, disrupting tile scheduling.
3. UMA (Unified Memory Architecture): Unlike desktop systems where VRAM and host RAM are separated by a high-latency PCIe bus, iOS devices feature unified physical LPDDR memory shared by both CPU and GPU. PBO abstractions introduce synchronization barriers and state machine validation without any physical DMA bypass.

Takeaway: Desktop graphics idioms should not be blindly transplanted to mobile unified memory architectures. Native OS sharing facilities are essential.

---

4.0 Shared GPU-CPU Memory via CVOpenGLESTextureCache

Examining the data flow revealed two redundant memory transfers:

1. GPU VRAM -> CPU malloc buffer (via synchronous glReadPixels);
2. CPU buffer -> AVFrame buffers (via memcpy).

Can CVPixelBuffer and an OpenGL texture point directly to the same underlying physical allocation?

Using Apple's CoreVideo framework, CVOpenGLESTextureCacheCreateTextureFromImage allows wrapping a CVPixelBuffer directly as an OpenGL texture target, sharing backing memory with zero host-to-device transfers.

Implementation

// 1. Create a CVPixelBufferPool for buffer reuse
var attributes = [String: Any]()
attributes[kCVPixelBufferPixelFormatTypeKey as String] = kCVPixelFormatType_32BGRA
attributes[kCVPixelBufferWidthKey as String] = yuvBufferSize.width
attributes[kCVPixelBufferHeightKey as String] = yuvBufferSize.height
attributes[kCVPixelBufferIOSurfacePropertiesKey as String] = [:] // Enable IOSurface cross-API backing

CVPixelBufferPoolCreate(kCFAllocatorDefault, nil, attributes as CFDictionary, &self.pixelBufferPool)
guard let pixelBufferPool = pixelBufferPool else { return }

// 2. Obtain a CVPixelBuffer from the pool
CVPixelBufferPoolCreatePixelBuffer(nil, pixelBufferPool, &self.pixelBuffer)

CVBufferSetAttachment(self.pixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, .shouldPropagate)
CVBufferSetAttachment(self.pixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_601_4, .shouldPropagate)
CVBufferSetAttachment(self.pixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, .shouldPropagate)

// 3. Create an OpenGL texture directly bound to the CVPixelBuffer
var cachedTextureRef: CVOpenGLESTexture? = nil
CVOpenGLESTextureCacheCreateTextureFromImage(
    kCFAllocatorDefault,
    sharedImageProcessingContext.coreVideoTextureCache,
    self.pixelBuffer!,
    nil,
    GLenum(GL_TEXTURE_2D),
    GL_RGBA,
    GLsizei(yuvBufferSize.width),
    GLsizei(yuvBufferSize.height),
    GLenum(GL_BGRA),
    GLenum(GL_UNSIGNED_BYTE),
    0,
    &cachedTextureRef
)

let cachedTexture = CVOpenGLESTextureGetName(cachedTextureRef!)
self.renderFramebuffer = try? Framebuffer(
    context: sharedImageProcessingContext,
    orientation: .portrait,
    size: yuvBufferSize,
    textureOnly: false,
    overriddenTexture: cachedTexture
)

After the shader finishes rendering, no glReadPixels call is required. We merely lock the base address and flush the GPU command queue:

// Lock CVPixelBuffer base address
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

// Synchronize GPU completion
glFinish()

// Read pixel data directly without intermediate readbacks, then unlock
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

This eliminated the 15ms+ glReadPixels stall, leaving only the final memory copy into AVFrame.

Results: Export latency dropped from 15s to ~12s.

---

5.0 Switching from I420 to NV21/NV12: Conquering Stride Padding

While Phase 4 optimized texture access, passing memory into the encoder surfaced another subtle performance drain: memory byte alignment and row padding (Stride / Linesize).

The Inherent Friction of I420 Alignment

To ensure optimal memory bus alignment, the CoreVideo driver enforces hardware strides (typically 16-pixel or 64-byte row alignments). For instance, a frame with a logical width of 327 pixels is padded to 336 bytes per row (9 trailing padding bytes):

Actual Data (327 bytes)       Padding Bytes (9 bytes)
[===========================][*********] -> 336 bytes (BytesPerRow)

Under the I420 planar layout:

• Y, U, and V are stored as three isolated planes;
• U and V have half the width of Y, resulting in independent and mismatched padding offsets;
• Packing or unpacking tightly-packed I420 data required the CPU to iterate through nested loops to manually strip padding row-by-row, consuming valuable clock cycles.

Architectural Advantages of NV21 / NV12

NV21 and NV12 are Semi-Planar formats:

• Y is stored in its own plane;
• UV samples are interleaved in a combined plane (NV21: VUVU...; NV12: UVUV...).

For a 4 x 4 image:

NV21 Layout (Y plane, VU interleaved)   NV12 Layout (Y plane, UV interleaved)
Y Y Y Y                                 Y Y Y Y
Y Y Y Y                                 Y Y Y Y
Y Y Y Y                                 Y Y Y Y
Y Y Y Y                                 Y Y Y Y
V U V U                                 U V U V
V U V U                                 U V U V

This structure provides two decisive benefits:

1. Reduced Shader Branching: I420 required 3 conditional sampling branches in the fragment shader; NV21 requires only two regions (Y and VU).
2. Harmonized Memory Stride: While the interleaved VU plane has half the pixel width, each sample comprises two bytes (V and U). Consequently, the byte stride (BytesPerRow) of the VU plane is identical to that of the Y plane: BytesPerRow_VU = (W / 2) * 2 = W = BytesPerRow_Y Both planes share identical alignment offsets! Instead of stripping padding row-by-row on the CPU, we set linesize[0] and linesize[1] directly on AVFrame, allowing the underlying encoder to skip padding hardware-natively.

NV21 Fragment Shader Implementation

#version 300 es
precision mediump float;
layout(location = 0) out vec4 outColor;

in vec2 v_texCoord;

uniform sampler2D inputImageTexture;
uniform vec2 u_ImgSize;

const vec3 COEF_Y = vec3( 0.299,  0.587,  0.114);
const vec3 COEF_U = vec3(-0.147, -0.289,  0.436);
const vec3 COEF_V = vec3( 0.615, -0.515, -0.100);
const float UV_DIVIDE_LINE = 2.0 / 3.0;

void main() {
    float offsetX = 1.0 / u_ImgSize.x;
    vec2 texelOffset = vec2(offsetX, 0.0);

    if (v_texCoord.y <= UV_DIVIDE_LINE) {
        // [Y Plane]
        vec2 texCoord = vec2(v_texCoord.x, v_texCoord.y * 3.0 / 2.0);
        vec4 color0 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texelOffset);
        vec4 color2 = texture(inputImageTexture, texCoord + texelOffset * 2.0);
        vec4 color3 = texture(inputImageTexture, texCoord + texelOffset * 3.0);

        float y0 = dot(color0.rgb, COEF_Y);
        float y1 = dot(color1.rgb, COEF_Y);
        float y2 = dot(color2.rgb, COEF_Y);
        float y3 = dot(color3.rgb, COEF_Y);
        outColor = vec4(y0, y1, y2, y3);
    } else {
        // [VU Interleaved Plane] 4 RGBA samples generate 1 packed (V0, U0, V1, U1)
        vec2 texCoord = vec2(v_texCoord.x, (v_texCoord.y - UV_DIVIDE_LINE) * 3.0);
        vec4 color0 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texelOffset);
        vec4 color2 = texture(inputImageTexture, texCoord + texelOffset * 2.0);
        vec4 color3 = texture(inputImageTexture, texCoord + texelOffset * 3.0);

        float v0 = dot(color0.rgb, COEF_V) + 0.5;
        float u0 = dot(color1.rgb, COEF_U) + 0.5;
        float v1 = dot(color2.rgb, COEF_V) + 0.5;
        float u1 = dot(color3.rgb, COEF_U) + 0.5;
        outColor = vec4(v0, u0, v1, u1);
    }
}

The CPU dispatches memory in two bulk block copies without manual padding iteration:

- (void)writeVideoData2:(CVPixelBufferRef)pixelBuffer {
    AVFrame *frame = _muxer->get_video_buffer();
    CVPixelBufferLockBaseAddress(pixelBuffer, 0);
    uint8_t *baseAddress = (uint8_t *)CVPixelBufferGetBaseAddress(pixelBuffer);

    // Bulk copy Y plane
    int ySize = frame->linesize[0] * frame->height;
    memcpy(frame->data[0], baseAddress, ySize);

    // Bulk copy VU plane (alignment matched, zero row-by-row stripping needed)
    int uSize = frame->linesize[1] * frame->height / 2.0;
    memcpy(frame->data[1], baseAddress + ySize, uSize);

    CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);
    _muxer->write_video_frame(frame);
}

Results: Total export duration dropped from 12s to 9s.

---

6.0 The Ultimate Architecture: VideoToolbox Hardware Encoding & True Zero-Copy

At 9s, the pipeline had improved significantly from the initial 60s+ baseline. However, a fundamental architectural question remained:

If the GPU is already rendering into an Apple-native CVPixelBuffer, why are we using CPU memcpy to shuttle pixels into AVFrame and burning CPU cycles running x264 software encoding? Why not pipe the buffer directly into Apple's dedicated hardware encoder?

The MRT (Multiple Render Targets) Dimension Trap

To render directly into iOS's preferred BiPlanar hardware format (kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange, commonly known as 420v), we initially attempted using OpenGL ES 3.0 MRT to write to two attached textures simultaneously (Attachment 0 at full size for Y, Attachment 1 at half size for UV).

The OpenGL ES 3.0 Caveat: Under the OpenGL ES 3.0 specification, if color attachments on an active framebuffer have mismatched dimensions, the rasterization viewport is clamped to the smallest attachment's bounds. As a result, the full-sized Y texture only rendered its bottom-left quadrant, leaving the remainder blank.

The Solution: Multi-Pass Target Rendering to BiPlanar Planes

Using CoreVideo's native BiPlanar CVPixelBuffer, Plane 0 (Y) and Plane 1 (UV) are encapsulated as independent sub-surfaces. We create a dedicated texture target for each plane and render them using lightweight specialized passes:

#version 300 es
precision mediump float;

layout (location = 0) out vec4 outColor;
in vec2 v_texCoord;

uniform sampler2D inputImageTexture;
uniform vec2 u_imgSize;     // Full frame resolution
uniform bool u_isYPlane;    // Selects Y vs UV plane pass

const vec3 COEF_Y = vec3( 0.299,  0.587,  0.114);
const vec3 COEF_U = vec3(-0.169, -0.331,  0.500);
const vec3 COEF_V = vec3( 0.500, -0.439, -0.081);

void main() {
    // Render Y Plane (Plane 0)
    if (u_isYPlane) {
        vec4 color0 = texture(inputImageTexture, v_texCoord);
        float y = dot(color0.rgb, COEF_Y);
        outColor = vec4(y, 0.0, 0.0, 1.0);
    }

    // Render UV Plane (Plane 1)
    if (!u_isYPlane && v_texCoord.x <= 0.5 && v_texCoord.y <= 0.5) {
        vec2 texOffset = vec2(1.0 / u_imgSize.x, 0.0);
        vec2 texCoord = vec2(v_texCoord.x * 2.0, v_texCoord.y * 2.0);
        vec4 color00 = texture(inputImageTexture, texCoord);
        vec4 color1 = texture(inputImageTexture, texCoord + texOffset);
        float v = 0.5 + dot(color00.rgb, COEF_V);
        float u = 0.5 + dot(color1.rgb, COEF_U);
        outColor = vec4(u, v, 0.0, 1.0);
    }
}

Direct Handle Passing: Zero-Copy Handshake with FFmpeg

Switching the encoder from libx264 to h264_videotoolbox allows using FFmpeg's hardware acceleration pass-through. AVFrame does not allocate any pixel memory; instead, by convention, the native CVPixelBufferRef handle is assigned directly to frame->data[3]:

/// Submits native 420v CVPixelBuffer to VideoToolbox hardware encoder
/// @param pixelBuffer Plane 0 contains Y, Plane 1 contains UV
- (void)writeVideoToolBoxPixelData:(CVPixelBufferRef)pixelBuffer {
    AVFrame *frame = _muxer->get_video_buffer();

    // Assign native iOS CVPixelBuffer handle directly to data[3]
    frame->data[3] = (uint8_t *)pixelBuffer;

    // Dispatch to hardware encoder; dedicated silicon reads backing memory directly
    _muxer->write_video_frame(frame);
}

The Zero-Copy Dataflow

[GPU Effect Render Passes Complete]
        ↓ (Internal GPU write, zero bus transfers)
[CoreVideo BiPlanar CVPixelBuffer (420v)]
        ↓ (Direct pointer handle passing, 0 CPU memory copies)
[Apple VideoToolbox Hardware Encoder (ASIC Silicon)]
        ↓ (Hardware-accelerated bitstream encoding)
[H.264 NALUs -> MP4 Container Muxing]

Results:

• Near-Zero CPU Load: The CPU performs zero color conversion, stride alignment, or pixel copying;
• Export latency plummeted from 9s to under 2 seconds, completely resolving thermal throttling concerns.

---

Performance Evolution Summary

Comparing metrics across a 20-second 1308 × 1530 video export:

---

Engineering Takeaways for Mobile Media Developers

1. Mobile SoC Architecture Differs Fundamentally from Desktop: Techniques designed for discrete desktop GPUs (such as asynchronous dual-PBO transfers) can backfire on mobile Tile-Based Deferred Rendering (TBDR) GPUs by triggering forced pipeline flushes.
2. Minimize Cross-Bus Data Transfers: In modern media processing, raw mathematical transformations (like color conversions) are fast on GPU shaders. The real performance killer is shuttling buffers across bus boundaries. Keep data inside unified GPU memory whenever possible.
3. Build Around Native Platform Abstractions: In the Apple ecosystem, CVPixelBuffer, IOSurface, and CVOpenGLESTextureCache / Metal represent the universal language across rendering, computer vision, and hardware codecs. Aligning your internal formats with native types like 420v unlocks zero-copy hardware acceleration across the entire OS.
4. Account for Memory Alignment and Stride Upfront: Hardware memory strides (typically 16-pixel or 64-byte padding) are standard across mobile SoCs. Structuring shader output as Semi-Planar (NV12/NV21) matches the byte stride of chrominance and luminance planes, eliminating expensive CPU loops dedicated to stripping padding bytes.