Custom coders

Implementing custom encoders and decoders

Mediabunny’s custom coder API allows you to extend the library with your own encoding and decoding implementations. This is useful for adding support for codecs not natively supported by browsers, implementing specialized encoders/decoders, or polyfilling missing WebCodecs support.

When to use custom coders

Missing codec support

Add support for codecs not available in the browser’s WebCodecs implementation (e.g., MP3, AC3).

Polyfilling

Provide fallback implementations when WebCodecs is unavailable or doesn’t support a specific codec.

Specialized encoding

Implement custom encoding logic with specific quality/performance characteristics.

Hardware integration

Interface with custom hardware encoders/decoders or external encoding services.

Architecture

Custom coders integrate seamlessly into Mediabunny’s encoding and decoding pipeline:

When you register a custom coder, Mediabunny automatically uses it when:

1. The custom coder’s supports() method returns true
2. A native WebCodecs implementation is unavailable or doesn’t support the configuration

Custom video encoder

Create a custom video encoder by extending CustomVideoEncoder:

src/custom-coder.ts

import { CustomVideoEncoder, registerEncoder } from 'mediabunny';

class MyVideoEncoder extends CustomVideoEncoder {
  // Check if this encoder can handle the given codec and config
  static supports(codec: VideoCodec, config: VideoEncoderConfig): boolean {
    return codec === 'hevc' && config.width <= 4096;
  }

  // Initialize encoder
  async init(): Promise<void> {
    // Set up your encoder
    // Access config via: this.codec, this.config
  }

  // Encode a video sample
  async encode(
    videoSample: VideoSample,
    options: VideoEncoderEncodeOptions
  ): Promise<void> {
    // Extract frame data
    const videoFrame = videoSample.toVideoFrame();
    
    // Your encoding logic here...
    const encodedData = await myEncodingFunction(videoFrame, options.keyFrame);
    
    // Create packet from encoded data
    const packet = new EncodedPacket(
      encodedData,
      options.keyFrame ? 'key' : 'delta',
      videoSample.timestamp,
      videoSample.duration
    );
    
    // Output the packet (with optional metadata)
    this.onPacket(packet, {
      decoderConfig: {
        codec: 'hev1.1.6.L93.B0',
        codedWidth: this.config.width,
        codedHeight: this.config.height,
      },
    });
    
    videoFrame.close();
  }

  // Flush any buffered frames
  async flush(): Promise<void> {
    // Flush your encoder
  }

  // Clean up resources
  async close(): Promise<void> {
    // Release encoder resources
  }
}

// Register the encoder
registerEncoder(MyVideoEncoder);

Key points:

• supports() determines when your encoder is used
• Access codec and config via readonly properties
• Call this.onPacket() for each encoded packet
• Include decoder config metadata in the first onPacket() call

Source code: src/custom-coder.ts:80-102

Custom video decoder

Create a custom video decoder by extending CustomVideoDecoder:

src/custom-coder.ts

import { CustomVideoDecoder, registerDecoder } from 'mediabunny';

class MyVideoDecoder extends CustomVideoDecoder {
  static supports(codec: VideoCodec, config: VideoDecoderConfig): boolean {
    return codec === 'hevc';
  }

  async init(): Promise<void> {
    // Initialize your decoder
    // Access config via: this.codec, this.config
  }

  async decode(packet: EncodedPacket): Promise<void> {
    // Your decoding logic
    const frameData = await myDecodingFunction(packet.data);
    
    // Create VideoFrame from decoded data
    const videoFrame = new VideoFrame(frameData, {
      format: 'I420',
      codedWidth: this.config.codedWidth,
      codedHeight: this.config.codedHeight,
      timestamp: packet.timestamp * 1e6, // Convert to microseconds
    });
    
    // Create VideoSample and output it
    const sample = new VideoSample(videoFrame);
    this.onSample(sample);
  }

  async flush(): Promise<void> {
    // Flush decoder, output any buffered frames
  }

  async close(): Promise<void> {
    // Clean up decoder resources
  }
}

registerDecoder(MyVideoDecoder);

Key points:

• Call this.onSample() for each decoded sample
• Create VideoFrames with proper format and timestamps
• Handle B-frames correctly (buffering may be needed)

Source code: src/custom-coder.ts:20-42

Custom audio encoder

Create a custom audio encoder by extending CustomAudioEncoder:

src/custom-coder.ts

import { CustomAudioEncoder, registerEncoder } from 'mediabunny';

class MyAudioEncoder extends CustomAudioEncoder {
  static supports(codec: AudioCodec, config: AudioEncoderConfig): boolean {
    return codec === 'mp3';
  }

  async init(): Promise<void> {
    // Initialize MP3 encoder
    // Access: this.codec, this.config
  }

  async encode(audioSample: AudioSample): Promise<void> {
    // Extract audio data
    const audioData = audioSample.toAudioData();
    
    // Encode to MP3
    const encodedData = await myMp3Encoder(audioData);
    
    // Create packet
    const packet = new EncodedPacket(
      encodedData,
      'key', // Audio packets are typically all key frames
      audioSample.timestamp,
      audioSample.duration
    );
    
    // Output packet with config on first frame
    this.onPacket(packet, {
      decoderConfig: {
        codec: 'mp3',
        numberOfChannels: this.config.numberOfChannels,
        sampleRate: this.config.sampleRate,
      },
    });
    
    audioData.close();
  }

  async flush(): Promise<void> {
    // Flush encoder
  }

  async close(): Promise<void> {
    // Clean up
  }
}

registerEncoder(MyAudioEncoder);

Source code: src/custom-coder.ts:110-132

Custom audio decoder

Create a custom audio decoder by extending CustomAudioDecoder:

src/custom-coder.ts

import { CustomAudioDecoder, registerDecoder } from 'mediabunny';

class MyAudioDecoder extends CustomAudioDecoder {
  static supports(codec: AudioCodec, config: AudioDecoderConfig): boolean {
    return codec === 'ac3';
  }

  async init(): Promise<void> {
    // Initialize AC3 decoder
  }

  async decode(packet: EncodedPacket): Promise<void> {
    // Decode AC3 packet
    const pcmData = await myAc3Decoder(packet.data);
    
    // Create AudioData
    const audioData = new AudioData({
      format: 'f32-planar',
      sampleRate: this.config.sampleRate,
      numberOfFrames: pcmData.length / this.config.numberOfChannels,
      numberOfChannels: this.config.numberOfChannels,
      timestamp: packet.timestamp * 1e6,
      data: pcmData,
    });
    
    // Create AudioSample and output
    const sample = new AudioSample(audioData);
    this.onSample(sample);
  }

  async flush(): Promise<void> {
    // Flush any buffered audio
  }

  async close(): Promise<void> {
    // Clean up
  }
}

registerDecoder(MyAudioDecoder);

Source code: src/custom-coder.ts:50-72

Registration

Register your custom coders before creating outputs or inputs:

import { registerEncoder, registerDecoder } from 'mediabunny';

// Register encoders
registerEncoder(MyVideoEncoder);
registerEncoder(MyAudioEncoder);

// Register decoders
registerDecoder(MyVideoDecoder);
registerDecoder(MyAudioDecoder);

// Now create outputs/inputs - custom coders will be used automatically
const output = new Output({ /* ... */ });
const input = new Input({ /* ... */ });

Register custom coders before creating Output or Input instances to ensure they’re available when needed.

Source code:

• registerEncoder: src/custom-coder.ts:175-197
• registerDecoder: src/custom-coder.ts:145-167

Polyfilling missing codecs

Custom coders are perfect for polyfilling codecs not supported by a browser:

import { CustomVideoEncoder, registerEncoder } from 'mediabunny';
import h264Wasm from 'h264-wasm-encoder'; // Hypothetical WASM encoder

class H264WasmEncoder extends CustomVideoEncoder {
  private encoder: any;

  // Only use WASM encoder if native WebCodecs doesn't support AVC
  static supports(codec: VideoCodec, config: VideoEncoderConfig): boolean {
    if (codec !== 'avc') return false;
    
    // Check if native encoder is available
    if (typeof VideoEncoder !== 'undefined') {
      // Let native encoder handle it
      return false;
    }
    
    // Native not available, use WASM fallback
    return true;
  }

  async init(): Promise<void> {
    this.encoder = await h264Wasm.create({
      width: this.config.width,
      height: this.config.height,
      bitrate: this.config.bitrate,
    });
  }

  async encode(
    videoSample: VideoSample,
    options: VideoEncoderEncodeOptions
  ): Promise<void> {
    const frame = videoSample.toVideoFrame();
    const result = await this.encoder.encode(frame, options.keyFrame);
    
    const packet = new EncodedPacket(
      result.data,
      result.isKeyFrame ? 'key' : 'delta',
      videoSample.timestamp,
      videoSample.duration
    );
    
    this.onPacket(packet, result.metadata);
    frame.close();
  }

  async flush(): Promise<void> {
    await this.encoder.flush();
  }

  async close(): Promise<void> {
    this.encoder.destroy();
  }
}

registerEncoder(H264WasmEncoder);

Real-world examples

Mediabunny extensions use custom coders:

MP3 Encoder

WASM-based MP3 encoder using LAME

@mediabunny/mp3-encoder

AC-3 Decoder

WASM-based AC-3/E-AC-3 decoder

@mediabunny/ac3

See the extension source code for complete implementations.

Best practices

1. Implement supports() carefully

   The supports() method determines when your coder is used. Return false if native support is better.

2. Handle errors gracefully

   Throw descriptive errors from your methods - they’ll be caught and surfaced to the user.

3. Manage resources

   Always clean up in close(). This includes WASM memory, worker threads, etc.

4. Provide metadata

   Include complete decoder config in the first onPacket() call for encoders.

5. Use async operations

   Methods can return void or Promise<void>. Use async when needed for WASM or workers.

6. Test thoroughly

   Test with various configurations, frame sizes, and edge cases.

API reference

CustomVideoEncoder

Properties

readonly codec: VideoCodec;        // The codec to encode (e.g., 'avc', 'hevc')
readonly config: VideoEncoderConfig; // WebCodecs-compatible encoder config
readonly onPacket: (              // Callback to output encoded packets
  packet: EncodedPacket,
  meta?: EncodedVideoChunkMetadata
) => unknown;

Methods

// Static method: Check if encoder supports this config
static supports(codec: VideoCodec, config: VideoEncoderConfig): boolean;

// Initialize encoder
abstract init(): Promise<void> | void;

// Encode a video sample
abstract encode(
  videoSample: VideoSample,
  options: VideoEncoderEncodeOptions
): Promise<void> | void;

// Flush buffered frames
abstract flush(): Promise<void> | void;

// Release resources
abstract close(): Promise<void> | void;

CustomVideoDecoder

Properties

readonly codec: VideoCodec;          // The codec to decode
readonly config: VideoDecoderConfig; // WebCodecs-compatible decoder config
readonly onSample: (                 // Callback to output decoded samples
  sample: VideoSample
) => unknown;

Methods

// Static method: Check if decoder supports this config
static supports(codec: VideoCodec, config: VideoDecoderConfig): boolean;

// Initialize decoder
abstract init(): Promise<void> | void;

// Decode an encoded packet
abstract decode(packet: EncodedPacket): Promise<void> | void;

// Flush buffered frames
abstract flush(): Promise<void> | void;

// Release resources
abstract close(): Promise<void> | void;

CustomAudioEncoder

Properties

readonly codec: AudioCodec;        // The codec to encode (e.g., 'opus', 'mp3')
readonly config: AudioEncoderConfig; // WebCodecs-compatible encoder config
readonly onPacket: (              // Callback to output encoded packets
  packet: EncodedPacket,
  meta?: EncodedAudioChunkMetadata
) => unknown;

Methods

// Static method: Check if encoder supports this config
static supports(codec: AudioCodec, config: AudioEncoderConfig): boolean;

// Initialize encoder
abstract init(): Promise<void> | void;

// Encode an audio sample
abstract encode(audioSample: AudioSample): Promise<void> | void;

// Flush buffered samples
abstract flush(): Promise<void> | void;

// Release resources
abstract close(): Promise<void> | void;

CustomAudioDecoder

Properties

readonly codec: AudioCodec;          // The codec to decode
readonly config: AudioDecoderConfig; // WebCodecs-compatible decoder config
readonly onSample: (                 // Callback to output decoded samples
  sample: AudioSample
) => unknown;

Methods

// Static method: Check if decoder supports this config
static supports(codec: AudioCodec, config: AudioDecoderConfig): boolean;

// Initialize decoder
abstract init(): Promise<void> | void;

// Decode an encoded packet
abstract decode(packet: EncodedPacket): Promise<void> | void;

// Flush buffered samples
abstract flush(): Promise<void> | void;

// Release resources
abstract close(): Promise<void> | void;

See also

• Extensions - Pre-built codec extensions
• Media sources - Using custom encoders with media sources
• Media sinks - Using custom decoders with media sinks
• Packets and samples - Understanding media data structures