Formats and Codecs

Understanding container formats, codecs, and their compatibility in Mediabunny

In media processing, it’s crucial to understand the difference between container formats and codecs. Mediabunny supports a wide range of both, giving you flexibility in how you process media files.

Container formats vs codecs

Think of a media file as a shipping box:

Container Format

The box itself - defines how media data, metadata, and multiple tracks are organized and stored together. Examples: MP4, WebM, MKV.

Codec

What’s inside the box - the compression algorithm used to encode/decode the actual video or audio data. Examples: H.264, VP9, AAC.

A single container format can hold media encoded with various codecs, but not all combinations are valid. For example:

• ✅ MP4 can contain H.264 video and AAC audio
• ✅ WebM can contain VP9 video and Opus audio
• ❌ WebM cannot contain H.264 video (not in WebM specification)

Supported container formats

Mediabunny supports these container formats for both reading and writing:

Video containers

MP4

MPEG-4 Part 14 - The most widely supported video container format.

• Extension: .mp4
• MIME Type: video/mp4
• Best for: Universal compatibility, web delivery, mobile devices
• Video codecs: H.264 (AVC), H.265 (HEVC), VP9, AV1, VP8
• Audio codecs: AAC, MP3, Opus, Vorbis, FLAC, AC-3, E-AC-3, most PCM variants

input-format.ts:83-104

export class Mp4InputFormat extends IsobmffInputFormat {
  async _canReadInput(input: Input) {
    const majorBrand = await this._getMajorBrand(input);
    return !!majorBrand && majorBrand !== 'qt  ';
  }

  get name() {
    return 'MP4';
  }

  get mimeType() {
    return 'video/mp4';
  }
}

MP4 is based on the ISO Base Media File Format (ISOBMFF), which QuickTime also uses.

WebM

WebM - An open, royalty-free format optimized for web use.

• Extension: .webm
• MIME Type: video/webm
• Best for: Web streaming, HTML5 video, open-source projects
• Video codecs: VP8, VP9, AV1 only
• Audio codecs: Opus, Vorbis only

output-format.ts:526-561

export class WebMOutputFormat extends MkvOutputFormat {
  override getSupportedCodecs(): MediaCodec[] {
    return [
      ...VIDEO_CODECS.filter(codec => ['vp8', 'vp9', 'av1'].includes(codec)),
      ...AUDIO_CODECS.filter(codec => ['opus', 'vorbis'].includes(codec)),
      ...SUBTITLE_CODECS,
    ];
  }
}

WebM has strict codec requirements - only VP8/VP9/AV1 for video and Opus/Vorbis for audio.

Matroska (MKV)

Matroska - A flexible, open standard container format.

• Extension: .mkv
• MIME Type: video/x-matroska
• Best for: Archival, high-quality video, multiple audio/subtitle tracks
• Video codecs: All supported video codecs
• Audio codecs: All non-PCM codecs, most PCM variants

input-format.ts:138-228

export class MatroskaInputFormat extends InputFormat {
  get name() {
    return 'Matroska';
  }

  get mimeType() {
    return 'video/x-matroska';
  }
}

Matroska supports transparent video when alpha side data is provided.

QuickTime (MOV)

QuickTime File Format - Apple’s multimedia container.

• Extension: .mov
• MIME Type: video/quicktime
• Best for: Apple ecosystem, professional video editing
• Video codecs: All supported video codecs
• Audio codecs: All supported audio codecs (including all PCM variants)

input-format.ts:114-128

export class QuickTimeInputFormat extends IsobmffInputFormat {
  async _canReadInput(input: Input) {
    const majorBrand = await this._getMajorBrand(input);
    return majorBrand === 'qt  ';
  }

  get name() {
    return 'QuickTime File Format';
  }
}

MPEG-TS

MPEG Transport Stream - Designed for streaming and broadcast.

• Extension: .ts
• MIME Type: video/MP2T
• Best for: Live streaming, broadcast, network transmission
• Video codecs: H.264 (AVC), H.265 (HEVC)
• Audio codecs: AAC, MP3, AC-3, E-AC-3

input-format.ts:517-553

export class MpegTsInputFormat extends InputFormat {
  get name() {
    return 'MPEG Transport Stream';
  }

  get mimeType() {
    return 'video/MP2T';
  }
}

Audio-only containers

MP3

MPEG-1/2 Audio Layer 3 - The ubiquitous audio format.

• Extension: .mp3
• MIME Type: audio/mpeg
• Audio codec: MP3 only
• Track limit: 1 audio track

output-format.ts:589-648

export class Mp3OutputFormat extends OutputFormat {
  getSupportedCodecs(): MediaCodec[] {
    return ['mp3'];
  }

  get supportsTimestampedMediaData() {
    return false;
  }
}

WAV

Waveform Audio File Format - Uncompressed audio standard.

• Extension: .wav
• MIME Type: audio/wav
• Audio codecs: PCM variants (s16, s24, s32, f32, u8, μ-law, A-law)
• Track limit: 1 audio track

output-format.ts:684-750

export class WavOutputFormat extends OutputFormat {
  getSupportedCodecs(): MediaCodec[] {
    return [
      ...PCM_AUDIO_CODECS.filter(codec =>
        ['pcm-s16', 'pcm-s24', 'pcm-s32', 'pcm-f32', 'pcm-u8', 'ulaw', 'alaw'].includes(codec),
      ),
    ];
  }
}

FLAC

Free Lossless Audio Codec - Lossless compression for audio.

• Extension: .flac
• MIME Type: audio/flac
• Audio codec: FLAC only
• Track limit: 1 audio track

input-format.ts:412-434

export class FlacInputFormat extends InputFormat {
  get name() {
    return 'FLAC';
  }

  get mimeType() {
    return 'audio/flac';
  }
}

Ogg

Ogg - Open container format for free codecs.

• Extension: .ogg
• MIME Type: application/ogg
• Audio codecs: Vorbis, Opus
• Track limit: Multiple audio tracks supported

output-format.ts:779-845

export class OggOutputFormat extends OutputFormat {
  getSupportedCodecs(): MediaCodec[] {
    return [
      ...AUDIO_CODECS.filter(codec => ['vorbis', 'opus'].includes(codec)),
    ];
  }
}

ADTS

Audio Data Transport Stream - AAC elementary stream.

• Extension: .aac
• MIME Type: audio/aac
• Audio codec: AAC only
• Track limit: 1 audio track

input-format.ts:444-507

export class AdtsInputFormat extends InputFormat {
  get name() {
    return 'ADTS';
  }

  get mimeType() {
    return 'audio/aac';
  }
}

Supported codecs

Mediabunny supports 25+ codecs across video, audio, and subtitle categories.

Video codecs (5)

codec.ts:34-40

export const VIDEO_CODECS = [
  'avc',     // H.264/MPEG-4 AVC
  'hevc',    // H.265/HEVC
  'vp9',     // VP9
  'av1',     // AV1
  'vp8',     // VP8
] as const;

Video codec details

• AVC (H.264): Most widely supported, good compression, broad device compatibility
• HEVC (H.265): Better compression than H.264, 4K/8K video, newer devices
• VP9: Open-source, YouTube standard, good for web delivery
• AV1: Next-gen open codec, superior compression, growing support
• VP8: Predecessor to VP9, legacy web video

Audio codecs (19)

Audio codecs are divided into compressed and uncompressed (PCM) formats:

Compressed

export const NON_PCM_AUDIO_CODECS = [
  'aac',      // Advanced Audio Coding
  'opus',     // Opus
  'mp3',      // MPEG-1/2 Audio Layer 3
  'vorbis',   // Vorbis
  'flac',     // Free Lossless Audio Codec
  'ac3',      // Dolby Digital AC-3
  'eac3',     // Dolby Digital Plus E-AC-3
] as const;

Subtitle codecs (1)

codec.ts:90-92

export const SUBTITLE_CODECS = [
  'webvtt',   // WebVTT text tracks
] as const;

Format and codec compatibility

Not all codecs work with all formats. Use this compatibility matrix:

Format	Video Codecs	Audio Codecs	Subtitles
MP4	All video codecs	All non-PCM + some PCM	WebVTT
MOV	All video codecs	All audio codecs	No
WebM	VP8, VP9, AV1 only	Opus, Vorbis only	WebVTT
MKV	All video codecs	All non-PCM + most PCM	WebVTT
MPEG-TS	AVC, HEVC	AAC, MP3, AC-3, E-AC-3	No
MP3	No	MP3 only	No
WAV	No	PCM only (7 variants)	No
FLAC	No	FLAC only	No
Ogg	No	Vorbis, Opus	No
ADTS	No	AAC only	No

Checking compatibility

You can programmatically check if a codec is supported by a format:

import { Mp4OutputFormat, WebMOutputFormat } from 'mediabunny';

const mp4Format = new Mp4OutputFormat();
const webmFormat = new WebMOutputFormat();

// Check video codec support
const mp4VideoCodecs = mp4Format.getSupportedVideoCodecs();
console.log(mp4VideoCodecs); // ['avc', 'hevc', 'vp9', 'av1', 'vp8']

const webmVideoCodecs = webmFormat.getSupportedVideoCodecs();
console.log(webmVideoCodecs); // ['vp8', 'vp9', 'av1']

// Check audio codec support
const mp4AudioCodecs = mp4Format.getSupportedAudioCodecs();
const webmAudioCodecs = webmFormat.getSupportedAudioCodecs();

// Check subtitle support
const mp4Subtitles = mp4Format.getSupportedSubtitleCodecs();

Error handling for incompatible codecs

Mediabunny will throw an error if you try to use an incompatible codec:

import { Output, WebMOutputFormat, BufferTarget, VideoPacketSource } from 'mediabunny';

const output = new Output({
  format: new WebMOutputFormat(),
  target: new BufferTarget()
});

const videoSource = new VideoPacketSource({
  codec: 'avc', // H.264 - NOT supported in WebM!
  width: 1920,
  height: 1080
});

try {
  output.addVideoTrack(videoSource);
} catch (error) {
  console.error(error.message);
  // "Codec 'avc' cannot be contained within WebM. 
  //  Supported video codecs are: 'vp8', 'vp9', 'av1'. 
  //  Switching to MKV will grant support for this codec."
}

Mediabunny provides helpful error messages suggesting alternative formats when codec compatibility issues arise.

Choosing the right format and codec

Consider these factors when selecting formats and codecs:

Compatibility

For maximum compatibility across devices and platforms, use MP4 with H.264 video and AAC audio.

Web Delivery

For web streaming, use WebM (VP9/Opus) or MP4 (H.264/AAC). Modern browsers support both.

Quality

For high quality with efficient compression, use HEVC, VP9, or AV1 video codecs.

Open Source

For open-source projects avoiding patent issues, use WebM, MKV, or Ogg with open codecs.

Next steps

Input and Output

Learn how to read and write media files

Sources and Targets

Understand different ways to read and write data