Media sinks

Different media sink types and when to use them

Media sinks provide APIs for extracting media data from input files. Like media sources, sinks come at different abstraction levels, letting you choose between convenience and control.

Overview of media sinks

Media sinks can be organized into three abstraction levels:

High-level sinks

CanvasSink, AudioBufferSink

Easy to use, provides processed output ready for display/playback.

Mid-level sinks

VideoSampleSink, AudioSampleSink

Access to decoded samples, handles decoding internally.

Low-level sinks

EncodedPacketSink

Direct packet access, you handle decoding.

When to use each sink

EncodedPacketSink

Best for: Metadata extraction, remuxing without decoding, custom decoding pipelines

EncodedPacketSink provides direct access to encoded packets without decoding. Use this when you only need packet metadata or want to implement custom decoding.

src/media-sink.ts

const sink = new EncodedPacketSink(videoTrack);

// Get packet at specific timestamp
const packet = await sink.getPacket(5.0);

// Get key frame at timestamp
const keyPacket = await sink.getKeyPacket(5.0);

// Iterate over all packets
for await (const packet of sink.packets()) {
  console.log(packet.timestamp, packet.type);
}

Advantages:

• No decoding overhead
• Access to packet metadata
• Perfect for remuxing operations
• Fast iteration over packet structure

Source code: src/media-sink.ts:120-375

VideoSampleSink

Best for: Frame-by-frame processing, video analysis, custom rendering

VideoSampleSink decodes video packets into raw VideoFrames, giving you access to decoded pixel data.

src/media-sink.ts

const sink = new VideoSampleSink(videoTrack);

// Get frame at specific timestamp
const sample = await sink.getSample(5.0);

// Iterate over all frames
for await (const sample of sink.samples()) {
  // Process the VideoFrame
  console.log(sample.codedWidth, sample.codedHeight);
  sample.close(); // Don't forget to close!
}

// Get frames at specific timestamps (efficient)
for await (const sample of sink.samplesAtTimestamps([0, 1, 2, 3, 4])) {
  // Process sample
  sample?.close();
}

Advantages:

• Direct access to decoded frames
• Integration with VideoFrame API
• Efficient sparse sampling
• Automatic decoding pipeline

Important: Always call close() on VideoSamples when done to free memory.

Source code: src/media-sink.ts:1361-1440

CanvasSink

Best for: Thumbnail generation, video preview, frame export, display in browser

CanvasSink provides the most convenient way to extract video frames as canvases, with built-in support for resizing, rotation, and cropping.

src/media-sink.ts

const sink = new CanvasSink(videoTrack, {
  width: 1280,
  height: 720,
  fit: 'contain',
  rotation: 90,
  poolSize: 3, // Reuse canvases for efficiency
});

// Get canvas at timestamp
const { canvas, timestamp, duration } = await sink.getCanvas(5.0);

// Iterate over frames
for await (const { canvas, timestamp } of sink.canvases()) {
  // Canvas is ready to display or export
  document.body.appendChild(canvas);
}

Advantages:

• Ready-to-display canvases
• Built-in resizing, rotation, cropping
• Canvas pooling for memory efficiency
• Perfect for thumbnails and previews

Source code: src/media-sink.ts:1516-1719

AudioSampleSink

Best for: Audio analysis, waveform generation, custom audio processing

AudioSampleSink decodes audio packets into raw AudioData, giving you access to decoded audio samples.

src/media-sink.ts

const sink = new AudioSampleSink(audioTrack);

// Get sample at timestamp
const sample = await sink.getSample(5.0);

// Iterate over all samples
for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Process audio data
  sample.close();
}

Advantages:

• Access to raw audio samples
• Integration with AudioData API
• Precise sample-level control
• Automatic decoding

Source code: src/media-sink.ts:2034-2092

AudioBufferSink

Best for: Web Audio API integration, playback, audio processing with Web Audio

AudioBufferSink provides decoded audio as AudioBuffers, ready for use with the Web Audio API.

src/media-sink.ts

const sink = new AudioBufferSink(audioTrack);
const audioContext = new AudioContext();

// Play audio from timestamp
for await (const { buffer, timestamp } of sink.buffers(10.0, 20.0)) {
  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  source.start(audioContext.currentTime + timestamp - 10.0);
}

Advantages:

• Direct AudioBuffer support
• Perfect for Web Audio API
• Ready for playback
• Easy audio processing

Source code: src/media-sink.ts:2094+

Advanced usage patterns

Efficient sparse sampling

When you need samples at specific timestamps, use samplesAtTimestamps instead of multiple getSample calls:

// Inefficient - decodes same packets multiple times
const sample1 = await sink.getSample(1.0);
const sample2 = await sink.getSample(2.0);
const sample3 = await sink.getSample(3.0);

// Efficient - optimized decode pipeline
for await (const sample of sink.samplesAtTimestamps([1.0, 2.0, 3.0])) {
  // Process sample
  sample?.close();
}

Generating thumbnails

Generate evenly-spaced thumbnails efficiently:

const sink = new CanvasSink(videoTrack, {
  width: 160,
  height: 90,
  fit: 'cover',
  poolSize: 1, // Only need one canvas at a time
});

const duration = await videoTrack.computeDuration();
const thumbnailCount = 10;
const timestamps = Array.from(
  { length: thumbnailCount },
  (_, i) => (duration * i) / (thumbnailCount - 1)
);

const thumbnails = [];
for await (const { canvas } of sink.canvasesAtTimestamps(timestamps)) {
  // Export canvas to blob
  const blob = await new Promise(resolve => 
    canvas.toBlob(resolve, 'image/jpeg', 0.9)
  );
  thumbnails.push(blob);
}

Extracting key frames only

Combine EncodedPacketSink with VideoSampleSink to extract only key frames:

const packetSink = new EncodedPacketSink(videoTrack);
const sampleSink = new VideoSampleSink(videoTrack);

const keyFrameTimestamps = (async function* () {
  let packet = await packetSink.getFirstPacket();
  
  while (packet) {
    if (packet.type === 'key') {
      yield packet.timestamp;
    }
    packet = await packetSink.getNextPacket(packet);
  }
})();

for await (const sample of sampleSink.samplesAtTimestamps(keyFrameTimestamps)) {
  // Process only key frames
  sample?.close();
}

Audio waveform generation

Generate a waveform visualization:

const sink = new AudioSampleSink(audioTrack);
const waveformData = [];

for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Calculate RMS for this chunk
  let sum = 0;
  for (let i = 0; i < floats.length; i++) {
    sum += floats[i] ** 2;
  }
  const rms = Math.sqrt(sum / floats.length);
  waveformData.push(rms);
  
  sample.close();
}

Range iteration with break

Exit iteration early while ensuring proper cleanup:

let frameCount = 0;
for await (const sample of sink.samples()) {
  // Process frame
  sample.close();
  
  // Stop after 100 frames
  if (++frameCount >= 100) {
    break; // Automatically cleans up decoder
  }
}

Canvas pool optimization

Use canvas pooling to minimize memory allocation:

No pool (default)

// Creates new canvas for each frame - can be slow
const sink = new CanvasSink(videoTrack);

for await (const { canvas } of sink.canvases()) {
  // New canvas every iteration
}

With pool

// Reuses canvases from pool - much faster
const sink = new CanvasSink(videoTrack, { poolSize: 3 });

for await (const { canvas } of sink.canvases()) {
  // Canvas reused from pool
}

For sequential iteration, poolSize: 1 is sufficient and optimal.

Verifying key packets

Some files incorrectly mark packet types. Verify key packets to ensure decoder compatibility:

const sink = new EncodedPacketSink(videoTrack);

// Without verification (faster, but may be incorrect)
const packet1 = await sink.getKeyPacket(5.0);

// With verification (slower, but guaranteed correct)
const packet2 = await sink.getKeyPacket(5.0, { verifyKeyPackets: true });

Metadata-only packet retrieval

When you only need packet metadata, avoid loading packet data:

const sink = new EncodedPacketSink(videoTrack);

// Get metadata without loading packet data
const packet = await sink.getPacket(5.0, { metadataOnly: true });

console.log(packet.timestamp); // Available
console.log(packet.type);      // Available
console.log(packet.data);      // Empty Uint8Array

Decode vs. presentation order

Understanding the difference between decode and presentation order is crucial:

• Presentation order: The order in which frames are displayed (sorted by timestamp)
• Decode order: The order in which packets must be decoded (may differ due to B-frames)

B-frames example

Consider frames with B-frames:

Presentation order: Frame1(0.0s) → Frame2(0.1s) → Frame3(0.2s)
Decode order:       Frame1(0.0s) → Frame3(0.2s) → Frame2(0.1s)

EncodedPacketSink methods:

• packets() - Returns packets in decode order
• getPacket(timestamp) - Searches by presentation timestamp

VideoSampleSink and CanvasSink methods:

• All methods use presentation order

Best practices

1. Choose the right abstraction

   Use high-level sinks (CanvasSink, AudioBufferSink) unless you need sample-level control.

2. Close samples and frames

   Always call close() on VideoSamples and AudioSamples to prevent memory leaks.

3. Use sparse sampling wisely

   Use samplesAtTimestamps() instead of multiple getSample() calls for efficiency.

4. Enable canvas pooling

   Use poolSize option in CanvasSink to reduce memory allocation overhead.

5. Break early when needed

   Use break in for-await loops to exit early while ensuring proper cleanup.

6. Verify key packets when needed

   Enable verifyKeyPackets if you encounter decoder errors with key frames.

Performance considerations

Memory management

• Always close VideoSamples and AudioSamples
• Use canvas pooling for CanvasSink
• Use metadata-only packets when possible
• Break out of iterations early if possible

Decoding efficiency

• Use sparse sampling for non-sequential access
• Prefer range iteration for sequential access
• Use EncodedPacketSink to skip decoding entirely
• Consider decoder queue sizes

See also

• Reading media files - Complete guide to reading input files
• Packets and samples - Understanding media data structures
• Custom coders - Implementing custom decoders