> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/Vanilagy/mediabunny/llms.txt
> Use this file to discover all available pages before exploring further.

# Media sinks

> Different media sink types and when to use them

Media sinks provide APIs for extracting media data from input files. Like media sources, sinks come at different abstraction levels, letting you choose between convenience and control.

## Overview of media sinks

Media sinks can be organized into three abstraction levels:

<CardGroup cols={3}>
  <Card title="High-level sinks" icon="layer-group">
    `CanvasSink`, `AudioBufferSink`

    Easy to use, provides processed output ready for display/playback.
  </Card>

  <Card title="Mid-level sinks" icon="code">
    `VideoSampleSink`, `AudioSampleSink`

    Access to decoded samples, handles decoding internally.
  </Card>

  <Card title="Low-level sinks" icon="gear">
    `EncodedPacketSink`

    Direct packet access, you handle decoding.
  </Card>
</CardGroup>

## When to use each sink

### EncodedPacketSink

**Best for:** Metadata extraction, remuxing without decoding, custom decoding pipelines

`EncodedPacketSink` provides direct access to encoded packets without decoding. Use this when you only need packet metadata or want to implement custom decoding.

```typescript src/media-sink.ts theme={null}
const sink = new EncodedPacketSink(videoTrack);

// Get packet at specific timestamp
const packet = await sink.getPacket(5.0);

// Get key frame at timestamp
const keyPacket = await sink.getKeyPacket(5.0);

// Iterate over all packets
for await (const packet of sink.packets()) {
  console.log(packet.timestamp, packet.type);
}
```

**Advantages:**

* No decoding overhead
* Access to packet metadata
* Perfect for remuxing operations
* Fast iteration over packet structure

**Source code:** [src/media-sink.ts:120-375](/home/daytona/workspace/source/src/media-sink.ts#L120-L375)

### VideoSampleSink

**Best for:** Frame-by-frame processing, video analysis, custom rendering

`VideoSampleSink` decodes video packets into raw VideoFrames, giving you access to decoded pixel data.

```typescript src/media-sink.ts theme={null}
const sink = new VideoSampleSink(videoTrack);

// Get frame at specific timestamp
const sample = await sink.getSample(5.0);

// Iterate over all frames
for await (const sample of sink.samples()) {
  // Process the VideoFrame
  console.log(sample.codedWidth, sample.codedHeight);
  sample.close(); // Don't forget to close!
}

// Get frames at specific timestamps (efficient)
for await (const sample of sink.samplesAtTimestamps([0, 1, 2, 3, 4])) {
  // Process sample
  sample?.close();
}
```

**Advantages:**

* Direct access to decoded frames
* Integration with VideoFrame API
* Efficient sparse sampling
* Automatic decoding pipeline

**Important:** Always call `close()` on VideoSamples when done to free memory.

**Source code:** [src/media-sink.ts:1361-1440](/home/daytona/workspace/source/src/media-sink.ts#L1361-L1440)

### CanvasSink

**Best for:** Thumbnail generation, video preview, frame export, display in browser

`CanvasSink` provides the most convenient way to extract video frames as canvases, with built-in support for resizing, rotation, and cropping.

```typescript src/media-sink.ts theme={null}
const sink = new CanvasSink(videoTrack, {
  width: 1280,
  height: 720,
  fit: 'contain',
  rotation: 90,
  poolSize: 3, // Reuse canvases for efficiency
});

// Get canvas at timestamp
const { canvas, timestamp, duration } = await sink.getCanvas(5.0);

// Iterate over frames
for await (const { canvas, timestamp } of sink.canvases()) {
  // Canvas is ready to display or export
  document.body.appendChild(canvas);
}
```

**Advantages:**

* Ready-to-display canvases
* Built-in resizing, rotation, cropping
* Canvas pooling for memory efficiency
* Perfect for thumbnails and previews

**Source code:** [src/media-sink.ts:1516-1719](/home/daytona/workspace/source/src/media-sink.ts#L1516-L1719)

### AudioSampleSink

**Best for:** Audio analysis, waveform generation, custom audio processing

`AudioSampleSink` decodes audio packets into raw AudioData, giving you access to decoded audio samples.

```typescript src/media-sink.ts theme={null}
const sink = new AudioSampleSink(audioTrack);

// Get sample at timestamp
const sample = await sink.getSample(5.0);

// Iterate over all samples
for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Process audio data
  sample.close();
}
```

**Advantages:**

* Access to raw audio samples
* Integration with AudioData API
* Precise sample-level control
* Automatic decoding

**Source code:** [src/media-sink.ts:2034-2092](/home/daytona/workspace/source/src/media-sink.ts#L2034-L2092)

### AudioBufferSink

**Best for:** Web Audio API integration, playback, audio processing with Web Audio

`AudioBufferSink` provides decoded audio as AudioBuffers, ready for use with the Web Audio API.

```typescript src/media-sink.ts theme={null}
const sink = new AudioBufferSink(audioTrack);
const audioContext = new AudioContext();

// Play audio from timestamp
for await (const { buffer, timestamp } of sink.buffers(10.0, 20.0)) {
  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  source.start(audioContext.currentTime + timestamp - 10.0);
}
```

**Advantages:**

* Direct AudioBuffer support
* Perfect for Web Audio API
* Ready for playback
* Easy audio processing

**Source code:** [src/media-sink.ts:2094+](/home/daytona/workspace/source/src/media-sink.ts#L2094)

## Advanced usage patterns

### Efficient sparse sampling

When you need samples at specific timestamps, use `samplesAtTimestamps` instead of multiple `getSample` calls:

```typescript theme={null}
// Inefficient - decodes same packets multiple times
const sample1 = await sink.getSample(1.0);
const sample2 = await sink.getSample(2.0);
const sample3 = await sink.getSample(3.0);

// Efficient - optimized decode pipeline
for await (const sample of sink.samplesAtTimestamps([1.0, 2.0, 3.0])) {
  // Process sample
  sample?.close();
}
```

### Generating thumbnails

Generate evenly-spaced thumbnails efficiently:

```typescript theme={null}
const sink = new CanvasSink(videoTrack, {
  width: 160,
  height: 90,
  fit: 'cover',
  poolSize: 1, // Only need one canvas at a time
});

const duration = await videoTrack.computeDuration();
const thumbnailCount = 10;
const timestamps = Array.from(
  { length: thumbnailCount },
  (_, i) => (duration * i) / (thumbnailCount - 1)
);

const thumbnails = [];
for await (const { canvas } of sink.canvasesAtTimestamps(timestamps)) {
  // Export canvas to blob
  const blob = await new Promise(resolve => 
    canvas.toBlob(resolve, 'image/jpeg', 0.9)
  );
  thumbnails.push(blob);
}
```

### Extracting key frames only

Combine `EncodedPacketSink` with `VideoSampleSink` to extract only key frames:

```typescript theme={null}
const packetSink = new EncodedPacketSink(videoTrack);
const sampleSink = new VideoSampleSink(videoTrack);

const keyFrameTimestamps = (async function* () {
  let packet = await packetSink.getFirstPacket();
  
  while (packet) {
    if (packet.type === 'key') {
      yield packet.timestamp;
    }
    packet = await packetSink.getNextPacket(packet);
  }
})();

for await (const sample of sampleSink.samplesAtTimestamps(keyFrameTimestamps)) {
  // Process only key frames
  sample?.close();
}
```

### Audio waveform generation

Generate a waveform visualization:

```typescript theme={null}
const sink = new AudioSampleSink(audioTrack);
const waveformData = [];

for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Calculate RMS for this chunk
  let sum = 0;
  for (let i = 0; i < floats.length; i++) {
    sum += floats[i] ** 2;
  }
  const rms = Math.sqrt(sum / floats.length);
  waveformData.push(rms);
  
  sample.close();
}
```

### Range iteration with break

Exit iteration early while ensuring proper cleanup:

```typescript theme={null}
let frameCount = 0;
for await (const sample of sink.samples()) {
  // Process frame
  sample.close();
  
  // Stop after 100 frames
  if (++frameCount >= 100) {
    break; // Automatically cleans up decoder
  }
}
```

### Canvas pool optimization

Use canvas pooling to minimize memory allocation:

<Tabs>
  <Tab title="No pool (default)">
    ```typescript theme={null}
    // Creates new canvas for each frame - can be slow
    const sink = new CanvasSink(videoTrack);

    for await (const { canvas } of sink.canvases()) {
      // New canvas every iteration
    }
    ```
  </Tab>

  <Tab title="With pool">
    ```typescript theme={null}
    // Reuses canvases from pool - much faster
    const sink = new CanvasSink(videoTrack, { poolSize: 3 });

    for await (const { canvas } of sink.canvases()) {
      // Canvas reused from pool
    }
    ```
  </Tab>
</Tabs>

<Note>
  For sequential iteration, `poolSize: 1` is sufficient and optimal.
</Note>

### Verifying key packets

Some files incorrectly mark packet types. Verify key packets to ensure decoder compatibility:

```typescript theme={null}
const sink = new EncodedPacketSink(videoTrack);

// Without verification (faster, but may be incorrect)
const packet1 = await sink.getKeyPacket(5.0);

// With verification (slower, but guaranteed correct)
const packet2 = await sink.getKeyPacket(5.0, { verifyKeyPackets: true });
```

### Metadata-only packet retrieval

When you only need packet metadata, avoid loading packet data:

```typescript theme={null}
const sink = new EncodedPacketSink(videoTrack);

// Get metadata without loading packet data
const packet = await sink.getPacket(5.0, { metadataOnly: true });

console.log(packet.timestamp); // Available
console.log(packet.type);      // Available
console.log(packet.data);      // Empty Uint8Array
```

## Decode vs. presentation order

Understanding the difference between decode and presentation order is crucial:

* **Presentation order:** The order in which frames are displayed (sorted by timestamp)
* **Decode order:** The order in which packets must be decoded (may differ due to B-frames)

<Accordion title="B-frames example">
  Consider frames with B-frames:

  ```
  Presentation order: Frame1(0.0s) → Frame2(0.1s) → Frame3(0.2s)
  Decode order:       Frame1(0.0s) → Frame3(0.2s) → Frame2(0.1s)
  ```

  `EncodedPacketSink` methods:

  * `packets()` - Returns packets in **decode order**
  * `getPacket(timestamp)` - Searches by **presentation timestamp**

  `VideoSampleSink` and `CanvasSink` methods:

  * All methods use **presentation order**
</Accordion>

## Best practices

<Steps>
  <Step title="Choose the right abstraction">
    Use high-level sinks (CanvasSink, AudioBufferSink) unless you need sample-level control.
  </Step>

  <Step title="Close samples and frames">
    Always call `close()` on VideoSamples and AudioSamples to prevent memory leaks.
  </Step>

  <Step title="Use sparse sampling wisely">
    Use `samplesAtTimestamps()` instead of multiple `getSample()` calls for efficiency.
  </Step>

  <Step title="Enable canvas pooling">
    Use `poolSize` option in CanvasSink to reduce memory allocation overhead.
  </Step>

  <Step title="Break early when needed">
    Use `break` in for-await loops to exit early while ensuring proper cleanup.
  </Step>

  <Step title="Verify key packets when needed">
    Enable `verifyKeyPackets` if you encounter decoder errors with key frames.
  </Step>
</Steps>

## Performance considerations

<CardGroup cols={2}>
  <Card title="Memory management" icon="memory">
    * Always close VideoSamples and AudioSamples
    * Use canvas pooling for CanvasSink
    * Use metadata-only packets when possible
    * Break out of iterations early if possible
  </Card>

  <Card title="Decoding efficiency" icon="gauge">
    * Use sparse sampling for non-sequential access
    * Prefer range iteration for sequential access
    * Use EncodedPacketSink to skip decoding entirely
    * Consider decoder queue sizes
  </Card>
</CardGroup>

## See also

* [Reading media files](/guides/reading-files) - Complete guide to reading input files
* [Packets and samples](/concepts/packets-and-samples) - Understanding media data structures
* [Custom coders](/advanced/custom-coders) - Implementing custom decoders