VideoPlayer Class Reference

High-level video playback pipeline and controller - the video counterpart of AudioPlayer (CoreAudio/AudioPlayer.h). Wraps a container Demuxer, a VideoDecoder driven through a PacedVideoOutput, and optionally an audio decode chain synced against it, behind one object driven by a single copy() call per loop() iteration. More...

#include <VideoPlayer.h>

Inheritance diagram for VideoPlayer:

Public Member Functions
	VideoPlayer ()=default
	VideoPlayer (Demuxer &demuxer, VideoOutput &videoOutput)
	VideoPlayer (Demuxer &demuxer, VideoOutput &videoOutput, AudioOutput &audioOutput)
	VideoPlayer (Demuxer &demuxer, VideoOutput &videoOutput, AudioStream &audioOutput)
	VideoPlayer (Demuxer &demuxer, VideoOutput &videoOutput, Print &audioOutput)
	VideoPlayer (VideoPlayer const &)=delete
void	addAudioDecoder (AudioDecoder &decoder, const char *mime=nullptr)
void	addDemuxer (Demuxer &demuxer)
void	addVideoDecoder (VideoDecoder &decoder)
MultiDecoder &	audioDecoder ()
bool	begin (Stream &source)
size_t	copy ()
size_t	copyAll ()
	Copies until the source is exhausted (blocking).
MultiVideoDemuxer &	demuxer ()
void	end ()
Stream *	getStream ()
	The Stream given to begin() - nullptr before the first begin() call.
bool	isActive () const
void	logTo (Print &out)
	operator bool () const
	Same as isActive().
VideoPlayer &	operator= (VideoPlayer const &)=delete
void	setActive (bool isActive)
void	setAudioOutput (AudioOutput &out)
void	setAudioOutput (AudioStream &out)
void	setAudioOutput (Print &out)
void	setBufferSize (int size)
void	setCatchUpThresholdFrames (float frames)
void	setIgnorePFrames (bool active)
void	setMaxQueuedIFrames (int count)
void	setQueueBytes (size_t bytes)
void	setQueueUsePSRAM (bool flag)
void	setResyncQueueFillFraction (float fraction)
void	setResyncThresholdMs (uint32_t ms)
void	setSchedulingDelayMs (uint32_t delayMs)
void	setTaskParameters (uint32_t stackSizeWords, uint8_t priority, int core=-1)
void	setUseAudioClock (bool active)
void	setVideoOutput (VideoOutput &out)
bool	useAudioClock () const
	Current setUseAudioClock() setting.
MultiVideoDecoder &	videoDecoder ()
PacedVideoOutput &	videoSyncTask ()

Protected Attributes
bool	active = false
AudioTimeSourceStream	audio_clock
EncodedAudioStream	audio_out
int	buffer_size = 1024
MultiDecoder	default_audio_decoder
MultiVideoDemuxer	default_demuxer
MultiVideoDecoder	default_video_decoder
Print *	p_audio_output = nullptr
AudioOutput *	p_audio_output_typed = nullptr
AudioStream *	p_audio_stream = nullptr
Stream *	p_source = nullptr
VideoOutput *	p_video_output = nullptr
bool	use_audio_clock = true
PacedVideoOutput	video_sync {default_video_decoder}

Detailed Description

High-level video playback pipeline and controller - the video counterpart of AudioPlayer (CoreAudio/AudioPlayer.h). Wraps a container Demuxer, a VideoDecoder driven through a PacedVideoOutput, and optionally an audio decode chain synced against it, behind one object driven by a single copy() call per loop() iteration.

Nothing is pre-registered (no container/codec library dependency by default) - register what your content needs:

DemuxerAVI aviDemuxer;
VideoPlayer player(aviDemuxer, tftOutput, audioOut);
player.addVideoDecoder(h264Decoder);
player.addAudioDecoder(mp3Decoder, "audio/mpeg");
player.begin(file);
// loop(): if (player.copy() == 0) { file.close(); ... }

Use VideoPlayerFull (VideoPlayerFull.h) instead to pre-register every common container/video/audio codec at once, at the cost of pulling in all of their libraries unconditionally.

Pipeline: Stream -> copy() -> Demuxer -> [audio: EncodedAudioStream -> audio output] / [video: PacedVideoOutput -> VideoDecoder -> video output, own background task].

See the wiki's Video Playback page for the full picture: audio-clock scheduling (setUseAudioClock()), running video decode on its own core (setTaskParameters()), the seek-backed/spooled MP4 exception, and more.

Author

Phil Schatzmann

Copyright

GPLv3

Constructor & Destructor Documentation

VideoPlayer() [1/6]

VideoPlayer ( )

default

VideoPlayer() [2/6]

VideoPlayer ( Demuxer &  demuxer, VideoOutput &  videoOutput  )

inline

Video-only playback (no audio track) - auto-detected video codec. demuxer is registered the same way an addDemuxer() call would (this is just a convenience for the common single-format case - call addDemuxer() separately, as many times as needed, for additional container formats).

VideoPlayer() [3/6]

VideoPlayer ( Demuxer &  demuxer, VideoOutput &  videoOutput, AudioOutput &  audioOutput  )

inline

Video + audio playback (AudioOutput target, e.g. AudioBoardStream/ I2SStream) with auto-detected codecs. By default the video frame schedule is driven off this audio output's own playback progress (an "audio clock") rather than a free-running timer, so video stays in sync with audio even as the two decode at slightly different rates; call setUseAudioClock(false) if the content's audio track never actually delivers bytes (e.g. a silent/empty track), since the clock would then never advance and video would never play. demuxer is registered the same way as the video-only constructor above.

VideoPlayer() [4/6]

VideoPlayer ( Demuxer &  demuxer, VideoOutput &  videoOutput, Print &  audioOutput  )

inline

Video + audio playback (a generic Print target) with auto-detected codecs - same audio-clock-driven scheduling as the AudioOutput overload above, and demuxer is registered the same way as the video-only constructor.

VideoPlayer() [5/6]

VideoPlayer ( Demuxer &  demuxer, VideoOutput &  videoOutput, AudioStream &  audioOutput  )

inline

Video + audio playback (AudioStream target, e.g. PortAudioStream) with auto-detected codecs - same audio-clock-driven scheduling as the AudioOutput overload above, and demuxer is registered the same way as the video-only constructor.

VideoPlayer() [6/6]

VideoPlayer ( VideoPlayer const &  )

delete

Non-copyable: video_sync/audio_out below are wired against this object's own member addresses - copying would leave the copy's internal wiring pointing at the original.

Member Function Documentation

addAudioDecoder()

void addAudioDecoder ( AudioDecoder &  decoder, const char *  mime = nullptr  )

inline

Registers an audio codec decoder (MP3DecoderHelix/AACDecoderHelix/...) with the built-in MultiDecoder, keyed by the mime string given here (or the decoder's own default mime type if mime is null). The built-in decoder selector starts out completely empty, so this needs at least one call per audio codec your content actually uses before begin() is called - call it multiple times to support more than one codec.

addDemuxer()

void addDemuxer ( Demuxer &  demuxer )

inline

Registers a container demuxer implementation (DemuxerAVI/DemuxerMP4/ DemuxerMPG/...) with the built-in MultiVideoDemuxer, keyed under the demuxer's own Demuxer::mimeVideo(). The built-in demuxer selector starts out completely empty, so this needs at least one call per container format your content actually uses before begin() is called

• call it multiple times to support more than one format.

addVideoDecoder()

void addVideoDecoder ( VideoDecoder &  decoder )

inline

Registers a video codec decoder (H264Decoder/MJPEGDecoder/MPGDecoder/ ...) with the built-in MultiVideoDecoder, keyed under the decoder's own VideoDecoder::codecFormat(). The built-in decoder selector starts out completely empty, so this needs at least one call per video codec your content actually uses before begin() is called - call it multiple times to support more than one codec.

audioDecoder()

MultiDecoder & audioDecoder ( )

inline

The built-in audio multi-decoder, empty until addAudioDecoder() has registered at least one audio codec decoder - also useful for diagnostics (e.g. selectedMime()).

begin()

bool begin ( Stream &  source )

inline

Wires the full pipeline (video decoder -> output via PacedVideoOutput; audio decoder -> [audio clock ->] output, if an audio output was given) and starts the demuxer. source (e.g. an open File) is read from on every subsequent copy() call - the caller owns it and is responsible for closing it once copy()/copyAll() signal end of stream (return 0).

copy()

size_t copy ( )

inline

Reads and demuxes one chunk (setBufferSize(), default 1024 bytes) from the source given to begin() - call this every loop() iteration. Also keeps the video schedule's fps in sync with the demuxer's own parsed rate, since VideoInfo::fps only becomes known partway through (once the container's header has been parsed) - no separate polling needed in the caller's loop(), unlike the *.ino examples this class replaces.

Returns

bytes actually copied - 0 once the source is exhausted (or playback is inactive/not begin()'d), matching CodecCopy::copy()'s own "0 means stop" convention every current *.ino example already checks for.

copyAll()

size_t copyAll ( )

inline

Copies until the source is exhausted (blocking).

demuxer()

MultiVideoDemuxer & demuxer ( )

inline

The built-in demuxer multi-selector, empty until addDemuxer() has registered at least one container demuxer - also useful for diagnostics (e.g. selectedDemuxer()/mimeVideo()).

end()

void end ( )

inline

Stops playback and releases the video/audio decoders + the sync task's background render thread. The Stream given to begin() is left for the caller to close.

getStream()

Stream * getStream ( )

inline

The Stream given to begin() - nullptr before the first begin() call.

isActive()

bool isActive ( ) const

inline

True between a successful begin() and end()/setActive(false)/source exhaustion.

logTo()

void logTo ( Print &  out )

inline

Logs periodic playback diagnostics (frame count, average frame time, output fps, ...) to out - convenience shortcut for videoSyncTask().logTo(out).

operator bool()

operator bool ( ) const

inline

Same as isActive().

operator=()

VideoPlayer & operator= ( VideoPlayer const &  )

delete

setActive()

void setActive ( bool  isActive )

inline

Halts copy()/copyAll() without tearing down the pipeline (unlike end()) - resume with setActive(true).

setAudioOutput() [1/3]

void setAudioOutput ( AudioOutput &  out )

inline

setAudioOutput() [2/3]

void setAudioOutput ( AudioStream &  out )

inline

setAudioOutput() [3/3]

void setAudioOutput ( Print &  out )

inline

Defines the final audio output target and enables the audio playback path - leave unset entirely for video-only content. The caller must already have configured/begin()'d it. Call before begin(). Also enables audio-clock-driven video scheduling by default (see setUseAudioClock()) - call setUseAudioClock(false) afterwards to turn that off.

Three overloads, mirroring AudioPlayer::setOutput(): AudioOutput and AudioStream are unrelated types (both derive from Print, but neither derives from the other), so the most specific one actually available should be used. Using the specific overload lets begin() wire the audio clock/EncodedAudioStream through the matching setOutput(AudioOutput&)/setStream(AudioStream&) overload instead of falling back to the generic Print& one - that's what makes things like audio-info change notifications reach the real output correctly.

setBufferSize()

void setBufferSize ( int  size )

inline

Sets the read-buffer size (bytes) copy() uses per call (default 1024, same as CodecCopy's own default). Call before begin().

setCatchUpThresholdFrames()

void setCatchUpThresholdFrames ( float  frames )

inline

How many frame periods behind schedule playback must fall before non-keyframes start being proactively dropped to catch up, instead of only dropping once the frame queue is completely full. 1.0 (the default) starts dropping once the most recently rendered frame was at least one frame period late; a lower value catches up faster at the cost of more dropped frames, a higher value tolerates more backlog before reacting. Forwards to the built-in PacedVideoOutput; call before begin().

setIgnorePFrames()

void setIgnorePFrames ( bool  active )

inline

Unconditionally discards every non-keyframe (P/B-frame) as it arrives, before it is even queued for decode - a codec-agnostic way to render only keyframes, cheaper than letting the decoder decode and then discard each one. Off by default. Independent of the catch-up/backlog-driven dropping described above, which still applies when this is off. Forwards to the built-in PacedVideoOutput; call before begin().

setMaxQueuedIFrames()

void setMaxQueuedIFrames ( int  count )

inline

Number of not-yet-consumed keyframes sitting in the queue that triggers the same resync as the other two thresholds - catches a growing backlog earlier than either: once several groups-of-pictures worth of unconsumed keyframes have piled up, none of them are worth rendering in order anymore, so the freshest one is kept and the rest discarded. 3 by default (deliberately small); 0 disables this trigger. Forwards to the built-in PacedVideoOutput; call before begin().

setQueueBytes()

void setQueueBytes ( size_t  bytes )

inline

Byte capacity of the frame queue sitting between the demuxer/decoder and the paced render task - default 32KB. Too small behaves like a 1-frame queue, where any single slow frame blocks the writer immediately; a larger queue absorbs transient jitter at the cost of more RAM and more frames buffered ahead of what's actually on screen. It does not change what happens under a sustained rate mismatch - the queue still eventually fills, just later. Forwards to the built-in PacedVideoOutput; call before begin()/the first frame - not supported afterwards.

setQueueUsePSRAM()

void setQueueUsePSRAM ( bool  flag )

inline

Whether the frame queue above is allocated from PSRAM instead of the internal heap - silently falls back to internal heap on boards without PSRAM. On by default, since internal heap is a scarcer shared resource on most ESP32 boards and video decoding already needs PSRAM for other buffers (decoded picture buffers, scaling scratch space, ...). Turn it off for a queue small enough that internal heap is preferable, or on a board with no PSRAM where the fallback path is undesired. Forwards to the built-in PacedVideoOutput; call before begin()/the first frame - has no effect on an already-allocated queue.

setResyncQueueFillFraction()

void setResyncQueueFillFraction ( float  fraction )

inline

Byte-occupancy fraction (0..1) of the internal frame queue that triggers the same forward-jump resync as setResyncThresholdMs(), but from a different signal: dropped frames can keep the most recently rendered frame's own lateness low even while newer, not-yet-rendered bytes keep piling up faster than they're consumed. 0.8 (80% full) by default; 0 disables this trigger. Forwards to the built-in PacedVideoOutput; call before begin().

setResyncThresholdMs()

void setResyncThresholdMs ( uint32_t  ms )

inline

How far behind schedule (milliseconds) playback must fall before it gives up trying to catch up frame-by-frame and instead jumps its internal schedule forward to the current time - dropping frames alone can only slow a backlog's growth, never actually shrink it, since decoding can't outrun real time. Content in the skipped gap is never shown - this trades a visible jump for recovering instead of lagging further and further behind. 2000ms by default; 0 disables this resync behavior entirely. Forwards to the built-in PacedVideoOutput; call before begin().

setSchedulingDelayMs()

void setSchedulingDelayMs ( uint32_t  delayMs )

inline

Corrects for the real audio output's own buffering latency: the audio clock advances as soon as bytes are accepted by write(), not when they actually become audible - e.g. ~100ms ahead on a device with ~100ms of internal buffering, which would otherwise make every video frame appear that much too early. Delaying each frame's schedule by this amount cancels that out. There's no way to derive it automatically - set it to roughly match the audio output's own buffering latency. 0 (the default) applies no correction. Forwards to the built-in PacedVideoOutput; call before begin().

setTaskParameters()

void setTaskParameters ( uint32_t  stackSizeWords, uint8_t  priority, int  core = -1  )

inline

Stack size (in words), priority, and (optionally) CPU core for the background task that renders queued video frames on schedule. core pins that task to a specific core - e.g. core 0, to keep it off whichever core runs copy()/loop() (audio decode/demuxing), giving audio and video decoding their own separate cores with no further setup. -1 (the default) leaves core assignment up to the RTOS. Forwards to the built-in PacedVideoOutput; call before begin() - has no effect on an already-running render task.

setUseAudioClock()

void setUseAudioClock ( bool  active )

inline

Whether video frames are scheduled against the real audio output's playback progress (an "audio clock", via AudioTimeSourceStream) instead of a free-running timer - keeps video in sync with audio even as the two decode at slightly different rates, since the audio clock only advances as audio is actually consumed. True (on) by default once an audio output is set via setAudioOutput() - calling setAudioOutput() again does not reset this. Turn it off if the content's audio track never actually delivers bytes (e.g. a silent/empty track), since the clock would then never advance and video would never play. Call before begin().

setVideoOutput()

void setVideoOutput ( VideoOutput &  out )

inline

Defines the final video display target (e.g. OutputTinyGPU, OutputOpenCV) - the caller must already have configured/begin()'d it (board/panel init, such as pin setup or a TFT library's own begin(), is outside this class's scope). Call before begin().

useAudioClock()

bool useAudioClock ( ) const

inline

Current setUseAudioClock() setting.

videoDecoder()

MultiVideoDecoder & videoDecoder ( )

inline

The built-in video multi-decoder, empty until addVideoDecoder() has registered at least one video codec decoder - also useful for diagnostics (e.g. selectedFormat()).

videoSyncTask()

PacedVideoOutput & videoSyncTask ( )

inline

Escape hatch for tuning beyond this class's own surface - e.g. setTaskParameters()/setQueueBytes()/setMaxQueuedIFrames()/ setIgnorePFrames()/setSchedulingDelayMs(), or the frameCount()/ avgFrameMs()/outputFPS() family of diagnostics - returns the PacedVideoOutput instance that VideoPlayer's own tuning forwarders above delegate to.

Member Data Documentation

active

bool active = false

protected

audio_clock

AudioTimeSourceStream audio_clock

protected

audio_out

EncodedAudioStream audio_out

protected

buffer_size

int buffer_size = 1024

protected

default_audio_decoder

MultiDecoder default_audio_decoder

protected

default_demuxer

MultiVideoDemuxer default_demuxer

protected

default_video_decoder

MultiVideoDecoder default_video_decoder

protected

p_audio_output

Print* p_audio_output = nullptr

protected

p_audio_output_typed

AudioOutput* p_audio_output_typed = nullptr

protected

p_audio_stream

AudioStream* p_audio_stream = nullptr

protected

p_source

Stream* p_source = nullptr

protected

p_video_output

VideoOutput* p_video_output = nullptr

protected

use_audio_clock

bool use_audio_clock = true

protected

video_sync

PacedVideoOutput video_sync {default_video_decoder}

protected

---

The documentation for this class was generated from the following file:

• src/AudioTools/Video/VideoPlayer.h