MultiVideoDecoder Class Reference

Manages multiple VideoDecoders with automatic format detection - the video-side counterpart of MultiDecoder (AudioCodecs/MultiDecoder.h). No decoders are registered by default and this header has no codec- library dependency of its own - register whatever your content needs via addDecoder(), or use MultiVideoDecoderFull (MultiVideoDecoderFull.h) for one pre-registered with every video codec this library ships a portable (no hardware-specific backend) software decoder for (H264/MJPEG/MPEG-1). More...

#include <MultiVideoDecoder.h>

Inheritance diagram for MultiVideoDecoder:

Classes
struct	DecoderInfo

Public Member Functions
void	addDecoder (VideoDecoder &decoder)
bool	begin () override
VideoFormat	codecFormat () override
void	end () override
	Releases the decoder's resources.
void	flush () override
virtual uint32_t	getWriteTimeMs () const
	Optional: returns the time (ms) spent in the last write() call.
virtual bool	hadOutput () const
bool	isKeyFrame (const uint8_t *data, size_t len) override
virtual bool	isValid (const uint8_t *data, size_t len)
VideoFormat	selectedFormat () const
void	setOutput (Print &out) override
	Defines the target each decoded picture is written to.
void	setOutput (VideoOutput &out) override
void	setSkipRender (bool skip) override
void	setVideoFormat (VideoFormat format) override
void	setVideoInfoSource (VideoInfoSource &source)
uint64_t	totalDecodeMs () const override
VideoInfo	videoInfo () override
size_t	write (const uint8_t *data, size_t len) override

Protected Member Functions
void	ensureSelected (const uint8_t *data, size_t len)
void	select (DecoderInfo &info)

Protected Attributes
Vector< DecoderInfo >	decoders {0}
bool	is_first = true
VideoDecoder *	p_selected = nullptr
VideoFormat	p_selected_format = VideoFormat::UNKNOWN
VideoInfoSource *	p_video_info_source = nullptr

Detailed Description

Manages multiple VideoDecoders with automatic format detection - the video-side counterpart of MultiDecoder (AudioCodecs/MultiDecoder.h). No decoders are registered by default and this header has no codec- library dependency of its own - register whatever your content needs via addDecoder(), or use MultiVideoDecoderFull (MultiVideoDecoderFull.h) for one pre-registered with every video codec this library ships a portable (no hardware-specific backend) software decoder for (H264/MJPEG/MPEG-1).

Drop it into a demuxer's setOutputVideo() the same way any single decoder would go, and it self-selects the right registered decoder from the bitstream's own framing instead of the caller having to know the codec up front.

Content-sniffing detection (the fallback path - see below) is done via each registered decoder's own VideoDecoder::isValid() - a virtual method with a default-false implementation, so a decoder only takes part in auto-detection if it actually overrides it (the built-in H264Decoder/MJPEGDecoder/MPGDecoder each do - see their own class comments); a decoder that doesn't (e.g. H264DecoderESP32S3, not auto-detected by default - see MultiVideoDecoderFull's own comment) can still be registered and selected via a VideoInfoSource answer.

Detection runs once, on the very first write() OR isKeyFrame() call (whichever comes first - see isKeyFrame()'s own comment for why it also triggers detection, not just write()), preferring the container's own answer over guessing from raw bytes: if setVideoInfoSource() was given a source (typically the demuxer feeding this object - DemuxerAVI/DemuxerMP4/DemuxerMPG all implement VideoInfoSource and already parse the real codec from their own container metadata, e.g. DemuxerAVI::getVideoInfo().format from the AVI 'strf' chunk's FOURCC - see ContainerAVI.h's own comment) and its videoInfo().format matches a registered decoder, that decoder is selected directly, no sniffing needed. Otherwise (no source set, or its format is UNKNOWN/unregistered) falls back to each registered decoder's isValid(), tried in registration order, against that first call's bytes. Reliable either way because every current caller already hands over one complete access unit per write() call. Every subsequent call, plus flush()/isKeyFrame()/setSkipRender(), is forwarded to that same decoder for the rest of the stream - never re-detected.

setOutput()/setVideoFormat() are applied to every registered decoder eagerly (cheap - just stores pointers/an enum, no allocation), but begin() is only ever called on the one decoder actually selected - matching MultiDecoder's own memory-efficient lazy-init rationale, since a decoder's begin() is what allocates its picture buffers.

Author

Phil Schatzmann

Copyright

GPLv3

Member Function Documentation

addDecoder()

void addDecoder ( VideoDecoder &  decoder )

inline

Registers a decoder under its own VideoDecoder::codecFormat() - see the class comment for when/how it's later selected (VideoInfoSource match, or its own VideoDecoder::isValid() as a content-sniffing fallback). Call before the first write() reaches this object.

Replaces, rather than adds to, any decoder already registered for that format - at most one entry per format, so e.g. registering H264DecoderESP32S3 (codecFormat() == H264) actually overrides a previously-registered H264Decoder instead of silently losing to it: both write()'s VideoInfoSource-format lookup and its isValid() fallback loop match the first entry found for a format, so a stale second entry would otherwise be permanently unreachable dead weight, never actually selected.

begin()

bool begin ( )

inlineoverridevirtual

Defers actual decoder init to the first write() (see class comment)

• always succeeds itself.

Implements VideoDecoder.

codecFormat()

VideoFormat codecFormat ( )

inlineoverridevirtual

The codec of the currently selected decoder - VideoFormat::UNKNOWN before the first write()/if none matched. Same value as selectedFormat(), just satisfying VideoDecoder's own interface (e.g. so a MultiVideoDecoder can itself be registered under another MultiVideoDecoder's addDecoder()).

Implements VideoDecoder.

end()

void end ( )

inlineoverridevirtual

Releases the decoder's resources.

Implements VideoDecoder.

ensureSelected()

void ensureSelected ( const uint8_t *  data, size_t  len  )

inlineprotected

Runs detection (VideoInfoSource match, else each decoder's own isValid() as a content-sniffing fallback - see the class comment) exactly once, off whichever data/len it's first called with - shared by write() and isKeyFrame() so either one can trigger it (see isKeyFrame()'s own comment for why that matters). A no-op on every call after the first.

flush()

void flush ( )

inlineoverridevirtual

Finalizes the frame most recently written via one or more write() calls - see class comment. Default no-op for implementations that display/decode synchronously in write() instead.

Reimplemented from VideoOutput.

getWriteTimeMs()

virtual uint32_t getWriteTimeMs ( ) const

inlinevirtualinherited

Optional: returns the time (ms) spent in the last write() call.

Reimplemented in OutputTFT_eSPI, OutputTinyGPU, and OutputOpenCV.

hadOutput()

virtual bool hadOutput ( ) const

inlinevirtualinherited

True if the most recent write()+flush() call actually produced a displayable picture - default true, matching every synchronous decoder (H264Decoder, MJPEGDecoder, ...), which always decodes and pushes pixels fully within that one call. Override this only if your decoder can legitimately accept/decode a frame's bytes without emitting a picture during that same call - e.g. MPGDecoder, whose B-picture display-order reordering can hold a just-decoded picture back and instead emit an earlier one (or nothing at all) from a given write(), see its own override. Used by PacedVideoOutput to avoid counting/timing a call that did no real rendering work as a rendered frame - without this, its outputFPS()/frameCountI()/ frameCountP()/avgFrameMs() would overcount for such a decoder.

Reimplemented in MPGDecoder.

isKeyFrame()

bool isKeyFrame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Also triggers detection (see ensureSelected()) if it hasn't run yet - a caller upstream of write() (e.g. PacedVideoOutput::write(), which checks isKeyFrame() before deciding whether to even queue/forward a frame - see its own setIgnorePFrames()) can otherwise call this first, on the very same bytes write() would have detected from, and get a permanent false: no decoder ever selected (since write() never runs on a frame classified "drop"), so isKeyFrame() never returns true either - a deadlock. Detecting here too, off the exact same data/len write() would have used, closes that gap.

Reimplemented from VideoOutput.

isValid()

virtual bool isValid ( const uint8_t *  data, size_t  len  )

inlinevirtualinherited

True if data (the start of an access unit, as handed to write()) looks like this decoder's own bitstream format - content-sniffing, not a guarantee (see the concrete class for exactly what's checked). Used by MultiVideoDecoder to auto-select a registered decoder when no VideoInfoSource answer is available (see its own class comment); not otherwise part of the write()/flush() decode path. Default false, matching VideoOutput::isKeyFrame()'s own default - a decoder not meant to be auto-detected this way (e.g. a hardware-accelerated backend not registered by default, still usable via an explicit VideoInfoSource-based selection) simply never overrides it.

Reimplemented in H264Decoder, MJPEGDecoder, and MPGDecoder.

select()

void select ( DecoderInfo &  info )

inlineprotected

selectedFormat()

VideoFormat selectedFormat ( ) const

inline

The codec detection picked for the current stream - VideoFormat::UNKNOWN before the first write()/isKeyFrame(), or if none matched.

setOutput() [1/2]

void setOutput ( Print &  out )

inlineoverridevirtual

Defines the target each decoded picture is written to.

Implements VideoDecoder.

setOutput() [2/2]

void setOutput ( VideoOutput &  out )

inlineoverridevirtual

See VideoDecoder::setOutput(VideoOutput&) - forwarded to every registered decoder eagerly (same rationale as the Print& overload above), so whichever one gets selected on the first write() is already wired.

Implements VideoDecoder.

setSkipRender()

void setSkipRender ( bool  skip )

inlineoverridevirtual

Hint to skip the expensive part of displaying the next frame(s) (e.g. the panel refresh) while still accepting and fully processing write() calls - used to recover from falling behind the playback schedule without breaking a codec's decode state (e.g. H.264 inter-prediction reference chain, which requires every frame to still be decoded even if it's never shown). Default no-op: implementations that can't skip rendering cheaply just ignore it and always render.

Reimplemented from VideoOutput.

setVideoFormat()

void setVideoFormat ( VideoFormat  format )

inlineoverridevirtual

Selects the pixel format written to setOutput()'s target - e.g. VideoFormat::RGB565 (the common TFT wire format), RGB666/RGB888 for higher color depth displays, or I420 to pass the decoded planes through unconverted. Not every decoder backend supports every value (e.g. RGB666/RGB888 are TinyH264-only, not available on the esp_h264 backend) - unsupported values are logged and ignored (the previously selected format stays in effect); see the concrete class for exactly which ones it supports. Call before begin().

Implements VideoDecoder.

setVideoInfoSource()

void setVideoInfoSource ( VideoInfoSource &  source )

inlinevirtual

Provides the container's own answer for which codec the video track actually is - takes precedence over content-sniffing when set (see the class comment). Pass the demuxer feeding this object, e.g. multiVideoDecoder.setVideoInfoSource(aviDemuxer). Must outlive this object; call before the first write().

Reimplemented from VideoOutput.

totalDecodeMs()

uint64_t totalDecodeMs ( ) const

inlineoverridevirtual

Optional: sum of time (ms) spent purely decoding (excluding any surrounding convert/render/SPI work a subclass's write() also does) since begin() - see H264Decoder's own override for the only current implementation. Default 0: only meaningful for a decoder that separates decode time from render time internally: PacedVideoOutput:: logTo() prints a decode-vs-render split under "avg decode ms:" only when this returns nonzero.

Reimplemented from VideoOutput.

videoInfo()

VideoInfo videoInfo ( )

inlineoverridevirtual

Reports the format/dimensions of the picture written to setOutput()'s target - VideoInfo::format is always the format most recently selected via setVideoFormat() (RGB565 if never called), the reliable way to determine it (rather than assuming); width/ height reflect the most recently decoded picture, 0 before any picture has been decoded.

Implements VideoDecoder.

write()

size_t write ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Implements VideoOutput.

Member Data Documentation

decoders

Vector<DecoderInfo> decoders {0}

protected

is_first

bool is_first = true

protected

p_selected

VideoDecoder* p_selected = nullptr

protected

p_selected_format

VideoFormat p_selected_format = VideoFormat::UNKNOWN

protected

p_video_info_source

VideoInfoSource* p_video_info_source = nullptr

protected

---

The documentation for this class was generated from the following file:

• src/AudioTools/Video/MultiVideoDecoder.h