H264Decoder Class Reference

H.264 video decoder: wraps TinyH264Decoder (https://github.com/pschatzmann/TinyH264) as a VideoOutput, so it can be plugged directly into e.g. DemuxerAVI::setOutputVideo()/ DemuxerMP4::setOutputVideo(). Consumes an Annex-B H.264 bitstream via write() and, once a complete picture has been decoded, writes it - converted to setVideoFormat()'s format (RGB565 by default, the common TFT wire format) - to the Print target configured via setOutput(), one write() call per decoded picture. More...

#include <CodecH264.h>

Inheritance diagram for H264Decoder:

Public Member Functions
	H264Decoder ()
	H264Decoder (Print &out)
	H264Decoder (VideoOutput &out)
bool	begin () override
VideoFormat	codecFormat () override
tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32< uint8_t > > &	driver ()
void	end () override
	Releases the decoder's picture buffers - see TinyH264Decoder::end().
virtual void	flush ()
virtual uint32_t	getWriteTimeMs () const
	Optional: returns the time (ms) spent in the last write() call.
virtual bool	hadOutput () const
bool	hasError ()
bool	isKeyFrame (const uint8_t *data, size_t len) override
	See VideoOutput::isKeyFrame() - scans for an Annex-B IDR slice NAL.
bool	isValid (const uint8_t *data, size_t len) override
void	setMaxRefFrames (int n)
void	setOutput (Print &out) override
void	setOutput (VideoOutput &out) override
virtual void	setSkipRender (bool skip)
void	setVideoFormat (VideoFormat format) override
virtual void	setVideoInfoSource (VideoInfoSource &source)
uint64_t	totalDecodeMs () const override
VideoInfo	videoInfo () override
size_t	write (const uint8_t *data, size_t len) override

Protected Member Functions
void	writeToOutput ()
size_t	writeToOutput (uint8_t *data, size_t len)

Static Protected Member Functions
static void	onFrame (tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32< uint8_t > > &decoder, void *userData)

Protected Attributes
tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32< uint8_t > >	decoder_
Vector< uint8_t >	frame_buffer
Print *	p_out = nullptr
VideoOutput *	p_out_video = nullptr
VideoFormat	pixel_format = VideoFormat::RGB565
uint64_t	total_decode_ms_ = 0
uint32_t	total_output_ms_ = 0

Detailed Description

H.264 video decoder: wraps TinyH264Decoder (https://github.com/pschatzmann/TinyH264) as a VideoOutput, so it can be plugged directly into e.g. DemuxerAVI::setOutputVideo()/ DemuxerMP4::setOutputVideo(). Consumes an Annex-B H.264 bitstream via write() and, once a complete picture has been decoded, writes it - converted to setVideoFormat()'s format (RGB565 by default, the common TFT wire format) - to the Print target configured via setOutput(), one write() call per decoded picture.

write()/flush() implement the VideoOutput contract (see Video.h): write() may be called one or more times per frame - bytes are forwarded to the decoder immediately (Annex-B start codes are self-delimiting, so no internal buffering is needed here; a picture may complete partway through any write() call, not just at flush()). flush() is VideoOutput's no-op default - some producers (e.g. DemuxerAVI/DemuxerMP4) call it unconditionally after each frame, but this class has nothing to do with it.

Uses H264_DEFAULT_ALLOCATOR for the decoded picture buffers - see that macro's own comment above for the PSRAM-on-ESP32/heap- elsewhere default it picks, and how to override it.

Author

Phil Schatzmann

Copyright

GPLv3

Constructor & Destructor Documentation

H264Decoder() [1/3]

H264Decoder ( )

inline

H264Decoder() [2/3]

H264Decoder ( Print &  out )

inline

H264Decoder() [3/3]

H264Decoder ( VideoOutput &  out )

inline

Member Function Documentation

begin()

bool begin ( )

inlineoverridevirtual

Reserves the decoder's picture buffers up front - see TinyH264Decoder::begin().

Implements VideoDecoder.

codecFormat()

VideoFormat codecFormat ( )

inlineoverridevirtual

The compressed bitstream codec this decoder consumes (e.g. VideoFormat::H264, VideoFormat::MJPEG) - NOT the same axis as setVideoFormat()/videoInfo().format below, which is the decoded pixel format this decoder produces. Used by MultiVideoDecoder::addDecoder() to register a decoder under its own codec instead of the caller having to pass it separately (and possibly get it wrong).

Implements VideoDecoder.

driver()

tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32< uint8_t > > & driver ( )

inline

Direct access to the wrapped TinyH264Decoder, e.g. for its width()/ height()/y()/u()/v()/getY()/getU()/getV() accessors.

end()

void end ( )

inlineoverridevirtual

Releases the decoder's picture buffers - see TinyH264Decoder::end().

Implements VideoDecoder.

flush()

virtual void flush ( )

inlinevirtualinherited

Finalizes the frame most recently written via one or more write() calls - see class comment. Default no-op for implementations that display/decode synchronously in write() instead.

Reimplemented in Muxer, H264DecoderESP32S3< Alloc >, MJPEGDecoder, MPGDecoder, MultiVideoDecoder, OutputFPSMeter, OutputOpenCV, PacedVideoOutput, and VideoFrameMeter.

getWriteTimeMs()

virtual uint32_t getWriteTimeMs ( ) const

inlinevirtualinherited

Optional: returns the time (ms) spent in the last write() call.

Reimplemented in OutputTFT_eSPI, OutputTinyGPU, and OutputOpenCV.

hadOutput()

virtual bool hadOutput ( ) const

inlinevirtualinherited

True if the most recent write()+flush() call actually produced a displayable picture - default true, matching every synchronous decoder (H264Decoder, MJPEGDecoder, ...), which always decodes and pushes pixels fully within that one call. Override this only if your decoder can legitimately accept/decode a frame's bytes without emitting a picture during that same call - e.g. MPGDecoder, whose B-picture display-order reordering can hold a just-decoded picture back and instead emit an earlier one (or nothing at all) from a given write(), see its own override. Used by PacedVideoOutput to avoid counting/timing a call that did no real rendering work as a rendered frame - without this, its outputFPS()/frameCountI()/ frameCountP()/avgFrameMs() would overcount for such a decoder.

Reimplemented in MPGDecoder.

hasError()

bool hasError ( )

inline

True if the most recent write() call ended in a bitstream error or hit an unsupported stream feature.

isKeyFrame()

bool isKeyFrame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

See VideoOutput::isKeyFrame() - scans for an Annex-B IDR slice NAL.

Reimplemented from VideoOutput.

isValid()

bool isValid ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

True if data starts with an Annex-B start code (00 00 01) followed by a structurally valid NAL header (forbidden_zero_bit == 0, a defined nal_unit_type) - see VideoDecoder::isValid(). Checked after MPGDecoder::isValid() (CodecMPG.h) by MultiVideoDecoder (see MultiVideoDecoderFull's constructor), since a handful of low NAL-type values numerically overlap MPEG-1's slice_start_code range and are only disambiguated by MPEG-1's own sequence_header having already been ruled out first.

Reimplemented from VideoDecoder.

onFrame()

static void onFrame ( tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32< uint8_t > > &  decoder, void *  userData  )

inlinestaticprotected

setMaxRefFrames()

void setMaxRefFrames ( int  n )

inline

Runtime-active maximum stored reference pictures - see TinyH264Decoder::setMaxRefFrames().

setOutput() [1/2]

void setOutput ( Print &  out )

inlineoverridevirtual

Defines the target the decoded picture is written to, one write() call per decoded picture, in the format selected via setVideoFormat() (RGB565 by default).

Implements VideoDecoder.

setOutput() [2/2]

void setOutput ( VideoOutput &  out )

inlineoverridevirtual

Defines the target each decoded picture is written to - the VideoOutput-specific counterpart of setOutput(Print&), for a target like OutputTinyGPU/OutputOpenCV that implements VideoOutput but not Print (the two are unrelated types - see VideoFrameMeter's own comment). Part of the base interface (unlike other decoder-specific extras) so code holding only a VideoDecoder& (e.g. VideoPlayer) can wire it to an arbitrary VideoOutput& without knowing the concrete decoder type - every current decoder already implemented this identically before it was pulled up into the interface.

Implements VideoDecoder.

setSkipRender()

virtual void setSkipRender ( bool  skip )

inlinevirtualinherited

Hint to skip the expensive part of displaying the next frame(s) (e.g. the panel refresh) while still accepting and fully processing write() calls - used to recover from falling behind the playback schedule without breaking a codec's decode state (e.g. H.264 inter-prediction reference chain, which requires every frame to still be decoded even if it's never shown). Default no-op: implementations that can't skip rendering cheaply just ignore it and always render.

Reimplemented in MultiVideoDecoder, OutputFPSMeter, OutputTinyGPU, PacedVideoOutput, and VideoFrameMeter.

setVideoFormat()

void setVideoFormat ( VideoFormat  format )

inlineoverridevirtual

Selects the pixel format written to setOutput()'s target - RGB565 (the default) is what most TFT display libraries expect; RGB666/ RGB888 for higher color depth displays; I420 to pass the decoded planes through unconverted (tightly packed: Y bytes, then U bytes, then V bytes, no row padding - see TinyH264Decoder::toYUV420()). Any other VideoFormat is logged and ignored (previous format kept).

Implements VideoDecoder.

setVideoInfoSource()

virtual void setVideoInfoSource ( VideoInfoSource &  source )

inlinevirtualinherited

Optional: registers where width/height/fps/format (VideoInfo) come from - e.g. the demuxer feeding this output, so it can size its own buffers/panel setup without the caller having to duplicate that call per sketch (VideoPlayer::begin() does this automatically for whichever VideoOutput it was given). Default no-op: only implementations that actually need VideoInfo (OutputTinyGPU/ OutputOpenCV/OutputTFT_eSPI) override this.

Reimplemented in MultiVideoDecoder, OutputOpenCV, OutputTFT_eSPI, and OutputTinyGPU.

totalDecodeMs()

uint64_t totalDecodeMs ( ) const

inlineoverridevirtual

Sum of time spent purely inside decoder_.write() (CAVLC decode + reconstruction) since begin() - excludes SD reads and demux overhead a caller's own outer timing bundles in alongside it, AND excludes the picture-ready callback's own cost (pixel-format conversion + pushing the result to setOutput()'s target, which decoder_.write() invokes synchronously mid-call - see writeToOutput()) - measured separately and subtracted in write(), so a caller comparing this against its own outer wall-clock time around write() (e.g. OutputFPSMeter) gets a real decode-vs-render split instead of a number that already secretly includes render.

Reimplemented from VideoOutput.

videoInfo()

VideoInfo videoInfo ( )

inlineoverridevirtual

Reports the format/dimensions of the picture written to setOutput()'s target - format is setVideoFormat()'s most recently selected value (RGB565 if never called); width/height are the most recently decoded picture's (0 before any picture has been decoded).

Implements VideoDecoder.

write()

size_t write ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Feeds one chunk of Annex-B H.264 data - may be called more than once per frame (e.g. from a demuxer that hands over payload in pieces). Decodes immediately, invoking setOutput()'s Print once per completed picture from within this call (not deferred to flush()).

Implements VideoOutput.

writeToOutput() [1/2]

void writeToOutput ( )

inlineprotected

writeToOutput() [2/2]

size_t writeToOutput ( uint8_t *  data, size_t  len  )

inlineprotected

Member Data Documentation

decoder_

tinyh264::TinyH264Decoder< tinyh264::PSRAMAllocatorESP32<uint8_t> > decoder_

protected

frame_buffer

Vector<uint8_t> frame_buffer

protected

p_out

Print* p_out = nullptr

protected

p_out_video

VideoOutput* p_out_video = nullptr

protected

pixel_format

VideoFormat pixel_format = VideoFormat::RGB565

protected

total_decode_ms_

uint64_t total_decode_ms_ = 0

protected

total_output_ms_

uint32_t total_output_ms_ = 0

protected

---

The documentation for this class was generated from the following file:

• src/AudioTools/Video/CodecH264.h