Buffers a small, configurable amount of video (see setQueueBytes()) and renders it frame by frame from a dedicated background task, timed against an audio clock - so a demuxer's own dispatch loop never blocks on video pacing. Wrap the real VideoOutput (e.g. H264Decoder) in this and pass it to setOutputVideo() instead of the decoder itself. More...
#include <PacedVideoOutput.h>
Classes | |
| struct | FrameHeader |
Public Member Functions | |
| PacedVideoOutput (VideoOutput &target, float fps=0, uint32_t schedulingDelayMs=0) | |
| float | avgFrameMs () const |
| float | avgIFrameMs () const |
| float | avgPFrameMs () const |
| bool | begin () |
| uint32_t | droppedFrameCount () const |
| uint32_t | droppedIFrameCount () const |
| void | end () |
| Stops the background render task. | |
| void | flush () override |
| uint32_t | frameCount () const |
| uint32_t | frameCountI () const |
| uint32_t | frameCountP () const |
| Number of P-frames (non-key frames) actually rendered so far. | |
| virtual uint32_t | getWriteTimeMs () const |
| Optional: returns the time (ms) spent in the last write() call. | |
| virtual bool | hadOutput () const |
| uint32_t | ignoredFrameCount () const |
| bool | ignorePFrames () const |
| float | inputFPS () |
| float | outputFPS () |
| size_t | queueCapacityBytes () |
| size_t | queuedBytes () |
| int | queuedIFrameCount () const |
| void | setAudioClock (TimeSource &clock) |
| void | setCatchUpThresholdFrames (float frames) |
| void | setFps (float fps) |
| void | setIgnorePFrames (bool active) |
| void | setMaxQueuedIFrames (int count) |
| void | setQueueBytes (size_t bytes) |
| void | setQueueUsePSRAM (bool flag) |
| void | setResyncQueueFillFraction (float fraction) |
| void | setResyncThresholdMs (uint32_t ms) |
| void | setSchedulingDelayMs (uint32_t delayMs) |
| void | setSkipRender (bool skip) override |
| void | setTaskParameters (uint32_t stackSizeWords, uint8_t priority, int core=-1) |
| size_t | write (const uint8_t *data, size_t len) override |
Protected Member Functions | |
| uint32_t | clockMs () |
| bool | drainQueueKeepingLastKeyframe (uint32_t &out_target_ms) |
| bool | isKeyFrame (const uint8_t *data, size_t len) |
| void | taskLoop () |
Protected Attributes | |
| bool | awaiting_keyframe = false |
| float | catch_up_threshold_frames = 1.0f |
| FrameHeader | current_header |
| uint32_t | dropped_frame_count = 0 |
| uint32_t | dropped_i_frame_count = 0 |
| Vector< uint8_t > | frame_buf |
| uint32_t | frame_count = 0 |
| uint64_t | frame_index = 0 |
| float | frame_period_ms = 0 |
| bool | have_header = false |
| std::atomic< uint32_t > | i_frame_count {0} |
| std::atomic< uint64_t > | i_frame_total_ms {0} |
| bool | ignore_p_frames = false |
| uint32_t | ignored_frame_count = 0 |
| uint32_t | input_start_ms = 0 |
| bool | input_start_set = false |
| bool | logged_drop_burst = false |
| int | max_queued_i_frames = 4 |
| std::atomic< uint32_t > | output_start_ms {0} |
| std::atomic< bool > | output_start_set {false} |
| TimeSource * | p_clock = nullptr |
| std::atomic< uint32_t > | p_frame_count {0} |
| std::atomic< uint64_t > | p_frame_total_ms {0} |
| VideoOutput * | p_target |
| RingBufferSPSC< uint8_t > | queue |
| size_t | queue_bytes = 32 * 1024 |
| size_t | queue_bytes_allocated = 0 |
| bool | queue_use_psram = false |
| std::atomic< int32_t > | queued_i_frame_count {0} |
| std::atomic< int32_t > | render_lateness_ms {0} |
| float | resync_queue_fill_fraction = 0.8f |
| uint32_t | resync_threshold_ms = 2000 |
| uint32_t | scheduling_delay_ms = 0 |
| uint32_t | start_ms = 0 |
| bool | start_set = false |
| Task | task |
| int | task_core = -1 |
| uint8_t | task_priority = 2 |
| uint32_t | task_stack_size = 4096 |
| bool | task_started = false |
Static Protected Attributes | |
| static constexpr int | kDrainYieldEvery = 32 |
Detailed Description
Buffers a small, configurable amount of video (see setQueueBytes()) and renders it frame by frame from a dedicated background task, timed against an audio clock - so a demuxer's own dispatch loop never blocks on video pacing. Wrap the real VideoOutput (e.g. H264Decoder) in this and pass it to setOutputVideo() instead of the decoder itself.
write() requires one whole frame per call, immediately followed by flush() (a no-op) - every current caller (DemuxerAVI/DemuxerMP4/ DemuxerMPG) already does this. Frame N renders once the audio clock (setAudioClock(), defaults to wall clock) reaches N / fps (setFps()); setSchedulingDelayMs() corrects for the audio output's own buffering latency.
decode+render is assumed to keep up with the recording frame rate on average. When it falls behind, write() drops non-keyframes instead of blocking - once the queue is full, or proactively once the render task is more than setCatchUpThresholdFrames() frame periods late (see write()'s own comment). Keyframes are never dropped this way.
Dropping alone can only prevent the backlog from growing, not shrink it (the producer can't deliver frames faster than real time either). setResyncThresholdMs(), setResyncQueueFillFraction(), and setMaxQueuedIFrames() are the fallback: once any fires, taskLoop() gives up on the backlog - keeping the freshest keyframe found in it (see drainQueueKeepingLastKeyframe()) and jumping the schedule forward to render it - so playback recovers (a visible jump) instead of lagging indefinitely. Any P-frame after a resync is discarded until the next real keyframe arrives, since a P-frame may reference a picture the decoder no longer holds.
Thread-safety: write() (the caller's thread) is the sole producer, the background task the sole consumer of the frame queue - a lock-free RingBufferSPSC, not a mutex.
Diagnostics: frameCount()/frameCountI()/frameCountP()/ droppedFrameCount()/droppedIFrameCount(), avgFrameMs()/avgIFrameMs()/ avgPFrameMs(), inputFPS()/outputFPS(), queuedBytes()/ queuedIFrameCount(). Also logs its own processing (LOGI lifecycle, LOGD per-frame, LOGW when falling behind/dropping).
- Copyright
- GPLv3
Constructor & Destructor Documentation
◆ PacedVideoOutput()
|
inline |
- Parameters
-
target every frame is eventually forwarded here (write() + flush()), from the background task only - never from the caller's own write()/flush() calls. fps nominal frames/second frames are scheduled against - 0 (the default) means "not yet known"; set it later via setFps() (e.g. once the demuxer's parsed VideoInfo::fps becomes available) before the first write(). schedulingDelayMs see setSchedulingDelayMs() - 0 (the default) applies no correction.
Member Function Documentation
◆ avgFrameMs()
|
inline |
◆ avgIFrameMs()
|
inline |
◆ avgPFrameMs()
|
inline |
◆ begin()
|
inline |
Starts the background render task - optional: write() calls this itself (idempotently) on first use if you never do, so the common case needs no explicit begin() at all. Call it yourself only if you need setTaskParameters()/setQueueBytes() to take effect (call those first) before any frame arrives.
◆ clockMs()
|
inlineprotected |
playbackTime() (not millis()) - scheduling against actual audio progress, not just time passing, is the point of setAudioClock(). Falls back to wall-clock millis() when no audio clock is set.
◆ drainQueueKeepingLastKeyframe()
|
inlineprotected |
Drains every whole frame currently sitting in 'queue' - used by taskLoop()'s resync trigger to abandon a backlog. Consumer-thread- only.
Keeps the LAST (freshest) keyframe found along the way instead of discarding it too: copies it into frame_buf, reports its schedule time via out_target_ms, returns true. Any keyframe found earlier in the same backlog is superseded (counted into dropped_i_frame_count). Every non-keyframe is discarded (counted into dropped_frame_count). Returns false (frame_buf untouched) if the backlog held no complete keyframe.
Stops as soon as the queue doesn't hold a complete next record (typically a header whose payload hasn't fully arrived - write() writes header and payload as two separate, non-atomic writeArray() calls), leaving that partial state for the next call - never touches queue.reset() (unsafe with a producer concurrently active) or discards anything mid-record, which would desync the framing permanently.
Yields every kDrainYieldEvery frames: this loop is pure memory copies with no hardware wait to implicitly yield the way normal rendering does - running it straight through against a large backlog can starve the FreeRTOS idle task long enough to trip the watchdog.
◆ droppedFrameCount()
|
inline |
Number of P-frames write() dropped instead of enqueueing - queue was full, the render task had fallen behind schedule (see write()'s own comment), or setIgnorePFrames() is on (see ignoredFrameCount() for that specific subset). 0 under normal operation with setIgnorePFrames() off; a rising count otherwise means decode+render can't sustain the recording frame rate - each dropped frame is simply never shown, the next queued one still renders at its own correct time, so this doesn't desync the audio/video clock.
◆ droppedIFrameCount()
|
inline |
Number of keyframes discarded by a resync instead of being rendered. 0 under normal operation - unlike droppedFrameCount(), this only ever comes from a resync, never from write()'s own proactive check.
◆ end()
|
inline |
Stops the background render task.
◆ flush()
|
inlineoverridevirtual |
No-op: write() (see its own comment) already does all the work as soon as one complete frame arrives - kept only because every current caller still calls this unconditionally right after write().
Reimplemented from VideoOutput.
◆ frameCount()
|
inline |
Total number of frames handed off via write() so far - includes dropped frames (see droppedFrameCount()), since those still count as "handed off", just not enqueued.
◆ frameCountI()
|
inline |
Number of I-frames (key frames) actually rendered so far - see isKeyFrame() for how a frame is classified.
◆ frameCountP()
|
inline |
Number of P-frames (non-key frames) actually rendered so far.
◆ getWriteTimeMs()
|
inlinevirtualinherited |
Optional: returns the time (ms) spent in the last write() call.
Reimplemented in OutputTFT_eSPI, OutputTinyGPU, and OutputOpenCV.
◆ hadOutput()
|
inlinevirtualinherited |
True if the most recent write()+flush() call actually produced a displayable picture - default true, matching every synchronous decoder (H264Decoder, MJPEGDecoder, ...), which always decodes and pushes pixels fully within that one call. Override this only if your decoder can legitimately accept/decode a frame's bytes without emitting a picture during that same call - e.g. MPGDecoder, whose B-picture display-order reordering can hold a just-decoded picture back and instead emit an earlier one (or nothing at all) from a given write(), see its own override. Used by PacedVideoOutput to avoid counting/timing a call that did no real rendering work as a rendered frame - without this, its outputFPS()/frameCountI()/ frameCountP()/avgFrameMs() would overcount for such a decoder.
Reimplemented in MPGDecoder.
◆ ignoredFrameCount()
|
inline |
Number of this stream's droppedFrameCount() specifically caused by setIgnorePFrames() rather than backlog (queue-full/falling-behind) - always 0 unless that's on.
◆ ignorePFrames()
|
inline |
◆ inputFPS()
|
inline |
Rate (frames/sec) frames are being written in, averaged from the first write() up to now - drops visibly if whatever feeds write() stalls, instead of freezing. Measured against setAudioClock()'s clock if set, wall clock otherwise - see clockMs(). Compare against outputFPS(): input consistently higher means the queue is filling up before write() has to block.
◆ isKeyFrame()
|
inlineprotectedvirtual |
I/P classification, both for the frameCountI()/frameCountP() stats and for deciding which frames write()/a resync may ever drop - delegates to the target's own VideoOutput::isKeyFrame() (see its comment), since the target is the one that actually knows its bitstream format.
Reimplemented from VideoOutput.
◆ outputFPS()
|
inline |
Rate (frames/sec) frames are actually being rendered - same averaging as inputFPS(), counting the background task's completed render calls. Always measured against wall-clock time regardless of setAudioClock(): answers "how fast can this hardware render", which shouldn't depend on audio's clock (playbackTime() freezes with any audio stall, which would otherwise distort this into an artificial spike on resume).
◆ queueCapacityBytes()
|
inline |
Total byte capacity of the frame queue - may be larger than requested (rounded up to the next power of two, see RingBufferSPSC). 0 before the queue has ever been allocated.
◆ queuedBytes()
|
inline |
Bytes currently sitting in the queue, waiting for the render task. 0 most of the time under normal operation; a consistently non-zero value is an early warning sign before write() has to block. Compare against queueCapacityBytes() for a fill-level percentage.
◆ queuedIFrameCount()
|
inline |
Number of keyframes currently queued, not yet consumed (rendered or dropped) - see setMaxQueuedIFrames(). 0 or 1 under normal operation; a rising count is an early warning sign of a backlog.
◆ setAudioClock()
|
inline |
Provides the audio clock frames are scheduled against - its playbackTime() (not millis(), see TimeSource's own comment for the difference) must return elapsed audio playback time; a ready-made implementation is AudioTimeSourceStream, AudioTools/CoreAudio/ AudioIO.h. Must outlive this object. Leave unset to schedule against the wall clock instead.
◆ setCatchUpThresholdFrames()
|
inline |
How many frame periods behind schedule the render task must fall before write() starts proactively dropping non-keyframes to catch up (see write()'s own comment), rather than only dropping once the queue is actually full. 1.0 (the default) drops once the most recently rendered frame was at least one frame period late; lower catches up faster at the cost of more drops, higher tolerates more backlog. No effect until setFps() reports a nonzero rate.
◆ setFps()
|
inline |
Sets/overrides the nominal frame rate frames are scheduled against - safe to call anytime, including before begin().
◆ setIgnorePFrames()
|
inline |
Unconditionally drops every non-keyframe in write() itself, before it's even queued - codec-agnostic (works for H264Decoder/MPGDecoder/ MJPEGDecoder/MultiVideoDecoder alike, unlike a decoder-specific flag such as MPGDecoder::setIgnorePFrames()) and cheaper than letting the decoder no-op an ignored frame after it: the bytes never get copied into the queue or scheduled at all. Off by default. See ignoredFrameCount() for how many this has skipped. Independent of setCatchUpThresholdFrames()'s own conditional dropping, which still applies when this is off.
◆ setMaxQueuedIFrames()
|
inline |
Number of not-yet-consumed keyframes in the queue that triggers the same resync as the other two thresholds - catches a backlog earlier than either: several GOPs' worth of unconsumed keyframes already means none of them are worth rendering in order (see drainQueueKeepingLastKeyframe(), shared by all three triggers, for how the freshest one gets kept instead). 3 (the default) is deliberately small; 0 disables this trigger.
◆ setQueueBytes()
|
inline |
Byte capacity of the frame queue - default 32KB. Too small behaves like a 1-frame queue (any one-off slow frame blocks write() immediately); larger absorbs transient jitter at the cost of RAM (and video trailing decode by however many frames end up buffered). Does not change what happens under a sustained mismatch: the queue still eventually fills, just later. Actual capacity may round up to the next power of two (see RingBufferSPSC). Call before begin()/the first frame - not supported afterwards.
◆ setQueueUsePSRAM()
|
inline |
Opts the frame queue into PSRAM-backed allocation instead of internal heap (see RingBufferSPSC::setUsePSRAM()) - falls back silently on boards without PSRAM. Worth enabling once setQueueBytes() is sized in the hundreds of KB+, since internal heap is a scarcer shared resource on most ESP32 boards. Call before begin()/the first frame - no effect on an already-allocated queue.
◆ setResyncQueueFillFraction()
|
inline |
Byte-occupancy fraction (0..1) of the queue that triggers the same resync as setResyncThresholdMs(), off a different signal: dropped P-frames can keep the most recently rendered frame's own lateness low even while newer, not-yet-rendered bytes keep piling up, since overall render throughput can still trail the arrival rate. 0.8 (80% full, the default) catches that before the queue_full drop condition (100%) would otherwise be the only thing bounding it. 0 disables this trigger.
◆ setResyncThresholdMs()
|
inline |
How far behind schedule (ms) the render task must fall before it stops trying to catch up and instead jumps the schedule anchor forward to the current clock time (see taskLoop()) - dropping frames alone can't reduce a backlog, only slow its growth, since the producer can't deliver frames faster than real time. Content in the skipped gap is never shown - a visible jump, but playback recovers instead of lagging further. 2000ms (the default); 0 disables resyncing.
◆ setSchedulingDelayMs()
|
inline |
Corrects for the real audio output's own buffering latency: an AudioTimeSourceStream clock advances as soon as bytes are accepted by write(), not when they become audible - e.g. ~100ms ahead on a device with ~100ms of internal buffering, which would otherwise show each frame that much early. Delaying every frame's schedule by this amount (added to target_ms, only ever later, never earlier) cancels it out. No way to derive this automatically - pick it to roughly match your audio output's configured buffering latency; 0 (the default) applies no correction. Also settable via the constructor.
◆ setSkipRender()
|
inlineoverridevirtual |
Hint to skip the expensive part of displaying the next frame(s) (e.g. the panel refresh) while still accepting and fully processing write() calls - used to recover from falling behind the playback schedule without breaking a codec's decode state (e.g. H.264 inter-prediction reference chain, which requires every frame to still be decoded even if it's never shown). Default no-op: implementations that can't skip rendering cheaply just ignore it and always render.
Reimplemented from VideoOutput.
◆ setTaskParameters()
|
inline |
◆ taskLoop()
|
inlineprotected |
One background-task iteration: pulls one {header, payload} frame out of 'queue' (across as many calls as needed for the payload to fully arrive - see have_header), waits until its scheduled presentation time, then renders it - else just yields. Never called from the caller's thread.
◆ write()
|
inlineoverridevirtual |
Enqueues one complete frame. Unlike the general VideoOutput contract (write() may be called several times per frame, finalized by flush()), this class requires the whole frame in one call - every current caller (DemuxerAVI/DemuxerMP4/DemuxerMPG) already does this; a producer that split a frame across multiple write() calls would have each piece enqueued as its own separate, undecodable "frame" - undetected.
Tags the frame with its scheduled presentation time (frame_index / fps), classifies it I vs P (see isKeyFrame()), and appends header + bytes to the background task's queue. Never renders inline - only the background task's write()+flush() into the real target does that, once the frame's scheduled time arrives.
- Parameters
-
data the complete frame's bytes - copied before this call returns, so the caller can reuse/discard the buffer immediately. len number of bytes in data; 0 is a harmless no-op.
- Returns
len, always - even for a dropped frame (see droppedFrameCount()), so the caller's own bookkeeping sees no difference.
Implements VideoOutput.
Member Data Documentation
◆ awaiting_keyframe
|
protected |
◆ catch_up_threshold_frames
|
protected |
◆ current_header
|
protected |
◆ dropped_frame_count
|
protected |
◆ dropped_i_frame_count
|
protected |
◆ frame_buf
|
protected |
Scratch buffer taskLoop() reads each frame's payload into - reused across frames, consumer-side-only, same as have_header above.
◆ frame_count
|
protected |
◆ frame_index
|
protected |
◆ frame_period_ms
|
protected |
◆ have_header
|
protected |
◆ i_frame_count
|
protected |
◆ i_frame_total_ms
|
protected |
◆ ignore_p_frames
|
protected |
◆ ignored_frame_count
|
protected |
◆ input_start_ms
|
protected |
◆ input_start_set
|
protected |
◆ kDrainYieldEvery
|
staticconstexprprotected |
How often drainQueueKeepingLastKeyframe() yields (delay(1)) instead of running straight through - see that method's own comment.
◆ logged_drop_burst
|
protected |
◆ max_queued_i_frames
|
protected |
◆ output_start_ms
|
protected |
◆ output_start_set
|
protected |
◆ p_clock
|
protected |
◆ p_frame_count
|
protected |
◆ p_frame_total_ms
|
protected |
◆ p_target
|
protected |
◆ queue
|
protected |
Lock-free single-producer (write())/single-consumer (taskLoop()) byte queue - see setQueueBytes(). Safe without a mutex specifically because there is exactly one writer and one reader (see the class comment) - RingBufferSPSC is not safe for more than that.
◆ queue_bytes
|
protected |
◆ queue_bytes_allocated
|
protected |
◆ queue_use_psram
|
protected |
◆ queued_i_frame_count
|
protected |
◆ render_lateness_ms
|
protected |
◆ resync_queue_fill_fraction
|
protected |
◆ resync_threshold_ms
|
protected |
◆ scheduling_delay_ms
|
protected |
◆ start_ms
|
protected |
◆ start_set
|
protected |
◆ task
|
protected |
◆ task_core
|
protected |
◆ task_priority
|
protected |
◆ task_stack_size
|
protected |
◆ task_started
|
protected |
The documentation for this class was generated from the following file:
- src/AudioTools/Video/PacedVideoOutput.h
