MuxerMTS Class Reference

ISO/IEC 13818-1 MPEG Transport Stream Encoder: packetizes an already-encoded video elementary stream (H.264 by default - see MuxerVideoConfig::format) and an optional audio elementary stream (AAC ADTS by default - see setAudioInfo()) into 188-byte TS packets, with a PAT and PMT describing the one program/two streams, written to a Print (a local File to record a .ts, or e.g. a network Client to publish a live/HLS-segment stream). More...

#include <ContainerMTS.h>

Inheritance diagram for MuxerMTS:

Public Member Functions
	MuxerMTS ()
	MuxerMTS (Print &out)
size_t	addAudioFrame (const uint8_t *data, size_t len) override
size_t	addI420Frame (const uint8_t *data, size_t len) override
size_t	addJpegFrame (const uint8_t *data, size_t len) override
size_t	addRGB565Frame (const uint8_t *data, size_t len) override
size_t	addVideoFrame (const uint8_t *data, size_t len, bool isKeyFrame=true) override
size_t	addYUV422Frame (const uint8_t *data, size_t len) override
uint32_t	audioFrameCount ()
AudioInfoFormat &	audioInfo () override
	Provides read/write access to the audio track's AudioInfoFormat.
bool	begin () override
void	end () override
	MPEG-TS has no defined trailer - simply stops accepting frames.
virtual void	flush ()
float	getAudioSamplesPerVideoFrame ()
MuxerVideoConfig	getVideoInfo () override
	Provides the video track configuration.
virtual uint32_t	getWriteTimeMs () const
	Optional: returns the time (ms) spent in the last write() call.
virtual bool	hadOutput () const
virtual bool	isKeyFrame (const uint8_t *data, size_t len)
const char *	mimeVideo () override
	operator bool () override
void	setAudioInfo (AudioInfoFormat info) override
void	setOutput (Print &out) override
	Defines the output: e.g. a local File or a network Client.
virtual void	setSkipRender (bool skip)
void	setStreamType (StreamContentType type) override
	Selects whether write() feeds the video or the audio track.
void	setVideoInfo (MuxerVideoConfig config) override
	Defines the video track configuration - call before begin()
virtual void	setVideoInfoSource (VideoInfoSource &source)
StreamContentType	streamType () override
	The track write() currently targets (see setStreamType())
virtual uint64_t	totalDecodeMs () const
uint32_t	videoFrameCount ()
size_t	write (const uint8_t *data, size_t len) override

Protected Member Functions
uint64_t	audioPts ()
void	packBytesToTs (uint16_t pid, uint8_t &cc, const uint8_t *header, size_t header_len, const uint8_t *payload, size_t payload_len, bool withPcr, uint64_t pcr)
uint64_t	videoPts ()
void	writeAccessUnit (uint16_t pid, uint8_t stream_id, uint8_t &cc, const uint8_t *data, size_t len, uint64_t pts, bool withPcr)
void	writePat ()
void	writePmt ()
void	writeStreamEntry (uint8_t *pkt, size_t &pos, uint8_t stream_type, uint16_t pid)

Static Protected Member Functions
static void	copyVirtual (uint8_t *dst, size_t voffset, size_t n, const uint8_t *a, size_t alen, const uint8_t *b, size_t blen)
static void	writePcr (uint8_t *buf, uint64_t pcr_base)
static void	writeTsTimestamp (uint8_t *b, uint64_t ts)

Protected Attributes
uint8_t	audio_cc = 0
uint32_t	audio_frame_count = 0
AudioInfoFormat	audio_info
uint32_t	audio_samples_per_frame = 1024
bool	has_audio = false
bool	is_open = false
Print *	p_out = nullptr
Print *	p_print = nullptr
uint8_t	pat_cc = 0
uint8_t	pmt_cc = 0
uint8_t	video_cc = 0
MuxerVideoConfig	video_cfg
uint32_t	video_frame_count = 0
StreamContentType	write_stream_type = StreamContentType::Video

Detailed Description

ISO/IEC 13818-1 MPEG Transport Stream Encoder: packetizes an already-encoded video elementary stream (H.264 by default - see MuxerVideoConfig::format) and an optional audio elementary stream (AAC ADTS by default - see setAudioInfo()) into 188-byte TS packets, with a PAT and PMT describing the one program/two streams, written to a Print (a local File to record a .ts, or e.g. a network Client to publish a live/HLS-segment stream).

Like MuxerMPG/MuxerAVI, this is a streaming writer: PAT+PMT are (re-)written before every video keyframe (so a player/segmenter joining mid-stream, or starting a new HLS segment, always sees them up front), and each addVideoFrame()/addAudioFrame() call becomes exactly one PES packet, split across as many TS packets as its size needs (no PES length limit is enforced: PES_packet_length is written literally when it fits 16 bits, 0 ("unbounded", the standard convention for a video elementary stream whose access units routinely exceed 64KB) otherwise - DemuxerMTS itself never relies on this field, only on payload_unit_start_indicator, for framing). A PCR (derived from the same PTS clock as the video track, not a wall clock) is attached to every video access unit's first TS packet, matching the PMT's PCR_PID (always the video PID).

MJPEG is also supported (VideoFormat::MJPEG + addJpegFrame(), one complete JPEG image per call) - see MTS_STREAM_TYPE_MJPEG's own comment for how it's signalled, since MPEG-TS has no ISO-registered stream_type for it.

Usage:

MuxerMTS mts(client);  // any Print: File, WiFiClient, ...
MuxerVideoConfig cfg;
cfg.width = 1280;
cfg.height = 720;
cfg.fps = 30;
cfg.format = VideoFormat::H264;
mts.setVideoInfo(cfg);
mts.setAudioInfo(AudioInfoFormat(48000, 2, 16, AudioFormat::AAC));
mts.begin();
// for each encoded H.264 access unit (Annex-B, starting with its own
// start codes):
mts.addVideoFrame(h264_access_unit, len, isKeyFrame);
// for each raw ADTS-framed AAC frame:
mts.addAudioFrame(adts_frame, len);

(VideoMuxer/VideoMuxerWithTasks drive the video side, and setStreamType()-toggled write(), automatically - MuxerMTS implements the same Muxer interface as MuxerAVI/MuxerMP4/MuxerMPG, so it plugs into them unmodified.)

Author

Phil Schatzmann

Copyright

GPLv3

Constructor & Destructor Documentation

MuxerMTS() [1/2]

MuxerMTS ( )

inline

MuxerMTS() [2/2]

MuxerMTS ( Print &  out )

inline

Member Function Documentation

addAudioFrame()

size_t addAudioFrame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Writes one complete audio frame (e.g. one ADTS-framed AAC frame, or one MP3 frame) as a single PES packet. Ignored (returns 0) if no audio track was configured via setAudioInfo().

Implements Muxer.

addI420Frame()

size_t addI420Frame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Writes one planar 4:2:0 YUV (I420/IYUV) frame. Expects exactly width*height*3/2 bytes.

Implements Muxer.

addJpegFrame()

size_t addJpegFrame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Writes one complete, already-encoded JPEG image as a single PES packet (see MTS_STREAM_TYPE_MJPEG's own comment for how this container signals MJPEG - there is no ISO-registered stream_type for it). isKeyFrame is always true: every JPEG image is independently decodable, so - unlike addVideoFrame()'s H.264-oriented default - this always re-sends PAT+PMT first too (same convention MuxerVideoSink, ContainerCommon.h, already uses for MJPEG).

Implements Muxer.

addRGB565Frame()

size_t addRGB565Frame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Writes one uncompressed RGB565 (16-bit, 5-6-5) frame. Expects exactly width*height*2 bytes.

Implements Muxer.

addVideoFrame()

size_t addVideoFrame ( const uint8_t *  data, size_t  len, bool  isKeyFrame = true  )

inlineoverridevirtual

Writes one complete, already-encoded video access unit as a single PES packet (see class comment). Re-sends PAT+PMT first when isKeyFrame is true.

Implements Muxer.

addYUV422Frame()

size_t addYUV422Frame ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Writes one packed 4:2:2 YUV frame (YUY2/YUYV byte order). Expects exactly width*height*2 bytes.

Implements Muxer.

audioFrameCount()

uint32_t audioFrameCount ( )

inline

audioInfo()

AudioInfoFormat & audioInfo ( )

inlineoverridevirtual

Provides read/write access to the audio track's AudioInfoFormat.

Implements Muxer.

audioPts()

uint64_t audioPts ( )

inlineprotected

begin()

bool begin ( )

inlineoverridevirtual

Writes the initial PAT + PMT. Call after configuring video (and audio, if any) and before writing any frames.

Implements Muxer.

copyVirtual()

static void copyVirtual ( uint8_t *  dst, size_t  voffset, size_t  n, const uint8_t *  a, size_t  alen, const uint8_t *  b, size_t  blen  )

inlinestaticprotected

Copies 'n' bytes starting at virtual offset 'voffset' out of the logical concatenation of [a, a+alen) followed by [b, b+blen) - lets packBytesToTs() packetize a small PES header and a (possibly large) ES payload as one continuous byte stream without ever copying the payload into a combined buffer first.

end()

void end ( )

inlineoverridevirtual

MPEG-TS has no defined trailer - simply stops accepting frames.

Implements Muxer.

flush()

virtual void flush ( )

inlinevirtualinherited

Finalizes the frame most recently written via one or more write() calls - see class comment. Default no-op for implementations that display/decode synchronously in write() instead.

Reimplemented from VideoOutput.

getAudioSamplesPerVideoFrame()

float getAudioSamplesPerVideoFrame ( )

inlineinherited

Average number of audio samples per video frame, derived from audioInfo().sample_rate and getVideoInfo().fps - the natural audio chunk size to write once per video frame if you want to keep both tracks advancing at roughly the same pace as you write them (not a hard requirement - see addAudioFrame()/addVideoFrame()). 0 if fps hasn't been set.

getVideoInfo()

MuxerVideoConfig getVideoInfo ( )

inlineoverridevirtual

Provides the video track configuration.

Implements Muxer.

getWriteTimeMs()

virtual uint32_t getWriteTimeMs ( ) const

inlinevirtualinherited

Optional: returns the time (ms) spent in the last write() call.

Reimplemented in OutputTFT_eSPI, OutputTinyGPU, and OutputOpenCV.

hadOutput()

virtual bool hadOutput ( ) const

inlinevirtualinherited

True if the most recent write()+flush() call actually produced a displayable picture - default true, matching every synchronous decoder (H264Decoder, MJPEGDecoder, ...), which always decodes and pushes pixels fully within that one call. Override this only if your decoder can legitimately accept/decode a frame's bytes without emitting a picture during that same call - e.g. MPGDecoder, whose B-picture display-order reordering can hold a just-decoded picture back and instead emit an earlier one (or nothing at all) from a given write(), see its own override. Used by PacedVideoOutput to avoid counting/timing a call that did no real rendering work as a rendered frame - without this, its outputFPS()/frameCountI()/ frameCountP()/avgFrameMs() would overcount for such a decoder.

Reimplemented in MPGDecoder.

isKeyFrame()

virtual bool isKeyFrame ( const uint8_t *  data, size_t  len  )

inlinevirtualinherited

True if data (one complete encoded frame, as handed to write()) is a keyframe/sync-sample - self-contained, decodable without any earlier frame. Used e.g. by PacedVideoOutput to decide which frames are safe to drop, and whether it's safe to resume decoding after abandoning a backlog (see its own class comment). Default false: a plain VideoOutput doesn't know or care about codec structure - override this in a decoder for the bitstream format it actually parses (see H264Decoder/H264DecoderESP32S3's isH264KeyFrame()-based override, MPGDecoder's isMpeg1KeyFrame()- based one). Getting this right matters beyond bookkeeping: a target whose frames are never recognized as keyframes can leave a caller like PacedVideoOutput unable to ever resume after a resync.

Reimplemented in PacedVideoOutput, H264Decoder, H264DecoderESP32S3< Alloc >, MJPEGDecoder, MPGDecoder, MultiVideoDecoder, and OutputFPSMeter.

mimeVideo()

const char * mimeVideo ( )

inlineoverridevirtual

The container's MIME type (e.g. "video/avi", "video/mp4") - useful for e.g. an HTTP Content-Type header when streaming the muxed output to a client.

Implements Muxer.

operator bool()

operator bool ( )

inlineoverridevirtual

Implements Muxer.

packBytesToTs()

void packBytesToTs ( uint16_t  pid, uint8_t &  cc, const uint8_t *  header, size_t  header_len, const uint8_t *  payload, size_t  payload_len, bool  withPcr, uint64_t  pcr  )

inlineprotected

Splits the logical concatenation of a PES header (header/header_len) and its ES payload (payload/payload_len) into as many 188-byte TS packets as needed. payload_unit_start_indicator is set on the first packet only; the continuity_counter 'cc' is threaded through and incremented per packet. When 'withPcr' is set, the first packet additionally carries a PCR in its adaptation field (stealing from its own payload capacity for that packet only) - see writePcr(). Every packet that doesn't fill its 184-byte payload capacity exactly is padded to it via adaptation-field stuffing (0xFF bytes) - this only ever applies to the last packet of the run.

setAudioInfo()

void setAudioInfo ( AudioInfoFormat  info )

inlineoverridevirtual

Adds an (optional) audio track. info.format selects the PMT stream_type (AudioFormat::AAC -> ADTS/0x0F, AudioFormat::MP3 -> MPEG-1 audio/0x03 - the only two with a standard MPEG-TS mapping). Call before begin().

Implements Muxer.

setOutput()

void setOutput ( Print &  out )

inlineoverridevirtual

Defines the output: e.g. a local File or a network Client.

Implements Muxer.

setSkipRender()

virtual void setSkipRender ( bool  skip )

inlinevirtualinherited

Hint to skip the expensive part of displaying the next frame(s) (e.g. the panel refresh) while still accepting and fully processing write() calls - used to recover from falling behind the playback schedule without breaking a codec's decode state (e.g. H.264 inter-prediction reference chain, which requires every frame to still be decoded even if it's never shown). Default no-op: implementations that can't skip rendering cheaply just ignore it and always render.

Reimplemented in MultiVideoDecoder, OutputFPSMeter, OutputTinyGPU, PacedVideoOutput, and VideoFrameMeter.

setStreamType()

void setStreamType ( StreamContentType  type )

inlineoverridevirtual

Selects whether write() feeds the video or the audio track.

Implements Muxer.

setVideoInfo()

void setVideoInfo ( MuxerVideoConfig  config )

inlineoverridevirtual

Defines the video track configuration - call before begin()

Implements Muxer.

setVideoInfoSource()

virtual void setVideoInfoSource ( VideoInfoSource &  source )

inlinevirtualinherited

Optional: registers where width/height/fps/format (VideoInfo) come from - e.g. the demuxer feeding this output, so it can size its own buffers/panel setup without the caller having to duplicate that call per sketch (VideoPlayer::begin() does this automatically for whichever VideoOutput it was given). Default no-op: only implementations that actually need VideoInfo (OutputTinyGPU/ OutputOpenCV/OutputTFT_eSPI) override this.

Reimplemented in MultiVideoDecoder, OutputOpenCV, OutputTFT_eSPI, and OutputTinyGPU.

streamType()

StreamContentType streamType ( )

inlineoverridevirtual

The track write() currently targets (see setStreamType())

Implements Muxer.

totalDecodeMs()

virtual uint64_t totalDecodeMs ( ) const

inlinevirtualinherited

Optional: sum of time (ms) spent purely decoding (excluding any surrounding convert/render/SPI work a subclass's write() also does) since begin() - see H264Decoder's own override for the only current implementation. Default 0: only meaningful for a decoder that separates decode time from render time internally: PacedVideoOutput:: logTo() prints a decode-vs-render split under "avg decode ms:" only when this returns nonzero.

Reimplemented in H264Decoder, and MultiVideoDecoder.

videoFrameCount()

uint32_t videoFrameCount ( )

inline

videoPts()

uint64_t videoPts ( )

inlineprotected

write()

size_t write ( const uint8_t *  data, size_t  len  )

inlineoverridevirtual

Implements VideoOutput.

writeAccessUnit()

void writeAccessUnit ( uint16_t  pid, uint8_t  stream_id, uint8_t &  cc, const uint8_t *  data, size_t  len, uint64_t  pts, bool  withPcr  )

inlineprotected

Builds the (ISO/IEC 13818-1) PES header for one PTS-tagged access unit and hands the whole PES packet (header + payload) to packBytesToTs().

writePat()

void writePat ( )

inlineprotected

Writes the Program Association Section: one program (program_number 1) pointing at the PMT PID.

writePcr()

static void writePcr ( uint8_t *  buf, uint64_t  pcr_base  )

inlinestaticprotected

Encodes a PCR (program_clock_reference) field: a 33-bit base (this muxer's 90kHz PTS-style clock, matching the video track's own timestamps rather than a wall clock - same simplification MuxerMPG makes for its SCR) plus a 9-bit extension, always 0 here (adequate precision for a streaming muxer, not frame-accurate 27MHz jitter control).

writePmt()

void writePmt ( )

inlineprotected

Writes the Program Map Section: the video track (PCR_PID is always the video PID) plus, if configured, the audio track.

writeStreamEntry()

void writeStreamEntry ( uint8_t *  pkt, size_t &  pos, uint8_t  stream_type, uint16_t  pid  )

inlineprotected

writeTsTimestamp()

static void writeTsTimestamp ( uint8_t *  b, uint64_t  ts  )

inlinestaticprotected

Writes a 5-byte PTS/DTS-only timestamp field (ISO/IEC 13818-1 2.4.3.7): 4-bit marker (0x02 for a PTS-only field), 3x15-bit chunks of the 33-bit timestamp each terminated by a marker_bit.

Member Data Documentation

audio_cc

uint8_t audio_cc = 0

protected

audio_frame_count

uint32_t audio_frame_count = 0

protected

audio_info

AudioInfoFormat audio_info

protected

audio_samples_per_frame

uint32_t audio_samples_per_frame = 1024

protected

has_audio

bool has_audio = false

protected

is_open

bool is_open = false

protected

p_out

Print* p_out = nullptr

protected

p_print

Print* p_print = nullptr

protectedinherited

pat_cc

uint8_t pat_cc = 0

protected

pmt_cc

uint8_t pmt_cc = 0

protected

video_cc

uint8_t video_cc = 0

protected

video_cfg

MuxerVideoConfig video_cfg

protected

video_frame_count

uint32_t video_frame_count = 0

protected

write_stream_type

StreamContentType write_stream_type = StreamContentType::Video

protected

---

The documentation for this class was generated from the following file:

• src/AudioTools/AudioCodecs/ContainerMTS.h