H.264 RTP Encoder - Packetizes Annex-B H.264 access units per RFC 6184 for RTSP video streaming. More...

#include <H264RtpEncoder.h>

Inheritance diagram for H264RtpEncoder:

[legend]

Collaboration diagram for H264RtpEncoder:

[legend]

Public Member Functions
	H264RtpEncoder ()=default

	H264RtpEncoder (size_t maxFragmentSize)
	Constructor with maximum fragment size.

AudioInfo	audioInfo () override
	provides the actual input AudioInfo

virtual AudioInfo	audioInfoOut ()

bool	begin () override

virtual bool	begin (AudioInfo info)

void	end () override

virtual uint32_t	frameDurationUs ()
	Optional rtsp function: provide the frame duration in microseconds.

virtual int	frameSize ()

RTSPFormat &	getFormat () override
	Get the media format configuration.

const char *	mime () override
	Provides the mime type of the encoded result.

	operator bool () override

int	packetSize () override
	Size of the next queued, ready-to-send fragment, for sources that provide already RTP-payload-ready, self-delimited fragments (e.g. RFC 2435 JPEG) instead of a continuous byte stream. For these sources, readBytes() must always be called with exactly packetSize() as maxBytes, so each call retrieves exactly one complete fragment instead of the transport having to guess where one RTP payload ends and the next begins.

int	readBytes (void *dest, int maxBytes) override

virtual uint16_t	samplesPerFrame ()
	Optional rtsp function: provide samples per the frame.

void	setAudioInfo (AudioInfo info) override
	Defines the sample rate, number of channels and bits per sample.

void	setFormat (RTSPFormat &format) override

void	setMaxFragmentSize (size_t size)
	Set the maximum RTP payload size per packet.

void	setMaxQueuedFrames (size_t frames)
	Maximum number of complete frames buffered when the consumer falls behind.

virtual void	setOutput (Print &out_stream) override
	Default output assignment (encoders may override to store Print reference)

void	start () override
	Initialize media source for streaming.

void	stop () override
	Cleanup media source after streaming.

size_t	write (const uint8_t *data, size_t len) override
	Process one complete Annex-B access unit (all NAL units of one encoded frame) and queue the resulting RTP fragments.

Protected Member Functions
void	appendFragment (const uint8_t *payload, size_t len, bool last)

void	cachePps (const uint8_t *nal, size_t len)

void	cacheSps (const uint8_t *nal, size_t len)

void	clearQueue ()

void	dropOldestFrame ()

void	emitNal (const uint8_t *nal, size_t nalLen, bool isLastOfFrame)
	Queue one NAL unit as a Single NAL Unit packet, or split it into FU-A fragments (RFC 6184 §5.6/§5.8) if it is larger than m_maxFragmentSize.

void	extractNALs (const uint8_t *data, size_t len, Vector< size_t > &offsets, Vector< size_t > &lengths)

size_t	findStartCode (const uint8_t *data, size_t len, size_t from)
	Locate the next Annex-B start code (00 00 01 or 00 00 00 01) at or after `from`.

int	queuePacketSize ()

int	queueReadBytes (void *dest, int maxBytes)

uint32_t	readQueueU32 (size_t pos)

void	updateSpropParameterSets ()

void	writeBlocking (Print out, uint8_t data, size_t len)

Static Protected Member Functions
static String	base64Encode (const uint8_t *data, size_t len)

Protected Attributes
RTSPFormatH264	default_format

AudioInfo	info

bool	m_isStarted = false

size_t	m_maxFragmentSize = 1400

size_t	m_maxQueuedFrames = 1

Vector< uint8_t >	m_pps

Vector< uint8_t >	m_queue

size_t	m_queuedFrames = 0

size_t	m_queueHead = 0

Vector< uint8_t >	m_scratch

Vector< uint8_t >	m_sps

RTSPFormat *	p_format = nullptr

Detailed Description

H.264 RTP Encoder - Packetizes Annex-B H.264 access units per RFC 6184 for RTSP video streaming.

Like JPEGRtpEncoder, this class plays both roles needed to get a video frame onto the wire:

As an AudioEncoder, write() accepts one complete Annex-B access unit (all the NAL units for one encoded frame, e.g. the output of a single H264Encoder::encode()/captureH264() call from the codec-h264-ESP32S3 library) per call - there is no cross-call accumulation, since Annex-B has no reliable in-stream "end of frame" marker to detect the way JPEG has 0xFFD9.
As an IMediaSource, packetSize()/readBytes() hand out the resulting RTP payloads one fragment at a time: NAL units that fit in one packet are sent as Single NAL Unit packets (payload = NAL bytes verbatim); larger ones are split into FU-A fragments, each with its own 2-byte FU header.

SPS (type 7) and PPS (type 8) NAL units are cached as soon as they are seen and re-sent (as their own Single NAL Unit packets) immediately before every IDR frame, so clients that join mid-stream or lose a packet can still resync - the same approach used by RFC 6184-based servers in general. The cached SPS is also used to derive profile-level-id and, together with the cached PPS, sprop-parameter-sets for the SDP - pushed into the associated RTSPFormatH264 as soon as both are available (see RTSPFormatH264 for why this can only happen after the first IDR frame).

Usage with RTSPOutput:

RTSPFormatH264 h264Format(1280, 720, 15.0f);
H264RtpEncoder h264Encoder;
RTSPOutput<RTSPPlatformWiFi> rtspOutput(h264Format, h264Encoder);

Author: Phil Schatzmann

Constructor & Destructor Documentation

◆ H264RtpEncoder() [1/2]

H264RtpEncoder ( )

default

◆ H264RtpEncoder() [2/2]

H264RtpEncoder ( size_t maxFragmentSize )

inline

Constructor with maximum fragment size.

Parameters

maxFragmentSize Maximum RTP payload size per packet (default: 1400 bytes)

Member Function Documentation

◆ appendFragment()

void appendFragment	(	const uint8_t *	payload,
		size_t	len,
		bool	last
	)

inlineprotectedinherited

◆ audioInfo()

AudioInfo audioInfo ( )

inlineoverridevirtualinherited

provides the actual input AudioInfo

Implements AudioInfoSupport.

Reimplemented in CodecNOP, MP3EncoderShine, BinaryContainerEncoder, and MP3ParserEncoder.

◆ audioInfoOut()

virtual AudioInfo audioInfoOut ( )

inlinevirtualinherited

provides the actual output AudioInfo: this is usually the same as audioInfo() unless we use a transforming stream

Reimplemented in MP3EncoderShine, PureDataStream, PWMAudioOutput< PWMDriverT >, ChannelFormatConverterStreamT< T >, ChannelFormatConverterStream, NumberFormatConverterStreamT< TFrom, TTo >, NumberFormatConverterStream, FormatConverterStream, Pipeline, ResampleStream, ResampleStreamT< TInterpolator >, and SupportedRatesStream.

◆ base64Encode()

static String base64Encode	(	const uint8_t *	data,
		size_t	len
	)

inlinestaticprotected

◆ begin() [1/2]

bool begin ( )

inlineoverridevirtual

Implements AudioWriter.

◆ begin() [2/2]

virtual bool begin ( AudioInfo info )

inlinevirtualinherited

Reimplemented in OpusOggEncoder, OggContainerEncoder, DecoderBasic, AudioDecoder, AACEncoderFAAC, AACEncoderFDK, AACEncoderVO, OggContainerDecoder, and MetaDataFilterEncoder.

◆ cachePps()

void cachePps	(	const uint8_t *	nal,
		size_t	len
	)

inlineprotected

◆ cacheSps()

void cacheSps	(	const uint8_t *	nal,
		size_t	len
	)

inlineprotected

◆ clearQueue()

void clearQueue ( )

inlineprotectedinherited

◆ dropOldestFrame()

void dropOldestFrame ( )

inlineprotectedinherited

◆ emitNal()

void emitNal	(	const uint8_t *	nal,
		size_t	nalLen,
		bool	isLastOfFrame
	)

inlineprotected

Queue one NAL unit as a Single NAL Unit packet, or split it into FU-A fragments (RFC 6184 §5.6/§5.8) if it is larger than m_maxFragmentSize.

Parameters

isLastOfFrame True if this is the last NAL unit of the access unit - propagated to its last fragment's isLast flag.

◆ end()

void end ( )

inlineoverridevirtual

Implements AudioWriter.

◆ extractNALs()

void extractNALs	(	const uint8_t *	data,
		size_t	len,
		Vector< size_t > &	offsets,
		Vector< size_t > &	lengths
	)

inlineprotected

Splits an Annex-B buffer into its individual NAL units (start codes stripped), recording each one's offset/length into data.

◆ findStartCode()

size_t findStartCode	(	const uint8_t *	data,
		size_t	len,
		size_t	from
	)

inlineprotected

Locate the next Annex-B start code (00 00 01 or 00 00 00 01) at or after from.

Returns: Index of the first 0x00 of the start code, or len if none found

◆ frameDurationUs()

virtual uint32_t frameDurationUs ( )

inlinevirtualinherited

Optional rtsp function: provide the frame duration in microseconds.

Reimplemented in ADPCMEncoder, MP3EncoderShine, OpusOggEncoder, MP3ParserEncoder, and MetaDataFilterEncoder.

◆ frameSize()

virtual int frameSize ( )

inlinevirtualinherited

Optional rtsp function: provide the encoded size (in bytes) of one frame, once known (e.g. after the encoder has seen real data). Lets an RTSPFormat size its RTP fragments to match one frame instead of a fixed guess, so throughput actually tracks frameDurationUs() - a fragment size covering several frames' worth of bytes sent at one-frame timing overruns real-time bandwidth and, over UDP (no flow control), causes packet loss. Returns 0 if not yet known/not applicable (fixed-frame-size codecs have no need to override this).

Reimplemented in ADPCMEncoder, EncoderALAC, AMRNBEncoder, AMRWBEncoder, and MP3ParserEncoder.

◆ getFormat()

RTSPFormat & getFormat ( )

inlineoverridevirtual

Get the media format configuration.

Returns the RTSPFormat object that describes the media data characteristics including sample rate, bit depth, number of channels, and RTP packaging parameters. If no format has been explicitly set, creates a default 16-bit PCM format at 16kHz mono.

Returns: Pointer to RTSPFormat describing the media characteristics

Note: Default format: 16-bit PCM, 16000 Hz, 1 channel

See also: setFormat(), RTSPFormatPCM

Implements IMediaSource.

◆ mime()

const char * mime ( )

inlineoverridevirtual

Provides the mime type of the encoded result.

Implements AudioEncoder.

◆ operator bool()

operator bool ( )

inlineoverridevirtual

Implements AudioWriter.

◆ packetSize()

int packetSize ( )

inlineoverridevirtual

Size of the next queued, ready-to-send fragment, for sources that provide already RTP-payload-ready, self-delimited fragments (e.g. RFC 2435 JPEG) instead of a continuous byte stream. For these sources, readBytes() must always be called with exactly packetSize() as maxBytes, so each call retrieves exactly one complete fragment instead of the transport having to guess where one RTP payload ends and the next begins.

Call it once before each readBytes() to size that call, and again right after to tell whether the fragment just read was the last one currently available - a 0 result means the caller should set the RTP marker bit on the fragment it just sent (and may advance the timestamp for the next frame).

Returns: -1 if this source is not packetized (use the plain readBytes() stream instead), 0 if packetized but nothing is queued right now, otherwise the size in bytes of the next fragment

Reimplemented from IMediaSource.

◆ queuePacketSize()

int queuePacketSize ( )

inlineprotectedinherited

◆ queueReadBytes()

int queueReadBytes	(	void *	dest,
		int	maxBytes
	)

inlineprotectedinherited

Retrieves the next queued fragment. Call queuePacketSize() first and pass its result as maxBytes so exactly one fragment is read.

◆ readBytes()

int readBytes	(	void *	dest,
		int	maxBytes
	)

inlineoverridevirtual

Retrieves the next queued fragment. Call packetSize() first and pass its result as maxBytes so exactly one fragment is read.

Implements IMediaSource.

◆ readQueueU32()

uint32_t readQueueU32 ( size_t pos )

inlineprotectedinherited

◆ samplesPerFrame()

virtual uint16_t samplesPerFrame ( )

inlinevirtualinherited

Optional rtsp function: provide samples per the frame.

Reimplemented in OpusAudioEncoder, OpusMultiStreamAudioEncoder, MP3EncoderShine, MP3ParserEncoder, and MetaDataFilterEncoder.

◆ setAudioInfo()

void setAudioInfo ( AudioInfo from )

inlineoverridevirtual

Defines the sample rate, number of channels and bits per sample.

Reimplemented from AudioEncoder.

◆ setFormat()

void setFormat ( RTSPFormat & format )

inlineoverridevirtual

Associates the RTSPFormat (an RTSPFormatH264) that receives sprop-parameter-sets/profile-level-id once SPS/PPS are known (via the virtual RTSPFormat::setSpropParameterSets()/setProfileLevelId() hooks) and supplies width/height/framerate.

Implements RTSPVideoEncoder.

◆ setMaxFragmentSize()

void setMaxFragmentSize ( size_t size )

inline

Set the maximum RTP payload size per packet.

◆ setMaxQueuedFrames()

void setMaxQueuedFrames ( size_t frames )

inline

Maximum number of complete frames buffered when the consumer falls behind.

◆ setOutput()

virtual void setOutput ( Print & out_stream )

inlineoverridevirtualinherited

Default output assignment (encoders may override to store Print reference)

Implements AudioWriter.

Reimplemented in EncoderBasic, DSFEncoder, WAVEncoder, EncoderBase64, CodecNOP, APTXEncoder, Codec2Encoder, CopyEncoder, G722Encoder, GSMEncoder, ILBCEncoder, LC3Encoder, MP3EncoderLAME, EncoderNetworkFormat, SBCEncoder, AACEncoderFAAC, AACEncoderFDK, AACEncoderVO, ADPCMEncoder, ADPCMEncoderXQ, EncoderALAC, AMRNBEncoder, AMRWBEncoder, CodecChain, FLACEncoder, EncoderFloat, G729Encoder, G7xxEncoder, EncoderL16, EncoderL8, MP3EncoderShine, OpusAudioEncoder, OpusMultiStreamAudioEncoder, MP3ParserEncoder, MetaDataFilterEncoder, BinaryContainerEncoder, OSCContainerEncoder, and OggContainerEncoder.

◆ start()

void start ( )

inlineoverridevirtual

Initialize media source for streaming.

Called when streaming is about to begin. Implementations should use this to initialize hardware, allocate buffers, start media capture, or perform any other setup required for media data generation.

Note: Called by RTSPMediaStreamer.start(); Default implementation does nothing (suitable for sources that need no setup)

Reimplemented from IMediaSource.

◆ stop()

void stop ( )

inlineoverridevirtual

Cleanup media source after streaming.

Called when streaming has ended. Implementations should use this to release hardware resources, deallocate buffers, stop media capture, or perform cleanup operations.

Note: Called by RTSPMediaStreamer.stop(); Default implementation does nothing (suitable for sources that need no cleanup)

Reimplemented from IMediaSource.

◆ updateSpropParameterSets()

void updateSpropParameterSets ( )

inlineprotected

◆ write()

size_t write	(	const uint8_t *	data,
		size_t	len
	)

inlineoverridevirtual

Process one complete Annex-B access unit (all NAL units of one encoded frame) and queue the resulting RTP fragments.

Parameters

data	Pointer to Annex-B H.264 data (one or more start-code delimited NAL units belonging to a single frame)
len	Length of the data in bytes

Returns: Number of bytes processed (always the full input, matching AudioEncoder::write() semantics; call once per encoded frame)

Implements AudioWriter.

◆ writeBlocking()

void writeBlocking	(	Print *	out,
		uint8_t *	data,
		size_t	len
	)

RTSPFormat* p_format = nullptr

protected

The documentation for this class was generated from the following file:

src/AudioTools/Communication/RTSP/H264RtpEncoder.h

Public Member Functions

Protected Member Functions

Static Protected Member Functions

Protected Attributes

Detailed Description

Constructor & Destructor Documentation

◆ H264RtpEncoder() [1/2]

◆ H264RtpEncoder() [2/2]

Member Function Documentation

◆ appendFragment()

◆ audioInfo()

◆ audioInfoOut()

◆ base64Encode()

◆ begin() [1/2]

◆ begin() [2/2]

◆ cachePps()

◆ cacheSps()

◆ clearQueue()

◆ dropOldestFrame()

◆ emitNal()

◆ end()

◆ extractNALs()

◆ findStartCode()

◆ frameDurationUs()

◆ frameSize()

◆ getFormat()

◆ mime()

◆ operator bool()

◆ packetSize()

◆ queuePacketSize()

◆ queueReadBytes()

◆ readBytes()

◆ readQueueU32()

◆ samplesPerFrame()

◆ setAudioInfo()

◆ setFormat()

◆ setMaxFragmentSize()

◆ setMaxQueuedFrames()

◆ setOutput()

◆ start()

◆ stop()

◆ updateSpropParameterSets()

◆ write()

◆ writeBlocking()

Member Data Documentation

◆ default_format

◆ info

◆ m_isStarted

◆ m_maxFragmentSize

◆ m_maxQueuedFrames

◆ m_pps

◆ m_queue

◆ m_queuedFrames

◆ m_queueHead

◆ m_scratch

◆ m_sps

◆ p_format