|
arduino-audio-tools
|
Video playback. More...
Modules | |
| H264 | |
| H.264 encoding/decoding using https://github.com/pschatzmann/TinyH264. | |
| H264ESP32S3 | |
| H.264 encoding/decoding on ESP32-S3 using https://github.com/pschatzmann/ESP32S3-h264 (esp_h264, hardware-assisted where available). ESP32-S3 only - H264Config.h in that library fails the build on any other target. | |
| MJPEG | |
| Motion-JPEG decoding using https://github.com/Bodmer/JPEGDecoder. | |
| MPG | |
| MPEG-1 part 2 (video) encoding/decoding using https://github.com/pschatzmann/TinyMPG. | |
Classes | |
| class | CameraFrameSource |
| VideoFrameSource that pulls frames from the ESP32 camera (esp_camera.h, the Arduino-ESP32 "esp32-camera" component) - nextFrame() returns esp_camera_fb_get()'s buffer, releasing the previous frame (esp_camera_fb_return()) first; as with any VideoFrameSource, the returned pointer is only valid until the next nextFrame() call. More... | |
| class | CameraFrameSourceOpenCV |
| VideoFrameSource that pulls frames from an OpenCV cv::VideoCapture (e.g. a USB/CSI webcam via V4L2 on Linux/Raspberry Pi, or any other OpenCV-supported backend) - the desktop/Linux counterpart to CameraFrameSource, which targets the ESP32 esp_camera.h driver instead. More... | |
| class | Demuxer |
| Common interface for demuxers (DemuxerAVI, DemuxerMP4) that split a container's video and (optional) audio tracks apart. Write code against this interface instead of a concrete class if it should work with either container format. More... | |
| class | DemuxerAVI |
| AVI Container Decoder which can be fed with small chunks of data. The minimum length must be bigger then the header size! The file structure is documented at https://learn.microsoft.com/en-us/windows/win32/directshow/avi-riff-file-reference. More... | |
| class | DemuxerMP4 |
| DemuxerMP4 extracts both the audio and (H.264) video track from a general, interleaved MP4/ISO-BMFF stream - unlike M4AAudioDemuxer/ M4AAudioFileDemuxer, which only handle a single (audio) track. More... | |
| class | DemuxerMPG |
| MPEG-1 System (Program) Stream Demuxer, as defined by ISO/IEC 11172-1: splits the pack_header/system_header/PES_packet framing apart and forwards the raw MPEG-1 video (ISO/IEC 11172-2) and MPEG-1 audio (ISO/IEC 11172-3, Layer I/II/III) elementary streams to setOutputVideo()/ setOutputAudio(). Both elementary streams are self-delimiting (the video ES has its own picture_start_code sequence, the audio ES its own frame sync word) so, like DemuxerAVI/DemuxerMP4, no separate decoding is done here - point the outputs at whatever decoder understands the codec (e.g. an MPEG1 video decoder, or an EncodedAudioStream wrapping an MP3/MP2 AudioDecoder). More... | |
| class | Muxer |
| Common interface for muxers (MuxerAVI, MuxerMP4) that combine an already-encoded video track (and optionally an audio track) into a container written to a Print (a local File to record, or e.g. a network Client to publish a live stream to an HTTP/TCP client). Write code against this interface instead of a concrete class if it should work with either container format. More... | |
| class | MuxerAVI |
| Configuration for the (single) video track written by MuxerAVI. More... | |
| class | MuxerMP4 |
| Video track configuration for MuxerMP4 - update before calling begin(). More... | |
| class | MuxerMPG |
| MPEG-1 System (Program) Stream Encoder, as defined by ISO/IEC 11172-1: muxes an already-encoded MPEG-1 video elementary stream (ISO/IEC 11172-2) and an optional MPEG-1 audio elementary stream (ISO/IEC 11172-3, Layer I/II/III) into pack_header/system_header/PES_packet framing written to a Print (a local File to record, or e.g. a network Client to publish a live stream). More... | |
| struct | MuxerVideoConfig |
| Shared video track configuration for muxers (MuxerAVI, MuxerMP4) - call before begin(). More... | |
| class | MuxerVideoSink |
| Print sink that writes each frame it receives to a Muxer's video track via addVideoFrame(), determining a real isKeyFrame value instead of relying on that method's isKeyFrame=true default - via isH264KeyFrame() (Video.h) when the Muxer's video track is VideoFormat::H264, otherwise always true (correct for e.g. MJPEG, where every frame is independently decodable). More... | |
| class | OutputOpenCV |
| Display a video frame with OpenCV, to be used on the desktop - VideoFormat::MJPEG (the default) expects one complete, already-encoded JPEG image assembled across write() calls and closed off by flush(), as produced by e.g. DemuxerAVI. Any other setVideoFormat() value (e.g. RGB565, the common raw picture format decoders like H264Decoder produce) expects one complete, already-decoded picture per write() call instead - setSize() must be called too in that case, since unlike JPEG a raw picture doesn't carry its own dimensions. More... | |
| class | OutputTFT_eSPI |
| Bridges VideoDecoder's write() calls to the TFT - VideoDecoder always hands over one complete frame per write() call (see its class comment), so this can push the whole frame in one go. Usually the data is in RGB565 format, but other formats are supported as well. More... | |
| class | OutputTinyGPU |
| Bridges VideoDecoder's write() calls to a TinyGPU ILI9341Driver - VideoDecoder always hands over one complete frame per write() call (see its class comment), so this can push the whole frame in one go. Usually the data is in RGB565 format, but other formats are supported as well. More... | |
| class | VideoAudioBufferedSync |
| Logic to Synchronize video and audio output: we use a buffer to store the audio and instead of delaying the frames with delay() we play audio. The bufferSize defines the audio buffer in bytes. The correctionMs is used to slow down or speed up the playback of the video to prevent any audio buffer underflows. More... | |
| class | VideoAudioSync |
| Logic to Synchronize video and audio output: This is the minimum implementatin which actually does not synchronize, but directly processes the data. No additinal memory is used! Provide your own optimized platform specific implementation. More... | |
| class | VideoDecoder |
| Common interface for video decoders (e.g. H264Decoder, H264DecoderESP32S3 - CodecH264.h/CodecH264ESP32S3.h) - standardizes lifecycle (begin()/end()), the Print target decoded pictures are written to, and the pixel format they're written in (setVideoFormat()), on top of VideoOutput's write()/flush() (the encoded-bitstream input side, inherited unchanged). Concrete decoders may still expose their own additional config knobs beyond this shared surface. More... | |
| class | VideoEncoder |
| Common interface for video encoders (e.g. H264Encoder, H264EncoderESP32S3 - CodecH264.h/CodecH264ESP32S3.h) - standardizes lifecycle (begin()/end()), the Print target the encoded bitstream is written to, the raw-picture input format (setVideoFormat()), and a single write() that encodes one picture in that format and writes the result to setOutput()'s target - one write() call per picture, mirroring VideoOutput's write()-per-frame convention on the decoder side. Concrete encoders may still expose their own additional config knobs (e.g. bitrate/QP) beyond this shared surface. More... | |
| class | VideoFrameSource |
| Pull-based provider of one already-encoded video/image frame at a time - e.g. wraps a camera capture + H264Encoder/MJPEG capture pipeline. Used by VideoMuxerWithTasks's video task, which calls nextFrame() once per iteration, at the rate given by videoInfo().fps. Implementations should produce/capture the frame here directly (not defer it): a slow nextFrame() only delays whatever is pulling from it. More... | |
| struct | VideoInfo |
| Basic video information (width/height/codec/frame size), analogous to AudioInfo - common to both DemuxerAVI and DemuxerMP4, accessible via their getVideoInfo() getter. More... | |
| class | VideoMuxer |
| Feeds a Muxer (MuxerAVI, MuxerMP4, ...) from a single copy() call. More... | |
| class | VideoMuxerWithTasks |
| Feeds a Muxer (MuxerAVI, MuxerMP4, ...) from two background Tasks instead of a single loop()/copy() call - one per track, each write wrapped in a LockGuard over a shared mutex (real by default; override via setMutex()), since a Muxer is not safe to call from two threads concurrently. More... | |
| class | VideoOutput |
| Abstract class for video playback. This class is used to assemble a complete video frame in memory. A video frame is written via one or more write() calls, then finalized with flush() - implementations use flush() to know a frame is complete (there is no separate frame-size hint, unlike a length-prefixed chunk format). More... | |
Enumerations | |
| enum class | AudioFormat : uint16_t { UNKNOWN = 0x0000 , PCM = 0x0001 , ADPCM = 0x0002 , IEEE_FLOAT = 0x0003 , ALAW = 0x0006 , MULAW = 0x0007 , OKI_ADPCM = 0x0010 , DVI_ADPCM = 0x0011 , IMA_ADPCM = DVI_ADPCM , MEDIASPACE_ADPCM = 0x0012 , SIERRA_ADPCM = 0x0013 , G723_ADPCM = 0x0014 , DIALOGIC_OKI_ADPCM = 0x0017 , MEDIAVISION_ADPCM = 0x0018 , YAMAHA_ADPCM = 0x0020 , ANTEX_ADPCME = 0x0033 , DIGIADPCM = 0x0036 , NMS_VBXADPCM = 0x0038 , CS_IMAADPCM = 0x0039 , ROCKWELL_ADPCM = 0x003B , G721_ADPCM = 0x0040 , G726_ADPCM = 0x0064 , G722_ADPCM = 0x0065 , INFOCOM_ITS_G721_ADPCM = 0x008B , ZYXEL_ADPCM = 0x0097 , RHETOREX_ADPCM = 0x0100 , SANYO_LD_ADPCM = 0x0125 , G726ADPCM = 0x0140 , UNISYS_NAP_ADPCM = 0x0170 , MP3 = 0x0055 , ALAC = 0x6C61 , AAC = 0xA106 } |
| Audio format codes used by Microsoft e.g. in avi or wav files. More... | |
| enum | StreamContentType { Audio , Video } |
| Which track write() feeds, for muxers (MuxerAVI, MuxerMP4) that double as a plain, generic Print-like sink for both tracks - see setStreamType()/streamType() on those classes. More... | |
| enum class | VideoFormat { H264 , MJPEG , MPEG4 , RAW , YUV422 , RGB565 , RGB666 , RGB888 , I420 , MPEG1 , UNKNOWN } |
| Video codec/pixel-format identifier, shared by two unrelated uses: the (single) video stream of a container (DemuxerAVI/MuxerAVI and DemuxerMP4 - much like AudioFormat is shared across the audio demuxers, despite its WAV-flavored name), and the decoded-picture pixel format VideoDecoder::setVideoFormat() selects. H264 is the primary target for muxing; the others are AVI-specific conveniences or decoder-only. More... | |
Functions | |
| AudioFormat | fromMime (const char *mime) |
Best-effort inverse of toMime(): maps a mime type (e.g. an AudioEncoder's mime()) back to the AudioFormat wav code it came from - AudioFormat::UNKNOWN if mime is null or doesn't match any of toMime()'s mappings (most codecs - Opus, FLAC, GSM, ... - have no wav code at all, so this can never be exhaustive; "audio/wav" itself maps back to PCM specifically, even though toMime() also uses it for every ADPCM variant, since PCM is the common case). | |
| bool | isH264KeyFrame (const uint8_t *data, size_t len) |
| True if the given Annex-B H.264 access unit contains an IDR slice NAL unit (nal_unit_type 5) - the reliable way to tell a real keyframe/sync-sample apart from a P-frame, since nothing in the byte layout itself says so without inspecting NAL headers. Used e.g. to determine Muxer::addVideoFrame()'s isKeyFrame argument for an already-encoded VideoFormat::H264 frame. | |
| bool | isWavFormat (AudioFormat format) |
| True if the wav code is handled via the WAV decoder (i.e. toMime() maps it to "audio/wav": PCM and all ADPCM variants). | |
| const char * | toMime (AudioFormat format) |
| Provides the mime type for a AudioFormat wav code, or nullptr if not known/mapped. | |
| size_t | videoFrameSizeBytes (VideoFormat format, uint16_t width, uint16_t height) |
| Fixed per-frame size (bytes) for a raw/uncompressed VideoFormat at the given resolution - 0 for compressed formats (H264/MJPEG/MPEG4) or VideoFormat::UNKNOWN, since their frame size varies per frame. | |
Video playback.
|
strong |
Audio format codes used by Microsoft e.g. in avi or wav files.
| enum StreamContentType |
|
strong |
Video codec/pixel-format identifier, shared by two unrelated uses: the (single) video stream of a container (DemuxerAVI/MuxerAVI and DemuxerMP4 - much like AudioFormat is shared across the audio demuxers, despite its WAV-flavored name), and the decoded-picture pixel format VideoDecoder::setVideoFormat() selects. H264 is the primary target for muxing; the others are AVI-specific conveniences or decoder-only.
| Enumerator | |
|---|---|
| H264 | |
| MJPEG | |
| MPEG4 | |
| RAW | |
| YUV422 | |
| RGB565 | |
| RGB666 | |
| RGB888 | |
| I420 | |
| MPEG1 | |
| UNKNOWN | |
|
inline |
Best-effort inverse of toMime(): maps a mime type (e.g. an AudioEncoder's mime()) back to the AudioFormat wav code it came from - AudioFormat::UNKNOWN if mime is null or doesn't match any of toMime()'s mappings (most codecs - Opus, FLAC, GSM, ... - have no wav code at all, so this can never be exhaustive; "audio/wav" itself maps back to PCM specifically, even though toMime() also uses it for every ADPCM variant, since PCM is the common case).
|
inline |
True if the given Annex-B H.264 access unit contains an IDR slice NAL unit (nal_unit_type 5) - the reliable way to tell a real keyframe/sync-sample apart from a P-frame, since nothing in the byte layout itself says so without inspecting NAL headers. Used e.g. to determine Muxer::addVideoFrame()'s isKeyFrame argument for an already-encoded VideoFormat::H264 frame.
|
inline |
True if the wav code is handled via the WAV decoder (i.e. toMime() maps it to "audio/wav": PCM and all ADPCM variants).
|
inline |
Provides the mime type for a AudioFormat wav code, or nullptr if not known/mapped.
|
inline |
Fixed per-frame size (bytes) for a raw/uncompressed VideoFormat at the given resolution - 0 for compressed formats (H264/MJPEG/MPEG4) or VideoFormat::UNKNOWN, since their frame size varies per frame.
1.9.8