https://github.com/ossrs/ffmpeg-webrtc/pull/1 Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} ossrs / ffmpeg-webrtc Public forked from FFmpeg/FFmpeg * Notifications * Fork 10.9k * Star 25 * Code * Pull requests 1 * Actions * Security * Insights More * Code * Pull requests * Actions * Security * Insights New issue Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community. Pick a username [ ] Email Address [ ] Password [ ] [ ] Sign up for GitHub By clicking "Sign up for GitHub", you agree to our terms of service and privacy statement. We'll occasionally send you account related emails. Already on GitHub? Sign in to your account Jump to bottom FFmpeg: Add Whip Muxer support for subsecond latency streaming #1 Open winlinvip wants to merge 44 commits into ossrs:feature/whip base: feature/whip Choose a base branch [ ] Branches Tags Could not load branches Branch not found: {{ refName }} {{ refName }} default Could not load tags Nothing to show {{ refName }} default Are you sure you want to change the base? Some commits from the old base branch may be removed from the timeline, and old review comments may become outdated. Change base from winlinvip:feature/rtc-muxer Open FFmpeg: Add Whip Muxer support for subsecond latency streaming #1 winlinvip wants to merge 44 commits into ossrs:feature/whip from winlinvip:feature/rtc-muxer +2,291 -0 Conversation 66 Commits 44 Checks 0 Files changed 7 Conversation This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters. Learn more about bidirectional Unicode characters Show hidden characters winlinvip Copy link Member @winlinvip winlinvip commented Apr 16, 2023 * edited WHIP stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers. Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C. However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg. To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 2k lines of C code. Note: We've included Janus, Pion, Millicast, and SRS as examples of WHIP servers you can use. However, there are many other options available, such as Galene, Deadsfu, and more. For more information, please see the ietf112-hackathon-whip.pdf. Usage Please select a WHIP server to work with FFmpeg. * Usage: FFmpeg + Janus * Usage: FFmpeg + SRS If you encounter any issues or get stuck, please leave us a message on Discord. Usage: FFmpeg + Janus We referred to WISH, WHIP and Janus: Part II to make it possible to use FFmpeg for publishing a stream to Janus via WHIP. We have also created a demo docker image janus-docker for quick testing. git clone https://github.com/winlinvip/janus-docker.git cd janus-docker Initially, run the demo Docker that includes Janus server using the following command: cd ~/git/janus-docker ip="192.168.3.85" && sed -i '' "s/nat_1_1_mapping.*/nat_1_1_mapping=\"$ip\"/g" janus.jcfg docker run --rm -it -p 8081:8080 -p 8188:8188 -p 8443:8443 -p 20000-20010:20000-20010/udp \ -v $(pwd)/janus.jcfg:/usr/local/etc/janus/janus.jcfg \ -v $(pwd)/janus.plugin.videoroom.jcfg:/usr/local/etc/janus/janus.plugin.videoroom.jcfg \ -v $(pwd)/janus.transport.http.jcfg:/usr/local/etc/janus/janus.transport.http.jcfg \ -v $(pwd)/janus.transport.websockets.jcfg:/usr/local/etc/janus/janus.transport.websockets.jcfg \ -v $(pwd)/videoroomtest.js:/usr/local/share/janus/demos/videoroomtest.js \ ossrs/janus:v1.0.11 Note: Kindly modify the IP address to your own IP address. You can use ifconfig or ipconfig to determine it. After that, access the URL http://localhost:8081/videoroomtest.html? room=2345 in your browser to join the Janus room. Next, download and run the Simple WHIP Server for Janus using the following command: git clone https://github.com/meetecho/simple-whip-server.git cd simple-whip-server npm install npm run build npm run start Generate a WHIP handler using curl, which will enable ffmpeg to join the same Janus room through WHIP: curl -H 'Content-Type: application/json' -d '{"id": "abc123", "room": 2345}' \ http://localhost:7080/whip/create To download the code and build FFmpeg, you can use the following command. cd ~/git git clone -b feature/rtc-muxer https://github.com/winlinvip/ffmpeg-webrtc.git ./configure --enable-muxer=rtc --enable-openssl --enable-version3 \ --enable-libx264 --enable-gpl --enable-libopus make -j10 Note: To enable DTLS handshake, OpenSSL is mandatory. Please install OpenSSL, for instance, brew install openssl, and then configure the environment by running export PKG_CONFIG_PATH="/usr /local/opt/openssl@3/lib/pkgconfig". Note: For demonstration purposes, you can install libx264 by running brew install x264 and libopus by running brew install opus. Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg. ~/git/FFmpeg/ffmpeg_g -re -i ~/git/srs/trunk/doc/source.flv \ -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \ -f rtc 'http://localhost:7080/whip/endpoint/abc123' # Or you can also capture your screen, and measure the end-to-end latency. ~/git/FFmpeg/ffmpeg_g -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \ -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \ -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \ -acodec libopus -ar 48000 -ac 2 \ -f rtc 'http://localhost:7080/whip/endpoint/abc123' After publishing the stream to the Janus room, you will be able to view it on the previously opened webpage. The image below shows that the latency is around 480ms. image Note: Initially, the latency was about 150ms, but it later increased to approximately 480ms. We have reported this issue at here. Usage: FFmpeg + Pion On the way. Usage: FFmpeg + Millicast On the way. Usage: FFmpeg + SRS To enable FFmpeg to publish a stream, SRS can be used as the WHIP server. It is recommended to use docker. ip="192.168.3.85" docker run --rm -it -p 1935:1935 -p 1985:1985 -p 8080:8080 \ --env CANDIDATE=$ip -p 8000:8000/udp \ ossrs/srs:5 ./objs/srs -c conf/rtc2rtmp.conf Alternatively, you can build SRS from its source code. cd ~/git git clone -b develop https://github.com/ossrs/srs.git cd srs/trunk ./configure make -j10 # After building SRS, you may run it using a configuration file. cd ~/git/srs/trunk ./objs/srs -c conf/rtc2rtmp.conf Note: Please upgrade to SRS version 5.0.153 or higher, or 6.0.43 or higher. To download the code and build FFmpeg, you can use the following command. cd ~/git git clone -b feature/rtc-muxer https://github.com/winlinvip/ffmpeg-webrtc.git ./configure --enable-muxer=rtc --enable-openssl --enable-version3 \ --enable-libx264 --enable-gpl --enable-libopus make -j10 Note: To enable DTLS handshake, OpenSSL is mandatory. Please install OpenSSL, for instance, brew install openssl, and then configure the environment by running export PKG_CONFIG_PATH="/usr /local/opt/openssl@3/lib/pkgconfig". Note: For demonstration purposes, you can install libx264 by running brew install x264 and libopus by running brew install opus. Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg. ~/git/FFmpeg/ffmpeg_g -re -i ~/git/srs/trunk/doc/source.flv \ -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \ -f rtc "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream" # Or you can also capture your screen, and measure the end-to-end latency. ~/git/FFmpeg/ffmpeg_g -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \ -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \ -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \ -acodec libopus -ar 48000 -ac 2 \ -f rtc "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream" After publishing stream to SRS, you can play the WHIP stream in web browser such as Chrome, using srs-player. * WHEP: http://localhost:1985/rtc/v1/whip-play/?app=live&stream= livestream The image below shows that the latency is around 150ms. image The RTMP, HTTP-FLV, or HTTP-TS stream remuxed by SRS can be played using ffplay, VLC, or srs-player. Known Issues The current version has some known issues that we plan to fix in the future. You are welcome to help us fix them by sending a pull request. The current known issues include: * Long First Screen Wait Time: Respond to the PLI request for quick decoding and rendering; otherwise, it takes a GOP's duration to display the first decoded image. A RTC encoder ought to have the capability to refresh the IDR frame when a new player joins or there is packet loss, as requested by the player through a PLI. When working within the ffmpeg framework, it's not possible to control the encoder in muxer. This means we cannot force an encoder to generate an IDR frame right away. One possible solution could be to cache the IDR frame or a GOP of frames in the RTC muxer and send it to the server upon receiving a PLI request. The player can then handle any latency issues that may arise. * Multiple Slices Issue: If the -tune zerolatency option is enabled and threads>1, FFmpeg will encode the frame in multiple slices. This process can cause stuttering in Chrome's decoding, with only the IDR being able to be decoded. Both the IDR and P frames may be encoded to multiple slices and sent by multiple RTP packets with the same timestamp. RTC players, such as Chrome, may have issues decoding frames that have been encoded using multiple slices. As a result, Chrome may only decode some of the slices of the IDR frame and drop the P frames. If this issue occurs, it may appear as if the video player is stuttering and the decoded framerate will be equivalent to the GOP size. * In addition to OpenSSL, support for other cryptographic libraries such as GnuTLS, mbedtls, etc., is needed. To achieve this, a dtls.c file should be extracted, similar to tls.c, for better utilization. * Presently, only the client role is supported for DTLS, while the server role remains unsupported. Although not mandatory, the DTLS client is commonly used in ffmpeg as a WebRTC client. * Compatibility is limited to OpenSSL 1.1.1b or newer versions. To prevent build failures, it may be necessary to ensure compatibility with older versions. * Features such as congestion control, NACK, and FEC are not yet implemented. While NACK is essential, FEC remains optional. * Currently, only the first candidate (UDP and host type) is utilized. A mechanism to determine the fastest candidate, select the best one, or switch during publishing is needed. * When remuxing WebRTC to RTMP or HTTP-FLV, stuttering can occur every N seconds, which may be the same as the GOP size. Note that this bug occurs during screen streaming, but there is no issues when converting a file to WHIP streaming. * While capturing the screen using FFmpeg, the h.264 profile may be negative (e.g. -99), in which case we set the profile to 0x42 (baseline). However, in such a situation, it may be necessary to parse from SPS/PPS. Below are the issues that have already been fixed: * Need to improve the end-to-end latency to 150ms from 800ms. Fixed by @T-bagwell @mypopydev @winlinvip * For SRS to convert RTC to RTMP using time information, it is essential to send an RTCP SR report. Fixed by @duiniuluantanqin * Only send binding requests to the server and cannot receive binding requests from the server. Fixed by @cloudwebrtc. In addition, he assists in enabling ICE use-candidate for Janus or Pion to finalize the ICE handshake. * WHIP currently supports only HTTP, with HTTPS yet to be implemented. However, incorporating HTTPS should be straightforward, as ffmpeg already contains the necessary code. Note: The ffmpeg HTTP library has the functionality to support both the HTTP and HTTPS protocols. * Support baseline/main/high profile without B frames. Even though WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. See bd9f7d1 @duiniuluantanqin * During the DTLS handshake, when FFmpeg receives a ServerHello from the server, it generates a ClientHello for retransmission with a Certificate. This causes the server to retransmit the ServerHello, which still works but is not efficient. We should try to eliminate the unnecessary retransmission of the ClientHello. See 3b3b17a @winlinvip Latency To test the end-to-end latency of a WIHP stream published using FFmpeg, you can capture your desktop using FFmpeg, open a stopwatch or miaobiao in the browser, and compare the player with the original stopwatch. ~/git/FFmpeg/ffmpeg_g -f avfoundation -framerate 25 -pixel_format yuyv422 -i "2:0" \ -vcodec libx264 -pix_fmt yuv420p -profile:v baseline -preset:v ultrafast \ -b:v 800k -s 1024x576 -r 25 -g 50 -tune zerolatency -threads 1 -bf 0 \ -acodec libopus -ar 48000 -ac 2 \ -f rtc 'http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream' Note: The parameter -i "2:0" is formatted as a video:audio device ID. Please use the ~/git/FFmpeg/ffmpeg_g -f avfoundation -list_devices true -i "" command to list device ID and information. Note: It is recommended to keep the threads=1 setting to prevent the occurrence of the Multiple Slices Issue. Please refer to Known Issues for more information. The test results are incredible! The image below shows that the latency is around 150ms. image OpenSSL Below is a list of OpenSSL versions that are supported. * OpenSSL 1.0.1k and newer. Note that OpenSSL 1.0.1j does not work, please refer to this link. * OpenSSL 1.0.2 and newer. * OpenSSL 1.1.0h and newer. Note that the compilation of OpenSSL 1.1.0g failed, but the API should work if you can fix the compilation bug. * OpenSSl 3.0.0 and newer. * OpenSSl 3.1.0 and newer. In short, OpenSSL 1.0.1k and newer versions should work. Authorization Set option -authorization token to use the authorization of WHIP, please refer to the Authentication and authorization. ~/git/FFmpeg/ffmpeg_g -re -i ~/git/srs/trunk/doc/source.flv \ -vcodec libx264 -profile:v baseline -r 25 -g 50 -acodec libopus -ar 48000 -ac 2 \ -f rtc -authorization "mF_9.B5f-4.1JqM" \ "http://localhost:1985/rtc/v1/whip/?app=live&stream=livestream" Note: The token is mF_9.B5f-4.1JqM, and the HTTP header is set to Authorization: Bearer mF_9.B5f-4.1JqM. [241973872-] Authors This patch has been created and is maintained by the developers below. * Steven Liu liuqi05@kuaishou.com * winlin winlinvip@gmail.com * yangrtc yangrtc@aliyun.com * cloudwebrtc duanweiwei1982@gmail.com * Haibo Chen 495810242@qq.com Links WHIP: https://datatracker.ietf.org/doc/draft-ietf-wish-whip/ metaRTC: feature/metaRTC SRS: ossrs/srs#3170 For RTP plaintext mode: Add RTP Plaintext Mode for Debugging with Wireshark Sorry, something went wrong. [?] 3 feniljain, cloudwebrtc, and griimick reacted with heart emoji All reactions * [?] 3 reactions @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 5 times, most recently from 8c1b958 to 15cf4e4 Compare April 21, 2023 22:41 @winlinvip WHIP: Add WebRTC WHIP muxer. 9587390 @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 2 times, most recently from 598aecc to cf69b01 Compare April 21, 2023 23:16 @winlinvip winlinvip changed the title [DEL:WHIP: Support FFmpeg WebRTC muxer via WHP protocol.:DEL] [INS:FFmpeg: Support WebRTC muxer via WHP protocol.:INS] Apr 21, 2023 @winlinvip winlinvip changed the title [DEL:FFmpeg: Support WebRTC muxer via WHP protocol.:DEL] [INS:FFmpeg: Support WebRTC muxer via WHIP protocol.:INS] Apr 21, 2023 @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 4 times, most recently from 6af6864 to 2bfb273 Compare April 21, 2023 23:52 winlinvip and others added 3 commits April 22, 2023 09:06 @winlinvip WHIP: Only support h264+opus codec. ... 8a9c829 1. Input stream codec should be h264 or opus. 2. For video codec profile, should be h264 baseline or constrained baseline. 3. For audio, should be 48000HZ and stereo. 4. Only support one video and audio stream. @yangrtc @winlinvip WHIP: Generate and exchange offer server to get answer. ... 54e45e4 1. Generate random ice ufrag, pwd and ssrc. 2. Use HTTP POST to send offer to server and read answer. 3. Logging offer and answer in verbose level. @winlinvip WHIP: Support parse ice from answer. ... 2e26baf 1. Parse ice username and pwd. 2. Parse ice candidate, the priority and host. 3. Only support udp. @winlinvip winlinvip force-pushed the feature/rtc-muxer branch from 2bfb273 to 2e26baf Compare April 22, 2023 01:08 T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed Apr 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 3 times, most recently from 48d93eb to 92c4187 Compare April 22, 2023 23:23 @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 3 times, most recently from c8cff8d to c909292 Compare May 1, 2023 04:37 101 hidden items Load more... @winlinvip WHIP: Merge write header to init. 4a81e3c @winlinvip winlinvip requested review from T-bagwell and duiniuluantanqin May 18, 2023 22:16 @duiniuluantanqin WHIP:Support baseline/main/high profile without B frames (#2) bd9f7d1 T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 22, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved @winlinvip WHIP: Fix the SSL deprecated warning by replacing EC_KEY_new with EVP... ... eeb462e ..._EC_gen. @winlinvip winlinvip requested a review from T-bagwell May 23, 2023 04:14 winlinvip added 3 commits May 23, 2023 12:30 @winlinvip WHIP: Refine the code to be shorter. 9c1c81a @winlinvip WHIP: Increase the base timeout and thereby reduce the number of unne... ... 3b3b17a ...cessary ClientHello. @winlinvip WHIP: Refine ARQ for DTLS with bug fixed. 5c7c95d T-bagwell T-bagwell reviewed May 29, 2023 View reviewed changes libavformat/rtcenc.c Outdated Show resolved Hide resolved T-bagwell T-bagwell reviewed May 29, 2023 View reviewed changes libavformat/rtcenc.c Show resolved Hide resolved T-bagwell T-bagwell reviewed May 29, 2023 View reviewed changes libavformat/rtcenc.c Show resolved Hide resolved @winlinvip WHIP: Update muxers.texi for RTC. 928c98a @winlinvip winlinvip requested a review from T-bagwell May 29, 2023 11:34 @winlinvip winlinvip force-pushed the feature/rtc-muxer branch from 1fc9a69 to 928c98a Compare May 29, 2023 11:58 @flying1314 Copy link flying1314 commented May 30, 2023 * edited Is there any plan to add whip demuxer? All reactions Sorry, something went wrong. @winlinvip winlinvip changed the base branch from master to feature/ whip May 30, 2023 13:17 @winlinvip winlinvip force-pushed the feature/rtc-muxer branch 2 times, most recently from ad1bbae to 47173e8 Compare May 30, 2023 14:47 @winlinvip WHIP: Fix openssl build error. Support authorization. Change pkt_size... ... 83678f0 ... to 1200. @winlinvip winlinvip force-pushed the feature/rtc-muxer branch from 47173e8 to 83678f0 Compare May 30, 2023 16:07 Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment Reviewers @yangrtc yangrtc yangrtc approved these changes @mypopydev mypopydev @notedit notedit @cloudwebrtc cloudwebrtc @duiniuluantanqin duiniuluantanqin @T-bagwell T-bagwell At least 6 approving reviews are required to merge this pull request. Assignees No one assigned Labels None yet Projects None yet Milestone No milestone 6 participants @winlinvip @flying1314 @T-bagwell @duiniuluantanqin @cloudwebrtc @yangrtc Add this suggestion to a batch that can be applied as a single commit. This suggestion is invalid because no changes were made to the code. Suggestions cannot be applied while the pull request is closed. Suggestions cannot be applied while viewing a subset of changes. Only one suggestion per line can be applied in a batch. Add this suggestion to a batch that can be applied as a single commit. Applying suggestions on deleted lines is not supported. You must change the existing code in this line in order to create a valid suggestion. Outdated suggestions cannot be applied. This suggestion has been applied or marked resolved. Suggestions cannot be applied from pending reviews. Suggestions cannot be applied on multi-line comments. Suggestions cannot be applied while the pull request is queued to merge. Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.