How to download a video from an URL like *.m3u8 ? lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- Hartmut · 2024-04-04 13:48 Many videos in the internet have a mp4-Format, but sometimes the URL is like this: https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20240317_00 0539_4703373_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/index- f1-v1-a1.m3u8 https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20240320_00 5905_4753932_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/index- f1-v1-a1.m3u8 https://apasfiis.sf.apa.at/ipad/cms- worldwide_episodes/14220193_0005_Q6A.mp4/playlist.m3u8 When I download such an URL via 'TFPHTTPClient' or via synapse, then I only get a small textfile with a content like this: Example1: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i #EXTM3U#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXT-X-PLAYLIST- TYPE:VOD#EXT-X-VERSION:3#EXT-X-MEDIA-SEQUENCE:1#EXTINF:4.000,segment- 1-f1-v1-a1.ts#EXTINF:6.000,segment-2-f1-v1- a1.ts#EXTINF:10.000,segment-3-f1-v1-a1.ts...segment-35-f1-v1- a1.ts#EXTINF:10.000,segment-36-f1-v1-a1.ts#EXTINF:2.200,segment-37-f1- v1-a1.ts#EXT-X-ENDLIST Example2: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i #EXTM3U#EXT-X-TARGETDURATION:10#EXT-X-ALLOW-CACHE:YES#EXT-X-PLAYLIST- TYPE:VOD#EXT-X-VERSION:3#EXT-X-MEDIA-SEQUENCE:1#EXTINF:4.000,segment- 1-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a- bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subt itles.eai-general.aws.srf.ch#EXTINF:6.000,segment-2-f1-v1- a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a- bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subt itles.eai-general.aws.srf.ch#EXTINF:10.000,...#EXTINF:10.000,segment- 830-f1-v1-a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a- bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subt itles.eai-general.aws.srf.ch#EXTINF:8.960,segment-831-f1-v1- a1.ts?caption=srf/c871f14e-cfd9-41a7-bc3a- bb7d09cda4d8/episode/de/vod/vod.m3u8:de:Deutsch:sdh&webvttbaseurl=subt itles.eai-general.aws.srf.ch#EXT-X-ENDLIST Example3: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i #EXTM3U#EXT-X-VERSION:3#EXT-X-STREAM-INF:BANDWIDTH=2254982,CODECS="avc 1.64001f,mp4a.40.2",RESOLUTION=960x540chunklist.m3u8 My question is: how can my program download such a video? Must I reinvent the wheel or does FPC have some "library" for that? Is this somehow possible via 'TFPHTTPClient' or via synapse? Or something other which needs no package installation? Then I would prefer that. I'm on Linux Ubuntu 22.04 with FPC 3.2.2. Thanks in advance. ---------------------------------------------------------------------- lainz · 2024-04-04 14:06 You need to parse the text file and append at the URL the .ts file to download it. Quotehttps://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_202403 17_000539_4703373_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csmil/s egment-1-f1-v1-a1.ts ---------------------------------------------------------------------- Hartmut · 2024-04-04 16:23 Thanks lainz. Yes, but then all possible formats have to be considered and all the single segments have to be concatenated together. I made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC-player (only 1706 MB). As said, I would reinvent the wheel... Does FPC not have some unit or "library" for that? Do 'TFPHTTPClient' or synapse have this feature? Or something other which needs no package installation? Then I would prefer that. ---------------------------------------------------------------------- TRon · 2024-04-04 16:32 Quote from: Hartmut on April 04, 2024, 06:23:02 pmI made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC- player (only 1706 MB). As said, I would reinvent the wheel... You might want to consider reading up on MPEG transport. QuoteDoes FPC not have some unit or "library" for that? Sure. It is called ffmpeg  Kidding aside, no afaik there is not a native implementation for decoding mpeg streams. There are plenty of 3th party libraries that are able to do that for you (such as ffmpeg) and all solutions that are available for FPC/Lazarus uses such a 3th party library one way or another. ---------------------------------------------------------------------- Hartmut · 2024-04-04 16:59 Quote from: TRon on April 04, 2024, 06:32:47 pmQuote from: Hartmut on April 04, 2024, 06:23:02 pmI made a quick test and saw, that the sum of all downloaded segment files (1747 MB) is clearly bigger then if I download the same video by ffmpeg or by VLC-player (only 1706 MB). As said, I would reinvent the wheel... You might want to consider reading up on MPEG transport. Do you mean, that each single downloaded segment file contains several control data from the MPEG transport stream (which must be deleted)? Looks like pure horror... QuoteQuoteDoes FPC not have some unit or "library" for that? Sure. It is called ffmpeg  I used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg. ---------------------------------------------------------------------- TRon · 2024-04-04 17:09 Quote from: Hartmut on April 04, 2024, 06:59:36 pmDo you mean, that each single downloaded segment file contains several control data from the MPEG transport stream (which must be deleted)? Looks like pure horror... Yes and besides that synchronization frames. It /is/ horror which is probably why (again afaik) there isn't a native solution. Let others worry about the nitty gritty details and let us (simply) make use of it  QuoteI used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg. I am not sure how individual solutions solve it but it can be either the commandline or the library (the latter which provides more fine grained control). Note, that I do mention ffpmpeg but there are other libraries out there as well (it is just my limitation of not knowing others from mind). ---------------------------------------------------------------------- Hartmut · 2024-04-04 17:20 Quote from: TRon on April 04, 2024, 07:09:29 pmQuote from: Hartmut on April 04, 2024, 06:59:36 pmI used ffmpeg manually in the past to download *.m3u8 videos. But multiple times these videos had gaps and were bumpy/jerky. Sometimes I saw messages like "HTTP error 404 Not Found" or "Failed to open segment of playlist" in the huge output of ffmpeg. It looks like ffmpeg does not retry on problems and simple continues... So I hesitate to trust on ffmpeg. I am not sure how individual solutions solve it but it can be either the commandline or the library (the latter which provides more fine grained control). Thank you TRon. How can I access ffmpeg through a library? Can you give me some details please? I'm a bloody beginner to that stuff. ---------------------------------------------------------------------- TRon · 2024-04-04 17:33 Quote from: Hartmut on April 04, 2024, 07:20:14 pmThank you TRon. How can I access ffmpeg through a library? Can you give me some details please? I'm a bloody beginner to that stuff. First of all there is a wiki page on multimedia/video here. ffmpeg libraries can be "accessed" by Free Pascal by using/installing the ffmpeg header files, which is mentioned on that wiki-page and can be found here. Other than that the installation of the ffmpeg libraries is required. How to do that (properly) depends on the platform. If you are looking for example code then I would have to come up with something (new) because last time I did something like that I was still on windows (Linux now) so that might take quite some time due to time constraints at my side. There are probably plenty of c examples out there (and which usually can be translated into Pascal). ---------------------------------------------------------------------- cdbc · 2024-04-04 19:44 Hi Search for user @Metis & ffmpeg here on the forum, he does a lot with ffmpeg edit: See here:https://forum.lazarus.freepascal.org/index.php/topic,26 666.msg464784.html#msg464784 Reply #140 Regards Benny ---------------------------------------------------------------------- Hartmut · 2024-04-05 11:08 Thank you TRon for those links. I checked them but got stuck at https://github.com/DJMaster/ffmpeg-fpc. Don't know what "headers binding" is (has it to do with C?). There are plenty of source files, I had a look in some of them, but did not see something which helped me. Thanks cdbc for your post. My search got 26 matches, but I found nothing which helped me to access ffmpeg as a library. I think ffmpeg as a library is far beyond my horizon. And as I said in reply #4, I hesitate to trust on ffmpeg. So I'm looking for another solution. There exists a FPC unit for curl called 'libcurl'. Does somebody know, if curl can download a video from an URL like *.m3u8? If the answer is definetively 'yes' then I would dive into it. ---------------------------------------------------------------------- cdbc · 2024-04-05 11:22 Hi Vlc is another possibility ...and fpc has got bindings for that and I think a component too... Regards Benny ---------------------------------------------------------------------- TRon · 2024-04-05 15:29 Quote from: Hartmut on April 05, 2024, 01:08:24 pmI think ffmpeg as a library is far beyond my horizon. And as I said in reply #4, I hesitate to trust on ffmpeg. Why ? and I really meant why. There is nothing wrong with ffmpeg and is a trustworthy tool (that is when compiling yourself or downloaded from a trustworthy source). Let's do a rewind first... When copying the streams with ffmpeg (e.g. no conversion at all) URL1: 93.953.449 bytes URL2: 2.214.927.005 bytes URL3: 7.527.485 bytes The first 2 URL's do not have any issue copying the stream but the 3th last one does. Code: [Select][https @ 0x7f32a4062d00] HTTP error 403 Forbidden [hls @ 0x55816ff06200] Failed to open segment 14 of playlist 0 [hls @ 0x55816ff06200] Segment 14 of playlist 0 failed too many times, skipping If attempting that URL a second time: Code: [Select][https @ 0x563518add340] HTTP error 403 Forbidden [hls @ 0x563518a57200] parse_playlist error Server returned 403 Forbidden (access denied) [https://vod-ww.mdn.ors.at/cms- worldwide_episodes_nas/_definst_/nas/cms- worldwide_episodes/online/14220193_0005_Q6A.mp4/chunklist.m3u8] [AVIOContext @ 0x563518adce40] Statistics: 126 bytes read, 0 seeks https://apasfiis.sf.apa.at/ipad/cms- worldwide_episodes/14220193_0005_Q6A.mp4/playlist.m3u8: Server returned 403 Forbidden (access denied) That tells me that you are not suppose to download that stream without proper authentication (that is, if you are ever suppose/allowed to download it at all). There are solutions to that problem but people have been going to jail for that or their projects taken down by DMCA. You do have to understand what you are playing with as there are risks involved. Personally I do not judge one way or another. So, I am going to mention a tool that can be used (you can invoke it from within Pascal code using TProcess for example) but you would have to decode it with rot47 and is named "JE\5=A" (without the quotes). Keen people already know what tool it is but it is the only tool I know of that is open source and reasonably reliable in downloading such URL's. ---------------------------------------------------------------------- Hartmut · 2024-04-05 17:18 Hi cdbc, I spent some hours searching for those vlc components and their documentation and investigating them. I found FPC units 'libvlc' and 'vlc' and package 'lazvlc'. The only documentation for them I found was https://www.freepascal.org/~michael/articles/lazvlc/lazvlc.pdf, but this deals only with playing a video by VLC, not with downloading a video into a file. So I see no way/sense to proceed with VLC with a reasonable effort. Thank you TRon for your detailed post. I will investigate it tomorrow. What about curl? There exists a FPC unit for curl called 'libcurl'. Does somebody know, if this curl unit can download a video from an URL like *.m3u8? ---------------------------------------------------------------------- TRon · 2024-04-05 17:42 Quote from: Hartmut on April 05, 2024, 07:18:06 pmWhat about curl? There exists a FPC unit for curl called 'libcurl'. Does somebody know, if this curl unit can download a video from an URL like *.m3u8? Not with the URL's that you provided, or at least not that I am aware of. More information is required.  You can check that out yourself by using the Curl command-line program as that is just a wrapper around libCurl. That is why I proposed to use a 3th party tool so that you do not have to worry about nitty gritty details (and there are quite a lot of them associated with proper authentication). ---------------------------------------------------------------------- Hartmut · 2024-04-06 17:16 Quote from: TRon on April 05, 2024, 05:29:24 pmThe first 2 URL's do not have any issue copying the stream but the 3th last one does. Code: [Select][https @ 0x7f32a4062d00] HTTP error 403 Forbidden [hls @ 0x55816ff06200] Failed to open segment 14 of playlist 0 [hls @ 0x55816ff06200] Segment 14 of playlist 0 failed too many times, skipping I tried the 3rd URL with ffmpeg and got also "HTTP error 403 Forbidden". Then I downloaded from the same URL via VLC-Player the whole video without problems... One more reason to avoid ffmpeg. I got the 3 URL's from a public database where you can search, view and download videos from the German public television. So nothing not legal. My intention is not to download illegal videos. Then I checked the curl command-line program to download those URL's: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i curl -o I got always only the segment list, not the video. I found no curl option to change this. Same results with wget: Code: Text  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i wget -O I got always only the segment list, not the video. I found no wget option to change this. Seams that VLC-Player is the best solution. ---------------------------------------------------------------------- toby · 2024-04-26 21:34 did you get something working? i use sysutils.executeprocess with yt-dlp https://github.com/yt-dlp ---------------------------------------------------------------------- Hartmut · 2024-04-27 05:42 Quote from: toby on April 26, 2024, 11:34:39 pmdid you get something working? i use sysutils.executeprocess with yt-dlp https://github.com/yt-dlp Yes, I wrote a small program which can download the single segment files of a m3u8 video and calls ffmpeg to concat them to a video file. If I'll get further problems, I'll give yt-dlp a try. Thank you toby for that suggestion. ---------------------------------------------------------------------- toby · 2024-04-27 16:41 i assume you are using curl to get the m3u8 to get the urls and number of fragments and then curl to get the fragments? sounds like you are doing in fpc what yt-dlp is doing but in python would you be up to posting the code you have written in fpc?  it would be interesting to test it this is what yt-dlp is doing (from cli but same with sysutils.executeprocess in fpc) > yt-dlp https://srf-vod-amd.akamaized.net/ch/hls/film/2024/03/film_20 240317_000539_4703373_v_webcast_h264_,q40,q10,q20,q30,q50,q60,.mp4.csm il/index-f1-v1-a1.m3u8 [generic] Extracting URL: https://srf-vod-amd.akamaized.net/ch/hls/fil m/2024/03/film_20240317_000539_4703373_v_webcast_h264.../index-f1-v1- a1.m3u8 [generic] index-f1-v1-a1: Downloading webpage [generic] index-f1-v1-a1: Downloading m3u8 information [generic] index-f1-v1-a1: Checking m3u8 live status [info] index-f1-v1-a1: Downloading 1 format(s): 0 [hlsnative] Downloading m3u8 manifest WARNING: Live HLS streams are not supported by the native downloader. If this is a livestream, please add "--downloader ffmpeg --hls-use- mpegts" to your command [hlsnative] Total fragments: 37 [download] Destination: index-f1-v1-a1 [index-f1-v1-a1].mp4 [download] 100% of   91.77MiB in 00:01:38 at 954.51KiB/s        <-- gets each fragment then gives this total for downloads [FixupM3u8] Fixing MPEG-TS in MP4 container of "index-f1-v1-a1 [index-f1-v1-a1].mp4" ---------------------------------------------------------------------- Hartmut · 2024-04-28 08:47 Quote from: toby on April 27, 2024, 06:41:13 pmi assume you are using curl to get the m3u8 to get the urls and number of fragments and then curl to get the fragments? No, I use class 'TFPHttpClient' to download each file. Quotewould you be up to posting the code you have written in fpc?  it would be interesting to test it I am sorry, no, for two reasons. 1st this program was a quick hack and is no recommendation to publish it. 2nd the program uses several common libraries, which I don't have the rights to publish them. But if you are deeply interested, I could extract for you 1 procedure (which downloads 1 URL from type *.m3u8 into a small textfile and then extracts all needed segments into an array) and send it by PM to you. But as said, that procedure would not compile (because of missing libraries) and all comments are in German. ---------------------------------------------------------------------- phoenix27 · 2024-08-13 23:18 Why don't you show us that routine? Do you want us to reinvent the wheel? ---------------------------------------------------------------------- akdev000 · 2024-10-11 16:26 This method seems to work when you receive 403 Forbidden errors while downloading the video (it emulates the same headers that your browser sends): Using Firefox, open Developer Tools using Ctrl + Shift + I, or click on the 3-line hamburger menu in the top-right (1.) > More Tools (2.) > Web Developer Tools (3.). Screenshot of Firefox: 3-line Hamburger Menu > More Tools: https://i.sstatic.net/GP1U7fWQ.png Screenshot of Firefox: More Tools > Developer Tools: https://i.sstatic.net/x2mulkiI.png Go to the Network tab (4.), reload the page, find the appropriate .m3u8 entry, typically named stream.m3u8 or something similar (5.), right-click on it, Copy Value (6.) > Copy as cURL (7.). Screenshot of Firefox: right-click on stream.m3u8: Copy Value > Copy as cURL https://i.sstatic.net/jYTZbyFd.png Then paste the resulting cURL command into here: https://windyakin.github.io/curl2ffmpeg. Download and install the free, open source FFMpeg tool from https://www.ffmpeg.org/ if you don't already have it, paste the resulting ffmpeg command into your commandline (you may have to add a .exe suffix, eg: ffmpeg becomes ffmpeg.exe if you are using Windows) and enjoy the video!