Tool to extract m3u8 stream from embedded video player? lazarus / 38-Networking and Web Programming ---------------------------------------------------------------------- BosseB · 2023-08-28 07:33 This question is about Lazarus/fpc on Linux Ubuntu. I would like to create a tool that can extract the m3u8 video stream URL given the URL of the webpage where the player is embedded. It would be used on streaming video sites that change this URL regularly with short time intervals. With the m3u8 url I can use a call to ffmpeg to do the actual stream download I want to do. Some such sites can be processed by a command like this in a shell script to extract the url: Code: [Select]curl -s "" | grep -o -e \"https://.\+m3u8\" But others (most) need some other way of finding the m3u8 url for the stream. When playing such a site in FireFox one can hit F12 to bring up a debug window where one can find the m3u8 URL below the "Network" tab and right-click to copy the url. The problem with this is that the m3u8 url on some sites change at intervals ranging from hours to days, so a manal examination and extraction is not practical when downloading the video using ffmpeg in unattended mode. I would like to create an fpc utility that can do this on an arbitrary webpage such that when the download script starts it will call the utility to get the m3u8 url and send that into an ffmpeg command that does the download itself. So my question is really if someone here can suggest a way using fpc/lazarus to create a command line utility that will run on Ubuntu to extract the m3u8 url given the main webpage url where the player is embedded into? Note: There is a FireFox extension named "Video Controller", which can do this extraction, but I have not found the source for this so I cannot see how it is being done... And it cannot be automated because of this. Any suggestions? ---------------------------------------------------------------------- Fibonacci · 2023-08-28 09:10 Hardest thing is to catch the URL to the m3u8 file. It can be well hidden, dynamically created, obfuscated or sent via WebSockets. In pure FPC you can do as much as your curl | grep command. How would I do it? I would do it as an extension for the browser, in JavaScript. It would catch all HTTP requests, search for m3u8 in URL and send it together with cookies to FPC app, which would be a simple HTTP server just to receive m3u8 URL and cookies. I would then download that file and pass it to ffmpeg or ytdl. Probably instead of extension I would do it in some headless browser such as PhantomJS or Selenium Also using chromedriver (or other browser driver) is an option. Forum member ported WebDriver4D to Lazarus (click). I don't think capturing network traffic is possible with this unit but you can inject some JS code (I guess). BTW Arent firefox extensions just JavaScript files? Search for *.xpi files and open it as ZIP, it will contain full source code of this extension you mentioned ---------------------------------------------------------------------- BosseB · 2023-08-28 10:44 Quote from: Fibonacci on August 28, 2023, 11:10:38 amBTW Arent firefox extensions just JavaScript files? Search for *.xpi files and open it as ZIP, it will contain full source code of this extension you mentioned Maybe, I don't know. I looked inside this folder: Code: [Select]C:\Users\Username\AppData\Roaming\Mozilla\Firefox\Profil es\lxxxxxl9m.default-release\extensions and I found a number of files of this kind there: Code: [Select]langpack-sv-SE@firefox.mozilla.org.xpi privatkopiera@firefox.stefansundin.com.xpi swedish@dictionaries.addons.mozilla.org.xpi uBlock0@raymondhill.net.xpi {06b3e550-d8ff-4273-8fa0-03cd60feee8a}.xpi {6cc0a66e-ae3d-4cd8-9a03-5cd93b392903}.xpi {943b8007-a895-44af-a672-4f4ea548c95f}.xpi {b9db16a4-6edc-47ec-a1f4-b86292ed211d}.xpi {e4a8a97b-f2ed-450b-b12d-ee082ba24781}.xpi How can I know which file belongs to the extension I mentioned? Those with actual names I recognize but not the others with GUID names... ---------------------------------------------------------------------- Fibonacci · 2023-08-28 10:49 Extract all and check, you will know which one is it if you see whats inside ---------------------------------------------------------------------- TRon · 2023-08-28 13:04 Quote from: BosseB on August 28, 2023, 09:33:26 amSome such sites can be processed by a command like this in a shell script to extract the url: curl -s "" | grep -o -e \"https://.\+m3u8\" In such cases you can use fphttpclient QuoteBut others (most) need some other way of finding the m3u8 url for the stream. In case you are able to 'locate' the m3u8 file in the HTML source-code then you can try and extract it using jurassic pork's webdriver. In case you have no idea what that does then do a search for selenium webdriver -> you control your (embedded and if wanted hiding in the background) browser. The trickery used to hide files/links in webpages can be daunting so be warned. ---------------------------------------------------------------------- BosseB · 2023-08-28 13:09 Quote from: Fibonacci Extract all and check, you will know which one is it if you see whats insideOK, will do. Meanwhile I have been checking manually on one of the video streams I want to use and it turns out that every hour it changes by modifying a 3 hour timestamp by 1 hour... The part of the url that that changes looks like this: Code: [Select]FUu9Jj3Tre1H_KYVTSBE0g==,1693238086 (=August 28, 2023 17:54:46 GMT+02:00 DST) 6OwnmgK447Rhi7Z8z6NTeQ==,1693241698 (=August 28, 2023 18:54:58 GMT+02:00 DST) so the last numeric part seems to be the expire time, decoded by me inside the ().. Every hour it changes by 1 hour, but the preceding mumbo-jumbo also changes... So if I could get the proper url once every 2 hours or so it would probably work. ---------------------------------------------------------------------- BosseB · 2023-08-28 13:53 Quote from: Fibonacci on August 28, 2023, 12:49:01 pmExtract all and check, you will know which one is it if you see whats inside I found it, but to no avail... It is apparently a lot of javascript code put into files with line lengths of >> 10000 chars. Not possible for me to decode. ---------------------------------------------------------------------- delphius · 2023-08-29 07:34 Quote from: TRon on August 28, 2023, 03:04:19 pmIn case you have no idea what that does then do a search for selenium webdriver -> you control your (embedded and if wanted hiding in the background) browser.At the moment, there is an easier way to get a similar result with the chrome browser running in headless mode using the webui library Webdriver is tied to the browser version and is not small in size, and in general, the chain turns out to be very long, time-consuming and not reliable in practical use, except for testing purposes (which is what it was originally intended for) ---------------------------------------------------------------------- BosseB · 2023-08-29 12:44 Well, the server that my download code is running on is really an Ubuntu *server* with no GUI so I doubt that there are any web browsers or the like installed to hook into... ---------------------------------------------------------------------- delphius · 2023-08-29 15:57 Quote from: BosseB on August 29, 2023, 02:44:51 pmWell, the server that my download code is running on is really an Ubuntu *server* with no GUI so I doubt that there are any web browsers or the like installed to hook into... Of course, there are certain difficulties with installing on the server, but they are mostly solvable In any case, before trying any solution directly on the server, you need to fully understand the problem and the algorithm for solving it, while, as far as I understand, there is no final clarity. Are you sure that the algorithms for obfuscating links to the stream will be the same for different sites? In order for someone here to be able to competently advise a way to do this in the most convenient way, give any example of such a site with such links Just firing a cannon at sparrows is also not the case ---------------------------------------------------------------------- Fibonacci · 2023-08-29 16:33 Quote from: delphius on August 29, 2023, 09:34:23 amAt the moment, there is an easier way to get a similar result with the chrome browser running in headless mode using the webui library I checked this webui lib you linked to cos Im interested in alternatives to LCL. It looks nice, I was even able to static-compile this lib into my app so no DLL file needed. Single exe file, 261 KB. But one thing, with webui its not possible to browse other webpages. It creates socket connection between browser and your app, and as soon as you close the connection the webui no longer works. Socket connection management is JavaScript appended to your HTML. So if for example you execute JS "document.location.href = 'some domain...'", then webui socket script is no longer running and connection breaks and your app closes - webui_wait() thinks the browser window was closed. So you cant use it the same way as webdriver, you cant make it to go to some URL and play with DOM. URL changes = socket connection breaks = webui loses control. ---------------------------------------------------------------------- BosseB · 2023-08-29 20:51 Today I discovered that my commands using curl which did not get results failed because something curl does makes the server send false data... So I changed the code as follows below and now I get the m3u8 url for half of the sites I am using. Half of the remaining sources do not change the m3u8 very often so there I can do the manual extraction and save into the url-file. I still have a handful of sources where the extraction needs to dig deeper... Here is a snippet of the code I have changed to where I use wget instead: Code: Bash  [Select][+][-]window.onload = function(){var x1 = document.getElementById("main_content_section"); if (x1) { var x = document.getElementsByClassName("geshi");for (var i = 0; i    #New command working for sources 1-6:   TMPFILE=$(mktemp)   CMD="wget -q ${STREAMURL} -O ${TMPFILE}"   eval $CMD #Download source to temp file   CMD="grep m3u8 ${TMPFILE} | cut -d '\"' -f2"   M3U8=$(eval $CMD)   rm $TMPFILE  STREAMURL is the URL to the webpage hosting the streaming player. ---------------------------------------------------------------------- delphius · 2023-08-30 04:35 Quote from: Fibonacci on August 29, 2023, 06:33:38 pmI checked this webui lib you linked to cos Im interested in alternatives to LCL. It looks nice, I was even able to static-compile this lib into my app so no DLL file needed. Single exe file, 261 KB. It's great that you managed to try the library and link it statically. Could you write a little instruction on how to do this for other users? Quote from: Fibonacci on August 29, 2023, 06:33:38 pmBut one thing, with webui its not possible to browse other webpages. It creates socket connection between browser and your app, and as soon as you close the connection the webui no longer works. Socket connection management is JavaScript appended to your HTML. So if for example you execute JS "document.location.href = 'some domain...'", then webui socket script is no longer running and connection breaks and your app closes - webui_wait() thinks the browser window was closed.The library itself is not intended for testing web pages, it is precisely the replacement of the interface using the browser. But I use some tricks, at least they worked in the old version of the library, which allowed me to work around this problem, something like this, wich makes to autoload webui js code, or you can manipulate dom manually, including webui.js in each page you load. And it was my suggestion to use the library in headless mode, because I'm not so interested in using it as an interface for applications, but I really need a full-fledged replacement of the heavy webdriver. Quote from: BosseB on August 29, 2023, 10:51:00 pmHere is a snippet of the code I have changed to where I use wget insteadAt first glance, the presented command code is not difficult to repeat on pascal, even without third-party modules, since the entire page is simply downloaded and the necessary link is located in the text, given statically from the server when the page is generated. Difficulties will begin when link generation occurs on the fly, when the page is running in the browser, then it will require the methods discussed in the topic. ---------------------------------------------------------------------- BosseB · 2023-08-30 06:03 Quote from: delphius on August 30, 2023, 06:35:24 amQuote from: BosseB on August 29, 2023, 10:51:00 pmHere is a snippet of the code I have changed to where I use wget insteadAt first glance, the presented command code is not difficult to repeat on pascal, even without third- party modules, since the entire page is simply downloaded and the necessary link is located in the text, given statically from the server when the page is generated. Difficulties will begin when link generation occurs on the fly, when the page is running in the browser, then it will require the methods discussed in the topic. I thought that these pages too were doing some cloaking like the few I still have to handle manually, but that was not the case - just a curl problem. But there is another problem this script does not handle: Pages with multiple video players... It is not very common but my the page I used back in 2018 suddenly stuffed a second player on the same page and I guess this would need to be handled. That page is long gone, though, so currently no problem like that. ---------------------------------------------------------------------- delphius · 2023-08-30 06:20 Quote from: BosseB on August 30, 2023, 08:03:10 amBut there is another problem this script does not handle: Pages with multiple video players... It is not very common but my the page I used back in 2018 suddenly stuffed a second player on the same page and I guess this would need to be handled. That page is long gone, though, so currently no problem like that. Theoretically, there should be one video source, and there may be several players, but they all use the same link to the stream, or is it not so? ---------------------------------------------------------------------- BosseB · 2023-08-30 13:52 Quote from: delphius on August 30, 2023, 08:20:46 amQuote from: BosseB on August 30, 2023, 08:03:10 amBut there is another problem this script does not handle: Pages with multiple video players... It is not very common but my the page I used back in 2018 suddenly stuffed a second player on the same page and I guess this would need to be handled. That page is long gone, though, so currently no problem like that. Theoretically, there should be one video source, and there may be several players, but they all use the same link to the stream, or is it not so? That site is decommissioned, just mentioned it as a possible problem when extracting the m38u url... I have another site where I cannot get the m3u8 at all, not using my extractors or the debug view (F12) or even the VideoController extension in Firefox. Nothing works for this site... ---------------------------------------------------------------------- delphius · 2023-08-30 14:15 Quote from: BosseB on August 30, 2023, 03:52:19 pmI have another site where I cannot get the m3u8 at all, not using my extractors or the debug view (F12) or even the VideoController extension in Firefox. Nothing works for this site... It looks like this is a live broadcast of the twitch streaming service, try to work in this direction, and here is a similar utility for YouTube. ---------------------------------------------------------------------- BosseB · 2023-08-30 17:10 Quote from: delphius on August 30, 2023, 04:15:18 pmQuote from: BosseB on August 30, 2023, 03:52:19 pmI have another site where I cannot get the m3u8 at all, not using my extractors or the debug view (F12) or even the VideoController extension in Firefox. Nothing works for this site... It looks like this is a live broadcast of the twitch streaming service, try to work in this direction, and here is a similar utility for YouTube. I am using yt-dlp to download static video files and to check the available resolutions in a video. But I don't know how to make it work for a streaming service where one needs to specify also the download duration. Seems like yt-dlp is not capable of downloading from a stream... If it could start downloading from time now and then run until being stopped by a Ctrl-C or similar and when it does stop then finish the downloaded file so it is playable, then it could perhaps work. But is this possible? And if so how? Remember that the problem is finding the stream URL, which I have failed to do for this site. ---------------------------------------------------------------------- TRon · 2023-08-30 17:38 Quote from: BosseB on August 30, 2023, 07:10:14 pmBut is this possible? And if so how? Remember that the problem is finding the stream URL, which I have failed to do for this site. Sure but this is becoming close to not being legal (depends on country). If you want a hint then have a look at these instructions (just checked, these instructions work for your example stream). If you can do that stuff manually then you can also do it automated (and has been done so by many tools out there that are available). ---------------------------------------------------------------------- BosseB · 2023-08-30 19:06 Quote from: TRon on August 30, 2023, 07:38:10 pmQuote from: BosseB on August 30, 2023, 07:10:14 pmBut is this possible? And if so how? Remember that the problem is finding the stream URL, which I have failed to do for this site. Sure but this is becoming close to not being legal (depends on country). If you want a hint then have a look at these instructions (just checked, these instructions work for your example stream). If you can do that stuff manually then you can also do it automated (and has been done so by many tools out there that are available). Thanks, I persisted in following the instructions in your linked page and got the m3u8 URL for the stream.  It turns out to be a 998 char long m3u8 URL where the content seems to be Base64 encoded, but it won't decode as such so it is some other kind of m3u8 encryption. But I put it as-is into my download script for the source and made a 30 second download using it. This seemed to be exactly like I want it to be. So now I am doing a complete 1 hour download using this m3u8. Interesting to see when/if it will fail. The question now is how to extract the m3u8 url programmatically (if it expires regularly, if not then it probably will work into the future). ---------------------------------------------------------------------- TRon · 2023-08-30 20:08 Quote from: BosseB on August 30, 2023, 09:06:04 pmIt turns out to be a 998 char long m3u8 URL where the content seems to be Base64 encoded, but it won't decode as such so it is some other kind of m3u8 encryption. huh ? If I download that m3u8 URI with wget I get a 'normal' text file that represents a m3u file (with video related extensions ofc). Did you use/do something special to download the m3u8 file for that URL ? QuoteSo now I am doing a complete 1 hour download using this m3u8. You should be able to automate that, but depends a little on the download utility that you use. I assume that you are only interested in 'some' parts of that live stream (for example the regular weather report that is aired at specific times/time intervals) in which case you should be able to write a small utility that runs in the background, checking the date/time (intervals) and invoke the recording utility for a specific amount of time (or kill/close it's process ID when that is not possible). QuoteThe question now is how to extract the m3u8 url programmatically (if it expires regularly, if not then it probably will work into the future). Usuaulyl sites suh as twitch that display those videos have a API that you can use to extract all kinds of information. Some API's are free, some paid, other sometimes require an account. I do not know anything about Twitch so I have no idea. I did came across this utility that has some interesting comments on its twitch plugin (be sure to read that, also to understand better why it is not always legal to do as you request and platforms like twitch tries to protect itself against such practices). Also as a hint, make sure to do a search for twitch and record/save as there are many other such kind of tools like the utility mentioned above (also those that use twitch API to extract all kinds of information from a stream, most come with sourcecode, hint: include github (or any other source hosting sites) to your searches  ). ---------------------------------------------------------------------- Fibonacci · 2023-09-01 03:47 Quote from: delphius on August 30, 2023, 06:35:24 amQuote from: Fibonacci on August 29, 2023, 06:33:38 pmI checked this webui lib you linked to cos Im interested in alternatives to LCL. It looks nice, I was even able to static-compile this lib into my app so no DLL file needed. Single exe file, 261 KB. It's great that you managed to try the library and link it statically. Could you write a little instruction on how to do this for other users? Already replied to you PM but I think more people may be interesed in static compiling so I'll include source in the attachment. These files you need to find yourself: {$linklib libkernel32.a} {$linklib libadvapi32.a} {$linklib libuser32.a} {$linklib libmingwex.a} {$linklib libucrt.a} {$linklib libmincore.a} Quote from: BosseB on August 30, 2023, 09:06:04 pmThe question now is how to extract the m3u8 url programmatically (if it expires regularly, if not then it probably will work into the future). Obviously it will expire (its a valid assumption). Also you try to decode that base64, why? Dont do it, its some kind of auth token, you shouldnt care about it. Just get valid URL and use it, if it expires then extract a new one. ---------------------------------------------------------------------- BosseB · 2023-09-01 06:26 Quote from: Fibonacci on September 01, 2023, 05:47:43 amQuote from: BosseB on August 30, 2023, 09:06:04 pmThe question now is how to extract the m3u8 url programmatically (if it expires regularly, if not then it probably will work into the future). Obviously it will expire (its a valid assumption). Also you try to decode that base64, why? Dont do it, its some kind of auth token, you shouldnt care about it. Just get valid URL and use it, if it expires then extract a new one. So I have made some more tests now using the FireFox F12 debug window to get the m3u8 url to use with the ffmpeg download. I have discovered this: 1) The main site url will display a player but sometimes the player itself shows a message to "refresh the browser" instead of the actual video. Refreshing the browser does not do anything... 2) In this situation there is below the player two links "STREAM-1" and "STREAM-2" and STREAM-1 seems to be the default which displays the message above. 3) By hitting STREAM-2 the video starts playing and it shows the expected content but with a different overlay from the other stream (when it worked). This stream is less reliable than the original, though. It refreshes visibly now and then... 4) The lifetime of the m3u8 url extracted using the FireFox F12 tool seems to have a lifetime of about a day, so the automation of extraction would help here. But when now dealing with two streams complicates things a lot... I am using this source as a backup only so if the standard source works it is not used, only when that fails. ---------------------------------------------------------------------- Fibonacci · 2023-09-01 07:05 It displays "PLEASE REFRESH THE BROWSER" even in thumbnail https://www.twitch.tv/solanna89 ---------------------------------------------------------------------- Fibonacci · 2023-09-01 07:29 Stream 2 works, so 1. Go to https://www.twitch.tv/grhntlk 2. Open devtools, F12 3. Reload 4. Search for "ttvnw.net/v1/playlist" 5. You will get a bunch of links, get the first one 6. Copy it and open in VLC 7. Works, you can pass this link to ffmpeg or ytdl How to do it in pascal? I wouldn't even bother. Just allow it to process all javascript and catch first URL to m3u8. Use browser extension or webdriver ---------------------------------------------------------------------- BosseB · 2023-09-01 09:11 Quote from: Fibonacci on September 01, 2023, 09:29:13 amStream 2 works, so 1. Go to https://www.twitch.tv/grhntlk 2. Open devtools, F12 3. Reload 4. Search for "ttvnw.net/v1/playlist" 5. You will get a bunch of links, get the first one 6. Copy it and open in VLC 7. Works, you can pass this link to ffmpeg or ytdl How to do it in pascal? I wouldn't even bother. Just allow it to process all javascript and catch first URL to m3u8. Use browser extension or webdriver For me it does not display anything but an "offline" message: (how can you make the attachment image display inside the post itself rather than as a listed attachment?) ---------------------------------------------------------------------- Fibonacci · 2023-09-01 09:38 Quote from: BosseB on September 01, 2023, 11:11:48 amFor me it does not display anything but an "offline" message: At this moment its actually offline Test with other stream Quote from: BosseB on September 01, 2023, 11:11:48 am(how can you make the attachment image display inside the post itself rather than as a listed attachment?) I dont think you can ---------------------------------------------------------------------- delphius · 2023-09-01 20:31 Quote from: Fibonacci on September 01, 2023, 05:47:43 amI think more people may be interesed in static compiling so I'll include source in the attachment. Slightly modified, taking into account the changes in the static version of the library in August (+$linklib libs/libmsvcr120.a) Posted here to make it convenient to test ---------------------------------------------------------------------- TRon · 2023-09-01 21:28 Quote from: Fibonacci on September 01, 2023, 09:29:13 amHow to do it in pascal? I wouldn't even bother. Just allow it to process all javascript and catch first URL to m3u8. Use browser extension or webdriver Another possible solution (next to the already mentioned stream tools) is to use something like youtube_dl (search for the working URL as the original is down after court ruling) Code: [Select]youtube-dl -g https://www.twitch.tv/"nameofchannel/livestream" Which will then return the link to the m3u8 file for you. fwiw: as suspected the lokinau link is down for reasons mentioned before.