ytdl-sub: Automate YouTube downloads and metadata generation for usage in Kodi/Jellyfin/Plex/etc + more reddit / selfhosted ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 02:48 Hey all, it has been almost a year since I last posted about ytdl-sub. For folks who are new, ytdl-sub is a command-line tool that uses yt- dlp to download and format media for any self-hosted use case. Three main uses cases are: - Channels/playlists/etc as TV Shows - with Plex, Jellyfin, Emby, Kodi support - Music (YouTube, SoundCloud, Bandcamp) - with tag-support for Navidrome/Gonic/etc usage - Music Videos When I last posted, ytdl-sub's learning curve was quite high. We've been focusing on adding things to make it easier for users to start downloading hassle-free. A few features I want to highlight are: Usability: - `ytdl-sub` can now be used in-browser using the ytdl-sub-gui Docker image - This image runs VS-Code in browser with ytdl-sub preinstalled for users to edit subscriptions and run ytdl-sub from the terminal - Portable downloads for Linux, ARM, Windows, Pip - Docker is not required Ease-of-use: - We've built many presets for many use-cases into the app, which means little-to-no configuring is required to start downloading and watching/listening asap - Simplified subscription syntax to express downloads much easier And now, for a quick demo. To download and only keep the last two months of Linus Tech Tips videos, and the entirety of my toddler's favorite train channel for Plex, all you need is this file: ``` # subscriptions.yaml # global overrides for all subscriptions __preset__: overrides: tv_show_directory: "/tv_shows" date_range: "2months" # All subs under this use the `Plex TV Show by Date` preset Plex TV Show by Date: # Sets genre and rating to "Kids" and "TV-Y" = Kids | = TV-Y: "Jake Trains": "https://www.youtube.com/@JakeTrains # Uses `Only Recent` preset to keep 2months worth Only Recent | = Tech | = TV-14: "Linus Tech Tips": "https://www.youtube.com/@LinusTechTips" ``` And the command: ``` ytdl-sub sub subscriptions.yaml ``` That's it! Successive downloads will start right where you left off. Will take a while to download, but that's the nature of scraping with yt-dlp. Any part of the download/naming/formatting process is configurable, but will require some reading in our extensive documentation. We support all popular players, scraping music with proper tagging, music videos, and more! Check out our repo for more info: https://github.com/jmbannon/ytdl-sub Thanks for reading, hope you find it as useful as I do! ---------------------------------------------------------------------- smackson · 2023-11-04 03:52 This looks lkke exactly what I've bee thinking is needed. ---------------------------------------------------------------------- aamfk · 2023-11-04 04:24 Thanks. I'm looking forward to finding something new. Tartube has been my goto for many years because it works for SOOOOOOOOOOOOOOOOOOOOOOOOOO many different video websites. ---------------------------------------------------------------------- Lalaz4lyf · 2023-11-04 04:31 Wish there was something that worked similarly for Twitch vods ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 04:32 I think yt-dlp supports twitch, if so then ytdl-sub should as well even as a 'TV show' ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 04:34 Both Tartube and ytdl-sub use yt-dlp, so any website should be compatible between the two ---------------------------------------------------------------------- Lalaz4lyf · 2023-11-04 04:37 It supports downloading from Twitch if passed a vod url. However it won't scrape a channel for vods like it will for YouTube channels. I asked in the Discord about it detailing what I have tried and the outcome but only received a non-answer from the dev. ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 04:40 Ah I remember this. Yeah I'm surprised someone hasn't added that support in yt-dlp yet. ---------------------------------------------------------------------- Lalaz4lyf · 2023-11-04 04:48 Works great for what it's intended for! Glad to see it being posted here. ---------------------------------------------------------------------- thekrautboy · 2023-11-04 04:50 Thanks for sharing it here, i would think a lot of readers are already aware of your project, but clearly not all of them! ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 04:55 New self-hosters are born every day and need a post here and there ☺️ ---------------------------------------------------------------------- thekrautboy · 2023-11-04 05:02 Yes they are and thats great! Sadly a percentage of them are not able to read the basic infos from the subreddit sidebar and follow those, and also not to simply use the subreddit search function. ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 05:15 Hmmm I used markdown mode, looks good for me on the official Android app and desktop. Which part is unformatted for you? ---------------------------------------------------------------------- thekrautboy · 2023-11-04 05:21 Basically everything starting from all you need is this file: And using plain desktop reddit (firefox). ---------------------------------------------------------------------- Cybasura · 2023-11-04 05:48 AYOOOO How am I just hearing about this I'm gonna give this a shot when I get home. ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 05:56 Docker compose can be found here my friend, with multiple flavors: https://ytdl-sub.readthedocs.io/en/latest/install.html#install ---------------------------------------------------------------------- XxNerdAtHeartxX · 2023-11-04 05:59 Its called Ganymede - and I love it ---------------------------------------------------------------------- austozi · 2023-11-04 08:02 Some Reddit clients don't render code blocks well if you use ```. Reddit recommends using a four-space indent instead of that. ---------------------------------------------------------------------- dotinho · 2023-11-04 11:03 Hey, it’s possible also YouTube music artist / albums? ---------------------------------------------------------------------- Count-ant · 2023-11-04 11:18 Good timing, installed on unraid a couple days ago! Downloading is all going great, but struggling with Plex naming. I’m using the worked example files with Plex preset and it is assigning E110201 as the tv show name and not picking up the title in the file name as well. I’ve created a new library in a seperate directory as a tv show, any initial thoughts of what I’m doing wrong? I’m on mobile atm, but can provide more detail if it helps just let me know what? ---------------------------------------------------------------------- 4LAc · 2023-11-04 13:21 Fantastic, spun up in a minute and am watching already on Kodi :) Thanks a million! ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 13:34 This is my most frequently asked question, follow https://github.com/jmbannon/ytdl-sub/wiki/FAQ#plex-is-not-showing-my- tv-shows-correctly and you should be good ---------------------------------------------------------------------- Count-ant · 2023-11-04 13:41 Thank you, will get reading. #Rtfm 😂 ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 13:42 Yes! See ytdl-sub/blob/master/examples/music_subscriptions.yaml for more details. It can help Lidarr by getting music in proper name/directory format. ytdl-sub will tag files though, so IMO Lidarr isn't adding much other than importing into its app and maybe slightly retagging. ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 13:48 Lol no worries, this question always makes me glad. You are about to cross the finish line - enjoy! ---------------------------------------------------------------------- flyingvwap · 2023-11-04 14:14 Been using this to download YouTube channels and playlists for my kids to watch on Plex and it's fantastic! ---------------------------------------------------------------------- mreim · 2023-11-04 17:17 Awesome, thank you! I'm relatively new to self-hosting. I've been looking for something to watch the YouTube videos that are relevant to me outside of YouTube lately. This sounds pretty much like what I've been looking for. Tomorrow, when I have more time, I'll take a closer look. ---------------------------------------------------------------------- io-x · 2023-11-04 18:41 I'm confused, Does it download to the machine its run? Does it query YT daily? ---------------------------------------------------------------------- Illeazar · 2023-11-04 18:52 Perfect timing, YouTube archiving was next on my selfhost to-do list! ---------------------------------------------------------------------- Legitimate_Farm_3095 · 2023-11-04 18:53 info is in the docs ---------------------------------------------------------------------- mallrat32 · 2023-11-04 18:58 Is Sponsorblock a potential addition at some point? ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 19:14 It is supported now! But needs a custom config (will eventually make it a prebuilt preset). See https://github.com/jmbannon/ytdl- sub/blob/master/examples/advanced/tv_show_config.yaml for an example ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 19:15 You can point outputs to a (remote) mounted directory, and you can set up cron within Docker to do a daily download ---------------------------------------------------------------------- ohv_ · 2023-11-04 19:30 Possible to include IG posts? Personally having issues with pulling Jiu-Jitsu content and the task is kinda of cumbersome IG and YT. ---------------------------------------------------------------------- FrankMagecaster · 2023-11-04 19:42 I'm not sure. If yt-dlp can, then so can ytdl-sub ---------------------------------------------------------------------- pigers1986 · 2023-11-04 21:10 Some years ago I scripted yt-dlp to do dirty work for me - now I see someone replicated it - good job mate! ---------------------------------------------------------------------- Legitimate_Farm_3095 · 2023-11-04 22:45 thanks for all the work! i've been running emby alongside plex for years because i could never get something like this to work with plex. i appreciate the windows exe too, my plex server is still on my windows server and i dont plan on moving it anytime soon. ---------------------------------------------------------------------- d4nm3d · 2023-11-04 23:43 Incredible.. thank you.. i've dabbled with archiving some of my favorite channels a few times but gave up most of the time because of no easy way to consume them.. I've been a Kodi user since it was XBMC on the original XBOX and to have an app that will just download it all and present it in the right format is a god send! one thing i havn't seen.. (and i've not looked with much gusto)... is there any support for sponsor block? ---------------------------------------------------------------------- Jonteponte71 · 2023-11-05 00:08 I suppose you know about tubearchvist? Even simpler to use than this. ---------------------------------------------------------------------- Jonteponte71 · 2023-11-05 00:12 Yes. Look further up… ---------------------------------------------------------------------- d4nm3d · 2023-11-05 00:58 apologies and thank you ! ---------------------------------------------------------------------- Illeazar · 2023-11-05 01:14 Not familiar with that one. I haven't started researching the subject yet, but my to-do list just says: "sonarr for Youtube?" I'll look into tubearchivist. What do you like about it? ---------------------------------------------------------------------- eye_can_do_that · 2023-11-05 01:36 This looks great. I've been manually downloading critical role each week to watch on Plex, but this looks better. ---------------------------------------------------------------------- Jonteponte71 · 2023-11-05 10:47 I should say that I haven’t actually installed it yet. But I am a heavy YT user that also uses yt-dlp with an extension to archive the things I want to keep. And have done for years. I just haven’t gotten to installing it yet. The thing with tubearchivist is that it has it’s own web gui that you can manage subscriptions and downloads through. And watch the actual videos in as well. Or you can sync to Plex and/or Jellyfin including metadata. And not have to mess with config files or the CLI. A And on top of that TA has it’s own search engine so you can always find stuff when your local library grows. It just seems like a great project with a responsive and active developer… ---------------------------------------------------------------------- Virindi · 2023-11-06 16:33 I'm still reading all the docs, but: is it possible to strip some content from the episode name using a regex? For example, sometimes the channel adds some cruft to the end like: Episode Name | Channel Name I'd like to strip off everything matching (and after) **| Channel Name** to keep the titles clean. Same for the description field, sometimes they fill it with ads for their other videos, but they all start with something reasonably predicable. Or.. does anyone know of another tool that simplifies this that I can run after this tool? ---------------------------------------------------------------------- FrankMagecaster · 2023-11-06 19:18 Yes, regex is a plugin that you can use to capture and create new variables to use anywhere (filenames, metadata, etc). See https://ytdl-sub.readthedocs.io/en/latest/config.html#regex ---------------------------------------------------------------------- Virindi · 2023-11-06 20:19 Thank you! ---------------------------------------------------------------------- sn4201 · 2023-11-11 01:08 I think i'm not smart enough to actually understand how to use this. I can't seem to understand what is supposed to go in config, and what is supposed to go in subscrptions. I ran the docker gui, and my config/subscriptions files that i created aren't shown. So i dont know what will happen if i can actually figure out how to run the program... Would love some more example files and a little bit clearer/plainer language on how to use this properly...but maybe its my fault for not understanding. Appreciate the efforts to make the program available, anyway. ---------------------------------------------------------------------- eye_can_do_that · 2023-11-13 23:31 So I have been looking at ytdl-sub but I am a bit confused how I could pull in my yt-dlp command line options in to the config. I download Critical Role Campaign 3 (I noticed it made it in to your sample image on github). Over time I made a fairly complicated yt-dlp command to get it formatted just right: yt-dlp -f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]' --merge-output- format mp4 --convert-thumb jpg --embed-thumbnail --write-thumbnail --sub-lang en --embed-subs --add-metadata --parse-metadata "description:(?P.+)((\s)+)CAPTION STATUS" --parse-metadata "title:\s*(?P.+?)\s*[|].+Campaign\s*(?P[\d]+),\s*Episode\s*(?P[\d]+)" --parse-metadata "episode_name:%(meta_title)s" --parse-metadata ":(?P)" -o "Critical Role - s%(season_num)03de%(episode_num)03d - %(episode_name)s" url Looking at the ytdl-sub configuration options, I don't see how I can use the parse-metadata option to pull the right description from the description and the right title from the youtube title. Is there a way to do this with ytdl-sub? ---------------------------------------------------------------------- FrankMagecaster · 2023-11-14 01:10 Yes, the regex plugin: https://ytdl- sub.readthedocs.io/en/latest/config.html#regex ---------------------------------------------------------------------- classicksworld · 2023-11-15 06:32 Thank you for this. Will be testing it out tomorrow! ---------------------------------------------------------------------- aamfk · 2023-11-19 01:05 thanks for that info ---------------------------------------------------------------------- Terrible_Ad_4678 · 2023-11-20 20:48 This is awesome! One thing I'm struggling with is downloading a custom playlist instead of an entire channel or some time-frame of a channel. Anyone have any advice on how to do it? ---------------------------------------------------------------------- eye_can_do_that · 2023-11-21 02:32 If you don't mind a little more help. I get the episoide num from a regexp, say 77, I want to save the file with 3 digits with leading 0, so 077; is there a way to put the string format specifier in the output\_options? Something like %{epidose\_num}03d? Also, I couldn't figure out how to replace the description, I use a regexp to pull out an actual description from the youtube description (which contains all the sponsor, website and other info); but can I replace the description in the meta data? In my yt-dlp command which updates the metadata that gets added to the video file: --add-metadata --parse-metadata "description:(?P.+)((\s)+)CAPTION STATUS" ---------------------------------------------------------------------- FrankMagecaster · 2023-11-21 04:03 For the first part, not at this time but it's something (function support) that I'm actively working on and hoping to release in the next month. You could use `download_index_padded` as an alternative for a workaround. The second part, the variable you want to override is `episode_plot` (https://github.com/jmbannon/ytdl-sub/blob/master/src/ytdl_sub/prebuil t_presets/tv_show/episode.yaml#L41) ---------------------------------------------------------------------- FrankMagecaster · 2023-11-21 04:05 More on the first part, ytdl-sub supports renaming/retagging video files without redownloading so long as you save .info.json files (TV presets do by default). So if you do use an alternative method, you should be able update it later with `--update-with-info-json` ---------------------------------------------------------------------- FrankMagecaster · 2023-11-21 04:10 See how `Only Recent` preset is used here: https://github.com/jmbannon/ytdl- sub/blob/master/examples/tv_show_subscriptions.yaml#L57 This limits subscriptions underneath it to the only_recent variables defined at the top of the file ---------------------------------------------------------------------- Rich4477 · 2023-11-26 18:10 I'm running it from Windows in PowerShell and getting an error but it pops up in a terminal window and closes immediately so I can't read it. Any ideas? Does it need yl-dl installed too? ---------------------------------------------------------------------- FrankMagecaster · 2023-11-26 19:52 You're invoking from the command line, right? Shouldn't need any extra dependencies besides ffmpeg (which is in the install instructions) ---------------------------------------------------------------------- Rich4477 · 2023-11-26 22:44 I sorted it out. My antivirus deleted the exe but I never noticed. Thanks. Still sorting my config but making good progress. ---------------------------------------------------------------------- LoadingStill · 2023-11-28 19:26 I see references to a gui for it hut I see no images of the gui. Would you update your guide to show some images of the gui? ---------------------------------------------------------------------- FrankMagecaster · 2023-11-29 00:28 its in the README on the bottom, it's not really a true gui, just a console/editor in browser via code-server ---------------------------------------------------------------------- LoadingStill · 2023-11-29 00:32 I will be honest I thought that was a self host of vscodium. ---------------------------------------------------------------------- -Blasting-Off-Again- · 2023-12-12 03:25 is there any easy guide on setting this up? so much i read seems to assume i know anything about docker. I'm fairly tech savvy but i cant even figure out how to get started here ---------------------------------------------------------------------- FrankMagecaster · 2023-12-12 09:03 Docker is not required - I really need to update the wiki. Download the executable, copy/paste the readme subscriptions into a file, and run `ytdl-sub sub path/to/subscription_file.yaml` from the command-line / power shell if on Windows ---------------------------------------------------------------------- -Blasting-Off-Again- · 2023-12-12 16:47 So I installed it with powershell but I keep getting told the subscriptions file doesn't exist when I try to use the example with Linus tech tips. When I try to download a channel I designated, it says I don't have ffmpeg but I directed it there but it didn't work. I'm really bad at this, I'm really having a hard time understanding how to use it and the commands lol. But I really want to get it working as the alternative is doing the metadata myself which sounds incredibly tedious ---------------------------------------------------------------------- FrankMagecaster · 2023-12-12 17:29 Mind hopping on our discord and making a support thread, it'd be easier for me to help you there. And thanks for the kind words ☺️