Youtube hoarders, what do you use? reddit / selfhosted ---------------------------------------------------------------------- ECrispy · 2023-12-12 03:02 What do you use to store channels/videos, along with their comments, and keep them updated? TubeArchivist ([https://github.com/tubearchivist/tubearchivist](https: //github.com/tubearchivist/tubearchivist)) looks great - it seems to do everything, and has a nice searchable videos/comments, dashboard etc. It has one big limitation and thats the file names, which are fixed to .ext. I've looked at their many github issues opened for this and exchanged messages with the author. They have their reasons for doing so, and they say its a technical limitation, which I don't really agree with, but it is what it is. The only other comparable project seems to be [https://github.com/jmbannon/ytdl- sub](https://github.com/jmbannon/ytdl-sub), but it requires a lot more manual config and lacks the dashboard/automation/search etc from TA. They both also have integrations with media centers like Jellyfin, but TA has its own UI as well which works like your personal youtube. Everything else is just a script which is params passed to yt-dlp. I've used these, and written my own scripts, but its nice to have an all in one system. update - I found [https://github.com/RoninTech/ta- helper](https://github.com/RoninTech/ta-helper), this uses the TA api to create symlinks with the title in filename. Looks very useful. ---------------------------------------------------------------------- pea_gravel · 2023-12-12 03:57 I use ytdl-sub. In fact it's on the advanced side of things, but once you get it right, it's great. They're working on a UI for it. Once it's done I think the user base will grow significantly. My lib is scanned by Plex where I created smart collections using the Genre tags and each channel has a Season per year. In the end it's like a regular TV Show https://imgbox.com/moI3oi87 https://imgbox.com/7TMVK9lo ---------------------------------------------------------------------- smegheadkryten · 2023-12-12 04:24 I started out with ytdl-sub, but I could never get it to consistently fetch new releases. So I switched back over to tubearchivist with the tubearchivist-plex integration, and I haven't had any issues since. ---------------------------------------------------------------------- ECrispy · 2023-12-12 04:37 How would this compare to the Plex plugin for TA? Could you share your config file please, did you make any changes? ---------------------------------------------------------------------- fy_pool_day · 2023-12-12 05:05 Tubesync ---------------------------------------------------------------------- ECrispy · 2023-12-12 05:32 I dont think it downloads comments ---------------------------------------------------------------------- -Blasting-Off-Again- · 2023-12-12 06:53 im just trying it out now, (im a noob) but it just keps telling me the subscriptions.yaml doesnt exist and im stuck but i really want to like it! ---------------------------------------------------------------------- Theon · 2023-12-12 07:22 I'm fine with TubeArchivist, I don't really see a problem with the file names, from the POV of everyday use I'm accessing the videos through the Web UI anyway, and it even makes scripting and such easier since I don't have to match filenames with titles that may change, contain emojis, get malformed by the FS, etc. ---------------------------------------------------------------------- BigResolution2160 · 2023-12-12 07:25 A custom mess of scripts. None of the solutions i tried work smoothly after 300k of videos and 800 channels ---------------------------------------------------------------------- ECrispy · 2023-12-12 07:36 >match filenames with titles that may change, contain emojis, get malformed by the FS, etc This is the same reason given by TA devs and I don't get it. The filename should be title-id.ext, not a user definable one. You can keep using the id for any scripts and automation. Why does a title changing matter? The id part of the name will never change, so the new video witll still match. emoji etc doesn't matter, its all unicode and standardized. This is how all the other programs like ytdl-sub work and none of them seem to have any issues. ---------------------------------------------------------------------- SamStarnes · 2023-12-12 07:42 My own software. https://github.com/samstarnes/vdm It's not done yet. Not every feature is included that I'd like but I have nearly 1000 videos saved with it so far. I've taken a two month break and returned as of two days ago to work on it. I've been working on it since March 2023 and only taken two (short) breaks. Tonight I was working on the search feature and will be implementing tags and embeds and afterwards filters then integrating user management and private/public uploads (currently only public, no users). After that I will be integrating more database options as Mongodb will have issues with tens of thousands of entries. Everything will be highly configurable so you'll be able to decide what database you'd like, if you want to save json files or not, themes, etc. Don't expect fast development if you use it though as I'm only one person with a job outside of coding but expect everything to upgrade flawlessly and be seamless as possible. "If you want something done, do it yourself." It's the motto I've lived by making this. ---------------------------------------------------------------------- AlteRedditor · 2023-12-12 07:54 Title matters a lot, especially when it comes to having lots of files. Then imagine what if someone has titles that your OS does not allow? Question marks are a good example. How do you decide what to delete and what not to delete from the titles? Not to mention that titles can change at any time on YouTube as well. Should files be renamed when the new data is fetched? This is a whole can of firm that does little to help imo. Shouldn't Plex show accurate data based on the meta information? I think it should. Other software should also be able to have show that as well. ---------------------------------------------------------------------- Sym0n · 2023-12-12 08:04 Out of interest, why do you want the comments? ---------------------------------------------------------------------- crysisnotaverted · 2023-12-12 08:09 Likely archival reasons. If you download content from very technical and instructional channels, the comments are troves of technical information and often expound on parts of the video where not enough info was said to get a full understanding of what is going on. ---------------------------------------------------------------------- Theon · 2023-12-12 08:10 > This is the same reason given by TA devs and I don't get it. Haha, I wasn't aware of that, but I'll have you know I'm speaking from experience of years of scraping YT and other sites myself :) Anyway, what you're saying isn't exactly false, but it introduces so much complications. For one, suddenly you can't rely on a known filepath, because the title isn't stable, so every single file access becomes a lookup. Emojis and unicode weirdness *shouldn't* matter (in 2023!), but much software seems to disagree - I started my data hoarder journey a while back while still dual booting Windows, and the NTFS>Not to mention that titles can change at any time on YouTube as well. Should files be renamed when the new data is fetched? This is a whole can of firm that does little to help imo. Shouldn't Plex show accurate data based on the meta information? I think it should. Other software should also be able to have show that as well. if you redownload the video, you can rename it. else not. plex/emby etc don't care about the filename and never have. thats what metadata is for and it has no limitations as its just a string. The only issue is fs limits on what can be a filename. And that has a easy solution as mentioned above. ---------------------------------------------------------------------- ECrispy · 2023-12-12 08:22 thanks, that helps put things in perspective and no doubt the same issues were faced by the TA team. The format used by ytdl-sub isn't rigid and it shouldn't matter as it and all the other apps are just a wrapper for yt-dlp (youtube-dl) which does all the work and it has no such limitations. You can use a %title% and it will download and replace unallowed characters, there's no error. Are you saying some videos will not download? maybe I haven't come across them. I guess I will stick with TA. I know it has an API to query and that will help. ---------------------------------------------------------------------- ECrispy · 2023-12-12 08:23 yes, this. there is a lot of value in the comments. ---------------------------------------------------------------------- AlteRedditor · 2023-12-12 08:37 I see, I think you're right. Perhaps one way to resolve it is by creating the solution and doing a pull request. 😅 ---------------------------------------------------------------------- ECrispy · 2023-12-12 08:50 please see an update I made, I found a helper script ---------------------------------------------------------------------- dotinho · 2023-12-12 08:51 I have try that using TA, but file names are strings not YouTube titles. ---------------------------------------------------------------------- Perpetual_Nuisance · 2023-12-12 09:24 I have installed and tried both ytdl and TubeArchivist and find TA's UI vastly better. ---------------------------------------------------------------------- AlteRedditor · 2023-12-12 10:26 Thank you so much! ---------------------------------------------------------------------- [deleted] · 2023-12-12 11:18 [deleted] ---------------------------------------------------------------------- pea_gravel · 2023-12-12 14:48 First check if your indentations are correct: https://www.yamllint.com/ . Here's an example of how you're gonna run ytdl-sub: `/opt/ytdl- sub/ytdl-sub sub -c /opt/ytdl-sub/config.yaml /opt/ytdl- sub/subscriptions.yaml` ---------------------------------------------------------------------- pea_gravel · 2023-12-12 15:20 I've never used TA because my understanding is that you have to consume the content in their app. I just read about the Plex plugin and it still looks very basic. Here's some of the stuff I'm doing with ytdl-sub. I don't know if you can or cannot do the same things with TA: • downloading files to a tmpfs before moving to its final destination. That extends the life of my ssd. • removing sponsor blocks. • creating a json with all the metadata. You can have .NFO for Jellyfin/Kodi too. • you can for example download all vireos from the last 6 months or the last 10 videos from a channel and ytdl-dl will automatically delete anything that doesn't match those conditions. • download videos longer/shorter than X. • download only videos that contain/not contain certain keywords. For example I have this crime channel that I only download videos where the title contains the keyword "Solved". • naming files in a pattern that media centers can understand: https://imgbox.com/Qbfg13Yr. Once the content is on my Plex Library, I can, not only watch it but also share it with other users if I want to. ---------------------------------------------------------------------- binary_flame · 2023-12-12 16:42 Yt-dlp actually has an option to restrict filenames, to strip out any illegal characters that the filesystem would reject ---------------------------------------------------------------------- barry_flash · 2023-12-12 17:46 Love it - does it download playlists? ---------------------------------------------------------------------- SamStarnes · 2023-12-12 17:58 Honestly? Not sure lol but **I'm going to assume no. Never been tested.** It fails with video grids on Twitter so I'd assume an entire playlist is also something that it wouldn't like. **But don't worry, that will be in the works as well (it just may not be in my todo list).** I too download entire playlists of users but this is the usual "create directory, cd there, yt-dlp the playlist as normal, come back later." I'm going to try to make that a little more organized as downloading potentially hundreds (or thousands) of videos into your normal set of downloads would be incredibly messy. A lot of my ideas came from [YoutubeDL- Material](https://github.com/Tzahi12345/YoutubeDL-Material) but there are just *some issues* I have a problem with its structure of organization and other problems. There's also the fact that for some reason, somehow, it suddenly stopped working and stopped downloading for no explicable reason. Even recreating new instances gave me the same result. ---------------------------------------------------------------------- ibfreeekout · 2023-12-12 18:27 I just installed the Plex plugin last night and it works very well. It's very nice to be able to just watch the videos now through Plex on my Roku TV. That was the biggest hold up for me until I found that. ---------------------------------------------------------------------- jojotdfb · 2023-12-12 19:55 I've had good luck with this one. My use case is probably different than op's thou. I mostly wanted to have an offline ad free copy of some animation series and some of the stuff the kids watch. I don't mind them watching some things, but the recommendations have them wandering off into some stuff that I really don't want them into. This has helped a lot. The Boy can watch his Blippi without following it up with some minecraft tuber that's better suited for an older audience. ---------------------------------------------------------------------- Antonaros · 2023-12-12 20:14 Ideally I would use tube archivist but as a beginner in self-hosting I never managed to get it working. Instead I use ytdl-material. ---------------------------------------------------------------------- ECrispy · 2023-12-12 22:01 these are all great features, thanks so much for replying. I believe yt-dlp creates the info.json which also includes the comments, but its just a json file and not really viewable, I'm hoping TA makes this easier? how do you update your channels? do you manually do it? does ytdl-sub create playlists from the ones in the channel? I mean the ones you can see in youtube, not created by the app like for each channel/year etc. ---------------------------------------------------------------------- pea_gravel · 2023-12-12 23:43 You can use crontab to schedule your downloads. Ytdl-sub doesn't generate playlists, but you can download videos from playlists. For example, I have this playlist called Documentaries, ytdl-sub is monitoring it and every time I add something to that playlist, it gets added to my Plex. ---------------------------------------------------------------------- Darkchamber292 · 2023-12-13 03:16 OMG I didn't know about the plugin. I need to try it. I love TA but I have an issue with TA consistently giving playback errors in the middle of videos. Happens over Chrome/Edge/FF/Opera over multiple devices. I've even tried multiple installations. I have 80+ containers on this server. Maybe the Plex Integration can save me here ---------------------------------------------------------------------- murdaBot · 2023-12-13 07:15 Just browsing through the threads in here and wanted to say kudos for offering this to everyone and putting the effort into making it usable for folks other than you. ---------------------------------------------------------------------- murdaBot · 2023-12-13 07:17 I just use yt-dlp and yt-dlp-gui manually. I don't store them frequently enough to need an automated solution, like I do for usenet/torrents and Plex, but by now I do have a few thousand saved. ---------------------------------------------------------------------- ECrispy · 2023-12-13 07:57 you have more than me, at this point I think automation is called for. depends on how many channels you want to keep and how much work you want to do. ---------------------------------------------------------------------- Darkchamber292 · 2023-12-13 17:07 Set it up last night and it's awesome! I ran into a bug last night but was able to resolve it on my own and document it https://github.com/tubearchivist/tubearchivist-plex/issues/32 Thanks for the suggestion! ---------------------------------------------------------------------- FrankMagecaster · 2023-12-17 18:23 Hey, ytdl-sub author here. We recently fixed a bug in the cron instructions that should resolve your issue in case you decide to switch back. Happy scraping 🙂 ---------------------------------------------------------------------- smegheadkryten · 2023-12-17 22:14 I appreciate you reaching out. I didn't have a problem getting ytdl- sub to execute. It would run and chunk download fine on a schedule. The problem I had was it wouldn't detect videos that were released after I created the subscription.yaml . Any video that was in the playlist before I created the subscription downloaded fine, but it wouldn't detect any new videos added after that date. ---------------------------------------------------------------------- FrankMagecaster · 2023-12-18 00:20 I see, the playlist probably has videos added to the end (ytdl-sub looks at the top/front, then breaks by default if you hit a vid that exists). This can be resolved by either setting ytdl_options.break_on_existing to False or ytdl_options.playlistreverse to True