Pipe between containers? What do you guys use? reddit / selfhosted ---------------------------------------------------------------------- laterral · 2023-10-31 13:32 Not sure I'm even asking the right questions (definitely my google searches are coming empty). What I want is ideally an intuitive way to chain containers together (e.g. I want one to read up inputs from a telegram bot -> send the link to another that downloads the YouTube video -> depending on the size, for another to move the file around). There must be solutions to orchestrate all this - can you help me go in the right direction please? ---------------------------------------------------------------------- lesigh · 2023-10-31 13:47 The containers need to be coded to be able to talk to each other thru apis. Example discord bot -> overseer -> radarr -> nzbget -> plex ---------------------------------------------------------------------- GolemancerVekk · 2023-10-31 13:54 Don't over complicate this. Use a single container that runs the bot and can write to a host directory. Have the bot run yt-dlp whenever it finds a youtube link and download to the host directory. For moving the videos based on size you can write code in the bot to do it after download, or you can run a cron job to do it (either in the container or on the host), or maybe yt-dlp has an option for that. ---------------------------------------------------------------------- revereddesecration · 2023-10-31 13:56 I have a series of containers that use the host network and share messages over MQTT. Works well. One of them launches other containers when specific trigger phrases are said in specific topics. Another one sends trigger phrases based on a schedule. ---------------------------------------------------------------------- ClassicGOD · 2023-10-31 14:04 For stitching different applications and APIs together without writing custom software I recommend Node-Red. ---------------------------------------------------------------------- amcco1 · 2023-10-31 14:10 Just use docker-compose? Put multiple containers in one compose file. Create an internal network for them to communicate with. ---------------------------------------------------------------------- Murky-Sector · 2023-10-31 14:44 Each task has a dedicated queue. One task hands over to the next. each task can be scaled up to suit the environment by adding more instances. Reliable and highly scalable. ---------------------------------------------------------------------- ninjersteve · 2023-10-31 15:24 As far as the general solution to the higher level question asked, this is the way. For the specific application, can certainly question the question of the multiple containers in the first place. ---------------------------------------------------------------------- SamSausages · 2023-10-31 15:51 Not sure why you're getting downvoted. Docker networks is exactly how you would make containers communicate with each other. I use docker compose and when I have a database & app in the stack, those two get their own private docker network. ---------------------------------------------------------------------- naxxfish · 2023-10-31 18:14 You could definitely do all this in one container and use a pipe, or just spawn a new process when you get a message that does all the things you need. If you must put it in multiple containers, have an API on the downloader container that the telegram bot can poke with the messages when it recieves them, via a Docker network. If you wanted to really over engineer it, stick a message broker between the two containers so they can push/pull from a queue or topic. This would enable you to scale up to thousands of concurrent workers should you wish. But I guess you probably don't need that? ---------------------------------------------------------------------- thekrautboy · 2023-10-31 20:14 [Pipe between containers?](https://imgur.com/Q5jYIbb) ---------------------------------------------------------------------- the_idiot_monster · 2023-10-31 21:57 Aren't you looking for some kind of automation workflow tool ? Like n8n or huginn ? ---------------------------------------------------------------------- hunterhulk · 2023-11-02 01:59 why not just use redis ---------------------------------------------------------------------- AsP3X4R3AL · 2023-11-02 09:54 I like your thinking, it makes sense to split the application process into multiple sections when you plan to scale this later but I would recommend to build a crude version of your idea to test if it would even work as you intend.