XPipe – A reimagined SFTP client for modern server infrastructure reddit / selfhosted ---------------------------------------------------------------------- milchshakee · 2023-11-25 09:22 I'm proud to share a status update of [XPipe](https://github.com/xpipe-io/xpipe), a new type of shell connection hub and remote file manager that allows you to access your entire server infrastructure from your local machine. It works on top of your installed command-line programs and does not require any setup on your remote systems. ## A new scripting system XPipe 1.7 comes with a new scripting system, so now you can take your shell environment everywhere. The idea is to create modular and reusable shell scripts in XPipe that you can then use for various different use cases. You can set certain scripts to be run on init for every connection independently of your profile files, allowing you to set up a consistent environment across all remote systems without any manual setup. In addition, you can choose to bring scripts to all your remote systems. This will make XPipe automatically copy and update these scripts to a target system if needed and put them in your PATH so that you're able to call them from anywhere. As of now, there is one set of predefined scripts included for enabling the starship prompt in your shells, mainly as a proof of concept. What you will use the scripting system for is up to you. If you like, you can contribute scripts to be included by default. ## Large connection sets A lot of work went into improving the application for large use cases when you're managing hundreds of connections. This includes hierarchical organization features to group all your connections into different categories and subcategories. Furthermore, there have been multiple processing and memory optimizations to ensure that the user experience stays smooth all the time. As a side effect, the memory footprint also has gone down. For people who have to use a potato as their workstation, there's also now a performance mode setting to disable any visual effects that are not required. You can also now tag connections by color for organizational purposes to help in situations when many connections are opened in the file browser and terminals at the same time. These colors will be shown to identify tabs everywhere within XPipe and also outside of XPipe, for example in terminal titles using unicode color symbols. ## Other improvements - In terms of availability, there is now a new [xpipe nixpkg package](https://github.com/NixOS/nixpkgs/pull/259855) which works but not merged yet, a [Microsoft Store package](https://apps.microsoft.com/store/detail/XP9KK2PJ9JDQ6G), and the [choco package](https://community.chocolatey.org/packages/xpipe) is finally marked as trusted - You can now optionally sync your connection configurations with a remote git repository - Additional actions for containers have been added, such as attaching to a container or printing the live logs of a container in a terminal session - A transparency slider has been added so that you can make all windows partially transparent just as you like - Support for many more terminals and text editors across all platforms has been added - Support for BSD systems and special login shells like OPNsense has been added - There's now support to open an SSH connection in your default installed SFTP client or Termius - The powershell execution policy usage has been reworked to always preserve the system default setting - There have been many performance improvements to reduce the startup time, memory usage, file browser loading speed, and more - The homepage at https://xpipe.io/ got an upgrade - Of course, a lot of bugs have been fixed across the board ## Going full-time A few messages I received and the demand for XPipe so far convinced that there is a market for developing XPipe full-time and financing it by special commercial and enterprise plans for interested customers. It essentially encompasses support for enterprise systems and tools that you normally don't find outside of enterprises. This will improve the development speed and quality as I can now fully focus on creating the best possible application. The scope is very small and only involves me, so no investors or other employees. This drastically lowers the break-even value compared to most other tools and allows me to implement a very lenient commercialization. Essentially, you can use most current features without any limitation for free. Furthermore, most upcoming features will also be included in the free version. The open-source model and license also won't change. The only features that require a license are integrations for enterprise systems. For example, if you're trying to connect to a licensed RHEL system or an OpenShift cluster, it will ask you to buy a license. Conversely, with a Rocky Linux system and a k3s cluster, you can use everything for free. These commercial-exclusive implementations will probably not be included in the repository though. Other than that, there are no restrictions so ## Outlook So if you gave this project a try a while ago or it sounds interesting to you, you can check it out [on GitHub](https://github.com/xpipe- io/xpipe)! In the near future, I'm looking to add support for VMs through their CLIs. On top of that, maybe also an automatic network scan functionality to detect all available systems in a network plus support for more device types and shell types like cisco devices. If you have any other ideas, please share them with me. I'm always interested in what kind of use cases other people can come up with. Enjoy! ---------------------------------------------------------------------- Fluffer_Wuffer · 2023-11-25 09:44 Congratulation, I enjoy seeing people take an idea an it flourishes, I really hope it works out for you. ---------------------------------------------------------------------- FluffyMumbles · 2023-11-25 09:59 Looks ace! But why "X" Pipe? ---------------------------------------------------------------------- milchshakee · 2023-11-25 10:05 Internally, the whole functionality is realized by interacting with processes by piping data from and into the stdin/stdout/stderr. I called the internal implementation xpipe as in eXtended pipe because it enables higher level operations on pipes. ---------------------------------------------------------------------- japanfred · 2023-11-25 11:48 Starred, will look at it this week! Great work ---------------------------------------------------------------------- themedleb · 2023-11-25 12:02 > Note that this is a desktop application ... Since it is a desktop app, is there any plan to provide a Flatpak version of it? Or that's not possible because of the its nature? ---------------------------------------------------------------------- milchshakee · 2023-11-25 12:05 Sadly the sandboxed nature of flatpak it prevents xpipe from working properly as it can't run your local programs. I already experimented with a flatpak build, but since there is no option to disable the sandboxing, it doesn't work. ---------------------------------------------------------------------- -SHINSTER007 · 2023-11-25 12:15 Can you do scoop? ---------------------------------------------------------------------- milchshakee · 2023-11-25 12:17 For now only choco and winget are supported, but I will continue adding support for more package managers over time ---------------------------------------------------------------------- PowerfulAttorney3780 · 2023-11-25 12:31 Thank you man You're awesome! ---------------------------------------------------------------------- dotinho · 2023-11-25 12:35 Nice, I’m a Mac user, I’ll try for sure 👍 ---------------------------------------------------------------------- themedleb · 2023-11-25 12:54 I see. Thank you sir. ---------------------------------------------------------------------- valdearg · 2023-11-25 13:49 Just on winget, looks like the winget version is 1.5.3 at the moment, would it be possible to push the updates through to it? Name Id Version Match Source --------------------------------------------------------------------- XPipe XP9KK2PJ9JDQ6G Unknown msstore XPipe xpipe-io.xpipe 1.5.3 winget XPipe Portable xpipe-io.xpipe.portable 1.5.3 Moniker: xpipe winget ---------------------------------------------------------------------- milchshakee · 2023-11-25 13:50 The msstore version is the correct one, the others are community-made. I honestly don't know why the version is not shown there. So you should be able to run `winget install xpipe --source msstore`. ---------------------------------------------------------------------- BigKitten · 2023-11-25 13:55 This is great work. I feel I am gonna shift my windows system to yours. But do you have plans to natively support arm on Mac? Your arm version seems to be based on rosetta? ---------------------------------------------------------------------- milchshakee · 2023-11-25 13:57 It should be natively supported, the artifacts are build on ARM runners. I will double check now that you mention it, I hope that there wasn't a mix up. ---------------------------------------------------------------------- valdearg · 2023-11-25 13:58 Oh, got you! Maybe something to consider for the future, just to cover all bases as I know the Windows Store source isn't always available. The winget source is pretty cool and I've seen a fair number of people automate the submission process, e.g. GitHub Action on release, write the pull request. ---------------------------------------------------------------------- milchshakee · 2023-11-25 14:05 Sure, I can look into that. Do you have any links for the window store source availability? I wasn't aware of that, probably because I'm not really familiar with winget. ---------------------------------------------------------------------- BigKitten · 2023-11-25 14:08 Thank you so much for looking into that! ---------------------------------------------------------------------- milchshakee · 2023-11-25 14:12 I just did, all ARM64 releases I downloaded were arm executables for me. How did you install XPipe? ---------------------------------------------------------------------- BigKitten · 2023-11-25 14:23 Thank you. I did both homebrew and the pkg directly from releases. They are arm executables but in the installation process both versions (well they are the same pkg file to be specific) require rosetta. ---------------------------------------------------------------------- milchshakee · 2023-11-25 14:25 That is interesting, I will look into that. I probably messed something up in the pkg generation. The app itself should run without rosetta for you ---------------------------------------------------------------------- BigKitten · 2023-11-25 14:32 Thank you. Just tried the portable version and it works perfectly (and I have no idea it works with warp. Thank you!) ---------------------------------------------------------------------- valdearg · 2023-11-25 14:32 Generally Windows Store is mostly available on the desktop OS, but it can be locked down heavily by organisation group policies, geo- restrictions, and can also not be available on Windows Server installations. WinGet itself can be a bit lighter and can be better for transparency and easier to use in deployment. I think it's been a fantastic inclusion with Windows. Of course, this is not absolutely required (and absolutely not a detraction from your great work on XPipe!) but may be beneficial for users if it's a quick setup once procedure. Looks like there's a GitHub Actions marketplace item available [here](https://github.com/marketplace/actions/winget-releaser) which might be an option, looks also like MS have an option also which seems to be automatable [here](https://github.com/microsoft/winget-create). ---------------------------------------------------------------------- milchshakee · 2023-11-25 14:34 Thanks for the extensive information, I will look into publishing to winget itself as well. ---------------------------------------------------------------------- Darkchamber292 · 2023-11-25 15:05 Not exactly true. Look into Flatseal. Only issue is you'd have to rely on the user to set it up. ---------------------------------------------------------------------- milchshakee · 2023-11-25 15:11 I'm not very familiar with that, but can it disable the built-in flatpak sandbox? From what I can see it can just modify the privileges of packages, but there is no privilege to disable the sandbox. ---------------------------------------------------------------------- Darkchamber292 · 2023-11-25 15:16 Yea I don't think you can disable the sandbox but you can give it access to binaries and convinced and directories on the system. I figured that'd be sufficient but I'm not a developer just a humble Sys Admin :) ---------------------------------------------------------------------- milchshakee · 2023-11-25 15:18 In practice that would be a lot of binaries (shells, coreutils commands, ssh, etc.), maybe I can somehow automate that by providing a script ---------------------------------------------------------------------- edthesmokebeard · 2023-11-25 15:18 Can you explain "for modern server infrastructure" ? ---------------------------------------------------------------------- Darkchamber292 · 2023-11-25 15:22 You can give your flatpak access to the directory like /usr/bin or /etc and then it can run any system binaries there You can use ` --filesystem=host` and then access them in the sandbox under /run/host. ---------------------------------------------------------------------- milchshakee · 2023-11-25 15:23 Ten years ago, you would maybe have a couple of servers all reachable via SSH And a normal SFTP client work perfectly there. Nowadays, you have many containers, VMs, different local shell environments, the windows subsystem, maybe clusters, and more. The traditional SFTP clients fall flat here as they simply can't connect to them as there is no SSH protocol available many times, or it is tedious to set up. You also have to keep up with organizing may be hundreds of system connections. The goal is to provide a tool that feels just like a normal SFTP client but works on many more systems. ---------------------------------------------------------------------- milchshakee · 2023-11-25 15:24 Alright, I will play around with that to see what's possible. ---------------------------------------------------------------------- LukeLC · 2023-11-25 16:16 This looks amazing! Definitely going to give it a try! ---------------------------------------------------------------------- root_switch · 2023-11-25 16:42 I really don’t like the UI, it looks like a 90s-early 2000s pc game lol. ---------------------------------------------------------------------- milchshakee · 2023-11-25 16:46 There are multiple themes available, this is just one of them. Ideally, there should be something for everyone. The main challenge for the UI is creating a consistent look across all platforms, so it can't compete with native UIs. ---------------------------------------------------------------------- HadManySons · 2023-11-25 18:29 Not gonna lie, this is pretty amazing. Well done. ---------------------------------------------------------------------- rez410 · 2023-11-25 19:08 This looks great! I haven’t tried it out yet, but why do you keep on emphasizing that it can do SFTP? It is seemingly much more than that. More like a competitor to Royal TS. Just feels like you are selling it short. I personally would emphasize its ability to manage all of your connections, (passwords, ssh keys?) etc. I almost skimmed by this post because I can’t remember which decade I last cared to look for a simple sftp client. > I’m looking to add support for VMs through their CLI What do you mean by this? ---------------------------------------------------------------------- milchshakee · 2023-11-25 19:41 I experimented with different names for it but I found out that people know instantly what you're talking about when you mention the words SFTP client. Usually I referred to it as a shell connection hub and remote file manager, but that wasn't quite so catchy. I am open for catchy one liner suggestions. VMware and VirtualBox desktop hypervisors come with a CLI and I experimented with adding support to manage and to connect to your VMs via the CLI in combination with SSH. Experimental VMware support is already implemented. ---------------------------------------------------------------------- rez410 · 2023-11-25 20:53 Ah ok, that makes sense. I would probably word that like “looking to add CLI support for type 2 hypervisors” ---------------------------------------------------------------------- milchshakee · 2023-11-25 21:00 I think it's the same problem here with the technical correct naming. I would argue that many people don't exactly know what type 2 hypervisor refers to. If you just casually run some VMs with VMware player or virtual box, then you don't really come into contact with that name. ---------------------------------------------------------------------- linkinx · 2023-11-25 21:43 Can I use it for my ssh connections, like to replace securecrt? ---------------------------------------------------------------------- milchshakee · 2023-11-25 21:46 XPipe comes with full support for ssh connections, so in theory yes. In the end it is up to you which one you like better and whether you want to switch from securecrt. ---------------------------------------------------------------------- linkinx · 2023-11-25 21:49 I'll give it a shot, thanks! ---------------------------------------------------------------------- aguerooo_9320 · 2023-11-25 22:11 Can I use it from a mobile browser? ---------------------------------------------------------------------- milchshakee · 2023-11-25 22:12 It is a desktop application, so you sadly can't use it on a mobile device or a browser. ---------------------------------------------------------------------- Lkcarfrey · 2023-11-25 23:02 Ok forgive me but I'm fairly new to self hosting. Still researching but what type of program is this similar to. (Forgive me if I'm not using right terminology ---------------------------------------------------------------------- milchshakee · 2023-11-25 23:08 If you have heard about it, it is similar to things like MobaXterm, WinSCP, or Termius, which can be categorized as SSH/SFTP clients, connection managers, or remote file managers. But this is only a rough categorization as there are still differences between every single product. ---------------------------------------------------------------------- Lkcarfrey · 2023-11-25 23:44 I am currently learning CompTIA LINUX+ so I am familiar with ssh but thank you ---------------------------------------------------------------------- -pLx- · 2023-11-26 01:04 This looks insanely good, I can’t wait to try it out! ---------------------------------------------------------------------- -pLx- · 2023-11-26 01:05 This looks insanely good, I can’t wait to try it out! ---------------------------------------------------------------------- mundane_motto · 2023-11-26 04:55 How can I get the "in program" terminal as in [this screenshot](https: //preview.redd.it/ge3pwm7uog2c1.png?width=108&crop=smart&auto=webp&s=a b269f49eb76955bf52c4106ec1cd9c3a0c9f016) on KDE Linux? For me XPipe just opens an external terminal window... ---------------------------------------------------------------------- milchshakee · 2023-11-26 08:31 Oh I just notice that this screenshot is a little bit misleading, that is just the Windows Terminal window opened by XPipe. I moved it around a bit in order to not cut off any content halfway. I will have to think about maybe adjusting this screenshot so it becomes more clear that it is an external window. ---------------------------------------------------------------------- mundane_motto · 2023-11-26 09:28 Oh :/. Thank you for your reply. This would have been such an awesome feature! At least in KDE Linux it should be possible to implement since e.g. [their file explorer Dolphin also let's you show a terminal window](https://userbase.kde.org/Dolphin/File_Management#Terminal) - so there must be some kind of component for it. ---------------------------------------------------------------------- milchshakee · 2023-11-26 09:35 Yeah, that would be nice to have. Sadly implementing an integrated terminal is very difficult and time consuming. Especially when you are trying to on the level of other existing terminals who have dedicated teams behind them that just work on the terminal core functionality. ---------------------------------------------------------------------- maevian · 2023-11-26 19:57 Does it have support for ssh connection protected by FIDO? ---------------------------------------------------------------------- milchshakee · 2023-11-26 19:59 That is actually currently being worked on. I just received my yubikey a few days ago that I ordered to implement this ---------------------------------------------------------------------- maevian · 2023-11-26 20:31 Thanks ---------------------------------------------------------------------- Morgan-CS · 2023-11-26 20:40 Congrats! This is a great project. ---------------------------------------------------------------------- Oujii · 2023-11-27 21:54 It doesn't open on my MacOS 10.15.7 machine. Does it not support Catalina? Thank you! ---------------------------------------------------------------------- milchshakee · 2023-11-27 21:56 I honestly did not back test and check the minimum OS version for macOS. You can try to run `/Applications/XPipe.app/Contents/Resources/scripts/xpiped_debug.sh` and see whether there is maybe an error. ---------------------------------------------------------------------- Oujii · 2023-11-27 22:08 The app "opens", but you can never see the user interface. It doesn't load. Should I run this script standalone or after I open the app? Or maybe run the script and then open the app? ---------------------------------------------------------------------- milchshakee · 2023-11-27 22:11 This script should be run standalone in a terminal. If it opens but never shows, you can try running it with `/Applications/XPipe.app/Contents/Resources/scripts/xpiped\_debug.sh -Dio.xpipe.app.debugPlatform=true -Dprism.order=j2d` and see what happens. ---------------------------------------------------------------------- Oujii · 2023-11-27 22:20 Okay, it did complain of the GUI not loading. ``` 19:19:51:984 - info: [mode] Finished base components initialization 19:19:51:985 - info: Attempting to switch mode to gui 19:19:51:985 - info: Gui is not available, using base instead ``` ---------------------------------------------------------------------- milchshakee · 2023-11-27 22:24 Hmm ... and there is nothing in the logs that looks like an error? Ideally it should tell us why it couldn't initialize the GUI ---------------------------------------------------------------------- pushc6 · 2023-11-28 13:39 Downloaded it, seemed really neat. Can't use it because most of my hosts in my homelab are RHEL and it want's me to pay for a commercial license. :( ---------------------------------------------------------------------- milchshakee · 2023-11-28 13:43 Oh I did not anticipate that people use many RHEL systems in their homelab. I thought that with RHEL there was a clear distinction of commercial usage intent. Is there any particular reason why you use RHEL itself and not any other RHEL-based systems? I can think of refining the commercial check to allow for some kind of home usage even with commercial operating systems ---------------------------------------------------------------------- pushc6 · 2023-11-28 16:01 I think for most homelabs that's a safe assumption. For me, at work we use RHEL (shocker) and I wanted to be able to have 1:1 learnings in my home lab that will transfer to work. I could use something like centos, but since I get RHEL for free in my lab, that's the direction I went. I do have a handful of ubuntu machines for temporary stuff as well as a centos server, but at least that's my rationalization. I think you're probably fine with RHEL, I'm probably in the minority :) ---------------------------------------------------------------------- milchshakee · 2023-11-28 16:35 I'm not super familiar with RHEL licensing, do they have free versions available? If so, I could just check whether or not you are using the free version. ---------------------------------------------------------------------- pushc6 · 2023-11-28 17:13 They are full versions, so you wouldn't know. It's through their developer program. I don't know how you would know me running 3 rhel instances at home is any different than someone running 3 commercially. I manage FAR more rhel instances at work, so maybe limiting the number of RHEL\commercial type OS locally may help. ---------------------------------------------------------------------- milchshakee · 2023-11-28 18:04 Alright, maybe I can find a solution here with local operating systems. ---------------------------------------------------------------------- Internal_Seesaw5612 · 2023-11-28 19:04 This was fine until you're making me pay just to ssh into a rhel machine. Looks nice though. ---------------------------------------------------------------------- milchshakee · 2023-11-29 00:17 I decided to make the cutoff between free and licensed use at operating systems that are typically used in enterprise environments. I hoped that this would play nicer with most people as it doesn't impose any other limits, e.g. on how much connections you can have like many other tools do. But yeah this it does not apply to everyone. ---------------------------------------------------------------------- maevian · 2023-11-29 20:43 If you’re using putty as a back end, putty cac works like a charm with Fido. It’s what I use when I am on Windows ---------------------------------------------------------------------- milchshakee · 2023-12-01 00:31 Alright, I check the minimum requirements for the libraries I use and it is sadly osx 11+. However, I used your problems as motivation to rework the error output so that in the future people are more aware of minimum requirements. ---------------------------------------------------------------------- Oujii · 2023-12-01 01:06 Oh, thank you for the feedback! I imagined it had to do with my MacOS version, I will be upgrading soon as I have other software not running too. ---------------------------------------------------------------------- milchshakee · 2023-12-05 16:08 This issue should be fixed in 1.7.9. I did not have the host architecture parameter set before and it seems like it automatically assumes x86_64 then. ---------------------------------------------------------------------- milchshakee · 2023-12-17 08:56 Alright I was able to work on smartcard support for PKCS#11, GPG agent support, FIDO, and more. I am focusing on using the native Windows OpenSSH client which works fine for FIDO if you install newer versions with v8.3+. So you can try it out if you want. I was only able to test it on my few systems and with one yubikey, so I have no idea whether it will work for everyone. ---------------------------------------------------------------------- maevian · 2023-12-20 05:54 I need top updates OpenSSH first? ---------------------------------------------------------------------- milchshakee · 2023-12-20 08:33 I think you need OpenSSH 8.3p1 or higher for that