[HN Gopher] Socat - A utility for data transfer between two addr...
___________________________________________________________________
Socat - A utility for data transfer between two addresses (2018)
Author : brigandish
Score : 212 points
Date : 2021-03-25 07:20 UTC (2 days ago)
(HTM) web link (copyconstruct.medium.com)
(TXT) w3m dump (copyconstruct.medium.com)
| prhn wrote:
| I use socat daily.
|
| Nearly all of the microservices we develop have a way to be run
| locally with all dependent services using docker-compose.
|
| The problem is most teams don't provide a way to do efficient
| local development for their particular service. Some do remote
| debugging into the built/running container (slow and requires
| rebuilding docker image for every change). Some inspect logs to
| debug (yikes!).
|
| When I want to debug a particular service in IntelliJ, I simply
| remove it from the docker network and replace it with a container
| running socat, forwarding the appropriate port to
| host.docker.internal (the host machine). This is on a MAC, of
| course, where Docker runs in a VM.
|
| This allows me to do in-IDE debugging of any service that runs in
| the docker network without having to rebuild the image every time
| I make a change.
|
| I know there are IntelliJ plugins that provide a similar
| workflow, but I believe they all build an image and deploy a
| container to the network for every change. I could be wrong.
| e12e wrote:
| > remove it from the docker network and replace it with a
| container running socat, forwarding the appropriate port to
| host.docker.internal (the host machine
|
| Would you happen to have a fully worked example of this? We to
| have some services that we can and do run via docker compose -
| but it's often a cumbersome process to move one and one service
| out of the docker compose setup for debug/devel.
| tomerbd wrote:
| Interesting .. Can you provide an example?
| robbles wrote:
| How do you swap out the container in docker-compose?
|
| Do you just have a service defined for each, and only start one
| of them?
| Bluecobra wrote:
| Great article! I use socat all the time to test multicast
| subscriptions in Linux.
| oofnik wrote:
| I've used socat in the past to debug all kinds of odd network
| behavior. It helped me discover a networking issue with some code
| by comparing performance of piping a Unix socket over TCP using
| socat vs. the native TCP implementation, which turned out to be
| caused by an overly conservative default send buffer limit.
| cperciva wrote:
| A similar tool: spiped[0]. Connects two arbitrary TCP4, TCP6, or
| UNIX socket addresses, encrypting + authenticating data using a
| preshared key. For many applications it's basically socat +
| security.
|
| [0] https://www.tarsnap.com/spiped.html
| qbasic_forever wrote:
| Have they done any audits or had critical analysis of their
| security code? These days I'm super hesitant to consider
| anything that hasn't had a lot of eyes on it. It's just too
| easy for a tiny screwup in protocol design/implementation,
| random number generation, etc. that puts everything at risk.
| Why risk it when I can just use wireguard.
| boulos wrote:
| cperciva is the author of spiped and a security expert [0].
|
| But your question about an audit is still a good one! Tarsnap
| has bug bounties, but the top hit for "spiped security audit"
| led to a sort of amusing flame war on HN [1].
|
| [0] https://www.tarsnap.com/about.html
|
| [1] https://news.ycombinator.com/item?id=7539903
| cperciva wrote:
| Lots of people have looked at the spiped source code; it has
| been around since 2011 and I've even handed out a few bug
| bounties.
|
| It's also considerably simpler than wireguard, which reduces
| the potential for vulnerabilities; and operates on a per-
| connection basis, which means you can do things like
| establishing a secure connection to another host but have a
| socket endpoint which can only be accessed by certain users.
| iJohnDoe wrote:
| Is it possible to use Socat (or any utility) to redirect a UDP
| connection on a local Windows machine to TCP port 443/80 to a
| remote system listening on 443/80 that will connect to the
| original intended UDP connection?
|
| I can also do port redirects with iptables that can assist with
| solving the problem.
|
| I need to host a microservice that uses UDP which won't connect
| out on most corporate networks due to firewalls. I need to get
| the connection out via a standard TCP port.
|
| Nothing malicious. It's a basic corporate client/server app, but
| it's bound to a UDP port that I can't change.
| cyberpunk wrote:
| I think your best bet is to setup a simple WireGuard for this,
| also udp but put the server on 53 or something it's unlikely to
| be blocked.
| cyberpunk wrote:
| Oh or skip that abd just use pen ( http://siag.nu/pen/ ) to
| get the traffic from 53->whatever port (can't remember if
| haproxy supports udp)
| alexeldeib wrote:
| huh. I haven't heard of this one. Seems like it's been
| around a while? Why might I use this compared to something
| more well known? haproxy doesn't seem to do generic udp,
| but maybe linux virtual server or something like that?
| tssva wrote:
| I use socat along with npiperelay to allow the ssh client in my
| WSL sessions to access the Windows ssh-agent.
| strictfp wrote:
| Interesting! Do you mind elaborating a bit? I ssh into WSL for
| this reason.
| shawnz wrote:
| Just to throw another one of these bridge applications into
| the mix, here is one that allows Pageant apps to communicate
| with Windows' native ssh-agent.
|
| https://github.com/ndbeals/winssh-pageant
|
| I use both this and rupor's wsl-ssh-agent from the other
| thread extensively.
| tssva wrote:
| When I looked for a similar solution quite a while ago one
| didn't exist so I wrote my own. I have been meaning to
| clean the code up and post it for about 18 months now but
| have never gotten around to it. Need to check this out and
| see if it is worth still using what I wrote.
| shawnz wrote:
| Let that be a lesson to not abandon your side projects :)
|
| I was in need of a solution to this issue for a long time
| before ndbeals thankfully released this about 9mo ago.
| And I was checking around the web regularly! I nearly
| wrote my own as well.
|
| The use case for me was to allow WinSCP and Sourcetree to
| be used with the Windows ssh agent.
| tssva wrote:
| WinSCP is the reason I wrote mine. I'll probably stick
| with the one I wrote since it is win32 native and seems
| to consume less resources. I'm not a developer by trade
| and barely one by hobby. This was the first and so far
| only win32 app I have written so the code while
| functional is very ugly. I'm sure anyone familiar with
| win32 programming would be horrified. I should just throw
| it over the wall as is. Obviously the cleanup I was going
| to do before releasing it is never going to happen.
| nullify88 wrote:
| Likely something like this https://github.com/rupor-
| github/wsl-ssh-agent#wsl-2-compatib...
|
| Similarly, you can also do this with KeepassXC
| tssva wrote:
| I followed the directions at
| https://stuartleeks.com/posts/wsl-ssh-key-forward-to-
| windows... to get the combination of socat and npiperelay
| to work for this purpose. I did slightly deviate from his
| final script by using pgrep instead of the combination of
| ps and grep to detect if npiperelay was already running.
|
| Edit: These directions are very similar to those linked in
| the comment this is in reply to and either will work.
| dataflow wrote:
| I haven't tried on WSL2, but on WSL1 this script also lets
| you integrate with KeePass's KeeAgent:
|
| https://gist.github.com/duebbert/4298b5f4eb7cc064b09e9d865dd.
| ..
| shawnz wrote:
| In the past I also used https://github.com/benpye/wsl-ssh-
| pageant to go from WSL to Pageant (which I believe KeeAgent
| is also compatible with)
| justin_oaks wrote:
| I often use a socat docker container to proxy network traffic to
| docker containers. It effectively allows creating dynamic docker
| port mappings.
|
| For example, I have a database with no docker port mappings (only
| a web application container in the same docker network accesses
| it directly). I can use a socat container with port mappings to
| proxy network traffic to the database.
| barbazoo wrote:
| Interesting, what's the benefit over exposing the port through
| Docker?
| throwaway888abc wrote:
| Security
|
| https://en.wikipedia.org/wiki/DMZ_(computing)
| Datagenerator wrote:
| Shameless plug [1], socat combined with WireGuard
|
| [1] https://github.com/lrvl/tunnel-wireguard-udp2tcp
| je42 wrote:
| very versatile - socat PTY,Link is awesome.
| prussian wrote:
| very underrated. I've used this trick before to mock serial
| ports on some Linux-running industrial devices so I could debug
| them.
| inivekin wrote:
| I made a small utility that'll replay socat log files
| captured by the -lf -x -v flags with the timing it collects
| if used in this way. It is very simple and is basically only
| there if you can't be bothered to roll your own:
| https://gitlab.com/inivekin/socatplayer/-/releases
|
| So if you're commonly doing a socat process like: socat -x
| OPEN:/dev/ttyS0,b115200,echo=0,icanon=0
| PTY,link=/dev/ttyO0,rawer 2> socat_log.log
|
| then it may be of interest to you.
| fguerraz wrote:
| I use socat a lot, especially when wanting to transfer files
| without incurring the ssh encryption overhead or rsync daemon
| setup. Just tar | scocat and voila, Superfast whole directory
| tree transfer. Sprinkle a bit of mbuffer or openssl on top and
| you can do a lot!
| AnotherGoodName wrote:
| It talks serial too. I've used socat to send out data to the
| network from a set of serial connected devices for environmental
| modelling. Ended up being one of those jobs I got done from end
| to end in an afternoon with the socat part taking 10mins
| megous wrote:
| Oh! I might as well use it for line editing on serial line with
| the readline local and serial remote addresses, for example
| when talking AT to a modem. Great! :)
|
| And with history and all that stuff, too! I guess it can be
| used to forward serial port over network too.
| jonenst wrote:
| We use socat to buffer logs sent from containers (docker-compose)
| during the short window at startup where logstash and
| elasticsearch are not ready yet. Works great!
| [deleted]
| mrspuratic wrote:
| Similar: using it to get syslog out of self-chrooting ssh with
| minimal trauma: socat -u UNIX-
| RECV:/home/sftponly/user/dev/log,mode=666 UNIX-SENDTO:/dev/log
|
| It's also an CLI IMAPS client, with history: socat
| READLINE,history=$HOME/.imaps_history EXEC:'"openssl s_client
| -connect mailserver:993"'
|
| or a CLI web browser with history: socat -d -d
| READLINE,history=$HOME/.http_history
| TCP4:www.domain.com:www,crnl
|
| or, if you have to interact with something that has no
| readline, say sendmail: socat READLINE EXEC:"sendmail -bt"
| nwjsmith wrote:
| Have you happened to blog about this? I'd love to be able to
| fix our setups with this trick!
| jorl17 wrote:
| socat is an invaluable tool. I first came about it at my first
| job, where we (just two clueless dudes) had to migrate the mostly
| undocumented infrastructure of about 15 machines at Digital Ocean
| to Linode, adding replication and failover on top of that (making
| it ~35 or so machines).
|
| During that migration, we tried to minimize downtime, so we made
| sure that requests made to the old infrastructure were aptly
| forwarded to the new infrastructure wherever possible, until DNS
| properly propagated. I found out about socat and we went for it.
| It worked great, and I marveled at the simplicity of the
| solution!
|
| Ever since then, I've used socat both as an emergency saviour and
| as an actual service. I seem to recall using it once to go around
| some docker bug or shenanigans.
| bombcar wrote:
| This is a brilliant idea and I've never even considered doing
| that form of port forwarding.
|
| I'll have to remember it for the next time I migrate.
| maddyboo wrote:
| Excellent article!
|
| I've tried to grok socat a few times but gave up quickly upon
| being faced with its screenfuls of dense help text.
|
| This post sheds light on the core concepts behind the tool in a
| way that the man page and help text just didn't do for me.
|
| Now that I 'get' it, I'm actually excited to find an opportunity
| to use socat! I can think of so many interesting use cases.
| ldiracdelta wrote:
| I use these two socat commands for MITM all the time on my own
| stuff so I can see the traffic: socat -v TCP-
| LISTEN:666,forever,reuseaddr,fork TCP:example.com:8080
| socat -v TCP-LISTEN:8080,forever,reuseaddr,fork
| ssl:example.com:443
| axiolite wrote:
| Socat is terrible.
|
| When used to establish a persistent connection to a remote
| listening (TCP) port, it has NO WAY to detect that the remote end
| has dropped the connect, and automatically reconnect. It just
| does not work.
|
| Yes, I know all about all the keep-alive and retry options in the
| manual. Try it yourself.
|
| It's one of those horrible tools that you can get up and running
| quickly, but can never make work reliably.
| bandie91 wrote:
| i doubt that socat did not handle TCP RST. is not it the
| network environtment which eats TCP RST in some cases (like AWS
| NLB).
| sundbry wrote:
| I don't think you understand how TCP works. A TCP connection
| can stay alive through an unplugged Ethernet cable if it hasn't
| sent any packets.
| vesinisa wrote:
| This tool is like netcat on steroids. Definitely part of my
| stable toolbelt.
| rwmj wrote:
| Whenever someone mentions netcat or nc I always ask "which
| one?". There are at last count 3 different implementations, all
| subtly incompatible. Which is another reason why socat is
| better since there's only one to deal with.
| cogburnd02 wrote:
| How do you refer to each implementation & what are some of
| the subtle incompatibilities?
| vesinisa wrote:
| The original one is the "TCP/IP Swiss army knife" released
| in 1995:
|
| https://nc110.sourceforge.io/ (original release:
| https://seclists.org/bugtraq/1995/Oct/28)
|
| Although it is "freely given away to the Internet
| community" with "an obligation to give credit where due",
| at least OpenBSD and GNU have seen the need to write their
| own versions under their project licenses:
|
| GNU netcat: http://netcat.sourceforge.net/
|
| OpenBSD nc: https://man.openbsd.org/nc.1
|
| (The OpenBSD version has been ported to at least FreeBSD
| and Apple Macintosh OS.)
|
| All of them have the same basic telnet's `host port` syntax
| for outbound TCP connections, but annoyingly the syntax for
| opening a local listening TCP socket varies. Say, you want
| to open a TCP socket listening on port 1234 (local), and a
| confirmation when it is ready:
|
| The original and GNU netcats: netcat -v -l -p 1234
|
| BSD netcat: nc -v -l 1234
| rwmj wrote:
| That makes _4_ implementations, because nmap also has one
| (https://nmap.org/ncat/).
| vesinisa wrote:
| Dammit, they have of course gone with the BSD syntax for
| opening a local listen socket (ncat -l 1234):
|
| https://nmap.org/book/ncat-man-examples.html
|
| Means half of implemations use `-l -p 1234` and half `-l
| 1234`.
| tyingq wrote:
| Indeed. There are examples for making local/remote shells. It
| can make a pty, use setsid() to be a daemon, reset the terminal
| state, then listen(), etc, all in a one liner. Or route a
| serial port over ip via ptys in raw mode, again, with a one-
| liner.
| cowmix wrote:
| Finally socat is getting the luv it deserves.
| rgovostes wrote:
| I'll join the chorus praising socat, but also share my gratitude
| for Gerhard, the developer / maintainer.
|
| We exchanged emails back and forth about a use case I had, which
| was to split a serial device into two PTYs, allowing a process to
| control one while I monitor the other with picocom. We couldn't
| get it to work, so I gave up, but then Gerhard wrote back a few
| months later saying he added it as a feature.
|
| It can be frustrating to figure out the right incantation for
| some advanced use cases, but this is more a complaint about
| esoteric corners of the operating system than socat which exposes
| the full complexity to you.
|
| One of the invocations I recently worked out was to wrap an
| interactive process with a PTY, so that I could make a mock
| serial device that emulates a piece of hardware.
| socat PTY,link=/tmp/dev,rawer,wait-slave \
| EXEC:"./emulator",pty,setsid,ctty,echo=0
| lukax wrote:
| Socat is the default implementation for Kubernetes port-
| forwarding [1] when using dockershim for the kubelet runtime.
| Similar to how tar is used as the default Kubernetes cp command
| implementation.
|
| [1]
| https://github.com/kubernetes/kubernetes/blob/770d3f181c5d7e...
___________________________________________________________________
(page generated 2021-03-27 23:00 UTC)