https://vibhavstechdiary.substack.com/p/mirrord-mirrord-on-the-wall-whos [https] Vibhav's Tech Diary SubscribeSign in Share this post [https] Mirrord Mirrord on the wall, who's most processed of them all vibhavstechdiary.substack.com Copy link Facebook Email Note Other Mirrord Mirrord on the wall, who's most processed of them all Exploring Mirrord 1: The quickest process mirroring from Local to Kubernetes [https] Vibhav Bobade Nov 01, 2024 2 Share this post [https] Mirrord Mirrord on the wall, who's most processed of them all vibhavstechdiary.substack.com Copy link Facebook Email Note Other 2 2 Share In today's post, we are going to explore mirrord. Mirrord is a process mirroring tool which mirrors your local running process and related dependencies to a Kuebrnetes cluster. We will see how to quickly get started with mirrord, and how it works. 31st October 2024 This will take a minute. Support me on Patreon Thanks for reading Vibhav's Tech Diary! Subscribe for free to receive new posts and support my work. [ ] Subscribe Installing Mirrord CLI on the Local system I am currently operating a M2 Mac so running the command below to install the mirrord CLI. brew install metalbear-co/mirrord/mirrord [https] We need a service running in the Kubernetes cluster against which we can test our process mirroring. We will be sending a HTTP GET request to the echo-server through process mirroring. Installing Echo Server on Kubernetes Start minikube. minikube start Let's install the echo-server with the following command. helm repo add ealenn https://ealenn.github.io/charts helm repo update helm install echo-server ealenn/echo-server --namespace echo-server --force --create-namespace Run the command below to send HTTP GET request to the echo-server ClusterIP Service. mirrord exec curl echo-server The echo-server Service is only available from within the context of the Kubernetes cluster (and namespace). The command above runs curl echo-server in the context the Kubernetes cluster and we get the output below. * Running binary "/var/folders/v7/yqyq_d6x2996hfnnwvgs5f080000gn/T/mirrord-bin-ghu3278mz/usr/bin/curl" with arguments: ["echo-server"]. * mirrord will run without a target, no configuration file was loaded * operator: the operator will be used if possible * env: all environment variables will be fetched * fs: file operations will default to read only from the remote * incoming: incoming traffic will be mirrored * outgoing: forwarding is enabled on TCP and UDP * dns: DNS will be resolved remotely a mirrord exec running on latest (3.122.1)! ready to launch process layer extracted operator not found agent pod created pod is ready arm64 layer library extracted config summary {"host":{"hostname":"echo-server","ip":"::ffff:10.244.0.10","ips":[]},"http":{"method":"GET","baseUrl":"","originalUrl":"/","protocol":"http"},"request":{"params":{"0":"/"},"query":{},"cookies":{},"body":{},"headers":{"host":"echo-server","user-agent":"curl/8.4.0","accept":"*/*"}},"environment":{"PATH":"/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","HOSTNAME":"echo-server-cbc7ddb7b-s9c6q","ENABLE__COOKIES":"true","ENABLE__HTTP":"true","ENABLE__HEADER":"true","ENABLE__HOST":"true","ENABLE__REQUEST":"true","LOGS__IGNORE__PING":"false","PORT":"80","ENABLE__ENVIRONMENT":"true","ENABLE__FILE":"true","KUBERNETES_PORT_443_TCP_ADDR":"10.96.0.1","ECHO_SERVER_SERVICE_HOST":"10.103.113.87","ECHO_SERVER_PORT_80_TCP":"tcp://10.103.113.87:80","ECHO_SERVER_PORT_80_TCP_ADDR":"10.103.113.87","KUBERNETES_SERVICE_PORT":"443","KUBERNETES_SERVICE_PORT_HTTPS":"443","KUBERNETES_PORT_443_TCP_PROTO":"tcp","KUBERNETES_PORT_443_TCP_PORT":"443","KUBERNETES_SERVICE_HOST":"10.96.0.1","KUBERNETES_PORT_443_TCP":"tcp://10.96 .0.1:443","ECHO_SERVER_SERVICE_PORT":"80","ECHO_SERVER_SERVICE_PORT_HTTP":"80","ECHO_SERVER_PORT_80_TCP_PROTO":"tcp","ECHO_SERVER_PORT_80_TCP_PORT":"80","KUBERNETES_PORT":"tcp://10.96.0.1:443","ECHO_SERVER_PORT":"tcp://10.103.113.87:80","NODE_VERSION":"16.16.0","YARN_VERSION":"1.22.19","HOME":"/root"}}% The last part of the output after "config summary" was returned from the echo-server Service. The Mirrord Agent (Targetless mode) The mirrord-agent acts as a proxy to the local process due to which it can run in the context of the Kubernetes cluster. More on the agent here. When we run `mirrord exec curl echo-server` the CLI basically creates a Job called the mirrord-agent which looks like the one below. k get job NAME COMPLETIONS DURATION AGE mirrord-agent-95u3z3q18t 0/1 6s 6s And below are the logs for the agent. Had to quickly run it as the exec command runs. kubectl logs -f job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}') agent ready - version 3.122.1 2024-10-30T22:21:14.035109Z WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(5) 2024-10-30T22:21:19.172187Z WARN mirrord_agent::outgoing::udp: interceptor_task -> no messages left at mirrord/agent/src/outgoing/udp.rs:225 on ThreadId(7) 2024-10-30T22:21:19.172908Z INFO mirrord_agent::entrypoint: main -> mirrord-agent `start` exiting successfully. at mirrord/agent/src/entrypoint.rs:824 on ThreadId(1) We don't have any critical information which we can work with here for sure except an observation that there is no UDP data available which can be forwarded to the user on the local host, hence the warning messages above. Apart from that the agent process is exiting successfully. Let's take apart the Job spec now. Let's run the command below and stare at the spec for a bit. kubectl edit job/$(kubectl get jobs --output=jsonpath='{.items[*].metadata.name}') We get the following output. I have annotated important/relevant lines, apiVersion: batch/v1 kind: Job metadata: annotations: batch.kubernetes.io/job-tracking: "" linkerd.io/inject: disabled sidecar.istio.io/inject: "false" creationTimestamp: "2024-10-30T22:32:02Z" generation: 1 labels: app: mirrord kuma.io/sidecar-injection: disabled ### name: mirrord-agent-q4xcvitguv namespace: echo-server resourceVersion: "7817" uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3 spec: backoffLimit: 0 completionMode: NonIndexed completions: 1 parallelism: 1 selector: matchLabels: batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3 suspend: false template: metadata: annotations: linkerd.io/inject: disabled ### sidecar.istio.io/inject: "false" ### creationTimestamp: null labels: app: mirrord batch.kubernetes.io/controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3 batch.kubernetes.io/job-name: mirrord-agent-q4xcvitguv controller-uid: 24167a9e-4c32-4ef7-b233-e9c98c2e73b3 job-name: mirrord-agent-q4xcvitguv kuma.io/sidecar-injection: disabled spec: containers: - command: - ./mirrord-agent - -l - "33863" - targetless ### env: - name: RUST_LOG value: info - name: MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS ## value: "true" - name: MIRRORD_AGENT_NFTABLES value: "false" - name: MIRRORD_AGENT_JSON_LOG value: "false" image: ghcr.io/metalbear-co/mirrord:3.122.1 imagePullPolicy: IfNotPresent name: mirrord-agent resources: limits: cpu: 100m memory: 100Mi requests: cpu: 1m memory: 1Mi securityContext: privileged: false ### terminationMessagePath: /dev/termination-log terminationMessagePolicy: File dnsPolicy: ClusterFirst restartPolicy: Never schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 tolerations: - operator: Exists ttlSecondsAfterFinished: 1 status: active: 1 ready: 1 startTime: "2024-10-30T22:32:02Z" uncountedTerminatedPods: {} Sidecar injection disabled Based on the labels and annotations given above we can tell that these have the Kuma, Istio and LinkerD sidecar injection disabled. Targetless Targetless is the mode because of which we were able to run `curl echo-server` in the content of the Kubernetes cluster without specifying a particular target pod or container. The opposite of this,l targets a particular pod and intercepts/mirrors the traffic and environment of the process in the pod (Targeted mode). Flush existing connections MIRRORD_AGENT_STEALER_FLUSH_CONNECTIONS is an environment variable is set to true which means that existing connections are flushed for more accurate interception of traffic. Unprivileged Container The agent container runs in an unprivileged mode but needs the below Linux capabilities to act as a proxy for processes. These capabilities in Linux allow granular control to be given to processes based on what they can or cannot do. We can see these being used in the targeted mode and not in the targetless mode we were using right now. * CAP_NET_ADMIN and CAP_NET_RAW - used for modifying routing tables. + CAP_NET_ADMIN allows network interface, firewall, routing tables, socket permissions configurations etc. + CAP_NET_RAW allows for creation of RAW and PACKET socket types. * CAP_SYS_PTRACE - used for reading target pod environment. + Gives the ability to use the `ptrace` system call which is usually used for debugging and process control and in this case we can use it to get information about the process running in the Pod. * CAP_SYS_ADMIN - used for joining target pod network namespace. + Gives the ability to mount filesystems, change system time, manage devices etc. (this discussion is continued in the next batch of relevant diary entries/post ...) Support me on Patreon Thanks for reading Vibhav's Tech Diary! Subscribe for free to receive new posts and support my work. [ ] Subscribe [https] 2 Share this post [https] Mirrord Mirrord on the wall, who's most processed of them all vibhavstechdiary.substack.com Copy link Facebook Email Note Other 2 2 Share Discussion about this post Comments Restacks [https] [ ] Yuval Shechori Nov 1Liked by Vibhav Bobade [https] Great guide, thanks! Expand full comment Reply Share 1 reply by Vibhav Bobade 1 more comment... Top Latest Discussions No posts Ready for more? [ ] Subscribe (c) 2024 Vibhav Bobade Privacy [?] Terms [?] Collection notice Start WritingGet the app Substack is the home for great culture Share Copy link Facebook Email Note Other This site requires JavaScript to run correctly. Please turn on JavaScript or unblock scripts