https://github.com/felixge/fgtrace Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code + Explore + All features + Documentation + GitHub Skills + Changelog * Solutions + By Plan + Enterprise + Teams + Compare all + By Solution + CI/CD & Automation + DevOps + DevSecOps + Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles + Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this user All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} felixge / fgtrace Public * Notifications * Fork 2 * Star 369 fgtrace is an experimental profiler/tracer that is capturing wallclock timelines for each goroutine. It's very similar to the Chrome profiler. License MIT license 369 stars 2 forks Star Notifications * Code * Issues 0 * Pull requests 0 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights felixge/fgtrace This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 1 branch 1 tag Code * Clone HTTPS GitHub CLI [https://github.com/f] Use Git or checkout with SVN using the web URL. [gh repo clone felixg] Work fast with our official CLI. Learn more. * Open with GitHub Desktop * Download ZIP Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @felixge felixge Update README.md ... 624b70f Sep 19, 2022 Update README.md 624b70f Git stats * 18 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github/workflows assets internal testdata LICENSE.txt README.md fgtrace.go fgtrace_test.go go.mod go.sum go.work go.work.sum handler.go handler_test.go View code [ ] fgtrace - The Full Go Tracer Quick Start Comparison with Similar Tools fgtrace fgprof runtime/trace How it Works The Future License README.md fgtrace - The Full Go Tracer ci test status documentation fgtrace is an experimental profiler/tracer that is capturing wallclock timelines for each goroutine. It's very similar to the Chrome profiler. [?][?] fgtrace may cause noticeable stop-the-world pauses in your applications. It is intended for dev and testing environments for now. [fgtrace-ex] Quick Start To capture an fgtrace of your program, simply add the one-liner shown below. This will cause the creation of a fgtrace.json file in the current working directory that you can view by opening it in the Perfetto UI. package main import "github.com/felixge/fgtrace" func main() { defer fgtrace.Config{Dst: fgtrace.File("fgtrace.json")}.Trace().Stop() // } Alternatively you can configure fgtrace as a http.Handler and request traces on-demand by hitting http://localhost:1234/debug/fgtrace? seconds=30&hz=100. package main import ( "net/http" "github.com/felixge/fgtrace" ) func main() { http.DefaultServeMux.Handle("/debug/fgtrace", fgtrace.Config{}) http.ListenAndServe(":1234", nil) } For more advanced use cases, have a look at the API Documentation. Comparison with Similar Tools Below is a simple program that spends its time sleeping, requesting a website, capturing the response body and then hashing it a few times. for i := 0; i < 10; i++ { time.Sleep(10 * time.Millisecond) } res, err := http.Get("https://github.com/") if err != nil { panic(err) } defer res.Body.Close() var buf bytes.Buffer if _, err := io.Copy(&buf, res.Body); err != nil { panic(err) } for i := 0; i < 1000; i++ { sha1.Sum(buf.Bytes()) } Now let's have a look at how fgtrace and other tools allow you to understand the performance of such a program. fgtrace Looking at our main goroutine (G1), we can easily recognize the operations of the program, their order, and how long they are taking (~100ms time.Sleep, ~65ms http.Get, ~30ms io.Copying the response and ~300ms calling sha1.Sum to hash it). However, it's important to note that this data is captured by sampling goroutine stack traces rather than actual tracing. Therefore fgtrace does not know that there were ten time.Sleep() function calls lasting 10ms each. Instead it just merges its samples into one big time.Sleep() call that appears to take 100ms. Another detail are the virtual goroutine state indicators on top, e.g. sleep, select, sync.Cond.Wait and running/runnable. These are not part of the real stack traces and meant to help understanding On-CPU activity (running/runnable) vs Off-CPU states. You can disable them via configuration. [fgtrace-ex] To break down the latency of our main goroutine, we can also look at other goroutines used by the program. E.g. below is a closer look on how the http.Get operation is broken down into resolving the IP address, connecting to it, and performing a TLS handshake. [fgtrace-ex] So as you can see, fgtrace offers an intuitive, yet powerful way to understand the operation of Go programs. However, since it always captures the activity of all goroutines and has no information about how they communicate with each other, it may create overwhelming amounts of data in some cases. fgprof You can think of fgprof as a more simplified version of fgtrace. Instead of capturing a timeline for each goroutine, it aggregates the same data into a single profile as shown in the flame graph below. [fgprof-exa] This means that the x-axis represents duration rather than time, so function calls are ordered alphabetically rather than chronologically. E.g. notice how time.Sleep is shown after sha1.Sum in the graph above even so it's the first operation completed by our program. Additionally the data of all goroutines ends up in the same graph which can be difficult to read without having a good understanding of the underlaying code and number of goroutines that are involved. Despite these disadvantages, fgprof may still be useful in certain situations where the detail provided by the timeline may be overwhelming and a simpler view of the average program behavior is desirable. Additionally fgprof under Go 1.19 has less negative impact on the performance of the profiled program than fgtrace. runtime/trace The runtime/trace package is a true execution tracer that is capable of capturing even more detailed information than fgtrace. However, it's mostly designed to understand the decisions made by the Go scheduler. So the default timeline is focused on how goroutines are scheduled onto the CPU (processors). This means only the sha1.Sum operation stands out in green, and full stack traces can only be seen by clicking on the individual scheduler activities. [runtime-ex] The goroutine analysis view offers a more useful breakdown. Here we can see that our goroutine is spending 271ms in Execution on CPU, but it's not clear from this view alone that this is the sha1.Sum operation. Our networking activity (http.Get and io.Copy) gets grouped into Sync block rather than Network wait because the networking is done through channels via other goroutines. And our time.Sleep activity is shown as a grey component of the bar diagram, but not explicitly listed in the table. So while a lot of information is available here, it's difficult to interpret for casual users. [runtime-ex] Last but not least it's possible to click on the goroutine id in the view above in order to see a timeline for the individual goroutine, as well as the other goroutines it is communicating with. However, the view is also CPU-centric, so remains difficult to understand the sleep and networking operations of our program. [runtime-ex] That being said, some of the limitations of runtime/trace could probably be resolved with changes to the UI or converting the traces into a format that Perfetto UI can understand which might be a fun project for another time. How it Works The current implementation of fgtrace is incredibly hacky. It calls runtime.Stack() on a regular frequency (default 100 Hz) to capture textual stack traces of all goroutines and parses them using the gostackparse package. Each call to runtime.Stack() is a blocking stop-the-world operation, so it scales very poorly to programs using ten thousand or more goroutines. After the data is captured, it is converted into the Trace Event Format which is one of the data formats understood by Perfetto UI. The Future fgtrace is mostly a "Do Things that Don't Scale" kind of project. If enough people like it, it will motivate me and perhaps others to invest into putting it on a solid technical foundation. The Go team has previously declined the idea of adding wallclock profiling capabilities similar to fgprof (which is similar to fgtrace) to the Go project and is more likely to invest in runtime/ trace going forward. That being said, I still think fgtrace can help by: 1. Showing the usefulness of stack-trace/wallclock focused timeline views in addition to the CPU-centric views used by runtime/trace to perhaps implement the future developement of the runtime tracer. 2. Starting a conversation (link to GH issue will follow ...) to offer more powerful goroutine profiling APIs to allow user-space tooling like this to thrive without having to hack around the existing APIs while reducing their overhead. License fgtrace is licensed under the MIT License. About fgtrace is an experimental profiler/tracer that is capturing wallclock timelines for each goroutine. It's very similar to the Chrome profiler. Resources Readme License MIT license Stars 369 stars Watchers 4 watching Forks 2 forks Releases 1 v0.1.0: Initial Release Latest Sep 18, 2022 Packages 0 No packages published Contributors 2 * @felixge felixge Felix Geisendorfer * @carloscarnero carloscarnero Carlos Alberto Carnero Delgado Languages * Go 100.0% Footer (c) 2022 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.