[HN Gopher] Tokio Console
___________________________________________________________________
Tokio Console
Author : hasheddan
Score : 444 points
Date : 2021-12-17 16:28 UTC (6 hours ago)
(HTM) web link (tokio.rs)
(TXT) w3m dump (tokio.rs)
| dthul wrote:
| Can't wait to try this out! I always wanted the ability to see
| the tasks that are currently running or waiting.
|
| In the screenshots I can see that tasks have descriptive names.
| Does anybody know how to set the name for a task? tokio::spawn
| doesn't take a name parameter. Does it require `tracing`?
| lkt wrote:
| Looks like they have added a builder for tokio::task[1] that
| will allow you to set a name. It's unstable at the moment so
| you would need to set the tokio_unstable cfg flag.
|
| [1]: https://github.com/tokio-
| rs/tokio/blob/master/tokio/src/task...
| carllerche wrote:
| Exactly, this is the first release of Tokio Console. We will
| keep adding functionality over time, which will help inform
| the APIs in Tokio to better provide the necessary
| instrumentation.
|
| I expect the new APIs like the task builder will stabilize
| (no longer require the `tokio_unstable` flag) over the course
| of 2022.
| teknopurge wrote:
| this is great. thank you.
| 7357 wrote:
| 1st time i laughed reading a readme for such a brandifious
| project.
| phab wrote:
| What does brandifious mean? A google search doesn't return
| anything.
| xrisk wrote:
| Has anybody done this for Python?
| alexchamberlain wrote:
| trio has a similar utility I believe
| stjohnswarts wrote:
| Oh I've been waiting for this for sooooo long. Thank you tokio
| team. I think this is going to make my life a lot easier in the
| debugging sphere.
| nih0 wrote:
| very cool, reminds me a bit of jmx which I miss alot back from my
| java days.
| marmada wrote:
| this is very exciting b/c it feels like one of the 1st language
| features centered around runtime debugging.
|
| Most languages don't seem to do a lot for run-time debugging.
| Being able to `gdb` and step-through on a local binary is a far-
| cry from detailed metrics / visualizations. We end up resorting
| to stuff like Honeycomb, but I'm waiting for the days of a
| programming language built for the ground-up for runtime
| debugging.
|
| Anyways, this feels like an important step in the right direction
| and I'm excited to try this out
| ReactiveJelly wrote:
| I've been waiting years for some cooperation between languages
| and editors to achieve things Bret Victor wanted years ago,
| like "Just change this constant while the program is running",
| where you can have very tight iteration loops for things that
| don't need a full recompile.
|
| I mean, Visual Studio has some kind of hot reloading, but I'm
| not gonna VS, I want mainstream support for this fast iteration
| and deep runtime debugging.
| echelon wrote:
| The Rust ecosystem is on _fire_!
|
| This is such amazing tooling. There's so much best in class
| engineering going on with this language, Tokio/async runtimes,
| graphics, web libraries, etc.
| mycoliza wrote:
| Thank you, that's really nice to hear!
| Thaxll wrote:
| I'm not sure about that, I don't use Rust but I don't see how it
| could be relevant because its real time, what you want is to dump
| things and analyze later, if you have to stay in front of the
| console to see what's happening it's not very useful. Is it
| currently possible to dump the trace and analyze it later with
| that console? ( like Go or Java does with JFR ).
|
| The idea is great it's the way to consume the information that I
| think misses the point.
| carllerche wrote:
| Dumping to a file for later analysis is on the roadmap.
| pcwalton wrote:
| It's helpful if you're, for example, running unit or
| integration tests and see stuck threads.
| pastrami_panda wrote:
| I only read the first paragraph but it sounds like it addresses
| this?
|
| > It gives you a live, easy-to-navigate view into the program's
| tasks and resources, summarizing both their current status and
| their historical behavior.
| pjscott wrote:
| Often you can find a misbehaving process that somehow got into
| a weird state, one that you weren't tracing beforehand because
| you weren't expecting trouble and didn't want the overhead, and
| in those cases it can be tremendously useful to be able to
| attach some inspection tool to figure out what's going on so
| you can fix the underlying bug.
| xtagon wrote:
| Very cool. Reminds me of Erlang Observer.
| javajosh wrote:
| Praise, 2 suggestions and a question.
|
| Color coding OOM (order-of-magnitude) is smart, but the colors
| chosen seem quite hard to distinguish. Consider picking different
| default colors; also consider adding a redundant representation
| of scale, like an integer representing OOM scale.
|
| It is strange that a thoughtful design around OOM would also
| choose keep so many digits. If the goal is to summarize, then
| please throw those extra digits away. (e.g. 10.3992s is
| specifying the time down to the millisecond, but those extra ms
| are not meaningful at the second time scale.)
|
| What is this about the runtime polling tasks? Does that happen in
| Tokio? Why? I am only familiar with 2 async runtimes (node,
| vert.x) and I was under the impression that neither of these
| "poll their threads" in any meaningful way. Threads (or process
| or fiber or whatever you want to call it in this context) never
| initiate action on their own. They are ALWAYS waiting for
| something to happen to them, and that includes timeouts, which
| would be triggered by passing a thread a clock tick. The
| runtime's job is to centrally manage resources that must persist
| between thread behavior, so at most it is going to be polling
| external resources, and not it's own threads.
|
| Or maybe I don't understand what polling means here? I would
| interpret it as meaning "keep checking a well-known place for
| changes, and then do something if a change is found". But since
| async threads can't initiate any change on their own, this is
| nonsensical.
| carllerche wrote:
| I'll answer the polling question. The Tokio runtime (and async
| rust in general) works a bit differently than other async
| runtimes like node. With node, callbacks are provided and
| executed when an OS event is received. With Tokio, there are no
| callbacks. Instead, async logic is organized in terms of tasks
| (kind of like async green threads). When the task is blocked on
| external events, it goes into a waiting state. When external OS
| events are received, the task is scheduled by the runtime and
| eventually the runtime "polls" it. Because the poll happens in
| response to an OS event, most times, the poll results in the
| task making progress. Sometimes there are false positive polls.
|
| This page goes into a bit more depth and shows an example of
| how one would implement a (very simple) runtime/executor:
| https://tokio.rs/tokio/tutorial/async
| Hercuros wrote:
| I sometimes wonder what is the fundamental distinction
| between a callback-based API and this wakeup-based task API.
| I guess the main difference is that in a callback you
| generally provide the result as an argument whereas with a
| wakeup-based API you just wake the task and it has to look
| for the result in some stored state somewhere.
|
| But ultimately both of them take the "rest of the
| computation"/continuation and store it somewhere (i.e. on
| some sleeping task/callback list) to be awakened/invoked
| later.
| tijsvd wrote:
| The callback is really hard to implement without allocating
| memory for each wakeup. The poll mechanism can simply leave
| the task in place. I suspect the poll thing is also easier
| to generate.
| carllerche wrote:
| It is fairly subtle and _mostly_ an implementation detail.
| In Rust, the concept of "polling tasks" was very exposed
| before the async/await keywords were introduced, so the
| lingo kind of stuck. There is an argument that we should
| move away from that lingo now that it is mostly hidden as
| an implementation detail, but we haven't yet.
| mycoliza wrote:
| (primary author of the console here) you're right that there
| are too many digits of precision right now...the reason for
| that is that it's actually _not_ a thoughtful design at all,
| though I appreciate you saying that it is; I just picked an
| arbitrary number when I was writing the format string and
| didn't really think about it. We probably don't want to display
| that much precision --- for smaller units, we probably don't
| want any fractional digits, for larger units like seconds, we
| probably want two digits of precision maximum.
|
| Regarding the color scheme, glad you like the idea. Because
| it's a terminal application, the choice of the colors was
| constrained a bit by the ANSI 256 color palette
| (https://www.ditig.com/256-colors-cheat-sheet); I wanted it to
| be obviously a gradient, so I just picked colors that were
| immediately adjacent to each other in the ANSI palette. It
| might be better to pick colors that are one step apart from
| each other, instead, so they're more distinguishable
| visually...but there's kind of a balancing act between
| distinguishability and having a clear gradient. We'll keep
| playing with it!
| chc4 wrote:
| Quite a lot of terminal emulators support 24bit TrueColor
| escape codes, so it might be worth using that along with a
| color space designed for visual intensity corrolation
| (there's a lot of study in this for e.g. heatgraphs for maps)
| mycoliza wrote:
| Yeah, currently, the console knows how to _detect_
| TrueColor, but in this case, I just used the ANSI 256
| palette rather than picking a better one when TrueColor is
| available...we should probably fix that!
|
| Side note, it turns out that detecting what color palettes
| a terminal supports 24-bit colors is surprisingly fraught.
| There are a couple env variables that may be set...but not
| every terminal emulator will set them. And then you can use
| `tput`...but the terminal may not have correct data in the
| tput database. So that was fun to learn about!
| hinkley wrote:
| > we probably want two digits of precision maximum
|
| I think it depends a lot on jitter in the system. Sigfigs are
| one kind of error bar, one where I often have to haul my
| coworkers or myself out of trying to read things into the
| data that aren't there.
|
| There are times where a 10ms change in a 2 second response
| actually matter to me, because that's half a percent and not
| all improvements which are easy are also straightforward.
| Sometimes you're scrambling for 3% here and 2.2% there. But
| if the noise in the system is +-50ms then people declaring
| that they've shaved 15ms off of response time are likely
| deluding themselves and then deluding the rest of us.
|
| I know how to do some of these things by hand, I'm not sure
| how you automate them, or in the case of a dashboard, typeset
| them.
| carllerche wrote:
| Also, thanks for the thought re: digit precision. I am tracking
| it here: https://github.com/tokio-rs/console/issues/224
| aliceryhl wrote:
| Generally the way async Rust works is that you have a Future
| trait with a poll method, and if you call it the future will
| attempt to make progress if it can -- e.g. if the task is a
| timer it will check the time and complete the task if so and
| otherwise return Pending.
|
| However, async Rust includes an additional concept: Wakers.
| When your runtime (Tokio) calls poll on a future, it gives the
| future a waker object, and when the future is ready to continue
| work, something needs to call wake on the waker. Once this
| happens, Tokio will poll the task again soon, and Tokio wont
| poll tasks that have not been woken.
|
| For example, for timers, Tokio includes a timer wheel (a sort
| of priority queue) with all the registered timers, and the
| timer wheel calls wake on the appropriate waker whenever a
| timer expires. Similarly with a message passing channel, the
| sender calls wake on the receiver's waker when a message is
| sent.
| mycoliza wrote:
| Hi, I'm one of the main authors of `tokio-console`, so if folks
| have any questions, I'm happy to answer them!
| IMTDb wrote:
| Is there any question we _should_ ask ? About some challenge or
| interesting discovery / aspect that most of us wouldn't even
| realise ?
| sam0x17 wrote:
| Have you delved at all into the potential for having a true
| "rust repl"? This is something I have very much wanted for some
| time, where instead of just recompiling and re-running
| everything every time, it just compiles the new line and
| executes it in the memory context of the already running rust
| program. I'm just not enough of a low level rust hacker to get
| it working, but imagine the web frameworks and things that
| could exist with a good REPL for rapid debugging and
| prototyping in rust.
|
| I actually spent a whole summer trying to do this in Crystal
| and I was very nearly successful, however a few low level
| limitations got in my way at the end of the day. In Crystal it
| is actually possible to do this kind of REPL if you have a
| perfect ability to deep marshal/copy any object, and I almost,
| almost got that working here:
| https://github.com/sam0x17/marshal
| mycoliza wrote:
| Something like that would definitely be useful! It's not
| really in scope for this project, which is intended as a
| telemetry and diagnostics tool, but I can imagine a Rust REPL
| being useful. Of course, in order to do that, you'd need to
| implement a general-purpose Rust interpreter, which seems
| like a fairly large amount of work.
|
| In practice, I personally just use REPLs mostly for quick
| testing out of a small expression or something...and
| honestly, I usually just use the Rust playground
| (https://play.rust-lang.org/) for this. Small examples are
| compiled fast enough in the playground that it's kind of a
| REPL-like experience for testing stuff out semi-
| interactively...but it's not the same as connecting to a
| running application and running new code inside of that
| application. That's something that seems very difficult to
| add to Rust, a compiled, statically-linked language with
| limited support for hot reloading...
| carreau wrote:
| There is https://github.com/google/evcxr/blob/main/evcxr_re
| pl/README....
| AceJohnny2 wrote:
| Can tokio-console be used on Rust async programs that don't use
| the tokio runtime (assuming such arises on desktop)?
| carllerche wrote:
| Yeah, it is decoupled from Tokio. Tokio emits instrumentation
| via the `tracing` crate. Tokio Console just listens to the
| tracing events. Any runtime that emits the same events can be
| used with Tokio Console. This is the tracking issue:
| https://github.com/tokio-rs/console/issues/130
| ninkendo wrote:
| Could you please go get hired by Apple and implement this for
| Swift? Pretty please!
| staticassertion wrote:
| What's the intended workflow? Would I run a console an all of
| my rust services, and then when debugging some prod issue
| connect to it? Or would I flip a switch? Or is it more for
| CLIs?
|
| Curious to hear, in general, how it's been used.
| carllerche wrote:
| This first release is geared primarily towards local
| debugging. That said, it is designed to be able to enable /
| disable instrumentation at runtime and it will be able to
| support connecting to a process in production, enable the
| instrumentation, and debug there.
|
| Right now, we wanted to get the first release out and start
| getting people using it and collect feedback to help
| prioritize future development.
| staticassertion wrote:
| Got it, thanks. A follow up - is this work funded in any
| way?
| mycoliza wrote:
| Tokio is a non-profit, community-supported project,
| although many of us work on it as part of our day jobs.
| If you want to support Tokio development, you can
| contribute to it on GitHub Sponsors
| (https://github.com/sponsors/tokio-rs) and on
| OpenCollective (https://opencollective.com/tokio).
|
| I've also recently started accepting donations on my
| personal GitHub Sponsors page
| (https://github.com/sponsors/hawkw) if you're interested
| in supporting my open-source work in particular.
| staticassertion wrote:
| Perfect! Thanks.
| kevinmgranger wrote:
| `valuable`[1] was initially written to support tracing, but I
| see this 0.1 release doesn't use it. Is valuable seen as more
| of an add-on to this approach, rather than core to it?
|
| [1]: https://tokio.rs/blog/2021-05-valuable
| carllerche wrote:
| The current release of `tracing` includes the predecessor of
| valuable (https://docs.rs/tracing-
| core/latest/tracing_core/span/struct...). Valuable extracts
| this functionality and improves on it, but hasn't quite made
| its way back into tracing yet. I expect that it will be
| included in upcoming releases (I know eliza has been poking
| me to release valuable and get it in tracing, I probably
| should get on that!)
| mycoliza wrote:
| Yeah, the goal is for `valuable` to replace `tracing`'s
| (currently much more limited) `Value` trait entirely, when
| we release `tracing` 0.2. Before making a breaking change,
| though, we want to release opt-in support for `valuable` in
| the current v0.1.x `tracing` ecosystem, so people can start
| trying it out and we can figure out if there's anything
| missing.
|
| You can follow the progress of that here:
| https://github.com/tokio-rs/tracing/pull/1608
|
| I believe it's currently just waiting for a crates.io
| release of `valuable`!
| kevinmgranger wrote:
| I stopped following development of `valuable` since I was
| too busy with non-rust-stuff, but weren't there still
| some big API-shaping questions open for it?
| carllerche wrote:
| Not too many now. What it really needs is real usage to
| find any potential limitations.
| nicoburns wrote:
| This looks great! I wonder how hard it would be to add
| support for this to debuggers like lldb. I believe they
| currently lack a way to inspect rust objects.
| seanp2k2 wrote:
| Is there something like this for Golang? I've inherited a golang
| project at work and it's really not fun to debug compared to
| Python.
| ReactiveJelly wrote:
| Is there even something like this for Python?
|
| I've used Tracy for C++, which does similar tracing on OS-level
| threads, but I don't know much about Python.
| mycoliza wrote:
| Go has pprof (https://github.com/google/pprof), which I've
| heard good things about --- and, the pprof data model was one
| of the influences I looked at when designing the Tokio
| console's wire format. But, I'm not sure if pprof has any
| similar UIs to the one we've implemented for the Tokio console;
| and I haven't actually _used_ it all that much.
| artursapek wrote:
| This is awesome
| iillexial wrote:
| Is there sth like this for Go?
| mjibson wrote:
| Having programmed in both Rust and Go: I mostly didn't need
| what this does in Go because my problem was usually solved by
| panic'ing, which prints a stack trace of each go routine,
| allowing me to get enough of the way through figuring out my
| current problem (usually something was stuck waiting or
| sending). In tokio, there's no such print-stack behavior, so
| it's much harder to get a snapshot of what's going on. (I'm
| relatively new to tokio and Rust, so there's perhaps a handy
| "print all the tokio tasks and their stack traces" method, but
| I haven't come across it yet.)
|
| If folks use this console thing for perf reasons and not debug
| reasons, then yeah, maybe cool to have in Go.
| hcarvalhoalves wrote:
| Very interesting! I can't help but wonder why these technologies
| (the framework, and accompanying observability) aren't an OS-
| level feature; wondering if at this point we need an OS at all,
| and why not ship a minimal kernel + the binary of an application
| built w/ this to run directly on the VM.
| pcwalton wrote:
| In Linux there's work being done on a language-agnostic
| lightweight threading model, User-Mode Concurrency Groups or
| UMCG: https://lwn.net/Articles/863386/
|
| One could imagine a similar, runtime-independent console for
| UMCG. Note, however, that the programming model for such a
| runtime would be much more similar to 1:1 threading (i.e.
| blocking I/O with threads) than async/await.
| xtagon wrote:
| It's been done, check out https://www.nerves-project.org/ for
| an example of running a minimal kernel + Elixir (BEAM VM) on
| devices. As I understand it, it's powered by Buildroot and
| should be possible to do for other VMs/languages. Don't think
| I've seen anything like this for desktop OS's though!
| kristjansson wrote:
| Unikernels are a thing e.g. https://mirage.io. AFIACT uptake
| has been ... slow. You give up a lot to erase most or all of
| your OS, and only the most performance sensitive applications
| realize a benefit
| ReactiveJelly wrote:
| They're not OS-level because there isn't wide consensus on the
| right way to do things yet.
|
| Rust, C#, and JS all have similar concepts of async, but
| they're all slightly different. None of them would be trivial
| to adapt to other system langs like C and C++ - Rust requires
| compiler support to take apart async functions and put them
| back together as state machine, and the others lean on their
| GC. (And also compiler support, IIRC) I think there is a
| proposal to add coroutines in the new C++ standard, but I'm not
| sure how it would be done in the kernel.
|
| And sometimes I see people saying, "async is very bad, just use
| coroutines." Having only used Lua coroutines, I don't
| understand what the big difference is supposed to be.
|
| But mostly, these runtimes don't need OS-level support. Async
| is sort of a way to do concurrency without a kernel-level
| context switch for every task switch, right? If it's working so
| well in-process, why involve the OS at all?
|
| > wondering if at this point we need an OS at all
|
| Depends what you mean by OS. If you deploy in a container, of
| course your OS shares its kernel with the host. But for some
| user stories, (glares at Android) "OS" means all the software,
| including a Blink-based web browser, a plethora of GUI
| programs, and other things that I would rather call a "desktop
| environment" than an OS.
| billfruit wrote:
| Yes truly, is there something like this for linux threads and
| processes?
| pkaye wrote:
| The Linux Perf tool is pretty powerful. I've barely just
| scratched the surface myself.
|
| https://perf.wiki.kernel.org/index.php/Main_Page
|
| As as a side note, Go and pretty good profiling tools. You
| can see a trace of goroutine execution and why a goroutine
| got scheduled out and when GC kicks in.
|
| https://about.sourcegraph.com/go/an-introduction-to-go-
| tool-...
| pjmlp wrote:
| With languages with rich runtimes that is exactly the point.
|
| Actually in a way, POSIX is the missing C's runtime that wasn't
| made part of ISO C.
| tpmx wrote:
| I think a submission title regarding something specifically about
| Rust development should contain the word Rust.
| dnautics wrote:
| i think most people know tokio is rust (I'm not a rust dev)
| [deleted]
| tpmx wrote:
| It's the german word for Tokyo. There is little reason for a
| non-rust dev to know this other meaning.
|
| I hate these ambigious HN _vaguebait_ headlines and you
| should too.
| awestroke wrote:
| There is little reason for a non-german to know this other
| meaning
| [deleted]
| andrewshadura wrote:
| Actually, that's how Tokyo is spelled in many languages
| besides German.
| kristianpaul wrote:
| Is this the beginning of a new way of deploying applications?
| outside1234 wrote:
| Thank you so much for building the things in the Tokio ecosystem
| including Console.
|
| So much great stuff!
___________________________________________________________________
(page generated 2021-12-17 23:00 UTC)