[HN Gopher] Show HN: Benthos Studio - A modern take on Yahoo Pipes
___________________________________________________________________
Show HN: Benthos Studio - A modern take on Yahoo Pipes
Benthos Studio lets you plug and play various components to build a
Data Streaming pipeline through a graphic interface. It also
allows you to mock inputs to emit dummy data and run the rest of
the pipeline to inspect the output of each step. The project is
running https://www.benthos.dev/ under the hood.
Author : mihaitodor
Score : 65 points
Date : 2022-11-23 17:16 UTC (5 hours ago)
(HTM) web link (studio.benthos.dev)
(TXT) w3m dump (studio.benthos.dev)
| dchuk wrote:
| I like the playful vibe of the site, but it's incredibly unclear
| what this actually is. Needs more specific description of the
| problems being solved.
| mihaitodor wrote:
| Benthos Studio is an application that provides visual editing
| capabilities to the Benthos (https://www.benthos.dev) stream
| processor, which lets you craft and test yaml-based
| configurations that you can then run using Benthos.
|
| Benthos itself is a stateless command line (CLI) app written in
| Go. It supports quite a few types of "macro" building blocks
| (aka components) which are various flavours of inputs, outputs,
| processors, caches, rate limits, buffers, metrics, tracers and
| loggers. The most important processor is the `mapping` one
| which lets you execute Bloblang code against each message which
| passes through it. Bloblang is a functional programming
| language embedded in Benthos as a DSL for manipulating
| structured data. You can read more about it over here:
| https://www.benthos.dev/docs/guides/bloblang/about Also, if
| you'd like to use it outside of Benthos, you can import it as a
| library:
| https://pkg.go.dev/github.com/benthosdev/benthos/v4@v4.10.0/...
|
| Since I mentioned importing Bloblang as a library, you can
| import the entire Benthos framework as a library and inject
| your own custom plugins to create a custom Benthos build with
| whatever components and extra functionality you need. It's also
| a great way to slim down the existing distribution and only
| import the components that you require. See some examples here:
| https://github.com/benthosdev/benthos-plugin-example
| jeffail wrote:
| Yeah that's my bad, the site assumes you already know what
| benthos is as it's an early stages UI for it. The best place to
| start is https://www.benthos.dev, or if you like dumb videos:
| https://youtu.be/88DSzCFV4Ng
| moklick wrote:
| This looks nice! Which library are you using for rendering the
| node based UI? Or is it a custom solution? (just added benthos
| here: https://github.com/wbkd/awesome-node-based-uis)
| jeffail wrote:
| Hey! The visuals are all via https://reactflow.dev/, which has
| been pretty awesome to use.
| moklick wrote:
| I am glad to hear that :)
| NortySpock wrote:
| I've been using Benthos for some one-off bulk ETL needs and it's
| been very easy to write a config to shovel data from here-to-
| there.
|
| I'll keep an eye on Benthos Studio, though I confess I'd prefer a
| self-hostable version at some point.
| Scotrix wrote:
| Didn't look too much into Benthos but what's the difference to
| Nifi?
| mihaitodor wrote:
| Benthos is much simpler, since it's stateless and it's a
| single static binary written in Go. I don't really know much
| about NiFi, but if you need to use a messages bus with
| Benthos, such as Kafka, you can. However, you don't have to.
| throwthere wrote:
| Benthos looks like a cool project but thanks for turning me
| on to Nifi. Nifi has a lot more processors out of the box.
|
| Edit: I'm speechless but see below
| mihaitodor wrote:
| Which ones would you need? Happy to add more to Benthos.
| Feel free to open issues here:
| https://github.com/benthosdev/benthos/issues
| throwthere wrote:
| Google Sheets, Drive and Slack. I'm not comfortable
| asking for it because I can't contribute.
| mihaitodor wrote:
| OK, that's not a problem. There are a bunch of other
| channels that you can use to propose new features. Feel
| free to reach out via https://www.benthos.dev/community
|
| LE: I took note of those 3 and they should be quite
| straightforward to add. Thanks!
| NortySpock wrote:
| I don't know much about Nifi, but from re-reviewing the
| documentation, it looks like Nifi is a stateful application
| with orchestration and multi-user access, etc.
|
| Benthos requires something else handle orchestration and
| source control and multi-user access and the conditions that
| trigger an ETL run.
|
| On the plus side, the stateless nature of Benthos means there
| isn't any big setup process or cluster of containers to login
| to or anything. You just call benthos from the cli and pass
| in a config file.
|
| Last week I needed to get 120k rows of data from an Oracle
| view that would crash on a few bad rows, and shove rows that
| worked into a SQL Server table.
|
| Writing a Benthos config to tick through every id, fetch the
| relevant rows one at a time, annotate with batch information,
| and drop error rows and error messages in a log and healthy
| rows into SQL Server was, all told, about 120 lines of YAML.
| Roughly 3 hours to write while consulting the extensive
| documentation, and then trying performance tweaks until I
| could saturate my network connection (increase threads to
| 256) only took another 30 minutes.
|
| Tweaking that YAML to cover a similar second scenario as a
| new config then was only another 15 minutes.
|
| The week before that, I was trying Benthos at home to stress
| test MQTT on a Raspberry Pi with synthetic data to see which
| messages got dropped or mis-ordered.
|
| (Depends on your QoS and in-flight settings, obviously)
|
| I've never had "basic" performance testing be so simple.
|
| If you are stuck on a Windows platform, it even works there,
| which is nice for one-off runs for dev work, troubleshooting,
| or break-fix work.
| mihaitodor wrote:
| Really glad to hear the Oracle driver I added to it is
| proving useful! <3
| NortySpock wrote:
| Let me tell you Benthos is a real breath of fresh air
| after fighting a few different ETL tools that are more
| GUI, chrome and administrivia than function.
|
| Thanks for contributing to Benthos!
| jeffail wrote:
| Thanks for sharing, this is awesome to hear!
| rmorey wrote:
| I believe I've seen this before, but did this used to be called
| blobfish? Or am I misremembering that due to the mascot (which is
| excellent)
___________________________________________________________________
(page generated 2022-11-23 23:01 UTC)