[HN Gopher] Show HN: Encore - Go framework for building distribu...
___________________________________________________________________
Show HN: Encore - Go framework for building distributed systems
Author : eandre
Score : 152 points
Date : 2021-04-14 13:30 UTC (9 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| ceesharp wrote:
| Is this something like https://dapr.io/?
| rusht wrote:
| I'd be interested in OP's answer here as well.
|
| I know dapr injects a sidecar so you can code in any language
| and then use their SDK to call other microservices. Encore
| looks to be Go centric and uses code generation to create
| actual Go functions, which provides much better IDE support.
|
| I did not see this in the docs but I'm not sure if Encore
| supports retries with or without exponential backoffs; dapr
| supports both.
| domano wrote:
| Is it possible to integrate this with event based systems or
| message queues or is encore not made to be integrated into
| existing service landscapes anyways?
| eandre wrote:
| Since Encore supports deploying to your own cloud, it's
| definitely possible to integrate with existing systems you
| already have in place.
|
| We're also working on expanding the set of infrastructure
| primitives supported, so that you get the Encore experience for
| doing more types of things. Things like Pub/Sub and message
| queues are the next things we'd like to add support for.
| nautilus12 wrote:
| It's just rails built in go. Why do we keep doing the same thing
| over and over again but with our favorite language? It's as
| though we are just doing these things to be the first one there
| and build a new empire around ourselves. It's like digital
| colonialism...did we learn anything from the old world before
| forging ahead into a new one?
| eandre wrote:
| It might be like rails in the sense of prioritizing the
| developer experience, but the goals are quite different. Rails
| is a web framework designed to build websites. Encore is a
| backend framework designed to build distributed systems. In
| practice that means the feature sets are extremely different,
| with very little overlap.
| void_mint wrote:
| It looks like the secret management is just pulling env variables
| into a map? Maybe it's a language thing but this isn't really how
| I interpret people describing the need for "secrets management",
| which is more to do with secure storage and access control. A map
| of key/values pulled from the environment doesn't really handle
| any of that, right?
| eandre wrote:
| No, we do support storing secrets and securely delivering them
| to the running application. See `encore secret set --help`.
| hnarn wrote:
| I'm sure this is awesome but do we really need phrases like
| "state of the art developer experience" giving you "unmatched
| productivity"? Is this the new normal? Do people feel the need to
| add this type of crazy hyperbole to stand out? I'm sure most
| people who read this are technical enough to appreciate this
| framework for what it can achieve technically, there's no need
| for an open source framework presentation to use the same
| language as a brochure for selling a sports car.
| thwarted wrote:
| This has escalated from the claims of being "simple" and "fun
| to use", and a bunch of other highly subjective, and
| distracting, terms.
| ggregoire wrote:
| "a blazing fast library [rocket emoji] [fire emoji]"
| dhagz wrote:
| This looks neat, but man. I'm not a fan of code generation magic.
| It's part of the reason I moved away from the Rails ecosystem -
| even though I knew what Rails was doing under the hood, the magic
| started to give me the heebie-jeebies. Maybe I'm weird, but I'd
| rather spend 10-15 minutes writing some boilerplate for a service
| and control the code than trust a third-party tool to do it for
| me.
|
| I also feel like this is doing too much to be a framework, in a
| way. Like, if it just did the code generation, cool. But add in
| the secret management, cloud deployment and migration stuff, and
| it starts to feel like a platform - which, coincidentally, it is.
| With pricing and everything. I can't use this to build a service
| that runs on AWS/GCP/my own servers - at least it doesn't look
| like it from first glace.
|
| So if this were actually just a framework that could be separated
| from the platform stuff, I'd be willing to try it out. But as is,
| I'd rather write boilerplate and be able to 'go run' my service
| and deploy it wherever I want.
|
| EDIT: Just found that I could deploy to another cloud provider,
| which is nice. I wish I didn't have to manage that via a web
| interface, and that it supported just building a container I
| could deploy anywhere.
| dastx wrote:
| One thing I absolute hate about magic in code is that it's
| often done at runtime. Considering the idiomatic way for go is
| actually have code generated and checked into git makes a huge
| difference.
| freedomben wrote:
| I agree regarding magic, but I've struggled to find the balance
| because if you have copy/pasted boilerplate, it's much harder
| to upgrade major versions because you have a lot of stuff to
| update/review. If you've modified the boilerplate, it may be
| impossible.
|
| How do you balance that?
| itake wrote:
| The language should have more features and less boiler plate.
| In Golang code, there is first class code generation support.
|
| Compared to Ruby or Python, there is minimal code generation
| outside of the initial framework setups.
| eandre wrote:
| Thanks for the feedback. I think there are many different types
| of magic. In my (biased) opinion I think the code generation we
| do is much more on the side of "how to do this is obvious but
| annoying" rather than on the "I don't know what's happening"
| side.
|
| As for platform vs framework, my realization is that the
| traditional separation of frameworks/libraries from
| infrastructure means that it's almost impossible to improve
| many parts of the developer experience.
|
| The annoying parts of building backends and distributed systems
| come from the integration between the two. Setting up
| databases, secrets, and so on all require bridging that gap.
| Another big area that we're investing in is providing insights
| about how your application works at runtime, in production,
| directly at your fingertips when you're writing code. That's
| also not something you can do without integrating across that
| gap.
|
| As you pointed out in the edit, you can absolutely use Encore
| with your own cloud. We're working on making the deployment
| options much more flexible, but it's very much not another
| "let's make an infrastructure tool and then charge a premium on
| top of your cloud bill to manage it for you".
|
| Our business model is much more about aligning the incentives
| with the developers. We want to provide the best developer
| experience out there, and by charging for _that_ as opposed to
| for infrastructure, we 'll happily invest time to reduce our
| users' and customers' cloud bills, rather than just selling
| them more servers they don't really need.
| dhagz wrote:
| I will say I do like the focus on telemetry and tracing -
| honestly that's the thing I have the hardest time setting up
| in applications, distributed or otherwise.
| AtNightWeCode wrote:
| Platforms are typically garbage. This is garbage.
|
| Love this quote by the way.
|
| "With Encore you write your apps using Go, a modern programming
| language developed by Google to make backend development simpler.
| Rapidly growing and designed to be easy to learn, Go is the
| language of the cloud."
|
| I think the irony of Go is that it is easier to "script" things
| in Go than in NodeJS for instance. This makes it the right tool
| for infrastructure programming. I always use Go for these kind of
| things.
|
| Writing enterprise apps in Go is a bad idea with or without
| Encore. One lib can't simply make up for the large ecosystem in
| the more modern technologies.
| eandre wrote:
| I've been working on Encore for the past 3 years.
|
| I was at Spotify for the past 8 years as a Staff Software
| Engineer. We grew frustrated with the disproportionally large
| effort needed to build even simple backend applications, and
| created Encore to solve this problem.
|
| Encore uses static analysis and code generation to reduce the
| boilerplate you have to write, resulting in an extremely
| productive developer experience. It's a combination of a
| framework that combines with a cloud platform in order to
| simplify all aspects of building distributed systems, from how
| you write code to shipping it to production.
|
| I shared some more background on Encore here [1]. Feedback very
| much appreciated :)
|
| [1] https://twitter.com/_eandre/status/1381668476662734849
| Philip-J-Fry wrote:
| I've built my fair share of Go microservices and have never
| felt the need to develop an entire framework like this.
|
| We have everything like CI/CD, distributed tracing and one line
| project generation.
|
| It's just based on some foundation code and some simple bash
| scripts to find and replace some service specific things.
|
| And then it's just the basic routing and HTTP handlers to
| unmarshal and marshal responses.
|
| When all your services are built off the same foundation then
| creating a new one is pretty much and copy and paste job in my
| experience.
|
| It's a cool project but it seems like unnecessary complexity,
| something I strive to avoid.
| ashtar wrote:
| Curious - how do you end up handling transactions that span
| services?
|
| I'm getting into go, but there doesn't seem to be a great
| pattern for this. Used to @Transactional on the service, so
| feels annoying in having to build that.
| mrath wrote:
| How does @Transactional span services? With a
| JtaTransaction manager?
|
| If you are saying about micro services here which talk
| grpc/http then @Transactional does not help.
|
| If you are talking about inside one service then you have
| write all that boilerplates, and it will not look as simple
| as Java
| eandre wrote:
| If that works for you, all the better! There are quite a few
| things that Encore does beyond what you mentioned:
|
| - Automatically sets up dedicated Preview Environments for
| your Pull Requests
|
| - Provisions all the infrastructure you need, both for local
| development as well as in production (and other environments
| like staging or testing)
|
| - Provides auto-completion and compile-time type safety for
| making cross-service API calls, with a single line of code
|
| - Integrates secrets management, API documentation, frontend
| client generation, etc :)
| marcus_holmes wrote:
| > it seems like unnecessary complexity, something I strive to
| avoid.
|
| This. Adding complexity to reduce boilerplate seems like the
| wrong direction to go in (pun intended).
|
| I like boilerplate. I can overwrite it with special cases
| whenever I need to, and it takes up very little mental space
| (as opposed to the rules needed to generate it).
|
| I hated working in Rails because of all the magic. I love
| working in Go because of the lack of magic.
|
| But y'know. I'm sure it'll suit someone. Good luck with it
| OP.
| kbar13 wrote:
| isnt code generation just smarter copypasta? since there will
| be no difference in complexity during runtime i feel like
| it's equivalent, just maybe more powerful
|
| what i mean is that you have complexity in your existing
| setup too... its just that you understand the ins and outs.
| and so the same applies to encore.
| outworlder wrote:
| No it isn't.
|
| If you did down enough, you'll find some form of code
| generation in your environment of choice. If nothing else,
| compilers, but it's not limited to them.
|
| The largest difference between that and copypasta is -
| generated code can be easily updated, by regenerating it.
|
| It's also an implementation detail. I've read elsewhere in
| this thread that code generation is bad and language
| supported features are good. I'm not sure it's so clear
| cut. It shouldn't matter if the generated code is mostly
| invisible (when it's baked inside your binary by the
| compiler, due to language support), versus code that is
| generated in the source language and then compiled.
|
| One thing that's always bad is mixing generated and non-
| generated code. You have all the disadvantages and little
| benefit.
| Philip-J-Fry wrote:
| I don't think it is. It's fundamentally taking the code out
| of your control. With my services all the code is there, we
| can tweak it if necessary, we're not tied to a framework,
| it's just a barebones foundation of a Go application.
|
| Nothing is hidden away from you with code generation.
| There's no in-between steps from what you write to what you
| execute other than Go compilation.
|
| I think it's the wrong mentality to require a whole new set
| of third-party tools to maintain a Go application. It then
| starts feeling like you're writing Javascript, and does
| anyone really want that?
| skrtskrt wrote:
| What is your setup for distributed tracing?
| wdb wrote:
| Personally, I am using Opentelemetry to create and export
| my distributed traces. At first sight it looks like they
| have reinvented the wheel for this project.
| Philip-J-Fry wrote:
| Elastic APM. As far as I am aware it's an implementation of
| the OpenTracing specification.
|
| We just use the Elastic client libraries. Tell it where our
| APM server is, do the little bit of application set up and
| then view all our application traces in Kibana.
|
| All applications follow the same basic setup. Probably less
| than 20 lines of code and then wrapping some HTTP handlers
| and HTTP clients. Then we have some application specific
| spans.
| wdb wrote:
| Any plans to switch to Opentelemetry?
| Philip-J-Fry wrote:
| Not that I know of
| nvarsj wrote:
| Hi Andre. Encore looks really cool! It must have taken a huge
| amount of work to put this together. I think streamlining from
| IDE to production is the future of tooling and frameworks, and
| it looks like you are going in that direction. I feel like that
| is the entire hype behind "serverless", but no one has really
| been able to pull it off yet imo.
|
| I wonder what you think about K8s integration? Spotify is all
| K8s right? I'm guessing that is tougher to monetize. I tried at
| a prior startup to work on something like this, offering a PaaS
| like experience on K8s with development libraries (it's
| actually still in progress - check out https://cloudstate.io/).
| It is a large effort to build such a thing in a nice way.
|
| Anyways, I'm looking forward to seeing how Encore develops.
| Good luck!
| datavinci wrote:
| Hey thanks a lot for sharing this, Could you recommend a couple
| of resources for someone to get into distributed systems? I
| read Data Intensive Applications, and have been finding such
| sources since.
| eandre wrote:
| A lot of the seminal research papers are surprisingly
| readable! I really recommend reading them to get a good feel
| for the sort of problems you face at scale and good ways of
| handling them.
|
| Off the top of my head: a lot of stuff from Google (Dapper,
| Stubby, Zanzibar). Dynamo from Amazon. Leslie Lamport's guide
| on TLA+. The two generals problem, distributed consensus, and
| so on.
|
| Most distributed systems don't need all that fancy stuff, but
| it's useful to know regardless. Even the simplest systems
| will get weird and interesting bottlenecks with scale.
|
| And then a lot of just doing it in practice! Find a job at a
| startup with some scale, and learn on the job.
| latch wrote:
| Did you look at whether Go was part of the problem or were you
| using something else and you found that Go was part of the
| solution?
| bketelsen wrote:
| We built a conference management system using an early release of
| Encore this past winter, and it was an amazing experience. I can
| absolutely recommend you check it out if you're building services
| in Go. The developer experience is so WELL thought out, it's a
| pleasure to use it.
| throwaway894345 wrote:
| I'd love to hear more about this. Did you deploy it to the
| encore managed environment, or private cloud? Which "escape
| hatches" (e.g., dropping down into plain http requests) did you
| end up using and where did you encounter friction?
| lux wrote:
| This looks pretty awesome!
|
| I'd love to see an example using websockets. I've used Go for
| websockets in the past but find the boilerplate-to-custom code
| ratio isn't great.
|
| Knowing Go has a solid webrtc library
| (https://github.com/pion/webrtc), another cool feature would be
| webrtc support. Tons you can build with that!
| eandre wrote:
| Thanks! Encore does support dropping down to plain HTTP
| requests which lets you use Websockets, see [1]. We hope to
| make it more ergonomic and less boilerplate-y in the future,
| but it is supported :)
|
| [1] https://github.com/encoredev/examples/tree/main/websocket-
| ec...
| lux wrote:
| Good to know! Thanks for the example. Looking forward to
| giving this a spin!
| ideas-are-cheap wrote:
| Looks quite interesting to me. The telemetry and tracing aspects
| are cool I think. That sort of thing is a 'must have' really but
| I find it is quite a pain to manage the tools for it yourself,
| especially on a small team.
|
| You mentioned on another comment that you would like to introduce
| a pub/sub element to this. Could you talk a bit more about that?
| I can see how that work for communicating between services
| managed by Encore but could it be integrated with existing event
| sources?
|
| In general, I guess it's hard to see how some of this would
| integrate with existing systems nicely. If you're starting from a
| blank slate I'm sure the devex is excellent, but what would the
| workflow be like if only a small percentage of my infrastructure
| was done this way? (Which is how I imagine all of your customers
| would start out).
| gigatexal wrote:
| I kinda like this a lot. Raw SQL in the examples. And generated
| and consumable api services. I will surely play with this later.
| domano wrote:
| I have a very minor issue with it: The API endpoints naming
| scheme. /greet.Person seems quite unusual - wouldn't something
| like /greet/person/abc be better?
| eandre wrote:
| It's like gRPC in the sense that it's endpoint-centric rather
| than URL-centric.
| domano wrote:
| At least for a public facing API i would like some more
| customization possibilities, but this can maybe be done with
| a seperate BFF / API Gateway / GraphQL gateway etc. Thanks
| for the answer.
| eandre wrote:
| Fair enough :) We're working on adding GraphQL support as
| an alternative approach to exposing your API, and what you
| suggested wouldn't be that difficult to add either.
| tyingq wrote:
| The page didn't mention retries, exponential back off, etc. Is
| there somewhere where it shows how that sort of thing is handled?
| neeeeees wrote:
| Looks pretty cool! Couple of questions:
|
| - Is there a story behind the name?
|
| - Out of curiosity, does "Encore Cloud" also run on AWS?
|
| - I think a lot of developers have a mentality of "rather write
| boilerplate than be tied to a platform not endorsed by
| $bigCompany". Do you have a strategy to mitigate this?
| eandre wrote:
| Good questions! "Encore Cloud" is currently deployed to AWS and
| DigitalOcean under the hood.
|
| As for being tied to us, that's a big part of why we're open
| sourcing it. We see that Encore can rather reduce lock-in, as
| the code you end up writing with Encore is very cloud-agnostic,
| but also quite agnostic of Encore itself (by virtue of using
| static analysis, you end up writing mostly regular business
| logic).
|
| And when you use it you can deploy the applications to your own
| account at any major cloud provider, so you can use your
| existing trust relationships with your cloud provider of
| choice.
|
| As for the name, it's a bit of a wordplay :). "en-" for "to
| bring about [something]", and "core" for "the part of something
| that is central to its existence or character" (in our case
| backend development).
| cultofmetatron wrote:
| Nice to see elixir get a but of competition in this space.
|
| They are fighting an uphill battle to beat the OTP + lib-cluster
| + horde combo that elixir developers have but that autogenerated
| api stuff looks neat.
| wskinner wrote:
| Looks interesting. How do the infrastructure layer features
| (deployment, cloud resource management) compare to the open
| source alternatives? If I am already using Kubernetes and
| terraform, would it still make sense for me to build applications
| on Encore?
| eandre wrote:
| Currently Encore deploys to Kubernetes, and uses Terraform
| under the hood to provision the infrastructure. Over time we'll
| add much more flexibility in how you deploy the application and
| the infrastructure we support.
|
| It's not another infrastructure offering but rather a new
| developer experience for building backend applications and
| distributed systems. The idea is to get the developer feedback
| loop to be about building your product and writing business
| logic.
|
| Encore then uses static analysis to understand what your
| application does, the different backend services involved, and
| the infrastructure you need. It can then provision all the
| infrastructure, servers, databases, and hook everything
| together.
| itake wrote:
| Any ideas how this compares with [go-kit](https://github.com/go-
| kit/kit)?
|
| I like how go-kit's transport not just JSON and HTTP (e.g. gRPC
| and protobufs are supported).
___________________________________________________________________
(page generated 2021-04-14 23:01 UTC)