[HN Gopher] Airbyte makes 100 alpha / beta connectors free
___________________________________________________________________
Airbyte makes 100 alpha / beta connectors free
Author : jeanlaf
Score : 73 points
Date : 2023-01-26 16:01 UTC (7 hours ago)
(HTM) web link (airbyte.com)
(TXT) w3m dump (airbyte.com)
| bnchrch wrote:
| Oh man, so happy to see this.
|
| For me right now, Airbyte is that tool I wish we had at my last
| startup.
|
| We we're pulling data from a lot of weird places (servers in the
| back of mom and pop vet clinics). This meant writing a lot of one
| off scripts to populate our databases. We learned the hard way
| about scheduling, retries, resource monitoring, error reporting
| etc..
|
| Would've loved to have someone else take care of all that for us.
|
| Anyway love letter over.
|
| I'm currently wondering if I can use this to power some of my web
| scraping scripts....
| alexmarquardt wrote:
| This article from Simon Spati shows how to do some scraping.
| https://airbyte.com/tutorials/configure-airbyte-with-python-...
| tootie wrote:
| I have recently inherited the small and unsophisticated data
| engineering practice at my company and it's been a learning
| experience. The market for tooling seems to be incredibly frothy
| right now and I'm almost at a loss to make good selections. Is
| airbyte a direct competitor to stuff like fivetran or AWS Glue?
| jeanlaf wrote:
| Yep. Airbyte is the leading open-source solution (with a Cloud
| hosted solution too). Fivetran is the leading closed-sourced
| one.
|
| Open source makes it the solution future-proof, in the sense
| that it will address your future long-tail or custom needs,
| while a closed-source solution won't which will require you to
| build/maintain connectors in-house again.
| dangwhy wrote:
| anyone use this care to comment about how this compares to
| meltano.
| leag wrote:
| Meltano also supports Airbyte connectors
| https://meltano.com/blog/meltano-community-contribution-enab...
| tayloramurphy wrote:
| We're exploring whether or not we can support Airbyte
| connectors on our upcoming Meltano Cloud offering. As it
| stands we have some community members running Airbyte
| connectors in their production environment with Meltano and
| they seem quite happy with it!
|
| As with anything there are tradeoffs though - gaining the
| ability to have a connector in a non-Python language comes
| with the overhead of running (likely) Docker-in-docker. Also,
| connectors not built on our SDK[0] are missing out on some
| nice features like batch message[1] support (for bulk
| loading) and stream maps[2] for inline data transformations.
|
| [0] https://sdk.meltano.com/ [1]
| https://sdk.meltano.com/en/latest/batch.html [2]
| https://sdk.meltano.com/en/latest/stream_maps.html
| klaussilveira wrote:
| Airbyte is a godsend for us. It works really well for most use
| cases. Unfortunately, we had to write our custom thing for a
| large table (8 billion rows) and "tricked" Airbyte into thinking
| it had done the first sync. After that, it continued to sync
| happily.
|
| Hoping that more users will bring more maturity and better
| solutions to those edge cases.
| jeanlaf wrote:
| We'll get to that maturity level! Which database was that out
| of curiosity?
| Bedon292 wrote:
| How do you know what the status of a connector is? I just see
| https://airbyte.com/connectors and have no obvious indication of
| status.
| muspimerol wrote:
| Looks like the status of each connector is shown here:
| https://docs.airbyte.com/integrations/
| swyx wrote:
| coverage on venturebeat https://venturebeat.com/data-
| infrastructure/etl-company-airb...
| swyx wrote:
| (i work at airbyte) this was announced last month at our conf to
| a very positive response (https://movedata.airbyte.com/, keynote)
|
| The context is that Airbyte is now (after pivoting 3x during YC
| https://airbyte.com/blog/how-we-pivoted-3-times-in-the-1st-m...)
| the largest/fastest growing open source community (see our github
| https://github.com/airbytehq/airbyte) of data pipeline
| connectors[0], so in a sense they have always been free if you
| are self hosting. But now _using them on Airbyte Cloud_ is going
| to be free as well aka "we will do your ELT for free no matter
| the volume as long as our connectors are not GA yet".
|
| This is a massive commitment to improve the quality of our
| connectors, which is also something we have been pushing the
| industry on: https://airbyte.com/blog/connector-release-stages :
|
| Alpha: new, basic docs, works, passes acceptance tests
|
| Beta: Alpha + at least 25 active users + >90% sync success rate +
| snapshot tests + all streams + severe issues handled + security +
| supports checkpointing + SLA on cloud
|
| GA: Beta + >99% sync success rate + more than 50 active users +
| <24 hours downtime + polished docs + performant
|
| It's been going very well; you can see how many connectors we
| promote to GA each month in our slack (https://slack.airbyte.io/)
| and changelogs, and our new lowcode CDK
| (https://www.youtube.com/watch?v=i7VSL2bDvmw) is helping new
| connectors insta-promote to beta.
|
| We hope to set the new standard in data integration and this is
| still only day 1.
|
| [0]: good explainer on why companies are moving towards ELT in
| the first place for the uninitiated https://airbyte.com/blog/elt-
| pipeline
| sixhobbits wrote:
| I like the idea but practically I wonder how it works out. I
| feel like I would have a double disincentive to use an
| alpha/beta connection as a) it might fail and b) it might hit
| GA and then suddenly my free workflow breaks?
|
| Maybe I am thinking about it wrong and this is more aimed at
| people who were previously paying for something and now get the
| same thing for free.
| swyx wrote:
| i dont understand, why would it break when it goes from beta
| to GA?
|
| and failures happen, anyone who promises you otherwise is
| lying. you'd have them yourself if you DIY. what helps is
| having good monitoring, a large open source community with
| strong first party support, and a good development/testing
| framework so most breakages can be fixed the same day they
| happen.
| sixhobbits wrote:
| Sorry I wasn't clear. I mean two personas
|
| - a) maybe I don't care about 99% reliability and I am
| attracted by the fact that it's free. But I have no control
| over when it will stop being free, so I don't implement it.
|
| - b) I have money to pay but I need reliability.
|
| They pull in different directions but both seem to pull
| away from a free alpha version. I would imagine if you
| could offer them as alternatives you might get both
| personas but with this model you kind of lose both?
|
| Again, I am sure this is a hole in my understanding, not
| the model. Just curious as to where it is.
| hiddencost wrote:
| Previously it wasn't free, so you wouldn't ever have used
| it.
| jeanlaf wrote:
| Hi! Here's how we think about it. Given the economical
| context, I would say pricing becomes a strong argument.
| It's also one of the reasons we built the program, it's a
| way for us to give back to the community.
|
| Having more usage on our side will help us tremendously
| at bringing all those connectors to higher reliability
| standards as we're exposed to more and more use cases. We
| do believe most people will be willing to pay for
| reliability and will stay with us when those connectors
| will become GA. We will also give a grace period for them
| to switch if they want to (I guess Airbyte Open Source
| will be the best alternative then :) )
| Centigonal wrote:
| nice way of internally motivating connector stability and
| adoption! It literally costs you money to keep connectors in an
| immature state for too long.
| yevpats wrote:
| Congrats! Shameless plug (Original Author and Founder) - Go High
| Performance alternative
| (http://github.com/cloudquery/cloudquery). no backend, no ui,
| everything stateless, as code and you can run it anywhere
| including ECS.
___________________________________________________________________
(page generated 2023-01-26 23:01 UTC)