[HN Gopher] How to build a custom data connector in 2 hours
___________________________________________________________________
How to build a custom data connector in 2 hours
Author : jeanlaf
Score : 89 points
Date : 2021-05-19 17:51 UTC (5 hours ago)
(HTM) web link (airbyte.io)
(TXT) w3m dump (airbyte.io)
| dvaun wrote:
| This looks great. I'd love to throw up an instance locally and
| play with it at work, but that likely wouldn't happen for a long
| time (local gov -> slow processes).
|
| That being said, I've got a fun project in mind which I may find
| a use case for this. I'll give it a shot.
| mtricot wrote:
| Thanks! let us know if we can help.
| thibaut_barrere wrote:
| Does anyone have some mid-long-term feedback on the use of the
| community edition of Airbyte in particular?
| arjie wrote:
| What do you want to know? I've been using since late-Jan /
| early-Feb.
|
| I fired up an EC2 instance, docker-compose'd up, added the
| initial few syncs and then let people just add their own.
|
| Mostly without trouble since then, though I'm running an old
| version as a result of the fire-and-forget.
|
| (edit: also, text me on Keybase or email me since I don't get
| notifications for responses here)
| smarx007 wrote:
| Been waiting for some practical uses since I read
| https://news.ycombinator.com/item?id=26650493. I forgot to ask
| back then if someone had experience with Apache Camel, Spring
| Integration, or similar and could compare it to the crop of SaaS
| offerings like Fivetran (that Airbyte itself is taking on by
| providing an OSS offering).
| Evidlo wrote:
| Was hoping for something about rapid prototyping of electrical
| connectors.
| therein wrote:
| Same, I was expecting some 3D printing, maybe some
| experimentation about how much spacing is needed between pins
| for insulation and how large the contact surfaces should be
| etc.
|
| I was excited.
| tmh88j wrote:
| That's what I was expecting as well. Guessing that "connector"
| an industry term that I'm just not aware of. Seems like some
| pre-made adapter patterns for ETL's from browsing the docs.
| buildbot wrote:
| I actually would love a service that did this for something
| mortally/individually affordable.
| mtricot wrote:
| In the end, data is transported as electrical current so it is
| not completely off :)
| afandian wrote:
| This is a fun story in that vein:
| http://www.bigmessowires.com/2016/06/04/db-19-resurrecting-a...
| geoduck14 wrote:
| Low supply of DB 19 connectors? I'll bet this is why there
| are 10k Teslas that can't be delivered right now.
| ncmncm wrote:
| It took a substantial fraction of two hours for the page to load.
| vincentdm wrote:
| Cool!
|
| We built something similar for on-premise data sources, in
| Clojure. https://basil.net
| cgardens wrote:
| The next step is to make the tool easy enough to use such that
| GPT-3 could build a connector in 2 hours.
| skadamat wrote:
| Exciting! Hoping to build Reddit & Stack Overflow connectors
| mtricot wrote:
| Let us know if we can help on slack (https://slack.airbyte.io)
| #connector-development or we can schedule a coding call call
| with you
| justinzollars wrote:
| cool! Nice work
| jeanlaf wrote:
| Thx!
| renewiltord wrote:
| Very cool. Honestly, I've been using a mostly vanilla Airbyte.
| There are already enough connectors that I usually don't need to
| write a new one.
|
| Pretty thrilled with how fast these guys have been going
| honestly. Only picked up their stuff a few months ago and there
| are already more and better connectors today than there were back
| then.
|
| I've been slowly moving over my error prone sync scripts to my
| Airbyte instance. Life is already better.
| jeanlaf wrote:
| Makes our day to read your comment :)
| justinzollars wrote:
| One question: who is your customer archetype?
| mtricot wrote:
| Data engineers, data scientist & analytics engineers. People
| that be users of Airbyte but also can be contributors.
|
| For data engineers, the objectives is to help them empower
| other roles to be data fluent and have the freedom to move data
| without bugging data engineers
| hobofan wrote:
| How suited is Airbyte for multi-tennant use-cases?
|
| I might have a project coming up to build a semi-analytics
| solution where you would obviously want to allow the users to
| easily connect to their data sources, and Airbyte looks like
| it might be a good fit for that (for now my rough idea was to
| build some plumbing around Singer taps). Are you aware of any
| users that use it for such purposes and are there any
| specific pitfalls too look out for there?
| mtricot wrote:
| We do support separate workspaces but we don't have auth
| yet. However because there is an API available, you can
| always build the account segmentation and permissionning at
| your app level and only interact with Airbyte using the
| API.
| jeanlaf wrote:
| If you want to have a look at the open-source GitHub project,
| here's the link: https://github.com/airbytehq/airbyte
| ducttapecrown wrote:
| What is a connector?
| sherifnada wrote:
| It's a module which extracts data from a source (an API,
| database, etc..) or pushes data to a destination (warehouse,
| blob storage, API, etc..). They're ubiquitous in data
| engineering & analytics workflows where users often need to
| consolidate their data in one database or warehouse to build
| dashboards, run analyses, etc..
| tmh88j wrote:
| I already responded to a different comment in this thread,
| but you seem to be knowledgable on the subject. Is this a
| tool to assist with ETL's between otherwise incompatible
| sources/destinations?
___________________________________________________________________
(page generated 2021-05-19 23:01 UTC)