[HN Gopher] Launch HN: MovingLake (YC S22) - Real-time data conn...
___________________________________________________________________
Launch HN: MovingLake (YC S22) - Real-time data connectors for
almost anything
Hello HN! We are Andres and Edgar and we are building MovingLake
(https://movinglake.com). We are creating real-time, bidirectional
data connectors which aim to solve automated ETL (eg moving data
from an API to Snowflake), reverse ETL (we pipe the data straight
to your CRM instead of going through the data warehouse), and
transactional integrations (e.g. for an ERP integration) in one
swoop. Polling, webhooks, websockets, REST, GraphQL, SOAP--we merge
it all into one reliable, replayable, real-time data stream which
can be sent anywhere. Here are two demos: Whatsapp to Google
Sheets (https://youtu.be/Qy4eInGgIhw) and Postgres to Webhooks
(https://youtu.be/cmrkUan8o1w). Moving data around is still
difficult and a pain. Companies spend thousands of hours and lots
of money making data pipelines. General solutions are difficult in
that there are too many sources and too many destinations. It also
entails a ton of schlep that no one really wants to deal with. At
the same time, the API / data connector market is vast. Even when
there are already multiple billion-dollar companies in the space,
we continue to see big verticals unattended. We previously worked
at big tech as well as have been CTOs of Series B startups in
Latam. We got into this problem when we had the experience of
paying for pre-built data connectors, but at the same time having
to do custom integrations to the API, at which point it didn't make
sense to have a paid data connector. After a few paid connectors
and manual connectors we thought that there should be an event-
driven, realtime data connector company which could solve all of
our API integration needs in one go. We also are not huge
believers in operationalizing the data warehouse, i.e. the trend to
use the data warehouse as a processing tool out of which data is
extracted and pushed to other systems. Since data warehouses are
built on an OLAP frame of mind, we think using it as a computation
source for automated workflows is not the best idea. At MovingLake
we propose using rather realtime connectors with a transformation
pipeline which is specifically designed to do these things. We
combine polling with webhooks and websockets to provide a single
reliable stream. If there's websockets, we use them. If there's
webhooks we use them. If there's only GET endpoints then we poll as
fast as the API lets us. Either way we ensure you'll get the data
as fast as possible. We provide destination adapters so that this
data can be sent anywhere to as many destinations as you want for
the same price. We provide CDC (Change Data Capture) plugins for
databases to pull data as it is written and then send it anywhere.
We also provide automatic JSON to SQL converters with Schema
Evolution. On our roadmap we still have to add the data
transformation layer as well as add support for more bidirectional
connectors. "Get started" on our home page takes you to a "book a
demo" thing, but there's an open beta at https://app.movinglake.com
which everyone is welcome to try out. Everything is free for now,
though eventually we'll have to charge. Although we wanted to
charge a standard fee per event extracted, this would have tied our
hands to deliver tougher-to-build connectors. So most of our
connectors are charged at $0.00004 per event, but some which
include scraping are charged higher (we want to be very explicit
about which connectors are charged differently). Please let us
know what you think! Roasting our product would be super helpful :)
Author : paezand
Score : 72 points
Date : 2022-10-27 16:19 UTC (6 hours ago)
(HTM) web link (app.movinglake.com)
(TXT) w3m dump (app.movinglake.com)
| [deleted]
| lyime wrote:
| How are you different than Airbyte
| mritchie712 wrote:
| it skips over the data warehouse. directly connect hubspot to a
| google sheet without a warehouse in the middle.
| mdaniel wrote:
| Congratulations on the launch!
|
| Because the href is `text-decoration:none`, this page appears to
| be blank: https://www.movinglake.com/docs/security/ only by
| hovering over "Data Privacy" does one realize it's a sub-page
|
| I would guess in general the docs need some TLC, since
| https://www.movinglake.com/docs/destinations/http/ doesn't help
| me know what knobs I have influence over, as best I can tell
| "page 2" is the same as page 1
| (https://www.movinglake.com/docs/destinations/page/2/), and _for
| sure_ please do not require static AWS credentials
| (https://www.movinglake.com/docs/destinations/s3/) -- we actually
| have a Service Control Policy that outright bans the use of
| `iam:CreateAccessKey` across our entire AWS Org. Also, the "IAM
| view" is missing its protocol, causing the browser to think that
| the link is
| https://www.movinglake.com/docs/destinations/s3/console.aws....
| aleandros wrote:
| Thanks a lot for the feedback. You are absolutely right
| regarding the AWS credentials. We aim to fix it as soon as
| possible. And thanks a lot for pointing out the problems in the
| documentation. We we'll work on them ASAP!
| codeaken wrote:
| What are the other option(s) for AWS credentials which would be
| more secure in this scenario?
| mdaniel wrote:
| sts:AssumeRole or its friend sts:AssumeRoleWithWebIdentity
| aleandros wrote:
| The target account would need to allow explicit access to our
| account via an AWS policy. See the docs:
| https://aws.amazon.com/premiumsupport/knowledge-
| center/cross...
|
| This is actually the approach we take for connecting to
| Google Services.
| IlyaFriedman wrote:
| Congratulations on the launch!
| aleandros wrote:
| Thanks a lot!
| Epamar wrote:
| The connector for Whatsapp is amazing, it has so many use cases
| across several industries.
| danielmarkbruce wrote:
| This is really awesome. Great work guys. So many use cases, such
| annoying stuff to work on, huge value add.
| aleandros wrote:
| Thanks Daniel! We have solved this before at our previous jobs
| several times, so we are thrilled to help other companies with
| this.
| nub_s wrote:
| Congrats, this is super useful, esp. the whatsapp connector
| propagating in real time to google sheets opens up a lot of
| possibilities. I understand pricing would be based on the data
| volume?
| aleandros wrote:
| Thanks a lot! Just an FYI, the google sheets destination is
| great for getting started but we support several destinations
| for more advanced usage and workflows.
|
| Regarding pricing, you are mostly right. It is connector-
| dependent and volume based. Most of our connectors are charged
| at $0.00004USD per event.
| Fiahil wrote:
| Hi and congrats !
|
| > bidirectional data connectors which aim to solve automated ETL
| (eg moving data from an API to Snowflake), reverse ETL (we pipe
| the data straight to your CRM)
|
| I need exactly this at work. However, my requirements are :
|
| - it has to be "self-hosted", usually in my clients' VPC, sending
| data to a third party is a big no-no.
|
| - most of them use SAP, Salesforces and many other esoteric CRMs
| & ERPs. It has to work for at least the most important ones.
|
| - the target medium is usually the cloud provider's object
| storage (S3, ...), snowflake is quite rare. It also have to be
| flexible enough to support our own versioning strategy.
|
| Would it work with your product ?
| danielmarkbruce wrote:
| +1 to a version which works in a VPC in a public cloud. This is
| starting to become table stakes for some large enterprise
| products.
| aleandros wrote:
| We completely understand. We don't support it right now, but
| we built the product to be completely cloud native so this is
| straight forward for us to implement. But yeah, it's
| definitely in our roadmap.
| danielmarkbruce wrote:
| The economics can be nice too, run everything on their bill
| :)
| aleandros wrote:
| Thanks a lot for the comment Fiahil!
|
| All of this is in our roadmap so we are glad to hear that we
| are on the right track. However we would love to know more
| about your last point. If you can email us to
| andres@movinglake.com and edgar@movinglake.com so we can
| discuss it, we would greatly appreciate it!
| colinchartier wrote:
| Just sends me to a login page?
| paezand wrote:
| Please use the Sign-Up link at the bottom or use this URL
| directly https://app.movinglake.com/#/authentication/signup. We
| are working on improving the copy of that page, sorry for the
| confusion here.
| Timja wrote:
| ako wrote:
| I'd expect anybody doing enterprise software and having to
| integrate different systems would be aware of these.
|
| Google probably has better explanations for these, but to
| summarize:
|
| * data connectors - connectors that allow you to get data from
| all systems you use in your organization, from databases to
| ERPS, to custom build systems.
|
| * ETL - a method for moving data from your data sources to your
| datawarehouses (DWH) - extract from source, transform to
| desired format, load in you DWH.
|
| * Snowflake - cloud based highly scalable DWH for structured
| and unstructured data
| Timja wrote:
| aleandros wrote:
| Hi Timja, sorry about that! We have been so focused on this for
| the past months that is easy to loose perspective.
|
| Data connectors, in the loosest definition possible, is simply
| a piece of software that moves data from one place to the
| other. This can be from database A to database B or (as in our
| case) from a given API into a database.
|
| ETL stands for "Extract, Transform, Load", a process in which
| you get data from some place, clean it/do something with it,
| then store it into a desired destination. Probably this term is
| most frequently used in the context of data warehousing, in
| which you move data from one or more OLTP databases from the
| application side, into an OLAP database.
|
| Finally, Snowflake is a very popular (and very cool) database
| focused on Data Warehousing needs.
| Timja wrote:
| Thanks for the explanations!
|
| I get kind of a "no code" vibe from it?
|
| When I want to read data from an API, transform it and put it
| into a DB, my approach would be to write a few lines of
| Python. A script that does an http request to the API,
| transforms it and then writes the data to SQLite.
|
| That seems much easier and more future proof to me than to
| bring in a 3rd party service.
| aleandros wrote:
| Yes, the idea is for it to be a no-code solution. We
| completely understand why some people would like to this
| work in-house, specially for simple use-cases such as the
| one you mentioned.
| dang wrote:
| Users flagged it. We can only guess why users flag things.
| Maybe they thought it was offtopic, or that the terms were
| obvious? not sure. I've unflagged it now.
|
| When editing the launch blurbs, I usually tell founders to add
| explanations of any concepts or jargon that HN readers might
| not be familiar with. It's not always easy to know where to
| draw that line.
|
| https://news.ycombinator.com/item?id=33360234 was pretty
| rightly flagged though - you're not supposed to put down other
| people's work in HN threads, especially not Launch or Show HN
| threads (https://news.ycombinator.com/showhn.html).
| Timja wrote:
| Interesting that
| https://news.ycombinator.com/item?id=33360234 is considered
| to put other people's work down.
|
| I asked about some terms and if they are common among the HN
| crowd. Ako replied that people who are doing enterprise
| software integrations know them. Then I ask if the HN crowd
| is nowadays people who are doing enterprise software
| integrations. (Because that is not what I would associate
| with hackers / startup founders. So maybe what initially was
| a community of hackers + startup founders is now a community
| of grown ups with enterprise jobs.)
|
| Whose work have I put down by asking this?
| ako wrote:
| There are plenty of startups that focus on enterprise
| (integration) software, don't see how these are mutually
| exclusive. And you can be both a hacker at night and an
| enterprise software engineer by day.
| pvg wrote:
| It's not hard to figure out what an unfamiliar (to you)
| product or forum community is about - by a bit of googling
| for the former and reading and participation for the
| latter. Maybe it's not your intent but you come off as if
| you're conducting some sort of interrogation which most
| people aren't all that keen on. It's also mentioned in the
| site guidelines:
|
| _Have curious conversation; don 't cross-examine._
|
| https://news.ycombinator.com/newsguidelines.html
| squidlogic wrote:
| I tried to sign up but never received the verification email.
| Might just be me.
| paezand wrote:
| Hi!
|
| Let me know if you still haven't gotten the email. I just
| tested it out and worked for me. But we can activate manually
| your user if you send me an email at andres@movinglake.com.
| squidlogic wrote:
| Thanks, I sent an email. IMO it would be helpful to list your
| pricing model on the landing page as well.
| paezand wrote:
| Noted! Although we wanted to charge a standard fee per
| event extracted, this would have tied our hands to deliver
| tougher-to-build connectors. So most of our connectors are
| charged at $0.00004 per event, but some which include
| scraping are charged higher (we want to be very explicit
| about which connectors are charged differently).
| [deleted]
| max_hammer wrote:
| Looks awesome.
|
| What is the pricing model ?
|
| I would love a good real time connector for database like
| Oracle/postgres. There are few options like AWS DMS. But don't
| have many options for target
| aleandros wrote:
| Hi Max, thanks a lot for the interest!
|
| Pricing is connector-dependent and volume based. Most of our
| connectors are charged at $0.00004USD per event. You could
| email us to andres@movinglake.com and edgar@movinglake.com so
| we can discuss if we are a good fit for your needs.
| Jchimal wrote:
| Hola, solo para comentar que hay un pequeno error en la
| traduccion, dice "tostar" la aplicacion, cuando deberia decir
| "testar". Su aplicacion podria funcionar para pequenas o micro
| empresas...?... Algun ejemplo...? Gracias.
| paezand wrote:
| Hola!
|
| Gracias por el comentario! Supongo que en la traduccion de
| ingles a espanol se perdio ahi algo.
|
| Pero si claro! Nuestra aplicacion puede funcionar para pequenas
| o micro empresas para cualquiera de nuestros conectores.
| Cualquier pregunta por favor escribanos a andres@movinglake.com
| o edgar@movinglake.com
| sert121 wrote:
| Are you hiring :)?
| paezand wrote:
| Shoot me an email to andres@movinglake.com if you are
| interested!
| jonheller wrote:
| Congrats on the launch! I do have a quick bit of feedback based
| solely on looking at the website:
|
| Seeing a clearly fake testimonial immediately makes me lose trust
| in your product. It was pretty obvious the images in your
| testimonials were stock images, and a quick reverse image search
| confirms this (https://tineye.com/search/2f7aa5f73e0cee27b9ff54b1
| ad29cc8353...).
|
| You obviously put a lot of work into your product, and it'd be a
| bummer to turn people off for something like that.
| a5seo wrote:
| Hah, I thought that, too. I Googled for this person and didn't
| find anything. Funny how something just didn't seem right in
| that testimonial.
| aleandros wrote:
| We just removed them! But we will be uploading real
| testimonials soon! Again, thanks a lot!
| aleandros wrote:
| Thanks a lot for bringing this up! Yeah, you are right. Those
| were meant to be placeholders, not to mislead customers but
| should not be there. We'll make sure to remove them!
| ianbutler wrote:
| > We also are not huge believers in operationalizing the data
| warehouse, i.e. the trend to use the data warehouse as a
| processing tool out of which data is extracted and pushed to
| other systems. Since data warehouses are built on an OLAP frame
| of mind, we think using it as a computation source for automated
| workflows is not the best idea. At MovingLake we propose using
| rather realtime connectors with a transformation pipeline which
| is specifically designed to do these things.
|
| Absolutely, I think this area is under tapped currently. Good
| luck, fantastic area to work on.
| aleandros wrote:
| Thanks a lot!!!
| tareknh wrote:
| I heard you were creating connectors for WhatsApp! That's a great
| feature and it benefits any company using WhatsApp for
| communication with their own customers. Thank you, great idea!
| paezand wrote:
| Great to hear thanks! Let me know if you want more info about
| it!
___________________________________________________________________
(page generated 2022-10-27 23:00 UTC)