[HN Gopher] Build API integrations with SQL and YAML - no SaaS l...
___________________________________________________________________
Build API integrations with SQL and YAML - no SaaS lock-in, no
drag-and-drop UIs
Author : maxgrinev
Score : 35 points
Date : 2025-05-30 15:29 UTC (7 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| katamaster818 wrote:
| seems like sequor.dev (docs site) is down..
| maxgrinev wrote:
| Thank you for letting me know! I'm not seeing any issues on my
| end - both https://docs.sequor.dev/ and https://sequor.dev/ are
| loading fine for me. This might be a temporary issue or could
| be related to your location/network. Try to refresh it. Let me
| know if you're still having trouble!
| katamaster818 wrote:
| well, that's one way of learning that my company is now
| blocking some traffic on the vpn... works fine when I drop
| off the VPN, thanks.
| sscarduzio wrote:
| > Fuses API execution with SQL logic to provide an open, flexible
| platform (...)
|
| But there's a gigantic multi line string in your yaml. Full of
| Python.
| maxgrinev wrote:
| The Python here is doing exactly what it should - extracting
| nested data and flattening it into proper relational tables.
| Most of the 'Python' is actually just the return statement
| defining table schemas. The schema can be defined separately
| and reused across API calls. With such reuse the Python part
| would be really compact. This is the right balance between
| declarative config and the flexibility needed for real-world
| API responses. I can confirm it from my experience developing
| many integrations with Sequor - this approach handles real-
| world complexity much better than pure declarative tools.
| slt2021 wrote:
| >> intuitive YAML-based workflow definition.
|
| in my opinion, there is very little intuitive in a mix of yaml +
| templating engine, sql, python, and custom DSL logic
| maxgrinev wrote:
| Fair point - 'intuitive' is subjective and depends on your
| background. Let me explain where each technology fits: HTTP
| request configuration (URL, headers, parameters) is pretty
| intuitive and similar to Postman that most people know. Python
| is used mainly for response parsing - without a general-purpose
| language in this place it would be very limiting. Templating is
| mainly for variable substitution, familiar to anyone using
| frameworks like dbt. And SQL stays separate from Python -
| they're not mixed together.
| captn3m0 wrote:
| This looks quite similar to Arazzo:
| https://github.com/OAI/Arazzo-Specification/
| maxgrinev wrote:
| Thank you for pointing this out! I wasn't aware of Arazzo
| before. It does look very similar indeed. I'll definitely look
| more into it and see how we might align with their format or
| adapt it to Sequor's capabilities and execution model. Really
| appreciate the heads up!
| dangets wrote:
| The request / data fetching is interesting in how "easy" it is to
| write. I did basic perusal of the examples, but I'd be interested
| to see what it looks like with rate-limited endpoints and
| concurrent requests.
|
| Another tangentially related project is https://steampipe.io/
| though it is for exposing APIs via Postgres tables and the
| clients are written using Go code and shared through a
| marketplace.
| maxgrinev wrote:
| Great question! Rate limiting and concurrency are absolutely
| critical for production API integrations. Here's how Sequor
| handles these challenges: Rate Limiting:
|
| * Built-in rate limiting controls at the source level (requests
| per second/minute/hour): each http_request operations refers to
| an http source defined separately * Automatic backoff and retry
| logic with delays * There is a option for per-endpoint rate
| limit configuration since different API calls can have
| different limits * Because it is at the source level, it works
| properly even for parallel requests to the same source.
|
| The key idea is that rate limits are handled by the engine - no
| need to handle it explicitely by the user.
|
| Concurrency is explicit by the user: * Inter-operation
| parallelism is activated by adding begin_parallel_block and
| end_parallel_block - between these two operations all the
| operations are executed at once in parallel * Intra-operation
| parallelism: many operations have parameters to partition input
| data and run in parallel on partitions. For example,
| http_request takes an input table that contains data to be
| updated via API and you can partition the input table by key
| columns into specified number of partitions.
|
| Thanks for the Steampipe reference! That's a really interesting
| approach - exposing APIs as Postgres tables is clever, and I'm
| definitely going to play with it.
| throwaway7783 wrote:
| I'm biased, but this is yet another thing my team has to build
| and maintain now.
|
| I am not in the business of building and maintaining
| integrations, but trying to derive and surface signals that are
| useful to my business, from all the data (often crappy data)
| that's generated. Even just dumping data in a database is not
| sufficient, because it's not clean, not unified with other data,
| highly duplicative in many cases and so on.
|
| I'm glad these options exist though, for teams that thrive on
| building everything themselves.
| maxgrinev wrote:
| Totally understand - you want clean, unified data for business
| insights, not another integration tool to maintain. Sequor
| actually grew out of our Master Data Management (MDM) work
| where data cleaning and deduplication are core challenges. We
| focused on API integration for this release, but have mature
| data cleaning/deduplication components that we plan to open
| source as well. What specific data quality issues are you
| dealing with? Happy to share what we've learned from MDM
| projects and our data quality engine.
| throwaway7783 wrote:
| That's exactly where I landed too. We didn't need a modern
| data stack managed by a data team, where everything is coded
| with significant turnaround times. We ended up using an MDM
| (Syncari).
|
| MDMs are unsexy and have a lot of baggage filled with legacy,
| expensive vendors. But the principles are sound, and more
| modern platforms have turned out to be pretty good.
| tnolet wrote:
| But, this is just leaky abstractions to the max. You replaced a
| python code base with adding a layer of YAML on top of it, but
| then still requiring you to write Python?
|
| My IDE will have a stroke parsing this.
| gervwyk wrote:
| If you enjoy building stuff in yaml, check out lowdefy.com
| pan69 wrote:
| This looks more website oriented, i.e. building HTML pages
| within a YAML construct. OP's project seems to be more
| API/middleware oriented.
| gervwyk wrote:
| It's for building web apps, including the backend. The latest
| version also allows to define backend flow logic for APIs but
| the docs is still in progress.. Sharing this because we've
| building abstraction to yaml for a few years so can look at
| lowdefy for some ideas.
|
| For us, building business logic in yaml scales really well
| across projects and teams. We run advanced ticketing systems,
| crms, call center solutions etc all defined in yaml config.
___________________________________________________________________
(page generated 2025-05-30 23:00 UTC)