[HN Gopher] Show HN: Xorq - open-source Python-first Pandas-styl...
___________________________________________________________________
Show HN: Xorq - open-source Python-first Pandas-style pipelines
Hi HN, Dan, Hussain and Daniel here... After years of struggling
with data pipelines that worked in notebooks but failed in
production, we decided to do something about it. We created xorq to
eliminate the constant headaches of SQL/pandas impedance mismatch,
runtime debugging, wasteful recomputations and unreliable research-
to-production deployments that plague traditional pandas-style
pipeline workflows. xorq is built on Ibis and DataFusion. We'd
love your feedback and contributions. xorq is [Apache 2.0
licensed](https://github.com/letsql/xorq/blob/main/LICENSE) to
encourage open collaboration. _Repo_ :
https://github.com/letsql/xorq _Docs_ : https://docs.xorq.dev
_Roadmap Issues_ : https://github.com/letsql/xorq You can get
started `pip install xorq`. Or, if you use nix, you can simply run
`nix run github:xorq-labs/xorq` and drop into an IPython shell.
Demo video: https://youtu.be/jUk8vrR6bCw Here are some vignettes
to look into next: 1. _MCP Server + Flight + XGBoost:_
https://docs.xorq.dev/vignettes/mcp_flight_server 2. _1 DuckDB + 2
Writers + 1 Reader:_
https://docs.xorq.dev/vignettes/duckdb_concurrent 3. _OpenAI UDF:_
https://docs.xorq.dev/tutorials/hn_data_prep Some features to
note: - Ibis-based multi-engine expression system: effortless
engine-to-engine streaming - Cache expressions with `.cache`
operator - Portable DataFusion-backed UDF engine with first class
support for pandas dataframes - Serialize Expressions to and from
YAML - Easily build Flight end-points by composing UDFs thanks
for checking this out, and we're here to answer any questions!
Author : secretasiandan
Score : 13 points
Date : 2025-03-27 17:27 UTC (5 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| esafak wrote:
| Interesting!
|
| 1. How does it compare against alternatives?
|
| 2. Do you have benchmarks?
| secretasiandan wrote:
| 1. I would argue there are no "real alternatives". The two most
| proximate alternatives in feature space are Ibis and Snowpark.
|
| - Ibis because while it can target multiple engines (as we
| state in our docs, we are built on and heavily reliant on
| Ibis), it aims to be "single engine, single session" in its
| execution in that nothing is expected to persist beyond the
| current session and an Ibis expression can only have a single
| engine. We want to be multi-engine and have some artifacts
| durable across sessions (by way of caching)
|
| - Snowpark because it is sort of "multi-engine" by way of
| external functions or python stages, but locked to Snowflake.
| In some sense, we want to be Anypark: Snowpark like
| functionality but centered on whatever engine of choice is
| desired and performant interop with any other engines.
|
| 2. We don't have anything I would hold out as benchmarks yet.
| We don't aim to be "best in class" / the "fastest engine", we
| aim to be "in class" for as many operations as possible (we use
| the word performant). Our goal is to make it easy for an org to
| choose whichever engine(s) they feel most performant in when
| they consider the full space of {developer,computation} x
| {time,cost}. However, Hussain has demonstrated how having
| information from the "whole pipeline" available but execution
| deferred can allow for specialized optimization by way of
| predicate pushdowns (https://ibis-project.org/posts/udf-
| rewriting/)
|
| Thanks for your interest and please feel free challenge any of
| the above or point us to anything you think we might have
| overlooked!
|
| Best Dan
| esafak wrote:
| What size data is "in class" for Xorq? Can it process data
| out-of-core?
| secretasiandan wrote:
| Yes, "we" are out of core to the extent that the engines
| used in the deferred expressions we execute are out-of-core
| (our "batteries-included" engine is a modified Datafusion).
|
| We have previously demonstrated the capability of doing
| iterative batch training by way of our "batteries-included"
| engine. I'll try to post a reference later but need to run
| now due to family obligations.
___________________________________________________________________
(page generated 2025-03-27 23:02 UTC)