[HN Gopher] Roapi: An API Server for Static Datasets
___________________________________________________________________
Roapi: An API Server for Static Datasets
Author : todsacerdoti
Score : 90 points
Date : 2021-10-08 12:17 UTC (10 hours ago)
(HTM) web link (tech.marksblogg.com)
(TXT) w3m dump (tech.marksblogg.com)
| iAm25626 wrote:
| very cool! For a project I used
| https://github.com/ranaroussi/pystore to stored TS(time series)
| data. As TS are mostly read-only so I try to avoid OLTP for
| certain use case that it's perfect for Parquet/DASK type of
| storage. I used Pandas to explore the data. This seems to be
| suited for high volume, query rate.
| houqp wrote:
| Yes, I am aiming for production grade online serving + many
| more query frontends and data types.
| gavinray wrote:
| What's the difference between this, and Datasette which has been
| in development multiple years by the Django co-creator?
|
| Sorry if it's a dumb question but I skimmed the docs and sample
| and it looks similar + didn't see any mention:
|
| https://github.com/simonw/datasette
| houqp wrote:
| I looked into Datasette before starting ROAPI. From a
| product/use-case point of view, to me Datasette focuses more on
| quick and easy ad-hoc data exploration type of work. ROAPI
| focuses more production ready online serving of static
| datasets. So I would expect users to use ROAPI to power micro-
| services in production with high QPS.
|
| From a technical design point of view, ROAPI authors owns the
| full stack end to end from query parsing, data format parsing
| to query execution because I am also a maintainer of Apache
| arrow and it's sub-project datafusion. The whole project is
| built with Rust end to end from scratch. Datasette is mostly a
| wrapper around sqlite. It translates user actions into SQL
| queries, then execute them on sqlite. In ROAPI, we work at a
| lower level. We translate REST APIs, GraphQL and SQLs into
| datafusion logical plans and execute them. Datafusion is also a
| analytical compute engine optimized for columnar data, so it
| will be a lot faster for OLAP workload, while sqlite is
| optimized for OLTP. I also plan to add other type of query
| capabilities like nearest neighbor vector search for ML
| applications, etc.
| gavinray wrote:
| Thank you for a genuine, well-thought answer. Much
| appreciated.
|
| I'm not in the Data Science space so I only know of
| Datasette, but maybe worth copy-pasting that on a FAQ page
| under "How does ROAPI compare to X?" to avoid repeating it.
| houqp wrote:
| yeah, that's a good idea. thanks for the suggestion :)
| FraaJad wrote:
| datasette is only sqlite, ROAPI serves up CSV, Parquet storage
| etc.,
| gavinray wrote:
| Datasette serves JSON, CSV, SQLite, and a bunch of others as
| well. Don't think it handles Parquet tho
| https://docs.datasette.io/en/stable/ecosystem.html#sqlite-
| utils
|
| _" Insert data into a SQLite database from JSON, CSV or TSV,
| automatically creating tables with the correct schema or
| altering existing tables to add missing columns."_
|
| There's a bunch of plugins (mix of official and user-
| developed) that might add other formats too, I'm not 100%
| sure:
|
| https://datasette.io/plugins?sort=downloads-this-week
| suicas wrote:
| Looks super interesting and potentially useful. Curious how it
| compares with Apache Drill (https://drill.apache.org/).
| zerkten wrote:
| According to linked in the author worked on Apache Arrow, so I
| suspect they knew about it. It feels like this is intended as a
| much more lightweight option, but would be interested to hear
| their take.
| houqp wrote:
| That's right, it's intended to be more lightweight since it's
| built with only Rust from the ground up. Apache Drill also
| only focuses on serving SQL as the user interface while ROAPI
| wants to provide a pluggable interface to support all use-
| cases. For example, we can plan graphql and rest api calls
| into query plan and efficiently execute them using
| Datafusion.
| rexreed wrote:
| Looks cool and useful. One note:
|
| "ROAPI is made up of 4K lines of Rust. This line count is low due
| to the intense use of 3rd party libraries."
|
| This actually seems like a high line count, and that's not
| counting all the 3rd party libraries.
|
| While not sexy, this is the sort of thing that PHP can do fairly
| easily in a much smaller # of lines of code and operate with
| lower resource requirements. I venture that this could be done
| using CSV parse, json decode, and other built-in PHP functions
| and use small infrastructure to make it work. I know PHP doesn't
| have a lot of love, but isn't this the sort of thing PHP is made
| for? Simple processing and hosting for API-based access to static
| file information? Is there a reason why Rust is needed with all
| the baggage?
| marklit wrote:
| The Parquet support, with the ability to handle row groups,
| skip over data not of interest to a query and aggregations that
| will likely be 3 orders of magnitude quicker are a pretty
| unique feature.
|
| Also, there's no need to write any code here, it's a CLI app.
| There's much less that could go wrong versus rolling your own.
| rexreed wrote:
| Interesting. I've used this PHP Parquet implementation in the
| past: https://github.com/jocoon/php-parquet. Whatever works!
| houqp wrote:
| This is true, the core of it is Apache arrow datafusion query
| engine, which is also a project I help maintain. I doubt you
| will be able to beat it with PHP though ;) The VM overhead
| alone will cause a big hit to your performance even if we can
| get JIT to work.
| houqp wrote:
| Author of the project here, thanks for writing about ropai! Happy
| to answer any question.
| dpeck wrote:
| I always love when people realize the power and simplicity that
| can come with static serving of mostly unchanging datasets. It
| can vastly speed up development in many cases.
|
| If you need a simple version of this, Hugo is a great and well
| proven option. Push onto your CDN of choice and you've got a
| blazing fast static json api.
|
| https://gohugo.io/templates/data-templates/
| distantsounds wrote:
| If you have a large dataset, Hugo will choke your clients in
| memory. https://github.com/gohugoio/hugo/issues/1065
|
| I fail to see how this method is not "simple enough"
| dpeck wrote:
| Fine to use whatever tool that works for you. Using the words
| "clients" here with reference to Hugo is confusing though, as
| clients would never be effected by it. The way I read that
| bug, it would be Hugo itself that is effected and crashes at
| generation time.
| tobgu wrote:
| This looks really awesome, I need to explore this project more!
|
| Here's a project which in part is built to fulfill a similar
| need: https://github.com/tobgu/qocache (I'm the author). Most of
| the background/rationale for it and example usage can be found in
| the README of the original QCache project, linked from the above
| repo.
|
| There are of course big differences between the projects but I
| find that they share the same goal of making random, file based,
| datasets easily accessible for querying.
|
| I'll definitely let myself be inspired by Roapi, thanks!
| houqp wrote:
| Thanks, nice work on qocache and qframe too :)
___________________________________________________________________
(page generated 2021-10-08 23:01 UTC)