[HN Gopher] Analysing Data in Web3 with SQL
___________________________________________________________________
Analysing Data in Web3 with SQL
Author : webmaven
Score : 38 points
Date : 2022-05-20 14:36 UTC (1 days ago)
(HTM) web link (thisgoke.medium.com)
(TXT) w3m dump (thisgoke.medium.com)
| hans1729 wrote:
| > Web3 is going through rapid development and high variance
|
| Well thats one way to put it.
| halfmatthalfcat wrote:
| "We are searching for the problem our solution 'solves'."
| [deleted]
| mritchie712 wrote:
| If you need an API (dune doesn't have one), check out Luabase
| (https://luabase.com/). We're built on Clickhouse (instead of
| Postgres), so most queries are orders of magnitude faster.
|
| We also support scheduling, webhooks and a bunch more. I'm one of
| the founders (mike@luabase.com), let me know if you have
| questions!
| ushakov wrote:
| if you don't need API just use an Ethereum ETL:
| https://github.com/blockchain-etl
|
| query Ethereum via RPC
| https://ethereum.org/en/developers/docs/apis/json-rpc/
|
| or use a free one hosted one by Google:
| https://cloud.google.com/blog/products/data-analytics/introd...
|
| there's also open-source Graph Protocol:
| https://github.com/graphprotocol
| ushakov wrote:
| do you stream ETL data to Clickhouse?
|
| what kind of delay do you have between when the event occurs
| and when it's in the database?
| mritchie712 wrote:
| Exactly, we stream blockchains (3 now!) to our warehouse with
| < 60 second delay from the live data
| hbarka wrote:
| > We're built on Clickhouse (instead of Postgres), so most
| queries are orders of magnitude faster.
|
| Woah woah dems fightin words. Is this true?
| mritchie712 wrote:
| Yes, quite true
|
| https://www.timescale.com/blog/what-is-clickhouse-how-
| does-i...
| akulkarni wrote:
| Results of query benchmarking between TimescaleDB [built on
| PostgreSQL] and ClickHouse. TimescaleDB outperforms in
| almost every query category
|
| I don't think that post says what you think it says ;-)
|
| (Disclaimer, Timescale co-founder)
| jakewins wrote:
| As I understand, the comparison doesn't make sense.
| ClickHouse is an OLAP db, PG is an OLTP system. I haven't
| used ClickHouse, but it seems it's the kind of thing to back
| reporting functions, data science workloads etc, not back a
| transactional website load?
| gfody wrote:
| postgres isn't known for being fast, quite the opposite
|
| if your usecase is one clickhouse can handle
| (timeseries/olap) most queries are orders of magnitude
| faster.
| akulkarni wrote:
| Time-series queries are faster on TimescaleDB (built on
| PostgreSQL) than ClickHouse.
|
| https://www.timescale.com/blog/what-is-clickhouse-how-
| does-i...
|
| Disclaimer: Timescale co-founder
| i_like_waiting wrote:
| I was wondering regarding Web3 data, if anyone was succesful to
| self-host the transactions in some local database.
|
| When I was trying to do that, my local system could keep up only
| with doing writes, but not much more. What specs you need to have
| for that? I guess now its like 10tb of transactions in Ethereum,
| 1m transactions per day.
| i_like_waiting wrote:
| Also if somebody wants to get into this type, I would recommend
| infura as source of data, or some other similar providers.
| ushakov wrote:
| i'd also add alchemy (alchemy.com) to the list
| ushakov wrote:
| Google was
|
| https://cloud.google.com/blog/products/data-analytics/introd...
|
| storing billions of records in database is hard, but querying
| them is harder
|
| if you want to try, get yourself some proper tools first:
| https://github.com/blockchain-etl
|
| or you can just query transactions via RPC
| shuntress wrote:
| Is anyone willing to try to explain for me what this article is
| actually about in a way that doesn't need so much jargon?
|
| Is this actually about some kind of Web3 Data Analytics or is
| this "I downloaded the entire block chain and parsed it into a
| relational database"?
___________________________________________________________________
(page generated 2022-05-21 23:01 UTC)