[HN Gopher] Directly running DuckDB queries on data stored in SQ...
       ___________________________________________________________________
        
       Directly running DuckDB queries on data stored in SQLite files
        
       Author : eatonphil
       Score  : 23 points
       Date   : 2022-03-25 12:57 UTC (10 hours ago)
        
 (HTM) web link (twitter.com)
 (TXT) w3m dump (twitter.com)
        
       | samwillis wrote:
       | Would love to see a benchmark comparing SQLite, DuckDB and this
       | SQLite/DuckDB combo with the same queries and data.
       | 
       | Is this adapter copying the contents of the SQLite db into
       | DuckDBs own memory layout in order to optimise the queries or is
       | it just "proxying" to the SQLite library?
        
         | 1egg0myegg0 wrote:
         | It is just scanning the data (in parallel), not copying it into
         | DuckDB!
        
       | eatonphil wrote:
       | I'm excited to use duckdb in DataStation/dsq but the Go bindings
       | aren't yet super mature and I haven't had time to work on them
       | myself yet.
        
       | nattaylor wrote:
       | Can't wait to see what simonw comes up for using this with
       | datasette!
        
         | samwillis wrote:
         | This would be super useful for something I'm working on using
         | datasette, querying across three tables with over 1.5m rows.
         | Going to try the same queries on this tomorrow.
        
         | simonw wrote:
         | Setting up a plugin hook that lets you use alternative database
         | backends is getting more and more tempting...
        
       | wenc wrote:
       | Exciting stuff!
       | 
       | I've been using duckdb instead of Pandas these days because it's
       | much faster on larger datasets plus I can write more compact and
       | complex SQL than I can Pandas constructs. The fact that duckdb
       | can query in memory Pandas dataframes faster than Pandas itself
       | is a plus.
       | 
       | It's like having a local performant database engine that can
       | query and join across Parquet, CSV, Pandas and now SQLite.
       | 
       | Kinda like a quick efficient local Spark.
        
       ___________________________________________________________________
       (page generated 2022-03-25 23:00 UTC)