[HN Gopher] Geospatial data science with Julia
___________________________________________________________________
Geospatial data science with Julia
Author : juliohm
Score : 148 points
Date : 2023-10-11 00:13 UTC (15 hours ago)
(HTM) web link (juliaearth.github.io)
(TXT) w3m dump (juliaearth.github.io)
| jstrickshire wrote:
| I have a passion project 4x4anarchy.com that operates with a
| Python-MariaDB system for querying map data by latitude and
| longitude, transforming it into GeoJSON for map display. The
| website deals with sizable tables, approximately 1 GB in size.
| I've made extensive optimizations, relying on well-structured
| indexes, caching mechanisms, and query optimization to enhance
| performance.
|
| Given these circumstances, how might the incorporation of Julia
| and some geospatial DB (PostGIS) contribute to further optimizing
| geospatial data retrieval and presentation, especially when
| dealing with large datasets and intricate geospatial operations?
| tony_cannistra wrote:
| Interesting! I work on a very similar product.
|
| I don't know Julia well, but I definitely would suggest
| exploring whether PostGIS can help improve the speed of your DB
| queries.
|
| I'd also consider how you deliver your geospatial data to your
| clients -- I'm not sure GeoJSON is your best bet. Protobuf
| tiles might be better for your use-case (e.g. the Mapbox Vector
| Tiles spec).
| p4ul wrote:
| I completely agree! It would be hard to overstate the power
| of PostGIS!
|
| For anyone working with GIS data, it's absolutely worth
| investigating what PostGIS provides and the ease of
| integration to your existing application!
| gabegm wrote:
| It would depend on where most of the processing is happening.
|
| PostGIS gives you the benefit of spatial indexes which are
| extremely performant.
|
| I've seen Python GeoSpatial applications taking hours to finish
| processing which only took a few minutes when shifted onto
| PostGIS.
|
| If you're also doing a lot of processing in Python, exploring
| other languages could also help. In the case of Julia you get a
| typed language that's also JIT compiled.
| qfwfq_ wrote:
| Geopandas has had spatial indexing available for quite a long
| time...
|
| https://geopandas.org/en/stable/docs/reference/sindex.html
|
| I think that the challenge for most is that the PostGIS query
| planner does the indexing for you in most queries, while a
| naive all-pairs comparison in geopandas/shapely won't tell
| you to use the .sindex attribute instead.
| mustknow2201 wrote:
| I know you said it's a passion project, but you should probably
| still give the correct OSM attribution
|
| https://osmfoundation.org/wiki/Licence/Attribution_Guideline...
| jstrickshire wrote:
| I appreciate you calling that out, I will get that done.
| benzofuran wrote:
| Cool site! Any chance of a adding a simple KMZ export for
| offline use for a given area of interest?
| jstrickshire wrote:
| Yeah, I can do that. Will get to it tomorrow!
| benzofuran wrote:
| Awesome - getting KMZs of 4x4 routes is way harder than it
| should be. All the Colorado data is there but extracting it
| is challenging.
| fiedzia wrote:
| If all you do is "find records within x miles from lat,lon",
| solr/ES is the best solution. I think it can match a shape too.
| juliohm wrote:
| Geospatial Data Science with Julia presents a fresh approach to
| data science with geospatial data and the Julia programming
| language. It contains best practices for writing clean, readable
| and performant code in geoscientific applications involving
| sophisticated representations of the (sub)surface of the Earth
| such as unstructured meshes made of 2D and 3D geometries.
| [deleted]
| eigenket wrote:
| Are you a bot? Why did you copy and paste the top paragraph of
| the linked page?
| forgotpwd16 wrote:
| Seems to be the author and copied it as some form of
| abstract. @juliohm no need to be doing that.
| alekseiprokopev wrote:
| Nice thing about Julia is that you randomly find cool projects
| like this.
| nraynaud wrote:
| Be mindful that most of julia's geometry code is a wrapper of
| libGEOS (C version) and libGDAL, that means that you can't easy
| extend the algorithms, everythig is behind a black box on the C
| side. Source: I have worked in the field last year, I have a
| small patch in LibGEOS.jl .
| juliohm wrote:
| This is not true. Please read the book.
| nraynaud wrote:
| Scanning the site see mostly points algorithms, the only
| mention of polygons is a textbook LibGEOS call, I see no
| network at all. And I see no smart manipulation of anything
| else than points, I see no subdivision of space, etc.
| juliohm wrote:
| You probably need to re-scan the book. Meshes.jl is the
| submodule of the project entirely written in Julia with
| geometric processing algorithms.
| nraynaud wrote:
| I have worked with it. It was just stating, very little
| useful code in it. Going back to the source code, I see
| they added a bit more. A quick look around suggest that
| only one algorithm uses an indexing structure. Clipping
| seems limited between a convex polygon and a concave one.
| ZeroCool2u wrote:
| The book is quite interesting, but it does seem like a lot
| of the underlying work is farmed out to GeoStats.jl, which
| doesn't really seem to use the same vocabulary I'd expect
| in other languages using PostGIS or Geopandas etc. For
| example, I don't see many mentions of Polygons or
| MultiPolygons when I search. However, I do find this
| page[1] which seems to define similar(?) equivalents. Can I
| expect equivalent geospatial joins/queries to be available?
| I don't see many mentions of the types that I would
| normally do, especially overlay operations[2].
|
| [1]: https://juliaearth.github.io/GeoStatsDocs/stable/domai
| ns.htm...
|
| [2]: https://geopandas.org/en/stable/docs/user_guide/set_op
| eratio...
| cbkeller wrote:
| So in other Julia geometry-related projects that may be true,
| but for this particular corner of the ecosystem the main
| author (Julio Hoffimann) has actually implemented much of the
| underlyin code from scratch (to the best of my understanding)
| in pure Julia in a whole set of packages, including
|
| https://github.com/JuliaGeometry/Meshes.jl
| https://github.com/JuliaGeometry/Rotations.jl
| https://github.com/JuliaEarth/GeoStatsBase.jl
| https://github.com/JuliaEarth/PointPatterns.jl
| beeburrt wrote:
| In the preface you list:
|
| - Generate high-performance code
|
| - Specialize on multiple arguments
|
| - Evaluate code interactively
|
| - Exploit parallel hardware
|
| > This list of requirements eliminates Python, R and other
| mainstream languages used for data science.
|
| Can you elaborate on why/how? Awesome work by the way
| ekianjo wrote:
| R can exploit parallel hardware just fine with Parallel, Future
| and other libraries like Mirai. The problem is that execution
| speed is going to be a bottleneck for anything large and when
| you reach some optimizations, maybe R is not the best language
| to do the job. But it depends a lot on the use case.
| wodenokoto wrote:
| Python and R do not generate high performing code. At best they
| generate calls to high performing code.
| BrandonS113 wrote:
| I used to think so, but I have a function that gets called
| about a billion times each and every day as new data comes
| in, and and takes about 0.01 seconds to evaluate
| (optimizaiton with nlopt). I tried to code it in c (30% speed
| improvement) python (twice as slow), Julia (about the same
| speed). Reason is that call has 5 parameters that operate on
| a vector of length 50 to return a value to minimize. Turns
| out R is pretty good at such vector calculations.
| n4r9 wrote:
| Is this what you mean by nlopt?
| https://github.com/stevengj/nlopt
|
| If so, it looks like you're interfacing from R to high-
| performing code written in C. Isn't that exactly what OP
| was describing?
| trostaft wrote:
| > At best they generate calls to high performing code.
|
| It should be noted that this is usually sufficient. But
| particularly for earth scale problems it can often not be.
| th0ma5 wrote:
| Julia is designed to seem to win arguments as best I can
| tell... If you complain about the need to break abstractions
| and the lack of general purpose application you're accused of
| not understanding. When you say it slow they say you can
| inline assembler, and when you say that's dumb why have a
| high level language then, they then say well you don't have
| to it is fast as is and everyone else is slow, and it just
| devolves into circular arguments. Abstractions exist in
| layers for reasons.
| sainez wrote:
| You can obviously provide the same abstraction with
| different implementations that yield different performance
| characteristics. Julia provides the same level of
| flexibility (if not more) as Python without any of the
| design decisions which cause Python to be so slow. I fail
| to see how this is a contentious point.
| adgjlsfhk1 wrote:
| when you say Julia is slow, what are you talking about?
| even without any fancy tricks, normal Julia code is usually
| the same speed as the equivalent normal C code
___________________________________________________________________
(page generated 2023-10-11 16:02 UTC)