[HN Gopher] Show HN: ADS-B visualizer
___________________________________________________________________
Show HN: ADS-B visualizer
I've created a web app for querying and visualization of ADS-B
datasets: https://adsb.exposed/ Source code:
https://github.com/ClickHouse/adsb.exposed/ The results
significantly exceeded my expectations because the pictures are
insanely beautiful, and the data is a treasure trove. It proves
many statements that were not certain: - it is feasible to generate
tiles by aggregation on a pixel level (instead of hexagons or
rectangular grid); - it does not require JPG/PNG tiles - we can
transfer raw bitmap data with zstd compression; - it is possible to
do it in real time;
Author : zX41ZdbW
Score : 220 points
Date : 2024-04-10 13:15 UTC (9 hours ago)
(HTM) web link (adsb.exposed)
(TXT) w3m dump (adsb.exposed)
| systemz wrote:
| Hi, thanks for sharing! Unfortunately besides empty map I don't
| see any visualization. Browser dev console is showing some CORS
| errors.
|
| EDIT: Nevermind, clickhouse.com was on disconnect's ad blocking
| list which I used for DNS blocking.
| parker-3461 wrote:
| It seems to be all working on my end though.
| zX41ZdbW wrote:
| We bought clickhouse.com from the previous owner 2.5 years ago,
| who used it for some ad network - it was in some block lists,
| but I hope we managed to clean up most of them.
| tamimio wrote:
| Still blocked in Disconnect list due to "Malvertising"
| rrix2 wrote:
| I'm not sure that list should be used any more. I tracked
| down a pihole issue a while ago to find this "The list you
| referenced was used in our legacy products. It is not
| maintained, has not been updated, and is not actively
| distributed us."
|
| https://www.reddit.com/r/pihole/comments/wtizpa/deprecation
| _... (ETA: https://github.com/disconnectme/disconnect-
| tracking-protecti...)
| tamimio wrote:
| Interesting, thanks for the heads up, just had it
| removed.
| Solvency wrote:
| fully empty map on iOS safari for me.
| panki27 wrote:
| Wow, this is beautiful!
|
| I predict this will get a hug of death $soon
| zX41ZdbW wrote:
| I keep the dashboard open :) So far ok with ~2000 QPS.
| windexh8er wrote:
| This is really cool when you dig into how much fidelity there is
| here. Also, a fantastic marketing campaign for Clickhouse!
|
| As for the dataset - is this continually updated or how "fresh"
| is it at any given moment?
| zX41ZdbW wrote:
| There are two data sources: adsb.lol and adsbexchange. The
| first is updated each day from
| https://github.com/adsblol/globe_history_2024. The second
| should be updated each month (they provide only sample data for
| the first day of each month), but I still have to put it in
| cron.
|
| The update scripts are also open-source, published here:
| https://github.com/ClickHouse/adsb.exposed/blob/main/prepare...
| underyx wrote:
| https://adsb.lol provides this data licensed under ODbL, this
| site violates the attribution and share alike clauses
| zX41ZdbW wrote:
| You can find the attribution and the link to the details at the
| bottom right corner of the main page. It links to the
| documentation: https://github.com/ClickHouse/adsb.exposed,
| which provides the full details. Additionally, you can read the
| license here:
| https://github.com/adsblol/globe_history_2023/blob/main/LICE...
| oq1ik6631rldh00 wrote:
| https://opendatacommons.org/licenses/odbl/1-0/
|
| 4.3.a. is clear:
|
| a. Example notice. The following text will satisfy notice
| under Section 4.3:
|
| Contains information from DATABASE NAME, which is made
| available here under the Open Database License (ODbL).
|
| At the time if writing, the attribution does not make it
| clear where to get the data and what the terms of the license
| of the data are.
|
| Only a (data: adsb.lol) which does not even comply with
| copyright attribution (Which funnily enough you are complying
| with for OpenStreetMap, while only using that for display,
| while creating your derivative database of the
| globe_history_2023 and globe_history_2024 database...)
| dirkhh wrote:
| I find it super frustrating to see well funded tech
| companies boldly abuse open data resources and assume they
| can just get away with it. I guess that's the theme of
| 2024. But to make it clear: you are in violation of the
| license the data is provided under, and as one of the
| copyright holders I object to that.
| zX41ZdbW wrote:
| Could you please clarify the details?
|
| I took extra care to provide all the needed attributions
| and credits and I believe it is complete and sufficient.
| If I missed something, please describe it at
| https://github.com/ClickHouse/adsb.exposed/issues, and I
| will correct it.
| dirkhh wrote:
| please read the license and its requirements. It is YOUR
| responsibility to comply with those requirements in order
| to be allowed to use the data. It is ridiculous that a
| well funded company asks a volunteer community to tell
| them how to comply with the license - after they have
| violated the license and used it for a marketing stunt.
| bilekas wrote:
| You might be frustrated and that's fine, personally I
| don't know what's missing either but you simply saying
| "YPU need to figure it out" isn't helpful at all.
| dirkhh wrote:
| Well, here's the language in the license that apparently
| is too hard for you to find:
|
| 4.3 Notice for using output (Contents). Creating and
| Using a Produced Work does not require the notice in
| Section 4.2. However, if you Publicly Use a Produced
| Work, You must include a notice associated with the
| Produced Work reasonably calculated to make any Person
| that uses, views, accesses, interacts with, or is
| otherwise exposed to the Produced Work aware that Content
| was obtained from the Database, Derivative Database, or
| the Database as part of a Collective Database, and that
| it is available under this License.
|
| I can't read it for you, but I can summarize it for you.
| You are required to make sure that someone who uses the
| product (i.e., Clickhouse's marketing stunt thingy)
| becomes aware of the license and origin of the underlying
| data. And not by digging into some GitHub repo, but right
| there, on the page.
| malwrar wrote:
| Their banner on the bottom links to adsb.lol and
| adsbexchange.com, is your specific concern that they
| don't have the odbl license called out? I personally
| didn't have much trouble figuring out where the data for
| this project came from based on their banner alone, and
| thought it was honorable of them to publish their process
| for obtaining the data.
|
| I have no horse in this race, but am really confused by
| this aggressive reaction to what I perceive to be a good-
| faith use of this data. Is this the prelude to some
| scheme by which you plan to extract money from
| ClickHouse? The grievance in these replies is genuinely
| unclear to me.
| ryandrake wrote:
| I don't think anyone is questioning the correctness of
| what you are saying, just the aggressive tone and
| assumption of malice over mistake. Maybe take this as an
| opportunity to educate licensees rather than ridicule
| them.
| singleshot_ wrote:
| Licenses get abused for one of two reasons, more or less:
| 1) the license is unenforceable; 2) no one enforces the
| license.
|
| In some cases, this is only one reason.
| rafram wrote:
| This is some copyleft troll-level pedantry [0]. They clearly
| made a good effort to comply with the license. Additionally,
| ADSB.lol requires contributors to license their contributions
| under CC0. Databases that don't have some sort of creative work
| involved in their compilation aren't copyrightable, so it's
| very dubious that anyone could enforce any sort of restrictive
| license over ADSB.lol's database as a whole when its individual
| contributions are CC0.
|
| [0]: https://pluralistic.net/2022/01/24/a-bug-in-early-
| creative-c...
| dirkhh wrote:
| Not at all. It is the polite request to please correctly
| attribute the data that a company is using for their
| marketing stunts.
|
| I love the fact that techbros being called out for violating
| other people's intellectual property immediately revert to
| "TROLL!" or "this is too hard!" instead of actually engaging
| with the question at hand. And btw - until you actually
| create the software stack to collect the data and run an
| aggregator, don't condescend on people who do the hard work
| that you seem to feel free to copy in violation of said
| license.
| jjwiseman wrote:
| No one said complying was too hard. This is one of the
| coolest things anyone's done with your data, they made a
| good faith effort to comply, and you've done nothing but
| act like a jerk and project your issues all over this
| thread. If I ran an ADS-B aggregator that was based on the
| same software as a half-dozen others, which hadn't seen any
| significant innovation in years, sitting on tons of data
| with absolutely amazing potential, I might consider
| praising the person who just revolutionized the world of
| ADS-B analysis & visualization, and possibly begging them
| to help me. And "P.S., could you throw in [specific
| attribution text]."
| johtso wrote:
| This is beautiful! Definitely worth looking at the examples in
| the Github repo:
|
| https://github.com/ClickHouse/adsb.exposed/
|
| I particularly like the example of helicopters following the
| river Thames in London:
|
| https://github.com/ClickHouse/adsb.exposed/?tab=readme-ov-fi...
| JosephRedfern wrote:
| Very pretty! As it happens, I too have started ingesting ADS-B
| Data into ClickHouse recently, but have nothing nearly as
| beautiful as this.
|
| I'm hitting the airplanes.live API every 10 seconds using
| ClickHouse's URL table function and storing in a MergeTree:
| https://github.com/JosephRedfern/airhoover/blob/main/airhoov....
| Would love to use refreshable materialised views for this, but at
| the moment there's no append functionality (refresh only), so
| have to use Python to to trigger the query.
|
| There's an open instance here:
| https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU....
| Only 1.5 days or so of data, I truncated before setting up tiered
| storage (local disk + backblaze b2).
|
| Cool being able to e.g. get a break-down of aircraft type by
| operator
| (https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU...)
| or who (supposedly) flies their planes fastest
| (https://airhoover.joesstuff.co.uk/play?user=default#U0VMRUNU...)
| kqr wrote:
| Wow. I have no idea what I'm looking at but it's pretty. And I do
| recognise the extended centrelines of local runways, as well as
| the typical instrument approach tracks into them!
|
| Edit: Having read the readme, I have a better idea of what I'm
| looking at. Really impressive technically as well.
| mmaunder wrote:
| Im surprised to see military traffic over the US. They tend to
| fly with ADSB out turned off and have an agreement with FAA to be
| able to do that. Also surprised to see gliders who generally
| don't broadcast ADSB out. In the US, we are required to have ADSB
| out within most controlled airspace and within the mode C veil of
| a major airport (within 30nm and up to 10,000 MSL). So most GA
| planes have it but remote areas have planes without ADSB out.
| nick238 wrote:
| For general transport/rebasing, it seems like it would just
| make life easier on air traffic controllers, and probably also
| other traffic in the air (I think TCAS uses ADS-B?). By way of
| analogy, the military might have permission to drive around at
| night with their lights off, but it'd make me more comfortable
| if they did that only when it was operationally useful (i.e. I
| assume B-2s taking off from Missouri to go bombing have
| transponders off the whole way.)
| larrywright wrote:
| One of the "tells" for military aircraft flying without ADSB
| is when you see refueling tankers (KC-135) doing loops for
| periods of time, and no other military aircraft around. Those
| tankers are refueling _something_ , they don't just fly
| around for grins.
| jjwiseman wrote:
| Fighters often don't have ADS-B, but there are lots of other
| military aircraft that routinely use it (transports, etc.). Go
| to https://globe.adsbexchange.com and press U to show only
| military.
| petee wrote:
| I'd read about small retro reflectors they bolt on when not
| in combat so the stealth fighters will show on domestic
| radars
| petee wrote:
| Edit: they're called Luneburg Lenses
| EvanAnderson wrote:
| My town, north of Dayton, OH was recently overflown by a
| group of fighters (I believe F-18's). I was surprised that
| they didn't appear on the ADS-B tracker site I ran inside and
| looked at. I guess it makes sense.
|
| I'm just a couple of miles from DAY and see a lot of traffic
| every day. It would be interesting to know how military
| aircraft like that coordinate with civilian air traffic
| control.
|
| (I like to listen to Dayton approach while watching an ADS-B
| tracker site. I enjoy seeing the traffic fly over my
| neighborhood. I find it oddly amusing to look up at a plane I
| just heard getting clearance to land knowing that I just
| heard the voice of somebody up there thru my speakers. I
| don't know why it's so pleasing...)
| fullstop wrote:
| I tracked, briefly, a F15E flying past yesterday.
| mshockwave wrote:
| I'm surprised about this, I thought it's pretty easy to add
| ADS-B to modern IFF module and they can just switch between
| different modes.
| fullstop wrote:
| I started playing around with ADS-B and military aircraft are
| all over the place, at least on the east coast of the USA.
|
| In fact, this Seahawk just flew over:
| https://globe.airplanes.live/?icao=ae6904
| larrywright wrote:
| If you look at https://globe.adsbexchange.com, and filter for
| military/interesting (the U button at the top of the screen),
| you'll see that at any given time there are a LOT of military
| flights over the US. Most are transport, refueling tankers and
| what seem to be pilots-in-training (mostly in Texas and
| Florida, some in Colorado), but there are plenty of helicopters
| and smaller Lear-type jets. When the president or vice
| president is flying somewhere there are typically one or more
| E-3 AWACS planes in the air to provide radar coverage.
|
| There are certainly military planes that fly without ADSB, but
| for flights where secrecy doesn't matter, they seem to fly with
| it on. I've seen all manner of planes with ADSB, from U2 spy
| planes, F-15, F-16, A-10, the occasional B-52, and more.
| secondcoming wrote:
| Those Reaper drones show up on FlightRadar24 and ADSBExchange
| while patrolling over the Black Sea monitoring Crimea and
| beyond.
| morkalork wrote:
| Sometimes they forget to turn it off:
|
| https://theaviationist.com/2023/11/22/usaf-ac-130j-iraq/
| rlpb wrote:
| That article states that it's unlikely that they forgot, and
| much more likely that it was deliberate.
| morkalork wrote:
| Interesting, when it happened at the time the theory was
| they forgot.
| yegle wrote:
| Sometimes I feel lucky that human's electromagnetic spectrum
| perception is very limited to the so called "visible spectrum".
|
| Imaging alien species that evolved to perceive a wider
| electromagnetic spectrum, the earth must look like a disco ball
| when their spaceship approaches.
| kqr wrote:
| Much like we're lucky to have eyes that happen to see the sun's
| strongest frequencies?
| throwaway743 wrote:
| This is fantastic!
| nickphx wrote:
| Is it possible to export a report of traffic by a selected
| region?
| zX41ZdbW wrote:
| Yes, it is possible to connect to the database directly and run
| an arbitrary query. Example: $ clickhouse-
| client --user website --host kvzqttvc2n.eu-
| west-1.aws.clickhouse-staging.com clickhouse-cloud
| :) SELECT t, desc, count() AS c FROM planes_mercator_sample100
| GROUP BY ALL ORDER BY c DESC LIMIT 10
| +-t----+-desc----------------------------+--------c-+
| 1. | B738 | BOEING 737-800 | 51530781 |
| 2. | A320 | AIRBUS A-320 | 37196762 |
| 3. | C172 | CESSNA 172 Skyhawk | 20049393 |
| 4. | A321 | AIRBUS A-321 | 19983151 |
| 5. | A20N | AIRBUS A-320neo | 14938832 |
| 6. | B38M | BOEING 737 MAX 8 | 14200826 |
| 7. | B737 | BOEING 737-700 | 13929403 |
| 8. | A319 | AIRBUS A-319 | 13906164 |
| 9. | E75L | EMBRAER ERJ-170-200 (long wing) | 12006441 |
| 10. | A21N | AIRBUS A-321neo | 10965047 |
| +------+---------------------------------+----------+
|
| Add FORMAT CSV to output in CSV (or any other format).
|
| To obtain an SQL query for a particular region, you can open
| the browser dev tools (F12), switch to Network, and copy a
| particular request that is made when you select an area with
| the rectangle selection tool.
| j1897 wrote:
| Super cool! Visually dazzling.
|
| If you want to build something similar on a raspberry pi, here is
| a tutorial: https://questdb.io/blog/create-flight-radar-
| raspberry-pi-que...
| consumer451 wrote:
| The one glider path over the LAX area is interesting.
|
| What was going on there?
| sllabres wrote:
| I think what you see is CIVIL AIR PATROL and CIVIL AIR PATROL
| INC
|
| Types: GLID (SGS 2-33A)
|
| Flights: N7589, N2037T
|
| Registration: N7589, N2037T
|
| When you select the dottet square button in the lower left and
| select a rectangle the planes within this rectangle are listed
|
| Very interesting project!
| consumer451 wrote:
| Oh, thanks. I somehow missed that UI.
|
| The glider I was thinking about is actually N914SF, a
| Pipistrel Sinus. That's a motorized glider, which makes a lot
| more sense flying though Class D airspace, right above LAX.
| jjwiseman wrote:
| It's like writing webgl shaders for ADS-B. In SQL.
|
| Incredible work, and I hope this kicks off a lot of innovation in
| the world of aircraft traffic analysis & visualization, which I
| think has been kind of stuck in a rut for a while.
| spdustin wrote:
| The "strange hole near Mexico City" example in the Github repo's
| README is another volcano.
| ptero wrote:
| This is fantastic! Is the airplanes.live related to ADS-B
| exchange or is this an unrelated effort?
___________________________________________________________________
(page generated 2024-04-10 23:00 UTC)