[HN Gopher] Show HN: IPDetective - An API for IP bot detection
___________________________________________________________________
Show HN: IPDetective - An API for IP bot detection
IPDetective collects data from about 60+ different sources such as
official cloud provider endpoints and public VPN/Proxy/Tor/Bot net
lists. Then aggregates this data into a fast and easy to use API
that can be integrated into applications or scripts easily.
IPDetective started as a hobby project for my other hobby projects
:) and I decided to wrap a simple website around and offer it as a
service. Let me know what your thoughts, if you find value in this
service or if you have any feature requests.
Author : AndrewCopeland
Score : 29 points
Date : 2022-09-12 19:57 UTC (1 days ago)
(HTM) web link (ipdetective.io)
(TXT) w3m dump (ipdetective.io)
| bootsmann wrote:
| How do you plan to keep the list up to date and especially
| reliably accurate? (providers change, bot nets get cleaned, ASN
| especially in 3rd world countries are very unreliable, crawlers
| might scrap weird things etc.)
|
| Disclaimer: I work for a company that operates in a similar space
| so don't go into too much secret sauce if you don't want to :)
| AndrewCopeland wrote:
| So I have a scraper that gathers the information from about 60
| different sources using a wide array of techniques which is run
| everyday.
|
| The largest thing I try to look at is to make sure my sources
| are not older than 1 year. Sometimes these sources break and I
| update them or they go offline all together. I am always
| looking for new hosting providers, vpn and proxies.
|
| Also I can do further port analytics on these ips as well. A
| lot of the VPN services have specific ports open used by VPN
| services. I would say that automating every source from the
| start was a good way to start so I can stay up to date.
| stevenicr wrote:
| I checked the posted privacy policy - and I have no idea what
| your system logs and for how long.
|
| There are a few sites I could use this for, but some of them
| would also end up sending private customer ips for lookups, and
| for that reason I just manually check the suspicious ones and
| manually don't look up the ones I know have passed the 'already a
| member and human not hacker' gate.
|
| I keep look for something I can just add lists to my server and
| check against that, but not going to spend thousands on it.
| marvinblum wrote:
| Same here. I would like to integrate a simple blacklist into
| our system. Does anybody know of a good IP lists that contains
| known bots, data centers, and so on?
|
| Also from a performance perspective, I can't do hundreds of
| millions of requests a month over the wire. That just a waste.
| bragr wrote:
| >Let me know what your thoughts
|
| Do you have any data showing your service is better/more
| accurate/better false positive rate than competing offerings?
|
| >IPDetective collects data from about 60+ different sources such
| as official cloud provider endpoints and public VPN/Proxy/Tor/Bot
| net lists.
|
| Are you on the up and up with all those public sources? I'm not
| sure which ones you're sourcing, but many do not allow commercial
| use, resale, or at the very least have some attribution clause to
| keep security companies from mooching off crowdsourced data.
|
| >No, currently IPDetective does not support ipv6 addresses.
| However this feature is on the road map.
|
| That's a major shortcoming in 2022.
|
| Finally, as itake pointed out bellow, I'm not giving you my email
| just to run a simple test query and see what the results look
| like.
| AndrewCopeland wrote:
| Thanks for your input.
|
| >Do you have any data showing your service is better/more
| accurate/better false positive rate than competing offerings
|
| I have signed up to some of the competitors and my service
| averaged about 20ms per request for a single IP address. From
| the competition it was typically around 200ms to 300ms.
| Regarding better false positives that would be rather hard to
| do.
|
| > Are you on the up and up with all those public sources?
|
| I gathered from sources that did not have any licensing related
| to non-commercial use. Would you recommend I reach out to all
| of the sources regardless even if they say you can use it for
| commercial use? Or do not say anything about commercial use at
| all?
|
| > No, currently IPDetective does not support ipv6 addresses.
| However this feature is on the road map.
|
| Yeah I know, just focusing on IPv4 right now. I am still
| collecting ipv6 addresses however I have not implemented it in
| the service yet.
|
| Thanks a bunch for writing the comments above.
| kosolam wrote:
| Yeap... measure the amount of requests for ipv6 support you
| get from your users, not hn users :-)
| bragr wrote:
| https://www.google.com/intl/en/ipv6/statistics.html
| itake wrote:
| 1/ needs authenticationless way to search. all other ip look up
| tools don't require an account for me to test this. I'm not going
| to register.
|
| 2/ I've tried blocking vpn or proxy users (by ip hosting
| provider), but found too many false positives. Using a VPN is
| common for IT professionals or consumers trying to 'protect'
| their privacy and this impacted the growth of my app.
|
| How is your tool better at detecting a bot vs a human using a
| vpn?
| AndrewCopeland wrote:
| >1/ needs authenticationless way to search. all other ip look
| up tools don't require an account for me to test this. I'm not
| going to register.
|
| Okay I will look into a solution.
|
| 2/ I've tried blocking vpn or proxy users (by ip hosting
| provider), but found too many false positives. Using a VPN is
| common for IT professionals or consumers trying to 'protect'
| their privacy and this impacted the growth of my app.
|
| That good to know, I do not find that I have a lot of false
| postivies but I would imagine it all depends on audience.
|
| >How is your tool better at detecting a bot vs a human using a
| vpn?
|
| It does not know the difference as or right now. I was thinking
| about adding a user-agent validation as well which could add
| another layer.
| michaelmior wrote:
| > I was thinking about adding a user-agent validation as well
| which could add another layer.
|
| Presumably this service exists because bots try to avoid
| detection. I don't think UA validation would really help much
| and there are plenty of libraries already that do this.
| joshmn wrote:
| Does this handle residential or LTE proxies? Obviously harder to
| identify, but it's not unfeasible.
| AndrewCopeland wrote:
| Currently we collect the data across several free public proxy
| lists. Definitely more difficult to do wit the proxies you
| mentioned. I think traffic analytics and port scanning would
| need to be implemented to get these residential and LTE
| proxies.
| mousetree wrote:
| Good work on the pricing and free tier. Much cheaper than your
| competitors (we're busy testing out ipinfo.io). But would
| recommend you add a bit more data in your API response if you
| already have it or can built it out easily. For example, any
| geocoding, ASN name, whether its a proxy or a VPN etc.
| AndrewCopeland wrote:
| Please feel free to leave comments if you have any questions or
| any feedback.
| mxuribe wrote:
| I didn't see my question in the FAQ, so asking here: do you have
| a process to contest accidental inclusion of an IP address?
| AndrewCopeland wrote:
| I do not, but it seems like a great feature request. I will
| look into adding something like this.
| mxuribe wrote:
| Thanks so much @AndrewCopeland!
|
| I should have added context (or at least an anecdote) to help
| you during any product roadmap meetings...I used to oversee
| web ops for global real estate company (one of the biggest in
| the U.S. and the woirld)...and our consumer-facing websites
| would show tons and tons and tons of listings of residential
| homes for sale. Of course we really were just showing
| listings from our data store as well as other real estate
| companies who agreed to share listings data. As in many data
| sharing and data synch arenas, there are data issues. The
| most common scenario: "hey, you're showing a home that sold X
| time ago...stop showing it!" And, in all cases it was
| "someone else's data". But to the customer, or even other
| realtors, we were the bad guys. Even realtors who should have
| known betteer that there are always data issues that we can
| not fully control; at least nopt at the source...would
| complain to us. Now, one might assume that once the source
| data updates properly, the "correct data" should flow
| through, right? Well, not in real estate! Clearly this is a
| different arena. But i've learned that even if its only to
| remove a local cahce of data, its a good idea to give users
| and/or customers a mechanism to at least properly communicate
| to you about stale data...and if appropriate, maybe evenallow
| self-service for a user/customer to get rid of the "bad
| data". Obviously this merits putting in place protocols to
| avoid abuse...but i hope you get the idea. Good luck!
___________________________________________________________________
(page generated 2022-09-13 23:01 UTC)