https://grayhatwarfare.medium.com/how-to-search-urls-exposed-by-shortener-services-f68e199cd560 Get started Open in app GrayhatWarfare 195 Followers About Follow Sign in Get started Follow 195 Followers About Get started Open in app How to search URLs exposed by Shortener services GrayhatWarfare GrayhatWarfare 3 days ago*3 min read Almost two years after the release of our Buckets Search tool, we are happy to announce the release of http://shorteners.grayhatwarfare.com , the second tool in the GrayhatWarfare arsenal, that allows you to search exposed Shortened Urls. Image for post shorteners.grayhatwarfare.com - Shortened URLs search engine TinyURL, bit.ly, and other similar services allow long URLs to be converted to smaller ones hosted on their service; the small URL is visited by a consumer and their web browser is redirected to the long URL. Common use-case is Twitter where the number of characters is limited but also emails and DMs. There is always the risk, that a Shortener service, for whatever reason might stop delivering urls. Teams like 301Work and URLTeam are trying to mitigate the risk. URLTeam is periodically visiting the urls and archiving the results. The design of Shortener services made this process easier, because they use the minimum characters to accommodate the number of links they host, meaning the number of possible urls is really low and can be brute forced. There are some security risks associated with Shortener services. First, you never know where you are redirected when you click a shortened url -- there could be a malicious url on the other end. Second Shortener services can track users and their behavior. Also there are other risks when shortening urls that should remain private. Those include things like: * Links to shared/unlocked files in Cloud services (Dropbox, Google Drive, OneDrive). OneDrive in particular used to generate short URLs for documents and folders using the 1drv.ms domain. This is a "branded short domain" operated by Bitly and uses the same tokens as bit.ly. Searching by any cloud service domain (dropbox.com, drive.google.com), reveals a lot of downloadable files. This reveals files and directories and some of them even have write access. * Links to locations (Google Maps or other maps/location services), which often contain personal information, or routes to sensitive areas. * Links to internal/hidden but accessible urls with sensitive information. Those could be, Admin panels, Hidden directories, or private documents * Accidentally public trello.com boards or other similar services. It's a common practice for pentesters to search for sensitive urls, or gather information for a domain name, through Shortened urls. Utku Sen did an excellent job creating https://github.com/utkusen/ urlhunter, which is the most known tool that tackles the problem of searching through the urls. However there are some limitations: * The tool needs to download the files from Archive.org before it's operational. This process is rather slow because Archive.org limits the bandwidth. * The user must have 500GB of free disk space available to save all data. * Filtering the results is relatively slow. * It's a command line tool which might be intimidating for some people. * Most of the urls are inactive (404 links, expired domains etc). In order to improve the search experience we: * Removed urls from expired domains. * Removed all non existing urls (404) * Removed urls from parked domains, that falsely respond with 200 to all requests. * Deduced content-type from the contents of all urls and guessed the file extension. The functionality we offer is: * Filter by keywords * Filter by extension, calculated from the contents of the url and not its url extension. * Allow to filter by domain and keywords. * Sort by size when known. * Sort by discovery date, based on URLTeam release date. Our primary goal for the first release was to clean up as much junk as possible and create a robust filtering mechanism while keeping the tool as minimal as possible. As always we want your input on what functionality you want to see in the tool. shorteners.grayhatwarfare.com is the second tool we released and we have a lot more on the way. Follow us on twitter to get announcements for new tools and updates first: Thanks for your support, it makes delivering new tools feasible. GrayhatWarfare Follow 3 3 3 * Penetration Testing * Url Shorteners * Hacking * Leak Detection * Cybersecurity More from GrayhatWarfare Follow More From Medium Communicating through UUID conflicts Evin Sellin [1] [1] Rocket 2 and the importance of good software development practices, documentation, and testing Cheuk Yin Ng in UBC Launch Pad Software Engineering [0] [0] Machine Learning Powered Data Pipeline Kai Sahling in The Startup [1] [1] RipTrace: Debugging Microservice and Container Applications Cooper Thompson in The Startup [1] [1] Tips for Running End to End Testing for Web Frameworks Computer Software [1] [1] How does event storming help in understanding processes/systems better Sreenidhi Chandar in Mind Boggler [1] [1] GoodReads API from Android with Kotlin Andrey Suvorov [1] [1] Handling Databases from a Developer's Perspective Nikolai Averin in Miro Engineering [0] [0] About Help Legal Get the Medium app A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store