https://github.com/facundoolano/ngtop Skip to content Navigation Menu Toggle navigation Sign in * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + GitHub Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions By size + Enterprise + Teams + Startups By industry + Healthcare + Financial services + Manufacturing By use case + CI/CD & Automation + DevOps + DevSecOps * Resources Topics + AI + DevOps + Security + Software Development Explore + Learning Pathways + White papers, Ebooks, Webinars + Customer Stories + Partners * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Enterprise + Enterprise platform AI-powered developer platform Available add-ons + Advanced Security Enterprise-grade security features + GitHub Copilot Enterprise-grade AI features + Premium Support Enterprise-grade 24/7 support * Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search [ ] Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. [ ] [ ] Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name [ ] Query [ ] To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} facundoolano / ngtop Public * Notifications You must be signed in to change notification settings * Fork 2 * Star 214 Request analytics from the nginx access logs License GPL-3.0 license 214 stars 2 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings * Code * Issues 8 * Pull requests 0 * Actions * Projects 0 * Security * Insights Additional navigation options * Code * Issues * Pull requests * Actions * Projects * Security * Insights facundoolano/ngtop This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main BranchesTags Go to file Code Folders and files Name Name Last commit Last commit message date Latest commit History 92 Commits .github/workflows .github/ workflows ngtop ngtop .gitignore .gitignore LICENSE LICENSE Makefile Makefile README.md README.md go.mod go.mod go.sum go.sum main.go main.go main_test.go main_test.go View all files Repository files navigation * README * GPL-3.0 license ngtop ngtop is a command-line program to query request counts from nginx's access.log files. $ ngtop url user_agent --since 1d --where url=/blog/% --where status=200 --limit 5 PATH USER_AGENT #REQS /blog/deconstructing-the-role-playing-videogame/ Safari 120 /blog/on-ai-assistance/ Go-http-client 101 /blog/a-note-on-essential-complexity/ Safari 91 /blog/deconstructing-the-role-playing-videogame/ Chrome 84 /blog/from-rss-to-my-kindle/ Safari 79 Installation Download the latest release binary for your platform, for example: $ wget https://github.com/facundoolano/ngtop/releases/latest/download/ngtop-linux-arm64 \ -O ngtop && chmod +x ngtop && mv ngtop /usr/local/bin Alternatively, install with go: $ go install github.com/facundoolano/ngtop@latest Usage examples Count requests from the last hour: $ ngtop --since 1h $ ngtop -s 1h $ ngtop Count requests from the last second, minute, day, week, or month: $ ngtop -s 1s $ ngtop -s 1m $ ngtop -s 1d $ ngtop -s 1w $ ngtop -s 1M Count requests from the day before: $ ngtop --since 2d --until 1d $ ngtop -s 2d -u 1d Show the top 5 urls in the last hour: $ ngtop url $ ngtop path Show the top 5 urls in the last minute: $ ngtop url -s 1m Show the top 10 urls in the last hour: $ ngtop url --limit 10 $ ngtop url -l 10 Count total requests to a specific url in the last hour: $ ngtop --where url=/blog/code-is-run-more-than-read $ ngtop --where path=/blog/code-is-run-more-than-read $ ngtop -w url=/blog/code-is-run-more-than-read Count total requests to urls matching a pattern: $ ngtop -w url=/blog/% Count total requests to urls excluding a value or pattern: $ ngtop -w url!=/feed.xml $ ngtop -w url!=/feed% Count total requests to one of mutliple urls (one OR another): $ ngtop -w url=/blog/code-is-run-more-than-read -w url=/blog/a-note-on-essential-complexity Count total requests to a specific urls AND referer: $ ngtop -w url=/blog/code-is-run-more-than-read -w referer=news.ycombinator.com Show the top visited urls matching a pattern: $ ngtop url -w url=/blog/% Show the top requesting ips: $ ngtop ip Show the top url visits by ip: $ ngtop url -w ip=77.16.76.86 Show the top user agents by url: $ ngtop user_agent -w url=/blog/code-is-run-more-than-read $ ngtop ua -w url=/blog/code-is-run-more-than-read Show the top urls by user agent details (parsed with mileusna/ useagent): $ ngtop url -w ua=Firefox $ ngtop url -w ua_type=bot $ ngtop url -w device=iPhone $ ngtop url -w os=Linux Show the top referers for a url pattern: $ ngtop referer -w url=/blog/% Show the top user agent and referer combination $ ngtop ua referer Show the top user agent and referer combination for a specific url $ ngtop ua referer -w url=/blog/code-is-run-more-than-read Count total 404 status responses: $ ngtop -w status=404 Count non successful responses: $ ngtop status -w status=4% -status=5% How it works * Whenever the program is run, it looks for the nginx access.logs, parses them and stores the data into an SQLite DB. + By default, the logs are looked up at /var/log/nginx/ access.log*, which can be overridden with the NGTOP_LOGS_PATH environment variable. + By default, the logs are assumed to have the nginx combined log format. The format can be customized with NGTOP_LOG_FORMAT. o This could likely be made to work with non nginx logs, although that hasn't been tested. + Subsequent runs of the program only parse and store the logs up until the time of the previous run. + The SQLite DB is stored at ./ngtop.db, which can be overridden with the NGTOP_DB environment variable. * The command line arguments express a filtering criteria, used to build the SQL query that counts the requests. + For instance, the command ngtop url -w url=/blog/% produces: SELECT path,count(1) '#reqs' FROM access_logs WHERE time > ? AND time < ? AND (path LIKE ?) GROUP BY 1 ORDER BY count(1) DESC LIMIT 5 Configuration The command-line arguments and flags are intended exclusively to express a requests count query. The configuration, which isn't expected to change across command invocations, is left to environment variables: * NGTOP_LOGS_PATH: path pattern to find the nginx access logs. Defaults to /var/log/nginx/access.log*. The pattern is expanded using Go's path/filepath.Glob. * NGTOP_LOG_FORMAT: The nginx log_format specification to parse the log entries. By default combined logs are assumed, which is equivalent to: NGTOP_LOG_FORMAT='$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"' * NGTOP_DEBUG: when set, internal logs will be printed to standard output. * NGTOP_DB: location of the SQLite db where the parsed logs are stored. Defaults to ./ngtop.db. About Request analytics from the nginx access logs Topics nginx cli analytics metrics self-hosted Resources Readme License GPL-3.0 license Activity Stars 214 stars Watchers 3 watching Forks 2 forks Report repository Releases 5 v0.4.1 Latest Jul 31, 2024 + 4 releases Packages 0 No packages published Languages * Go 97.3% * Makefile 2.7% Footer (c) 2024 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact * Manage cookies * Do not share my personal information You can't perform that action at this time.