https://github.com/PostHog/HouseWatch Skip to content Toggle navigation Sign up * Product + Actions Automate any workflow + Packages Host and manage packages + Security Find and fix vulnerabilities + Codespaces Instant dev environments + Copilot Write better code with AI + Code review Manage code changes + Issues Plan and track work + Discussions Collaborate outside of code Explore + All features + Documentation + GitHub Skills + Blog * Solutions For + Enterprise + Teams + Startups + Education By Solution + CI/CD & Automation + DevOps + DevSecOps Case Studies + Customer Stories + Resources * Open Source + GitHub Sponsors Fund open source developers + The ReadME Project GitHub community articles Repositories + Topics + Trending + Collections * Pricing [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. {{ message }} PostHog / HouseWatch Public * Notifications * Fork 12 * Star 212 Open source tool for monitoring and managing ClickHouse clusters License MIT license 212 stars 12 forks Star Notifications * Code * Issues 1 * Pull requests 1 * Actions * Projects 0 * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Security * Insights PostHog/HouseWatch This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. main Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags Name already in use A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch? Cancel Create 6 branches 0 tags Code * Local * Codespaces * Clone HTTPS GitHub CLI [https://github.com/P] Use Git or checkout with SVN using the web URL. [gh repo clone PostHo] Work fast with our official CLI. Learn more about the CLI. * Open with GitHub Desktop * Download ZIP Sign In Required Please sign in to use Codespaces. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching GitHub Desktop If nothing happens, download GitHub Desktop and try again. Launching Xcode If nothing happens, download Xcode and try again. Launching Visual Studio Code Your codespace will open once ready. There was a problem preparing your codespace, please try again. Latest commit @yakkomajuri yakkomajuri chore: update readme env vars ... 1021c3d Jun 14, 2023 chore: update readme env vars 1021c3d Git stats * 222 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time bin update bin/start to use celery March 29, 2023 13:29 frontend feat: support time ranges in query performance page June 13, 2023 13:12 housewatch feat: support time ranges in query performance page June 13, 2023 13:12 .gitignore feat: cleanup + make operations polling stop when user navigates away May 24, 2023 14:42 Dockerfile feat: support undefined cluster May 23, 2023 14:18 LICENSE chore: add LICENSE May 25, 2023 11:24 README.md chore: update readme env vars June 14, 2023 08:49 banner-light.png feat: yet another banner May 23, 2023 12:40 benchmark1.png chore: add query benchmarking feature screenshots May 26, 2023 15:51 benchmark2.png chore: update benchmark feature screenshot 2 (again) May 26, 2023 16:04 disk-usage.png feat: add screenshots May 23, 2023 11:26 docker-compose.yml chore: run redis on a more random port on the host May 24, 2023 15:21 errors.png feat: add screenshots May 23, 2023 11:26 explain.png feat: add screenshots May 23, 2023 11:26 logs.png feat: add screenshots May 23, 2023 11:26 manage.py initial March 28, 2023 20:55 mypy.ini initial March 28, 2023 20:55 normalized-query.png feat: add screenshots May 23, 2023 11:26 operations.png chore: add operations feature screenshot May 24, 2023 15:13 overview.png feat: add screenshots May 23, 2023 11:26 pyproject.toml initial March 28, 2023 20:55 pytest.ini initial March 28, 2023 20:55 query-editor.png feat: add screenshots May 23, 2023 11:26 query-stats.png feat: add screenshots May 23, 2023 11:26 requirements.txt feat: alpha natural language query editor (#7) June 9, 2023 14:47 running-queries.png feat: add screenshots May 23, 2023 11:26 runtime.txt initial March 28, 2023 20:55 schema-drilldown.png feat: add screenshots May 23, 2023 11:26 schema.png feat: add screenshots May 23, 2023 11:26 slow-queries.png feat: add screenshots May 23, 2023 11:26 View code [ ] Open source tool for monitoring and managing ClickHouse clusters Deploy Motivation [?] Status of the project [?] Contributing Features Query performance Schema stats Query benchmarking Logs Query editor Disk usage Errors Operations [?] To-do list README.md [banner-lig] [overview] Open source tool for monitoring and managing ClickHouse clusters * Get an overview of cluster load and performance * Drill down into your queries and understand the load they put on your cluster * Search through logs and errors * Monitor and kill running queries with the click of a button * Get stats on your disk usage per node, and understand how much disk space tables, columns, and parts take up * Run your own queries straight from the interface to further dig into performance and cluster issues * Setup operations to run in the background with automatic rollbacks for failures Deploy To deploy HouseWatch, clone this repo and then run the following, substituting the environment variables for the relevant values of one of your ClickHouse instances: CLICKHOUSE_HOST=localhost \ CLICKHOUSE_CLUSTER=mycluster \ CLICKHOUSE_USER=default \ CLICKHOUSE_PASSWORD=xxxxxxxxxxx \ docker compose -f docker-compose.yml up After running the above, the UI will be running on http:// localhost:3000. For production installs, you might want to setup something like Caddy or NGINX with a Let's Encrypt TLS certificate. Read more The following are the supported environment variables for configuring your HouseWatch deployment: * CLICKHOUSE_HOST: Required - hostname of the instance to connect to. * CLICKHOUSE_USER: Required - username to access ClickHouse. Can be a read-only user, but in that case not all features will work. * CLICKHOUSE_PASSWORD: Required - password for the specified user. * CLICKHOUSE_DATABASE: Optional - database to connect to by default. * CLICKHOUSE_CLUSTER: Optional - cluster name, to analyze data from the whole cluster. * CLICKHOUSE_SECURE: Optional - see clickhouse-driver docs for more information * CLICKHOUSE_VERIFY: Optional - see clickhouse-driver docs for more information * CLICKHOUSE_CA: Optional - see clickhouse-driver docs for more information * OPENAI_API_KEY: Optional - enables the experimental "AI Tools" page, which currently features a natural language query editor * OPENAI_MODEL: Optional - a valid OpenAI model (e.g. gpt-3.5-turbo, gpt-4) that you have access to with the key above to be used for the AI features Motivation At PostHog we manage a few large ClickHouse clusters and found ourselves in need of a tool to monitor and manage these more easily. ClickHouse is fantastic at introspection, providing a lot of metadata about the system in its system tables so that it can be easily queried. However, knowing exactly how to query and parse the available information can be a difficult task. Over the years at PostHog, we've developed great intuition for how to debug ClickHouse issues using ClickHouse, and HouseWatch is the compilation of this knowledge into a tool. Beyond monitoring, we also built internal systems and processes for managing the clusters that spanned various platforms. We would use Grafana to look at metrics, SSH into nodes for running operations and using specialized tooling, query via Metabase to dig deeper into the data in the system tables and create dashboards, and then a combination of tools baked into the PostHog product for further debugging and streamlined operations such as our async migrations tool, and internal views for listing queries and analyzing their performance. As a result, we felt it was appropriate to have these tools live in one place. Ultimately, our vision for HouseWatch is that it can both serve the purpose of a pganalyze for the ClickHouse ecosystem, while also including tooling for taking action on insights derived from the analysis. [?] Status of the project HouseWatch is in its early days and we have a lot more features in mind that we'd like to build into it going forward. The code could also use some cleaning up :) As of right now, it is considered Beta software and you should exercise caution when using it in production. One potential approach is to connect HouseWatch to ClickHouse using a read-only user. In this case, the cluster management features will not work (e.g. operations, query editor), but the analysis toolset will function normally. [?] Contributing Contributions are certainly welcome! However, if you'd like to build a new feature, please open up an issue first. Features Query performance [slow-queries] [normalized-query] [query-stats] [explain] Schema stats [schema] [schema-drilldown] Query benchmarking [benchmark1] [benchmark2] Logs [logs] Query editor [query-edit] Disk usage [disk-usage] Errors [errors] Operations [operations] [?] To-do list A public list of things we intend to do with HouseWatch in the near future. See list Features * [ ] System issues tab * [ ] EXPLAIN visualizer * [ ] Multiple instance support * [ ] Stats on page cache hit percentage * [ ] Make operations resilient to Celery going down (as we do in PostHog with async migrations) * [ ] Read-only mode * [ ] Button to force refresh running queries list * [ ] Logs pagination * [ ] Allow copying example queries * [ ] Configurable time ranges * [ ] Whole cluster schema stats * [ ] More operation controls: view, delete, edit, re-run, display errors Developer experience * [ ] Configure instance from UI * [ ] Publish a Docker image * [ ] Development docker-compose.yml with baked in ClickHouse Cleanup * [ ] Extract README images out of repo * [ ] Make banner subtitle work on dark mode * [ ] Fetch data independently on the query analyzer * [ ] Breakpoint for logs search * [ ] Run Django "production server" * [ ] Write tests :) * [ ] Query editor pipe all errors to client * [ ] Abstraction to load data from API as JSON About Open source tool for monitoring and managing ClickHouse clusters Topics react open-source django database clickhouse database-management mit-license Resources Readme License MIT license Stars 212 stars Watchers 13 watching Forks 12 forks Report repository Releases No releases published Packages 0 No packages published Contributors 4 * @yakkomajuri yakkomajuri Yakko Majuri * @timgl timgl timgl * @liyiy liyiy Li Yi Yu * @sindhubb sindhubb Sindhu S Languages * TypeScript 60.9% * Python 37.1% * Other 2.0% Footer (c) 2023 GitHub, Inc. Footer navigation * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time.