https://tembo.io/blog/pgxn-ecosystem-jobs tembo log HomeDocsBlogGithub Try Free arrow pointing left Go Back The Jobs to be Done by the Ideal Postgres Extension Ecosystem Feb 21, 2024 * 6 min read David E. Wheeler David E. Wheeler Principal Architect blog post hero image The past year has seen a surge in interest in Postgres extension distribution. A number of people have noted in particular the challenges in finding and installing extensions. PGXN, released way back in 2011, aimed to be the canonical registry for the community, but currently indexes only the 350 distributions that developers have made the effort to publish -- perhaps a third of all known extensions. Furthermore, PGXN distributes source code packages, requiring developers to download, compile, install, and test them. As a result of these challenges, a number of new entrants have emerged in recent months with a focus on discovery and ease of installation. These include dbdev, pgxman, pgpm, and our own trunk. However, they too exhibit limitations, such as minimal OS and platform availability and, notably, the manual process to add extensions, further constraining inclusion. None has even reached the number provided by PGXN. These challenges and the interest and energy put into exploring new solutions make clear that the time has come to revisit the whole idea of the PostgreSQL extension ecosystem: to work though the jobs to be done, specify the tools to do those jobs, and outline a plan for the broader Postgres community to design and build them. Future posts will dream up the tools and make the plan; today we begin with the jobs. Let's get started. --------------------------------------------------------------------- Jobs to be Done Theodore Levitt cites a famous adage: "Last year 1 million quarter-inch drills were sold," Leo McGivena once said, "not because people wanted quarter-inch drills but because they wanted quarter-inch holes." People don't buy products; they buy the expectation of benefits. Today the most cited descriptor underpinning Jobs to be Done Theory, thanks to Clayton Christensen, we bear it in mind to think through the "jobs" of an idealized Postgres extension ecosystem -- without reference to existing Postgres solutions. However, we cite examples for the jobs from other projects and communities, both to clarify the jobs to be done and to reference prior art from which to borrow. So what are the jobs to be done by the ideal Postgres extension ecosystem? Authoritative Job: Be the official, canonical source of record for all Postgres extensions Developers need to know the one place to reliably release and find all publicly-available Postgres extensions. It can't be a subset, but must be comprehensive, the assumed default recognized and used by the community and included in the Postgres documentation. Exemplars: * The Go Module Index indexes Go package that it automatically discovers * crates.io indexes Rust crates published to it * PAUSE indexes Perl modules uploaded to it * RubyGems indexes Ruby Gems published to it * PyPI indexes Python distributions uploaded to it Beneficial Job: It's expected for extension developers to develop and publish their extensions in the ecosystem. Extension developers want their extensions to be a part of the ecosystem, to gain all the advantages and privileges of participation, without undo effort. Publishing an extension should be as straightforward as possible. Ideally it simply requires maintaining a metadata file and publishing releases to the authoritative index. Exemplars: * Go Package Discovery automatically discovers and indexes go.mod -configured packages * Registered crates.io developers publish Rust crates defined by Cargo Manifest Files via the cargo CLI * Registered NPM developers publish NPM projects defined by package.json files via the npm publish command * Registered PAUSE developers publish Perl modules defined by CPAN Meta Spec files via the cpan-upload CLI or web API Integrated Job: It's easy for developers to start writing and publishing extensions. Extension developers need excellent tooling to simplify the process of creating, maintaining, testing, and distributing extensions. Such tools should be well-integrated into the extension ecosystem, and empower developers to get started and easily follow best practices throughout the extension development lifecycle: from beginning development, managing dependencies and other metadata, testing it, and releasing it. Exemplars: * cargo * Go module-aware commands * Language Servers Discoverable Job: Make extensions easy to find and understand There's no use in being the canonical index of extensions if no one can find them. The ecosystem needs a well-known web site where people find extensions, learn their purposes, distinguish them from each other, and read their docs. Ideally most people use integrated search rather than a search engine, but extension pages should appear high up in search engine results, too. Exemplars: * pkg.go.dev * crates.io * metacpan.org * rubygems.org * pypi.org Informative Job: Make extension documentation consistent, standardized, and comprehensive Extensions are only as easy to use as their documentation enables. It's therefore important to provide well-organized, standardized, and accessible documentation -- and that it be expedient for developers to document their extension packages. Ideally development tools encourage extension documentation. A documentation standard enables publishing docs in multiple formats, especially on the canonical discovery site. Exemplars: * Go Doc Comments * rustdoc Programmable Job: Provide stable, easy-to-use, and comprehensive APIs to empower downstream developers to build and integrate other tools and products Provide excellent, well-documented, intuitive web APIs that allow anyone to build useful tools and services on the index. The jobs below require it, but creative coders will invent and mashup APIs in ways we haven't thought of. Web APIs enable imaginative solutions within the ecosystem and without. Some example APIs: * Recent releases feeds * Extension metadata * Full text search * Event webhooks to notify registered services Better still, let trusted services report back results for listing -- even badging! -- in the authoritative index, such as build results, test results, binary packaging availability, and more. Exemplars: * metacpan-api * RubyGems API * PyPI API * PyPI project releases RSS feed Installable Job: Provide automatic binary packaging for wide variety of platforms It must be as easy as possible for users to install all indexed extensions. Ideally, independent services rely on the index APIs for notification of new releases and automatically build binary packages for as wide a variety of OSes, OS versions, architectures (arm64, amd64, etc.), and PostgreSQL versions as possible. Some example repositories to support or enable to build and distribute extensions from the core registry: * Community packaging: apt.postgresql.org, yum.postgresql.org * Binary packaging: Homebrew, Chocolatey, etc. * Vendor packaging & distribution: pgxman, dbdev, pgpm, trunk Exemplars: Trusted Job: Validate extensions and protect from supply chain vulnerabilities Provide tools and features to earn and retain user trust by ensuring that that extensions indexed by the canonical registry have been validated, come from trustworthy sources, and are unlikely to be a vector for supply chain attacks. This means some combination of author validation (e.g. badging projects from well-known sources), checksums, public key signing, and perhaps static code analysis. Exemplars: * How Go Mitigates Supply Chain Attacks * Improving Supply Chain Security for Rust Through Artifact Signing * GitHub Code Scanning * Docker Scout Manageable Job: Provide intuitive, powerful interfaces for installing and managing extensions Provide applications to manage the extensions in one or more Postgres clusters. At a minimum distribute a command line interface that manages extensions for a locally-installed Postgres -- as currently required for it to install packages, for example. Users need to easily manage their Postgres extensions in a single, consistent manner. The extensions manager should be aware of installed extensions, available extensions, dependency graphs, and known vulnerabilities; allow for search (via the APIs); and of course install or upgrade extensions. Exemplars: * cpan, cpanminus, cpm * cargo * Go module-aware commands * pip * gem * apt-get * yum Omissions and Next Steps With these jobs in mind, we begin to identify the tools necessary to serve them in the next post. But first, what's missing here? Are these nine jobs to be done sufficiently comprehensive to meet most extension indexing, discovery, packaging, and distribution needs? What's missing? We don't want to leave any good ideas behind! Please drop yours into the #extensions channel on the Postgres Slack. What's next? Github cat logo Try Tembo Cloud for free Github cat logo Star on Github Youtube play button logo Subscribe on Youtube Github cat logo View our RSS feed Github cat logo Follow on X Github cat logo Go back Next post Github cat logo MongoDB capabilities on Postgres with Managed FerretDB on Tembo Cloud Feb 15, 2024 postgres extensions ecosystem packaging distribution architecture jobs to be done On this page Jobs to be DoneAuthoritativeBeneficialIntegratedDiscoverable InformativeProgrammableInstallableTrustedManageableOmissions and Next Steps Share this article tembo log Company Docs Blog Pricing Product Cloud Trunk Roadmap Changelog Connect Github Twitter LinkedIn Youtube Tembo Slack Trunk Slack Resources Status page Careers Support Privacy policy Terms of service checkmark SOC2 Type 1 compliant