https://diziet.dreamwidth.org/20436.html
Account name: [ ] Password [ ] [Log in]
(OpenID?) (Forgot it?) [ ] Remember Me
You're viewing [personal profile] diziet's journal
Create a Dreamwidth Account Learn More
[ ] [Interest ] [Go]
Reload page in style: site light
Ian Jackson
* Recent Entries
* Archive
* Reading
* Tags
* Memories
* Profile
Debian's git transition
Dec. 21st, 2025 11:24 pm
diziet: (Default)
[personal profile] diziet
tl;dr:
There is a Debian git transition plan. It's going OK so far but we
need help, especially with outreach and updating Debian's
documentation.
* Goals of the Debian git transition project
+ Achievements so far, and current status
* Core engineering principle
+ Correspondence between dsc and git
+ Patches-applied vs patches-unapplied
+ Consequences, some of which are annoying
* Distributing the source code as git
+ Tracking the relevant git data, when changes are made in the
legacy Archive
+ Why *.dgit.debian.org is not Salsa
* Roadmap
+ In progress
+ Future Technology
* Mindshare and adoption - please help!
+ A rant about publishing the source code
+ Documentation
* Personnel
+ Thanks
Goals of the Debian git transition project
0. Everyone who interacts with Debian source code should be able to
do so entirely in git.
That means, more specifically:
1. All examination and edits to the source should be performed via
normal git operations.
2. Source code should be transferred and exchanged as git data, not
tarballs. git should be the canonical form everywhere.
3. Upstream git histories should be re-published, traceably, as part
of formal git releases published by Debian.
4. No-one should have to learn about Debian Source Packages, which
are bizarre, and have been obsoleted by modern version control.
This is very ambitious, but we have come a long way!
Achievements so far, and current status
We have come a very long way. But, there is still much to do -
especially, the git transition team needs your help with adoption,
developer outreach, and developer documentation overhaul.
We've made big strides towards goals 1 and 4. Goal 2 is partially
achieved: we currently have dual running. Goal 3 is within our reach
but depends on widespread adoption of tag2upload (and/or dgit push).
Downstreams and users can obtain the source code of any Debian
package in git form. (dgit clone, 2013). They can then work with this
source code completely in git, including building binaries, merging
new versions, even automatically (eg Raspbian, 2016), and all without
having to deal with source packages at all (eg Wikimedia 2025).
A Debian maintainer can maintain their own package entirely in git.
They can obtain upstream source code from git, and do their packaging
work in git (git-buildpackage, 2006).
Every Debian maintainer can (and should!) release their package from
git reliably and in a standard form (dgit push, 2013; tag2upload,
2025). This is not only more principled, but also more convenient,
and with better UX, than pre-dgit tooling like dput.
Indeed a Debian maintainer can now often release their changes to
Debian, from git, using only git branches (so no tarballs). Releasing
to Debian can be simply pushing a signed tag (tag2upload, 2025).
A Debian maintainer can maintain a stack of changes to upstream
source code in git (gbp pq 2009). They can even maintain such a delta
series as a rebasing git branch, directly buildable, and use normal
git rebase style operations to edit their changes, (git-dpm, 2010;
git-debrebase, 2018)
An authorised Debian developer can do a modest update to any package
in Debian, even one maintained by someone else, working entirely in
git in a standard and convenient way (dgit, 2013).
Debian contributors can share their work-in-progress on git forges
and collaborate using merge requests, git based code review, and so
on. (Alioth, 2003; Salsa, 2018.)
Core engineering principle
The Debian git transition project is based on one core engineering
principle:
Every Debian Source Package can be losslessly converted to and from
git.
In order to transition away from Debian Source Packages, we need to
gateway between the old dsc approach, and the new git approach.
This gateway obviously needs to be bidirectional: source packages
uploaded with legacy tooling like dput need to be imported into a
canonical git representation; and of course git branches prepared by
developers need to be converted to source packages for the benefit of
legacy downstream systems (such as the Debian Archive and apt
source).
This bidirectional gateway is implemented in src:dgit, and is
allowing us to gradually replace dsc-based parts of the Debian system
with git-based ones.
Correspondence between dsc and git
A faithful bidirectional gateway must define an invariant:
The canonical git tree, corresponding to a .dsc, is the tree
resulting from dpkg-source -x.
This canonical form is sometimes called the "dgit view". It's
sometimes not the same as the maintainer's git branch, because many
maintainers are still working with "patches-unapplied" git branches.
More on this below.
(For 3.0 (quilt) .dscs, the canonical git tree doesn't include the
quilt .pc directory.)
Patches-applied vs patches-unapplied
The canonical git format is "patches applied". That is:
If Debian has modified the upstream source code, a normal git clone
of the canonical branch gives the modified source tree, ready for
reading and building.
Many Debian maintainers keep their packages in a different git branch
format, where the changes made by Debian, to the upstream source
code, are in actual patch files in a debian/patches/ subdirectory.
Patches-applied has a number of important advantages over
patches-unapplied:
* It is familiar to, and doesn't trick, outsiders to Debian. Debian
insiders radically underestimate how weird "patches-unapplied"
is. Even expert software developers can get very confused or even
accidentally build binaries without security patches!
* Making changes can be done with just normal git commands, eg git
commit. Many Debian insiders working with patches-unapplied are
still using quilt(1), a footgun-rich contraption for working with
patch files!
* When developing, one can make changes to upstream code, and to
Debian packaging, together, without ceremony. There is no need to
switch back and forth between patch queue and packaging branches
(as with gbp pq), no need to "commit" patch files, etc. One can
always edit every file and commit it with git commit.
The downside is that, with the (bizarre) 3.0 (quilt) source format,
the patch files files in debian/patches/ must somehow be kept up to
date. Nowadays though, tools like git-debrebase and git-dpm (and dgit
for NMUs) make it very easy to work with patches-applied git
branches. git-debrebase can deal very ergonomically even with big
patch stacks.
(For smaller packages which usually have no patches, plain git merge
with an upstream git branch, and a much simpler dsc format, sidesteps
the problem entirely.)
Prioritising Debian's users (and other outsiders)
We want everyone to be able to share and modify the software that
they interact with. That means we should make source code truly
accessible, on the user's terms.
Many of Debian's processes assume everyone is an insider. It's okay
that there are Debian insiders and that people feel part of something
that they worked hard to become involved with. But lack of
perspective can lead to software which fails to uphold our values.
Our source code practices -- in particular, our determination to share
properly (and systematically) -- are a key part of what makes Debian
worthwhile at all. Like Debian's installer, we want our source code
to be useable by Debian outsiders.
This is why we have chosen to privilege a git branch format which is
more familiar to the world at large, even if it's less popular in
Debian.
Consequences, some of which are annoying
The requirement that the conversion be bidirectional, lossless, and
context-free can be inconvenient.
For example, we cannot support .gitattributes which modify files
during git checkin and checkout. .gitattributes cause the meaning of
a git tree to depend on the context, in possibly arbitrary ways, so
the conversion from git to source package wouldn't be stable. And,
worse, some source packages might not to be representable in git at
all.
Another example: Maintainers often have existing git branches for
their packages, generated with pre-dgit tooling which is less careful
and less principled than ours. That can result in discrepancies
between git and dsc, which need to be resolved before a proper
git-based upload can succeed.
That some maintainers use patches-unapplied, and some
patches-unapplied, means that there has to be some kind of conversion
to a standard git representation. Choosing the less-popular
patches-applied format as the canonical form, means that many
packages need their git representation converted. It also means that
user- and outsider-facing branches from {browse,git}.dgit.d.o and
dgit clone are not always compatible with maintainer branches on
Salsa. User-contributed changes need cherry-picking rather than
merging, or conversion back to the maintainer format. The good news
is that dgit can automate much of this, and the manual parts are
usually easy git operations.
Distributing the source code as git
Our source code management should be normal, modern, and based on
git. That means the Debian Archive is obsolete and needs to be
replaced with a set of git repositories.
The replacement repository for source code formally released to
Debian is *.dgit.debian.org. This contains all the git objects for
every git-based upload since 2013, including the signed tag for each
released package version.
The plan is that it will contain a git view of every uploaded Debian
package, by centrally importing all legacy uploads into git.
Tracking the relevant git data, when changes are made in the legacy
Archive
Currently, many critical source code management tasks are done by
changes to the legacy Debian Archive, which works entirely with dsc
files (and the associated tarballs etc). The contents of the Archive
are therefore still an important source of truth. But, the Archive's
architecture means it cannot sensibly directly contain git data.
To track changes made in the Archive, we added the Dgit: field to the
.dsc of a git-based upload (2013). This declares which git commit
this package was converted from. and where those git objects can be
obtained.
Thus, given a Debian Source Package from a git-based upload, it is
possible for the new git tooling to obtain the equivalent git
objects. If the user is going to work in git, there is no need for
any tarballs to be downloaded: the git data could be obtained from
the depository using the git protocol.
The signed tags, available from the git depository, have standardised
metdata which gives traceability back to the uploading Debian
contributor.
Why *.dgit.debian.org is not Salsa
We need a git depository - a formal, reliable and permanent git
repository of source code actually released to Debian.
Git forges like Gitlab can be very convenient. But Gitlab is not
sufficiently secure, and too full of bugs, to be the principal and
only archive of all our source code. (The "open core" business model
of the Gitlab corporation, and the constant-churn development
approach, are critical underlying problems.)
Our git depository lacks forge features like Merge Requests. But:
* It is dependable, both in terms of reliability and security.
* It is append-only: once something is pushed, it is permanently
recorded.
* Its access control is precisely that of the Debian Archive.
* Its ref namespace is standardised and corresponds to Debian
releases.
* Pushes are authorised by PGP signatures, not ssh keys, so
traceable.
The dgit git depository outlasted Alioth and it may well outlast
Salsa.
We need both a good forge, and the *.dgit.debian.org formal git
depository.
Roadmap
In progress
Right now we are quite focused on tag2upload.
We are working hard on eliminating the remaining issues that we feel
need to be addressed before declaring the service out of beta.
Future Technology
Whole-archive dsc importer
Currently, the git depository only has git data for git-based package
updates (tag2upload and dgit push). Legacy dput-based uploads are not
currently present there. This means that the git-based and legacy
uploads must be resolved client-side, by dgit clone.
We will want to start importing legacy uploads to git.
Then downstreams and users will be able to get the source code for
any package simply with git clone, even if the maintainer is using
legacy upload tools like dput.
Support for git-based uploads to security.debian.org
Security patching is a task which would particularly benefit from
better and more formal use of git. git-based approaches to applying
and backporting security patches are much more convenient than
messing about with actual patch files.
Currently, one can use git to help prepare a security upload, but it
often involves starting with a dsc import (which lacks the proper git
history) or figuring out a package maintainer's unstandardised git
usage conventions on Salsa.
And it is not possible to properly perform the security release as
git.
Internal Debian consumers switch to getting source from git
Buildds, QA work such as lintian checks, and so on, could be simpler
if they don't need to deal with source packages.
And since git is actually the canonical form, we want them to use it
directly.
Problems for the distant future
For decades, Debian has been built around source packages. Replacing
them is a long and complex process. Certainly source packages are
going to continue to be supported for the foreseeable future.
There are no doubt going to be unanticipated problems. There are also
foreseeable issues: for example, perhaps there are packages that work
very badly when represented in git. We think we can rise to these
challenges as they come up.
Mindshare and adoption - please help!
We and our users are very pleased with our technology. It is
convenient and highly dependable.
dgit in particular is superb, even if we say so ourselves. As
technologists, we have been very focused on building good software,
but it seems we have fallen short in the marketing department.
A rant about publishing the source code
git is the preferred form for modification.
Our upstreams are overwhelmingly using git. We are overwhelmingly
using git. It is a scandal that for many packages, Debian does not
properly, formally and officially publish the git history.
Properly publishing the source code as git means publishing it in a
way that means that anyone can automatically and reliably obtain and
build the exact source code corresponding to the binaries. The test
is: could you use that to build a derivative?
Putting a package in git on Salsa is often a good idea, but it is not
sufficient. No standard branch structure git on Salsa is enforced,
nor should it be (so it can't be automatically and reliably
obtained), the tree is not in a standard form (so it can't be
automatically built), and is not necessarily identical to the source
package. So Vcs-Git fields, and git from Salsa, will never be
sufficient to make a derivative.
Debian is not publishing the source code!
The time has come for proper publication of source code by Debian to
no longer be a minority sport. Every maintainer of a package whose
upstream is using git (which is nearly all packages nowadays) should
be basing their work on upstream git, and properly publishing that
via tag2upload or dgit.
And it's not even difficult! The modern git-based tooling provides a
far superior upload experience.
A common misunderstanding
dgit push is not an alternative to gbp pq or quilt. Nor is
tag2upload. These upload tools complement your existing git workflow.
They replace and improve source package building/signing and the
subsequent dput. If you are using one of the usual git layouts on
salsa, and your package is in good shape, you can adopt tag2upload
and/or dgit push right away.
git-debrebase is distinct and does provides an alternative way to
manage your git packaging, do your upstream rebases, etc.
Documentation
Debian's documentation all needs to be updated, including
particularly instructions for packaging, to recommend use of
git-first workflows. Debian should not be importing git-using
upstreams' "release tarballs" into git. (Debian outsiders who
discover this practice are typically horrified.) We should use only
upstream git, work only in git, and properly release (and publish) in
git form.
We, the git transition team, are experts in the technology, and can
provide good suggestions. But we do not have the bandwidth to also
engage in the massive campaigns of education and documentation
updates that are necessary -- especially given that (as with any
programme for change) many people will be sceptical or even hostile.
So we would greatly appreciate help with writing and outreach.
Personnel
We consider ourselves the Debian git transition team.
Currently we are:
* Ian Jackson. Author and maintainer of dgit and git-debrebase.
Co-creator of tag2upload. Original author of dpkg-source, and
inventor in 1996 of Debian Source Packages. Alumnus of the Debian
Technical Committee.
* Sean Whitton. Co-creator of the tag2upload system; author and
maintainer of git-debpush. Co-maintainer of dgit. Debian Policy
co-Editor. Former Chair of the Debian Technical Committee.
We wear the following hats related to the git transition:
* Maintainers of src:dgit
* tag2upload Delegates; operators of the tag2upload service.
* service operators of the git depository *.dgit.debian.org.
You can contact us:
* By email: Ian Jackson ijackson@chiark.greenend.org.uk; Sean
Whitton spwhitton@spwhitton.name; git-debpush@packages.d.o.
* By filing bugs in the Debian Bug System against src:dgit.
* On OFTC IRC, as Diziet and spwhitton.
We do most of our heavy-duty development on Salsa.
Thanks
Particular thanks are due to Joey Hess, who, in the now-famous design
session in Vaumarcus in 2013, helped invent dgit. Since then we have
had a lot of support: most recently political support to help get
tag2upload deployed, but also, over the years, helpful bug reports
and kind words from our users, as well as translations and code
contributions.
Many other people have contributed more generally to support for
working with Debian source code in git. We particularly want to
mention Guido Gunther (git-buildpackage); and of course Alexander
Wirt, Joerg Jaspert, Thomas Goirand and Antonio Terceiro (Salsa
administrators); and before them the Alioth administrators.
Tags:
* debian,
* dgit,
* git,
* tag2upload
* Previous Entry
* Add Memory
* Share This Entry
* Next Entry
---------------------------------------------------------------------
* Reply
---------------------------------------------------------------------
Profile
diziet: (Default)
Ian Jackson
My Website
December 2025
S M T W T F S
1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31
Most Popular Tags
* 3d printing - 3 uses
* administrivia - 1 use
* bicon - 1 use
* board games - 5 uses
* chiark - 6 uses
* chiark-utils - 1 use
* climate emergency - 1 use
* computers - 58 uses
* covid - 2 uses
* cycling - 2 uses
* debian - 12 uses
* derive-adhoc - 2 uses
* derive-deftly - 2 uses
* derril water - 1 use
* dgit - 4 uses
* diversity - 2 uses
* dkim-rotate - 2 uses
* engineering - 2 uses
* eudcc - 1 use
* games - 1 use
* git - 4 uses
* hippotat - 2 uses
* legal - 4 uses
* nailing-cargo - 3 uses
* otter - 3 uses
* outflank-mailman - 1 use
* partial-borrow - 1 use
* personal - 4 uses
* phone - 5 uses
* politics - 8 uses
* prefork-interp - 1 use
* rust - 23 uses
* rust-polyglot - 2 uses
* secnet - 1 use
* subdirmk - 2 uses
* tag2upload - 2 uses
* tor - 1 use
* userv - 1 use
* xen - 3 uses
Style Credit
* Style: Basic for Transmogrified by Yvonne
Expand Cut Tags
No cut tags
Powered by Dreamwidth Studios
Top of page