Bug#859867: Please add a package which automatically configures sbuild for Debian packaging

[Michael Stapelberg]

Package: sbuild
Version: 0.73.0-4
Severity: wishlist

I’ve been using sbuild instead of pbuilder for a few years now, and I
generally like it: it seems almost universally better than pbuilder.

One area where sbuild sorely lacks is configuration, though: pbuilder
is very easy to set up, whereas sbuild requires reading through
https://wiki.debian.org/sbuild, performing a bunch of steps, only to
end up with a setup which works fine for unstable, but seems very
clumsy when building packages for experimental or backports.

One solution to this issue that I can see is to add a new binary
package to src:sbuild which — possibly after a brief debconf prompt —
performs all the necessary steps to end up with a setup that just
works.

What are your thoughts on this? Would patches be welcome to add such a
package?

-- System Information:
Debian Release: 9.0
APT prefers testing
APT policy: (990, 'testing'), (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386, armel, mipsel, arm64

Kernel: Linux 4.9.0-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages sbuild depends on:
ii adduser 3.115
ii libsbuild-perl 0.73.0-4
pn perl:any <none>

Versions of packages sbuild recommends:
ii autopkgtest 4.3
ii debootstrap 1.0.88
ii schroot 1.6.10-3+b1

Versions of packages sbuild suggests:
ii deborphan 1.7.28.8-0.3+b1
ii kmod 23-2
ii wget 1.

[Johannes Schauer]

Hi,

Quoting Michael Stapelberg (2017-04-08 11:28:12)

One area where sbuild sorely lacks is configuration, though: pbuilder is very
easy to set up, whereas sbuild requires reading through
https://wiki.debian.org/sbuild, performing a bunch of steps, only to end up
with a setup which works fine for unstable, but seems very clumsy when
building packages for experimental or backports.
One solution to this issue that I can see is to add a new binary
package to src:sbuild which — possibly after a brief debconf prompt —
performs all the necessary steps to end up with a setup that just
works.
What are your thoughts on this? Would patches be welcome to add such a
package?

patches totally welcome! :D

This is a nice idea!

Maybe these packages could be named sbuild-backend-${foo} where $foo is the
respective backend? At first, a package sbuild-backend-schroot would be cool
because schroot is the default backend. It would be nice if that would set up
sbuild schroots for stable-backports, unstable and experimental. Maybe that
package should also install and activate cron-jobs to regularly update those
schroots?

What irks me is, that this setup would be Debian specific. It doesn't make much
sense for Debian's downstreams to have have schroots for Debian. Maybe the
distribution name should be part of the package name? Or maybe it should be
easy for downstreams that care to override the set of distributions the
schroots are created for?

Thanks!

cheers, josch

[Michael Stapelberg]

Sorry for the late reply.

Post by Johannes Schauer
Hi,
Quoting Michael Stapelberg (2017-04-08 11:28:12)

One area where sbuild sorely lacks is configuration, though: pbuilder is

very

easy to set up, whereas sbuild requires reading through
https://wiki.debian.org/sbuild, performing a bunch of steps, only to

end up

with a setup which works fine for unstable, but seems very clumsy when
building packages for experimental or backports.
One solution to this issue that I can see is to add a new binary
package to src:sbuild which — possibly after a brief debconf prompt —
performs all the necessary steps to end up with a setup that just
works.
What are your thoughts on this? Would patches be welcome to add such a
package?

patches totally welcome! :D

Cool! I’ll see if I can whip up such a package by working through the wiki
page.

Post by Johannes Schauer
This is a nice idea!
Maybe these packages could be named sbuild-backend-${foo} where $foo is the
respective backend? At first, a package sbuild-backend-schroot would be cool
because schroot is the default backend. It would be nice if that would set up
sbuild schroots for stable-backports, unstable and experimental. Maybe that
package should also install and activate cron-jobs to regularly update those
schroots?

Which other backends even are there, and do we really need to offer our
users that choice, when we’re talking about a package with sane defaults
for people who prefer not to make these choices right now? :)

Post by Johannes Schauer
What irks me is, that this setup would be Debian specific. It doesn't make much
sense for Debian's downstreams to have have schroots for Debian. Maybe the
distribution name should be part of the package name? Or maybe it should be
easy for downstreams that care to override the set of distributions the
schroots are created for?

Putting Debian into the package name seems reasonable.

How about sbuild-debian-dev-setup?

I originally thought of sbuild-debian-setup, but it could be confused to
mean “a setup which reflects Debian’s official setup”, i.e. the buildds.

The “setup” suffix to me conveys that this package offers only
configuration, not new software. Perhaps there’s a more idiomatic term for
that in Debian?

Post by Johannes Schauer
Thanks!
cheers, josch

[Michael Stapelberg]

Post by Michael Stapelberg
Sorry for the late reply.

Post by Johannes Schauer
Hi,
Quoting Michael Stapelberg (2017-04-08 11:28:12)

Post by Michael Stapelberg
One area where sbuild sorely lacks is configuration, though: pbuilder

is very

Post by Michael Stapelberg
easy to set up, whereas sbuild requires reading through
https://wiki.debian.org/sbuild, performing a bunch of steps, only to

end up

Post by Michael Stapelberg
with a setup which works fine for unstable, but seems very clumsy when
building packages for experimental or backports.
One solution to this issue that I can see is to add a new binary
package to src:sbuild which — possibly after a brief debconf prompt —
performs all the necessary steps to end up with a setup that just
works.
What are your thoughts on this? Would patches be welcome to add such a
package?

patches totally welcome! :D

Cool! I’ll see if I can whip up such a package by working through the wiki
page.

Find attached the first draft of my suggestion. I implemented it as a
separate package purely so that I can build it more quickly, but I assume
we’d want to fold this into src:sbuild eventually.

The resulting package (I built it using dpkg-buildpackage -b) depends on
sbuild, schroot, debootstrap, perl-base, lintian. Upon installation, it
will create an unstable sbuild schroot, modify its configuration, add all
users to sbuild and create a modified ~/.sbuildrc for all users.

If you want to read through the entire behavior, I recommend the
entrypoints debian/postinst and update-sbuild-chroots.

I tested this package on my notebook, which is a Debian installation on
which I never had sbuild installed, so I’m reasonably confident that the
package works — at least to the point that one gets an sbuild installation
that builds packages.

I’d be happy about any feedback. Thanks!

Post by Michael Stapelberg

Post by Johannes Schauer
This is a nice idea!
Maybe these packages could be named sbuild-backend-${foo} where $foo is the
respective backend? At first, a package sbuild-backend-schroot would be cool
because schroot is the default backend. It would be nice if that would set up
sbuild schroots for stable-backports, unstable and experimental. Maybe that
package should also install and activate cron-jobs to regularly update those
schroots?

Which other backends even are there, and do we really need to offer our
users that choice, when we’re talking about a package with sane defaults
for people who prefer not to make these choices right now? :)

Post by Johannes Schauer
What irks me is, that this setup would be Debian specific. It doesn't make much
sense for Debian's downstreams to have have schroots for Debian. Maybe the
distribution name should be part of the package name? Or maybe it should be
easy for downstreams that care to override the set of distributions the
schroots are created for?

Putting Debian into the package name seems reasonable.
How about sbuild-debian-dev-setup?
I originally thought of sbuild-debian-setup, but it could be confused to
mean “a setup which reflects Debian’s official setup”, i.e. the buildds.
The “setup” suffix to me conveys that this package offers only
configuration, not new software. Perhaps there’s a more idiomatic term for
that in Debian?

Post by Johannes Schauer
Thanks!
cheers, josch

--
Best regards,
Michael

[Geert Stappers]

Post by Michael Stapelberg
Find attached the first draft of my suggestion. I implemented it as a
separate package purely so that I can build it more quickly, but I assume
we???d want to fold this into src:sbuild eventually.
The resulting package (I built it using dpkg-buildpackage -b) depends on
sbuild, schroot, debootstrap, perl-base, lintian. Upon installation, it
will create an unstable sbuild schroot, modify its configuration, add all
users to sbuild and create a modified ~/.sbuildrc for all users.
If you want to read through the entire behavior, I recommend the
entrypoints debian/postinst and update-sbuild-chroots.

the debian/postinst now here inline


#!/bin/sh

set -e

#DEBHELPER#

# TODO: make this package pass piuparts

# Add to group sbuild all “dynamically allocated user accounts”; see
# https://www.debian.org/doc/debian-policy/ch-opersys.html
for user in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <= 59999 && say $F[0]')
do
# Strictly speaking, we should use sbuild-adduser, but sbuild-adduser prints
# setup instructions to STDERR which do not make sense in the context of
# this package. Filtering STDERR is cumbersome, so we call adduser directly.
adduser --quiet -- "$user" sbuild
done

# TODO: maybe add a “setup” suffix into the generated chroot name for easier trouble-shooting (we’ll immediately know who created the chroot initially)

# Create a chroot if it does not already exist
chroot="unstable-$(dpkg --print-architecture)-sbuild"
if ! schroot -i -c chroot:${chroot} >/dev/null 2>&1
then
sbuild-createchroot \
--include=eatmydata,ccache,gnupg \
unstable \
/srv/chroot/${chroot} \
http://deb.debian.org/debian

# At this point, sbuild-createchroot created an schroot configuration with a
# random suffix, e.g. /etc/schroot/chroot.d/unstable-amd64-sbuild-pyViYe. As
# sbuild-createchroot recommends, we rename that file before making
# adjustments.
mv /etc/schroot/chroot.d/${chroot}-* /etc/schroot/chroot.d/${chroot}
fi

# schroot config customizations:
config=/etc/schroot/chroot.d/${chroot}
tmp=$(mktemp ${config}-XXXXXX.dpkg-tmp)
trap 'rm -f "${tmp}"' TERM INT EXIT QUIT

grep -v -E '^(aliases|command-prefix)=' "${config}" > "${tmp}"

# For convenience, treat UNRELEASED as an alias for unstable (so that
# debian/changelog files containing UNRELEASED do not need to be modified before
# building). Also sid, because it is short to type when specifying -d.
echo "aliases=UNRELEASED,sid" >> "${tmp}"

# Enable eatmydata: occasionally losing a test build is preferable over longer
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"

chmod 644 "${tmp}"
mv "${tmp}" "${config}"

# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <= 59999 && say $F[5]')
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)" /usr/share/doc/sbuild/examples/example.sbuildrc && cat /usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"
fi
done

# bind-mount the apt archive cache into chroots, so that packages are downloaded
# only once. The assumption is that users will not typically have a local apt
# mirror or caching proxy.
if ! grep -q '^/var/cache/apt/archives' /etc/schroot/sbuild/fstab
then
echo "/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0" \

Post by Michael Stapelberg

/etc/schroot/sbuild/fstab

fi

if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in tmpfs using:"
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d/overlays-in-tmpfs /etc/schroot/setup.d/04tmpfs"
echo ""
fi

exit 0

Post by Michael Stapelberg
I tested this package on my notebook, which is a Debian installation on
which I never had sbuild installed, so I???m reasonably confident that the
package works ??? at least to the point that one gets an sbuild installation
that builds packages.
I???d be happy about any feedback. Thanks!

What was/is the idea of doing this in an additional packages?
Why not in an additional script?


Groeten
Geert Stappers

[Michael Stapelberg]

Doing it in a script is one more step. The point of this endavour is to
make the setup as simple as possible.

On that note, the awkward newgrp step afterwards still irks me.

Post by Geert Stappers

Post by Michael Stapelberg
Find attached the first draft of my suggestion. I implemented it as a
separate package purely so that I can build it more quickly, but I assume
we???d want to fold this into src:sbuild eventually.
The resulting package (I built it using dpkg-buildpackage -b) depends on
sbuild, schroot, debootstrap, perl-base, lintian. Upon installation, it
will create an unstable sbuild schroot, modify its configuration, add all
users to sbuild and create a modified ~/.sbuildrc for all users.
If you want to read through the entire behavior, I recommend the
entrypoints debian/postinst and update-sbuild-chroots.

the debian/postinst now here inline
#!/bin/sh
set -e
#DEBHELPER#
# TODO: make this package pass piuparts
# Add to group sbuild all “dynamically allocated user accounts”; see
# https://www.debian.org/doc/debian-policy/ch-opersys.html
for user in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <= 59999 && say $F[0]')
do
# Strictly speaking, we should use sbuild-adduser, but sbuild-adduser prints
# setup instructions to STDERR which do not make sense in the context of
# this package. Filtering STDERR is cumbersome, so we call adduser directly.
adduser --quiet -- "$user" sbuild
done
# TODO: maybe add a “setup” suffix into the generated chroot name for
easier trouble-shooting (we’ll immediately know who created the chroot
initially)
# Create a chroot if it does not already exist
chroot="unstable-$(dpkg --print-architecture)-sbuild"
if ! schroot -i -c chroot:${chroot} >/dev/null 2>&1
then
sbuild-createchroot \
--include=eatmydata,ccache,gnupg \
unstable \
/srv/chroot/${chroot} \
http://deb.debian.org/debian
# At this point, sbuild-createchroot created an schroot configuration with a
# random suffix, e.g. /etc/schroot/chroot.d/unstable-amd64-sbuild-pyViYe. As
# sbuild-createchroot recommends, we rename that file before making
# adjustments.
mv /etc/schroot/chroot.d/${chroot}-* /etc/schroot/chroot.d/${chroot}
fi
config=/etc/schroot/chroot.d/${chroot}
tmp=$(mktemp ${config}-XXXXXX.dpkg-tmp)
trap 'rm -f "${tmp}"' TERM INT EXIT QUIT
grep -v -E '^(aliases|command-prefix)=' "${config}" > "${tmp}"
# For convenience, treat UNRELEASED as an alias for unstable (so that
# debian/changelog files containing UNRELEASED do not need to be modified before
# building). Also sid, because it is short to type when specifying -d.
echo "aliases=UNRELEASED,sid" >> "${tmp}"
# Enable eatmydata: occasionally losing a test build is preferable over longer
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"
chmod 644 "${tmp}"
mv "${tmp}" "${config}"
# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <=
59999 && say $F[5]')
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)" /usr/share/doc/sbuild/examples/example.sbuildrc
&& cat /usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"
fi
done
# bind-mount the apt archive cache into chroots, so that packages are downloaded
# only once. The assumption is that users will not typically have a local apt
# mirror or caching proxy.
if ! grep -q '^/var/cache/apt/archives' /etc/schroot/sbuild/fstab
then
echo "/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0" \

Post by Michael Stapelberg

Post by Geert Stappers
/etc/schroot/sbuild/fstab

fi
if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in tmpfs using:"
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d/overlays-in-tmpfs
/etc/schroot/setup.d/04tmpfs"
echo ""
fi
exit 0

Post by Michael Stapelberg
I tested this package on my notebook, which is a Debian installation on
which I never had sbuild installed, so I???m reasonably confident that

the

Post by Michael Stapelberg
package works ??? at least to the point that one gets an sbuild

installation

Post by Michael Stapelberg
that builds packages.
I???d be happy about any feedback. Thanks!

What was/is the idea of doing this in an additional packages?
Why not in an additional script?
Groeten
Geert Stappers
--
Leven en laten leven

[Johannes Schauer]

Quoting Geert Stappers (2017-05-24 10:56:00)

Thanks for the feedback. Any suggestions as to how the script should be
called, and which options it should have, if any?

sbuild-setup [ triplet [ /my/place/for/the/chroot ]]

If we introduce sensible defaults (see also my other mail in this thread), what
is the difference between this and sbuild-createchroot?

[Raphael Hertzog]

Thanks for the feedback. Any suggestions as to how the script should be
called, and which options it should have, if any?

In my case, I believe that we should have a cron job/systemd timer that
updates the chroot each week and that cron job could create the chroots on
its first invocation.

What chroots are created should be dictated by configuration files in a .d
directory. So that a package can ship additional configuration files and
possibly also tweak the parameters of the standard Debian chroots that
you will include by default.

Cheers,

[Johannes Schauer]

Hi,

Quoting Geert Stappers (2017-05-21 08:43:31)

Post by Geert Stappers
the debian/postinst now here inline

thanks, that allows me to comment easily.

Post by Geert Stappers
# Add to group sbuild all “dynamically allocated user accounts”; see
# https://www.debian.org/doc/debian-policy/ch-opersys.html
for user in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <= 59999 && say $F[0]')
do
# Strictly speaking, we should use sbuild-adduser, but sbuild-adduser prints
# setup instructions to STDERR which do not make sense in the context of
# this package. Filtering STDERR is cumbersome, so we call adduser directly.
adduser --quiet -- "$user" sbuild
done

I'm doubtful whether adding *all* users with ids between 1000 and 59999 to the
sbuild group is something sensible to do. This would give *all* these users
instant access to schroot-based chroots. I don't think this is something
anybody expects when running a script, much less something that anybody expects
to happen in a package's postinstall.

Post by Geert Stappers
# Create a chroot if it does not already exist
chroot="unstable-$(dpkg --print-architecture)-sbuild"
if ! schroot -i -c chroot:${chroot} >/dev/null 2>&1
then
sbuild-createchroot \
--include=eatmydata,ccache,gnupg \

Why include ccache? I would not expect that a single build will compile the
same source multiple times. Without bind-mounting the cache directory from the
outside this will probably not help much.

Why include gnupg?

Post by Geert Stappers
unstable \
/srv/chroot/${chroot} \
http://deb.debian.org/debian

Why not let this new package depend on apt-cacher-ng and let
http://127.0.0.1:3142/deb.debian.org/debian be the default mirror?

Post by Geert Stappers
# At this point, sbuild-createchroot created an schroot configuration with a
# random suffix, e.g. /etc/schroot/chroot.d/unstable-amd64-sbuild-pyViYe. As
# sbuild-createchroot recommends, we rename that file before making
# adjustments.
mv /etc/schroot/chroot.d/${chroot}-* /etc/schroot/chroot.d/${chroot}

What happens with any existing /etc/schroot/chroot.d/${chroot}? You just
overwrite it?

What happens if more than one file match /etc/schroot/chroot.d/${chroot}-*?

Post by Geert Stappers
fi
config=/etc/schroot/chroot.d/${chroot}
tmp=$(mktemp ${config}-XXXXXX.dpkg-tmp)
trap 'rm -f "${tmp}"' TERM INT EXIT QUIT
grep -v -E '^(aliases|command-prefix)=' "${config}" > "${tmp}"
# For convenience, treat UNRELEASED as an alias for unstable (so that
# debian/changelog files containing UNRELEASED do not need to be modified before
# building). Also sid, because it is short to type when specifying -d.
echo "aliases=UNRELEASED,sid" >> "${tmp}"

You can achieve the same effect using sbuild-createchroot by using the --alias
option. No need to manually mangle the schroot config.

Post by Geert Stappers
# Enable eatmydata: occasionally losing a test build is preferable over longer
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"

I don't see sbuild-createchroot mangling with command-prefix at all. So why do
you filter it out initially?

Would it not be better to add a command line option to sbuild-createchroot
which allows setting a custom command-prefix? Then you would not need to
manually mangle the created config file at all and lots of the problems I
mentioned above and other fragile things would go away.

Post by Geert Stappers
chmod 644 "${tmp}"
mv "${tmp}" "${config}"
# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <= 59999 && say $F[5]')
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)" /usr/share/doc/sbuild/examples/example.sbuildrc && cat /usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"
fi
done

Instead of crafting a custom sbuildrc, we can also talk about changing the
existing defaults.

For example you set "$build_arch_all = 1;" I can certainly see how it makes
sense to change this default for sbuild but keep it at 0 for buildd.

Then you set "$build_source = 1;" but I don't know why you would need this.
Sbuild is not there to build the source package. The source package is its
input not its output. Why would you want sbuild to build it again?

Lastly, you set "$run_lintian = 1;" which I also think would be a sensible
thing to change in the defaults.

As a result, you would not need to manually craft a custom sbuildrc anymore.

Post by Geert Stappers
# bind-mount the apt archive cache into chroots, so that packages are downloaded
# only once. The assumption is that users will not typically have a local apt
# mirror or caching proxy.
if ! grep -q '^/var/cache/apt/archives' /etc/schroot/sbuild/fstab
then
echo "/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0 0" \

Post by Geert Stappers
/etc/schroot/sbuild/fstab

fi

And who cleans up /var/cache/apt/archives regularly when it becomes cluttered
over time?

I also don't like the idea of just randomly mixing the /var/cache/apt/archives
directory of the host with multiple sbuild instances which might be running at
the same time. Just imagine sbuild running for different distributions which
have packages with the same name and version but different content. That is
just calling for trouble.

I think a much better option is to use something like apt-cacher-ng. This new
package could easily depend on it and then use that as its default mirror as
already explained above.

Post by Geert Stappers
if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in tmpfs using:"
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d/overlays-in-tmpfs /etc/schroot/setup.d/04tmpfs"
echo ""
fi

This will affect *all* schroot sessions and not just those used by sbuild. As
such, this should rather go into documentation for schroot. But maybe there is
a way to make this mounting specific to sbuild schroots?

I think in summary much of what this script does can be achieved by:

- improving existing documentation
- changing configuration defaults to more sensible values
- adding more options to existing tools like sbuild-createchroot
- using existing tools instead of using creative ways to work around them
(apt-cacher-ng)

If we do all that, what is left that a new script should take care of?

Thanks!

cheers, josch

[Michael Stapelberg]

Post by Johannes Schauer
Hi,
Quoting Geert Stappers (2017-05-21 08:43:31)

Post by Geert Stappers
the debian/postinst now here inline

thanks, that allows me to comment easily.

Post by Geert Stappers
# Add to group sbuild all “dynamically allocated user accounts†; see
# https://www.debian.org/doc/debian-policy/ch-opersys.html
for user in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <=

59999 && say $F[0]')

Post by Geert Stappers
do
# Strictly speaking, we should use sbuild-adduser, but

sbuild-adduser prints

Post by Geert Stappers
# setup instructions to STDERR which do not make sense in the

context of

Post by Geert Stappers
# this package. Filtering STDERR is cumbersome, so we call adduser

directly.

Post by Geert Stappers
adduser --quiet -- "$user" sbuild
done

I'm doubtful whether adding *all* users with ids between 1000 and 59999 to the
sbuild group is something sensible to do. This would give *all* these users
instant access to schroot-based chroots. I don't think this is something
anybody expects when running a script, much less something that anybody expects
to happen in a package's postinstall.

Post by Geert Stappers
# Create a chroot if it does not already exist
chroot="unstable-$(dpkg --print-architecture)-sbuild"
if ! schroot -i -c chroot:${chroot} >/dev/null 2>&1
then
sbuild-createchroot \
--include=eatmydata,ccache,gnupg \

Why include ccache? I would not expect that a single build will compile the
same source multiple times. Without bind-mounting the cache directory from the
outside this will probably not help much.
Why include gnupg?

This was copy&pasted from the wiki page. Removed.

Post by Johannes Schauer

Post by Geert Stappers
unstable \
/srv/chroot/${chroot} \
http://deb.debian.org/debian

Why not let this new package depend on apt-cacher-ng and let
http://127.0.0.1:3142/deb.debian.org/debian be the default mirror?

Done.

Post by Johannes Schauer

Post by Geert Stappers
# At this point, sbuild-createchroot created an schroot

configuration with a

Post by Geert Stappers
# random suffix, e.g. /etc/schroot/chroot.d/unstable-amd64-sbuild-pyViYe.

As

Post by Geert Stappers
# sbuild-createchroot recommends, we rename that file before making
# adjustments.
mv /etc/schroot/chroot.d/${chroot}-* /etc/schroot/chroot.d/${chroot}

What happens with any existing /etc/schroot/chroot.d/${chroot}? You just
overwrite it?
What happens if more than one file match /etc/schroot/chroot.d/${
chroot}-*?

Post by Geert Stappers
fi
config=/etc/schroot/chroot.d/${chroot}
tmp=$(mktemp ${config}-XXXXXX.dpkg-tmp)
trap 'rm -f "${tmp}"' TERM INT EXIT QUIT
grep -v -E '^(aliases|command-prefix)=' "${config}" > "${tmp}"
# For convenience, treat UNRELEASED as an alias for unstable (so that
# debian/changelog files containing UNRELEASED do not need to be

modified before

Post by Geert Stappers
# building). Also sid, because it is short to type when specifying -d.
echo "aliases=UNRELEASED,sid" >> "${tmp}"

You can achieve the same effect using sbuild-createchroot by using the --alias
option. No need to manually mangle the schroot config.

Done.

Post by Johannes Schauer

Post by Geert Stappers
# Enable eatmydata: occasionally losing a test build is preferable over

longer

Post by Geert Stappers
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"

I don't see sbuild-createchroot mangling with command-prefix at all. So why do
you filter it out initially?
Would it not be better to add a command line option to sbuild-createchroot
which allows setting a custom command-prefix? Then you would not need to
manually mangle the created config file at all and lots of the problems I
mentioned above and other fragile things would go away.

Sure. Can you take care of this or do you want me to send a patch?

Post by Johannes Schauer

Post by Geert Stappers
chmod 644 "${tmp}"
mv "${tmp}" "${config}"
# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2]

<= 59999 && say $F[5]')

Post by Geert Stappers
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)" /usr/share/doc/sbuild/examples/example.sbuildrc

&& cat /usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"

Post by Geert Stappers
fi
done

Instead of crafting a custom sbuildrc, we can also talk about changing the
existing defaults.
For example you set "$build_arch_all = 1;" I can certainly see how it makes
sense to change this default for sbuild but keep it at 0 for buildd.

Sounds good. Same question: can you take care of this, or should I send a
patch?

Post by Johannes Schauer
Then you set "$build_source = 1;" but I don't know why you would need this.
Sbuild is not there to build the source package. The source package is its
input not its output. Why would you want sbuild to build it again?

The wiki page contained some confusing advice, which I now removed.

Post by Johannes Schauer
Lastly, you set "$run_lintian = 1;" which I also think would be a sensible
thing to change in the defaults.

Sounds good, see above.

Post by Johannes Schauer
As a result, you would not need to manually craft a custom sbuildrc anymore.

Post by Geert Stappers
# bind-mount the apt archive cache into chroots, so that packages are

downloaded

Post by Geert Stappers
# only once. The assumption is that users will not typically have a

local apt

Post by Geert Stappers
# mirror or caching proxy.
if ! grep -q '^/var/cache/apt/archives' /etc/schroot/sbuild/fstab
then
echo "/var/cache/apt/archives /var/cache/apt/archives none rw,bind 0

0" \

Post by Geert Stappers

Post by Geert Stappers
/etc/schroot/sbuild/fstab

fi

And who cleans up /var/cache/apt/archives regularly when it becomes cluttered
over time?
I also don't like the idea of just randomly mixing the
/var/cache/apt/archives
directory of the host with multiple sbuild instances which might be running at
the same time. Just imagine sbuild running for different distributions which
have packages with the same name and version but different content. That is
just calling for trouble.
I think a much better option is to use something like apt-cacher-ng. This new
package could easily depend on it and then use that as its default mirror as
already explained above.

Done.

Post by Johannes Schauer

Post by Geert Stappers
if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in tmpfs

using:"

Post by Geert Stappers
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d/overlays-in-tmpfs

/etc/schroot/setup.d/04tmpfs"

Post by Geert Stappers
echo ""
fi

This will affect *all* schroot sessions and not just those used by sbuild. As
such, this should rather go into documentation for schroot. But maybe there is
a way to make this mounting specific to sbuild schroots?

I don’t know, you’re the expert :). I think users who know what schroot is
(to me it’s “the thing which sbuild uses”) and who use it for anything but
sbuild will realize what’s happening here. We could add some wording to
that extent.

Post by Johannes Schauer
- improving existing documentation

I updated the wiki page based on your review.

Post by Johannes Schauer
- changing configuration defaults to more sensible values
- adding more options to existing tools like sbuild-createchroot
- using existing tools instead of using creative ways to work around them
(apt-cacher-ng)
If we do all that, what is left that a new script should take care of?

The following things are still left:

• Adding the user(s) to the sbuild group
• Creating the chroot with all the options we discussed
• Suggesting to build in tmpfs
• Periodically updating the schroots (not strictly speaking part of the
script itself, but listing it here anyway)

Post by Johannes Schauer
Thanks!
cheers, josch

[Michael Stapelberg]

Hi,
Quoting Michael Stapelberg (2017-06-02 18:23:02)
sorry for the delay. I'm under a pile of work and this wasn't on the top
of my

Post by Michael Stapelberg

Post by Geert Stappers

Post by Geert Stappers
# Enable eatmydata: occasionally losing a test build is preferable

over

Post by Michael Stapelberg

Post by Geert Stappers
longer

Post by Geert Stappers
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"

I don't see sbuild-createchroot mangling with command-prefix at all. So why do
you filter it out initially?
Would it not be better to add a command line option to

sbuild-createchroot

Post by Michael Stapelberg

Post by Geert Stappers
which allows setting a custom command-prefix? Then you would not need

to

Post by Michael Stapelberg

Post by Geert Stappers
manually mangle the created config file at all and lots of the

problems I

Post by Michael Stapelberg

Post by Geert Stappers
mentioned above and other fragile things would go away.

Sure. Can you take care of this or do you want me to send a patch?

Please file a new bug with a patch.

Done: https://bugs.debian.org/870260

Post by Michael Stapelberg

Post by Geert Stappers

Post by Geert Stappers
chmod 644 "${tmp}"
mv "${tmp}" "${config}"
# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 &&

$F[2]

Post by Michael Stapelberg

Post by Geert Stappers
<= 59999 && say $F[5]')

Post by Geert Stappers
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)"

/usr/share/doc/sbuild/examples/example.sbuildrc

Post by Michael Stapelberg

Post by Geert Stappers
&& cat /usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"

Post by Geert Stappers
fi
done

Instead of crafting a custom sbuildrc, we can also talk about changing

the

Post by Michael Stapelberg

Post by Geert Stappers
existing defaults.
For example you set "$build_arch_all = 1;" I can certainly see how it

makes

Post by Michael Stapelberg

Post by Geert Stappers
sense to change this default for sbuild but keep it at 0 for buildd.

Sounds good. Same question: can you take care of this, or should I send a
patch?

For personal users of sbuild (users of sbuild and not buildd) they will
probably always want to build arch:all packages. So lets change the
default!
Again, a short bug with patch is appreciated.

Done: https://bugs.debian.org/870263

Post by Michael Stapelberg

Post by Geert Stappers
Then you set "$build_source = 1;" but I don't know why you would need

this.

Post by Michael Stapelberg

Post by Geert Stappers
Sbuild is not there to build the source package. The source package is

its

Post by Michael Stapelberg

Post by Geert Stappers
input not its output. Why would you want sbuild to build it again?

The wiki page contained some confusing advice, which I now removed.

Thanks!

Post by Michael Stapelberg

Post by Geert Stappers
Lastly, you set "$run_lintian = 1;" which I also think would be a

sensible

Post by Michael Stapelberg

Post by Geert Stappers
thing to change in the defaults.

Sounds good, see above.

You can include that in the other bug.

See above.

Post by Michael Stapelberg

Post by Geert Stappers

Post by Geert Stappers
if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in

tmpfs

Post by Michael Stapelberg

Post by Geert Stappers
using:"

Post by Geert Stappers
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d

/overlays-in-tmpfs

Post by Michael Stapelberg

Post by Geert Stappers
/etc/schroot/setup.d/04tmpfs"

Post by Geert Stappers
echo ""
fi

This will affect *all* schroot sessions and not just those used by

sbuild.

Post by Michael Stapelberg

Post by Geert Stappers
As such, this should rather go into documentation for schroot. But

maybe

Post by Michael Stapelberg

Post by Geert Stappers
there is a way to make this mounting specific to sbuild schroots?

I don’t know, you’re the expert :). I think users who know what schroot

is

Post by Michael Stapelberg
(to me it’s “the thing which sbuild uses”) and who use it for anything

but

Post by Michael Stapelberg
sbuild will realize what’s happening here. We could add some wording to
that extent.

More docs in this direction would be appreciated.

Post by Michael Stapelberg

Post by Geert Stappers
- improving existing documentation

I updated the wiki page based on your review.

Thanks a lot!

Post by Michael Stapelberg

Post by Geert Stappers
- changing configuration defaults to more sensible values
- adding more options to existing tools like sbuild-createchroot
- using existing tools instead of using creative ways to work around

them

Post by Michael Stapelberg

Post by Geert Stappers
(apt-cacher-ng)
If we do all that, what is left that a new script should take care of?

• Adding the user(s) to the sbuild group

That's just one call to sbuild-adduser

Post by Michael Stapelberg
• Creating the chroot with all the options we discussed

Yet another single call to sbuild-createchroot

Post by Michael Stapelberg
• Suggesting to build in tmpfs

I'd consider that an optional tip that can be listed in the docs somewhere.

Post by Michael Stapelberg
• Periodically updating the schroots (not strictly speaking part of the
script itself, but listing it here anyway)

This can be done using the cron script from
/usr/share/doc/sbuild/examples/sbuild-update-all
Bug #870102 is about making this cron script more visible by adding better
documentation.
So all in all, we are down to running two scripts to setup sbuild. One of
that
is only required once after installation while the other can be run
multiple
times to setup multiple chroots. I don't think a wrapper script for these
two
scripts makes much sense. Lets rather better document which two commands
one
needs to run after installing sbuild for the first time.

Unless I’m mistaken, the following is what we’d need to recommend to new
users:

% sudo apt install sbuild apt-cacher-ng lintian

% sudo adduser --quiet -- "$USER" sbuild

% sudo sbuild-createchroot \
--command-prefix=eatmydata \
--include=eatmydata \
--alias=UNRELEASED \
--alias=sid \
unstable \
/srv/chroot/unstable-amd64-sbuild \
http://localhost:3142/deb.debian.org/debian

% echo 15 */6 * * * root /usr/share/doc/sbuild/examples/sbuild-update-all |
sudo tee /etc/cron.d/sbuild-update-all

% newgrp sbuild

That seems quite involved over, say, “apt install sbuild-setup &&
sbuild-setup unstable”.

Hence, I’d definitely appreciate a script which does all the over having to
refer to a wiki page and copy&paste long commands.

What do you think?

[Johannes Schauer]

Quoting Michael Stapelberg (2017-07-31 14:19:16)

Post by Michael Stapelberg
Unless I’m mistaken, the following is what we’d need to recommend to new
% sudo apt install sbuild apt-cacher-ng lintian

Why install lintian?

Post by Michael Stapelberg
% sudo adduser --quiet -- "$USER" sbuild

Better:

sudo sbuild-adduser $USER

Post by Michael Stapelberg
% sudo sbuild-createchroot \
--command-prefix=eatmydata \
--include=eatmydata \
--alias=UNRELEASED \
--alias=sid \
unstable \
/srv/chroot/unstable-amd64-sbuild \
http://localhost:3142/deb.debian.org/debian

That is *if* the machine of the user is amd64.

Also, this part would be Debian-specific. Downstream distributions would have
to adapt the alias and mirror values.

Also, didn't you also propose to make the schroot be run in a tmpfs the
default? In that case, eatmydata would be quite pointless, no?

Post by Michael Stapelberg
% echo 15 */6 * * * root /usr/share/doc/sbuild/examples/sbuild-update-all |
sudo tee /etc/cron.d/sbuild-update-all

Every six hours? I find that a bit excessive. This should certainly be
configurable. Not everybody is behind an internet connection which is fast
enough and/or where one doesn't pay per MB.

Post by Michael Stapelberg
% newgrp sbuild

This would only have an effect on the currently open terminal and would have to
be executed again on every new terminal session until the user *really* logs
out and in again.

Post by Michael Stapelberg
That seems quite involved over, say, “apt install sbuild-setup &&
sbuild-setup unstable”.
Hence, I’d definitely appreciate a script which does all the over having to
refer to a wiki page and copy&paste long commands.

Except that the sbuild-setup command would need to become quite complex because
it the user has to be able to control:

- how to setup schroot (overlayfs? tmpfs?)
- where to put the chroots
- which distribution aliases (distribution specific)
- which extra packages to include (like eatmydata)
- whether this is the first run or not (warn if the script is run for a second
time)
- how often to update the chroots via cron

And then we have a script with a complexity which is close to where
sbuild-createchroot already is.

Or are you actually convinced that it is possible to find a set of defaults
which fits even half the userbase of sbuild?

Since we are down to two mandatory (and two optional) commands after running
"apt install sbuild", I'd argue that a superior solution would be to improve
the documentation of which commands to run for a "typical" setup. I fear that
trying to create a "one-size-fits-all" script can have many unintended
side-effects (thinking of users behind bad or costly internet, who use schroot
for other purposes, who don't want to install another deamon like
apt-cacher-ng, who are not building for Debian but for downstreams...). I'm not
convinced that the time that the user would invest to *really* understand the
things that an sbuild-setup script is actually doing would not be better spent
in learning how to use the individual tools.

What do you think?

cheers, josch

[Michael Stapelberg]

josch, friendly ping? :)

Post by Michael Stapelberg

Post by Johannes Schauer
Hi,
Quoting Geert Stappers (2017-05-21 08:43:31)

Post by Geert Stappers
the debian/postinst now here inline

thanks, that allows me to comment easily.

Post by Geert Stappers
# Add to group sbuild all “dynamically allocated user accounts†; see
# https://www.debian.org/doc/debian-policy/ch-opersys.html
for user in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2] <=

59999 && say $F[0]')

Post by Geert Stappers
do
# Strictly speaking, we should use sbuild-adduser, but

sbuild-adduser prints

Post by Geert Stappers
# setup instructions to STDERR which do not make sense in the

context of

Post by Geert Stappers
# this package. Filtering STDERR is cumbersome, so we call adduser

directly.

Post by Geert Stappers
adduser --quiet -- "$user" sbuild
done

I'm doubtful whether adding *all* users with ids between 1000 and 59999 to the
sbuild group is something sensible to do. This would give *all* these users
instant access to schroot-based chroots. I don't think this is something
anybody expects when running a script, much less something that anybody expects
to happen in a package's postinstall.

Post by Geert Stappers
# Create a chroot if it does not already exist
chroot="unstable-$(dpkg --print-architecture)-sbuild"
if ! schroot -i -c chroot:${chroot} >/dev/null 2>&1
then
sbuild-createchroot \
--include=eatmydata,ccache,gnupg \

Why include ccache? I would not expect that a single build will compile the
same source multiple times. Without bind-mounting the cache directory from the
outside this will probably not help much.
Why include gnupg?

This was copy&pasted from the wiki page. Removed.

Post by Johannes Schauer

Post by Geert Stappers
unstable \
/srv/chroot/${chroot} \
http://deb.debian.org/debian

Why not let this new package depend on apt-cacher-ng and let
http://127.0.0.1:3142/deb.debian.org/debian be the default mirror?

Done.

Post by Johannes Schauer

Post by Geert Stappers
# At this point, sbuild-createchroot created an schroot

configuration with a

Post by Geert Stappers
# random suffix, e.g. /etc/schroot/chroot.d/unstable-amd64-sbuild-pyViYe.

As

Post by Geert Stappers
# sbuild-createchroot recommends, we rename that file before making
# adjustments.
mv /etc/schroot/chroot.d/${chroot}-* /etc/schroot/chroot.d/${chroot

}
What happens with any existing /etc/schroot/chroot.d/${chroot}? You just
overwrite it?
What happens if more than one file match /etc/schroot/chroot.d/${chroot
}-*?

Post by Geert Stappers
fi
config=/etc/schroot/chroot.d/${chroot}
tmp=$(mktemp ${config}-XXXXXX.dpkg-tmp)
trap 'rm -f "${tmp}"' TERM INT EXIT QUIT
grep -v -E '^(aliases|command-prefix)=' "${config}" > "${tmp}"
# For convenience, treat UNRELEASED as an alias for unstable (so that
# debian/changelog files containing UNRELEASED do not need to be

modified before

Post by Geert Stappers
# building). Also sid, because it is short to type when specifying -d.
echo "aliases=UNRELEASED,sid" >> "${tmp}"

You can achieve the same effect using sbuild-createchroot by using the --alias
option. No need to manually mangle the schroot config.

Done.

Post by Johannes Schauer

Post by Geert Stappers
# Enable eatmydata: occasionally losing a test build is preferable over

longer

Post by Geert Stappers
# build times and disk wear.
echo "command-prefix=eatmydata" >> "${tmp}"

I don't see sbuild-createchroot mangling with command-prefix at all. So why do
you filter it out initially?
Would it not be better to add a command line option to sbuild-createchroot
which allows setting a custom command-prefix? Then you would not need to
manually mangle the created config file at all and lots of the problems I
mentioned above and other fragile things would go away.

Sure. Can you take care of this or do you want me to send a patch?

Post by Johannes Schauer

Post by Geert Stappers
chmod 644 "${tmp}"
mv "${tmp}" "${config}"
# Copy a modified example sbuildrc config file
for homedir in $(getent passwd | perl -F: -nlE '$F[2] >= 1000 && $F[2]

<= 59999 && say $F[5]')

Post by Geert Stappers
do
userconfig="${homedir}/.sbuildrc"
if [ ! -e "${userconfig}" ]
then
(grep -v -E "^(# don't remove this|1;\$)"

/usr/share/doc/sbuild/examples/example.sbuildrc && cat
/usr/share/doc/sbuild-debian-setup/sbuildrc) > "${userconfig}"

Post by Geert Stappers
fi
done

Instead of crafting a custom sbuildrc, we can also talk about changing the
existing defaults.
For example you set "$build_arch_all = 1;" I can certainly see how it makes
sense to change this default for sbuild but keep it at 0 for buildd.

Sounds good. Same question: can you take care of this, or should I send a
patch?

Post by Johannes Schauer
Then you set "$build_source = 1;" but I don't know why you would need this.
Sbuild is not there to build the source package. The source package is its
input not its output. Why would you want sbuild to build it again?

The wiki page contained some confusing advice, which I now removed.

Post by Johannes Schauer
Lastly, you set "$run_lintian = 1;" which I also think would be a sensible
thing to change in the defaults.

Sounds good, see above.

Post by Johannes Schauer
As a result, you would not need to manually craft a custom sbuildrc anymore.

Post by Geert Stappers
# bind-mount the apt archive cache into chroots, so that packages are

downloaded

Post by Geert Stappers
# only once. The assumption is that users will not typically have a

local apt

Post by Geert Stappers
# mirror or caching proxy.
if ! grep -q '^/var/cache/apt/archives' /etc/schroot/sbuild/fstab
then
echo "/var/cache/apt/archives /var/cache/apt/archives none rw,bind

0 0" \

Post by Geert Stappers

Post by Geert Stappers
/etc/schroot/sbuild/fstab

fi

And who cleans up /var/cache/apt/archives regularly when it becomes cluttered
over time?
I also don't like the idea of just randomly mixing the
/var/cache/apt/archives
directory of the host with multiple sbuild instances which might be running at
the same time. Just imagine sbuild running for different distributions which
have packages with the same name and version but different content. That is
just calling for trouble.
I think a much better option is to use something like apt-cacher-ng. This new
package could easily depend on it and then use that as its default mirror as
already explained above.

Done.

Post by Johannes Schauer

Post by Geert Stappers
if [ ! -e "/etc/schroot/setup.d/04tmpfs" ]
then
echo ""
echo " If you can spare the RAM, you can enable building in tmpfs

using:"

Post by Geert Stappers
echo ""
echo " sudo ln -s /etc/schroot/setup-available.d/overlays-in-tmpfs

/etc/schroot/setup.d/04tmpfs"

Post by Geert Stappers
echo ""
fi

This will affect *all* schroot sessions and not just those used by sbuild. As
such, this should rather go into documentation for schroot. But maybe there is
a way to make this mounting specific to sbuild schroots?

I don’t know, you’re the expert :). I think users who know what schroot is
(to me it’s “the thing which sbuild uses”) and who use it for anything but
sbuild will realize what’s happening here. We could add some wording to
that extent.

Post by Johannes Schauer
- improving existing documentation

I updated the wiki page based on your review.

Post by Johannes Schauer
- changing configuration defaults to more sensible values
- adding more options to existing tools like sbuild-createchroot
- using existing tools instead of using creative ways to work around them
(apt-cacher-ng)
If we do all that, what is left that a new script should take care of?

• Adding the user(s) to the sbuild group
• Creating the chroot with all the options we discussed
• Suggesting to build in tmpfs
• Periodically updating the schroots (not strictly speaking part of the
script itself, but listing it here anyway)

Post by Johannes Schauer
Thanks!
cheers, josch

--
Best regards,
Michael