Bug#869557: apt: please make the output of apt-ftparchive reproducible

[Chris Lamb]

Source: apt
Version: 1.5~beta1
Severity: wishlist
User: reproducible-***@lists.alioth.debian.org

Hi,

According to Colin Percival, the output of apt-ftparchive is not
reproducible:

https://news.ycombinator.com/item?id=14834520

At least for the "packages" subcommand, it appears we do non-deterministic
directory traversal using ftw(3) and output the results as we go along.


(Sorry for the usual lack of patch (!) but I was wondering if you had any
implementation ideas/pointers before I went down the wrong rabbithole. AIUI
we either need to make the traversal determinstic — is this even possible
with ftw(3)? — or collect all the results, sort, and then output which seems
a little bit more invasive. Thoughts?)


Regards,

[Chris Lamb]

tags 869557 + patch
thanks

Hi,

apt: please make the output of apt-ftparchive reproducible

*Extremely* WIP patch attached:

commit 489251d0bcf32883e9028aa0278168c167f2f6e9
Author: Chris Lamb <***@debian.org>
Date: Mon Jul 24 15:04:03 2017 +0100

Make the output of ftp-archive reproducible by sorting non-deterministic file orders prior to output. (Closes: #869557)

ftparchive/writer.cc | 19 ++++++++++++++++++-
ftparchive/writer.h | 3 +++
2 files changed, 21 insertions(+), 1 deletion(-)


Regards,

[Chris Lamb]

Hi David,

entirely broken as they weren't generating content anymore, so,
in summary: no biggy. ;)

Odd, I wasn't getting that. Just to point out that my diff was
_explicitly_ labelled "*Extremely* WIP"! Curious I wasn't getting a
warning too? C++ is not my mother tongue, alas.

Anyway, Thanks for fixing everything up!


Regards,

[Colin Percival]

[Original complainer chiming in here]

Thanks Chris & David for working on this. As it turns out, the problem was
in a sense a case of PEBKAC -- we didn't realize that apt-sortpkgs existed!
Now that we know that, the random ordering coming out of apt-ftparchive is
no longer an issue for us... although one could argue that there is instead
a documentation bug, in that apt-ftparchive(1) should say something along
the lines of "packages will be listed in an unpredictable order; you may
wish to run apt-sortpkgs to fix this".

Since the question of why we're using apt-ftparchive: For reasons of paranoia
(we run an encrypted online backup service, so it's important to ensure that
the binaries people are running are the right ones) we build packages and sign
everything in a very locked-down environment. Using low level tools makes it
possible to do exactly what we want under exactly the right conditions, in a
way which isn't possible with a high level tool which does everything.