Discussion:
Bug#758788: cryptsetup: Passphrase caching broken in decrypt_keyctl
(too old to reply)
Marek Królikowski
2014-08-21 11:22:24 UTC
Permalink
Package: cryptsetup
Version: 2:1.6.6-1
Severity: normal

Dear Maintainer,
after upgrade to 2:1.6.6-1 I have to enter passphrase for each of crypttab entries.
Desired behavior is for cryptsetup to try cached passphrase for crypptab entries with
the same identifier (pw1 in this case).

Working configurations:
- cryptsetup 2:1.6.4-4
- cryptsetup 2:1.6.6-1 with decrypt_keyctl copied from 2:1.6.4-4


-- Package-specific info:
-- /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.16-trunk-amd64 root=/dev/mapper/debian-system ro quiet

-- /etc/crypttab
sdaX_crypt UUID=redacted pw1 luks,keyscript=decrypt_keyctl
sdbX_crypt UUID=redacted pw1 luks,discard,keyscript=decrypt_keyctl

-- System Information:
Debian Release: jessie/sid
APT prefers unstable
APT policy: (995, 'unstable'), (991, 'experimental'), (101, 'testing'), (100, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 3.16-trunk-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages cryptsetup depends on:
ii cryptsetup-bin 2:1.6.6-1
ii debconf [debconf-2.0] 1.5.53
ii dmsetup 2:1.02.88-1
ii libc6 2.19-9

Versions of packages cryptsetup recommends:
ii busybox 1:1.22.0-8
ii console-setup 1.111
ii initramfs-tools [linux-initramfs-tool] 0.115
ii kbd 1.15.5-1

Versions of packages cryptsetup suggests:
ii dosfstools 3.0.26-3
ii keyutils 1.5.9-5
ii liblocale-gettext-perl 1.05-8+b1

-- debconf information excluded
Jonas Meurer
2014-08-21 14:22:21 UTC
Permalink
Hello Marek,

thanks for your bugreport.
Post by Marek Królikowski
Dear Maintainer,
after upgrade to 2:1.6.6-1 I have to enter passphrase for each of crypttab entries.
Desired behavior is for cryptsetup to try cached passphrase for crypptab entries with
the same identifier (pw1 in this case).
- cryptsetup 2:1.6.4-4
- cryptsetup 2:1.6.6-1 with decrypt_keyctl copied from 2:1.6.4-4
The decrypt_keyctl keyscript was changed with the last upload of
cryptsetup in order to fix a bug when a wrong passphrase was entered.
See the bugreport for more details:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=748368

Do you unlock your devices in initramfs or with the initscript?

I guess that something is wrong with the environment variable
CRYPTTAB_TRIED. This variable should contain the count of tries to
unlock one particular dm-crypt device.

Can you add some debugging code to your script in order to print the
value of CRYPTTAB_TRIED?

A simple 'echo "CRYPTTAB_TRIED: $CRYPTTAB_TRIED"' at line 36 of the
script should already do the job. Please report back your observations.

Also, can you please report back what happens if you enter a wrong
password (both with the new version of decrypt_keyctl and with the old one).

Kind regards,
jonas
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Luc Maisonobe
2014-09-03 07:46:25 UTC
Permalink
Hi encountered the exact same problem on a laptop with two different
encrypted partition (one for a LVM on a SSD and one for a LVM on a HDD).

Setting echo "CRYPTTAB_TRIED: $CRYPTTAB_TRIED"' at line 36 of the
script does NOT work, and in fact it completely prevent the computer to
boot, as it gets stuck in an infinite loop complaining about wrong
passwphrase. The echo is not displayed at all during this loop.

I don't know were the echo goes and I don't know what passphrase is read
(I wonder the echo may garble the input).

I had to reboot the computer on a live system containing cryptsetup to
manually reset everything back to work (running all the commands
manually: cryptsetup luksOpen for the two encrypted partitions, vgchange
-a y to find the LVM components, mounting everything, mounting /dev,
/sys and /proc with --bind, chroot, edit the script to remove the echo,
update-initramfs -u -k all to make sure the updated script is in the
initramfs, and reboot).

So, back to square one, how can we identify the reason why the caching
script does not work anymore?

best regards,
Luc
Jonas Meurer
2014-10-03 15:34:23 UTC
Permalink
Hey Marek and Luc,
Post by Luc Maisonobe
Hi encountered the exact same problem on a laptop with two different
encrypted partition (one for a LVM on a SSD and one for a LVM on a HDD).
I failed to reproduce the bug you discovered so far. Can you please give
the latest packages from
https://people.debian.org/~mejo/debian/mejo-unstable/ a try and see
whether decrypt_keyctl still doesn't work for you?

In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.

Kind regards,
jonas
Luc Maisonobe
2014-10-03 19:15:24 UTC
Permalink
Post by Jonas Meurer
Hey Marek and Luc,
Hi Jonas,
Post by Jonas Meurer
Post by Luc Maisonobe
Hi encountered the exact same problem on a laptop with two different
encrypted partition (one for a LVM on a SSD and one for a LVM on a HDD).
I failed to reproduce the bug you discovered so far. Can you please give
the latest packages from
https://people.debian.org/~mejo/debian/mejo-unstable/ a try and see
whether decrypt_keyctl still doesn't work for you?
The new packages allow to boot, but I still have to enter the key twice,
once for each encrypted device.
Post by Jonas Meurer
In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.
Here they are:

/etc/fstab:
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point> <type> <options> <dump> <pass>
/dev/mapper/vg--ssd-root / ext4 errors=remount-ro 0 1
# /boot was on /dev/sda1 during installation
UUID=4d228296-ceae-45c5-bbeb-86677a4fdfe2 /boot ext2
defaults 0 2
/dev/mapper/vg--hdd-home /home ext4 defaults 0 2
/dev/mapper/vg--ssd-usr /usr ext4 defaults 0 2
/dev/mapper/vg--ssd-var /var ext4 defaults 0 2
/dev/mapper/vg--hdd-swap_1 none swap sw 0 0
/dev/sr0 /media/cdrom0 udf,iso9660 user,noauto 0 0
# /dev/sdc1 /media/usb0 auto rw,user,noauto 0 0
# /dev/sdc2 /media/usb1 auto rw,user,noauto 0 0



/etc/crypttab:

sdb1_crypt UUID=9aa983b5-0224-406b-a177-7481162c6172
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
sda5_crypt UUID=3764df68-de26-4a24-a7dc-1498cb6b20ab
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl


Note that the two partitions contain physical volumes for LVM, as shown
here:

***@marislae:~# pvdisplay
--- Physical volume ---
PV Name /dev/mapper/sda5_crypt
VG Name vg-hdd
PV Size 698,40 GiB / not usable 0
Allocatable yes (but full)
PE Size 4,00 MiB
Total PE 178789
Free PE 0
Allocated PE 178789
PV UUID E1pkgZ-BFN4-1JFd-4HGA-YW1B-XXww-mubuR1

--- Physical volume ---
PV Name /dev/mapper/sdb1_crypt
VG Name vg-ssd
PV Size 29,82 GiB / not usable 2,84 MiB
Allocatable yes (but full)
PE Size 4,00 MiB
Total PE 7632
Free PE 0
Allocated PE 7632
PV UUID eaY3yu-PvoQ-1u6o-P7tK-wpRB-tVUJ-sWyMqh

***@marislae:~#


best regards,
Luc
Post by Jonas Meurer
Kind regards,
jonas
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Jonas Meurer
2014-10-03 19:55:18 UTC
Permalink
Hey Luc,

thanks for the prompt reply.
Post by Luc Maisonobe
Post by Jonas Meurer
I failed to reproduce the bug you discovered so far. Can you please give
the latest packages from
https://people.debian.org/~mejo/debian/mejo-unstable/ a try and see
whether decrypt_keyctl still doesn't work for you?
The new packages allow to boot, but I still have to enter the key twice,
once for each encrypted device.
Very strange. I'm still unable to reproduce the issues you encounter.
Could you do some futher testing for me?

I test the decrypt_keyctl script with the following setup, and it works
as expected. Maybe you could try a similar setup:

- create two small lvm logical volumes (5MB are more than enough)
- luksformat both logical volumes
- add them to your crypttab:

clv1_crypt /dev/<VG>/<LV1> testkey1 luks,keyscript=decrypt_keyctl
clv2_crypt /dev/<VG>/<LV2> testkey1 luks,keyscript=decrypt_keyctl

- try unlocking them via cryptdisks_start:

# cryptdisks_start clv1_crypt
# cryptdisks_start clv2_crypt

The second unlocking should use the key cached during first unlocking.

It would be awesome if you could test this. I as well tested this setup
during boot process, and it works as expected as well. Also tested with
UUID instead of source device path in crypttab, same result.

I've no glue what's different on your setups, and any help with
debugging would be highly appreciated.
Post by Luc Maisonobe
Post by Jonas Meurer
In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.
sdb1_crypt UUID=9aa983b5-0224-406b-a177-7481162c6172
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
sda5_crypt UUID=3764df68-de26-4a24-a7dc-1498cb6b20ab
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
Nothing suspicious here, looks ok to me.
Post by Luc Maisonobe
Note that the two partitions contain physical volumes for LVM, as shown
Actually the content of your encrypted devices should not matter at all.

Kind regards,
jonas
luc
2014-10-09 07:54:38 UTC
Permalink
Post by Jonas Meurer
Hey Luc,
Hi Jonas,
Post by Jonas Meurer
Post by Jonas Meurer
Post by Luc Maisonobe
Post by Jonas Meurer
I failed to reproduce the bug you discovered so far. Can you please give
the latest packages from
https://people.debian.org/~mejo/debian/mejo-unstable/ a try and see
whether decrypt_keyctl still doesn't work for you?
The new packages allow to boot, but I still have to enter the key twice,
once for each encrypted device.
Very strange. I'm still unable to reproduce the issues you encounter.
Could you do some futher testing for me?
Did you find time to do the additional testing/debugging yet? I'd like
to fix this bug in time for Debian Jessie, provided that it's really a
bug in the package and not an issue on your side ;)
Not yet, but I intend to do so.
The problem occurs on a computer with some critical information on it,
and the partitions already use the full disk space. This implies I have
to do
some careful work of shrinking filesystems, logical volumes and so on
before I can
set up a test partition aside.
Post by Jonas Meurer
As already mentioned - up to now I'm unable to reproduce the bug. For
me, decrypt_keyctl works in all tested setups. The provided passphrase
is stored in kernel keyring with first invokation and used for all the
following device unlockings that have the same keyscript/keyname
configured.
I understand your point. It is difficult to debug here (as mentioned
earlier putting
some echo commands completely trashed the boot sequence). I'll do my
best.

best regards,
Luc
Post by Jonas Meurer
Kind regards,
jonas
Post by Jonas Meurer
I test the decrypt_keyctl script with the following setup, and it works
- create two small lvm logical volumes (5MB are more than enough)
- luksformat both logical volumes
clv1_crypt /dev/<VG>/<LV1> testkey1 luks,keyscript=decrypt_keyctl
clv2_crypt /dev/<VG>/<LV2> testkey1 luks,keyscript=decrypt_keyctl
# cryptdisks_start clv1_crypt
# cryptdisks_start clv2_crypt
The second unlocking should use the key cached during first unlocking.
It would be awesome if you could test this. I as well tested this setup
during boot process, and it works as expected as well. Also tested with
UUID instead of source device path in crypttab, same result.
I've no glue what's different on your setups, and any help with
debugging would be highly appreciated.
Post by Luc Maisonobe
Post by Jonas Meurer
In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.
sdb1_crypt UUID=9aa983b5-0224-406b-a177-7481162c6172
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
sda5_crypt UUID=3764df68-de26-4a24-a7dc-1498cb6b20ab
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
Nothing suspicious here, looks ok to me.
Post by Luc Maisonobe
Note that the two partitions contain physical volumes for LVM, as shown
Actually the content of your encrypted devices should not matter at all.
Kind regards,
jonas
_______________________________________________
pkg-cryptsetup-devel mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-cryptsetup-devel
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Jonas Meurer
2014-10-09 08:39:29 UTC
Permalink
Hey Luc,
Post by luc
Post by Jonas Meurer
Did you find time to do the additional testing/debugging yet? I'd like
to fix this bug in time for Debian Jessie, provided that it's really a
bug in the package and not an issue on your side ;)
Not yet, but I intend to do so.
The problem occurs on a computer with some critical information on it,
and the partitions already use the full disk space. This implies I have
to do
some careful work of shrinking filesystems, logical volumes and so on
before I can
set up a test partition aside.
I see. But you don't need to resize your filesystems or go through
similar hassle. Simply use file containers for testing. The following
commands should setup a testing environment (use carefully, even though
I tested them):

# dd if=/dev/urandom of=/cont1 bs=1M count=3
# dd if=/dev/urandom of=/cont2 bs=1M count=3
# echo "testpw" | cryptsetup luksFormat /cont1
# echo "testpw" | cryptsetup luksFormat /cont2
# echo "cont1_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by luc
Post by Jonas Meurer
/etc/crypttab
# echo "cont2_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by luc
Post by Jonas Meurer
/etc/crypttab
# cryptdisks_start cont1_crypt
# cryptdisks_start cont2_crypt
Post by luc
Post by Jonas Meurer
As already mentioned - up to now I'm unable to reproduce the bug. For
me, decrypt_keyctl works in all tested setups. The provided passphrase
is stored in kernel keyring with first invokation and used for all the
following device unlockings that have the same keyscript/keyname
configured.
I understand your point. It is difficult to debug here (as mentioned
earlier putting
some echo commands completely trashed the boot sequence). I'll do my best.
I'm sorry that I brought you in troubles here. The echo command was
untested and I at least should have written that. It was meant for
debugging purposes only but it completely broke the keyscript. I'll try
to not make such premature requests again :-/

Cheers,
jonas
Post by luc
Post by Jonas Meurer
Post by Jonas Meurer
I test the decrypt_keyctl script with the following setup, and it works
- create two small lvm logical volumes (5MB are more than enough)
- luksformat both logical volumes
clv1_crypt /dev/<VG>/<LV1> testkey1 luks,keyscript=decrypt_keyctl
clv2_crypt /dev/<VG>/<LV2> testkey1 luks,keyscript=decrypt_keyctl
# cryptdisks_start clv1_crypt
# cryptdisks_start clv2_crypt
The second unlocking should use the key cached during first unlocking.
It would be awesome if you could test this. I as well tested this setup
during boot process, and it works as expected as well. Also tested with
UUID instead of source device path in crypttab, same result.
I've no glue what's different on your setups, and any help with
debugging would be highly appreciated.
Post by Luc Maisonobe
Post by Jonas Meurer
In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.
sdb1_crypt UUID=9aa983b5-0224-406b-a177-7481162c6172
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
sda5_crypt UUID=3764df68-de26-4a24-a7dc-1498cb6b20ab
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
Nothing suspicious here, looks ok to me.
Post by Luc Maisonobe
Note that the two partitions contain physical volumes for LVM, as shown
Actually the content of your encrypted devices should not matter at all.
Kind regards,
jonas
_______________________________________________
pkg-cryptsetup-devel mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-cryptsetup-devel
_______________________________________________
pkg-cryptsetup-devel mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-cryptsetup-devel
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Luc Maisonobe
2014-10-10 08:29:33 UTC
Permalink
Hi Jonas,
Post by Jonas Meurer
Hey Luc,
Post by luc
Post by Jonas Meurer
Did you find time to do the additional testing/debugging yet? I'd like
to fix this bug in time for Debian Jessie, provided that it's really a
bug in the package and not an issue on your side ;)
Not yet, but I intend to do so.
The problem occurs on a computer with some critical information on it,
and the partitions already use the full disk space. This implies I have
to do
some careful work of shrinking filesystems, logical volumes and so on
before I can
set up a test partition aside.
I see. But you don't need to resize your filesystems or go through
similar hassle. Simply use file containers for testing. The following
commands should setup a testing environment (use carefully, even though
# dd if=/dev/urandom of=/cont1 bs=1M count=3
# dd if=/dev/urandom of=/cont2 bs=1M count=3
# echo "testpw" | cryptsetup luksFormat /cont1
# echo "testpw" | cryptsetup luksFormat /cont2
# echo "cont1_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by luc
Post by Jonas Meurer
/etc/crypttab
# echo "cont2_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by luc
Post by Jonas Meurer
/etc/crypttab
# cryptdisks_start cont1_crypt
# cryptdisks_start cont2_crypt
On the first invocation (for cont1_crypt), I got this dialog:

***@marislae:~# cryptdisks_start cont1_crypt
[[....] Starting crypto disk...[info] cont1_crypt (starting)...
Caching passphrase for /cont1: ******
keyctl_set_timeout: Permission denied
Error setting timeout on key (2524288), removing
Caching passphrase for /cont1: ******
keyctl_set_timeout: Permission denied
Error setting timeout on key (612589418), removing

[Here I pressed <ctrl-C> to stop the attempts]

Caching passphrase for /cont1: Erreur de lecture de la phrase secrète.


I was running the commands from root. I initially logged in to the
computer from SSH to a regular user, than did "sudo -i" to get root
access if this matters. As I suspected this may be a problem, I allowed
root direct SSH access and tried again, login directly to root account,
this time it worked:

***@marislae:~# cryptdisks_start cont1_crypt
[....] Starting crypto disk...[info] cont1_crypt (starting)...
Caching passphrase for /cont1: ******
[ ok _crypt (started)...done.
***@marislae:~# cryptdisks_start cont2_crypt
[....] Starting crypto disk...[info] cont2_crypt (starting)...
Using cached passphrase for /cont2.
[ ok _crypt (started)...done.
***@marislae:~#

The /dev/mapper/cont1_crypt and /dev/mapper/cont2_crypt did appear
correctly.

Is there a way I could check the keyring just after boot, before it is
cleared? I could probably set an independent init script to run after
disks are mounted to dump the list of the keys in the keyring to some
file in /tmp so I can retrieve them once the system is up and debug. I
thing I could do this using some keyctl command, but don't know which
one to use for a given entry in /etc/crypttab. Should it be simply
"keyctl list pw1" in the case of your example or something else? I saw
in the decrypt_keyctl script some cryptkey-$1 identifier (probably used
with an _ appended). How could I use this?

best regards

Luc
Post by Jonas Meurer
Post by luc
Post by Jonas Meurer
As already mentioned - up to now I'm unable to reproduce the bug. For
me, decrypt_keyctl works in all tested setups. The provided passphrase
is stored in kernel keyring with first invokation and used for all the
following device unlockings that have the same keyscript/keyname
configured.
I understand your point. It is difficult to debug here (as mentioned
earlier putting
some echo commands completely trashed the boot sequence). I'll do my best.
I'm sorry that I brought you in troubles here. The echo command was
untested and I at least should have written that. It was meant for
debugging purposes only but it completely broke the keyscript. I'll try
to not make such premature requests again :-/
Cheers,
jonas
Post by luc
Post by Jonas Meurer
Post by Jonas Meurer
I test the decrypt_keyctl script with the following setup, and it works
- create two small lvm logical volumes (5MB are more than enough)
- luksformat both logical volumes
clv1_crypt /dev/<VG>/<LV1> testkey1 luks,keyscript=decrypt_keyctl
clv2_crypt /dev/<VG>/<LV2> testkey1 luks,keyscript=decrypt_keyctl
# cryptdisks_start clv1_crypt
# cryptdisks_start clv2_crypt
The second unlocking should use the key cached during first unlocking.
It would be awesome if you could test this. I as well tested this setup
during boot process, and it works as expected as well. Also tested with
UUID instead of source device path in crypttab, same result.
I've no glue what's different on your setups, and any help with
debugging would be highly appreciated.
Post by Luc Maisonobe
Post by Jonas Meurer
In case that you still encounter the bug, please paste your full
/etc/fstab and /etc/crypttab again.
sdb1_crypt UUID=9aa983b5-0224-406b-a177-7481162c6172
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
sda5_crypt UUID=3764df68-de26-4a24-a7dc-1498cb6b20ab
sda5_sdb1_common_key luks,keyscript=decrypt_keyctl
Nothing suspicious here, looks ok to me.
Post by Luc Maisonobe
Note that the two partitions contain physical volumes for LVM, as shown
Actually the content of your encrypted devices should not matter at all.
Kind regards,
jonas
_______________________________________________
pkg-cryptsetup-devel mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-cryptsetup-devel
_______________________________________________
pkg-cryptsetup-devel mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-cryptsetup-devel
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Jonas Meurer
2014-10-10 13:34:16 UTC
Permalink
Hey Luc,
Post by Luc Maisonobe
Post by Jonas Meurer
I see. But you don't need to resize your filesystems or go through
similar hassle. Simply use file containers for testing. The following
commands should setup a testing environment (use carefully, even though
# dd if=/dev/urandom of=/cont1 bs=1M count=3
# dd if=/dev/urandom of=/cont2 bs=1M count=3
# echo "testpw" | cryptsetup luksFormat /cont1
# echo "testpw" | cryptsetup luksFormat /cont2
# echo "cont1_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by Marek Królikowski
/etc/crypttab
# echo "cont2_crypt /cont1 pw1 luks,keyscript=decrypt_keyctl" \
Post by Marek Królikowski
/etc/crypttab
# cryptdisks_start cont1_crypt
# cryptdisks_start cont2_crypt
[[....] Starting crypto disk...[info] cont1_crypt (starting)...
Caching passphrase for /cont1: ******
keyctl_set_timeout: Permission denied
Error setting timeout on key (2524288), removing
Caching passphrase for /cont1: ******
keyctl_set_timeout: Permission denied
Error setting timeout on key (612589418), removing
[Here I pressed <ctrl-C> to stop the attempts]
Caching passphrase for /cont1: Erreur de lecture de la phrase secrète.
I was running the commands from root. I initially logged in to the
computer from SSH to a regular user, than did "sudo -i" to get root
access if this matters. As I suspected this may be a problem, I allowed
root direct SSH access and tried again, login directly to root account,
Interesting, never saw these kind of problems before. I'm testing on a
Laptop with Ubuntu 14.04 installed and use 'sudo -i' a lot. Indeed I
used it for testing the commands above as well. For me it worked. But
let's keep that aside. It's another issue and out of scope for this
bugreport ;)
Post by Luc Maisonobe
[....] Starting crypto disk...[info] cont1_crypt (starting)...
Caching passphrase for /cont1: ******
[ ok _crypt (started)...done.
[....] Starting crypto disk...[info] cont2_crypt (starting)...
Using cached passphrase for /cont2.
[ ok _crypt (started)...done.
The /dev/mapper/cont1_crypt and /dev/mapper/cont2_crypt did appear
correctly.
Looks like it worked for you as well this time. Do you still discover
the bug during the boot process with your older decrypt_keyring setup?
Did you try the test setup from above during boot process? It would be
helpful do know whether the keyscript works on boot process with the
test setup (it does work for me), and what's the exact difference
between your older decrypt_keyctl devices and the new test ones.
Post by Luc Maisonobe
Is there a way I could check the keyring just after boot, before it is
cleared? I could probably set an independent init script to run after
disks are mounted to dump the list of the keys in the keyring to some
file in /tmp so I can retrieve them once the system is up and debug. I
thing I could do this using some keyctl command, but don't know which
one to use for a given entry in /etc/crypttab. Should it be simply
"keyctl list pw1" in the case of your example or something else? I saw
in the decrypt_keyctl script some cryptkey-$1 identifier (probably used
with an _ appended). How could I use this?
The script uses the user keyring '@u' for storing the cryptdisks key.
You can list the keyring with 'keyring list @u'. In order to get the
particular key from the keyring, do 'keyring search @u user
<IDENTIFIER>', where <IDENTIFIER>' is cryptkey-pw1 in the example.
Then you get the key id, which is needed to get the key itself using
'keyctl print <KEY-ID>' or 'keyctl pipe <KEY-ID>'.

If you want to output all values during keyscript invokation for
debugging purposes, the easiest way is to modify the script at
/lib/cryptsetup/decrypt_keyctl. Add the following just after the last
line to the script:

echo -e "BEGIN DECRYPT_KEYCTL DEBUG OUTPUT\nid: $ID_ | key id:
$KID_\nkey content: $(keyctl pipe $KID_)\nkey list:\n$(keyctl list
@u)\nEND DECRYPT_KEYCTL DEBUG OUTPUT\n" >&2

Be warned, this line will print your key in plaintext to the console. If
you want to redirect the output to a logfile, simply replace '>&2' by
e.g. '>/root/decrypt_keyctl.log'.

In order to not break your boot process again, after modifying first
test the decrypt_keyctl keyscript with the test setup from above by
using cryptdisks_start/_stop and only do a reboot after you verified
that the keyscript still works.

Kind regards,
jonas
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Luc Maisonobe
2014-10-12 10:57:25 UTC
Permalink
Here are some news about the analysis of this boot problem.

First, starting and stopping test partitions with cryptdisks_start and
cryptdisks_stop does work properly after the computer has boot, but does
not work at boot time. Here is the example of the command (with
decrypt_keyctl modified to display the keys at the end, as suggested in
an earlier post).

***@marislae:/lib/systemd/system# cryptdisks_start sda3_crypt
[....] Starting crypto disk...[info] sda3_crypt (starting)...
Caching passphrase for
/dev/disk/by-uuid/4f04656d-00fc-4fea-8c8a-44a7bdc7b6e5: ******
BEGIN DECRYPT_KEYCTL DEBUG OUTPUT
id: cryptkey-pw1 | key id: 657815207
key content: testpw
key list:
1 key in keyring:
657815207: --alswrv 0 0 user: cryptkey-pw1
END DECRYPT_KEYCTL DEBUG OUTPUT

[ ok crypt (started)...done.
***@marislae:/lib/systemd/system# cryptdisks_start sda5_crypt
[ ok ] Starting crypto disk...sda5_crypt (running)...done.
***@marislae:/lib/systemd/system#

As you can see, running it on an already started system is fine.

On the other hand, the same setting does not work at boot time.

Back to the original problem, without the test partitions and using only
my regular two work partitions sda5 and sdb1 holding LVM volumes. The
modification of the decrypt_keyctl to display the key on boot showed
that after the key for the first partition was entered, it was stored in
the user keyring. I got something like:

id: cryptkey-sda5b1 | key id: 1007139041
key list:
1 key in keyring:
1007139041: --alswrv 0 0 user: cryptkey-sda5b1

Then the system asked for the second key, ignoring the one it already
known, and the following is displayed:

id: cryptkey-sda5b1 | key id: 422064942
key list:
1 key in keyring:
422064942: --alswrv 0 0 user: cryptkey-sda5b1

So the id was exactly the same, but the key id was a new one. It seems
the keyring is cleared sometime between the two runs of the script and
in each case only one key was stored in the keyring. I think the culprit
is there: why did key 1007139041 disappear from keyring between the two
commands?

Another important information is that this system uses systemd, and it
seems decrypting is handled differently with systemd. If I run
journalctl to get information, I see this message:

oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Password file path
'sda5b1' is not absolute. Ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Encountered unknown
/etc/crypttab option 'keyscript=decrypt_keyctl', ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Volume sda5_crypt
already active.
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Password file path
'sda5b1' is not absolute. Ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Encountered unknown
/etc/crypttab option 'keyscript=decrypt_keyctl', ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Volume sdb1_crypt
already active.

However, this message seems to be generated after the partitions have
already been activated, and from the message displayed on the console, I
know decrypt_keyctl did run, despite these messages.

Yet another information is I noticed earlier in journalctl the message
oct. 12 12:08:05 marislae systemd[1]: /usr appears to be on its own
filesytem and is not already mounted. This is not a supported setup.
This problem is explained here:
<http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/>.
I don't know if it is related to my problem, I will try to fix this by
moving /usr content into the / partition and see if it helps.

best regards,
Luc
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Jonas Meurer
2014-10-12 12:20:11 UTC
Permalink
Hey Luc,
Post by Luc Maisonobe
Here are some news about the analysis of this boot problem.
First, great that you finally managed to use the debugging version that
I sent earlier.
Post by Luc Maisonobe
First, starting and stopping test partitions with cryptdisks_start and
cryptdisks_stop does work properly after the computer has boot, but does
not work at boot time. Here is the example of the command (with
decrypt_keyctl modified to display the keys at the end, as suggested in
an earlier post).
[....] Starting crypto disk...[info] sda3_crypt (starting)...
Caching passphrase for
/dev/disk/by-uuid/4f04656d-00fc-4fea-8c8a-44a7bdc7b6e5: ******
BEGIN DECRYPT_KEYCTL DEBUG OUTPUT
id: cryptkey-pw1 | key id: 657815207
key content: testpw
657815207: --alswrv 0 0 user: cryptkey-pw1
END DECRYPT_KEYCTL DEBUG OUTPUT
[ ok crypt (started)...done.
[ ok ] Starting crypto disk...sda5_crypt (running)...done.
As you can see, running it on an already started system is fine.
Looks good so far.
Post by Luc Maisonobe
On the other hand, the same setting does not work at boot time.
Back to the original problem, without the test partitions and using only
my regular two work partitions sda5 and sdb1 holding LVM volumes. The
modification of the decrypt_keyctl to display the key on boot showed
that after the key for the first partition was entered, it was stored in
id: cryptkey-sda5b1 | key id: 1007139041
1007139041: --alswrv 0 0 user: cryptkey-sda5b1
Then the system asked for the second key, ignoring the one it already
id: cryptkey-sda5b1 | key id: 422064942
422064942: --alswrv 0 0 user: cryptkey-sda5b1
So the id was exactly the same, but the key id was a new one. It seems
the keyring is cleared sometime between the two runs of the script and
in each case only one key was stored in the keyring. I think the culprit
is there: why did key 1007139041 disappear from keyring between the two
commands?
Is this output from the boot process or did you try to unlock the
encrypted LVM volumes after boot process finished?
Post by Luc Maisonobe
Another important information is that this system uses systemd, and it
seems decrypting is handled differently with systemd. If I run
This information changes everything! I never tried systemd myself so far
and I don't know nothing about how the systemd boot process handles
dm-crypt devices/cryptdisks/crypttab at all.
Post by Luc Maisonobe
oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Password file path
'sda5b1' is not absolute. Ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Encountered unknown
/etc/crypttab option 'keyscript=decrypt_keyctl', ignoring.
This looks suspicious to me. Maybe systemd has its own crypttab
implementation which doesn't support all the crypttab options/functions
from Debian sysvinit init and initramfs scripts? At least some
'systemd-cryptsetup' process seems to process crypttab entries and
detects an unsupported option 'keyscript'.
Post by Luc Maisonobe
oct. 12 12:08:05 marislae systemd-cryptsetup[574]: Volume sda5_crypt
already active.
At the same time, the encrypted device is detected as 'already active'.
Doesn't make sense to me.

But nevertheless, I can imagine that systemd-cryptsetup detects unknown
crypttab options and falls back to the default password prompt
afterwards. That would explain why you're asked for the password of
every encrypted device.
Post by Luc Maisonobe
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Password file path
'sda5b1' is not absolute. Ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Encountered unknown
/etc/crypttab option 'keyscript=decrypt_keyctl', ignoring.
oct. 12 12:08:05 marislae systemd-cryptsetup[575]: Volume sdb1_crypt
already active.
However, this message seems to be generated after the partitions have
already been activated, and from the message displayed on the console, I
know decrypt_keyctl did run, despite these messages.
Ok, another option would be that the initscript is executed first, and
afterwards the systemd-cryptsetup helper binary iterates over crypttab
entries again and detects that the devices are already unlocked.
Post by Luc Maisonobe
Yet another information is I noticed earlier in journalctl the message
oct. 12 12:08:05 marislae systemd[1]: /usr appears to be on its own
filesytem and is not already mounted. This is not a supported setup.
<http://freedesktop.org/wiki/Software/systemd/separate-usr-is-broken/>.
I don't know if it is related to my problem, I will try to fix this by
moving /usr content into the / partition and see if it helps.
Yeah, I remember I've read somewhere that systemd doesn't support
separate /usr partitions. But again, I don't know nothing about systemd
yet. Never tried it so far.

So I don't even know at which state systemd executes the cryptdisks
initscript, and how it interacts with the kernel keyrings.

I'm eager to say that your problems are related to systemd init system.
Honestly, I don't know when I will find time to give systemd a try and
further debug this issue :-/

Cheers,
jonas
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Luc Maisonobe
2014-10-12 12:33:31 UTC
Permalink
Post by Jonas Meurer
Hey Luc,
Post by Luc Maisonobe
Back to the original problem, without the test partitions and using only
my regular two work partitions sda5 and sdb1 holding LVM volumes. The
modification of the decrypt_keyctl to display the key on boot showed
that after the key for the first partition was entered, it was stored in
id: cryptkey-sda5b1 | key id: 1007139041
1007139041: --alswrv 0 0 user: cryptkey-sda5b1
Then the system asked for the second key, ignoring the one it already
id: cryptkey-sda5b1 | key id: 422064942
422064942: --alswrv 0 0 user: cryptkey-sda5b1
So the id was exactly the same, but the key id was a new one. It seems
the keyring is cleared sometime between the two runs of the script and
in each case only one key was stored in the keyring. I think the culprit
is there: why did key 1007139041 disappear from keyring between the two
commands?
Is this output from the boot process or did you try to unlock the
encrypted LVM volumes after boot process finished?
It was the output of the boot process.
Post by Jonas Meurer
So I don't even know at which state systemd executes the cryptdisks
initscript, and how it interacts with the kernel keyrings.
I'm eager to say that your problems are related to systemd init system.
Honestly, I don't know when I will find time to give systemd a try and
further debug this issue :-/
If you can have a running system without systemd, you are lucky. I was
really forced to migrate by some packages updates and so far am really
not happy with it. It changed a lot of things and it seems not all
packages are ready for it.

I'll try to debug further and understand what happens to the keyring.

best regards,
Luc
Post by Jonas Meurer
Cheers,
jonas
--
To UNSUBSCRIBE, email to debian-bugs-dist-***@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact ***@lists.debian.org
Loading...