Bug#851132: /usr/sbin/mysqld: ssl_ciphers not working; mariadb built without TLS support?

[Matthew Sackman]

Package: mariadb-server-core-10.1
Version: 10.1.20-3
Severity: important
File: /usr/sbin/mysqld

Hi,

In my /etc/mysql/mariadb.conf.d/50-server.cnf file, I have
ssl_cipher=TLSv1.2

However, on startup, the logs contain:

2017-01-12 8:53:04 139750636693376 [Warning] Failed to setup SSL
2017-01-12 8:53:04 139750636693376 [Warning] SSL error: Failed to set ciphers to use

It's difficult to verify at this point, but I'm fairly sure this used to
work fine. Even more interesting is that ldd gives:

# ldd /usr/sbin/mysqld
linux-vdso.so.1 (0x00007ffc393bf000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f559a013000)
libaio.so.1 => /lib/x86_64-linux-gnu/libaio.so.1 (0x00007f5599e11000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5599bf7000)
libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f5599984000)
libjemalloc.so.1 => /usr/lib/x86_64-linux-gnu/libjemalloc.so.1 (0x00007f559974d000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f5599513000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f559930f000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f5598f8e000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f5598c8a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f55988ec000)
/lib64/ld-linux-x86-64.so.2 (0x0000557e1ecbc000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f55986d5000)

which does not contain any ssl/tls lib that I can see. This makes me
wonder whether mariadb is being built currently without any ssl/tls
support?

-- System Information:
Debian Release: stretch/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.8.0-2-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mariadb-server-core-10.1 depends on:
ii libaio1 0.3.110-3
ii libc6 2.24-8
ii libjemalloc1 3.6.0-9
ii libpcre3 2:8.39-2
ii libstdc++6 6.2.1-5
ii mariadb-common 10.1.20-3
ii zlib1g 1:1.2.8.dfsg-4

mariadb-server-core-10.1 recommends no packages.

mariadb-server-core-10.1 suggests no packages.

-- no debconf information

Many thanks for your time.

Matthew

[Otto Kekäläinen]

Hello!

Thanks for reporting. MariaDB 10.1 is built with a bundled YaSSL. I
don't directly see why you don't see the ciphers. Thanks if you can
help with bugging.

This issue might also be of interest as it explains why we have YaSSL
instead of OpenSSL:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=787118

[Otto Kekäläinen]

Hello,
Thank you for your reply. I have read through the links you provided,
though I'm afraid I'm still unable to fix this.
I have ssl-ciphers set in my config, and I have even modified the init
script so that --ssl is passed to mysql (though that should be implied).
According to
https://mariadb.com/kb/en/mariadb/secure-connections-overview/ that
MariaDB [(none)]> SHOW VARIABLES LIKE 'version_ssl_library';
+---------------------+-------------+
| Variable_name | Value |
+---------------------+-------------+
| version_ssl_library | YaSSL 2.4.2 |
+---------------------+-------------+
1 row in set (0.00 sec)
MariaDB [(none)]> SHOW VARIABLES LIKE 'have_ssl';
+---------------+----------+
| Variable_name | Value |
+---------------+----------+
| have_ssl | DISABLED |
+---------------+----------+
1 row in set (0.00 sec)
I can find nothing further in any docs as to how to get this DISABLED to
change to a YES.
Matthew

I looked at the build logs and any SSL tests pass, e.g.

sys_vars.have_openssl_basic w3 [ pass ] 4
sys_vars.have_ssl_basic w3 [ pass ] 4

sys_vars.ssl_cert_basic w1 [ pass ] 4
sys_vars.ssl_cipher_basic w1 [ pass ] 6
sys_vars.ssl_crl_basic w1 [ pass ] 1
sys_vars.ssl_crlpath_basic w1 [ pass ] 1
sys_vars.ssl_key_basic w1 [ pass ] 6

No test fails.

In the build options there is -DWITH_SSL=bundled

In general the build log is full of YaSSL references. Everything should be OK.


Please somebody help debug this :)



PS. Matthew: Remember to keep the bug e-mail included in your replies
so that others can track progress of the issue.

[Otto Kekäläinen]

For the record, I also tested this on Ubuntu Yakkety with the version
10.0.28-0ubuntu0.16.10.1 and I get the same warning in syslog:

mysqld: 170119 9:47:47 [Warning] Failed to setup SSL
mysqld: 170119 9:47:47 [Warning] SSL error: Failed to set ciphers to use

It seems something with how we do " -DWITH_SSL=bundled" and YaSSL is
(and maybe always was) broken.

[Clint Byrum]

Post by Otto Kekäläinen
For the record, I also tested this on Ubuntu Yakkety with the version
mysqld: 170119 9:47:47 [Warning] Failed to setup SSL
mysqld: 170119 9:47:47 [Warning] SSL error: Failed to set ciphers to use
It seems something with how we do " -DWITH_SSL=bundled" and YaSSL is
(and maybe always was) broken.

Otto, have you tried building with WolfSSL, which is the updated and
renamed version of YaSSL? That would definitely be preferred over an
embedded version.

https://tracker.debian.org/pkg/wolfssl

[Georg Richter]

Hi,

WolfSSL has another interface (it's not C++ anymore, but C) and can't
replace Yassl on the fly. Beside TLS/SSL communication also hash functions
and crypto functions need to be migrated.

I'm currently working on a new tls/crypto wrapper for MariaDB, supporting
OpenSSL/LibreSSL, GnuTLS and SChannel (Windows) - likely it will be part of
10.3 or a later 10.2 version of MariaDB server.

/Georg

Post by Clint Byrum

Post by Otto Kekäläinen
For the record, I also tested this on Ubuntu Yakkety with the version
mysqld: 170119 9:47:47 [Warning] Failed to setup SSL
mysqld: 170119 9:47:47 [Warning] SSL error: Failed to set ciphers to use
It seems something with how we do " -DWITH_SSL=bundled" and YaSSL is
(and maybe always was) broken.

Otto, have you tried building with WolfSSL, which is the updated and
renamed version of YaSSL? That would definitely be preferred over an
embedded version.
https://tracker.debian.org/pkg/wolfssl
_______________________________________________
pkg-mysql-maint mailing list
http://lists.alioth.debian.org/cgi-bin/mailman/listinfo/pkg-mysql-maint

[Clint Byrum]

Post by Georg Richter
Hi,
WolfSSL has another interface (it's not C++ anymore, but C) and can't
replace Yassl on the fly. Beside TLS/SSL communication also hash functions
and crypto functions need to be migrated.

That's interesting. WolfSSL also has an OpenSSL compatibility layer,
could that be used?

https://www.wolfssl.com/wolfSSL/Docs-wolfssl-manual-13-openssl-compatibility.html

[Georg Richter]

Hi Clint,

I didn't check the compatibility layer for TLS/SSL stuff, there is no layer
for crypto and hashing. I did a quick hack framework for crypto wrapper
(supporting OpenSSL, WolfSSL and GnuTLS) at http://github.com/9EOR9/mrl.

A main difference between OpenSSL and WolfSSL is, that WolfSSL expects
always a ca from client - if you don't specify one verification needs to
be skipped/turned off explicitly - I'm also not sure if the compatibility
layer works well for OpenSSL 1.1 (which had a bunch of incompatible API
changes).

The best solution for MariaDB would be a wrapper library which could be
used by both MariaDB Server and Connector/C - however WolfSSL would not fit
for LGPL licensed Connector/C since it's GPL/commercial licensed.

Also the GnuTLS compatibility layer didn't work well, another hack/proof of
concept for Yassl replacement by GnuTLS can be found at
https://github.com/MariaDB/server/tree/10.2-good_bye_yassl.

We are aware of all the Yassl problems (no TLS v.1.2 and 1.3, no session
ticket support, no session renegotiation, missing ciphers, limited block
cipher support, etc) and are working on it. Connector/C 3.0 already
supports GnuTLS beside OpenSSL, and SChannel for Windows platforms.

/Georg

Post by Georg Richter

Post by Georg Richter
Hi,
WolfSSL has another interface (it's not C++ anymore, but C) and can't
replace Yassl on the fly. Beside TLS/SSL communication also hash

functions

Post by Georg Richter
and crypto functions need to be migrated.

That's interesting. WolfSSL also has an OpenSSL compatibility layer,
could that be used?
https://www.wolfssl.com/wolfSSL/Docs-wolfssl-manual-
13-openssl-compatibility.html

[Clint Byrum]

Post by Georg Richter
Hi Clint,
I didn't check the compatibility layer for TLS/SSL stuff, there is no layer
for crypto and hashing. I did a quick hack framework for crypto wrapper
(supporting OpenSSL, WolfSSL and GnuTLS) at http://github.com/9EOR9/mrl.
A main difference between OpenSSL and WolfSSL is, that WolfSSL expects
always a ca from client - if you don't specify one verification needs to
be skipped/turned off explicitly - I'm also not sure if the compatibility
layer works well for OpenSSL 1.1 (which had a bunch of incompatible API
changes).
The best solution for MariaDB would be a wrapper library which could be
used by both MariaDB Server and Connector/C - however WolfSSL would not fit
for LGPL licensed Connector/C since it's GPL/commercial licensed.
Also the GnuTLS compatibility layer didn't work well, another hack/proof of
concept for Yassl replacement by GnuTLS can be found at
https://github.com/MariaDB/server/tree/10.2-good_bye_yassl.
We are aware of all the Yassl problems (no TLS v.1.2 and 1.3, no session
ticket support, no session renegotiation, missing ciphers, limited block
cipher support, etc) and are working on it. Connector/C 3.0 already
supports GnuTLS beside OpenSSL, and SChannel for Windows platforms.

What a mess!

Note that WolfSSL includes WolfCrypt, so there are at least native crypto
primitives, just not in the OpenSSL compat layer.

However, it sounds like the way to go is probably to just use GnuTLS's
API directly rather than try and make it all work with the compatibility
layers.

IMO this is going to have to be addressed _rapidly_, as it sounds like
YaSSL is a security nightmare at this point.

[Otto Kekäläinen]

Ok, this is now figured out.

To activate YaSSL you must have 'ssl=on' in the config and no
ssl_cipher defined.


50-server.cnf: ssl=on

=>

MariaDB [(none)]> SHOW VARIABLES LIKE '%ssl%';
+---------------------+-------------+
| Variable_name | Value |
+---------------------+-------------+
| have_openssl | NO |
| have_ssl | YES |
| ssl_ca | |
| ssl_capath | |
| ssl_cert | |
| ssl_cipher | |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | |
| version_ssl_library | YaSSL 2.4.2 |
+---------------------+-------------+


YaSSL does not support TLS v1.2 for sure and when I tried 1.1 and 1.0
it didn't work either. Apparently you cannot define a cipher, as it
seems to disable YaSSL.

The solution here is to fix the documentation part:
https://anonscm.debian.org/git/pkg-mysql/mariadb-10.1.git/commit/?id=a27f5ace9786032a5d97d40964dcf8f1a6829a40

Props Vicentiu!

[Matthew Sackman]

Post by Otto Kekäläinen
Ok, this is now figured out.
To activate YaSSL you must have 'ssl=on' in the config and no
ssl_cipher defined.

Erm, ok, but this is somewhat terrifying - I can't disable insecure and
broken ciphers? I basically would consider anything < TLSv1.2
insecure and I would expect to be able to restrict even the ciphers
within TLSv1.2. This is in keeping with standard practise for apache,
dovecot, postfix etc etc.

Matthew

[Otto Kekäläinen]

Post by Matthew Sackman

Post by Otto Kekäläinen
Ok, this is now figured out.
To activate YaSSL you must have 'ssl=on' in the config and no
ssl_cipher defined.

Erm, ok, but this is somewhat terrifying - I can't disable insecure and
broken ciphers? I basically would consider anything < TLSv1.2
insecure and I would expect to be able to restrict even the ciphers
within TLSv1.2. This is in keeping with standard practise for apache,
dovecot, postfix etc etc.

I am sorry but we have basically been forbidden from using OpenSSL in
Debian due to license reasons:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761911

If you can get somebody to change their opinion, then we could use
OpenSSL. But still it is sad OpenSSL has such an license. In the long
term migrating to something else would be good.

[Matthew Sackman]

Post by Otto Kekäläinen
I am sorry but we have basically been forbidden from using OpenSSL in
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=761911
If you can get somebody to change their opinion, then we could use
OpenSSL. But still it is sad OpenSSL has such an license. In the long
term migrating to something else would be good.

Understood. I'm sorry if I came across as whining - been a long week.

I would just hope that these issues are thoroughly and prominently
documented so that we can take extra steps to try to secure
installations and make informed decisions.

Matthew