This happened a while ago. After applying some updates to my LXD containers and restarted everything, I was surprised to discover that all my websites depending on MariaDB weren’t able to connect to the server.

When I tried to connect for the CLI on one of the affected containers, I go this weird error:

root@wordpress ~# mysql --host mysql.kokoro.infra -u wordpress -p
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 107 "Transport endpoint is not connected"

With the IPv4 of the MariaDB container:

root@wordpress ~# mysql --host 10.80.245.19 -u wordpress -p
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 "Internal error/check (Not system error)"

These are some unusual errors that seem more low-level than I’m used to. I couldn’t find much about them. I tried to disable AppArmor on the containers without luck.

Then this helped. What happens if we use netcat to connect to MariaDB?

root@wordpress ~# nc 10.80.245.19 3306
Host '10.80.245.15' is not allowed to connect to this MariaDB server

Now, that’s a much more explicit error compared to what the MariaDB client was returning.

The MySQL users were using their respective @<container>.lxd hostname. For example [email protected]. I don’t know what changed with the updates I made, but something did. After updating all my users to use the IPv4 of their respective container instead of the .lxd domains, I was able to connect:

root@wordpress ~# nc 10.80.245.19 330
u
5.5.5-10.4.13-MariaDB-1:10.4.13+maria~stretch[...]

root@wordpress ~# mysql --host 10.80.245.19 -u wordpress -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \\g.
Your MariaDB connection id is 80
Server version: 10.4.13-MariaDB-1:10.4.13+maria~stretch-log mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.

MariaDB [(none)]>

Not sure what happened…