Use netcat when the MySQL client lies to you
This happened a while ago. After applying updates to my LXD containers and restarting everything, I discovered that none of my websites depending on MariaDB could connect to the server.
When I tried to connect from the CLI on one of the affected containers, I got this weird error:
root@wordpress ~# mysql --host mysql.kokoro.infra -u wordpress -p
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 107 "Transport endpoint is not connected"
And with the IPv4 of the MariaDB container:
root@wordpress ~# mysql --host 10.80.245.19 -u wordpress -p
Enter password:
ERROR 2013 (HY000): Lost connection to MySQL server at 'reading initial communication packet', system error: 0 "Internal error/check (Not system error)"
These errors seemed more low-level than usual, and I couldn’t find much documentation about them. Even disabling AppArmor on the containers didn’t help.
Then this article gave me an idea. What if we used netcat to connect to MariaDB?
root@wordpress ~# nc 10.80.245.19 3306
Host '10.80.245.15' is not allowed to connect to this MariaDB server
Now, that’s a much more explicit error compared to what the MariaDB client was returning.
The MySQL users were configured to use their respective @<container>.lxd hostname (for example, [email protected]). Something in the updates must have changed how these hostnames were resolved. After updating all users to use the IPv4 of their respective container instead of the .lxd domains, everything started working again.
root@wordpress ~# nc 10.80.245.19 330
u
5.5.5-10.4.13-MariaDB-1:10.4.13+maria~stretch[...]
root@wordpress ~# mysql --host 10.80.245.19 -u wordpress -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \\g.
Your MariaDB connection id is 80
Server version: 10.4.13-MariaDB-1:10.4.13+maria~stretch-log mariadb.org binary distribution
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\\h' for help. Type '\\c' to clear the current input statement.
MariaDB [(none)]>
The root cause remains unclear, but using netcat helped reveal the actual error message that the MySQL client was hiding.