Several things conspired. First, I intended the yum updates to Tempest to *not* happen automatically, because if anything went wrong, I wanted them to happen moments after I had personally typed in the “yum update” command.
On the clients, updates happen automatically via cron at 1am on Friday morning, via a link in /etc/cron.weekly to /usr/network/sbin/client-up2date, which is mostly a wrapper around “yum update” to log a bunch of stuff.
At some point (Sept 25th, 2012, I think) a link to client-up2date was put in cron.weekly on Tempest, so it started automatically doing updates every Friday. How we didn’t notice this is a mystery.
So, that explains the following:
[root@tempest ~] grep httpd /var/log/yum.log
Feb 22 01:34:08 Updated: httpd-tools-2.2.15-26.el6.x86_64
Feb 22 01:35:47 Updated: httpd-2.2.15-26.el6.x86_64
So, at 1:35, httpd was updated. That, I think, put into /etc/httpd/conf.d the following files:
ssl.conf
authz_ldap.conf
Now, we had had those files in there, but when we installed the CAS authentication, it requested that we rename those to be
00_ssl.conf
10_authz_ldap.conf
and to put the following in there as well:
05_auth_cas.conf
Presumably, so that they would be loaded in the correct order (default is directory order). So, that means we had both ssl.conf and 00_ssl.conf, both of which said, among other things, to Listen on port 443 and configured a virtual host on that port. Two such modules conflict, and so when httpd was restarted after the yum update, httpd failed to restart.
It actually is continuing to have trouble re-starting, which I think is due to not having the correct PID file; I’m still working on that.
Anyhow, now that httpd is running again, the next question is now to prevent this from happening again. Step one is to remove that link and to modify the client-up2date script so that even if it’s inadvertently run on the server it will automatically exit instead of running yum. My first thought, which I implemented, was to check the server name, but that will fail if we rename the server or something. Now that I think about it, I think a better approach would be for the script to look for something that will default to failing, such as looking for a special file such as “/etc/wellesley-cs-client-machine” before proceeding. Alternatively, it could look for its own name in the /etc/centos-clients file, but we’d need to move that file to /usr/network.
Step two is to prevent the conflicting ssl.conf file to be put in there by yum. If the ssl.conf file isn’t there, yum update will always put one there, and we’ll again have the conflict. But we need (I think) to have the 00_ssl.conf file.
Ah! I think I just had an idea. If we create a ssl.conf file, but make it *empty* or innocuous, yum should create a ssl.conf.rpmnew file *instead* of replacing the empty one. At least, that’s an idea. Need to investigate how yum decides whether to create .rpmnew (keep local version, skipping yum updated version) versus .rpmsave (the reverse).
