CS SysAdmin

Installing the necessary modules for CAS authentication on Tempest should be a wall in the park, thanks to all the work we did last summer to do the same for Puma to install mod_auth_cas. Those directions were base on these directions for installing CAS authentication.  First, we have to install some packages:

[root@tempest ~] yum --enablerepo=epel search php-pear-CAS
php-pear-CAS.noarch : Central Authentication Service client library in php
[root@tempest ~] yum --enablerepo=epel -y install php-pear-CAS mod_auth_cas
Package mod_auth_cas-1.0.8.1-2.el6.x86_64 already installed and latest version
Installed:
  php-pear-CAS.noarch 0:1.3.2-1.el6                                                                                                  

Complete!
[root@tempest ~] yum install mod_authz_ldap
Package mod_authz_ldap-0.26-16.el6.x86_64 already installed and latest version
Nothing to do
[root@tempest ~]

Then, we need to set the order in which they are loaded.  Apparently, we did that last summer:

[root@tempest conf.d] pwd
/etc/httpd/conf.d
[root@tempest conf.d] ls -l | head
total 116
-rw-r--r--. 1 root root  9473 Feb  7  2012 00_ssl.conf
-rw-r--r--. 1 root root   793 Aug 15  2012 05_auth_cas.conf
-rw-r--r--. 1 root root  1145 Mar 22  2007 10_authz_ldap.conf

One thing I learned, painfully, is that when there is an update to the ssl.conf file or any of these other files, they will be installed under their normal names (without the leading numbers) and then there are two copies of the content, thereby breaking Apache.  So I created empty dummy versions of the originals. Hopefully, I will get .rpmnew files instead of a broken web server:

[root@tempest conf.d] touch ssl.conf auth_cas.conf authz_ldap.conf
[root@tempest conf.d] ls -l ssl.conf auth_cas.conf authz_ldap.conf
-rw-rw----. 1 root root 0 Mar 26 16:27 auth_cas.conf
-rw-rw----. 1 root root 0 Mar 26 16:27 authz_ldap.conf
-rw-rw----. 1 root root 0 Mar 26 16:27 ssl.conf
[root@tempest conf.d]

Don’t forget to make SELinux happy:

[root@tempest conf.d] restorecon *

Next, restart the web server:

[root@tempest conf.d] apachectl graceful
[root@tempest conf.d] service httpd status
httpd (pid  13608) is running...
[root@tempest conf.d] psg httpd
apache    1454 13608  0 Mar25 ?        00:00:01 /usr/sbin/httpd
root     13608     1  0 Feb22 ?        00:01:04 /usr/sbin/httpd
apache   22198 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22199 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22200 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22201 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22202 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22203 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22204 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22205 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
apache   22206 13608  0 16:42 ?        00:00:00 /usr/sbin/httpd
[root@tempest conf.d]

And test CAS authentication on puma and on tempest.

Yay!  Seems to work.

The automatic nightly incremental backups are written to the puma:/root/snapshots directory, which is itself a mounted disk partition (to one of our external drives).  To allow users to retrieve their own lost files, but not to modify them, that directory is re-mounted read-only as /archives.  This idea, which I believe I learned about from the rsnapshot website, has worked well for a long time.

One thing I’ve known is that after un-mounting /archives, I can’t immediately un-mount /root/snapshots — it will say “device busy.”  I need to restart the nfs server (so that it will let go of /archives) and then I can do the second un-mount.  A little annoying, but not really a problem.  (It’s annoying because I haven’t scripted this as part of a server shut-down/reboot, so I have to do it by hand if I want to reboot the server.  But that’s something I do 1-2 times a year, so not a big deal.)

However, today I wanted to reboot that server.  I un-mounted /archives and then tried to restart nfs, and got:

[root@puma ~] service nfs restart
Shutting down NFS mountd:                                  [FAILED]
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down NFS services:                                [  OK  ]
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [FAILED]
[root@puma ~]

Hunh?  This is new.  Some diagnostic info:

[root@puma ~] showmount -e
mount clntudp_create: RPC: Program not registered
[root@puma ~] exportfs -a
[root@puma ~] rpcinfo -p
program vers proto   port
100000    2   tcp    111  portmapper
100000    2   udp    111  portmapper
100021    1   udp  58804  nlockmgr
100021    3   udp  58804  nlockmgr
100021    4   udp  58804  nlockmgr
100021    1   tcp  43429  nlockmgr
100021    3   tcp  43429  nlockmgr
100021    4   tcp  43429  nlockmgr
100003    2   udp   2049  nfs
100003    3   udp   2049  nfs
100003    4   udp   2049  nfs
100003    2   tcp   2049  nfs
100003    3   tcp   2049  nfs
100003    4   tcp   2049  nfs
100024    1   udp    745  status
100024    1   tcp    748  status
100011    1   udp    704  rquotad
100011    2   udp    704  rquotad
100011    1   tcp    707  rquotad
100011    2   tcp    707  rquotad

Still working on this:

[root@puma ~] mount -v /archives/
mount: trying 127.0.0.1 prog 100003 vers 3 prot tcp port 2049
mount: trying 127.0.0.1 prog 100003 vers 2 prot tcp port 2049
mount: trying 127.0.0.1 prog 100003 vers 2 prot tcp port 2049
mount: mount to NFS server ‘localhost’ failed: RPC Error: Program not registered.
[root@puma ~]

I really can’t see what’s wrong.

Our NAS (bigdata) is filling up.  Here’s the latest data:

Unfortunately, that means we don’t have room to put a live copy of Tempest’s data on bigdata, so we’ll have to think of something else.

In the meantime, we need to do a better job of keeping track of disk usage, so that we’ll have records of growth and forewarning of shortages.

To avoid network bandwidth issues when all our CentOS workstations are updated, we keep a copy of the CentOS repos on Puma in /share.

I recently updated the scripts that synchronize the copies repo from CentOS mirrors, choosing one of the “rsync” repos from this list:  http://www.centos.org/modules/tinycontent/index.php?id=30 However, the rsync ran out of space in /share, before it even got to downloading the CentOS 6.4 packages.  So, I wanted to double the space to 80GB.  All the space on Puma is allocated, but /home isn’t being used, so I wanted to reduce it and xfer the space to /share:

[root@puma ~] lvreduce --size -40G --resizefs /dev/vg0/lvhome
fsck 1.39 (29-May-2006)
/dev/mapper/vg0/vg0-lvhome: /lost+found not found.  CREATED.
/dev/mapper/vg0/vg0-lvhome:  11/35225600 files (9.1% non-contiguous), 1128681/70443008 blocks
 Command failed with status code 5.

Darn.  Searching for that error code yielded not much, but some hints that the PV (physical volume) is corrupted, so that’s bad.  Hopefully, it’s something else.

So, I just removed the lvhome partition and grew the lvstud partition (the former students partition, which we are now using for /share):

[root@puma ~] lvremove -v /dev/vg0/lvhome
Using logical volume(s) on command line
Do you really want to remove active logical volume lvhome? [y/n]: y
    Archiving volume group "vg0" metadata (seqno 45).
    Found volume group "vg0"
    Removing vg0-lvhome (253:1)
    Releasing logical volume "lvhome"
    Creating volume group backup "/etc/lvm/backup/vg0" (seqno 46)
  Logical volume "lvhome" successfully removed
[root@puma ~] lvextend -v --size=+40G --resizefs /dev/vg0/lvstud 
    Finding volume group vg0
    Executing: fsadm --verbose check /dev/vg0/lvstud 
fsadm: "ext3" filesystem found on "/dev/mapper/vg0-lvstud"
fsadm: Skipping filesystem check for device "/dev/mapper/vg0-lvstud" as the filesystem is mounted on /share
    fsadm failed: 3
    Archiving volume group "vg0" metadata (seqno 46).
  Extending logical volume lvstud to 77.50 GB
    Found volume group "vg0"
    Found volume group "vg0"
    Loading vg0-lvstud table (253:5)
    Suspending vg0-lvstud (253:5) with device flush
    Found volume group "vg0"
    Resuming vg0-lvstud (253:5)
    Creating volume group backup "/etc/lvm/backup/vg0" (seqno 47).
  Logical volume lvstud successfully resized
    Executing: fsadm --verbose resize /dev/vg0/lvstud 81264640K 
fsadm: "ext3" filesystem found on "/dev/mapper/vg0-lvstud"
fsadm: Device "/dev/mapper/vg0-lvstud" size is 83214991360 bytes
fsadm: Parsing tune2fs -l "/dev/mapper/vg0-lvstud"
fsadm: Resizing filesystem on device "/dev/mapper/vg0-lvstud" to 83214991360 bytes (9830400 -> 20316160 blocks of 4096 bytes)
fsadm: Executing resize2fs /dev/mapper/vg0-lvstud 20316160
resize2fs 1.39 (29-May-2006)
Filesystem at /dev/mapper/vg0-lvstud is mounted on /share; on-line resizing required
Performing an on-line resize of /dev/mapper/vg0-lvstud to 20316160 (4k) blocks.
The filesystem on /dev/mapper/vg0-lvstud is now 20316160 blocks long.
[root@puma ~] df -H | grep share
/dev/mapper/vg0-lvstud     82G  40G   42G  49% /share

Good enough!

We keep most disk space on our server un-allocated and grow partitions as needed.  Here’s how:

To see how much space is used:

root@tempest ~] df -H
Filesystem                         Size Used Avail Use% Mountpoint
/dev/mapper/vg_tempest-lv_root      42G 825M   39G   3% /         
none                                34G 697k   34G   1% /dev/shm  
/dev/sda1                          508M  87M  396M  18% /boot     
/dev/mapper/vg_tempest-lv_home     627G 576G   20G  97% /home     
/dev/mapper/vg_tempest-lv_students  52G  46G  3.0G  94% /students 
/dev/mapper/vg_tempest-lv_usr       17G  15G  925M  95% /usr      
/dev/mapper/vg_tempest-lv_var       49G  24G   22G  53% /var      
/dev/mapper/vg_tempest-lv_tmp       11G 159M  9.7G   2% /tmp      
bigdata:/volume1/data              3.0T 2.6T  368G  88% /data

To expand the /usr partition by 1GB, do:

[root@tempest ~] lvextend --size +1G --resizefs /dev/mapper/vg_tempest-lv_usr 
  Extending logical volume lv_usr to 16.62 GiB
  Logical volume lv_usr successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_tempest-lv_usr is mounted on /usr; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/mapper/vg_tempest-lv_usr to 4358144 (4k) blocks.
The filesystem on /dev/mapper/vg_tempest-lv_usr is now 4358144 blocks long.

We’d like to keep the /home partion not more than about 90% full, so this brings it back up to snuff:

[root@tempest ~] lvextend --size +37G --resizefs /dev/mapper/vg_tempest-lv_home 
  Extending logical volume lv_home to 629.97 GiB
  Logical volume lv_home successfully resized
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/mapper/vg_tempest-lv_home is mounted on /home; on-line resizing required
old desc_blocks = 38, new_desc_blocks = 40
Performing an on-line resize of /dev/mapper/vg_tempest-lv_home to 165142528 (4k) blocks.
The filesystem on /dev/mapper/vg_tempest-lv_home is now 165142528 blocks long.
[root@tempest ~] df -H
Filesystem                         Size Used Avail Use% Mountpoint
/dev/mapper/vg_tempest-lv_root      42G 825M   39G   3% /         
none                                34G 697k   34G   1% /dev/shm  
/dev/sda1                          508M  87M  396M  18% /boot     
/dev/mapper/vg_tempest-lv_home     666G 576G   57G  92% /home     
/dev/mapper/vg_tempest-lv_students  52G  46G  3.0G  94% /students 
/dev/mapper/vg_tempest-lv_usr       18G  15G  2.0G  89% /usr      
/dev/mapper/vg_tempest-lv_var       49G  24G   22G  53% /var      
/dev/mapper/vg_tempest-lv_tmp       11G 159M  9.7G   2% /tmp      
bigdata:/volume1/data              3.0T 2.6T  368G  88% /data     
[root@tempest ~] 

I wasn’t sure whether a particular file from a yum-managed package had changed.  (/etc/sysconfig/httpd as it happens, but that’s not essential to this problem.

So, I wanted to (1) download the RPM, and (2) look at the files in it.  This page seems to show how:  http://www.cyberciti.biz/faq/yum-downloadonly-plugin/

I first tried the download-only command-line argument:

root@tempest conf.d] yum update httpd -y --downloadonly
Loaded plugins: downloadonly, product-id, refresh-packagekit, rhnplugin, security, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
This system is receiving updates from RHN Classic or RHN Satellite.
Setting up Update Process
No Packages marked for Update
[root@tempest conf.d] ls /var/cache/yum/
x86_64
[root@tempest conf.d] find /var/cache/yum/ -name "*httpd*"
[root@tempest conf.d]

Okay, so that’s useless.  Let me try method #2:

[root@tempest conf.d] cd /root/tmp/
[root@tempest tmp] yumdownloader httpd
Loaded plugins: product-id, refresh-packagekit, rhnplugin
This system is receiving updates from RHN Classic or RHN Satellite.
httpd-2.2.15-26.el6.x86_64.rpm                                                                          | 821 kB     00:00     
[root@tempest tmp] ls -l httpd-2.2.15-26.el6.x86_64.rpm 
-rw-rw----. 1 root root 840348 Feb 18 15:45 httpd-2.2.15-26.el6.x86_64.rpm
[root@tempest tmp]

Yes, that’s much better.

Now to find that file:

[root@tempest tmp] rpm2cpio httpd-2.2.15-26.el6.x86_64.rpm | cpio -idmv [root@tempest tmp] ls -l ./etc/sysconfig/httpd 
-rw-r--r--. 1 root root 947 Dec  5 03:59 ./etc/sysconfig/httpd
[root@tempest tmp]

Excellent!  This is almost easy.

[root@tempest tmp] diff !$ /etc/sysconfig/httpd.orig 
diff ./etc/sysconfig/httpd /etc/sysconfig/httpd.orig 
31c31
< #PIDFILE=/var/run/httpd/httpd.pid
---
> #PIDFILE=/var/run/httpd.pid
[root@tempest tmp]

Ah, so that did change.  What about the startup script?

[root@tempest tmp] diff ./etc/rc.d/init.d/httpd /etc/rc.d/init.d/httpd

No change there. What about the httpd.conf file?

[root@tempest tmp] diff ./etc/httpd/conf/httpd.conf /etc/httpd/conf/httpd.conf
65c65
< PidFile run/httpd.pid
---
> PidFile /var/run/httpd.pid

Weird.  Why can’t they make up their minds?

The problem I’m trying to solve is this:

[root@tempest sysconfig] service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd: (98)Address already in use: make_sock: could not bind to address [::]:80
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
Unable to open logs
                                                           [FAILED]
[root@tempest sysconfig]

But I think that’s because it’s looking in the wrong place for the PID file, so it can’t stop httpd, and then it can’t start it because it’s already running and listening on port 80.  But, change where it’s looking for the PID file, and all is well:

[root@tempest sysconfig] pwd
/etc/sysconfig
[root@tempest sysconfig] diff httpd.orig httpd
31c31,32
< #PIDFILE=/var/run/httpd.pid
---
> # Wellesley mod:  uncomment this
> PIDFILE=/var/run/httpd.pid
[root@tempest sysconfig]

And now:

[root@tempest sysconfig] service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]
[root@tempest sysconfig]

Several things conspired.  First, I intended the yum updates to Tempest to *not* happen automatically, because if anything went wrong, I wanted them to happen moments after I had personally typed in the “yum update” command.

On the clients, updates happen automatically via cron at 1am on Friday morning, via a link in /etc/cron.weekly to /usr/network/sbin/client-up2date, which is mostly a wrapper around “yum update” to log a bunch of stuff.

At some point (Sept 25th, 2012, I think) a link to client-up2date was put in cron.weekly on Tempest, so it started automatically doing updates every Friday.  How we didn’t notice this is a mystery.

So, that explains the following:

[root@tempest ~] grep httpd /var/log/yum.log
Feb 22 01:34:08 Updated: httpd-tools-2.2.15-26.el6.x86_64
Feb 22 01:35:47 Updated: httpd-2.2.15-26.el6.x86_64

So, at 1:35, httpd was updated.  That, I think, put into /etc/httpd/conf.d the following files:

ssl.conf
authz_ldap.conf

Now, we had had those files in there, but when we installed the CAS authentication, it requested that we rename those to be

00_ssl.conf
10_authz_ldap.conf

and to put the following in there as well:

05_auth_cas.conf

Presumably, so that they would be loaded in the correct order (default is directory order).  So, that means we had both ssl.conf and 00_ssl.conf, both of which said, among other things, to Listen on port 443 and configured a virtual host on that port.   Two such modules conflict, and so when httpd was restarted after the yum update, httpd failed to restart.

It actually is continuing to have trouble re-starting, which I think is due to not having the correct PID file; I’m still working on that.

Anyhow, now that httpd is running again, the next question is now to prevent this from happening again.  Step one is to remove that link and to modify the client-up2date script so that even if it’s inadvertently run on the server it will automatically exit instead of running yum.  My first thought, which I implemented, was to check the server name, but that will fail if we rename the server or something.  Now that I think about it, I think a better approach would be for the script to look for something that will default to failing, such as looking for a special file such as “/etc/wellesley-cs-client-machine” before proceeding.  Alternatively, it could look for its own name in the /etc/centos-clients file, but we’d need to move that file to /usr/network.

Step two is to prevent the conflicting ssl.conf file to be put in there by yum.  If the ssl.conf file isn’t there, yum update will always put one there, and we’ll again have the conflict.  But we need (I think) to have the 00_ssl.conf file.

Ah!  I think I just had an idea.  If we create a ssl.conf file, but make it *empty* or innocuous, yum should create a ssl.conf.rpmnew file *instead* of replacing the empty one.  At least, that’s an idea.  Need to investigate how yum decides whether to create .rpmnew (keep local version, skipping yum updated version) versus .rpmsave (the reverse).