CS SysAdmin

With some effort and frustration, I got VirtualBox running on four of our CentOS 6.3 client machines, jay, finch, lark, and robin.  It can now run headless VMs that (re)start when the host (re)boots, and can even run 64-bit guests, such as the 2012 MIT/LL Capture the Flag VM.  Networking is a work-in-progress; presently the guests use NAT with selected ports forwarded.  Here’s how to do it.

But, first, why VirtualBox?  It’s free (as in beer, and speech, except for a proprietary extension pack that provides a few extra features).  And it it’s been actively developed for many years and seems pretty solid, if not quite as polished as VMWare.  It’s cross-platform, running pretty-much identically on Mac, Windows, and Linux machines.  And it’s backed by a giant company, Oracle, which isn’t going away anytime soon, even if it’s focused on other things and is evil.

VMWare would have been nice; it’s the king of the hill and is very mature, polished, and enterprise-y.  The MIT/LL CTF VMs are native VMWare and probably easiest to run on that platform.  But it costs a ton, so it’s a non-starter, unless you’re willing to use a 1-month trial, or score an activation code from the Grace Hopper conference.

There’s a free VMWare player that might work, but it doesn’t run on Macs, and lacks useful features such as (I think) snapshots and virtual network tweaking.

I’d really like to use KVM, the main Red Hat virtualization technology, to learn it.  I was fiddling with it on Tempest last year and it seemed pretty nice.  But, it’s Linux only, and so useless on our home machines, and I don’t have the time or patience to deal with 2 platforms at once.

So, VirtualBox it is.

BIOS Tweaks

First, reboot and enter the setup screen.  Get there by hitting Enter repeatedly until the machine beeps, then hit F1.

Power –> After Power Loss:
Change from “Last State” to “Power On”.  You’d think this would be unnecessary, but if power comes on, then goes off immediately, then comes back up, “Last State” doesn’t work.

Update:  Maybe not.  This was true for the previous micro-focus machines.  At least I inferred it after most of them stayed off after a Science Center power outage last spring.  But wren has “Last State” set, and I just unplugged/plugged it in a bunch of times, and it came back on every time.

Advanced –> CPU Setup:
–> Enable Intel(R) Virtualization Technology
–> Enable VT-d (this may be unnecessary)
–> Leave TxT disabled (explanation)

That will allow you to run 64-bit guests even under the 32-bit host OS.  These steps required being at the machine; everything else can be done remotely.

Software Installation

Reference:  http://www.if-not-true-then-false.com/2010/install-virtualbox-with-yum-on-fedora-centos-red-hat-rhel/

Get the VBox repo and make sure everything’s up to date (note that the wget line is wrapped, but it will copy-paste just fine):

mount puma:/share /share
cd /etc/yum.repos.d/
wget http://download.virtualbox.org/virtualbox/rpm/rhel/virtualbox.repo
yum update

Observe whether the kernel was upgraded at the last step, and reboot if so.  If not sure, compare the outputs of  ‘rpm -qa kernel |sort |tail -n 1’ and ‘uname -r’.

Oops, all packages are updated periodically, including VBox, if it’s not running.  Empirically, that’s bad:  /etc/vbox/vbox.cfg will be removed for no good reason, and the VMs won’t run unless the new extension pack is installed, which is a PITA.  So  on each machine, change the ‘enabled=1’ line in  /etc/yum.repos.d/virtualbox.repo to ‘enabled=1’.  And then if you want to update VBox later, you can do it manually, with ‘yum –enablerepo=virtualbox update’.

We need EPEL for dkms (Dynamic Kernel Module Support), which enables kernel device drivers, such as those for VBox, to be automatically rebuilt when a new kernel is installed:

rpm -Uvh http://epel.mirror.freedomvoice.com/6/i386/epel-release-6-7.noarch.rpm

Now install a bunch of necessary packages (all but dkms are probably already installed), then VBox:

yum install binutils gcc make patch libgomp glibc-headers glibc-devel kernel-headers kernel-devel dkms
yum install VirtualBox-4.2

Specify who can use VBox:

usermod -aG vboxusers <user1>
usermod -aG vboxusers <user2>
.
.

Now you can stop being root, and can start the graphical VBox Manager from the Gnome panel if you’re local, or if you’ve ssh -Y’d in, with:

VirtualBox &

VBox Manager settings

You should make a couple tweaks in the VBox Manager prefs/settings.  First, the default host key for Linux is the right control key.  You need this to escape from the VM to the host environment in certain situations, such as when guest additions are not installed on the client.  There’s no right-ctrl on my Mac, so I make it F3, which is more OS-neutral.

Second, we need to install the proprietary Extension Pack to get VRDP support (discussed below) as well as some hardware-y stuff like USB support that we probably don’t need.  This is problematic, either from the GUI prefs, or from the command line:

[cs342@finch InstallingVBoxOnClients] VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack
0%...
Progress state: NS_ERROR_FAILURE
VBoxManage: error: Failed to install "/home/cs342/InstallingVBoxOnClients/Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack"
VBoxManage: error: The installer failed with exit code 127: Error executing command as another user: No authentication agent was found.
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component ExtPackManager, interface IExtPackManager
VBoxManage: error: Context: "int handleExtPack(HandlerArg*)" at line 1112 of file VBoxManageMisc.cpp

Is this because of the GID conflict between the local group vboxusers and the LDAP group cs342?  No idea.  But we can do it as root, provided we move the file off the root-squashed NFS-mounted filesystem:

[cs342@finch InstallingVBoxOnClients] cp Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack /tmp
[cs342@finch InstallingVBoxOnClients] su
Password: 
[root@finch InstallingVBoxOnClients] VBoxManage extpack install /tmp/Oracle_VM_VirtualBox_Extension_Pack-4.2.0-80737.vbox-extpack
0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Successfully installed "Oracle VM VirtualBox Extension Pack".
[root@finch InstallingVBoxOnClients]

Now we need to do the following:

• Install a VM and set up SSH if necessary (internally) and port forwarding (externally).
• Arrange for external access to the console in case SSH isn’t working.
• Arrange for the VM to start on host boot.

I won’t discuss setting up a VM (but see the section “Putting VM files on clients” below).  On finch, I’m going to install a copy of the CTF 2012 VM.  SSH is already enabled; if not, ‘sudo apt-get install openssh-server’ is all you need for Ubuntu.  In the VBox Manager, I’m forwarding port 2222 to 22 in Settings > Network > Advanced > Port forwarding.  And now you can SSH in like so:

ssh -Y -p 2222 <username>@finch

Note that the host port needs to be >1024.  I believe that the guest can only listen to ports < 1024 if it’s run as root, which sounds like a bad idea.  We need to use a different port for SSH anyway because we want to be able to SSH into the host and guest separately.  But if we run a web server in the guest, we’ll need to either change the port it’s listening on or forward, say, 8080 to 80.

VRDP Access (or lack thereof)

VBox allows you to run VMs “headless”, but access a virtual console via the Microsoft Remote Desktop Protocol when necessary.  This requires opening up a port, default=3389.  Security for this under VBox is a mess.  I’ll record the details below for posterity, but I think that, for us, it should just be disabled.  When the machine is running in its normal headless mode you’ll need to ssh in.  If ssh is hosed or not set up yet, ssh into the host, open the VBox Manager with ‘VirtualBox &’, and enable VRDP with null authentication via Settings > Display > Remote Display > Enable Server (this can be done with the machine running).  Then access the machine with RDP (more below).  When done, disable the RDP server and close the VBox Manager; the VM will continue to run.

To connect to a VM on finch, say, just enter ‘rdesktop finch’ at a machine with rdesktop installed (currently jay, finch, and robin).  If there are multiple VMs running the port may be greater than 3389; see the machine’s settings.  Other RDP clients might work, e.g. MS’s Remote Desktop Connection for Mac, or there’s obviously a native Windows client.

An alternative to this is to shut down the and re-start the VM non-headless through the VBox Manager, bringing up the console.  This requires an X11 connection.  I’ve found that doing this through X11 on a Mac can cause the keyboard to be remapped unusably.  This only happens sometimes, not always, and I don’t know why.  Interestingly, I had very similar problems accessing KVM running on tempest through my Mac last year.

Here are some notes on VRDP authentication.  As mentioned above, this is broken, and it’s probably best to use null authentication and disable it most of the time.  There are four authentication modes:

• Null.  Works fine, but insecure.
• Guest.  Uses authentication on the guest.  Experimental, not supported, and in fact it doesn’t work for us.
• External.  Uses authentication on the host.  Causes segfaults.
• External with simple authentication.  Sort-of-works, see below.

External/simple uses authentication provided by VBox.  You supply VBox with a hashed password to insert into a machine’s XML config file, <Machine Name>.vbox.  Then you supply the password at login.  Problem is that, for us anyway, entering the password interactively doesn’t work and you need to type:

rdesktop -u <user> -p <password> HOST

and then the password’s in your bash history file.  Also, less importantly,  MS’s Remote Desktop Connection for Mac doesn’t work for this.

Anyway, here’s how to set it up if you want to.  I’m following Sec. 7.1.5, “RDP authentication” of the VBox manual:

[cs342@jay Minimal-Ubuntu-Server] VBoxManage setproperty vrdeauthlibrary "VBoxAuthSimple"
[cs342@jay Minimal-Ubuntu-Server] VBoxManage modifyvm <Machine-name> --vrdeauthtype external
[cs342@jay Minimal-Ubuntu-Server] VBoxManage internalcommands passwordhash "<password>"
Password hash: 447a9e95f60aa7c586558c6192283c22786981632b035e3978c78b143d9396c3
[cs342@jay Minimal-Ubuntu-Server] VBoxManage setextradata Minimal-Ubuntu-Server "VBoxAuthSimple/users/<User>" 447a9e95f60aa7c586558c6192283c22786981632b035e3978c78b143d9396c3

VM Auto-start

To make a VM auto-start on the host’s reboot requires that we create a service and configure it appropriately.  Fortunately, this capability was introduced in VBox 4.2, just released a few weeks ago.  (An alternate solution from 2009 might have worked, with some customization:
http://www.kernelhardware.org/virtualbox-auto-start-vm-centos-fedora-redhat/ )

Refs:

• The latest VBox manual: Sec. 9.24.1.: “Linux: starting the autostart service via init”, and sec. 8.2, with some commands for setting machines to auto-start/stop.
• A recent blog post with more detail.

I’ll follow the blog post, mainly.

As root, create /etc/default/virtualbox, containing:

# virtualbox defaults file
VBOXAUTOSTART_DB=/etc/vbox
VBOXAUTOSTART_CONFIG=/etc/vbox/vbox.cfg

And create /etc/vbox/vbox.cfg containing:

# Default policy is to deny starting a VM, the other option is "allow".
default_policy=deny
# Create an entry for each user allowed to run autostart
cs342={allow=true}
fturbak={allow=true}
mdawson={allow=true}

(The examples in the manual and blog post has more line feeds, but produce errors for me.  A CRLF translation issue?)

Permissions/ownership:

chown root:vboxusers /etc/default/virtualbox /etc/vbox /etc/vbox/vbox.cfg
chmod 1775 /etc/vbox
chmod 660 /etc/default/virtualbox /etc/vbox/vbox.cfg

It may be that /etc/default/virtualbox and /etc/vbox/vbox.cfg can have group root, not vboxusers, I’m not sure.

Now become cs342 and do the following.  For modifyvm, the VM needs to
be  off.

VBoxManage setproperty autostartdbpath /etc/vbox
VBoxManage modifyvm <Machine Name> --autostart-enabled on
VBoxManage modifyvm <Machine Name> --autostop-type [disabled|savestate|poweroff|acpishutdown]

(I’m going with savestate, but maybe acpishutdown would be safer)

Problems:

• ‘service vboxautostart-service start’ indicates that it’s starting VMs, but doesn’t.
• ‘service vboxautostart-service stop’ doesn’t stop running VMs
• The ‘savestate’ option doesn’t work; apparently the plug is pulled on the VM, since it will indicate zero uptime when restarted.  Is some sort of delay required?  Or is this connected to the fact that ‘service vboxautostart-service stop’ doesn’t work?

However, the autostart-enabled machines do start up on host system boot (albeit without saved state), which is the main thing we want.

Wait, hold on, that’s true on jay, finch, and lark, but not robin.  ‘VBoxManage modifyvm CTF-2012-robin –autostart-enabled on’ doesn’t create the 1-byte ‘cs342.start’ and ‘cs342.stop’ files in /etc/vbox that it does on the other machines.  Don’t know why, despite a fruitless comparison of  configurations, permissions and ownership, etc. on finch vs. robin.  The steps I took setting up the CTF-2012-robin VM were a little different than for the others, but I don’t know what could have created this difference.  Anyway, manually creating these two files with the right contents  (a single ‘1’) and permownership did the trick.

To shutdown a running VM properly, log in to it and execute ‘shutdown -P now’.  Use -P, not -h, since -h causes the VM to get stuck in the GRUB boot screen on restart.  As far as I can tell, you can’t effect a controlled shutdown externally; the best you can do is

VBoxManage controlvm <vm> savestate

To start a VM that you’ve shut down without rebooting, you should use VBoxHeadless.  Otherwise your VM will die when your session closes:

VBoxHeadless --startvm <vm> --vrde off &

Moving Configuration and VM files to clients

By default, the VM files live in ~/VirtualBox\ VMs.  That’s on the NFS server, so access is slower, and VMs are simultaneously accessible, and even simultaneously runnable, which is bad!  So let’s put the VM files on the clients.  Note that the VMs will no longer be automatically backed up, which is a problem I should think about.

Also, a user’s VBox config and log directory is by default ~/.Virtualbox/.  This means that all hosts running VBox under the same user will share configuration, including the list of available VMs; again, bad.  So lets move that too.  This can be done with symlinks to the local file system or by changing the environment variable VBOX_USER_HOME; since VBox starts running in various ways and at various times, including soon after boot, and I’m not an expert on environment variables, I’ll use symlinks.

So I’ll create /var/VBox-local/<account-name>/ for each account, containing subdirectories ‘VMs’ and ‘Config’, and symlink these from ~acount-name/VirtualBox\ VMs and ~account-name/.VirtualBox, respectively.  As root:

mkdir /var/VBox-local
chgrp vboxusers /var/VBox-local/
chmod 770 /var/VBox-local/

mkdir /var/VBox-local/cs342
chown cs342 /var/VBox-local/cs342/
chmod 770 /var/VBox-local/cs342/

Now backup ~/.Virtualbox by mv-ing it to ~/.Virtualbox-backup, and likewise for ~/VirtualBox\ VMs.  The second backup takes up lots of space, so get rid of it once everything’s running properly.  Create the symlinks mentioned above, which you only have to do once.  Then on each machine that will be running VBox:

• cp -a ~/.Virtualbox  /var/VBox-local/cs342/Config
• cp -a ~/VirtualBox\ VMs /var/VBox-local/cs342/VirtualBox\ VMs
• Remove VMs you don’t want on this machine
• Remove the associated lines from the <MachineRegistry> and any other sections of …Config/VirtualBox.xml

Through the VBox Manager, each machine’s settings should be tweaked like so:

• Choose a new random Mac address if another copy of the same VM exists on another host and there’s any chance they’ll be on the same network.
• Adjust port forwarding if necessary
• Turn off the Remote Display server
• Under Storage, select the controller for the VMs virtual disk, and click ‘Use Host I/O Cache’.  Otherwise, you’ll get a scary warning about a linux kernel bug possibly corrupting the virtual disk when it’s mounted on an ext4 partition.

External Port Forwarding

We need to be able to ssh independently to both a VM and its host, so I’ve been forwarding external port 2222 to the VM’s port 22.  Why not 222?  Because non-root processes can’t serve ports below 1024 on Linux.  Likewise, I’ve been forwarding port 8080 to 80 for HTTP, even though the hosts aren’t running web servers.  Now to connect from outside the host, we need to open host ports.  I’ll open 2222, and forward 80 to 8080 so it will look from the outside like we’re using the standard HTTP port.  This is pretty basic firewall stuff so we can just use system-config-firewall.  Use “Other Ports”, “User Defined” to open a port, and “Port Forwarding”, “Local Forwarding” to forward a port.

If there’s a possibility of running multiple VMs simultaneously on a host, you’ll need to be careful about that, e.g. use ports 2223, 2224, etc. for SSH, and likewise for HTTP.

The CTF VM uses WordPress, which has been giving me huge headaches with port forwarding.  I was able to get it working with an open high port, so I may do it that way rather than forwarding to port 80.

Setting up a Virtual Network Environment on One Host (added 11/25/2012)

I’m helping a student with a botnet project, which will require a little network of VMs.  We want the network to have these properties:

1. VMs should be able to freely communicate with each other, without our having to set up lots of complicated ad hoc port forwarding rules.
2. VMs should be able to connect to the internet for software downloads, etc.
3. Host should be able to connect to VMs via SSH at least.  Connection from global internet to host isn’t necessary.

In sensible virtualization environments like VMWare and (I think) KVM, all 3 can be satisfied with NAT, which behaves like a home router setup.  But VMWare costs money, and KVM requires a 64-bit host OS; it won’t work on our 32-bit CentOS machines.  Maybe we should have installed 64-bit; the hardware would accept it.  But it’s too late now.

VBox’s NAT setup won’t work because the guests are isolated from one another, unlike VMWare and KVM.  Don’t know why, but that’s the way it is.

So we’ll use VBox’s host-only network.  That satisfies requirements 1 and 3.  For #2, we’ll modify the host’s firewall.  (I’ve also seen suggestions that one use both NAT and host-only on each guest.  That sounds complicated.  Maybe it’s not–I didn’t try it–but I’d prefer that the student’s VM network settings be as simple as possible.)

I’m going to set everything up on one host, finch.  I hope the resources will be sufficient.  Memory-wise, finch has 4 GB, so 6 VMs using 1/2 GB each will leave 1 GB for the host, which should be enough.  I don’t think the VMs will need lots of simultaneous CPUs; they’d better not, since there are only 2 of them.  There shouldn’t be a lot of network traffic.  Hard drive space is a concern, there’s only 30 GB available of a 50GB hard drive (small nowadays).  The student could save space by using linked clones.  And if things get tight, I could free up 7 GB by deleting a couple VMs.

If we needed to split things up on multiple hosts it would be more complicated–maybe we’d use openvpn?

Anyway, first, I shut down the running VM, and prevented it from auto-starting with:

VBoxManage modifyvm <machine> --autostart-enabled off

Then, on finch, I added the user to the vboxusers group and set up her local VBox config and VM directories:

# Allow to use VBox
[root@finch ~] usermod -aG vboxusers <user>
# Set up directories and fix perms/ownership
[root@finch ~] cd /var/VBox-local/
[root@finch VBox-local] mkdir -p -m770 <user>/VMs
[root@finch VBox-local] mkdir -m700 <user>/Config
[root@finch VBox-local] chown -R <user> <user>
[root@finch VBox-local] chgrp vboxusers <user>/*
# Make symlinks.  Need to be <user> since root on
# public clients not trusted
[root@finch <user>] su - <user>
[<user>@finch ~] ln -s /var/VBox-local/<user>/VMs/ VirtualBox\ VMs
[<user>@finch ~] ln -s /var/VBox-local/<user>/Config/ .VirtualBox

Then I logged in to finch as this user, started up VBox with ‘Virtualbox &’, and added the default host-only network vboxnet0 with its default settings.

At this point, VMs on the host-only network should be able to communicate with one another, and the host should be able to talk to them.  But we want the VMs to be able to connect to the wider internet.  So we add iptables rules to finch’s firewall.

I learned a lot about iptables at this point.  My main references were:

1. The man page for iptables
2. The book Linux Firewalls
3. The Packet Filtering HOWTO

But, I ended up basically copying the rules for KVM’s virtual bridge and IP range (virbr0, 192.168.122.0/24) over for VBox (vboxnet0, 192.168.56.0/24).  To add the new rules, I used system-config-firewall, which I’d already used to open/forward some ports for the CTF practice VMs.  In the “Custom Rules” section, I specified files containing the new rules, one each for the forward, nat, and mangle tables.  The file contents are copied below for reference.

Probably some of these rules are unnecessary or redundant.  But they seem to work, and finding a minimal set isn’t worth my time right now.

Now we just give the student some advice about working in this new environment:

• Use static IPs in 192.168.56.(2-99).
• Make sure the MAC addresses are different.   When you change MACs on Ubuntu machines, there’s a network config problem; it’s fixed by deleting /etc/udev/rules.d/70-persistent-net.rules and rebooting.
• Look into VBoxManage for scripting.
• Run headlessly with VBoxHeadless to save resources and prevent the VMs from dying when the connection drops (make sure you can ssh in first).

And hopefully everything will Just Work.  We’ll see…

Custom Firewall Rules:

[root@finch MikeVBox-relatedIPtablesFiddling] pwd
/root/MikeVBox-relatedIPtablesFiddling
[root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-filter-rules

# VBox iptables rules for the filter table
# Loaded with system-config-firewall's custom rules
#
# Cribbed from the firewall rules created by libvirtd
# (obtained by 'iptables-save|grep virbr0 > virbr0-rules',
# and see http://libvirt.org/firewall.html )
#
# Accept DNS and DHCP requests from guests (not sure I need this)
-A INPUT -i vboxnet0 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -i vboxnet0 -p tcp -m tcp --dport 53 -j ACCEPT
-A INPUT -i vboxnet0 -p udp -m udp --dport 67 -j ACCEPT
-A INPUT -i vboxnet0 -p tcp -m tcp --dport 67 -j ACCEPT
#
# Allow incoming connections iff related to existing outbound connections
-A FORWARD -d 192.168.56.0/24 -o vboxnet0 -m state --state RELATED,ESTABLISHED -j ACCEPT
#
# ... and allow outgoing connections
-A FORWARD -s 192.168.56.0/24 -i vboxnet0 -j ACCEPT
##
# Allow guests to communicate among themselves
-A FORWARD -i vboxnet0 -o vboxnet0 -j ACCEPT
#
# Nuke everything else to/from vboxnet0
-A FORWARD -o vboxnet0 -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i vboxnet0 -j REJECT --reject-with icmp-port-unreachable

[root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-nat-rules

# VBox iptables rules for the nat table
# Loaded with system-config-firewall's custom rules
#
# Cribbed from the firewall rules created by libvirtd
# (manually copied and altered from the nat section of iptables-save)
#
-A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -p tcp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -p udp -j MASQUERADE --to-ports 1024-65535
-A POSTROUTING -s 192.168.56.0/24 ! -d 192.168.56.0/24 -j MASQUERADE

[root@finch MikeVBox-relatedIPtablesFiddling] cat vboxnet0-iptables-mangle-rules

# VBox iptables rules for the mangle table
# Loaded with system-config-firewall's custom rules
#
# This has something to do with DHCP replies.  Not sure I need it.
# Wish I understood it better.
-A POSTROUTING -o vboxnet0 -p udp -m udp --dport 68 -j CHECKSUM --checksum-fill
[root@finch MikeVBox-relatedIPtablesFiddling]

There’s a package (Python module) that interfaces a Python Script with a MySQL database back-end.  It’s called MySQLdb.  For example:

[root@puma ~] python2.4
Python 2.4.3 (#1, Jun 18 2012, 08:55:23) 
>>> import MySQLdb
>>> 
[root@puma ~] python2.6
Python 2.6.5 (r265:79063, Jun  8 2010, 15:27:41) 
>>> import MySQLdb
>>> 
[root@puma ~] python2.7
Python 2.7 (r27:82500, Aug 29 2012, 15:28:26) 
>>> import MySQLdb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named MySQLdb
>>> 
[root@puma ~]

Oops.  So, that last one is a bit of a problem.  No problem, thought I.  That’s what pip is for:

[root@puma ~] pip-2.7 install MySQLdb
Downloading/unpacking MySQLdb
  Real name of requirement MySQLdb is MySQLdb
  Could not find any downloads that satisfy the requirement MySQLdb
No distributions at all found for MySQLdb
Storing complete log in /root/.pip/pip.log
[root@puma ~]

Hmm.  Not as easy as I hoped.  Okay, so, what is the module actually called?

[root@puma ~] pip-2.7 search MySQLdb
jaraco.mysql              - MySQLDB-compatible MySQL wrapper by Jason R. Coombs
mysqldbda                 - MySQL Database adapter
raisin.mysqldb            - UNKNOWN
sqlShort                  - A tiny wrapper for the Python modules MySQLdb and Sqlite
sqlwitch                  - sqlwitch offers idiomatic SQL generation on top of MySQLdb.
torndb                    - A lightweight wrapper around MySQLdb.  Originally part of the Tornado framework.

Hmm.  I’m still not sure.  None of those look quite right, though many of them look interesting.

Okay, we know that we have MySQLdb for python2.4 and python2.6, so let’s see where those come from:

[root@puma ~] locate MySQLdb
/usr/lib64/python2.4/site-packages/MySQLdb
/usr/lib64/python2.4/site-packages/MySQLdb/__init__.py
/usr/lib64/python2.4/site-packages/MySQLdb/__init__.pyc
/usr/lib64/python2.4/site-packages/MySQLdb/__init__.pyo
/usr/lib64/python2.4/site-packages/MySQLdb/connections.py
...
/usr/lib64/python2.4/site-packages/MySQLdb/constants/__init__.pyc
/usr/lib64/python2.4/site-packages/MySQLdb/constants/__init__.pyo
/usr/share/doc/MySQL-python-1.2.3/MySQLdb.txt
[root@puma ~]

Grrr.  That’s distressing.  So there’s no file/directory called MySQLdb except for Python2.4  So, where is the one for Python2.6?  Back to the drawing board:

[cs304@puma ps2] python2.6
Python 2.6.5 (r265:79063, Jun  8 2010, 15:27:41)
>>> import sys
>>> sys.path
['', '/usr/local/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg',
'/usr/local/lib/python2.6/site-packages/MySQL_python-1.2.3-py2.6-linux-x86_64.egg',
'/usr/local/lib/python2.6/site-packages/Theano-0.3.0-py2.6.egg',
...,
'/usr/local/lib/python2.6/site-packages']
>>> import MySQLdb
>>>
[cs304@puma ps2] cd /usr/local/lib/python2.6/site-packages/
[cs304@puma site-packages] file MySQL_python-1.2.3-py2.6-linux-x86_64.egg
MySQL_python-1.2.3-py2.6-linux-x86_64.egg: Zip archive data, at least v2.0
to extract
[cs304@puma site-packages] zipinfo MySQL_python-1.2.3-py2.6-linux-x86_64.egg
Archive:  MySQL_python-1.2.3-py2.6-linux-x86_64.egg   107724   37
-rw-rw----  2.0 unx      524 b- defN 28-Feb-11 13:37 _mysql.pyc
-rw-rw----  2.0 unx      276 b- defN 28-Feb-11 13:37 _mysql.py
-rwxrwx---  2.0 unx   147488 b- defN 28-Feb-11 13:34 _mysql.so
-rw-rw----  2.0 unx     2306 b- defN 17-Jun-10 03:21 _mysql_exceptions.py
-rw-rw----  2.0 unx     4139 b- defN 28-Feb-11 13:37 _mysql_exceptions.pyc
-rw-rw----  2.0 unx     3121 b- defN 17-Jun-10 03:21 MySQLdb/__init__.py
-rw-rw----  2.0 unx     6317 b- defN 28-Feb-11 13:37
MySQLdb/converters.pyc
-rw-rw----  2.0 unx    12338 b- defN 28-Feb-11 13:37
MySQLdb/connections.pyc
-rw-rw----  2.0 unx     4529 b- defN 28-Feb-11 13:37 MySQLdb/times.pyc
...
EGG-INFO/native_libs.txt
-rw-rw-r--  2.0 unx      868 b- defN 28-Feb-11 13:37 EGG-INFO/SOURCES.txt
-rw-rw----  2.0 unx        1 b- defN 28-Feb-11 13:37 EGG-INFO/zip-safe
-rw-rw-r--  2.0 unx     1751 b- defN 28-Feb-11 13:37 EGG-INFO/PKG-INFO
-rw-rw-r--  2.0 unx       33 b- defN 28-Feb-11 13:37
EGG-INFO/top_level.txt
37 files, 279214 bytes uncompressed, 103268 bytes compressed:  63.0%
scottas-imac91:~ scotta$

Wow, that was quite a journey.  So, it seems the MySQLdb module is inside an “egg” called MySQL_python.  Can we find *that* with pip?

Okay, we can, and we were able to upgrade it.

[root@puma cgi-bin] pip-2.6 search MySQL_python
MySQL-python              - Python interface to MySQL
  INSTALLED: 1.2.3
  LATEST:    1.2.4b5
[root@puma cgi-bin] pip-2.6 install --upgrade MySQL_python
 Downloading/unpacking MySQL-python from
 http://pypi.python.org/packages/source/M/MySQL-python/MySQL-python-1.2.4b5.tar.gz#md5=2d760ee948aff4f50d01afdf8afff48c
 Downloading MySQL-python-1.2.4b5.tar.gz (82Kb): 82Kb
 downloaded
 Running setup.py egg_info for package MySQL-python
 Downloading
 Installing collected packages: MySQL-python
   Found existing installation: MySQL-python 1.2.3
   Uninstalling MySQL-python:
     Successfully uninstalled MySQL-python
   Running setup.py install for MySQL-python
 Successfully installed MySQL-python
 Cleaning up...
[root@puma cgi-bin]

One quick test:

[root@puma ~] python2.6
Python 2.6.5 (r265:79063, Jun  8 2010, 15:27:41) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> 
[root@puma ~] su - anderson
[anderson@puma ~] python2.6
Python 2.6.5 (r265:79063, Jun  8 2010, 15:27:41) 
[GCC 4.1.2 20080704 (Red Hat 4.1.2-46)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named MySQLdb
>>> 
[anderson@puma ~] logout
[root@puma ~] ls -lt /usr/local/lib/python2.6/site-packages/
total 732
drwxrwx---  3 root root   4096 Oct 11 21:45 MySQLdb
-rw-rw-r--  1 root root   4305 Oct 11 21:45 _mysql_exceptions.pyc
drwxrwxr-x  2 root root   4096 Oct 11 21:45 MySQL_python-1.2.4b5-py2.6.egg-info
-rwxrwxr--  1 root root 148882 Oct 11 21:45 _mysql.so
-rw-rw-r--  1 root root    445 Oct 11 21:45 easy-install.pth
-rw-rw-r--  1 root root   2352 Oct 11 21:45 _mysql_exceptions.py
drwxrwxr-x  4 root root   4096 Jul 26 20:57 pip-1.1-py2.6.egg
drwxrwxr-x  5 root root   4096 Jun 17 15:00 networkx-1.7rc1-py2.6.egg
drwxrwxr-x  4 root root   4096 Jun 17 14:03 Orange-2.5a4-py2.6-linux-x86_64.egg

Darn!  pip never sets the directory permissions properly, and I always forget. (Oh, and I see that the upgrade has created a MySQLdb directory, instead of hiding the module inside an egg.) A quick fix:

[root@puma ~] opendir /usr/local/lib/python2.6/site-packages/

And we’re done:

[anderson@puma ~] python2.6 -c 'import MySQLdb'

whew!  Now, can we finally install MySQLdb for python2.7?

[root@puma ~] pip-2.7 search MySQL_python
MySQL-python              - Python interface to MySQL
[root@puma ~] pip-2.7 install MySQL_python
Downloading/unpacking MySQL-python
  Downloading MySQL-python-1.2.4b5.tar.gz (82Kb): 82Kb downloaded
  Running setup.py egg_info for package MySQL-python
    Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
    Extracting in /tmp/tmpGemlI5
    Now working in /tmp/tmpGemlI5/distribute-0.6.28
    Building a Distribute egg in /root/build/MySQL-python
    /root/build/MySQL-python/distribute-0.6.28-py2.7.egg
    
Installing collected packages: MySQL-python
  Running setup.py install for MySQL-python
    building '_mysql' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Dversion_info=(1,2,4,'beta',5) -D__version__=1.2.4b5 -I/usr/include/mysql -I/usr/local/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv
    gcc -pthread -shared build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib64/mysql -L/usr/lib64 -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread -lssl -lcrypto -o build/lib.linux-x86_64-2.7/_mysql.so
    
Successfully installed MySQL-python
Cleaning up...
[root@puma ~] ls -lt /usr/local/lib/python2.7/site-packages/ | head
total 592
drwxrwx---  3 root root   4096 Oct 11 22:36 MySQLdb
-rw-rw----  1 root root   4303 Oct 11 22:36 _mysql_exceptions.pyc
drwxrwx---  2 root root   4096 Oct 11 22:36 MySQL_python-1.2.4b5-py2.7.egg-info
-rwxrwx---  1 root root 148898 Oct 11 22:36 _mysql.so
-rw-rw----  1 root root   2352 Oct 11 22:36 _mysql_exceptions.py
drwxrwx--- 18 root root   4096 Aug 31 10:27 numpy
drwxrwx---  2 root root   4096 Aug 31 10:27 numpy-1.6.2-py2.7.egg-info
drwxrwx---  2 root root   4096 Aug 31 10:23 simplejson-2.6.1-py2.7.egg-info
drwxrwx---  3 root root   4096 Aug 31 10:23 simplejson
[root@puma ~] opendir /usr/local/lib/python2.7/site-packages/
[root@puma ~] su - anderson
[anderson@puma ~] python2.7 -c 'import MySQLdb'
[anderson@puma ~] 

Yes, we can!  Woo-hoo!  (And, as a side-effect, we made numpy and simplejson and other things available in Python2.7.)

Now for Tempest:

[root@tempest ~] pip-2.6 search MySQL_python
MySQL-python              - Python interface to MySQL
[root@tempest ~] pip-2.6 install MySQL_python
Downloading/unpacking MySQL-python
  Downloading MySQL-python-1.2.4b5.tar.gz (82Kb): 82Kb downloaded
  Running setup.py egg_info for package MySQL-python
    The required version of distribute (>=0.6.28) is not available,
    and can't be installed while this script is running. Please
    install a more recent version first, using
    'easy_install -U distribute'.
    
    (Currently using distribute 0.6.10 (/usr/lib/python2.6/site-packages))
    Complete output from command python setup.py egg_info:
    The required version of distribute (>=0.6.28) is not available,

and can't be installed while this script is running. Please

install a more recent version first, using

'easy_install -U distribute'.



(Currently using distribute 0.6.10 (/usr/lib/python2.6/site-packages))

----------------------------------------
Command python setup.py egg_info failed with error code 2
Storing complete log in /root/.pip/pip.log
[root@tempest ~] easy_install
easy_install      easy_install-2.6  easy_install-2.7  
[root@tempest ~] easy_install-2.6 -U distribute
Searching for distribute
Reading http://pypi.python.org/simple/distribute/
Reading http://packages.python.org/distribute
Best match: distribute 0.6.28
Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz#md5=b400b532e33f78551e6847c1f5965e56
Processing distribute-0.6.28.tar.gz
Running distribute-0.6.28/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ScwCrB/distribute-0.6.28/egg-dist-tmp-1aKoLe
Before install bootstrap.
Scanning installed packages
Setuptools installation detected at /usr/lib/python2.6/site-packages
Non-egg installation
Removing elements out of the way...
Already patched.
/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info already patched.
After install bootstrap.
/usr/lib/python2.6/site-packages/setuptools-0.6c11-py2.6.egg-info already exists
Adding distribute 0.6.28 to easy-install.pth file
Installing easy_install script to /usr/bin
Installing easy_install-2.6 script to /usr/bin

Installed /usr/lib/python2.6/site-packages/distribute-0.6.28-py2.6.egg
Processing dependencies for distribute
Finished processing dependencies for distribute
[root@tempest ~] pip-2.6 install MySQL_python
Downloading/unpacking MySQL-python
  Running setup.py egg_info for package MySQL-python
Installing collected packages: MySQL-python
  Running setup.py install for MySQL-python
    building '_mysql' extension
    gcc -pthread -fno-strict-aliasing -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -DNDEBUG -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -mtune=generic -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Dversion_info=(1,2,4,'beta',5) -D__version__=1.2.4b5 -I/usr/include/mysql -I/usr/include/python2.6 -c _mysql.c -o build/temp.linux-x86_64-2.6/_mysql.o -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX -DUNIV_LINUX
    In file included from /usr/include/mysql/my_config.h:14,
                     from _mysql.c:44:
    /usr/include/mysql/my_config_x86_64.h:1082:1: warning: "HAVE_WCSCOLL" redefined
    In file included from /usr/include/python2.6/pyconfig.h:6,
                     from /usr/include/python2.6/Python.h:8,
                     from _mysql.c:29:
    /usr/include/python2.6/pyconfig-64.h:808:1: warning: this is the location of the previous definition
    gcc -pthread -shared build/temp.linux-x86_64-2.6/_mysql.o -L/usr/lib64/mysql -L/usr/lib64 -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread -lssl -lcrypto -lpython2.6 -o build/lib.linux-x86_64-2.6/_mysql.so
Successfully installed MySQL-python
Cleaning up...
[root@tempest ~]

Okay, a bit of a detour in the middle, there, but ultimately successful.  Wasn’t it?  Let’s check:

[root@tempest ~] python2.6
Python 2.6.6 (r266:84292, Aug 28 2012, 10:55:56) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>> MySQLdb.__file__
'/usr/lib64/python2.6/site-packages/MySQLdb/__init__.pyc'
>>> 
[root@tempest ~] ls -lt /usr/lib64/python2.6/site-packages/ | head
total 8988
drwxrwx---.  2 root root   4096 Oct 11 22:42 MySQL_python-1.2.4b5-py2.6.egg-info
-rw-rw----.  1 root root   4257 Oct 11 22:42 _mysql_exceptions.pyc
drwxrwx---.  3 root root   4096 Oct 11 22:42 MySQLdb
-rwxrwx---.  1 root root 141945 Oct 11 22:42 _mysql.so
-rw-rw----.  1 root root   2352 Oct 11 22:40 _mysql_exceptions.py
drwxr-xr-x.  2 root root   4096 Oct  3 23:20 qmf
-rwxr-xr-x.  1 root root  46472 Sep 11 10:10 SpiceClientGtk.so
-rw-r--r--.  2 root root  11329 Sep  5 04:29 drv_libxml2.pyc
-rw-r--r--.  2 root root  11329 Sep  5 04:29 drv_libxml2.pyo
[root@tempest ~] opendir /usr/lib64/python2.6/

Okay, so the libraries are in /usr/lib64 instead of /usr/local/lib as on Puma.  We should check what other library directories python2.6 is on Tempest:

[root@tempest ~] ls -lt /usr/lib/python2.6/site-packages/ | head
total 1228
-rw-r--r--. 1 root root    30 Oct 11 22:42 setuptools.pth
-rw-rw-r--. 1 root root   457 Oct 11 22:42 easy-install.pth
drwxrwx---. 4 root root  4096 Oct 11 22:42 distribute-0.6.28-py2.6.egg
drwxr-xr-x. 2 root root  4096 Oct  3 23:20 mllib
drwxr-xr-x. 5 root root  4096 Oct  3 23:20 qpid
[root@tempest ~] opendir /usr/lib/python2.6/site-packages/

So, we’ve opened that up, too. One last test:

[root@tempest ~] su - anderson
[anderson@tempest ~] python2.6 -c 'import MySQLdb'

Now to install MySQLdb for python2.7:

[root@tempest ~] pip-2.7 search MySQL_python
MySQL-python              - Python interface to MySQL
[root@tempest ~] pip-2.7 install MySQL_python
Downloading/unpacking MySQL-python
  Downloading MySQL-python-1.2.4b5.tar.gz (82Kb): 82Kb downloaded
  Running setup.py egg_info for package MySQL-python
    Downloading http://pypi.python.org/packages/source/d/distribute/distribute-0.6.28.tar.gz
    Extracting in /tmp/tmpwuvghL
    Now working in /tmp/tmpwuvghL/distribute-0.6.28
    Building a Distribute egg in /root/build/MySQL-python
    /root/build/MySQL-python/distribute-0.6.28-py2.7.egg
    
Installing collected packages: MySQL-python
  Running setup.py install for MySQL-python
    building '_mysql' extension
    gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Dversion_info=(1,2,4,'beta',5) -D__version__=1.2.4b5 -I/usr/include/mysql -I/usr/local/include/python2.7 -c _mysql.c -o build/temp.linux-x86_64-2.7/_mysql.o -g -pipe -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector --param=ssp-buffer-size=4 -m64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -fno-strict-aliasing -fwrapv -fPIC -DUNIV_LINUX -DUNIV_LINUX
    In file included from /usr/include/mysql/my_config.h:14,
                     from _mysql.c:44:
    /usr/include/mysql/my_config_x86_64.h:1082:1: warning: "HAVE_WCSCOLL" redefined
    In file included from /usr/local/include/python2.7/Python.h:8,
                     from _mysql.c:29:
    /usr/local/include/python2.7/pyconfig.h:881:1: warning: this is the location of the previous definition
    gcc -pthread -shared build/temp.linux-x86_64-2.7/_mysql.o -L/usr/lib64/mysql -lmysqlclient_r -lz -lpthread -lcrypt -lnsl -lm -lpthread -lssl -lcrypto -o build/lib.linux-x86_64-2.7/_mysql.so
    
Successfully installed MySQL-python
Cleaning up...
[root@tempest ~] opendir /usr/local/lib
lib/     lib64/   libexec/ 
[root@tempest ~] opendir /usr/local/lib/python2.7/
[root@tempest ~] su - anderson
[anderson@tempest ~] python2.7 -c 'import MySQLdb'
[anderson@tempest ~]

Whew!  I think we’re good now.

Generally, users receive emails warning them if they are near or over their quota usage. However, users haven’t been receiving these notifications for some time now, which is mysterious.

I checked /root/sysadmin/bin/ for quota-related scripts, i.e., scripts containing the word ‘quota’. Most, if not all, of those scripts contain this line:

for name in `cat /root/etc/passwd.ldap | grep student | cut -f1 -d’:’`; do

Unfortunately, /root/etc/passwd.ldap doesn’t exist on tempest or on puma, so it made sense that the scripts were not able to do very much. On puma, I typed “find . -name “passwd.ldap”” at the home directory and found that passwd.ldap was in the directory /ldapdumps. So I moved a copy of it to /root/etc. I did the same thing on tempest.

However, when I checked the modification dates of passwd.ldap on puma and tempest, I found that they weren’t modified on the same date. Don’t know why that is the case.

tempest
-rw-r–r–. 1 root root 152794 Sep 26 03:41 passwd.ldap
puma
-rw-r—– 1 root root 152794 Sep 30 05:08 passwd.ldap

In conclusion, I think the quota scripts should work now when run, but I’m not sure. I didn’t run them.

** As a side note: tempest contains a file in /root/etc called ‘slapcat’, while puma does not. Don’t know why this is the case either.

We have several layers of backups for our data. BigData (drive enclosure T) is a 2.7 TB drive with backups for a week. tempest (drive enclosure V) has backups for 90 days (it’s useful to have incrementals). There’s also a 3 TB drive (drive enclosure U) with a copy of the whole machine (in case of hackers, fumbles, or hardware failures).  Last week, Erin brought up that she’d been receiving several emails about “/root/snapshots not mounted!” The cause of this was a backup scare: both tempest and the 3TB drive could not be read by their drive enclosures.  It was thought for a few days that we’d lost all those backups…

Forumgoers suggested that the problem might be with the quality of the drives (consumer drives are 3x cheaper than more quality drives, so we use the cheap ones), or that moving the drives in and out of the cabinets wasn’t a good idea… or that something could be wrong with their drive enclosures, which turned out to be correct. Luckily, when they were read by T instead, it turned out that the drives weren’t unreadable and everything was fine.

Unfortunately, we’re still getting kernel errors and the like sometimes from the drives, but at least we have backups for now.

We have a mystery on the CentOS workstations.  ssh works, but su does not.  Here’s an example:

[luser@gibbon ~] ssh anderson@gibbon
anderson@gibbon's password: 
Last login: Wed Sep 26 10:56:41 2012 from gibbon.wellesley.edu
[anderson@gibbon ~] logout
Connection to gibbon closed.
[luser@gibbon ~] su - anderson
Password: 
su: incorrect password
[luser@gibbon ~] su - anderson
Password: 
su: incorrect password
[luser@gibbon ~]

You can see that I double-checked to make sure I typed my password correctly on the su command.  A difference like this probably points to a difference in pam.  Since both su and ssh work on Sampras, an older machine running Fedora 14, I thought it made sense to compare their authentication configuration.  I can do this by using authconfig on Gibbon and Sampras to dump out all the relevant files for both Gibbon and Sampras to a directory and then compare those directories:

[root@gibbon ~] authconfig --savebackup=/usr/network/tmp/authconfig-gibbon
[root@gibbon ~] cd /usr/network/tmp
[root@gibbon tmp] diff -qr authconfig-gibbon/ authconfig-sampras
Files authconfig-gibbon/authconfig and authconfig-sampras/authconfig differ
Files authconfig-gibbon/fingerprint-auth-ac and authconfig-sampras/fingerprint-auth-ac differ
Files authconfig-gibbon/login.defs and authconfig-sampras/login.defs differ
Files authconfig-gibbon/network and authconfig-sampras/network differ
Only in authconfig-sampras: nss_ldap.conf
Files authconfig-gibbon/nsswitch.conf and authconfig-sampras/nsswitch.conf differ
Files authconfig-gibbon/openldap.conf and authconfig-sampras/openldap.conf differ
Files authconfig-gibbon/pam_ldap.conf and authconfig-sampras/pam_ldap.conf differ
Only in authconfig-sampras: pam_pkcs11.conf
Files authconfig-gibbon/smartcard-auth-ac and authconfig-sampras/smartcard-auth-ac differ
Files authconfig-gibbon/smb.conf and authconfig-sampras/smb.conf differ
Files authconfig-gibbon/system-auth-ac and authconfig-sampras/system-auth-ac differ
[root@gibbon tmp]

Now, many of these differences are irrelevant.  For example, we don’t use any fingerprint or smartcard stuff, so we can ignore those.  Let’s start with authconfig, which certainly seems relevant:

[root@gibbon tmp] diff authconfig-gibbon/authconfig authconfig-sampras/authconfig 
1d0
< IPADOMAINJOINED=no
8d6
< USESSSD=no
14,15c12,13
< USELDAPAUTH=no
< IPAV2NONTP=no
---
> USEDB=no
> USELDAPAUTH=yes
18d15
< USEIPAV2=no
23c20
< USEKERBEROS=yes
---
> USEKERBEROS=no
25c22
< USEDB=no
---
> USESSSD=no
[root@gibbon tmp]

Hmmm.  Well, the two files are in a different order, but they both say USESSSD=no, which is a little surprising because we use sssd, according to /etc/nsswitch.conf:

[root@gibbon tmp] service sssd status
sssd (pid  1697) is running...
[root@gibbon tmp] grep pass /etc/nsswitch.conf 
#passwd:    db files nisplus nis
passwd:     files sss
[root@gibbon tmp]

So, that’s puzzling. Still, Sampras works, and we’re looking for a difference, not necessarily looking to understand each item.

Hmm, I also notice that Gibbon has USELDAPAUTH=no, while Sampras has that set to yes.  That’s the kind of difference we’re looking for.  Is there a way to change that using authconfig or do we edit files by hand?

It turns out authconfig can also write out info in a different format:

[root@gibbon tmp] authconfig --test > gibbon-test1

It also turns out that there’s a command line arg called –enableldapauth which certainly sounds promising:

[root@gibbon tmp] authconfig --enableldapauth --test > gibbon-test2
[root@gibbon tmp] diff gibbon-test1 gibbon-test2 
37c37
< pam_ldap is disabled
---
> pam_ldap is enabled
[root@gibbon tmp]

I don’t know if that will edit the authconfig file in the way we want, but it does seem promising.  By the way, the “authconfig” file is in face /etc/sysconfig/authconfig, which is local specifications, so that does seem like the right file.

Let’s compare to sampras (I’m not showing the step where we spit out the authconfig data on sampras, but it’s the same command as on Gibbon.)

[root@gibbon tmp] grep pam_ldap sampras-test1 
pam_ldap is enabled
[root@gibbon tmp]

Oh, very promising. Let’s look more closely:

[root@gibbon tmp] diff gibbon-test1 sampras-test1 
26d25
< nss_mdns4_minimal is disabled
31c30
< pam_krb5 is enabled
---
> pam_krb5 is disabled
37c36
< pam_ldap is disabled
---
> pam_ldap is enabled
44,45c43,44
<  smartcard module = ""
<  smartcard removal action = ""
---
>  smartcard module = "coolkey"
>  smartcard removal action = "Ignore"
53c52
<  credential caching in SSSD is disabled
---
>  credential caching in SSSD is enabled
55,59d53
< IPAv2 is disabled
< IPAv2 domain was not joined
<  IPAv2 server = ""
<  IPAv2 realm = ""
<  IPAv2 domain = ""
[root@gibbon tmp]

Interesting.  So we should probably disable krb, since we don’t use kerberos, and we should probably enable sssd caching, for performance and consistency.  Let’s try those changes:

[root@gibbon tmp] authconfig --enableldapauth --disablekrb5 --enablecachecreds --test > gibbon-test2
[root@gibbon tmp] diff gibbon-test2 sampras-test1 
26d25
< nss_mdns4_minimal is disabled
44,45c43,44
<  smartcard module = ""
<  smartcard removal action = ""
---
>  smartcard module = "coolkey"
>  smartcard removal action = "Ignore"
55,59d53
< IPAv2 is disabled
< IPAv2 domain was not joined
<  IPAv2 server = ""
<  IPAv2 realm = ""
<  IPAv2 domain = ""
[root@gibbon tmp]

This is looking really good.  Okay, time to bite the bullet and go from –test to –update:

[root@gibbon tmp] authconfig --enableldapauth --disablekrb5 --enablecachecreds --update
Starting sssd:                                             [  OK  ]
[root@gibbon tmp]

Now, time to test this change:

[luser@gibbon ~] ssh anderson@gibbon
anderson@gibbon's password: 
Permission denied, please try again.
anderson@gibbon's password: 
Permission denied, please try again.
anderson@gibbon's password: 
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
[luser@gibbon ~]

*sigh*.  Complete and utter failure.  Somehow, our change means sshd no longer works.  What about su?

[luser@gibbon ~] su - anderson
Password: 
su: incorrect password
[luser@gibbon ~]

Okay, no luck there.  What changes did that command do?  I saved the authconfig info in authconfig-gibbon2 and compared:

[root@gibbon tmp] diff -qr authconfig-gibbon authconfig-gibbon2
Files authconfig-gibbon/authconfig and authconfig-gibbon2/authconfig differ
Files authconfig-gibbon/password-auth-ac and authconfig-gibbon2/password-auth-ac differ
Files authconfig-gibbon/smartcard-auth-ac and authconfig-gibbon2/smartcard-auth-ac differ
Files authconfig-gibbon/sssd.conf and authconfig-gibbon2/sssd.conf differ
[root@gibbon tmp]

Not many, which is good.  What are the exact differences?

[root@gibbon tmp] cd authconfig-gibbon
[root@gibbon authconfig-gibbon] sort authconfig > authconfig.sorted
[root@gibbon authconfig-gibbon] cd ../authconfig-gibbon2
[root@gibbon authconfig-gibbon2] sort authconfig > authconfig.sorted
[root@gibbon authconfig-gibbon2] cd ..
[root@gibbon tmp] diff authconfig-gibbon/authconfig.sorted authconfig-gibbon2/authconfig.sorted 
12,13c12,13
< USEKERBEROS=yes
< USELDAPAUTH=no
---
> USEKERBEROS=no
> USELDAPAUTH=yes
[root@gibbon tmp]

Hmm.  So turning *on* LDAPAUTH has disabled ssh.  That’s weird.  Maybe there’s some insight in the other files?

[root@gibbon tmp] diff authconfig-gibbon/password-auth-ac authconfig-gibbon2/password-auth-ac 
7c7
< auth        sufficient    pam_ldap.so use_first_pass
---
> auth        sufficient    pam_sss.so use_first_pass
13c13
< account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
---
> account     [default=bad success=ok user_unknown=ignore] pam_sss.so
18c18
< password    sufficient    pam_ldap.so use_authtok
---
> password    sufficient    pam_sss.so use_authtok
23d22
< -session     optional      pam_systemd.so
26c25
< session     optional      pam_ldap.so
---
> session     optional      pam_sss.so
[root@gibbon tmp]

Very weird.  Turning *on* LDAPAUTH means replacing pam_ldap with pam_sss in the password-auth-ac file.  (I wonder what turning *off* LDAP would do?)

It’s weird that the smartcard-auth-ac file should change.  What differences there?

[root@gibbon tmp] diff authconfig-gibbon/smartcard-auth-ac authconfig-gibbon2/smartcard-auth-ac 
5,7c5
< auth        [success=ok ignore=2 default=die] pam_pkcs11.so wait_for_card card_only
< auth        optional      pam_krb5.so use_first_pass no_subsequent_prompt
< auth        sufficient    pam_permit.so
---
> auth        [success=done ignore=ignore default=die] pam_pkcs11.so wait_for_card card_only
[root@gibbon tmp]

Oh, probably from disabling kerberos.  Okay, I guess that makes sense.

Next, of course, there’s the /etc/sssd.conf file.  Let’s filter out the comments and compare:

[root@gibbon tmp] cd authconfig-gibbon
[root@gibbon authconfig-gibbon] cat sssd.conf | egrep -v "#|;" | egrep "\w" > sssd.conf.min
[root@gibbon authconfig-gibbon] cd ../authconfig-gibbon2
[root@gibbon authconfig-gibbon2] cat sssd.conf | egrep -v "#|;" | egrep "\w" > sssd.conf.min
[root@gibbon authconfig-gibbon2] cd ..
[root@gibbon tmp] diff authconfig-gibbon/sssd.conf.min authconfig-gibbon2/sssd.conf.min 
0a1,11
> [domain/default]
> ldap_id_use_start_tls = False
> cache_credentials = True
> ldap_search_base = dc=cs,dc=wellesley,dc=edu
> krb5_realm = EXAMPLE.COM
> krb5_server = kerberos.example.com
> id_provider = ldap
> auth_provider = ldap
> chpass_provider = ldap
> ldap_uri = ldap://tempest.wellesley.edu/
> ldap_tls_cacertdir = /etc/openldap/cacerts
6c17
< domains = LDAP
---
> domains = default, LDAP
[root@gibbon tmp]

Interesting.  So, it added a new SSSD domain (default) and then configured it.  Is it different from the one we already had?

[domain/LDAP]
id_provider = ldap
auth_provider = ldap
ldap_id_use_start_tls = False
cache_credentials = False
enumerate = false
ldap_search_base = dc=cs,dc=wellesley,dc=edu
chpass_provider = ldap
debug_level = 0
ldap_uri = ldap://tempest.wellesley.edu/
ldap_tls_cacertdir = /etc/openldap/cacerts
min_id = 500
[root@gibbon tmp]

No, not much.  Every variable in the new domain has the same value it had in the old domain.  So, the trouble has to be with the sssd stuff in pam.d/password-auth-ac.  Let’s compare with Sampras, the working machine:

[root@gibbon tmp] diff authconfig-gibbon2/password-auth-ac authconfig-sampras/password-auth-ac 
7c7
< auth        sufficient    pam_sss.so use_first_pass
---
> auth        sufficient    pam_ldap.so use_first_pass
13c13
< account     [default=bad success=ok user_unknown=ignore] pam_sss.so
---
> account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
18c18
< password    sufficient    pam_sss.so use_authtok
---
> password    sufficient    pam_ldap.so use_authtok
22a23
> -session     optional      pam_systemd.so
25c26
< session     optional      pam_sss.so
---
> session     optional      pam_ldap.so
[root@gibbon tmp]

Yes, that seems to confirm it.  Using pam_sss makes things not work.  Let’s look particularly in the /etc/pam.d/ directory on Sampras and Gibbon.  Sampras first:

[root@sampras ~] cd /etc/pam.d/
[root@sampras pam.d] more su
#%PAM-1.0
auth        sufficient    pam_rootok.so
# Uncomment the following line to implicitly trust users in the "wheel" group.
#auth        sufficient    pam_wheel.so trust use_uid
# Uncomment the following line to require a user to be in the "wheel" group.
#auth        required    pam_wheel.so use_uid
auth        include        system-auth
account        sufficient    pam_succeed_if.so uid = 0 use_uid quiet
account        include        system-auth
password    include        system-auth
session        include        system-auth
session        optional    pam_xauth.so
[root@sampras pam.d] more sshd
#%PAM-1.0
auth       required    pam_sepermit.so
auth       include      password-auth
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
# pam_selinux.so close should be the first session rule
session    required     pam_selinux.so close
session    required     pam_loginuid.so
# pam_selinux.so open should only be followed by sessions to be executed in the user context
session    required     pam_selinux.so open env_params
session    optional     pam_keyinit.so force revoke
session    include      password-auth
[root@sampras pam.d]

Okay, I don’t know why su uses system-auth and sshd uses password-auth. How do those files differ?

[root@sampras pam.d] grep sss system-auth password-auth
[root@sampras pam.d] grep ldap system-auth password-auth
system-auth:auth        sufficient    pam_ldap.so use_first_pass
system-auth:account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
system-auth:password    sufficient    pam_ldap.so use_authtok
system-auth:session     optional      pam_ldap.so
password-auth:auth        sufficient    pam_ldap.so use_first_pass
password-auth:account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
password-auth:password    sufficient    pam_ldap.so use_authtok
password-auth:session     optional      pam_ldap.so
[root@sampras pam.d]

Okay, so the working machine never uses sss and always uses ldap.  What about Gibbon?

[root@gibbon ~] cd /etc/pam.d
[root@gibbon pam.d] grep auth su
auth        sufficient    pam_rootok.so
#auth        sufficient    pam_wheel.so trust use_uid
#auth        required    pam_wheel.so use_uid
auth        include        system-auth
account        include        system-auth
password    include        system-auth
session        include        system-auth
session        optional    pam_xauth.so
[root@gibbon pam.d] grep auth sshd
auth       required    pam_sepermit.so
auth       include      password-auth
account    include      password-auth
password   include      password-auth
session    include      password-auth
[root@gibbon pam.d]

Again, su uses system-auth and sshd uses password-auth.  And how do those differ?

[root@gibbon pam.d] grep sss system-auth password-auth
system-auth:auth        sufficient    pam_sss.so use_first_pass
system-auth:account     [default=bad success=ok user_unknown=ignore] pam_sss.so
system-auth:password    sufficient    pam_sss.so use_authtok
system-auth:session     optional      pam_sss.so
password-auth:auth        sufficient    pam_sss.so use_first_pass
password-auth:account     [default=bad success=ok user_unknown=ignore] pam_sss.so
password-auth:password    sufficient    pam_sss.so use_authtok
password-auth:session     optional      pam_sss.so
[root@gibbon pam.d] grep ldap system-auth password-auth
[root@gibbon pam.d]

Yep, the both use sss and neither uses ldap, so they both fail.  What was the previous, partially working version of Gibbon?

[root@gibbon tmp] grep sss authconfig-gibbon/system-auth-ac authconfig-gibbon/password-auth-ac 
authconfig-gibbon/system-auth-ac:auth        sufficient    pam_sss.so use_first_pass
authconfig-gibbon/system-auth-ac:account     [default=bad success=ok user_unknown=ignore] pam_sss.so
authconfig-gibbon/system-auth-ac:password    sufficient    pam_sss.so use_authtok
authconfig-gibbon/system-auth-ac:session     optional      pam_sss.so
[root@gibbon tmp] grep ldap authconfig-gibbon/system-auth-ac authconfig-gibbon/password-auth-ac 
authconfig-gibbon/password-auth-ac:auth        sufficient    pam_ldap.so use_first_pass
authconfig-gibbon/password-auth-ac:account     [default=bad success=ok user_unknown=ignore] pam_ldap.so
authconfig-gibbon/password-auth-ac:password    sufficient    pam_ldap.so use_authtok
authconfig-gibbon/password-auth-ac:session     optional      pam_ldap.so
[root@gibbon tmp]

Yep, since sshd uses password-auth and password-auth used pam_ldap, that’s why that succeeded, while su uses system-auth and system-auth used pam_sss, so that failed.

Is sssd running?  Any errors?

[root@gibbon tmp] service sssd status
sssd (pid  1755) is running...
[root@gibbon tmp] ls -lt /var/log/sssd/
total 12
-rw-------. 1 root root 327 Sep 26 11:59 sssd.log
-rw-------. 1 root root 177 Sep 23 03:24 sssd.log-20120923.gz
-rw-------. 1 root root 161 Sep 20 16:31 sssd.log-20120920.gz
-rw-------. 1 root root   0 Sep  7 12:49 sssd_LDAP.log
-rw-------. 1 root root   0 Sep  4 12:40 sssd_nss.log
-rw-------. 1 root root   0 Sep  4 12:40 sssd_pam.log
-rw-------. 1 root root   0 Sep  4 12:40 krb5_child.log
-rw-------. 1 root root   0 Sep  4 12:40 sssd_default.log
[root@gibbon tmp] tail /var/log/sssd/sssd.log
(Mon Sep 24 14:48:58 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children
(Wed Sep 26 11:36:26 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children
(Wed Sep 26 11:59:58 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children
[root@gibbon tmp]

So, no error messages from sssd.

I’m stumped for now.  I’m going to revert the authconfig for gibbon so that at least sshd works and I’ll post to linuxquestions.org and hope for some help.

[root@gibbon tmp] authconfig --restorebackup=/usr/network/tmp/authconfig-gibbon
[root@gibbon tmp] ssh anderson@gibbon
anderson@gibbon's password: 
Last login: Wed Sep 26 10:56:56 2012 from gibbon.wellesley.edu
[anderson@gibbon ~] logout
Connection to gibbon closed.
[root@gibbon tmp]

Great.  That’s it for now.

Wait, I’m not giving up quite yet.  On LinuxQuestions, I found this: http://www.linuxquestions.org/questions/linux-enterprise-47/rhel-6-ldap-now-requires-tls-843917/. So it has a few ideas:

[root@gibbon pam.d] authconfig --enableldap --enableldapauth --ldapserver=tempest.wellesley.edu --ldapbasedn="dc=tempest,dc=wellesley,dc=edu" --update
Stopping sssd:                                             [  OK  ]
[root@gibbon pam.d] emacs -nw /etc/sysconfig/authconfig 
[root@gibbon pam.d] grep FORCELEGACY /etc/sysconfig/authconfig
FORCELEGACY=yes
[root@gibbon pam.d]

Does that work? Nope, it doesn’t. Here’re the lines from /var/log/secure:

Sep 26 15:19:47 gibbon sshd[2613]: Invalid user anderson from 149.130.136.34
Sep 26 15:19:47 gibbon sshd[2614]: input_userauth_request: invalid user anderson
Sep 26 15:19:49 gibbon sshd[2613]: pam_unix(sshd:auth): check pass; user unknown
Sep 26 15:19:49 gibbon sshd[2613]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gibbon.wellesley.edu 
Sep 26 15:19:49 gibbon sshd[2613]: pam_succeed_if(sshd:auth): error retrieving information about user anderson
Sep 26 15:19:52 gibbon sshd[2613]: Failed password for invalid user anderson from 149.130.136.34 port 60323 ssh2
Sep 26 15:19:52 gibbon sshd[2614]: Connection closed by 149.130.136.34
[root@gibbon ~]

And the info from ldapsearch:

root@gibbon ~] ldapsearch -x "uid=anderson"
# extended LDIF
#
# LDAPv3
# base <dc=cs,dc=wellesley,dc=edu> (default) with scope subtree
# filter: uid=anderson
# requesting: ALL
#

# anderson, People, cs.wellesley.edu
dn: uid=anderson,ou=People,dc=cs,dc=wellesley,dc=edu
uid: anderson
cn: Scott D. Anderson
objectClass: account
objectClass: posixAccount
objectClass: top
objectClass: shadowAccount
shadowLastChange: 12494
shadowMax: 99999
shadowWarning: 7
shadowFlag: 134523415
loginShell: /bin/bash
uidNumber: 716
gidNumber: 501
homeDirectory: /home/anderson
gecos: Scott D. Anderson,E114

# search result
search: 2
result: 0 Success

# numResponses: 2
# numEntries: 1
[root@gibbon ~]

And from getent:

getent passwd anderson
[root@gibbon ~] grep passwd /etc/nsswitch.conf 
#passwd:    db files nisplus nis
passwd:     files sss
[root@gibbon ~] service sssd status
sssd is stopped
[root@gibbon ~]

Just start sssd?

[root@gibbon ~] getent passwd anderson
anderson:*:716:501:Scott D. Anderson,E114:/home/anderson:/bin/bash
[root@gibbon ~] ssh anderson@gibbon
anderson@gibbon's password: 
Last login: Wed Sep 26 14:23:51 2012 from gibbon.wellesley.edu
[anderson@gibbon ~]

Interesting.  Can I also su?

[anderson@gibbon ~] su - anderson
Password: 
su: incorrect password
[anderson@gibbon ~] su - anderson
Password: 
su: incorrect password
[anderson@gibbon ~] logout
Connection to gibbon closed.
[root@gibbon ~] tail /var/log/secure
Sep 26 15:24:12 gibbon sshd[2693]: Accepted password for anderson from 149.130.136.34 port 60328 ssh2
Sep 26 15:24:12 gibbon sshd[2693]: pam_unix(sshd:session): session opened for user anderson by (uid=0)
Sep 26 15:24:18 gibbon su: pam_unix(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost=  user=anderson
Sep 26 15:24:18 gibbon su: pam_sss(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson
Sep 26 15:24:18 gibbon su: pam_sss(su-l:auth): received for user anderson: 4 (System error)
Sep 26 15:24:35 gibbon su: pam_unix(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost=  user=anderson
Sep 26 15:24:35 gibbon su: pam_sss(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson
Sep 26 15:24:35 gibbon su: pam_sss(su-l:auth): received for user anderson: 9 (Authentication service cannot retrieve authentication info)
Sep 26 15:24:39 gibbon sshd[2695]: Received disconnect from 149.130.136.34: 11: disconnected by user
Sep 26 15:24:39 gibbon sshd[2693]: pam_unix(sshd:session): session closed for user anderson
[root@gibbon ~]

No, of course not.  Reverting again.

I just tried to ssh from Puma to Gibbon (as root) and got the following:

[root@puma ~] ssh gibbon
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
63:20:c2:e9:53:95:ab:f8:d3:d9:50:76:16:60:cb:06.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /etc/ssh/ssh_known_hosts:23
RSA host key for gibbon has changed and you have requested strict checking.
Host key verification failed.
[root@puma ~]

Notice that the ssh completely failed, and you have to edit a file (in this case a system file, which ordinary users can’t do) in order to fix the problem.  The script /usr/network/scripts/centos6.3-client-part2.script is the script that pushes out the correct ssh keys for a re-installed client.  (Maybe we should rename that script?)  That needs to be run on tempest, because the sshkeys live on that machine under /root. (They should be protected so that inspired hackers don’t modify them to perpetrate an actual man-in-the-middle attack.)

Note that since “allow-root-connect” works by ssh’ing to the client machine, invoking that command near the beginning of the client-part2 script will fail.  So we’ll have to re-think that idea.  In any event, I’m going to use the un-modified client-part2 script:

[root@tempest ~] /usr/network/scripts/centos6.3-client-part2.script gibbon
Modify known hosts to comment out client pub key
delete root's known hosts, to avoid trouble
copying ssh keys to client.
We *expect* scp to complain, so just say 'yes' 
The authenticity of host 'gibbon (149.130.136.34)' can't be established.
RSA key fingerprint is 63:20:c2:e9:53:95:ab:f8:d3:d9:50:76:16:60:cb:06.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'gibbon,149.130.136.34' (RSA) to the list of known hosts.
root@gibbon's password: 
moduli                                                                                    100%  130KB 129.7KB/s   00:00    
ssh_host_dsa_key                                                                          100%  668     0.7KB/s   00:00    
ssh_host_dsa_key.pub                                                                      100%  590     0.6KB/s   00:01    
ssh_host_key                                                                              100%  963     0.9KB/s   00:00    
ssh_host_key.pub                                                                          100%  627     0.6KB/s   00:00    
ssh_host_rsa_key                                                                          100% 1675     1.6KB/s   00:00    
ssh_host_rsa_key.pub                                                                      100%  382     0.4KB/s   00:00    
restoring known hosts files
delete root's known hosts file after accepting key
root@gibbon's password: 
Stopping sshd: [  OK  ]
Starting sshd: [  OK  ]
Unmounting NFS filesystems:  [  OK  ]
Mounting NFS filesystems:  [  OK  ]
Mounting other filesystems:  [  OK  ]
root@gibbon's password: 
[root@tempest ~]

Okay, that’s normal behavior.  Now, we should be able to ssh to gibbon:

[root@tempest ~] ssh gibbon
root@gibbon's password: 
[root@gibbon ~]

Good, but we don’t want to be asked for a password, so one last step:

[root@tempest ~] allow-root-connect gibbon
root@gibbon's password: 
mkdir: cannot create directory `.ssh': File exists
root@gibbon's password: 
[root@tempest ~]

And make sure that worked:

[root@tempest ~] ssh gibbon
Last login: Wed Sep 26 10:42:35 2012 from tempest.wellesley.edu
[root@gibbon ~]

Good!  We’re done.  I’ve modified the client part2 script to run allow-root-connect at the end.  We’ll have to type the client root password three times in all.  I don’t see any way to improve that, but you’re welcome to try.

Lyn and Mike want to install LimeSurvey on Tempest. Given that it requires PHP and MySQL, and I’ve been having trouble with that combination, we may be in for trouble, but maybe something will go well.

First, we need to have the PHP modules mbstring (for multi-byte string; handles unicode character sets).  Looking at phpinfo for Puma, I don’t find it.  Their advice is to install the php-mbstring package using Yum.  That’s easy enough:

[root@tempest ~]# yum search php-mbstring
================================================ N/S Matched: php-mbstring =================================================
php-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling

  Name and summary matches only, use "search all" for everything.
[root@tempest ~]# rpm -q php-mbstring
package php-mbstring is not installed
[root@tempest ~]# yum -y install php-mbstring
Installed:
  php-mbstring.x86_64 0:5.3.3-14.el6_3                                                                                      
Complete!

Okay, now let’s try Puma…

[root@puma ~] rpm -q php-mbstring
package php-mbstring is not installed
[root@puma ~] yum search php-mbstring
================================================== Matched: php-mbstring ===================================================
php-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
php53-mbstring.x86_64 : A module for PHP applications which need multi-byte string handling
[root@puma ~] yum -y install php-mbstring
Resolving Dependencies
--> Running transaction check
---> Package php-mbstring.x86_64 0:5.1.6-39.el5_8 set to be updated
--> Processing Dependency: php-common = 5.1.6-39.el5_8 for package: php-mbstring
--> Finished Dependency Resolution
php-mbstring-5.1.6-39.el5_8.x86_64 from updates has depsolving problems
  --> Missing Dependency: php-common = 5.1.6-39.el5_8 is needed by package php-mbstring-5.1.6-39.el5_8.x86_64 (updates)
Error: Missing Dependency: php-common = 5.1.6-39.el5_8 is needed by package php-mbstring-5.1.6-39.el5_8.x86_64 (updates)
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
[root@puma ~] rpm -q php-common
php-common-5.2.13-jason.1
[root@puma ~] rpm -q php
php-5.2.13-jason.1
[root@puma ~] yum install php53-mbstring
Resolving Dependencies
--> Running transaction check
---> Package php53-mbstring.x86_64 0:5.3.3-13.el5_8 set to be updated
--> Processing Dependency: php53-common = 5.3.3-13.el5_8 for package: php53-mbstring
--> Running transaction check
---> Package php53-common.x86_64 0:5.3.3-13.el5_8 set to be updated
--> Processing Conflict: php53-common conflicts php-common
--> Finished Dependency Resolution
php53-common-5.3.3-13.el5_8.x86_64 from updates has depsolving problems
  --> php53-common conflicts with php-common
Error: php53-common conflicts with php-common
 You could try using --skip-broken to work around the problem
 You could try running: package-cleanup --problems
                        package-cleanup --dupes
                        rpm -Va --nofiles --nodigest
[root@puma ~]

I see that Puma is going to fight us on this.  Let’s turn back to Tempest.  If I now look at the output of phpinfo (after restarting Apache), I see mbstring.  Good!

The same php-info shows the mysql is 5.1.61, which is the same version reported by the mysql shell command, so I think we are okay on that.

Other “standard” things we’re supposed to have.  Again, I’m looking at the output of php-info:

• session support?  yes
• pcre?  yes
• ctype?  yes

Okay, hopefully we’re set for those.

Optional PHP Extensions:

• GD-Library with FreeType support installed is needed for captchas or nice charts in statistics – see PHP GD-Library Extension documentation? No, not this.
• IMAP (pretty standard) is needed for the Email bounce tracking system – see PHP IMAP Extension documentation?  No, not this.
• LDAP installed is needed to import tokens using LDAP – see PHP LDAP Documentation? Do we really have to have this?
• Zip (pretty standard) for template upload and import resources Zip . archives – see PHP Zip Extension documentation
  ? Yes, we have this.
• Zlib (pretty standard) is needed for the ComfortUpdate – see PHP Zlib Extension documentation?  Yes, we have this.

Let’s see about installing these.

[root@tempest ~]# yum search php | grep -i gd
php-gd.x86_64 : A module for PHP applications for using the gd graphics library
[root@tempest ~]# rpm -q php-gd
package php-gd is not installed
[root@tempest ~]# yum search php | grep -i imap
php-imap.x86_64 : A module for PHP applications that use IMAP
[root@tempest ~]# rpm -q php-imap
package php-imap is not installed
[root@tempest ~]# yum search php | grep -i ldap
php-ldap.x86_64 : A module for PHP applications that use LDAP
[root@tempest ~]# rpm -q php-ldap
package php-ldap is not installed
[root@tempest ~]# yum -y install php-gd php-imap php-ldap
Installed products updated.
  Verifying  : php-ldap-5.3.3-14.el6_3.x86_64                                                                           1/4 
  Verifying  : libc-client-2007e-11.el6.x86_64                                                                          2/4 
  Verifying  : php-imap-5.3.3-14.el6_3.x86_64                                                                           3/4 
  Verifying  : php-gd-5.3.3-14.el6_3.x86_64                                                                             4/4 

Installed:
  php-gd.x86_64 0:5.3.3-14.el6_3         php-imap.x86_64 0:5.3.3-14.el6_3         php-ldap.x86_64 0:5.3.3-14.el6_3        

Dependency Installed:
  libc-client.x86_64 0:2007e-11.el6                                                                                         

Complete!
[root@tempest ~]# apachectl graceful

Okay, that wasn’t too hard.  php-info shows all those modules.  Woo-hoo!

Now, to install LimeSurvey, from the zip file.