Bootable USB drive

I wanted to make a bootable USB drive to install Linux (CentOS 6.6) on a PC.  I downloaded the CentOS .iso file using Bittorrent, but then I needed to write it onto the USB drive. My office machine is a Mac, so I found these directions:

Those directions worked great, except I had to think to make one small modification, which is that I don’t have to convert the .iso file to a Mac .dmg file.  So the key steps are:

Insert the USB drive, which will then automatically be mounted, and then unmount it. Do the “dd” copy, and then eject the drive:

diskutil unmountDisk /dev/diskN
sudo dd if=CentOS-6.7-x86_64-bin-DVD1to2/CentOS-6.7-x86_64-bin-DVD1.iso of=/dev/disk3 bs=1m
3715+0 records in
3715+0 records out
3895459840 bytes transferred in 2268.510931 secs (1717188 bytes/sec)
diskutil eject /dev/diskN

I was shocked at how long the “dd” took. Copying a 3.6 GB iso file took well over half an hour! I think that’s longer than writing a DVD!

Posted in Uncategorized | Leave a comment

Configure rsnapshot on Tempest

I’m configuring rsnapshot to run on Tempest. It had been running, for many years, on Puma, but we’re going to re-install on Puma and I want the two machines to be as identical as possible, so both will be acting for themselves, not for each other.

Rsnapshot is very nice software for backing up machines and conserving disk space by using hard-links to identical files instead of copies. This isn’t perfect, but it’s very good. The imperfections are

  • if a file hasn’t changed, so that there are many hard-links to a single copy, but that single copy gets corrupted, you essentially have no backups, and
  • a filesystem with many hard-links can prove challenging to fsck, as I learned recently

We now use CrashPlan for backups of user files, but I like using rsnapshot because we can easily see and compare system configuration changes.

Here’s how I set it up on Tempest.

First, install using Yum:

[root@tempest ~] yum search rsnapshot
No Matches found
[root@tempest ~]

Darn, not in the standard repos. So, guess that it might be in EPEL:

[root@tempest ~] yum --enablerepo=epel search rsnapshot
============================== N/S Matched: rsnapshot ==============================
rsnapshot.noarch : Local and remote filesystem snapshot utility
Name and summary matches only, use "search all" for everything.
[root@tempest ~]

Win! Okay, install it:

[root@tempest ~] yum --enablerepo=epel install rsnapshot
 Setting up Install Process
 Resolving Dependencies
 --> Running transaction check
 ---> Package rsnapshot.noarch 0:1.3.1-12.el6 will be installed
 --> Finished Dependency Resolution
Dependencies Resolved
============= Package Arch Version Repository Size =================================
 rsnapshot noarch 1.3.1-12.el6 epel 116 k
Transaction Summary
 Install 1 Package(s)
Total download size: 116 k
 Installed size: 377 k
 Is this ok [y/N]: y
 Downloading Packages:
 rsnapshot-1.3.1-12.el6.noarch.rpm | 116 kB 00:00
 Running rpm_check_debug
 Running Transaction Test
 Transaction Test Succeeded
 Running Transaction
 Installing : rsnapshot-1.3.1-12.el6.noarch 1/1
 warning: /etc/rsnapshot.conf created as /etc/rsnapshot.conf.rpmnew
 Verifying : rsnapshot-1.3.1-12.el6.noarch 1/1
 rsnapshot.noarch 0:1.3.1-12.el6
 [root@tempest ~]

The .rpmnew is because I had previously copied over the old rsnapshot.conf file. Let’s see what the differences are:

[root@tempest etc] diff rsnapshot.conf rsnapshot.conf.rpmnew
< snapshot_root /root/snapshots/tempest/ --- > snapshot_root /.snapshots/

That’s just where the copies will be stored. We’ll store them on /data, which is a different piece of hardware (a NAS), so that if Tempest completely dies, we’ll see what the configuration files are on /data

< no_create_root 1
> #no_create_root 1

This variable says not to do the backups if the snapshot_root isn’t there. If the NAS isn’t there or isn’t mounted, we don’t want to fill up this machine.

< #interval hourly 6
> interval hourly 6
< interval monthly 6
> #interval monthly 3

These say how many hourly or monthly backups to retain before they are deleted. We don’t use hourly, and 6 months is more than enough.

< one_fs 1
> #one_fs 0

This says that if you get to another file system, when recursively traversing one that you are backing up, skip it. Setting this to 1 keeps us from backing up /home and /students and /data2 and even /data when backing up /. However, it means we have to list all the ones we do want to back up.

< backup / slash/ exclude=/proc,exclude=/dev,exclude=/tmp,exclude=/sys
< backup /boot/ boot/
< backup /var/www www/ exclude=/var/www/html/fedora/
< backup /var/lib/tomcat6 tomcat/
> backup /home/ localhost/
> backup /etc/ localhost/
> backup /usr/local/ localhost/
> #backup /var/log/rsnapshot localhost/
[root@tempest etc]

And the preceding is the stuff we want to back up. We don’t back up /usr, because that’s mostly executable files that we don’t need. We similarly don’t need to back up most of /var, but we do want to back up the web pages and stuff in /var/www. Similarly, all the web applications in /var/lib/tomcat6.

Next, we need to configure when these things happen:

[root@tempest etc] cat /etc/crontab
# rsnapshot. Need to allow enough time for the deletes to finish
# so I did these every hour
10 0 1 * * root rsnapshot monthly
10 1 * * 0 root rsnapshot weekly
10 2 * * * root rsnapshot daily

Check the syntax of the config file:

[root@tempest etc] rsnapshot configtest
 rsnapshot encountered an error! The program was invoked with these options:
 /usr/bin/rsnapshot configtest
 ERROR: /data/rsnapshots/tempest does not exist.
 ERROR: rsnapshot refuses to create snapshot_root when no_create_root is enabled
 [root@tempest etc] mkdir /data/rsnapshots/tempest
 [root@tempest etc] rsnapshot configtest
 Syntax OK

Good. Now, run it for the first time.


Posted in Uncategorized | Leave a comment

HTTPS certificates

This is for a RHEL/CentOS 6.6 machine.

There isn’t a genkey command, but in /etc/pki/tls/certs/ there is a Makefile that seems to do what we want:

make usage

prints some helpful and encouraging messages. Since I want to set up HTTPS to this machine and it uses apache, I did:

make genkey

Hmph. That says there’s nothing to be done. Okay, so maybe the keys already exist (were created when this server was set up), but I need to generate a certificate request. So:

make certreq

Aha, that prompts me for a number of fields:

Country Name:US
State or Province Name:Massachusetts
Locality Name:Wellesley
Organization Name:Wellesley College
Organizational Unit:Computer Science

The only one that gave me any pause was the Common Name, but apparently that’s the fully qualified domain name of the server. I found this site helpful:

That created a file in the local directory called localhost.csr.

I copy/pasted that into the box on the form on the website of the cert authority that Tim Cantin said we have a contract with, and clicked “get common name from CSR” and it complained that the public key size has to be a minimum of 2048 bits. Apparently the Makefile that RHEL/CentOS comes with is less than that. Back to the drawing board:

Referencing this website: and I went back to the server and its /etc/pki/tls/certs/ directory and did:

openssl genrsa -out studentorgs.key 2048

That generates the private key. Then I did

openssl req -new -key studentorgs.key -out studentorgs.csr

That prompts for the same info as above and creates the CSR file.

I then copy/pasted that into the Cert Authority’s web form and that worked!

Posted in Uncategorized | Leave a comment


Two very very late posts about what to do if you think you screwed up a config file or other system file (you do not necessarily an update — might affect other dependencies) and how to set up varnish.

Part I: What to do if you think you screwed up a config file

When we attempted to download varnish on 11/8, we hit some issues with our httpd.conf file that was just outdated/incorrect (for reasons unknown, problem revealed later). For context, on our machines, we edit the httpd.conf file in the sysadmin account and use a script to update the tempest server’s httpd.conf.

Our problem: When we tried to restart httpd, we got an error that it “cannot load httpd/modules/” We had restarted apache the previous Wednesday to test out varnish on puma, so this was peculiar.

Scott did some detective work in the rsnapshots since April, saw that the other httpd.conf files hadn’t needed to load After grepping through the yum.logs, we saw that httpd had last updated on october 14th. Did this actually help? We were still confused.

We first tried just commenting out the module in httpd.conf. Then it failed and wanted another module. Trying again: we thought maybe we should just redownload a copy of httpd.conf (using yum reinstall). However, we were concerned that yum reinstall would overwrite the entire httpd directory: there were some (not many) Wellesley specific edits (wellesley.conf) that we didn’t want to overwrite, so we backed them up in the sysadmin account…but then found this article! The yum downloadonly flag makes yum only download the package and not install/update them.

>>yumdownloader -v httpd

which puts the httpd files in the current directory. After moving them to /var/tmp/, we used rpm2cpio to extract the files. Ideally, we thought this way we would be able to minimize the amount of edits transferring needed. To untar, we used cpio -idmv:

d: make directories

i: extract

m: preserve modification times

v: vebose

After trying this out, it created subdirectory /usr/var/etc. We looked into etc and saw httpd and logrotate.d and sysconfig. We went into httpd/conf, and saw a httpd.conf file, last updated August 15th.

Interesting: we diff’d the new file and our current httpd.conf (the weird reverted version) and saw that there was a lot of student stuff that had been removed before. Purely looking at the size of the different httpd.conf files, the one we just downloaded using yum –downloadonly was ~35k (as were the rsnapshot daily backups we looked at before) while the current httpd.conf file was ~55k, this confirmed that the version of httpd.conf was odd. Hmm. We ended up just copying the rsnapshot version of httpd.conf (since the snapshot versions had the Wellesley conf edits) into the sysadmin/etc/ version. Comparing the rsnapshot version with the newly downloaded version, confirmed that they weren’t all that different, so we tested, it worked, and we proceeded to use Scott’s httpd.conf update script to update the httpd.conf

>> service httpd start

*now works!

After this side journey in fixing the httpd.conf file, we then cleaned up the /var/tmp (can delete the usr and var and etc), and we kept the yum –downloadonly file so that we can always cpio it again if we want some more info.

So takeaways from the mystery httpd.conf file problem: use yum –downloadonly if necessary so you don’t overwrite your entire directory, use cpio to untar (know that yum –download only downloads into the current directory that you are in). Have an excellent week!

Part II: Setting up varnish

In light of some recent research completions, there were some concerns about whether tempest could handle the brunt of requests hitting the server. After some weeks of exploration, we decided to try out varnish and set it up on tempest. We’re still keeping track of whether varnish will work best for our system, but this is what we did to configure it:

  1. Edit /etc/varnish/default.vcl

We changed the backend default to be port 800 since we wanted varnish pointing to

  1. We changed apache to be listening on port on /etc/httpd/conf/httpd.conf to listen to varnish.

So what we have now is we’ve set up varnish backend to point to port 800 and for apache to see varnish on port 800.

  1. Now we need to set varnish to listen on port 80 (where apache used to be). To do this, we configured /etc/sysconfig/varnish to alternative option 3. We hit a problem when trying to test this out on puma where we forgot to comment out another alternative option. Doing that will make varnish not work.

    1. minor edit: we set the varnish cache file size from its default 256M to 1GB

After doing >>service varnish start, it seemed to work.

Note: you can’t test by having varnish and apache going at the same time(if both listen to port 80…), so have to restart apache first (Won’t be listening on port 80) then start varnish to listen on port 80.

  1. We tested varnish to see if it impacted our requests by using >>GET -Used to see what happens when we request a file. To check varnish stats, look at… >>varnishstats which will show the cache hits. When testing it on puma, we also used >>nmap puma to see what was on port 80.

So excellent! Varnish works. We’ll probably have another post soon about observed changes with varnish.

Posted in Uncategorized | Leave a comment


A few months ago, we talked about adding a robots.txt file so that webcrawlers won’t index archived courses, such as the course material for cs110 back in fall of 2012.

The page is, unsurprisingly, the authority on creating a robots.txt file. You put the robots.txt file in the top level directory of the webserver, which on tempest is /var/www/html. When I looked, there was actually already a robots.txt page, so I just added a few lines like  “Disallow:/~cs110f12/”. Hopefully this will prevent robots from indexing those archived courses.

Posted in Uncategorized | Leave a comment

Installing HTTPS certificates

To create a signed certificate, we first ran genkey and created a file that I happened to call  I put that in /etc/pki/tls/certs.

When I uploaded it to the certificate company’s website, they complained that it didn’t meet the 2048 bit minimum, so I created a second as

After they did their work, I got an email back that looked like this:

* Click the following link to download your SSL certificate (generally try to use a version that includes intermediates & root or your certificate may be rejected by some older clients)

Format(s) most suitable for your server software:
as X509 Certificate only, Base64 encoded:
as X509 Intermediates/root only, Base64 encoded:

And no further information.  Thanks to help from Andy Maroney, I now know that these files are the targets of two config variables in /etc/httpd/conf.d/ssl.conf.  There’s also a third variable that needs to be configured.

  • The first one downloads as cs_wellesley_edu_cert.cer. I put it in /etc/pki/tls/certs and made it the value of SSLCertificateFile
  • The second one downloads as cs_wellesley_edu_interm.cer. I put it in /etc/pki/tls/certs and made it the value of SSLCertificateChainFile
  • The third variable is SSLCertificateKeyFile and the value of that is set to a file that was generated simultaneously with the original .csr file, and that is /etc/pki/tls/private/ (That file holds the private key and is readable only by root.)

Restart Apache and you’re done!

Posted in Uncategorized | Leave a comment

Decreasing Logwatch Output

In the recent past, logwatch has been spitting out way, way too much info about activity on tempest. The output that the cs-sysadmin email receives is actually truncated. The main culprit is HTTP, which spits out every link that gets a 401, 403, and 404 error. On puma, we’ve turned off http messages completely, but this time we wanted to figure out a more refined way to filter messages.

I found the documentation for logwatch in /usr/share/doc. The default configuration files are found in the directory /usr/share/logwatch/default.conf/services for services such as http, sshd, pam_unix, etc. The default scripts are in /usr/share/logwatch/scripts/services. These scripts are all written in Perl.

To override the default configuration, the documentation recommends 2 ways. You can add in whatever overriding variables in the /etc/logwatch/conf directory in a file called logwatch.conf. Or you could copy the default from /usr/share/logwatch to the corresponding directory in /etc/logwatch, and modify the file in the /etc/logwatch folder.

You could set variables in the /etc/logwatch/conf directory. On puma, this is how we turned off the output for http in the logwatch.conf file, since the output was getting voluminous. However, the configuration files did not give us fine enough control over the output. I had to modify the scripts themselves.

To modify scripts, I copied the appropriate default script from /usr/share/logwatch/scripts/services over to /etc/logwatch/scripts/services, where I fiddled around with the output. For example for the http script, I changed it so that it prints out the total number of 404 errors tempest collects for the day, instead of every single URL.

Once you’ve made the change, you can just run logwatch and have it print to your terminal with the following command:

logwatch --print [--service=<servicename>]

In general, to configure logwatch to your liking, copy the original file from /usr/share/logwatch to its corresponding place in /etc/logwatch, and make your edits to the /etc/logwatch file. The output will be much more readable.




Posted in Uncategorized | Leave a comment

CentoOS Reinstall Adventures – Password Trouble

Karina and I sat down in 173 this evening to make sure that the clients are working. We’re pretty sure orangutan, gorilla, gibbon, and tamarin are all good to go, and we’re currently installing centos on chimp.
However, when I log into these machines, I can’t log in with my new password. My old password gets me in fine, and my new one doesn’t! I thought we resolved this issue when I ran the password-change script on tempest a couple weeks ago…
In our start-up notes, we did say “no” to using auth config, then “yes” to flat-files, keeping in mind that we need to do authentication later. Perhaps this is related to my login issues?


Posted in Uncategorized | Leave a comment

cp -al taking a long time

Our backups on Puma have been taking a long time, finishing late in the day, almost in time for the next backup.  The problem seemed to be in the cp -al step, not in the rsync step.  I investigated, looking at how long the cp -al on each directory took, using code like this:

for a in `ls -A $from`; do
now=`date +%T`
echo "$now cp -al $from/$a to $to/$a"
cp -al $from/$a $to/$a

and the result looked like:

16:38:06 cp -al 2014-03-26/alice to today/alice
16:38:06 cp -al 2014-03-26/anderson to today/anderson
16:39:37 cp -al 2014-03-26/apache-tomcat-5.5.26 to today/apache-tomcat-5.5.26
16:39:42 cp -al 2014-03-26/appinvstats to today/appinvstats
22:59:14 cp -al 2014-03-26/appinv-stats to today/appinv-stats
22:59:14 cp -al 2014-03-26/btjaden to today/btjaden
23:11:11 cp -al 2014-03-26/compbio to today/compbio
23:11:12 cp -al 2014-03-26/cs to today/cs
23:11:24 cp -al 2014-03-26/cs110f11 to today/cs110f11

It finished at about midnight (so, less than 8 hours total), but essentially all of that time was in the appinvstats directory.

Sure enough, some subdirectories of that account had a *lot* of inodes.  Here are some useful references:

A count of the inodes:

ls -fR collectedStats | wc -l
ls -fR errorFiles | wc -l

So, about 1 million files/folders or inodes.

Turning these in to tarfiles would reduce these to two inodes.  There’s also a savings in space:

du -csh errorFiles errorFiles.tar
129G    errorFiles
106G    errorFiles.tar
du -csh collectedStats collectedStats.tar
11G     collectedStats
3.3G       collectedStats.tar

We’ll look at replacing these directories with the tar files.


Posted in Uncategorized | Leave a comment

Install Python2.7 w/ Tkinter

Rhys asked for us to install Tkinter for Python2.7.  [Remember, that Python2.6 is the default Python on Tempest because it’s what RHEL comes with, but most users are using Python 2.7.]  Here’s the current behavior:

[anderson@tempest ~] python2.6
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *
[anderson@tempest ~] python2.7
Python 2.7 (r27:82500, Sep 20 2012, 17:09:01) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/lib-tk/", line 39, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named _tkinter

Which is obviously no good.  This seemed pretty straightforward, since the fact that it works for Python2.6 suggests that I have all the necessary .so files and such.  However, I couldn’t find any pre-compiled binaries for Python2.7 for RHEL6/CentOS6.  I tried compiling Python for myself, just like I did last August (  However, that resulted in exactly the same ImportError.

Here’s the best of the “how to compile Python” info pages that I found.  However, I tried one more time, but this time, I discovered a comment that explained that I have to install tk-devel.  That, however, is a piece of cake.

# yum -y install tk-devel

To avoid any issue with over-writing the existing Python2.7, which is installed in /usr/local/, I installed the new, improved, Python2.7 in /opt:

# wget
# tar xf Python-2.7.6.tar.xz
# cd Python-2.7.6
# ./configure --prefix=/opt --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /opt/lib"
# make && make altinstall

The first time I tried that, it failed miserably:

checking build system type... x86_64-unknown-linux-gnu
 checking host system type... x86_64-unknown-linux-gnu
 checking for --enable-universalsdk... no
 checking for --with-universal-archs... 32-bit
 checking MACHDEP... linux2
 checking EXTRAPLATDIR...
 checking for --without-gcc... no
 checking for gcc... gcc
 checking whether the C compiler works... no
 configure: error: in `/var/tmp/install-python2.7/Python-2.7.6':
 configure: error: C compiler cannot create executables
 See `config.log' for more details

It turned out that the trouble was that there wasn’t a /opt/lib directory.  Doing a “mkdir /opt/lib” and re-doing the config command worked perfectly, as did the “make” and “make altinstall”

So, now we have:

Python 2.7.6 (default, Mar 24 2014, 13:29:51)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *

But, we also need to make sure that the new Python2.7 has all the libraries that the old Python2.7 had.

I learned that Pip can tell you the libraries it has loaded:

# /usr/local/bin/pip-2.7 freeze

Compare to:

# /opt/bin/pip2.7 freeze

We can dump the former into a file and use it to set up the new Python:

# umask 0002
# /opt/bin/pip2.7 freeze
# rm old-py2.7-modules
# /usr/local/bin/pip-2.7 freeze > old-py2.7-modules
# /opt/bin/pip2.7 install -r old-py2.7-modules
Downloading/unpacking CouchDB==0.9 (from -r old-py2.7-modules (line 1))
Downloading CouchDB-0.9.tar.gz (55kB): 55kB downloaded
Running (path:/tmp/pip_build_root/CouchDB/ egg_info for package CouchDB

Downloading/unpacking MySQL-python==1.2.4b5 (from -r old-py2.7-modules (line 2))
Downloading MySQL-python-1.2.4b5.tar.gz (82kB): 82kB downloaded

Well, that *mostly* worked.  We ran into trouble after a while:

Downloading/unpacking scikit-learn==0.12.1 (from -r old-py2.7-modules (line 27))
Downloading scikit-learn-0.12.1.tar.gz (3.0MB): 3.0MB downloaded
Running (path:/tmp/pip_build_root/scikit-learn/ egg_info for package scikit-learn
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/scikit-learn/", line 36, in <module>
from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core
Complete output from command python egg_info:
Partial import of sklearn during the build process.

Traceback (most recent call last):

File "<string>", line 17, in <module>

File "/tmp/pip_build_root/scikit-learn/", line 36, in <module>

from numpy.distutils.core import setup

ImportError: No module named numpy.distutils.core

Cleaning up...
Command python egg_info failed with error code 1 in /tmp/pip_build_root/scikit-learn
Traceback (most recent call last):
File "/opt/bin/pip2.7", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip2.7')()
File "/opt/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/", line 185, in main
return command.main(cmd_args)
File "/opt/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 70: ordinal not in range(128)

How much got installed?  Let’s see:

# /opt/bin/pip2.7 freeze

Well, that stinks! Is it all or nothing?  The error output suggests that there may be a dependency on numpy, so let’s install that first:

# /opt/bin/pip2.7 install numpy
# /opt/bin/pip2.7 freeze

Okay, and I notice that the version is newer than the numpy in the old Python2.7. So, this will be annoying and tedious, but I guess I may have to install things by hand. One more try:

# /usr/local/bin/pip-2.7 freeze | cut -d= -f1 | grep -v numpy > old-py2.7-module-names
# /opt/bin/pip2.7 install -r old-py2.7-module-names

Wait a *long* time, and it succeeds!

# diff old-py2.7-module-names new-py2.7-module-names
> numpy
> scipy

Woo hoo!

Note that pip can be used to upgrade Python packages.

Posted in Uncategorized | Leave a comment