CS SysAdmin

My CentOS 6 machine is running Python2.7 alongside the default Python 2.6. A user wanted me to install Pygame. Pygame is in the yum repos but only for python 2.6. How to get it for Python 2.7?

I started with this: http://www.pygame.org/wiki/CompileRedHat?parent=Compilation

That had me install a bunch of RPMs (remember to enable repos like EPEL and RPMFusion). Unfortunately, that repo does *not* have the smpeg and smpeg-devel packages. I can find smpeg and smpeg-devel for Centos5 but not for Centos6. For CentOS 6, I can only find libsmpeg and libsmpeg-devel.

I crossed my fingers and installed that.

I then did:

# As user do the following
hg clone https://bitbucket.org/pygame/pygame
cd pygame
python2.7 setup.py
No Arguments Given, Perform Default Install? [Y/n]Y

WARNING, No "Setup" File Exists, Running "config.py"
Using UNIX configuration...

Hunting dependencies...
SDL : found 1.2.14
FONT : found
IMAGE : found
MIXER : found
SMPEG : found 0.4.5
PNG : found
JPEG : found
SCRAP : found
PORTMIDI: found
PORTTIME: not found
AVFORMAT: not found
SWSCALE : not found
FREETYPE: found 2.3.11

Warning, some of the pygame dependencies were not found. Pygame can still
compile and install, but games that depend on those missing dependencies
will not run. Would you like to continue the configuration? [Y/n]:

Note that it *did* find SMPEG!  (The first time I ran setup.py (not shown), it failed to find SMPEG.)

Success!

I needed to renew the certificate for my server and the certificate request (CSR) needed to be 2048 bits and SHA-2. My server is a RHEL (RedHat Enterprise Linux) system, so there is a Makefile in /etc/pki/tls/certs that should do the trick. However, I had a devil of a time. Here’s what finally worked.

First, I added “-sha256” to the openssl command line in the Makefile. (Sha-2 is a family of cryptographic hash functions, and SHA-256 is in that family.) Here is the relevant entry in the Makefile:

$(CSR): $(KEY)
       umask 77 ; \
       /usr/bin/openssl req -sha256 $(UTF8) -new -key $(KEY) -out $(CSR) 

In the Makefile, there are lines that specify where the private key file is/should be and where the CSR file is/should be

KEY=$(TLSROOT)/private/localhost.key

CSR=$(TLSROOT)/certs/localhost.csr

It’s the private key that needs to be 2048 bits. I think I had one that was 2048 bits, but I think I also had an older one that was smaller (probably 1024). The .key file doesn’t label itself with its length, but I noticed that the two files differed in length:

root@tempest private] ls -l *.key
-r--------. 1 root root 1737 Apr 13  2014 cs.wellesley.edu.key
-r--------. 1 root root  916 Jan 26  2014 localhost.key

The “make genkey” command said nothing needed to be done, but generating a csr using “make certreq” produced a CSR file that the signing authority rejected (too few bits). So, I edited the Makefile to point to the longer .key file and generated a new certreq. That one seems to have worked.

I wanted to make a bootable USB drive to install Linux (CentOS 6.6) on a PC.  I downloaded the CentOS .iso file using Bittorrent, but then I needed to write it onto the USB drive. My office machine is a Mac, so I found these directions:

http://www.ubuntu.com/download/desktop/create-a-usb-stick-on-mac-osx

Those directions worked great, except I had to think to make one small modification, which is that I don’t have to convert the .iso file to a Mac .dmg file.  So the key steps are:

Insert the USB drive, which will then automatically be mounted, and then unmount it. Do the “dd” copy, and then eject the drive:

diskutil unmountDisk /dev/diskN
sudo dd if=CentOS-6.7-x86_64-bin-DVD1to2/CentOS-6.7-x86_64-bin-DVD1.iso of=/dev/disk3 bs=1m
Password:
3715+0 records in
3715+0 records out
3895459840 bytes transferred in 2268.510931 secs (1717188 bytes/sec)
diskutil eject /dev/diskN

I was shocked at how long the “dd” took. Copying a 3.6 GB iso file took well over half an hour! I think that’s longer than writing a DVD!

I’m configuring rsnapshot to run on Tempest. It had been running, for many years, on Puma, but we’re going to re-install on Puma and I want the two machines to be as identical as possible, so both will be acting for themselves, not for each other.

Rsnapshot is very nice software for backing up machines and conserving disk space by using hard-links to identical files instead of copies. This isn’t perfect, but it’s very good. The imperfections are

• if a file hasn’t changed, so that there are many hard-links to a single copy, but that single copy gets corrupted, you essentially have no backups, and
• a filesystem with many hard-links can prove challenging to fsck, as I learned recently

We now use CrashPlan for backups of user files, but I like using rsnapshot because we can easily see and compare system configuration changes.

Here’s how I set it up on Tempest.

First, install using Yum:

[root@tempest ~] yum search rsnapshot
No Matches found
[root@tempest ~]

Darn, not in the standard repos. So, guess that it might be in EPEL:

[root@tempest ~] yum --enablerepo=epel search rsnapshot
============================== N/S Matched: rsnapshot ==============================
rsnapshot.noarch : Local and remote filesystem snapshot utility
Name and summary matches only, use "search all" for everything.
[root@tempest ~]

Win! Okay, install it:

[root@tempest ~] yum --enablerepo=epel install rsnapshot
 Setting up Install Process
 Resolving Dependencies
 --> Running transaction check
 ---> Package rsnapshot.noarch 0:1.3.1-12.el6 will be installed
 --> Finished Dependency Resolution
Dependencies Resolved
============= Package Arch Version Repository Size =================================
 Installing:
 rsnapshot noarch 1.3.1-12.el6 epel 116 k
Transaction Summary
 =================================
 Install 1 Package(s)
Total download size: 116 k
 Installed size: 377 k
 Is this ok [y/N]: y
 Downloading Packages:
 rsnapshot-1.3.1-12.el6.noarch.rpm | 116 kB 00:00
 Running rpm_check_debug
 Running Transaction Test
 Transaction Test Succeeded
 Running Transaction
 Installing : rsnapshot-1.3.1-12.el6.noarch 1/1
 warning: /etc/rsnapshot.conf created as /etc/rsnapshot.conf.rpmnew
 Verifying : rsnapshot-1.3.1-12.el6.noarch 1/1
Installed:
 rsnapshot.noarch 0:1.3.1-12.el6
Complete!
 [root@tempest ~]

The .rpmnew is because I had previously copied over the old rsnapshot.conf file. Let’s see what the differences are:

[root@tempest etc] diff rsnapshot.conf rsnapshot.conf.rpmnew
32c27
< snapshot_root /root/snapshots/tempest/ --- > snapshot_root /.snapshots/

That’s just where the copies will be stored. We’ll store them on /data, which is a different piece of hardware (a NAS), so that if Tempest completely dies, we’ll see what the configuration files are on /data

38c33
< no_create_root 1
> #no_create_root 1

This variable says not to do the backups if the snapshot_root isn’t there. If the NAS isn’t there or isn’t mounted, we don’t want to fill up this machine.

94c89
< #interval hourly 6
> interval hourly 6
97c92
< interval monthly 6
> #interval monthly 3

These say how many hourly or monthly backups to retain before they are deleted. We don’t use hourly, and 6 months is more than enough.

149c144
< one_fs 1
> #one_fs 0

This says that if you get to another file system, when recursively traversing one that you are backing up, skip it. Setting this to 1 keeps us from backing up /home and /students and /data2 and even /data when backing up /. However, it means we have to list all the ones we do want to back up.

196,209c195,199
< # TEMPEST
< backup / slash/ exclude=/proc,exclude=/dev,exclude=/tmp,exclude=/sys
< backup /boot/ boot/
< backup /var/www www/ exclude=/var/www/html/fedora/
< backup /var/lib/tomcat6 tomcat/
---
> # LOCALHOST
> backup /home/ localhost/
> backup /etc/ localhost/
> backup /usr/local/ localhost/
> #backup /var/log/rsnapshot localhost/
213c203
[root@tempest etc]

And the preceding is the stuff we want to back up. We don’t back up /usr, because that’s mostly executable files that we don’t need. We similarly don’t need to back up most of /var, but we do want to back up the web pages and stuff in /var/www. Similarly, all the web applications in /var/lib/tomcat6.

Next, we need to configure when these things happen:

[root@tempest etc] cat /etc/crontab
# rsnapshot. Need to allow enough time for the deletes to finish
# so I did these every hour
10 0 1 * * root rsnapshot monthly
10 1 * * 0 root rsnapshot weekly
10 2 * * * root rsnapshot daily

Check the syntax of the config file:

[root@tempest etc] rsnapshot configtest
----------------------------------------------------------------
 rsnapshot encountered an error! The program was invoked with these options:
 /usr/bin/rsnapshot configtest
--------------------------------------------------------------
 ERROR: /data/rsnapshots/tempest does not exist.
 ERROR: rsnapshot refuses to create snapshot_root when no_create_root is enabled
 [root@tempest etc] mkdir /data/rsnapshots/tempest
 [root@tempest etc] rsnapshot configtest
 Syntax OK

Good. Now, run it for the first time.

This is for a RHEL/CentOS 6.6 machine.

There isn’t a genkey command, but in /etc/pki/tls/certs/ there is a Makefile that seems to do what we want:

make usage

prints some helpful and encouraging messages. Since I want to set up HTTPS to this machine and it uses apache, I did:

make genkey

Hmph. That says there’s nothing to be done. Okay, so maybe the keys already exist (were created when this server was set up), but I need to generate a certificate request. So:

make certreq

Aha, that prompts me for a number of fields:

Country Name:US
State or Province Name:Massachusetts
Locality Name:Wellesley
Organization Name:Wellesley College
Organizational Unit:Computer Science
Common Name:studentorgs.wellesley.edu
Email Address:someone@wellesley.edu

The only one that gave me any pause was the Common Name, but apparently that’s the fully qualified domain name of the server. I found this site helpful: http://www.rackspace.com/knowledge_center/article/generate-a-csr-with-openssl

That created a file in the local directory called localhost.csr.

I copy/pasted that into the box on the form on the website of the cert authority that Tim Cantin said we have a contract with, and clicked “get common name from CSR” and it complained that the public key size has to be a minimum of 2048 bits. Apparently the Makefile that RHEL/CentOS comes with is less than that. Back to the drawing board:

Referencing this website: http://tecadmin.net/simple-steps-to-generate-csr-on-centos/ and http://wiki.centos.org/HowTos/Https. I went back to the server and its /etc/pki/tls/certs/ directory and did:

openssl genrsa -out studentorgs.key 2048

That generates the private key. Then I did

openssl req -new -key studentorgs.key -out studentorgs.csr

That prompts for the same info as above and creates the CSR file.

I then copy/pasted that into the Cert Authority’s web form and that worked!

Soon thereafter, I got a response from the Cert Authority with two links to download:

as X509 Certificate only, Base64 encoded: <link to download domain_cert.cer>
as X509 Intermediates/root only, Base64 encoded:  <link to download domain_interm.cer>

Copy both of those files to the /etc/pki/tls/certs directory on the server. Edit the /etc/httpd/conf.d/ssl.conf file and make the full pathname to the cert.cer file the value of SSLCertificateFile and the other the value of SSLCertificateChainFile

Note that the full pathname to domain.key should be the value of SSLCertificateKeyFile

restart Apache and you’re done!

Two very very late posts about what to do if you think you screwed up a config file or other system file (you do not necessarily an update — might affect other dependencies) and how to set up varnish.

Part I: What to do if you think you screwed up a config file

When we attempted to download varnish on 11/8, we hit some issues with our httpd.conf file that was just outdated/incorrect (for reasons unknown, problem revealed later). For context, on our machines, we edit the httpd.conf file in the sysadmin account and use a script to update the tempest server’s httpd.conf.

Our problem: When we tried to restart httpd, we got an error that it “cannot load httpd/modules/mod_file_cache.so.” We had restarted apache the previous Wednesday to test out varnish on puma, so this was peculiar.

Scott did some detective work in the rsnapshots since April, saw that the other httpd.conf files hadn’t needed to load mod_file_cache.so. After grepping through the yum.logs, we saw that httpd had last updated on october 14th. Did this actually help? We were still confused.

We first tried just commenting out the module in httpd.conf. Then it failed and wanted another module. Trying again: we thought maybe we should just redownload a copy of httpd.conf (using yum reinstall). However, we were concerned that yum reinstall would overwrite the entire httpd directory: there were some (not many) Wellesley specific edits (wellesley.conf) that we didn’t want to overwrite, so we backed them up in the sysadmin account…but then found this article http://www.cyberciti.biz/faq/yum-downloadonly-plugin/! The yum downloadonly flag makes yum only download the package and not install/update them.

>>yumdownloader -v httpd

which puts the httpd files in the current directory. After moving them to /var/tmp/, we used rpm2cpio to extract the files. Ideally, we thought this way we would be able to minimize the amount of edits transferring needed. To untar, we used cpio -idmv:

d: make directories

i: extract

m: preserve modification times

v: vebose

After trying this out, it created subdirectory /usr/var/etc. We looked into etc and saw httpd and logrotate.d and sysconfig. We went into httpd/conf, and saw a httpd.conf file, last updated August 15th.

Interesting: we diff’d the new file and our current httpd.conf (the weird reverted version) and saw that there was a lot of student stuff that had been removed before. Purely looking at the size of the different httpd.conf files, the one we just downloaded using yum –downloadonly was ~35k (as were the rsnapshot daily backups we looked at before) while the current httpd.conf file was ~55k, this confirmed that the version of httpd.conf was odd. Hmm. We ended up just copying the rsnapshot version of httpd.conf (since the snapshot versions had the Wellesley conf edits) into the sysadmin/etc/ version. Comparing the rsnapshot version with the newly downloaded version, confirmed that they weren’t all that different, so we tested, it worked, and we proceeded to use Scott’s httpd.conf update script to update the httpd.conf

>> service httpd start

*now works!

After this side journey in fixing the httpd.conf file, we then cleaned up the /var/tmp (can delete the usr and var and etc), and we kept the yum –downloadonly file so that we can always cpio it again if we want some more info.

So takeaways from the mystery httpd.conf file problem: use yum –downloadonly if necessary so you don’t overwrite your entire directory, use cpio to untar (know that yum –download only downloads into the current directory that you are in). Have an excellent week!

Part II: Setting up varnish

In light of some recent research completions, there were some concerns about whether tempest could handle the brunt of requests hitting the server. After some weeks of exploration, we decided to try out varnish and set it up on tempest. We’re still keeping track of whether varnish will work best for our system, but this is what we did to configure it:

1. Edit /etc/varnish/default.vcl

We changed the backend default to be port 800 since we wanted varnish pointing to 127.0.0.1:800.

2. We changed apache to be listening on port 127.0.0.1:800 on /etc/httpd/conf/httpd.conf to listen to varnish.

So what we have now is we’ve set up varnish backend to point to port 800 and for apache to see varnish on port 800.

3. Now we need to set varnish to listen on port 80 (where apache used to be). To do this, we configured /etc/sysconfig/varnish to alternative option 3. We hit a problem when trying to test this out on puma where we forgot to comment out another alternative option. Doing that will make varnish not work.

   1. minor edit: we set the varnish cache file size from its default 256M to 1GB

After doing >>service varnish start, it seemed to work.

Note: you can’t test by having varnish and apache going at the same time(if both listen to port 80…), so have to restart apache first (Won’t be listening on port 80) then start varnish to listen on port 80.

4. We tested varnish to see if it impacted our requests by using >>GET -Used http://cs.wellesley.edu/~cs110 to see what happens when we request a file. To check varnish stats, look at… >>varnishstats which will show the cache hits. When testing it on puma, we also used >>nmap puma to see what was on port 80.

So excellent! Varnish works. We’ll probably have another post soon about observed changes with varnish.

A few months ago, we talked about adding a robots.txt file so that webcrawlers won’t index archived courses, such as the course material for cs110 back in fall of 2012.

The robotstxt.org page is, unsurprisingly, the authority on creating a robots.txt file. You put the robots.txt file in the top level directory of the webserver, which on tempest is /var/www/html. When I looked, there was actually already a robots.txt page, so I just added a few lines like  “Disallow:/~cs110f12/”. Hopefully this will prevent robots from indexing those archived courses.

To create a signed certificate, we first ran genkey and created a file that I happened to call cs.wellesley.edu.1.csr.  I put that in /etc/pki/tls/certs.

When I uploaded it to the certificate company’s website, they complained that it didn’t meet the 2048 bit minimum, so I created a second as cs.wellesley.edu.2.csr

After they did their work, I got an email back that looked like this:

* Click the following link to download your SSL certificate (generally try to use a version that includes intermediates & root or your certificate may be rejected by some older clients)

Format(s) most suitable for your server software:
as X509 Certificate only, Base64 encoded: https://cert-manager.com/customer/InCommon/ssl?action=download&sslId=12345678&format=x509CO
as X509 Intermediates/root only, Base64 encoded: https://cert-manager.com/customer/InCommon/ssl?action=download&sslId=12345678&format=x509IO

And no further information.  Thanks to help from Andy Maroney, I now know that these files are the targets of two config variables in /etc/httpd/conf.d/ssl.conf.  There’s also a third variable that needs to be configured.

• The first one downloads as cs_wellesley_edu_cert.cer. I put it in /etc/pki/tls/certs and made it the value of SSLCertificateFile
• The second one downloads as cs_wellesley_edu_interm.cer. I put it in /etc/pki/tls/certs and made it the value of SSLCertificateChainFile
• The third variable is SSLCertificateKeyFile and the value of that is set to a file that was generated simultaneously with the original .csr file, and that is /etc/pki/tls/private/cs.wellesley.edu.key. (That file holds the private key and is readable only by root.)

Restart Apache and you’re done!

In the recent past, logwatch has been spitting out way, way too much info about activity on tempest. The output that the cs-sysadmin email receives is actually truncated. The main culprit is HTTP, which spits out every link that gets a 401, 403, and 404 error. On puma, we’ve turned off http messages completely, but this time we wanted to figure out a more refined way to filter messages.

I found the documentation for logwatch in /usr/share/doc. The default configuration files are found in the directory /usr/share/logwatch/default.conf/services for services such as http, sshd, pam_unix, etc. The default scripts are in /usr/share/logwatch/scripts/services. These scripts are all written in Perl.

To override the default configuration, the documentation recommends 2 ways. You can add in whatever overriding variables in the /etc/logwatch/conf directory in a file called logwatch.conf. Or you could copy the default from /usr/share/logwatch to the corresponding directory in /etc/logwatch, and modify the file in the /etc/logwatch folder.

You could set variables in the /etc/logwatch/conf directory. On puma, this is how we turned off the output for http in the logwatch.conf file, since the output was getting voluminous. However, the configuration files did not give us fine enough control over the output. I had to modify the scripts themselves.

To modify scripts, I copied the appropriate default script from /usr/share/logwatch/scripts/services over to /etc/logwatch/scripts/services, where I fiddled around with the output. For example for the http script, I changed it so that it prints out the total number of 404 errors tempest collects for the day, instead of every single URL.

Once you’ve made the change, you can just run logwatch and have it print to your terminal with the following command:

logwatch --print [--service=<servicename>]

In general, to configure logwatch to your liking, copy the original file from /usr/share/logwatch to its corresponding place in /etc/logwatch, and make your edits to the /etc/logwatch file. The output will be much more readable.

Sources: http://serverfault.com/questions/293226/linux-logwatch8-is-too-noisy-how-can-i-control-the-noise-level/293233#293233

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-logwatch-log-analyzer-and-reporter-on-a-vps