cp -al taking a long time

Our backups on Puma have been taking a long time, finishing late in the day, almost in time for the next backup.  The problem seemed to be in the cp -al step, not in the rsync step.  I investigated, looking at how long the cp -al on each directory took, using code like this:

for a in `ls -A $from`; do
now=`date +%T`
echo "$now cp -al $from/$a to $to/$a"
cp -al $from/$a $to/$a
done

and the result looked like:

16:38:06 cp -al 2014-03-26/alice to today/alice
16:38:06 cp -al 2014-03-26/anderson to today/anderson
16:39:37 cp -al 2014-03-26/apache-tomcat-5.5.26 to today/apache-tomcat-5.5.26
16:39:42 cp -al 2014-03-26/appinvstats to today/appinvstats
22:59:14 cp -al 2014-03-26/appinv-stats to today/appinv-stats
22:59:14 cp -al 2014-03-26/btjaden to today/btjaden
23:11:11 cp -al 2014-03-26/compbio to today/compbio
23:11:12 cp -al 2014-03-26/cs to today/cs
23:11:24 cp -al 2014-03-26/cs110f11 to today/cs110f11
...

It finished at about midnight (so, less than 8 hours total), but essentially all of that time was in the appinvstats directory.

Sure enough, some subdirectories of that account had a *lot* of inodes.  Here are some useful references:

http://unixetc.co.uk/2012/05/20/large-directory-causes-ls-to-hang/

http://www.olark.com/spw/2011/08/you-can-list-a-directory-with-8-million-files-but-not-with-ls/

http://www.pronego.com/helpdesk/knowledgebase.php?article=59

A count of the inodes:

ls -fR collectedStats | wc -l
5057712
ls -fR errorFiles | wc -l
4438339

So, about 1 million files/folders or inodes.

Turning these in to tarfiles would reduce these to two inodes.  There’s also a savings in space:

du -csh errorFiles errorFiles.tar
129G    errorFiles
106G    errorFiles.tar
du -csh collectedStats collectedStats.tar
11G     collectedStats
3.3G       collectedStats.tar

We’ll look at replacing these directories with the tar files.

 

Posted in Uncategorized | Leave a comment

Install Python2.7 w/ Tkinter

Rhys asked for us to install Tkinter for Python2.7.  [Remember, that Python2.6 is the default Python on Tempest because it’s what RHEL comes with, but most users are using Python 2.7.]  Here’s the current behavior:

[anderson@tempest ~] python2.6
Python 2.6.6 (r266:84292, Nov 21 2013, 10:50:32) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *
>>> 
[anderson@tempest ~] python2.7
Python 2.7 (r27:82500, Sep 20 2012, 17:09:01) 
[GCC 4.4.6 20120305 (Red Hat 4.4.6-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/lib-tk/Tkinter.py", line 39, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ImportError: No module named _tkinter
>>>

Which is obviously no good.  This seemed pretty straightforward, since the fact that it works for Python2.6 suggests that I have all the necessary .so files and such.  However, I couldn’t find any pre-compiled binaries for Python2.7 for RHEL6/CentOS6.  I tried compiling Python for myself, just like I did last August (https://blogs.wellesley.edu/cssysadmin/2012/08/29/python-2-7/  However, that resulted in exactly the same ImportError.

Here’s the best of the “how to compile Python” info pages that I found.  However, I tried one more time, but this time, I discovered a comment that explained that I have to install tk-devel.  That, however, is a piece of cake.

# yum -y install tk-devel

To avoid any issue with over-writing the existing Python2.7, which is installed in /usr/local/, I installed the new, improved, Python2.7 in /opt:

# wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tar.xz
# tar xf Python-2.7.6.tar.xz
# cd Python-2.7.6
# ./configure --prefix=/opt --enable-unicode=ucs4 --enable-shared LDFLAGS="-Wl,-rpath /opt/lib"
# make && make altinstall

The first time I tried that, it failed miserably:

checking build system type... x86_64-unknown-linux-gnu
 checking host system type... x86_64-unknown-linux-gnu
 checking for --enable-universalsdk... no
 checking for --with-universal-archs... 32-bit
 checking MACHDEP... linux2
 checking EXTRAPLATDIR...
 checking for --without-gcc... no
 checking for gcc... gcc
 checking whether the C compiler works... no
 configure: error: in `/var/tmp/install-python2.7/Python-2.7.6':
 configure: error: C compiler cannot create executables
 See `config.log' for more details

It turned out that the trouble was that there wasn’t a /opt/lib directory.  Doing a “mkdir /opt/lib” and re-doing the config command worked perfectly, as did the “make” and “make altinstall”

So, now we have:

Python 2.7.6 (default, Mar 24 2014, 13:29:51)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-4)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from Tkinter import *
>>>

But, we also need to make sure that the new Python2.7 has all the libraries that the old Python2.7 had.

I learned that Pip can tell you the libraries it has loaded:

# /usr/local/bin/pip-2.7 freeze
CouchDB==0.9
MySQL-python==1.2.4b5
Orange==2.5a4
anyjson==0.3.3
beautifulsoup4==4.3.2
biopython==1.63
couchdbkit==0.6.5
dumptruck==0.1.5
gensim==0.8.6
numpy=1.6.2
scikit-learn=0.12.1
...

Compare to:

# /opt/bin/pip2.7 freeze
wsgiref==0.1.2

We can dump the former into a file and use it to set up the new Python:

# umask 0002
# /opt/bin/pip2.7 freeze
wsgiref==0.1.2
# rm old-py2.7-modules
# /usr/local/bin/pip-2.7 freeze > old-py2.7-modules
# /opt/bin/pip2.7 install -r old-py2.7-modules
Downloading/unpacking CouchDB==0.9 (from -r old-py2.7-modules (line 1))
Downloading CouchDB-0.9.tar.gz (55kB): 55kB downloaded
Running setup.py (path:/tmp/pip_build_root/CouchDB/setup.py) egg_info for package CouchDB

Downloading/unpacking MySQL-python==1.2.4b5 (from -r old-py2.7-modules (line 2))
Downloading MySQL-python-1.2.4b5.tar.gz (82kB): 82kB downloaded
...

Well, that *mostly* worked.  We ran into trouble after a while:

Downloading/unpacking scikit-learn==0.12.1 (from -r old-py2.7-modules (line 27))
Downloading scikit-learn-0.12.1.tar.gz (3.0MB): 3.0MB downloaded
Running setup.py (path:/tmp/pip_build_root/scikit-learn/setup.py) egg_info for package scikit-learn
Partial import of sklearn during the build process.
Traceback (most recent call last):
File "<string>", line 17, in <module>
File "/tmp/pip_build_root/scikit-learn/setup.py", line 36, in <module>
from numpy.distutils.core import setup
ImportError: No module named numpy.distutils.core
Complete output from command python setup.py egg_info:
Partial import of sklearn during the build process.

Traceback (most recent call last):

File "<string>", line 17, in <module>

File "/tmp/pip_build_root/scikit-learn/setup.py", line 36, in <module>

from numpy.distutils.core import setup

ImportError: No module named numpy.distutils.core

----------------------------------------
Cleaning up...
Command python setup.py egg_info failed with error code 1 in /tmp/pip_build_root/scikit-learn
Traceback (most recent call last):
File "/opt/bin/pip2.7", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip2.7')()
File "/opt/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/opt/lib/python2.7/site-packages/pip-1.5.4-py2.7.egg/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 70: ordinal not in range(128)

How much got installed?  Let’s see:

# /opt/bin/pip2.7 freeze
wsgiref==0.1.2

Well, that stinks! Is it all or nothing?  The error output suggests that there may be a dependency on numpy, so let’s install that first:

# /opt/bin/pip2.7 install numpy
# /opt/bin/pip2.7 freeze
numpy==1.8.0
wsgiref==0.1.2

Okay, and I notice that the version is newer than the numpy in the old Python2.7. So, this will be annoying and tedious, but I guess I may have to install things by hand. One more try:

# /usr/local/bin/pip-2.7 freeze | cut -d= -f1 | grep -v numpy > old-py2.7-module-names
# /opt/bin/pip2.7 install -r old-py2.7-module-names

Wait a *long* time, and it succeeds!

# diff old-py2.7-module-names new-py2.7-module-names
15a16
> numpy
26a28
> scipy

Woo hoo!

Note that pip can be used to upgrade Python packages.

Posted in Uncategorized | Leave a comment

March 15? Power outage…and troubles.

Apparently early on Saturday morning there was a power outage for a flicker or something. I didn’t notice this at all, but Karishma did. Flash forward to today around 11:30, when we get this email from Celia, “I just brought my use down to 65092 MB, but I cannot access the linux machines. Any chance I could be given access again?” Scott responds asking if she is using the right password and still having trouble. She responds that she is using the right password and still having trouble. I arrive at the Science Center at ~2:30 pm.

Celia is in the Microfocus and I ask her if she’s still having trouble and what she’s tried. She is still having trouble and points to about 5 different client machines that she’s tried logging into. I ask if she can ssh into tempest. She successfully ssh’s into tempest. I ask her if she can ssh into puma. She cannot. Then I attempt logging into jay. It fails with edavis5, but I want to make sure Celia is able to do her work. I login to wren as luser, try to ssh into tempest, it fails (doesn’t know what tempest is). I become root@wren and ifup eth0. Then Celia is able to ssh into tempest and do her work.

In this debugging process, I run ah-broadcast ping (which I now realize is a bad thing to do with an ah-broadcast, ah-broadcast hostname would be much better, but this gave me the message I needed). None of the clients were up. This was a larger problem than just Celia and I not being able to login.

I logged in as luser to jay, brought eth0 back up, mounted all and rebooted. I was now able to login as edavis5. For finch, I did a hard reboot (pressing the button and then pressing it again). This didn’t work. I was then not able to login as edavis5, so I figured that the ifup eth0 and mount -a was necessary.

Unfortunately, I also have a life. So I had a meeting and church and was able to get back to the clients at 5 pm. Lulu joined me around 5:15. By the time Lulu had joined, I had brought eth0 up and rebooted on 5 machines, so there were only 3 left in the microfocus for her. But! When I then returned to Finch to ifup eth0 on Finch…I got “Trying to connect…is the cord connected?” which is a very nice error message if you ask me. I was expecting it to just fail and I would flail around to see what was wrong. Turns out that ethernet port was broken. Because I tried connecting my laptop to the internet using the other side connected to the port, I tried using a different cord but neither worked so it must be the port. I then proceeded to try all ports that weren’t already connected to a client that was working. There were not very many of these, but none of them worked either. This seemed crazy. All of the ethernet ports should work. Just like all of the power outlets should work…but they don’t.

Then Lulu told me that she had rebooted cardinal and swallow and irwin but none of them worked. I already knew that irwin wasn’t working because I had given a sticky note to it and had tried logging in on Wednesday but that didn’t work. But swallow I was also on on Wednesday and it should’ve worked with the reboot.

So I tried logging into them, and it worked for me. So we decided that there was another LDAP style problem with the hye account not working but my edavis5 account working. Why would this happen?? 🙁

Then Lulu rebooted all of the clients not in the microfocus. Thank you! While I tried to figure out how to tell the difference between ones that knew hye and ones that didn’t. I couldn’t figure anything else out, got frustrated and needed to eat dinner so I left.

So now the outstanding issues:

-why can edavis5 login when hye can’t?

-why is there an ethernet drop down?

-Then something that I noticed when trying to look into Celia’s account specifically, was that her entry in tempest’s /etc/passwd was different from her entry in puma’s /etc/passwd. Specifically in the type of shell– scponly v bash. Why would this be like that? Should we make sure those are the same?

-Also running ah-broadcast after Lulu brought some of the machines in 173 back up, we still need to copy the ssh keys for orangutan and tamarin over to tempest, because when I was ah-broadcasting it asked for root@orangutan and root@tamarin’s passwords. This shouldn’t be hard and I have a blog post about it from before.

Posted in Uncategorized | Leave a comment

Install Tomcat 6 on RHEL 6

I’m following the instructions here: http://newpush.com/how-to-install-tomcat-6-on-rhel-6-or-centos-6/

First up, install yum-priorities. That allows us to prioritize some repos over others. So, for example, we can make sure the RHEL repos take priority over EPEL, I guess.  He also installs two repo files (enabling those repos) and a jpackage-utils (but not the jpackage repo).  Hmm.

I’ve typically gotten the effect of yum-priorities by leaving EPEL disabled, and enabling it when desired by doing:

yum –enablerepo=epel search foo

I’m going to skip that, then, and just search for the three packages he specifies:

[root@tempest ~] yum –enablerepo=epel,rpmforge search tomcat6 tomcat6-webapps tomcat6-admin-webapps
rhel-6-server-cf-tools-1-rpms                                                                | 2.8 kB     00:00
rhel-6-server-rhev-agent-rpms                                                                | 3.1 kB     00:00
rhel-6-server-rpms                                                                           | 3.7 kB     00:00
rpmforge                                                                                     | 1.9 kB     00:00
rpmforge/primary_db                                                                          | 2.7 MB     00:05
=============================================== N/S Matched: tomcat6 ===============================================
glite-security-trustmanager-tomcat6.noarch : Java trustmanager interface supporting a GSI grid name space
tomcat6.noarch : Apache Servlet/JSP Engine, RI for Servlet 2.5/JSP 2.1 API
tomcat6-el-2.1-api.noarch : Expression Language v1.0 API
tomcat6-jsp-2.1-api.noarch : Apache Tomcat JSP API implementation classes
tomcat6-lib.noarch : Libraries needed to run the Tomcat Web container
tomcat6-servlet-2.5-api.noarch : Apache Tomcat Servlet API implementation classes

Name and summary matches mostly, use “search all” for everything.
Warning: No matches found for: tomcat6-webapps
Warning: No matches found for: tomcat6-admin-webapps
[root@tempest ~]

Hmm.  Now I’m puzzled. The two missing packages aren’t in EPEL or in RPMFORGE, yet those are the only repositories he enables.  Very strange.  Let’s take a leap and just slavishly do as he tells us:

[root@tempest ~] yum install yum-priorities
No package yum-priorities available.
Error: Nothing to do
[root@tempest ~]

I did some searching and it turns out that on CentOS 6, It bothers me that the instructions fail at the first step.  On the other hand, this page: http://wiki.centos.org/PackageManagement/Yum/Priorities says that it should exist, which I find very bothersome:

[root@tempest ~] yum –enablerepo=epel search yum
Loaded plugins: downloadonly, product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-6-server-cf-tools-1-rpms                                                                | 2.8 kB     00:00
rhel-6-server-rhev-agent-rpms                                                                | 3.1 kB     00:00
rhel-6-server-rpms                                                                           | 3.7 kB     00:00
================================================= N/S Matched: yum =================================================
anaconda-yum-plugins.noarch : Installation-related yum plugins
yum-arch.noarch : Extract headers from rpm in a old yum repository
yum-metadata-parser.x86_64 : A fast metadata parser for yum
yum-presto.noarch : Presto plugin for yum
yum-rhn-plugin.noarch : RHN support for yum
yumex.noarch : Yum Extender graphical package management tool
PackageKit-yum.x86_64 : PackageKit YUM backend
PackageKit-yum-plugin.x86_64 : Tell PackageKit to check for updates when yum exits
fusioninventory-agent-yum-plugin.noarch : Ask FusionInventory agent to send an inventory when yum exits
kabi-yum-plugins.noarch : The Red Hat Enterprise Linux kernel ABI yum plugin
repoview.noarch : Creates a set of static HTML pages in a yum repository
yum-dellsysid.x86_64 : YUM plugin to retrieve the Dell System ID
yum-plugin-aliases.noarch : Yum plugin to enable aliases filters
yum-plugin-changelog.noarch : Yum plugin for viewing package changelogs before/after updating
yum-plugin-downloadonly.noarch : Yum plugin to add downloadonly command option
yum-plugin-protect-packages.noarch : Yum plugin to prevents Yum from removing itself and other protected packages
yum-plugin-security.noarch : Yum plugin to enable security filters
yum-plugin-tmprepo.noarch : Yum plugin to add temporary repositories
yum-plugin-verify.noarch : Yum plugin to add verify command, and options
yum-plugin-versionlock.noarch : Yum plugin to lock specified packages from being updated
yum-utils.noarch : Utilities based around the yum package manager
createrepo_c.x86_64 : Creates a common metadata repository
grinder.noarch : A tool for synchronizing content from yum repositories
mash.noarch : Koji buildsystem to yum repository converter
mrepo.noarch : A tool to set up a yum/apt mirror from various sources
remi-release.noarch : YUM configuration for remi repository
yum.noarch : RPM installer/updater

Name and summary matches only, use “search all” for everything

 

I’m going to skip yum-priorities and go on to the rest of step one:

[root@tempest ~] rpm -Uvh http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
Retrieving http://apt.sw.be/redhat/el6/en/x86_64/rpmforge/RPMS/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
Preparing…                ########################################### [100%]
package rpmforge-release-0.5.2-2.el6.rf.x86_64 is already installed
[root@tempest ~] rpm -Uvh http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
Retrieving http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm
curl: (6) Couldn’t resolve host ‘download.fedora.redhat.com’
error: skipping http://download.fedora.redhat.com/pub/epel/6/x86_64/epel-release-6-5.noarch.rpm – transfer failed
[root@tempest ~] rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Retrieving http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
Preparing…                ########################################### [100%]
package epel-release-6-8.noarch is already installed
[root@tempest ~]

So, he has a package name wrong (the correction was posted as a comment to his page) and we already have both repos installed.  So if installing jpackage-utils doesn’t allow us to find the tomcat6-webapps and tomcat6-manager-webapps, I’ll be very unhappy.

[root@tempest ~] rpm -Uvh http://mirrors.dotsrc.org/jpackage/6.0/generic/free/RPMS/jpackage-utils-5.0.0-7.jpp6.noarch.rpm
Retrieving http://mirrors.dotsrc.org/jpackage/6.0/generic/free/RPMS/jpackage-utils-5.0.0-7.jpp6.noarch.rpm
warning: /var/tmp/rpm-tmp.htRz2i: Header V3 DSA/SHA1 Signature, key ID c431416d: NOKEY
Preparing…                ########################################### [100%]
1:jpackage-utils         warning: /etc/maven/maven2-depmap.xml created as /etc/maven/maven2-depmap.xml.rpmnew
########################################### [100%]
[root@tempest ~]

Okay, at least that added something to our system.  Let’s see how the search goes:

yum –enablerepo=epel,rpmforge search tomcat6 tomcat6-webapps tomcat6-admin-webapps
Loaded plugins: downloadonly, product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-6-server-cf-tools-1-rpms                                                                | 2.8 kB     00:00
rhel-6-server-rhev-agent-rpms                                                                | 3.1 kB     00:00
rhel-6-server-rpms                                                                           | 3.7 kB     00:00
=============================================== N/S Matched: tomcat6 ===============================================
glite-security-trustmanager-tomcat6.noarch : Java trustmanager interface supporting a GSI grid name space
tomcat6.noarch : Apache Servlet/JSP Engine, RI for Servlet 2.5/JSP 2.1 API
tomcat6-el-2.1-api.noarch : Expression Language v1.0 API
tomcat6-jsp-2.1-api.noarch : Apache Tomcat JSP API implementation classes
tomcat6-lib.noarch : Libraries needed to run the Tomcat Web container
tomcat6-servlet-2.5-api.noarch : Apache Tomcat Servlet API implementation classes

Name and summary matches mostly, use “search all” for everything.
Warning: No matches found for: tomcat6-webapps
Warning: No matches found for: tomcat6-admin-webapps
[root@tempest yum.repos.d]

Nope, utter failure.

Hang on, (more googling and, more importantly, logging into my old redhat account and searching the knowledge base finds this page:

https://access.redhat.com/site/solutions/56374

Which promises to exactly answer my question, but since we are now until LTS and I don’t have the password, I can’t read the answer.  So, I asked Andrew Maroney to look at it for me.  Turns out that the two -webapps packages are in RHEL optional, so we have to enable that repo:

[root@tempest yum.repos.d] yum-config-manager –enable rhel-6-server-optional-rpms

and now:

[root@tempest yum.repos.d] yum search tomcat6-webapps tomcat6-admin-webapps
Loaded plugins: downloadonly, product-id, refresh-packagekit, security, subscription-manager
This system is receiving updates from Red Hat Subscription Management.
rhel-6-server-cf-tools-1-rpms                                                                | 2.8 kB     00:00
rhel-6-server-optional-rpms                                                                  | 3.5 kB     00:00
rhel-6-server-rhev-agent-rpms                                                                | 3.1 kB     00:00
rhel-6-server-rpms                                                                           | 3.7 kB     00:00
=========================================== N/S Matched: tomcat6-webapps ===========================================
tomcat6-webapps.noarch : The ROOT and examples web applications for Apache Tomcat

======================================== N/S Matched: tomcat6-admin-webapps ========================================
tomcat6-admin-webapps.noarch : The host-manager and manager web applications for Apache Tomcat

Name and summary matches mostly, use “search all” for everything.
[root@tempest yum.repos.d]

Yay!  So I don’t need EPEL or Rpmforge at all:

[root@tempest yum.repos.d] yum install tomcat6 tomcat6-webapps tomcat6-admin-webapps
Setting up Install Process
Resolving Dependencies
–> Running transaction check
—> Package tomcat6-admin-webapps.noarch 0:6.0.24-62.el6 will be installed
—> Package tomcat6-webapps.noarch 0:6.0.24-62.el6 will be installed
–> Processing Dependency: jakarta-taglibs-standard >= 1.1 for package: tomcat6-webapps-6.0.24-62.el6.noarch
–> Running transaction check
—> Package jakarta-taglibs-standard.noarch 0:1.1.1-11.4.el6 will be installed
—> Package tomcat6.noarch 0:6.0.24-62.el6 will be installed
10gen/filelists                                                                              | 5.3 kB     00:00
rhel-6-server-cf-tools-1-rpms/filelists_db                                                   |  11 kB     00:00
rhel-6-server-optional-rpms/filelists_db                                                     | 4.5 MB     00:01
rhel-6-server-rhev-agent-rpms/filelists_db                                                   |  12 kB     00:00
rhel-6-server-rpms/filelists_db                                                              |  14 MB     00:07
–> Processing Dependency: tomcat6-lib = 6.0.24-62.el6 for package: tomcat6-6.0.24-62.el6.noarch
–> Processing Dependency: jakarta-commons-pool for package: tomcat6-6.0.24-62.el6.noarch
–> Processing Dependency: jakarta-commons-dbcp for package: tomcat6-6.0.24-62.el6.noarch
–> Processing Dependency: jakarta-commons-daemon for package: tomcat6-6.0.24-62.el6.noarch
–> Processing Dependency: jakarta-commons-collections for package: tomcat6-6.0.24-62.el6.noarch
–> Running transaction check
—> Package jakarta-commons-collections.noarch 0:3.2.1-3.4.el6 will be installed
—> Package jakarta-commons-daemon.x86_64 1:1.0.1-8.9.el6 will be installed
—> Package jakarta-commons-dbcp.noarch 0:1.2.1-13.8.el6 will be installed
—> Package jakarta-commons-pool.x86_64 0:1.3-12.7.el6 will be installed
—> Package tomcat6-lib.noarch 0:6.0.24-62.el6 will be installed
–> Processing Dependency: tomcat6-servlet-2.5-api = 6.0.24-62.el6 for package: tomcat6-lib-6.0.24-62.el6.noarch
–> Processing Dependency: tomcat6-jsp-2.1-api = 6.0.24-62.el6 for package: tomcat6-lib-6.0.24-62.el6.noarch
–> Processing Dependency: tomcat6-el-2.1-api = 6.0.24-62.el6 for package: tomcat6-lib-6.0.24-62.el6.noarch
–> Running transaction check
—> Package tomcat6-el-2.1-api.noarch 0:6.0.24-62.el6 will be installed
—> Package tomcat6-jsp-2.1-api.noarch 0:6.0.24-62.el6 will be installed
—> Package tomcat6-servlet-2.5-api.noarch 0:6.0.24-62.el6 will be installed
–> Finished Dependency Resolution

Dependencies Resolved

Installed:
tomcat6-admin-webapps.noarch 0:6.0.24-62.el6                tomcat6-webapps.noarch 0:6.0.24-62.el6

Dependency Installed:
jakarta-commons-collections.noarch 0:3.2.1-3.4.el6         jakarta-commons-daemon.x86_64 1:1.0.1-8.9.el6
jakarta-commons-dbcp.noarch 0:1.2.1-13.8.el6               jakarta-commons-pool.x86_64 0:1.3-12.7.el6
jakarta-taglibs-standard.noarch 0:1.1.1-11.4.el6           tomcat6.noarch 0:6.0.24-62.el6
tomcat6-el-2.1-api.noarch 0:6.0.24-62.el6                  tomcat6-jsp-2.1-api.noarch 0:6.0.24-62.el6
tomcat6-lib.noarch 0:6.0.24-62.el6                         tomcat6-servlet-2.5-api.noarch 0:6.0.24-62.el6

Complete!
[root@tempest yum.repos.d]

Whew!  Okay, let’s try starting tomcat:

[root@tempest yum.repos.d] service tomcat6 start
Starting tomcat6:                                          [  OK  ]

That’s nice.  If we browse to the Tomcat home page, we see the logo; yay!

Now we need to start configuring it.  We should look at the kinds of things we did with Tomcat5 on Puma; hopefully the changes will be similar.

We should also look at these:

http://tomcat.apache.org/tomcat-6.0-doc/setup.html

http://tomcat.apache.org/tomcat-6.0-doc/manager-howto.html

 

 

 

Posted in Uncategorized | Leave a comment

cs111 tutor account

Sohie wants the cs111 tutors to be able to read the contents of /home/cs111/drop so that they can grade dropped assignments, but not to be able to write to other cs111 files and directories. Currently the drop folder is owned by cs111, so there’s no choice but to give the tutors the cs111 password and allow them to write other files.

My idea is to create a cs111tutor account, make its home directory be the cs111/drop folder. It will then have a separate password. If we put the drop folder in the cs111tutor group, the cs111 tutors will also be able to read the files, but won’t own any of them, so they won’t even be able to modify the permissions. The drop folders can still be owned by cs111, but gid of cs111tutor.

One concern is that /home/cs111/drop is a symlink, not a real directory, but that may not matter.

So:

useradd -c "CS111 tutor has access to drop directory" -d /home/cs111/drop -M -U cs111tutor
[root@tempest ~] /usr/bin/passwd cs111tutor
 Changing password for user cs111tutor.
 New password:
 Retype new password:
 passwd: all authentication tokens updated successfully.
 [root@tempest ~] cd ~cs111tutor
[root@tempest drop] ls -ld .
 drwxr-xr-x. 2 cs111 cs111 4096 Feb  2 13:44 .

Now, all the elements of the “drop” directory are symlinks, and we can modify the group of a symlink, but that’s not what we want to do. Instead, we want to modify the group of the thing that the simlink is pointing to:

[root@tempest drop] for a in *; do chgrp -R cs111tutor /students/$a/cs111/; done
[root@tempest drop] ls[root@tempest drop] su - cs111tutor
-bash-4.1$ pwd
/home/cs111/drop/
-bash-4.1$ cd egstu/
-bash-4.1$ ls
exam2  ps01  ps02  ps03  ps04  ps05  ps06  ps07  ps08  ps09  ps10  ps11
-bash-4.1$ ls -l
total 48
dr-xr-s---. 5 egstu cs111tutor 4096 Nov 12 07:20 exam2
drwxr-s---. 4 egstu cs111tutor 4096 Sep  9 17:22 ps01
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps02
drwxr-s---. 3 egstu cs111tutor 4096 Sep 18 18:12 ps03
drwxr-s---. 4 egstu cs111tutor 4096 Sep 30 22:33 ps04
drwxr-s---. 3 egstu cs111tutor 4096 Oct  6 22:33 ps05
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps06
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps07
drwxr-s---. 3 egstu cs111tutor 4096 Oct 30 17:53 ps08
drwxr-s---. 3 egstu cs111tutor 4096 Nov 17 15:37 ps09
drwxr-s---. 4 egstu cs111tutor 4096 Nov 24 11:47 ps10
drwxr-s---. 3 egstu cs111tutor 4096 Dec  4 17:43 ps11
-bash-4.1$logout

We also need to add cs111tutor as a supplementary group for the cs111 user and for Sohie, too:

[root@tempest drop] grep cs111tutor /etc/group
cs111tutor:x:6163:cs111,slee
[root@tempest drop]

If that works, we should be able to read students’ submissions as cs111:

[root@tempest drop] su - cs111
[cs111@tempest ~] cd ~/drop
[cs111@tempest drop] id
uid=709(cs111) gid=709(cs111) groups=709(cs111),3740(cs111web),6163(cs111tutor) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
[cs111@tempest drop] cd egstu/
[cs111@tempest egstu] ls -l
total 48
dr-xr-s---. 3 egstu cs111tutor 4096 Nov 11 21:50 exam2
drwxr-s---. 3 egstu cs111tutor 4096 Sep  8 19:27 ps01
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps02
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps03
drwxr-s---. 3 egstu cs111tutor 4096 Sep 30 21:19 ps04
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps05
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps06
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps07
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps08
drwxr-s---. 3 egstu cs111tutor 4096 Nov 18 20:12 ps09
drwxr-s---. 3 egstu cs111tutor 4096 Nov 25 20:54 ps10
drwxr-s---. 2 egstu cs111tutor 4096 Sep  3 21:00 ps11
[cs111@tempest egstu] cd ps01
[cs111@tempest ps01] ls -l
total 4
drwxr-sr-x. 2 egstu cs111tutor 4096 Sep  8 19:27 ps01_programs
[cs111@tempest ps01]

It works!

Posted in Uncategorized | Leave a comment

make semester dir (page of links)

Faculty member wants to have a page like the CS 110 students page, a list of hyperlinks to student accounts, with the student’s real name as well.  Maybe for multiple sections. Here’s how I did it for CS 230 in spring 2014.

Login to Banner, navigate to the course list for cs230-01, click on the “Download Class List to Excel” link. The description is a lie: the file is a “tab-separated values” text file (TSV).  It’ll be saved to ~/Downloads/class_list.xls (on your Mac). So:

cd ~/Downloads
mv class_list.xls > cs230-01.tsv
# download other section
mv class_list.xls > cs230-02.tsv
cat cs230-0*.tsv > cs230.tsv
scp cs230.tsv anderson@cs:/tmp
ssh anderson@cs
chmod a+r /tmp/cs230.tsv
su - cs230
cd public_html
cp /tmp/cs230.tsv .
mkdir studentsS14
cp studentsS13/template.html studentsS14  # happened to have one around from last year
~cs110/bin/make-semester-dir-general.pl studentsS14 cs230.tsv

Finally, hand-edit the file to get exactly what you want. The result looks like:

http://cs.wellesley.edu/~cs230/studentsS14/accounts.html

This still needs some work to be really right, and could use some additional automation, but it’s getting there.

Posted in Uncategorized | Leave a comment

quota_near updates and installs

I updated quota_near.py because I moved the email template from a string within the script to a file separate from this, and it was giving an error of not knowing where that file was (since its pathname was relative). Originally I knew that quota_near wasn’t running correctly when it was running from /etc/cron.daily but I couldn’t figure out exactly why that was, because if I ran it with “at” it would work fine. I also knew from the logs that only one user was being reported as being near quota each day, which I knew was false. Yesterday I tried the simple answer of creating a symlink from the /etc/cron.daily directory to the email template. But that didn’t work for last night. So today I moved the string of email template from a separate file into the python file. Hopefully tonight/tomorrow morning, the emails will go out to quota_near. I tested this with my own account, changing around my quota to make myself receive both emails. But now I know where to look for the error outputs from cron jobs, in the email about cron jobs.

I started Karina and Lulu with updates/installs to CentOS in 173. Of the 7 machines in there, we got through most of the installation on one, and through part 1 on 3 others. One of them, baboon, didn’t have enough free disk space on it, and the partition tools that we sometimes used within the CentOS boot disk didn’t help. So I think this one will need some Gpartd disk separately. And Lemur is the testing Fedora machine. I’m hoping that Karina and Lulu can work independently on bringing the rest of them to the same level as tamarin.

Posted in Uncategorized | Tagged | Leave a comment

Remove/stop Anacron

For unknown reasons, we installed anacron on Tempest, and so many of our /etc/cron.daily scripts have been running multiple times.  My best guess, based on the timestamp on /etc/cron.hourly/0anacron is that we installed anacron on September 12th.

There’s no obvious way to turn anacron off (it’s not in chkconfig, for example), so it seems to me the best thing is to delete the file in /etc/cron.hourly. I’ve done that.

 

Posted in Uncategorized | Leave a comment

renew https server certificate

We use a self-signed https certificate for our server, and they expire every year or so. When they do, and you visit a page using https, you see something like:

The site's security certificate is not trusted!
The site’s security certificate is not trusted!

So, we have to renew or re-create the certificate. I think I succeeded in writing a script to do this, namely “server-certificate” (poor name), which is a wrapper for genkey.wc, which is a wellesley modification of an older genkey script from RedHat. But, apparently, that’s obsolete, and the genkey.wc script doesn’t run.

I found the following: https://access.redhat.com/site/documentation/en-US/Red_Hat_Certificate_System/8.0/html/Admin_Guide/Renewing_Certificates.html. It seems to be on the right topic, but it’s incomprehensible to me. That page describes a choice between “re-generate” and “re-create,” and I think we want “re-generate.”

Ack, this makes no sense. I googled for various things like “RHEL 6 renew https certificate” and got nowhere.  Eventually, I stumbled on http://superuser.com/questions/622434/can-self-signed-ssl-certificate-be-renewed-how which explained that you can’t renew a self-signed certificate, so I just need to create a new one, and I might as well give it a long expiration time.  That may be easier.

Boy, it’s hard to find RHEL 6 documentation. Eventually, I found the following: https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html-single/Deployment_Guide/index.html#s2-apache-mod_ssl which seems to be trying to explain how to create SSL certificates for Apache.

The first substantive step I did was to install crypto-utils:

yum install crypto-utils

Aha! That installs the “genkey” command! Oh, but they’ve really changed the script:

[root@tempest tls] diff -bB /usr/bin/genkey /root/bin/genkey.wc | wc
 1008 4500 31119
[root@tempest tls]

Oh well. Let’s just follow the instructions and not try to script this, yet.

genkey cs.wellesley.edu

That starts a GUI that prompts for the various pieces of information:

1. It informs me that the key  will be stored in

/etc/pki/tls/private/cs.wellesley.edu

and the certificate will be stored in

/etc/pki/tls/certs/cs.wellesley.edu.crt

2. I chose the default 1024-bit key size. It then starts generating the keys, but it needs to collect some randomness, and, wow, that takes a long time. I think it’s been 5 minutes or more, and it’s only 75% done.

3. I said “no” to whether I want to send a certificate request (CSR) to a certificate authority (CA). That means I’ll get a self-signed certificate.

4. I skipped having a passphrase, to make it easier for apache to start up at boot time (or whatever) without having to give a passphrase.

5. Finally, I get to put in the data about our server:

Country Name: US
State or Province: Massachusetts
Locality: Wellesley
Organization Name: Wellesley College
Organizational Unit: Computer Science Department
Common Name:  cs.wellesley.edu

That finishes the job and the program exited.

Next, I restarted the web server:

apachectl graceful

and re-tried the https connection. Hmm. No luck.

I ran /etc/cron.daily/certwatch and it generated an email saying that the certificate had expired and explicitly said the certificate was in /etc/pki/tls/certs/localhost.crt. So if I just alias (via ln) the localhost files to the cs.wellesley.edu files, that should work:

cd /etc/pki/tls/certs/
[root@tempest certs] pwd
/etc/pki/tls/certs
[root@tempest certs] mv localhost.crt localhost.crt.old
[root@tempest certs] ln cs.wellesley.edu.crt localhost.crt
[root@tempest certs] cd ../private/
[root@tempest private] ls -l *.key
-r--------. 1 root root 937 Jan 26 18:44 cs.wellesley.edu.key
-rw-------. 1 root root 887 Aug 15 2012 localhost.key
[root@tempest private] mv localhost.key localhost.key.old
[root@tempest private] ln cs.wellesley.edu.key localhost.key
[root@tempest private] apachectl graceful
[root@tempest private]

And re-try the web connection…. In Chrome, I still get all kinds of warnings, with no good way to say “add exception” but it *is* using the new certificate. Firefox is much better, allowing me to add an exception. However, the new certificate expires in one month!  (2/26/2014).  What the heck?!

More googling and frustration. Aha, I found this excellent post where all the steps seem to make a lot of sense: http://stevejenkins.com/blog/2010/08/renewing-a-self-signed-ssl-certificate-on-fedoracentos/ 

Furthermore, I found a nice script that fits well with that blog post, called make-dummy-cert, which is in the openssl package.

So, let’s first follow step 1 of Steve’s blog post:

[root@tempest certs] grep SSLCertificate /etc/httpd/conf.d/00_ssl.conf
# Point SSLCertificateFile at a PEM encoded certificate. If
SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key
# Point SSLCertificateChainFile at a file containing the
# the referenced file can be the same as SSLCertificateFile
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt
[root@tempest certs]

Okay, so that’s why I have to use “localhost” and not cs.wellesley.edu; fine. Let’s check the permissions, as Steve suggests:

[root@tempest certs] ls -l /etc/pki/tls/certs/localhost.crt /etc/pki/tls/private/localhost.key
-rw-rw----. 2 root root 995 Jan 26 18:44 /etc/pki/tls/certs/localhost.crt
-r--------. 2 root root 937 Jan 26 18:44 /etc/pki/tls/private/localhost.key
[root@tempest certs]

Okay, I can tighten those a bit.

Now, I’m going to modify step three, to put a wrapper around the openssl command, so that I don’t have to retype all that stuff each time. I based this wrapper on /etc/pki/tls/certs/make-dummy-cert.  I’m going to copy that to /home/sysadmin/fixes and put the derivative script, make-wellesley-cert in the same directory.

I made some modifications to that script, reducing the number of bits to 1024 (per RedHat’s recommendations) and hard-coding the site info and filename locations. Here’s the finished script:

#!/bin/sh
umask 077

answers() {
 echo US
 echo Massachusetts
 echo Wellesley
 echo Wellesley College
 echo Computer Science Department
 echo cs.wellesley.edu
 echo cs-sysadmin@wellesley.edu
}

name="localhost"
key="/etc/pki/tls/private/$name.key"
crt="/etc/pki/tls/certs/$name.crt"
answers | /usr/bin/openssl req -newkey rsa:1024 -keyout $key -nodes -x509 -days 365 -out $crt 2> /dev/null

ls -l $key $crt
apachectl graceful

Okay, that seems to have worked. Woo-hoo! Thank you, Steve Jenkins!

Posted in Uncategorized | Leave a comment

Purchase and configure NAS

We ran out of space on our old NAS, the one bought by Tyler, and Brian asked us to help purchase and configure a new one. After considering several alternatives and consulting several people in LTS, we settled on the following:

QNAP TurboNAS TS-569L and 5 of the following:

Seagate NAS HDD model ST4000VN000

I opened the NAS and installed all the hard drives, moved the NAS into 121A and powered it on. Then I took their installation CD and inserted it into my Mac and mounted the QFinderMac_1.0.13_Build0607.dmg file. I dragged the QFinder application into my Applications folder, as instructed.

I then launched the QFinder application, which found a NAS on the network, so that’s presumably ours. It called it NASDB5EB3 and popped up a window to ask whether I want to configure it.

I noticed in the other window, which I can’t select as long as the popup window is here, that there is info about my new NAS, including its IP address (0.0.0.0), hardware address 00-08-9B-DB-5E-B3 and its firmware status, which says that there is an update available.

Later I could click on details of the NAS; here’s the screenshot:

Screen shot 2013-12-19 at 12.27.16 PM

I clicked yes in the popup window, which tells me I’ll be re-directed to the web browser. Unsurprisingly, it says it can’t connect to 0.0.0.0:8080.  So, this is a pretty idiotic interface.  Or, it may be that DHCP isn’t working because of Cisco CleanAccess?  Time to check with Tim and Don.

Okay, eventually that worked, and it has a valid Wellesley IP address, and the web browser is able to talk to it.  It offered me a choice of “automatic” versus “manual” installation, and I chose the latter even though I intend to choose the default option at most step.

Step 1 is choosing name for the NAS and an administrator password. Default name is NASDB5EB3.  Admin account is “admin.” I used the same password as Tempest. Also an email address; I used cs-sysadmin. Oh, but I’m not allowed to! It says “Invalid Email Format.”  Stupid. I used my Wellesley email address.

Step 2 is date/time.  Chose Eastern Time and made it sync with ntp.wellesley.edu

Step 3 is network. Default is to get IP address automatically, and I kept that.

Step 4 is services. Default os is Mac, and it says multiple are allowed. I’m assuming that these are for client machines, since under Linux it mentions NFS, as well as “Web File Manager” (whatever that is) and FTP.  I enabled Linux only.

Also in this step is other applications.  I disabled all of those. We don’t need it to be an iTunes server, among other things.

Step 5 is disk. Default is RAID 5 and an Ext4 filesystem. That’s exactly what I want. (Another option is RAID 6.) It calculates that this will give us 14,904.08 GB of storage capacity.

It recognized the drives as Seagate ST4000VN000-1H41SC43.  There are also checkboxes to choose some drives as “hot spares.” There are also checkboxes for encrypted disk volume and bad block scan.  I decided to check the latter; we might as well find out if there are any bad blocks on these drives, though it may run overnight to do so! They recommend only doing this if the installation is unsuccessful, so maybe that’s reasonable.

That’s it! Here’s a screen shot of the summary:

Screen shot 2013-12-19 at 4.44.34 PMThen a final few clicks on “apply” and “confirm,” and off we go!

It gives us a progress bar. Interestingly, it’s taking measurable time to even do step 1, of changing the admin password.  But, the whole process didn’t take too long, maybe 10-15 minutes.

Logging in via the web interface is frustrating: there’s nothing to click on, nothing active, nothing to read. A complete dead end.

I can ssh as root to the machine. That’s cool.

Oh, the web interface is now working. Dunno why it wasn’t earlier.

It’s a pretty nice interface. I’m using it to update the firmware right now. I’ve also been able to mount the NAS onto Tempest, read-and-write, so I’m starting to copy Brian’s data. This will take a long time!

 

 

 

Posted in Uncategorized | 1 Comment