Saving a web page to a client (say, Sampras), and then going to cs.wellesley.edu and refreshing the page often fails to get the new files, because of NFS delays. I think the saving is nearly instantaneous I think this is mostly at Puma’s end; it’s not reading the new file. We could work on the parameters for NFS, or we could make Tempest be the web server, which we need to do anyhow.
I checked for differences between the directory tree at /etc/httpd for each system. There are lots of extra files in /etc/httpd/conf.d on Puma, unsurprisingly. Some of them are unnecessary (manual.conf).
Some, I’m not sure we use, but they can’t hurt: perl.conf enables mod_perl and python.conf enables mod_python, which advanced web programmers might use.
[root@tempest ~]# yum -y install mod_perl Resolving Dependencies --> Running transaction check ---> Package mod_perl.x86_64 0:2.0.4-10.el6 will be installed --> Processing Dependency: perl(BSD::Resource) for package: mod_perl-2.0.4-10.el6.x86_64 --> Running transaction check ---> Package perl-BSD-Resource.x86_64 0:1.29.03-3.el6 will be installed --> Finished Dependency Resolution Dependencies Resolved ===================================================================================================================================== Package Arch Version Repository Size ===================================================================================================================================== Installing: mod_perl x86_64 2.0.4-10.el6 rhel-x86_64-server-6 3.2 M Installing for dependencies: perl-BSD-Resource x86_64 1.29.03-3.el6 rhel-x86_64-server-6 35 k Installed: mod_perl.x86_64 0:2.0.4-10.el6 Dependency Installed: perl-BSD-Resource.x86_64 0:1.29.03-3.el6 Complete!
Now, we made some changes to the perl.conf file, so
[root@tempest ~]# cd /etc/httpd/conf.d [root@tempest conf.d]# mv perl.conf perl.conf.orig [root@tempest conf.d]# cp /home/sysadmin/etc/httpd/conf.d/perl.conf . [root@tempest conf.d]# ls -Z perl.conf* -rw-rw----. root root unconfined_u:object_r:httpd_config_t:s0 perl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 perl.conf.orig [root@tempest conf.d]# chcon --reference=perl.conf.orig perl.conf [root@tempest conf.d]# ls -Z perl.conf* -rw-rw----. root root system_u:object_r:httpd_config_t:s0 perl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 perl.conf.orig [root@tempest conf.d]# ls -Z -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 mod_dnssd.conf -rw-rw----. root root system_u:object_r:httpd_config_t:s0 perl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 perl.conf.orig -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 php.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 README -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 welcome.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 wsgi.conf
And now for python:
[root@tempest ~]# yum -y install mod_python No package mod_python available. Error: Nothing to do [root@tempest ~]#
Hunh. That was unexpected. mod_python *does* exist on Puma. Let’s see ….
[root@tempest ~]# yum search mod_python ======================================================== Matched: mod_python ======================================================== mod_wsgi.x86_64 : A WSGI interface for Python web applications in Apache [root@tempest ~]# yum -y install mod_wsgi ---> Package mod_wsgi.x86_64 0:3.2-1.el6 will be installed Installed: mod_wsgi.x86_64 0:3.2-1.el6 Complete! [root@tempest ~]#
Okay, that’s better. Now, I think Consuelo wanted this, to do subversion over the web:
[root@tempest conf.d]# yum -y install mod_dav_svn --> Running transaction check ---> Package mod_dav_svn.x86_64 0:1.6.11-7.el6 will be installed Installed: mod_dav_svn.x86_64 0:1.6.11-7.el6 Complete!
I’m not going to install squid or proxy_ajp or webalizer, so we can ignore those modules out of the list on puma:
[root@tempest conf.d]# ssh puma ls -l /etc/httpd/conf.d/ total 140 -rw-r--r-- 1 root root 793 Jul 19 14:37 00_auth_cas.conf -rw-r--r-- 1 root root 517 Sep 9 2004 10_authz_ldap.conf -rw-r--r-- 1 root root 9835 Mar 9 2008 20_ssl.conf -rw-r--r-- 1 root root 295 Jun 6 10:04 manual.conf -rw-rw---- 1 root root 1824 Feb 1 2008 perl.conf -rw-r--r-- 1 root root 1796 Apr 22 2005 perl.conf.orig -rw-r--r-- 1 root root 1853 Feb 1 2008 perl.conf.wellesley -rw-r--r-- 1 root root 560 Jun 2 2010 php.conf -rw-r--r-- 1 root root 566 Jun 6 10:04 proxy_ajp.conf -rw-r--r-- 1 root root 1671 Jul 12 2006 python.conf -rw-r--r-- 1 root root 392 Jun 6 10:04 README -rw-r--r-- 1 root root 334 Feb 16 2010 squid.conf -rw-r--r-- 1 root root 1315 Apr 4 2005 subversion.conf -rw-r--r-- 1 root root 8159 Feb 1 2008 twiki.conf -rw-r--r-- 1 root root 352 Jul 12 2006 webalizer.conf -rw-r--r-- 1 root root 299 Jun 6 10:04 welcome.conf [root@tempest conf.d]#
That leaves on twiki, which I’ll put off for now, and the CAS stuff. I implement a script to duplicate the CAS setup. Here it is:
[root@tempest conf.d]# cat /home/sysadmin/fixes/install-cas.sh #!/bin/bash set -e yum -y install --enablerepo=epel mod_auth_cas yum -y install mod_authz_ldap # rename modules to load first and in order cd /etc/httpd/conf.d/ mv authz_ldap.conf 10_authz_ldap.conf mv ssl.conf 20_ssl.conf # This file was developed by following http://wiki.jasig.org/pages/viewpage.action?pageid=47874958 # and Jim Billing's (of Wellesley's LTS) advice: cp /home/sysadmin/etc/httpd/conf.d/00_auth_cas.conf /etc/httpd/conf.d cd /var/lib mkdir cas chown apache:apache cas chmod 750 cas semanage fcontext -a -s system_u -t httpd_var_lib_t /var/lib/cas restorecon -r -v ./
Let’s go ahead and do that:
[root@tempest conf.d]# /home/sysadmin/fixes/install-cas.sh epel/metalink | 12 kB 00:00 epel | 4.3 kB 00:00 epel/primary_db | 4.6 MB 00:01 Setting up Install Process Resolving Dependencies --> Running transaction check ---> Package mod_auth_cas.x86_64 0:1.0.8.1-2.el6 will be installed --> Finished Dependency Resolution Installed: mod_auth_cas.x86_64 0:1.0.8.1-2.el6 Complete! Resolving Dependencies --> Running transaction check ---> Package mod_authz_ldap.x86_64 0:0.26-15.el6 will be installed --> Finished Dependency Resolution Installed: mod_authz_ldap.x86_64 0:0.26-15.el6 Complete! mv: cannot stat `ssl.conf': No such file or directory [root@tempest conf.d]#
Oops. Hmm. Where does that come from?
[root@tempest conf.d]# ssh puma rpm -qf /etc/httpd/conf.d/ssl.conf mod_ssl-2.2.3-65.el5.centos [root@tempest conf.d]# yum -y install mod_ssl Resolving Dependencies --> Running transaction check ---> Package mod_ssl.x86_64 1:2.2.15-15.el6_2.1 will be installed --> Finished Dependency Resolution Installed: mod_ssl.x86_64 1:2.2.15-15.el6_2.1 Complete! [root@tempest conf.d]#
Okay, that’s better. I updated the script to include that step, and then undid the “mv” step (since that’s not idempotent) and re-ran the script.
root@tempest conf.d]# mv 10_authz_ldap.conf authz_ldap.conf [root@tempest conf.d]# /home/sysadmin/fixes/install-cas.sh Package mod_auth_cas-1.0.8.1-2.el6.x86_64 already installed and latest version Nothing to do Package mod_authz_ldap-0.26-15.el6.x86_64 already installed and latest version Nothing to do Package 1:mod_ssl-2.2.15-15.el6_2.1.x86_64 already installed and latest version Nothing to do mkdir: cannot create directory `cas': File exists
Hmm. Did that directory already exist?
[root@tempest conf.d]# ls -ld /var/lib/cas drwxr-xr-x. 3 root root 4096 Feb 3 2012 /var/lib/cas
Seems so. *sigh*. Okay, edit the script to make it idempotent, and re-run. It should only do the last steps of “semanage” and “restorecon”:
[root@tempest conf.d]# /home/sysadmin/fixes/install-cas.sh Package mod_auth_cas-1.0.8.1-2.el6.x86_64 already installed and latest version Nothing to do Package mod_authz_ldap-0.26-15.el6.x86_64 already installed and latest version Nothing to do Package 1:mod_ssl-2.2.15-15.el6_2.1.x86_64 already installed and latest version Nothing to do restorecon reset /var/lib/ldap.24549 context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.004 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/objectClass.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.006 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/ou.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/cn.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/id2entry.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.001 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.003 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/DB_CONFIG context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/gidNumber.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/nisMapName.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/memberUid.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/loginShell.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/dn2id.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/uidNumber.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/log.0000000001 context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/alock context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/log.0000000003 context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/log.0000000002 context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.002 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/uid.bdb context system_u:object_r:slapd_db_t:s0->system_u:object_r:var_lib_t:s0 restorecon reset /var/lib/ldap.24549/__db.005 context unconfined_u:object_r:slapd_db_t:s0->unconfined_u:object_r:var_lib_t:s0 restorecon reset /var/lib/denyhosts context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/offset context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/hosts-valid context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/allowed-hosts~ context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/hosts-restricted context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/hosts-root context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/allowed-hosts context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/users-valid context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/suspicious-logins context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/users-hosts context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/hosts context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/denyhosts/users-invalid context unconfined_u:object_r:var_lib_t:s0->unconfined_u:object_r:denyhosts_var_lib_t:s0 restorecon reset /var/lib/rsyslog context system_u:object_r:var_lib_t:s0->system_u:object_r:syslogd_var_lib_t:s0 restorecon reset /var/lib/cas context system_u:object_r:var_lib_t:s0->system_u:object_r:httpd_var_lib_t:s0 [root@tempest conf.d]#
Wow, I didn’t expect those other operations by restorecon. I wonder if that explains the trouble we’ve been having with denyhosts? Seems likely: whenever you have unexplained difficulties, consider blaming selinux!
Now, we should check the httpd.conf file. First, compare the httpd versions:
[root@tempest conf.d]# rpm -q httpd httpd-2.2.15-15.el6_2.1.x86_64 [root@tempest conf.d]# ssh puma rpm -q httpd httpd-2.2.3-65.el5.centos
I think those are similar enough that they probably haven’t changed the httpd.conf file. Eventually, I should clean out the cruft in there, such as cgi-script-user directives for students who have graduated, but that’s not for today.
[root@tempest conf.d]# cd ../conf [root@tempest conf]# ls httpd.conf magic [root@tempest conf]# mv httpd.conf httpd.conf.orig [root@tempest conf]# cp /home/sysadmin/etc/httpd/conf/httpd.conf . [root@tempest conf]# ls -Z httpd.conf* -rw-r-----. root root unconfined_u:object_r:httpd_config_t:s0 httpd.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 httpd.conf.orig [root@tempest conf]# chcon --reference=httpd.conf.orig httpd.conf [root@tempest conf]# chmod a+r httpd.conf [root@tempest conf]# ls -l httpd.conf* -rw-r--r--. 1 root root 54451 Aug 15 12:16 httpd.conf -rw-r--r--. 1 root root 34418 Feb 7 2012 httpd.conf.orig [root@tempest conf]# ls -Z httpd.conf* -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 httpd.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 httpd.conf.orig [root@tempest conf]#
Okay, I’m ready to ask apachectl to test the configuration files:
[root@tempest conf]# apachectl -t httpd: Syntax error on line 196 of /etc/httpd/conf/httpd.conf: Cannot load /etc/httpd/modules/mod_file_cache.so into server: /etc/httpd/modules/mod_file_cache.so: cannot open shared object file: No such file or directory [root@tempest conf]# ssh puma rpm -qf /etc/httpd/modules/mod_file_cache.so httpd-2.2.3-65.el5.centos
So, I think that file goes with the version of httpd on Puma, and not the one on Tempest, so I think the right thing to do is simply to delete that line from the httpd.conf file. In fact, there may be other differences in default settings, so let’s check:
[root@tempest conf]# diff httpd.conf httpd.conf.orig | more 61c61,63 < # identification number when it starts. --- > # identification number when it starts. Note the PIDFILE variable in > # /etc/sysconfig/httpd must be set appropriately if this location is > # changed. 68c70 < Timeout 120 --- > Timeout 60 117,118c119,120 < StartServers 2 < MaxClients 150 --- > StartServers 4 > MaxClients 300 186a189 > LoadModule substitute_module modules/mod_substitute.so 191a195 > LoadModule proxy_ajp_module modules/mod_proxy_ajp.so 196d199 < LoadModule mem_cache_module modules/mod_mem_cache.so 200,210d202 < # Wellesley mod: Added these for Subversion support. See http://svnbook.red-bean.com/en/1.5/svn.serverconfig.httpd.html < # Actually, these aren't necessary, because the .so files are in /etc/httpd/modules/ < # LoadModule dav_module modules/mod_dav.so < # LoadModule dav_svn_module modules/mod_dav_svn.so < LoadModule unique_id_module modules/mod_unique_id.so < < # Wellesley mod: Added this for Django support. See < # http://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/ < # TODO < < 214d205 < #LoadModule cern_meta_module modules/mod_cern_meta.so 215a207,216 > #LoadModule authn_dbd_module modules/mod_authn_dbd.so > #LoadModule cern_meta_module modules/mod_cern_meta.so > #LoadModule cgid_module modules/mod_cgid.so > #LoadModule dbd_module modules/mod_dbd.so > #LoadModule dumpio_module modules/mod_dumpio.so > #LoadModule filter_module modules/mod_filter.so > #LoadModule ident_module modules/mod_ident.so > #LoadModule log_forensic_module modules/mod_log_forensic.so > #LoadModule unique_id_module modules/mod_unique_id.so > # 276d276 < ServerName cs.wellesley.edu:80 331c331 < Options Indexes FollowSymLinks ExecCGI --- > Options Indexes FollowSymLinks ...
Ick. The difference is modules and such has me concerned. So, I think it’s time to bite the bullet and factor the system httpd.conf file from the Wellesley augmentations and overrides. I think if we put all those in a file in /etc/httpd/conf.d/, it’ll get loaded. So, I did that, and we’ll see if this all works. We have to undo the modified httpd.conf file and put a file in conf.d:
[root@tempest conf]# rm httpd.conf [root@tempest conf]# mv httpd.conf httpd.conf~ httpd.conf.orig [root@tempest conf]# rm httpd.conf~ [root@tempest conf]# mv httpd.conf.orig httpd.conf [root@tempest conf]# cd ../conf.d [root@tempest conf.d]# cp /home/sysadmin/etc/httpd/conf.d/wellesley.conf . [root@tempest conf.d]# ls -Z w* -rw-r-----. root root unconfined_u:object_r:httpd_config_t:s0 00_auth_cas.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 10_authz_ldap.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 20_ssl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 auth_cas.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 mod_dnssd.conf -rw-rw----. root root system_u:object_r:httpd_config_t:s0 perl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 perl.conf.orig -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 php.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 README -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 subversion.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 welcome.conf -rw-rw----. root root unconfined_u:object_r:httpd_config_t:s0 wellesley.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 wsgi.conf [root@tempest conf.d]# chcon --reference=wsgi.conf wellesley.conf [root@tempest conf.d]# chcon --reference=10_authz_ldap.conf 00_auth_cas.conf [root@tempest conf.d]# ls -Z -rw-r-----. root root unconfined_u:object_r:httpd_config_t:s0 00_auth_cas.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 10_authz_ldap.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 20_ssl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 auth_cas.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 mod_dnssd.conf -rw-rw----. root root system_u:object_r:httpd_config_t:s0 perl.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 perl.conf.orig -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 php.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 README -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 subversion.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 welcome.conf -rw-rw----. root root system_u:object_r:httpd_config_t:s0 wellesley.conf -rw-r--r--. root root system_u:object_r:httpd_config_t:s0 wsgi.conf [root@tempest conf.d]# chmod 644 *
Okay, let’s see how that goes:
[root@tempest conf.d]# !apac apachectl -t httpd: Syntax error on line 221 of /etc/httpd/conf/httpd.conf: Syntax error on line 6 of /etc/httpd/conf.d/00_auth_cas.conf: Cannot load /etc/httpd/modules/mod_auth_cas.so into server: /etc/httpd/modules/mod_auth_cas.so: undefined symbol: SSL_connect
*sigh*.
Okay, first check that it’s not just some module that is out of date:
[root@tempest conf.d]# rpm -V mod_auth_cas [root@tempest conf.d]# rpm -V httpd S.5....T. c /var/www/error/include/bottom.html [root@tempest conf.d]# yum -y update httpd mod_auth_cas Loaded plugins: downloadonly, product-id, refresh-packagekit, rhnplugin, security, subscription-manager Updating certificate-based repositories. Unable to read consumer identity Setting up Update Process No Packages marked for Update [root@tempest conf.d]#
I did some googling for the error message and I found the following Bugzilla entry: https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id=708550. So, I added the following lines to the top of 00_auth_cas.conf:
[root@tempest conf.d]# head -12 00_auth_cas.conf # # mod_auth_cas is an Apache 2.0/2.2 compliant module that supports the # CASv1 and CASv2 protocols # # The following three lines may fix a bug where the SSL_connect symbol # is not defined when the mod_auth_cas.so is loaded, because the # mod_ssl is loaded later. <IfModule !mod_ssl.c> LoadModule ssl_module modules/mod_ssl.so </IfModule>
When I do that, I get:
[root@tempest conf.d]# apachectl -t [Wed Aug 15 14:35:21 2012] [warn] module ssl_module is already loaded, skipping [Wed Aug 15 14:35:21 2012] [warn] module auth_cas_module is already loaded, skipping Syntax OK
Interesting! So, maybe the thing to do is to re-arrange the order of these .conf files. Instead of
[root@tempest conf.d]# ls -1 *0* 00_auth_cas.conf 10_authz_ldap.conf 20_ssl.conf
we do
[root@tempest conf.d]# mv 00_auth_cas.conf 05_auth_cas.conf [root@tempest conf.d]# mv 20_ssl.conf 00_ssl.conf
to get:
[root@tempest conf.d]# ls -1 *0* 00_ssl.conf 05_auth_cas.conf 10_authz_ldap.conf
So, we get:
[root@tempest conf.d]# apachectl -t [Wed Aug 15 14:40:20 2012] [warn] module auth_cas_module is already loaded, skipping Syntax OK
Hunh? How does that happen?
[root@tempest conf.d]# grep mod_auth_cas.so * 05_auth_cas.conf:# is not defined when the mod_auth_cas.so is loaded, because the 05_auth_cas.conf:LoadModule auth_cas_module modules/mod_auth_cas.so auth_cas.conf:LoadModule auth_cas_module modules/mod_auth_cas.so
Oh, we don’t need the auth_cas.conf file, since we have the 05 version to replace it:
[root@tempest conf.d]# rm auth_cas.conf [root@tempest conf.d]# apachectl -t Syntax OK
Oh, good. I also went back and deleted those extra lines at the top of the file that is now 05_auth_cas.conf.
[root@tempest conf.d]# service httpd start Starting httpd: [ OK ] [root@tempest conf.d]# chkconfig httpd on
We can do some testing, by visiting these urls:
Hmm. That results in a page that says:
This page is used to test the proper operation of the Apache HTTP server after it has been installed. If you can read this page, it means that the Apache HTTP server installed at this site is working properly.
If you are a member of the general public:
The fact that you are seeing this page indicates that the website you just visited is either experiencing problems, or is undergoing routine maintenance.
Ah, the /var/log/httpd/error_log shows:
[root@tempest conf.d]# tail /var/log/httpd/error_log [Wed Aug 15 14:43:44 2012] [notice] Apache/2.2.15 (Unix) DAV/2 mod_ssl/2.2.15 OpenSSL/1.0.0-fips PHP/5.3.3 SVN/1.6.11 mod_wsgi/3.2 Python/2.6.6 mod_perl/2.0.4 Perl/v5.10.1 configured -- resuming normal operations [Wed Aug 15 14:48:05 2012] [error] [client 149.130.167.203] Symbolic link not allowed or link target not accessible: /var/www/html/index.html [Wed Aug 15 14:48:05 2012] [error] [client 149.130.167.203] Symbolic link not allowed or link target not accessible: /var/www/html/index.html [root@tempest conf.d]# ls -l /var/www/html/index.html lrwxrwxrwx. 1 root root 31 May 10 2011 /var/www/html/index.html -> /home/cs/public_html/index.html
So, the symlink out of /var/www/html is not working. Hmm, but I see that
<Directory "/var/www/html"> ... Options Indexes FollowSimLinks
So, why isn’t the simlink being followed? Is it an issue with showing the target of the link? So, try the following URL:
http://tempest.wellesley.edu/~cs/
Ah, that says 404, and the error log shows:
[root@tempest conf.d]# tail /var/log/httpd/error_log [Wed Aug 15 14:58:05 2012] [error] [client 149.130.167.203] File does not exist: /var/www/html/~cs [root@tempest conf.d]#
So, we need to enable home directories. But that should be done in conf.d/wellesley.conf.
[root@tempest conf.d]# ls -l wellesley.conf -rw-r--r--. 1 root root 22129 Aug 15 12:35 wellesley.conf [root@tempest conf.d]# grep public_html !$ grep public_html wellesley.conf # To enable requests to /~user/ to serve the user's public_html UserDir public_html <Directory /home/*/public_html> ...
I wasn’t sure whether we could override the configuration of mod_userdir in the main httpd.conf file, so let’s try that, first.
[root@tempest conf.d]# cd /etc/httpd/conf/ [root@tempest conf]# ls httpd.conf* httpd.conf httpd.conf.orig [root@tempest conf]# diff httpd.conf* 366,368c366 < # Wellesley mod: allow UserDir < #UserDir disabled < UserDir public_html --- > UserDir disabled [root@tempest conf]# apachectl graceful
Okay, now I get 403 forbidden:
[root@tempest conf]# tail -1 /var/log/httpd/error_log [Wed Aug 15 15:08:45 2012] [error] [client 149.130.167.203] (13)Permission denied: access to /~cs/ denied
Okay, now what? I think it might be that apache isn’t able to read those files. Let’s learn how to test that:
[root@tempest conf]# echo "success" > /tmp/apache-no [root@tempest conf]# echo "success" > /tmp/apache-only [root@tempest conf]# chown apache.apache /tmp/apache-only [root@tempest conf]# chmod 600 /tmp/apache* [root@tempest conf]# su -s /bin/sh apache -c "cat /tmp/apache-only" success [root@tempest conf]# su -s /bin/sh apache -c "cat /tmp/apache-no" cat: /tmp/apache-no: Permission denied
Let’s see if that command works on these web pages:
[root@tempest conf]# su -s /bin/sh apache -c "head /home/cs/public_html/index.html" <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <!-- Edited by lyn on 03/04/2012 to edit the intro to "A Brief History" (removing mention that it was written back in 1999.) --> <html xmlns="http://www.w3.org/1999/xhtml"> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1" /> [root@tempest conf]#
Okay, so that’s not it. Hmm…. Okay, I’ll modify the main httpd.conf file to use, exactly, the <Directory configuration they have.
[root@tempest conf]# grep -A12 "^<Directory .*public_html" httpd.conf <Directory /home/*/public_html> AllowOverride FileInfo AuthConfig Limit Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec <Limit GET POST OPTIONS> Order allow,deny Allow from all </Limit> <LimitExcept GET POST OPTIONS> Order deny,allow Deny from all </LimitExcept> </Directory> [root@tempest conf]# apachectl graceful [root@tempest conf]# HEAD http://tempest.wellesley.edu/~cs/index.html 403 Forbidden Connection: close Date: Wed, 15 Aug 2012 20:06:00 GMT Server: Apache/2.2.15 (Red Hat) Content-Type: text/html; charset=iso-8859-1 Client-Date: Wed, 15 Aug 2012 20:06:00 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 [root@tempest conf]# tail -1 /var/log/httpd/error_log [Wed Aug 15 16:06:00 2012] [error] [client 127.0.0.1] (13)Permission denied: access to /~cs/index.html denied
@#*!! Let’s try running the server in debug mode:
[root@tempest conf]# service httpd stop Stopping httpd: [ OK ] [root@tempest conf]# /usr/sbin/httpd -X -c "LogLevel debug" & [1] 17983 [root@tempest conf]# HEAD http://tempest.wellesley.edu/~cs/index.html 200 OK Connection: close Date: Wed, 15 Aug 2012 20:16:52 GMT Accept-Ranges: bytes Server: Apache/2.2.15 (Red Hat) Content-Length: 4579 Content-Type: text/html; charset=UTF-8 Last-Modified: Wed, 04 Apr 2012 15:11:06 GMT Client-Date: Wed, 15 Aug 2012 20:16:52 GMT Client-Peer: 127.0.0.1:80 Client-Response-Num: 1 [root@tempest conf]#
Bizarre! Running it in debug mode solves the problem! Let me bump up the logging in the configuration file:
[root@tempest conf]# grep LogLevel httpd.conf # LogLevel: Control the number of messages logged to the error_log. #LogLevel warn LogLevel debug [root@tempest conf]# service httpd start Starting httpd: [root@tempest conf]# service httpd status httpd (pid 17983) is running... [root@tempest conf]# HEAD http://tempest.wellesley.edu/~cs/index.html 200 OK Connection: close
Okay, so bumping up the logging works. What if we take it back down?
[root@tempest conf]# grep LogLevel httpd.conf # LogLevel: Control the number of messages logged to the error_log. LogLevel warn #LogLevel debug [root@tempest conf]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [1]+ Done /usr/sbin/httpd -X -c "LogLevel debug" [root@tempest conf]# HEAD http://tempest.wellesley.edu/~cs/index.html 403 Forbidden Connection: close
Ah, I see. It wasn’t the loglevel, it was the server process running in debug mode (see how it exits when we restart the server). Let’s try once more:
[root@tempest conf]# service httpd stop Stopping httpd: [ OK ] [root@tempest conf]# /usr/sbin/httpd -X
That hangs, as it should. Now, in another window, I do:
[root@tempest ~]# HEAD http://tempest.wellesley.edu/~cs/index.html 200 OK
Hmm. Good, but what I’d really like is for it to fail, so that I can try to figure out why.
Some Googling found this: http://stackoverflow.com/questions/8816836/apache-403-error-13permission-denied-access-to-denied-fedora-16, which suggests it’s an selinux problem (big surprise). But where are the darn selinux logs? Ah, this page http://wiki.centos.org/HowTos/SELinux/ says they’re in /var/log/audit/audit.log, and sure enough:
root@tempest conf]# grep httpd /var/log/audit/audit.log | wc
17194 378268 5100511
[root@tempest conf]# grep /usr/sbin/httpd /var/log/audit/audit.log | wc
8597 240716 3128595
[root@tempest conf]# grep httpd /var/log/audit/audit.log | tail -5
type=SYSCALL msg=audit(1345062965.521:18406): arch=c000003e syscall=4 success=no exit=-13 a0=7f26fddca878 a1=7fff177324c0 a2=7fff177324c0 a3=1999999999999999 items=0 ppid=18276 pid=18278 auid=716 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=1019 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1345062971.986:18407): avc: denied { search } for pid=18280 comm="httpd" name="cs" dev=dm-4 ino=5636097 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=SYSCALL msg=audit(1345062971.986:18407): arch=c000003e syscall=4 success=no exit=-13 a0=7f26fe2305c0 a1=7fff177321a0 a2=7fff177321a0 a3=0 items=0 ppid=18276 pid=18280 auid=716 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=1019 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1345062971.986:18408): avc: denied { getattr } for pid=18280 comm="httpd" path="/home/cs" dev=dm-4 ino=5636097 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:user_home_dir_t:s0 tclass=dir
type=SYSCALL msg=audit(1345062971.986:18408): arch=c000003e syscall=6 success=no exit=-13 a0=7f26fe2306a0 a1=7fff177321a0 a2=7fff177321a0 a3=1 items=0 ppid=18276 pid=18280 auid=716 uid=48 gid=48 euid=48 suid=48 fsuid=48 egid=48 sgid=48 fsgid=48 tty=(none) ses=1019 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
[root@tempest conf]#
Absolutely unreadable. So, use “sealert”
[root@tempest conf]# sealert -b -bash: sealert: command not found [root@tempest conf]# yum -y install setroubleshoot ============================================================================================================================================================================ Package Arch Version Repository Size ============================================================================================================================================================================ Installing: setroubleshoot x86_64 3.0.47-3.el6_3 rhel-x86_64-server-6 118 k Installing for dependencies: setroubleshoot-plugins noarch 3.0.40-1.el6 rhel-x86_64-server-6 505 k setroubleshoot-server x86_64 3.0.47-3.el6_3 rhel-x86_64-server-6 1.3 M Running Transaction Installing : setroubleshoot-plugins-3.0.40-1.el6.noarch 1/3 Installing : setroubleshoot-server-3.0.47-3.el6_3.x86_64 2/3 Installing : setroubleshoot-3.0.47-3.el6_3.x86_64 3/3 Installed products updated. Verifying : setroubleshoot-server-3.0.47-3.el6_3.x86_64 1/3 Verifying : setroubleshoot-plugins-3.0.40-1.el6.noarch 2/3 Verifying : setroubleshoot-3.0.47-3.el6_3.x86_64 3/3 Installed: setroubleshoot.x86_64 0:3.0.47-3.el6_3 Dependency Installed: setroubleshoot-plugins.noarch 0:3.0.40-1.el6 setroubleshoot-server.x86_64 0:3.0.47-3.el6_3 Complete!
Let’s have sealert analyze the logs. Wait, this is taking *forever*. Let’s have it just focus on the errors we want it to:
[root@tempest conf]# grep "/home/cs" /var/log/audit/audit.log > /tmp/audit-home-cs [root@tempest conf]# sealert -a /tmp/audit-home-cs 100% donefound 1 alerts in /tmp/audit-home-cs -------------------------------------------------------------------------------- SELinux is preventing httpd from getattr access on the directory /home/cs. ***** Plugin catchall_boolean (47.5 confidence) suggests ******************* If you want to allow httpd to read user content Then you must tell SELinux about this by enabling the 'httpd_read_user_content' boolean.You can read 'user_selinux' man page for more details. Do setsebool -P httpd_read_user_content 1 ***** Plugin catchall_boolean (47.5 confidence) suggests ******************* If you want to allow httpd to read home directories Then you must tell SELinux about this by enabling the 'httpd_enable_homedirs' boolean.You can read 'user_selinux' man page for more details. Do setsebool -P httpd_enable_homedirs 1 ***** Plugin catchall (6.38 confidence) suggests *************************** If you believe that httpd should be allowed getattr access on the cs directory by default. Then you should report this as a bug. You can generate a local policy module to allow this access. Do allow this access for now by executing: # grep httpd /var/log/audit/audit.log | audit2allow -M mypol # semodule -i mypol.pp
Okay, let’s follow its instructions:
[root@tempest conf]# setsebool -P httpd_read_user_content 1 [root@tempest conf]# setsebool -P http_enable_homedirs 1 libsemanage.dbase_llist_set: record not found in the database libsemanage.dbase_llist_set: could not set record value Could not change boolean http_enable_homedirs Could not change policy booleans [root@tempest conf]# setsebool -P httpd_enable_homedirs 1 [root@tempest conf]# service httpd restart Stopping httpd: [ OK ] Starting httpd: [ OK ] [root@tempest conf]# HEAD http://tempest.wellesley.edu/~cs/index.html 200 OK
Yay! This suggests, though, that we may have lots of difficulties with student accounts, since they’re not under /home. We’ll see that next.
[root@tempest conf]# getent passwd gdome gdome:x:707:707:Georgia B. Dome:/students/gdome:/bin/bash [root@tempest conf]# HEAD http://tempest.wellesley.edu/~gdome/index.html 200 OK
Whew! No problem here.
Finally, we should check out the python, perl, CAS and other module-like stuff. But this is enough for today.
