Two very very late posts about what to do if you think you screwed up a config file or other system file (you do not necessarily an update — might affect other dependencies) and how to set up varnish.
Part I: What to do if you think you screwed up a config file
When we attempted to download varnish on 11/8, we hit some issues with our httpd.conf file that was just outdated/incorrect (for reasons unknown, problem revealed later). For context, on our machines, we edit the httpd.conf file in the sysadmin account and use a script to update the tempest server’s httpd.conf.
Our problem: When we tried to restart httpd, we got an error that it “cannot load httpd/modules/mod_file_cache.so.” We had restarted apache the previous Wednesday to test out varnish on puma, so this was peculiar.
Scott did some detective work in the rsnapshots since April, saw that the other httpd.conf files hadn’t needed to load mod_file_cache.so. After grepping through the yum.logs, we saw that httpd had last updated on october 14th. Did this actually help? We were still confused.
We first tried just commenting out the module in httpd.conf. Then it failed and wanted another module. Trying again: we thought maybe we should just redownload a copy of httpd.conf (using yum reinstall). However, we were concerned that yum reinstall would overwrite the entire httpd directory: there were some (not many) Wellesley specific edits (wellesley.conf) that we didn’t want to overwrite, so we backed them up in the sysadmin account…but then found this article http://www.cyberciti.biz/faq/yum-downloadonly-plugin/! The yum downloadonly flag makes yum only download the package and not install/update them.
>>yumdownloader -v httpd
which puts the httpd files in the current directory. After moving them to /var/tmp/, we used rpm2cpio to extract the files. Ideally, we thought this way we would be able to minimize the amount of edits transferring needed. To untar, we used cpio -idmv:
d: make directories
i: extract
m: preserve modification times
v: vebose
After trying this out, it created subdirectory /usr/var/etc. We looked into etc and saw httpd and logrotate.d and sysconfig. We went into httpd/conf, and saw a httpd.conf file, last updated August 15th.
Interesting: we diff’d the new file and our current httpd.conf (the weird reverted version) and saw that there was a lot of student stuff that had been removed before. Purely looking at the size of the different httpd.conf files, the one we just downloaded using yum –downloadonly was ~35k (as were the rsnapshot daily backups we looked at before) while the current httpd.conf file was ~55k, this confirmed that the version of httpd.conf was odd. Hmm. We ended up just copying the rsnapshot version of httpd.conf (since the snapshot versions had the Wellesley conf edits) into the sysadmin/etc/ version. Comparing the rsnapshot version with the newly downloaded version, confirmed that they weren’t all that different, so we tested, it worked, and we proceeded to use Scott’s httpd.conf update script to update the httpd.conf
>> service httpd start
*now works!
After this side journey in fixing the httpd.conf file, we then cleaned up the /var/tmp (can delete the usr and var and etc), and we kept the yum –downloadonly file so that we can always cpio it again if we want some more info.
So takeaways from the mystery httpd.conf file problem: use yum –downloadonly if necessary so you don’t overwrite your entire directory, use cpio to untar (know that yum –download only downloads into the current directory that you are in). Have an excellent week!
Part II: Setting up varnish
In light of some recent research completions, there were some concerns about whether tempest could handle the brunt of requests hitting the server. After some weeks of exploration, we decided to try out varnish and set it up on tempest. We’re still keeping track of whether varnish will work best for our system, but this is what we did to configure it:
-
Edit /etc/varnish/default.vcl
We changed the backend default to be port 800 since we wanted varnish pointing to 127.0.0.1:800.
-
We changed apache to be listening on port 127.0.0.1:800 on /etc/httpd/conf/httpd.conf to listen to varnish.
So what we have now is we’ve set up varnish backend to point to port 800 and for apache to see varnish on port 800.
-
Now we need to set varnish to listen on port 80 (where apache used to be). To do this, we configured /etc/sysconfig/varnish to alternative option 3. We hit a problem when trying to test this out on puma where we forgot to comment out another alternative option. Doing that will make varnish not work.
-
minor edit: we set the varnish cache file size from its default 256M to 1GB
-
After doing >>service varnish start, it seemed to work.
Note: you can’t test by having varnish and apache going at the same time(if both listen to port 80…), so have to restart apache first (Won’t be listening on port 80) then start varnish to listen on port 80.
-
We tested varnish to see if it impacted our requests by using >>GET -Used http://cs.wellesley.edu/~cs110 to see what happens when we request a file. To check varnish stats, look at… >>varnishstats which will show the cache hits. When testing it on puma, we also used >>nmap puma to see what was on port 80.
So excellent! Varnish works. We’ll probably have another post soon about observed changes with varnish.