Fedora 19 Adventures

When Karishma and I came back to Lemur today, we found that the local user account was impossible to get into, the password was not what we had set it to, so we logged in as root and changed luser’s password (passwd luser), but even then we couldn’t login to luser. But when we looked into the System Preferences about the users, the luser account had a different length than we were expecting, and it said that we had logged in 8 minutes earlier. So that’s confusing.

Then when we were in, we couldn’t get the network running. After a bit of googling, since

service network status

didn’t work. We found someone that told us

systemctl restart NetworkManger.service.

for Fedora 19. From this website, https://ask.fedoraproject.org/question/32861/how-do-i-get-networking-to-work-after-using-hibernate/ So we did that, and then eth0 was connected.

Going through the rest of the startup script is confusing, since some commands have changed and others are no longer valid.

Even though I had changed the /etc/fstab, the new items weren’t mounted. This is because of the network configuration.

We copied the flat files, /etc/group|password|shadow. And then we were about to login and with /students mounted we could access our accounts.

Then Erin decided to try to restart. That went…badly. The UI is really slow, and we could no longer login from the console, just like what had happened to luser earlier. But we could still ssh into the machine. We googled some to find that gnome was the process that was breaking. 🙁 but we can still ssh to do some digging around.

Posted in Uncategorized | Leave a comment

Quota (Near) Script

I’ve been working the past couple weeks on updating the quota_near script. In the last edition, the bash script would get hung on an awk command because it would just keep reading. So I’ve updated the script, and written it in python!

It lives in /home/sysadmin/quota_near.py and /root/bin/quota_near.py with the email being sent in /root/bin/quota_near_email

It uses MIMEText to send emails with Python. And uses the ‘repquota /students’ command to figure out everyone’s quotas.

I’m pretty sure that it can easily be combined to also serve as quota_over by adding a couple lines and the email format. So I’ve done most of that too.

Now to put this in the actual cronjob!

Posted in Uncategorized | Leave a comment

Updating to Firefox 25

Whenever you open firefox on a client, it asks you if you want to update firefox. But normal users don’t have the power to update firefox on the machine. So I went through and did that.

First I updated firefox on tempest using the remi package. I followed the instructions at http://www.tecmint.com/install-firefox-in-rhelcentos-6-3-fedora-17-16/

Then Scott and I checked out the remi package to make sure we weren’t just downloading from a sketchy package. But it seems like this guy who manages the remi package is nice and lots of people trust him. If nothing else, we might be able to get the next firefox release through this package and that would save us some trouble.

Today I ran ah-broadcast of the commands from the link above. And they seemed to work on some of the machines, ones that were up. I still have to check that these changes actually made an effect, but I think they did!

Turns out that the firefox update did not work on at least thrush, so I still need to go in and check whether each has actually updated. And also to read the logs to see if the ah-broadcast worked.

Posted in Uncategorized | Tagged , | Leave a comment

Configuring procmail

For the Boston Preliminary Programming contest (BOSPRE), we set up an account where email to the account sends the email to a printer.  That’s how teams get printouts of their draft code.

The basic setup is pretty straightforward, and is described here: http://www.seas.harvard.edu/hc3/bospre/managers/printer-account.html

The particular procmail incantation isn’t too bad either, and is described here: http://www.seas.harvard.edu/hc3/bospre/managers/email_printer.txt

However, it didn’t work.  What I got in the logs and in the bounce message to the root account was

   ----- The following addresses had permanent fatal errors -----
"| exec /usr/bin/procmail"
    (reason: Service unavailable)
    (expanded from: <hidden_account@cs.wellesley.edu>)

   ----- Transcript of session follows -----
smrsh: "procmail" not available for sendmail programs (stat failed)
554 5.0.0 Service unavailable

X-Actual-Recipient: X-Unix; | exec /usr/bin/procmail
Action: failed
Status: 5.5.0
Diagnostic-Code: X-Unix; 69
Last-Attempt-Date: Tue, 15 Oct 2013 15:40:55 -0400

Not particularly informative, but googling for

smrsh: "procmail" not available for sendmail programs (stat failed)

yielded some advice to put a symlink to /usr/bin/procmail into /etc/smrsh:

cd /etc/smrsh
ln -s /usr/bin/procmail .

and that seemed to work fine!

Posted in Uncategorized | Leave a comment

Updating Papercut

Karishma and I updated papercut last weekend. This was relatively simple but a good exercise for a new SysAdmin.

First I copied all of the configurations from irwin onto tempest. An LTS person had updated irwin to papercut, so we just had to mimic irwin’s configuration on all of the machines.

Then, we updated /usr/network/scripts/parts/printing.sh. One of the hard parts was checking to make sure that our script was idempotent, ie when we run it again it doesn’t necessarily download the new things. So to do this check, we checked the version on tempest against the version on this machine, to see if the versions matched. If they did, we didn’t update. We decided that this was the best technique..so that we could perform similar updates in the future.

We ran all-centos-hosts and it worked! And now this week I went around to all the machines to check to see if it worked, and it did 🙂

Posted in Uncategorized | Tagged | Leave a comment

Node.js install and firewall changes

Installed node.js on Tempest using the directions here

https://github.com/joyent/node/wiki/Installation

and the directions in the README file in the source tarball, available here:

http://nodejs.org/download/

By default, it installs into /usr/local/bin/node and /usr/local/lib/node_modules and /usr/local/lib/dtrace.

So that I could play with it, I opened up a port in the firewall.  The port will not, I think, be available from off-campus, due to the campus firewall. To open up the port, I did:

system-config-firewall

I went to “Other Ports,” clicked on “add,” clicked on “user defined” and put in the port number I wanted to open.  Then I clicked on “apply” and confirmed.

I ran my node.js server and used netstat to show that it was listening:

netstat -t –listen –program

 

Posted in Uncategorized | 7 Comments

Clients going down 4/29/13-4/30/13

The warning errors I was getting when I tried to login as edavis5: Authentification failure. And then getting in as luser: /usr/network not available! Contact your System Administrator.

On Cardinal,

ifconfig showed that it wasn’t connected to the internet.

ifup eth0, and now it’s connected.

uptime told me that it’s been up for 1 day, 36 minutes. So something weird happened at about 6 am on 4/29/13.

So I saved the logfiles from 4/49/13 in /var/log/messages.429outage on cardinal. And then looked at the same ones on tempest, and saved them in a similar file. My guess is that with this little power outage blip that the clients all went down and in that time that they were trying to come back up, tempest was down or at least not receiving their requests to connect.

For now I’m just going around logging into luser, ifup eth0-ing, and then rebooting.

Posted in Uncategorized | Tagged | Leave a comment

Rebooting Puma 4/17/13

During my meeting with Scott, we rebooted puma since it had been up for 258 days and /archives wouldn’t mount. It went down with relatively few complaints (/var, ipmapd, …). But then coming back up it complained about being able to be mount to tempest. We looked at open ports from puma’s side and from tempest’s side and one of them didn’t match up, the one that we needed wasn’t open. From tempest, we then opened that port for upd as necessary with the firewall. Then the mount was successful. Umount also needed an additional port, so when we opened that port we could unmount from tempest. I will be looking into NFS and firewalls playing nicely together, but I wanted to document what we did/saw today.

Posted in Uncategorized | Tagged | Leave a comment

SELynux Running Python Scripts

For Lyn’s CS 118 class he wants to get Python CGI scripts on tempest working. So he changed the SE Linux permissions for httpd_can_network_connect. This also solved the problem we were having with Eni’s PHP/python scripts working on puma but not on tempest. So.. thank you Lyn!

Here’s a transcript of his email, with a link to the article he referenced to figure this out.

Dear CS sysadmins —
For CS118, I need to get have python CGI scripts on tempest that can run urllib.urlopen().
Although python programs with this function run fine when executed as a normal user, they fail with the error IOError<urlopen error [Errno 13] Permission denied> when run in a CGI script. The article at

http://serverfault.com/questions/65362/is-there-a-work-around-for-the-permission-denied-error-from-urllib2-urlopen

suggests this is due to the SELINUX flag httpd_can_network_relay being turned off and suggests turning it on. I tried this, and it did not solve the problem. But instead I tried turning on httpd_can_network_connect and this did solve the problem.  However, the article http://beginlinux.com/blog/2009/05/apache-security-with-selinux/ warns that by default this is “Disabled to prevent hackers from attacking other machines from httpd.”
I don’t want to make tempest a haven for hackers. But I do want my CGI scripts to work. Is it problematic that I changed this flag? Is there a better way to fix this problem?
Below is a transcript of what I did.
[root@tempest ~] getsebool -a | grep httpd
httpd_can_network_connect –> off
[root@tempest ~] setsebool -P httpd_can_network_connect=1
[root@tempest ~] getsebool -a | grep httpd
httpd_can_network_connect –> on
 – lyn –
Posted in Uncategorized | Tagged , | Leave a comment

modifying PHP settings

Consider the following trivial PHP script:

<?php

if( $rv = mail("scott.anderson@wellesley.edu",
   "test email from PHP",
   "body of message" ) ) {
   echo "Success!";
} else {
   echo "Failed:  $rv\n";
}

?>

When I ran this, either from the command line (running as an ordinary user) or via Apache, it *worked* (I got the email and the word “success” was printed), but there was also the following error message:

Warning: mail(/var/log/maillog): failed to open stream: 
Permission denied in /home/ruhlman/public_html/rapp/test-email1.php 
on line 5 Success!

This seemed like an easy thing to fix, but it was anything but.  My first thought was that PHP was connecting to sendmail to send the message, and sendmail wasn’t configured properly.  But every other method of sending mail worked without error.  Furthermore, the /var/log/maillog file was showing the mail messages, so that couldn’t be it.

I used “strace” on the php script, running it as an ordinary user (not Apache):

strace php test-email1.php

And that showed data like this:

lstat("/var/log/maillog", {st_mode=S_IFREG|0620, st_size=35752, ...}) = 0
lstat("/var/log", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/var", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/log/maillog", O_WRONLY|O_CREAT|O_APPEND, 0666) = -1 EACCES (Permission denied)
write(2, "PHP Warning:  mail(/var/log/mail"..., 139PHP Warning:  mail(/var/log/maillog): failed to open stream: Permission denied in /home/ruhlman/public_html/rapp/test-email1.php on line 5
) = 139
write(1, "\nWarning: mail(/var/log/maillog)"..., 135
Warning: mail(/var/log/maillog): failed to open stream: Permission denied in /home/ruhlman/public_html/rapp/test-email1.php on line 5
) = 135

which I interpreted to mean that PHP was trying to write to /var/log/maillog by itself!  Which means the script would be trying to write to the maillog as me if I ran the script.  Well, that was never going to work and shouldn’t work.  There’s no way I’m going to change /var/log/maillog to be world-writable, and that seemed the only solution.

So, I need to change the place that PHP tries to log mail messages. That’s set in /etc/php.ini to be /var/log/maillog, which is the default.  (Usually, a default is a pretty good value, but I don’t see how this one is ever going to be right.)

Now, the PHP manual tells you that for any PHP configuration variable, you can change it in various places, depending on the variable.  The variable mail.log, according to the manual listing:  http://www.php.net/manual/en/ini.list.php can be changed at PHP_INI_ALL, which means anywhere.  Cool, that’s easy.  I can change it right in the script itself, using the ini_set function to set the mail.log file to a file that is writable by owner and by apache:

 [ruhlman@tempest rapp] ls -l ./maillog 
-rw-rw----. 1 ruhlman apache 521 Apr 15 23:59 ./maillog
[ruhlman@tempest rapp]

Here’s the script that should work fine.

<?php

$rv = ini_set('mail.log','/home/ruhlman/public_html/rapp/maillog');
if( ! $rv ) {
    echo "failed to set mail.log\n";
} else {
    echo "changed mail.log from $rv\n";
}

$log = ini_get('mail.log');

$stat = stat( $log );
$permnum = $stat['mode'];
$permstr = sprintf("%o",$permnum);
echo "<p>mail.log is $log with perms $permstr\n";

if( $rv = mail("scott.anderson@wellesley.edu",
    "test email from PHP",
    "body of message" ) ) {
    echo "Success!\n";
} else {
    echo "Failed:  $rv\n";
}

?>

However, that was a miserable failure.  Here’s the output from apache running that script:

failed to set mail.log

mail.log is /var/log/maillog with perms 100620 
Warning: mail(/var/log/maillog): failed to open stream: 
Permission denied in /home/ruhlman/public_html/rapp/test-email2.php 
on line 19 Success!

Hunh.  So, I guess I can’t set it in the script.  Maybe I can set it in the .htaccess file?  Yes, that can be done:

[ruhlman@tempest rapp] cat .htaccess
php_value mail.log /home/ruhlman/public_html/rapp/maillog

But first, you have to tell Apache that .htaccess files in this directory tree can change php_values.  Here’s the relevant lines from a file in /etc/httpd/conf.d/ that configures this directory:

# Added in April 2013 to see if we can set a PHP value in .htaccess in that directory
<Directory /home/ruhlman/public_html>
AllowOverride FileInfo AuthConfig Limit Options
</Directory>

It’s the adding of options that makes the difference.  Before that, my efforts with .htaccess were fruitless.  I’ve also since learned from this manual page on configuring PHP mail http://www.php.net/manual/en/mail.configuration.php that the mail.log variable might be PHP_INI_PERDIR instead of PHP_INI_ALL, which would explain why I couldn’t change it in the script.  There’s also mention, in http://www.php.net/manual/en/configuration.changes.modes.php   of a .user.ini file, which might be easier than changing the <Directory> element in the system httpd file just so that .htaccess can be used. to change the mail.log variable.  However, this page http://php.net/manual/en/configuration.file.per-user.php suggests that maybe that’s for non-Apache installations.

Anyhow, this was hard to debug, but ultimately successful.  Hopefully, this post will save someone else some heartache.

Posted in Uncategorized | Leave a comment