CS SysAdmin

Randy just asked me to add Karishma Chadha and Smaranda Sandu to the cs251stu group.

There are basically two strategies:

1. edit the /etc/group entry, extract the whole group, use the migrate_group.pl script to convert it to ldif format, then delete the old LDAP entry using ldapdelgroup and use ldapadd2 to insert the new one.

2. edit the /etc/group entry.  Extract the existing group in LDIF format from LDAP using ldapsearchgroup, edit that to add the new entries, and delete the old and add the new.

Here is what I did:

[root@tempest ~] grep cs251stu /etc/group
cs251stu:*:2726:cs251,fturbak,gdome,rshull,achang4,ahu2,alevy2,choef,dgranger,dsuggs,esegreto,ggoodwin,gwunnava,hbond,hlu,hye,jkilleng,jlu3,jokerlun,klevine2,nshih,okotsopo,rbobbins,rshull,skim22,slee14,ssastry,sshah2,ssunier,swang6,vbrown,vlin,xlu2,ytrujill
[root@tempest ~] emacs -nw !$
emacs -nw /etc/group
[root@tempest ~] grep cs251stu /etc/group
cs251stu:*:2726:cs251,fturbak,gdome,rshull,achang4,ahu2,alevy2,choef,dgranger,dsuggs,esegreto,ggoodwin,gwunnava,hbond,hlu,hye,jkilleng,jlu3,jokerlun,klevine2,nshih,okotsopo,rbobbins,rshull,skim22,slee14,ssastry,sshah2,ssunier,swang6,vbrown,vlin,xlu2,ytrujill,asandu,kchadha
[root@tempest ~] cd /root/acctmp/
[root@tempest acctmp] grep cs251stu /etc/group > cs251stu.group
[root@tempest acctmp] migrate_group.pl cs251stu.group > cs251stu.ldif
[root@tempest acctmp] ldapdelgroup cs251stu
dn: cn=cs251stu,ou=Group,dc=cs,dc=wellesley,dc=edu
objectClass: posixGroup
objectClass: top
cn: cs251stu
gidNumber: 2726
memberUid: gdome
...
delete? y
[root@tempest acctmp] ldapadd2 cs251stu.ldif 
adding new entry "cn=cs251stu,ou=Group,dc=cs,dc=wellesley,dc=edu"
[root@tempest acctmp]

Works!

Even easier, I now have a script “groupupdate” which reads the (updated) entry from /etc/group and does the rest, so just edit /etc/group and use the script.

Upgrading Firefox on a Mac or Windows machine that isn’t shared with others is usually a piece of cake:  click on the “upgrade” button that the browser gives you and follow the directions.  Upgrading on a Linux workstation, particularly one that is shared by many users, isn’t quite as easy, though it’s not too hard.

As far as I can tell, the Firefox desktop icon on CentOS and Fedora simply launches the “firefox” command in a shell.  This means that we can intercept that and replace it with an upgraded version of firefox.  In particular, the “firefox” RPM from CentOS installs to /usr/bin/firefox.  Here’s the data on an un-upgraded CentOS 6.3 machine:

[root@irwin ~] cat /etc/issue
CentOS release 6.3 (Final)
[root@irwin ~] rpm -q firefox
firefox-10.0.7-1.el6.centos.i686
[root@irwin ~] rpm -ql firefox | grep bin/
/usr/bin/firefox
[root@irwin ~] which firefox
/usr/bin/firefox
[root@irwin ~] /usr/bin/firefox -version
Mozilla Firefox 10.0.7
[root@irwin ~]

Firefox 10.0.7 is very out-of-date, so we want to upgrade right away, and we’ll want to keep upgrading as new versions come out.

If you go to getfirefox.com, and click on the appropriate buttons, it will download a .bz2 file to your home directory.  Unzipping that file results in a .tar file that create a directory called “firefox” that contains a program called “firefox.”  So, do the following:

[root@irwin tmp] cp firefox-18.0.tar.bz2 /usr/local/
[root@irwin tmp] cd /usr/local
[root@irwin local] bzcat firefox-18.0.tar.bz2 | tar xf -
[root@irwin local] ls -l firefox/firefox
-rwxr-xr-x 1 root root 96524 Jan  4 19:44 firefox/firefox
[root@irwin local] ./firefox/firefox -version
Mozilla Firefox 18.0
[root@irwin local] cd /usr/local/bin
[root@irwin bin] ln -s ../firefox/firefox .
[root@irwin bin] which firefox
/usr/local/bin/firefox
[root@irwin bin] firefox -version
Mozilla Firefox 18.0
[root@irwin bin]

Good.  Firefox 18.0 is much better and this last step shows that we succeeded.

I’ve written a script called upgrade-firefox to automate this.

Randy recently asked me to add some students to the cs251stu group.  There is a script that does that, addLDAPusertogroup.pl, but it (1) doesn’t add anything to the entry in /etc/group, and (2) just appends new members.  In this case, the member list was getting kinda long, including many students who had graduated or simply taken the course in the past.

So, I did the operation as follows:

* edit /etc/group to remove the old entries and put the new entries in

* extract just that line, say by grep “^cs251stu:” /etc/group > /root/acctmp/cs251stu.group

* convert that to LDIF:  migrate_group.pl cs251stu.group > cs251stu.ldif

* delete the old group from LDAP:  ldapdelgroup cs251stu

* add the new group to LDAP:  ldapadd2 cs251stu.lif

* confirm the results:  ldapsearchgroup cs251stu

Here’s an edited history:

1001 grep cs251stu /etc/group > /root/acctmp/cs251stu.group
1002 emacs -nw /root/acctmp/cs251stu.group
1010 migrate_group.pl cs251stu.group > cs251stu.ldif
1017 ldapsearchgroup cs251stu
1018 ldapdelgroup cs251stu
1020 ldapadd2 cs251stu.ldif 
1021 ldapsearchgroup cs251stu

We should think about ways to automate this process a bit better.

Scott

Mike Dawson did the early detective work for this:

A message from Larry Baldwin that we need the XMLWriter class compiled into PHP to enable a feature that he wants.  Is that hard to arrange?  The word “compile” scares me but is it just a matter of enabling/installing a module?

The cs304 phpinfo page has ‘–disable-xmlwriter’ in the Configure Command section, but XMLWriter enabled elsewhere.  It’s also enabled in the file
/etc/php.d/xmlwriter.ini.

Oh, wait, that’s on puma, but Larry’s PHP is on tempest, right?

This may be helpful, the person solves the same problem by installing the php-xml package:

http://www.limesurvey.org/de/forum/installation-a-update-issues/59119-xmlwriter-class-not-enabled

That page suggests that installing php-xml, which was easy enough:

[root@tempest cgi-bin] yum search php-xml
php-xml.x86_64 : A module for PHP applications which use XML
php-xmlrpc.x86_64 : A module for PHP applications which use the XML-RPC protocol
[root@tempest cgi-bin] rpm -q php-xml
package php-xml is not installed
[root@tempest cgi-bin] rpm -q php-xmlrpc
package php-xmlrpc is not installed
[root@tempest cgi-bin] yum -y install php-xml php-xmlrpc
Installed:
  php-xml.x86_64 0:5.3.3-14.el6_3                                  php-xmlrpc.x86_64 0:5.3.3-14.el6_3                                 

Complete!
[root@tempest cgi-bin] rpm -q libxslt
libxslt-1.1.26-2.el6_3.1.x86_64
[root@tempest cgi-bin]

Okay, now time to check on xmlwriter in the phpinfo page for Tempest:  http://tempest.wellesley.edu/LocalOnly/phpinfo.php

Ah, unfortunately, Tempest’s php has –disable-xmlwriter while the person who posted the limesurvey forum link above didn’t have that.  So it looks like we’ll have to compile PHP from source.  Yuck!  May want to consult RedHat so that we do this right.  Could also try it on one of our CentOS 6.3 client machines as a test machine.

Scott

Brian Tjaden was having some trouble with CGI scripts running on Puma and on Tempest.  Since the trouble was also on Puma, we could ignore selinux as the problem.

He could run simple CGI/Python scripts, but the trouble came when he tried to call out to another script or program from his Python script.  The error he was getting was “permission denied” The examples we worked with were hello.cgi and hello2.py.  Here’s the CGI script, after a few iterations:

#!/usr/local/bin/python2.6

import sys,os,subprocess

print ("Content-Type: text/html\n")

#os.chdir("../TargetRNA2/trash/")
#os.system("java Trash &> output.txt")
#os.system("./RNAfold -version &> output.txt")
out_file = open("/var/tmp/output.txt", "w")
os.chmod("/var/tmp/output.txt",0666)

#subprocess.call(["/usr/bin/python","./hello2.py"], stdin=None, stdout=out_file, stderr=subprocess.STDOUT)
result = subprocess.call(["/home/btjaden/public_html/cgi-bin/blastn"], stdin=None, stdout=out_file, stderr=subprocess.STDOUT, shell=Tr
ue)
#result = subprocess.call(["/home/btjaden/public_html/cgi-bin/blastn"], stdin=None, stdout=out_file, stderr=subprocess.STDOUT)
#result = subprocess.call(["/home/btjaden/public_html/cgi-bin/blastn"], stdin=None)
print "<p>",result
print "<p>euid is ",os.geteuid()
#subprocess.call(["./hello2.py"], stdin=None, stdout=out_file, stderr=subprocess.STDOUT, shell=True)
#subprocess.call(["blastn"], stdin=None, stdout=out_file, stderr=subprocess.STDOUT, shell=True)
out_file.close()

The solution to calling out to an external python script was to include the Python interpreter as the first argument.

The next problem was to call out to an external binary (ELF) file.  We got that to work on Puma by adding the shell=True keyword argument. We’re not sure why that works, but it does.

However, it didn’t work on Tempest.  Again, the error was “permission denied.”  Since the filesystems on Puma and Tempest are the same, it had to be something else.  Ah, yes, it’s SELinux.  There are a few of the log entries:

[root@tempest ~] grep denied /var/log/audit/audit.log | grep blastn | tail -3
type=AVC msg=audit(1355349606.090:272209): avc:  denied  { execute } for  pid=7945 comm="hello.cgi" name="blastn" dev=dm-4 ino=5151086 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file
type=AVC msg=audit(1355349789.856:272288): avc:  denied  { execute } for  pid=8091 comm="sh" name="blastn" dev=dm-4 ino=5151086 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file
type=AVC msg=audit(1355349789.856:272289): avc:  denied  { execute } for  pid=8091 comm="sh" name="blastn" dev=dm-4 ino=5151086 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file
[root@tempest ~]

The timestamps on these are hard to read (the number starting 1355… is the seconds since the beginning of the epoch), but there’s a program called ausearch that will also reformat them in more human-readable form:

[root@tempest cgi-bin] grep denied /var/log/audit/audit.log | grep blastn | ausearch -i --just-one
----
type=AVC msg=audit(12/12/2012 14:17:57.886:271123) : avc:  denied  { execute } for  pid=4974 comm=sh name=blastn dev=dm-4 ino=5151086 scontext=unconfined_u:system_r:httpd_sys_script_t:s0 tcontext=unconfined_u:object_r:httpd_user_content_t:s0 tclass=file

There are more sophisticated ways to use ausearch (different search criteria), but this is enough for now.

Since the issue is the SElinux context, and Brian’s CGI scripts can run python but not blastn, I was curious about that.  The -Z argument to “ls” lists the SElinux context:

[root@tempest cgi-bin] ls -Z /usr/bin/python ./blastn 
-rwxr-xr-x. btjaden btjaden unconfined_u:object_r:httpd_user_content_t:s0 ./blastn
-rwxr-xr-x. root    root    system_u:object_r:bin_t:s0       /usr/bin/python
[root@tempest cgi-bin]

It’s hard to know what’s critical here, but maybe setroubleshoot/sealert will help.

Alas, sealert is not helping.  It shows a denial where java is prevented from running blastn, but that isn’t what’s happening.  Very strange.  Also, the solution isn’t a simple “seboolean” kind of solution, which is what I expected.  More work is needed.

Irwin recently experienced one of those random crashes and currently is in a bad state.  One symptom:

[root@tempest ~] ssh irwin
Last login: Tue Dec 11 14:31:20 2012 from puma.wellesley.edu
[root@irwin ~] getent passwd anderson

So, somehow one of the essential authentication services has failed.  What could it be?  Thrush is working fine, so I did the following on both irwin and thrush

[root@irwin ~] service --status-all > /usr/network/tmp/services-irwin 
JAVA_EXECUTABLE or HSQLDB_JAR_PATH in '/etc/sysconfig/hsqldb' is set to a non-file.
No sensors found!
Make sure you loaded all the kernel drivers you need.
Try sensors-detect to find out which these are.

Now, compare:

[root@irwin tmp] diff services-irwin services-thrush | grep -v pid
1,7c1,7
< auditd is stopped
---
9,11c9
< Running
< cgred is stopped
---
13,17c11,15
< 2010
---
> dnsmasq is stopped
> 2014

Hmm. That’s interesting. I would have expected something involving sssd or ldap.  Let’s look specifically at the failing command:

[root@irwin tmp] strace getent passwd anderson

close(3)                                = 0
munmap(0xb7484000, 99604)               = 0
getpid()                                = 15639
fstat64(-1, 0xbff2f278)                 = -1 EBADF (Bad file descriptor)
time(NULL)                              = 1355255435
socket(PF_FILE, SOCK_STREAM, 0)         = 3
fcntl64(3, F_GETFL)                     = 0x2 (flags O_RDWR)
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK)  = 0
fcntl64(3, F_GETFD)                     = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
connect(3, {sa_family=AF_FILE, path="/var/lib/sss/pipes/nss"}, 110) = -1 ECONNREFUSED (Connection refused)
close(3)                                = 0
exit_group(2)

Connection refused?  By whom?  Why?

[root@irwin tmp] cd /var/log/sssd/
[root@irwin sssd] grep 'connection refused' *
[root@irwin sssd] grep 'connection' *
[root@irwin sssd] grep 'refused' *
[root@irwin sssd] cd ..
[root@irwin log] grep 'refused' *
[root@irwin log] grep 'connection' *
anaconda.syslog:16:24:25,558 NOTICE NetworkManager:    ifcfg-rh:     read connection 'System eth0'
anaconda.syslog:16:24:25,558 NOTICE NetworkManager:    ifcfg-rh: Ignoring connection 'System eth0' and its device due to NM_CONTROLLED/BRIDGE/VLAN.
anaconda.syslog:16:24:26,555 WARNING NetworkManager: <warn> error requesting auth for org.freedesktop.NetworkManager.use-user-connections: (26) Remote Exception invoking org.freedesktop.PolicyKit1.Authority.CheckAuthorization() on /org/freedesktop/PolicyKit1/Authority at name org.freedesktop.PolicyKit1: org.freedesktop.DBus.Error.Spawn.ExecFailed: Cannot launch daemon, file not found or permissions invalid
anaconda.syslog:16:25:05,936 NOTICE NetworkManager:    ifcfg-rh: Managing connection 'System eth0' and its device because NM_CONTROLLED was true.
[root@irwin log]

So, not logged by the client, it seems.  What’s with this pipe?

[root@irwin log] lsof | grep pipes
sssd       1851      root   13u     unix 0xdf043500      0t0      10659 /var/lib/sss/pipes/private/sbus-monitor
sssd       1851      root   15u     unix 0xf56a0ac0      0t0      10779 /var/lib/sss/pipes/private/sbus-monitor
sssd       1851      root   17u     unix 0xdf040580      0t0      10842 /var/lib/sss/pipes/private/sbus-monitor
sssd_be    1862      root   22u     unix 0xf56a04c0      0t0      10780 /var/lib/sss/pipes/private/sbus-dp_LDAP.1862
sssd_be    1862      root   30u     unix 0xf5a5f480      0t0      10844 /var/lib/sss/pipes/private/sbus-dp_LDAP.1862
sssd_pam   1893      root   23u     unix 0xdf040780      0t0      10845 /var/lib/sss/pipes/pam
sssd_pam   1893      root   24u     unix 0xdf040980      0t0      10847 /var/lib/sss/pipes/private/pam
[root@irwin log] service sssd status
sssd (pid  1851) is running...
[root@irwin log]

Hmm.  I expected nss to be there.  What happens if we just restart sssd?

[root@irwin log] service sssd restart
Stopping sssd:                                             [  OK  ]
Starting sssd:                                             [  OK  ]
[root@irwin log] getent passwd anderson
anderson:*:716:501:Scott D. Anderson,E114:/home/anderson:/bin/bash
[root@irwin log]

Well, that’s good, I guess.  At least we can do this remotely.  We could also set up a cron job that just restarts sssd every hour or so.  Can anyone think of anything better?  Can we tell when sssd gets in trouble?

[root@irwin log] grep sssd /var/log/messages
Dec 10 22:42:37 irwin abrt[9800]: Saved core dump of pid 1892 (/usr/libexec/sssd/sssd_nss) to /var/spool/abrt/ccpp-2012-12-10-22:42:37-1892 (819200 bytes)
Dec 10 22:42:38 irwin sssd[nss]: Starting up
Dec 10 22:43:09 irwin abrt[9979]: Saved core dump of pid 9801 (/usr/libexec/sssd/sssd_nss) to /var/spool/abrt/ccpp-2012-12-10-22:43:09-9801 (712704 bytes)
Dec 10 22:43:11 irwin sssd[nss]: Starting up
Dec 10 22:43:34 irwin abrt[10156]: Saved core dump of pid 9980 (/usr/libexec/sssd/sssd_nss) to /var/spool/abrt/ccpp-2012-12-10-22:43:34-9980 (712704 bytes)
Dec 10 22:43:36 irwin sssd[nss]: Starting up
Dec 10 22:43:58 irwin abrt[10333]: Saved core dump of pid 10157 (/usr/libexec/sssd/sssd_nss) to /var/spool/abrt/ccpp-2012-12-10-22:43:58-10157 (712704 bytes)
Dec 10 22:44:00 irwin sssd[nss]: Starting up
Dec 10 22:44:23 irwin abrt[10512]: Saved core dump of pid 10334 (/usr/libexec/sssd/sssd_nss) to /var/spool/abrt/ccpp-2012-12-10-22:44:23-10334 (712704 bytes)
Dec 11 14:56:06 irwin sssd[pam]: Shutting down
Dec 11 14:56:06 irwin sssd[be[LDAP]]: Shutting down
Dec 11 14:56:06 irwin sssd: Starting up
Dec 11 14:56:06 irwin sssd[be[LDAP]]: Starting up
Dec 11 14:56:06 irwin sssd[nss]: Starting up
Dec 11 14:56:06 irwin sssd[pam]: Starting up
[root@irwin log]

Oh, that’s interesting.  And we see:

[root@irwin log] cd /var/spool/abrt
[root@irwin abrt] ls
abrt-db                        ccpp-2012-10-02-21:46:07-1899   ccpp-2012-12-10-22:42:37-1892
ccpp-2012-10-02-21:45:13-3727  ccpp-2012-12-10-22:42:28-17063  last-ccpp
[root@irwin abrt] ls -lt
total 20
drwxr-x---  2 abrt root 4096 Dec 10 22:45 ccpp-2012-12-10-22:42:37-1892
drwxr-xr-x  3 root root 4096 Dec 10 22:45 ccpp-2012-10-02-21:45:13-3727
drwxr-x---  2 abrt root 4096 Dec 10 22:45 ccpp-2012-10-02-21:46:07-1899
drwxr-x---  2 abrt gdm  4096 Dec 10 22:45 ccpp-2012-12-10-22:42:28-17063
-rw-------  1 root root   26 Dec 10 22:44 last-ccpp
-rw-r--r--. 1 root root    0 Aug 29 12:53 abrt-db
[root@irwin abrt] cd ccpp-2012-12-10-22\:42\:37-1892/
[root@irwin ccpp-2012-12-10-22:42:37-1892] ls
abrt_version  component  environ     limits      package  sosreport.tar.xz  uuid
analyzer      coredump   executable  maps        pid      time              var_log_messages
architecture  count      hostname    open_fds    pwd      uid
cmdline       dso_list   kernel      os_release  reason   username
[root@irwin ccpp-2012-12-10-22:42:37-1892] more cmdline 
/usr/libexec/sssd/sssd_nss --debug-to-files
[root@irwin ccpp-2012-12-10-22:42:37-1892] more reason 
Process /usr/libexec/sssd/sssd_nss was killed by signal 6 (SIGABRT)
[root@irwin ccpp-2012-12-10-22:42:37-1892] more var_log_messages 
Dec 10 22:42:37 irwin abrt[9800]: Saved core dump of pid 1892 (/usr/libexec/sssd/sssd_nss) to /var/spool
/abrt/ccpp-2012-12-10-22:42:37-1892 (819200 bytes)
Dec 10 22:43:09 irwin abrt[9979]: Saved core dump of pid 9801 (/usr/libexec/sssd/sssd_nss) to /var/spool
/abrt/ccpp-2012-12-10-22:43:09-9801 (712704 bytes)

Okay, so nothing diagnostic in here, but interesting nevertheless.  Googling for “sssd_nss sigabrt” yields some results like:

https://bugzilla.redhat.com/show_bug.cgi?format=multiple&id...
Nov 13, 2011 – Summary: sssd_nss crashes when passed invalid UTF-8 
for the .../usr/libexec/sssd/sssd_nss was killed by signal 6 (SIGABRT) 
time: Sun Nov ...

[root@irwin ccpp-2012-12-10-22:42:37-1892] rpm -q sssd
sssd-1.8.0-32.el6.i686