We have a mystery on the CentOS workstations. ssh works, but su does not. Here’s an example:
[luser@gibbon ~] ssh anderson@gibbon anderson@gibbon's password: Last login: Wed Sep 26 10:56:41 2012 from gibbon.wellesley.edu [anderson@gibbon ~] logout Connection to gibbon closed. [luser@gibbon ~] su - anderson Password: su: incorrect password [luser@gibbon ~] su - anderson Password: su: incorrect password [luser@gibbon ~]
You can see that I double-checked to make sure I typed my password correctly on the su command. A difference like this probably points to a difference in pam. Since both su and ssh work on Sampras, an older machine running Fedora 14, I thought it made sense to compare their authentication configuration. I can do this by using authconfig on Gibbon and Sampras to dump out all the relevant files for both Gibbon and Sampras to a directory and then compare those directories:
[root@gibbon ~] authconfig --savebackup=/usr/network/tmp/authconfig-gibbon [root@gibbon ~] cd /usr/network/tmp [root@gibbon tmp] diff -qr authconfig-gibbon/ authconfig-sampras Files authconfig-gibbon/authconfig and authconfig-sampras/authconfig differ Files authconfig-gibbon/fingerprint-auth-ac and authconfig-sampras/fingerprint-auth-ac differ Files authconfig-gibbon/login.defs and authconfig-sampras/login.defs differ Files authconfig-gibbon/network and authconfig-sampras/network differ Only in authconfig-sampras: nss_ldap.conf Files authconfig-gibbon/nsswitch.conf and authconfig-sampras/nsswitch.conf differ Files authconfig-gibbon/openldap.conf and authconfig-sampras/openldap.conf differ Files authconfig-gibbon/pam_ldap.conf and authconfig-sampras/pam_ldap.conf differ Only in authconfig-sampras: pam_pkcs11.conf Files authconfig-gibbon/smartcard-auth-ac and authconfig-sampras/smartcard-auth-ac differ Files authconfig-gibbon/smb.conf and authconfig-sampras/smb.conf differ Files authconfig-gibbon/system-auth-ac and authconfig-sampras/system-auth-ac differ [root@gibbon tmp]
Now, many of these differences are irrelevant. For example, we don’t use any fingerprint or smartcard stuff, so we can ignore those. Let’s start with authconfig, which certainly seems relevant:
[root@gibbon tmp] diff authconfig-gibbon/authconfig authconfig-sampras/authconfig 1d0 < IPADOMAINJOINED=no 8d6 < USESSSD=no 14,15c12,13 < USELDAPAUTH=no < IPAV2NONTP=no --- > USEDB=no > USELDAPAUTH=yes 18d15 < USEIPAV2=no 23c20 < USEKERBEROS=yes --- > USEKERBEROS=no 25c22 < USEDB=no --- > USESSSD=no [root@gibbon tmp]
Hmmm. Well, the two files are in a different order, but they both say USESSSD=no, which is a little surprising because we use sssd, according to /etc/nsswitch.conf:
[root@gibbon tmp] service sssd status sssd (pid 1697) is running... [root@gibbon tmp] grep pass /etc/nsswitch.conf #passwd: db files nisplus nis passwd: files sss [root@gibbon tmp]
So, that’s puzzling. Still, Sampras works, and we’re looking for a difference, not necessarily looking to understand each item.
Hmm, I also notice that Gibbon has USELDAPAUTH=no, while Sampras has that set to yes. That’s the kind of difference we’re looking for. Is there a way to change that using authconfig or do we edit files by hand?
It turns out authconfig can also write out info in a different format:
[root@gibbon tmp] authconfig --test > gibbon-test1
It also turns out that there’s a command line arg called –enableldapauth which certainly sounds promising:
[root@gibbon tmp] authconfig --enableldapauth --test > gibbon-test2 [root@gibbon tmp] diff gibbon-test1 gibbon-test2 37c37 < pam_ldap is disabled --- > pam_ldap is enabled [root@gibbon tmp]
I don’t know if that will edit the authconfig file in the way we want, but it does seem promising. By the way, the “authconfig” file is in face /etc/sysconfig/authconfig, which is local specifications, so that does seem like the right file.
Let’s compare to sampras (I’m not showing the step where we spit out the authconfig data on sampras, but it’s the same command as on Gibbon.)
[root@gibbon tmp] grep pam_ldap sampras-test1 pam_ldap is enabled [root@gibbon tmp]
Oh, very promising. Let’s look more closely:
[root@gibbon tmp] diff gibbon-test1 sampras-test1 26d25 < nss_mdns4_minimal is disabled 31c30 < pam_krb5 is enabled --- > pam_krb5 is disabled 37c36 < pam_ldap is disabled --- > pam_ldap is enabled 44,45c43,44 < smartcard module = "" < smartcard removal action = "" --- > smartcard module = "coolkey" > smartcard removal action = "Ignore" 53c52 < credential caching in SSSD is disabled --- > credential caching in SSSD is enabled 55,59d53 < IPAv2 is disabled < IPAv2 domain was not joined < IPAv2 server = "" < IPAv2 realm = "" < IPAv2 domain = "" [root@gibbon tmp]
Interesting. So we should probably disable krb, since we don’t use kerberos, and we should probably enable sssd caching, for performance and consistency. Let’s try those changes:
[root@gibbon tmp] authconfig --enableldapauth --disablekrb5 --enablecachecreds --test > gibbon-test2 [root@gibbon tmp] diff gibbon-test2 sampras-test1 26d25 < nss_mdns4_minimal is disabled 44,45c43,44 < smartcard module = "" < smartcard removal action = "" --- > smartcard module = "coolkey" > smartcard removal action = "Ignore" 55,59d53 < IPAv2 is disabled < IPAv2 domain was not joined < IPAv2 server = "" < IPAv2 realm = "" < IPAv2 domain = "" [root@gibbon tmp]
This is looking really good. Okay, time to bite the bullet and go from –test to –update:
[root@gibbon tmp] authconfig --enableldapauth --disablekrb5 --enablecachecreds --update Starting sssd: [ OK ] [root@gibbon tmp]
Now, time to test this change:
[luser@gibbon ~] ssh anderson@gibbon anderson@gibbon's password: Permission denied, please try again. anderson@gibbon's password: Permission denied, please try again. anderson@gibbon's password: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password). [luser@gibbon ~]
*sigh*. Complete and utter failure. Somehow, our change means sshd no longer works. What about su?
[luser@gibbon ~] su - anderson Password: su: incorrect password [luser@gibbon ~]
Okay, no luck there. What changes did that command do? I saved the authconfig info in authconfig-gibbon2 and compared:
[root@gibbon tmp] diff -qr authconfig-gibbon authconfig-gibbon2 Files authconfig-gibbon/authconfig and authconfig-gibbon2/authconfig differ Files authconfig-gibbon/password-auth-ac and authconfig-gibbon2/password-auth-ac differ Files authconfig-gibbon/smartcard-auth-ac and authconfig-gibbon2/smartcard-auth-ac differ Files authconfig-gibbon/sssd.conf and authconfig-gibbon2/sssd.conf differ [root@gibbon tmp]
Not many, which is good. What are the exact differences?
[root@gibbon tmp] cd authconfig-gibbon [root@gibbon authconfig-gibbon] sort authconfig > authconfig.sorted [root@gibbon authconfig-gibbon] cd ../authconfig-gibbon2 [root@gibbon authconfig-gibbon2] sort authconfig > authconfig.sorted [root@gibbon authconfig-gibbon2] cd .. [root@gibbon tmp] diff authconfig-gibbon/authconfig.sorted authconfig-gibbon2/authconfig.sorted 12,13c12,13 < USEKERBEROS=yes < USELDAPAUTH=no --- > USEKERBEROS=no > USELDAPAUTH=yes [root@gibbon tmp]
Hmm. So turning *on* LDAPAUTH has disabled ssh. That’s weird. Maybe there’s some insight in the other files?
[root@gibbon tmp] diff authconfig-gibbon/password-auth-ac authconfig-gibbon2/password-auth-ac 7c7 < auth sufficient pam_ldap.so use_first_pass --- > auth sufficient pam_sss.so use_first_pass 13c13 < account [default=bad success=ok user_unknown=ignore] pam_ldap.so --- > account [default=bad success=ok user_unknown=ignore] pam_sss.so 18c18 < password sufficient pam_ldap.so use_authtok --- > password sufficient pam_sss.so use_authtok 23d22 < -session optional pam_systemd.so 26c25 < session optional pam_ldap.so --- > session optional pam_sss.so [root@gibbon tmp]
Very weird. Turning *on* LDAPAUTH means replacing pam_ldap with pam_sss in the password-auth-ac file. (I wonder what turning *off* LDAP would do?)
It’s weird that the smartcard-auth-ac file should change. What differences there?
[root@gibbon tmp] diff authconfig-gibbon/smartcard-auth-ac authconfig-gibbon2/smartcard-auth-ac 5,7c5 < auth [success=ok ignore=2 default=die] pam_pkcs11.so wait_for_card card_only < auth optional pam_krb5.so use_first_pass no_subsequent_prompt < auth sufficient pam_permit.so --- > auth [success=done ignore=ignore default=die] pam_pkcs11.so wait_for_card card_only [root@gibbon tmp]
Oh, probably from disabling kerberos. Okay, I guess that makes sense.
Next, of course, there’s the /etc/sssd.conf file. Let’s filter out the comments and compare:
[root@gibbon tmp] cd authconfig-gibbon [root@gibbon authconfig-gibbon] cat sssd.conf | egrep -v "#|;" | egrep "\w" > sssd.conf.min [root@gibbon authconfig-gibbon] cd ../authconfig-gibbon2 [root@gibbon authconfig-gibbon2] cat sssd.conf | egrep -v "#|;" | egrep "\w" > sssd.conf.min [root@gibbon authconfig-gibbon2] cd .. [root@gibbon tmp] diff authconfig-gibbon/sssd.conf.min authconfig-gibbon2/sssd.conf.min 0a1,11 > [domain/default] > ldap_id_use_start_tls = False > cache_credentials = True > ldap_search_base = dc=cs,dc=wellesley,dc=edu > krb5_realm = EXAMPLE.COM > krb5_server = kerberos.example.com > id_provider = ldap > auth_provider = ldap > chpass_provider = ldap > ldap_uri = ldap://tempest.wellesley.edu/ > ldap_tls_cacertdir = /etc/openldap/cacerts 6c17 < domains = LDAP --- > domains = default, LDAP [root@gibbon tmp]
Interesting. So, it added a new SSSD domain (default) and then configured it. Is it different from the one we already had?
[domain/LDAP] id_provider = ldap auth_provider = ldap ldap_id_use_start_tls = False cache_credentials = False enumerate = false ldap_search_base = dc=cs,dc=wellesley,dc=edu chpass_provider = ldap debug_level = 0 ldap_uri = ldap://tempest.wellesley.edu/ ldap_tls_cacertdir = /etc/openldap/cacerts min_id = 500 [root@gibbon tmp]
No, not much. Every variable in the new domain has the same value it had in the old domain. So, the trouble has to be with the sssd stuff in pam.d/password-auth-ac. Let’s compare with Sampras, the working machine:
[root@gibbon tmp] diff authconfig-gibbon2/password-auth-ac authconfig-sampras/password-auth-ac 7c7 < auth sufficient pam_sss.so use_first_pass --- > auth sufficient pam_ldap.so use_first_pass 13c13 < account [default=bad success=ok user_unknown=ignore] pam_sss.so --- > account [default=bad success=ok user_unknown=ignore] pam_ldap.so 18c18 < password sufficient pam_sss.so use_authtok --- > password sufficient pam_ldap.so use_authtok 22a23 > -session optional pam_systemd.so 25c26 < session optional pam_sss.so --- > session optional pam_ldap.so [root@gibbon tmp]
Yes, that seems to confirm it. Using pam_sss makes things not work. Let’s look particularly in the /etc/pam.d/ directory on Sampras and Gibbon. Sampras first:
[root@sampras ~] cd /etc/pam.d/ [root@sampras pam.d] more su #%PAM-1.0 auth sufficient pam_rootok.so # Uncomment the following line to implicitly trust users in the "wheel" group. #auth sufficient pam_wheel.so trust use_uid # Uncomment the following line to require a user to be in the "wheel" group. #auth required pam_wheel.so use_uid auth include system-auth account sufficient pam_succeed_if.so uid = 0 use_uid quiet account include system-auth password include system-auth session include system-auth session optional pam_xauth.so [root@sampras pam.d] more sshd #%PAM-1.0 auth required pam_sepermit.so auth include password-auth account required pam_nologin.so account include password-auth password include password-auth # pam_selinux.so close should be the first session rule session required pam_selinux.so close session required pam_loginuid.so # pam_selinux.so open should only be followed by sessions to be executed in the user context session required pam_selinux.so open env_params session optional pam_keyinit.so force revoke session include password-auth [root@sampras pam.d]
Okay, I don’t know why su uses system-auth and sshd uses password-auth. How do those files differ?
[root@sampras pam.d] grep sss system-auth password-auth [root@sampras pam.d] grep ldap system-auth password-auth system-auth:auth sufficient pam_ldap.so use_first_pass system-auth:account [default=bad success=ok user_unknown=ignore] pam_ldap.so system-auth:password sufficient pam_ldap.so use_authtok system-auth:session optional pam_ldap.so password-auth:auth sufficient pam_ldap.so use_first_pass password-auth:account [default=bad success=ok user_unknown=ignore] pam_ldap.so password-auth:password sufficient pam_ldap.so use_authtok password-auth:session optional pam_ldap.so [root@sampras pam.d]
Okay, so the working machine never uses sss and always uses ldap. What about Gibbon?
[root@gibbon ~] cd /etc/pam.d [root@gibbon pam.d] grep auth su auth sufficient pam_rootok.so #auth sufficient pam_wheel.so trust use_uid #auth required pam_wheel.so use_uid auth include system-auth account include system-auth password include system-auth session include system-auth session optional pam_xauth.so [root@gibbon pam.d] grep auth sshd auth required pam_sepermit.so auth include password-auth account include password-auth password include password-auth session include password-auth [root@gibbon pam.d]
Again, su uses system-auth and sshd uses password-auth. And how do those differ?
[root@gibbon pam.d] grep sss system-auth password-auth system-auth:auth sufficient pam_sss.so use_first_pass system-auth:account [default=bad success=ok user_unknown=ignore] pam_sss.so system-auth:password sufficient pam_sss.so use_authtok system-auth:session optional pam_sss.so password-auth:auth sufficient pam_sss.so use_first_pass password-auth:account [default=bad success=ok user_unknown=ignore] pam_sss.so password-auth:password sufficient pam_sss.so use_authtok password-auth:session optional pam_sss.so [root@gibbon pam.d] grep ldap system-auth password-auth [root@gibbon pam.d]
Yep, the both use sss and neither uses ldap, so they both fail. What was the previous, partially working version of Gibbon?
[root@gibbon tmp] grep sss authconfig-gibbon/system-auth-ac authconfig-gibbon/password-auth-ac authconfig-gibbon/system-auth-ac:auth sufficient pam_sss.so use_first_pass authconfig-gibbon/system-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_sss.so authconfig-gibbon/system-auth-ac:password sufficient pam_sss.so use_authtok authconfig-gibbon/system-auth-ac:session optional pam_sss.so [root@gibbon tmp] grep ldap authconfig-gibbon/system-auth-ac authconfig-gibbon/password-auth-ac authconfig-gibbon/password-auth-ac:auth sufficient pam_ldap.so use_first_pass authconfig-gibbon/password-auth-ac:account [default=bad success=ok user_unknown=ignore] pam_ldap.so authconfig-gibbon/password-auth-ac:password sufficient pam_ldap.so use_authtok authconfig-gibbon/password-auth-ac:session optional pam_ldap.so [root@gibbon tmp]
Yep, since sshd uses password-auth and password-auth used pam_ldap, that’s why that succeeded, while su uses system-auth and system-auth used pam_sss, so that failed.
Is sssd running? Any errors?
[root@gibbon tmp] service sssd status sssd (pid 1755) is running... [root@gibbon tmp] ls -lt /var/log/sssd/ total 12 -rw-------. 1 root root 327 Sep 26 11:59 sssd.log -rw-------. 1 root root 177 Sep 23 03:24 sssd.log-20120923.gz -rw-------. 1 root root 161 Sep 20 16:31 sssd.log-20120920.gz -rw-------. 1 root root 0 Sep 7 12:49 sssd_LDAP.log -rw-------. 1 root root 0 Sep 4 12:40 sssd_nss.log -rw-------. 1 root root 0 Sep 4 12:40 sssd_pam.log -rw-------. 1 root root 0 Sep 4 12:40 krb5_child.log -rw-------. 1 root root 0 Sep 4 12:40 sssd_default.log [root@gibbon tmp] tail /var/log/sssd/sssd.log (Mon Sep 24 14:48:58 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children (Wed Sep 26 11:36:26 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children (Wed Sep 26 11:59:58 2012) [sssd] [monitor_quit] (0x0010): Monitor received Terminated: terminating children [root@gibbon tmp]
So, no error messages from sssd.
I’m stumped for now. I’m going to revert the authconfig for gibbon so that at least sshd works and I’ll post to linuxquestions.org and hope for some help.
[root@gibbon tmp] authconfig --restorebackup=/usr/network/tmp/authconfig-gibbon [root@gibbon tmp] ssh anderson@gibbon anderson@gibbon's password: Last login: Wed Sep 26 10:56:56 2012 from gibbon.wellesley.edu [anderson@gibbon ~] logout Connection to gibbon closed. [root@gibbon tmp]
Great. That’s it for now.
Wait, I’m not giving up quite yet. On LinuxQuestions, I found this: http://www.linuxquestions.org/questions/linux-enterprise-47/rhel-6-ldap-now-requires-tls-843917/. So it has a few ideas:
[root@gibbon pam.d] authconfig --enableldap --enableldapauth --ldapserver=tempest.wellesley.edu --ldapbasedn="dc=tempest,dc=wellesley,dc=edu" --update Stopping sssd: [ OK ] [root@gibbon pam.d] emacs -nw /etc/sysconfig/authconfig [root@gibbon pam.d] grep FORCELEGACY /etc/sysconfig/authconfig FORCELEGACY=yes [root@gibbon pam.d]
Does that work? Nope, it doesn’t. Here’re the lines from /var/log/secure:
Sep 26 15:19:47 gibbon sshd[2613]: Invalid user anderson from 149.130.136.34 Sep 26 15:19:47 gibbon sshd[2614]: input_userauth_request: invalid user anderson Sep 26 15:19:49 gibbon sshd[2613]: pam_unix(sshd:auth): check pass; user unknown Sep 26 15:19:49 gibbon sshd[2613]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=gibbon.wellesley.edu Sep 26 15:19:49 gibbon sshd[2613]: pam_succeed_if(sshd:auth): error retrieving information about user anderson Sep 26 15:19:52 gibbon sshd[2613]: Failed password for invalid user anderson from 149.130.136.34 port 60323 ssh2 Sep 26 15:19:52 gibbon sshd[2614]: Connection closed by 149.130.136.34 [root@gibbon ~]
And the info from ldapsearch:
root@gibbon ~] ldapsearch -x "uid=anderson" # extended LDIF # # LDAPv3 # base <dc=cs,dc=wellesley,dc=edu> (default) with scope subtree # filter: uid=anderson # requesting: ALL # # anderson, People, cs.wellesley.edu dn: uid=anderson,ou=People,dc=cs,dc=wellesley,dc=edu uid: anderson cn: Scott D. Anderson objectClass: account objectClass: posixAccount objectClass: top objectClass: shadowAccount shadowLastChange: 12494 shadowMax: 99999 shadowWarning: 7 shadowFlag: 134523415 loginShell: /bin/bash uidNumber: 716 gidNumber: 501 homeDirectory: /home/anderson gecos: Scott D. Anderson,E114 # search result search: 2 result: 0 Success # numResponses: 2 # numEntries: 1 [root@gibbon ~]
And from getent:
getent passwd anderson [root@gibbon ~] grep passwd /etc/nsswitch.conf #passwd: db files nisplus nis passwd: files sss [root@gibbon ~] service sssd status sssd is stopped [root@gibbon ~]
Just start sssd?
[root@gibbon ~] getent passwd anderson anderson:*:716:501:Scott D. Anderson,E114:/home/anderson:/bin/bash [root@gibbon ~] ssh anderson@gibbon anderson@gibbon's password: Last login: Wed Sep 26 14:23:51 2012 from gibbon.wellesley.edu [anderson@gibbon ~]
Interesting. Can I also su?
[anderson@gibbon ~] su - anderson Password: su: incorrect password [anderson@gibbon ~] su - anderson Password: su: incorrect password [anderson@gibbon ~] logout Connection to gibbon closed. [root@gibbon ~] tail /var/log/secure Sep 26 15:24:12 gibbon sshd[2693]: Accepted password for anderson from 149.130.136.34 port 60328 ssh2 Sep 26 15:24:12 gibbon sshd[2693]: pam_unix(sshd:session): session opened for user anderson by (uid=0) Sep 26 15:24:18 gibbon su: pam_unix(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson Sep 26 15:24:18 gibbon su: pam_sss(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson Sep 26 15:24:18 gibbon su: pam_sss(su-l:auth): received for user anderson: 4 (System error) Sep 26 15:24:35 gibbon su: pam_unix(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson Sep 26 15:24:35 gibbon su: pam_sss(su-l:auth): authentication failure; logname=anderson uid=716 euid=0 tty=pts/1 ruser=anderson rhost= user=anderson Sep 26 15:24:35 gibbon su: pam_sss(su-l:auth): received for user anderson: 9 (Authentication service cannot retrieve authentication info) Sep 26 15:24:39 gibbon sshd[2695]: Received disconnect from 149.130.136.34: 11: disconnected by user Sep 26 15:24:39 gibbon sshd[2693]: pam_unix(sshd:session): session closed for user anderson [root@gibbon ~]
No, of course not. Reverting again.
One Response to ssh vs su on CentOS clients