puppet module gets 'Permission denied' applying NFS-based home directories - linux

Question

Some disclosure:
I'm using a master/agent setup in which I own the agent but do not have permission to the master console. The puppetmaster is git-backed, and I control the source for the module(s) in question.
I have 2 relevant modules for my question. One of them, which appears to work just fine, ensures autofs is installed and has 2 file resources for auto.master and a custom auto.home to mount home directories.
#auto.home
#this file is used by auto.master to automount home directories
#from the nfs cluster when a user logs in.
* -fstype=nfs,rw,nosuid,soft <IPaddress>:/homedirs/&
In the module to add home directories, I'm creating users and deploying their public ssh keys via a file resource. This module "works" on systems when I comment out the Class dependency and I'm not mounting /home to NFS, and it *sometimes works when I'm deploying it as-is over NFS.
define local_user(
$fullname,
$username = $title,
$userid,
$gid = 9999,
$homedir_mode = 0700
) {
$white_gid = $gid
user { $username:
ensure => present,
comment => $fullname,
gid => $white_gid,
uid => $userid,
home => $homedir,
require => Group[ "white" ],
}
exec { "chage -M 99999 ${username}":
command => "chage -M 99999 ${username}",
path => "/bin:/sbin:/usr/bin:/usr/sbin",
# chage(1) only works on local users, not on LDAP users,
# so make sure this is a local user before we try to
# change their password expiration.
onlyif => "grep -q '^${username}:' /etc/passwd",
subscribe => User[ $username ],
refreshonly => true,
}
file { $homedir:
ensure => directory,
owner => $username,
group => $white_gid,
mode => $homedir_mode,
require => User[ $username ],
}
file { "$homedir/.ssh":
ensure => directory,
owner => $username,
group => $white_gid,
mode => 0700,
require => File[ "$homedir" ],
}
file { "$homedir/.ssh/authorized_keys":
ensure => present,
owner => $username,
group => $white_gid,
mode => 0600,
source => "puppet:///modules/ssh_keys/${username}_authorized_keys",
require => File["$homedir/.ssh"],
}
}
class ssh_keys {
group { "white":
ensure => present,
gid => 9999,
require => Class["nfs_homedirs"],
}
#### add users below this line
local_user { "userA" : fullname => "userA", userid => "123" }
Some things I'm puzzled by and could use expertise with:
In order for the NFS home directories to work at all, I had to run the module on a machine to create the users locally, then mount the root directory of the NFS mount for home directories and create those user's folders owned by their uid/gid for the autofs to actually work when they log in.
When the module fails to "work" against the NFS-mounted home directories, the error is 'Permission denied' when it tries to create the home folder. I've tried no_root_squash to combat the error, but to no avail. I have tried running the agent as root, as not-root via sudo, as not-root at all, etc.
Error:
/Stage[main]/Ssh_keys/Local_user[userA]/File[/home/userA]/ensure:
change from absent to directory failed: Could not set 'directory' on ensure:
Permission denied - /home/userA at
80:/app/puppet/conf/environments/puppet_dev/modules/ssh_keys/manifests/init.pp
It's seemingly harmless to put ensure => present statements on these directories and file resources. They're technically already created on the NFS share, but the way autofs seems to work is that it won't actually "mount" that user's share until they login. It's not my expertise, but that's what I experience. When this module does run successfully, every user's home directory it creates shows as a mount in the df output.
I suspect that there's something on the machine itself that's preventing this module from working the way it should. Knowing that there are probably 500 things that I could diff between a machine where this module runs clean and one where it doesn't, what are some places I should investigate?
Any assistance would be greatly appreciated.

Answer 1

The way auto.home works is to mount the directory when the user logs in. If the user hasn't logged in, no mount exists -- and thus your directory/file resources fail.
Personally I wouldn't try creating home directories over an nfs mount. Plus you don't want multiple servers trying to manage the same physical resources. Split this out to run only on the NFS server if possible and run all your file resources related to the home directories there. Have the nfs clients just ensure nfs is configured and the local user accounts exist.
If you can't run puppet on the NFS server, pick 1 server to mount it as a regular mount -- i.e. mount the root of the home dirs section so they are all visible. Have no_root_squash set also. Then you should be able to have puppet create the directories.
Also the ssh_authorized_key resource is handy. I use it often.

Answer 2

It sounds to me like selinux is being enforced, which would cause a permission denied as described even if you have the right user/uid owning the directories. If you have selinux enforced, then you'll want to check to see if using nfs_home_dirs is allowed. First, check by running:
getsebool use_nfs_home_dirs
If it comes back as use_nfs_home_dirs --> off, then you can either manually correct this using setsebool -P use_nfs_home_dirs 1, or you can use puppet to manage this as well:
include selinux
selinux::boolean {'use_nfs_home_dirs':
ensure => 'on',
}