Salt-minion fails to start


#1

Thanks to the guys over at BSD Now, found a tutorial on setting up and running saltstack (I like python better than ruby).

So I was walking through the tutorial, got the saltmaster configured and started. It is running in a jail on my desktop machine. But when I tried to set up the first minion on my laptop, it failed. The first attempt at starting it gave me

[root@yukon salt]# service salt_minion start

  • Caching service dependencies …
    /libexec/rc/sh/gendepends.sh: shell_var: not found
    /libexec/rc/sh/gendepends.sh: shell_var: not found
    /libexec/rc/sh/gendepends.sh: shell_var: not found [ ok ]
    /libexec/rc/sh/openrc-run.sh: ebegin: not found
    /libexec/rc/sh/openrc-run.sh: eend: not found
  • ERROR: salt_minion failed to start

Subsequent attempt gave

[root@yukon salt]# service salt_minion restart
/libexec/rc/sh/openrc-run.sh: ebegin: not found
/libexec/rc/sh/openrc-run.sh: eend: not found

  • ERROR: salt_minion failed to start

Never seen the ebegin and eend errors before. Haven’t really dug in to openrc, so I am throwing myself on the mercy of the court. :slight_smile:


#2

Are You talking about this?

% pkg search 'py.*salt'
py27-salt-2017.7.2             Distributed remote execution and configuration management system

#3

That almost sounds like the OpenRC service has the wrong shebang at the top:
If it has #!/bin/sh as the first line, you need to change that to #!/sbin/openrc-run.
That will enable all the OpenRC-specific definitions like “ebegin” and “eend”


#4

Thanks Ken. I checked the files (salt_api salt_master salt_minion salt_proxy salt_syndic) I checked, and the ones in /usr/local/etc/init.d all had #!/sbin/openrc-run, and the ones in /usr/local/etc/rc.d have #!/bin/sh.


#5

I finally got a few more minutes play time wiht the salt problem. On my FreeBSD 11.0-RELEASE jails, salt minion seems to be running fine, just like the salt master which is also running in a jail.

It’s just the ones running on TrueOS that are having a problem. @beanpole135, your thought about the incorrect shebang sounds like it may be the case.

As I said above, the ones in /usr/local/etc/init.d had openrc-run shebangs, and the ones in /usr/local/etc/rc.d is /bin/sh Could putting

salt_minion_enable=“YES”

in /etc/rc.d be confusing the system into calling openrc but pointing it to the init file in /usr/local/etc/rc.d?


#6

I don’t think that “[service]_enable=YES” is causing any problems - those get ignored on TrueOS systems because OpenRC looks in a different place for which services need to be enabled on boot rc-update add [service].
I am not seeing the salt/client package on my system right now, could you copy/paste the OpenRC service for it here so I can look it over? It might be something really simple that just was not converted properly from the rc.d service.


#7

Thanks Ken.

$ cat /usr/local/etc/init.d/salt_minion 
#!/sbin/openrc-run

# Add the following to /etc/rc.conf[.local] to enable this service
#
# salt_minion_paths (string):      Set to "/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin" by default.
#               Default $PATH for salt_minion
# salt_minion_eggcache (string):   Set to "/tmp" by default.
#               Allows defining egg cache directory to fix runtime on diskless systems.

name=salt_minion

: ${salt_minion_paths=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin}
: ${salt_minion_pidfile:=/var/run/salt-minion.pid}
: ${salt_minion_eggcache=/tmp}

command="/usr/local/bin/salt-minion"
required_files="/usr/local/etc/salt"
command_args="-c ${required_files} -d"
pidfile=${salt_minion_pidfile}

export PATH="${salt_minion_paths}"
export PYTHON_EGG_CACHE="${salt_minion_eggcache}"

depend() {
        keyword -shutdown
}

#8

ok, I have not tested this yet, but see copy this over as your /usr/local/etc/init.d/salt_minion service and see if it starts working now:

#!/sbin/openrc-run

# Configuration options for this service
# =======================
# salt_minion_paths (string):      Set to "/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin" by default.
#               Default $PATH for salt_minion
# salt_minion_eggcache (string):   Set to "/tmp" by default.
#               Allows defining egg cache directory to fix runtime on diskless systems.

name=salt_minion
command="/usr/local/bin/salt-minion"
required_dirs="/usr/local/etc/salt"
command_args="-c ${required_dirs}"
pidfile="${salt_minion_pidfile:-/var/run/salt-minion.pid}"

start_stop_daemon_args=" -e PATH=${salt_minion_paths:-/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin} -e PYTHON_EGG_CACHE=\"${salt_minion_eggcache:-/tmp}\""

depend() {
	keyword -shutdown
}

The main things I changed were the default variable expansions (probably fine as it was before - just cleaning things up), convert “required_files” to “required_dirs” (since /usr/local/etc/salt is a directory, not a file), and move the “export” lines into environment options for start-stop-daemon (-e FOO=BAR).

EDIT: I just removed the “-d” argument to the salt-minion utility as well: With OpenRC the daemon-ization is managed by OpenRC directly - the utility no longer needs to manually daemonize/background itself.


#9

Thank you again, Ken. It works better than it did before, but I’m not sure it is completely working. When I did a salt_minion restart, it hung there. In another window, after about 5 minutes, I did:

[storm@yukon ~]$ service salt_minion status

  • status: starting

Looking at processes, I have

[storm@yukon ports]$ ps auxwww | grep salt
root 3815 0.0 0.6 63340 48704 - Is 07:13 0:01.03 /usr/local/bin/python2.7 /usr/local/bin/salt-minion -c /usr/local/etc/salt
root 3839 0.0 0.8 83460 60456 - S 07:13 0:00.85 /usr/local/bin/python2.7 /usr/local/bin/salt-minion -c /usr/local/etc/salt
root 3840 0.0 0.6 65816 49124 - I 07:13 0:00.00 /usr/local/bin/python2.7 /usr/local/bin/salt-minion -c /usr/local/etc/salt
root 3473 0.0 0.0 11296 2192 v0 I+ 07:13 0:00.01 /sbin/openrc-run /usr/local/etc/init.d/salt_minion --lockfd 26 start
root 3805 0.0 0.0 12048 2280 v0 I+ 07:13 0:00.01 /bin/sh /libexec/rc/sh/openrc-run.sh /usr/local/etc/init.d/salt_minion start
root 3813 0.0 0.0 11500 2216 v0 I+ 07:13 0:00.00 start-stop-daemon --start --exec /usr/local/bin/salt-minion --pidfile /var/run/salt-minion.pid -e PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin -e PYTHON_EGG_CACHE=/tmp – -c /usr/local/etc/salt

So somewhere it is getting hung up, but at least it is starting it…albeit multiple times. :slight_smile:

Edit: I rebooted to start on a level playing field. salt_minion started to start onboot, but it stopped at the same place. Status starting, etc…Just like above.


#10

Ok, that might mean that the application does need to be backgrounded then.
Try putthing this line in at the top really quick:

command_background="true"

That will tell OpenRC to force the daemon into the background (and requires the pidfile flags, but you already have those set).


#11

The other alternative is to tell salt_minion to go into the background itself, but in a way that OpenRC knows about it (alternative to “command_background”)

command_args_background="-d --pidfile ${pidfile}"

or something like that - I don’t have the man page for the salt_minion utility handy at the moment. You just need to ensure that you can pass the pidfile to it somehow in addition to the daemonize flag (-d).


#12

I added the line after pidfile=

Gettning ready to try it, but when I tried to stop salt_minion, I got:

[root@yukon ~]# service salt_minion stop

  • Call to flock failed: Resource temporarily unavailable
  • ERROR: salt_minion stopped by something else

Yet the status is still “starting” and there are all of the processes listed in my previous message.

Apparently that was what it needed. it came back instantly, and the status is "started.

Thanks Ken. Just out of curiosity, I have modified the one for salt_master (though none of the others yet). How do these get added to the distro?


#13

That stop error was probably a holdover from when the previous version(s) of the service were being used.
If you do a full reboot it will clean out any leftover artifacts like that, and then do some tests on your service to ensure it is working properly:

  1. start service -> verify that the utility is indeed running and doing what it should
  2. stop service - > verify that the utility was indeed stopped and no leftover processes are still hanging around

Once you are satisfied that both those checks are passing, go ahead and either send me the new service file(s), or submit them as a pull request on GitHub (https://github.com/trueos/freebsd-ports/tree/trueos-master/sysutils/py-salt/files). The OpenRC services are the files called “openrc-salt_[something].in”. You can even edit/submit changes to those files directly through the GitHub web interface (much easier than all the CLI commands for git).


#14

Ken, I think I set up the pull request. Can you check for VulcanRidr-patch-1? I am still very, very new to git.


#15

@VulcanRidr : I do not see any pull requests from you yet.
The basic methodology to create one is:

  1. Fork the repo to your personal github account
  2. Clone that fork on your local box
  3. Make a new branch for the repo, make/commit your changes
  4. Push your local commits back to your account on github (you should be able to view the changes in your browser at this point)
  5. In the Github web interface, click the button to create a pull request for those changes back to the original repository.

That was the full/longer method for it, but the entire process can be streamlined just by using the web interface exclusively.

  1. Go to the official repository/file on Github in your browser.
  2. Click the “edit” button in the upper-right corner when viewing an individual file (you need to be logged in to Github first).
  3. Make the changes in the editor (copy/paste from the working local file on your system is how I do it).
  4. Add a commit message at the bottom of the page and click commit. Github will automatically perform the fork/branch for you and open a pull request automatically.

#16

Okay, I’m showing two pull requests. Do you see them?


#17

Yep - I just merged in one of the pull requests. I will get to the other one after a meeting I have in a couple minutes.


#18

ok, the second one is merged in now. Thanks for testing/sending in those fixes!