TemplateNotFound: salt/formulas.jinja when running salt-ssh

[LYY]

I used salt-formula to install salt-minion on other ubuntu 14.04 servers.

master file_roots config:

file_roots:
   base:
     - /srv/salt
     - /data/salstack_forumals/salt-formula

version:

# salt-ssh --version
salt-ssh 2014.7.1 (Helium)

command:

salt-ssh 'es4log*' state.sls salt.minion

error:

----------
          ID: salt-minion
    Function: pkg.installed
      Result: True
     Comment: Package salt-minion is already installed.
     Started: 16:02:52.474110
    Duration: 1210.034 ms
     Changes:
----------
          ID: salt-minion
    Function: file.recurse
        Name: /etc/salt/minion.d
      Result: False
     Comment: #### /etc/salt/minion.d/_defaults.conf ####
              Unable to manage file: Jinja error: salt/formulas.jinja
              Traceback (most recent call last):
                File "/tmp/.root_76a713__salt/salt/utils/templates.py", line 286, in render_jinja_tmpl
                  output = template.render(**unicode_context)
                File "/tmp/.root_76a713__salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_76a713__salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 15, in top-level template code
                File "/tmp/.root_76a713__salt/salt/utils/jinja.py", line 132, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: salt/formulas.jinja

              ; line 15

              ---
              [...]
              {{ configname }}: {{ cfg_salt[configname] }}
              {%- else -%}
              #{{ configname }}: {{ default_value }}
              {%- endif -%}
              {%- endmacro -%}
              {%- from 'salt/formulas.jinja' import file_roots, formulas with context -%}    <======================
              ##### Primary configuration settings #####
              ##########################################

              # Per default the minion will automatically include all config files
              # from minion.d/*.conf (minion.d is a directory in the same directory
              [...]
              ---
     Started: 16:02:53.684336
    Duration: 108.172 ms
     Changes:
----------
          ID: salt-minion
    Function: service.running
      Result: False
     Comment: One or more requisite failed: {'salt.minion.salt-minion': '#### /etc/salt/minion.d/_defaults.conf ####\nUnable to manage file: Jinja error: salt/formulas.jinja\nTraceback (most recent call last):\n  File "/tmp/.root_76a713__salt/salt/utils/templates.py", line 286, in render_jinja_tmpl\n    output = template.render(**unicode_context)\n  File "/tmp/.root_76a713__salt/jinja2/environment.py", line 969, in render\n    return self.environment.handle_exception(exc_info, True)\n  File "/tmp/.root_76a713__salt/jinja2/environment.py", line 742, in handle_exception\n    reraise(exc_type, exc_value, tb)\n  File "<template>", line 15, in top-level template code\n  File "/tmp/.root_76a713__salt/salt/utils/jinja.py", line 132, in get_source\n    raise TemplateNotFound(template)\nTemplateNotFound: salt/formulas.jinja\n\n; line 15\n\n---\n[...]\n{{ configname }}: {{ cfg_salt[configname] }}\n{%- else -%}\n#{{ configname }}: {{ default_value }}\n{%- endif -%}\n{%- endmacro -%}\n{%- from \'salt/formulas.jinja\' import file_roots, formulas with context -%}    <======================\n##### Primary configuration settings #####\n##########################################\n\n# Per default the minion will automatically include all config files\n# from minion.d/*.conf (minion.d is a directory in the same directory\n[...]\n---'}
     Started:
    Duration:
     Changes:

Summary
------------
Succeeded: 1
Failed:    2
------------
Total states run:     3

[rallytime]

@LYY Have you been able to get this working via a regular salt command (instead of using salt-ssh)? I suspect this might be a problem with the formula itself.

[LYY]

@rallytime I'm a beginner of salt user, I really don't know how to use this formula via a regular salt command. If you can give me some help, I think I can give more info.

[rallytime]

@LYY Welcome! Now that I think about it, I am not sure if I was asking a useful question. At a glance from the info you've provided here, it looks like you have everything set up correctly.

ping @nmadhok - I haven't used this formula before. Have you seen this particular error? I can try and reproduce this tomorrow.

[nmadhok]

@rallytime I haven't come across this error in tests before. Seeing this for the first time. I will have to test this using salt-ssh since I've never tested this formula using salt-ssh

Also, salt/formulas.jinja is a part of the salt-formula. Look here: https://github.com/saltstack-formulas/salt-formula/blob/master/salt/formulas.jinja

It's probably not able to find the path to the jinja template. @LYY make sure that you have cloned the salt-formula and have specified the correct path to the jinja template in your state file.

BTW, you can use the bootstrap script to install salt-minion on the servers using salt-ssh.

[LYY]

@nmadhok Thank you very much!
I cloned salt-formula to /data/salstack_forumals/salt-formula, and setted the path to /etc/salt/master with file_roots config. Is there any path setting for use?

[gravyboat]

@rallytime We're getting the same thing over here: saltstack-formulas/mysql-formula#92 where @ogabrielsantos has completed some additional testing for us. If you want to take a look at that there is additional information that could help resolve this.

[rallytime]

Thanks for the updates everyone. I've edited the labels so that we'll hopefully get some more eyes on this one.

@basepi FYI.

[basepi]

Any files required by jinja templating must be synced down as extra_filerefs, which is just a list of salt:// paths that should be synced with the rest of the state data. Salt tries to find many of these references, but it cannot find jinja references. You can pass in extra_filerefs as a kwarg to your state run, or just set it in the master config and have those extra files synced down with every state run.

[ogabrielsantos]

Is this supposed to work without "extra_filerefs" only in certain scenarios?

[basepi]

Basically, the only refs we find are references to salt://, I can't remember off the top of my head how we're parsing those out. So if your file.managed has a source argument such as salt://myfile.txt then it should be picked up.

[gravyboat]

@basepi I think this would be something worth investigating as files that end in .yaml clearly aren't being picked up. @ogabrielsantos was able to get this working via adding the following to his /etc/salt/master file for salt-ssh:

extra_filerefs:
  - salt://mysql/defaults.yaml
  - salt://mysql/supported_sections.yaml

Seems both unnecessary and confusing to have to add that especially for people who may just be testing out Salt via salt-ssh and a formula.

[ogabrielsantos]

I can't reproduce the same issue with masterless / master-slave operation. Currently only salt-ssh has had this extra_filerefs requirement.

@basepi comment

Any files required by jinja templating must be synced down as extra_filerefs,

Related: salt-ssh jinja TemplateNotFound (#9878)

[basepi]

Right. This is not a problem with normal salt, because minions can request new files at any time. With salt-ssh, however, we have to anticipate all files that will be needed before sending those files to the minion to be evaluated.

It has nothing to do with file extension. Rather, the master does a best effort job of finding file references to send down so the minion will have a complete picture when it runs the states.

That said, if I remember correctly, states are compiled down to low data master-side for salt-ssh, so any references in the jinja should be incorporated before the files are sent down to the minion. The only time you should have to use the extra filerefs is for templates and the like, unless I'm misremembering.

[basepi]

(I know the above conflicts with what I said before, I'm trying to resolve that discrepancy now, going through the code)

[gravyboat]

Awesome thanks for taking a look!

[Xiol]

Came across this whilst having the same issue. For clarity, if you're encountering this issue you should run salt-ssh like so:

salt-ssh '*' state.sls salt.minion extra_filerefs=salt://salt/formulas.jinja

[alfredopalhares]

Hello,

Is there any progress on this issue?
@Xiol workaround works, but its not practical o use if you have many formulas that use default values like myformula/map.jinja.

[baby-gnu]

@baby-gnu Could you try the following one-line patch? #31531 (comment)

It packs the whole salt:// file tree and requires you to specify just one --extra-filerefs arg.

Without using --extra-filerefs, I manage to make 3004.1 working correctly using #61895

[corycandia]

Does anyone have an update on this?

I looked through the release notes of 3005.1 and didn't see any mention of this issue unless I missed it.

Using Xoil's provided example, I can make single states work now:
salt-ssh '*' state.sls salt.minion extra_filerefs=salt://salt/formulas.jinja

I cannot seem to make a complete state.apply (highstate) for with salt-ssh.

The master is still running 3004.2

[baby-gnu]

Another workaround, that can be triggered from a sls-file and can be used without impact such as the /dev/null approach is to use file.cached :

cache_map_jinja:
  file:
    - cached
    - name: salt://accounts/map.jinja
    - require_in:
        - file: file_that_needs_map

file_that_needs_map:
  file:
    - managed
    - ...

Thanks a lot, I will use that to work around the issue of include in file.managed rendered jinja templates.

[baby-gnu]

Thanks a lot, I will use that to work around the issue of include in file.managed rendered jinja templates.

You can find an example of how I use file.cache to workaround salt-ssh bug in my PR for collectd-formula.

[baby-gnu]

Thanks a lot, I will use that to work around the issue of include in file.managed rendered jinja templates.

You can find an example of how I use file.cache to workaround salt-ssh bug in my PR for collectd-formula.

To me, using --extra-filerefs is not usable:

• salt-ssh 'mytarget' state.apply collectd.disk test=True take 2 minutes
• salt-ssh --extra-filerefs='salt://' 'mytarget' state.apply collectd.disk test=True take 12 minutes

Does someone work on this file.managed template rendering issue?

[twangboy]

I think this has been fixed here: #63184
Please test this on version 3006.2 and create a new issue if the problem persists. The new issue template has more information and will allow us to track and reproduce the issue more effectively. Thanks!