Can't find "mysql/defaults.yaml" during mysql.server state #92

ogabrielsantos · 2015-09-28T05:42:16Z

I can't install MySQL server because Jinja can't find mysql/defaults.yaml (related with this not solved but closed issue)

Currently I've tested this with a clean new VM deploying from my notebook (thanks to salt-ssh) without success.

I hope that this can be solved.

etc/salt/master

root_dir: .
file_roots:
  base:
    - srv/salt/states
    - srv/salt/formulas/mysql-formula
pillar_roots:
  base:
    - srv/salt/pillars

srv/salt/states/top.sls

base:
  '*':
    - mysql.server

srv/salt/pillars/top.sls

base:
  '*':
    - database

srv/salt/pillars/database.sls

mysql:
  server:
    root_user: 'root'
    root_password: 'pass'
    user: mysql
{% if grains['os_family'] == 'RedHat' %}
  lookup:
    server: mariadb-server
    client: mariadb
{% endif %}

Error:

          ID: mysql_config
    Function: file.managed
        Name: /etc/my.cnf
      Result: False
     Comment: Unable to manage file: Jinja error: mysql/defaults.yaml
              Traceback (most recent call last):
                File "/tmp/.root_b91e95_salt/salt/utils/templates.py", line 368, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/tmp/.root_b91e95_salt/jinja2/environment.py", line 969, in render
                  return self.environment.handle_exception(exc_info, True)
                File "/tmp/.root_b91e95_salt/jinja2/environment.py", line 742, in handle_exception
                  reraise(exc_type, exc_value, tb)
                File "<template>", line 7, in top-level template code
                File "/tmp/.root_b91e95_salt/salt/utils/jinja.py", line 144, in get_source
                  raise TemplateNotFound(template)
              TemplateNotFound: mysql/defaults.yaml

              ; line 7

              ---
              [...]
              # This config is generated by SALTSTACK
              # and all change will be overrided on next salt call
              {#-
              ===== FETCH DATA =====
              -#}
              {%- from "mysql/defaults.yaml" import rawmap with context -%}    <======================
              {%- from "mysql/supported_sections.yaml" import supported_sections with context -%}
              {%- set datamap = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) -%}
              {#-
              ===== COMBINE DATA =====
              -#}
              [...]
              ---
     Started: 05:25:44.050898
    Duration: 30.905 ms
     Changes:

salt-ssh --versions-report

Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: 1.0.0
         PyYAML: 3.11
          PyZMQ: 14.4.1
         Python: 2.7.9 (default, Apr  2 2015, 15:33:21)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.2
          gitdb: 0.5.4
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.2
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: 0.8.2
        timelib: Not Installed

System Versions:
           dist: Ubuntu 15.04 vivid
        machine: x86_64
        release: 3.19.0-30-generic
         system: Ubuntu 15.04 vivid

The text was updated successfully, but these errors were encountered:

ogabrielsantos · 2015-09-28T06:04:23Z

Tried with different salt version without success:

                  Salt: 2015.5.0
                Python: 2.7.9 (default, Apr  2 2015, 15:33:21)
                Jinja2: 2.7.3
              M2Crypto: Not Installed
        msgpack-python: 0.4.2
          msgpack-pure: Not Installed
              pycrypto: 2.6.1
               libnacl: Not Installed
                PyYAML: 3.11
                 ioflo: Not Installed
                 PyZMQ: Not Installed
                  RAET: Not Installed
                   ZMQ: Not Installed
                  Mako: 1.0.0
 Debian source package: 2015.5.0+ds-1utopic1

gravyboat · 2015-09-28T06:09:04Z

I'll try to take a look at this sometimes this week, @iggy this is similar to the other issue with this same problem, however @ogabrielsantos hasn't tried workarounds so we have a clean slate, any suggestions/insight?

puneetk · 2015-09-28T07:29:48Z

change the file root ?

root_dir: .
file_roots:
base:
- srv/salt/states
- srv/salt/formulas
pillar_roots:
base:
- srv/salt/pillars

ogabrielsantos · 2015-09-28T07:33:45Z

@puneetk srv/salt/formulas/mysql-formula refer to a git module clone inside formulas. Changing to srv/salt/formulas will invalidate the formula/state.

├── srv
│   └── salt
│       ├── formulas
│       │   └── mysql-formula
│       │       ├── FORMULA
│       │       ├── import_users.py
│       │       ├── LICENSE
│       │       ├── mysql
│       │       │   ├── client.sls
│       │       │   ├── database.sls
│       │       │   ├── defaults.yaml
│       │       │   ├── dev.sls
│       │       │   ├── disabled.sls
│       │       │   ├── files
│       │       │   │   ├── my.cnf
│       │       │   │   └── usr-my.cnf
│       │       │   ├── init.sls
│       │       │   ├── python.sls
│       │       │   ├── remove_test_database.sls
│       │       │   ├── server.sls
│       │       │   ├── supported_sections.yaml
│       │       │   └── user.sls
│       │       ├── pillar.example
│       │       ├── README.rst
│       │       └── scripts
│       │           └── import_users.py
│       ├── pillars
│       │   ├── database.sls
│       │   └── top.sls
│       └── states
│           └── top.sls

gravyboat · 2015-09-28T07:43:52Z

I find it interesting that this issue exists: saltstack/salt#21370. @ogabrielsantos can you spin a VM and install a minion then try from there? So via salt itself, and not salt-ssh?

ogabrielsantos · 2015-09-28T07:52:22Z

I'll try as soon as possible @gravyboat

I don't think that this is a issue related to mysql-formula. Tested a sample state with same error:

etc/salt/master

root_dir: .
file_roots:
  base:
    - srv/salt/states

srv/salt/states/top.sls

base:
  '*':
    - sample

srv/salt/states/sample/init.sls

{% from "sample/sample.jinja" import sampledata with context %}

{{ sampledata.file }}:
  file.managed:
    - source: salt://sample/sample.cnf
    - template: jinja

srv/salt/states/sample/sample.cnf

{% from "sample/sample.jinja" import sampledata with context %}

srv/salt/states/sample/sample.jinja

{% load_yaml as rawmap %}
RedHat:
  file: /etc/my.cnf
{% endload %}

{% set sampledata = rawmap.get(grains.os_family) %}

gravyboat · 2015-09-28T07:56:54Z

@ogabrielsantos Interesting, thanks for testing that. I'm headed out for the evening, let me know what it looks like when you're able to test with the minion.

ogabrielsantos · 2015-09-28T08:32:59Z

Currently confirmed that CentOS 6.7 (minion) works fine with a CentOS 7 (master) using both salt and salt-ssh. Maybe it is a Saltstack dependency related issue?

Minion

[root@li1289-145 ~]# salt-minion --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: unknown
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.5.0
         Python: 2.6.6 (r266:84292, Jul 23 2015, 15:22:56)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 6.7 Final
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: CentOS 6.7 Final

Master

[root@li1278-54 sample]# salt-ssh --versions-report
Salt Version:
           Salt: 2015.8.0

Dependency Versions:
         Jinja2: 2.7.3
       M2Crypto: Not Installed
           Mako: Not Installed
         PyYAML: 3.11
          PyZMQ: 14.7.0
         Python: 2.7.5 (default, Jun 24 2015, 00:41:19)
           RAET: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
        libnacl: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
         pygit2: Not Installed
   python-gnupg: Not Installed
          smmap: Not Installed
        timelib: Not Installed

System Versions:
           dist: centos 7.1.1503 Core
        machine: x86_64
        release: 4.1.5-x86_64-linode61
         system: CentOS Linux 7.1.1503 Core

ogabrielsantos · 2015-09-28T21:35:43Z

I've patched this workaround which remove extra_filerefs and jinja templating file inclusion requirements as per @basepi comment

From f42f48666503a970636fd9c06be5dd350be889c5 Mon Sep 17 00:00:00 2001
From: Gabriel Santos <[email protected]>
Date: Mon, 28 Sep 2015 18:24:42 -0300
Subject: [PATCH] Workaround Jinja extra_filerefs requirement

---
 mysql/files/my.cnf | 6 ------
 mysql/server.sls   | 4 ++++
 2 files changed, 4 insertions(+), 6 deletions(-)

diff --git a/mysql/files/my.cnf b/mysql/files/my.cnf
index 8e97641..5c5b794 100644
--- a/mysql/files/my.cnf
+++ b/mysql/files/my.cnf
@@ -2,12 +2,6 @@
 # This config is generated by SALTSTACK
 # and all change will be overrided on next salt call
 {#-
-===== FETCH DATA =====
--#}
-{%- from "mysql/defaults.yaml" import rawmap with context -%}
-{%- from "mysql/supported_sections.yaml" import supported_sections with context -%}
-{%- set datamap = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) -%}
-{#-
 ===== COMBINE DATA =====
 -#}
 {%- set goodParamList = datamap.config.sections -%}
diff --git a/mysql/server.sls b/mysql/server.sls
index 1a28cc6..3624107 100644
--- a/mysql/server.sls
+++ b/mysql/server.sls
@@ -1,4 +1,5 @@
 {% from "mysql/defaults.yaml" import rawmap with context %}
+{% from "mysql/supported_sections.yaml" import supported_sections with context %}
 {%- set mysql = salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:lookup')) %}

 {% set os = salt['grains.get']('os', None) %}
@@ -75,6 +76,9 @@ mysql_config:
     - name: {{ mysql.config.file }}
     - template: jinja
     - source: salt://mysql/files/my.cnf
+    - defaults:
+      supported_sections: supported_sections
+      datamap: {{ salt['grains.filter_by'](rawmap, grain='os', merge=salt['pillar.get']('mysql:server:lookup')) }}
     - watch_in:
       - service: mysqld
     {% if os_family in ['Debian', 'Gentoo', 'RedHat'] %}
-- 
2.1.4

ogabrielsantos · 2015-09-30T06:28:06Z

Editing /etc/salt/master solve this odd issue:

extra_filerefs:
  - salt://mysql/defaults.yaml
  - salt://mysql/supported_sections.yaml

Maybe documentation could be improved or templating file inclusion avoided in order to prevent future issues to be opened.

If there are no new considerations this issue can be closed. Thanks.

gravyboat · 2015-09-30T06:32:18Z

@ogabrielsantos This only had to be added for salt-ssh though right? Worked fine with the master/minion and masterless?

ogabrielsantos · 2015-09-30T06:35:11Z

@gravyboat Yep. Currently only salt-ssh needs the extra_filerefs. master/minion and masterless works fine without any intervention.

gravyboat · 2015-09-30T06:36:48Z

Hmm, this is very interesting. I don't really FEEL like you should have to do that. I'm going to leave this open for right now and comment in the other issue. Stealing your extra_filerefs content and will attribute to you.

iggy · 2015-10-01T22:33:30Z

There have been a couple issues in the past about extra_filerefs in a couple different places. It would be really nice if SaltStack could handle that on it's own, but until then, maybe we should start adding something to the README in all the formulas?

gravyboat · 2015-10-01T23:09:54Z

@iggy Yeah that would probably be a good plan until we see what happens with saltstack/salt#21370

ogabrielsantos changed the title ~~Jinja can't find mysql/defaults.yaml during mysql.server state~~ Jinja can't find mysql/defaults.yaml during mysql.server state Sep 28, 2015

ogabrielsantos changed the title ~~Jinja can't find mysql/defaults.yaml during mysql.server state~~ Can't find "mysql/defaults.yaml" during mysql.server state Sep 28, 2015

gravyboat mentioned this issue Sep 28, 2015

TemplateNotFound: salt/formulas.jinja when running salt-ssh saltstack/salt#21370

Closed

gravyboat mentioned this issue Sep 30, 2015

mysql/defaults.yaml not found #85

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't find "mysql/defaults.yaml" during mysql.server state #92

Can't find "mysql/defaults.yaml" during mysql.server state #92

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

puneetk commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 30, 2015

gravyboat commented Sep 30, 2015

ogabrielsantos commented Sep 30, 2015

gravyboat commented Sep 30, 2015

iggy commented Oct 1, 2015

gravyboat commented Oct 1, 2015

Can't find "mysql/defaults.yaml" during mysql.server state #92

Can't find "mysql/defaults.yaml" during mysql.server state #92

Comments

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

puneetk commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

gravyboat commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 28, 2015

ogabrielsantos commented Sep 30, 2015

gravyboat commented Sep 30, 2015

ogabrielsantos commented Sep 30, 2015

gravyboat commented Sep 30, 2015

iggy commented Oct 1, 2015

gravyboat commented Oct 1, 2015