Adding external includes from existing schema

[edager]

Hi and thank you for yamale :)

I would like to reuse some definitions from an old schema to generate a new schema. From your documentation it indeed looks like this should be possible:

Adding external includes

After you construct a schema you can add extra, external include definitions by calling
schema.add_include(dict). This method takes a dictionary and adds each key as another include.

I thought it could be done like shown in this dummy example:

old_schema.yaml

def_from_old_schema:
    element : str()
    other_element : num()

new_schema.yaml

list(include('def_from_old_schema'))

import yamale

old_schema = yamale.make_schema('old_schema.yaml')
new_schema = yamale.make_schema('new_schema.yaml')
new_schema.add_include(old_schema.dict)

However this raises a rather convoluted error message (see below). I thought the old_schema.dict would have had the correct structure to be used for Schema.add_include(), however values like 'String((), {})' throws it off. What is the simplest way to generate a dict of the right structure?

Traceback (most recent call last):
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/syntax/parser.py", line 39, in parse
tree = ast.parse(validator_string, mode='eval')
File "/usr/lib/python3.10/ast.py", line 50, in parse
return compile(source, filename, mode, flags,
TypeError: compile() arg 1 must be a string, bytes or AST object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 47, in _parse_schema_item
return syntax.parse(expression, validators)
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/syntax/parser.py", line 46, in parse
raise SyntaxError(
SyntaxError: Invalid schema expression: 'String((), {})'. compile() arg 1 must be a string, bytes or AST object
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/c71chilltown/.local/lib/python3.10/site-packages/IPython/core/interactiveshell.py", line 3553, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in <cell line: 1>
new_schema.add_include(old_schema.dict)
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 26, in add_include
t = Schema(custom_type, name=include_name,
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 17, in init
self._schema = self._process_schema(DataPath(),
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 36, in _process_schema
schema_data[key] = self._process_schema(path + DataPath(key),
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 40, in _process_schema
schema_data = self._parse_schema_item(path,
File "/home/c71chilltown/.local/lib/python3.10/site-packages/yamale/schema/schema.py", line 51, in _parse_schema_item
raise SyntaxError(error)
SyntaxError: Invalid schema expression: 'String((), {})'. compile() arg 1 must be a string, bytes or AST object at node 'element'

[mechie]

Unsure whether intended, but I think yamale.schema.Schema._process_schema() mutates self.dict in the process of making self._schema, leading to your situation.

Had never noticed, since my includes are never in the primary document:

schema_for_standalone_use: stuff()
---
# includes go here
---
# more includes

So I just do main_schema.add_include(other_schema.includes) and go on my merry way. That might be a decent workaround for you.

---

Some possible solutions:

• self.dict shouldn't be mutated at __init__(), dict() or copy() when passing it out, treat as a bug
• Add an @property (e.g. self.raw_schema) that serves __init__()'s schema_dict unchanged, since someone might depend on self.dict being equivalent to self._schema, given it's not marked private

Neutral on what to do--I'd lean for the latter since self.dict is a public part of Schema. Either way it's a tiny change.

[edager]

Thank you for the swift response :)

old_schema.yaml

random: int()

#include in different document now
---
def_from_old_schema:
    element : str()
    other_element : num()

new_schema.yaml

list(include('def_from_old_schema'))

import yamale

old_schema = yamale.make_schema('old_schema.yaml')
new_schema = yamale.make_schema('new_schema.yaml')
new_schema.add_include(old_schema.includes)                   #  Changed from .dict to .includes  

Unfortunately the work around does not appear to be work for me as the old_schema.includes contains a Schema object as its item, which gives the same type of parsing error as above.

I'm loading old_schema.yaml through yamale.make_schema(), is that a weird way of doing it?

It, indeed, works if I bypass making Schema and generate a raw_schema from old_schema.yaml and use that as a start point (i.e. borrowing from yamale's source code) it just feels like I'm doing something clumsy by reading old_schema.yaml in like that.

I'm of course willing to contribute, if you want me to look more into it?