a custom file metadata integration between Hammerspace and Shotgrid allowing simple selection of files for instantiation in other sites
Explore the docs »
View Demo
·
Report Bug
·
Request Feature
Table of Contents
Shothammer is a shotgunEvents plugin that can respond to status change events by setting custom metadata on files and directories. This metadata can be used to drive data placement and location using SmartObjectives on a Hammerspace Global Data Environment.
Currently, shothammer.py watches for tags added to or removed from Shots, Sequences, or Tasks in Shotgrid. When
it sees tags in the allowed namespace (currently tags named SGHS_*) it adds them as Hammerspace keywords to the root
of the specified template folders on a Hammerspace file system. The tag schema is controlled in Shotgrid,
and the tag names are passed through directly as Hammerspace keywords.
It finds the appropriate folder(s) by using a per-project Pipeline Configuration containing path templates fed to it by
Shotgrid. This config can be specified in Shotgrid by adding the plugin id sghs. to the specific pipeline
configuration for the project.
Shothammer requests the templates specified in shothammer_config.yml under the appropriate object name in the
SGHS_PATH_TEMPLATES dictionary.
sghs is intended to grow as a package over time with multiple plugin modules that do different things with Shotgrid's events or webhooks.
Here are some basics on getting up and running. Everyone's Shotgrid instance is configured a bit differently and will require some configuration depending on paths and pipeline configuration names and such.
- Shotgrid API installed and configured
- Shotgrid event daemon installed and configured
- Hammerspace Toolkit (hstk) installed:
$ pip install hstk, though this will also be taken care of when installing Python requirements below. - Hammerspace file system mounted
- Clone this repository
git clone https://github.com/mabott/sghs.git - Copy or link shothammer.py to your shotgunEvents plugin directory, specified in
shotgunEventDaemon.confas one of potentially severalpathsin the[plugins]section. - Install requirements
pip install -r requirements.txt
- Adjust shothammer_config.yml to fit environment (paths, fixing namespace overlap, etc.)
- Copy or link shothammer_config.yml to shotgunEvents working directory
- One or more Hammerspace clusters set up with keyword-based objectives to drive data placement
(e.g.
IF HAS_KEYWORD("SGHS_LOCATION") THEN {SLO('place-on-local-volumes)})
Given the appropriate values in shothammer_config.yml, the tests in test_shothammer.py should pass. If the plugin gets disabled by shotgunEvents then there is a fatal problem somewhere. Running the tests using nose/pytest/your favorite IDE should give enough details to show what is broken. Open an issue with details and someone will try to help.
- Make sure hstk is installed in the same environment (venv, preferably) running shotgunEvents.
- Make sure the authentication information for Shotgrid is complete, both name and application key, for the shotgunEvents config as well as the shothammer config.
Once the plugin has been added to a ShotgunEvents server, it will recognize Tag Change events on Shots, Sequences, or Tasks and act accordingly. Before anything can happen, you need to perform a little more configuration of Shotgrid and Hammerspace.
Add SGHS-specific tags to your Shotgrid Tags via the admin menu. Click on your user portrait in the upper right and scroll down a bit):
Click the Add Tag button in the upper left and add a tag in the text field of the dialog box that pops up. By default, we want to use tags in the SGHS_ namespace, but this can be modified in the plugin config.
On your Hammerspace clusters, create Hammerspace Objectives on all sites that look for Keywords (a flavor of
Hammerspace metadata) that match the Tags you added to Shotgrid. For example, on a Site called AZ you could make an
Objective called local-if-keyword-az, expressing "If a file is in the live tree (i.e. not in a snapshot) and either
it or a parent has the Hammerspace Keyword "SGHS_AZ" then instantiate this file on a volume in the Hammerspace Volume
Group called local-volumes":
IF (IS_LIVE && HAS_KEYWORD("SGHS_AZ"))
THEN {SLO('place-on-local-volumes')}
Apply the per-site objective to the appropriate shares:
View a list of shots in Shotgrid (Project->Shots Tab) and enable viewing of the 'Tags' column:
Check the Remote status and Alignment of a shot's directories. The directory paths are determined by the value of
SGHS_PATH_TEMPLATES['Shot'] in shothammer_config.yml, which might be something like "shot_root" in the
default shotgrid toolkit config. If there are no local file
instances in the path you are observing, you will see Remote Yes and Green alignment for files, meaning that all
Objectives have been satisfied.
Set a tag on a Shot that matches the Keyword the Objective is looking for:
The plugin will set the keyword on the file system at the top level directory for the shot, causing the AZ system to recognize it needs some local data instances and request them from HQ. You can observe this by looking at the same path. Alignment will be Yellow while data is transferring, and once a file has been transferred it will be Remote Yes(cached), and Alignment will be green again, meaning that the shot now has local data instances in AZ.
Access to these files on the AZ Hammerspace cluster will now be local access, instead of remote!
- Reduce the weight of the recommended boilerplate pipeline config
- Find a path for more Shotgrid objects (currently just shots)
- Sequences
- Episodes
- Specific assets
- Tasks
See the open issues for a full list of proposed features (and known issues).
We welcome contributions to this project so that we can make this plugin applicable to many different workflows!
Feel free to fork this repo and create a pull request. You can also just open an issue with the tag "enhancement". Please give the project a star if you like it. Thanks!
- Fork the Project
- Create your Feature Branch (
git checkout -b feature/AmazingFeature) - Commit your Changes (
git commit -m 'Add some AmazingFeature') - Push to the Branch (
git push origin feature/AmazingFeature) - Open a Pull Request
Distributed under the MIT License. See LICENSE for more information.
Mike Bott, Principal Systems Engineer - @mabott - [email protected]
Project Link: https://github.com/hammer-space/sghs
This project wouldn't exist without these folks' contributions:
- Jeremy Smith of Jellyfish Pictures for his technical vision, guidance, and support
- Dan Hutchinson formerly of Jellyfish Pictures for timely application of his Python and Shotgrid knowledge
- Natasha Kelkar of Jellyfish Pictures for technical direction-setting, Python skills, deep Shotgrid knowledge, and steering me away from several dead ends.
- Our excellent crew at Hammerspace for developing cool storage technology that solves interesting distributed storage problems