Global: persistent staging directory for renders (#4583)

* OP-4258 - Settings for transient template

* OP-4258 - added collector for transient staging dir

Allows setting profiles to create persistent stagingDir.

* OP-4258 - implemented persistent stagingDir in cleanup

* OP-4258 - updated logging

* OP-4258 - updated settings

* OP-4258 - Hound

* OP-4258 - renamed class to better name

* OP-4258 - changed location of Settings

Should be used in create and collecting phase also.

* OP-4258 - remove version placeholder from transient template

It was discussed that it shouldn't be used for now.

* OP-4258 - extracted transient dir query logic

This should be used in collection and creation phase for DCCs which are storing staging dir path directly into nodes.

* OP-4258 - added use of scene_name placeholder in collector

DCC dependent, way how to implement versioning, might not be used.

* OP-4258 - fix scene_name

* OP-4258 - remove wrong defaults

* OP-4258 - added possibility of different template name

Studio might want to put renders to different place from caches.

* OP-4258 - renamed according to GH comments

* OP-4258 - use is active filter

* OP-4258 - use is active filter

* OP-4793 - added project_settings to signature

* OP-4793 - updated logging message

* OP-4793 - added documentation

* OP-4258 - fix function arguments

* OP-4258 - updates to documentation


* OP-4258 - added known issues to documentation

---------

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

openpype/pipeline/publish/contants.py

@@ -1,2 +1,3 @@
 DEFAULT_PUBLISH_TEMPLATE = "publish"
 DEFAULT_HERO_PUBLISH_TEMPLATE = "hero"
+TRANSIENT_DIR_TEMPLATE = "transient"

openpype/pipeline/publish/lib.py

@@ -20,13 +20,15 @@ from openpype.settings import (
     get_system_settings,
 )
 from openpype.pipeline import (
-    tempdir
+    tempdir,
+    Anatomy
 )
 from openpype.pipeline.plugin_discover import DiscoverResult
 
 from .contants import (
     DEFAULT_PUBLISH_TEMPLATE,
     DEFAULT_HERO_PUBLISH_TEMPLATE,
+    TRANSIENT_DIR_TEMPLATE
 )
 
 
@@ -690,3 +692,79 @@ def get_publish_repre_path(instance, repre, only_published=False):
     if os.path.exists(src_path):
         return src_path
     return None
+
+
+def get_custom_staging_dir_info(project_name, host_name, family, task_name,
+                                task_type, subset_name,
+                                project_settings=None,
+                                anatomy=None, log=None):
+    """Checks profiles if context should use special custom dir as staging.
+
+    Args:
+        project_name (str)
+        host_name (str)
+        family (str)
+        task_name (str)
+        task_type (str)
+        subset_name (str)
+        project_settings(Dict[str, Any]): Prepared project settings.
+        anatomy (Dict[str, Any])
+        log (Logger) (optional)
+
+    Returns:
+        (tuple)
+    Raises:
+        ValueError - if misconfigured template should be used
+    """
+    settings = project_settings or get_project_settings(project_name)
+    custom_staging_dir_profiles = (settings["global"]
+                                           ["tools"]
+                                           ["publish"]
+                                           ["custom_staging_dir_profiles"])
+    if not custom_staging_dir_profiles:
+        return None, None
+
+    if not log:
+        log = Logger.get_logger("get_custom_staging_dir_info")
+
+    filtering_criteria = {
+        "hosts": host_name,
+        "families": family,
+        "task_names": task_name,
+        "task_types": task_type,
+        "subsets": subset_name
+    }
+    profile = filter_profiles(custom_staging_dir_profiles,
+                              filtering_criteria,
+                              logger=log)
+
+    if not profile or not profile["active"]:
+        return None, None
+
+    if not anatomy:
+        anatomy = Anatomy(project_name)
+
+    template_name = profile["template_name"] or TRANSIENT_DIR_TEMPLATE
+    _validate_transient_template(project_name, template_name, anatomy)
+
+    custom_staging_dir = anatomy.templates[template_name]["folder"]
+    is_persistent = profile["custom_staging_dir_persistent"]
+
+    return custom_staging_dir, is_persistent
+
+
+def _validate_transient_template(project_name, template_name, anatomy):
+    """Check that transient template is correctly configured.
+
+    Raises:
+        ValueError - if misconfigured template
+    """
+    if template_name not in anatomy.templates:
+        raise ValueError(("Anatomy of project \"{}\" does not have set"
+                          " \"{}\" template key!"
+                          ).format(project_name, template_name))
+
+    if "folder" not in anatomy.templates[template_name]:
+        raise ValueError(("There is not set \"folder\" template in \"{}\" anatomy"  # noqa
+                             " for project \"{}\"."
+                         ).format(template_name, project_name))

openpype/plugins/publish/cleanup.py

@@ -93,6 +93,10 @@ class CleanUp(pyblish.api.InstancePlugin):
             self.log.info("No staging directory found: %s" % staging_dir)
             return
 
+        if instance.data.get("stagingDir_persistent"):
+            self.log.info("Staging dir: %s should be persistent" % staging_dir)
+            return
+
         self.log.info("Removing staging directory {}".format(staging_dir))
         shutil.rmtree(staging_dir)
 

openpype/plugins/publish/cleanup_farm.py

@@ -37,7 +37,7 @@ class CleanUpFarm(pyblish.api.ContextPlugin):
         dirpaths_to_remove = set()
         for instance in context:
             staging_dir = instance.data.get("stagingDir")
-            if staging_dir:
+            if staging_dir and not instance.data.get("stagingDir_persistent"):
                 dirpaths_to_remove.add(os.path.normpath(staging_dir))
 
             if "representations" in instance.data:

openpype/plugins/publish/collect_custom_staging_dir.py

@@ -0,0 +1,67 @@
+"""
+Requires:
+    anatomy
+
+
+Provides:
+    instance.data     -> stagingDir (folder path)
+                      -> stagingDir_persistent (bool)
+"""
+import copy
+import os.path
+
+import pyblish.api
+
+from openpype.pipeline.publish.lib import get_custom_staging_dir_info
+
+
+class CollectCustomStagingDir(pyblish.api.InstancePlugin):
+    """Looks through profiles if stagingDir should be persistent and in special
+    location.
+
+    Transient staging dir could be useful in specific use cases where is
+    desirable to have temporary renders in specific, persistent folders, could
+    be on disks optimized for speed for example.
+
+    It is studio responsibility to clean up obsolete folders with data.
+
+    Location of the folder is configured in `project_anatomy/templates/others`.
+    ('transient' key is expected, with 'folder' key)
+
+    Which family/task type/subset is applicable is configured in:
+    `project_settings/global/tools/publish/custom_staging_dir_profiles`
+
+    """
+    label = "Collect Custom Staging Directory"
+    order = pyblish.api.CollectorOrder + 0.4990
+
+    template_key = "transient"
+
+    def process(self, instance):
+        family = instance.data["family"]
+        subset_name = instance.data["subset"]
+        host_name = instance.context.data["hostName"]
+        project_name = instance.context.data["projectName"]
+
+        anatomy = instance.context.data["anatomy"]
+        anatomy_data = copy.deepcopy(instance.data["anatomyData"])
+        task = anatomy_data.get("task", {})
+
+        transient_tml, is_persistent = get_custom_staging_dir_info(
+            project_name, host_name, family, task.get("name"),
+            task.get("type"), subset_name, anatomy=anatomy, log=self.log)
+        result_str = "Not adding"
+        if transient_tml:
+            anatomy_data["root"] = anatomy.roots
+            scene_name = instance.context.data.get("currentFile")
+            if scene_name:
+                anatomy_data["scene_name"] = os.path.basename(scene_name)
+            transient_dir = transient_tml.format(**anatomy_data)
+            instance.data["stagingDir"] = transient_dir
+
+            instance.data["stagingDir_persistent"] = is_persistent
+            result_str = "Adding '{}' as".format(transient_dir)
+
+        self.log.info("{} custom staging dir for instance with '{}'".format(
+            result_str, family
+        ))

openpype/settings/defaults/project_anatomy/templates.json

@@ -58,12 +58,16 @@
             "file": "{originalBasename}.{ext}",
             "path": "{@folder}/{@file}"
         },
+        "transient": {
+            "folder": "{root[work]}/{project[name]}/{hierarchy}/{asset}/work/{family}/{subset}"
+        },
         "__dynamic_keys_labels__": {
             "maya2unreal": "Maya to Unreal",
             "simpleUnrealTextureHero": "Simple Unreal Texture - Hero",
             "simpleUnrealTexture": "Simple Unreal Texture",
             "online": "online",
-            "source": "source"
+            "source": "source",
+            "transient": "transient"
         }
     }
 }

openpype/settings/defaults/project_settings/global.json

@@ -591,7 +591,8 @@
                     "task_names": [],
                     "template_name": "simpleUnrealTextureHero"
                 }
-            ]
+            ],
+            "custom_staging_dir_profiles": []
         }
     },
     "project_folder_structure": "{\"__project_root__\": {\"prod\": {}, \"resources\": {\"footage\": {\"plates\": {}, \"offline\": {}}, \"audio\": {}, \"art_dept\": {}}, \"editorial\": {}, \"assets\": {\"characters\": {}, \"locations\": {}}, \"shots\": {}}}",

openpype/settings/entities/schemas/projects_schema/schemas/schema_global_tools.json

@@ -408,6 +408,71 @@
                             }
                         ]
                     }
+                },
+                {
+                    "type": "list",
+                    "key": "custom_staging_dir_profiles",
+                    "label": "Custom Staging Dir Profiles",
+                    "use_label_wrap": true,
+                    "docstring": "Profiles to specify special location and persistence for staging dir. Could be used in Creators and Publish phase!",
+                    "object_type": {
+                        "type": "dict",
+                        "children": [
+                            {
+                                "type": "boolean",
+                                "key": "active",
+                                "label": "Is active",
+                                "default": true
+                            },
+                            {
+                                "type": "separator"
+                            },
+                            {
+                                "key": "hosts",
+                                "label": "Host names",
+                                "type": "hosts-enum",
+                                "multiselection": true
+                            },
+                            {
+                                "key": "task_types",
+                                "label": "Task types",
+                                "type": "task-types-enum"
+                            },
+                            {
+                                "key": "task_names",
+                                "label": "Task names",
+                                "type": "list",
+                                "object_type": "text"
+                            },
+                            {
+                                "key": "families",
+                                "label": "Families",
+                                "type": "list",
+                                "object_type": "text"
+                            },
+                            {
+                                "key": "subsets",
+                                "label": "Subset names",
+                                "type": "list",
+                                "object_type": "text"
+                            },
+                            {
+                                "type": "separator"
+                            },
+                            {
+                                "key": "custom_staging_dir_persistent",
+                                "label": "Custom Staging Folder Persistent",
+                                "type": "boolean",
+                                "default": false
+                            },
+                            {
+                                "key": "template_name",
+                                "label": "Template Name",
+                                "type": "text",
+                                "placeholder": "transient"
+                            }
+                        ]
+                    }
                 }
             ]
         }

website/docs/project_settings/settings_project_global.md

@@ -230,10 +230,10 @@ Applicable context filters:
 ## Tools
 Settings for OpenPype tools.
 
-## Creator
+### Creator
 Settings related to [Creator tool](artist_tools_creator).
 
-### Subset name profiles
+#### Subset name profiles
 ![global_tools_creator_subset_template](assets/global_tools_creator_subset_template.png)
 
 Subset name helps to identify published content. More specific name helps with organization and avoid mixing of published content. Subset name is defined using one of templates defined in **Subset name profiles settings**. The template is filled with context information at the time of creation.
@@ -263,10 +263,31 @@ Template may look like `"{family}{Task}{Variant}"`.
 Some creators may have other keys as their context may require more information or more specific values. Make sure you've read documentation of host you're using.
 
 
-## Workfiles
+### Publish
+
+#### Custom Staging Directory Profiles
+With this feature, users can specify a custom data folder path based on presets, which can be used during the creation and publishing stages.
+
+![global_tools_custom_staging_dir](assets/global_tools_custom_staging_dir.png)
+
+Staging directories are used as a destination for intermediate files (as renders) before they are renamed and copied to proper location during the integration phase. They could be created completely dynamically in the temp folder or for some DCCs in the `work` area.
+Example could be Nuke where artist might want to temporarily render pictures into `work` area to check them before they get published with the choice of "Use existing frames" on the write node.
+
+One of the key advantages of this feature is that it allows users to choose the folder for writing such intermediate files to take advantage of faster storage for rendering, which can help improve workflow efficiency. Additionally, this feature allows users to keep their intermediate extracted data persistent, and use their own infrastructure for regular cleaning.
+
+In some cases, these DCCs (Nuke, Houdini, Maya) automatically add a rendering path during the creation stage, which is then used in publishing. Creators and extractors of such DCCs need to use these profiles to fill paths in DCC's nodes to use this functionality.
+
+The custom staging folder uses a path template configured in `project_anatomy/templates/others` with `transient` being a default example path that could be used. The template requires a 'folder' key for it to be usable as custom staging folder.
+
+##### Known issues
+- Any DCC that uses prefilled paths and store them inside of workfile nodes needs to implement resolving these paths with a configured profiles.
+- If studio uses Site Sync remote artists need to have access to configured custom staging folder!
+- Each node on the rendering farm must have access to configured custom staging folder!
+
+### Workfiles
 All settings related to Workfile tool.
 
-### Open last workfile at launch
+#### Open last workfile at launch
 This feature allows you to define a rule for each task/host or toggle the feature globally to all tasks as they are visible in the picture.
 
 ![global_tools_workfile_open_last_version](assets/global_tools_workfile_open_last_version.png)