Global: persistent staging directory for renders (#4583)

* OP-4258 - Settings for transient template

* OP-4258 - added collector for transient staging dir

Allows setting profiles to create persistent stagingDir.

* OP-4258 - implemented persistent stagingDir in cleanup

* OP-4258 - updated logging

* OP-4258 - updated settings

* OP-4258 - Hound

* OP-4258 - renamed class to better name

* OP-4258 - changed location of Settings

Should be used in create and collecting phase also.

* OP-4258 - remove version placeholder from transient template

It was discussed that it shouldn't be used for now.

* OP-4258 - extracted transient dir query logic

This should be used in collection and creation phase for DCCs which are storing staging dir path directly into nodes.

* OP-4258 - added use of scene_name placeholder in collector

DCC dependent, way how to implement versioning, might not be used.

* OP-4258 - fix scene_name

* OP-4258 - remove wrong defaults

* OP-4258 - added possibility of different template name

Studio might want to put renders to different place from caches.

* OP-4258 - renamed according to GH comments

* OP-4258 - use is active filter

* OP-4258 - use is active filter

* OP-4793 - added project_settings to signature

* OP-4793 - updated logging message

* OP-4793 - added documentation

* OP-4258 - fix function arguments

* OP-4258 - updates to documentation


* OP-4258 - added known issues to documentation

---------

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

openpype/plugins/publish/cleanup.py

@@ -93,6 +93,10 @@ class CleanUp(pyblish.api.InstancePlugin):
             self.log.info("No staging directory found: %s" % staging_dir)
             return
 
+        if instance.data.get("stagingDir_persistent"):
+            self.log.info("Staging dir: %s should be persistent" % staging_dir)
+            return
+
         self.log.info("Removing staging directory {}".format(staging_dir))
         shutil.rmtree(staging_dir)
 

openpype/plugins/publish/cleanup_farm.py

@@ -37,7 +37,7 @@ class CleanUpFarm(pyblish.api.ContextPlugin):
         dirpaths_to_remove = set()
         for instance in context:
             staging_dir = instance.data.get("stagingDir")
-            if staging_dir:
+            if staging_dir and not instance.data.get("stagingDir_persistent"):
                 dirpaths_to_remove.add(os.path.normpath(staging_dir))
 
             if "representations" in instance.data:

openpype/plugins/publish/collect_custom_staging_dir.py

@@ -0,0 +1,67 @@
+"""
+Requires:
+    anatomy
+
+
+Provides:
+    instance.data     -> stagingDir (folder path)
+                      -> stagingDir_persistent (bool)
+"""
+import copy
+import os.path
+
+import pyblish.api
+
+from openpype.pipeline.publish.lib import get_custom_staging_dir_info
+
+
+class CollectCustomStagingDir(pyblish.api.InstancePlugin):
+    """Looks through profiles if stagingDir should be persistent and in special
+    location.
+
+    Transient staging dir could be useful in specific use cases where is
+    desirable to have temporary renders in specific, persistent folders, could
+    be on disks optimized for speed for example.
+
+    It is studio responsibility to clean up obsolete folders with data.
+
+    Location of the folder is configured in `project_anatomy/templates/others`.
+    ('transient' key is expected, with 'folder' key)
+
+    Which family/task type/subset is applicable is configured in:
+    `project_settings/global/tools/publish/custom_staging_dir_profiles`
+
+    """
+    label = "Collect Custom Staging Directory"
+    order = pyblish.api.CollectorOrder + 0.4990
+
+    template_key = "transient"
+
+    def process(self, instance):
+        family = instance.data["family"]
+        subset_name = instance.data["subset"]
+        host_name = instance.context.data["hostName"]
+        project_name = instance.context.data["projectName"]
+
+        anatomy = instance.context.data["anatomy"]
+        anatomy_data = copy.deepcopy(instance.data["anatomyData"])
+        task = anatomy_data.get("task", {})
+
+        transient_tml, is_persistent = get_custom_staging_dir_info(
+            project_name, host_name, family, task.get("name"),
+            task.get("type"), subset_name, anatomy=anatomy, log=self.log)
+        result_str = "Not adding"
+        if transient_tml:
+            anatomy_data["root"] = anatomy.roots
+            scene_name = instance.context.data.get("currentFile")
+            if scene_name:
+                anatomy_data["scene_name"] = os.path.basename(scene_name)
+            transient_dir = transient_tml.format(**anatomy_data)
+            instance.data["stagingDir"] = transient_dir
+
+            instance.data["stagingDir_persistent"] = is_persistent
+            result_str = "Adding '{}' as".format(transient_dir)
+
+        self.log.info("{} custom staging dir for instance with '{}'".format(
+            result_str, family
+        ))