Merge branch 'develop' into enhancement/OP-3302_maya-camera-content-validation

CHANGELOG.md

@@ -1,8 +1,42 @@
 # Changelog
 
-## [3.11.0-nightly.2](https://github.com/pypeclub/OpenPype/tree/HEAD)
+## [3.11.1](https://github.com/pypeclub/OpenPype/tree/3.11.1) (2022-06-20)
 
-[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.10.0...HEAD)
+[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.11.0...3.11.1)
+
+**🆕 New features**
+
+- Flame: custom export temp folder [\#3346](https://github.com/pypeclub/OpenPype/pull/3346)
+- Nuke: removing third-party plugins   [\#3344](https://github.com/pypeclub/OpenPype/pull/3344)
+
+**🚀 Enhancements**
+
+- Pyblish Pype: Hiding/Close issues [\#3367](https://github.com/pypeclub/OpenPype/pull/3367)
+- Ftrack: Removed requirement of pypeclub role from default settings [\#3354](https://github.com/pypeclub/OpenPype/pull/3354)
+- Kitsu: Prevent crash on missing frames information [\#3352](https://github.com/pypeclub/OpenPype/pull/3352)
+- Ftrack: Open browser from tray [\#3320](https://github.com/pypeclub/OpenPype/pull/3320)
+- Enhancement: More control over thumbnail processing. [\#3259](https://github.com/pypeclub/OpenPype/pull/3259)
+
+**🐛 Bug fixes**
+
+- Nuke: bake streams with slate on farm [\#3368](https://github.com/pypeclub/OpenPype/pull/3368)
+- Harmony: audio validator has wrong logic [\#3364](https://github.com/pypeclub/OpenPype/pull/3364)
+- Nuke: Fix missing variable in extract thumbnail [\#3363](https://github.com/pypeclub/OpenPype/pull/3363)
+- Nuke: Fix precollect writes [\#3361](https://github.com/pypeclub/OpenPype/pull/3361)
+- AE- fix validate\_scene\_settings and renderLocal [\#3358](https://github.com/pypeclub/OpenPype/pull/3358)
+- deadline: fixing misidentification of revieables [\#3356](https://github.com/pypeclub/OpenPype/pull/3356)
+- General: Create only one thumbnail per instance [\#3351](https://github.com/pypeclub/OpenPype/pull/3351)
+- General: Fix last version function [\#3345](https://github.com/pypeclub/OpenPype/pull/3345)
+- Deadline: added OPENPYPE\_MONGO to filter [\#3336](https://github.com/pypeclub/OpenPype/pull/3336)
+- Nuke: fixing farm publishing if review is disabled [\#3306](https://github.com/pypeclub/OpenPype/pull/3306)
+
+**🔀 Refactored code**
+
+- Webpublisher: Use client query functions [\#3333](https://github.com/pypeclub/OpenPype/pull/3333)
+
+## [3.11.0](https://github.com/pypeclub/OpenPype/tree/3.11.0) (2022-06-17)
+
+[Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.11.0-nightly.4...3.11.0)
 
 ### 📖 Documentation
 
@@ -11,7 +45,9 @@
 
 **🚀 Enhancements**
 
+- Settings: Settings can be extracted from UI [\#3323](https://github.com/pypeclub/OpenPype/pull/3323)
 - updated poetry installation source [\#3316](https://github.com/pypeclub/OpenPype/pull/3316)
+- Ftrack: Action to easily create daily review session [\#3310](https://github.com/pypeclub/OpenPype/pull/3310)
 - TVPaint: Extractor use mark in/out range to render [\#3309](https://github.com/pypeclub/OpenPype/pull/3309)
 - Ftrack: Delivery action can work on ReviewSessions [\#3307](https://github.com/pypeclub/OpenPype/pull/3307)
 - Maya: Look assigner UI improvements [\#3298](https://github.com/pypeclub/OpenPype/pull/3298)
@@ -22,13 +58,13 @@
 - Maya: reference loaders could store placeholder in referenced url [\#3264](https://github.com/pypeclub/OpenPype/pull/3264)
 - TVPaint: Init file for TVPaint worker also handle guideline images [\#3250](https://github.com/pypeclub/OpenPype/pull/3250)
 - Nuke: Change default icon path in settings [\#3247](https://github.com/pypeclub/OpenPype/pull/3247)
-- Maya: publishing of animation and pointcache on a farm [\#3225](https://github.com/pypeclub/OpenPype/pull/3225)
-- Maya: Look assigner UI improvements [\#3208](https://github.com/pypeclub/OpenPype/pull/3208)
-- Nuke: add pointcache and animation to loader [\#3186](https://github.com/pypeclub/OpenPype/pull/3186)
 
 **🐛 Bug fixes**
 
+- General: Handle empty source key on instance [\#3342](https://github.com/pypeclub/OpenPype/pull/3342)
 - Houdini: Fix Houdini VDB manage update wrong file attribute name [\#3322](https://github.com/pypeclub/OpenPype/pull/3322)
+- Nuke: anatomy compatibility issue hacks [\#3321](https://github.com/pypeclub/OpenPype/pull/3321)
+- hiero: otio p3 compatibility issue - metadata on effect use update 3.11 [\#3314](https://github.com/pypeclub/OpenPype/pull/3314)
 - General: Vendorized modules for Python 2 and update poetry lock [\#3305](https://github.com/pypeclub/OpenPype/pull/3305)
 - Fix - added local targets to install host [\#3303](https://github.com/pypeclub/OpenPype/pull/3303)
 - Settings: Add missing default settings for nuke gizmo [\#3301](https://github.com/pypeclub/OpenPype/pull/3301)
@@ -44,12 +80,17 @@
 - Unreal: Fix Camera Loading if Layout is missing [\#3255](https://github.com/pypeclub/OpenPype/pull/3255)
 - Unreal: Fixed Animation loading in UE5 [\#3240](https://github.com/pypeclub/OpenPype/pull/3240)
 - Unreal: Fixed Render creation in UE5 [\#3239](https://github.com/pypeclub/OpenPype/pull/3239)
-- Unreal: Fixed Camera loading in UE5 [\#3238](https://github.com/pypeclub/OpenPype/pull/3238)
-- Flame: debugging [\#3224](https://github.com/pypeclub/OpenPype/pull/3224)
+
+**🔀 Refactored code**
+
+- Blender: Use client query functions [\#3331](https://github.com/pypeclub/OpenPype/pull/3331)
+- General: Define query functions [\#3288](https://github.com/pypeclub/OpenPype/pull/3288)
 
 **Merged pull requests:**
 
 - Maya: add pointcache family to gpu cache loader [\#3318](https://github.com/pypeclub/OpenPype/pull/3318)
+- Maya look: skip empty file attributes [\#3274](https://github.com/pypeclub/OpenPype/pull/3274)
+- Harmony: 21.1 fix [\#3248](https://github.com/pypeclub/OpenPype/pull/3248)
 
 ## [3.10.0](https://github.com/pypeclub/OpenPype/tree/3.10.0) (2022-05-26)
 
@@ -59,9 +100,6 @@
 
 - Maya: FBX camera export [\#3253](https://github.com/pypeclub/OpenPype/pull/3253)
 - General: updating common vendor `scriptmenu` to 1.5.2 [\#3246](https://github.com/pypeclub/OpenPype/pull/3246)
-- Project Manager: Allow to paste Tasks into multiple assets at the same time [\#3226](https://github.com/pypeclub/OpenPype/pull/3226)
-- Project manager: Sped up project load [\#3216](https://github.com/pypeclub/OpenPype/pull/3216)
-- Loader UI: Speed issues of loader with sync server [\#3199](https://github.com/pypeclub/OpenPype/pull/3199)
 
 **🐛 Bug fixes**
 
@@ -70,47 +108,16 @@
 - Maya: renderman displays needs to be filtered [\#3242](https://github.com/pypeclub/OpenPype/pull/3242)
 - Ftrack: Validate that the user exists on ftrack [\#3237](https://github.com/pypeclub/OpenPype/pull/3237)
 - Maya: Fix support for multiple resolutions [\#3236](https://github.com/pypeclub/OpenPype/pull/3236)
-- TVPaint: Look for more groups than 12 [\#3228](https://github.com/pypeclub/OpenPype/pull/3228)
-- Hiero: debugging frame range and other 3.10 [\#3222](https://github.com/pypeclub/OpenPype/pull/3222)
-- Project Manager: Fix persistent editors on project change [\#3218](https://github.com/pypeclub/OpenPype/pull/3218)
-- Deadline: instance data overwrite fix [\#3214](https://github.com/pypeclub/OpenPype/pull/3214)
-- Ftrack: Push hierarchical attributes action works [\#3210](https://github.com/pypeclub/OpenPype/pull/3210)
-- Standalone Publisher: Always create new representation for thumbnail [\#3203](https://github.com/pypeclub/OpenPype/pull/3203)
-- Photoshop: skip collector when automatic testing [\#3202](https://github.com/pypeclub/OpenPype/pull/3202)
-- Nuke: render/workfile version sync doesn't work on farm [\#3185](https://github.com/pypeclub/OpenPype/pull/3185)
-- Ftrack: Review image only if there are no mp4 reviews [\#3183](https://github.com/pypeclub/OpenPype/pull/3183)
-
-**🔀 Refactored code**
-
-- Avalon repo removed from Jobs workflow [\#3193](https://github.com/pypeclub/OpenPype/pull/3193)
 
 **Merged pull requests:**
 
 - Harmony: message length in 21.1 [\#3257](https://github.com/pypeclub/OpenPype/pull/3257)
 - Harmony: 21.1 fix [\#3249](https://github.com/pypeclub/OpenPype/pull/3249)
-- Maya: added jpg to filter for Image Plane Loader [\#3223](https://github.com/pypeclub/OpenPype/pull/3223)
 
 ## [3.9.8](https://github.com/pypeclub/OpenPype/tree/3.9.8) (2022-05-19)
 
 [Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.9.7...3.9.8)
 
-**🚀 Enhancements**
-
-- nuke: generate publishing nodes inside render group node  [\#3206](https://github.com/pypeclub/OpenPype/pull/3206)
-- Loader UI: Speed issues of loader with sync server [\#3200](https://github.com/pypeclub/OpenPype/pull/3200)
-- Backport of fix for attaching renders to subsets [\#3195](https://github.com/pypeclub/OpenPype/pull/3195)
-- Looks: add basic support for Renderman [\#3190](https://github.com/pypeclub/OpenPype/pull/3190)
-
-**🐛 Bug fixes**
-
-- Standalone Publisher: Always create new representation for thumbnail [\#3204](https://github.com/pypeclub/OpenPype/pull/3204)
-- Nuke: render/workfile version sync doesn't work on farm  [\#3184](https://github.com/pypeclub/OpenPype/pull/3184)
-- Ftrack: Review image only if there are no mp4 reviews [\#3182](https://github.com/pypeclub/OpenPype/pull/3182)
-
-**Merged pull requests:**
-
-- hiero: otio p3 compatibility issue - metadata on effect use update  [\#3194](https://github.com/pypeclub/OpenPype/pull/3194)
-
 ## [3.9.7](https://github.com/pypeclub/OpenPype/tree/3.9.7) (2022-05-11)
 
 [Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.9.6...3.9.7)

openpype/api.py

@@ -44,7 +44,6 @@ from . import resources
 
 from .plugin import (
     Extractor,
-    Integrator,
 
     ValidatePipelineOrder,
     ValidateContentsOrder,
@@ -87,7 +86,6 @@ __all__ = [
 
     # plugin classes
     "Extractor",
-    "Integrator",
     # ordering
     "ValidatePipelineOrder",
     "ValidateContentsOrder",

openpype/client/__init__.py

@@ -5,6 +5,7 @@ from .entities import (
     get_asset_by_id,
     get_asset_by_name,
     get_assets,
+    get_archived_assets,
     get_asset_ids_with_subsets,
 
     get_subset_by_id,
@@ -41,6 +42,7 @@ __all__ = (
     "get_asset_by_id",
     "get_asset_by_name",
     "get_assets",
+    "get_archived_assets",
     "get_asset_ids_with_subsets",
 
     "get_subset_by_id",

openpype/client/entities.py

@@ -23,7 +23,7 @@ def _get_project_connection(project_name=None):
     return mongodb
 
 
-def _prepare_fields(fields):
+def _prepare_fields(fields, required_fields=None):
     if not fields:
         return None
 
@@ -33,6 +33,10 @@ def _prepare_fields(fields):
     }
     if "_id" not in output:
         output["_id"] = True
+
+    if required_fields:
+        for key in required_fields:
+            output[key] = True
     return output
 
 
@@ -135,8 +139,16 @@ def get_asset_by_name(project_name, asset_name, fields=None):
     return conn.find_one(query_filter, _prepare_fields(fields))
 
 
-def get_assets(
-    project_name, asset_ids=None, asset_names=None, archived=False, fields=None
+# NOTE this could be just public function?
+# - any better variable name instead of 'standard'?
+# - same approach can be used for rest of types
+def _get_assets(
+    project_name,
+    asset_ids=None,
+    asset_names=None,
+    standard=True,
+    archived=False,
+    fields=None
 ):
     """Assets for specified project by passed filters.
 
@@ -149,6 +161,8 @@ def get_assets(
         project_name (str): Name of project where to look for queried entities.
         asset_ids (list[str|ObjectId]): Asset ids that should be found.
         asset_names (list[str]): Name assets that should be found.
+        standard (bool): Query standart assets (type 'asset').
+        archived (bool): Query archived assets (type 'archived_asset').
         fields (list[str]): Fields that should be returned. All fields are
             returned if 'None' is passed.
 
@@ -157,10 +171,15 @@ def get_assets(
             passed filters.
     """
 
-    asset_types = ["asset"]
+    asset_types = []
+    if standard:
+        asset_types.append("asset")
     if archived:
         asset_types.append("archived_asset")
 
+    if not asset_types:
+        return []
+
     if len(asset_types) == 1:
         query_filter = {"type": asset_types[0]}
     else:
@@ -182,6 +201,68 @@ def get_assets(
     return conn.find(query_filter, _prepare_fields(fields))
 
 
+def get_assets(
+    project_name,
+    asset_ids=None,
+    asset_names=None,
+    archived=False,
+    fields=None
+):
+    """Assets for specified project by passed filters.
+
+    Passed filters (ids and names) are always combined so all conditions must
+    match.
+
+    To receive all assets from project just keep filters empty.
+
+    Args:
+        project_name (str): Name of project where to look for queried entities.
+        asset_ids (list[str|ObjectId]): Asset ids that should be found.
+        asset_names (list[str]): Name assets that should be found.
+        archived (bool): Add also archived assets.
+        fields (list[str]): Fields that should be returned. All fields are
+            returned if 'None' is passed.
+
+    Returns:
+        Cursor: Query cursor as iterable which returns asset documents matching
+            passed filters.
+    """
+
+    return _get_assets(
+        project_name, asset_ids, asset_names, True, archived, fields
+    )
+
+
+def get_archived_assets(
+    project_name,
+    asset_ids=None,
+    asset_names=None,
+    fields=None
+):
+    """Archived assets for specified project by passed filters.
+
+    Passed filters (ids and names) are always combined so all conditions must
+    match.
+
+    To receive all archived assets from project just keep filters empty.
+
+    Args:
+        project_name (str): Name of project where to look for queried entities.
+        asset_ids (list[str|ObjectId]): Asset ids that should be found.
+        asset_names (list[str]): Name assets that should be found.
+        fields (list[str]): Fields that should be returned. All fields are
+            returned if 'None' is passed.
+
+    Returns:
+        Cursor: Query cursor as iterable which returns asset documents matching
+            passed filters.
+    """
+
+    return _get_assets(
+        project_name, asset_ids, asset_names, False, True, fields
+    )
+
+
 def get_asset_ids_with_subsets(project_name, asset_ids=None):
     """Find out which assets have existing subsets.
 
@@ -428,6 +509,7 @@ def _get_versions(
     project_name,
     subset_ids=None,
     version_ids=None,
+    versions=None,
     standard=True,
     hero=False,
     fields=None
@@ -458,6 +540,16 @@ def _get_versions(
             return []
         query_filter["_id"] = {"$in": version_ids}
 
+    if versions is not None:
+        versions = list(versions)
+        if not versions:
+            return []
+
+        if len(versions) == 1:
+            query_filter["name"] = versions[0]
+        else:
+            query_filter["name"] = {"$in": versions}
+
     conn = _get_project_connection(project_name)
 
     return conn.find(query_filter, _prepare_fields(fields))
@@ -467,6 +559,7 @@ def get_versions(
     project_name,
     version_ids=None,
     subset_ids=None,
+    versions=None,
     hero=False,
     fields=None
 ):
@@ -480,6 +573,8 @@ def get_versions(
             Filter ignored if 'None' is passed.
         subset_ids (list[str]): Subset ids that will be queried.
             Filter ignored if 'None' is passed.
+        versions (list[int]): Version names (as integers).
+            Filter ignored if 'None' is passed.
         hero (bool): Look also for hero versions.
         fields (list[str]): Fields that should be returned. All fields are
             returned if 'None' is passed.
@@ -492,6 +587,7 @@ def get_versions(
         project_name,
         subset_ids,
         version_ids,
+        versions,
         standard=True,
         hero=hero,
         fields=fields
@@ -655,9 +751,8 @@ def get_last_versions(project_name, subset_ids, fields=None):
         doc["_version_id"]
         for doc in conn.aggregate(_pipeline)
     ]
-    fields = _prepare_fields(fields)
-    if fields and "parent" not in fields:
-        fields.append("parent")
+
+    fields = _prepare_fields(fields, ["parent"])
 
     version_docs = get_versions(
         project_name, version_ids=version_ids, fields=fields

openpype/hosts/aftereffects/api/pipeline.py

@@ -65,14 +65,14 @@ def on_pyblish_instance_toggled(instance, old_value, new_value):
     instance[0].Visible = new_value
 
 
-def get_asset_settings():
+def get_asset_settings(asset_doc):
     """Get settings on current asset from database.
 
     Returns:
         dict: Scene data.
 
     """
-    asset_data = lib.get_asset()["data"]
+    asset_data = asset_doc["data"]
     fps = asset_data.get("fps")
     frame_start = asset_data.get("frameStart")
     frame_end = asset_data.get("frameEnd")

openpype/hosts/aftereffects/plugins/create/workfile_creator.py

@@ -1,4 +1,5 @@
 import openpype.hosts.aftereffects.api as api
+from openpype.client import get_asset_by_name
 from openpype.pipeline import (
     AutoCreator,
     CreatedInstance,
@@ -41,10 +42,7 @@ class AEWorkfileCreator(AutoCreator):
         host_name = legacy_io.Session["AVALON_APP"]
 
         if existing_instance is None:
-            asset_doc = legacy_io.find_one({
-                "type": "asset",
-                "name": asset_name
-            })
+            asset_doc = get_asset_by_name(project_name, asset_name)
             subset_name = self.get_subset_name(
                 variant, task_name, asset_doc, project_name, host_name
             )
@@ -69,10 +67,7 @@ class AEWorkfileCreator(AutoCreator):
             existing_instance["asset"] != asset_name
             or existing_instance["task"] != task_name
         ):
-            asset_doc = legacy_io.find_one({
-                "type": "asset",
-                "name": asset_name
-            })
+            asset_doc = get_asset_by_name(project_name, asset_name)
             subset_name = self.get_subset_name(
                 variant, task_name, asset_doc, project_name, host_name
             )

openpype/hosts/aftereffects/plugins/publish/collect_render.py

@@ -21,7 +21,7 @@ class AERenderInstance(RenderInstance):
     projectEntity = attr.ib(default=None)
     stagingDir = attr.ib(default=None)
     app_version = attr.ib(default=None)
-    publish_attributes = attr.ib(default=None)
+    publish_attributes = attr.ib(default={})
     file_name = attr.ib(default=None)
 
 
@@ -90,7 +90,7 @@ class CollectAERender(abstract_collect_render.AbstractCollectRender):
 
             subset_name = inst.data["subset"]
             instance = AERenderInstance(
-                family=family,
+                family="render",
                 families=inst.data.get("families", []),
                 version=version,
                 time="",
@@ -116,7 +116,7 @@ class CollectAERender(abstract_collect_render.AbstractCollectRender):
                 toBeRenderedOn='deadline',
                 fps=fps,
                 app_version=app_version,
-                publish_attributes=inst.data.get("publish_attributes"),
+                publish_attributes=inst.data.get("publish_attributes", {}),
                 file_name=render_q.file_name
             )
 

openpype/hosts/aftereffects/plugins/publish/validate_scene_settings.py

@@ -1,5 +1,9 @@
 # -*- coding: utf-8 -*-
-"""Validate scene settings."""
+"""Validate scene settings.
+Requires:
+    instance    -> assetEntity
+    instance    -> anatomyData
+"""
 import os
 import re
 
@@ -54,7 +58,7 @@ class ValidateSceneSettings(OptionalPyblishPluginMixin,
 
     order = pyblish.api.ValidatorOrder
     label = "Validate Scene Settings"
-    families = ["render.farm", "render"]
+    families = ["render.farm", "render.local", "render"]
     hosts = ["aftereffects"]
     optional = True
 
@@ -67,7 +71,8 @@ class ValidateSceneSettings(OptionalPyblishPluginMixin,
         if not self.is_active(instance.data):
             return
 
-        expected_settings = get_asset_settings()
+        asset_doc = instance.data["assetEntity"]
+        expected_settings = get_asset_settings(asset_doc)
         self.log.info("config from DB::{}".format(expected_settings))
 
         task_name = instance.data["anatomyData"]["task"]["name"]

openpype/hosts/blender/api/pipeline.py

@@ -10,6 +10,7 @@ from . import ops
 
 import pyblish.api
 
+from openpype.client import get_asset_by_name
 from openpype.pipeline import (
     schema,
     legacy_io,
@@ -83,11 +84,9 @@ def uninstall():
 
 
 def set_start_end_frames():
+    project_name = legacy_io.active_project()
     asset_name = legacy_io.Session["AVALON_ASSET"]
-    asset_doc = legacy_io.find_one({
-        "type": "asset",
-        "name": asset_name
-    })
+    asset_doc = get_asset_by_name(project_name, asset_name)
 
     scene = bpy.context.scene
 

openpype/hosts/blender/plugins/publish/extract_layout.py

@@ -1,13 +1,11 @@
 import os
 import json
 
-from bson.objectid import ObjectId
-
 import bpy
 import bpy_extras
 import bpy_extras.anim_utils
 
-from openpype.pipeline import legacy_io
+from openpype.client import get_representation_by_name
 from openpype.hosts.blender.api import plugin
 from openpype.hosts.blender.api.pipeline import AVALON_PROPERTY
 import openpype.api
@@ -131,43 +129,32 @@ class ExtractLayout(openpype.api.Extractor):
 
         fbx_count = 0
 
+        project_name = instance.context.data["projectEntity"]["name"]
         for asset in asset_group.children:
             metadata = asset.get(AVALON_PROPERTY)
 
-            parent = metadata["parent"]
+            version_id = metadata["parent"]
             family = metadata["family"]
 
-            self.log.debug("Parent: {}".format(parent))
+            self.log.debug("Parent: {}".format(version_id))
             # Get blend reference
-            blend = legacy_io.find_one(
-                {
-                    "type": "representation",
-                    "parent": ObjectId(parent),
-                    "name": "blend"
-                },
-                projection={"_id": True})
+            blend = get_representation_by_name(
+                project_name, "blend", version_id, fields=["_id"]
+            )
             blend_id = None
             if blend:
                 blend_id = blend["_id"]
             # Get fbx reference
-            fbx = legacy_io.find_one(
-                {
-                    "type": "representation",
-                    "parent": ObjectId(parent),
-                    "name": "fbx"
-                },
-                projection={"_id": True})
+            fbx = get_representation_by_name(
+                project_name, "fbx", version_id, fields=["_id"]
+            )
             fbx_id = None
             if fbx:
                 fbx_id = fbx["_id"]
             # Get abc reference
-            abc = legacy_io.find_one(
-                {
-                    "type": "representation",
-                    "parent": ObjectId(parent),
-                    "name": "abc"
-                },
-                projection={"_id": True})
+            abc = get_representation_by_name(
+                project_name, "abc", version_id, fields=["_id"]
+            )
             abc_id = None
             if abc:
                 abc_id = abc["_id"]

openpype/hosts/flame/plugins/publish/extract_subset_resources.py

@@ -1,5 +1,6 @@
 import os
 import re
+import tempfile
 from pprint import pformat
 from copy import deepcopy
 
@@ -420,3 +421,30 @@ class ExtractSubsetResources(openpype.api.Extractor):
                 "Path `{}` is containing more that one clip".format(path)
             )
         return clips[0]
+
+    def staging_dir(self, instance):
+        """Provide a temporary directory in which to store extracted files
+
+        Upon calling this method the staging directory is stored inside
+        the instance.data['stagingDir']
+        """
+        staging_dir = instance.data.get('stagingDir', None)
+        openpype_temp_dir = os.getenv("OPENPYPE_TEMP_DIR")
+
+        if not staging_dir:
+            if openpype_temp_dir and os.path.exists(openpype_temp_dir):
+                staging_dir = os.path.normpath(
+                    tempfile.mkdtemp(
+                        prefix="pyblish_tmp_",
+                        dir=openpype_temp_dir
+                    )
+                )
+            else:
+                staging_dir = os.path.normpath(
+                    tempfile.mkdtemp(prefix="pyblish_tmp_")
+                )
+            instance.data['stagingDir'] = staging_dir
+
+        instance.context.data["cleanupFullPaths"].append(staging_dir)
+
+        return staging_dir

openpype/hosts/harmony/api/README.md

@@ -610,7 +610,8 @@ class ImageSequenceLoader(load.LoaderPlugin):
     def update(self, container, representation):
         node = container.pop("node")
 
-        version = legacy_io.find_one({"_id": representation["parent"]})
+        project_name = legacy_io.active_project()
+        version = get_version_by_id(project_name, representation["parent"])
         files = []
         for f in version["data"]["files"]:
             files.append(

openpype/hosts/harmony/api/pipeline.py

@@ -2,10 +2,10 @@ import os
 from pathlib import Path
 import logging
 
-from bson.objectid import ObjectId
 import pyblish.api
 
 from openpype import lib
+from openpype.client import get_representation_by_id
 from openpype.lib import register_event_callback
 from openpype.pipeline import (
     legacy_io,
@@ -104,22 +104,20 @@ def check_inventory():
     If it does it will colorize outdated nodes and display warning message
     in Harmony.
     """
-    if not lib.any_outdated():
-        return
 
+    project_name = legacy_io.active_project()
     outdated_containers = []
     for container in ls():
-        representation = container['representation']
-        representation_doc = legacy_io.find_one(
-            {
-                "_id": ObjectId(representation),
-                "type": "representation"
-            },
-            projection={"parent": True}
+        representation_id = container['representation']
+        representation_doc = get_representation_by_id(
+            project_name, representation_id, fields=["parent"]
         )
         if representation_doc and not lib.is_latest(representation_doc):
             outdated_containers.append(container)
 
+    if not outdated_containers:
+        return
+
     # Colour nodes.
     outdated_nodes = []
     for container in outdated_containers:

openpype/hosts/harmony/plugins/publish/validate_audio.py

@@ -47,6 +47,6 @@ class ValidateAudio(pyblish.api.InstancePlugin):
         formatting_data = {
             "audio_url": audio_path
         }
-        if os.path.isfile(audio_path):
+        if not os.path.isfile(audio_path):
             raise PublishXmlValidationError(self, msg,
                                             formatting_data=formatting_data)

openpype/hosts/houdini/api/lib.py

@@ -130,6 +130,8 @@ def get_output_parameter(node):
     elif node_type == "arnold":
         if node.evalParm("ar_ass_export_enable"):
             return node.parm("ar_ass_file")
+    elif node_type == "Redshift_Proxy_Output":
+        return node.parm("RS_archive_file")
 
     raise TypeError("Node type '%s' not supported" % node_type)
 

openpype/hosts/houdini/plugins/create/create_redshift_proxy.py

@@ -0,0 +1,48 @@
+from openpype.hosts.houdini.api import plugin
+
+
+class CreateRedshiftProxy(plugin.Creator):
+    """Redshift Proxy"""
+
+    label = "Redshift Proxy"
+    family = "redshiftproxy"
+    icon = "magic"
+
+    def __init__(self, *args, **kwargs):
+        super(CreateRedshiftProxy, self).__init__(*args, **kwargs)
+
+        # Remove the active, we are checking the bypass flag of the nodes
+        self.data.pop("active", None)
+
+        # Redshift provides a `Redshift_Proxy_Output` node type which shows
+        # a limited set of parameters by default and is set to extract a
+        # Redshift Proxy. However when "imprinting" extra parameters needed
+        # for OpenPype it starts showing all its parameters again. It's unclear
+        # why this happens.
+        # TODO: Somehow enforce so that it only shows the original limited
+        #       attributes of the Redshift_Proxy_Output node type
+        self.data.update({"node_type": "Redshift_Proxy_Output"})
+
+    def _process(self, instance):
+        """Creator main entry point.
+
+        Args:
+            instance (hou.Node): Created Houdini instance.
+
+        """
+        parms = {
+            "RS_archive_file": '$HIP/pyblish/`chs("subset")`.$F4.rs',
+        }
+
+        if self.nodes:
+            node = self.nodes[0]
+            path = node.path()
+            parms["RS_archive_sopPath"] = path
+
+        instance.setParms(parms)
+
+        # Lock some Avalon attributes
+        to_lock = ["family", "id"]
+        for name in to_lock:
+            parm = instance.parm(name)
+            parm.lock(True)

openpype/hosts/houdini/plugins/publish/collect_frames.py

@@ -20,7 +20,7 @@ class CollectFrames(pyblish.api.InstancePlugin):
 
     order = pyblish.api.CollectorOrder
     label = "Collect Frames"
-    families = ["vdbcache", "imagesequence", "ass"]
+    families = ["vdbcache", "imagesequence", "ass", "redshiftproxy"]
 
     def process(self, instance):
 

openpype/hosts/houdini/plugins/publish/collect_output_node.py

@@ -12,6 +12,7 @@ class CollectOutputSOPPath(pyblish.api.InstancePlugin):
         "imagesequence",
         "usd",
         "usdrender",
+        "redshiftproxy"
     ]
 
     hosts = ["houdini"]
@@ -54,6 +55,8 @@ class CollectOutputSOPPath(pyblish.api.InstancePlugin):
             else:
                 out_node = node.parm("loppath").evalAsNode()
 
+        elif node_type == "Redshift_Proxy_Output":
+            out_node = node.parm("RS_archive_sopPath").evalAsNode()
         else:
             raise ValueError(
                 "ROP node type '%s' is" " not supported." % node_type

openpype/hosts/houdini/plugins/publish/extract_redshift_proxy.py

@@ -0,0 +1,48 @@
+import os
+
+import pyblish.api
+import openpype.api
+from openpype.hosts.houdini.api.lib import render_rop
+
+
+class ExtractRedshiftProxy(openpype.api.Extractor):
+
+    order = pyblish.api.ExtractorOrder + 0.1
+    label = "Extract Redshift Proxy"
+    families = ["redshiftproxy"]
+    hosts = ["houdini"]
+
+    def process(self, instance):
+
+        ropnode = instance[0]
+
+        # Get the filename from the filename parameter
+        # `.evalParm(parameter)` will make sure all tokens are resolved
+        output = ropnode.evalParm("RS_archive_file")
+        staging_dir = os.path.normpath(os.path.dirname(output))
+        instance.data["stagingDir"] = staging_dir
+        file_name = os.path.basename(output)
+
+        self.log.info("Writing Redshift Proxy '%s' to '%s'" % (file_name,
+                                                               staging_dir))
+
+        render_rop(ropnode)
+
+        output = instance.data["frames"]
+
+        if "representations" not in instance.data:
+            instance.data["representations"] = []
+
+        representation = {
+            "name": "rs",
+            "ext": "rs",
+            "files": output,
+            "stagingDir": staging_dir,
+        }
+
+        # A single frame may also be rendered without start/end frame.
+        if "frameStart" in instance.data and "frameEnd" in instance.data:
+            representation["frameStart"] = instance.data["frameStart"]
+            representation["frameEnd"] = instance.data["frameEnd"]
+
+        instance.data["representations"].append(representation)

openpype/hosts/maya/api/lib.py

@@ -2126,9 +2126,11 @@ def set_scene_resolution(width, height, pixelAspect):
 
     control_node = "defaultResolution"
     current_renderer = cmds.getAttr("defaultRenderGlobals.currentRenderer")
+    aspect_ratio_attr = "deviceAspectRatio"
 
     # Give VRay a helping hand as it is slightly different from the rest
     if current_renderer == "vray":
+        aspect_ratio_attr = "aspectRatio"
         vray_node = "vraySettings"
         if cmds.objExists(vray_node):
             control_node = vray_node
@@ -2141,7 +2143,8 @@ def set_scene_resolution(width, height, pixelAspect):
     cmds.setAttr("%s.height" % control_node, height)
 
     deviceAspectRatio = ((float(width) / float(height)) * float(pixelAspect))
-    cmds.setAttr("%s.deviceAspectRatio" % control_node, deviceAspectRatio)
+    cmds.setAttr(
+        "{}.{}".format(control_node, aspect_ratio_attr), deviceAspectRatio)
     cmds.setAttr("%s.pixelAspect" % control_node, pixelAspect)
 
 

openpype/hosts/maya/plugins/create/create_multiverse_usd.py

@@ -16,7 +16,7 @@ class CreateMultiverseUsd(plugin.Creator):
         self.data.update(lib.collect_animation_data(True))
 
         self.data["fileFormat"] = ["usd", "usda", "usdz"]
-        self.data["stripNamespaces"] = False
+        self.data["stripNamespaces"] = True
         self.data["mergeTransformAndShape"] = False
         self.data["writeAncestors"] = True
         self.data["flattenParentXforms"] = False
@@ -37,15 +37,15 @@ class CreateMultiverseUsd(plugin.Creator):
         self.data["writeUVs"] = True
         self.data["writeColorSets"] = False
         self.data["writeTangents"] = False
-        self.data["writeRefPositions"] = False
+        self.data["writeRefPositions"] = True
         self.data["writeBlendShapes"] = False
-        self.data["writeDisplayColor"] = False
+        self.data["writeDisplayColor"] = True
         self.data["writeSkinWeights"] = False
         self.data["writeMaterialAssignment"] = False
         self.data["writeHardwareShader"] = False
         self.data["writeShadingNetworks"] = False
         self.data["writeTransformMatrix"] = True
-        self.data["writeUsdAttributes"] = False
+        self.data["writeUsdAttributes"] = True
         self.data["writeInstancesAsReferences"] = False
         self.data["timeVaryingTopology"] = False
         self.data["customMaterialNamespace"] = ''

openpype/hosts/maya/plugins/create/create_yeti_cache.py

@@ -22,7 +22,8 @@ class CreateYetiCache(plugin.Creator):
         # Add animation data without step and handles
         anim_data = lib.collect_animation_data()
         anim_data.pop("step")
-        anim_data.pop("handles")
+        anim_data.pop("handleStart")
+        anim_data.pop("handleEnd")
         self.data.update(anim_data)
 
         # Add samples

openpype/hosts/maya/plugins/load/load_yeti_cache.py

@@ -1,15 +1,13 @@
 import os
 import json
 import re
-import glob
 from collections import defaultdict
-from pprint import pprint
 
+import clique
 from maya import cmds
 
 from openpype.api import get_project_settings
 from openpype.pipeline import (
-    legacy_io,
     load,
     get_representation_path
 )
@@ -17,7 +15,15 @@ from openpype.hosts.maya.api import lib
 from openpype.hosts.maya.api.pipeline import containerise
 
 
+def set_attribute(node, attr, value):
+    """Wrapper of set attribute which ignores None values"""
+    if value is None:
+        return
+    lib.set_attribute(node, attr, value)
+
+
 class YetiCacheLoader(load.LoaderPlugin):
+    """Load Yeti Cache with one or more Yeti nodes"""
 
     families = ["yeticache", "yetiRig"]
     representations = ["fur"]
@@ -28,6 +34,16 @@ class YetiCacheLoader(load.LoaderPlugin):
     color = "orange"
 
     def load(self, context, name=None, namespace=None, data=None):
+        """Loads a .fursettings file defining how to load .fur sequences
+
+        A single yeticache or yetiRig can have more than a single pgYetiMaya
+        nodes and thus load more than a single yeti.fur sequence.
+
+        The .fursettings file defines what the node names should be and also
+        what "cbId" attribute they should receive to match the original source
+        and allow published looks to also work for Yeti rigs and its caches.
+
+        """
 
         try:
             family = context["representation"]["context"]["family"]
@@ -43,22 +59,11 @@ class YetiCacheLoader(load.LoaderPlugin):
         if not cmds.pluginInfo("pgYetiMaya", query=True, loaded=True):
             cmds.loadPlugin("pgYetiMaya", quiet=True)
 
-        # Get JSON
-        fbase = re.search(r'^(.+)\.(\d+|#+)\.fur', self.fname)
-        if not fbase:
-            raise RuntimeError('Cannot determine fursettings file path')
-        settings_fname = "{}.fursettings".format(fbase.group(1))
-        with open(settings_fname, "r") as fp:
-            fursettings = json.load(fp)
-
-        # Check if resources map exists
-        # Get node name from JSON
-        if "nodes" not in fursettings:
-            raise RuntimeError("Encountered invalid data, expect 'nodes' in "
-                               "fursettings.")
-
-        node_data = fursettings["nodes"]
-        nodes = self.create_nodes(namespace, node_data)
+        # Create Yeti cache nodes according to settings
+        settings = self.read_settings(self.fname)
+        nodes = []
+        for node in settings["nodes"]:
+            nodes.extend(self.create_node(namespace, node))
 
         group_name = "{}:{}".format(namespace, name)
         group_node = cmds.group(nodes, name=group_name)
@@ -111,28 +116,14 @@ class YetiCacheLoader(load.LoaderPlugin):
 
     def update(self, container, representation):
 
-        legacy_io.install()
         namespace = container["namespace"]
         container_node = container["objectName"]
 
-        fur_settings = legacy_io.find_one(
-            {"parent": representation["parent"], "name": "fursettings"}
-        )
-
-        pprint({"parent": representation["parent"], "name": "fursettings"})
-        pprint(fur_settings)
-        assert fur_settings is not None, (
-            "cannot find fursettings representation"
-        )
-
-        settings_fname = get_representation_path(fur_settings)
         path = get_representation_path(representation)
-        # Get all node data
-        with open(settings_fname, "r") as fp:
-            settings = json.load(fp)
+        settings = self.read_settings(path)
 
         # Collect scene information of asset
-        set_members = cmds.sets(container["objectName"], query=True)
+        set_members = lib.get_container_members(container)
         container_root = lib.get_container_transforms(container,
                                                       members=set_members,
                                                       root=True)
@@ -147,7 +138,7 @@ class YetiCacheLoader(load.LoaderPlugin):
         # Re-assemble metadata with cbId as keys
         meta_data_lookup = {n["cbId"]: n for n in settings["nodes"]}
 
-        # Compare look ups and get the nodes which ar not relevant any more
+        # Delete nodes by "cbId" that are not in the updated version
         to_delete_lookup = {cb_id for cb_id in scene_lookup.keys() if
                             cb_id not in meta_data_lookup}
         if to_delete_lookup:
@@ -163,25 +154,18 @@ class YetiCacheLoader(load.LoaderPlugin):
                                                 fullPath=True) or []
                 to_remove.extend(shapes + transforms)
 
-                # Remove id from look uop
+                # Remove id from lookup
                 scene_lookup.pop(_id, None)
 
             cmds.delete(to_remove)
 
-        # replace frame in filename with %04d
-        RE_frame = re.compile(r"(\d+)(\.fur)$")
-        file_name = re.sub(RE_frame, r"%04d\g<2>", os.path.basename(path))
-        for cb_id, data in meta_data_lookup.items():
-
-            # Update cache file name
-            data["attrs"]["cacheFileName"] = os.path.join(
-                os.path.dirname(path), file_name)
+        for cb_id, node_settings in meta_data_lookup.items():
 
             if cb_id not in scene_lookup:
-
+                # Create new nodes
                 self.log.info("Creating new nodes ..")
 
-                new_nodes = self.create_nodes(namespace, [data])
+                new_nodes = self.create_node(namespace, node_settings)
                 cmds.sets(new_nodes, addElement=container_node)
                 cmds.parent(new_nodes, container_root)
 
@@ -218,14 +202,8 @@ class YetiCacheLoader(load.LoaderPlugin):
                                                     children=True)
                     yeti_node = yeti_nodes[0]
 
-                    for attr, value in data["attrs"].items():
-                        # handle empty attribute strings. Those are reported
-                        # as None, so their type is NoneType and this is not
-                        # supported on attributes in Maya. We change it to
-                        # empty string.
-                        if value is None:
-                            value = ""
-                        lib.set_attribute(attr, value, yeti_node)
+                    for attr, value in node_settings["attrs"].items():
+                        set_attribute(attr, value, yeti_node)
 
         cmds.setAttr("{}.representation".format(container_node),
                      str(representation["_id"]),
@@ -235,7 +213,6 @@ class YetiCacheLoader(load.LoaderPlugin):
         self.update(container, representation)
 
     # helper functions
-
     def create_namespace(self, asset):
         """Create a unique namespace
         Args:
@@ -253,100 +230,122 @@ class YetiCacheLoader(load.LoaderPlugin):
 
         return namespace
 
-    def validate_cache(self, filename, pattern="%04d"):
-        """Check if the cache has more than 1 frame
+    def get_cache_node_filepath(self, root, node_name):
+        """Get the cache file path for one of the yeti nodes.
 
-        All caches with more than 1 frame need to be called with `%04d`
-        If the cache has only one frame we return that file name as we assume
+        All caches with more than 1 frame need cache file name set with `%04d`
+        If the cache has only one frame we return the file name as we assume
         it is a snapshot.
 
+        This expects the files to be named after the "node name" through
+        exports with <Name> in Yeti.
+
         Args:
-            filename(str)
-            pattern(str)
+            root(str): Folder containing cache files to search in.
+            node_name(str): Node name to search cache files for
 
         Returns:
-            str
+            str: Cache file path value needed for cacheFileName attribute
 
         """
 
-        glob_pattern = filename.replace(pattern, "*")
+        name = node_name.replace(":", "_")
+        pattern = r"^({name})(\.[0-4]+)?(\.fur)$".format(name=re.escape(name))
 
-        escaped = re.escape(filename)
-        re_pattern = escaped.replace(pattern, "-?[0-9]+")
-
-        files = glob.glob(glob_pattern)
-        files = [str(f) for f in files if re.match(re_pattern, f)]
+        files = [fname for fname in os.listdir(root) if re.match(pattern,
+                                                                 fname)]
+        if not files:
+            self.log.error("Could not find cache files for '{}' "
+                           "with pattern {}".format(node_name, pattern))
+            return
 
         if len(files) == 1:
-            return files[0]
-        elif len(files) == 0:
-            self.log.error("Could not find cache files for '%s'" % filename)
+            # Single file
+            return os.path.join(root, files[0])
 
-        return filename
+        # Get filename for the sequence with padding
+        collections, remainder = clique.assemble(files)
+        assert not remainder, "This is a bug"
+        assert len(collections) == 1, "This is a bug"
+        collection = collections[0]
 
-    def create_nodes(self, namespace, settings):
+        # Formats name as {head}%d{tail} like cache.%04d.fur
+        fname = collection.format("{head}{padding}{tail}")
+        return os.path.join(root, fname)
+
+    def create_node(self, namespace, node_settings):
         """Create nodes with the correct namespace and settings
 
         Args:
             namespace(str): namespace
-            settings(list): list of dictionaries
+            node_settings(dict): Single "nodes" entry from .fursettings file.
 
         Returns:
-             list
+             list: Created nodes
 
         """
-
         nodes = []
-        for node_settings in settings:
 
-            # Create pgYetiMaya node
-            original_node = node_settings["name"]
-            node_name = "{}:{}".format(namespace, original_node)
-            yeti_node = cmds.createNode("pgYetiMaya", name=node_name)
+        # Get original names and ids
+        orig_transform_name = node_settings["transform"]["name"]
+        orig_shape_name = node_settings["name"]
 
-            # Create transform node
-            transform_node = node_name.rstrip("Shape")
+        # Add namespace
+        transform_name = "{}:{}".format(namespace, orig_transform_name)
+        shape_name = "{}:{}".format(namespace, orig_shape_name)
 
-            lib.set_id(transform_node, node_settings["transform"]["cbId"])
-            lib.set_id(yeti_node, node_settings["cbId"])
+        # Create pgYetiMaya node
+        transform_node = cmds.createNode("transform",
+                                         name=transform_name)
+        yeti_node = cmds.createNode("pgYetiMaya",
+                                    name=shape_name,
+                                    parent=transform_node)
 
-            nodes.extend([transform_node, yeti_node])
+        lib.set_id(transform_node, node_settings["transform"]["cbId"])
+        lib.set_id(yeti_node, node_settings["cbId"])
 
-            # Ensure the node has no namespace identifiers
-            attributes = node_settings["attrs"]
+        nodes.extend([transform_node, yeti_node])
 
-            # Check if cache file name is stored
+        # Update attributes with defaults
+        attributes = node_settings["attrs"]
+        attributes.update({
+            "viewportDensity": 0.1,
+            "verbosity": 2,
+            "fileMode": 1,
 
-            # get number of # in path and convert it to C prinf format
-            # like %04d expected by Yeti
-            fbase = re.search(r'^(.+)\.(\d+|#+)\.fur', self.fname)
-            if not fbase:
-                raise RuntimeError('Cannot determine file path')
-            padding = len(fbase.group(2))
-            if "cacheFileName" not in attributes:
-                cache = "{}.%0{}d.fur".format(fbase.group(1), padding)
+            # Fix render stats, like Yeti's own
+            # ../scripts/pgYetiNode.mel script
+            "visibleInReflections": True,
+            "visibleInRefractions": True
+        })
 
-                self.validate_cache(cache)
-                attributes["cacheFileName"] = cache
+        # Apply attributes to pgYetiMaya node
+        for attr, value in attributes.items():
+            set_attribute(attr, value, yeti_node)
 
-            # Update attributes with requirements
-            attributes.update({"viewportDensity": 0.1,
-                               "verbosity": 2,
-                               "fileMode": 1})
-
-            # Apply attributes to pgYetiMaya node
-            for attr, value in attributes.items():
-                if value is None:
-                    continue
-                lib.set_attribute(attr, value, yeti_node)
-
-            # Fix for : YETI-6
-            # Fixes the render stats (this is literally taken from Perigrene's
-            # ../scripts/pgYetiNode.mel script)
-            cmds.setAttr("{}.visibleInReflections".format(yeti_node), True)
-            cmds.setAttr("{}.visibleInRefractions".format(yeti_node), True)
-
-            # Connect to the time node
-            cmds.connectAttr("time1.outTime", "%s.currentTime" % yeti_node)
+        # Connect to the time node
+        cmds.connectAttr("time1.outTime", "%s.currentTime" % yeti_node)
 
         return nodes
+
+    def read_settings(self, path):
+        """Read .fursettings file and compute some additional attributes"""
+
+        with open(path, "r") as fp:
+            fur_settings = json.load(fp)
+
+        if "nodes" not in fur_settings:
+            raise RuntimeError("Encountered invalid data, "
+                               "expected 'nodes' in fursettings.")
+
+        # Compute the cache file name values we want to set for the nodes
+        root = os.path.dirname(path)
+        for node in fur_settings["nodes"]:
+            cache_filename = self.get_cache_node_filepath(
+                root=root, node_name=node["name"])
+
+            attrs = node.get("attrs", {})       # allow 'attrs' to not exist
+            attrs["cacheFileName"] = cache_filename
+            node["attrs"] = attrs
+
+        return fur_settings

openpype/hosts/maya/plugins/publish/collect_instances.py

@@ -28,18 +28,19 @@ def get_all_children(nodes):
         dag = sel.getDagPath(0)
 
         iterator.reset(dag)
-        next(iterator)  # ignore self
+        # ignore self
+        iterator.next()  # noqa: B305
         while not iterator.isDone():
 
             path = iterator.fullPathName()
 
             if path in traversed:
                 iterator.prune()
-                next(iterator)
+                iterator.next()  # noqa: B305
                 continue
 
             traversed.add(path)
-            next(iterator)
+            iterator.next()  # noqa: B305
 
     return list(traversed)
 

openpype/hosts/maya/plugins/publish/collect_yeti_rig.py

@@ -43,11 +43,12 @@ class CollectYetiRig(pyblish.api.InstancePlugin):
 
         instance.data["resources"] = yeti_resources
 
-        # Force frame range for export
-        instance.data["frameStart"] = cmds.playbackOptions(
-                                        query=True, animationStartTime=True)
-        instance.data["frameEnd"] = cmds.playbackOptions(
-                                        query=True, animationStartTime=True)
+        # Force frame range for yeti cache export for the rig
+        start = cmds.playbackOptions(query=True, animationStartTime=True)
+        for key in ["frameStart", "frameEnd",
+                    "frameStartHandle", "frameEndHandle"]:
+            instance.data[key] = start
+        instance.data["preroll"] = 0
 
     def collect_input_connections(self, instance):
         """Collect the inputs for all nodes in the input_SET"""

openpype/hosts/maya/plugins/publish/extract_camera_alembic.py

@@ -61,6 +61,7 @@ class ExtractCameraAlembic(openpype.api.Extractor):
 
             if bake_to_worldspace:
                 job_str += ' -worldSpace'
+
                 for member in members:
                     transform = cmds.listRelatives(
                         member, parent=True, fullPath=True)

openpype/hosts/maya/plugins/publish/extract_camera_mayaScene.py

@@ -172,6 +172,7 @@ class ExtractCameraMayaScene(openpype.api.Extractor):
                                                       dag=True,
                                                       shapes=True,
                                                       long=True)
+
                     attrs = {"backgroundColorR": 0.0,
                              "backgroundColorG": 0.0,
                              "backgroundColorB": 0.0,
@@ -189,6 +190,7 @@ class ExtractCameraMayaScene(openpype.api.Extractor):
                         unlock(plug)
                         cmds.setAttr(plug, value)
 
+
                     self.log.info("Performing extraction..")
                     cmds.select(cmds.ls(members, dag=True,
                                         shapes=True, long=True), noExpand=True)

openpype/hosts/maya/plugins/publish/extract_playblast.py

@@ -111,7 +111,8 @@ class ExtractPlayblast(openpype.api.Extractor):
         self.log.debug("playblast path  {}".format(path))
 
         collected_files = os.listdir(stagingdir)
-        collections, remainder = clique.assemble(collected_files)
+        collections, remainder = clique.assemble(collected_files,
+                                                 minimum_items=1)
 
         self.log.debug("filename {}".format(filename))
         frame_collection = None
@@ -134,10 +135,15 @@ class ExtractPlayblast(openpype.api.Extractor):
         # Add camera node name to representation data
         camera_node_name = pm.ls(camera)[0].getTransform().name()
 
+        collected_files = list(frame_collection)
+        # single frame file shouldn't be in list, only as a string
+        if len(collected_files) == 1:
+            collected_files = collected_files[0]
+
         representation = {
             'name': 'png',
             'ext': 'png',
-            'files': list(frame_collection),
+            'files': collected_files,
             "stagingDir": stagingdir,
             "frameStart": start,
             "frameEnd": end,

openpype/hosts/maya/plugins/publish/extract_yeti_cache.py

@@ -25,13 +25,10 @@ class ExtractYetiCache(openpype.api.Extractor):
         # Define extract output file path
         dirname = self.staging_dir(instance)
 
-        # Yeti related staging dirs
-        data_file = os.path.join(dirname, "yeti.fursettings")
-
         # Collect information for writing cache
-        start_frame = instance.data.get("frameStartHandle")
-        end_frame = instance.data.get("frameEndHandle")
-        preroll = instance.data.get("preroll")
+        start_frame = instance.data["frameStartHandle"]
+        end_frame = instance.data["frameEndHandle"]
+        preroll = instance.data["preroll"]
         if preroll > 0:
             start_frame -= preroll
 
@@ -57,32 +54,35 @@ class ExtractYetiCache(openpype.api.Extractor):
         cache_files = [x for x in os.listdir(dirname) if x.endswith(".fur")]
 
         self.log.info("Writing metadata file")
-        settings = instance.data.get("fursettings", None)
-        if settings is not None:
-            with open(data_file, "w") as fp:
-                json.dump(settings, fp, ensure_ascii=False)
+        settings = instance.data["fursettings"]
+        fursettings_path = os.path.join(dirname, "yeti.fursettings")
+        with open(fursettings_path, "w") as fp:
+            json.dump(settings, fp, ensure_ascii=False)
 
         # build representations
         if "representations" not in instance.data:
             instance.data["representations"] = []
 
         self.log.info("cache files: {}".format(cache_files[0]))
-        instance.data["representations"].append(
-            {
-                'name': 'fur',
-                'ext': 'fur',
-                'files': cache_files[0] if len(cache_files) == 1 else cache_files,
-                'stagingDir': dirname,
-                'frameStart': int(start_frame),
-                'frameEnd': int(end_frame)
-            }
-        )
+
+        # Workaround: We do not explicitly register these files with the
+        # representation solely so that we can write multiple sequences
+        # a single Subset without renaming - it's a bit of a hack
+        # TODO: Implement better way to manage this sort of integration
+        if 'transfers' not in instance.data:
+            instance.data['transfers'] = []
+
+        publish_dir = instance.data["publishDir"]
+        for cache_filename in cache_files:
+            src = os.path.join(dirname, cache_filename)
+            dst = os.path.join(publish_dir, os.path.basename(cache_filename))
+            instance.data['transfers'].append([src, dst])
 
         instance.data["representations"].append(
             {
-                'name': 'fursettings',
+                'name': 'fur',
                 'ext': 'fursettings',
-                'files': os.path.basename(data_file),
+                'files': os.path.basename(fursettings_path),
                 'stagingDir': dirname
             }
         )

openpype/hosts/maya/plugins/publish/extract_yeti_rig.py

@@ -124,8 +124,8 @@ class ExtractYetiRig(openpype.api.Extractor):
         settings_path = os.path.join(dirname, "yeti.rigsettings")
 
         # Yeti related staging dirs
-        maya_path = os.path.join(
-            dirname, "yeti_rig.{}".format(self.scene_type))
+        maya_path = os.path.join(dirname,
+                                 "yeti_rig.{}".format(self.scene_type))
 
         self.log.info("Writing metadata file")
 
@@ -157,7 +157,7 @@ class ExtractYetiRig(openpype.api.Extractor):
         input_set = next(i for i in instance if i == "input_SET")
 
         # Get all items
-        set_members = cmds.sets(input_set, query=True)
+        set_members = cmds.sets(input_set, query=True) or []
         set_members += cmds.listRelatives(set_members,
                                           allDescendents=True,
                                           fullPath=True) or []
@@ -167,7 +167,7 @@ class ExtractYetiRig(openpype.api.Extractor):
         resources = instance.data.get("resources", {})
         with disconnect_plugs(settings, members):
             with yetigraph_attribute_values(resources_dir, resources):
-                with maya.attribute_values(attr_value):
+                with lib.attribute_values(attr_value):
                     cmds.select(nodes, noExpand=True)
                     cmds.file(maya_path,
                               force=True,

openpype/hosts/maya/plugins/publish/validate_camera_contents.py

@@ -54,7 +54,6 @@ class ValidateCameraContents(pyblish.api.InstancePlugin):
             cls.log.info("not validating shapes in the content")
             return invalid
 
-
         # non-camera shapes
         valid_shapes = cmds.ls(shapes, type=('camera', 'locator'), long=True)
         shapes = set(shapes) - set(valid_shapes)

openpype/hosts/maya/plugins/publish/validate_mesh_shader_connections.py

@@ -12,28 +12,41 @@ def pairs(iterable):
         yield i, y
 
 
-def get_invalid_sets(shape):
-    """Get sets that are considered related but do not contain the shape.
+def get_invalid_sets(shapes):
+    """Return invalid sets for the given shapes.
 
-    In some scenarios Maya keeps connections to multiple shaders
-    even if just a single one is assigned on the full object.
+    This takes a list of shape nodes to cache the set members for overlapping
+    sets in the queries. This avoids many Maya set member queries.
 
-    These are related sets returned by `maya.cmds.listSets` that don't
-    actually have the shape as member.
+    Returns:
+        dict: Dictionary of shapes and their invalid sets, e.g.
+            {"pCubeShape": ["set1", "set2"]}
 
     """
 
-    invalid = []
-    sets = cmds.listSets(object=shape, t=1, extendToShape=False) or []
-    for s in sets:
-        members = cmds.sets(s, query=True, nodesOnly=True)
-        if not members:
-            invalid.append(s)
-            continue
+    cache = dict()
+    invalid = dict()
 
-        members = set(cmds.ls(members, long=True))
-        if shape not in members:
-            invalid.append(s)
+    # Collect the sets from the shape
+    for shape in shapes:
+        invalid_sets = []
+        sets = cmds.listSets(object=shape, t=1, extendToShape=False) or []
+        for set_ in sets:
+
+            members = cache.get(set_, None)
+            if members is None:
+                members = set(cmds.ls(cmds.sets(set_,
+                                                query=True,
+                                                nodesOnly=True), long=True))
+                cache[set_] = members
+
+            # If the shape is not actually present as a member of the set
+            # consider it invalid
+            if shape not in members:
+                invalid_sets.append(set_)
+
+        if invalid_sets:
+            invalid[shape] = invalid_sets
 
     return invalid
 
@@ -92,15 +105,9 @@ class ValidateMeshShaderConnections(pyblish.api.InstancePlugin):
     @staticmethod
     def get_invalid(instance):
 
-        shapes = cmds.ls(instance[:], dag=1, leaf=1, shapes=1, long=True)
-
-        # todo: allow to check anything that can have a shader
-        shapes = cmds.ls(shapes, noIntermediate=True, long=True, type="mesh")
-
-        invalid = []
-        for shape in shapes:
-            if get_invalid_sets(shape):
-                invalid.append(shape)
+        nodes = instance[:]
+        shapes = cmds.ls(nodes, noIntermediate=True, long=True, type="mesh")
+        invalid = get_invalid_sets(shapes).keys()
 
         return invalid
 
@@ -108,7 +115,7 @@ class ValidateMeshShaderConnections(pyblish.api.InstancePlugin):
     def repair(cls, instance):
 
         shapes = cls.get_invalid(instance)
-        for shape in shapes:
-            invalid_sets = get_invalid_sets(shape)
+        invalid = get_invalid_sets(shapes)
+        for shape, invalid_sets in invalid.items():
             for set_node in invalid_sets:
                 disconnect(shape, set_node)

openpype/hosts/maya/plugins/publish/validate_rendersettings.py

@@ -94,6 +94,7 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin):
     def get_invalid(cls, instance):
 
         invalid = False
+        multipart = False
 
         renderer = instance.data['renderer']
         layer = instance.data['setMembers']
@@ -113,6 +114,7 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin):
             "{aov_separator}", instance.data.get("aovSeparator", "_"))
 
         required_prefix = "maya/<scene>"
+        default_prefix = cls.ImagePrefixTokens[renderer]
 
         if not anim_override:
             invalid = True
@@ -213,14 +215,16 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin):
                     cls.log.error("Wrong image prefix [ {} ] - "
                                   "You can't use '<renderpass>' token "
                                   "with merge AOVs turned on".format(prefix))
+                default_prefix = re.sub(
+                    cls.R_AOV_TOKEN, "", default_prefix)
+                # remove aov token from prefix to pass validation
+                default_prefix = default_prefix.split("{aov_separator}")[0]
             elif not re.search(cls.R_AOV_TOKEN, prefix):
                 invalid = True
                 cls.log.error("Wrong image prefix [ {} ] - "
                               "doesn't have: '<renderpass>' or "
                               "token".format(prefix))
 
-        # prefix check
-        default_prefix = cls.ImagePrefixTokens[renderer]
         default_prefix = default_prefix.replace(
             "{aov_separator}", instance.data.get("aovSeparator", "_"))
         if prefix.lower() != default_prefix.lower():

openpype/hosts/nuke/plugins/publish/extract_render_local.py

@@ -7,7 +7,7 @@ import clique
 
 class NukeRenderLocal(openpype.api.Extractor):
     # TODO: rewrite docstring to nuke
-    """Render the current Fusion composition locally.
+    """Render the current Nuke composition locally.
 
     Extract the result of savers by starting a comp render
     This will run the local render of Fusion.

openpype/hosts/nuke/plugins/publish/extract_thumbnail.py

@@ -23,9 +23,13 @@ class ExtractThumbnail(openpype.api.Extractor):
     families = ["review"]
     hosts = ["nuke"]
 
-    # presets
+    # settings
+    use_rendered = False
+    bake_viewer_process = True
+    bake_viewer_input_process = True
     nodes = {}
 
+
     def process(self, instance):
         if "render.farm" in instance.data["families"]:
             return
@@ -38,11 +42,17 @@ class ExtractThumbnail(openpype.api.Extractor):
             self.render_thumbnail(instance)
 
     def render_thumbnail(self, instance):
+        first_frame = instance.data["frameStartHandle"]
+        last_frame = instance.data["frameEndHandle"]
+
+        # find frame range and define middle thumb frame
+        mid_frame = int((last_frame - first_frame) / 2)
+
         node = instance[0]  # group node
         self.log.info("Creating staging dir...")
 
         if "representations" not in instance.data:
-            instance.data["representations"] = list()
+            instance.data["representations"] = []
 
         staging_dir = os.path.normpath(
             os.path.dirname(instance.data['path']))
@@ -53,7 +63,11 @@ class ExtractThumbnail(openpype.api.Extractor):
             "StagingDir `{0}`...".format(instance.data["stagingDir"]))
 
         temporary_nodes = []
+
+        # try to connect already rendered images
+        previous_node = node
         collection = instance.data.get("collection", None)
+        self.log.debug("__ collection: `{}`".format(collection))
 
         if collection:
             # get path
@@ -61,40 +75,45 @@ class ExtractThumbnail(openpype.api.Extractor):
                 "{head}{padding}{tail}"))
             fhead = collection.format("{head}")
 
-            # get first and last frame
-            first_frame = min(collection.indexes)
-            last_frame = max(collection.indexes)
+            thumb_fname = list(collection)[mid_frame]
         else:
-            fname = os.path.basename(instance.data.get("path", None))
+            fname = thumb_fname = os.path.basename(
+                instance.data.get("path", None))
             fhead = os.path.splitext(fname)[0] + "."
-            first_frame = instance.data.get("frameStart", None)
-            last_frame = instance.data.get("frameEnd", None)
+
+        self.log.debug("__ fhead: `{}`".format(fhead))
 
         if "#" in fhead:
             fhead = fhead.replace("#", "")[:-1]
 
-        path_render = os.path.join(staging_dir, fname).replace("\\", "/")
-        # check if file exist otherwise connect to write node
-        if os.path.isfile(path_render):
+        path_render = os.path.join(
+            staging_dir, thumb_fname).replace("\\", "/")
+        self.log.debug("__ path_render: `{}`".format(path_render))
+
+        if self.use_rendered and os.path.isfile(path_render):
+            # check if file exist otherwise connect to write node
             rnode = nuke.createNode("Read")
 
             rnode["file"].setValue(path_render)
 
-            rnode["first"].setValue(first_frame)
-            rnode["origfirst"].setValue(first_frame)
-            rnode["last"].setValue(last_frame)
-            rnode["origlast"].setValue(last_frame)
+            # turn it raw if none of baking is ON
+            if all([
+                not self.bake_viewer_input_process,
+                not self.bake_viewer_process
+            ]):
+                rnode["raw"].setValue(True)
+
             temporary_nodes.append(rnode)
             previous_node = rnode
-        else:
-            previous_node = node
 
-        # get input process and connect it to baking
-        ipn = self.get_view_process_node()
-        if ipn is not None:
-            ipn.setInput(0, previous_node)
-            previous_node = ipn
-            temporary_nodes.append(ipn)
+        # bake viewer input look node into thumbnail image
+        if self.bake_viewer_input_process:
+            # get input process and connect it to baking
+            ipn = self.get_view_process_node()
+            if ipn is not None:
+                ipn.setInput(0, previous_node)
+                previous_node = ipn
+                temporary_nodes.append(ipn)
 
         reformat_node = nuke.createNode("Reformat")
 
@@ -110,10 +129,12 @@ class ExtractThumbnail(openpype.api.Extractor):
         previous_node = reformat_node
         temporary_nodes.append(reformat_node)
 
-        dag_node = nuke.createNode("OCIODisplay")
-        dag_node.setInput(0, previous_node)
-        previous_node = dag_node
-        temporary_nodes.append(dag_node)
+        # bake viewer colorspace into thumbnail image
+        if self.bake_viewer_process:
+            dag_node = nuke.createNode("OCIODisplay")
+            dag_node.setInput(0, previous_node)
+            previous_node = dag_node
+            temporary_nodes.append(dag_node)
 
         # create write node
         write_node = nuke.createNode("Write")
@@ -128,26 +149,18 @@ class ExtractThumbnail(openpype.api.Extractor):
         temporary_nodes.append(write_node)
         tags = ["thumbnail", "publish_on_farm"]
 
-        # retime for
-        mid_frame = int((int(last_frame) - int(first_frame)) / 2) \
-            + int(first_frame)
-        first_frame = int(last_frame) / 2
-        last_frame = int(last_frame) / 2
-
         repre = {
             'name': name,
             'ext': "jpg",
             "outputName": "thumb",
             'files': file,
             "stagingDir": staging_dir,
-            "frameStart": first_frame,
-            "frameEnd": last_frame,
             "tags": tags
         }
         instance.data["representations"].append(repre)
 
         # Render frames
-        nuke.execute(write_node.name(), int(mid_frame), int(mid_frame))
+        nuke.execute(write_node.name(), mid_frame, mid_frame)
 
         self.log.debug(
             "representations: {}".format(instance.data["representations"]))

openpype/hosts/nuke/plugins/publish/precollect_instances.py

@@ -151,15 +151,11 @@ class PreCollectNukeInstances(pyblish.api.ContextPlugin):
                 "resolutionWidth": resolution_width,
                 "resolutionHeight": resolution_height,
                 "pixelAspect": pixel_aspect,
-                "review": review
+                "review": review,
+                "representations": []
 
             })
             self.log.info("collected instance: {}".format(instance.data))
             instances.append(instance)
 
-        # create instances in context data if not are created yet
-        if not context.data.get("instances"):
-            context.data["instances"] = list()
-
-        context.data["instances"].extend(instances)
         self.log.debug("context: {}".format(context))

openpype/hosts/nuke/plugins/publish/precollect_workfile.py

@@ -17,7 +17,7 @@ class CollectWorkfile(pyblish.api.ContextPlugin):
     label = "Pre-collect Workfile"
     hosts = ['nuke']
 
-    def process(self, context):
+    def process(self, context):  # sourcery skip: avoid-builtin-shadow
         root = nuke.root()
 
         current_file = os.path.normpath(nuke.root().name())
@@ -74,20 +74,6 @@ class CollectWorkfile(pyblish.api.ContextPlugin):
         }
         context.data.update(script_data)
 
-        # creating instance data
-        instance.data.update({
-            "subset": subset,
-            "label": base_name,
-            "name": base_name,
-            "publish": root.knob('publish').value(),
-            "family": family,
-            "families": [family],
-            "representations": list()
-        })
-
-        # adding basic script data
-        instance.data.update(script_data)
-
         # creating representation
         representation = {
             'name': 'nk',
@@ -96,12 +82,18 @@ class CollectWorkfile(pyblish.api.ContextPlugin):
             "stagingDir": staging_dir,
         }
 
-        instance.data["representations"].append(representation)
+        # creating instance data
+        instance.data.update({
+            "subset": subset,
+            "label": base_name,
+            "name": base_name,
+            "publish": root.knob('publish').value(),
+            "family": family,
+            "families": [family],
+            "representations": [representation]
+        })
+
+        # adding basic script data
+        instance.data.update(script_data)
 
         self.log.info('Publishing script version')
-
-        # create instances in context data if not are created yet
-        if not context.data.get("instances"):
-            context.data["instances"] = list()
-
-        context.data["instances"].append(instance)

openpype/hosts/nuke/plugins/publish/precollect_writes.py

@@ -72,12 +72,12 @@ class CollectNukeWrites(pyblish.api.InstancePlugin):
             if "representations" not in instance.data:
                 instance.data["representations"] = list()
 
-                representation = {
-                    'name': ext,
-                    'ext': ext,
-                    "stagingDir": output_dir,
-                    "tags": list()
-                }
+            representation = {
+                'name': ext,
+                'ext': ext,
+                "stagingDir": output_dir,
+                "tags": list()
+            }
 
             try:
                 collected_frames = [f for f in os.listdir(output_dir)
@@ -175,6 +175,11 @@ class CollectNukeWrites(pyblish.api.InstancePlugin):
                 "frameEndHandle": last_frame,
             })
 
+            # make sure rendered sequence on farm will
+            # be used for exctract review
+            if not instance.data["review"]:
+                instance.data["useSequenceForReview"] = False
+
         # * Add audio to instance if exists.
         # Find latest versions document
         version_doc = pype.get_latest_version(

openpype/hosts/nuke/startup/KnobScripter/__init__.py

@@ -1 +0,0 @@
-import knob_scripter

openpype/hosts/nuke/startup/init.py

@@ -1,4 +0,0 @@
-import nuke
-
-# default write mov
-nuke.knobDefault('Write.mov.colorspace', 'sRGB')

openpype/hosts/photoshop/plugins/create/workfile_creator.py

@@ -1,4 +1,5 @@
 import openpype.hosts.photoshop.api as api
+from openpype.client import get_asset_by_name
 from openpype.pipeline import (
     AutoCreator,
     CreatedInstance,
@@ -40,10 +41,7 @@ class PSWorkfileCreator(AutoCreator):
         task_name = legacy_io.Session["AVALON_TASK"]
         host_name = legacy_io.Session["AVALON_APP"]
         if existing_instance is None:
-            asset_doc = legacy_io.find_one({
-                "type": "asset",
-                "name": asset_name
-            })
+            asset_doc = get_asset_by_name(project_name, asset_name)
             subset_name = self.get_subset_name(
                 variant, task_name, asset_doc, project_name, host_name
             )
@@ -67,10 +65,7 @@ class PSWorkfileCreator(AutoCreator):
             existing_instance["asset"] != asset_name
             or existing_instance["task"] != task_name
         ):
-            asset_doc = legacy_io.find_one({
-                "type": "asset",
-                "name": asset_name
-            })
+            asset_doc = get_asset_by_name(project_name, asset_name)
             subset_name = self.get_subset_name(
                 variant, task_name, asset_doc, project_name, host_name
             )

openpype/hosts/standalonepublisher/plugins/publish/collect_bulk_mov_instances.py

@@ -3,7 +3,7 @@ import json
 import pyblish.api
 
 from openpype.lib import get_subset_name_with_asset_doc
-from openpype.pipeline import legacy_io
+from openpype.client import get_asset_by_name
 
 
 class CollectBulkMovInstances(pyblish.api.InstancePlugin):
@@ -24,12 +24,9 @@ class CollectBulkMovInstances(pyblish.api.InstancePlugin):
 
     def process(self, instance):
         context = instance.context
+        project_name = context.data["projectEntity"]["name"]
         asset_name = instance.data["asset"]
-
-        asset_doc = legacy_io.find_one({
-            "type": "asset",
-            "name": asset_name
-        })
+        asset_doc = get_asset_by_name(project_name, asset_name)
         if not asset_doc:
             raise AssertionError((
                 "Couldn't find Asset document with name \"{}\""
@@ -52,7 +49,7 @@ class CollectBulkMovInstances(pyblish.api.InstancePlugin):
             self.subset_name_variant,
             task_name,
             asset_doc,
-            legacy_io.Session["AVALON_PROJECT"]
+            project_name
         )
         instance_name = f"{asset_name}_{subset_name}"
 

openpype/hosts/standalonepublisher/plugins/publish/collect_hierarchy.py

@@ -3,7 +3,7 @@ import re
 from copy import deepcopy
 import pyblish.api
 
-from openpype.pipeline import legacy_io
+from openpype.client import get_asset_by_id
 
 
 class CollectHierarchyInstance(pyblish.api.ContextPlugin):
@@ -61,27 +61,32 @@ class CollectHierarchyInstance(pyblish.api.ContextPlugin):
             **instance.data["anatomyData"])
 
     def create_hierarchy(self, instance):
-        parents = list()
-        hierarchy = list()
-        visual_hierarchy = [instance.context.data["assetEntity"]]
+        asset_doc = instance.context.data["assetEntity"]
+        project_doc = instance.context.data["projectEntity"]
+        project_name = project_doc["name"]
+        visual_hierarchy = [asset_doc]
+        current_doc = asset_doc
         while True:
-            visual_parent = legacy_io.find_one(
-                {"_id": visual_hierarchy[-1]["data"]["visualParent"]}
-            )
-            if visual_parent:
-                visual_hierarchy.append(visual_parent)
-            else:
-                visual_hierarchy.append(
-                    instance.context.data["projectEntity"])
+            visual_parent_id = current_doc["data"]["visualParent"]
+            visual_parent = None
+            if visual_parent_id:
+                visual_parent = get_asset_by_id(project_name, visual_parent_id)
+
+            if not visual_parent:
+                visual_hierarchy.append(project_doc)
                 break
+            visual_hierarchy.append(visual_parent)
+            current_doc = visual_parent
 
         # add current selection context hierarchy from standalonepublisher
+        parents = list()
         for entity in reversed(visual_hierarchy):
             parents.append({
                 "entity_type": entity["data"]["entityType"],
                 "entity_name": entity["name"]
             })
 
+        hierarchy = list()
         if self.shot_add_hierarchy:
             parent_template_patern = re.compile(r"\{([a-z]*?)\}")
             # fill the parents parts from presets
@@ -129,9 +134,8 @@ class CollectHierarchyInstance(pyblish.api.ContextPlugin):
         self.log.debug(f"Hierarchy: {hierarchy}")
         self.log.debug(f"parents: {parents}")
 
+        tasks_to_add = dict()
         if self.shot_add_tasks:
-            tasks_to_add = dict()
-            project_doc = legacy_io.find_one({"type": "project"})
             project_tasks = project_doc["config"]["tasks"]
             for task_name, task_data in self.shot_add_tasks.items():
                 _task_data = deepcopy(task_data)
@@ -150,9 +154,7 @@ class CollectHierarchyInstance(pyblish.api.ContextPlugin):
                             task_name,
                             list(project_tasks.keys())))
 
-            instance.data["tasks"] = tasks_to_add
-        else:
-            instance.data["tasks"] = dict()
+        instance.data["tasks"] = tasks_to_add
 
         # updating hierarchy data
         instance.data["anatomyData"].update({

openpype/hosts/standalonepublisher/plugins/publish/collect_matching_asset.py

@@ -4,7 +4,7 @@ import collections
 import pyblish.api
 from pprint import pformat
 
-from openpype.pipeline import legacy_io
+from openpype.client import get_assets
 
 
 class CollectMatchingAssetToInstance(pyblish.api.InstancePlugin):
@@ -119,8 +119,9 @@ class CollectMatchingAssetToInstance(pyblish.api.InstancePlugin):
 
     def _asset_docs_by_parent_id(self, instance):
         # Query all assets for project and store them by parent's id to list
+        project_name = instance.context.data["projectEntity"]["name"]
         asset_docs_by_parent_id = collections.defaultdict(list)
-        for asset_doc in legacy_io.find({"type": "asset"}):
+        for asset_doc in get_assets(project_name):
             parent_id = asset_doc["data"]["visualParent"]
             asset_docs_by_parent_id[parent_id].append(asset_doc)
         return asset_docs_by_parent_id

openpype/hosts/standalonepublisher/plugins/publish/validate_task_existence.py

@@ -1,9 +1,7 @@
 import pyblish.api
 
-from openpype.pipeline import (
-    PublishXmlValidationError,
-    legacy_io,
-)
+from openpype.client import get_assets
+from openpype.pipeline import PublishXmlValidationError
 
 
 class ValidateTaskExistence(pyblish.api.ContextPlugin):
@@ -20,15 +18,11 @@ class ValidateTaskExistence(pyblish.api.ContextPlugin):
         for instance in context:
             asset_names.add(instance.data["asset"])
 
-        asset_docs = legacy_io.find(
-            {
-                "type": "asset",
-                "name": {"$in": list(asset_names)}
-            },
-            {
-                "name": 1,
-                "data.tasks": 1
-            }
+        project_name = context.data["projectEntity"]["name"]
+        asset_docs = get_assets(
+            project_name,
+            asset_names=asset_names,
+            fields=["name", "data.tasks"]
         )
         tasks_by_asset_names = {}
         for asset_doc in asset_docs:

openpype/hosts/tvpaint/api/communication_server.py

@@ -707,6 +707,9 @@ class BaseCommunicator:
         if exit_code is not None:
             self.exit_code = exit_code
 
+        if self.exit_code is None:
+            self.exit_code = 0
+
     def stop(self):
         """Stop communication and currently running python process."""
         log.info("Stopping communication")

openpype/hosts/tvpaint/api/pipeline.py

@@ -8,6 +8,7 @@ import requests
 
 import pyblish.api
 
+from openpype.client import get_project, get_asset_by_name
 from openpype.hosts import tvpaint
 from openpype.api import get_current_project_settings
 from openpype.lib import register_event_callback
@@ -442,14 +443,14 @@ def set_context_settings(asset_doc=None):
 
     Change fps, resolution and frame start/end.
     """
-    if asset_doc is None:
-        # Use current session asset if not passed
-        asset_doc = legacy_io.find_one({
-            "type": "asset",
-            "name": legacy_io.Session["AVALON_ASSET"]
-        })
 
-    project_doc = legacy_io.find_one({"type": "project"})
+    project_name = legacy_io.active_project()
+    if asset_doc is None:
+        asset_name = legacy_io.Session["AVALON_ASSET"]
+        # Use current session asset if not passed
+        asset_doc = get_asset_by_name(project_name, asset_name)
+
+    project_doc = get_project(project_name)
 
     framerate = asset_doc["data"].get("fps")
     if framerate is None:

openpype/hosts/tvpaint/plugins/load/load_workfile.py

@@ -1,5 +1,6 @@
 import os
 
+from openpype.client import get_project, get_asset_by_name
 from openpype.lib import (
     StringTemplate,
     get_workfile_template_key_from_context,
@@ -44,21 +45,17 @@ class LoadWorkfile(plugin.Loader):
 
         # Save workfile.
         host_name = "tvpaint"
+        project_name = context.get("project")
         asset_name = context.get("asset")
         task_name = context.get("task")
         # Far cases when there is workfile without context
         if not asset_name:
+            project_name = legacy_io.active_project()
             asset_name = legacy_io.Session["AVALON_ASSET"]
             task_name = legacy_io.Session["AVALON_TASK"]
 
-        project_doc = legacy_io.find_one({
-            "type": "project"
-        })
-        asset_doc = legacy_io.find_one({
-            "type": "asset",
-            "name": asset_name
-        })
-        project_name = project_doc["name"]
+        project_doc = get_project(project_name)
+        asset_doc = get_asset_by_name(project_name, asset_name)
 
         template_key = get_workfile_template_key_from_context(
             asset_name,

openpype/hosts/tvpaint/plugins/publish/collect_instances.py

@@ -2,6 +2,7 @@ import json
 import copy
 import pyblish.api
 
+from openpype.client import get_asset_by_name
 from openpype.lib import get_subset_name_with_asset_doc
 from openpype.pipeline import legacy_io
 
@@ -92,17 +93,15 @@ class CollectInstances(pyblish.api.ContextPlugin):
             if family == "review":
                 # Change subset name of review instance
 
+                # Project name from workfile context
+                project_name = context.data["workfile_context"]["project"]
+
                 # Collect asset doc to get asset id
                 # - not sure if it's good idea to require asset id in
                 #   get_subset_name?
                 asset_name = context.data["workfile_context"]["asset"]
-                asset_doc = legacy_io.find_one({
-                    "type": "asset",
-                    "name": asset_name
-                })
+                asset_doc = get_asset_by_name(project_name, asset_name)
 
-                # Project name from workfile context
-                project_name = context.data["workfile_context"]["project"]
                 # Host name from environment variable
                 host_name = context.data["hostName"]
                 # Use empty variant value

openpype/hosts/tvpaint/plugins/publish/collect_scene_render.py

@@ -2,8 +2,8 @@ import json
 import copy
 import pyblish.api
 
+from openpype.client import get_asset_by_name
 from openpype.lib import get_subset_name_with_asset_doc
-from openpype.pipeline import legacy_io
 
 
 class CollectRenderScene(pyblish.api.ContextPlugin):
@@ -56,14 +56,11 @@ class CollectRenderScene(pyblish.api.ContextPlugin):
         # - not sure if it's good idea to require asset id in
         #   get_subset_name?
         workfile_context = context.data["workfile_context"]
-        asset_name = workfile_context["asset"]
-        asset_doc = legacy_io.find_one({
-            "type": "asset",
-            "name": asset_name
-        })
-
         # Project name from workfile context
         project_name = context.data["workfile_context"]["project"]
+        asset_name = workfile_context["asset"]
+        asset_doc = get_asset_by_name(project_name, asset_name)
+
         # Host name from environment variable
         host_name = context.data["hostName"]
         # Variant is using render pass name

openpype/hosts/tvpaint/plugins/publish/collect_workfile.py

@@ -2,6 +2,7 @@ import os
 import json
 import pyblish.api
 
+from openpype.client import get_asset_by_name
 from openpype.lib import get_subset_name_with_asset_doc
 from openpype.pipeline import legacy_io
 
@@ -22,19 +23,17 @@ class CollectWorkfile(pyblish.api.ContextPlugin):
         basename, ext = os.path.splitext(filename)
         instance = context.create_instance(name=basename)
 
+        # Project name from workfile context
+        project_name = context.data["workfile_context"]["project"]
+
         # Get subset name of workfile instance
         # Collect asset doc to get asset id
         # - not sure if it's good idea to require asset id in
         #   get_subset_name?
         family = "workfile"
         asset_name = context.data["workfile_context"]["asset"]
-        asset_doc = legacy_io.find_one({
-            "type": "asset",
-            "name": asset_name
-        })
+        asset_doc = get_asset_by_name(project_name, asset_name)
 
-        # Project name from workfile context
-        project_name = context.data["workfile_context"]["project"]
         # Host name from environment variable
         host_name = os.environ["AVALON_APP"]
         # Use empty variant value

openpype/hosts/webpublisher/plugins/publish/collect_published_files.py

@@ -13,9 +13,13 @@ import tempfile
 import math
 
 import pyblish.api
+
+from openpype.client import (
+    get_asset_by_name,
+    get_last_version_by_subset_name
+)
 from openpype.lib import (
     prepare_template_data,
-    get_asset,
     get_ffprobe_streams,
     convert_ffprobe_fps_value,
 )
@@ -23,7 +27,6 @@ from openpype.lib.plugin_tools import (
     parse_json,
     get_subset_name_with_asset_doc
 )
-from openpype.pipeline import legacy_io
 
 
 class CollectPublishedFiles(pyblish.api.ContextPlugin):
@@ -56,8 +59,9 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
 
         self.log.info("task_sub:: {}".format(task_subfolders))
 
+        project_name = context.data["project_name"]
         asset_name = context.data["asset"]
-        asset_doc = get_asset()
+        asset_doc = get_asset_by_name(project_name, asset_name)
         task_name = context.data["task"]
         task_type = context.data["taskType"]
         project_name = context.data["project_name"]
@@ -80,7 +84,9 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
                 family, variant, task_name, asset_doc,
                 project_name=project_name, host_name="webpublisher"
             )
-            version = self._get_last_version(asset_name, subset_name) + 1
+            version = self._get_next_version(
+                project_name, asset_doc, subset_name
+            )
 
             instance = context.create_instance(subset_name)
             instance.data["asset"] = asset_name
@@ -219,55 +225,19 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
                 config["families"],
                 config["tags"])
 
-    def _get_last_version(self, asset_name, subset_name):
-        """Returns version number or 0 for 'asset' and 'subset'"""
-        query = [
-            {
-                "$match": {"type": "asset", "name": asset_name}
-            },
-            {
-                "$lookup":
-                    {
-                        "from": os.environ["AVALON_PROJECT"],
-                        "localField": "_id",
-                        "foreignField": "parent",
-                        "as": "subsets"
-                    }
-            },
-            {
-                "$unwind": "$subsets"
-            },
-            {
-                "$match": {"subsets.type": "subset",
-                           "subsets.name": subset_name}},
-            {
-                "$lookup":
-                    {
-                        "from": os.environ["AVALON_PROJECT"],
-                        "localField": "subsets._id",
-                        "foreignField": "parent",
-                        "as": "versions"
-                    }
-            },
-            {
-                "$unwind": "$versions"
-            },
-            {
-                "$group": {
-                    "_id": {
-                        "asset_name": "$name",
-                        "subset_name": "$subsets.name"
-                    },
-                    'version': {'$max': "$versions.name"}
-                }
-            }
-        ]
-        version = list(legacy_io.aggregate(query))
+    def _get_next_version(self, project_name, asset_doc, subset_name):
+        """Returns version number or 1 for 'asset' and 'subset'"""
 
-        if version:
-            return version[0].get("version") or 0
-        else:
-            return 0
+        version_doc = get_last_version_by_subset_name(
+            project_name,
+            subset_name,
+            asset_doc["_id"],
+            fields=["name"]
+        )
+        version = 1
+        if version_doc:
+            version += int(version_doc["name"])
+        return version
 
     def _get_number_of_frames(self, file_url):
         """Return duration in frames"""

openpype/hosts/webpublisher/webserver_service/webpublish_routes.py

@@ -2,11 +2,15 @@
 import os
 import json
 import datetime
-from bson.objectid import ObjectId
 import collections
-from aiohttp.web_response import Response
 import subprocess
+from bson.objectid import ObjectId
+from aiohttp.web_response import Response
 
+from openpype.client import (
+    get_projects,
+    get_assets,
+)
 from openpype.lib import (
     OpenPypeMongoConnection,
     PypeLogger,
@@ -16,30 +20,29 @@ from openpype.lib.remote_publish import (
     ERROR_STATUS,
     REPROCESS_STATUS
 )
-from openpype.pipeline import AvalonMongoDB
-from openpype_modules.avalon_apps.rest_api import _RestApiEndpoint
 from openpype.settings import get_project_settings
+from openpype_modules.webserver.base_routes import RestApiEndpoint
+
+log = PypeLogger.get_logger("WebpublishRoutes")
 
 
-
-log = PypeLogger.get_logger("WebServer")
+class ResourceRestApiEndpoint(RestApiEndpoint):
+    def __init__(self, resource):
+        self.resource = resource
+        super(ResourceRestApiEndpoint, self).__init__()
 
 
-class RestApiResource:
-    """Resource carrying needed info and Avalon DB connection for publish."""
-    def __init__(self, server_manager, executable, upload_dir,
-                 studio_task_queue=None):
-        self.server_manager = server_manager
-        self.upload_dir = upload_dir
-        self.executable = executable
+class WebpublishApiEndpoint(ResourceRestApiEndpoint):
+    @property
+    def dbcon(self):
+        return self.resource.dbcon
 
-        if studio_task_queue is None:
-            studio_task_queue = collections.deque().dequeu
-        self.studio_task_queue = studio_task_queue
 
-        self.dbcon = AvalonMongoDB()
-        self.dbcon.install()
+class JsonApiResource:
+    """Resource for json manipulation.
 
+    All resources handling sending output to REST should inherit from
+    """
     @staticmethod
     def json_dump_handler(value):
         if isinstance(value, datetime.datetime):
@@ -59,19 +62,33 @@ class RestApiResource:
         ).encode("utf-8")
 
 
-class OpenPypeRestApiResource(RestApiResource):
+class RestApiResource(JsonApiResource):
+    """Resource carrying needed info and Avalon DB connection for publish."""
+    def __init__(self, server_manager, executable, upload_dir,
+                 studio_task_queue=None):
+        self.server_manager = server_manager
+        self.upload_dir = upload_dir
+        self.executable = executable
+
+        if studio_task_queue is None:
+            studio_task_queue = collections.deque().dequeu
+        self.studio_task_queue = studio_task_queue
+
+
+class WebpublishRestApiResource(JsonApiResource):
     """Resource carrying OP DB connection for storing batch info into DB."""
-    def __init__(self, ):
+
+    def __init__(self):
         mongo_client = OpenPypeMongoConnection.get_mongo_client()
         database_name = os.environ["OPENPYPE_DATABASE_NAME"]
         self.dbcon = mongo_client[database_name]["webpublishes"]
 
 
-class ProjectsEndpoint(_RestApiEndpoint):
+class ProjectsEndpoint(ResourceRestApiEndpoint):
     """Returns list of dict with project info (id, name)."""
     async def get(self) -> Response:
         output = []
-        for project_doc in self.dbcon.projects():
+        for project_doc in get_projects():
             ret_val = {
                 "id": project_doc["_id"],
                 "name": project_doc["name"]
@@ -84,7 +101,7 @@ class ProjectsEndpoint(_RestApiEndpoint):
         )
 
 
-class HiearchyEndpoint(_RestApiEndpoint):
+class HiearchyEndpoint(ResourceRestApiEndpoint):
     """Returns dictionary with context tree from assets."""
     async def get(self, project_name) -> Response:
         query_projection = {
@@ -96,10 +113,7 @@ class HiearchyEndpoint(_RestApiEndpoint):
             "type": 1,
         }
 
-        asset_docs = self.dbcon.database[project_name].find(
-            {"type": "asset"},
-            query_projection
-        )
+        asset_docs = get_assets(project_name, fields=query_projection.keys())
         asset_docs_by_id = {
             asset_doc["_id"]: asset_doc
             for asset_doc in asset_docs
@@ -183,7 +197,7 @@ class TaskNode(Node):
         self["attributes"] = {}
 
 
-class BatchPublishEndpoint(_RestApiEndpoint):
+class BatchPublishEndpoint(WebpublishApiEndpoint):
     """Triggers headless publishing of batch."""
     async def post(self, request) -> Response:
         # Validate existence of openpype executable
@@ -288,7 +302,7 @@ class BatchPublishEndpoint(_RestApiEndpoint):
         )
 
 
-class TaskPublishEndpoint(_RestApiEndpoint):
+class TaskPublishEndpoint(WebpublishApiEndpoint):
     """Prepared endpoint triggered after each task - for future development."""
     async def post(self, request) -> Response:
         return Response(
@@ -298,8 +312,12 @@ class TaskPublishEndpoint(_RestApiEndpoint):
         )
 
 
-class BatchStatusEndpoint(_RestApiEndpoint):
-    """Returns dict with info for batch_id."""
+class BatchStatusEndpoint(WebpublishApiEndpoint):
+    """Returns dict with info for batch_id.
+
+    Uses 'WebpublishRestApiResource'.
+    """
+
     async def get(self, batch_id) -> Response:
         output = self.dbcon.find_one({"batch_id": batch_id})
 
@@ -318,8 +336,12 @@ class BatchStatusEndpoint(_RestApiEndpoint):
         )
 
 
-class UserReportEndpoint(_RestApiEndpoint):
-    """Returns list of dict with batch info for user (email address)."""
+class UserReportEndpoint(WebpublishApiEndpoint):
+    """Returns list of dict with batch info for user (email address).
+
+    Uses 'WebpublishRestApiResource'.
+    """
+
     async def get(self, user) -> Response:
         output = list(self.dbcon.find({"user": user},
                                       projection={"log": False}))
@@ -338,7 +360,7 @@ class UserReportEndpoint(_RestApiEndpoint):
         )
 
 
-class ConfiguredExtensionsEndpoint(_RestApiEndpoint):
+class ConfiguredExtensionsEndpoint(WebpublishApiEndpoint):
     """Returns dict of extensions which have mapping to family.
 
         Returns:
@@ -378,8 +400,12 @@ class ConfiguredExtensionsEndpoint(_RestApiEndpoint):
         )
 
 
-class BatchReprocessEndpoint(_RestApiEndpoint):
-    """Marks latest 'batch_id' for reprocessing, returns 404 if not found."""
+class BatchReprocessEndpoint(WebpublishApiEndpoint):
+    """Marks latest 'batch_id' for reprocessing, returns 404 if not found.
+
+    Uses 'WebpublishRestApiResource'.
+    """
+
     async def post(self, batch_id) -> Response:
         batches = self.dbcon.find({"batch_id": batch_id,
                                    "status": ERROR_STATUS}).sort("_id", -1)

openpype/hosts/webpublisher/webserver_service/webserver_cli.py

@@ -10,7 +10,7 @@ from openpype.lib import PypeLogger
 
 from .webpublish_routes import (
     RestApiResource,
-    OpenPypeRestApiResource,
+    WebpublishRestApiResource,
     HiearchyEndpoint,
     ProjectsEndpoint,
     ConfiguredExtensionsEndpoint,
@@ -27,7 +27,7 @@ from openpype.lib.remote_publish import (
 )
 
 
-log = PypeLogger().get_logger("webserver_gui")
+log = PypeLogger.get_logger("webserver_gui")
 
 
 def run_webserver(*args, **kwargs):
@@ -69,16 +69,14 @@ def run_webserver(*args, **kwargs):
     )
 
     # triggers publish
-    webpublisher_task_publish_endpoint = \
-        BatchPublishEndpoint(resource)
+    webpublisher_task_publish_endpoint = BatchPublishEndpoint(resource)
     server_manager.add_route(
         "POST",
         "/api/webpublish/batch",
         webpublisher_task_publish_endpoint.dispatch
     )
 
-    webpublisher_batch_publish_endpoint = \
-        TaskPublishEndpoint(resource)
+    webpublisher_batch_publish_endpoint = TaskPublishEndpoint(resource)
     server_manager.add_route(
         "POST",
         "/api/webpublish/task",
@@ -86,27 +84,26 @@ def run_webserver(*args, **kwargs):
     )
 
     # reporting
-    openpype_resource = OpenPypeRestApiResource()
-    batch_status_endpoint = BatchStatusEndpoint(openpype_resource)
+    webpublish_resource = WebpublishRestApiResource()
+    batch_status_endpoint = BatchStatusEndpoint(webpublish_resource)
     server_manager.add_route(
         "GET",
         "/api/batch_status/{batch_id}",
         batch_status_endpoint.dispatch
     )
 
-    user_status_endpoint = UserReportEndpoint(openpype_resource)
+    user_status_endpoint = UserReportEndpoint(webpublish_resource)
     server_manager.add_route(
         "GET",
         "/api/publishes/{user}",
         user_status_endpoint.dispatch
     )
 
-    webpublisher_batch_reprocess_endpoint = \
-        BatchReprocessEndpoint(openpype_resource)
+    batch_reprocess_endpoint = BatchReprocessEndpoint(webpublish_resource)
     server_manager.add_route(
         "POST",
         "/api/webpublish/reprocess/{batch_id}",
-        webpublisher_batch_reprocess_endpoint.dispatch
+        batch_reprocess_endpoint.dispatch
     )
 
     server_manager.start_server()

openpype/lib/avalon_context.py

@@ -7,7 +7,6 @@ import platform
 import logging
 import collections
 import functools
-import getpass
 
 from bson.objectid import ObjectId
 
@@ -19,6 +18,7 @@ from .anatomy import Anatomy
 from .profiles_filtering import filter_profiles
 from .events import emit_event
 from .path_templates import StringTemplate
+from .local_settings import get_openpype_username
 
 legacy_io = None
 
@@ -550,7 +550,7 @@ def get_workdir_data(project_doc, asset_doc, task_name, host_name):
         "asset": asset_doc["name"],
         "parent": parent_name,
         "app": host_name,
-        "user": getpass.getuser(),
+        "user": get_openpype_username(),
         "hierarchy": hierarchy,
     }
 
@@ -797,8 +797,14 @@ def update_current_task(task=None, asset=None, app=None, template_key=None):
         else:
             os.environ[key] = value
 
+    data = changes.copy()
+    # Convert env keys to human readable keys
+    data["project_name"] = legacy_io.Session["AVALON_PROJECT"]
+    data["asset_name"] = legacy_io.Session["AVALON_ASSET"]
+    data["task_name"] = legacy_io.Session["AVALON_TASK"]
+
     # Emit session change
-    emit_event("taskChanged", changes.copy())
+    emit_event("taskChanged", data)
 
     return changes
 

openpype/lib/transcoding.py

@@ -533,7 +533,7 @@ def convert_input_paths_for_ffmpeg(
     output_dir,
     logger=None
 ):
-    """Contert source file to format supported in ffmpeg.
+    """Convert source file to format supported in ffmpeg.
 
     Currently can convert only exrs. The input filepaths should be files
     with same type. Information about input is loaded only from first found

openpype/modules/base.py

@@ -463,6 +463,25 @@ class OpenPypeModule:
 
         pass
 
+    def on_host_install(self, host, host_name, project_name):
+        """Host was installed which gives option to handle in-host logic.
+
+        It is a good option to register in-host event callbacks which are
+        specific for the module. The module is kept in memory for rest of
+        the process.
+
+        Arguments may change in future. E.g. 'host_name' should be possible
+        to receive from 'host' object.
+
+        Args:
+            host (ModuleType): Access to installed/registered host object.
+            host_name (str): Name of host.
+            project_name (str): Project name which is main part of host
+                context.
+        """
+
+        pass
+
     def cli(self, module_click_group):
         """Add commands to click group.
 

openpype/modules/deadline/plugins/publish/submit_aftereffects_deadline.py

@@ -33,6 +33,7 @@ class AfterEffectsSubmitDeadline(
     hosts = ["aftereffects"]
     families = ["render.farm"]  # cannot be "render' as that is integrated
     use_published = True
+    targets = ["local"]
 
     priority = 50
     chunk_size = 1000000

openpype/modules/deadline/plugins/publish/submit_harmony_deadline.py

@@ -238,6 +238,7 @@ class HarmonySubmitDeadline(
     order = pyblish.api.IntegratorOrder + 0.1
     hosts = ["harmony"]
     families = ["render.farm"]
+    targets = ["local"]
 
     optional = True
     use_published = False
@@ -321,7 +322,9 @@ class HarmonySubmitDeadline(
         )
         unzip_dir = (published_scene.parent / published_scene.stem)
         with _ZipFile(published_scene, "r") as zip_ref:
-            zip_ref.extractall(unzip_dir.as_posix())
+            # UNC path (//?/) added to minimalize risk with extracting
+            # to large file paths
+            zip_ref.extractall("//?/" + str(unzip_dir.as_posix()))
 
         # find any xstage files in directory, prefer the one with the same name
         # as directory (plus extension)

openpype/modules/deadline/plugins/publish/submit_maya_deadline.py

@@ -287,6 +287,7 @@ class MayaSubmitDeadline(pyblish.api.InstancePlugin):
     order = pyblish.api.IntegratorOrder + 0.1
     hosts = ["maya"]
     families = ["renderlayer"]
+    targets = ["local"]
 
     use_published = True
     tile_assembler_plugin = "OpenPypeTileAssembler"

openpype/modules/deadline/plugins/publish/submit_maya_remote_publish_deadline.py

@@ -10,7 +10,7 @@ import openpype.api
 import pyblish.api
 
 
-class MayaSubmitRemotePublishDeadline(openpype.api.Integrator):
+class MayaSubmitRemotePublishDeadline(pyblish.api.InstancePlugin):
     """Submit Maya scene to perform a local publish in Deadline.
 
     Publishing in Deadline can be helpful for scenes that publish very slow.
@@ -31,6 +31,7 @@ class MayaSubmitRemotePublishDeadline(openpype.api.Integrator):
     order = pyblish.api.IntegratorOrder
     hosts = ["maya"]
     families = ["publish.farm"]
+    targets = ["local"]
 
     def process(self, instance):
         settings = get_project_settings(os.getenv("AVALON_PROJECT"))

openpype/modules/deadline/plugins/publish/submit_nuke_deadline.py

@@ -23,6 +23,7 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
     hosts = ["nuke", "nukestudio"]
     families = ["render.farm", "prerender.farm"]
     optional = True
+    targets = ["local"]
 
     # presets
     priority = 50
@@ -54,8 +55,8 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
         self._ver = re.search(r"\d+\.\d+", context.data.get("hostVersion"))
         self._deadline_user = context.data.get(
             "deadlineUser", getpass.getuser())
-        self._frame_start = int(instance.data["frameStartHandle"])
-        self._frame_end = int(instance.data["frameEndHandle"])
+        submit_frame_start = int(instance.data["frameStartHandle"])
+        submit_frame_end = int(instance.data["frameEndHandle"])
 
         # get output path
         render_path = instance.data['path']
@@ -81,13 +82,16 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
 
         # exception for slate workflow
         if "slate" in instance.data["families"]:
-            self._frame_start -= 1
+            submit_frame_start -= 1
 
-        response = self.payload_submit(instance,
-                                       script_path,
-                                       render_path,
-                                       node.name()
-                                       )
+        response = self.payload_submit(
+            instance,
+            script_path,
+            render_path,
+            node.name(),
+            submit_frame_start,
+            submit_frame_end
+        )
         # Store output dir for unified publisher (filesequence)
         instance.data["deadlineSubmissionJob"] = response.json()
         instance.data["outputDir"] = os.path.dirname(
@@ -95,20 +99,22 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
         instance.data["publishJobState"] = "Suspended"
 
         if instance.data.get("bakingNukeScripts"):
+            # exception for slate workflow
+            if "slate" in instance.data["families"]:
+                submit_frame_start += 1
+
             for baking_script in instance.data["bakingNukeScripts"]:
                 render_path = baking_script["bakeRenderPath"]
                 script_path = baking_script["bakeScriptPath"]
                 exe_node_name = baking_script["bakeWriteNodeName"]
 
-                # exception for slate workflow
-                if "slate" in instance.data["families"]:
-                    self._frame_start += 1
-
                 resp = self.payload_submit(
                     instance,
                     script_path,
                     render_path,
                     exe_node_name,
+                    submit_frame_start,
+                    submit_frame_end,
                     response.json()
                 )
 
@@ -125,13 +131,16 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
             families.insert(0, "prerender")
         instance.data["families"] = families
 
-    def payload_submit(self,
-                       instance,
-                       script_path,
-                       render_path,
-                       exe_node_name,
-                       responce_data=None
-                       ):
+    def payload_submit(
+        self,
+        instance,
+        script_path,
+        render_path,
+        exe_node_name,
+        start_frame,
+        end_frame,
+        responce_data=None
+    ):
         render_dir = os.path.normpath(os.path.dirname(render_path))
         script_name = os.path.basename(script_path)
         jobname = "%s - %s" % (script_name, instance.name)
@@ -191,8 +200,8 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
 
                 "Plugin": "Nuke",
                 "Frames": "{start}-{end}".format(
-                    start=self._frame_start,
-                    end=self._frame_end
+                    start=start_frame,
+                    end=end_frame
                 ),
                 "Comment": self._comment,
 
@@ -292,7 +301,13 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
         self.log.info(json.dumps(payload, indent=4, sort_keys=True))
 
         # adding expectied files to instance.data
-        self.expected_files(instance, render_path)
+        self.expected_files(
+            instance,
+            render_path,
+            start_frame,
+            end_frame
+        )
+
         self.log.debug("__ expectedFiles: `{}`".format(
             instance.data["expectedFiles"]))
         response = requests.post(self.deadline_url, json=payload, timeout=10)
@@ -338,9 +353,13 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
             self.log.debug("_ path: `{}`".format(path))
         return path
 
-    def expected_files(self,
-                       instance,
-                       path):
+    def expected_files(
+        self,
+        instance,
+        path,
+        start_frame,
+        end_frame
+    ):
         """ Create expected files in instance data
         """
         if not instance.data.get("expectedFiles"):
@@ -358,7 +377,7 @@ class NukeSubmitDeadline(pyblish.api.InstancePlugin):
             instance.data["expectedFiles"].append(path)
             return
 
-        for i in range(self._frame_start, (self._frame_end + 1)):
+        for i in range(start_frame, (end_frame + 1)):
             instance.data["expectedFiles"].append(
                 os.path.join(dir, (file % i)).replace("\\", "/"))
 

openpype/modules/deadline/plugins/publish/submit_publish_job.py

@@ -103,6 +103,7 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin):
     order = pyblish.api.IntegratorOrder + 0.2
     icon = "tractor"
     deadline_plugin = "OpenPype"
+    targets = ["local"]
 
     hosts = ["fusion", "maya", "nuke", "celaction", "aftereffects", "harmony"]
 
@@ -128,7 +129,8 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin):
         "OPENPYPE_LOG_NO_COLORS",
         "OPENPYPE_USERNAME",
         "OPENPYPE_RENDER_JOB",
-        "OPENPYPE_PUBLISH_JOB"
+        "OPENPYPE_PUBLISH_JOB",
+        "OPENPYPE_MONGO"
     ]
 
     # custom deadline attributes
@@ -640,6 +642,7 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin):
 
     def _solve_families(self, instance, preview=False):
         families = instance.get("families")
+
         # if we have one representation with preview tag
         # flag whole instance for review and for ftrack
         if preview:
@@ -719,10 +722,17 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin):
                 " This may cause issues."
             ).format(source))
 
-        families = ["render"]
+        family = "render"
+        if "prerender" in instance.data["families"]:
+            family = "prerender"
+        families = [family]
+
+        # pass review to families if marked as review
+        if data.get("review"):
+            families.append("review")
 
         instance_skeleton_data = {
-            "family": "render",
+            "family": family,
             "subset": subset,
             "families": families,
             "asset": asset,
@@ -744,11 +754,6 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin):
             "useSequenceForReview": data.get("useSequenceForReview", True)
         }
 
-        if "prerender" in instance.data["families"]:
-            instance_skeleton_data.update({
-                "family": "prerender",
-                "families": []})
-
         # skip locking version if we are creating v01
         instance_version = instance.data.get("version")  # take this if exists
         if instance_version != 1:

openpype/modules/ftrack/event_handlers_server/action_prepare_project.py

@@ -1,8 +1,8 @@
 import json
 
+from openpype.client import get_project
 from openpype.api import ProjectSettings
 from openpype.lib import create_project
-from openpype.pipeline import AvalonMongoDB
 from openpype.settings import SaveWarningExc
 
 from openpype_modules.ftrack.lib import (
@@ -363,12 +363,8 @@ class PrepareProjectServer(ServerAction):
         project_name = project_entity["full_name"]
 
         # Try to find project document
-        dbcon = AvalonMongoDB()
-        dbcon.install()
-        dbcon.Session["AVALON_PROJECT"] = project_name
-        project_doc = dbcon.find_one({
-            "type": "project"
-        })
+        project_doc = get_project(project_name)
+
         # Create project if is not available
         # - creation is required to be able set project anatomy and attributes
         if not project_doc:
@@ -376,9 +372,7 @@ class PrepareProjectServer(ServerAction):
             self.log.info("Creating project \"{} [{}]\"".format(
                 project_name, project_code
             ))
-            create_project(project_name, project_code, dbcon=dbcon)
-
-        dbcon.uninstall()
+            create_project(project_name, project_code)
 
         project_settings = ProjectSettings(project_name)
         project_anatomy_settings = project_settings["project_anatomy"]

openpype/modules/ftrack/event_handlers_server/event_sync_to_avalon.py

@@ -12,6 +12,12 @@ from pymongo import UpdateOne
 import arrow
 import ftrack_api
 
+from openpype.client import (
+    get_project,
+    get_assets,
+    get_archived_assets,
+    get_asset_ids_with_subsets
+)
 from openpype.pipeline import AvalonMongoDB, schema
 
 from openpype_modules.ftrack.lib import (
@@ -149,12 +155,11 @@ class SyncToAvalonEvent(BaseEvent):
     @property
     def avalon_entities(self):
         if self._avalon_ents is None:
+            project_name = self.cur_project["full_name"]
             self.dbcon.install()
-            self.dbcon.Session["AVALON_PROJECT"] = (
-                self.cur_project["full_name"]
-            )
-            avalon_project = self.dbcon.find_one({"type": "project"})
-            avalon_entities = list(self.dbcon.find({"type": "asset"}))
+            self.dbcon.Session["AVALON_PROJECT"] = project_name
+            avalon_project = get_project(project_name)
+            avalon_entities = list(get_assets(project_name))
             self._avalon_ents = (avalon_project, avalon_entities)
         return self._avalon_ents
 
@@ -284,28 +289,21 @@ class SyncToAvalonEvent(BaseEvent):
         self._avalon_ents_by_ftrack_id[ftrack_id] = doc
 
     @property
-    def avalon_subsets_by_parents(self):
-        if self._avalon_subsets_by_parents is None:
-            self._avalon_subsets_by_parents = collections.defaultdict(list)
-            self.dbcon.install()
-            self.dbcon.Session["AVALON_PROJECT"] = (
-                self.cur_project["full_name"]
+    def avalon_asset_ids_with_subsets(self):
+        if self._avalon_asset_ids_with_subsets is None:
+            project_name = self.cur_project["full_name"]
+            self._avalon_asset_ids_with_subsets = get_asset_ids_with_subsets(
+                project_name
             )
-            for subset in self.dbcon.find({"type": "subset"}):
-                self._avalon_subsets_by_parents[subset["parent"]].append(
-                    subset
-                )
-        return self._avalon_subsets_by_parents
+
+        return self._avalon_asset_ids_with_subsets
 
     @property
     def avalon_archived_by_id(self):
         if self._avalon_archived_by_id is None:
             self._avalon_archived_by_id = {}
-            self.dbcon.install()
-            self.dbcon.Session["AVALON_PROJECT"] = (
-                self.cur_project["full_name"]
-            )
-            for asset in self.dbcon.find({"type": "archived_asset"}):
+            project_name = self.cur_project["full_name"]
+            for asset in get_archived_assets(project_name):
                 self._avalon_archived_by_id[asset["_id"]] = asset
         return self._avalon_archived_by_id
 
@@ -327,7 +325,7 @@ class SyncToAvalonEvent(BaseEvent):
             avalon_project, avalon_entities = self.avalon_entities
             self._changeability_by_mongo_id[avalon_project["_id"]] = False
             self._bubble_changeability(
-                list(self.avalon_subsets_by_parents.keys())
+                list(self.avalon_asset_ids_with_subsets)
             )
 
         return self._changeability_by_mongo_id
@@ -449,14 +447,9 @@ class SyncToAvalonEvent(BaseEvent):
             if not entity:
                 # if entity is not found then it is subset without parent
                 if entity_id in unchangeable_ids:
-                    _subset_ids = [
-                        str(sub["_id"]) for sub in
-                        self.avalon_subsets_by_parents[entity_id]
-                    ]
-                    joined_subset_ids = "| ".join(_subset_ids)
                     self.log.warning((
-                        "Parent <{}> for subsets <{}> does not exist"
-                    ).format(str(entity_id), joined_subset_ids))
+                        "Parent <{}> with subsets does not exist"
+                    ).format(str(entity_id)))
                 else:
                     self.log.warning((
                         "In avalon are entities without valid parents that"
@@ -483,7 +476,7 @@ class SyncToAvalonEvent(BaseEvent):
         self._avalon_ents_by_parent_id = None
         self._avalon_ents_by_ftrack_id = None
         self._avalon_ents_by_name = None
-        self._avalon_subsets_by_parents = None
+        self._avalon_asset_ids_with_subsets = None
         self._changeability_by_mongo_id = None
         self._avalon_archived_by_id = None
         self._avalon_archived_by_name = None

openpype/modules/ftrack/event_handlers_server/event_user_assigment.py

@@ -1,11 +1,9 @@
 import re
 import subprocess
 
+from openpype.client import get_asset_by_id, get_asset_by_name
 from openpype_modules.ftrack.lib import BaseEvent
 from openpype_modules.ftrack.lib.avalon_sync import CUST_ATTR_ID_KEY
-from openpype.pipeline import AvalonMongoDB
-
-from bson.objectid import ObjectId
 
 from openpype.api import Anatomy, get_project_settings
 
@@ -36,8 +34,6 @@ class UserAssigmentEvent(BaseEvent):
         3) path to publish files of task user was (de)assigned to
     """
 
-    db_con = AvalonMongoDB()
-
     def error(self, *err):
         for e in err:
             self.log.error(e)
@@ -101,26 +97,16 @@ class UserAssigmentEvent(BaseEvent):
         :rtype: dict
         """
         parent = task['parent']
-        self.db_con.install()
-        self.db_con.Session['AVALON_PROJECT'] = task['project']['full_name']
-
+        project_name = task["project"]["full_name"]
         avalon_entity = None
         parent_id = parent['custom_attributes'].get(CUST_ATTR_ID_KEY)
         if parent_id:
-            parent_id = ObjectId(parent_id)
-            avalon_entity = self.db_con.find_one({
-                '_id': parent_id,
-                'type': 'asset'
-            })
+            avalon_entity = get_asset_by_id(project_name, parent_id)
 
         if not avalon_entity:
-            avalon_entity = self.db_con.find_one({
-                'type': 'asset',
-                'name': parent['name']
-            })
+            avalon_entity = get_asset_by_name(project_name, parent["name"])
 
         if not avalon_entity:
-            self.db_con.uninstall()
             msg = 'Entity "{}" not found in avalon database'.format(
                 parent['name']
             )
@@ -129,7 +115,6 @@ class UserAssigmentEvent(BaseEvent):
                 'success': False,
                 'message': msg
             }
-        self.db_con.uninstall()
         return avalon_entity
 
     def _get_hierarchy(self, asset):

openpype/modules/ftrack/event_handlers_user/action_applications.py

@@ -1,5 +1,6 @@
 import os
 
+from openpype.client import get_project
 from openpype_modules.ftrack.lib import BaseAction
 from openpype.lib.applications import (
     ApplicationManager,
@@ -7,7 +8,6 @@ from openpype.lib.applications import (
     ApplictionExecutableNotFound,
     CUSTOM_LAUNCH_APP_GROUPS
 )
-from openpype.pipeline import AvalonMongoDB
 
 
 class AppplicationsAction(BaseAction):
@@ -25,7 +25,6 @@ class AppplicationsAction(BaseAction):
         super(AppplicationsAction, self).__init__(*args, **kwargs)
 
         self.application_manager = ApplicationManager()
-        self.dbcon = AvalonMongoDB()
 
     @property
     def discover_identifier(self):
@@ -110,12 +109,7 @@ class AppplicationsAction(BaseAction):
             if avalon_project_doc is None:
                 ft_project = self.get_project_from_entity(entity)
                 project_name = ft_project["full_name"]
-                if not self.dbcon.is_installed():
-                    self.dbcon.install()
-                self.dbcon.Session["AVALON_PROJECT"] = project_name
-                avalon_project_doc = self.dbcon.find_one({
-                    "type": "project"
-                }) or False
+                avalon_project_doc = get_project(project_name) or False
                 event["data"]["avalon_project_doc"] = avalon_project_doc
 
             if not avalon_project_doc:

openpype/modules/ftrack/event_handlers_user/action_create_cust_attrs.py

@@ -140,9 +140,9 @@ class CustomAttributes(BaseAction):
     identifier = 'create.update.attributes'
     #: Action label.
     label = "OpenPype Admin"
-    variant = '- Create/Update Avalon Attributes'
+    variant = '- Create/Update Custom Attributes'
     #: Action description.
-    description = 'Creates Avalon/Mongo ID for double check'
+    description = 'Creates required custom attributes in ftrack'
     icon = statics_icon("ftrack", "action_icons", "OpenPypeAdmin.svg")
     settings_key = "create_update_attributes"
 

openpype/modules/ftrack/event_handlers_user/action_delete_asset.py

@@ -4,6 +4,7 @@ from datetime import datetime
 
 from bson.objectid import ObjectId
 
+from openpype.client import get_assets, get_subsets
 from openpype.pipeline import AvalonMongoDB
 from openpype_modules.ftrack.lib import BaseAction, statics_icon
 from openpype_modules.ftrack.lib.avalon_sync import create_chunks
@@ -91,10 +92,8 @@ class DeleteAssetSubset(BaseAction):
                 continue
 
             ftrack_id = entity.get("entityId")
-            if not ftrack_id:
-                continue
-
-            ftrack_ids.append(ftrack_id)
+            if ftrack_id:
+                ftrack_ids.append(ftrack_id)
 
         if project_in_selection:
             msg = "It is not possible to use this action on project entity."
@@ -120,48 +119,51 @@ class DeleteAssetSubset(BaseAction):
                 "message": "Invalid selection for this action (Bug)"
             }
 
-        if entities[0].entity_type.lower() == "project":
-            project = entities[0]
-        else:
-            project = entities[0]["project"]
-
+        project = self.get_project_from_entity(entities[0], session)
         project_name = project["full_name"]
         self.dbcon.Session["AVALON_PROJECT"] = project_name
 
-        selected_av_entities = list(self.dbcon.find({
-            "type": "asset",
-            "data.ftrackId": {"$in": ftrack_ids}
-        }))
+        asset_docs = list(get_assets(
+            project_name,
+            fields=["_id", "name", "data.ftrackId", "data.parents"]
+        ))
+        selected_av_entities = []
+        found_ftrack_ids = set()
+        asset_docs_by_name = collections.defaultdict(list)
+        for asset_doc in asset_docs:
+            ftrack_id = asset_doc["data"].get("ftrackId")
+            if ftrack_id:
+                found_ftrack_ids.add(ftrack_id)
+                if ftrack_id in entity_mapping:
+                    selected_av_entities.append(asset_doc)
+
+            asset_name = asset_doc["name"]
+            asset_docs_by_name[asset_name].append(asset_doc)
+
         found_without_ftrack_id = {}
-        if len(selected_av_entities) != len(ftrack_ids):
-            found_ftrack_ids = [
-                ent["data"]["ftrackId"] for ent in selected_av_entities
-            ]
-            for ftrack_id, entity in entity_mapping.items():
-                if ftrack_id in found_ftrack_ids:
+        for ftrack_id, entity in entity_mapping.items():
+            if ftrack_id in found_ftrack_ids:
+                continue
+
+            av_ents_by_name = asset_docs_by_name[entity["name"]]
+            if not av_ents_by_name:
+                continue
+
+            ent_path_items = [ent["name"] for ent in entity["link"]]
+            end_index = len(ent_path_items) - 1
+            parents = ent_path_items[1:end_index:]
+            # TODO we should say to user that
+            # few of them are missing in avalon
+            for av_ent in av_ents_by_name:
+                if av_ent["data"]["parents"] != parents:
                     continue
 
-                av_ents_by_name = list(self.dbcon.find({
-                    "type": "asset",
-                    "name": entity["name"]
-                }))
-                if not av_ents_by_name:
-                    continue
-
-                ent_path_items = [ent["name"] for ent in entity["link"]]
-                parents = ent_path_items[1:len(ent_path_items)-1:]
-                # TODO we should say to user that
-                # few of them are missing in avalon
-                for av_ent in av_ents_by_name:
-                    if av_ent["data"]["parents"] != parents:
-                        continue
-
-                    # TODO we should say to user that found entity
-                    # with same name does not match same ftrack id?
-                    if "ftrackId" not in av_ent["data"]:
-                        selected_av_entities.append(av_ent)
-                        found_without_ftrack_id[str(av_ent["_id"])] = ftrack_id
-                        break
+                # TODO we should say to user that found entity
+                # with same name does not match same ftrack id?
+                if "ftrackId" not in av_ent["data"]:
+                    selected_av_entities.append(av_ent)
+                    found_without_ftrack_id[str(av_ent["_id"])] = ftrack_id
+                    break
 
         if not selected_av_entities:
             return {
@@ -206,10 +208,7 @@ class DeleteAssetSubset(BaseAction):
 
         items.append(id_item)
         asset_ids = [ent["_id"] for ent in selected_av_entities]
-        subsets_for_selection = self.dbcon.find({
-            "type": "subset",
-            "parent": {"$in": asset_ids}
-        })
+        subsets_for_selection = get_subsets(project_name, asset_ids=asset_ids)
 
         asset_ending = ""
         if len(selected_av_entities) > 1:
@@ -459,13 +458,9 @@ class DeleteAssetSubset(BaseAction):
         if len(assets_to_delete) > 0:
             map_av_ftrack_id = spec_data["without_ftrack_id"]
             # Prepare data when deleting whole avalon asset
-            avalon_assets = self.dbcon.find(
-                {"type": "asset"},
-                {
-                    "_id": 1,
-                    "data.visualParent": 1,
-                    "data.ftrackId": 1
-                }
+            avalon_assets = get_assets(
+                project_name,
+                fields=["_id", "data.visualParent", "data.ftrackId"]
             )
             avalon_assets_by_parent = collections.defaultdict(list)
             for asset in avalon_assets:

openpype/modules/ftrack/event_handlers_user/action_delete_old_versions.py

@@ -5,7 +5,12 @@ import uuid
 import clique
 from pymongo import UpdateOne
 
-
+from openpype.client import (
+    get_assets,
+    get_subsets,
+    get_versions,
+    get_representations
+)
 from openpype.api import Anatomy
 from openpype.lib import StringTemplate, TemplateUnsolved
 from openpype.pipeline import AvalonMongoDB
@@ -198,10 +203,9 @@ class DeleteOldVersions(BaseAction):
         self.log.debug("Project is set to {}".format(project_name))
 
         # Get Assets from avalon database
-        assets = list(self.dbcon.find({
-            "type": "asset",
-            "name": {"$in": avalon_asset_names}
-        }))
+        assets = list(
+            get_assets(project_name, asset_names=avalon_asset_names)
+        )
         asset_id_to_name_map = {
             asset["_id"]: asset["name"] for asset in assets
         }
@@ -210,10 +214,9 @@ class DeleteOldVersions(BaseAction):
         self.log.debug("Collected assets ({})".format(len(asset_ids)))
 
         # Get Subsets
-        subsets = list(self.dbcon.find({
-            "type": "subset",
-            "parent": {"$in": asset_ids}
-        }))
+        subsets = list(
+            get_subsets(project_name, asset_ids=asset_ids)
+        )
         subsets_by_id = {}
         subset_ids = []
         for subset in subsets:
@@ -230,10 +233,9 @@ class DeleteOldVersions(BaseAction):
         self.log.debug("Collected subsets ({})".format(len(subset_ids)))
 
         # Get Versions
-        versions = list(self.dbcon.find({
-            "type": "version",
-            "parent": {"$in": subset_ids}
-        }))
+        versions = list(
+            get_versions(project_name, subset_ids=subset_ids)
+        )
 
         versions_by_parent = collections.defaultdict(list)
         for ent in versions:
@@ -295,10 +297,9 @@ class DeleteOldVersions(BaseAction):
                 "message": msg
             }
 
-        repres = list(self.dbcon.find({
-            "type": "representation",
-            "parent": {"$in": version_ids}
-        }))
+        repres = list(
+            get_representations(project_name, version_ids=version_ids)
+        )
 
         self.log.debug(
             "Collected representations to remove ({})".format(len(repres))

openpype/modules/ftrack/event_handlers_user/action_delivery.py

@@ -3,8 +3,13 @@ import copy
 import json
 import collections
 
-from bson.objectid import ObjectId
-
+from openpype.client import (
+    get_project,
+    get_assets,
+    get_subsets,
+    get_versions,
+    get_representations
+)
 from openpype.api import Anatomy, config
 from openpype_modules.ftrack.lib import BaseAction, statics_icon
 from openpype_modules.ftrack.lib.avalon_sync import CUST_ATTR_ID_KEY
@@ -18,11 +23,9 @@ from openpype.lib.delivery import (
     process_single_file,
     process_sequence
 )
-from openpype.pipeline import AvalonMongoDB
 
 
 class Delivery(BaseAction):
-
     identifier = "delivery.action"
     label = "Delivery"
     description = "Deliver data to client"
@@ -30,11 +33,6 @@ class Delivery(BaseAction):
     icon = statics_icon("ftrack", "action_icons", "Delivery.svg")
     settings_key = "delivery_action"
 
-    def __init__(self, *args, **kwargs):
-        self.dbcon = AvalonMongoDB()
-
-        super(Delivery, self).__init__(*args, **kwargs)
-
     def discover(self, session, entities, event):
         is_valid = False
         for entity in entities:
@@ -57,9 +55,7 @@ class Delivery(BaseAction):
 
         project_entity = self.get_project_from_entity(entities[0])
         project_name = project_entity["full_name"]
-        self.dbcon.install()
-        self.dbcon.Session["AVALON_PROJECT"] = project_name
-        project_doc = self.dbcon.find_one({"type": "project"}, {"name": True})
+        project_doc = get_project(project_name, fields=["name"])
         if not project_doc:
             return {
                 "success": False,
@@ -68,8 +64,7 @@ class Delivery(BaseAction):
                 ).format(project_name)
             }
 
-        repre_names = self._get_repre_names(session, entities)
-        self.dbcon.uninstall()
+        repre_names = self._get_repre_names(project_name, session, entities)
 
         items.append({
             "type": "hidden",
@@ -198,17 +193,21 @@ class Delivery(BaseAction):
             "title": title
         }
 
-    def _get_repre_names(self, session, entities):
-        version_ids = self._get_interest_version_ids(session, entities)
+    def _get_repre_names(self, project_name, session, entities):
+        version_ids = self._get_interest_version_ids(
+            project_name, session, entities
+        )
         if not version_ids:
             return []
-        repre_docs = self.dbcon.find({
-            "type": "representation",
-            "parent": {"$in": version_ids}
-        })
-        return list(sorted(repre_docs.distinct("name")))
+        repre_docs = get_representations(
+            project_name,
+            version_ids=version_ids,
+            fields=["name"]
+        )
+        repre_names = {repre_doc["name"] for repre_doc in repre_docs}
+        return list(sorted(repre_names))
 
-    def _get_interest_version_ids(self, session, entities):
+    def _get_interest_version_ids(self, project_name, session, entities):
         # Extract AssetVersion entities
         asset_versions = self._extract_asset_versions(session, entities)
         # Prepare Asset ids
@@ -235,14 +234,18 @@ class Delivery(BaseAction):
             subset_names.add(asset["name"])
             version_nums.add(asset_version["version"])
 
-        asset_docs_by_ftrack_id = self._get_asset_docs(session, parent_ids)
+        asset_docs_by_ftrack_id = self._get_asset_docs(
+            project_name, session, parent_ids
+        )
         subset_docs = self._get_subset_docs(
+            project_name,
             asset_docs_by_ftrack_id,
             subset_names,
             asset_versions,
             assets_by_id
         )
         version_docs = self._get_version_docs(
+            project_name,
             asset_docs_by_ftrack_id,
             subset_docs,
             version_nums,
@@ -290,6 +293,7 @@ class Delivery(BaseAction):
 
     def _get_version_docs(
         self,
+        project_name,
         asset_docs_by_ftrack_id,
         subset_docs,
         version_nums,
@@ -300,11 +304,11 @@ class Delivery(BaseAction):
             subset_doc["_id"]: subset_doc
             for subset_doc in subset_docs
         }
-        version_docs = list(self.dbcon.find({
-            "type": "version",
-            "parent": {"$in": list(subset_docs_by_id.keys())},
-            "name": {"$in": list(version_nums)}
-        }))
+        version_docs = list(get_versions(
+            project_name,
+            subset_ids=subset_docs_by_id.keys(),
+            versions=version_nums
+        ))
         version_docs_by_parent_id = collections.defaultdict(dict)
         for version_doc in version_docs:
             subset_doc = subset_docs_by_id[version_doc["parent"]]
@@ -345,6 +349,7 @@ class Delivery(BaseAction):
 
     def _get_subset_docs(
         self,
+        project_name,
         asset_docs_by_ftrack_id,
         subset_names,
         asset_versions,
@@ -354,11 +359,11 @@ class Delivery(BaseAction):
             asset_doc["_id"]
             for asset_doc in asset_docs_by_ftrack_id.values()
         ]
-        subset_docs = list(self.dbcon.find({
-            "type": "subset",
-            "parent": {"$in": asset_doc_ids},
-            "name": {"$in": list(subset_names)}
-        }))
+        subset_docs = list(get_subsets(
+            project_name,
+            asset_ids=asset_doc_ids,
+            subset_names=subset_names
+        ))
         subset_docs_by_parent_id = collections.defaultdict(dict)
         for subset_doc in subset_docs:
             asset_id = subset_doc["parent"]
@@ -385,15 +390,21 @@ class Delivery(BaseAction):
                 filtered_subsets.append(subset_doc)
         return filtered_subsets
 
-    def _get_asset_docs(self, session, parent_ids):
-        asset_docs = list(self.dbcon.find({
-            "type": "asset",
-            "data.ftrackId": {"$in": list(parent_ids)}
-        }))
+    def _get_asset_docs(self, project_name, session, parent_ids):
+        asset_docs = list(get_assets(
+            project_name, fields=["_id", "name", "data.ftrackId"]
+        ))
 
+        asset_docs_by_id = {}
+        asset_docs_by_name = {}
         asset_docs_by_ftrack_id = {}
         for asset_doc in asset_docs:
+            asset_id = str(asset_doc["_id"])
+            asset_name = asset_doc["name"]
             ftrack_id = asset_doc["data"].get("ftrackId")
+
+            asset_docs_by_id[asset_id] = asset_doc
+            asset_docs_by_name[asset_name] = asset_doc
             if ftrack_id:
                 asset_docs_by_ftrack_id[ftrack_id] = asset_doc
 
@@ -406,15 +417,15 @@ class Delivery(BaseAction):
         avalon_mongo_id_values = query_custom_attributes(
             session, [attr_def["id"]], parent_ids, True
         )
-        entity_ids_by_mongo_id = {
-            ObjectId(item["value"]): item["entity_id"]
-            for item in avalon_mongo_id_values
-            if item["value"]
-        }
-
         missing_ids = set(parent_ids)
-        for entity_id in set(entity_ids_by_mongo_id.values()):
-            if entity_id in missing_ids:
+        for item in avalon_mongo_id_values:
+            if not item["value"]:
+                continue
+            asset_id = item["value"]
+            entity_id = item["entity_id"]
+            asset_doc = asset_docs_by_id.get(asset_id)
+            if asset_doc:
+                asset_docs_by_ftrack_id[entity_id] = asset_doc
                 missing_ids.remove(entity_id)
 
         entity_ids_by_name = {}
@@ -427,36 +438,10 @@ class Delivery(BaseAction):
                 for entity in not_found_entities
             }
 
-        expressions = []
-        if entity_ids_by_mongo_id:
-            expression = {
-                "type": "asset",
-                "_id": {"$in": list(entity_ids_by_mongo_id.keys())}
-            }
-            expressions.append(expression)
-
-        if entity_ids_by_name:
-            expression = {
-                "type": "asset",
-                "name": {"$in": list(entity_ids_by_name.keys())}
-            }
-            expressions.append(expression)
-
-        if expressions:
-            if len(expressions) == 1:
-                filter = expressions[0]
-            else:
-                filter = {"$or": expressions}
-
-            asset_docs = self.dbcon.find(filter)
-            for asset_doc in asset_docs:
-                if asset_doc["_id"] in entity_ids_by_mongo_id:
-                    entity_id = entity_ids_by_mongo_id[asset_doc["_id"]]
-                    asset_docs_by_ftrack_id[entity_id] = asset_doc
-
-                elif asset_doc["name"] in entity_ids_by_name:
-                    entity_id = entity_ids_by_name[asset_doc["name"]]
-                    asset_docs_by_ftrack_id[entity_id] = asset_doc
+        for asset_name, entity_id in entity_ids_by_name.items():
+            asset_doc = asset_docs_by_name.get(asset_name)
+            if asset_doc:
+                asset_docs_by_ftrack_id[entity_id] = asset_doc
 
         return asset_docs_by_ftrack_id
 
@@ -490,7 +475,6 @@ class Delivery(BaseAction):
         session.commit()
 
         try:
-            self.dbcon.install()
             report = self.real_launch(session, entities, event)
 
         except Exception as exc:
@@ -516,7 +500,6 @@ class Delivery(BaseAction):
             else:
                 job["status"] = "failed"
             session.commit()
-            self.dbcon.uninstall()
 
         if not report["success"]:
             self.show_interface(
@@ -558,16 +541,15 @@ class Delivery(BaseAction):
             if not os.path.exists(location_path):
                 os.makedirs(location_path)
 
-        self.dbcon.Session["AVALON_PROJECT"] = project_name
-
         self.log.debug("Collecting representations to process.")
-        version_ids = self._get_interest_version_ids(session, entities)
-        repres_to_deliver = list(self.dbcon.find({
-            "type": "representation",
-            "parent": {"$in": version_ids},
-            "name": {"$in": repre_names}
-        }))
-
+        version_ids = self._get_interest_version_ids(
+            project_name, session, entities
+        )
+        repres_to_deliver = list(get_representations(
+            project_name,
+            representation_names=repre_names,
+            version_ids=version_ids
+        ))
         anatomy = Anatomy(project_name)
 
         format_dict = get_format_dict(anatomy, location_path)

openpype/modules/ftrack/event_handlers_user/action_fill_workfile_attr.py

@@ -7,6 +7,10 @@ import datetime
 
 import ftrack_api
 
+from openpype.client import (
+    get_project,
+    get_assets,
+)
 from openpype.api import get_project_settings
 from openpype.lib import (
     get_workfile_template_key,
@@ -14,7 +18,6 @@ from openpype.lib import (
     Anatomy,
     StringTemplate,
 )
-from openpype.pipeline import AvalonMongoDB
 from openpype_modules.ftrack.lib import BaseAction, statics_icon
 from openpype_modules.ftrack.lib.avalon_sync import create_chunks
 
@@ -248,10 +251,8 @@ class FillWorkfileAttributeAction(BaseAction):
         # Find matchin asset documents and map them by ftrack task entities
         # - result stored to 'asset_docs_with_task_entities' is list with
         #   tuple `(asset document, [task entitis, ...])`
-        dbcon = AvalonMongoDB()
-        dbcon.Session["AVALON_PROJECT"] = project_name
         # Quety all asset documents
-        asset_docs = list(dbcon.find({"type": "asset"}))
+        asset_docs = list(get_assets(project_name))
         job_entity["data"] = json.dumps({
             "description": "(1/3) Asset documents queried."
         })
@@ -276,7 +277,7 @@ class FillWorkfileAttributeAction(BaseAction):
         # Keep placeholders in the template unfilled
         host_name = "{app}"
         extension = "{ext}"
-        project_doc = dbcon.find_one({"type": "project"})
+        project_doc = get_project(project_name)
         project_settings = get_project_settings(project_name)
         anatomy = Anatomy(project_name)
         templates_by_key = {}

openpype/modules/ftrack/event_handlers_user/action_prepare_project.py

@@ -1,8 +1,8 @@
 import json
 
+from openpype.client import get_project
 from openpype.api import ProjectSettings
 from openpype.lib import create_project
-from openpype.pipeline import AvalonMongoDB
 from openpype.settings import SaveWarningExc
 
 from openpype_modules.ftrack.lib import (
@@ -389,12 +389,8 @@ class PrepareProjectLocal(BaseAction):
         project_name = project_entity["full_name"]
 
         # Try to find project document
-        dbcon = AvalonMongoDB()
-        dbcon.install()
-        dbcon.Session["AVALON_PROJECT"] = project_name
-        project_doc = dbcon.find_one({
-            "type": "project"
-        })
+        project_doc = get_project(project_name)
+
         # Create project if is not available
         # - creation is required to be able set project anatomy and attributes
         if not project_doc:
@@ -402,9 +398,7 @@ class PrepareProjectLocal(BaseAction):
             self.log.info("Creating project \"{} [{}]\"".format(
                 project_name, project_code
             ))
-            create_project(project_name, project_code, dbcon=dbcon)
-
-        dbcon.uninstall()
+            create_project(project_name, project_code)
 
         project_settings = ProjectSettings(project_name)
         project_anatomy_settings = project_settings["project_anatomy"]

openpype/modules/ftrack/event_handlers_user/action_rv.py

@@ -5,9 +5,16 @@ import json
 
 import ftrack_api
 
+from openpype.client import (
+    get_asset_by_name,
+    get_subset_by_name,
+    get_version_by_name,
+    get_representation_by_name
+)
+from openpype.api import Anatomy
 from openpype.pipeline import (
     get_representation_path,
-    legacy_io,
+    AvalonMongoDB,
 )
 from openpype_modules.ftrack.lib import BaseAction, statics_icon
 
@@ -255,9 +262,10 @@ class RVAction(BaseAction):
             "Component", list(event["data"]["values"].values())[0]
         )["version"]["asset"]["parent"]["link"][0]
         project = session.get(link["type"], link["id"])
-        os.environ["AVALON_PROJECT"] = project["name"]
-        legacy_io.Session["AVALON_PROJECT"] = project["name"]
-        legacy_io.install()
+        project_name = project["full_name"]
+        dbcon = AvalonMongoDB()
+        dbcon.Session["AVALON_PROJECT"] = project_name
+        anatomy = Anatomy(project_name)
 
         location = ftrack_api.Session().pick_location()
 
@@ -281,37 +289,38 @@ class RVAction(BaseAction):
             if online_source:
                 continue
 
-            asset = legacy_io.find_one({"type": "asset", "name": parent_name})
-            subset = legacy_io.find_one(
-                {
-                    "type": "subset",
-                    "name": component["version"]["asset"]["name"],
-                    "parent": asset["_id"]
-                }
+            subset_name = component["version"]["asset"]["name"]
+            version_name = component["version"]["version"]
+            representation_name = component["file_type"][1:]
+
+            asset_doc = get_asset_by_name(
+                project_name, parent_name, fields=["_id"]
             )
-            version = legacy_io.find_one(
-                {
-                    "type": "version",
-                    "name": component["version"]["version"],
-                    "parent": subset["_id"]
-                }
+            subset_doc = get_subset_by_name(
+                project_name,
+                subset_name=subset_name,
+                asset_id=asset_doc["_id"]
             )
-            representation = legacy_io.find_one(
-                {
-                    "type": "representation",
-                    "parent": version["_id"],
-                    "name": component["file_type"][1:]
-                }
+            version_doc = get_version_by_name(
+                project_name,
+                version=version_name,
+                subset_id=subset_doc["_id"]
             )
-            if representation is None:
-                representation = legacy_io.find_one(
-                    {
-                        "type": "representation",
-                        "parent": version["_id"],
-                        "name": "preview"
-                    }
+            repre_doc = get_representation_by_name(
+                project_name,
+                version_id=version_doc["_id"],
+                representation_name=representation_name
+            )
+            if not repre_doc:
+                repre_doc = get_representation_by_name(
+                    project_name,
+                    version_id=version_doc["_id"],
+                    representation_name="preview"
                 )
-            paths.append(get_representation_path(representation))
+
+            paths.append(get_representation_path(
+                repre_doc, root=anatomy.roots, dbcon=dbcon
+            ))
 
         return paths
 

openpype/modules/ftrack/event_handlers_user/action_store_thumbnails_to_avalon.py

@@ -5,6 +5,14 @@ import requests
 
 from bson.objectid import ObjectId
 
+from openpype.client import (
+    get_project,
+    get_asset_by_id,
+    get_assets,
+    get_subset_by_name,
+    get_version_by_name,
+    get_representations
+)
 from openpype_modules.ftrack.lib import BaseAction, statics_icon
 from openpype.api import Anatomy
 from openpype.pipeline import AvalonMongoDB
@@ -385,7 +393,7 @@ class StoreThumbnailsToAvalon(BaseAction):
 
         db_con.Session["AVALON_PROJECT"] = project_name
 
-        avalon_project = db_con.find_one({"type": "project"})
+        avalon_project = get_project(project_name)
         output["project"] = avalon_project
 
         if not avalon_project:
@@ -399,19 +407,17 @@ class StoreThumbnailsToAvalon(BaseAction):
         asset_mongo_id = parent["custom_attributes"].get(CUST_ATTR_ID_KEY)
         if asset_mongo_id:
             try:
-                asset_mongo_id = ObjectId(asset_mongo_id)
-                asset_ent = db_con.find_one({
-                    "type": "asset",
-                    "_id": asset_mongo_id
-                })
+                asset_ent = get_asset_by_id(project_name, asset_mongo_id)
             except Exception:
                 pass
 
         if not asset_ent:
-            asset_ent = db_con.find_one({
-                "type": "asset",
-                "data.ftrackId": parent["id"]
-            })
+            asset_docs = get_assets(project_name, asset_names=[parent["name"]])
+            for asset_doc in asset_docs:
+                ftrack_id = asset_doc.get("data", {}).get("ftrackId")
+                if ftrack_id == parent["id"]:
+                    asset_ent = asset_doc
+                    break
 
         output["asset"] = asset_ent
 
@@ -422,13 +428,11 @@ class StoreThumbnailsToAvalon(BaseAction):
             )
             return output
 
-        asset_mongo_id = asset_ent["_id"]
-
-        subset_ent = db_con.find_one({
-            "type": "subset",
-            "parent": asset_mongo_id,
-            "name": subset_name
-        })
+        subset_ent = get_subset_by_name(
+            project_name,
+            subset_name=subset_name,
+            asset_id=asset_ent["_id"]
+        )
 
         output["subset"] = subset_ent
 
@@ -439,11 +443,11 @@ class StoreThumbnailsToAvalon(BaseAction):
             ).format(subset_name, ent_path)
             return output
 
-        version_ent = db_con.find_one({
-            "type": "version",
-            "name": version,
-            "parent": subset_ent["_id"]
-        })
+        version_ent = get_version_by_name(
+            project_name,
+            version,
+            subset_ent["_id"]
+        )
 
         output["version"] = version_ent
 
@@ -454,10 +458,10 @@ class StoreThumbnailsToAvalon(BaseAction):
             ).format(version, subset_name, ent_path)
             return output
 
-        repre_ents = list(db_con.find({
-            "type": "representation",
-            "parent": version_ent["_id"]
-        }))
+        repre_ents = list(get_representations(
+            project_name,
+            version_ids=[version_ent["_id"]]
+        ))
 
         output["representations"] = repre_ents
         return output

openpype/modules/ftrack/lib/avalon_sync.py

@@ -6,6 +6,14 @@ import numbers
 
 import six
 
+from openpype.client import (
+    get_project,
+    get_assets,
+    get_archived_assets,
+    get_subsets,
+    get_versions,
+    get_representations
+)
 from openpype.api import (
     Logger,
     get_anatomy_settings
@@ -576,6 +584,10 @@ class SyncEntitiesFactory:
         self.ft_project_id = ft_project_id
         self.entities_dict = entities_dict
 
+    @property
+    def project_name(self):
+        return self.entities_dict[self.ft_project_id]["name"]
+
     @property
     def avalon_ents_by_id(self):
         """
@@ -660,9 +672,9 @@ class SyncEntitiesFactory:
             (list) of assets
         """
         if self._avalon_archived_ents is None:
-            self._avalon_archived_ents = [
-                ent for ent in self.dbcon.find({"type": "archived_asset"})
-            ]
+            self._avalon_archived_ents = list(
+                get_archived_assets(self.project_name)
+            )
         return self._avalon_archived_ents
 
     @property
@@ -730,7 +742,7 @@ class SyncEntitiesFactory:
         """
         if self._subsets_by_parent_id is None:
             self._subsets_by_parent_id = collections.defaultdict(list)
-            for subset in self.dbcon.find({"type": "subset"}):
+            for subset in get_subsets(self.project_name):
                 self._subsets_by_parent_id[str(subset["parent"])].append(
                     subset
                 )
@@ -1421,8 +1433,8 @@ class SyncEntitiesFactory:
         # Avalon entities
         self.dbcon.install()
         self.dbcon.Session["AVALON_PROJECT"] = ft_project_name
-        avalon_project = self.dbcon.find_one({"type": "project"})
-        avalon_entities = self.dbcon.find({"type": "asset"})
+        avalon_project = get_project(ft_project_name)
+        avalon_entities = get_assets(ft_project_name)
         self.avalon_project = avalon_project
         self.avalon_entities = avalon_entities
 
@@ -2258,46 +2270,37 @@ class SyncEntitiesFactory:
         self._delete_subsets_without_asset(subsets_to_remove)
 
     def _delete_subsets_without_asset(self, not_existing_parents):
-        subset_ids = []
-        version_ids = []
         repre_ids = []
         to_delete = []
 
+        subset_ids = []
         for parent_id in not_existing_parents:
             subsets = self.subsets_by_parent_id.get(parent_id)
             if not subsets:
                 continue
             for subset in subsets:
-                if subset.get("type") != "subset":
-                    continue
-                subset_ids.append(subset["_id"])
+                if subset.get("type") == "subset":
+                    subset_ids.append(subset["_id"])
 
-        db_subsets = self.dbcon.find({
-            "_id": {"$in": subset_ids},
-            "type": "subset"
-        })
-        if not db_subsets:
-            return
-
-        db_versions = self.dbcon.find({
-            "parent": {"$in": subset_ids},
-            "type": "version"
-        })
-        if db_versions:
-            version_ids = [ver["_id"] for ver in db_versions]
-
-        db_repres = self.dbcon.find({
-            "parent": {"$in": version_ids},
-            "type": "representation"
-        })
-        if db_repres:
-            repre_ids = [repre["_id"] for repre in db_repres]
+        db_versions = get_versions(
+            self.project_name,
+            subset_ids=subset_ids,
+            fields=["_id"]
+        )
+        version_ids = [ver["_id"] for ver in db_versions]
+        db_repres = get_representations(
+            self.project_name,
+            version_ids=version_ids,
+            fields=["_id"]
+        )
+        repre_ids = [repre["_id"] for repre in db_repres]
 
         to_delete.extend(subset_ids)
         to_delete.extend(version_ids)
         to_delete.extend(repre_ids)
 
-        self.dbcon.delete_many({"_id": {"$in": to_delete}})
+        if to_delete:
+            self.dbcon.delete_many({"_id": {"$in": to_delete}})
 
     # Probably deprecated
     def _check_changeability(self, parent_id=None):
@@ -2779,8 +2782,7 @@ class SyncEntitiesFactory:
 
     def report(self):
         items = []
-        project_name = self.entities_dict[self.ft_project_id]["name"]
-        title = "Synchronization report ({}):".format(project_name)
+        title = "Synchronization report ({}):".format(self.project_name)
 
         keys = ["error", "warning", "info"]
         for key in keys:

openpype/modules/ftrack/plugins/publish/integrate_hierarchy_ftrack.py

@@ -3,7 +3,8 @@ import collections
 import six
 import pyblish.api
 from copy import deepcopy
-from openpype.pipeline import legacy_io
+from openpype.client import get_asset_by_id
+
 
 # Copy of constant `openpype_modules.ftrack.lib.avalon_sync.CUST_ATTR_AUTO_SYNC`
 CUST_ATTR_AUTO_SYNC = "avalon_auto_sync"
@@ -82,9 +83,6 @@ class IntegrateHierarchyToFtrack(pyblish.api.ContextPlugin):
         auto_sync_state = project[
             "custom_attributes"][CUST_ATTR_AUTO_SYNC]
 
-        if not legacy_io.Session:
-            legacy_io.install()
-
         self.ft_project = None
 
         # disable termporarily ftrack project's autosyncing
@@ -93,14 +91,14 @@ class IntegrateHierarchyToFtrack(pyblish.api.ContextPlugin):
 
         try:
             # import ftrack hierarchy
-            self.import_to_ftrack(hierarchy_context)
+            self.import_to_ftrack(project_name, hierarchy_context)
         except Exception:
             raise
         finally:
             if auto_sync_state:
                 self.auto_sync_on(project)
 
-    def import_to_ftrack(self, input_data, parent=None):
+    def import_to_ftrack(self, project_name, input_data, parent=None):
         # Prequery hiearchical custom attributes
         hier_custom_attributes = get_pype_attr(self.session)[1]
         hier_attr_by_key = {
@@ -222,7 +220,7 @@ class IntegrateHierarchyToFtrack(pyblish.api.ContextPlugin):
                     six.reraise(tp, value, tb)
 
             # Incoming links.
-            self.create_links(entity_data, entity)
+            self.create_links(project_name, entity_data, entity)
             try:
                 self.session.commit()
             except Exception:
@@ -255,9 +253,9 @@ class IntegrateHierarchyToFtrack(pyblish.api.ContextPlugin):
             # Import children.
             if 'childs' in entity_data:
                 self.import_to_ftrack(
-                    entity_data['childs'], entity)
+                    project_name, entity_data['childs'], entity)
 
-    def create_links(self, entity_data, entity):
+    def create_links(self, project_name, entity_data, entity):
         # Clear existing links.
         for link in entity.get("incoming_links", []):
             self.session.delete(link)
@@ -270,9 +268,15 @@ class IntegrateHierarchyToFtrack(pyblish.api.ContextPlugin):
                 six.reraise(tp, value, tb)
 
         # Create new links.
-        for input in entity_data.get("inputs", []):
-            input_id = legacy_io.find_one({"_id": input})["data"]["ftrackId"]
-            assetbuild = self.session.get("AssetBuild", input_id)
+        for asset_id in entity_data.get("inputs", []):
+            asset_doc = get_asset_by_id(project_name, asset_id)
+            ftrack_id = None
+            if asset_doc:
+                ftrack_id = asset_doc["data"].get("ftrackId")
+            if not ftrack_id:
+                continue
+
+            assetbuild = self.session.get("AssetBuild", ftrack_id)
             self.log.debug(
                 "Creating link from {0} to {1}".format(
                     assetbuild["name"], entity["name"]

openpype/modules/ftrack/tray/ftrack_tray.py

@@ -2,6 +2,7 @@ import os
 import time
 import datetime
 import threading
+
 from Qt import QtCore, QtWidgets, QtGui
 
 import ftrack_api
@@ -48,6 +49,9 @@ class FtrackTrayWrapper:
         self.widget_login.activateWindow()
         self.widget_login.raise_()
 
+    def show_ftrack_browser(self):
+        QtGui.QDesktopServices.openUrl(self.module.ftrack_url)
+
     def validate(self):
         validation = False
         cred = credentials.get_credentials()
@@ -284,6 +288,13 @@ class FtrackTrayWrapper:
         tray_server_menu.addAction(self.action_server_stop)
 
         self.tray_server_menu = tray_server_menu
+
+        # Ftrack Browser
+        browser_open = QtWidgets.QAction("Open Ftrack...", tray_menu)
+        browser_open.triggered.connect(self.show_ftrack_browser)
+        tray_menu.addAction(browser_open)
+        self.browser_open = browser_open
+
         self.bool_logged = False
         self.set_menu_visibility()
 

openpype/modules/kitsu/utils/update_op_with_zou.py

@@ -85,7 +85,7 @@ def update_op_assets(
         # Frame in, fallback on 0
         frame_in = int(item_data.get("frame_in") or 0)
         item_data["frameStart"] = frame_in
-        item_data.pop("frame_in")
+        item_data.pop("frame_in", None)
         # Frame out, fallback on frame_in + duration
         frames_duration = int(item.get("nb_frames") or 1)
         frame_out = (
@@ -94,7 +94,7 @@ def update_op_assets(
             else frame_in + frames_duration
         )
         item_data["frameEnd"] = int(frame_out)
-        item_data.pop("frame_out")
+        item_data.pop("frame_out", None)
         # Fps, fallback to project's value when entity fps is deleted
         if not item_data.get("fps") and item_doc["data"].get("fps"):
             item_data["fps"] = project_doc["data"]["fps"]

openpype/modules/sync_server/sync_server.py

@@ -280,14 +280,13 @@ class SyncServerThread(threading.Thread):
         while self.is_running and not self.module.is_paused():
             try:
                 import time
-                start_time = None
+                start_time = time.time()
                 self.module.set_sync_project_settings()  # clean cache
-                for collection, preset in self.module.sync_project_settings.\
-                        items():
-                    if collection not in self.module.get_enabled_projects():
-                        continue
+                collection = None
+                enabled_projects = self.module.get_enabled_projects()
+                for collection in enabled_projects:
+                    preset = self.module.sync_project_settings[collection]
 
-                    start_time = time.time()
                     local_site, remote_site = self._working_sites(collection)
                     if not all([local_site, remote_site]):
                         continue

openpype/modules/sync_server/sync_server_module.py

@@ -926,9 +926,22 @@ class SyncServerModule(OpenPypeModule, ITrayModule):
 
         return enabled_projects
 
-    def is_project_enabled(self, project_name):
+    def is_project_enabled(self, project_name, single=False):
+        """Checks if 'project_name' is enabled for syncing.
+        'get_sync_project_setting' is potentially expensive operation (pulls
+        settings for all projects if cached version is not available), using
+        project_settings for specific project should be faster.
+        Args:
+            project_name (str)
+            single (bool): use 'get_project_settings' method
+        """
         if self.enabled:
-            project_settings = self.get_sync_project_setting(project_name)
+            if single:
+                project_settings = get_project_settings(project_name)
+                project_settings = \
+                    self._parse_sync_settings_from_settings(project_settings)
+            else:
+                project_settings = self.get_sync_project_setting(project_name)
             if project_settings and project_settings.get("enabled"):
                 return True
         return False
@@ -1026,21 +1039,13 @@ class SyncServerModule(OpenPypeModule, ITrayModule):
         """
         self.server_init()
 
-        from .tray.app import SyncServerWindow
-        self.widget = SyncServerWindow(self)
-
     def server_init(self):
         """Actual initialization of Sync Server."""
         # import only in tray or Python3, because of Python2 hosts
-        from .sync_server import SyncServerThread
-
         if not self.enabled:
             return
 
-        enabled_projects = self.get_enabled_projects()
-        if not enabled_projects:
-            self.enabled = False
-            return
+        from .sync_server import SyncServerThread
 
         self.lock = threading.Lock()
 
@@ -1060,7 +1065,7 @@ class SyncServerModule(OpenPypeModule, ITrayModule):
         self.server_start()
 
     def server_start(self):
-        if self.sync_project_settings and self.enabled:
+        if self.enabled:
             self.sync_server_thread.start()
         else:
             log.info("No presets or active providers. " +
@@ -1851,6 +1856,9 @@ class SyncServerModule(OpenPypeModule, ITrayModule):
         Returns:
             (int): in seconds
         """
+        if not project_name:
+            return 60
+
         ld = self.sync_project_settings[project_name]["config"]["loop_delay"]
         return int(ld)
 

openpype/modules/sync_server/tray/app.py

@@ -46,6 +46,14 @@ class SyncServerWindow(QtWidgets.QDialog):
 
         left_column_layout.addWidget(self.pause_btn)
 
+        checkbox = QtWidgets.QCheckBox("Show only enabled", self)
+        checkbox.setStyleSheet("QCheckBox{spacing: 5px;"
+                               "padding:5px 5px 5px 5px;}")
+        checkbox.setChecked(True)
+        self.show_only_enabled_chk = checkbox
+
+        left_column_layout.addWidget(self.show_only_enabled_chk)
+
         repres = SyncRepresentationSummaryWidget(
             sync_server,
             project=self.projects.current_project,
@@ -86,15 +94,27 @@ class SyncServerWindow(QtWidgets.QDialog):
         repres.message_generated.connect(self._update_message)
         self.projects.message_generated.connect(self._update_message)
 
+        self.show_only_enabled_chk.stateChanged.connect(
+            self._on_enabled_change
+        )
+
         self.representationWidget = repres
 
+    def showEvent(self, event):
+        self.representationWidget.set_project(self.projects.current_project)
+        self.projects.refresh()
+        self._set_running(True)
+        super().showEvent(event)
+
+    def closeEvent(self, event):
+        self._set_running(False)
+        super().closeEvent(event)
+
     def _on_project_change(self):
         if self.projects.current_project is None:
             return
 
-        self.representationWidget.table_view.model().set_project(
-            self.projects.current_project
-        )
+        self.representationWidget.set_project(self.projects.current_project)
 
         project_name = self.projects.current_project
         if not self.sync_server.get_sync_project_setting(project_name):
@@ -103,16 +123,12 @@ class SyncServerWindow(QtWidgets.QDialog):
             self.projects.refresh()
             return
 
-    def showEvent(self, event):
-        self.representationWidget.model.set_project(
-            self.projects.current_project)
+    def _on_enabled_change(self):
+        """Called when enabled projects only checkbox is toggled."""
+        self.projects.show_only_enabled = \
+            self.show_only_enabled_chk.isChecked()
         self.projects.refresh()
-        self._set_running(True)
-        super().showEvent(event)
-
-    def closeEvent(self, event):
-        self._set_running(False)
-        super().closeEvent(event)
+        self.representationWidget.set_project(None)
 
     def _set_running(self, running):
         self.representationWidget.model.is_running = running

openpype/modules/sync_server/tray/models.py

@@ -52,7 +52,8 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
 
             All queries should go through this (because of collection).
         """
-        return self.sync_server.connection.database[self.project]
+        if self.project:
+            return self.sync_server.connection.database[self.project]
 
     @property
     def project(self):
@@ -150,6 +151,9 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
     @property
     def can_edit(self):
         """Returns true if some site is user local site, eg. could edit"""
+        if not self.project:
+            return False
+
         return get_local_site_id() in (self.active_site, self.remote_site)
 
     def get_column(self, index):
@@ -190,7 +194,7 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
                     actually queried (scrolled a couple of times to list more
                     than single page of records)
         """
-        if self.is_editing or not self.is_running:
+        if self.is_editing or not self.is_running or not self.project:
             return
         self.refresh_started.emit()
         self.beginResetModel()
@@ -232,6 +236,9 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
             more records in DB than loaded.
         """
         log.debug("fetchMore")
+        if not self.dbcon:
+            return
+
         items_to_fetch = min(self._total_records - self._rec_loaded,
                              self.PAGE_SIZE)
         self.query = self.get_query(self._rec_loaded)
@@ -286,9 +293,10 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
         #           replace('False', 'false').\
         #           replace('True', 'true').replace('None', 'null'))
 
-        representations = self.dbcon.aggregate(pipeline=self.query,
-                                               allowDiskUse=True)
-        self.refresh(representations)
+        if self.dbcon:
+            representations = self.dbcon.aggregate(pipeline=self.query,
+                                                   allowDiskUse=True)
+            self.refresh(representations)
 
     def set_word_filter(self, word_filter):
         """
@@ -378,9 +386,9 @@ class _SyncRepresentationModel(QtCore.QAbstractTableModel):
                 project (str): name of project
         """
         self._project = project
-        self.sync_server.set_sync_project_settings()
         # project might have been deactivated in the meantime
         if not self.sync_server.get_sync_project_setting(project):
+            self._data = {}
             return
 
         self.active_site = self.sync_server.get_active_site(self.project)
@@ -509,25 +517,23 @@ class SyncRepresentationSummaryModel(_SyncRepresentationModel):
 
         self._word_filter = None
 
-        if not self._project or self._project == lib.DUMMY_PROJECT:
-            return
-
         self.sync_server = sync_server
         # TODO think about admin mode
+        self.sort_criteria = self.DEFAULT_SORT
+
+        self.timer = QtCore.QTimer()
+        if not self._project or self._project == lib.DUMMY_PROJECT:
+            self.active_site = sync_server.DEFAULT_SITE
+            self.remote_site = sync_server.DEFAULT_SITE
+            return
+
         # this is for regular user, always only single local and single remote
         self.active_site = self.sync_server.get_active_site(self.project)
         self.remote_site = self.sync_server.get_remote_site(self.project)
 
-        self.sort_criteria = self.DEFAULT_SORT
-
         self.query = self.get_query()
         self.default_query = list(self.get_query())
 
-        representations = self.dbcon.aggregate(pipeline=self.query,
-                                               allowDiskUse=True)
-        self.refresh(representations)
-
-        self.timer = QtCore.QTimer()
         self.timer.timeout.connect(self.tick)
         self.timer.start(self.REFRESH_SEC)
 
@@ -1003,9 +1009,6 @@ class SyncRepresentationDetailModel(_SyncRepresentationModel):
         self.sort_criteria = self.DEFAULT_SORT
 
         self.query = self.get_query()
-        representations = self.dbcon.aggregate(pipeline=self.query,
-                                               allowDiskUse=True)
-        self.refresh(representations)
 
         self.timer = QtCore.QTimer()
         self.timer.timeout.connect(self.tick)

openpype/modules/sync_server/tray/widgets.py

@@ -47,6 +47,7 @@ class SyncProjectListWidget(QtWidgets.QWidget):
     message_generated = QtCore.Signal(str)
 
     refresh_msec = 10000
+    show_only_enabled = True
 
     def __init__(self, sync_server, parent):
         super(SyncProjectListWidget, self).__init__(parent)
@@ -122,11 +123,15 @@ class SyncProjectListWidget(QtWidgets.QWidget):
         self._model_reset = False
 
         selected_item = None
-        for project_name in self.sync_server.sync_project_settings.\
-                keys():
+        sync_settings = self.sync_server.sync_project_settings
+        for project_name in sync_settings.keys():
             if self.sync_server.is_paused() or \
                self.sync_server.is_project_paused(project_name):
                 icon = self._get_icon("paused")
+            elif not sync_settings[project_name]["enabled"]:
+                if self.show_only_enabled:
+                    continue
+                icon = self._get_icon("disabled")
             else:
                 icon = self._get_icon("synced")
 
@@ -139,12 +144,12 @@ class SyncProjectListWidget(QtWidgets.QWidget):
             if self.current_project == project_name:
                 selected_item = item
 
+        if model.item(0) is None:
+            return
+
         if selected_item:
             selected_index = model.indexFromItem(selected_item)
 
-        if len(self.sync_server.sync_project_settings.keys()) == 0:
-            model.appendRow(QtGui.QStandardItem(lib.DUMMY_PROJECT))
-
         if not self.current_project:
             self.current_project = model.item(0).data(QtCore.Qt.DisplayRole)
 
@@ -248,6 +253,9 @@ class _SyncRepresentationWidget(QtWidgets.QWidget):
     active_changed = QtCore.Signal()  # active index changed
     message_generated = QtCore.Signal(str)
 
+    def set_project(self, project):
+        self.model.set_project(project)
+
     def _selection_changed(self, _new_selected, _all_selected):
         idxs = self.selection_model.selectedRows()
         self._selected_ids = set()
@@ -581,7 +589,6 @@ class SyncRepresentationSummaryWidget(_SyncRepresentationWidget):
         super(SyncRepresentationSummaryWidget, self).__init__(parent)
 
         self.sync_server = sync_server
-
         self._selected_ids = set()  # keep last selected _id
 
         txt_filter = QtWidgets.QLineEdit()
@@ -625,7 +632,6 @@ class SyncRepresentationSummaryWidget(_SyncRepresentationWidget):
         column = table_view.model().get_header_index("priority")
         priority_delegate = delegates.PriorityDelegate(self)
         table_view.setItemDelegateForColumn(column, priority_delegate)
-
         layout = QtWidgets.QVBoxLayout(self)
         layout.setContentsMargins(0, 0, 0, 0)
         layout.addLayout(top_bar_layout)
@@ -633,21 +639,16 @@ class SyncRepresentationSummaryWidget(_SyncRepresentationWidget):
 
         self.table_view = table_view
         self.model = model
-
         horizontal_header = HorizontalHeader(self)
-
         table_view.setHorizontalHeader(horizontal_header)
         table_view.setSortingEnabled(True)
-
         for column_name, width in self.default_widths:
             idx = model.get_header_index(column_name)
             table_view.setColumnWidth(idx, width)
-
         table_view.doubleClicked.connect(self._double_clicked)
         self.txt_filter.textChanged.connect(lambda: model.set_word_filter(
             self.txt_filter.text()))
         table_view.customContextMenuRequested.connect(self._on_context_menu)
-
         model.refresh_started.connect(self._save_scrollbar)
         model.refresh_finished.connect(self._set_scrollbar)
         model.modelReset.connect(self._set_selection)
@@ -963,7 +964,6 @@ class HorizontalHeader(QtWidgets.QHeaderView):
         super(HorizontalHeader, self).__init__(QtCore.Qt.Horizontal, parent)
         self._parent = parent
         self.checked_values = {}
-
         self.setModel(self._parent.model)
 
         self.setSectionsClickable(True)

openpype/modules/timers_manager/timers_manager.py

@@ -7,6 +7,7 @@ from openpype_interfaces import (
     ITrayService,
     ILaunchHookPaths
 )
+from openpype.lib.events import register_event_callback
 from openpype.pipeline import AvalonMongoDB
 
 from .exceptions import InvalidContextError
@@ -422,3 +423,20 @@ class TimersManager(OpenPypeModule, ITrayService, ILaunchHookPaths):
         }
 
         return requests.post(rest_api_url, json=data)
+
+    def on_host_install(self, host, host_name, project_name):
+        self.log.debug("Installing task changed callback")
+        register_event_callback("taskChanged", self._on_host_task_change)
+
+    def _on_host_task_change(self, event):
+        project_name = event["project_name"]
+        asset_name = event["asset_name"]
+        task_name = event["task_name"]
+        self.log.debug((
+            "Sending message that timer should change to"
+            " Project: {} Asset: {} Task: {}"
+        ).format(project_name, asset_name, task_name))
+
+        self.start_timer_with_webserver(
+            project_name, asset_name, task_name, self.log
+        )

openpype/pipeline/context_tools.py

@@ -16,9 +16,7 @@ from openpype.modules import load_modules, ModulesManager
 from openpype.settings import get_project_settings
 from openpype.lib import (
     Anatomy,
-    register_event_callback,
     filter_pyblish_plugins,
-    change_timer_to_current_context,
 )
 
 from . import (
@@ -33,6 +31,9 @@ from . import (
 _is_installed = False
 _registered_root = {"_": ""}
 _registered_host = {"_": None}
+# Keep modules manager (and it's modules) in memory
+# - that gives option to register modules' callbacks
+_modules_manager = None
 
 log = logging.getLogger(__name__)
 
@@ -44,6 +45,23 @@ PUBLISH_PATH = os.path.join(PLUGINS_DIR, "publish")
 LOAD_PATH = os.path.join(PLUGINS_DIR, "load")
 
 
+def _get_modules_manager():
+    """Get or create modules manager for host installation.
+
+    This is not meant for public usage. Reason is to keep modules
+    in memory of process to be able trigger their event callbacks if they
+    need any.
+
+    Returns:
+        ModulesManager: Manager wrapping discovered modules.
+    """
+
+    global _modules_manager
+    if _modules_manager is None:
+        _modules_manager = ModulesManager()
+    return _modules_manager
+
+
 def register_root(path):
     """Register currently active root"""
     log.info("Registering root: %s" % path)
@@ -74,6 +92,7 @@ def install_host(host):
     _is_installed = True
 
     legacy_io.install()
+    modules_manager = _get_modules_manager()
 
     missing = list()
     for key in ("AVALON_PROJECT", "AVALON_ASSET"):
@@ -95,8 +114,6 @@ def install_host(host):
 
     register_host(host)
 
-    register_event_callback("taskChanged", _on_task_change)
-
     def modified_emit(obj, record):
         """Method replacing `emit` in Pyblish's MessageHandler."""
         record.msg = record.getMessage()
@@ -112,7 +129,14 @@ def install_host(host):
     else:
         pyblish.api.register_target("local")
 
-    install_openpype_plugins()
+    project_name = os.environ.get("AVALON_PROJECT")
+    host_name = os.environ.get("AVALON_APP")
+
+    # Give option to handle host installation
+    for module in modules_manager.get_enabled_modules():
+        module.on_host_install(host, host_name, project_name)
+
+    install_openpype_plugins(project_name, host_name)
 
 
 def install_openpype_plugins(project_name=None, host_name=None):
@@ -124,7 +148,7 @@ def install_openpype_plugins(project_name=None, host_name=None):
     pyblish.api.register_discovery_filter(filter_pyblish_plugins)
     register_loader_plugin_path(LOAD_PATH)
 
-    modules_manager = ModulesManager()
+    modules_manager = _get_modules_manager()
     publish_plugin_dirs = modules_manager.collect_plugin_paths()["publish"]
     for path in publish_plugin_dirs:
         pyblish.api.register_plugin_path(path)
@@ -168,10 +192,6 @@ def install_openpype_plugins(project_name=None, host_name=None):
             register_inventory_action(path)
 
 
-def _on_task_change():
-    change_timer_to_current_context()
-
-
 def uninstall_host():
     """Undo all of what `install()` did"""
     host = registered_host()

openpype/pipeline/create/context.py

@@ -829,9 +829,10 @@ class CreateContext:
             discover_result = publish_plugins_discover()
             publish_plugins = discover_result.plugins
 
-            targets = pyblish.logic.registered_targets() or ["default"]
+            targets = set(pyblish.logic.registered_targets())
+            targets.add("default")
             plugins_by_targets = pyblish.logic.plugins_by_targets(
-                publish_plugins, targets
+                publish_plugins, list(targets)
             )
             # Collect plugins that can have attribute definitions
             for plugin in publish_plugins: