diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index 6ed6ae428c..96e768e420 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -6,6 +6,8 @@ labels: bug assignees: '' --- +**Running version** +[ex. 3.14.1-nightly.2] **Describe the bug** A clear and concise description of what the bug is. diff --git a/.gitignore b/.gitignore index ea5b20eb69..4b773e97ed 100644 --- a/.gitignore +++ b/.gitignore @@ -107,3 +107,6 @@ website/.docusaurus mypy.ini tools/run_eventserver.* + +# Developer tools +tools/dev_* diff --git a/CHANGELOG.md b/CHANGELOG.md index 7c8834dd49..24e02acc6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,145 +1,133 @@ # Changelog -## [3.14.2-nightly.1](https://github.com/pypeclub/OpenPype/tree/HEAD) +## [3.14.3-nightly.4](https://github.com/pypeclub/OpenPype/tree/HEAD) -[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.14.1...HEAD) +[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.14.2...HEAD) -**🆕 New features** +**🚀 Enhancements** -- Houdini: Publishing workfiles [\#3697](https://github.com/pypeclub/OpenPype/pull/3697) +- Maya: better logging in Maketx [\#3886](https://github.com/pypeclub/OpenPype/pull/3886) +- Photoshop: review can be turned off [\#3885](https://github.com/pypeclub/OpenPype/pull/3885) +- TrayPublisher: added persisting of last selected project [\#3871](https://github.com/pypeclub/OpenPype/pull/3871) +- TrayPublisher: added text filter on project name to Tray Publisher [\#3867](https://github.com/pypeclub/OpenPype/pull/3867) +- Github issues adding `running version` section [\#3864](https://github.com/pypeclub/OpenPype/pull/3864) +- Publisher: Increase size of main window [\#3862](https://github.com/pypeclub/OpenPype/pull/3862) +- Flame: make migratable projects after creation [\#3860](https://github.com/pypeclub/OpenPype/pull/3860) +- Photoshop: synchronize image version with workfile [\#3854](https://github.com/pypeclub/OpenPype/pull/3854) +- General: Transcoding handle float2 attr type [\#3849](https://github.com/pypeclub/OpenPype/pull/3849) +- General: Simple script for getting license information about used packages [\#3843](https://github.com/pypeclub/OpenPype/pull/3843) +- Houdini: Increment current file on workfile publish [\#3840](https://github.com/pypeclub/OpenPype/pull/3840) +- Publisher: Add new publisher to host tools [\#3833](https://github.com/pypeclub/OpenPype/pull/3833) +- General: lock task workfiles when they are working on [\#3810](https://github.com/pypeclub/OpenPype/pull/3810) +- Maya: Workspace mel loaded from settings [\#3790](https://github.com/pypeclub/OpenPype/pull/3790) **🐛 Bug fixes** -- Ftrack status fix typo prgoress -\> progress [\#3761](https://github.com/pypeclub/OpenPype/pull/3761) +- Flame: loading multilayer exr to batch/reel is working [\#3901](https://github.com/pypeclub/OpenPype/pull/3901) +- Hiero: Fix inventory check on launch [\#3895](https://github.com/pypeclub/OpenPype/pull/3895) +- WebPublisher: Fix import after refactor [\#3891](https://github.com/pypeclub/OpenPype/pull/3891) +- TVPaint: Fix renaming of rendered files [\#3882](https://github.com/pypeclub/OpenPype/pull/3882) +- Publisher: Nice checkbox visible in Python 2 [\#3877](https://github.com/pypeclub/OpenPype/pull/3877) +- Settings: Add missing default settings [\#3870](https://github.com/pypeclub/OpenPype/pull/3870) +- General: Copy of workfile does not use 'copy' function but 'copyfile' [\#3869](https://github.com/pypeclub/OpenPype/pull/3869) +- Tray Publisher: skip plugin if otioTimeline is missing [\#3856](https://github.com/pypeclub/OpenPype/pull/3856) +- Flame: retimed attributes are integrated with settings [\#3855](https://github.com/pypeclub/OpenPype/pull/3855) +- Maya: Extract Playblast fix textures + labelize viewport show settings [\#3852](https://github.com/pypeclub/OpenPype/pull/3852) +- Ftrack: Url validation does not require ftrackapp [\#3834](https://github.com/pypeclub/OpenPype/pull/3834) +- Maya+Ftrack: Change typo in family name `mayaascii` -\> `mayaAscii` [\#3820](https://github.com/pypeclub/OpenPype/pull/3820) +- Maya Deadline: Fix Tile Rendering by forcing integer pixel values [\#3758](https://github.com/pypeclub/OpenPype/pull/3758) **🔀 Refactored code** +- Houdini: Use new Extractor location [\#3894](https://github.com/pypeclub/OpenPype/pull/3894) +- Harmony: Use new Extractor location [\#3893](https://github.com/pypeclub/OpenPype/pull/3893) +- Hiero: Use new Extractor location [\#3851](https://github.com/pypeclub/OpenPype/pull/3851) +- Maya: Remove old legacy \(ftrack\) plug-ins that are of no use anymore [\#3819](https://github.com/pypeclub/OpenPype/pull/3819) +- Nuke: Use new Extractor location [\#3799](https://github.com/pypeclub/OpenPype/pull/3799) +- Maya: Use new Extractor location [\#3775](https://github.com/pypeclub/OpenPype/pull/3775) + +**Merged pull requests:** + +- Maya: RenderSettings set default image format for V-Ray+Redshift to exr [\#3879](https://github.com/pypeclub/OpenPype/pull/3879) +- Remove lockfile during publish [\#3874](https://github.com/pypeclub/OpenPype/pull/3874) + +## [3.14.2](https://github.com/pypeclub/OpenPype/tree/3.14.2) (2022-09-12) + +[Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.14.2-nightly.5...3.14.2) + +**🆕 New features** + +- Nuke: Build workfile by template [\#3763](https://github.com/pypeclub/OpenPype/pull/3763) + +**🚀 Enhancements** + +- Flame: Adding Creator's retimed shot and handles switch [\#3826](https://github.com/pypeclub/OpenPype/pull/3826) +- Flame: OpenPype submenu to batch and media manager [\#3825](https://github.com/pypeclub/OpenPype/pull/3825) +- General: Better pixmap scaling [\#3809](https://github.com/pypeclub/OpenPype/pull/3809) +- Photoshop: attempt to speed up ExtractImage [\#3793](https://github.com/pypeclub/OpenPype/pull/3793) +- SyncServer: Added cli commands for sync server [\#3765](https://github.com/pypeclub/OpenPype/pull/3765) + +**🐛 Bug fixes** + +- General: Fix Pattern access in client code [\#3828](https://github.com/pypeclub/OpenPype/pull/3828) +- Launcher: Skip opening last work file works for groups [\#3822](https://github.com/pypeclub/OpenPype/pull/3822) +- Maya: Publishing data key change [\#3811](https://github.com/pypeclub/OpenPype/pull/3811) +- Igniter: Fix status handling when version is already installed [\#3804](https://github.com/pypeclub/OpenPype/pull/3804) +- Resolve: Addon import is Python 2 compatible [\#3798](https://github.com/pypeclub/OpenPype/pull/3798) +- Hiero: retimed clip publishing is working [\#3792](https://github.com/pypeclub/OpenPype/pull/3792) +- nuke: validate write node is not failing due wrong type [\#3780](https://github.com/pypeclub/OpenPype/pull/3780) +- Fix - changed format of version string in pyproject.toml [\#3777](https://github.com/pypeclub/OpenPype/pull/3777) +- Ftrack status fix typo prgoress -\> progress [\#3761](https://github.com/pypeclub/OpenPype/pull/3761) +- Fix version resolution [\#3757](https://github.com/pypeclub/OpenPype/pull/3757) + +**🔀 Refactored code** + +- Photoshop: Use new Extractor location [\#3789](https://github.com/pypeclub/OpenPype/pull/3789) +- Blender: Use new Extractor location [\#3787](https://github.com/pypeclub/OpenPype/pull/3787) +- AfterEffects: Use new Extractor location [\#3784](https://github.com/pypeclub/OpenPype/pull/3784) +- General: Remove unused teshost [\#3773](https://github.com/pypeclub/OpenPype/pull/3773) +- General: Copied 'Extractor' plugin to publish pipeline [\#3771](https://github.com/pypeclub/OpenPype/pull/3771) +- General: Move queries of asset and representation links [\#3770](https://github.com/pypeclub/OpenPype/pull/3770) +- General: Move create project folders to pipeline [\#3768](https://github.com/pypeclub/OpenPype/pull/3768) +- General: Create project function moved to client code [\#3766](https://github.com/pypeclub/OpenPype/pull/3766) +- Maya: Refactor submit deadline to use AbstractSubmitDeadline [\#3759](https://github.com/pypeclub/OpenPype/pull/3759) +- General: Change publish template settings location [\#3755](https://github.com/pypeclub/OpenPype/pull/3755) - General: Move hostdirname functionality into host [\#3749](https://github.com/pypeclub/OpenPype/pull/3749) -- Webpublisher: Webpublisher is used as addon [\#3740](https://github.com/pypeclub/OpenPype/pull/3740) -- Houdini: Define houdini as addon [\#3735](https://github.com/pypeclub/OpenPype/pull/3735) -- Flame: Defined flame as addon [\#3732](https://github.com/pypeclub/OpenPype/pull/3732) +- General: Move publish utils to pipeline [\#3745](https://github.com/pypeclub/OpenPype/pull/3745) + +**Merged pull requests:** + +- Standalone Publisher: Ignore empty labels, then still use name like other asset models [\#3779](https://github.com/pypeclub/OpenPype/pull/3779) +- Kitsu - sync\_all\_project - add list ignore\_projects [\#3776](https://github.com/pypeclub/OpenPype/pull/3776) ## [3.14.1](https://github.com/pypeclub/OpenPype/tree/3.14.1) (2022-08-30) [Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.14.1-nightly.4...3.14.1) -### 📖 Documentation - -- Documentation: Few updates [\#3698](https://github.com/pypeclub/OpenPype/pull/3698) -- Documentation: Settings development [\#3660](https://github.com/pypeclub/OpenPype/pull/3660) - -**🆕 New features** - -- Webpublisher:change create flatten image into tri state [\#3678](https://github.com/pypeclub/OpenPype/pull/3678) -- Blender: validators code correction with settings and defaults [\#3662](https://github.com/pypeclub/OpenPype/pull/3662) - **🚀 Enhancements** - General: Thumbnail can use project roots [\#3750](https://github.com/pypeclub/OpenPype/pull/3750) -- Settings: Remove settings lock on tray exit [\#3720](https://github.com/pypeclub/OpenPype/pull/3720) -- General: Added helper getters to modules manager [\#3712](https://github.com/pypeclub/OpenPype/pull/3712) -- Unreal: Define unreal as module and use host class [\#3701](https://github.com/pypeclub/OpenPype/pull/3701) -- Settings: Lock settings UI session [\#3700](https://github.com/pypeclub/OpenPype/pull/3700) -- General: Benevolent context label collector [\#3686](https://github.com/pypeclub/OpenPype/pull/3686) -- Ftrack: Store ftrack entities on hierarchy integration to instances [\#3677](https://github.com/pypeclub/OpenPype/pull/3677) -- Blender: ops refresh manager after process events [\#3663](https://github.com/pypeclub/OpenPype/pull/3663) **🐛 Bug fixes** - Maya: Fix typo in getPanel argument `with\_focus` -\> `withFocus` [\#3753](https://github.com/pypeclub/OpenPype/pull/3753) - General: Smaller fixes of imports [\#3748](https://github.com/pypeclub/OpenPype/pull/3748) - General: Logger tweaks [\#3741](https://github.com/pypeclub/OpenPype/pull/3741) -- Nuke: missing job dependency if multiple bake streams [\#3737](https://github.com/pypeclub/OpenPype/pull/3737) -- Nuke: color-space settings from anatomy is working [\#3721](https://github.com/pypeclub/OpenPype/pull/3721) -- Settings: Fix studio default anatomy save [\#3716](https://github.com/pypeclub/OpenPype/pull/3716) -- Maya: Use project name instead of project code [\#3709](https://github.com/pypeclub/OpenPype/pull/3709) -- Settings: Fix project overrides save [\#3708](https://github.com/pypeclub/OpenPype/pull/3708) -- Workfiles tool: Fix published workfile filtering [\#3704](https://github.com/pypeclub/OpenPype/pull/3704) -- PS, AE: Provide default variant value for workfile subset [\#3703](https://github.com/pypeclub/OpenPype/pull/3703) -- Flame: retime is working on clip publishing [\#3684](https://github.com/pypeclub/OpenPype/pull/3684) -- Webpublisher: added check for empty context [\#3682](https://github.com/pypeclub/OpenPype/pull/3682) **🔀 Refactored code** - General: Move delivery logic to pipeline [\#3751](https://github.com/pypeclub/OpenPype/pull/3751) - General: Host addons cleanup [\#3744](https://github.com/pypeclub/OpenPype/pull/3744) -- Photoshop: Defined photoshop as addon [\#3736](https://github.com/pypeclub/OpenPype/pull/3736) -- Harmony: Defined harmony as addon [\#3734](https://github.com/pypeclub/OpenPype/pull/3734) -- General: Module interfaces cleanup [\#3731](https://github.com/pypeclub/OpenPype/pull/3731) -- AfterEffects: Move AE functions from general lib [\#3730](https://github.com/pypeclub/OpenPype/pull/3730) -- Blender: Define blender as module [\#3729](https://github.com/pypeclub/OpenPype/pull/3729) -- AfterEffects: Define AfterEffects as module [\#3728](https://github.com/pypeclub/OpenPype/pull/3728) -- General: Replace PypeLogger with Logger [\#3725](https://github.com/pypeclub/OpenPype/pull/3725) -- Nuke: Define nuke as module [\#3724](https://github.com/pypeclub/OpenPype/pull/3724) -- General: Move subset name functionality [\#3723](https://github.com/pypeclub/OpenPype/pull/3723) -- General: Move creators plugin getter [\#3714](https://github.com/pypeclub/OpenPype/pull/3714) -- General: Move constants from lib to client [\#3713](https://github.com/pypeclub/OpenPype/pull/3713) -- Loader: Subset groups using client operations [\#3710](https://github.com/pypeclub/OpenPype/pull/3710) -- TVPaint: Defined as module [\#3707](https://github.com/pypeclub/OpenPype/pull/3707) -- StandalonePublisher: Define StandalonePublisher as module [\#3706](https://github.com/pypeclub/OpenPype/pull/3706) -- TrayPublisher: Define TrayPublisher as module [\#3705](https://github.com/pypeclub/OpenPype/pull/3705) -- General: Move context specific functions to context tools [\#3702](https://github.com/pypeclub/OpenPype/pull/3702) - -**Merged pull requests:** - -- Hiero: Define hiero as module [\#3717](https://github.com/pypeclub/OpenPype/pull/3717) -- Deadline: better logging for DL webservice failures [\#3694](https://github.com/pypeclub/OpenPype/pull/3694) -- Photoshop: resize saved images in ExtractReview for ffmpeg [\#3676](https://github.com/pypeclub/OpenPype/pull/3676) +- Webpublisher: Webpublisher is used as addon [\#3740](https://github.com/pypeclub/OpenPype/pull/3740) ## [3.14.0](https://github.com/pypeclub/OpenPype/tree/3.14.0) (2022-08-18) [Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.14.0-nightly.1...3.14.0) -**🚀 Enhancements** - -- Ftrack: Addiotional component metadata [\#3685](https://github.com/pypeclub/OpenPype/pull/3685) -- Ftrack: Set task status on task creation in integrate hierarchy [\#3675](https://github.com/pypeclub/OpenPype/pull/3675) -- Maya: Disable rendering of all lights for render instances submitted through Deadline. [\#3661](https://github.com/pypeclub/OpenPype/pull/3661) -- General: Optimized OCIO configs [\#3650](https://github.com/pypeclub/OpenPype/pull/3650) - -**🐛 Bug fixes** - -- General: Switch from hero version to versioned works [\#3691](https://github.com/pypeclub/OpenPype/pull/3691) -- General: Fix finding of last version [\#3656](https://github.com/pypeclub/OpenPype/pull/3656) -- General: Extract Review can scale with pixel aspect ratio [\#3644](https://github.com/pypeclub/OpenPype/pull/3644) -- Maya: Refactor moved usage of CreateRender settings [\#3643](https://github.com/pypeclub/OpenPype/pull/3643) -- General: Hero version representations have full context [\#3638](https://github.com/pypeclub/OpenPype/pull/3638) -- Nuke: color settings for render write node is working now [\#3632](https://github.com/pypeclub/OpenPype/pull/3632) -- Maya: FBX support for update in reference loader [\#3631](https://github.com/pypeclub/OpenPype/pull/3631) - -**🔀 Refactored code** - -- General: Use client projects getter [\#3673](https://github.com/pypeclub/OpenPype/pull/3673) -- Resolve: Match folder structure to other hosts [\#3653](https://github.com/pypeclub/OpenPype/pull/3653) -- Maya: Hosts as modules [\#3647](https://github.com/pypeclub/OpenPype/pull/3647) -- TimersManager: Plugins are in timers manager module [\#3639](https://github.com/pypeclub/OpenPype/pull/3639) -- General: Move workfiles functions into pipeline [\#3637](https://github.com/pypeclub/OpenPype/pull/3637) - -**Merged pull requests:** - -- Deadline: Global job pre load is not Pype 2 compatible [\#3666](https://github.com/pypeclub/OpenPype/pull/3666) -- Maya: Remove unused get current renderer logic [\#3645](https://github.com/pypeclub/OpenPype/pull/3645) -- Kitsu|Fix: Movie project type fails & first loop children names [\#3636](https://github.com/pypeclub/OpenPype/pull/3636) -- fix the bug of failing to extract look when UDIMs format used in AiImage [\#3628](https://github.com/pypeclub/OpenPype/pull/3628) - ## [3.13.0](https://github.com/pypeclub/OpenPype/tree/3.13.0) (2022-08-09) [Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.13.0-nightly.1...3.13.0) -**🚀 Enhancements** - -- Editorial: Mix audio use side file for ffmpeg filters [\#3630](https://github.com/pypeclub/OpenPype/pull/3630) - -**🐛 Bug fixes** - -- Maya: fix aov separator in Redshift [\#3625](https://github.com/pypeclub/OpenPype/pull/3625) -- Fix for multi-version build on Mac [\#3622](https://github.com/pypeclub/OpenPype/pull/3622) -- Ftrack: Sync hierarchical attributes can handle new created entities [\#3621](https://github.com/pypeclub/OpenPype/pull/3621) - -**🔀 Refactored code** - -- General: Plugin settings handled by plugins [\#3623](https://github.com/pypeclub/OpenPype/pull/3623) - ## [3.12.2](https://github.com/pypeclub/OpenPype/tree/3.12.2) (2022-07-27) [Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.12.2-nightly.4...3.12.2) diff --git a/README.md b/README.md index b8c04f8b49..a3d3cf1dbb 100644 --- a/README.md +++ b/README.md @@ -41,7 +41,7 @@ It can be built and ran on all common platforms. We develop and test on the foll - **Linux** - **Ubuntu** 20.04 LTS - **Centos** 7 -- **Mac OSX** +- **Mac OSX** - **10.15** Catalina - **11.1** Big Sur (using Rosetta2) @@ -287,6 +287,14 @@ To run tests, execute `.\tools\run_tests(.ps1|.sh)`. **Note that it needs existing virtual environment.** + +Developer tools +------------- + +In case you wish to add your own tools to `.\tools` folder without git tracking, it is possible by adding it with `dev_*` suffix (example: `dev_clear_pyc(.ps1|.sh)`). + + + ## Contributors ✨ Thanks goes to these wonderful people ([emoji key](https://allcontributors.org/docs/en/emoji-key)): diff --git a/common/openpype_common/distribution/README.md b/common/openpype_common/distribution/README.md new file mode 100644 index 0000000000..212eb267b8 --- /dev/null +++ b/common/openpype_common/distribution/README.md @@ -0,0 +1,18 @@ +Addon distribution tool +------------------------ + +Code in this folder is backend portion of Addon distribution logic for v4 server. + +Each host, module will be separate Addon in the future. Each v4 server could run different set of Addons. + +Client (running on artist machine) will in the first step ask v4 for list of enabled addons. +(It expects list of json documents matching to `addon_distribution.py:AddonInfo` object.) +Next it will compare presence of enabled addon version in local folder. In the case of missing version of +an addon, client will use information in the addon to download (from http/shared local disk/git) zip file +and unzip it. + +Required part of addon distribution will be sharing of dependencies (python libraries, utilities) which is not part of this folder. + +Location of this folder might change in the future as it will be required for a clint to add this folder to sys.path reliably. + +This code needs to be independent on Openpype code as much as possible! \ No newline at end of file diff --git a/common/openpype_common/distribution/__init__.py b/common/openpype_common/distribution/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/common/openpype_common/distribution/addon_distribution.py b/common/openpype_common/distribution/addon_distribution.py new file mode 100644 index 0000000000..5e48639dec --- /dev/null +++ b/common/openpype_common/distribution/addon_distribution.py @@ -0,0 +1,208 @@ +import os +from enum import Enum +from abc import abstractmethod +import attr +import logging +import requests +import platform +import shutil + +from .file_handler import RemoteFileHandler +from .addon_info import AddonInfo + + +class UpdateState(Enum): + EXISTS = "exists" + UPDATED = "updated" + FAILED = "failed" + + +class AddonDownloader: + log = logging.getLogger(__name__) + + def __init__(self): + self._downloaders = {} + + def register_format(self, downloader_type, downloader): + self._downloaders[downloader_type.value] = downloader + + def get_downloader(self, downloader_type): + downloader = self._downloaders.get(downloader_type) + if not downloader: + raise ValueError(f"{downloader_type} not implemented") + return downloader() + + @classmethod + @abstractmethod + def download(cls, source, destination): + """Returns url to downloaded addon zip file. + + Args: + source (dict): {type:"http", "url":"https://} ...} + destination (str): local folder to unzip + Returns: + (str) local path to addon zip file + """ + pass + + @classmethod + def check_hash(cls, addon_path, addon_hash): + """Compares 'hash' of downloaded 'addon_url' file. + + Args: + addon_path (str): local path to addon zip file + addon_hash (str): sha256 hash of zip file + Raises: + ValueError if hashes doesn't match + """ + if not os.path.exists(addon_path): + raise ValueError(f"{addon_path} doesn't exist.") + if not RemoteFileHandler.check_integrity(addon_path, + addon_hash, + hash_type="sha256"): + raise ValueError(f"{addon_path} doesn't match expected hash.") + + @classmethod + def unzip(cls, addon_zip_path, destination): + """Unzips local 'addon_zip_path' to 'destination'. + + Args: + addon_zip_path (str): local path to addon zip file + destination (str): local folder to unzip + """ + RemoteFileHandler.unzip(addon_zip_path, destination) + os.remove(addon_zip_path) + + @classmethod + def remove(cls, addon_url): + pass + + +class OSAddonDownloader(AddonDownloader): + + @classmethod + def download(cls, source, destination): + # OS doesnt need to download, unzip directly + addon_url = source["path"].get(platform.system().lower()) + if not os.path.exists(addon_url): + raise ValueError("{} is not accessible".format(addon_url)) + return addon_url + + +class HTTPAddonDownloader(AddonDownloader): + CHUNK_SIZE = 100000 + + @classmethod + def download(cls, source, destination): + source_url = source["url"] + cls.log.debug(f"Downloading {source_url} to {destination}") + file_name = os.path.basename(destination) + _, ext = os.path.splitext(file_name) + if (ext.replace(".", '') not + in set(RemoteFileHandler.IMPLEMENTED_ZIP_FORMATS)): + file_name += ".zip" + RemoteFileHandler.download_url(source_url, + destination, + filename=file_name) + + return os.path.join(destination, file_name) + + +def get_addons_info(server_endpoint): + """Returns list of addon information from Server""" + # TODO temp + # addon_info = AddonInfo( + # **{"name": "openpype_slack", + # "version": "1.0.0", + # "addon_url": "c:/projects/openpype_slack_1.0.0.zip", + # "type": UrlType.FILESYSTEM, + # "hash": "4be25eb6215e91e5894d3c5475aeb1e379d081d3f5b43b4ee15b0891cf5f5658"}) # noqa + # + # http_addon = AddonInfo( + # **{"name": "openpype_slack", + # "version": "1.0.0", + # "addon_url": "https://drive.google.com/file/d/1TcuV8c2OV8CcbPeWi7lxOdqWsEqQNPYy/view?usp=sharing", # noqa + # "type": UrlType.HTTP, + # "hash": "4be25eb6215e91e5894d3c5475aeb1e379d081d3f5b43b4ee15b0891cf5f5658"}) # noqa + + response = requests.get(server_endpoint) + if not response.ok: + raise Exception(response.text) + + addons_info = [] + for addon in response.json(): + addons_info.append(AddonInfo(**addon)) + return addons_info + + +def update_addon_state(addon_infos, destination_folder, factory, + log=None): + """Loops through all 'addon_infos', compares local version, unzips. + + Loops through server provided list of dictionaries with information about + available addons. Looks if each addon is already present and deployed. + If isn't, addon zip gets downloaded and unzipped into 'destination_folder'. + Args: + addon_infos (list of AddonInfo) + destination_folder (str): local path + factory (AddonDownloader): factory to get appropriate downloader per + addon type + log (logging.Logger) + Returns: + (dict): {"addon_full_name": UpdateState.value + (eg. "exists"|"updated"|"failed") + """ + if not log: + log = logging.getLogger(__name__) + + download_states = {} + for addon in addon_infos: + full_name = "{}_{}".format(addon.name, addon.version) + addon_dest = os.path.join(destination_folder, full_name) + + if os.path.isdir(addon_dest): + log.debug(f"Addon version folder {addon_dest} already exists.") + download_states[full_name] = UpdateState.EXISTS.value + continue + + for source in addon.sources: + download_states[full_name] = UpdateState.FAILED.value + try: + downloader = factory.get_downloader(source.type) + zip_file_path = downloader.download(attr.asdict(source), + addon_dest) + downloader.check_hash(zip_file_path, addon.hash) + downloader.unzip(zip_file_path, addon_dest) + download_states[full_name] = UpdateState.UPDATED.value + break + except Exception: + log.warning(f"Error happened during updating {addon.name}", + exc_info=True) + if os.path.isdir(addon_dest): + log.debug(f"Cleaning {addon_dest}") + shutil.rmtree(addon_dest) + + return download_states + + +def check_addons(server_endpoint, addon_folder, downloaders): + """Main entry point to compare existing addons with those on server. + + Args: + server_endpoint (str): url to v4 server endpoint + addon_folder (str): local dir path for addons + downloaders (AddonDownloader): factory of downloaders + + Raises: + (RuntimeError) if any addon failed update + """ + addons_info = get_addons_info(server_endpoint) + result = update_addon_state(addons_info, + addon_folder, + downloaders) + if UpdateState.FAILED.value in result.values(): + raise RuntimeError(f"Unable to update some addons {result}") + + +def cli(*args): + raise NotImplementedError diff --git a/common/openpype_common/distribution/addon_info.py b/common/openpype_common/distribution/addon_info.py new file mode 100644 index 0000000000..00ece11f3b --- /dev/null +++ b/common/openpype_common/distribution/addon_info.py @@ -0,0 +1,80 @@ +import attr +from enum import Enum + + +class UrlType(Enum): + HTTP = "http" + GIT = "git" + FILESYSTEM = "filesystem" + + +@attr.s +class MultiPlatformPath(object): + windows = attr.ib(default=None) + linux = attr.ib(default=None) + darwin = attr.ib(default=None) + + +@attr.s +class AddonSource(object): + type = attr.ib() + + +@attr.s +class LocalAddonSource(AddonSource): + path = attr.ib(default=attr.Factory(MultiPlatformPath)) + + +@attr.s +class WebAddonSource(AddonSource): + url = attr.ib(default=None) + + +@attr.s +class VersionData(object): + version_data = attr.ib(default=None) + + +@attr.s +class AddonInfo(object): + """Object matching json payload from Server""" + name = attr.ib() + version = attr.ib() + title = attr.ib(default=None) + sources = attr.ib(default=attr.Factory(dict)) + hash = attr.ib(default=None) + description = attr.ib(default=None) + license = attr.ib(default=None) + authors = attr.ib(default=None) + + @classmethod + def from_dict(cls, data): + sources = [] + + production_version = data.get("productionVersion") + if not production_version: + return + + # server payload contains info about all versions + # active addon must have 'productionVersion' and matching version info + version_data = data.get("versions", {})[production_version] + + for source in version_data.get("clientSourceInfo", []): + if source.get("type") == UrlType.FILESYSTEM.value: + source_addon = LocalAddonSource(type=source["type"], + path=source["path"]) + if source.get("type") == UrlType.HTTP.value: + source_addon = WebAddonSource(type=source["type"], + url=source["url"]) + + sources.append(source_addon) + + return cls(name=data.get("name"), + version=production_version, + sources=sources, + hash=data.get("hash"), + description=data.get("description"), + title=data.get("title"), + license=data.get("license"), + authors=data.get("authors")) + diff --git a/tests/lib/file_handler.py b/common/openpype_common/distribution/file_handler.py similarity index 83% rename from tests/lib/file_handler.py rename to common/openpype_common/distribution/file_handler.py index ee3abc6ecb..f585c77632 100644 --- a/tests/lib/file_handler.py +++ b/common/openpype_common/distribution/file_handler.py @@ -21,7 +21,7 @@ class RemoteFileHandler: 'tar.gz', 'tar.xz', 'tar.bz2'] @staticmethod - def calculate_md5(fpath, chunk_size): + def calculate_md5(fpath, chunk_size=10000): md5 = hashlib.md5() with open(fpath, 'rb') as f: for chunk in iter(lambda: f.read(chunk_size), b''): @@ -33,17 +33,45 @@ class RemoteFileHandler: return md5 == RemoteFileHandler.calculate_md5(fpath, **kwargs) @staticmethod - def check_integrity(fpath, md5=None): + def calculate_sha256(fpath): + """Calculate sha256 for content of the file. + + Args: + fpath (str): Path to file. + + Returns: + str: hex encoded sha256 + + """ + h = hashlib.sha256() + b = bytearray(128 * 1024) + mv = memoryview(b) + with open(fpath, 'rb', buffering=0) as f: + for n in iter(lambda: f.readinto(mv), 0): + h.update(mv[:n]) + return h.hexdigest() + + @staticmethod + def check_sha256(fpath, sha256, **kwargs): + return sha256 == RemoteFileHandler.calculate_sha256(fpath, **kwargs) + + @staticmethod + def check_integrity(fpath, hash_value=None, hash_type=None): if not os.path.isfile(fpath): return False - if md5 is None: + if hash_value is None: return True - return RemoteFileHandler.check_md5(fpath, md5) + if not hash_type: + raise ValueError("Provide hash type, md5 or sha256") + if hash_type == 'md5': + return RemoteFileHandler.check_md5(fpath, hash_value) + if hash_type == "sha256": + return RemoteFileHandler.check_sha256(fpath, hash_value) @staticmethod def download_url( url, root, filename=None, - md5=None, max_redirect_hops=3 + sha256=None, max_redirect_hops=3 ): """Download a file from a url and place it in root. Args: @@ -51,7 +79,7 @@ class RemoteFileHandler: root (str): Directory to place downloaded file in filename (str, optional): Name to save the file under. If None, use the basename of the URL - md5 (str, optional): MD5 checksum of the download. + sha256 (str, optional): sha256 checksum of the download. If None, do not check max_redirect_hops (int, optional): Maximum number of redirect hops allowed @@ -64,7 +92,8 @@ class RemoteFileHandler: os.makedirs(root, exist_ok=True) # check if file is already present locally - if RemoteFileHandler.check_integrity(fpath, md5): + if RemoteFileHandler.check_integrity(fpath, + sha256, hash_type="sha256"): print('Using downloaded and verified file: ' + fpath) return @@ -76,7 +105,7 @@ class RemoteFileHandler: file_id = RemoteFileHandler._get_google_drive_file_id(url) if file_id is not None: return RemoteFileHandler.download_file_from_google_drive( - file_id, root, filename, md5) + file_id, root, filename, sha256) # download the file try: @@ -92,20 +121,21 @@ class RemoteFileHandler: raise e # check integrity of downloaded file - if not RemoteFileHandler.check_integrity(fpath, md5): + if not RemoteFileHandler.check_integrity(fpath, + sha256, hash_type="sha256"): raise RuntimeError("File not found or corrupted.") @staticmethod def download_file_from_google_drive(file_id, root, filename=None, - md5=None): + sha256=None): """Download a Google Drive file from and place it in root. Args: file_id (str): id of file to be downloaded root (str): Directory to place downloaded file in filename (str, optional): Name to save the file under. If None, use the id of the file. - md5 (str, optional): MD5 checksum of the download. + sha256 (str, optional): sha256 checksum of the download. If None, do not check """ # Based on https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url # noqa @@ -119,8 +149,8 @@ class RemoteFileHandler: os.makedirs(root, exist_ok=True) - if os.path.isfile(fpath) and RemoteFileHandler.check_integrity(fpath, - md5): + if os.path.isfile(fpath) and RemoteFileHandler.check_integrity( + fpath, sha256, hash_type="sha256"): print('Using downloaded and verified file: ' + fpath) else: session = requests.Session() diff --git a/common/openpype_common/distribution/tests/test_addon_distributtion.py b/common/openpype_common/distribution/tests/test_addon_distributtion.py new file mode 100644 index 0000000000..765ea0596a --- /dev/null +++ b/common/openpype_common/distribution/tests/test_addon_distributtion.py @@ -0,0 +1,167 @@ +import pytest +import attr +import tempfile + +from common.openpype_common.distribution.addon_distribution import ( + AddonDownloader, + OSAddonDownloader, + HTTPAddonDownloader, + AddonInfo, + update_addon_state, + UpdateState +) +from common.openpype_common.distribution.addon_info import UrlType + + +@pytest.fixture +def addon_downloader(): + addon_downloader = AddonDownloader() + addon_downloader.register_format(UrlType.FILESYSTEM, OSAddonDownloader) + addon_downloader.register_format(UrlType.HTTP, HTTPAddonDownloader) + + yield addon_downloader + + +@pytest.fixture +def http_downloader(addon_downloader): + yield addon_downloader.get_downloader(UrlType.HTTP.value) + + +@pytest.fixture +def temp_folder(): + yield tempfile.mkdtemp() + + +@pytest.fixture +def sample_addon_info(): + addon_info = { + "versions": { + "1.0.0": { + "clientPyproject": { + "tool": { + "poetry": { + "dependencies": { + "nxtools": "^1.6", + "orjson": "^3.6.7", + "typer": "^0.4.1", + "email-validator": "^1.1.3", + "python": "^3.10", + "fastapi": "^0.73.0" + } + } + } + }, + "hasSettings": True, + "clientSourceInfo": [ + { + "type": "http", + "url": "https://drive.google.com/file/d/1TcuV8c2OV8CcbPeWi7lxOdqWsEqQNPYy/view?usp=sharing" # noqa + }, + { + "type": "filesystem", + "path": { + "windows": ["P:/sources/some_file.zip", + "W:/sources/some_file.zip"], # noqa + "linux": ["/mnt/srv/sources/some_file.zip"], + "darwin": ["/Volumes/srv/sources/some_file.zip"] + } + } + ], + "frontendScopes": { + "project": { + "sidebar": "hierarchy" + } + } + } + }, + "description": "", + "title": "Slack addon", + "name": "openpype_slack", + "productionVersion": "1.0.0", + "hash": "4be25eb6215e91e5894d3c5475aeb1e379d081d3f5b43b4ee15b0891cf5f5658" # noqa + } + yield addon_info + + +def test_register(printer): + addon_downloader = AddonDownloader() + + assert len(addon_downloader._downloaders) == 0, "Contains registered" + + addon_downloader.register_format(UrlType.FILESYSTEM, OSAddonDownloader) + assert len(addon_downloader._downloaders) == 1, "Should contain one" + + +def test_get_downloader(printer, addon_downloader): + assert addon_downloader.get_downloader(UrlType.FILESYSTEM.value), "Should find" # noqa + + with pytest.raises(ValueError): + addon_downloader.get_downloader("unknown"), "Shouldn't find" + + +def test_addon_info(printer, sample_addon_info): + """Tests parsing of expected payload from v4 server into AadonInfo.""" + valid_minimum = { + "name": "openpype_slack", + "productionVersion": "1.0.0", + "versions": { + "1.0.0": { + "clientSourceInfo": [ + { + "type": "filesystem", + "path": { + "windows": [ + "P:/sources/some_file.zip", + "W:/sources/some_file.zip"], + "linux": [ + "/mnt/srv/sources/some_file.zip"], + "darwin": [ + "/Volumes/srv/sources/some_file.zip"] # noqa + } + } + ] + } + } + } + + assert AddonInfo.from_dict(valid_minimum), "Missing required fields" + + valid_minimum["versions"].pop("1.0.0") + with pytest.raises(KeyError): + assert not AddonInfo.from_dict(valid_minimum), "Must fail without version data" # noqa + + valid_minimum.pop("productionVersion") + assert not AddonInfo.from_dict( + valid_minimum), "none if not productionVersion" # noqa + + addon = AddonInfo.from_dict(sample_addon_info) + assert addon, "Should be created" + assert addon.name == "openpype_slack", "Incorrect name" + assert addon.version == "1.0.0", "Incorrect version" + + with pytest.raises(TypeError): + assert addon["name"], "Dict approach not implemented" + + addon_as_dict = attr.asdict(addon) + assert addon_as_dict["name"], "Dict approach should work" + + +def test_update_addon_state(printer, sample_addon_info, + temp_folder, addon_downloader): + """Tests possible cases of addon update.""" + addon_info = AddonInfo.from_dict(sample_addon_info) + orig_hash = addon_info.hash + + addon_info.hash = "brokenhash" + result = update_addon_state([addon_info], temp_folder, addon_downloader) + assert result["openpype_slack_1.0.0"] == UpdateState.FAILED.value, \ + "Update should failed because of wrong hash" + + addon_info.hash = orig_hash + result = update_addon_state([addon_info], temp_folder, addon_downloader) + assert result["openpype_slack_1.0.0"] == UpdateState.UPDATED.value, \ + "Addon should have been updated" + + result = update_addon_state([addon_info], temp_folder, addon_downloader) + assert result["openpype_slack_1.0.0"] == UpdateState.EXISTS.value, \ + "Addon should already exist" diff --git a/igniter/bootstrap_repos.py b/igniter/bootstrap_repos.py index c5003b062e..ccc9d4ac52 100644 --- a/igniter/bootstrap_repos.py +++ b/igniter/bootstrap_repos.py @@ -63,7 +63,7 @@ class OpenPypeVersion(semver.VersionInfo): """ staging = False path = None - _VERSION_REGEX = re.compile(r"(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?$") # noqa: E501 + _VERSION_REGEX = re.compile(r"(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)\.(?P0|[1-9]\d*)(?:-(?P(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(?:\.(?:0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(?:\+(?P[0-9a-zA-Z-]+(?:\.[0-9a-zA-Z-]+)*))?") # noqa: E501 _installed_version = None def __init__(self, *args, **kwargs): diff --git a/igniter/install_dialog.py b/igniter/install_dialog.py index b09529f5c5..65ddd58735 100644 --- a/igniter/install_dialog.py +++ b/igniter/install_dialog.py @@ -388,8 +388,11 @@ class InstallDialog(QtWidgets.QDialog): install_thread.start() def _installation_finished(self): + # TODO we should find out why status can be set to 'None'? + # - 'InstallThread.run' should handle all cases so not sure where + # that come from status = self._install_thread.result() - if status >= 0: + if status is not None and status >= 0: self._update_progress(100) QtWidgets.QApplication.processEvents() self.done(3) diff --git a/openpype/client/__init__.py b/openpype/client/__init__.py index 4b2d56168f..7831afd8ad 100644 --- a/openpype/client/__init__.py +++ b/openpype/client/__init__.py @@ -45,6 +45,12 @@ from .entities import ( get_workfile_info, ) +from .entity_links import ( + get_linked_asset_ids, + get_linked_assets, + get_linked_representation_id, +) + from .operations import ( create_project, ) @@ -94,5 +100,9 @@ __all__ = ( "get_workfile_info", + "get_linked_asset_ids", + "get_linked_assets", + "get_linked_representation_id", + "create_project", ) diff --git a/openpype/client/entities.py b/openpype/client/entities.py index f9afde0f5d..43afccf2f1 100644 --- a/openpype/client/entities.py +++ b/openpype/client/entities.py @@ -14,6 +14,8 @@ from bson.objectid import ObjectId from .mongo import get_project_database, get_project_connection +PatternType = type(re.compile("")) + def _prepare_fields(fields, required_fields=None): if not fields: @@ -32,17 +34,37 @@ def _prepare_fields(fields, required_fields=None): return output -def _convert_id(in_id): +def convert_id(in_id): + """Helper function for conversion of id from string to ObjectId. + + Args: + in_id (Union[str, ObjectId, Any]): Entity id that should be converted + to right type for queries. + + Returns: + Union[ObjectId, Any]: Converted ids to ObjectId or in type. + """ + if isinstance(in_id, six.string_types): return ObjectId(in_id) return in_id -def _convert_ids(in_ids): +def convert_ids(in_ids): + """Helper function for conversion of ids from string to ObjectId. + + Args: + in_ids (Iterable[Union[str, ObjectId, Any]]): List of entity ids that + should be converted to right type for queries. + + Returns: + List[ObjectId]: Converted ids to ObjectId. + """ + _output = set() for in_id in in_ids: if in_id is not None: - _output.add(_convert_id(in_id)) + _output.add(convert_id(in_id)) return list(_output) @@ -115,7 +137,7 @@ def get_asset_by_id(project_name, asset_id, fields=None): None: Asset was not found by id. """ - asset_id = _convert_id(asset_id) + asset_id = convert_id(asset_id) if not asset_id: return None @@ -196,7 +218,7 @@ def _get_assets( query_filter = {"type": {"$in": asset_types}} if asset_ids is not None: - asset_ids = _convert_ids(asset_ids) + asset_ids = convert_ids(asset_ids) if not asset_ids: return [] query_filter["_id"] = {"$in": asset_ids} @@ -207,7 +229,7 @@ def _get_assets( query_filter["name"] = {"$in": list(asset_names)} if parent_ids is not None: - parent_ids = _convert_ids(parent_ids) + parent_ids = convert_ids(parent_ids) if not parent_ids: return [] query_filter["data.visualParent"] = {"$in": parent_ids} @@ -307,7 +329,7 @@ def get_asset_ids_with_subsets(project_name, asset_ids=None): "type": "subset" } if asset_ids is not None: - asset_ids = _convert_ids(asset_ids) + asset_ids = convert_ids(asset_ids) if not asset_ids: return [] subset_query["parent"] = {"$in": asset_ids} @@ -347,7 +369,7 @@ def get_subset_by_id(project_name, subset_id, fields=None): Dict: Subset document which can be reduced to specified 'fields'. """ - subset_id = _convert_id(subset_id) + subset_id = convert_id(subset_id) if not subset_id: return None @@ -374,7 +396,7 @@ def get_subset_by_name(project_name, subset_name, asset_id, fields=None): if not subset_name: return None - asset_id = _convert_id(asset_id) + asset_id = convert_id(asset_id) if not asset_id: return None @@ -428,13 +450,13 @@ def get_subsets( query_filter = {"type": {"$in": subset_types}} if asset_ids is not None: - asset_ids = _convert_ids(asset_ids) + asset_ids = convert_ids(asset_ids) if not asset_ids: return [] query_filter["parent"] = {"$in": asset_ids} if subset_ids is not None: - subset_ids = _convert_ids(subset_ids) + subset_ids = convert_ids(subset_ids) if not subset_ids: return [] query_filter["_id"] = {"$in": subset_ids} @@ -449,7 +471,7 @@ def get_subsets( for asset_id, names in names_by_asset_ids.items(): if asset_id and names: or_query.append({ - "parent": _convert_id(asset_id), + "parent": convert_id(asset_id), "name": {"$in": list(names)} }) if not or_query: @@ -510,7 +532,7 @@ def get_version_by_id(project_name, version_id, fields=None): Dict: Version document which can be reduced to specified 'fields'. """ - version_id = _convert_id(version_id) + version_id = convert_id(version_id) if not version_id: return None @@ -537,7 +559,7 @@ def get_version_by_name(project_name, version, subset_id, fields=None): Dict: Version document which can be reduced to specified 'fields'. """ - subset_id = _convert_id(subset_id) + subset_id = convert_id(subset_id) if not subset_id: return None @@ -567,7 +589,7 @@ def version_is_latest(project_name, version_id): bool: True if is latest version from subset else False. """ - version_id = _convert_id(version_id) + version_id = convert_id(version_id) if not version_id: return False version_doc = get_version_by_id( @@ -610,13 +632,13 @@ def _get_versions( query_filter = {"type": {"$in": version_types}} if subset_ids is not None: - subset_ids = _convert_ids(subset_ids) + subset_ids = convert_ids(subset_ids) if not subset_ids: return [] query_filter["parent"] = {"$in": subset_ids} if version_ids is not None: - version_ids = _convert_ids(version_ids) + version_ids = convert_ids(version_ids) if not version_ids: return [] query_filter["_id"] = {"$in": version_ids} @@ -690,7 +712,7 @@ def get_hero_version_by_subset_id(project_name, subset_id, fields=None): Dict: Hero version entity data. """ - subset_id = _convert_id(subset_id) + subset_id = convert_id(subset_id) if not subset_id: return None @@ -720,7 +742,7 @@ def get_hero_version_by_id(project_name, version_id, fields=None): Dict: Hero version entity data. """ - version_id = _convert_id(version_id) + version_id = convert_id(version_id) if not version_id: return None @@ -786,7 +808,7 @@ def get_output_link_versions(project_name, version_id, fields=None): links for passed version. """ - version_id = _convert_id(version_id) + version_id = convert_id(version_id) if not version_id: return [] @@ -812,7 +834,7 @@ def get_last_versions(project_name, subset_ids, fields=None): dict[ObjectId, int]: Key is subset id and value is last version name. """ - subset_ids = _convert_ids(subset_ids) + subset_ids = convert_ids(subset_ids) if not subset_ids: return {} @@ -898,7 +920,7 @@ def get_last_version_by_subset_id(project_name, subset_id, fields=None): Dict: Version document which can be reduced to specified 'fields'. """ - subset_id = _convert_id(subset_id) + subset_id = convert_id(subset_id) if not subset_id: return None @@ -971,7 +993,7 @@ def get_representation_by_id(project_name, representation_id, fields=None): "type": {"$in": repre_types} } if representation_id is not None: - query_filter["_id"] = _convert_id(representation_id) + query_filter["_id"] = convert_id(representation_id) conn = get_project_connection(project_name) @@ -996,7 +1018,7 @@ def get_representation_by_name( to specified 'fields'. """ - version_id = _convert_id(version_id) + version_id = convert_id(version_id) if not version_id or not representation_name: return None repre_types = ["representation", "archived_representations"] @@ -1034,11 +1056,11 @@ def _regex_filters(filters): for key, value in filters.items(): regexes = [] a_values = [] - if isinstance(value, re.Pattern): + if isinstance(value, PatternType): regexes.append(value) elif isinstance(value, (list, tuple, set)): for item in value: - if isinstance(item, re.Pattern): + if isinstance(item, PatternType): regexes.append(item) else: a_values.append(item) @@ -1089,7 +1111,7 @@ def _get_representations( query_filter = {"type": {"$in": repre_types}} if representation_ids is not None: - representation_ids = _convert_ids(representation_ids) + representation_ids = convert_ids(representation_ids) if not representation_ids: return default_output query_filter["_id"] = {"$in": representation_ids} @@ -1100,7 +1122,7 @@ def _get_representations( query_filter["name"] = {"$in": list(representation_names)} if version_ids is not None: - version_ids = _convert_ids(version_ids) + version_ids = convert_ids(version_ids) if not version_ids: return default_output query_filter["parent"] = {"$in": version_ids} @@ -1111,7 +1133,7 @@ def _get_representations( for version_id, names in names_by_version_ids.items(): if version_id and names: or_query.append({ - "parent": _convert_id(version_id), + "parent": convert_id(version_id), "name": {"$in": list(names)} }) if not or_query: @@ -1174,7 +1196,7 @@ def get_representations( as filter. Filter ignored if 'None' is passed. version_ids (Iterable[str]): Subset ids used as parent filter. Filter ignored if 'None' is passed. - context_filters (Dict[str, List[str, re.Pattern]]): Filter by + context_filters (Dict[str, List[str, PatternType]]): Filter by representation context fields. names_by_version_ids (dict[ObjectId, list[str]]): Complex filtering using version ids and list of names under the version. @@ -1220,7 +1242,7 @@ def get_archived_representations( as filter. Filter ignored if 'None' is passed. version_ids (Iterable[str]): Subset ids used as parent filter. Filter ignored if 'None' is passed. - context_filters (Dict[str, List[str, re.Pattern]]): Filter by + context_filters (Dict[str, List[str, PatternType]]): Filter by representation context fields. names_by_version_ids (dict[ObjectId, List[str]]): Complex filtering using version ids and list of names under the version. @@ -1361,7 +1383,7 @@ def get_thumbnail_id_from_source(project_name, src_type, src_id): if not src_type or not src_id: return None - query_filter = {"_id": _convert_id(src_id)} + query_filter = {"_id": convert_id(src_id)} conn = get_project_connection(project_name) src_doc = conn.find_one(query_filter, {"data.thumbnail_id"}) @@ -1388,7 +1410,7 @@ def get_thumbnails(project_name, thumbnail_ids, fields=None): """ if thumbnail_ids: - thumbnail_ids = _convert_ids(thumbnail_ids) + thumbnail_ids = convert_ids(thumbnail_ids) if not thumbnail_ids: return [] @@ -1416,7 +1438,7 @@ def get_thumbnail(project_name, thumbnail_id, fields=None): if not thumbnail_id: return None - query_filter = {"type": "thumbnail", "_id": _convert_id(thumbnail_id)} + query_filter = {"type": "thumbnail", "_id": convert_id(thumbnail_id)} conn = get_project_connection(project_name) return conn.find_one(query_filter, _prepare_fields(fields)) @@ -1444,7 +1466,7 @@ def get_workfile_info( query_filter = { "type": "workfile", - "parent": _convert_id(asset_id), + "parent": convert_id(asset_id), "task_name": task_name, "filename": filename } diff --git a/openpype/client/entity_links.py b/openpype/client/entity_links.py new file mode 100644 index 0000000000..66214f469c --- /dev/null +++ b/openpype/client/entity_links.py @@ -0,0 +1,232 @@ +from .mongo import get_project_connection +from .entities import ( + get_assets, + get_asset_by_id, + get_representation_by_id, + convert_id, +) + + +def get_linked_asset_ids(project_name, asset_doc=None, asset_id=None): + """Extract linked asset ids from asset document. + + One of asset document or asset id must be passed. + + Note: + Asset links now works only from asset to assets. + + Args: + asset_doc (dict): Asset document from DB. + + Returns: + List[Union[ObjectId, str]]: Asset ids of input links. + """ + + output = [] + if not asset_doc and not asset_id: + return output + + if not asset_doc: + asset_doc = get_asset_by_id( + project_name, asset_id, fields=["data.inputLinks"] + ) + + input_links = asset_doc["data"].get("inputLinks") + if not input_links: + return output + + for item in input_links: + # Backwards compatibility for "_id" key which was replaced with + # "id" + if "_id" in item: + link_id = item["_id"] + else: + link_id = item["id"] + output.append(link_id) + return output + + +def get_linked_assets( + project_name, asset_doc=None, asset_id=None, fields=None +): + """Return linked assets based on passed asset document. + + One of asset document or asset id must be passed. + + Args: + project_name (str): Name of project where to look for queried entities. + asset_doc (Dict[str, Any]): Asset document from database. + asset_id (Union[ObjectId, str]): Asset id. Can be used instead of + asset document. + fields (Iterable[str]): Fields that should be returned. All fields are + returned if 'None' is passed. + + Returns: + List[Dict[str, Any]]: Asset documents of input links for passed + asset doc. + """ + + if not asset_doc: + if not asset_id: + return [] + asset_doc = get_asset_by_id( + project_name, + asset_id, + fields=["data.inputLinks"] + ) + if not asset_doc: + return [] + + link_ids = get_linked_asset_ids(project_name, asset_doc=asset_doc) + if not link_ids: + return [] + + return list(get_assets(project_name, asset_ids=link_ids, fields=fields)) + + +def get_linked_representation_id( + project_name, repre_doc=None, repre_id=None, link_type=None, max_depth=None +): + """Returns list of linked ids of particular type (if provided). + + One of representation document or representation id must be passed. + Note: + Representation links now works only from representation through version + back to representations. + + Args: + project_name (str): Name of project where look for links. + repre_doc (Dict[str, Any]): Representation document. + repre_id (Union[ObjectId, str]): Representation id. + link_type (str): Type of link (e.g. 'reference', ...). + max_depth (int): Limit recursion level. Default: 0 + + Returns: + List[ObjectId] Linked representation ids. + """ + + if repre_doc: + repre_id = repre_doc["_id"] + + if repre_id: + repre_id = convert_id(repre_id) + + if not repre_id and not repre_doc: + return [] + + version_id = None + if repre_doc: + version_id = repre_doc.get("parent") + + if not version_id: + repre_doc = get_representation_by_id( + project_name, repre_id, fields=["parent"] + ) + version_id = repre_doc["parent"] + + if not version_id: + return [] + + if max_depth is None: + max_depth = 0 + + match = { + "_id": version_id, + "type": {"$in": ["version", "hero_version"]} + } + + graph_lookup = { + "from": project_name, + "startWith": "$data.inputLinks.id", + "connectFromField": "data.inputLinks.id", + "connectToField": "_id", + "as": "outputs_recursive", + "depthField": "depth" + } + if max_depth != 0: + # We offset by -1 since 0 basically means no recursion + # but the recursion only happens after the initial lookup + # for outputs. + graph_lookup["maxDepth"] = max_depth - 1 + + query_pipeline = [ + # Match + {"$match": match}, + # Recursive graph lookup for inputs + {"$graphLookup": graph_lookup} + ] + + conn = get_project_connection(project_name) + result = conn.aggregate(query_pipeline) + referenced_version_ids = _process_referenced_pipeline_result( + result, link_type + ) + if not referenced_version_ids: + return [] + + ref_ids = conn.distinct( + "_id", + filter={ + "parent": {"$in": list(referenced_version_ids)}, + "type": "representation" + } + ) + + return list(ref_ids) + + +def _process_referenced_pipeline_result(result, link_type): + """Filters result from pipeline for particular link_type. + + Pipeline cannot use link_type directly in a query. + + Returns: + (list) + """ + + referenced_version_ids = set() + correctly_linked_ids = set() + for item in result: + input_links = item["data"].get("inputLinks") + if not input_links: + continue + + _filter_input_links( + input_links, + link_type, + correctly_linked_ids + ) + + # outputs_recursive in random order, sort by depth + outputs_recursive = item.get("outputs_recursive") + if not outputs_recursive: + continue + + for output in sorted(outputs_recursive, key=lambda o: o["depth"]): + output_links = output["data"].get("inputLinks") + if not output_links: + continue + + # Leaf + if output["_id"] not in correctly_linked_ids: + continue + + _filter_input_links( + output_links, + link_type, + correctly_linked_ids + ) + + referenced_version_ids.add(output["_id"]) + + return referenced_version_ids + + +def _filter_input_links(input_links, link_type, correctly_linked_ids): + for input_link in input_links: + if link_type and input_link["type"] != link_type: + continue + + link_id = input_link.get("id") or input_link.get("_id") + if link_id is not None: + correctly_linked_ids.add(link_id) diff --git a/openpype/hooks/pre_add_last_workfile_arg.py b/openpype/hooks/pre_add_last_workfile_arg.py index 8edccd48d4..3609620917 100644 --- a/openpype/hooks/pre_add_last_workfile_arg.py +++ b/openpype/hooks/pre_add_last_workfile_arg.py @@ -19,6 +19,7 @@ class AddLastWorkfileToLaunchArgs(PreLaunchHook): "hiero", "houdini", "nukestudio", + "fusion", "blender", "photoshop", "tvpaint", diff --git a/openpype/hooks/pre_create_extra_workdir_folders.py b/openpype/hooks/pre_create_extra_workdir_folders.py index d79c5831ee..c5af620c87 100644 --- a/openpype/hooks/pre_create_extra_workdir_folders.py +++ b/openpype/hooks/pre_create_extra_workdir_folders.py @@ -1,8 +1,6 @@ import os -from openpype.lib import ( - PreLaunchHook, - create_workdir_extra_folders -) +from openpype.lib import PreLaunchHook +from openpype.pipeline.workfile import create_workdir_extra_folders class AddLastWorkfileToLaunchArgs(PreLaunchHook): diff --git a/openpype/host/__init__.py b/openpype/host/__init__.py index 519888fce3..da1237c739 100644 --- a/openpype/host/__init__.py +++ b/openpype/host/__init__.py @@ -5,6 +5,7 @@ from .host import ( from .interfaces import ( IWorkfileHost, ILoadHost, + IPublishHost, INewPublisher, ) @@ -16,6 +17,7 @@ __all__ = ( "IWorkfileHost", "ILoadHost", + "IPublishHost", "INewPublisher", "HostDirmap", diff --git a/openpype/host/interfaces.py b/openpype/host/interfaces.py index cbf12b0d13..cfd089a0ad 100644 --- a/openpype/host/interfaces.py +++ b/openpype/host/interfaces.py @@ -282,7 +282,7 @@ class IWorkfileHost: return self.workfile_has_unsaved_changes() -class INewPublisher: +class IPublishHost: """Functions related to new creation system in new publisher. New publisher is not storing information only about each created instance @@ -306,7 +306,7 @@ class INewPublisher: workflow. """ - if isinstance(host, INewPublisher): + if isinstance(host, IPublishHost): return [] required = [ @@ -330,7 +330,7 @@ class INewPublisher: MissingMethodsError: If there are missing methods on host implementation. """ - missing = INewPublisher.get_missing_publish_methods(host) + missing = IPublishHost.get_missing_publish_methods(host) if missing: raise MissingMethodsError(host, missing) @@ -368,3 +368,17 @@ class INewPublisher: """ pass + + +class INewPublisher(IPublishHost): + """Legacy interface replaced by 'IPublishHost'. + + Deprecated: + 'INewPublisher' is replaced by 'IPublishHost' please change your + imports. + There is no "reasonable" way hot mark these classes as deprecated + to show warning of wrong import. Deprecated since 3.14.* will be + removed in 3.15.* + """ + + pass diff --git a/openpype/hosts/aftereffects/plugins/publish/extract_local_render.py b/openpype/hosts/aftereffects/plugins/publish/extract_local_render.py index 7323a0b125..dc65cee61d 100644 --- a/openpype/hosts/aftereffects/plugins/publish/extract_local_render.py +++ b/openpype/hosts/aftereffects/plugins/publish/extract_local_render.py @@ -2,14 +2,18 @@ import os import sys import six -import openpype.api +from openpype.lib import ( + get_ffmpeg_tool_path, + run_subprocess, +) +from openpype.pipeline import publish from openpype.hosts.aftereffects.api import get_stub -class ExtractLocalRender(openpype.api.Extractor): +class ExtractLocalRender(publish.Extractor): """Render RenderQueue locally.""" - order = openpype.api.Extractor.order - 0.47 + order = publish.Extractor.order - 0.47 label = "Extract Local Render" hosts = ["aftereffects"] families = ["renderLocal", "render.local"] @@ -53,7 +57,7 @@ class ExtractLocalRender(openpype.api.Extractor): instance.data["representations"] = [repre_data] - ffmpeg_path = openpype.lib.get_ffmpeg_tool_path("ffmpeg") + ffmpeg_path = get_ffmpeg_tool_path("ffmpeg") # Generate thumbnail. thumbnail_path = os.path.join(staging_dir, "thumbnail.jpg") @@ -66,7 +70,7 @@ class ExtractLocalRender(openpype.api.Extractor): ] self.log.debug("Thumbnail args:: {}".format(args)) try: - output = openpype.lib.run_subprocess(args) + output = run_subprocess(args) except TypeError: self.log.warning("Error in creating thumbnail") six.reraise(*sys.exc_info()) diff --git a/openpype/hosts/aftereffects/plugins/publish/extract_save_scene.py b/openpype/hosts/aftereffects/plugins/publish/extract_save_scene.py index eb2977309f..343838eb49 100644 --- a/openpype/hosts/aftereffects/plugins/publish/extract_save_scene.py +++ b/openpype/hosts/aftereffects/plugins/publish/extract_save_scene.py @@ -1,13 +1,13 @@ import pyblish.api -import openpype.api +from openpype.pipeline import publish from openpype.hosts.aftereffects.api import get_stub class ExtractSaveScene(pyblish.api.ContextPlugin): """Save scene before extraction.""" - order = openpype.api.Extractor.order - 0.48 + order = publish.Extractor.order - 0.48 label = "Extract Save Scene" hosts = ["aftereffects"] diff --git a/openpype/hosts/aftereffects/plugins/publish/remove_publish_highlight.py b/openpype/hosts/aftereffects/plugins/publish/remove_publish_highlight.py index 5f3fcc3089..370f916f04 100644 --- a/openpype/hosts/aftereffects/plugins/publish/remove_publish_highlight.py +++ b/openpype/hosts/aftereffects/plugins/publish/remove_publish_highlight.py @@ -1,8 +1,8 @@ -import openpype.api +from openpype.pipeline import publish from openpype.hosts.aftereffects.api import get_stub -class RemovePublishHighlight(openpype.api.Extractor): +class RemovePublishHighlight(publish.Extractor): """Clean utf characters which are not working in DL Published compositions are marked with unicode icon which causes @@ -10,7 +10,7 @@ class RemovePublishHighlight(openpype.api.Extractor): rendering, add it later back to avoid confusion. """ - order = openpype.api.Extractor.order - 0.49 # just before save + order = publish.Extractor.order - 0.49 # just before save label = "Clean render comp" hosts = ["aftereffects"] families = ["render.farm"] diff --git a/openpype/hosts/blender/plugins/publish/collect_current_file.py b/openpype/hosts/blender/plugins/publish/collect_current_file.py index 72976c490b..c3097a0694 100644 --- a/openpype/hosts/blender/plugins/publish/collect_current_file.py +++ b/openpype/hosts/blender/plugins/publish/collect_current_file.py @@ -1,6 +1,19 @@ +import os import bpy import pyblish.api +from openpype.pipeline import legacy_io +from openpype.hosts.blender.api import workio + + +class SaveWorkfiledAction(pyblish.api.Action): + """Save Workfile.""" + label = "Save Workfile" + on = "failed" + icon = "save" + + def process(self, context, plugin): + bpy.ops.wm.avalon_workfiles() class CollectBlenderCurrentFile(pyblish.api.ContextPlugin): @@ -8,12 +21,52 @@ class CollectBlenderCurrentFile(pyblish.api.ContextPlugin): order = pyblish.api.CollectorOrder - 0.5 label = "Blender Current File" - hosts = ['blender'] + hosts = ["blender"] + actions = [SaveWorkfiledAction] def process(self, context): """Inject the current working file""" - current_file = bpy.data.filepath - context.data['currentFile'] = current_file + current_file = workio.current_file() - assert current_file != '', "Current file is empty. " \ - "Save the file before continuing." + context.data["currentFile"] = current_file + + assert current_file, ( + "Current file is empty. Save the file before continuing." + ) + + folder, file = os.path.split(current_file) + filename, ext = os.path.splitext(file) + + task = legacy_io.Session["AVALON_TASK"] + + data = {} + + # create instance + instance = context.create_instance(name=filename) + subset = "workfile" + task.capitalize() + + data.update({ + "subset": subset, + "asset": os.getenv("AVALON_ASSET", None), + "label": subset, + "publish": True, + "family": "workfile", + "families": ["workfile"], + "setMembers": [current_file], + "frameStart": bpy.context.scene.frame_start, + "frameEnd": bpy.context.scene.frame_end, + }) + + data["representations"] = [{ + "name": ext.lstrip("."), + "ext": ext.lstrip("."), + "files": file, + "stagingDir": folder, + }] + + instance.data.update(data) + + self.log.info("Collected instance: {}".format(file)) + self.log.info("Scene path: {}".format(current_file)) + self.log.info("staging Dir: {}".format(folder)) + self.log.info("subset: {}".format(subset)) diff --git a/openpype/hosts/blender/plugins/publish/extract_abc.py b/openpype/hosts/blender/plugins/publish/extract_abc.py index a26a92f7e4..1cab9d225b 100644 --- a/openpype/hosts/blender/plugins/publish/extract_abc.py +++ b/openpype/hosts/blender/plugins/publish/extract_abc.py @@ -2,12 +2,12 @@ import os import bpy -from openpype import api +from openpype.pipeline import publish from openpype.hosts.blender.api import plugin from openpype.hosts.blender.api.pipeline import AVALON_PROPERTY -class ExtractABC(api.Extractor): +class ExtractABC(publish.Extractor): """Extract as ABC.""" label = "Extract ABC" diff --git a/openpype/hosts/blender/plugins/publish/extract_blend.py b/openpype/hosts/blender/plugins/publish/extract_blend.py index 9add633f05..6a001b6f65 100644 --- a/openpype/hosts/blender/plugins/publish/extract_blend.py +++ b/openpype/hosts/blender/plugins/publish/extract_blend.py @@ -2,10 +2,10 @@ import os import bpy -import openpype.api +from openpype.pipeline import publish -class ExtractBlend(openpype.api.Extractor): +class ExtractBlend(publish.Extractor): """Extract a blend file.""" label = "Extract Blend" diff --git a/openpype/hosts/blender/plugins/publish/extract_blend_animation.py b/openpype/hosts/blender/plugins/publish/extract_blend_animation.py index 4917223331..477411b73d 100644 --- a/openpype/hosts/blender/plugins/publish/extract_blend_animation.py +++ b/openpype/hosts/blender/plugins/publish/extract_blend_animation.py @@ -2,10 +2,10 @@ import os import bpy -import openpype.api +from openpype.pipeline import publish -class ExtractBlendAnimation(openpype.api.Extractor): +class ExtractBlendAnimation(publish.Extractor): """Extract a blend file.""" label = "Extract Blend" diff --git a/openpype/hosts/blender/plugins/publish/extract_camera.py b/openpype/hosts/blender/plugins/publish/extract_camera.py index b2c7611b58..9fd181825c 100644 --- a/openpype/hosts/blender/plugins/publish/extract_camera.py +++ b/openpype/hosts/blender/plugins/publish/extract_camera.py @@ -2,11 +2,11 @@ import os import bpy -from openpype import api +from openpype.pipeline import publish from openpype.hosts.blender.api import plugin -class ExtractCamera(api.Extractor): +class ExtractCamera(publish.Extractor): """Extract as the camera as FBX.""" label = "Extract Camera" diff --git a/openpype/hosts/blender/plugins/publish/extract_fbx.py b/openpype/hosts/blender/plugins/publish/extract_fbx.py index 3ac66f33a4..0ad797c226 100644 --- a/openpype/hosts/blender/plugins/publish/extract_fbx.py +++ b/openpype/hosts/blender/plugins/publish/extract_fbx.py @@ -2,12 +2,12 @@ import os import bpy -from openpype import api +from openpype.pipeline import publish from openpype.hosts.blender.api import plugin from openpype.hosts.blender.api.pipeline import AVALON_PROPERTY -class ExtractFBX(api.Extractor): +class ExtractFBX(publish.Extractor): """Extract as FBX.""" label = "Extract FBX" diff --git a/openpype/hosts/blender/plugins/publish/extract_fbx_animation.py b/openpype/hosts/blender/plugins/publish/extract_fbx_animation.py index 4b4a92932a..062b42e99d 100644 --- a/openpype/hosts/blender/plugins/publish/extract_fbx_animation.py +++ b/openpype/hosts/blender/plugins/publish/extract_fbx_animation.py @@ -5,12 +5,12 @@ import bpy import bpy_extras import bpy_extras.anim_utils -from openpype import api +from openpype.pipeline import publish from openpype.hosts.blender.api import plugin from openpype.hosts.blender.api.pipeline import AVALON_PROPERTY -class ExtractAnimationFBX(api.Extractor): +class ExtractAnimationFBX(publish.Extractor): """Extract as animation.""" label = "Extract FBX" diff --git a/openpype/hosts/blender/plugins/publish/extract_layout.py b/openpype/hosts/blender/plugins/publish/extract_layout.py index 8502c6fbd4..f2d04f1178 100644 --- a/openpype/hosts/blender/plugins/publish/extract_layout.py +++ b/openpype/hosts/blender/plugins/publish/extract_layout.py @@ -6,12 +6,12 @@ import bpy_extras import bpy_extras.anim_utils from openpype.client import get_representation_by_name +from openpype.pipeline import publish from openpype.hosts.blender.api import plugin from openpype.hosts.blender.api.pipeline import AVALON_PROPERTY -import openpype.api -class ExtractLayout(openpype.api.Extractor): +class ExtractLayout(publish.Extractor): """Extract a layout.""" label = "Extract Layout" diff --git a/openpype/hosts/celaction/plugins/publish/collect_audio.py b/openpype/hosts/celaction/plugins/publish/collect_audio.py deleted file mode 100644 index c6e3bf2c03..0000000000 --- a/openpype/hosts/celaction/plugins/publish/collect_audio.py +++ /dev/null @@ -1,113 +0,0 @@ -import os -import collections -from pprint import pformat - -import pyblish.api - -from openpype.client import ( - get_subsets, - get_last_versions, - get_representations -) -from openpype.pipeline import legacy_io - - -class AppendCelactionAudio(pyblish.api.ContextPlugin): - - label = "Colect Audio for publishing" - order = pyblish.api.CollectorOrder + 0.1 - - def process(self, context): - self.log.info('Collecting Audio Data') - asset_doc = context.data["assetEntity"] - - # get all available representations - subsets = self.get_subsets( - asset_doc, - representations=["audio", "wav"] - ) - self.log.info(f"subsets is: {pformat(subsets)}") - - if not subsets.get("audioMain"): - raise AttributeError("`audioMain` subset does not exist") - - reprs = subsets.get("audioMain", {}).get("representations", []) - self.log.info(f"reprs is: {pformat(reprs)}") - - repr = next((r for r in reprs), None) - if not repr: - raise "Missing `audioMain` representation" - self.log.info(f"representation is: {repr}") - - audio_file = repr.get('data', {}).get('path', "") - - if os.path.exists(audio_file): - context.data["audioFile"] = audio_file - self.log.info( - 'audio_file: {}, has been added to context'.format(audio_file)) - else: - self.log.warning("Couldn't find any audio file on Ftrack.") - - def get_subsets(self, asset_doc, representations): - """ - Query subsets with filter on name. - - The method will return all found subsets and its defined version - and subsets. Version could be specified with number. Representation - can be filtered. - - Arguments: - asset_doct (dict): Asset (shot) mongo document - representations (list): list for all representations - - Returns: - dict: subsets with version and representations in keys - """ - - # Query all subsets for asset - project_name = legacy_io.active_project() - subset_docs = get_subsets( - project_name, asset_ids=[asset_doc["_id"]], fields=["_id"] - ) - # Collect all subset ids - subset_ids = [ - subset_doc["_id"] - for subset_doc in subset_docs - ] - - # Check if we found anything - assert subset_ids, ( - "No subsets found. Check correct filter. " - "Try this for start `r'.*'`: asset: `{}`" - ).format(asset_doc["name"]) - - last_versions_by_subset_id = get_last_versions( - project_name, subset_ids, fields=["_id", "parent"] - ) - - version_docs_by_id = {} - for version_doc in last_versions_by_subset_id.values(): - version_docs_by_id[version_doc["_id"]] = version_doc - - repre_docs = get_representations( - project_name, - version_ids=version_docs_by_id.keys(), - representation_names=representations - ) - repre_docs_by_version_id = collections.defaultdict(list) - for repre_doc in repre_docs: - version_id = repre_doc["parent"] - repre_docs_by_version_id[version_id].append(repre_doc) - - output_dict = {} - for version_id, repre_docs in repre_docs_by_version_id.items(): - version_doc = version_docs_by_id[version_id] - subset_id = version_doc["parent"] - subset_doc = last_versions_by_subset_id[subset_id] - # Store queried docs by subset name - output_dict[subset_doc["name"]] = { - "representations": repre_docs, - "version": version_doc - } - - return output_dict diff --git a/openpype/hosts/flame/api/__init__.py b/openpype/hosts/flame/api/__init__.py index 76c1c93379..c00ee958b6 100644 --- a/openpype/hosts/flame/api/__init__.py +++ b/openpype/hosts/flame/api/__init__.py @@ -51,7 +51,8 @@ from .pipeline import ( ) from .menu import ( FlameMenuProjectConnect, - FlameMenuTimeline + FlameMenuTimeline, + FlameMenuUniversal ) from .plugin import ( Creator, @@ -131,6 +132,7 @@ __all__ = [ # menu "FlameMenuProjectConnect", "FlameMenuTimeline", + "FlameMenuUniversal", # plugin "Creator", diff --git a/openpype/hosts/flame/api/lib.py b/openpype/hosts/flame/api/lib.py index 94c46fe937..b7f7b24e51 100644 --- a/openpype/hosts/flame/api/lib.py +++ b/openpype/hosts/flame/api/lib.py @@ -766,11 +766,11 @@ class MediaInfoFile(object): _drop_mode = None _file_pattern = None - def __init__(self, path, **kwargs): + def __init__(self, path, logger=None): # replace log if any - if kwargs.get("logger"): - self.log = kwargs["logger"] + if logger: + self.log = logger # test if `dl_get_media_info` paht exists self._validate_media_script_path() diff --git a/openpype/hosts/flame/api/menu.py b/openpype/hosts/flame/api/menu.py index 7f1a6a24e2..f72a352bba 100644 --- a/openpype/hosts/flame/api/menu.py +++ b/openpype/hosts/flame/api/menu.py @@ -201,3 +201,53 @@ class FlameMenuTimeline(_FlameMenuApp): if self.flame: self.flame.execute_shortcut('Rescan Python Hooks') self.log.info('Rescan Python Hooks') + + +class FlameMenuUniversal(_FlameMenuApp): + + # flameMenuProjectconnect app takes care of the preferences dialog as well + + def __init__(self, framework): + _FlameMenuApp.__init__(self, framework) + + def __getattr__(self, name): + def method(*args, **kwargs): + project = self.dynamic_menu_data.get(name) + if project: + self.link_project(project) + return method + + def build_menu(self): + if not self.flame: + return [] + + menu = deepcopy(self.menu) + + menu['actions'].append({ + "name": "Load...", + "execute": lambda x: self.tools_helper.show_loader() + }) + menu['actions'].append({ + "name": "Manage...", + "execute": lambda x: self.tools_helper.show_scene_inventory() + }) + menu['actions'].append({ + "name": "Library...", + "execute": lambda x: self.tools_helper.show_library_loader() + }) + return menu + + def refresh(self, *args, **kwargs): + self.rescan() + + def rescan(self, *args, **kwargs): + if not self.flame: + try: + import flame + self.flame = flame + except ImportError: + self.flame = None + + if self.flame: + self.flame.execute_shortcut('Rescan Python Hooks') + self.log.info('Rescan Python Hooks') diff --git a/openpype/hosts/flame/api/pipeline.py b/openpype/hosts/flame/api/pipeline.py index da44be1b15..324d13bc3f 100644 --- a/openpype/hosts/flame/api/pipeline.py +++ b/openpype/hosts/flame/api/pipeline.py @@ -90,8 +90,7 @@ def containerise(flame_clip_segment, def ls(): """List available containers. """ - # TODO: ls - pass + return [] def parse_container(tl_segment, validate=True): @@ -107,6 +106,7 @@ def update_container(tl_segment, data=None): # TODO: update_container pass + def on_pyblish_instance_toggled(instance, old_value, new_value): """Toggle node passthrough states on instance toggles.""" diff --git a/openpype/hosts/flame/api/plugin.py b/openpype/hosts/flame/api/plugin.py index efbabb6a55..1a26e96c79 100644 --- a/openpype/hosts/flame/api/plugin.py +++ b/openpype/hosts/flame/api/plugin.py @@ -361,6 +361,8 @@ class PublishableClip: index_from_segment_default = False use_shot_name_default = False include_handles_default = False + retimed_handles_default = True + retimed_framerange_default = True def __init__(self, segment, **kwargs): self.rename_index = kwargs["rename_index"] @@ -496,6 +498,14 @@ class PublishableClip: "audio", {}).get("value") or False self.include_handles = self.ui_inputs.get( "includeHandles", {}).get("value") or self.include_handles_default + self.retimed_handles = ( + self.ui_inputs.get("retimedHandles", {}).get("value") + or self.retimed_handles_default + ) + self.retimed_framerange = ( + self.ui_inputs.get("retimedFramerange", {}).get("value") + or self.retimed_framerange_default + ) # build subset name from layer name if self.subset_name == "[ track name ]": @@ -668,6 +678,7 @@ class ClipLoader(LoaderPlugin): `update` logic. """ + log = log options = [ qargparse.Boolean( @@ -684,16 +695,20 @@ class OpenClipSolver(flib.MediaInfoFile): log = log - def __init__(self, openclip_file_path, feed_data): + def __init__(self, openclip_file_path, feed_data, logger=None): self.out_file = openclip_file_path + # replace log if any + if logger: + self.log = logger + # new feed variables: feed_path = feed_data.pop("path") # initialize parent class super(OpenClipSolver, self).__init__( feed_path, - **feed_data + logger=logger ) # get other metadata @@ -741,17 +756,18 @@ class OpenClipSolver(flib.MediaInfoFile): self.log.info("Building new openClip") self.log.debug(">> self.clip_data: {}".format(self.clip_data)) - # clip data comming from MediaInfoFile - tmp_xml_feeds = self.clip_data.find('tracks/track/feeds') - tmp_xml_feeds.set('currentVersion', self.feed_version_name) - for tmp_feed in tmp_xml_feeds: - tmp_feed.set('vuid', self.feed_version_name) + for tmp_xml_track in self.clip_data.iter("track"): + tmp_xml_feeds = tmp_xml_track.find('feeds') + tmp_xml_feeds.set('currentVersion', self.feed_version_name) - # add colorspace if any is set - if self.feed_colorspace: - self._add_colorspace(tmp_feed, self.feed_colorspace) + for tmp_feed in tmp_xml_track.iter("feed"): + tmp_feed.set('vuid', self.feed_version_name) - self._clear_handler(tmp_feed) + # add colorspace if any is set + if self.feed_colorspace: + self._add_colorspace(tmp_feed, self.feed_colorspace) + + self._clear_handler(tmp_feed) tmp_xml_versions_obj = self.clip_data.find('versions') tmp_xml_versions_obj.set('currentVersion', self.feed_version_name) @@ -764,6 +780,17 @@ class OpenClipSolver(flib.MediaInfoFile): self.write_clip_data_to_file(self.out_file, self.clip_data) + def _get_xml_track_obj_by_uid(self, xml_data, uid): + # loop all tracks of input xml data + for xml_track in xml_data.iter("track"): + track_uid = xml_track.get("uid") + self.log.debug( + ">> track_uid:uid: {}:{}".format(track_uid, uid)) + + # get matching uids + if uid == track_uid: + return xml_track + def _update_open_clip(self): self.log.info("Updating openClip ..") @@ -773,52 +800,81 @@ class OpenClipSolver(flib.MediaInfoFile): self.log.debug(">> out_xml: {}".format(out_xml)) self.log.debug(">> self.clip_data: {}".format(self.clip_data)) - # Get new feed from tmp file - tmp_xml_feed = self.clip_data.find('tracks/track/feeds/feed') + # loop tmp tracks + updated_any = False + for tmp_xml_track in self.clip_data.iter("track"): + # get tmp track uid + tmp_track_uid = tmp_xml_track.get("uid") + self.log.debug(">> tmp_track_uid: {}".format(tmp_track_uid)) - self._clear_handler(tmp_xml_feed) + # get out data track by uid + out_track_element = self._get_xml_track_obj_by_uid( + out_xml, tmp_track_uid) + self.log.debug( + ">> out_track_element: {}".format(out_track_element)) - # update fps from MediaInfoFile class - if self.fps: - tmp_feed_fps_obj = tmp_xml_feed.find( - "startTimecode/rate") - tmp_feed_fps_obj.text = str(self.fps) + # loop tmp feeds + for tmp_xml_feed in tmp_xml_track.iter("feed"): + new_path_obj = tmp_xml_feed.find( + "spans/span/path") + new_path = new_path_obj.text - # update start_frame from MediaInfoFile class - if self.start_frame: - tmp_feed_nb_ticks_obj = tmp_xml_feed.find( - "startTimecode/nbTicks") - tmp_feed_nb_ticks_obj.text = str(self.start_frame) + # check if feed path already exists in track's feeds + if ( + out_track_element is not None + and self._feed_exists(out_track_element, new_path) + ): + continue - # update drop_mode from MediaInfoFile class - if self.drop_mode: - tmp_feed_drop_mode_obj = tmp_xml_feed.find( - "startTimecode/dropMode") - tmp_feed_drop_mode_obj.text = str(self.drop_mode) + # rename versions on feeds + tmp_xml_feed.set('vuid', self.feed_version_name) + self._clear_handler(tmp_xml_feed) - new_path_obj = tmp_xml_feed.find( - "spans/span/path") - new_path = new_path_obj.text + # update fps from MediaInfoFile class + if self.fps is not None: + tmp_feed_fps_obj = tmp_xml_feed.find( + "startTimecode/rate") + tmp_feed_fps_obj.text = str(self.fps) - feed_added = False - if not self._feed_exists(out_xml, new_path): - tmp_xml_feed.set('vuid', self.feed_version_name) - # Append new temp file feed to .clip source out xml - out_track = out_xml.find("tracks/track") - # add colorspace if any is set - if self.feed_colorspace: - self._add_colorspace(tmp_xml_feed, self.feed_colorspace) + # update start_frame from MediaInfoFile class + if self.start_frame is not None: + tmp_feed_nb_ticks_obj = tmp_xml_feed.find( + "startTimecode/nbTicks") + tmp_feed_nb_ticks_obj.text = str(self.start_frame) - out_feeds = out_track.find('feeds') - out_feeds.set('currentVersion', self.feed_version_name) - out_feeds.append(tmp_xml_feed) + # update drop_mode from MediaInfoFile class + if self.drop_mode is not None: + tmp_feed_drop_mode_obj = tmp_xml_feed.find( + "startTimecode/dropMode") + tmp_feed_drop_mode_obj.text = str(self.drop_mode) - self.log.info( - "Appending new feed: {}".format( - self.feed_version_name)) - feed_added = True + # add colorspace if any is set + if self.feed_colorspace is not None: + self._add_colorspace(tmp_xml_feed, self.feed_colorspace) - if feed_added: + # then append/update feed to correct track in output + if out_track_element: + self.log.debug("updating track element ..") + # update already present track + out_feeds = out_track_element.find('feeds') + out_feeds.set('currentVersion', self.feed_version_name) + out_feeds.append(tmp_xml_feed) + + self.log.info( + "Appending new feed: {}".format( + self.feed_version_name)) + else: + self.log.debug("adding new track element ..") + # create new track as it doesnt exists yet + # set current version to feeds on tmp + tmp_xml_feeds = tmp_xml_track.find('feeds') + tmp_xml_feeds.set('currentVersion', self.feed_version_name) + out_tracks = out_xml.find("tracks") + out_tracks.append(tmp_xml_track) + + updated_any = True + + if updated_any: # Append vUID to versions out_xml_versions_obj = out_xml.find('versions') out_xml_versions_obj.set( diff --git a/openpype/hosts/flame/hooks/pre_flame_setup.py b/openpype/hosts/flame/hooks/pre_flame_setup.py index ad2b0dc897..0173eb8e3b 100644 --- a/openpype/hosts/flame/hooks/pre_flame_setup.py +++ b/openpype/hosts/flame/hooks/pre_flame_setup.py @@ -22,6 +22,7 @@ class FlamePrelaunch(PreLaunchHook): in environment var FLAME_SCRIPT_DIR. """ app_groups = ["flame"] + permissions = 0o777 wtc_script_path = os.path.join( opflame.HOST_DIR, "api", "scripts", "wiretap_com.py") @@ -38,6 +39,7 @@ class FlamePrelaunch(PreLaunchHook): """Hook entry method.""" project_doc = self.data["project_doc"] project_name = project_doc["name"] + volume_name = _env.get("FLAME_WIRETAP_VOLUME") # get image io project_anatomy = self.data["anatomy"] @@ -81,7 +83,7 @@ class FlamePrelaunch(PreLaunchHook): data_to_script = { # from settings "host_name": _env.get("FLAME_WIRETAP_HOSTNAME") or hostname, - "volume_name": _env.get("FLAME_WIRETAP_VOLUME"), + "volume_name": volume_name, "group_name": _env.get("FLAME_WIRETAP_GROUP"), "color_policy": str(imageio_flame["project"]["colourPolicy"]), @@ -99,8 +101,41 @@ class FlamePrelaunch(PreLaunchHook): app_arguments = self._get_launch_arguments(data_to_script) + # fix project data permission issue + self._fix_permissions(project_name, volume_name) + self.launch_context.launch_args.extend(app_arguments) + def _fix_permissions(self, project_name, volume_name): + """Work around for project data permissions + + Reported issue: when project is created locally on one machine, + it is impossible to migrate it to other machine. Autodesk Flame + is crating some unmanagable files which needs to be opened to 0o777. + + Args: + project_name (str): project name + volume_name (str): studio volume + """ + dirs_to_modify = [ + "/usr/discreet/project/{}".format(project_name), + "/opt/Autodesk/clip/{}/{}.prj".format(volume_name, project_name), + "/usr/discreet/clip/{}/{}.prj".format(volume_name, project_name) + ] + + for dirtm in dirs_to_modify: + for root, dirs, files in os.walk(dirtm): + try: + for name in set(dirs) | set(files): + path = os.path.join(root, name) + st = os.stat(path) + if oct(st.st_mode) != self.permissions: + os.chmod(path, self.permissions) + + except OSError as exc: + self.log.warning("Not able to open files: {}".format(exc)) + + def _get_flame_fps(self, fps_num): fps_table = { float(23.976): "23.976 fps", diff --git a/openpype/hosts/flame/plugins/create/create_shot_clip.py b/openpype/hosts/flame/plugins/create/create_shot_clip.py index fa239ea420..4fb041a4b2 100644 --- a/openpype/hosts/flame/plugins/create/create_shot_clip.py +++ b/openpype/hosts/flame/plugins/create/create_shot_clip.py @@ -23,10 +23,11 @@ class CreateShotClip(opfapi.Creator): # nested dictionary (only one level allowed # for sections and dict) for _k, _v in v["value"].items(): - if presets.get(_k): + if presets.get(_k) is not None: gui_inputs[k][ "value"][_k]["value"] = presets[_k] - if presets.get(k): + + if presets.get(k) is not None: gui_inputs[k]["value"] = presets[k] # open widget for plugins inputs @@ -276,6 +277,22 @@ class CreateShotClip(opfapi.Creator): "target": "tag", "toolTip": "By default handles are excluded", # noqa "order": 3 + }, + "retimedHandles": { + "value": True, + "type": "QCheckBox", + "label": "Retimed handles", + "target": "tag", + "toolTip": "By default handles are retimed.", # noqa + "order": 4 + }, + "retimedFramerange": { + "value": True, + "type": "QCheckBox", + "label": "Retimed framerange", + "target": "tag", + "toolTip": "By default framerange is retimed.", # noqa + "order": 5 } } } diff --git a/openpype/hosts/flame/plugins/load/load_clip.py b/openpype/hosts/flame/plugins/load/load_clip.py index b12f2f9690..0843dde76a 100644 --- a/openpype/hosts/flame/plugins/load/load_clip.py +++ b/openpype/hosts/flame/plugins/load/load_clip.py @@ -4,6 +4,7 @@ from pprint import pformat import openpype.hosts.flame.api as opfapi from openpype.lib import StringTemplate + class LoadClip(opfapi.ClipLoader): """Load a subset to timeline as clip @@ -60,8 +61,6 @@ class LoadClip(opfapi.ClipLoader): "path": self.fname.replace("\\", "/"), "colorspace": colorspace, "version": "v{:0>3}".format(version_name), - "logger": self.log - } self.log.debug(pformat( loading_context @@ -69,7 +68,8 @@ class LoadClip(opfapi.ClipLoader): self.log.debug(openclip_path) # make openpype clip file - opfapi.OpenClipSolver(openclip_path, loading_context).make() + opfapi.OpenClipSolver( + openclip_path, loading_context, logger=self.log).make() # prepare Reel group in actual desktop opc = self._get_clip( diff --git a/openpype/hosts/flame/plugins/load/load_clip_batch.py b/openpype/hosts/flame/plugins/load/load_clip_batch.py index fb4a3dc6e9..3b049b861b 100644 --- a/openpype/hosts/flame/plugins/load/load_clip_batch.py +++ b/openpype/hosts/flame/plugins/load/load_clip_batch.py @@ -64,8 +64,6 @@ class LoadClipBatch(opfapi.ClipLoader): "path": self.fname.replace("\\", "/"), "colorspace": colorspace, "version": "v{:0>3}".format(version_name), - "logger": self.log - } self.log.debug(pformat( loading_context @@ -73,7 +71,8 @@ class LoadClipBatch(opfapi.ClipLoader): self.log.debug(openclip_path) # make openpype clip file - opfapi.OpenClipSolver(openclip_path, loading_context).make() + opfapi.OpenClipSolver( + openclip_path, loading_context, logger=self.log).make() # prepare Reel group in actual desktop opc = self._get_clip( diff --git a/openpype/hosts/flame/plugins/publish/collect_timeline_instances.py b/openpype/hosts/flame/plugins/publish/collect_timeline_instances.py index 992db62c75..76d48dded2 100644 --- a/openpype/hosts/flame/plugins/publish/collect_timeline_instances.py +++ b/openpype/hosts/flame/plugins/publish/collect_timeline_instances.py @@ -131,6 +131,9 @@ class CollectTimelineInstances(pyblish.api.ContextPlugin): "fps": self.fps, "workfileFrameStart": workfile_start, "sourceFirstFrame": int(first_frame), + "retimedHandles": marker_data.get("retimedHandles"), + "shotDurationFromSource": ( + not marker_data.get("retimedFramerange")), "path": file_path, "flameAddTasks": self.add_tasks, "tasks": { diff --git a/openpype/hosts/flame/plugins/publish/extract_subset_resources.py b/openpype/hosts/flame/plugins/publish/extract_subset_resources.py index 3e1e8db986..5482af973c 100644 --- a/openpype/hosts/flame/plugins/publish/extract_subset_resources.py +++ b/openpype/hosts/flame/plugins/publish/extract_subset_resources.py @@ -1,7 +1,6 @@ import os import re import tempfile -from pprint import pformat from copy import deepcopy import pyblish.api @@ -80,40 +79,53 @@ class ExtractSubsetResources(openpype.api.Extractor): retimed_data = self._get_retimed_attributes(instance) # get individual keys - r_handle_start = retimed_data["handle_start"] - r_handle_end = retimed_data["handle_end"] - r_source_dur = retimed_data["source_duration"] - r_speed = retimed_data["speed"] + retimed_handle_start = retimed_data["handle_start"] + retimed_handle_end = retimed_data["handle_end"] + retimed_source_duration = retimed_data["source_duration"] + retimed_speed = retimed_data["speed"] # get handles value - take only the max from both handle_start = instance.data["handleStart"] handle_end = instance.data["handleEnd"] handles = max(handle_start, handle_end) include_handles = instance.data.get("includeHandles") + retimed_handles = instance.data.get("retimedHandles") # get media source range with handles source_start_handles = instance.data["sourceStartH"] source_end_handles = instance.data["sourceEndH"] + # retime if needed - if r_speed != 1.0: - source_start_handles = ( - instance.data["sourceStart"] - r_handle_start) - source_end_handles = ( - source_start_handles - + (r_source_dur - 1) - + r_handle_start - + r_handle_end - ) + if retimed_speed != 1.0: + if retimed_handles: + # handles are retimed + source_start_handles = ( + instance.data["sourceStart"] - retimed_handle_start) + source_end_handles = ( + source_start_handles + + (retimed_source_duration - 1) + + retimed_handle_start + + retimed_handle_end + ) + + else: + # handles are not retimed + source_end_handles = ( + source_start_handles + + (retimed_source_duration - 1) + + handle_start + + handle_end + ) # get frame range with handles for representation range frame_start_handle = frame_start - handle_start repre_frame_start = frame_start_handle if include_handles: - if r_speed == 1.0: + if retimed_speed == 1.0 or not retimed_handles: frame_start_handle = frame_start else: frame_start_handle = ( - frame_start - handle_start) + r_handle_start + frame_start - handle_start) + retimed_handle_start self.log.debug("_ frame_start_handle: {}".format( frame_start_handle)) @@ -124,6 +136,9 @@ class ExtractSubsetResources(openpype.api.Extractor): source_duration_handles = ( source_end_handles - source_start_handles) + 1 + self.log.debug("_ source_duration_handles: {}".format( + source_duration_handles)) + # create staging dir path staging_dir = self.staging_dir(instance) @@ -147,15 +162,30 @@ class ExtractSubsetResources(openpype.api.Extractor): if version_data: instance.data["versionData"].update(version_data) - if r_speed != 1.0: - instance.data["versionData"].update({ - "frameStart": frame_start_handle, - "frameEnd": ( - (frame_start_handle + source_duration_handles - 1) - - (r_handle_start + r_handle_end) - ) - }) - self.log.debug("_ i_version_data: {}".format( + # version data start frame + version_frame_start = frame_start + if include_handles: + version_frame_start = frame_start_handle + if retimed_speed != 1.0: + if retimed_handles: + instance.data["versionData"].update({ + "frameStart": version_frame_start, + "frameEnd": ( + (version_frame_start + source_duration_handles - 1) + - (retimed_handle_start + retimed_handle_end) + ) + }) + else: + instance.data["versionData"].update({ + "handleStart": handle_start, + "handleEnd": handle_end, + "frameStart": version_frame_start, + "frameEnd": ( + (version_frame_start + source_duration_handles - 1) + - (handle_start + handle_end) + ) + }) + self.log.debug("_ version_data: {}".format( instance.data["versionData"] )) diff --git a/openpype/hosts/flame/startup/openpype_in_flame.py b/openpype/hosts/flame/startup/openpype_in_flame.py index f2ac23b19e..d07aaa6b7d 100644 --- a/openpype/hosts/flame/startup/openpype_in_flame.py +++ b/openpype/hosts/flame/startup/openpype_in_flame.py @@ -73,6 +73,8 @@ def load_apps(): opfapi.FlameMenuProjectConnect(opfapi.CTX.app_framework)) opfapi.CTX.flame_apps.append( opfapi.FlameMenuTimeline(opfapi.CTX.app_framework)) + opfapi.CTX.flame_apps.append( + opfapi.FlameMenuUniversal(opfapi.CTX.app_framework)) opfapi.CTX.app_framework.log.info("Apps are loaded") @@ -191,3 +193,27 @@ def get_timeline_custom_ui_actions(): openpype_install() return _build_app_menu("FlameMenuTimeline") + + +def get_batch_custom_ui_actions(): + """Hook to create submenu in batch + + Returns: + list: menu object + """ + # install openpype and the host + openpype_install() + + return _build_app_menu("FlameMenuUniversal") + + +def get_media_panel_custom_ui_actions(): + """Hook to create submenu in desktop + + Returns: + list: menu object + """ + # install openpype and the host + openpype_install() + + return _build_app_menu("FlameMenuUniversal") diff --git a/openpype/hosts/fusion/__init__.py b/openpype/hosts/fusion/__init__.py index e69de29bb2..ddae01890b 100644 --- a/openpype/hosts/fusion/__init__.py +++ b/openpype/hosts/fusion/__init__.py @@ -0,0 +1,10 @@ +from .addon import ( + FusionAddon, + FUSION_HOST_DIR, +) + + +__all__ = ( + "FusionAddon", + "FUSION_HOST_DIR", +) diff --git a/openpype/hosts/fusion/addon.py b/openpype/hosts/fusion/addon.py new file mode 100644 index 0000000000..1913cc2e30 --- /dev/null +++ b/openpype/hosts/fusion/addon.py @@ -0,0 +1,32 @@ +import os +from openpype.modules import OpenPypeModule +from openpype.modules.interfaces import IHostAddon + +FUSION_HOST_DIR = os.path.dirname(os.path.abspath(__file__)) + + +class FusionAddon(OpenPypeModule, IHostAddon): + name = "fusion" + host_name = "fusion" + + def initialize(self, module_settings): + self.enabled = True + + def get_launch_hook_paths(self, app): + if app.host_name != self.host_name: + return [] + return [ + os.path.join(FUSION_HOST_DIR, "hooks") + ] + + def add_implementation_envs(self, env, _app): + # Set default values if are not already set via settings + defaults = { + "OPENPYPE_LOG_NO_COLORS": "Yes" + } + for key, value in defaults.items(): + if not env.get(key): + env[key] = value + + def get_workfile_extensions(self): + return [".comp"] diff --git a/openpype/hosts/fusion/api/__init__.py b/openpype/hosts/fusion/api/__init__.py index 19d1e092fe..ed70dbca50 100644 --- a/openpype/hosts/fusion/api/__init__.py +++ b/openpype/hosts/fusion/api/__init__.py @@ -5,10 +5,7 @@ from .pipeline import ( ls, imprint_container, - parse_container, - - get_current_comp, - comp_lock_and_undo_chunk + parse_container ) from .workio import ( @@ -22,8 +19,10 @@ from .workio import ( from .lib import ( maintained_selection, - get_additional_data, - update_frame_range + update_frame_range, + set_asset_framerange, + get_current_comp, + comp_lock_and_undo_chunk ) from .menu import launch_openpype_menu @@ -38,9 +37,6 @@ __all__ = [ "imprint_container", "parse_container", - "get_current_comp", - "comp_lock_and_undo_chunk", - # workio "open_file", "save_file", @@ -51,8 +47,10 @@ __all__ = [ # lib "maintained_selection", - "get_additional_data", "update_frame_range", + "set_asset_framerange", + "get_current_comp", + "comp_lock_and_undo_chunk", # menu "launch_openpype_menu", diff --git a/openpype/hosts/fusion/api/lib.py b/openpype/hosts/fusion/api/lib.py index 001eb636ee..4ef44dbb61 100644 --- a/openpype/hosts/fusion/api/lib.py +++ b/openpype/hosts/fusion/api/lib.py @@ -5,6 +5,7 @@ import contextlib from Qt import QtGui +from openpype.lib import Logger from openpype.client import ( get_asset_by_name, get_subset_by_name, @@ -17,13 +18,14 @@ from openpype.pipeline import ( switch_container, legacy_io, ) -from .pipeline import get_current_comp, comp_lock_and_undo_chunk +from openpype.pipeline.context_tools import get_current_project_asset self = sys.modules[__name__] self._project = None -def update_frame_range(start, end, comp=None, set_render_range=True): +def update_frame_range(start, end, comp=None, set_render_range=True, + handle_start=0, handle_end=0): """Set Fusion comp's start and end frame range Args: @@ -32,6 +34,8 @@ def update_frame_range(start, end, comp=None, set_render_range=True): comp (object, Optional): comp object from fusion set_render_range (bool, Optional): When True this will also set the composition's render start and end frame. + handle_start (float, int, Optional): frame handles before start frame + handle_end (float, int, Optional): frame handles after end frame Returns: None @@ -41,11 +45,16 @@ def update_frame_range(start, end, comp=None, set_render_range=True): if not comp: comp = get_current_comp() + # Convert any potential none type to zero + handle_start = handle_start or 0 + handle_end = handle_end or 0 + attrs = { - "COMPN_GlobalStart": start, - "COMPN_GlobalEnd": end + "COMPN_GlobalStart": start - handle_start, + "COMPN_GlobalEnd": end + handle_end } + # set frame range if set_render_range: attrs.update({ "COMPN_RenderStart": start, @@ -56,24 +65,116 @@ def update_frame_range(start, end, comp=None, set_render_range=True): comp.SetAttrs(attrs) -def get_additional_data(container): - """Get Fusion related data for the container +def set_asset_framerange(): + """Set Comp's frame range based on current asset""" + asset_doc = get_current_project_asset() + start = asset_doc["data"]["frameStart"] + end = asset_doc["data"]["frameEnd"] + handle_start = asset_doc["data"]["handleStart"] + handle_end = asset_doc["data"]["handleEnd"] + update_frame_range(start, end, set_render_range=True, + handle_start=handle_start, + handle_end=handle_end) - Args: - container(dict): the container found by the ls() function - Returns: - dict +def set_asset_resolution(): + """Set Comp's resolution width x height default based on current asset""" + asset_doc = get_current_project_asset() + width = asset_doc["data"]["resolutionWidth"] + height = asset_doc["data"]["resolutionHeight"] + comp = get_current_comp() + + print("Setting comp frame format resolution to {}x{}".format(width, + height)) + comp.SetPrefs({ + "Comp.FrameFormat.Width": width, + "Comp.FrameFormat.Height": height, + }) + + +def validate_comp_prefs(comp=None): + """Validate current comp defaults with asset settings. + + Validates fps, resolutionWidth, resolutionHeight, aspectRatio. + + This does *not* validate frameStart, frameEnd, handleStart and handleEnd. """ - tool = container["_tool"] - tile_color = tool.TileColor - if tile_color is None: - return {} + if comp is None: + comp = get_current_comp() - return {"color": QtGui.QColor.fromRgbF(tile_color["R"], - tile_color["G"], - tile_color["B"])} + log = Logger.get_logger("validate_comp_prefs") + + fields = [ + "name", + "data.fps", + "data.resolutionWidth", + "data.resolutionHeight", + "data.pixelAspect" + ] + asset_doc = get_current_project_asset(fields=fields) + asset_data = asset_doc["data"] + + comp_frame_format_prefs = comp.GetPrefs("Comp.FrameFormat") + + # Pixel aspect ratio in Fusion is set as AspectX and AspectY so we convert + # the data to something that is more sensible to Fusion + asset_data["pixelAspectX"] = asset_data.pop("pixelAspect") + asset_data["pixelAspectY"] = 1.0 + + validations = [ + ("fps", "Rate", "FPS"), + ("resolutionWidth", "Width", "Resolution Width"), + ("resolutionHeight", "Height", "Resolution Height"), + ("pixelAspectX", "AspectX", "Pixel Aspect Ratio X"), + ("pixelAspectY", "AspectY", "Pixel Aspect Ratio Y") + ] + + invalid = [] + for key, comp_key, label in validations: + asset_value = asset_data[key] + comp_value = comp_frame_format_prefs.get(comp_key) + if asset_value != comp_value: + # todo: Actually show dialog to user instead of just logging + log.warning( + "Comp {pref} {value} does not match asset " + "'{asset_name}' {pref} {asset_value}".format( + pref=label, + value=comp_value, + asset_name=asset_doc["name"], + asset_value=asset_value) + ) + + invalid_msg = "{} {} should be {}".format(label, + comp_value, + asset_value) + invalid.append(invalid_msg) + + if invalid: + + def _on_repair(): + attributes = dict() + for key, comp_key, _label in validations: + value = asset_data[key] + comp_key_full = "Comp.FrameFormat.{}".format(comp_key) + attributes[comp_key_full] = value + comp.SetPrefs(attributes) + + from . import menu + from openpype.widgets import popup + from openpype.style import load_stylesheet + dialog = popup.Popup(parent=menu.menu) + dialog.setWindowTitle("Fusion comp has invalid configuration") + + msg = "Comp preferences mismatches '{}'".format(asset_doc["name"]) + msg += "\n" + "\n".join(invalid) + dialog.setMessage(msg) + dialog.setButtonText("Repair") + dialog.on_clicked.connect(_on_repair) + dialog.show() + dialog.raise_() + dialog.activateWindow() + dialog.setStyleSheet(load_stylesheet()) def switch_item(container, @@ -195,3 +296,21 @@ def get_frame_path(path): padding = 4 # default Fusion padding return filename, padding, ext + + +def get_current_comp(): + """Hack to get current comp in this session""" + fusion = getattr(sys.modules["__main__"], "fusion", None) + return fusion.CurrentComp if fusion else None + + +@contextlib.contextmanager +def comp_lock_and_undo_chunk(comp, undo_queue_name="Script CMD"): + """Lock comp and open an undo chunk during the context""" + try: + comp.Lock() + comp.StartUndo(undo_queue_name) + yield + finally: + comp.Unlock() + comp.EndUndo() diff --git a/openpype/hosts/fusion/api/menu.py b/openpype/hosts/fusion/api/menu.py index 6234322d7f..7a6293807f 100644 --- a/openpype/hosts/fusion/api/menu.py +++ b/openpype/hosts/fusion/api/menu.py @@ -1,43 +1,25 @@ -import os import sys -from Qt import QtWidgets, QtCore +from Qt import QtWidgets, QtCore, QtGui -from openpype import style from openpype.tools.utils import host_tools - +from openpype.style import load_stylesheet +from openpype.lib import register_event_callback from openpype.hosts.fusion.scripts import ( set_rendermode, duplicate_with_inputs ) +from openpype.hosts.fusion.api.lib import ( + set_asset_framerange, + set_asset_resolution +) +from openpype.pipeline import legacy_io +from openpype.resources import get_openpype_icon_filepath +from .pulse import FusionPulse -def load_stylesheet(): - path = os.path.join(os.path.dirname(__file__), "menu_style.qss") - if not os.path.exists(path): - print("Unable to load stylesheet, file not found in resources") - return "" - - with open(path, "r") as file_stream: - stylesheet = file_stream.read() - return stylesheet - - -class Spacer(QtWidgets.QWidget): - def __init__(self, height, *args, **kwargs): - super(Spacer, self).__init__(*args, **kwargs) - - self.setFixedHeight(height) - - real_spacer = QtWidgets.QWidget(self) - real_spacer.setObjectName("Spacer") - real_spacer.setFixedHeight(height) - - layout = QtWidgets.QVBoxLayout(self) - layout.setContentsMargins(0, 0, 0, 0) - layout.addWidget(real_spacer) - - self.setLayout(layout) +self = sys.modules[__name__] +self.menu = None class OpenPypeMenu(QtWidgets.QWidget): @@ -46,15 +28,29 @@ class OpenPypeMenu(QtWidgets.QWidget): self.setObjectName("OpenPypeMenu") + icon_path = get_openpype_icon_filepath() + icon = QtGui.QIcon(icon_path) + self.setWindowIcon(icon) + self.setWindowFlags( QtCore.Qt.Window | QtCore.Qt.CustomizeWindowHint | QtCore.Qt.WindowTitleHint + | QtCore.Qt.WindowMinimizeButtonHint | QtCore.Qt.WindowCloseButtonHint | QtCore.Qt.WindowStaysOnTopHint ) self.render_mode_widget = None self.setWindowTitle("OpenPype") + + asset_label = QtWidgets.QLabel("Context", self) + asset_label.setStyleSheet("""QLabel { + font-size: 14px; + font-weight: 600; + color: #5f9fb8; + }""") + asset_label.setAlignment(QtCore.Qt.AlignHCenter) + workfiles_btn = QtWidgets.QPushButton("Workfiles...", self) create_btn = QtWidgets.QPushButton("Create...", self) publish_btn = QtWidgets.QPushButton("Publish...", self) @@ -62,77 +58,107 @@ class OpenPypeMenu(QtWidgets.QWidget): manager_btn = QtWidgets.QPushButton("Manage...", self) libload_btn = QtWidgets.QPushButton("Library...", self) rendermode_btn = QtWidgets.QPushButton("Set render mode...", self) + set_framerange_btn = QtWidgets.QPushButton("Set Frame Range", self) + set_resolution_btn = QtWidgets.QPushButton("Set Resolution", self) duplicate_with_inputs_btn = QtWidgets.QPushButton( "Duplicate with input connections", self ) - reset_resolution_btn = QtWidgets.QPushButton( - "Reset Resolution from project", self - ) layout = QtWidgets.QVBoxLayout(self) layout.setContentsMargins(10, 20, 10, 20) + layout.addWidget(asset_label) + + layout.addSpacing(20) + layout.addWidget(workfiles_btn) + + layout.addSpacing(20) + layout.addWidget(create_btn) - layout.addWidget(publish_btn) layout.addWidget(load_btn) + layout.addWidget(publish_btn) layout.addWidget(manager_btn) - layout.addWidget(Spacer(15, self)) + layout.addSpacing(20) layout.addWidget(libload_btn) - layout.addWidget(Spacer(15, self)) + layout.addSpacing(20) + layout.addWidget(set_framerange_btn) + layout.addWidget(set_resolution_btn) layout.addWidget(rendermode_btn) - layout.addWidget(Spacer(15, self)) + layout.addSpacing(20) layout.addWidget(duplicate_with_inputs_btn) - layout.addWidget(reset_resolution_btn) self.setLayout(layout) + # Store reference so we can update the label + self.asset_label = asset_label + workfiles_btn.clicked.connect(self.on_workfile_clicked) create_btn.clicked.connect(self.on_create_clicked) publish_btn.clicked.connect(self.on_publish_clicked) load_btn.clicked.connect(self.on_load_clicked) manager_btn.clicked.connect(self.on_manager_clicked) libload_btn.clicked.connect(self.on_libload_clicked) - rendermode_btn.clicked.connect(self.on_rendernode_clicked) + rendermode_btn.clicked.connect(self.on_rendermode_clicked) duplicate_with_inputs_btn.clicked.connect( self.on_duplicate_with_inputs_clicked) - reset_resolution_btn.clicked.connect(self.on_reset_resolution_clicked) + set_resolution_btn.clicked.connect(self.on_set_resolution_clicked) + set_framerange_btn.clicked.connect(self.on_set_framerange_clicked) + + self._callbacks = [] + self.register_callback("taskChanged", self.on_task_changed) + self.on_task_changed() + + # Force close current process if Fusion is closed + self._pulse = FusionPulse(parent=self) + self._pulse.start() + + def on_task_changed(self): + # Update current context label + label = legacy_io.Session["AVALON_ASSET"] + self.asset_label.setText(label) + + def register_callback(self, name, fn): + + # Create a wrapper callback that we only store + # for as long as we want it to persist as callback + def _callback(*args): + fn() + + self._callbacks.append(_callback) + register_event_callback(name, _callback) + + def deregister_all_callbacks(self): + self._callbacks[:] = [] def on_workfile_clicked(self): - print("Clicked Workfile") host_tools.show_workfiles() def on_create_clicked(self): - print("Clicked Create") host_tools.show_creator() def on_publish_clicked(self): - print("Clicked Publish") host_tools.show_publish() def on_load_clicked(self): - print("Clicked Load") host_tools.show_loader(use_context=True) def on_manager_clicked(self): - print("Clicked Manager") host_tools.show_scene_inventory() def on_libload_clicked(self): - print("Clicked Library") host_tools.show_library_loader() - def on_rendernode_clicked(self): - print("Clicked Set Render Mode") + def on_rendermode_clicked(self): if self.render_mode_widget is None: window = set_rendermode.SetRenderMode() - window.setStyleSheet(style.load_stylesheet()) + window.setStyleSheet(load_stylesheet()) window.show() self.render_mode_widget = window else: @@ -140,15 +166,16 @@ class OpenPypeMenu(QtWidgets.QWidget): def on_duplicate_with_inputs_clicked(self): duplicate_with_inputs.duplicate_with_input_connections() - print("Clicked Set Colorspace") - def on_reset_resolution_clicked(self): - print("Clicked Reset Resolution") + def on_set_resolution_clicked(self): + set_asset_resolution() + + def on_set_framerange_clicked(self): + set_asset_framerange() def launch_openpype_menu(): app = QtWidgets.QApplication(sys.argv) - app.setQuitOnLastWindowClosed(False) pype_menu = OpenPypeMenu() @@ -156,5 +183,8 @@ def launch_openpype_menu(): pype_menu.setStyleSheet(stylesheet) pype_menu.show() + self.menu = pype_menu - sys.exit(app.exec_()) + result = app.exec_() + print("Shutting down..") + sys.exit(result) diff --git a/openpype/hosts/fusion/api/menu_style.qss b/openpype/hosts/fusion/api/menu_style.qss deleted file mode 100644 index 12c474b070..0000000000 --- a/openpype/hosts/fusion/api/menu_style.qss +++ /dev/null @@ -1,29 +0,0 @@ -QWidget { - background-color: #282828; - border-radius: 3; -} - -QPushButton { - border: 1px solid #090909; - background-color: #201f1f; - color: #ffffff; - padding: 5; -} - -QPushButton:focus { - background-color: "#171717"; - color: #d0d0d0; -} - -QPushButton:hover { - background-color: "#171717"; - color: #e64b3d; -} - -#OpenPypeMenu { - border: 1px solid #fef9ef; -} - -#Spacer { - background-color: #282828; -} diff --git a/openpype/hosts/fusion/api/pipeline.py b/openpype/hosts/fusion/api/pipeline.py index 54a6c94b60..c92d072ef7 100644 --- a/openpype/hosts/fusion/api/pipeline.py +++ b/openpype/hosts/fusion/api/pipeline.py @@ -2,13 +2,14 @@ Basic avalon integration """ import os -import sys import logging -import contextlib import pyblish.api -from openpype.lib import Logger +from openpype.lib import ( + Logger, + register_event_callback +) from openpype.pipeline import ( register_loader_plugin_path, register_creator_plugin_path, @@ -18,12 +19,19 @@ from openpype.pipeline import ( deregister_inventory_action_path, AVALON_CONTAINER_ID, ) -import openpype.hosts.fusion +from openpype.pipeline.load import any_outdated_containers +from openpype.hosts.fusion import FUSION_HOST_DIR +from openpype.tools.utils import host_tools + +from .lib import ( + get_current_comp, + comp_lock_and_undo_chunk, + validate_comp_prefs +) log = Logger.get_logger(__name__) -HOST_DIR = os.path.dirname(os.path.abspath(openpype.hosts.fusion.__file__)) -PLUGINS_DIR = os.path.join(HOST_DIR, "plugins") +PLUGINS_DIR = os.path.join(FUSION_HOST_DIR, "plugins") PUBLISH_PATH = os.path.join(PLUGINS_DIR, "publish") LOAD_PATH = os.path.join(PLUGINS_DIR, "load") @@ -40,7 +48,7 @@ class CompLogHandler(logging.Handler): def install(): - """Install fusion-specific functionality of avalon-core. + """Install fusion-specific functionality of OpenPype. This is where you install menus and register families, data and loaders into fusion. @@ -52,7 +60,7 @@ def install(): """ # Remove all handlers associated with the root logger object, because - # that one sometimes logs as "warnings" incorrectly. + # that one always logs as "warnings" incorrectly. for handler in logging.root.handlers[:]: logging.root.removeHandler(handler) @@ -64,8 +72,6 @@ def install(): logger.addHandler(handler) logger.setLevel(logging.DEBUG) - log.info("openpype.hosts.fusion installed") - pyblish.api.register_host("fusion") pyblish.api.register_plugin_path(PUBLISH_PATH) log.info("Registering Fusion plug-ins..") @@ -78,6 +84,11 @@ def install(): "instanceToggled", on_pyblish_instance_toggled ) + # Fusion integration currently does not attach to direct callbacks of + # the application. So we use workfile callbacks to allow similar behavior + # on save and open + register_event_callback("workfile.open.after", on_after_open) + def uninstall(): """Uninstall all that was installed @@ -103,7 +114,7 @@ def uninstall(): ) -def on_pyblish_instance_toggled(instance, new_value, old_value): +def on_pyblish_instance_toggled(instance, old_value, new_value): """Toggle saver tool passthrough states on instance toggles.""" comp = instance.context.data.get("currentComp") if not comp: @@ -126,6 +137,38 @@ def on_pyblish_instance_toggled(instance, new_value, old_value): tool.SetAttrs({"TOOLB_PassThrough": passthrough}) +def on_after_open(_event): + comp = get_current_comp() + validate_comp_prefs(comp) + + if any_outdated_containers(): + log.warning("Scene has outdated content.") + + # Find OpenPype menu to attach to + from . import menu + + def _on_show_scene_inventory(): + # ensure that comp is active + frame = comp.CurrentFrame + if not frame: + print("Comp is closed, skipping show scene inventory") + return + frame.ActivateFrame() # raise comp window + host_tools.show_scene_inventory() + + from openpype.widgets import popup + from openpype.style import load_stylesheet + dialog = popup.Popup(parent=menu.menu) + dialog.setWindowTitle("Fusion comp has outdated content") + dialog.setMessage("There are outdated containers in " + "your Fusion comp.") + dialog.on_clicked.connect(_on_show_scene_inventory) + dialog.show() + dialog.raise_() + dialog.activateWindow() + dialog.setStyleSheet(load_stylesheet()) + + def ls(): """List containers from active Fusion scene @@ -211,19 +254,3 @@ def parse_container(tool): return container -def get_current_comp(): - """Hack to get current comp in this session""" - fusion = getattr(sys.modules["__main__"], "fusion", None) - return fusion.CurrentComp if fusion else None - - -@contextlib.contextmanager -def comp_lock_and_undo_chunk(comp, undo_queue_name="Script CMD"): - """Lock comp and open an undo chunk during the context""" - try: - comp.Lock() - comp.StartUndo(undo_queue_name) - yield - finally: - comp.Unlock() - comp.EndUndo() diff --git a/openpype/hosts/fusion/api/pulse.py b/openpype/hosts/fusion/api/pulse.py new file mode 100644 index 0000000000..5b61f3bd63 --- /dev/null +++ b/openpype/hosts/fusion/api/pulse.py @@ -0,0 +1,60 @@ +import os +import sys + +from Qt import QtCore + + +class PulseThread(QtCore.QThread): + no_response = QtCore.Signal() + + def __init__(self, parent=None): + super(PulseThread, self).__init__(parent=parent) + + def run(self): + app = getattr(sys.modules["__main__"], "app", None) + + # Interval in milliseconds + interval = os.environ.get("OPENPYPE_FUSION_PULSE_INTERVAL", 1000) + + while True: + if self.isInterruptionRequested(): + return + try: + app.Test() + except Exception: + self.no_response.emit() + + self.msleep(interval) + + +class FusionPulse(QtCore.QObject): + """A Timer that checks whether host app is still alive. + + This checks whether the Fusion process is still active at a certain + interval. This is useful due to how Fusion runs its scripts. Each script + runs in its own environment and process (a `fusionscript` process each). + If Fusion would go down and we have a UI process running at the same time + then it can happen that the `fusionscript.exe` will remain running in the + background in limbo due to e.g. a Qt interface's QApplication that keeps + running infinitely. + + Warning: + When the host is not detected this will automatically exit + the current process. + + """ + + def __init__(self, parent=None): + super(FusionPulse, self).__init__(parent=parent) + self._thread = PulseThread(parent=self) + self._thread.no_response.connect(self.on_no_response) + + def on_no_response(self): + print("Pulse detected no response from Fusion..") + sys.exit(1) + + def start(self): + self._thread.start() + + def stop(self): + self._thread.requestInterruption() diff --git a/openpype/hosts/fusion/api/workio.py b/openpype/hosts/fusion/api/workio.py index a1710c6e3a..939b2ff4be 100644 --- a/openpype/hosts/fusion/api/workio.py +++ b/openpype/hosts/fusion/api/workio.py @@ -2,13 +2,11 @@ import sys import os -from openpype.pipeline import HOST_WORKFILE_EXTENSIONS - -from .pipeline import get_current_comp +from .lib import get_current_comp def file_extensions(): - return HOST_WORKFILE_EXTENSIONS["fusion"] + return [".comp"] def has_unsaved_changes(): diff --git a/openpype/hosts/fusion/deploy/Config/openpype_menu.fu b/openpype/hosts/fusion/deploy/Config/openpype_menu.fu new file mode 100644 index 0000000000..8b8d448259 --- /dev/null +++ b/openpype/hosts/fusion/deploy/Config/openpype_menu.fu @@ -0,0 +1,60 @@ +{ + Action + { + ID = "OpenPype_Menu", + Category = "OpenPype", + Name = "OpenPype Menu", + + Targets = + { + Composition = + { + Execute = _Lua [=[ + local scriptPath = app:MapPath("OpenPype:MenuScripts/openpype_menu.py") + if bmd.fileexists(scriptPath) == false then + print("[OpenPype Error] Can't run file: " .. scriptPath) + else + target:RunScript(scriptPath) + end + ]=], + }, + }, + }, + Action + { + ID = "OpenPype_Install_PySide2", + Category = "OpenPype", + Name = "Install PySide2", + + Targets = + { + Composition = + { + Execute = _Lua [=[ + local scriptPath = app:MapPath("OpenPype:MenuScripts/install_pyside2.py") + if bmd.fileexists(scriptPath) == false then + print("[OpenPype Error] Can't run file: " .. scriptPath) + else + target:RunScript(scriptPath) + end + ]=], + }, + }, + }, + Menus + { + Target = "ChildFrame", + + Before "Help" + { + Sub "OpenPype" + { + "OpenPype_Menu{}", + "_", + Sub "Admin" { + "OpenPype_Install_PySide2{}" + } + } + }, + }, +} diff --git a/openpype/hosts/fusion/deploy/MenuScripts/README.md b/openpype/hosts/fusion/deploy/MenuScripts/README.md new file mode 100644 index 0000000000..f87eaea4a2 --- /dev/null +++ b/openpype/hosts/fusion/deploy/MenuScripts/README.md @@ -0,0 +1,6 @@ +### OpenPype deploy MenuScripts + +Note that this `MenuScripts` is not an official Fusion folder. +OpenPype only uses this folder in `{fusion}/deploy/` to trigger the OpenPype menu actions. + +They are used in the actions defined in `.fu` files in `{fusion}/deploy/Config`. \ No newline at end of file diff --git a/openpype/hosts/fusion/deploy/MenuScripts/install_pyside2.py b/openpype/hosts/fusion/deploy/MenuScripts/install_pyside2.py new file mode 100644 index 0000000000..ab9f13ce05 --- /dev/null +++ b/openpype/hosts/fusion/deploy/MenuScripts/install_pyside2.py @@ -0,0 +1,29 @@ +# This is just a quick hack for users running Py3 locally but having no +# Qt library installed +import os +import subprocess +import importlib + + +try: + from Qt import QtWidgets # noqa: F401 + from Qt import __binding__ + print(f"Qt binding: {__binding__}") + mod = importlib.import_module(__binding__) + print(f"Qt path: {mod.__file__}") + print("Qt library found, nothing to do..") + +except ImportError: + print("Assuming no Qt library is installed..") + print('Installing PySide2 for Python 3.6: ' + f'{os.environ["FUSION16_PYTHON36_HOME"]}') + + # Get full path to python executable + exe = "python.exe" if os.name == 'nt' else "python" + python = os.path.join(os.environ["FUSION16_PYTHON36_HOME"], exe) + assert os.path.exists(python), f"Python doesn't exist: {python}" + + # Do python -m pip install PySide2 + args = [python, "-m", "pip", "install", "PySide2"] + print(f"Args: {args}") + subprocess.Popen(args) diff --git a/openpype/hosts/fusion/utility_scripts/__OpenPype_Menu__.py b/openpype/hosts/fusion/deploy/MenuScripts/openpype_menu.py similarity index 51% rename from openpype/hosts/fusion/utility_scripts/__OpenPype_Menu__.py rename to openpype/hosts/fusion/deploy/MenuScripts/openpype_menu.py index 870e74280a..2918c552c8 100644 --- a/openpype/hosts/fusion/utility_scripts/__OpenPype_Menu__.py +++ b/openpype/hosts/fusion/deploy/MenuScripts/openpype_menu.py @@ -9,6 +9,10 @@ from openpype.pipeline import ( def main(env): + # This script working directory starts in Fusion application folder. + # However the contents of that folder can conflict with Qt library dlls + # so we make sure to move out of it to avoid DLL Load Failed errors. + os.chdir("..") from openpype.hosts.fusion import api from openpype.hosts.fusion.api import menu @@ -20,6 +24,11 @@ def main(env): menu.launch_openpype_menu() + # Initiate a QTimer to check if Fusion is still alive every X interval + # If Fusion is not found - kill itself + # todo(roy): Implement timer that ensures UI doesn't remain when e.g. + # Fusion closes down + if __name__ == "__main__": result = main(os.environ) diff --git a/openpype/hosts/fusion/utility_scripts/32bit/backgrounds_selected_to32bit.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/backgrounds_selected_to32bit.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/32bit/backgrounds_selected_to32bit.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/backgrounds_selected_to32bit.py diff --git a/openpype/hosts/fusion/utility_scripts/32bit/backgrounds_to32bit.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/backgrounds_to32bit.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/32bit/backgrounds_to32bit.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/backgrounds_to32bit.py diff --git a/openpype/hosts/fusion/utility_scripts/32bit/loaders_selected_to32bit.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/loaders_selected_to32bit.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/32bit/loaders_selected_to32bit.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/loaders_selected_to32bit.py diff --git a/openpype/hosts/fusion/utility_scripts/32bit/loaders_to32bit.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/loaders_to32bit.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/32bit/loaders_to32bit.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/32bit/loaders_to32bit.py diff --git a/openpype/hosts/fusion/utility_scripts/switch_ui.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/switch_ui.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/switch_ui.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/switch_ui.py diff --git a/openpype/hosts/fusion/utility_scripts/update_loader_ranges.py b/openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/update_loader_ranges.py similarity index 100% rename from openpype/hosts/fusion/utility_scripts/update_loader_ranges.py rename to openpype/hosts/fusion/deploy/Scripts/Comp/OpenPype/update_loader_ranges.py diff --git a/openpype/hosts/fusion/deploy/fusion_shared.prefs b/openpype/hosts/fusion/deploy/fusion_shared.prefs new file mode 100644 index 0000000000..998c6a6d66 --- /dev/null +++ b/openpype/hosts/fusion/deploy/fusion_shared.prefs @@ -0,0 +1,19 @@ +{ +Locked = true, +Global = { + Paths = { + Map = { + ["OpenPype:"] = "$(OPENPYPE_FUSION)/deploy", + ["Reactor:"] = "$(REACTOR)", + + ["Config:"] = "UserPaths:Config;OpenPype:Config", + ["Scripts:"] = "UserPaths:Scripts;Reactor:System/Scripts;OpenPype:Scripts", + ["UserPaths:"] = "UserData:;AllData:;Fusion:;Reactor:Deploy" + }, + }, + Script = { + PythonVersion = 3, + Python3Forced = true + }, + }, +} \ No newline at end of file diff --git a/openpype/hosts/fusion/hooks/pre_fusion_ocio_hook.py b/openpype/hosts/fusion/hooks/pre_fusion_ocio_hook.py new file mode 100644 index 0000000000..12fc640f5c --- /dev/null +++ b/openpype/hosts/fusion/hooks/pre_fusion_ocio_hook.py @@ -0,0 +1,40 @@ +import os +import platform + +from openpype.lib import PreLaunchHook, ApplicationLaunchFailed + + +class FusionPreLaunchOCIO(PreLaunchHook): + """Set OCIO environment variable for Fusion""" + app_groups = ["fusion"] + + def execute(self): + """Hook entry method.""" + + # get image io + project_settings = self.data["project_settings"] + + # make sure anatomy settings are having flame key + imageio_fusion = project_settings.get("fusion", {}).get("imageio") + if not imageio_fusion: + raise ApplicationLaunchFailed(( + "Anatomy project settings are missing `fusion` key. " + "Please make sure you remove project overrides on " + "Anatomy ImageIO") + ) + + ocio = imageio_fusion.get("ocio") + enabled = ocio.get("enabled", False) + if not enabled: + return + + platform_key = platform.system().lower() + ocio_path = ocio["configFilePath"][platform_key] + if not ocio_path: + raise ApplicationLaunchFailed( + "Fusion OCIO is enabled in project settings but no OCIO config" + f"path is set for your current platform: {platform_key}" + ) + + self.log.info(f"Setting OCIO config path: {ocio_path}") + self.launch_context.env["OCIO"] = os.pathsep.join(ocio_path) diff --git a/openpype/hosts/fusion/hooks/pre_fusion_setup.py b/openpype/hosts/fusion/hooks/pre_fusion_setup.py index e635a0ea74..d043d54322 100644 --- a/openpype/hosts/fusion/hooks/pre_fusion_setup.py +++ b/openpype/hosts/fusion/hooks/pre_fusion_setup.py @@ -1,114 +1,61 @@ import os -import shutil - -import openpype.hosts.fusion from openpype.lib import PreLaunchHook, ApplicationLaunchFailed +from openpype.hosts.fusion import FUSION_HOST_DIR class FusionPrelaunch(PreLaunchHook): - """ - This hook will check if current workfile path has Fusion - project inside. + """Prepares OpenPype Fusion environment + + Requires FUSION_PYTHON3_HOME to be defined in the environment for Fusion + to point at a valid Python 3 build for Fusion. That is Python 3.3-3.10 + for Fusion 18 and Fusion 3.6 for Fusion 16 and 17. + + This also sets FUSION16_MasterPrefs to apply the fusion master prefs + as set in openpype/hosts/fusion/deploy/fusion_shared.prefs to enable + the OpenPype menu and force Python 3 over Python 2. + """ app_groups = ["fusion"] def execute(self): - # making sure python 3.6 is installed at provided path - py36_dir = self.launch_context.env.get("PYTHON36") - if not py36_dir: + # making sure python 3 is installed at provided path + # Py 3.3-3.10 for Fusion 18+ or Py 3.6 for Fu 16-17 + py3_var = "FUSION_PYTHON3_HOME" + fusion_python3_home = self.launch_context.env.get(py3_var, "") + + self.log.info(f"Looking for Python 3 in: {fusion_python3_home}") + for path in fusion_python3_home.split(os.pathsep): + # Allow defining multiple paths to allow "fallback" to other + # path. But make to set only a single path as final variable. + py3_dir = os.path.normpath(path) + if os.path.isdir(py3_dir): + break + else: raise ApplicationLaunchFailed( - "Required environment variable \"PYTHON36\" is not set." - "\n\nFusion implementation requires to have" - " installed Python 3.6" + "Python 3 is not installed at the provided path.\n" + "Make sure the environment in fusion settings has " + "'FUSION_PYTHON3_HOME' set correctly and make sure " + "Python 3 is installed in the given path." + f"\n\nPYTHON36: {fusion_python3_home}" ) - py36_dir = os.path.normpath(py36_dir) - if not os.path.isdir(py36_dir): - raise ApplicationLaunchFailed( - "Python 3.6 is not installed at the provided path.\n" - "Either make sure the environments in fusion settings has" - " 'PYTHON36' set corectly or make sure Python 3.6 is installed" - f" in the given path.\n\nPYTHON36: {py36_dir}" - ) - self.log.info(f"Path to Fusion Python folder: '{py36_dir}'...") - self.launch_context.env["PYTHON36"] = py36_dir + self.log.info(f"Setting {py3_var}: '{py3_dir}'...") + self.launch_context.env[py3_var] = py3_dir - utility_dir = self.launch_context.env.get("FUSION_UTILITY_SCRIPTS_DIR") - if not utility_dir: - raise ApplicationLaunchFailed( - "Required Fusion utility script dir environment variable" - " \"FUSION_UTILITY_SCRIPTS_DIR\" is not set." - ) + # Fusion 18+ requires FUSION_PYTHON3_HOME to also be on PATH + self.launch_context.env["PATH"] += ";" + py3_dir - # setting utility scripts dir for scripts syncing - utility_dir = os.path.normpath(utility_dir) - if not os.path.isdir(utility_dir): - raise ApplicationLaunchFailed( - "Fusion utility script dir does not exist. Either make sure " - "the environments in fusion settings has" - " 'FUSION_UTILITY_SCRIPTS_DIR' set correctly or reinstall " - f"Fusion.\n\nFUSION_UTILITY_SCRIPTS_DIR: '{utility_dir}'" - ) + # Fusion 16 and 17 use FUSION16_PYTHON36_HOME instead of + # FUSION_PYTHON3_HOME and will only work with a Python 3.6 version + # TODO: Detect Fusion version to only set for specific Fusion build + self.launch_context.env["FUSION16_PYTHON36_HOME"] = py3_dir - self._sync_utility_scripts(self.launch_context.env) - self.log.info("Fusion Pype wrapper has been installed") + # Add our Fusion Master Prefs which is the only way to customize + # Fusion to define where it can read custom scripts and tools from + self.log.info(f"Setting OPENPYPE_FUSION: {FUSION_HOST_DIR}") + self.launch_context.env["OPENPYPE_FUSION"] = FUSION_HOST_DIR - def _sync_utility_scripts(self, env): - """ Synchronizing basic utlility scripts for resolve. - - To be able to run scripts from inside `Fusion/Workspace/Scripts` menu - all scripts has to be accessible from defined folder. - """ - if not env: - env = {k: v for k, v in os.environ.items()} - - # initiate inputs - scripts = {} - us_env = env.get("FUSION_UTILITY_SCRIPTS_SOURCE_DIR") - us_dir = env.get("FUSION_UTILITY_SCRIPTS_DIR", "") - us_paths = [os.path.join( - os.path.dirname(os.path.abspath(openpype.hosts.fusion.__file__)), - "utility_scripts" - )] - - # collect script dirs - if us_env: - self.log.info(f"Utility Scripts Env: `{us_env}`") - us_paths = us_env.split( - os.pathsep) + us_paths - - # collect scripts from dirs - for path in us_paths: - scripts.update({path: os.listdir(path)}) - - self.log.info(f"Utility Scripts Dir: `{us_paths}`") - self.log.info(f"Utility Scripts: `{scripts}`") - - # make sure no script file is in folder - if next((s for s in os.listdir(us_dir)), None): - for s in os.listdir(us_dir): - path = os.path.normpath( - os.path.join(us_dir, s)) - self.log.info(f"Removing `{path}`...") - - # remove file or directory if not in our folders - if not os.path.isdir(path): - os.remove(path) - else: - shutil.rmtree(path) - - # copy scripts into Resolve's utility scripts dir - for d, sl in scripts.items(): - # directory and scripts list - for s in sl: - # script in script list - src = os.path.normpath(os.path.join(d, s)) - dst = os.path.normpath(os.path.join(us_dir, s)) - - self.log.info(f"Copying `{src}` to `{dst}`...") - - # copy file or directory from our folders to fusion's folder - if not os.path.isdir(src): - shutil.copy2(src, dst) - else: - shutil.copytree(src, dst) + pref_var = "FUSION16_MasterPrefs" # used by Fusion 16, 17 and 18 + prefs = os.path.join(FUSION_HOST_DIR, "deploy", "fusion_shared.prefs") + self.log.info(f"Setting {pref_var}: {prefs}") + self.launch_context.env[pref_var] = prefs diff --git a/openpype/hosts/fusion/plugins/create/create_exr_saver.py b/openpype/hosts/fusion/plugins/create/create_exr_saver.py index 8bab5ee9b1..6d93fe710a 100644 --- a/openpype/hosts/fusion/plugins/create/create_exr_saver.py +++ b/openpype/hosts/fusion/plugins/create/create_exr_saver.py @@ -1,6 +1,9 @@ import os -from openpype.pipeline import LegacyCreator +from openpype.pipeline import ( + LegacyCreator, + legacy_io +) from openpype.hosts.fusion.api import ( get_current_comp, comp_lock_and_undo_chunk @@ -21,12 +24,9 @@ class CreateOpenEXRSaver(LegacyCreator): comp = get_current_comp() - # todo: improve method of getting current environment - # todo: pref avalon.Session over os.environ + workdir = os.path.normpath(legacy_io.Session["AVALON_WORKDIR"]) - workdir = os.path.normpath(os.environ["AVALON_WORKDIR"]) - - filename = "{}..tiff".format(self.name) + filename = "{}..exr".format(self.name) filepath = os.path.join(workdir, "render", filename) with comp_lock_and_undo_chunk(comp): @@ -39,10 +39,10 @@ class CreateOpenEXRSaver(LegacyCreator): saver["Clip"] = filepath saver["OutputFormat"] = file_format - # # # Set standard TIFF settings + # Check file format settings are available if saver[file_format] is None: - raise RuntimeError("File format is not set to TiffFormat, " - "this is a bug") + raise RuntimeError("File format is not set to {}, " + "this is a bug".format(file_format)) # Set file format attributes saver[file_format]["Depth"] = 1 # int8 | int16 | float32 | other diff --git a/openpype/hosts/fusion/plugins/load/load_sequence.py b/openpype/hosts/fusion/plugins/load/load_sequence.py index abd0f4e411..6f44c61d1b 100644 --- a/openpype/hosts/fusion/plugins/load/load_sequence.py +++ b/openpype/hosts/fusion/plugins/load/load_sequence.py @@ -101,6 +101,9 @@ def loader_shift(loader, frame, relative=True): else: shift = frame - old_in + if not shift: + return 0 + # Shifting global in will try to automatically compensate for the change # in the "ClipTimeStart" and "HoldFirstFrame" inputs, so we preserve those # input values to "just shift" the clip @@ -149,9 +152,8 @@ class FusionLoadSequence(load.LoaderPlugin): tool["Clip"] = path # Set global in point to start frame (if in version.data) - start = context["version"]["data"].get("frameStart", None) - if start is not None: - loader_shift(tool, start, relative=False) + start = self._get_start(context["version"], tool) + loader_shift(tool, start, relative=False) imprint_container(tool, name=name, @@ -214,12 +216,7 @@ class FusionLoadSequence(load.LoaderPlugin): # Get start frame from version data project_name = legacy_io.active_project() version = get_version_by_id(project_name, representation["parent"]) - start = version["data"].get("frameStart") - if start is None: - self.log.warning("Missing start frame for updated version" - "assuming starts at frame 0 for: " - "{} ({})".format(tool.Name, representation)) - start = 0 + start = self._get_start(version, tool) with comp_lock_and_undo_chunk(comp, "Update Loader"): @@ -256,3 +253,27 @@ class FusionLoadSequence(load.LoaderPlugin): """Get first file in representation root""" files = sorted(os.listdir(root)) return os.path.join(root, files[0]) + + def _get_start(self, version_doc, tool): + """Return real start frame of published files (incl. handles)""" + data = version_doc["data"] + + # Get start frame directly with handle if it's in data + start = data.get("frameStartHandle") + if start is not None: + return start + + # Get frame start without handles + start = data.get("frameStart") + if start is None: + self.log.warning("Missing start frame for version " + "assuming starts at frame 0 for: " + "{}".format(tool.Name)) + return 0 + + # Use `handleStart` if the data is available + handle_start = data.get("handleStart") + if handle_start: + start -= handle_start + + return start diff --git a/openpype/hosts/fusion/plugins/publish/collect_inputs.py b/openpype/hosts/fusion/plugins/publish/collect_inputs.py new file mode 100644 index 0000000000..8f9857b02f --- /dev/null +++ b/openpype/hosts/fusion/plugins/publish/collect_inputs.py @@ -0,0 +1,114 @@ +from bson.objectid import ObjectId + +import pyblish.api + +from openpype.pipeline import registered_host + + +def collect_input_containers(tools): + """Collect containers that contain any of the node in `nodes`. + + This will return any loaded Avalon container that contains at least one of + the nodes. As such, the Avalon container is an input for it. Or in short, + there are member nodes of that container. + + Returns: + list: Input avalon containers + + """ + + # Lookup by node ids + lookup = frozenset([tool.Name for tool in tools]) + + containers = [] + host = registered_host() + for container in host.ls(): + + name = container["_tool"].Name + + # We currently assume no "groups" as containers but just single tools + # like a single "Loader" operator. As such we just check whether the + # Loader is part of the processing queue. + if name in lookup: + containers.append(container) + + return containers + + +def iter_upstream(tool): + """Yields all upstream inputs for the current tool. + + Yields: + tool: The input tools. + + """ + + def get_connected_input_tools(tool): + """Helper function that returns connected input tools for a tool.""" + inputs = [] + + # Filter only to actual types that will have sensible upstream + # connections. So we ignore just "Number" inputs as they can be + # many to iterate, slowing things down quite a bit - and in practice + # they don't have upstream connections. + VALID_INPUT_TYPES = ['Image', 'Particles', 'Mask', 'DataType3D'] + for type_ in VALID_INPUT_TYPES: + for input_ in tool.GetInputList(type_).values(): + output = input_.GetConnectedOutput() + if output: + input_tool = output.GetTool() + inputs.append(input_tool) + + return inputs + + # Initialize process queue with the node's inputs itself + queue = get_connected_input_tools(tool) + + # We keep track of which node names we have processed so far, to ensure we + # don't process the same hierarchy again. We are not pushing the tool + # itself into the set as that doesn't correctly recognize the same tool. + # Since tool names are unique in a comp in Fusion we rely on that. + collected = set(tool.Name for tool in queue) + + # Traverse upstream references for all nodes and yield them as we + # process the queue. + while queue: + upstream_tool = queue.pop() + yield upstream_tool + + # Find upstream tools that are not collected yet. + upstream_inputs = get_connected_input_tools(upstream_tool) + upstream_inputs = [t for t in upstream_inputs if + t.Name not in collected] + + queue.extend(upstream_inputs) + collected.update(tool.Name for tool in upstream_inputs) + + +class CollectUpstreamInputs(pyblish.api.InstancePlugin): + """Collect source input containers used for this publish. + + This will include `inputs` data of which loaded publishes were used in the + generation of this publish. This leaves an upstream trace to what was used + as input. + + """ + + label = "Collect Inputs" + order = pyblish.api.CollectorOrder + 0.2 + hosts = ["fusion"] + + def process(self, instance): + + # Get all upstream and include itself + tool = instance[0] + nodes = list(iter_upstream(tool)) + nodes.append(tool) + + # Collect containers for the given set of nodes + containers = collect_input_containers(nodes) + + inputs = [ObjectId(c["representation"]) for c in containers] + instance.data["inputRepresentations"] = inputs + + self.log.info("Collected inputs: %s" % inputs) diff --git a/openpype/hosts/fusion/plugins/publish/collect_instances.py b/openpype/hosts/fusion/plugins/publish/collect_instances.py index b2192d1dd9..fe60b83827 100644 --- a/openpype/hosts/fusion/plugins/publish/collect_instances.py +++ b/openpype/hosts/fusion/plugins/publish/collect_instances.py @@ -4,19 +4,21 @@ import pyblish.api def get_comp_render_range(comp): - """Return comp's start and end render range.""" + """Return comp's start-end render range and global start-end range.""" comp_attrs = comp.GetAttrs() start = comp_attrs["COMPN_RenderStart"] end = comp_attrs["COMPN_RenderEnd"] + global_start = comp_attrs["COMPN_GlobalStart"] + global_end = comp_attrs["COMPN_GlobalEnd"] # Whenever render ranges are undefined fall back # to the comp's global start and end if start == -1000000000: - start = comp_attrs["COMPN_GlobalEnd"] + start = global_start if end == -1000000000: - end = comp_attrs["COMPN_GlobalStart"] + end = global_end - return start, end + return start, end, global_start, global_end class CollectInstances(pyblish.api.ContextPlugin): @@ -42,9 +44,11 @@ class CollectInstances(pyblish.api.ContextPlugin): tools = comp.GetToolList(False).values() savers = [tool for tool in tools if tool.ID == "Saver"] - start, end = get_comp_render_range(comp) + start, end, global_start, global_end = get_comp_render_range(comp) context.data["frameStart"] = int(start) context.data["frameEnd"] = int(end) + context.data["frameStartHandle"] = int(global_start) + context.data["frameEndHandle"] = int(global_end) for tool in savers: path = tool["Clip"][comp.TIME_UNDEFINED] @@ -78,8 +82,10 @@ class CollectInstances(pyblish.api.ContextPlugin): "label": label, "frameStart": context.data["frameStart"], "frameEnd": context.data["frameEnd"], + "frameStartHandle": context.data["frameStartHandle"], + "frameEndHandle": context.data["frameStartHandle"], "fps": context.data["fps"], - "families": ["render", "review", "ftrack"], + "families": ["render", "review"], "family": "render", "active": active, "publish": active # backwards compatibility diff --git a/openpype/hosts/fusion/plugins/publish/render_local.py b/openpype/hosts/fusion/plugins/publish/render_local.py index 601c2ffccf..79e458b40a 100644 --- a/openpype/hosts/fusion/plugins/publish/render_local.py +++ b/openpype/hosts/fusion/plugins/publish/render_local.py @@ -20,6 +20,8 @@ class Fusionlocal(pyblish.api.InstancePlugin): def process(self, instance): + # This plug-in runs only once and thus assumes all instances + # currently will render the same frame range context = instance.context key = "__hasRun{}".format(self.__class__.__name__) if context.data.get(key, False): @@ -28,8 +30,8 @@ class Fusionlocal(pyblish.api.InstancePlugin): context.data[key] = True current_comp = context.data["currentComp"] - frame_start = current_comp.GetAttrs("COMPN_RenderStart") - frame_end = current_comp.GetAttrs("COMPN_RenderEnd") + frame_start = context.data["frameStartHandle"] + frame_end = context.data["frameEndHandle"] path = instance.data["path"] output_dir = instance.data["outputDir"] @@ -40,7 +42,11 @@ class Fusionlocal(pyblish.api.InstancePlugin): self.log.info("End frame: {}".format(frame_end)) with comp_lock_and_undo_chunk(current_comp): - result = current_comp.Render() + result = current_comp.Render({ + "Start": frame_start, + "End": frame_end, + "Wait": True + }) if "representations" not in instance.data: instance.data["representations"] = [] diff --git a/openpype/hosts/fusion/scripts/fusion_switch_shot.py b/openpype/hosts/fusion/scripts/fusion_switch_shot.py deleted file mode 100644 index 49ef340679..0000000000 --- a/openpype/hosts/fusion/scripts/fusion_switch_shot.py +++ /dev/null @@ -1,284 +0,0 @@ -import os -import re -import sys -import logging - -from openpype.client import ( - get_asset_by_name, - get_versions, -) -from openpype.pipeline import ( - legacy_io, - install_host, - registered_host, -) -from openpype.lib import version_up -from openpype.hosts.fusion import api -from openpype.hosts.fusion.api import lib -from openpype.pipeline.context_tools import get_workdir_from_session - -log = logging.getLogger("Update Slap Comp") - - -def _format_version_folder(folder): - """Format a version folder based on the filepath - - Assumption here is made that, if the path does not exists the folder - will be "v001" - - Args: - folder: file path to a folder - - Returns: - str: new version folder name - """ - - new_version = 1 - if os.path.isdir(folder): - re_version = re.compile(r"v\d+$") - versions = [i for i in os.listdir(folder) if os.path.isdir(i) - and re_version.match(i)] - if versions: - # ensure the "v" is not included - new_version = int(max(versions)[1:]) + 1 - - version_folder = "v{:03d}".format(new_version) - - return version_folder - - -def _get_fusion_instance(): - fusion = getattr(sys.modules["__main__"], "fusion", None) - if fusion is None: - try: - # Support for FuScript.exe, BlackmagicFusion module for py2 only - import BlackmagicFusion as bmf - fusion = bmf.scriptapp("Fusion") - except ImportError: - raise RuntimeError("Could not find a Fusion instance") - return fusion - - -def _format_filepath(session): - - project = session["AVALON_PROJECT"] - asset = session["AVALON_ASSET"] - - # Save updated slap comp - work_path = get_workdir_from_session(session) - walk_to_dir = os.path.join(work_path, "scenes", "slapcomp") - slapcomp_dir = os.path.abspath(walk_to_dir) - - # Ensure destination exists - if not os.path.isdir(slapcomp_dir): - log.warning("Folder did not exist, creating folder structure") - os.makedirs(slapcomp_dir) - - # Compute output path - new_filename = "{}_{}_slapcomp_v001.comp".format(project, asset) - new_filepath = os.path.join(slapcomp_dir, new_filename) - - # Create new unique filepath - if os.path.exists(new_filepath): - new_filepath = version_up(new_filepath) - - return new_filepath - - -def _update_savers(comp, session): - """Update all savers of the current comp to ensure the output is correct - - This will refactor the Saver file outputs to the renders of the new session - that is provided. - - In the case the original saver path had a path set relative to a /fusion/ - folder then that relative path will be matched with the exception of all - "version" (e.g. v010) references will be reset to v001. Otherwise only a - version folder will be computed in the new session's work "render" folder - to dump the files in and keeping the original filenames. - - Args: - comp (object): current comp instance - session (dict): the current Avalon session - - Returns: - None - """ - - new_work = get_workdir_from_session(session) - renders = os.path.join(new_work, "renders") - version_folder = _format_version_folder(renders) - renders_version = os.path.join(renders, version_folder) - - comp.Print("New renders to: %s\n" % renders) - - with api.comp_lock_and_undo_chunk(comp): - savers = comp.GetToolList(False, "Saver").values() - for saver in savers: - filepath = saver.GetAttrs("TOOLST_Clip_Name")[1.0] - - # Get old relative path to the "fusion" app folder so we can apply - # the same relative path afterwards. If not found fall back to - # using just a version folder with the filename in it. - # todo: can we make this less magical? - relpath = filepath.replace("\\", "/").rsplit("/fusion/", 1)[-1] - - if os.path.isabs(relpath): - # If not relative to a "/fusion/" folder then just use filename - filename = os.path.basename(filepath) - log.warning("Can't parse relative path, refactoring to only" - "filename in a version folder: %s" % filename) - new_path = os.path.join(renders_version, filename) - - else: - # Else reuse the relative path - # Reset version in folder and filename in the relative path - # to v001. The version should be is only detected when prefixed - # with either `_v` (underscore) or `/v` (folder) - version_pattern = r"(/|_)v[0-9]+" - if re.search(version_pattern, relpath): - new_relpath = re.sub(version_pattern, - r"\1v001", - relpath) - log.info("Resetting version folders to v001: " - "%s -> %s" % (relpath, new_relpath)) - relpath = new_relpath - - new_path = os.path.join(new_work, relpath) - - saver["Clip"] = new_path - - -def update_frame_range(comp, representations): - """Update the frame range of the comp and render length - - The start and end frame are based on the lowest start frame and the highest - end frame - - Args: - comp (object): current focused comp - representations (list) collection of dicts - - Returns: - None - - """ - - project_name = legacy_io.active_project() - version_ids = {r["parent"] for r in representations} - versions = list(get_versions(project_name, version_ids)) - - versions = [v for v in versions - if v["data"].get("frameStart", None) is not None] - - if not versions: - log.warning("No versions loaded to match frame range to.\n") - return - - start = min(v["data"]["frameStart"] for v in versions) - end = max(v["data"]["frameEnd"] for v in versions) - - lib.update_frame_range(start, end, comp=comp) - - -def switch(asset_name, filepath=None, new=True): - """Switch the current containers of the file to the other asset (shot) - - Args: - filepath (str): file path of the comp file - asset_name (str): name of the asset (shot) - new (bool): Save updated comp under a different name - - Returns: - comp path (str): new filepath of the updated comp - - """ - - # If filepath provided, ensure it is valid absolute path - if filepath is not None: - if not os.path.isabs(filepath): - filepath = os.path.abspath(filepath) - - assert os.path.exists(filepath), "%s must exist " % filepath - - # Assert asset name exists - # It is better to do this here then to wait till switch_shot does it - project_name = legacy_io.active_project() - asset = get_asset_by_name(project_name, asset_name) - assert asset, "Could not find '%s' in the database" % asset_name - - # Go to comp - if not filepath: - current_comp = api.get_current_comp() - assert current_comp is not None, "Could not find current comp" - else: - fusion = _get_fusion_instance() - current_comp = fusion.LoadComp(filepath, quiet=True) - assert current_comp is not None, ( - "Fusion could not load '{}'").format(filepath) - - host = registered_host() - containers = list(host.ls()) - assert containers, "Nothing to update" - - representations = [] - for container in containers: - try: - representation = lib.switch_item( - container, - asset_name=asset_name) - representations.append(representation) - except Exception as e: - current_comp.Print("Error in switching! %s\n" % e.message) - - message = "Switched %i Loaders of the %i\n" % (len(representations), - len(containers)) - current_comp.Print(message) - - # Build the session to switch to - switch_to_session = legacy_io.Session.copy() - switch_to_session["AVALON_ASSET"] = asset['name'] - - if new: - comp_path = _format_filepath(switch_to_session) - - # Update savers output based on new session - _update_savers(current_comp, switch_to_session) - else: - comp_path = version_up(filepath) - - current_comp.Print(comp_path) - - current_comp.Print("\nUpdating frame range") - update_frame_range(current_comp, representations) - - current_comp.Save(comp_path) - - return comp_path - - -if __name__ == '__main__': - - # QUESTION: can we convert this to gui rather then standalone script? - # TODO: convert to gui tool - import argparse - - parser = argparse.ArgumentParser(description="Switch to a shot within an" - "existing comp file") - - parser.add_argument("--file_path", - type=str, - default=True, - help="File path of the comp to use") - - parser.add_argument("--asset_name", - type=str, - default=True, - help="Name of the asset (shot) to switch") - - args, unknown = parser.parse_args() - - install_host(api) - switch(args.asset_name, args.file_path) - - sys.exit(0) diff --git a/openpype/hosts/harmony/plugins/publish/extract_palette.py b/openpype/hosts/harmony/plugins/publish/extract_palette.py index fae778f6b0..69c6e098ff 100644 --- a/openpype/hosts/harmony/plugins/publish/extract_palette.py +++ b/openpype/hosts/harmony/plugins/publish/extract_palette.py @@ -6,10 +6,10 @@ import csv from PIL import Image, ImageDraw, ImageFont import openpype.hosts.harmony.api as harmony -import openpype.api +from openpype.pipeline import publish -class ExtractPalette(openpype.api.Extractor): +class ExtractPalette(publish.Extractor): """Extract palette.""" label = "Extract Palette" diff --git a/openpype/hosts/harmony/plugins/publish/extract_template.py b/openpype/hosts/harmony/plugins/publish/extract_template.py index d25b07bba3..458bf25a3c 100644 --- a/openpype/hosts/harmony/plugins/publish/extract_template.py +++ b/openpype/hosts/harmony/plugins/publish/extract_template.py @@ -3,12 +3,11 @@ import os import shutil -import openpype.api +from openpype.pipeline import publish import openpype.hosts.harmony.api as harmony -import openpype.hosts.harmony -class ExtractTemplate(openpype.api.Extractor): +class ExtractTemplate(publish.Extractor): """Extract the connected nodes to the composite instance.""" label = "Extract Template" @@ -50,7 +49,7 @@ class ExtractTemplate(openpype.api.Extractor): dependencies.remove(instance.data["setMembers"][0]) # Export template. - openpype.hosts.harmony.api.export_template( + harmony.export_template( unique_backdrops, dependencies, filepath ) diff --git a/openpype/hosts/harmony/plugins/publish/extract_workfile.py b/openpype/hosts/harmony/plugins/publish/extract_workfile.py index 7f25ec8150..9bb3090558 100644 --- a/openpype/hosts/harmony/plugins/publish/extract_workfile.py +++ b/openpype/hosts/harmony/plugins/publish/extract_workfile.py @@ -4,10 +4,10 @@ import os import shutil from zipfile import ZipFile -import openpype.api +from openpype.pipeline import publish -class ExtractWorkfile(openpype.api.Extractor): +class ExtractWorkfile(publish.Extractor): """Extract and zip complete workfile folder into zip.""" label = "Extract Workfile" diff --git a/openpype/hosts/hiero/api/lib.py b/openpype/hosts/hiero/api/lib.py index e288cea2b1..895e95e0c0 100644 --- a/openpype/hosts/hiero/api/lib.py +++ b/openpype/hosts/hiero/api/lib.py @@ -13,14 +13,10 @@ import hiero from Qt import QtWidgets -from openpype.client import ( - get_project, - get_versions, - get_last_versions, - get_representations, -) +from openpype.client import get_project from openpype.settings import get_anatomy_settings from openpype.pipeline import legacy_io, Anatomy +from openpype.pipeline.load import filter_containers from openpype.lib import Logger from . import tags @@ -1055,6 +1051,10 @@ def sync_clip_name_to_data_asset(track_items_list): print("asset was changed in clip: {}".format(ti_name)) +def set_track_color(track_item, color): + track_item.source().binItem().setColor(color) + + def check_inventory_versions(track_items=None): """ Actual version color idetifier of Loaded containers @@ -1066,68 +1066,29 @@ def check_inventory_versions(track_items=None): """ from . import parse_container - track_item = track_items or get_track_items() + track_items = track_items or get_track_items() # presets clip_color_last = "green" clip_color = "red" - item_with_repre_id = [] - repre_ids = set() + containers = [] # Find all containers and collect it's node and representation ids - for track_item in track_item: + for track_item in track_items: container = parse_container(track_item) if container: - repre_id = container["representation"] - repre_ids.add(repre_id) - item_with_repre_id.append((track_item, repre_id)) + containers.append(container) # Skip if nothing was found - if not repre_ids: + if not containers: return project_name = legacy_io.active_project() - # Find representations based on found containers - repre_docs = get_representations( - project_name, - repre_ids=repre_ids, - fields=["_id", "parent"] - ) - # Store representations by id and collect version ids - repre_docs_by_id = {} - version_ids = set() - for repre_doc in repre_docs: - # Use stringed representation id to match value in containers - repre_id = str(repre_doc["_id"]) - repre_docs_by_id[repre_id] = repre_doc - version_ids.add(repre_doc["parent"]) + filter_result = filter_containers(containers, project_name) + for container in filter_result.latest: + set_track_color(container["_track_item"], clip_color) - version_docs = get_versions( - project_name, version_ids, fields=["_id", "name", "parent"] - ) - # Store versions by id and collect subset ids - version_docs_by_id = {} - subset_ids = set() - for version_doc in version_docs: - version_docs_by_id[version_doc["_id"]] = version_doc - subset_ids.add(version_doc["parent"]) - - # Query last versions based on subset ids - last_versions_by_subset_id = get_last_versions( - project_name, subset_ids=subset_ids, fields=["_id", "parent"] - ) - - for item in item_with_repre_id: - # Some python versions of nuke can't unfold tuple in for loop - track_item, repre_id = item - - repre_doc = repre_docs_by_id[repre_id] - version_doc = version_docs_by_id[repre_doc["parent"]] - last_version_doc = last_versions_by_subset_id[version_doc["parent"]] - # Check if last version is same as current version - if version_doc["_id"] == last_version_doc["_id"]: - track_item.source().binItem().setColor(clip_color_last) - else: - track_item.source().binItem().setColor(clip_color) + for container in filter_result.outdated: + set_track_color(container["_track_item"], clip_color_last) def selection_changed_timeline(event): diff --git a/openpype/hosts/hiero/plugins/publish/extract_clip_effects.py b/openpype/hosts/hiero/plugins/publish/extract_clip_effects.py index 5b0aa270a7..7fb381ff7e 100644 --- a/openpype/hosts/hiero/plugins/publish/extract_clip_effects.py +++ b/openpype/hosts/hiero/plugins/publish/extract_clip_effects.py @@ -2,10 +2,11 @@ import os import json import pyblish.api -import openpype + +from openpype.pipeline import publish -class ExtractClipEffects(openpype.api.Extractor): +class ExtractClipEffects(publish.Extractor): """Extract clip effects instances.""" order = pyblish.api.ExtractorOrder diff --git a/openpype/hosts/hiero/plugins/publish/extract_frames.py b/openpype/hosts/hiero/plugins/publish/extract_frames.py index aa3eda2e9f..f865d2fb39 100644 --- a/openpype/hosts/hiero/plugins/publish/extract_frames.py +++ b/openpype/hosts/hiero/plugins/publish/extract_frames.py @@ -1,9 +1,14 @@ import os import pyblish.api -import openpype + +from openpype.lib import ( + get_oiio_tools_path, + run_subprocess, +) +from openpype.pipeline import publish -class ExtractFrames(openpype.api.Extractor): +class ExtractFrames(publish.Extractor): """Extracts frames""" order = pyblish.api.ExtractorOrder @@ -13,7 +18,7 @@ class ExtractFrames(openpype.api.Extractor): movie_extensions = ["mov", "mp4"] def process(self, instance): - oiio_tool_path = openpype.lib.get_oiio_tools_path() + oiio_tool_path = get_oiio_tools_path() staging_dir = self.staging_dir(instance) output_template = os.path.join(staging_dir, instance.data["name"]) sequence = instance.context.data["activeTimeline"] @@ -43,7 +48,7 @@ class ExtractFrames(openpype.api.Extractor): args.extend(["--powc", "0.45,0.45,0.45,1.0"]) args.extend([input_path, "-o", output_path]) - output = openpype.api.run_subprocess(args) + output = run_subprocess(args) failed_output = "oiiotool produced no output." if failed_output in output: diff --git a/openpype/hosts/hiero/plugins/publish/extract_thumbnail.py b/openpype/hosts/hiero/plugins/publish/extract_thumbnail.py index d12e7665bf..e64aa89b26 100644 --- a/openpype/hosts/hiero/plugins/publish/extract_thumbnail.py +++ b/openpype/hosts/hiero/plugins/publish/extract_thumbnail.py @@ -1,9 +1,10 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish -class ExtractThumnail(openpype.api.Extractor): +class ExtractThumnail(publish.Extractor): """ Extractor for track item's tumnails """ diff --git a/openpype/hosts/hiero/plugins/publish/precollect_instances.py b/openpype/hosts/hiero/plugins/publish/precollect_instances.py index 0c7dbc1f22..84f2927fc7 100644 --- a/openpype/hosts/hiero/plugins/publish/precollect_instances.py +++ b/openpype/hosts/hiero/plugins/publish/precollect_instances.py @@ -318,10 +318,9 @@ class PrecollectInstances(pyblish.api.ContextPlugin): @staticmethod def create_otio_time_range_from_timeline_item_data(track_item): - speed = track_item.playbackSpeed() timeline = phiero.get_current_sequence() frame_start = int(track_item.timelineIn()) - frame_duration = int((track_item.duration() - 1) / speed) + frame_duration = int(track_item.duration()) fps = timeline.framerate().toFloat() return hiero_export.create_otio_time_range( diff --git a/openpype/hosts/houdini/api/pipeline.py b/openpype/hosts/houdini/api/pipeline.py index 2ae8a4dbf7..e4af1913ef 100644 --- a/openpype/hosts/houdini/api/pipeline.py +++ b/openpype/hosts/houdini/api/pipeline.py @@ -14,7 +14,7 @@ from openpype.pipeline import ( ) from openpype.pipeline.load import any_outdated_containers from openpype.hosts.houdini import HOUDINI_HOST_DIR -from openpype.hosts.houdini.api import lib +from openpype.hosts.houdini.api import lib, shelves from openpype.lib import ( register_event_callback, @@ -73,6 +73,7 @@ def install(): # so it initializes into the correct scene FPS, Frame Range, etc. # todo: make sure this doesn't trigger when opening with last workfile _set_context_settings() + shelves.generate_shelves() def uninstall(): diff --git a/openpype/hosts/houdini/api/shelves.py b/openpype/hosts/houdini/api/shelves.py new file mode 100644 index 0000000000..248d99105c --- /dev/null +++ b/openpype/hosts/houdini/api/shelves.py @@ -0,0 +1,204 @@ +import os +import logging +import platform +import six + +from openpype.settings import get_project_settings + +import hou + +log = logging.getLogger("openpype.hosts.houdini.shelves") + +if six.PY2: + FileNotFoundError = IOError + + +def generate_shelves(): + """This function generates complete shelves from shelf set to tools + in Houdini from openpype project settings houdini shelf definition. + + Raises: + FileNotFoundError: Raised when the shelf set filepath does not exist + """ + current_os = platform.system().lower() + + # load configuration of houdini shelves + project_settings = get_project_settings(os.getenv("AVALON_PROJECT")) + shelves_set_config = project_settings["houdini"]["shelves"] + + if not shelves_set_config: + log.debug( + "No custom shelves found in project settings." + ) + return + + for shelf_set_config in shelves_set_config: + shelf_set_filepath = shelf_set_config.get('shelf_set_source_path') + + if shelf_set_filepath[current_os]: + if not os.path.isfile(shelf_set_filepath[current_os]): + raise FileNotFoundError( + "This path doesn't exist - {}".format( + shelf_set_filepath[current_os] + ) + ) + + hou.shelves.newShelfSet(file_path=shelf_set_filepath[current_os]) + continue + + shelf_set_name = shelf_set_config.get('shelf_set_name') + if not shelf_set_name: + log.warning( + "No name found in shelf set definition." + ) + return + + shelf_set = get_or_create_shelf_set(shelf_set_name) + + shelves_definition = shelf_set_config.get('shelf_definition') + + if not shelves_definition: + log.debug( + "No shelf definition found for shelf set named '{}'".format( + shelf_set_name + ) + ) + return + + for shelf_definition in shelves_definition: + shelf_name = shelf_definition.get('shelf_name') + if not shelf_name: + log.warning( + "No name found in shelf definition." + ) + return + + shelf = get_or_create_shelf(shelf_name) + + if not shelf_definition.get('tools_list'): + log.debug( + "No tool definition found for shelf named {}".format( + shelf_name + ) + ) + return + + mandatory_attributes = {'name', 'script'} + for tool_definition in shelf_definition.get('tools_list'): + # We verify that the name and script attibutes of the tool + # are set + if not all( + tool_definition[key] for key in mandatory_attributes + ): + log.warning( + "You need to specify at least the name and \ +the script path of the tool.") + continue + + tool = get_or_create_tool(tool_definition, shelf) + + if not tool: + return + + # Add the tool to the shelf if not already in it + if tool not in shelf.tools(): + shelf.setTools(list(shelf.tools()) + [tool]) + + # Add the shelf in the shelf set if not already in it + if shelf not in shelf_set.shelves(): + shelf_set.setShelves(shelf_set.shelves() + (shelf,)) + + +def get_or_create_shelf_set(shelf_set_label): + """This function verifies if the shelf set label exists. If not, + creates a new shelf set. + + Arguments: + shelf_set_label (str): The label of the shelf set + + Returns: + hou.ShelfSet: The shelf set existing or the new one + """ + all_shelves_sets = hou.shelves.shelfSets().values() + + shelf_sets = [ + shelf for shelf in all_shelves_sets if shelf.label() == shelf_set_label + ] + + if shelf_sets: + return shelf_sets[0] + + shelf_set_name = shelf_set_label.replace(' ', '_').lower() + new_shelf_set = hou.shelves.newShelfSet( + name=shelf_set_name, + label=shelf_set_label + ) + return new_shelf_set + + +def get_or_create_shelf(shelf_label): + """This function verifies if the shelf label exists. If not, creates + a new shelf. + + Arguments: + shelf_label (str): The label of the shelf + + Returns: + hou.Shelf: The shelf existing or the new one + """ + all_shelves = hou.shelves.shelves().values() + + shelf = [s for s in all_shelves if s.label() == shelf_label] + + if shelf: + return shelf[0] + + shelf_name = shelf_label.replace(' ', '_').lower() + new_shelf = hou.shelves.newShelf( + name=shelf_name, + label=shelf_label + ) + return new_shelf + + +def get_or_create_tool(tool_definition, shelf): + """This function verifies if the tool exists and updates it. If not, creates + a new one. + + Arguments: + tool_definition (dict): Dict with label, script, icon and help + shelf (hou.Shelf): The parent shelf of the tool + + Returns: + hou.Tool: The tool updated or the new one + """ + existing_tools = shelf.tools() + tool_label = tool_definition.get('label') + + existing_tool = [ + tool for tool in existing_tools if tool.label() == tool_label + ] + + if existing_tool: + tool_definition.pop('name', None) + tool_definition.pop('label', None) + existing_tool[0].setData(**tool_definition) + return existing_tool[0] + + tool_name = tool_label.replace(' ', '_').lower() + + if not os.path.exists(tool_definition['script']): + log.warning( + "This path doesn't exist - {}".format( + tool_definition['script'] + ) + ) + return + + with open(tool_definition['script']) as f: + script = f.read() + tool_definition.update({'script': script}) + + new_tool = hou.shelves.newTool(name=tool_name, **tool_definition) + + return new_tool diff --git a/openpype/hosts/houdini/plugins/publish/collect_inputs.py b/openpype/hosts/houdini/plugins/publish/collect_inputs.py index 8c7098c710..9ee0248bd9 100644 --- a/openpype/hosts/houdini/plugins/publish/collect_inputs.py +++ b/openpype/hosts/houdini/plugins/publish/collect_inputs.py @@ -1,3 +1,5 @@ +from bson.objectid import ObjectId + import pyblish.api from openpype.pipeline import registered_host @@ -115,7 +117,7 @@ class CollectUpstreamInputs(pyblish.api.InstancePlugin): # Collect containers for the given set of nodes containers = collect_input_containers(nodes) - inputs = [c["representation"] for c in containers] - instance.data["inputs"] = inputs + inputs = [ObjectId(c["representation"]) for c in containers] + instance.data["inputRepresentations"] = inputs self.log.info("Collected inputs: %s" % inputs) diff --git a/openpype/hosts/houdini/plugins/publish/collect_remote_publish.py b/openpype/hosts/houdini/plugins/publish/collect_remote_publish.py index c635a53074..d56d389be0 100644 --- a/openpype/hosts/houdini/plugins/publish/collect_remote_publish.py +++ b/openpype/hosts/houdini/plugins/publish/collect_remote_publish.py @@ -1,7 +1,7 @@ import pyblish.api -import openpype.api import hou +from openpype.pipeline.publish import RepairAction from openpype.hosts.houdini.api import lib @@ -13,7 +13,7 @@ class CollectRemotePublishSettings(pyblish.api.ContextPlugin): hosts = ["houdini"] targets = ["deadline"] label = "Remote Publish Submission Settings" - actions = [openpype.api.RepairAction] + actions = [RepairAction] def process(self, context): diff --git a/openpype/hosts/houdini/plugins/publish/extract_alembic.py b/openpype/hosts/houdini/plugins/publish/extract_alembic.py index 83b790407f..758d4c560b 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_alembic.py +++ b/openpype/hosts/houdini/plugins/publish/extract_alembic.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractAlembic(openpype.api.Extractor): +class ExtractAlembic(publish.Extractor): order = pyblish.api.ExtractorOrder label = "Extract Alembic" diff --git a/openpype/hosts/houdini/plugins/publish/extract_ass.py b/openpype/hosts/houdini/plugins/publish/extract_ass.py index e56e40df85..a302b451cb 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_ass.py +++ b/openpype/hosts/houdini/plugins/publish/extract_ass.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractAss(openpype.api.Extractor): +class ExtractAss(publish.Extractor): order = pyblish.api.ExtractorOrder + 0.1 label = "Extract Ass" diff --git a/openpype/hosts/houdini/plugins/publish/extract_composite.py b/openpype/hosts/houdini/plugins/publish/extract_composite.py index f300b6d28d..23e875f107 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_composite.py +++ b/openpype/hosts/houdini/plugins/publish/extract_composite.py @@ -1,12 +1,12 @@ import os import pyblish.api -import openpype.api +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractComposite(openpype.api.Extractor): +class ExtractComposite(publish.Extractor): order = pyblish.api.ExtractorOrder label = "Extract Composite (Image Sequence)" diff --git a/openpype/hosts/houdini/plugins/publish/extract_hda.py b/openpype/hosts/houdini/plugins/publish/extract_hda.py index 301dd4e297..7dd03a92b7 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_hda.py +++ b/openpype/hosts/houdini/plugins/publish/extract_hda.py @@ -4,10 +4,11 @@ import os from pprint import pformat import pyblish.api -import openpype.api + +from openpype.pipeline import publish -class ExtractHDA(openpype.api.Extractor): +class ExtractHDA(publish.Extractor): order = pyblish.api.ExtractorOrder label = "Extract HDA" diff --git a/openpype/hosts/houdini/plugins/publish/extract_redshift_proxy.py b/openpype/hosts/houdini/plugins/publish/extract_redshift_proxy.py index c754d60c59..ca9be64a47 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_redshift_proxy.py +++ b/openpype/hosts/houdini/plugins/publish/extract_redshift_proxy.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractRedshiftProxy(openpype.api.Extractor): +class ExtractRedshiftProxy(publish.Extractor): order = pyblish.api.ExtractorOrder + 0.1 label = "Extract Redshift Proxy" diff --git a/openpype/hosts/houdini/plugins/publish/extract_usd.py b/openpype/hosts/houdini/plugins/publish/extract_usd.py index 0fc26900fb..78c32affb4 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_usd.py +++ b/openpype/hosts/houdini/plugins/publish/extract_usd.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractUSD(openpype.api.Extractor): +class ExtractUSD(publish.Extractor): order = pyblish.api.ExtractorOrder label = "Extract USD" diff --git a/openpype/hosts/houdini/plugins/publish/extract_usd_layered.py b/openpype/hosts/houdini/plugins/publish/extract_usd_layered.py index 80919c023b..f686f712bb 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_usd_layered.py +++ b/openpype/hosts/houdini/plugins/publish/extract_usd_layered.py @@ -5,7 +5,6 @@ import sys from collections import deque import pyblish.api -import openpype.api from openpype.client import ( get_asset_by_name, @@ -16,6 +15,7 @@ from openpype.client import ( from openpype.pipeline import ( get_representation_path, legacy_io, + publish, ) import openpype.hosts.houdini.api.usd as hou_usdlib from openpype.hosts.houdini.api.lib import render_rop @@ -160,7 +160,7 @@ def parm_values(overrides): parm.set(value) -class ExtractUSDLayered(openpype.api.Extractor): +class ExtractUSDLayered(publish.Extractor): order = pyblish.api.ExtractorOrder label = "Extract Layered USD" diff --git a/openpype/hosts/houdini/plugins/publish/extract_vdb_cache.py b/openpype/hosts/houdini/plugins/publish/extract_vdb_cache.py index 113e1b0bcb..26ec423048 100644 --- a/openpype/hosts/houdini/plugins/publish/extract_vdb_cache.py +++ b/openpype/hosts/houdini/plugins/publish/extract_vdb_cache.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.houdini.api.lib import render_rop -class ExtractVDBCache(openpype.api.Extractor): +class ExtractVDBCache(publish.Extractor): order = pyblish.api.ExtractorOrder + 0.1 label = "Extract VDB Cache" diff --git a/openpype/hosts/houdini/plugins/publish/increment_current_file.py b/openpype/hosts/houdini/plugins/publish/increment_current_file.py index 5cb14d732a..c990f481d3 100644 --- a/openpype/hosts/houdini/plugins/publish/increment_current_file.py +++ b/openpype/hosts/houdini/plugins/publish/increment_current_file.py @@ -2,10 +2,9 @@ import pyblish.api from openpype.lib import version_up from openpype.pipeline import registered_host -from openpype.pipeline.publish import get_errored_plugins_from_context -class IncrementCurrentFile(pyblish.api.InstancePlugin): +class IncrementCurrentFile(pyblish.api.ContextPlugin): """Increment the current file. Saves the current scene with an increased version number. @@ -15,30 +14,10 @@ class IncrementCurrentFile(pyblish.api.InstancePlugin): label = "Increment current file" order = pyblish.api.IntegratorOrder + 9.0 hosts = ["houdini"] - families = ["colorbleed.usdrender", "redshift_rop"] - targets = ["local"] + families = ["workfile"] + optional = True - def process(self, instance): - - # This should be a ContextPlugin, but this is a workaround - # for a bug in pyblish to run once for a family: issue #250 - context = instance.context - key = "__hasRun{}".format(self.__class__.__name__) - if context.data.get(key, False): - return - else: - context.data[key] = True - - context = instance.context - errored_plugins = get_errored_plugins_from_context(context) - if any( - plugin.__name__ == "HoudiniSubmitPublishDeadline" - for plugin in errored_plugins - ): - raise RuntimeError( - "Skipping incrementing current file because " - "submission to deadline failed." - ) + def process(self, context): # Filename must not have changed since collecting host = registered_host() diff --git a/openpype/hosts/houdini/plugins/publish/increment_current_file_deadline.py b/openpype/hosts/houdini/plugins/publish/increment_current_file_deadline.py deleted file mode 100644 index cb0d7e3680..0000000000 --- a/openpype/hosts/houdini/plugins/publish/increment_current_file_deadline.py +++ /dev/null @@ -1,35 +0,0 @@ -import pyblish.api - -import hou -from openpype.lib import version_up -from openpype.pipeline.publish import get_errored_plugins_from_context - - -class IncrementCurrentFileDeadline(pyblish.api.ContextPlugin): - """Increment the current file. - - Saves the current scene with an increased version number. - - """ - - label = "Increment current file" - order = pyblish.api.IntegratorOrder + 9.0 - hosts = ["houdini"] - targets = ["deadline"] - - def process(self, context): - - errored_plugins = get_errored_plugins_from_context(context) - if any( - plugin.__name__ == "HoudiniSubmitPublishDeadline" - for plugin in errored_plugins - ): - raise RuntimeError( - "Skipping incrementing current file because " - "submission to deadline failed." - ) - - current_filepath = context.data["currentFile"] - new_filepath = version_up(current_filepath) - - hou.hipFile.save(file_name=new_filepath, save_to_recent_files=True) diff --git a/openpype/hosts/maya/api/lib.py b/openpype/hosts/maya/api/lib.py index 58e160cb2f..6a8447d6ad 100644 --- a/openpype/hosts/maya/api/lib.py +++ b/openpype/hosts/maya/api/lib.py @@ -2483,7 +2483,7 @@ def load_capture_preset(data=None): # DISPLAY OPTIONS id = 'Display Options' disp_options = {} - for key in preset['Display Options']: + for key in preset[id]: if key.startswith('background'): disp_options[key] = preset['Display Options'][key] if len(disp_options[key]) == 4: diff --git a/openpype/hosts/maya/api/lib_rendersettings.py b/openpype/hosts/maya/api/lib_rendersettings.py index 7cd2193086..777a6ffbc9 100644 --- a/openpype/hosts/maya/api/lib_rendersettings.py +++ b/openpype/hosts/maya/api/lib_rendersettings.py @@ -5,6 +5,7 @@ import maya.mel as mel import six import sys +from openpype.lib import Logger from openpype.api import ( get_project_settings, get_current_project_settings @@ -38,6 +39,8 @@ class RenderSettings(object): "underscore": "_" } + log = Logger.get_logger("RenderSettings") + @classmethod def get_image_prefix_attr(cls, renderer): return cls._image_prefix_nodes[renderer] @@ -133,20 +136,7 @@ class RenderSettings(object): cmds.setAttr( "defaultArnoldDriver.mergeAOVs", multi_exr) - # Passes additional options in from the schema as a list - # but converts it to a dictionary because ftrack doesn't - # allow fullstops in custom attributes. Then checks for - # type of MtoA attribute passed to adjust the `setAttr` - # command accordingly. self._additional_attribs_setter(additional_options) - for item in additional_options: - attribute, value = item - if (cmds.getAttr(str(attribute), type=True)) == "long": - cmds.setAttr(str(attribute), int(value)) - elif (cmds.getAttr(str(attribute), type=True)) == "bool": - cmds.setAttr(str(attribute), int(value), type = "Boolean") # noqa - elif (cmds.getAttr(str(attribute), type=True)) == "string": - cmds.setAttr(str(attribute), str(value), type = "string") # noqa reset_frame_range() def _set_redshift_settings(self, width, height): @@ -230,12 +220,20 @@ class RenderSettings(object): cmds.setAttr("defaultRenderGlobals.extensionPadding", 4) def _additional_attribs_setter(self, additional_attribs): - print(additional_attribs) for item in additional_attribs: attribute, value = item - if (cmds.getAttr(str(attribute), type=True)) == "long": - cmds.setAttr(str(attribute), int(value)) - elif (cmds.getAttr(str(attribute), type=True)) == "bool": - cmds.setAttr(str(attribute), int(value)) # noqa - elif (cmds.getAttr(str(attribute), type=True)) == "string": - cmds.setAttr(str(attribute), str(value), type = "string") # noqa + attribute = str(attribute) # ensure str conversion from settings + attribute_type = cmds.getAttr(attribute, type=True) + if attribute_type in {"long", "bool"}: + cmds.setAttr(attribute, int(value)) + elif attribute_type == "string": + cmds.setAttr(attribute, str(value), type="string") + elif attribute_type in {"double", "doubleAngle", "doubleLinear"}: + cmds.setAttr(attribute, float(value)) + else: + self.log.error( + "Attribute {attribute} can not be set due to unsupported " + "type: {attribute_type}".format( + attribute=attribute, + attribute_type=attribute_type) + ) diff --git a/openpype/hosts/maya/api/lib_rendersetup.py b/openpype/hosts/maya/api/lib_rendersetup.py index 0fdc54a068..e616f26e1b 100644 --- a/openpype/hosts/maya/api/lib_rendersetup.py +++ b/openpype/hosts/maya/api/lib_rendersetup.py @@ -348,3 +348,71 @@ def get_attr_overrides(node_attr, layer, break return reversed(plug_overrides) + + +def get_shader_in_layer(node, layer): + """Return the assigned shader in a renderlayer without switching layers. + + This has been developed and tested for Legacy Renderlayers and *not* for + Render Setup. + + Note: This will also return the shader for any face assignments, however + it will *not* return the components they are assigned to. This could + be implemented, but since Maya's renderlayers are famous for breaking + with face assignments there has been no need for this function to + support that. + + Returns: + list: The list of assigned shaders in the given layer. + + """ + + def _get_connected_shader(plug): + """Return current shader""" + return cmds.listConnections(plug, + source=False, + destination=True, + plugs=False, + connections=False, + type="shadingEngine") or [] + + # We check the instObjGroups (shader connection) for layer overrides. + plug = node + ".instObjGroups" + + # Ignore complex query if we're in the layer anyway (optimization) + current_layer = cmds.editRenderLayerGlobals(query=True, + currentRenderLayer=True) + if layer == current_layer: + return _get_connected_shader(plug) + + connections = cmds.listConnections(plug, + plugs=True, + source=False, + destination=True, + type="renderLayer") or [] + connections = filter(lambda x: x.endswith(".outPlug"), connections) + if not connections: + # If no overrides anywhere on the shader, just get the current shader + return _get_connected_shader(plug) + + def _get_override(connections, layer): + """Return the overridden connection for that layer in connections""" + # If there's an override on that layer, return that. + for connection in connections: + if (connection.startswith(layer + ".outAdjustments") and + connection.endswith(".outPlug")): + + # This is a shader override on that layer so get the shader + # connected to .outValue of the .outAdjustment[i] + out_adjustment = connection.rsplit(".", 1)[0] + connection_attr = out_adjustment + ".outValue" + override = cmds.listConnections(connection_attr) or [] + + return override + + override_shader = _get_override(connections, layer) + if override_shader is not None: + return override_shader + else: + # Get the override for "defaultRenderLayer" (=masterLayer) + return _get_override(connections, layer="defaultRenderLayer") diff --git a/openpype/hosts/maya/api/lib_template_builder.py b/openpype/hosts/maya/api/lib_template_builder.py deleted file mode 100644 index 34a8450a26..0000000000 --- a/openpype/hosts/maya/api/lib_template_builder.py +++ /dev/null @@ -1,253 +0,0 @@ -import json -from collections import OrderedDict -import maya.cmds as cmds - -import qargparse -from openpype.tools.utils.widgets import OptionDialog -from .lib import get_main_window, imprint - -# To change as enum -build_types = ["context_asset", "linked_asset", "all_assets"] - - -def get_placeholder_attributes(node): - return { - attr: cmds.getAttr("{}.{}".format(node, attr)) - for attr in cmds.listAttr(node, userDefined=True)} - - -def delete_placeholder_attributes(node): - ''' - function to delete all extra placeholder attributes - ''' - extra_attributes = get_placeholder_attributes(node) - for attribute in extra_attributes: - cmds.deleteAttr(node + '.' + attribute) - - -def create_placeholder(): - args = placeholder_window() - - if not args: - return # operation canceled, no locator created - - # custom arg parse to force empty data query - # and still imprint them on placeholder - # and getting items when arg is of type Enumerator - options = create_options(args) - - # create placeholder name dynamically from args and options - placeholder_name = create_placeholder_name(args, options) - - selection = cmds.ls(selection=True) - if not selection: - raise ValueError("Nothing is selected") - - placeholder = cmds.spaceLocator(name=placeholder_name)[0] - - # get the long name of the placeholder (with the groups) - placeholder_full_name = cmds.ls(selection[0], long=True)[ - 0] + '|' + placeholder.replace('|', '') - - if selection: - cmds.parent(placeholder, selection[0]) - - imprint(placeholder_full_name, options) - - # Some tweaks because imprint force enums to to default value so we get - # back arg read and force them to attributes - imprint_enum(placeholder_full_name, args) - - # Add helper attributes to keep placeholder info - cmds.addAttr( - placeholder_full_name, - longName="parent", - hidden=True, - dataType="string" - ) - cmds.addAttr( - placeholder_full_name, - longName="index", - hidden=True, - attributeType="short", - defaultValue=-1 - ) - - cmds.setAttr(placeholder_full_name + '.parent', "", type="string") - - -def create_placeholder_name(args, options): - placeholder_builder_type = [ - arg.read() for arg in args if 'builder_type' in str(arg) - ][0] - placeholder_family = options['family'] - placeholder_name = placeholder_builder_type.split('_') - - # add famlily in any - if placeholder_family: - placeholder_name.insert(1, placeholder_family) - - # add loader arguments if any - if options['loader_args']: - pos = 2 - loader_args = options['loader_args'].replace('\'', '\"') - loader_args = json.loads(loader_args) - values = [v for v in loader_args.values()] - for i in range(len(values)): - placeholder_name.insert(i + pos, values[i]) - - placeholder_name = '_'.join(placeholder_name) - - return placeholder_name.capitalize() - - -def update_placeholder(): - placeholder = cmds.ls(selection=True) - if len(placeholder) == 0: - raise ValueError("No node selected") - if len(placeholder) > 1: - raise ValueError("Too many selected nodes") - placeholder = placeholder[0] - - args = placeholder_window(get_placeholder_attributes(placeholder)) - - if not args: - return # operation canceled - - # delete placeholder attributes - delete_placeholder_attributes(placeholder) - - options = create_options(args) - - imprint(placeholder, options) - imprint_enum(placeholder, args) - - cmds.addAttr( - placeholder, - longName="parent", - hidden=True, - dataType="string" - ) - cmds.addAttr( - placeholder, - longName="index", - hidden=True, - attributeType="short", - defaultValue=-1 - ) - - cmds.setAttr(placeholder + '.parent', '', type="string") - - -def create_options(args): - options = OrderedDict() - for arg in args: - if not type(arg) == qargparse.Separator: - options[str(arg)] = arg._data.get("items") or arg.read() - return options - - -def imprint_enum(placeholder, args): - """ - Imprint method doesn't act properly with enums. - Replacing the functionnality with this for now - """ - enum_values = {str(arg): arg.read() - for arg in args if arg._data.get("items")} - string_to_value_enum_table = { - build: i for i, build - in enumerate(build_types)} - for key, value in enum_values.items(): - cmds.setAttr( - placeholder + "." + key, - string_to_value_enum_table[value]) - - -def placeholder_window(options=None): - options = options or dict() - dialog = OptionDialog(parent=get_main_window()) - dialog.setWindowTitle("Create Placeholder") - - args = [ - qargparse.Separator("Main attributes"), - qargparse.Enum( - "builder_type", - label="Asset Builder Type", - default=options.get("builder_type", 0), - items=build_types, - help="""Asset Builder Type -Builder type describe what template loader will look for. -context_asset : Template loader will look for subsets of -current context asset (Asset bob will find asset) -linked_asset : Template loader will look for assets linked -to current context asset. -Linked asset are looked in avalon database under field "inputLinks" -""" - ), - qargparse.String( - "family", - default=options.get("family", ""), - label="OpenPype Family", - placeholder="ex: model, look ..."), - qargparse.String( - "representation", - default=options.get("representation", ""), - label="OpenPype Representation", - placeholder="ex: ma, abc ..."), - qargparse.String( - "loader", - default=options.get("loader", ""), - label="Loader", - placeholder="ex: ReferenceLoader, LightLoader ...", - help="""Loader -Defines what openpype loader will be used to load assets. -Useable loader depends on current host's loader list. -Field is case sensitive. -"""), - qargparse.String( - "loader_args", - default=options.get("loader_args", ""), - label="Loader Arguments", - placeholder='ex: {"camera":"persp", "lights":True}', - help="""Loader -Defines a dictionnary of arguments used to load assets. -Useable arguments depend on current placeholder Loader. -Field should be a valid python dict. Anything else will be ignored. -"""), - qargparse.Integer( - "order", - default=options.get("order", 0), - min=0, - max=999, - label="Order", - placeholder="ex: 0, 100 ... (smallest order loaded first)", - help="""Order -Order defines asset loading priority (0 to 999) -Priority rule is : "lowest is first to load"."""), - qargparse.Separator( - "Optional attributes"), - qargparse.String( - "asset", - default=options.get("asset", ""), - label="Asset filter", - placeholder="regex filtering by asset name", - help="Filtering assets by matching field regex to asset's name"), - qargparse.String( - "subset", - default=options.get("subset", ""), - label="Subset filter", - placeholder="regex filtering by subset name", - help="Filtering assets by matching field regex to subset's name"), - qargparse.String( - "hierarchy", - default=options.get("hierarchy", ""), - label="Hierarchy filter", - placeholder="regex filtering by asset's hierarchy", - help="Filtering assets by matching field asset's hierarchy") - ] - dialog.create(args) - - if not dialog.exec_(): - return None - - return args diff --git a/openpype/hosts/maya/api/menu.py b/openpype/hosts/maya/api/menu.py index ebba706a6c..e20f29049b 100644 --- a/openpype/hosts/maya/api/menu.py +++ b/openpype/hosts/maya/api/menu.py @@ -9,16 +9,17 @@ import maya.cmds as cmds from openpype.settings import get_project_settings from openpype.pipeline import legacy_io from openpype.pipeline.workfile import BuildWorkfile -from openpype.pipeline.workfile.build_template import ( - build_workfile_template, - update_workfile_template -) from openpype.tools.utils import host_tools from openpype.hosts.maya.api import lib, lib_rendersettings from .lib import get_main_window, IS_HEADLESS from .commands import reset_frame_range -from .lib_template_builder import create_placeholder, update_placeholder +from .workfile_template_builder import ( + create_placeholder, + update_placeholder, + build_workfile_template, + update_workfile_template, +) log = logging.getLogger(__name__) @@ -104,13 +105,6 @@ def install(): cmds.menuItem(divider=True) - cmds.menuItem( - "Set Render Settings", - command=lambda *args: lib_rendersettings.RenderSettings().set_default_renderer_settings() # noqa - ) - - cmds.menuItem(divider=True) - cmds.menuItem( "Work Files...", command=lambda *args: host_tools.show_workfiles( @@ -132,6 +126,12 @@ def install(): "Set Colorspace", command=lambda *args: lib.set_colorspace(), ) + + cmds.menuItem( + "Set Render Settings", + command=lambda *args: lib_rendersettings.RenderSettings().set_default_renderer_settings() # noqa + ) + cmds.menuItem(divider=True, parent=MENU_NAME) cmds.menuItem( "Build First Workfile", @@ -162,12 +162,12 @@ def install(): cmds.menuItem( "Create Placeholder", parent=builder_menu, - command=lambda *args: create_placeholder() + command=create_placeholder ) cmds.menuItem( "Update Placeholder", parent=builder_menu, - command=lambda *args: update_placeholder() + command=update_placeholder ) cmds.menuItem( "Build Workfile from template", diff --git a/openpype/hosts/maya/api/pipeline.py b/openpype/hosts/maya/api/pipeline.py index 2401cc6aba..b3bf738a2b 100644 --- a/openpype/hosts/maya/api/pipeline.py +++ b/openpype/hosts/maya/api/pipeline.py @@ -16,6 +16,7 @@ from openpype.host import ( HostDirmap, ) from openpype.tools.utils import host_tools +from openpype.tools.workfiles.lock_dialog import WorkfileLockDialog from openpype.lib import ( register_event_callback, emit_event @@ -31,10 +32,17 @@ from openpype.pipeline import ( AVALON_CONTAINER_ID, ) from openpype.pipeline.load import any_outdated_containers +from openpype.pipeline.workfile.lock_workfile import ( + create_workfile_lock, + remove_workfile_lock, + is_workfile_locked, + is_workfile_lock_enabled +) from openpype.hosts.maya import MAYA_ROOT_DIR -from openpype.hosts.maya.lib import copy_workspace_mel +from openpype.hosts.maya.lib import create_workspace_mel from . import menu, lib +from .workfile_template_builder import MayaPlaceholderLoadPlugin from .workio import ( open_file, save_file, @@ -63,7 +71,7 @@ class MayaHost(HostBase, IWorkfileHost, ILoadHost): self._op_events = {} def install(self): - project_name = os.getenv("AVALON_PROJECT") + project_name = legacy_io.active_project() project_settings = get_project_settings(project_name) # process path mapping dirmap_processor = MayaDirmap("maya", project_name, project_settings) @@ -99,8 +107,13 @@ class MayaHost(HostBase, IWorkfileHost, ILoadHost): register_event_callback("open", on_open) register_event_callback("new", on_new) register_event_callback("before.save", on_before_save) + register_event_callback("after.save", on_after_save) + register_event_callback("before.close", on_before_close) + register_event_callback("before.file.open", before_file_open) register_event_callback("taskChanged", on_task_changed) + register_event_callback("workfile.open.before", before_workfile_open) register_event_callback("workfile.save.before", before_workfile_save) + register_event_callback("workfile.save.before", after_workfile_save) def open_workfile(self, filepath): return open_file(filepath) @@ -123,6 +136,11 @@ class MayaHost(HostBase, IWorkfileHost, ILoadHost): def get_containers(self): return ls() + def get_workfile_build_placeholder_plugins(self): + return [ + MayaPlaceholderLoadPlugin + ] + @contextlib.contextmanager def maintained_selection(self): with lib.maintained_selection(): @@ -143,6 +161,13 @@ class MayaHost(HostBase, IWorkfileHost, ILoadHost): OpenMaya.MSceneMessage.kBeforeSave, _on_scene_save ) + self._op_events[_after_scene_save] = ( + OpenMaya.MSceneMessage.addCallback( + OpenMaya.MSceneMessage.kAfterSave, + _after_scene_save + ) + ) + self._op_events[_before_scene_save] = ( OpenMaya.MSceneMessage.addCheckCallback( OpenMaya.MSceneMessage.kBeforeSaveCheck, @@ -161,15 +186,35 @@ class MayaHost(HostBase, IWorkfileHost, ILoadHost): ) ) - self._op_events[_on_scene_open] = OpenMaya.MSceneMessage.addCallback( - OpenMaya.MSceneMessage.kAfterOpen, _on_scene_open + self._op_events[_on_scene_open] = ( + OpenMaya.MSceneMessage.addCallback( + OpenMaya.MSceneMessage.kAfterOpen, + _on_scene_open + ) + ) + + self._op_events[_before_scene_open] = ( + OpenMaya.MSceneMessage.addCallback( + OpenMaya.MSceneMessage.kBeforeOpen, + _before_scene_open + ) + ) + + self._op_events[_before_close_maya] = ( + OpenMaya.MSceneMessage.addCallback( + OpenMaya.MSceneMessage.kMayaExiting, + _before_close_maya + ) ) self.log.info("Installed event handler _on_scene_save..") self.log.info("Installed event handler _before_scene_save..") + self.log.info("Installed event handler _on_after_save..") self.log.info("Installed event handler _on_scene_new..") self.log.info("Installed event handler _on_maya_initialized..") self.log.info("Installed event handler _on_scene_open..") + self.log.info("Installed event handler _check_lock_file..") + self.log.info("Installed event handler _before_close_maya..") def _set_project(): @@ -208,6 +253,10 @@ def _on_scene_new(*args): emit_event("new") +def _after_scene_save(*arg): + emit_event("after.save") + + def _on_scene_save(*args): emit_event("save") @@ -216,6 +265,14 @@ def _on_scene_open(*args): emit_event("open") +def _before_close_maya(*args): + emit_event("before.close") + + +def _before_scene_open(*args): + emit_event("before.file.open") + + def _before_scene_save(return_code, client_data): # Default to allowing the action. Registered @@ -229,6 +286,23 @@ def _before_scene_save(return_code, client_data): ) +def _remove_workfile_lock(): + """Remove workfile lock on current file""" + if not handle_workfile_locks(): + return + filepath = current_file() + log.info("Removing lock on current file {}...".format(filepath)) + if filepath: + remove_workfile_lock(filepath) + + +def handle_workfile_locks(): + if lib.IS_HEADLESS: + return False + project_name = legacy_io.active_project() + return is_workfile_lock_enabled(MayaHost.name, project_name) + + def uninstall(): pyblish.api.deregister_plugin_path(PUBLISH_PATH) pyblish.api.deregister_host("mayabatch") @@ -349,21 +423,13 @@ def containerise(name, ("id", AVALON_CONTAINER_ID), ("name", name), ("namespace", namespace), - ("loader", str(loader)), + ("loader", loader), ("representation", context["representation"]["_id"]), ] for key, value in data: - if not value: - continue - - if isinstance(value, (int, float)): - cmds.addAttr(container, longName=key, attributeType="short") - cmds.setAttr(container + "." + key, value) - - else: - cmds.addAttr(container, longName=key, dataType="string") - cmds.setAttr(container + "." + key, value, type="string") + cmds.addAttr(container, longName=key, dataType="string") + cmds.setAttr(container + "." + key, str(value), type="string") main_container = cmds.ls(AVALON_CONTAINERS, type="objectSet") if not main_container: @@ -434,6 +500,46 @@ def on_before_save(): return lib.validate_fps() +def on_after_save(): + """Check if there is a lockfile after save""" + check_lock_on_current_file() + + +def check_lock_on_current_file(): + + """Check if there is a user opening the file""" + if not handle_workfile_locks(): + return + log.info("Running callback on checking the lock file...") + + # add the lock file when opening the file + filepath = current_file() + + if is_workfile_locked(filepath): + # add lockfile dialog + workfile_dialog = WorkfileLockDialog(filepath) + if not workfile_dialog.exec_(): + cmds.file(new=True) + return + + create_workfile_lock(filepath) + + +def on_before_close(): + """Delete the lock file after user quitting the Maya Scene""" + log.info("Closing Maya...") + # delete the lock file + filepath = current_file() + if handle_workfile_locks(): + remove_workfile_lock(filepath) + + +def before_file_open(): + """check lock file when the file changed""" + # delete the lock file + _remove_workfile_lock() + + def on_save(): """Automatically add IDs to new nodes @@ -442,6 +548,8 @@ def on_save(): """ log.info("Running callback on save..") + # remove lockfile if users jumps over from one scene to another + _remove_workfile_lock() # # Update current task for the current scene # update_task_from_path(cmds.file(query=True, sceneName=True)) @@ -499,6 +607,9 @@ def on_open(): dialog.on_clicked.connect(_on_show_inventory) dialog.show() + # create lock file for the maya scene + check_lock_on_current_file() + def on_new(): """Set project resolution and fps when create a new file""" @@ -514,6 +625,7 @@ def on_new(): "from openpype.hosts.maya.api import lib;" "lib.add_render_layer_change_observer()") lib.set_context_settings() + _remove_workfile_lock() def on_task_changed(): @@ -541,7 +653,7 @@ def on_task_changed(): lib.update_content_on_context_change() msg = " project: {}\n asset: {}\n task:{}".format( - legacy_io.Session["AVALON_PROJECT"], + legacy_io.active_project(), legacy_io.Session["AVALON_ASSET"], legacy_io.Session["AVALON_TASK"] ) @@ -552,10 +664,26 @@ def on_task_changed(): ) +def before_workfile_open(): + if handle_workfile_locks(): + _remove_workfile_lock() + + def before_workfile_save(event): + project_name = legacy_io.active_project() + if handle_workfile_locks(): + _remove_workfile_lock() workdir_path = event["workdir_path"] if workdir_path: - copy_workspace_mel(workdir_path) + create_workspace_mel(workdir_path, project_name) + + +def after_workfile_save(event): + workfile_name = event["filename"] + if handle_workfile_locks(): + if workfile_name: + if not is_workfile_locked(workfile_name): + create_workfile_lock(workfile_name) class MayaDirmap(HostDirmap): diff --git a/openpype/hosts/maya/api/template_loader.py b/openpype/hosts/maya/api/template_loader.py deleted file mode 100644 index ecffafc93d..0000000000 --- a/openpype/hosts/maya/api/template_loader.py +++ /dev/null @@ -1,252 +0,0 @@ -import re -from maya import cmds - -from openpype.client import get_representations -from openpype.pipeline import legacy_io -from openpype.pipeline.workfile.abstract_template_loader import ( - AbstractPlaceholder, - AbstractTemplateLoader -) -from openpype.pipeline.workfile.build_template_exceptions import ( - TemplateAlreadyImported -) - -PLACEHOLDER_SET = 'PLACEHOLDERS_SET' - - -class MayaTemplateLoader(AbstractTemplateLoader): - """Concrete implementation of AbstractTemplateLoader for maya - """ - - def import_template(self, path): - """Import template into current scene. - Block if a template is already loaded. - Args: - path (str): A path to current template (usually given by - get_template_path implementation) - Returns: - bool: Wether the template was succesfully imported or not - """ - if cmds.objExists(PLACEHOLDER_SET): - raise TemplateAlreadyImported( - "Build template already loaded\n" - "Clean scene if needed (File > New Scene)") - - cmds.sets(name=PLACEHOLDER_SET, empty=True) - self.new_nodes = cmds.file(path, i=True, returnNewNodes=True) - cmds.setAttr(PLACEHOLDER_SET + '.hiddenInOutliner', True) - - for set in cmds.listSets(allSets=True): - if (cmds.objExists(set) and - cmds.attributeQuery('id', node=set, exists=True) and - cmds.getAttr(set + '.id') == 'pyblish.avalon.instance'): - if cmds.attributeQuery('asset', node=set, exists=True): - cmds.setAttr( - set + '.asset', - legacy_io.Session['AVALON_ASSET'], type='string' - ) - - return True - - def template_already_imported(self, err_msg): - clearButton = "Clear scene and build" - updateButton = "Update template" - abortButton = "Abort" - - title = "Scene already builded" - message = ( - "It's seems a template was already build for this scene.\n" - "Error message reveived :\n\n\"{}\"".format(err_msg)) - buttons = [clearButton, updateButton, abortButton] - defaultButton = clearButton - cancelButton = abortButton - dismissString = abortButton - answer = cmds.confirmDialog( - t=title, - m=message, - b=buttons, - db=defaultButton, - cb=cancelButton, - ds=dismissString) - - if answer == clearButton: - cmds.file(newFile=True, force=True) - self.import_template(self.template_path) - self.populate_template() - elif answer == updateButton: - self.update_missing_containers() - elif answer == abortButton: - return - - @staticmethod - def get_template_nodes(): - attributes = cmds.ls('*.builder_type', long=True) - return [attribute.rpartition('.')[0] for attribute in attributes] - - def get_loaded_containers_by_id(self): - try: - containers = cmds.sets("AVALON_CONTAINERS", q=True) - except ValueError: - return None - - return [ - cmds.getAttr(container + '.representation') - for container in containers] - - -class MayaPlaceholder(AbstractPlaceholder): - """Concrete implementation of AbstractPlaceholder for maya - """ - - optional_keys = {'asset', 'subset', 'hierarchy'} - - def get_data(self, node): - user_data = dict() - for attr in self.required_keys.union(self.optional_keys): - attribute_name = '{}.{}'.format(node, attr) - if not cmds.attributeQuery(attr, node=node, exists=True): - print("{} not found".format(attribute_name)) - continue - user_data[attr] = cmds.getAttr( - attribute_name, - asString=True) - user_data['parent'] = ( - cmds.getAttr(node + '.parent', asString=True) - or node.rpartition('|')[0] - or "" - ) - user_data['node'] = node - if user_data['parent']: - siblings = cmds.listRelatives(user_data['parent'], children=True) - else: - siblings = cmds.ls(assemblies=True) - node_shortname = user_data['node'].rpartition('|')[2] - current_index = cmds.getAttr(node + '.index', asString=True) - user_data['index'] = ( - current_index if current_index >= 0 - else siblings.index(node_shortname)) - - self.data = user_data - - def parent_in_hierarchy(self, containers): - """Parent loaded container to placeholder's parent - ie : Set loaded content as placeholder's sibling - Args: - containers (String): Placeholder loaded containers - """ - if not containers: - return - - roots = cmds.sets(containers, q=True) - nodes_to_parent = [] - for root in roots: - if root.endswith("_RN"): - refRoot = cmds.referenceQuery(root, n=True)[0] - refRoot = cmds.listRelatives(refRoot, parent=True) or [refRoot] - nodes_to_parent.extend(refRoot) - elif root in cmds.listSets(allSets=True): - if not cmds.sets(root, q=True): - return - else: - continue - else: - nodes_to_parent.append(root) - - if self.data['parent']: - cmds.parent(nodes_to_parent, self.data['parent']) - # Move loaded nodes to correct index in outliner hierarchy - placeholder_node = self.data['node'] - placeholder_form = cmds.xform( - placeholder_node, - q=True, - matrix=True, - worldSpace=True - ) - for node in set(nodes_to_parent): - cmds.reorder(node, front=True) - cmds.reorder(node, relative=self.data['index']) - cmds.xform(node, matrix=placeholder_form, ws=True) - - holding_sets = cmds.listSets(object=placeholder_node) - if not holding_sets: - return - for holding_set in holding_sets: - cmds.sets(roots, forceElement=holding_set) - - def clean(self): - """Hide placeholder, parent them to root - add them to placeholder set and register placeholder's parent - to keep placeholder info available for future use - """ - node = self.data['node'] - if self.data['parent']: - cmds.setAttr(node + '.parent', self.data['parent'], type='string') - if cmds.getAttr(node + '.index') < 0: - cmds.setAttr(node + '.index', self.data['index']) - - holding_sets = cmds.listSets(object=node) - if holding_sets: - for set in holding_sets: - cmds.sets(node, remove=set) - - if cmds.listRelatives(node, p=True): - node = cmds.parent(node, world=True)[0] - cmds.sets(node, addElement=PLACEHOLDER_SET) - cmds.hide(node) - cmds.setAttr(node + '.hiddenInOutliner', True) - - def get_representations(self, current_asset_doc, linked_asset_docs): - project_name = legacy_io.active_project() - - builder_type = self.data["builder_type"] - if builder_type == "context_asset": - context_filters = { - "asset": [current_asset_doc["name"]], - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representations": [self.data["representation"]], - "family": [self.data["family"]] - } - - elif builder_type != "linked_asset": - context_filters = { - "asset": [re.compile(self.data["asset"])], - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representation": [self.data["representation"]], - "family": [self.data["family"]] - } - - else: - asset_regex = re.compile(self.data["asset"]) - linked_asset_names = [] - for asset_doc in linked_asset_docs: - asset_name = asset_doc["name"] - if asset_regex.match(asset_name): - linked_asset_names.append(asset_name) - - context_filters = { - "asset": linked_asset_names, - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representation": [self.data["representation"]], - "family": [self.data["family"]], - } - - return list(get_representations( - project_name, - context_filters=context_filters - )) - - def err_message(self): - return ( - "Error while trying to load a representation.\n" - "Either the subset wasn't published or the template is malformed." - "\n\n" - "Builder was looking for :\n{attributes}".format( - attributes="\n".join([ - "{}: {}".format(key.title(), value) - for key, value in self.data.items()] - ) - ) - ) diff --git a/openpype/hosts/maya/api/workfile_template_builder.py b/openpype/hosts/maya/api/workfile_template_builder.py new file mode 100644 index 0000000000..ef043ed0f4 --- /dev/null +++ b/openpype/hosts/maya/api/workfile_template_builder.py @@ -0,0 +1,330 @@ +import json + +from maya import cmds + +from openpype.pipeline import registered_host +from openpype.pipeline.workfile.workfile_template_builder import ( + TemplateAlreadyImported, + AbstractTemplateBuilder, + PlaceholderPlugin, + LoadPlaceholderItem, + PlaceholderLoadMixin, +) +from openpype.tools.workfile_template_build import ( + WorkfileBuildPlaceholderDialog, +) + +from .lib import read, imprint + +PLACEHOLDER_SET = "PLACEHOLDERS_SET" + + +class MayaTemplateBuilder(AbstractTemplateBuilder): + """Concrete implementation of AbstractTemplateBuilder for maya""" + + def import_template(self, path): + """Import template into current scene. + Block if a template is already loaded. + + Args: + path (str): A path to current template (usually given by + get_template_path implementation) + + Returns: + bool: Wether the template was succesfully imported or not + """ + + if cmds.objExists(PLACEHOLDER_SET): + raise TemplateAlreadyImported(( + "Build template already loaded\n" + "Clean scene if needed (File > New Scene)" + )) + + cmds.sets(name=PLACEHOLDER_SET, empty=True) + cmds.file(path, i=True, returnNewNodes=True) + + cmds.setAttr(PLACEHOLDER_SET + ".hiddenInOutliner", True) + + return True + + +class MayaPlaceholderLoadPlugin(PlaceholderPlugin, PlaceholderLoadMixin): + identifier = "maya.load" + label = "Maya load" + + def _collect_scene_placeholders(self): + # Cache placeholder data to shared data + placeholder_nodes = self.builder.get_shared_populate_data( + "placeholder_nodes" + ) + if placeholder_nodes is None: + attributes = cmds.ls("*.plugin_identifier", long=True) + placeholder_nodes = {} + for attribute in attributes: + node_name = attribute.rpartition(".")[0] + placeholder_nodes[node_name] = ( + self._parse_placeholder_node_data(node_name) + ) + + self.builder.set_shared_populate_data( + "placeholder_nodes", placeholder_nodes + ) + return placeholder_nodes + + def _parse_placeholder_node_data(self, node_name): + placeholder_data = read(node_name) + parent_name = ( + cmds.getAttr(node_name + ".parent", asString=True) + or node_name.rpartition("|")[0] + or "" + ) + if parent_name: + siblings = cmds.listRelatives(parent_name, children=True) + else: + siblings = cmds.ls(assemblies=True) + node_shortname = node_name.rpartition("|")[2] + current_index = cmds.getAttr(node_name + ".index", asString=True) + if current_index < 0: + current_index = siblings.index(node_shortname) + + placeholder_data.update({ + "parent": parent_name, + "index": current_index + }) + return placeholder_data + + def _create_placeholder_name(self, placeholder_data): + placeholder_name_parts = placeholder_data["builder_type"].split("_") + + pos = 1 + # add famlily in any + placeholder_family = placeholder_data["family"] + if placeholder_family: + placeholder_name_parts.insert(pos, placeholder_family) + pos += 1 + + # add loader arguments if any + loader_args = placeholder_data["loader_args"] + if loader_args: + loader_args = json.loads(loader_args.replace('\'', '\"')) + values = [v for v in loader_args.values()] + for value in values: + placeholder_name_parts.insert(pos, value) + pos += 1 + + placeholder_name = "_".join(placeholder_name_parts) + + return placeholder_name.capitalize() + + def _get_loaded_repre_ids(self): + loaded_representation_ids = self.builder.get_shared_populate_data( + "loaded_representation_ids" + ) + if loaded_representation_ids is None: + try: + containers = cmds.sets("AVALON_CONTAINERS", q=True) + except ValueError: + containers = [] + + loaded_representation_ids = { + cmds.getAttr(container + ".representation") + for container in containers + } + self.builder.set_shared_populate_data( + "loaded_representation_ids", loaded_representation_ids + ) + return loaded_representation_ids + + def create_placeholder(self, placeholder_data): + selection = cmds.ls(selection=True) + if not selection: + raise ValueError("Nothing is selected") + if len(selection) > 1: + raise ValueError("More then one item are selected") + + placeholder_data["plugin_identifier"] = self.identifier + + placeholder_name = self._create_placeholder_name(placeholder_data) + + placeholder = cmds.spaceLocator(name=placeholder_name)[0] + # TODO: this can crash if selection can't be used + cmds.parent(placeholder, selection[0]) + + # get the long name of the placeholder (with the groups) + placeholder_full_name = ( + cmds.ls(selection[0], long=True)[0] + + "|" + + placeholder.replace("|", "") + ) + + imprint(placeholder_full_name, placeholder_data) + + # Add helper attributes to keep placeholder info + cmds.addAttr( + placeholder_full_name, + longName="parent", + hidden=True, + dataType="string" + ) + cmds.addAttr( + placeholder_full_name, + longName="index", + hidden=True, + attributeType="short", + defaultValue=-1 + ) + + cmds.setAttr(placeholder_full_name + ".parent", "", type="string") + + def update_placeholder(self, placeholder_item, placeholder_data): + node_name = placeholder_item.scene_identifier + new_values = {} + for key, value in placeholder_data.items(): + placeholder_value = placeholder_item.data.get(key) + if value != placeholder_value: + new_values[key] = value + placeholder_item.data[key] = value + + for key in new_values.keys(): + cmds.deleteAttr(node_name + "." + key) + + imprint(node_name, new_values) + + def collect_placeholders(self): + output = [] + scene_placeholders = self._collect_scene_placeholders() + for node_name, placeholder_data in scene_placeholders.items(): + if placeholder_data.get("plugin_identifier") != self.identifier: + continue + + # TODO do data validations and maybe updgrades if are invalid + output.append( + LoadPlaceholderItem(node_name, placeholder_data, self) + ) + + return output + + def populate_placeholder(self, placeholder): + self.populate_load_placeholder(placeholder) + + def repopulate_placeholder(self, placeholder): + repre_ids = self._get_loaded_repre_ids() + self.populate_load_placeholder(placeholder, repre_ids) + + def get_placeholder_options(self, options=None): + return self.get_load_plugin_options(options) + + def cleanup_placeholder(self, placeholder, failed): + """Hide placeholder, parent them to root + add them to placeholder set and register placeholder's parent + to keep placeholder info available for future use + """ + + node = placeholder._scene_identifier + node_parent = placeholder.data["parent"] + if node_parent: + cmds.setAttr(node + ".parent", node_parent, type="string") + + if cmds.getAttr(node + ".index") < 0: + cmds.setAttr(node + ".index", placeholder.data["index"]) + + holding_sets = cmds.listSets(object=node) + if holding_sets: + for set in holding_sets: + cmds.sets(node, remove=set) + + if cmds.listRelatives(node, p=True): + node = cmds.parent(node, world=True)[0] + cmds.sets(node, addElement=PLACEHOLDER_SET) + cmds.hide(node) + cmds.setAttr(node + ".hiddenInOutliner", True) + + def load_succeed(self, placeholder, container): + self._parent_in_hierarhchy(placeholder, container) + + def _parent_in_hierarchy(self, placeholder, container): + """Parent loaded container to placeholder's parent. + + ie : Set loaded content as placeholder's sibling + + Args: + container (str): Placeholder loaded containers + """ + + if not container: + return + + roots = cmds.sets(container, q=True) + nodes_to_parent = [] + for root in roots: + if root.endswith("_RN"): + refRoot = cmds.referenceQuery(root, n=True)[0] + refRoot = cmds.listRelatives(refRoot, parent=True) or [refRoot] + nodes_to_parent.extend(refRoot) + elif root not in cmds.listSets(allSets=True): + nodes_to_parent.append(root) + + elif not cmds.sets(root, q=True): + return + + if placeholder.data["parent"]: + cmds.parent(nodes_to_parent, placeholder.data["parent"]) + # Move loaded nodes to correct index in outliner hierarchy + placeholder_form = cmds.xform( + placeholder.scene_identifier, + q=True, + matrix=True, + worldSpace=True + ) + for node in set(nodes_to_parent): + cmds.reorder(node, front=True) + cmds.reorder(node, relative=placeholder.data["index"]) + cmds.xform(node, matrix=placeholder_form, ws=True) + + holding_sets = cmds.listSets(object=placeholder.scene_identifier) + if not holding_sets: + return + for holding_set in holding_sets: + cmds.sets(roots, forceElement=holding_set) + + +def build_workfile_template(*args): + builder = MayaTemplateBuilder(registered_host()) + builder.build_template() + + +def update_workfile_template(*args): + builder = MayaTemplateBuilder(registered_host()) + builder.rebuild_template() + + +def create_placeholder(*args): + host = registered_host() + builder = MayaTemplateBuilder(host) + window = WorkfileBuildPlaceholderDialog(host, builder) + window.exec_() + + +def update_placeholder(*args): + host = registered_host() + builder = MayaTemplateBuilder(host) + placeholder_items_by_id = { + placeholder_item.scene_identifier: placeholder_item + for placeholder_item in builder.get_placeholders() + } + placeholder_items = [] + for node_name in cmds.ls(selection=True, long=True): + if node_name in placeholder_items_by_id: + placeholder_items.append(placeholder_items_by_id[node_name]) + + # TODO show UI at least + if len(placeholder_items) == 0: + raise ValueError("No node selected") + + if len(placeholder_items) > 1: + raise ValueError("Too many selected nodes") + + placeholder_item = placeholder_items[0] + window = WorkfileBuildPlaceholderDialog(host, builder) + window.set_update_mode(placeholder_item) + window.exec_() diff --git a/openpype/hosts/maya/hooks/pre_copy_mel.py b/openpype/hosts/maya/hooks/pre_copy_mel.py index b11e18241e..6f90af4b7c 100644 --- a/openpype/hosts/maya/hooks/pre_copy_mel.py +++ b/openpype/hosts/maya/hooks/pre_copy_mel.py @@ -1,5 +1,5 @@ from openpype.lib import PreLaunchHook -from openpype.hosts.maya.lib import copy_workspace_mel +from openpype.hosts.maya.lib import create_workspace_mel class PreCopyMel(PreLaunchHook): @@ -10,9 +10,10 @@ class PreCopyMel(PreLaunchHook): app_groups = ["maya"] def execute(self): + project_name = self.launch_context.env.get("AVALON_PROJECT") workdir = self.launch_context.env.get("AVALON_WORKDIR") if not workdir: self.log.warning("BUG: Workdir is not filled.") return - copy_workspace_mel(workdir) + create_workspace_mel(workdir, project_name) diff --git a/openpype/hosts/maya/lib.py b/openpype/hosts/maya/lib.py index 6c142053e6..ffb2f0b27c 100644 --- a/openpype/hosts/maya/lib.py +++ b/openpype/hosts/maya/lib.py @@ -1,26 +1,24 @@ import os -import shutil +from openpype.settings import get_project_settings +from openpype.lib import Logger -def copy_workspace_mel(workdir): - # Check that source mel exists - current_dir = os.path.dirname(os.path.abspath(__file__)) - src_filepath = os.path.join(current_dir, "resources", "workspace.mel") - if not os.path.exists(src_filepath): - print("Source mel file does not exist. {}".format(src_filepath)) - return - - # Skip if workspace.mel already exists +def create_workspace_mel(workdir, project_name): dst_filepath = os.path.join(workdir, "workspace.mel") if os.path.exists(dst_filepath): return - # Create workdir if does not exists yet if not os.path.exists(workdir): os.makedirs(workdir) - # Copy file - print("Copying workspace mel \"{}\" -> \"{}\"".format( - src_filepath, dst_filepath - )) - shutil.copy(src_filepath, dst_filepath) + project_setting = get_project_settings(project_name) + mel_script = project_setting["maya"].get("mel_workspace") + + # Skip if mel script in settings is empty + if not mel_script: + log = Logger.get_logger("create_workspace_mel") + log.debug("File 'workspace.mel' not created. Settings value is empty.") + return + + with open(dst_filepath, "w") as mel_file: + mel_file.write(mel_script) diff --git a/openpype/hosts/maya/plugins/inventory/select_containers.py b/openpype/hosts/maya/plugins/inventory/select_containers.py new file mode 100644 index 0000000000..f85bf17ab0 --- /dev/null +++ b/openpype/hosts/maya/plugins/inventory/select_containers.py @@ -0,0 +1,46 @@ +from maya import cmds + +from openpype.pipeline import InventoryAction, registered_host +from openpype.hosts.maya.api.lib import get_container_members + + +class SelectInScene(InventoryAction): + """Select nodes in the scene from selected containers in scene inventory""" + + label = "Select in scene" + icon = "search" + color = "#888888" + order = 99 + + def process(self, containers): + + all_members = [] + for container in containers: + members = get_container_members(container) + all_members.extend(members) + cmds.select(all_members, replace=True, noExpand=True) + + +class HighlightBySceneSelection(InventoryAction): + """Select containers in scene inventory from the current scene selection""" + + label = "Highlight by scene selection" + icon = "search" + color = "#888888" + order = 100 + + def process(self, containers): + + selection = set(cmds.ls(selection=True, long=True, objectsOnly=True)) + host = registered_host() + + to_select = [] + for container in host.get_containers(): + members = get_container_members(container) + if any(member in selection for member in members): + to_select.append(container["objectName"]) + + return { + "objectNames": to_select, + "options": {"clear": True} + } diff --git a/openpype/hosts/maya/plugins/publish/collect_assembly.py b/openpype/hosts/maya/plugins/publish/collect_assembly.py index 1a65bf1fde..2aef9ab908 100644 --- a/openpype/hosts/maya/plugins/publish/collect_assembly.py +++ b/openpype/hosts/maya/plugins/publish/collect_assembly.py @@ -70,7 +70,7 @@ class CollectAssembly(pyblish.api.InstancePlugin): data[representation_id].append(instance_data) instance.data["scenedata"] = dict(data) - instance.data["hierarchy"] = list(set(hierarchy_nodes)) + instance.data["nodesHierarchy"] = list(set(hierarchy_nodes)) def get_file_rule(self, rule): return mel.eval('workspace -query -fileRuleEntry "{}"'.format(rule)) diff --git a/openpype/hosts/maya/plugins/publish/collect_inputs.py b/openpype/hosts/maya/plugins/publish/collect_inputs.py new file mode 100644 index 0000000000..470fceffc9 --- /dev/null +++ b/openpype/hosts/maya/plugins/publish/collect_inputs.py @@ -0,0 +1,215 @@ +import copy +from bson.objectid import ObjectId + +from maya import cmds +import maya.api.OpenMaya as om +import pyblish.api + +from openpype.pipeline import registered_host +from openpype.hosts.maya.api.lib import get_container_members +from openpype.hosts.maya.api.lib_rendersetup import get_shader_in_layer + + +def iter_history(nodes, + filter=om.MFn.kInvalid, + direction=om.MItDependencyGraph.kUpstream): + """Iterate unique upstream history for list of nodes. + + This acts as a replacement to maya.cmds.listHistory. + It's faster by about 2x-3x. It returns less than + maya.cmds.listHistory as it excludes the input nodes + from the output (unless an input node was history + for another input node). It also excludes duplicates. + + Args: + nodes (list): Maya node names to start search from. + filter (om.MFn.Type): Filter to only specific types. + e.g. to dag nodes using om.MFn.kDagNode + direction (om.MItDependencyGraph.Direction): Direction to traverse in. + Defaults to upstream. + + Yields: + str: Node names in upstream history. + + """ + if not nodes: + return + + sel = om.MSelectionList() + for node in nodes: + sel.add(node) + + it = om.MItDependencyGraph(sel.getDependNode(0)) # init iterator + handle = om.MObjectHandle + + traversed = set() + fn_dep = om.MFnDependencyNode() + fn_dag = om.MFnDagNode() + for i in range(sel.length()): + + start_node = sel.getDependNode(i) + start_node_hash = handle(start_node).hashCode() + if start_node_hash in traversed: + continue + + it.resetTo(start_node, + filter=filter, + direction=direction) + while not it.isDone(): + + node = it.currentNode() + node_hash = handle(node).hashCode() + + if node_hash in traversed: + it.prune() + it.next() # noqa: B305 + continue + + traversed.add(node_hash) + + if node.hasFn(om.MFn.kDagNode): + fn_dag.setObject(node) + yield fn_dag.fullPathName() + else: + fn_dep.setObject(node) + yield fn_dep.name() + + it.next() # noqa: B305 + + +def collect_input_containers(containers, nodes): + """Collect containers that contain any of the node in `nodes`. + + This will return any loaded Avalon container that contains at least one of + the nodes. As such, the Avalon container is an input for it. Or in short, + there are member nodes of that container. + + Returns: + list: Input avalon containers + + """ + # Assume the containers have collected their cached '_members' data + # in the collector. + return [container for container in containers + if any(node in container["_members"] for node in nodes)] + + +class CollectUpstreamInputs(pyblish.api.InstancePlugin): + """Collect input source inputs for this publish. + + This will include `inputs` data of which loaded publishes were used in the + generation of this publish. This leaves an upstream trace to what was used + as input. + + """ + + label = "Collect Inputs" + order = pyblish.api.CollectorOrder + 0.34 + hosts = ["maya"] + + def process(self, instance): + + # For large scenes the querying of "host.ls()" can be relatively slow + # e.g. up to a second. Many instances calling it easily slows this + # down. As such, we cache it so we trigger it only once. + # todo: Instead of hidden cache make "CollectContainers" plug-in + cache_key = "__cache_containers" + scene_containers = instance.context.data.get(cache_key, None) + if scene_containers is None: + # Query the scenes' containers if there's no cache yet + host = registered_host() + scene_containers = list(host.ls()) + for container in scene_containers: + # Embed the members into the container dictionary + container_members = set(get_container_members(container)) + container["_members"] = container_members + instance.context.data["__cache_containers"] = scene_containers + + # Collect the relevant input containers for this instance + if "renderlayer" in set(instance.data.get("families", [])): + # Special behavior for renderlayers + self.log.debug("Collecting renderlayer inputs....") + containers = self._collect_renderlayer_inputs(scene_containers, + instance) + + else: + # Basic behavior + nodes = instance[:] + + # Include any input connections of history with long names + # For optimization purposes only trace upstream from shape nodes + # looking for used dag nodes. This way having just a constraint + # on a transform is also ignored which tended to give irrelevant + # inputs for the majority of our use cases. We tend to care more + # about geometry inputs. + shapes = cmds.ls(nodes, + type=("mesh", "nurbsSurface", "nurbsCurve"), + noIntermediate=True) + if shapes: + history = list(iter_history(shapes, filter=om.MFn.kShape)) + history = cmds.ls(history, long=True) + + # Include the transforms in the collected history as shapes + # are excluded from containers + transforms = cmds.listRelatives(cmds.ls(history, shapes=True), + parent=True, + fullPath=True, + type="transform") + if transforms: + history.extend(transforms) + + if history: + nodes = list(set(nodes + history)) + + # Collect containers for the given set of nodes + containers = collect_input_containers(scene_containers, + nodes) + + inputs = [ObjectId(c["representation"]) for c in containers] + instance.data["inputRepresentations"] = inputs + + self.log.info("Collected inputs: %s" % inputs) + + def _collect_renderlayer_inputs(self, scene_containers, instance): + """Collects inputs from nodes in renderlayer, incl. shaders + camera""" + + # Get the renderlayer + renderlayer = instance.data.get("setMembers") + + if renderlayer == "defaultRenderLayer": + # Assume all loaded containers in the scene are inputs + # for the masterlayer + return copy.deepcopy(scene_containers) + else: + # Get the members of the layer + members = cmds.editRenderLayerMembers(renderlayer, + query=True, + fullNames=True) or [] + + # In some cases invalid objects are returned from + # `editRenderLayerMembers` so we filter them out + members = cmds.ls(members, long=True) + + # Include all children + children = cmds.listRelatives(members, + allDescendents=True, + fullPath=True) or [] + members.extend(children) + + # Include assigned shaders in renderlayer + shapes = cmds.ls(members, shapes=True, long=True) + shaders = set() + for shape in shapes: + shape_shaders = get_shader_in_layer(shape, layer=renderlayer) + if not shape_shaders: + continue + shaders.update(shape_shaders) + members.extend(shaders) + + # Explicitly include the camera being rendered in renderlayer + cameras = instance.data.get("cameras") + members.extend(cameras) + + containers = collect_input_containers(scene_containers, members) + + return containers diff --git a/openpype/hosts/maya/plugins/publish/collect_maya_scene.py b/openpype/hosts/maya/plugins/publish/collect_maya_scene.py deleted file mode 100644 index eb21b17989..0000000000 --- a/openpype/hosts/maya/plugins/publish/collect_maya_scene.py +++ /dev/null @@ -1,25 +0,0 @@ -from maya import cmds - -import pyblish.api - - -class CollectMayaScene(pyblish.api.InstancePlugin): - """Collect Maya Scene Data - - """ - - order = pyblish.api.CollectorOrder + 0.2 - label = 'Collect Model Data' - families = ["mayaScene"] - - def process(self, instance): - # Extract only current frame (override) - frame = cmds.currentTime(query=True) - instance.data["frameStart"] = frame - instance.data["frameEnd"] = frame - - # make ftrack publishable - if instance.data.get('families'): - instance.data['families'].append('ftrack') - else: - instance.data['families'] = ['ftrack'] diff --git a/openpype/hosts/maya/plugins/publish/collect_maya_scene_time.py b/openpype/hosts/maya/plugins/publish/collect_maya_scene_time.py new file mode 100644 index 0000000000..7e198df14d --- /dev/null +++ b/openpype/hosts/maya/plugins/publish/collect_maya_scene_time.py @@ -0,0 +1,26 @@ +from maya import cmds + +import pyblish.api + + +class CollectMayaSceneTime(pyblish.api.InstancePlugin): + """Collect Maya Scene playback range + + This allows to reproduce the playback range for the content to be loaded. + It does *not* limit the extracted data to only data inside that time range. + + """ + + order = pyblish.api.CollectorOrder + 0.2 + label = 'Collect Maya Scene Time' + families = ["mayaScene"] + + def process(self, instance): + instance.data.update({ + "frameStart": cmds.playbackOptions(query=True, minTime=True), + "frameEnd": cmds.playbackOptions(query=True, maxTime=True), + "frameStartHandle": cmds.playbackOptions(query=True, + animationStartTime=True), + "frameEndHandle": cmds.playbackOptions(query=True, + animationEndTime=True) + }) diff --git a/openpype/hosts/maya/plugins/publish/collect_render.py b/openpype/hosts/maya/plugins/publish/collect_render.py index ebda5e190d..14aac2f206 100644 --- a/openpype/hosts/maya/plugins/publish/collect_render.py +++ b/openpype/hosts/maya/plugins/publish/collect_render.py @@ -293,6 +293,7 @@ class CollectMayaRender(pyblish.api.ContextPlugin): "source": filepath, "expectedFiles": full_exp_files, "publishRenderMetadataFolder": common_publish_meta_path, + "renderProducts": layer_render_products, "resolutionWidth": lib.get_attr_in_layer( "defaultResolution.width", layer=layer_name ), @@ -359,7 +360,6 @@ class CollectMayaRender(pyblish.api.ContextPlugin): instance.data["label"] = label instance.data["farm"] = True instance.data.update(data) - self.log.debug("data: {}".format(json.dumps(data, indent=4))) def parse_options(self, render_globals): """Get all overrides with a value, skip those without. diff --git a/openpype/hosts/maya/plugins/publish/collect_rig.py b/openpype/hosts/maya/plugins/publish/collect_rig.py deleted file mode 100644 index 98ae1e8009..0000000000 --- a/openpype/hosts/maya/plugins/publish/collect_rig.py +++ /dev/null @@ -1,22 +0,0 @@ -from maya import cmds - -import pyblish.api - - -class CollectRigData(pyblish.api.InstancePlugin): - """Collect rig data - - Ensures rigs are published to Ftrack. - - """ - - order = pyblish.api.CollectorOrder + 0.2 - label = 'Collect Rig Data' - families = ["rig"] - - def process(self, instance): - # make ftrack publishable - if instance.data.get('families'): - instance.data['families'].append('ftrack') - else: - instance.data['families'] = ['ftrack'] diff --git a/openpype/hosts/maya/plugins/publish/extract_ass.py b/openpype/hosts/maya/plugins/publish/extract_ass.py index 760f410f91..5c21a4ff08 100644 --- a/openpype/hosts/maya/plugins/publish/extract_ass.py +++ b/openpype/hosts/maya/plugins/publish/extract_ass.py @@ -1,12 +1,12 @@ import os -import openpype.api - from maya import cmds + +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractAssStandin(openpype.api.Extractor): +class ExtractAssStandin(publish.Extractor): """Extract the content of the instance to a ass file Things to pay attention to: diff --git a/openpype/hosts/maya/plugins/publish/extract_assembly.py b/openpype/hosts/maya/plugins/publish/extract_assembly.py index 482930b76e..35932003ee 100644 --- a/openpype/hosts/maya/plugins/publish/extract_assembly.py +++ b/openpype/hosts/maya/plugins/publish/extract_assembly.py @@ -1,14 +1,13 @@ +import os import json -import os - -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import extract_alembic from maya import cmds -class ExtractAssembly(openpype.api.Extractor): +class ExtractAssembly(publish.Extractor): """Produce an alembic of just point positions and normals. Positions and normals are preserved, but nothing more, @@ -33,7 +32,7 @@ class ExtractAssembly(openpype.api.Extractor): json.dump(instance.data["scenedata"], filepath, ensure_ascii=False) self.log.info("Extracting point cache ..") - cmds.select(instance.data["hierarchy"]) + cmds.select(instance.data["nodesHierarchy"]) # Run basic alembic exporter extract_alembic(file=hierarchy_path, diff --git a/openpype/hosts/maya/plugins/publish/extract_assproxy.py b/openpype/hosts/maya/plugins/publish/extract_assproxy.py index 93720dbb82..4937a28a9e 100644 --- a/openpype/hosts/maya/plugins/publish/extract_assproxy.py +++ b/openpype/hosts/maya/plugins/publish/extract_assproxy.py @@ -3,17 +3,17 @@ import contextlib from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractAssProxy(openpype.api.Extractor): +class ExtractAssProxy(publish.Extractor): """Extract proxy model as Maya Ascii to use as arnold standin """ - order = openpype.api.Extractor.order + 0.2 + order = publish.Extractor.order + 0.2 label = "Ass Proxy (Maya ASCII)" hosts = ["maya"] families = ["ass"] diff --git a/openpype/hosts/maya/plugins/publish/extract_camera_alembic.py b/openpype/hosts/maya/plugins/publish/extract_camera_alembic.py index b744bfd0fe..aa445a0387 100644 --- a/openpype/hosts/maya/plugins/publish/extract_camera_alembic.py +++ b/openpype/hosts/maya/plugins/publish/extract_camera_alembic.py @@ -2,11 +2,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api import lib -class ExtractCameraAlembic(openpype.api.Extractor): +class ExtractCameraAlembic(publish.Extractor): """Extract a Camera as Alembic. The cameras gets baked to world space by default. Only when the instance's diff --git a/openpype/hosts/maya/plugins/publish/extract_camera_mayaScene.py b/openpype/hosts/maya/plugins/publish/extract_camera_mayaScene.py index 8d6c4b5f3c..7467fa027d 100644 --- a/openpype/hosts/maya/plugins/publish/extract_camera_mayaScene.py +++ b/openpype/hosts/maya/plugins/publish/extract_camera_mayaScene.py @@ -5,7 +5,7 @@ import itertools from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api import lib @@ -78,7 +78,7 @@ def unlock(plug): cmds.disconnectAttr(source, destination) -class ExtractCameraMayaScene(openpype.api.Extractor): +class ExtractCameraMayaScene(publish.Extractor): """Extract a Camera as Maya Scene. This will create a duplicate of the camera that will be baked *with* diff --git a/openpype/hosts/maya/plugins/publish/extract_fbx.py b/openpype/hosts/maya/plugins/publish/extract_fbx.py index fbbe8e06b0..9af3acef65 100644 --- a/openpype/hosts/maya/plugins/publish/extract_fbx.py +++ b/openpype/hosts/maya/plugins/publish/extract_fbx.py @@ -4,13 +4,13 @@ import os from maya import cmds # noqa import maya.mel as mel # noqa import pyblish.api -import openpype.api -from openpype.hosts.maya.api.lib import maintained_selection +from openpype.pipeline import publish +from openpype.hosts.maya.api.lib import maintained_selection from openpype.hosts.maya.api import fbx -class ExtractFBX(openpype.api.Extractor): +class ExtractFBX(publish.Extractor): """Extract FBX from Maya. This extracts reproducible FBX exports ignoring any of the diff --git a/openpype/hosts/maya/plugins/publish/extract_layout.py b/openpype/hosts/maya/plugins/publish/extract_layout.py index 991217684a..0f499b09b1 100644 --- a/openpype/hosts/maya/plugins/publish/extract_layout.py +++ b/openpype/hosts/maya/plugins/publish/extract_layout.py @@ -5,13 +5,11 @@ import json from maya import cmds from maya.api import OpenMaya as om -from bson.objectid import ObjectId - -from openpype.pipeline import legacy_io -import openpype.api +from openpype.client import get_representation_by_id +from openpype.pipeline import legacy_io, publish -class ExtractLayout(openpype.api.Extractor): +class ExtractLayout(publish.Extractor): """Extract a layout.""" label = "Extract Layout" @@ -30,6 +28,8 @@ class ExtractLayout(openpype.api.Extractor): instance.data["representations"] = [] json_data = [] + # TODO representation queries can be refactored to be faster + project_name = legacy_io.active_project() for asset in cmds.sets(str(instance), query=True): # Find the container @@ -43,11 +43,11 @@ class ExtractLayout(openpype.api.Extractor): representation_id = cmds.getAttr(f"{container}.representation") - representation = legacy_io.find_one( - { - "type": "representation", - "_id": ObjectId(representation_id) - }, projection={"parent": True, "context.family": True}) + representation = get_representation_by_id( + project_name, + representation_id, + fields=["parent", "context.family"] + ) self.log.info(representation) @@ -102,9 +102,10 @@ class ExtractLayout(openpype.api.Extractor): for i in range(0, len(t_matrix_list), row_length): t_matrix.append(t_matrix_list[i:i + row_length]) - json_element["transform_matrix"] = [] - for row in t_matrix: - json_element["transform_matrix"].append(list(row)) + json_element["transform_matrix"] = [ + list(row) + for row in t_matrix + ] basis_list = [ 1, 0, 0, 0, diff --git a/openpype/hosts/maya/plugins/publish/extract_look.py b/openpype/hosts/maya/plugins/publish/extract_look.py index ce3b265566..403b4ee6bc 100644 --- a/openpype/hosts/maya/plugins/publish/extract_look.py +++ b/openpype/hosts/maya/plugins/publish/extract_look.py @@ -13,8 +13,8 @@ from maya import cmds # noqa import pyblish.api -import openpype.api -from openpype.pipeline import legacy_io +from openpype.lib import source_hash, run_subprocess +from openpype.pipeline import legacy_io, publish from openpype.hosts.maya.api import lib # Modes for transfer @@ -68,7 +68,7 @@ def find_paths_by_hash(texture_hash): return legacy_io.distinct(key, {"type": "version"}) -def maketx(source, destination, *args): +def maketx(source, destination, args, logger): """Make `.tx` using `maketx` with some default settings. The settings are based on default as used in Arnold's @@ -79,7 +79,8 @@ def maketx(source, destination, *args): Args: source (str): Path to source file. destination (str): Writing destination path. - *args: Additional arguments for `maketx`. + args (list): Additional arguments for `maketx`. + logger (logging.Logger): Logger to log messages to. Returns: str: Output of `maketx` command. @@ -94,7 +95,7 @@ def maketx(source, destination, *args): "OIIO tool not found in {}".format(maketx_path)) raise AssertionError("OIIO tool not found") - cmd = [ + subprocess_args = [ maketx_path, "-v", # verbose "-u", # update mode @@ -103,27 +104,20 @@ def maketx(source, destination, *args): "--checknan", # use oiio-optimized settings for tile-size, planarconfig, metadata "--oiio", - "--filter lanczos3", - escape_space(source) + "--filter", "lanczos3", + source ] - cmd.extend(args) - cmd.extend(["-o", escape_space(destination)]) + subprocess_args.extend(args) + subprocess_args.extend(["-o", destination]) - cmd = " ".join(cmd) + cmd = " ".join(subprocess_args) + logger.debug(cmd) - CREATE_NO_WINDOW = 0x08000000 # noqa - kwargs = dict(args=cmd, stderr=subprocess.STDOUT) - - if sys.platform == "win32": - kwargs["creationflags"] = CREATE_NO_WINDOW try: - out = subprocess.check_output(**kwargs) - except subprocess.CalledProcessError as exc: - print(exc) - import traceback - - traceback.print_exc() + out = run_subprocess(subprocess_args) + except Exception: + logger.error("Maketx converion failed", exc_info=True) raise return out @@ -161,7 +155,7 @@ def no_workspace_dir(): os.rmdir(fake_workspace_dir) -class ExtractLook(openpype.api.Extractor): +class ExtractLook(publish.Extractor): """Extract Look (Maya Scene + JSON) Only extracts the sets (shadingEngines and alike) alongside a .json file @@ -505,7 +499,7 @@ class ExtractLook(openpype.api.Extractor): args = [] if do_maketx: args.append("maketx") - texture_hash = openpype.api.source_hash(filepath, *args) + texture_hash = source_hash(filepath, *args) # If source has been published before with the same settings, # then don't reprocess but hardlink from the original @@ -524,15 +518,17 @@ class ExtractLook(openpype.api.Extractor): if do_maketx and ext != ".tx": # Produce .tx file in staging if source file is not .tx converted = os.path.join(staging, "resources", fname + ".tx") - + additional_args = [ + "--sattrib", + "sourceHash", + texture_hash + ] if linearize: self.log.info("tx: converting sRGB -> linear") - colorconvert = "--colorconvert sRGB linear" - else: - colorconvert = "" + additional_args.extend(["--colorconvert", "sRGB", "linear"]) config_path = get_ocio_config_path("nuke-default") - color_config = "--colorconfig {0}".format(config_path) + additional_args.extend(["--colorconfig", config_path]) # Ensure folder exists if not os.path.exists(os.path.dirname(converted)): os.makedirs(os.path.dirname(converted)) @@ -541,12 +537,8 @@ class ExtractLook(openpype.api.Extractor): maketx( filepath, converted, - # Include `source-hash` as string metadata - "--sattrib", - "sourceHash", - escape_space(texture_hash), - colorconvert, - color_config + additional_args, + self.log ) return converted, COPY, texture_hash diff --git a/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py b/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py index 3a47cdadb5..3769ec3605 100644 --- a/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py +++ b/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py @@ -4,12 +4,11 @@ import os from maya import cmds -import openpype.api from openpype.hosts.maya.api.lib import maintained_selection -from openpype.pipeline import AVALON_CONTAINER_ID +from openpype.pipeline import AVALON_CONTAINER_ID, publish -class ExtractMayaSceneRaw(openpype.api.Extractor): +class ExtractMayaSceneRaw(publish.Extractor): """Extract as Maya Scene (raw). This will preserve all references, construction history, etc. diff --git a/openpype/hosts/maya/plugins/publish/extract_model.py b/openpype/hosts/maya/plugins/publish/extract_model.py index 0282d1e9c8..7c8c3a2981 100644 --- a/openpype/hosts/maya/plugins/publish/extract_model.py +++ b/openpype/hosts/maya/plugins/publish/extract_model.py @@ -4,11 +4,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api import lib -class ExtractModel(openpype.api.Extractor): +class ExtractModel(publish.Extractor): """Extract as Model (Maya Scene). Only extracts contents based on the original "setMembers" data to ensure diff --git a/openpype/hosts/maya/plugins/publish/extract_multiverse_look.py b/openpype/hosts/maya/plugins/publish/extract_multiverse_look.py index 82e2b41929..92137acb95 100644 --- a/openpype/hosts/maya/plugins/publish/extract_multiverse_look.py +++ b/openpype/hosts/maya/plugins/publish/extract_multiverse_look.py @@ -2,11 +2,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractMultiverseLook(openpype.api.Extractor): +class ExtractMultiverseLook(publish.Extractor): """Extractor for Multiverse USD look data. This will extract: diff --git a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd.py b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd.py index 3654be7b34..6c352bebe6 100644 --- a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd.py +++ b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd.py @@ -3,11 +3,11 @@ import six from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractMultiverseUsd(openpype.api.Extractor): +class ExtractMultiverseUsd(publish.Extractor): """Extractor for Multiverse USD Asset data. This will extract settings for a Multiverse Write Asset operation: diff --git a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_comp.py b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_comp.py index ad9303657f..a62729c198 100644 --- a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_comp.py +++ b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_comp.py @@ -2,11 +2,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractMultiverseUsdComposition(openpype.api.Extractor): +class ExtractMultiverseUsdComposition(publish.Extractor): """Extractor of Multiverse USD Composition data. This will extract settings for a Multiverse Write Composition operation: diff --git a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_over.py b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_over.py index d44e3878b8..0628623e88 100644 --- a/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_over.py +++ b/openpype/hosts/maya/plugins/publish/extract_multiverse_usd_over.py @@ -1,12 +1,12 @@ import os -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection from maya import cmds -class ExtractMultiverseUsdOverride(openpype.api.Extractor): +class ExtractMultiverseUsdOverride(publish.Extractor): """Extractor for Multiverse USD Override data. This will extract settings for a Multiverse Write Override operation: diff --git a/openpype/hosts/maya/plugins/publish/extract_playblast.py b/openpype/hosts/maya/plugins/publish/extract_playblast.py index 871adda0c3..81fdba2f98 100644 --- a/openpype/hosts/maya/plugins/publish/extract_playblast.py +++ b/openpype/hosts/maya/plugins/publish/extract_playblast.py @@ -1,18 +1,16 @@ import os -import glob -import contextlib import clique import capture +from openpype.pipeline import publish from openpype.hosts.maya.api import lib -import openpype.api from maya import cmds import pymel.core as pm -class ExtractPlayblast(openpype.api.Extractor): +class ExtractPlayblast(publish.Extractor): """Extract viewport playblast. Takes review camera and creates review Quicktime video based on viewport diff --git a/openpype/hosts/maya/plugins/publish/extract_pointcache.py b/openpype/hosts/maya/plugins/publish/extract_pointcache.py index bf6feecef3..7c1c6d5c12 100644 --- a/openpype/hosts/maya/plugins/publish/extract_pointcache.py +++ b/openpype/hosts/maya/plugins/publish/extract_pointcache.py @@ -2,7 +2,7 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import ( extract_alembic, suspended_refresh, @@ -11,7 +11,7 @@ from openpype.hosts.maya.api.lib import ( ) -class ExtractAlembic(openpype.api.Extractor): +class ExtractAlembic(publish.Extractor): """Produce an alembic of just point positions and normals. Positions and normals, uvs, creases are preserved, but nothing more, diff --git a/openpype/hosts/maya/plugins/publish/extract_redshift_proxy.py b/openpype/hosts/maya/plugins/publish/extract_redshift_proxy.py index 23cac9190d..4377275635 100644 --- a/openpype/hosts/maya/plugins/publish/extract_redshift_proxy.py +++ b/openpype/hosts/maya/plugins/publish/extract_redshift_proxy.py @@ -4,11 +4,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractRedshiftProxy(openpype.api.Extractor): +class ExtractRedshiftProxy(publish.Extractor): """Extract the content of the instance to a redshift proxy file.""" label = "Redshift Proxy (.rs)" diff --git a/openpype/hosts/maya/plugins/publish/extract_rendersetup.py b/openpype/hosts/maya/plugins/publish/extract_rendersetup.py index 6bdd5f590e..5970c038a4 100644 --- a/openpype/hosts/maya/plugins/publish/extract_rendersetup.py +++ b/openpype/hosts/maya/plugins/publish/extract_rendersetup.py @@ -1,10 +1,11 @@ -import json import os -import openpype.api +import json + import maya.app.renderSetup.model.renderSetup as renderSetup +from openpype.pipeline import publish -class ExtractRenderSetup(openpype.api.Extractor): +class ExtractRenderSetup(publish.Extractor): """ Produce renderSetup template file diff --git a/openpype/hosts/maya/plugins/publish/extract_rig.py b/openpype/hosts/maya/plugins/publish/extract_rig.py index 53c1eeb671..c71a2f710d 100644 --- a/openpype/hosts/maya/plugins/publish/extract_rig.py +++ b/openpype/hosts/maya/plugins/publish/extract_rig.py @@ -4,11 +4,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractRig(openpype.api.Extractor): +class ExtractRig(publish.Extractor): """Extract rig as Maya Scene.""" label = "Extract Rig (Maya Scene)" diff --git a/openpype/hosts/maya/plugins/publish/extract_thumbnail.py b/openpype/hosts/maya/plugins/publish/extract_thumbnail.py index 9380da5128..854301ea48 100644 --- a/openpype/hosts/maya/plugins/publish/extract_thumbnail.py +++ b/openpype/hosts/maya/plugins/publish/extract_thumbnail.py @@ -3,14 +3,14 @@ import glob import capture +from openpype.pipeline import publish from openpype.hosts.maya.api import lib -import openpype.api from maya import cmds import pymel.core as pm -class ExtractThumbnail(openpype.api.Extractor): +class ExtractThumbnail(publish.Extractor): """Extract viewport thumbnail. Takes review camera and creates a thumbnail based on viewport diff --git a/openpype/hosts/maya/plugins/publish/extract_unreal_skeletalmesh.py b/openpype/hosts/maya/plugins/publish/extract_unreal_skeletalmesh.py index 7ef7f2f181..258120db2f 100644 --- a/openpype/hosts/maya/plugins/publish/extract_unreal_skeletalmesh.py +++ b/openpype/hosts/maya/plugins/publish/extract_unreal_skeletalmesh.py @@ -6,7 +6,8 @@ from contextlib import contextmanager from maya import cmds # noqa import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.maya.api import fbx @@ -20,7 +21,7 @@ def renamed(original_name, renamed_name): cmds.rename(renamed_name, original_name) -class ExtractUnrealSkeletalMesh(openpype.api.Extractor): +class ExtractUnrealSkeletalMesh(publish.Extractor): """Extract Unreal Skeletal Mesh as FBX from Maya. """ order = pyblish.api.ExtractorOrder - 0.1 diff --git a/openpype/hosts/maya/plugins/publish/extract_unreal_staticmesh.py b/openpype/hosts/maya/plugins/publish/extract_unreal_staticmesh.py index 69d51f9ff1..44f0615a27 100644 --- a/openpype/hosts/maya/plugins/publish/extract_unreal_staticmesh.py +++ b/openpype/hosts/maya/plugins/publish/extract_unreal_staticmesh.py @@ -5,7 +5,8 @@ import os from maya import cmds # noqa import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import ( parent_nodes, maintained_selection @@ -13,7 +14,7 @@ from openpype.hosts.maya.api.lib import ( from openpype.hosts.maya.api import fbx -class ExtractUnrealStaticMesh(openpype.api.Extractor): +class ExtractUnrealStaticMesh(publish.Extractor): """Extract Unreal Static Mesh as FBX from Maya. """ order = pyblish.api.ExtractorOrder - 0.1 diff --git a/openpype/hosts/maya/plugins/publish/extract_vrayproxy.py b/openpype/hosts/maya/plugins/publish/extract_vrayproxy.py index 562ca078e1..38bf02245a 100644 --- a/openpype/hosts/maya/plugins/publish/extract_vrayproxy.py +++ b/openpype/hosts/maya/plugins/publish/extract_vrayproxy.py @@ -2,11 +2,11 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import maintained_selection -class ExtractVRayProxy(openpype.api.Extractor): +class ExtractVRayProxy(publish.Extractor): """Extract the content of the instance to a vrmesh file Things to pay attention to: diff --git a/openpype/hosts/maya/plugins/publish/extract_vrayscene.py b/openpype/hosts/maya/plugins/publish/extract_vrayscene.py index 5d41697e5f..8442df1611 100644 --- a/openpype/hosts/maya/plugins/publish/extract_vrayscene.py +++ b/openpype/hosts/maya/plugins/publish/extract_vrayscene.py @@ -3,14 +3,14 @@ import os import re -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.render_setup_tools import export_in_rs_layer from openpype.hosts.maya.api.lib import maintained_selection from maya import cmds -class ExtractVrayscene(openpype.api.Extractor): +class ExtractVrayscene(publish.Extractor): """Extractor for vrscene.""" label = "VRay Scene (.vrscene)" diff --git a/openpype/hosts/maya/plugins/publish/extract_xgen_cache.py b/openpype/hosts/maya/plugins/publish/extract_xgen_cache.py index 5728682abe..77350f343e 100644 --- a/openpype/hosts/maya/plugins/publish/extract_xgen_cache.py +++ b/openpype/hosts/maya/plugins/publish/extract_xgen_cache.py @@ -2,14 +2,14 @@ import os from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api.lib import ( suspended_refresh, maintained_selection ) -class ExtractXgenCache(openpype.api.Extractor): +class ExtractXgenCache(publish.Extractor): """Produce an alembic of just xgen interactive groom """ diff --git a/openpype/hosts/maya/plugins/publish/extract_yeti_cache.py b/openpype/hosts/maya/plugins/publish/extract_yeti_cache.py index cf6db00e9a..b61f599cab 100644 --- a/openpype/hosts/maya/plugins/publish/extract_yeti_cache.py +++ b/openpype/hosts/maya/plugins/publish/extract_yeti_cache.py @@ -3,10 +3,10 @@ import json from maya import cmds -import openpype.api +from openpype.pipeline import publish -class ExtractYetiCache(openpype.api.Extractor): +class ExtractYetiCache(publish.Extractor): """Producing Yeti cache files using scene time range. This will extract Yeti cache file sequence and fur settings. diff --git a/openpype/hosts/maya/plugins/publish/extract_yeti_rig.py b/openpype/hosts/maya/plugins/publish/extract_yeti_rig.py index 6e21bffa4e..1d0c5e88c3 100644 --- a/openpype/hosts/maya/plugins/publish/extract_yeti_rig.py +++ b/openpype/hosts/maya/plugins/publish/extract_yeti_rig.py @@ -7,7 +7,7 @@ import contextlib from maya import cmds -import openpype.api +from openpype.pipeline import publish from openpype.hosts.maya.api import lib @@ -90,7 +90,7 @@ def yetigraph_attribute_values(assumed_destination, resources): pass -class ExtractYetiRig(openpype.api.Extractor): +class ExtractYetiRig(publish.Extractor): """Extract the Yeti rig to a Maya Scene and write the Yeti rig data.""" label = "Extract Yeti Rig" diff --git a/openpype/hosts/maya/plugins/publish/save_scene.py b/openpype/hosts/maya/plugins/publish/save_scene.py index 50a2f2112a..45e62e7b44 100644 --- a/openpype/hosts/maya/plugins/publish/save_scene.py +++ b/openpype/hosts/maya/plugins/publish/save_scene.py @@ -1,4 +1,8 @@ import pyblish.api +from openpype.pipeline.workfile.lock_workfile import ( + is_workfile_lock_enabled, + remove_workfile_lock +) class SaveCurrentScene(pyblish.api.ContextPlugin): @@ -22,6 +26,10 @@ class SaveCurrentScene(pyblish.api.ContextPlugin): self.log.debug("Skipping file save as there " "are no modifications..") return - + project_name = context.data["projectName"] + project_settings = context.data["project_settings"] + # remove lockfile before saving + if is_workfile_lock_enabled("maya", project_name, project_settings): + remove_workfile_lock(current) self.log.info("Saving current file..") cmds.file(save=True, force=True) diff --git a/openpype/hosts/maya/plugins/publish/validate_assembly_transforms.py b/openpype/hosts/maya/plugins/publish/validate_assembly_transforms.py index f793846555..fb25b617be 100644 --- a/openpype/hosts/maya/plugins/publish/validate_assembly_transforms.py +++ b/openpype/hosts/maya/plugins/publish/validate_assembly_transforms.py @@ -48,7 +48,7 @@ class ValidateAssemblyModelTransforms(pyblish.api.InstancePlugin): from openpype.hosts.maya.api import lib # Get all transforms in the loaded containers - container_roots = cmds.listRelatives(instance.data["hierarchy"], + container_roots = cmds.listRelatives(instance.data["nodesHierarchy"], children=True, type="transform", fullPath=True) diff --git a/openpype/hosts/maya/plugins/publish/validate_rendersettings.py b/openpype/hosts/maya/plugins/publish/validate_rendersettings.py index 08ecc0d149..2474b2ead6 100644 --- a/openpype/hosts/maya/plugins/publish/validate_rendersettings.py +++ b/openpype/hosts/maya/plugins/publish/validate_rendersettings.py @@ -13,6 +13,12 @@ from openpype.pipeline.publish import ( from openpype.hosts.maya.api import lib +def get_redshift_image_format_labels(): + """Return nice labels for Redshift image formats.""" + var = "$g_redshiftImageFormatLabels" + return mel.eval("{0}={0}".format(var)) + + class ValidateRenderSettings(pyblish.api.InstancePlugin): """Validates the global render settings @@ -105,8 +111,9 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): # Get the node attributes for current renderer attrs = lib.RENDER_ATTRS.get(renderer, lib.RENDER_ATTRS['default']) + # Prefix attribute can return None when a value was never set prefix = lib.get_attr_in_layer(cls.ImagePrefixes[renderer], - layer=layer) + layer=layer) or "" padding = lib.get_attr_in_layer("{node}.{padding}".format(**attrs), layer=layer) @@ -183,18 +190,22 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): redshift_AOV_prefix )) invalid = True - # get aov format - aov_ext = cmds.getAttr( - "{}.fileFormat".format(aov), asString=True) - default_ext = cmds.getAttr( - "redshiftOptions.imageFormat", asString=True) + # check aov file format + aov_ext = cmds.getAttr("{}.fileFormat".format(aov)) + default_ext = cmds.getAttr("redshiftOptions.imageFormat") + aov_type = cmds.getAttr("{}.aovType".format(aov)) + if aov_type == "Cryptomatte": + # redshift Cryptomatte AOV always uses "Cryptomatte (EXR)" + # so we ignore validating file format for it. + pass - if default_ext != aov_ext: - cls.log.error(("AOV file format is not the same " - "as the one set globally " - "{} != {}").format(default_ext, - aov_ext)) + elif default_ext != aov_ext: + labels = get_redshift_image_format_labels() + cls.log.error( + "AOV file format {} does not match global file format " + "{}".format(labels[aov_ext], labels[default_ext]) + ) invalid = True if renderer == "renderman": @@ -302,6 +313,9 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): default = lib.RENDER_ATTRS['default'] render_attrs = lib.RENDER_ATTRS.get(renderer, default) + # Repair animation must be enabled + cmds.setAttr("defaultRenderGlobals.animation", True) + # Repair prefix if renderer != "renderman": node = render_attrs["node"] @@ -334,8 +348,7 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): cmds.optionMenuGrp("vrayRenderElementSeparator", v=instance.data.get("aovSeparator", "_")) cmds.setAttr( - "{}.fileNameRenderElementSeparator".format( - node), + "{}.fileNameRenderElementSeparator".format(node), instance.data.get("aovSeparator", "_"), type="string" ) diff --git a/openpype/hosts/maya/resources/workspace.mel b/openpype/hosts/maya/resources/workspace.mel deleted file mode 100644 index f7213fa4f6..0000000000 --- a/openpype/hosts/maya/resources/workspace.mel +++ /dev/null @@ -1,11 +0,0 @@ -//Maya 2018 Project Definition - -workspace -fr "shaders" "renderData/shaders"; -workspace -fr "alembicCache" "cache/alembic"; -workspace -fr "mayaAscii" ""; -workspace -fr "mayaBinary" ""; -workspace -fr "renderData" "renderData"; -workspace -fr "fileCache" "cache/nCache"; -workspace -fr "scene" ""; -workspace -fr "sourceImages" "sourceimages"; -workspace -fr "images" "renders"; diff --git a/openpype/hosts/nuke/api/__init__.py b/openpype/hosts/nuke/api/__init__.py index 962f31c177..c65058874b 100644 --- a/openpype/hosts/nuke/api/__init__.py +++ b/openpype/hosts/nuke/api/__init__.py @@ -21,6 +21,8 @@ from .pipeline import ( containerise, parse_container, update_container, + + get_workfile_build_placeholder_plugins, ) from .lib import ( maintained_selection, @@ -55,6 +57,8 @@ __all__ = ( "parse_container", "update_container", + "get_workfile_build_placeholder_plugins", + "maintained_selection", "reset_selection", "get_view_process_node", diff --git a/openpype/hosts/nuke/api/lib_template_builder.py b/openpype/hosts/nuke/api/lib_template_builder.py deleted file mode 100644 index 61baa23928..0000000000 --- a/openpype/hosts/nuke/api/lib_template_builder.py +++ /dev/null @@ -1,220 +0,0 @@ -from collections import OrderedDict - -import qargparse - -import nuke - -from openpype.tools.utils.widgets import OptionDialog - -from .lib import imprint, get_main_window - - -# To change as enum -build_types = ["context_asset", "linked_asset", "all_assets"] - - -def get_placeholder_attributes(node, enumerate=False): - list_atts = { - "builder_type", - "family", - "representation", - "loader", - "loader_args", - "order", - "asset", - "subset", - "hierarchy", - "siblings", - "last_loaded" - } - attributes = {} - for attr in node.knobs().keys(): - if attr in list_atts: - if enumerate: - try: - attributes[attr] = node.knob(attr).values() - except AttributeError: - attributes[attr] = node.knob(attr).getValue() - else: - attributes[attr] = node.knob(attr).getValue() - - return attributes - - -def delete_placeholder_attributes(node): - """Delete all extra placeholder attributes.""" - - extra_attributes = get_placeholder_attributes(node) - for attribute in extra_attributes.keys(): - try: - node.removeKnob(node.knob(attribute)) - except ValueError: - continue - - -def hide_placeholder_attributes(node): - """Hide all extra placeholder attributes.""" - - extra_attributes = get_placeholder_attributes(node) - for attribute in extra_attributes.keys(): - try: - node.knob(attribute).setVisible(False) - except ValueError: - continue - - -def create_placeholder(): - args = placeholder_window() - if not args: - # operation canceled, no locator created - return - - placeholder = nuke.nodes.NoOp() - placeholder.setName("PLACEHOLDER") - placeholder.knob("tile_color").setValue(4278190335) - - # custom arg parse to force empty data query - # and still imprint them on placeholder - # and getting items when arg is of type Enumerator - options = OrderedDict() - for arg in args: - if not type(arg) == qargparse.Separator: - options[str(arg)] = arg._data.get("items") or arg.read() - imprint(placeholder, options) - imprint(placeholder, {"is_placeholder": True}) - placeholder.knob("is_placeholder").setVisible(False) - - -def update_placeholder(): - placeholder = nuke.selectedNodes() - if not placeholder: - raise ValueError("No node selected") - if len(placeholder) > 1: - raise ValueError("Too many selected nodes") - placeholder = placeholder[0] - - args = placeholder_window(get_placeholder_attributes(placeholder)) - if not args: - return # operation canceled - # delete placeholder attributes - delete_placeholder_attributes(placeholder) - - options = OrderedDict() - for arg in args: - if not type(arg) == qargparse.Separator: - options[str(arg)] = arg._data.get("items") or arg.read() - imprint(placeholder, options) - - -def imprint_enum(placeholder, args): - """ - Imprint method doesn't act properly with enums. - Replacing the functionnality with this for now - """ - - enum_values = { - str(arg): arg.read() - for arg in args - if arg._data.get("items") - } - string_to_value_enum_table = { - build: idx - for idx, build in enumerate(build_types) - } - attrs = {} - for key, value in enum_values.items(): - attrs[key] = string_to_value_enum_table[value] - - -def placeholder_window(options=None): - options = options or dict() - dialog = OptionDialog(parent=get_main_window()) - dialog.setWindowTitle("Create Placeholder") - - args = [ - qargparse.Separator("Main attributes"), - qargparse.Enum( - "builder_type", - label="Asset Builder Type", - default=options.get("builder_type", 0), - items=build_types, - help="""Asset Builder Type -Builder type describe what template loader will look for. - -context_asset : Template loader will look for subsets of -current context asset (Asset bob will find asset) - -linked_asset : Template loader will look for assets linked -to current context asset. -Linked asset are looked in OpenPype database under field "inputLinks" -""" - ), - qargparse.String( - "family", - default=options.get("family", ""), - label="OpenPype Family", - placeholder="ex: image, plate ..."), - qargparse.String( - "representation", - default=options.get("representation", ""), - label="OpenPype Representation", - placeholder="ex: mov, png ..."), - qargparse.String( - "loader", - default=options.get("loader", ""), - label="Loader", - placeholder="ex: LoadClip, LoadImage ...", - help="""Loader - -Defines what openpype loader will be used to load assets. -Useable loader depends on current host's loader list. -Field is case sensitive. -"""), - qargparse.String( - "loader_args", - default=options.get("loader_args", ""), - label="Loader Arguments", - placeholder='ex: {"camera":"persp", "lights":True}', - help="""Loader - -Defines a dictionnary of arguments used to load assets. -Useable arguments depend on current placeholder Loader. -Field should be a valid python dict. Anything else will be ignored. -"""), - qargparse.Integer( - "order", - default=options.get("order", 0), - min=0, - max=999, - label="Order", - placeholder="ex: 0, 100 ... (smallest order loaded first)", - help="""Order - -Order defines asset loading priority (0 to 999) -Priority rule is : "lowest is first to load"."""), - qargparse.Separator( - "Optional attributes "), - qargparse.String( - "asset", - default=options.get("asset", ""), - label="Asset filter", - placeholder="regex filtering by asset name", - help="Filtering assets by matching field regex to asset's name"), - qargparse.String( - "subset", - default=options.get("subset", ""), - label="Subset filter", - placeholder="regex filtering by subset name", - help="Filtering assets by matching field regex to subset's name"), - qargparse.String( - "hierarchy", - default=options.get("hierarchy", ""), - label="Hierarchy filter", - placeholder="regex filtering by asset's hierarchy", - help="Filtering assets by matching field asset's hierarchy") - ] - dialog.create(args) - if not dialog.exec_(): - return None - - return args diff --git a/openpype/hosts/nuke/api/pipeline.py b/openpype/hosts/nuke/api/pipeline.py index bac42128cc..c6ccfaeb3a 100644 --- a/openpype/hosts/nuke/api/pipeline.py +++ b/openpype/hosts/nuke/api/pipeline.py @@ -22,10 +22,6 @@ from openpype.pipeline import ( AVALON_CONTAINER_ID, ) from openpype.pipeline.workfile import BuildWorkfile -from openpype.pipeline.workfile.build_template import ( - build_workfile_template, - update_workfile_template -) from openpype.tools.utils import host_tools from .command import viewer_update_and_undo_stop @@ -40,8 +36,12 @@ from .lib import ( set_avalon_knob_data, read_avalon_data, ) -from .lib_template_builder import ( - create_placeholder, update_placeholder +from .workfile_template_builder import ( + NukePlaceholderLoadPlugin, + build_workfile_template, + update_workfile_template, + create_placeholder, + update_placeholder, ) log = Logger.get_logger(__name__) @@ -141,6 +141,12 @@ def _show_workfiles(): host_tools.show_workfiles(parent=None, on_top=False) +def get_workfile_build_placeholder_plugins(): + return [ + NukePlaceholderLoadPlugin + ] + + def _install_menu(): # uninstall original avalon menu main_window = get_main_window() diff --git a/openpype/hosts/nuke/api/template_loader.py b/openpype/hosts/nuke/api/workfile_template_builder.py similarity index 51% rename from openpype/hosts/nuke/api/template_loader.py rename to openpype/hosts/nuke/api/workfile_template_builder.py index 5ff4b8fc41..7a2e442e32 100644 --- a/openpype/hosts/nuke/api/template_loader.py +++ b/openpype/hosts/nuke/api/workfile_template_builder.py @@ -1,13 +1,16 @@ -import re import collections import nuke -from openpype.client import get_representations -from openpype.pipeline import legacy_io -from openpype.pipeline.workfile.abstract_template_loader import ( - AbstractPlaceholder, - AbstractTemplateLoader, +from openpype.pipeline import registered_host +from openpype.pipeline.workfile.workfile_template_builder import ( + AbstractTemplateBuilder, + PlaceholderPlugin, + LoadPlaceholderItem, + PlaceholderLoadMixin, +) +from openpype.tools.workfile_template_build import ( + WorkfileBuildPlaceholderDialog, ) from .lib import ( @@ -25,19 +28,11 @@ from .lib import ( node_tempfile, ) -from .lib_template_builder import ( - delete_placeholder_attributes, - get_placeholder_attributes, - hide_placeholder_attributes -) - PLACEHOLDER_SET = "PLACEHOLDERS_SET" -class NukeTemplateLoader(AbstractTemplateLoader): - """Concrete implementation of AbstractTemplateLoader for Nuke - - """ +class NukeTemplateBuilder(AbstractTemplateBuilder): + """Concrete implementation of AbstractTemplateBuilder for maya""" def import_template(self, path): """Import template into current scene. @@ -58,224 +53,255 @@ class NukeTemplateLoader(AbstractTemplateLoader): return True - def preload(self, placeholder, loaders_by_name, last_representation): - placeholder.data["nodes_init"] = nuke.allNodes() - placeholder.data["last_repre_id"] = str(last_representation["_id"]) - def populate_template(self, ignored_ids=None): - processed_key = "_node_processed" +class NukePlaceholderPlugin(PlaceholderPlugin): + node_color = 4278190335 - processed_nodes = [] - nodes = self.get_template_nodes() - while nodes: - # Mark nodes as processed so they're not re-executed - # - that can happen if processing of placeholder node fails - for node in nodes: - imprint(node, {processed_key: True}) - processed_nodes.append(node) + def _collect_scene_placeholders(self): + # Cache placeholder data to shared data + placeholder_nodes = self.builder.get_shared_populate_data( + "placeholder_nodes" + ) + if placeholder_nodes is None: + placeholder_nodes = {} + all_groups = collections.deque() + all_groups.append(nuke.thisGroup()) + while all_groups: + group = all_groups.popleft() + for node in group.nodes(): + if isinstance(node, nuke.Group): + all_groups.append(node) - super(NukeTemplateLoader, self).populate_template(ignored_ids) + node_knobs = node.knobs() + if ( + "builder_type" not in node_knobs + or "is_placeholder" not in node_knobs + or not node.knob("is_placeholder").value() + ): + continue - # Recollect nodes to repopulate - nodes = [] - for node in self.get_template_nodes(): - # Skip already processed nodes - if ( - processed_key in node.knobs() - and node.knob(processed_key).value() - ): - continue - nodes.append(node) + if "empty" in node_knobs and node.knob("empty").value(): + continue - for node in processed_nodes: - knob = node.knob(processed_key) + placeholder_nodes[node.fullName()] = node + + self.builder.set_shared_populate_data( + "placeholder_nodes", placeholder_nodes + ) + return placeholder_nodes + + def create_placeholder(self, placeholder_data): + placeholder_data["plugin_identifier"] = self.identifier + + placeholder = nuke.nodes.NoOp() + placeholder.setName("PLACEHOLDER") + placeholder.knob("tile_color").setValue(self.node_color) + + imprint(placeholder, placeholder_data) + imprint(placeholder, {"is_placeholder": True}) + placeholder.knob("is_placeholder").setVisible(False) + + def update_placeholder(self, placeholder_item, placeholder_data): + node = nuke.toNode(placeholder_item.scene_identifier) + imprint(node, placeholder_data) + + def _parse_placeholder_node_data(self, node): + placeholder_data = {} + for key in self.get_placeholder_keys(): + knob = node.knob(key) + value = None if knob is not None: - node.removeKnob(knob) - - @staticmethod - def get_template_nodes(): - placeholders = [] - all_groups = collections.deque() - all_groups.append(nuke.thisGroup()) - while all_groups: - group = all_groups.popleft() - for node in group.nodes(): - if isinstance(node, nuke.Group): - all_groups.append(node) - - node_knobs = node.knobs() - if ( - "builder_type" not in node_knobs - or "is_placeholder" not in node_knobs - or not node.knob("is_placeholder").value() - ): - continue - - if "empty" in node_knobs and node.knob("empty").value(): - continue - - placeholders.append(node) - - return placeholders - - def update_missing_containers(self): - nodes_by_id = collections.defaultdict(list) - - for node in nuke.allNodes(): - node_knobs = node.knobs().keys() - if "repre_id" in node_knobs: - repre_id = node.knob("repre_id").getValue() - nodes_by_id[repre_id].append(node.name()) - - if "empty" in node_knobs: - node.removeKnob(node.knob("empty")) - imprint(node, {"empty": False}) - - for node_names in nodes_by_id.values(): - node = None - for node_name in node_names: - node_by_name = nuke.toNode(node_name) - if "builder_type" in node_by_name.knobs().keys(): - node = node_by_name - break - - if node is None: - continue - - placeholder = nuke.nodes.NoOp() - placeholder.setName("PLACEHOLDER") - placeholder.knob("tile_color").setValue(4278190335) - attributes = get_placeholder_attributes(node, enumerate=True) - imprint(placeholder, attributes) - pos_x = int(node.knob("x").getValue()) - pos_y = int(node.knob("y").getValue()) - placeholder.setXYpos(pos_x, pos_y) - imprint(placeholder, {"nb_children": 1}) - refresh_node(placeholder) - - self.populate_template(self.get_loaded_containers_by_id()) - - def get_loaded_containers_by_id(self): - repre_ids = set() - for node in nuke.allNodes(): - if "repre_id" in node.knobs(): - repre_ids.add(node.knob("repre_id").getValue()) - - # Removes duplicates in the list - return list(repre_ids) - - def delete_placeholder(self, placeholder): - placeholder_node = placeholder.data["node"] - last_loaded = placeholder.data["last_loaded"] - if not placeholder.data["delete"]: - if "empty" in placeholder_node.knobs().keys(): - placeholder_node.removeKnob(placeholder_node.knob("empty")) - imprint(placeholder_node, {"empty": True}) - return - - if not last_loaded: - nuke.delete(placeholder_node) - return - - if "last_loaded" in placeholder_node.knobs().keys(): - for node_name in placeholder_node.knob("last_loaded").values(): - node = nuke.toNode(node_name) - try: - delete_placeholder_attributes(node) - except Exception: - pass - - last_loaded_names = [ - loaded_node.name() - for loaded_node in last_loaded - ] - imprint(placeholder_node, {"last_loaded": last_loaded_names}) - - for node in last_loaded: - refresh_node(node) - refresh_node(placeholder_node) - if "builder_type" not in node.knobs().keys(): - attributes = get_placeholder_attributes(placeholder_node, True) - imprint(node, attributes) - imprint(node, {"is_placeholder": False}) - hide_placeholder_attributes(node) - node.knob("is_placeholder").setVisible(False) - imprint( - node, - { - "x": placeholder_node.xpos(), - "y": placeholder_node.ypos() - } - ) - node.knob("x").setVisible(False) - node.knob("y").setVisible(False) - nuke.delete(placeholder_node) + value = knob.getValue() + placeholder_data[key] = value + return placeholder_data -class NukePlaceholder(AbstractPlaceholder): - """Concrete implementation of AbstractPlaceholder for Nuke""" +class NukePlaceholderLoadPlugin(NukePlaceholderPlugin, PlaceholderLoadMixin): + identifier = "nuke.load" + label = "Nuke load" - optional_keys = {"asset", "subset", "hierarchy"} + def _parse_placeholder_node_data(self, node): + placeholder_data = super( + NukePlaceholderLoadPlugin, self + )._parse_placeholder_node_data(node) - def get_data(self, node): - user_data = dict() node_knobs = node.knobs() - for attr in self.required_keys.union(self.optional_keys): - if attr in node_knobs: - user_data[attr] = node_knobs[attr].getValue() - user_data["node"] = node - nb_children = 0 if "nb_children" in node_knobs: nb_children = int(node_knobs["nb_children"].getValue()) - user_data["nb_children"] = nb_children + placeholder_data["nb_children"] = nb_children siblings = [] if "siblings" in node_knobs: siblings = node_knobs["siblings"].values() - user_data["siblings"] = siblings + placeholder_data["siblings"] = siblings node_full_name = node.fullName() - user_data["group_name"] = node_full_name.rpartition(".")[0] - user_data["last_loaded"] = [] - user_data["delete"] = False - self.data = user_data + placeholder_data["group_name"] = node_full_name.rpartition(".")[0] + placeholder_data["last_loaded"] = [] + placeholder_data["delete"] = False + return placeholder_data - def parent_in_hierarchy(self, containers): - return + def _get_loaded_repre_ids(self): + loaded_representation_ids = self.builder.get_shared_populate_data( + "loaded_representation_ids" + ) + if loaded_representation_ids is None: + loaded_representation_ids = set() + for node in nuke.allNodes(): + if "repre_id" in node.knobs(): + loaded_representation_ids.add( + node.knob("repre_id").getValue() + ) - def create_sib_copies(self): - """ creating copies of the palce_holder siblings (the ones who were - loaded with it) for the new nodes added + self.builder.set_shared_populate_data( + "loaded_representation_ids", loaded_representation_ids + ) + return loaded_representation_ids + + def _before_repre_load(self, placeholder, representation): + placeholder.data["nodes_init"] = nuke.allNodes() + placeholder.data["last_repre_id"] = str(representation["_id"]) + + def collect_placeholders(self): + output = [] + scene_placeholders = self._collect_scene_placeholders() + for node_name, node in scene_placeholders.items(): + plugin_identifier_knob = node.knob("plugin_identifier") + if ( + plugin_identifier_knob is None + or plugin_identifier_knob.getValue() != self.identifier + ): + continue + + placeholder_data = self._parse_placeholder_node_data(node) + # TODO do data validations and maybe updgrades if are invalid + output.append( + LoadPlaceholderItem(node_name, placeholder_data, self) + ) + + return output + + def populate_placeholder(self, placeholder): + self.populate_load_placeholder(placeholder) + + def repopulate_placeholder(self, placeholder): + repre_ids = self._get_loaded_repre_ids() + self.populate_load_placeholder(placeholder, repre_ids) + + def get_placeholder_options(self, options=None): + return self.get_load_plugin_options(options) + + def cleanup_placeholder(self, placeholder, failed): + # deselect all selected nodes + placeholder_node = nuke.toNode(placeholder.scene_identifier) + + # getting the latest nodes added + # TODO get from shared populate data! + nodes_init = placeholder.data["nodes_init"] + nodes_loaded = list(set(nuke.allNodes()) - set(nodes_init)) + self.log.debug("Loaded nodes: {}".format(nodes_loaded)) + if not nodes_loaded: + return + + placeholder.data["delete"] = True + + nodes_loaded = self._move_to_placeholder_group( + placeholder, nodes_loaded + ) + placeholder.data["last_loaded"] = nodes_loaded + refresh_nodes(nodes_loaded) + + # positioning of the loaded nodes + min_x, min_y, _, _ = get_extreme_positions(nodes_loaded) + for node in nodes_loaded: + xpos = (node.xpos() - min_x) + placeholder_node.xpos() + ypos = (node.ypos() - min_y) + placeholder_node.ypos() + node.setXYpos(xpos, ypos) + refresh_nodes(nodes_loaded) + + # fix the problem of z_order for backdrops + self._fix_z_order(placeholder) + self._imprint_siblings(placeholder) + + if placeholder.data["nb_children"] == 0: + # save initial nodes postions and dimensions, update them + # and set inputs and outputs of loaded nodes + + self._imprint_inits() + self._update_nodes(placeholder, nuke.allNodes(), nodes_loaded) + self._set_loaded_connections(placeholder) + + elif placeholder.data["siblings"]: + # create copies of placeholder siblings for the new loaded nodes, + # set their inputs and outpus and update all nodes positions and + # dimensions and siblings names + + siblings = get_nodes_by_names(placeholder.data["siblings"]) + refresh_nodes(siblings) + copies = self._create_sib_copies(placeholder) + new_nodes = list(copies.values()) # copies nodes + self._update_nodes(new_nodes, nodes_loaded) + placeholder_node.removeKnob(placeholder_node.knob("siblings")) + new_nodes_name = get_names_from_nodes(new_nodes) + imprint(placeholder_node, {"siblings": new_nodes_name}) + self._set_copies_connections(placeholder, copies) + + self._update_nodes( + nuke.allNodes(), + new_nodes + nodes_loaded, + 20 + ) + + new_siblings = get_names_from_nodes(new_nodes) + placeholder.data["siblings"] = new_siblings + + else: + # if the placeholder doesn't have siblings, the loaded + # nodes will be placed in a free space + + xpointer, ypointer = find_free_space_to_paste_nodes( + nodes_loaded, direction="bottom", offset=200 + ) + node = nuke.createNode("NoOp") + reset_selection() + nuke.delete(node) + for node in nodes_loaded: + xpos = (node.xpos() - min_x) + xpointer + ypos = (node.ypos() - min_y) + ypointer + node.setXYpos(xpos, ypos) + + placeholder.data["nb_children"] += 1 + reset_selection() + # go back to root group + nuke.root().begin() + + def _move_to_placeholder_group(self, placeholder, nodes_loaded): + """ + opening the placeholder's group and copying loaded nodes in it. Returns : - copies (dict) : with copied nodes names and their copies + nodes_loaded (list): the new list of pasted nodes """ - copies = {} - siblings = get_nodes_by_names(self.data["siblings"]) - for node in siblings: - new_node = duplicate_node(node) + groups_name = placeholder.data["group_name"] + reset_selection() + select_nodes(nodes_loaded) + if groups_name: + with node_tempfile() as filepath: + nuke.nodeCopy(filepath) + for node in nuke.selectedNodes(): + nuke.delete(node) + group = nuke.toNode(groups_name) + group.begin() + nuke.nodePaste(filepath) + nodes_loaded = nuke.selectedNodes() + return nodes_loaded - x_init = int(new_node.knob("x_init").getValue()) - y_init = int(new_node.knob("y_init").getValue()) - new_node.setXYpos(x_init, y_init) - if isinstance(new_node, nuke.BackdropNode): - w_init = new_node.knob("w_init").getValue() - h_init = new_node.knob("h_init").getValue() - new_node.knob("bdwidth").setValue(w_init) - new_node.knob("bdheight").setValue(h_init) - refresh_node(node) - - if "repre_id" in node.knobs().keys(): - node.removeKnob(node.knob("repre_id")) - copies[node.name()] = new_node - return copies - - def fix_z_order(self): + def _fix_z_order(self, placeholder): """Fix the problem of z_order when a backdrop is loaded.""" - nodes_loaded = self.data["last_loaded"] + nodes_loaded = placeholder.data["last_loaded"] loaded_backdrops = [] bd_orders = set() for node in nodes_loaded: @@ -287,7 +313,7 @@ class NukePlaceholder(AbstractPlaceholder): return sib_orders = set() - for node_name in self.data["siblings"]: + for node_name in placeholder.data["siblings"]: node = nuke.toNode(node_name) if isinstance(node, nuke.BackdropNode): sib_orders.add(node.knob("z_order").getValue()) @@ -302,7 +328,56 @@ class NukePlaceholder(AbstractPlaceholder): backdrop_node.knob("z_order").setValue( z_order + max_order - min_order + 1) - def update_nodes(self, nodes, considered_nodes, offset_y=None): + def _imprint_siblings(self, placeholder): + """ + - add siblings names to placeholder attributes (nodes loaded with it) + - add Id to the attributes of all the other nodes + """ + + loaded_nodes = placeholder.data["last_loaded"] + loaded_nodes_set = set(loaded_nodes) + data = {"repre_id": str(placeholder.data["last_repre_id"])} + + for node in loaded_nodes: + node_knobs = node.knobs() + if "builder_type" not in node_knobs: + # save the id of representation for all imported nodes + imprint(node, data) + node.knob("repre_id").setVisible(False) + refresh_node(node) + continue + + if ( + "is_placeholder" not in node_knobs + or ( + "is_placeholder" in node_knobs + and node.knob("is_placeholder").value() + ) + ): + siblings = list(loaded_nodes_set - {node}) + siblings_name = get_names_from_nodes(siblings) + siblings = {"siblings": siblings_name} + imprint(node, siblings) + + def _imprint_inits(self): + """Add initial positions and dimensions to the attributes""" + + for node in nuke.allNodes(): + refresh_node(node) + imprint(node, {"x_init": node.xpos(), "y_init": node.ypos()}) + node.knob("x_init").setVisible(False) + node.knob("y_init").setVisible(False) + width = node.screenWidth() + height = node.screenHeight() + if "bdwidth" in node.knobs(): + imprint(node, {"w_init": width, "h_init": height}) + node.knob("w_init").setVisible(False) + node.knob("h_init").setVisible(False) + refresh_node(node) + + def _update_nodes( + self, placeholder, nodes, considered_nodes, offset_y=None + ): """Adjust backdrop nodes dimensions and positions. Considering some nodes sizes. @@ -314,7 +389,7 @@ class NukePlaceholder(AbstractPlaceholder): offset (int): distance between copies """ - placeholder_node = self.data["node"] + placeholder_node = nuke.toNode(placeholder.scene_identifier) min_x, min_y, max_x, max_y = get_extreme_positions(considered_nodes) @@ -330,7 +405,7 @@ class NukePlaceholder(AbstractPlaceholder): min_x = placeholder_node.xpos() min_y = placeholder_node.ypos() else: - siblings = get_nodes_by_names(self.data["siblings"]) + siblings = get_nodes_by_names(placeholder.data["siblings"]) minX, _, maxX, _ = get_extreme_positions(siblings) diff_y = max_y - min_y + 20 diff_x = abs(max_x - min_x - maxX + minX) @@ -369,59 +444,14 @@ class NukePlaceholder(AbstractPlaceholder): refresh_node(node) - def imprint_inits(self): - """Add initial positions and dimensions to the attributes""" - - for node in nuke.allNodes(): - refresh_node(node) - imprint(node, {"x_init": node.xpos(), "y_init": node.ypos()}) - node.knob("x_init").setVisible(False) - node.knob("y_init").setVisible(False) - width = node.screenWidth() - height = node.screenHeight() - if "bdwidth" in node.knobs(): - imprint(node, {"w_init": width, "h_init": height}) - node.knob("w_init").setVisible(False) - node.knob("h_init").setVisible(False) - refresh_node(node) - - def imprint_siblings(self): - """ - - add siblings names to placeholder attributes (nodes loaded with it) - - add Id to the attributes of all the other nodes - """ - - loaded_nodes = self.data["last_loaded"] - loaded_nodes_set = set(loaded_nodes) - data = {"repre_id": str(self.data["last_repre_id"])} - - for node in loaded_nodes: - node_knobs = node.knobs() - if "builder_type" not in node_knobs: - # save the id of representation for all imported nodes - imprint(node, data) - node.knob("repre_id").setVisible(False) - refresh_node(node) - continue - - if ( - "is_placeholder" not in node_knobs - or ( - "is_placeholder" in node_knobs - and node.knob("is_placeholder").value() - ) - ): - siblings = list(loaded_nodes_set - {node}) - siblings_name = get_names_from_nodes(siblings) - siblings = {"siblings": siblings_name} - imprint(node, siblings) - - def set_loaded_connections(self): + def _set_loaded_connections(self, placeholder): """ set inputs and outputs of loaded nodes""" - placeholder_node = self.data["node"] - input_node, output_node = get_group_io_nodes(self.data["last_loaded"]) + placeholder_node = nuke.toNode(placeholder.scene_identifier) + input_node, output_node = get_group_io_nodes( + placeholder.data["last_loaded"] + ) for node in placeholder_node.dependent(): for idx in range(node.inputs()): if node.input(idx) == placeholder_node: @@ -432,15 +462,45 @@ class NukePlaceholder(AbstractPlaceholder): if placeholder_node.input(idx) == node: input_node.setInput(0, node) - def set_copies_connections(self, copies): + def _create_sib_copies(self, placeholder): + """ creating copies of the palce_holder siblings (the ones who were + loaded with it) for the new nodes added + + Returns : + copies (dict) : with copied nodes names and their copies + """ + + copies = {} + siblings = get_nodes_by_names(placeholder.data["siblings"]) + for node in siblings: + new_node = duplicate_node(node) + + x_init = int(new_node.knob("x_init").getValue()) + y_init = int(new_node.knob("y_init").getValue()) + new_node.setXYpos(x_init, y_init) + if isinstance(new_node, nuke.BackdropNode): + w_init = new_node.knob("w_init").getValue() + h_init = new_node.knob("h_init").getValue() + new_node.knob("bdwidth").setValue(w_init) + new_node.knob("bdheight").setValue(h_init) + refresh_node(node) + + if "repre_id" in node.knobs().keys(): + node.removeKnob(node.knob("repre_id")) + copies[node.name()] = new_node + return copies + + def _set_copies_connections(self, placeholder, copies): """Set inputs and outputs of the copies. Args: copies (dict): Copied nodes by their names. """ - last_input, last_output = get_group_io_nodes(self.data["last_loaded"]) - siblings = get_nodes_by_names(self.data["siblings"]) + last_input, last_output = get_group_io_nodes( + placeholder.data["last_loaded"] + ) + siblings = get_nodes_by_names(placeholder.data["siblings"]) siblings_input, siblings_output = get_group_io_nodes(siblings) copy_input = copies[siblings_input.name()] copy_output = copies[siblings_output.name()] @@ -474,166 +534,45 @@ class NukePlaceholder(AbstractPlaceholder): siblings_input.setInput(0, copy_output) - def move_to_placeholder_group(self, nodes_loaded): - """ - opening the placeholder's group and copying loaded nodes in it. - Returns : - nodes_loaded (list): the new list of pasted nodes - """ +def build_workfile_template(*args): + builder = NukeTemplateBuilder(registered_host()) + builder.build_template() - groups_name = self.data["group_name"] - reset_selection() - select_nodes(nodes_loaded) - if groups_name: - with node_tempfile() as filepath: - nuke.nodeCopy(filepath) - for node in nuke.selectedNodes(): - nuke.delete(node) - group = nuke.toNode(groups_name) - group.begin() - nuke.nodePaste(filepath) - nodes_loaded = nuke.selectedNodes() - return nodes_loaded - def clean(self): - # deselect all selected nodes - placeholder_node = self.data["node"] +def update_workfile_template(*args): + builder = NukeTemplateBuilder(registered_host()) + builder.rebuild_template() - # getting the latest nodes added - nodes_init = self.data["nodes_init"] - nodes_loaded = list(set(nuke.allNodes()) - set(nodes_init)) - self.log.debug("Loaded nodes: {}".format(nodes_loaded)) - if not nodes_loaded: - return - self.data["delete"] = True +def create_placeholder(*args): + host = registered_host() + builder = NukeTemplateBuilder(host) + window = WorkfileBuildPlaceholderDialog(host, builder) + window.exec_() - nodes_loaded = self.move_to_placeholder_group(nodes_loaded) - self.data["last_loaded"] = nodes_loaded - refresh_nodes(nodes_loaded) - # positioning of the loaded nodes - min_x, min_y, _, _ = get_extreme_positions(nodes_loaded) - for node in nodes_loaded: - xpos = (node.xpos() - min_x) + placeholder_node.xpos() - ypos = (node.ypos() - min_y) + placeholder_node.ypos() - node.setXYpos(xpos, ypos) - refresh_nodes(nodes_loaded) +def update_placeholder(*args): + host = registered_host() + builder = NukeTemplateBuilder(host) + placeholder_items_by_id = { + placeholder_item.scene_identifier: placeholder_item + for placeholder_item in builder.get_placeholders() + } + placeholder_items = [] + for node in nuke.selectedNodes(): + node_name = node.fullName() + if node_name in placeholder_items_by_id: + placeholder_items.append(placeholder_items_by_id[node_name]) - self.fix_z_order() # fix the problem of z_order for backdrops - self.imprint_siblings() + # TODO show UI at least + if len(placeholder_items) == 0: + raise ValueError("No node selected") - if self.data["nb_children"] == 0: - # save initial nodes postions and dimensions, update them - # and set inputs and outputs of loaded nodes + if len(placeholder_items) > 1: + raise ValueError("Too many selected nodes") - self.imprint_inits() - self.update_nodes(nuke.allNodes(), nodes_loaded) - self.set_loaded_connections() - - elif self.data["siblings"]: - # create copies of placeholder siblings for the new loaded nodes, - # set their inputs and outpus and update all nodes positions and - # dimensions and siblings names - - siblings = get_nodes_by_names(self.data["siblings"]) - refresh_nodes(siblings) - copies = self.create_sib_copies() - new_nodes = list(copies.values()) # copies nodes - self.update_nodes(new_nodes, nodes_loaded) - placeholder_node.removeKnob(placeholder_node.knob("siblings")) - new_nodes_name = get_names_from_nodes(new_nodes) - imprint(placeholder_node, {"siblings": new_nodes_name}) - self.set_copies_connections(copies) - - self.update_nodes( - nuke.allNodes(), - new_nodes + nodes_loaded, - 20 - ) - - new_siblings = get_names_from_nodes(new_nodes) - self.data["siblings"] = new_siblings - - else: - # if the placeholder doesn't have siblings, the loaded - # nodes will be placed in a free space - - xpointer, ypointer = find_free_space_to_paste_nodes( - nodes_loaded, direction="bottom", offset=200 - ) - node = nuke.createNode("NoOp") - reset_selection() - nuke.delete(node) - for node in nodes_loaded: - xpos = (node.xpos() - min_x) + xpointer - ypos = (node.ypos() - min_y) + ypointer - node.setXYpos(xpos, ypos) - - self.data["nb_children"] += 1 - reset_selection() - # go back to root group - nuke.root().begin() - - def get_representations(self, current_asset_doc, linked_asset_docs): - project_name = legacy_io.active_project() - - builder_type = self.data["builder_type"] - if builder_type == "context_asset": - context_filters = { - "asset": [re.compile(self.data["asset"])], - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representations": [self.data["representation"]], - "family": [self.data["family"]] - } - - elif builder_type != "linked_asset": - context_filters = { - "asset": [ - current_asset_doc["name"], - re.compile(self.data["asset"]) - ], - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representation": [self.data["representation"]], - "family": [self.data["family"]] - } - - else: - asset_regex = re.compile(self.data["asset"]) - linked_asset_names = [] - for asset_doc in linked_asset_docs: - asset_name = asset_doc["name"] - if asset_regex.match(asset_name): - linked_asset_names.append(asset_name) - - if not linked_asset_names: - return [] - - context_filters = { - "asset": linked_asset_names, - "subset": [re.compile(self.data["subset"])], - "hierarchy": [re.compile(self.data["hierarchy"])], - "representation": [self.data["representation"]], - "family": [self.data["family"]], - } - - return list(get_representations( - project_name, - context_filters=context_filters - )) - - def err_message(self): - return ( - "Error while trying to load a representation.\n" - "Either the subset wasn't published or the template is malformed." - "\n\n" - "Builder was looking for:\n{attributes}".format( - attributes="\n".join([ - "{}: {}".format(key.title(), value) - for key, value in self.data.items()] - ) - ) - ) + placeholder_item = placeholder_items[0] + window = WorkfileBuildPlaceholderDialog(host, builder) + window.set_update_mode(placeholder_item) + window.exec_() diff --git a/openpype/hosts/nuke/plugins/publish/extract_backdrop.py b/openpype/hosts/nuke/plugins/publish/extract_backdrop.py index 0a2df0898e..d1e5c4cc5a 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_backdrop.py +++ b/openpype/hosts/nuke/plugins/publish/extract_backdrop.py @@ -4,7 +4,7 @@ import nuke import pyblish.api -import openpype +from openpype.pipeline import publish from openpype.hosts.nuke.api.lib import ( maintained_selection, reset_selection, @@ -12,7 +12,7 @@ from openpype.hosts.nuke.api.lib import ( ) -class ExtractBackdropNode(openpype.api.Extractor): +class ExtractBackdropNode(publish.Extractor): """Extracting content of backdrop nodes Will create nuke script only with containing nodes. diff --git a/openpype/hosts/nuke/plugins/publish/extract_camera.py b/openpype/hosts/nuke/plugins/publish/extract_camera.py index 54f65a0be3..b751bfab03 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_camera.py +++ b/openpype/hosts/nuke/plugins/publish/extract_camera.py @@ -5,11 +5,12 @@ from pprint import pformat import nuke import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.nuke.api.lib import maintained_selection -class ExtractCamera(openpype.api.Extractor): +class ExtractCamera(publish.Extractor): """ 3D camera exctractor """ label = 'Exctract Camera' diff --git a/openpype/hosts/nuke/plugins/publish/extract_gizmo.py b/openpype/hosts/nuke/plugins/publish/extract_gizmo.py index 2d5bfdeb5e..3047ad6724 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_gizmo.py +++ b/openpype/hosts/nuke/plugins/publish/extract_gizmo.py @@ -3,7 +3,7 @@ import nuke import pyblish.api -import openpype +from openpype.pipeline import publish from openpype.hosts.nuke.api import utils as pnutils from openpype.hosts.nuke.api.lib import ( maintained_selection, @@ -12,7 +12,7 @@ from openpype.hosts.nuke.api.lib import ( ) -class ExtractGizmo(openpype.api.Extractor): +class ExtractGizmo(publish.Extractor): """Extracting Gizmo (Group) node Will create nuke script only with the Gizmo node. diff --git a/openpype/hosts/nuke/plugins/publish/extract_model.py b/openpype/hosts/nuke/plugins/publish/extract_model.py index 0375263338..d82cb3110b 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_model.py +++ b/openpype/hosts/nuke/plugins/publish/extract_model.py @@ -2,14 +2,15 @@ import os from pprint import pformat import nuke import pyblish.api -import openpype.api + +from openpype.pipeline import publish from openpype.hosts.nuke.api.lib import ( maintained_selection, select_nodes ) -class ExtractModel(openpype.api.Extractor): +class ExtractModel(publish.Extractor): """ 3D model exctractor """ label = 'Exctract Model' diff --git a/openpype/hosts/nuke/plugins/publish/extract_render_local.py b/openpype/hosts/nuke/plugins/publish/extract_render_local.py index 8879f0c999..843d588786 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_render_local.py +++ b/openpype/hosts/nuke/plugins/publish/extract_render_local.py @@ -1,11 +1,13 @@ -import pyblish.api -import nuke import os -import openpype + +import pyblish.api import clique +import nuke + +from openpype.pipeline import publish -class NukeRenderLocal(openpype.api.Extractor): +class NukeRenderLocal(publish.Extractor): # TODO: rewrite docstring to nuke """Render the current Nuke composition locally. diff --git a/openpype/hosts/nuke/plugins/publish/extract_review_data.py b/openpype/hosts/nuke/plugins/publish/extract_review_data.py index 38a8140cff..3c85b21b08 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_review_data.py +++ b/openpype/hosts/nuke/plugins/publish/extract_review_data.py @@ -1,10 +1,11 @@ import os -import pyblish.api -import openpype from pprint import pformat +import pyblish.api + +from openpype.pipeline import publish -class ExtractReviewData(openpype.api.Extractor): +class ExtractReviewData(publish.Extractor): """Extracts review tag into available representation """ diff --git a/openpype/hosts/nuke/plugins/publish/extract_review_data_lut.py b/openpype/hosts/nuke/plugins/publish/extract_review_data_lut.py index 4cf2fd7d9f..67779e9599 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_review_data_lut.py +++ b/openpype/hosts/nuke/plugins/publish/extract_review_data_lut.py @@ -1,11 +1,12 @@ import os import pyblish.api -import openpype + +from openpype.pipeline import publish from openpype.hosts.nuke.api import plugin from openpype.hosts.nuke.api.lib import maintained_selection -class ExtractReviewDataLut(openpype.api.Extractor): +class ExtractReviewDataLut(publish.Extractor): """Extracts movie and thumbnail with baked in luts must be run after extract_render_local.py diff --git a/openpype/hosts/nuke/plugins/publish/extract_review_data_mov.py b/openpype/hosts/nuke/plugins/publish/extract_review_data_mov.py index fc16e189fb..3fcfc2a4b5 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_review_data_mov.py +++ b/openpype/hosts/nuke/plugins/publish/extract_review_data_mov.py @@ -1,13 +1,14 @@ import os -from pprint import pformat import re +from pprint import pformat import pyblish.api -import openpype + +from openpype.pipeline import publish from openpype.hosts.nuke.api import plugin from openpype.hosts.nuke.api.lib import maintained_selection -class ExtractReviewDataMov(openpype.api.Extractor): +class ExtractReviewDataMov(publish.Extractor): """Extracts movie and thumbnail with baked in luts must be run after extract_render_local.py diff --git a/openpype/hosts/nuke/plugins/publish/extract_slate_frame.py b/openpype/hosts/nuke/plugins/publish/extract_slate_frame.py index b5cad143db..e7197b4fa8 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_slate_frame.py +++ b/openpype/hosts/nuke/plugins/publish/extract_slate_frame.py @@ -6,7 +6,7 @@ import copy import pyblish.api import six -import openpype +from openpype.pipeline import publish from openpype.hosts.nuke.api import ( maintained_selection, duplicate_node, @@ -14,7 +14,7 @@ from openpype.hosts.nuke.api import ( ) -class ExtractSlateFrame(openpype.api.Extractor): +class ExtractSlateFrame(publish.Extractor): """Extracts movie and thumbnail with baked in luts must be run after extract_render_local.py diff --git a/openpype/hosts/nuke/plugins/publish/extract_thumbnail.py b/openpype/hosts/nuke/plugins/publish/extract_thumbnail.py index 2a919051d2..19eae9638b 100644 --- a/openpype/hosts/nuke/plugins/publish/extract_thumbnail.py +++ b/openpype/hosts/nuke/plugins/publish/extract_thumbnail.py @@ -2,7 +2,8 @@ import sys import os import nuke import pyblish.api -import openpype + +from openpype.pipeline import publish from openpype.hosts.nuke.api import ( maintained_selection, get_view_process_node @@ -13,7 +14,7 @@ if sys.version_info[0] >= 3: unicode = str -class ExtractThumbnail(openpype.api.Extractor): +class ExtractThumbnail(publish.Extractor): """Extracts movie and thumbnail with baked in luts must be run after extract_render_local.py diff --git a/openpype/hosts/nuke/plugins/publish/precollect_writes.py b/openpype/hosts/nuke/plugins/publish/precollect_writes.py index e37cc8a80a..17c4bc30cf 100644 --- a/openpype/hosts/nuke/plugins/publish/precollect_writes.py +++ b/openpype/hosts/nuke/plugins/publish/precollect_writes.py @@ -201,34 +201,6 @@ class CollectNukeWrites(pyblish.api.InstancePlugin): if not instance.data["review"]: instance.data["useSequenceForReview"] = False - project_name = legacy_io.active_project() - asset_name = instance.data["asset"] - # * Add audio to instance if exists. - # Find latest versions document - last_version_doc = get_last_version_by_subset_name( - project_name, "audioMain", asset_name=asset_name, fields=["_id"] - ) - - repre_doc = None - if last_version_doc: - # Try to find it's representation (Expected there is only one) - repre_docs = list(get_representations( - project_name, version_ids=[last_version_doc["_id"]] - )) - if not repre_docs: - self.log.warning( - "Version document does not contain any representations" - ) - else: - repre_doc = repre_docs[0] - - # Add audio to instance if representation was found - if repre_doc: - instance.data["audio"] = [{ - "offset": 0, - "filename": get_representation_path(repre_doc) - }] - self.log.debug("instance.data: {}".format(pformat(instance.data))) def is_prerender(self, families): diff --git a/openpype/hosts/nuke/plugins/publish/validate_write_nodes.py b/openpype/hosts/nuke/plugins/publish/validate_write_nodes.py index 5a8bc2022e..26a563b13b 100644 --- a/openpype/hosts/nuke/plugins/publish/validate_write_nodes.py +++ b/openpype/hosts/nuke/plugins/publish/validate_write_nodes.py @@ -2,7 +2,8 @@ import pyblish.api from openpype.pipeline.publish import get_errored_instances_from_context from openpype.hosts.nuke.api.lib import ( get_write_node_template_attr, - set_node_knobs_from_settings + set_node_knobs_from_settings, + color_gui_to_int ) from openpype.pipeline import PublishXmlValidationError @@ -76,8 +77,11 @@ class ValidateNukeWriteNode(pyblish.api.InstancePlugin): # fix type differences if type(node_value) in (int, float): - value = float(value) - node_value = float(node_value) + if isinstance(value, list): + value = color_gui_to_int(value) + else: + value = float(value) + node_value = float(node_value) else: value = str(value) node_value = str(node_value) diff --git a/openpype/hosts/photoshop/api/README.md b/openpype/hosts/photoshop/api/README.md index 80792a4da0..4a36746cb2 100644 --- a/openpype/hosts/photoshop/api/README.md +++ b/openpype/hosts/photoshop/api/README.md @@ -127,11 +127,11 @@ class CollectInstances(pyblish.api.ContextPlugin): ```python import os -import openpype.api -from avalon import photoshop +from openpype.pipeline import publish +from openpype.hosts.photoshop import api as photoshop -class ExtractImage(openpype.api.Extractor): +class ExtractImage(publish.Extractor): """Produce a flattened image file from instance This plug-in takes into account only the layers in the group. diff --git a/openpype/hosts/photoshop/api/lib.py b/openpype/hosts/photoshop/api/lib.py index 73a546604f..221b4314e6 100644 --- a/openpype/hosts/photoshop/api/lib.py +++ b/openpype/hosts/photoshop/api/lib.py @@ -64,10 +64,15 @@ def maintained_selection(): @contextlib.contextmanager -def maintained_visibility(): - """Maintain visibility during context.""" +def maintained_visibility(layers=None): + """Maintain visibility during context. + + Args: + layers (list) of PSItem (used for caching) + """ visibility = {} - layers = stub().get_layers() + if not layers: + layers = stub().get_layers() for layer in layers: visibility[layer.id] = layer.visible try: diff --git a/openpype/hosts/photoshop/api/ws_stub.py b/openpype/hosts/photoshop/api/ws_stub.py index b49bf1c73f..2c4d0ad5fc 100644 --- a/openpype/hosts/photoshop/api/ws_stub.py +++ b/openpype/hosts/photoshop/api/ws_stub.py @@ -229,10 +229,11 @@ class PhotoshopServerStub: return self._get_layers_in_layers(parent_ids) - def get_layers_in_layers_ids(self, layers_ids): + def get_layers_in_layers_ids(self, layers_ids, layers=None): """Return all layers that belong to layers (might be groups). Args: + layers_ids layers : Returns: @@ -240,10 +241,13 @@ class PhotoshopServerStub: """ parent_ids = set(layers_ids) - return self._get_layers_in_layers(parent_ids) + return self._get_layers_in_layers(parent_ids, layers) - def _get_layers_in_layers(self, parent_ids): - all_layers = self.get_layers() + def _get_layers_in_layers(self, parent_ids, layers=None): + if not layers: + layers = self.get_layers() + + all_layers = layers ret = [] for layer in all_layers: @@ -394,14 +398,17 @@ class PhotoshopServerStub: self.hide_all_others_layers_ids(extract_ids) - def hide_all_others_layers_ids(self, extract_ids): + def hide_all_others_layers_ids(self, extract_ids, layers=None): """hides all layers that are not part of the list or that are not children of this list Args: extract_ids (list): list of integer that should be visible + layers (list) of PSItem (used for caching) """ - for layer in self.get_layers(): + if not layers: + layers = self.get_layers() + for layer in layers: if layer.visible and layer.id not in extract_ids: self.set_visible(layer.id, False) diff --git a/openpype/hosts/photoshop/plugins/publish/collect_published_version.py b/openpype/hosts/photoshop/plugins/publish/collect_published_version.py new file mode 100644 index 0000000000..2502689e4b --- /dev/null +++ b/openpype/hosts/photoshop/plugins/publish/collect_published_version.py @@ -0,0 +1,55 @@ +"""Collects published version of workfile and increments it. + +For synchronization of published image and workfile version it is required +to store workfile version from workfile file name in context.data["version"]. +In remote publishing this name is unreliable (artist might not follow naming +convention etc.), last published workfile version for particular workfile +subset is used instead. + +This plugin runs only in remote publishing (eg. Webpublisher). + +Requires: + context.data["assetEntity"] + +Provides: + context["version"] - incremented latest published workfile version +""" + +import pyblish.api + +from openpype.client import get_last_version_by_subset_name + + +class CollectPublishedVersion(pyblish.api.ContextPlugin): + """Collects published version of workfile and increments it.""" + + order = pyblish.api.CollectorOrder + 0.190 + label = "Collect published version" + hosts = ["photoshop"] + targets = ["remotepublish"] + + def process(self, context): + workfile_subset_name = None + for instance in context: + if instance.data["family"] == "workfile": + workfile_subset_name = instance.data["subset"] + break + + if not workfile_subset_name: + self.log.warning("No workfile instance found, " + "synchronization of version will not work.") + return + + project_name = context.data["projectName"] + asset_doc = context.data["assetEntity"] + asset_id = asset_doc["_id"] + + version_doc = get_last_version_by_subset_name(project_name, + workfile_subset_name, + asset_id) + version_int = 1 + if version_doc: + version_int += int(version_doc["name"]) + + self.log.debug(f"Setting {version_int} to context.") + context.data["version"] = version_int diff --git a/openpype/hosts/photoshop/plugins/publish/collect_review.py b/openpype/hosts/photoshop/plugins/publish/collect_review.py index 7f395b46d7..7e598a8250 100644 --- a/openpype/hosts/photoshop/plugins/publish/collect_review.py +++ b/openpype/hosts/photoshop/plugins/publish/collect_review.py @@ -25,6 +25,8 @@ class CollectReview(pyblish.api.ContextPlugin): hosts = ["photoshop"] order = pyblish.api.CollectorOrder + 0.1 + publish = True + def process(self, context): family = "review" subset = get_subset_name( @@ -45,5 +47,6 @@ class CollectReview(pyblish.api.ContextPlugin): "family": family, "families": [], "representations": [], - "asset": os.environ["AVALON_ASSET"] + "asset": os.environ["AVALON_ASSET"], + "publish": self.publish }) diff --git a/openpype/hosts/photoshop/plugins/publish/collect_version.py b/openpype/hosts/photoshop/plugins/publish/collect_version.py new file mode 100644 index 0000000000..aff9f13bfb --- /dev/null +++ b/openpype/hosts/photoshop/plugins/publish/collect_version.py @@ -0,0 +1,24 @@ +import pyblish.api + + +class CollectVersion(pyblish.api.InstancePlugin): + """Collect version for publishable instances. + + Used to synchronize version from workfile to all publishable instances: + - image (manually created or color coded) + - review + + Dev comment: + Explicit collector created to control this from single place and not from + 3 different. + """ + order = pyblish.api.CollectorOrder + 0.200 + label = 'Collect Version' + + hosts = ["photoshop"] + families = ["image", "review"] + + def process(self, instance): + workfile_version = instance.context.data["version"] + self.log.debug(f"Applying version {workfile_version}") + instance.data["version"] = workfile_version diff --git a/openpype/hosts/photoshop/plugins/publish/extract_image.py b/openpype/hosts/photoshop/plugins/publish/extract_image.py index a133e33409..cdb28c742d 100644 --- a/openpype/hosts/photoshop/plugins/publish/extract_image.py +++ b/openpype/hosts/photoshop/plugins/publish/extract_image.py @@ -1,61 +1,99 @@ import os -import openpype.api +import pyblish.api +from openpype.pipeline import publish from openpype.hosts.photoshop import api as photoshop -class ExtractImage(openpype.api.Extractor): - """Produce a flattened image file from instance +class ExtractImage(pyblish.api.ContextPlugin): + """Extract all layers (groups) marked for publish. - This plug-in takes into account only the layers in the group. + Usually publishable instance is created as a wrapper of layer(s). For each + publishable instance so many images as there is 'formats' is created. + + Logic tries to hide/unhide layers minimum times. + + Called once for all publishable instances. """ + order = publish.Extractor.order - 0.48 label = "Extract Image" hosts = ["photoshop"] + families = ["image", "background"] formats = ["png", "jpg"] - def process(self, instance): - staging_dir = self.staging_dir(instance) - self.log.info("Outputting image to {}".format(staging_dir)) - - # Perform extraction + def process(self, context): stub = photoshop.stub() - files = {} + hidden_layer_ids = set() + + all_layers = stub.get_layers() + for layer in all_layers: + if not layer.visible: + hidden_layer_ids.add(layer.id) + stub.hide_all_others_layers_ids([], layers=all_layers) + with photoshop.maintained_selection(): - self.log.info("Extracting %s" % str(list(instance))) - with photoshop.maintained_visibility(): - ids = set() - layer = instance.data.get("layer") - if layer: - ids.add(layer.id) - add_ids = instance.data.pop("ids", None) - if add_ids: - ids.update(set(add_ids)) - extract_ids = set([ll.id for ll in stub. - get_layers_in_layers_ids(ids)]) - stub.hide_all_others_layers_ids(extract_ids) + with photoshop.maintained_visibility(layers=all_layers): + for instance in context: + if instance.data["family"] not in self.families: + continue - file_basename = os.path.splitext( - stub.get_active_document_name() - )[0] - for extension in self.formats: - _filename = "{}.{}".format(file_basename, extension) - files[extension] = _filename + staging_dir = self.staging_dir(instance) + self.log.info("Outputting image to {}".format(staging_dir)) - full_filename = os.path.join(staging_dir, _filename) - stub.saveAs(full_filename, extension, True) - self.log.info(f"Extracted: {extension}") + # Perform extraction + files = {} + ids = set() + layer = instance.data.get("layer") + if layer: + ids.add(layer.id) + add_ids = instance.data.pop("ids", None) + if add_ids: + ids.update(set(add_ids)) + extract_ids = set([ll.id for ll in stub. + get_layers_in_layers_ids(ids, all_layers) + if ll.id not in hidden_layer_ids]) - representations = [] - for extension, filename in files.items(): - representations.append({ - "name": extension, - "ext": extension, - "files": filename, - "stagingDir": staging_dir - }) - instance.data["representations"] = representations - instance.data["stagingDir"] = staging_dir + for extracted_id in extract_ids: + stub.set_visible(extracted_id, True) - self.log.info(f"Extracted {instance} to {staging_dir}") + file_basename = os.path.splitext( + stub.get_active_document_name() + )[0] + for extension in self.formats: + _filename = "{}.{}".format(file_basename, + extension) + files[extension] = _filename + + full_filename = os.path.join(staging_dir, + _filename) + stub.saveAs(full_filename, extension, True) + self.log.info(f"Extracted: {extension}") + + representations = [] + for extension, filename in files.items(): + representations.append({ + "name": extension, + "ext": extension, + "files": filename, + "stagingDir": staging_dir + }) + instance.data["representations"] = representations + instance.data["stagingDir"] = staging_dir + + self.log.info(f"Extracted {instance} to {staging_dir}") + + for extracted_id in extract_ids: + stub.set_visible(extracted_id, False) + + def staging_dir(self, instance): + """Provide a temporary directory in which to store extracted files + + Upon calling this method the staging directory is stored inside + the instance.data['stagingDir'] + """ + + from openpype.pipeline.publish import get_instance_staging_dir + + return get_instance_staging_dir(instance) diff --git a/openpype/hosts/photoshop/plugins/publish/extract_review.py b/openpype/hosts/photoshop/plugins/publish/extract_review.py index 5d37c86ed8..e5fee311f8 100644 --- a/openpype/hosts/photoshop/plugins/publish/extract_review.py +++ b/openpype/hosts/photoshop/plugins/publish/extract_review.py @@ -2,12 +2,15 @@ import os import shutil from PIL import Image -import openpype.api -import openpype.lib +from openpype.lib import ( + run_subprocess, + get_ffmpeg_tool_path, +) +from openpype.pipeline import publish from openpype.hosts.photoshop import api as photoshop -class ExtractReview(openpype.api.Extractor): +class ExtractReview(publish.Extractor): """ Produce a flattened or sequence image files from all 'image' instances. @@ -72,7 +75,7 @@ class ExtractReview(openpype.api.Extractor): }) processed_img_names = [img_list] - ffmpeg_path = openpype.lib.get_ffmpeg_tool_path("ffmpeg") + ffmpeg_path = get_ffmpeg_tool_path("ffmpeg") instance.data["stagingDir"] = staging_dir @@ -93,7 +96,7 @@ class ExtractReview(openpype.api.Extractor): thumbnail_path ] self.log.debug("thumbnail args:: {}".format(args)) - output = openpype.lib.run_subprocess(args) + output = run_subprocess(args) instance.data["representations"].append({ "name": "thumbnail", @@ -116,7 +119,7 @@ class ExtractReview(openpype.api.Extractor): mov_path ] self.log.debug("mov args:: {}".format(args)) - output = openpype.lib.run_subprocess(args) + output = run_subprocess(args) self.log.debug(output) instance.data["representations"].append({ "name": "mov", diff --git a/openpype/hosts/photoshop/plugins/publish/extract_save_scene.py b/openpype/hosts/photoshop/plugins/publish/extract_save_scene.py index 03086f389f..aa900fec9f 100644 --- a/openpype/hosts/photoshop/plugins/publish/extract_save_scene.py +++ b/openpype/hosts/photoshop/plugins/publish/extract_save_scene.py @@ -1,11 +1,11 @@ -import openpype.api +from openpype.pipeline import publish from openpype.hosts.photoshop import api as photoshop -class ExtractSaveScene(openpype.api.Extractor): +class ExtractSaveScene(publish.Extractor): """Save scene before extraction.""" - order = openpype.api.Extractor.order - 0.49 + order = publish.Extractor.order - 0.49 label = "Extract Save Scene" hosts = ["photoshop"] families = ["workfile"] diff --git a/openpype/hosts/resolve/utils.py b/openpype/hosts/resolve/utils.py index 382a7cf344..d5c133bbf5 100644 --- a/openpype/hosts/resolve/utils.py +++ b/openpype/hosts/resolve/utils.py @@ -17,7 +17,7 @@ def setup(env): # collect script dirs if us_env: - log.info(f"Utility Scripts Env: `{us_env}`") + log.info("Utility Scripts Env: `{}`".format(us_env)) us_paths = us_env.split( os.pathsep) + us_paths @@ -25,13 +25,13 @@ def setup(env): for path in us_paths: scripts.update({path: os.listdir(path)}) - log.info(f"Utility Scripts Dir: `{us_paths}`") - log.info(f"Utility Scripts: `{scripts}`") + log.info("Utility Scripts Dir: `{}`".format(us_paths)) + log.info("Utility Scripts: `{}`".format(scripts)) # make sure no script file is in folder for s in os.listdir(us_dir): path = os.path.join(us_dir, s) - log.info(f"Removing `{path}`...") + log.info("Removing `{}`...".format(path)) if os.path.isdir(path): shutil.rmtree(path, onerror=None) else: @@ -44,7 +44,7 @@ def setup(env): # script in script list src = os.path.join(d, s) dst = os.path.join(us_dir, s) - log.info(f"Copying `{src}` to `{dst}`...") + log.info("Copying `{}` to `{}`...".format(src, dst)) if os.path.isdir(src): shutil.copytree( src, dst, symlinks=False, diff --git a/openpype/hosts/traypublisher/api/pipeline.py b/openpype/hosts/traypublisher/api/pipeline.py index 2d9db7801e..0a8ddaa343 100644 --- a/openpype/hosts/traypublisher/api/pipeline.py +++ b/openpype/hosts/traypublisher/api/pipeline.py @@ -9,7 +9,7 @@ from openpype.pipeline import ( register_creator_plugin_path, legacy_io, ) -from openpype.host import HostBase, INewPublisher +from openpype.host import HostBase, IPublishHost ROOT_DIR = os.path.dirname(os.path.dirname( @@ -19,7 +19,7 @@ PUBLISH_PATH = os.path.join(ROOT_DIR, "plugins", "publish") CREATE_PATH = os.path.join(ROOT_DIR, "plugins", "create") -class TrayPublisherHost(HostBase, INewPublisher): +class TrayPublisherHost(HostBase, IPublishHost): name = "traypublisher" def install(self): diff --git a/openpype/hosts/tvpaint/lib.py b/openpype/hosts/tvpaint/lib.py index c67ab1e4fb..95653b6ecb 100644 --- a/openpype/hosts/tvpaint/lib.py +++ b/openpype/hosts/tvpaint/lib.py @@ -646,9 +646,6 @@ def rename_filepaths_by_frame_start( filepaths_by_frame, range_start, range_end, new_frame_start ): """Change frames in filenames of finished images to new frame start.""" - # Skip if source first frame is same as destination first frame - if range_start == new_frame_start: - return # Calculate frame end new_frame_end = range_end + (new_frame_start - range_start) @@ -669,14 +666,17 @@ def rename_filepaths_by_frame_start( source_range = range(range_start, range_end + 1) output_range = range(new_frame_start, new_frame_end + 1) + # Skip if source first frame is same as destination first frame new_dst_filepaths = {} for src_frame, dst_frame in zip(source_range, output_range): - src_filepath = filepaths_by_frame[src_frame] - src_dirpath = os.path.dirname(src_filepath) + src_filepath = os.path.normpath(filepaths_by_frame[src_frame]) + dirpath, src_filename = os.path.split(src_filepath) dst_filename = filename_template.format(frame=dst_frame) - dst_filepath = os.path.join(src_dirpath, dst_filename) + dst_filepath = os.path.join(dirpath, dst_filename) - os.rename(src_filepath, dst_filepath) + if src_filename != dst_filename: + os.rename(src_filepath, dst_filepath) new_dst_filepaths[dst_frame] = dst_filepath + return new_dst_filepaths diff --git a/openpype/lib/attribute_definitions.py b/openpype/lib/attribute_definitions.py index 17658eef93..37446f01f8 100644 --- a/openpype/lib/attribute_definitions.py +++ b/openpype/lib/attribute_definitions.py @@ -9,6 +9,49 @@ import six import clique +def get_attributes_keys(attribute_definitions): + """Collect keys from list of attribute definitions. + + Args: + attribute_definitions (List[AbtractAttrDef]): Objects of attribute + definitions. + + Returns: + Set[str]: Keys that will be created using passed attribute definitions. + """ + + keys = set() + if not attribute_definitions: + return keys + + for attribute_def in attribute_definitions: + if not isinstance(attribute_def, UIDef): + keys.add(attribute_def.key) + return keys + + +def get_default_values(attribute_definitions): + """Receive default values for attribute definitions. + + Args: + attribute_definitions (List[AbtractAttrDef]): Attribute definitions for + which default values should be collected. + + Returns: + Dict[str, Any]: Default values for passet attribute definitions. + """ + + output = {} + if not attribute_definitions: + return output + + for attr_def in attribute_definitions: + # Skip UI definitions + if not isinstance(attr_def, UIDef): + output[attr_def.key] = attr_def.default + return output + + class AbstractAttrDefMeta(ABCMeta): """Meta class to validate existence of 'key' attribute. diff --git a/openpype/lib/avalon_context.py b/openpype/lib/avalon_context.py index 08c221cb81..12f4a5198b 100644 --- a/openpype/lib/avalon_context.py +++ b/openpype/lib/avalon_context.py @@ -192,6 +192,7 @@ def get_system_general_anatomy_data(system_settings=None): return get_general_template_data(system_settings) +@deprecated("openpype.client.get_linked_asset_ids") def get_linked_asset_ids(asset_doc): """Return linked asset ids for `asset_doc` from DB @@ -200,26 +201,20 @@ def get_linked_asset_ids(asset_doc): Returns: (list): MongoDB ids of input links. + + Deprecated: + Function will be removed after release version 3.16.* """ - output = [] - if not asset_doc: - return output - input_links = asset_doc["data"].get("inputLinks") or [] - if input_links: - for item in input_links: - # Backwards compatibility for "_id" key which was replaced with - # "id" - if "_id" in item: - link_id = item["_id"] - else: - link_id = item["id"] - output.append(link_id) + from openpype.client import get_linked_asset_ids + from openpype.pipeline import legacy_io - return output + project_name = legacy_io.active_project() + + return get_linked_asset_ids(project_name, asset_doc=asset_doc) -@with_pipeline_io +@deprecated("openpype.client.get_linked_assets") def get_linked_assets(asset_doc): """Return linked assets for `asset_doc` from DB @@ -228,14 +223,17 @@ def get_linked_assets(asset_doc): Returns: (list) Asset documents of input links for passed asset doc. + + Deprecated: + Function will be removed after release version 3.15.* """ - link_ids = get_linked_asset_ids(asset_doc) - if not link_ids: - return [] + from openpype.pipeline import legacy_io + from openpype.client import get_linked_assets project_name = legacy_io.active_project() - return list(get_assets(project_name, link_ids)) + + return get_linked_assets(project_name, asset_doc=asset_doc) @deprecated("openpype.client.get_last_version_by_subset_name") @@ -1041,9 +1039,10 @@ def get_last_workfile( ) -@with_pipeline_io -def get_linked_ids_for_representations(project_name, repre_ids, dbcon=None, - link_type=None, max_depth=0): +@deprecated("openpype.client.get_linked_representation_id") +def get_linked_ids_for_representations( + project_name, repre_ids, dbcon=None, link_type=None, max_depth=0 +): """Returns list of linked ids of particular type (if provided). Goes from representations to version, back to representations @@ -1054,104 +1053,25 @@ def get_linked_ids_for_representations(project_name, repre_ids, dbcon=None, with Session. link_type (str): ['reference', '..] max_depth (int): limit how many levels of recursion + Returns: (list) of ObjectId - linked representations + + Deprecated: + Function will be removed after release version 3.16.* """ - # Create new dbcon if not passed and use passed project name - if not dbcon: - from openpype.pipeline import AvalonMongoDB - dbcon = AvalonMongoDB() - dbcon.Session["AVALON_PROJECT"] = project_name - # Validate that passed dbcon has same project - elif dbcon.Session["AVALON_PROJECT"] != project_name: - raise ValueError("Passed connection does not have right project") + + from openpype.client import get_linked_representation_id if not isinstance(repre_ids, list): repre_ids = [repre_ids] - version_ids = dbcon.distinct("parent", { - "_id": {"$in": repre_ids}, - "type": "representation" - }) - - match = { - "_id": {"$in": version_ids}, - "type": "version" - } - - graph_lookup = { - "from": project_name, - "startWith": "$data.inputLinks.id", - "connectFromField": "data.inputLinks.id", - "connectToField": "_id", - "as": "outputs_recursive", - "depthField": "depth" - } - if max_depth != 0: - # We offset by -1 since 0 basically means no recursion - # but the recursion only happens after the initial lookup - # for outputs. - graph_lookup["maxDepth"] = max_depth - 1 - - pipeline_ = [ - # Match - {"$match": match}, - # Recursive graph lookup for inputs - {"$graphLookup": graph_lookup} - ] - - result = dbcon.aggregate(pipeline_) - referenced_version_ids = _process_referenced_pipeline_result(result, - link_type) - - ref_ids = dbcon.distinct( - "_id", - filter={ - "parent": {"$in": list(referenced_version_ids)}, - "type": "representation" - } - ) - - return list(ref_ids) - - -def _process_referenced_pipeline_result(result, link_type): - """Filters result from pipeline for particular link_type. - - Pipeline cannot use link_type directly in a query. - Returns: - (list) - """ - referenced_version_ids = set() - correctly_linked_ids = set() - for item in result: - input_links = item["data"].get("inputLinks", []) - correctly_linked_ids = _filter_input_links(input_links, - link_type, - correctly_linked_ids) - - # outputs_recursive in random order, sort by depth - outputs_recursive = sorted(item.get("outputs_recursive", []), - key=lambda d: d["depth"]) - - for output in outputs_recursive: - if output["_id"] not in correctly_linked_ids: # leaf - continue - - correctly_linked_ids = _filter_input_links( - output["data"].get("inputLinks", []), - link_type, - correctly_linked_ids) - - referenced_version_ids.add(output["_id"]) - - return referenced_version_ids - - -def _filter_input_links(input_links, link_type, correctly_linked_ids): - for input_link in input_links: - if not link_type or input_link["type"] == link_type: - correctly_linked_ids.add(input_link.get("id") or - input_link.get("_id")) # legacy - - return correctly_linked_ids + output = [] + for repre_id in repre_ids: + output.extend(get_linked_representation_id( + project_name, + repre_id=repre_id, + link_type=link_type, + max_depth=max_depth + )) + return output diff --git a/openpype/lib/path_tools.py b/openpype/lib/path_tools.py index f60b2fa722..0b6d0a3391 100644 --- a/openpype/lib/path_tools.py +++ b/openpype/lib/path_tools.py @@ -1,21 +1,60 @@ import os import re -import abc -import json import logging -import six import platform +import functools +import warnings import clique -from openpype.client import get_project -from openpype.settings import get_project_settings - -from .profiles_filtering import filter_profiles - log = logging.getLogger(__name__) +class PathToolsDeprecatedWarning(DeprecationWarning): + pass + + +def deprecated(new_destination): + """Mark functions as deprecated. + + It will result in a warning being emitted when the function is used. + """ + + func = None + if callable(new_destination): + func = new_destination + new_destination = None + + def _decorator(decorated_func): + if new_destination is None: + warning_message = ( + " Please check content of deprecated function to figure out" + " possible replacement." + ) + else: + warning_message = " Please replace your usage with '{}'.".format( + new_destination + ) + + @functools.wraps(decorated_func) + def wrapper(*args, **kwargs): + warnings.simplefilter("always", PathToolsDeprecatedWarning) + warnings.warn( + ( + "Call to deprecated function '{}'" + "\nFunction was moved or removed.{}" + ).format(decorated_func.__name__, warning_message), + category=PathToolsDeprecatedWarning, + stacklevel=4 + ) + return decorated_func(*args, **kwargs) + return wrapper + + if func is None: + return _decorator + return _decorator(func) + + def format_file_size(file_size, suffix=None): """Returns formatted string with size in appropriate unit. @@ -232,107 +271,69 @@ def get_last_version_from_path(path_dir, filter): return None +@deprecated("openpype.pipeline.project_folders.concatenate_splitted_paths") def concatenate_splitted_paths(split_paths, anatomy): - pattern_array = re.compile(r"\[.*\]") - output = [] - for path_items in split_paths: - clean_items = [] - if isinstance(path_items, str): - path_items = [path_items] + """ + Deprecated: + Function will be removed after release version 3.16.* + """ - for path_item in path_items: - if not re.match(r"{.+}", path_item): - path_item = re.sub(pattern_array, "", path_item) - clean_items.append(path_item) + from openpype.pipeline.project_folders import concatenate_splitted_paths - # backward compatibility - if "__project_root__" in path_items: - for root, root_path in anatomy.roots.items(): - if not os.path.exists(str(root_path)): - log.debug("Root {} path path {} not exist on \ - computer!".format(root, root_path)) - continue - clean_items = ["{{root[{}]}}".format(root), - r"{project[name]}"] + clean_items[1:] - output.append(os.path.normpath(os.path.sep.join(clean_items))) - continue - - output.append(os.path.normpath(os.path.sep.join(clean_items))) - - return output + return concatenate_splitted_paths(split_paths, anatomy) +@deprecated def get_format_data(anatomy): - project_doc = get_project(anatomy.project_name, fields=["data.code"]) - project_code = project_doc["data"]["code"] + """ + Deprecated: + Function will be removed after release version 3.16.* + """ - return { - "root": anatomy.roots, - "project": { - "name": anatomy.project_name, - "code": project_code - }, - } + from openpype.pipeline.template_data import get_project_template_data + + data = get_project_template_data(project_name=anatomy.project_name) + data["root"] = anatomy.roots + return data +@deprecated("openpype.pipeline.project_folders.fill_paths") def fill_paths(path_list, anatomy): - format_data = get_format_data(anatomy) - filled_paths = [] + """ + Deprecated: + Function will be removed after release version 3.16.* + """ - for path in path_list: - new_path = path.format(**format_data) - filled_paths.append(new_path) + from openpype.pipeline.project_folders import fill_paths - return filled_paths + return fill_paths(path_list, anatomy) +@deprecated("openpype.pipeline.project_folders.create_project_folders") def create_project_folders(basic_paths, project_name): - from openpype.pipeline import Anatomy - anatomy = Anatomy(project_name) + """ + Deprecated: + Function will be removed after release version 3.16.* + """ - concat_paths = concatenate_splitted_paths(basic_paths, anatomy) - filled_paths = fill_paths(concat_paths, anatomy) + from openpype.pipeline.project_folders import create_project_folders - # Create folders - for path in filled_paths: - if os.path.exists(path): - log.debug("Folder already exists: {}".format(path)) - else: - log.debug("Creating folder: {}".format(path)) - os.makedirs(path) - - -def _list_path_items(folder_structure): - output = [] - for key, value in folder_structure.items(): - if not value: - output.append(key) - else: - paths = _list_path_items(value) - for path in paths: - if not isinstance(path, (list, tuple)): - path = [path] - - item = [key] - item.extend(path) - output.append(item) - - return output + return create_project_folders(project_name, basic_paths) +@deprecated("openpype.pipeline.project_folders.get_project_basic_paths") def get_project_basic_paths(project_name): - project_settings = get_project_settings(project_name) - folder_structure = ( - project_settings["global"]["project_folder_structure"] - ) - if not folder_structure: - return [] + """ + Deprecated: + Function will be removed after release version 3.16.* + """ - if isinstance(folder_structure, str): - folder_structure = json.loads(folder_structure) - return _list_path_items(folder_structure) + from openpype.pipeline.project_folders import get_project_basic_paths + + return get_project_basic_paths(project_name) +@deprecated("openpype.pipeline.workfile.create_workdir_extra_folders") def create_workdir_extra_folders( workdir, host_name, task_type, task_name, project_name, project_settings=None @@ -349,37 +350,18 @@ def create_workdir_extra_folders( project_name (str): Name of project on which task is. project_settings (dict): Prepared project settings. Are loaded if not passed. + + Deprecated: + Function will be removed after release version 3.16.* """ - # Load project settings if not set - if not project_settings: - project_settings = get_project_settings(project_name) - # Load extra folders profiles - extra_folders_profiles = ( - project_settings["global"]["tools"]["Workfiles"]["extra_folders"] + from openpype.pipeline.project_folders import create_workdir_extra_folders + + return create_workdir_extra_folders( + workdir, + host_name, + task_type, + task_name, + project_name, + project_settings ) - # Skip if are empty - if not extra_folders_profiles: - return - - # Prepare profiles filters - filter_data = { - "task_types": task_type, - "task_names": task_name, - "hosts": host_name - } - profile = filter_profiles(extra_folders_profiles, filter_data) - if profile is None: - return - - for subfolder in profile["folders"]: - # Make sure backslashes are converted to forwards slashes - # and does not start with slash - subfolder = subfolder.replace("\\", "/").lstrip("/") - # Skip empty strings - if not subfolder: - continue - - fullpath = os.path.join(workdir, subfolder) - if not os.path.exists(fullpath): - os.makedirs(fullpath) diff --git a/openpype/lib/plugin_tools.py b/openpype/lib/plugin_tools.py index 81d268ea1c..1e157dfbfd 100644 --- a/openpype/lib/plugin_tools.py +++ b/openpype/lib/plugin_tools.py @@ -3,7 +3,6 @@ import os import logging import re -import json import warnings import functools diff --git a/openpype/lib/transcoding.py b/openpype/lib/transcoding.py index 60d5d3ed4a..87ab6bf4ed 100644 --- a/openpype/lib/transcoding.py +++ b/openpype/lib/transcoding.py @@ -139,7 +139,7 @@ def convert_value_by_type_name(value_type, value, logger=None): return float(value) # Vectors will probably have more types - if value_type == "vec2f": + if value_type in ("vec2f", "float2"): return [float(item) for item in value.split(",")] # Matrix should be always have square size of element 3x3, 4x4 @@ -154,7 +154,7 @@ def convert_value_by_type_name(value_type, value, logger=None): elif parts_len == 4: divisor = 2 elif parts_len == 9: - divisor == 3 + divisor = 3 elif parts_len == 16: divisor = 4 else: @@ -204,8 +204,8 @@ def convert_value_by_type_name(value_type, value, logger=None): ) return output - logger.info(( - "MISSING IMPLEMENTATION:" + logger.debug(( + "Dev note (missing implementation):" " Unknown attrib type \"{}\". Value: {}" ).format(value_type, value)) return value @@ -263,8 +263,8 @@ def parse_oiio_xml_output(xml_string, logger=None): # - feel free to add more tags else: value = child.text - logger.info(( - "MISSING IMPLEMENTATION:" + logger.debug(( + "Dev note (missing implementation):" " Unknown tag \"{}\". Value \"{}\"" ).format(tag_name, value)) diff --git a/openpype/modules/deadline/abstract_submit_deadline.py b/openpype/modules/deadline/abstract_submit_deadline.py index 0bad981fdf..512ff800ee 100644 --- a/openpype/modules/deadline/abstract_submit_deadline.py +++ b/openpype/modules/deadline/abstract_submit_deadline.py @@ -9,6 +9,7 @@ import os from abc import abstractmethod import platform import getpass +from functools import partial from collections import OrderedDict import six @@ -66,6 +67,96 @@ def requests_get(*args, **kwargs): return requests.get(*args, **kwargs) +class DeadlineKeyValueVar(dict): + """ + + Serializes dictionary key values as "{key}={value}" like Deadline uses + for EnvironmentKeyValue. + + As an example: + EnvironmentKeyValue0="A_KEY=VALUE_A" + EnvironmentKeyValue1="OTHER_KEY=VALUE_B" + + The keys are serialized in alphabetical order (sorted). + + Example: + >>> var = DeadlineKeyValueVar("EnvironmentKeyValue") + >>> var["my_var"] = "hello" + >>> var["my_other_var"] = "hello2" + >>> var.serialize() + + + """ + def __init__(self, key): + super(DeadlineKeyValueVar, self).__init__() + self.__key = key + + def serialize(self): + key = self.__key + + # Allow custom location for index in serialized string + if "{}" not in key: + key = key + "{}" + + return { + key.format(index): "{}={}".format(var_key, var_value) + for index, (var_key, var_value) in enumerate(sorted(self.items())) + } + + +class DeadlineIndexedVar(dict): + """ + + Allows to set and query values by integer indices: + Query: var[1] or var.get(1) + Set: var[1] = "my_value" + Append: var += "value" + + Note: Iterating the instance is not guarantueed to be the order of the + indices. To do so iterate with `sorted()` + + """ + def __init__(self, key): + super(DeadlineIndexedVar, self).__init__() + self.__key = key + + def serialize(self): + key = self.__key + + # Allow custom location for index in serialized string + if "{}" not in key: + key = key + "{}" + + return { + key.format(index): value for index, value in sorted(self.items()) + } + + def next_available_index(self): + # Add as first unused entry + i = 0 + while i in self.keys(): + i += 1 + return i + + def update(self, data): + # Force the integer key check + for key, value in data.items(): + self.__setitem__(key, value) + + def __iadd__(self, other): + index = self.next_available_index() + self[index] = other + return self + + def __setitem__(self, key, value): + if not isinstance(key, int): + raise TypeError("Key must be an integer: {}".format(key)) + + if key < 0: + raise ValueError("Negative index can't be set: {}".format(key)) + dict.__setitem__(self, key, value) + + @attr.s class DeadlineJobInfo(object): """Mapping of all Deadline *JobInfo* attributes. @@ -218,24 +309,8 @@ class DeadlineJobInfo(object): # Environment # ---------------------------------------------- - _environmentKeyValue = attr.ib(factory=list) - - @property - def EnvironmentKeyValue(self): # noqa: N802 - """Return all environment key values formatted for Deadline. - - Returns: - dict: as `{'EnvironmentKeyValue0', 'key=value'}` - - """ - out = {} - for index, v in enumerate(self._environmentKeyValue): - out["EnvironmentKeyValue{}".format(index)] = v - return out - - @EnvironmentKeyValue.setter - def EnvironmentKeyValue(self, val): # noqa: N802 - self._environmentKeyValue.append(val) + EnvironmentKeyValue = attr.ib(factory=partial(DeadlineKeyValueVar, + "EnvironmentKeyValue")) IncludeEnvironment = attr.ib(default=None) # Default: false UseJobEnvironmentOnly = attr.ib(default=None) # Default: false @@ -243,121 +318,29 @@ class DeadlineJobInfo(object): # Job Extra Info # ---------------------------------------------- - _extraInfos = attr.ib(factory=list) - _extraInfoKeyValues = attr.ib(factory=list) - - @property - def ExtraInfo(self): # noqa: N802 - """Return all ExtraInfo values formatted for Deadline. - - Returns: - dict: as `{'ExtraInfo0': 'value'}` - - """ - out = {} - for index, v in enumerate(self._extraInfos): - out["ExtraInfo{}".format(index)] = v - return out - - @ExtraInfo.setter - def ExtraInfo(self, val): # noqa: N802 - self._extraInfos.append(val) - - @property - def ExtraInfoKeyValue(self): # noqa: N802 - """Return all ExtraInfoKeyValue values formatted for Deadline. - - Returns: - dict: as {'ExtraInfoKeyValue0': 'key=value'}` - - """ - out = {} - for index, v in enumerate(self._extraInfoKeyValues): - out["ExtraInfoKeyValue{}".format(index)] = v - return out - - @ExtraInfoKeyValue.setter - def ExtraInfoKeyValue(self, val): # noqa: N802 - self._extraInfoKeyValues.append(val) + ExtraInfo = attr.ib(factory=partial(DeadlineIndexedVar, "ExtraInfo")) + ExtraInfoKeyValue = attr.ib(factory=partial(DeadlineKeyValueVar, + "ExtraInfoKeyValue")) # Task Extra Info Names # ---------------------------------------------- OverrideTaskExtraInfoNames = attr.ib(default=None) # Default: false - _taskExtraInfos = attr.ib(factory=list) - - @property - def TaskExtraInfoName(self): # noqa: N802 - """Return all TaskExtraInfoName values formatted for Deadline. - - Returns: - dict: as `{'TaskExtraInfoName0': 'value'}` - - """ - out = {} - for index, v in enumerate(self._taskExtraInfos): - out["TaskExtraInfoName{}".format(index)] = v - return out - - @TaskExtraInfoName.setter - def TaskExtraInfoName(self, val): # noqa: N802 - self._taskExtraInfos.append(val) + TaskExtraInfoName = attr.ib(factory=partial(DeadlineIndexedVar, + "TaskExtraInfoName")) # Output # ---------------------------------------------- - _outputFilename = attr.ib(factory=list) - _outputFilenameTile = attr.ib(factory=list) - _outputDirectory = attr.ib(factory=list) + OutputFilename = attr.ib(factory=partial(DeadlineIndexedVar, + "OutputFilename")) + OutputFilenameTile = attr.ib(factory=partial(DeadlineIndexedVar, + "OutputFilename{}Tile")) + OutputDirectory = attr.ib(factory=partial(DeadlineIndexedVar, + "OutputDirectory")) - @property - def OutputFilename(self): # noqa: N802 - """Return all OutputFilename values formatted for Deadline. - - Returns: - dict: as `{'OutputFilename0': 'filename'}` - - """ - out = {} - for index, v in enumerate(self._outputFilename): - out["OutputFilename{}".format(index)] = v - return out - - @OutputFilename.setter - def OutputFilename(self, val): # noqa: N802 - self._outputFilename.append(val) - - @property - def OutputFilenameTile(self): # noqa: N802 - """Return all OutputFilename#Tile values formatted for Deadline. - - Returns: - dict: as `{'OutputFilenme#Tile': 'tile'}` - - """ - out = {} - for index, v in enumerate(self._outputFilenameTile): - out["OutputFilename{}Tile".format(index)] = v - return out - - @OutputFilenameTile.setter - def OutputFilenameTile(self, val): # noqa: N802 - self._outputFilenameTile.append(val) - - @property - def OutputDirectory(self): # noqa: N802 - """Return all OutputDirectory values formatted for Deadline. - - Returns: - dict: as `{'OutputDirectory0': 'dir'}` - - """ - out = {} - for index, v in enumerate(self._outputDirectory): - out["OutputDirectory{}".format(index)] = v - return out - - @OutputDirectory.setter - def OutputDirectory(self, val): # noqa: N802 - self._outputDirectory.append(val) + # Asset Dependency + # ---------------------------------------------- + AssetDependency = attr.ib(factory=partial(DeadlineIndexedVar, + "AssetDependency")) # Tile Job # ---------------------------------------------- @@ -381,7 +364,7 @@ class DeadlineJobInfo(object): """ def filter_data(a, v): - if a.name.startswith("_"): + if isinstance(v, (DeadlineIndexedVar, DeadlineKeyValueVar)): return False if v is None: return False @@ -389,15 +372,27 @@ class DeadlineJobInfo(object): serialized = attr.asdict( self, dict_factory=OrderedDict, filter=filter_data) - serialized.update(self.EnvironmentKeyValue) - serialized.update(self.ExtraInfo) - serialized.update(self.ExtraInfoKeyValue) - serialized.update(self.TaskExtraInfoName) - serialized.update(self.OutputFilename) - serialized.update(self.OutputFilenameTile) - serialized.update(self.OutputDirectory) + + # Custom serialize these attributes + for attribute in [ + self.EnvironmentKeyValue, + self.ExtraInfo, + self.ExtraInfoKeyValue, + self.TaskExtraInfoName, + self.OutputFilename, + self.OutputFilenameTile, + self.OutputDirectory, + self.AssetDependency + ]: + serialized.update(attribute.serialize()) + return serialized + def update(self, data): + """Update instance with data dict""" + for key, value in data.items(): + setattr(self, key, value) + @six.add_metaclass(AbstractMetaInstancePlugin) class AbstractSubmitDeadline(pyblish.api.InstancePlugin): @@ -521,68 +516,72 @@ class AbstractSubmitDeadline(pyblish.api.InstancePlugin): published. """ - anatomy = self._instance.context.data['anatomy'] - file_path = None - for i in self._instance.context: - if "workfile" in i.data["families"] \ - or i.data["family"] == "workfile": - # test if there is instance of workfile waiting - # to be published. - assert i.data["publish"] is True, ( - "Workfile (scene) must be published along") - # determine published path from Anatomy. - template_data = i.data.get("anatomyData") - rep = i.data.get("representations")[0].get("ext") - template_data["representation"] = rep - template_data["ext"] = rep - template_data["comment"] = None - anatomy_filled = anatomy.format(template_data) - template_filled = anatomy_filled["publish"]["path"] - file_path = os.path.normpath(template_filled) - self.log.info("Using published scene for render {}".format( - file_path)) + instance = self._instance + workfile_instance = self._get_workfile_instance(instance.context) + if workfile_instance is None: + return - if not os.path.exists(file_path): - self.log.error("published scene does not exist!") - raise + # determine published path from Anatomy. + template_data = workfile_instance.data.get("anatomyData") + rep = workfile_instance.data.get("representations")[0] + template_data["representation"] = rep.get("name") + template_data["ext"] = rep.get("ext") + template_data["comment"] = None - if not replace_in_path: - return file_path + anatomy = instance.context.data['anatomy'] + anatomy_filled = anatomy.format(template_data) + template_filled = anatomy_filled["publish"]["path"] + file_path = os.path.normpath(template_filled) - # now we need to switch scene in expected files - # because token will now point to published - # scene file and that might differ from current one - new_scene = os.path.splitext( - os.path.basename(file_path))[0] - orig_scene = os.path.splitext( - os.path.basename( - self._instance.context.data["currentFile"]))[0] - exp = self._instance.data.get("expectedFiles") + self.log.info("Using published scene for render {}".format(file_path)) - if isinstance(exp[0], dict): - # we have aovs and we need to iterate over them - new_exp = {} - for aov, files in exp[0].items(): - replaced_files = [] - for f in files: - replaced_files.append( - str(f).replace(orig_scene, new_scene) - ) - new_exp[aov] = replaced_files - # [] might be too much here, TODO - self._instance.data["expectedFiles"] = [new_exp] - else: - new_exp = [] - for f in exp: - new_exp.append( - str(f).replace(orig_scene, new_scene) - ) - self._instance.data["expectedFiles"] = new_exp + if not os.path.exists(file_path): + self.log.error("published scene does not exist!") + raise - self.log.info("Scene name was switched {} -> {}".format( - orig_scene, new_scene - )) + if not replace_in_path: + return file_path + + # now we need to switch scene in expected files + # because token will now point to published + # scene file and that might differ from current one + def _clean_name(path): + return os.path.splitext(os.path.basename(path))[0] + + new_scene = _clean_name(file_path) + orig_scene = _clean_name(instance.context.data["currentFile"]) + expected_files = instance.data.get("expectedFiles") + + if isinstance(expected_files[0], dict): + # we have aovs and we need to iterate over them + new_exp = {} + for aov, files in expected_files[0].items(): + replaced_files = [] + for f in files: + replaced_files.append( + str(f).replace(orig_scene, new_scene) + ) + new_exp[aov] = replaced_files + # [] might be too much here, TODO + instance.data["expectedFiles"] = [new_exp] + else: + new_exp = [] + for f in expected_files: + new_exp.append( + str(f).replace(orig_scene, new_scene) + ) + instance.data["expectedFiles"] = new_exp + + metadata_folder = instance.data.get("publishRenderMetadataFolder") + if metadata_folder: + metadata_folder = metadata_folder.replace(orig_scene, + new_scene) + instance.data["publishRenderMetadataFolder"] = metadata_folder + + self.log.info("Scene name was switched {} -> {}".format( + orig_scene, new_scene + )) return file_path @@ -645,3 +644,22 @@ class AbstractSubmitDeadline(pyblish.api.InstancePlugin): self._instance.data["deadlineSubmissionJob"] = result return result["_id"] + + @staticmethod + def _get_workfile_instance(context): + """Find workfile instance in context""" + for i in context: + + is_workfile = ( + "workfile" in i.data.get("families", []) or + i.data["family"] == "workfile" + ) + if not is_workfile: + continue + + # test if there is instance of workfile waiting + # to be published. + assert i.data["publish"] is True, ( + "Workfile (scene) must be published along") + + return i diff --git a/openpype/modules/deadline/plugins/publish/collect_deadline_server_from_instance.py b/openpype/modules/deadline/plugins/publish/collect_deadline_server_from_instance.py index a7035cd99f..9981bead3e 100644 --- a/openpype/modules/deadline/plugins/publish/collect_deadline_server_from_instance.py +++ b/openpype/modules/deadline/plugins/publish/collect_deadline_server_from_instance.py @@ -13,7 +13,7 @@ class CollectDeadlineServerFromInstance(pyblish.api.InstancePlugin): order = pyblish.api.CollectorOrder + 0.415 label = "Deadline Webservice from the Instance" - families = ["rendering"] + families = ["rendering", "renderlayer"] def process(self, instance): instance.data["deadlineUrl"] = self._collect_deadline_url(instance) diff --git a/openpype/modules/deadline/plugins/publish/submit_aftereffects_deadline.py b/openpype/modules/deadline/plugins/publish/submit_aftereffects_deadline.py index 1d68793d53..0c1ffa6bd7 100644 --- a/openpype/modules/deadline/plugins/publish/submit_aftereffects_deadline.py +++ b/openpype/modules/deadline/plugins/publish/submit_aftereffects_deadline.py @@ -67,9 +67,9 @@ class AfterEffectsSubmitDeadline( dln_job_info.Group = self.group dln_job_info.Department = self.department dln_job_info.ChunkSize = self.chunk_size - dln_job_info.OutputFilename = \ + dln_job_info.OutputFilename += \ os.path.basename(self._instance.data["expectedFiles"][0]) - dln_job_info.OutputDirectory = \ + dln_job_info.OutputDirectory += \ os.path.dirname(self._instance.data["expectedFiles"][0]) dln_job_info.JobDelay = "00:00:00" @@ -92,13 +92,12 @@ class AfterEffectsSubmitDeadline( environment = dict({key: os.environ[key] for key in keys if key in os.environ}, **legacy_io.Session) for key in keys: - val = environment.get(key) - if val: - dln_job_info.EnvironmentKeyValue = "{key}={value}".format( - key=key, - value=val) + value = environment.get(key) + if value: + dln_job_info.EnvironmentKeyValue[key] = value + # to recognize job from PYPE for turning Event On/Off - dln_job_info.EnvironmentKeyValue = "OPENPYPE_RENDER_JOB=1" + dln_job_info.EnvironmentKeyValue["OPENPYPE_RENDER_JOB"] = "1" return dln_job_info diff --git a/openpype/modules/deadline/plugins/publish/submit_harmony_deadline.py b/openpype/modules/deadline/plugins/publish/submit_harmony_deadline.py index 3f9c09b592..6327143623 100644 --- a/openpype/modules/deadline/plugins/publish/submit_harmony_deadline.py +++ b/openpype/modules/deadline/plugins/publish/submit_harmony_deadline.py @@ -284,14 +284,12 @@ class HarmonySubmitDeadline( environment = dict({key: os.environ[key] for key in keys if key in os.environ}, **legacy_io.Session) for key in keys: - val = environment.get(key) - if val: - job_info.EnvironmentKeyValue = "{key}={value}".format( - key=key, - value=val) + value = environment.get(key) + if value: + job_info.EnvironmentKeyValue[key] = value # to recognize job from PYPE for turning Event On/Off - job_info.EnvironmentKeyValue = "OPENPYPE_RENDER_JOB=1" + job_info.EnvironmentKeyValue["OPENPYPE_RENDER_JOB"] = "1" return job_info diff --git a/openpype/modules/deadline/plugins/publish/submit_maya_deadline.py b/openpype/modules/deadline/plugins/publish/submit_maya_deadline.py index 7966861358..44f2b5b2b4 100644 --- a/openpype/modules/deadline/plugins/publish/submit_maya_deadline.py +++ b/openpype/modules/deadline/plugins/publish/submit_maya_deadline.py @@ -18,7 +18,6 @@ Attributes: from __future__ import print_function import os -import json import getpass import copy import re @@ -27,45 +26,686 @@ from datetime import datetime import itertools from collections import OrderedDict -import clique -import requests +import attr from maya import cmds -import pyblish.api - -from openpype.lib import requests_post -from openpype.hosts.maya.api import lib from openpype.pipeline import legacy_io -# Documentation for keys available at: -# https://docs.thinkboxsoftware.com -# /products/deadline/8.0/1_User%20Manual/manual -# /manual-submission.html#job-info-file-options +from openpype_modules.deadline import abstract_submit_deadline +from openpype_modules.deadline.abstract_submit_deadline import DeadlineJobInfo -payload_skeleton_template = { - "JobInfo": { - "BatchName": None, # Top-level group name - "Name": None, # Job name, as seen in Monitor - "UserName": None, - "Plugin": "MayaBatch", - "Frames": "{start}-{end}x{step}", - "Comment": None, - "Priority": 50, - }, - "PluginInfo": { - "SceneFile": None, # Input - "OutputFilePath": None, # Output directory and filename - "OutputFilePrefix": None, - "Version": cmds.about(version=True), # Mandatory for Deadline - "UsingRenderLayers": True, - "RenderLayer": None, # Render only this layer - "Renderer": None, - "ProjectPath": None, # Resolve relative references - "RenderSetupIncludeLights": None, # Include all lights flag. - }, - "AuxFiles": [] # Mandatory for Deadline, may be empty -} + +@attr.s +class MayaPluginInfo: + SceneFile = attr.ib(default=None) # Input + OutputFilePath = attr.ib(default=None) # Output directory and filename + OutputFilePrefix = attr.ib(default=None) + Version = attr.ib(default=None) # Mandatory for Deadline + UsingRenderLayers = attr.ib(default=True) + RenderLayer = attr.ib(default=None) # Render only this layer + Renderer = attr.ib(default=None) + ProjectPath = attr.ib(default=None) # Resolve relative references + RenderSetupIncludeLights = attr.ib(default=None) # Include all lights flag + + +@attr.s +class PythonPluginInfo: + ScriptFile = attr.ib() + Version = attr.ib(default="3.6") + Arguments = attr.ib(default=None) + SingleFrameOnly = attr.ib(default=None) + + +@attr.s +class VRayPluginInfo: + InputFilename = attr.ib(default=None) # Input + SeparateFilesPerFrame = attr.ib(default=None) + VRayEngine = attr.ib(default="V-Ray") + Width = attr.ib(default=None) + Height = attr.ib(default=None) # Mandatory for Deadline + OutputFilePath = attr.ib(default=True) + OutputFileName = attr.ib(default=None) # Render only this layer + + +@attr.s +class ArnoldPluginInfo: + ArnoldFile = attr.ib(default=None) + + +class MayaSubmitDeadline(abstract_submit_deadline.AbstractSubmitDeadline): + + label = "Submit Render to Deadline" + hosts = ["maya"] + families = ["renderlayer"] + targets = ["local"] + + tile_assembler_plugin = "OpenPypeTileAssembler" + priority = 50 + tile_priority = 50 + limit = [] # limit groups + jobInfo = {} + pluginInfo = {} + group = "none" + + def get_job_info(self): + job_info = DeadlineJobInfo(Plugin="MayaBatch") + + # todo: test whether this works for existing production cases + # where custom jobInfo was stored in the project settings + job_info.update(self.jobInfo) + + instance = self._instance + context = instance.context + + # Always use the original work file name for the Job name even when + # rendering is done from the published Work File. The original work + # file name is clearer because it can also have subversion strings, + # etc. which are stripped for the published file. + src_filepath = context.data["currentFile"] + src_filename = os.path.basename(src_filepath) + + job_info.Name = "%s - %s" % (src_filename, instance.name) + job_info.BatchName = src_filename + job_info.Plugin = instance.data.get("mayaRenderPlugin", "MayaBatch") + job_info.UserName = context.data.get("deadlineUser", getpass.getuser()) + + # Deadline requires integers in frame range + frames = "{start}-{end}x{step}".format( + start=int(instance.data["frameStartHandle"]), + end=int(instance.data["frameEndHandle"]), + step=int(instance.data["byFrameStep"]), + ) + job_info.Frames = frames + + job_info.Pool = instance.data.get("primaryPool") + job_info.SecondaryPool = instance.data.get("secondaryPool") + job_info.ChunkSize = instance.data.get("chunkSize", 10) + job_info.Comment = context.data.get("comment") + job_info.Priority = instance.data.get("priority", self.priority) + job_info.FramesPerTask = instance.data.get("framesPerTask", 1) + + if self.group != "none" and self.group: + job_info.Group = self.group + + if self.limit: + job_info.LimitGroups = ",".join(self.limit) + + # Add options from RenderGlobals + render_globals = instance.data.get("renderGlobals", {}) + job_info.update(render_globals) + + keys = [ + "FTRACK_API_KEY", + "FTRACK_API_USER", + "FTRACK_SERVER", + "OPENPYPE_SG_USER", + "AVALON_PROJECT", + "AVALON_ASSET", + "AVALON_TASK", + "AVALON_APP_NAME", + "OPENPYPE_DEV", + "OPENPYPE_VERSION" + ] + # Add mongo url if it's enabled + if self._instance.context.data.get("deadlinePassMongoUrl"): + keys.append("OPENPYPE_MONGO") + + environment = dict({key: os.environ[key] for key in keys + if key in os.environ}, **legacy_io.Session) + + for key in keys: + value = environment.get(key) + if not value: + continue + job_info.EnvironmentKeyValue[key] = value + + # to recognize job from PYPE for turning Event On/Off + job_info.EnvironmentKeyValue["OPENPYPE_RENDER_JOB"] = "1" + job_info.EnvironmentKeyValue["OPENPYPE_LOG_NO_COLORS"] = "1" + + # Adding file dependencies. + if self.asset_dependencies: + dependencies = instance.context.data["fileDependencies"] + dependencies.append(context.data["currentFile"]) + for dependency in dependencies: + job_info.AssetDependency += dependency + + # Add list of expected files to job + # --------------------------------- + exp = instance.data.get("expectedFiles") + for filepath in self._iter_expected_files(exp): + job_info.OutputDirectory += os.path.dirname(filepath) + job_info.OutputFilename += os.path.basename(filepath) + + return job_info + + def get_plugin_info(self): + + instance = self._instance + context = instance.context + + plugin_info = MayaPluginInfo( + SceneFile=self.scene_path, + Version=cmds.about(version=True), + RenderLayer=instance.data['setMembers'], + Renderer=instance.data["renderer"], + RenderSetupIncludeLights=instance.data.get("renderSetupIncludeLights"), # noqa + ProjectPath=context.data["workspaceDir"], + UsingRenderLayers=True, + ) + + plugin_payload = attr.asdict(plugin_info) + + # Patching with pluginInfo from settings + for key, value in self.pluginInfo.items(): + plugin_payload[key] = value + + return plugin_payload + + def process_submission(self): + + instance = self._instance + context = instance.context + + filepath = self.scene_path # publish if `use_publish` else workfile + + # TODO: Avoid the need for this logic here, needed for submit publish + # Store output dir for unified publisher (filesequence) + expected_files = instance.data["expectedFiles"] + first_file = next(self._iter_expected_files(expected_files)) + output_dir = os.path.dirname(first_file) + instance.data["outputDir"] = output_dir + instance.data["toBeRenderedOn"] = "deadline" + + # Patch workfile (only when use_published is enabled) + if self.use_published: + self._patch_workfile() + + # Gather needed data ------------------------------------------------ + workspace = context.data["workspaceDir"] + default_render_file = instance.context.data.get('project_settings')\ + .get('maya')\ + .get('RenderSettings')\ + .get('default_render_image_folder') + filename = os.path.basename(filepath) + dirname = os.path.join(workspace, default_render_file) + + # Fill in common data to payload ------------------------------------ + # TODO: Replace these with collected data from CollectRender + payload_data = { + "filename": filename, + "dirname": dirname, + } + + # Submit preceding export jobs ------------------------------------- + export_job = None + assert not all(x in instance.data["families"] + for x in ['vrayscene', 'assscene']), ( + "Vray Scene and Ass Scene options are mutually exclusive") + + if "vrayscene" in instance.data["families"]: + self.log.debug("Submitting V-Ray scene render..") + vray_export_payload = self._get_vray_export_payload(payload_data) + export_job = self.submit(vray_export_payload) + + payload = self._get_vray_render_payload(payload_data) + + elif "assscene" in instance.data["families"]: + self.log.debug("Submitting Arnold .ass standalone render..") + ass_export_payload = self._get_arnold_export_payload(payload_data) + export_job = self.submit(ass_export_payload) + + payload = self._get_arnold_render_payload(payload_data) + else: + self.log.debug("Submitting MayaBatch render..") + payload = self._get_maya_payload(payload_data) + + # Add export job as dependency -------------------------------------- + if export_job: + job_info, _ = payload + job_info.JobDependency = export_job + + if instance.data.get("tileRendering"): + # Prepare tiles data + self._tile_render(payload) + else: + # Submit main render job + job_info, plugin_info = payload + self.submit(self.assemble_payload(job_info, plugin_info)) + + def _tile_render(self, payload): + """Submit as tile render per frame with dependent assembly jobs.""" + + # As collected by super process() + instance = self._instance + + payload_job_info, payload_plugin_info = payload + job_info = copy.deepcopy(payload_job_info) + plugin_info = copy.deepcopy(payload_plugin_info) + + # if we have sequence of files, we need to create tile job for + # every frame + job_info.TileJob = True + job_info.TileJobTilesInX = instance.data.get("tilesX") + job_info.TileJobTilesInY = instance.data.get("tilesY") + + tiles_count = job_info.TileJobTilesInX * job_info.TileJobTilesInY + + plugin_info["ImageHeight"] = instance.data.get("resolutionHeight") + plugin_info["ImageWidth"] = instance.data.get("resolutionWidth") + plugin_info["RegionRendering"] = True + + R_FRAME_NUMBER = re.compile( + r".+\.(?P[0-9]+)\..+") # noqa: N806, E501 + REPL_FRAME_NUMBER = re.compile( + r"(.+\.)([0-9]+)(\..+)") # noqa: N806, E501 + + exp = instance.data["expectedFiles"] + if isinstance(exp[0], dict): + # we have aovs and we need to iterate over them + # get files from `beauty` + files = exp[0].get("beauty") + # assembly files are used for assembly jobs as we need to put + # together all AOVs + assembly_files = list( + itertools.chain.from_iterable( + [f for _, f in exp[0].items()])) + if not files: + # if beauty doesn't exist, use first aov we found + files = exp[0].get(list(exp[0].keys())[0]) + else: + files = exp + assembly_files = files + + # Define frame tile jobs + frame_file_hash = {} + frame_payloads = {} + file_index = 1 + for file in files: + frame = re.search(R_FRAME_NUMBER, file).group("frame") + + new_job_info = copy.deepcopy(job_info) + new_job_info.Name += " (Frame {} - {} tiles)".format(frame, + tiles_count) + new_job_info.TileJobFrame = frame + + new_plugin_info = copy.deepcopy(plugin_info) + + # Add tile data into job info and plugin info + tiles_data = _format_tiles( + file, 0, + instance.data.get("tilesX"), + instance.data.get("tilesY"), + instance.data.get("resolutionWidth"), + instance.data.get("resolutionHeight"), + payload_plugin_info["OutputFilePrefix"] + )[0] + + new_job_info.update(tiles_data["JobInfo"]) + new_plugin_info.update(tiles_data["PluginInfo"]) + + self.log.info("hashing {} - {}".format(file_index, file)) + job_hash = hashlib.sha256( + ("{}_{}".format(file_index, file)).encode("utf-8")) + + file_hash = job_hash.hexdigest() + frame_file_hash[frame] = file_hash + + new_job_info.ExtraInfo[0] = file_hash + new_job_info.ExtraInfo[1] = file + + frame_payloads[frame] = self.assemble_payload( + job_info=new_job_info, + plugin_info=new_plugin_info + ) + file_index += 1 + + self.log.info( + "Submitting tile job(s) [{}] ...".format(len(frame_payloads))) + + # Submit frame tile jobs + frame_tile_job_id = {} + for frame, tile_job_payload in frame_payloads.items(): + job_id = self.submit(tile_job_payload) + frame_tile_job_id[frame] = job_id + + # Define assembly payloads + assembly_job_info = copy.deepcopy(job_info) + assembly_job_info.Plugin = self.tile_assembler_plugin + assembly_job_info.Name += " - Tile Assembly Job" + assembly_job_info.Frames = 1 + assembly_job_info.MachineLimit = 1 + assembly_job_info.Priority = instance.data.get("tile_priority", + self.tile_priority) + + assembly_plugin_info = { + "CleanupTiles": 1, + "ErrorOnMissing": True, + "Renderer": self._instance.data["renderer"] + } + + assembly_payloads = [] + output_dir = self.job_info.OutputDirectory[0] + for file in assembly_files: + frame = re.search(R_FRAME_NUMBER, file).group("frame") + + frame_assembly_job_info = copy.deepcopy(assembly_job_info) + frame_assembly_job_info.Name += " (Frame {})".format(frame) + frame_assembly_job_info.OutputFilename[0] = re.sub( + REPL_FRAME_NUMBER, + "\\1{}\\3".format("#" * len(frame)), file) + + file_hash = frame_file_hash[frame] + tile_job_id = frame_tile_job_id[frame] + + frame_assembly_job_info.ExtraInfo[0] = file_hash + frame_assembly_job_info.ExtraInfo[1] = file + frame_assembly_job_info.JobDependency = tile_job_id + + # write assembly job config files + now = datetime.now() + + config_file = os.path.join( + output_dir, + "{}_config_{}.txt".format( + os.path.splitext(file)[0], + now.strftime("%Y_%m_%d_%H_%M_%S") + ) + ) + try: + if not os.path.isdir(output_dir): + os.makedirs(output_dir) + except OSError: + # directory is not available + self.log.warning("Path is unreachable: " + "`{}`".format(output_dir)) + + with open(config_file, "w") as cf: + print("TileCount={}".format(tiles_count), file=cf) + print("ImageFileName={}".format(file), file=cf) + print("ImageWidth={}".format( + instance.data.get("resolutionWidth")), file=cf) + print("ImageHeight={}".format( + instance.data.get("resolutionHeight")), file=cf) + + tiles = _format_tiles( + file, 0, + instance.data.get("tilesX"), + instance.data.get("tilesY"), + instance.data.get("resolutionWidth"), + instance.data.get("resolutionHeight"), + payload_plugin_info["OutputFilePrefix"] + )[1] + for k, v in sorted(tiles.items()): + print("{}={}".format(k, v), file=cf) + + payload = self.assemble_payload( + job_info=frame_assembly_job_info, + plugin_info=assembly_plugin_info.copy(), + # todo: aux file transfers don't work with deadline webservice + # add config file as job auxFile + # aux_files=[config_file] + ) + assembly_payloads.append(payload) + + # Submit assembly jobs + assembly_job_ids = [] + num_assemblies = len(assembly_payloads) + for i, payload in enumerate(assembly_payloads): + self.log.info( + "submitting assembly job {} of {}".format(i + 1, + num_assemblies) + ) + assembly_job_id = self.submit(payload) + assembly_job_ids.append(assembly_job_id) + + instance.data["assemblySubmissionJobs"] = assembly_job_ids + + def _get_maya_payload(self, data): + + job_info = copy.deepcopy(self.job_info) + + if self.asset_dependencies: + # Asset dependency to wait for at least the scene file to sync. + job_info.AssetDependency += self.scene_path + + # Get layer prefix + render_products = self._instance.data["renderProducts"] + layer_metadata = render_products.layer_data + layer_prefix = layer_metadata.filePrefix + + # This hack is here because of how Deadline handles Renderman version. + # it considers everything with `renderman` set as version older than + # Renderman 22, and so if we are using renderman > 21 we need to set + # renderer string on the job to `renderman22`. We will have to change + # this when Deadline releases new version handling this. + renderer = self._instance.data["renderer"] + if renderer == "renderman": + try: + from rfm2.config import cfg # noqa + except ImportError: + raise Exception("Cannot determine renderman version") + + rman_version = cfg().build_info.version() # type: str + if int(rman_version.split(".")[0]) > 22: + renderer = "renderman22" + + plugin_info = copy.deepcopy(self.plugin_info) + plugin_info.update({ + # Output directory and filename + "OutputFilePath": data["dirname"].replace("\\", "/"), + "OutputFilePrefix": layer_prefix, + }) + + return job_info, plugin_info + + def _get_vray_export_payload(self, data): + + job_info = copy.deepcopy(self.job_info) + job_info.Name = self._job_info_label("Export") + + # Get V-Ray settings info to compute output path + vray_scene = self.format_vray_output_filename() + + plugin_info = { + "Renderer": "vray", + "SkipExistingFrames": True, + "UseLegacyRenderLayers": True, + "OutputFilePath": os.path.dirname(vray_scene) + } + + return job_info, attr.asdict(plugin_info) + + def _get_arnold_export_payload(self, data): + + try: + from openpype.scripts import export_maya_ass_job + except Exception: + raise AssertionError( + "Expected module 'export_maya_ass_job' to be available") + + module_path = export_maya_ass_job.__file__ + if module_path.endswith(".pyc"): + module_path = module_path[: -len(".pyc")] + ".py" + + script = os.path.normpath(module_path) + + job_info = copy.deepcopy(self.job_info) + job_info.Name = self._job_info_label("Export") + + # Force a single frame Python job + job_info.Plugin = "Python" + job_info.Frames = 1 + + renderlayer = self._instance.data["setMembers"] + + # add required env vars for the export script + envs = { + "AVALON_APP_NAME": os.environ.get("AVALON_APP_NAME"), + "OPENPYPE_ASS_EXPORT_RENDER_LAYER": renderlayer, + "OPENPYPE_ASS_EXPORT_SCENE_FILE": self.scene_path, + "OPENPYPE_ASS_EXPORT_OUTPUT": job_info.OutputFilename[0], + "OPENPYPE_ASS_EXPORT_START": int(self._instance.data["frameStartHandle"]), # noqa + "OPENPYPE_ASS_EXPORT_END": int(self._instance.data["frameEndHandle"]), # noqa + "OPENPYPE_ASS_EXPORT_STEP": 1 + } + for key, value in envs.items(): + if not value: + continue + job_info.EnvironmentKeyValue[key] = value + + plugin_info = PythonPluginInfo( + ScriptFile=script, + Version="3.6", + Arguments="", + SingleFrameOnly="True" + ) + + return job_info, attr.asdict(plugin_info) + + def _get_vray_render_payload(self, data): + + # Job Info + job_info = copy.deepcopy(self.job_info) + job_info.Name = self._job_info_label("Render") + job_info.Plugin = "Vray" + job_info.OverrideTaskExtraInfoNames = False + + # Plugin Info + plugin_info = VRayPluginInfo( + InputFilename=self.format_vray_output_filename(), + SeparateFilesPerFrame=False, + VRayEngine="V-Ray", + Width=self._instance.data["resolutionWidth"], + Height=self._instance.data["resolutionHeight"], + OutputFilePath=job_info.OutputDirectory[0], + OutputFileName=job_info.OutputFilename[0] + ) + + return job_info, attr.asdict(plugin_info) + + def _get_arnold_render_payload(self, data): + + # Job Info + job_info = copy.deepcopy(self.job_info) + job_info.Name = self._job_info_label("Render") + job_info.Plugin = "Arnold" + job_info.OverrideTaskExtraInfoNames = False + + # Plugin Info + ass_file, _ = os.path.splitext(data["output_filename_0"]) + ass_filepath = ass_file + ".ass" + + plugin_info = ArnoldPluginInfo( + ArnoldFile=ass_filepath + ) + + return job_info, attr.asdict(plugin_info) + + def format_vray_output_filename(self): + """Format the expected output file of the Export job. + + Example: + /_/ + "shot010_v006/shot010_v006_CHARS/CHARS_0001.vrscene" + Returns: + str + + """ + + # "vrayscene//_/" + vray_settings = cmds.ls(type="VRaySettingsNode") + node = vray_settings[0] + template = cmds.getAttr("{}.vrscene_filename".format(node)) + scene, _ = os.path.splitext(self.scene_path) + + def smart_replace(string, key_values): + new_string = string + for key, value in key_values.items(): + new_string = new_string.replace(key, value) + return new_string + + # Get workfile scene path without extension to format vrscene_filename + scene_filename = os.path.basename(self.scene_path) + scene_filename_no_ext, _ = os.path.splitext(scene_filename) + + layer = self._instance.data['setMembers'] + + # Reformat without tokens + output_path = smart_replace( + template, + {"": scene_filename_no_ext, + "": layer}) + + start_frame = int(self._instance.data["frameStartHandle"]) + workspace = self._instance.context.data["workspace"] + filename_zero = "{}_{:04d}.vrscene".format(output_path, start_frame) + filepath_zero = os.path.join(workspace, filename_zero) + + return filepath_zero.replace("\\", "/") + + def _patch_workfile(self): + """Patch Maya scene. + + This will take list of patches (lines to add) and apply them to + *published* Maya scene file (that is used later for rendering). + + Patches are dict with following structure:: + { + "name": "Name of patch", + "regex": "regex of line before patch", + "line": "line to insert" + } + + """ + project_settings = self._instance.context.data["project_settings"] + patches = ( + project_settings.get( + "deadline", {}).get( + "publish", {}).get( + "MayaSubmitDeadline", {}).get( + "scene_patches", {}) + ) + if not patches: + return + + if not os.path.splitext(self.scene_path)[1].lower() != ".ma": + self.log.debug("Skipping workfile patch since workfile is not " + ".ma file") + return + + compiled_regex = [re.compile(p["regex"]) for p in patches] + with open(self.scene_path, "r+") as pf: + scene_data = pf.readlines() + for ln, line in enumerate(scene_data): + for i, r in enumerate(compiled_regex): + if re.match(r, line): + scene_data.insert(ln + 1, patches[i]["line"]) + pf.seek(0) + pf.writelines(scene_data) + pf.truncate() + self.log.info("Applied {} patch to scene.".format( + patches[i]["name"] + )) + + def _job_info_label(self, label): + return "{label} {job.Name} [{start}-{end}]".format( + label=label, + job=self.job_info, + start=int(self._instance.data["frameStartHandle"]), + end=int(self._instance.data["frameEndHandle"]), + ) + + @staticmethod + def _iter_expected_files(exp): + if isinstance(exp[0], dict): + for _aov, files in exp[0].items(): + for file in files: + yield file + else: + for file in exp: + yield file def _format_tiles( @@ -94,7 +734,7 @@ def _format_tiles( Result for tile 0 for 4x4 will be: `maya///_tile_1x1_4x4__` - Calculating coordinates is tricky as in Job they are defined as top, + Calculating coordinates is tricky as in Job they are defined as top, left, bottom, right with zero being in top-left corner. But Assembler configuration file takes tile coordinates as X, Y, Width and Height and zero is bottom left corner. @@ -103,25 +743,32 @@ def _format_tiles( filename (str): Filename to process as tiles. index (int): Index of that file if it is sequence. tiles_x (int): Number of tiles in X. - tiles_y (int): Number if tikes in Y. + tiles_y (int): Number of tiles in Y. width (int): Width resolution of final image. height (int): Height resolution of final image. prefix (str): Image prefix. Returns: - (dict, dict): Tuple of two dictionaires - first can be used to + (dict, dict): Tuple of two dictionaries - first can be used to extend JobInfo, second has tiles x, y, width and height used for assembler configuration. """ - tile = 0 + # Math used requires integers for correct output - as such + # we ensure our inputs are correct. + assert type(tiles_x) is int, "tiles_x must be an integer" + assert type(tiles_y) is int, "tiles_y must be an integer" + assert type(width) is int, "width must be an integer" + assert type(height) is int, "height must be an integer" + out = {"JobInfo": {}, "PluginInfo": {}} cfg = OrderedDict() - w_space = width / tiles_x - h_space = height / tiles_y + w_space = width // tiles_x + h_space = height // tiles_y cfg["TilesCropped"] = "False" + tile = 0 for tile_x in range(1, tiles_x + 1): for tile_y in reversed(range(1, tiles_y + 1)): tile_prefix = "_tile_{}x{}_{}x{}_".format( @@ -129,1034 +776,38 @@ def _format_tiles( tiles_x, tiles_y ) - out_tile_index = "OutputFilename{}Tile{}".format( - str(index), tile - ) + new_filename = "{}/{}{}".format( os.path.dirname(filename), tile_prefix, os.path.basename(filename) ) - out["JobInfo"][out_tile_index] = new_filename + + top = height - (tile_y * h_space) + bottom = height - ((tile_y - 1) * h_space) - 1 + left = (tile_x - 1) * w_space + right = (tile_x * w_space) - 1 + + # Job info + out["JobInfo"]["OutputFilename{}Tile{}".format(index, tile)] = new_filename # noqa: E501 + + # Plugin Info out["PluginInfo"]["RegionPrefix{}".format(str(tile))] = \ "/{}".format(tile_prefix).join(prefix.rsplit("/", 1)) + out["PluginInfo"]["RegionTop{}".format(tile)] = top + out["PluginInfo"]["RegionBottom{}".format(tile)] = bottom + out["PluginInfo"]["RegionLeft{}".format(tile)] = left + out["PluginInfo"]["RegionRight{}".format(tile)] = right - out["PluginInfo"]["RegionTop{}".format(tile)] = int(height) - (tile_y * h_space) # noqa: E501 - out["PluginInfo"]["RegionBottom{}".format(tile)] = int(height) - ((tile_y - 1) * h_space) - 1 # noqa: E501 - out["PluginInfo"]["RegionLeft{}".format(tile)] = (tile_x - 1) * w_space # noqa: E501 - out["PluginInfo"]["RegionRight{}".format(tile)] = (tile_x * w_space) - 1 # noqa: E501 - + # Tile config cfg["Tile{}".format(tile)] = new_filename cfg["Tile{}Tile".format(tile)] = new_filename cfg["Tile{}FileName".format(tile)] = new_filename - cfg["Tile{}X".format(tile)] = (tile_x - 1) * w_space - - cfg["Tile{}Y".format(tile)] = int(height) - (tile_y * h_space) - + cfg["Tile{}X".format(tile)] = left + cfg["Tile{}Y".format(tile)] = top cfg["Tile{}Width".format(tile)] = w_space cfg["Tile{}Height".format(tile)] = h_space tile += 1 + return out, cfg - - -def get_renderer_variables(renderlayer, root): - """Retrieve the extension which has been set in the VRay settings. - - Will return None if the current renderer is not VRay - For Maya 2016.5 and up the renderSetup creates renderSetupLayer node which - start with `rs`. Use the actual node name, do NOT use the `nice name` - - Args: - renderlayer (str): the node name of the renderlayer. - root (str): base path to render - - Returns: - dict - - """ - renderer = lib.get_renderer(renderlayer or lib.get_current_renderlayer()) - render_attrs = lib.RENDER_ATTRS.get(renderer, lib.RENDER_ATTRS["default"]) - - padding = cmds.getAttr("{}.{}".format(render_attrs["node"], - render_attrs["padding"])) - - filename_0 = cmds.renderSettings( - fullPath=True, - gin="#" * int(padding), - lut=True, - layer=renderlayer or lib.get_current_renderlayer())[0] - filename_0 = re.sub('_', '_beauty', - filename_0, flags=re.IGNORECASE) - prefix_attr = "defaultRenderGlobals.imageFilePrefix" - - scene = cmds.file(query=True, sceneName=True) - scene, _ = os.path.splitext(os.path.basename(scene)) - - if renderer == "vray": - renderlayer = renderlayer.split("_")[-1] - # Maya's renderSettings function does not return V-Ray file extension - # so we get the extension from vraySettings - extension = cmds.getAttr("vraySettings.imageFormatStr") - - # When V-Ray image format has not been switched once from default .png - # the getAttr command above returns None. As such we explicitly set - # it to `.png` - if extension is None: - extension = "png" - - if extension in ["exr (multichannel)", "exr (deep)"]: - extension = "exr" - - prefix_attr = "vraySettings.fileNamePrefix" - filename_prefix = cmds.getAttr(prefix_attr) - # we need to determine path for vray as maya `renderSettings` query - # does not work for vray. - - filename_0 = re.sub('', scene, filename_prefix, flags=re.IGNORECASE) # noqa: E501 - filename_0 = re.sub('', renderlayer, filename_0, flags=re.IGNORECASE) # noqa: E501 - filename_0 = "{}.{}.{}".format( - filename_0, "#" * int(padding), extension) - filename_0 = os.path.normpath(os.path.join(root, filename_0)) - elif renderer == "renderman": - prefix_attr = "rmanGlobals.imageFileFormat" - # NOTE: This is guessing extensions from renderman display types. - # Some of them are just framebuffers, d_texture format can be - # set in display setting. We set those now to None, but it - # should be handled more gracefully. - display_types = { - "d_deepexr": "exr", - "d_it": None, - "d_null": None, - "d_openexr": "exr", - "d_png": "png", - "d_pointcloud": "ptc", - "d_targa": "tga", - "d_texture": None, - "d_tiff": "tif" - } - - extension = display_types.get( - cmds.listConnections("rmanDefaultDisplay.displayType")[0], - "exr" - ) or "exr" - - filename_prefix = "{}/{}".format( - cmds.getAttr("rmanGlobals.imageOutputDir"), - cmds.getAttr("rmanGlobals.imageFileFormat") - ) - - renderlayer = renderlayer.split("_")[-1] - - filename_0 = re.sub('', scene, filename_prefix, flags=re.IGNORECASE) # noqa: E501 - filename_0 = re.sub('', renderlayer, filename_0, flags=re.IGNORECASE) # noqa: E501 - filename_0 = re.sub('', "#" * int(padding), filename_0, flags=re.IGNORECASE) # noqa: E501 - filename_0 = re.sub('', extension, filename_0, flags=re.IGNORECASE) # noqa: E501 - filename_0 = os.path.normpath(os.path.join(root, filename_0)) - elif renderer == "redshift": - # mapping redshift extension dropdown values to strings - ext_mapping = ["iff", "exr", "tif", "png", "tga", "jpg"] - extension = ext_mapping[ - cmds.getAttr("redshiftOptions.imageFormat") - ] - else: - # Get the extension, getAttr defaultRenderGlobals.imageFormat - # returns an index number. - filename_base = os.path.basename(filename_0) - extension = os.path.splitext(filename_base)[-1].strip(".") - - filename_prefix = cmds.getAttr(prefix_attr) - return {"ext": extension, - "filename_prefix": filename_prefix, - "padding": padding, - "filename_0": filename_0} - - -class MayaSubmitDeadline(pyblish.api.InstancePlugin): - """Submit available render layers to Deadline. - - Renders are submitted to a Deadline Web Service as - supplied via settings key "DEADLINE_REST_URL". - - Attributes: - use_published (bool): Use published scene to render instead of the - one in work area. - - """ - - label = "Submit to Deadline" - order = pyblish.api.IntegratorOrder + 0.1 - hosts = ["maya"] - families = ["renderlayer"] - targets = ["local"] - - use_published = True - tile_assembler_plugin = "OpenPypeTileAssembler" - asset_dependencies = False - priority = 50 - tile_priority = 50 - limit_groups = [] - jobInfo = {} - pluginInfo = {} - group = "none" - - def process(self, instance): - """Plugin entry point.""" - instance.data["toBeRenderedOn"] = "deadline" - context = instance.context - - self._instance = instance - self.payload_skeleton = copy.deepcopy(payload_skeleton_template) - - # get default deadline webservice url from deadline module - self.deadline_url = instance.context.data.get("defaultDeadline") - # if custom one is set in instance, use that - if instance.data.get("deadlineUrl"): - self.deadline_url = instance.data.get("deadlineUrl") - assert self.deadline_url, "Requires Deadline Webservice URL" - - # just using existing names from Setting - self._job_info = self.jobInfo - - self._plugin_info = self.pluginInfo - - self.limit_groups = self.limit - - context = instance.context - workspace = context.data["workspaceDir"] - anatomy = context.data['anatomy'] - instance.data["toBeRenderedOn"] = "deadline" - - filepath = None - patches = ( - context.data["project_settings"].get( - "deadline", {}).get( - "publish", {}).get( - "MayaSubmitDeadline", {}).get( - "scene_patches", {}) - ) - - # Handle render/export from published scene or not ------------------ - if self.use_published: - patched_files = [] - for i in context: - if "workfile" not in i.data["families"]: - continue - assert i.data["publish"] is True, ( - "Workfile (scene) must be published along") - template_data = i.data.get("anatomyData") - rep = i.data.get("representations")[0].get("name") - template_data["representation"] = rep - template_data["ext"] = rep - template_data["comment"] = None - anatomy_filled = anatomy.format(template_data) - template_filled = anatomy_filled["publish"]["path"] - filepath = os.path.normpath(template_filled) - self.log.info("Using published scene for render {}".format( - filepath)) - - if not os.path.exists(filepath): - self.log.error("published scene does not exist!") - raise - # now we need to switch scene in expected files - # because token will now point to published - # scene file and that might differ from current one - new_scene = os.path.splitext( - os.path.basename(filepath))[0] - orig_scene = os.path.splitext( - os.path.basename(context.data["currentFile"]))[0] - exp = instance.data.get("expectedFiles") - - if isinstance(exp[0], dict): - # we have aovs and we need to iterate over them - new_exp = {} - for aov, files in exp[0].items(): - replaced_files = [] - for f in files: - replaced_files.append( - f.replace(orig_scene, new_scene) - ) - new_exp[aov] = replaced_files - instance.data["expectedFiles"] = [new_exp] - else: - new_exp = [] - for f in exp: - new_exp.append( - f.replace(orig_scene, new_scene) - ) - instance.data["expectedFiles"] = [new_exp] - - if instance.data.get("publishRenderMetadataFolder"): - instance.data["publishRenderMetadataFolder"] = \ - instance.data["publishRenderMetadataFolder"].replace( - orig_scene, new_scene) - self.log.info("Scene name was switched {} -> {}".format( - orig_scene, new_scene - )) - # patch workfile is needed - if filepath not in patched_files: - patched_file = self._patch_workfile(filepath, patches) - patched_files.append(patched_file) - - all_instances = [] - for result in context.data["results"]: - if (result["instance"] is not None and - result["instance"] not in all_instances): # noqa: E128 - all_instances.append(result["instance"]) - - # fallback if nothing was set - if not filepath: - self.log.warning("Falling back to workfile") - filepath = context.data["currentFile"] - - self.log.debug(filepath) - - # Gather needed data ------------------------------------------------ - default_render_file = instance.context.data.get('project_settings')\ - .get('maya')\ - .get('RenderSettings')\ - .get('default_render_image_folder') - filename = os.path.basename(filepath) - comment = context.data.get("comment", "") - dirname = os.path.join(workspace, default_render_file) - renderlayer = instance.data['setMembers'] # rs_beauty - deadline_user = context.data.get("user", getpass.getuser()) - - # Always use the original work file name for the Job name even when - # rendering is done from the published Work File. The original work - # file name is clearer because it can also have subversion strings, - # etc. which are stripped for the published file. - src_filename = os.path.basename(context.data["currentFile"]) - jobname = "%s - %s" % (src_filename, instance.name) - - # Get the variables depending on the renderer - render_variables = get_renderer_variables(renderlayer, dirname) - filename_0 = render_variables["filename_0"] - if self.use_published: - new_scene = os.path.splitext(filename)[0] - orig_scene = os.path.splitext( - os.path.basename(context.data["currentFile"]))[0] - filename_0 = render_variables["filename_0"].replace( - orig_scene, new_scene) - - output_filename_0 = filename_0 - - # this is needed because renderman handles directory and file - # prefixes separately - if self._instance.data["renderer"] == "renderman": - dirname = os.path.dirname(output_filename_0) - - # Create render folder ---------------------------------------------- - try: - # Ensure render folder exists - os.makedirs(dirname) - except OSError: - pass - - # Fill in common data to payload ------------------------------------ - payload_data = {} - payload_data["filename"] = filename - payload_data["filepath"] = filepath - payload_data["jobname"] = jobname - payload_data["deadline_user"] = deadline_user - payload_data["comment"] = comment - payload_data["output_filename_0"] = output_filename_0 - payload_data["render_variables"] = render_variables - payload_data["renderlayer"] = renderlayer - payload_data["workspace"] = workspace - payload_data["dirname"] = dirname - - self.log.info("--- Submission data:") - for k, v in payload_data.items(): - self.log.info("- {}: {}".format(k, v)) - self.log.info("-" * 20) - - frame_pattern = self.payload_skeleton["JobInfo"]["Frames"] - self.payload_skeleton["JobInfo"]["Frames"] = frame_pattern.format( - start=int(self._instance.data["frameStartHandle"]), - end=int(self._instance.data["frameEndHandle"]), - step=int(self._instance.data["byFrameStep"])) - - self.payload_skeleton["JobInfo"]["Plugin"] = self._instance.data.get( - "mayaRenderPlugin", "MayaBatch") - - self.payload_skeleton["JobInfo"]["BatchName"] = src_filename - # Job name, as seen in Monitor - self.payload_skeleton["JobInfo"]["Name"] = jobname - # Arbitrary username, for visualisation in Monitor - self.payload_skeleton["JobInfo"]["UserName"] = deadline_user - # Set job priority - self.payload_skeleton["JobInfo"]["Priority"] = \ - self._instance.data.get("priority", self.priority) - - if self.group != "none" and self.group: - self.payload_skeleton["JobInfo"]["Group"] = self.group - - if self.limit_groups: - self.payload_skeleton["JobInfo"]["LimitGroups"] = \ - ",".join(self.limit_groups) - # Optional, enable double-click to preview rendered - # frames from Deadline Monitor - self.payload_skeleton["JobInfo"]["OutputDirectory0"] = \ - os.path.dirname(output_filename_0).replace("\\", "/") - self.payload_skeleton["JobInfo"]["OutputFilename0"] = \ - output_filename_0.replace("\\", "/") - - self.payload_skeleton["JobInfo"]["Comment"] = comment - self.payload_skeleton["PluginInfo"]["RenderLayer"] = renderlayer - - self.payload_skeleton["PluginInfo"]["RenderSetupIncludeLights"] = instance.data.get("renderSetupIncludeLights") # noqa - # Adding file dependencies. - dependencies = instance.context.data["fileDependencies"] - dependencies.append(filepath) - if self.asset_dependencies: - for dependency in dependencies: - key = "AssetDependency" + str(dependencies.index(dependency)) - self.payload_skeleton["JobInfo"][key] = dependency - - # Handle environments ----------------------------------------------- - # We need those to pass them to pype for it to set correct context - keys = [ - "FTRACK_API_KEY", - "FTRACK_API_USER", - "FTRACK_SERVER", - "OPENPYPE_SG_USER", - "AVALON_PROJECT", - "AVALON_ASSET", - "AVALON_TASK", - "AVALON_APP_NAME", - "OPENPYPE_DEV", - "OPENPYPE_LOG_NO_COLORS", - "OPENPYPE_VERSION" - ] - # Add mongo url if it's enabled - if instance.context.data.get("deadlinePassMongoUrl"): - keys.append("OPENPYPE_MONGO") - - environment = dict({key: os.environ[key] for key in keys - if key in os.environ}, **legacy_io.Session) - environment["OPENPYPE_LOG_NO_COLORS"] = "1" - environment["OPENPYPE_MAYA_VERSION"] = cmds.about(v=True) - # to recognize job from PYPE for turning Event On/Off - environment["OPENPYPE_RENDER_JOB"] = "1" - self.payload_skeleton["JobInfo"].update({ - "EnvironmentKeyValue%d" % index: "{key}={value}".format( - key=key, - value=environment[key] - ) for index, key in enumerate(environment) - }) - # Add options from RenderGlobals------------------------------------- - render_globals = instance.data.get("renderGlobals", {}) - self.payload_skeleton["JobInfo"].update(render_globals) - - # Submit preceding export jobs ------------------------------------- - export_job = None - assert not all(x in instance.data["families"] - for x in ['vrayscene', 'assscene']), ( - "Vray Scene and Ass Scene options are mutually exclusive") - if "vrayscene" in instance.data["families"]: - export_job = self._submit_export(payload_data, "vray") - - if "assscene" in instance.data["families"]: - export_job = self._submit_export(payload_data, "arnold") - - # Prepare main render job ------------------------------------------- - if "vrayscene" in instance.data["families"]: - payload = self._get_vray_render_payload(payload_data) - elif "assscene" in instance.data["families"]: - payload = self._get_arnold_render_payload(payload_data) - else: - payload = self._get_maya_payload(payload_data) - - # Add export job as dependency -------------------------------------- - if export_job: - payload["JobInfo"]["JobDependency0"] = export_job - - # Add list of expected files to job --------------------------------- - exp = instance.data.get("expectedFiles") - exp_index = 0 - output_filenames = {} - - if isinstance(exp[0], dict): - # we have aovs and we need to iterate over them - for _aov, files in exp[0].items(): - col, rem = clique.assemble(files) - if not col and rem: - # we couldn't find any collections but have - # individual files. - assert len(rem) == 1, ("Found multiple non related files " - "to render, don't know what to do " - "with them.") - output_file = rem[0] - if not instance.data.get("tileRendering"): - payload['JobInfo']['OutputFilename' + str(exp_index)] = output_file # noqa: E501 - else: - output_file = col[0].format('{head}{padding}{tail}') - if not instance.data.get("tileRendering"): - payload['JobInfo']['OutputFilename' + str(exp_index)] = output_file # noqa: E501 - - output_filenames['OutputFilename' + str(exp_index)] = output_file # noqa: E501 - exp_index += 1 - else: - col, rem = clique.assemble(exp) - if not col and rem: - # we couldn't find any collections but have - # individual files. - assert len(rem) == 1, ("Found multiple non related files " - "to render, don't know what to do " - "with them.") - - output_file = rem[0] - if not instance.data.get("tileRendering"): - payload['JobInfo']['OutputFilename' + str(exp_index)] = output_file # noqa: E501 - else: - output_file = col[0].format('{head}{padding}{tail}') - if not instance.data.get("tileRendering"): - payload['JobInfo']['OutputFilename' + str(exp_index)] = output_file # noqa: E501 - - output_filenames['OutputFilename' + str(exp_index)] = output_file - - plugin = payload["JobInfo"]["Plugin"] - self.log.info("using render plugin : {}".format(plugin)) - - # Store output dir for unified publisher (filesequence) - instance.data["outputDir"] = os.path.dirname(output_filename_0) - - self.preflight_check(instance) - - # add jobInfo and pluginInfo variables from Settings - payload["JobInfo"].update(self._job_info) - payload["PluginInfo"].update(self._plugin_info) - - # Prepare tiles data ------------------------------------------------ - if instance.data.get("tileRendering"): - # if we have sequence of files, we need to create tile job for - # every frame - - payload["JobInfo"]["TileJob"] = True - payload["JobInfo"]["TileJobTilesInX"] = instance.data.get("tilesX") - payload["JobInfo"]["TileJobTilesInY"] = instance.data.get("tilesY") - payload["PluginInfo"]["ImageHeight"] = instance.data.get("resolutionHeight") # noqa: E501 - payload["PluginInfo"]["ImageWidth"] = instance.data.get("resolutionWidth") # noqa: E501 - payload["PluginInfo"]["RegionRendering"] = True - - assembly_payload = { - "AuxFiles": [], - "JobInfo": { - "BatchName": payload["JobInfo"]["BatchName"], - "Frames": 1, - "Name": "{} - Tile Assembly Job".format( - payload["JobInfo"]["Name"]), - "OutputDirectory0": - payload["JobInfo"]["OutputDirectory0"].replace( - "\\", "/"), - "Plugin": self.tile_assembler_plugin, - "MachineLimit": 1 - }, - "PluginInfo": { - "CleanupTiles": 1, - "ErrorOnMissing": True - } - } - assembly_payload["JobInfo"].update(output_filenames) - assembly_payload["JobInfo"]["Priority"] = self._instance.data.get( - "tile_priority", self.tile_priority) - assembly_payload["JobInfo"]["UserName"] = deadline_user - - frame_payloads = [] - assembly_payloads = [] - - R_FRAME_NUMBER = re.compile(r".+\.(?P[0-9]+)\..+") # noqa: N806, E501 - REPL_FRAME_NUMBER = re.compile(r"(.+\.)([0-9]+)(\..+)") # noqa: N806, E501 - - if isinstance(exp[0], dict): - # we have aovs and we need to iterate over them - # get files from `beauty` - files = exp[0].get("beauty") - # assembly files are used for assembly jobs as we need to put - # together all AOVs - assembly_files = list( - itertools.chain.from_iterable( - [f for _, f in exp[0].items()])) - if not files: - # if beauty doesn't exists, use first aov we found - files = exp[0].get(list(exp[0].keys())[0]) - else: - files = exp - assembly_files = files - - frame_jobs = {} - - file_index = 1 - for file in files: - frame = re.search(R_FRAME_NUMBER, file).group("frame") - new_payload = copy.deepcopy(payload) - new_payload["JobInfo"]["Name"] = \ - "{} (Frame {} - {} tiles)".format( - payload["JobInfo"]["Name"], - frame, - instance.data.get("tilesX") * instance.data.get("tilesY") # noqa: E501 - ) - self.log.info( - "... preparing job {}".format( - new_payload["JobInfo"]["Name"])) - new_payload["JobInfo"]["TileJobFrame"] = frame - - tiles_data = _format_tiles( - file, 0, - instance.data.get("tilesX"), - instance.data.get("tilesY"), - instance.data.get("resolutionWidth"), - instance.data.get("resolutionHeight"), - payload["PluginInfo"]["OutputFilePrefix"] - )[0] - new_payload["JobInfo"].update(tiles_data["JobInfo"]) - new_payload["PluginInfo"].update(tiles_data["PluginInfo"]) - - self.log.info("hashing {} - {}".format(file_index, file)) - job_hash = hashlib.sha256( - ("{}_{}".format(file_index, file)).encode("utf-8")) - frame_jobs[frame] = job_hash.hexdigest() - new_payload["JobInfo"]["ExtraInfo0"] = job_hash.hexdigest() - new_payload["JobInfo"]["ExtraInfo1"] = file - - frame_payloads.append(new_payload) - file_index += 1 - - file_index = 1 - for file in assembly_files: - frame = re.search(R_FRAME_NUMBER, file).group("frame") - new_assembly_payload = copy.deepcopy(assembly_payload) - new_assembly_payload["JobInfo"]["Name"] = \ - "{} (Frame {})".format( - assembly_payload["JobInfo"]["Name"], - frame) - new_assembly_payload["JobInfo"]["OutputFilename0"] = re.sub( - REPL_FRAME_NUMBER, - "\\1{}\\3".format("#" * len(frame)), file) - - new_assembly_payload["PluginInfo"]["Renderer"] = self._instance.data["renderer"] # noqa: E501 - new_assembly_payload["JobInfo"]["ExtraInfo0"] = frame_jobs[frame] # noqa: E501 - new_assembly_payload["JobInfo"]["ExtraInfo1"] = file - assembly_payloads.append(new_assembly_payload) - file_index += 1 - - self.log.info( - "Submitting tile job(s) [{}] ...".format(len(frame_payloads))) - - url = "{}/api/jobs".format(self.deadline_url) - tiles_count = instance.data.get("tilesX") * instance.data.get("tilesY") # noqa: E501 - - for tile_job in frame_payloads: - response = requests_post(url, json=tile_job) - if not response.ok: - raise Exception(response.text) - - job_id = response.json()["_id"] - hash = response.json()["Props"]["Ex0"] - - for assembly_job in assembly_payloads: - if assembly_job["JobInfo"]["ExtraInfo0"] == hash: - assembly_job["JobInfo"]["JobDependency0"] = job_id - - for assembly_job in assembly_payloads: - file = assembly_job["JobInfo"]["ExtraInfo1"] - # write assembly job config files - now = datetime.now() - - config_file = os.path.join( - os.path.dirname(output_filename_0), - "{}_config_{}.txt".format( - os.path.splitext(file)[0], - now.strftime("%Y_%m_%d_%H_%M_%S") - ) - ) - - try: - if not os.path.isdir(os.path.dirname(config_file)): - os.makedirs(os.path.dirname(config_file)) - except OSError: - # directory is not available - self.log.warning( - "Path is unreachable: `{}`".format( - os.path.dirname(config_file))) - - # add config file as job auxFile - assembly_job["AuxFiles"] = [config_file] - - with open(config_file, "w") as cf: - print("TileCount={}".format(tiles_count), file=cf) - print("ImageFileName={}".format(file), file=cf) - print("ImageWidth={}".format( - instance.data.get("resolutionWidth")), file=cf) - print("ImageHeight={}".format( - instance.data.get("resolutionHeight")), file=cf) - - tiles = _format_tiles( - file, 0, - instance.data.get("tilesX"), - instance.data.get("tilesY"), - instance.data.get("resolutionWidth"), - instance.data.get("resolutionHeight"), - payload["PluginInfo"]["OutputFilePrefix"] - )[1] - sorted(tiles) - for k, v in tiles.items(): - print("{}={}".format(k, v), file=cf) - - job_idx = 1 - instance.data["assemblySubmissionJobs"] = [] - for ass_job in assembly_payloads: - self.log.info("submitting assembly job {} of {}".format( - job_idx, len(assembly_payloads) - )) - self.log.debug(json.dumps(ass_job, indent=4, sort_keys=True)) - response = requests_post(url, json=ass_job) - if not response.ok: - raise Exception(response.text) - - instance.data["assemblySubmissionJobs"].append( - response.json()["_id"]) - job_idx += 1 - - instance.data["jobBatchName"] = payload["JobInfo"]["BatchName"] - self.log.info("Setting batch name on instance: {}".format( - instance.data["jobBatchName"])) - else: - # Submit job to farm -------------------------------------------- - self.log.info("Submitting ...") - self.log.debug(json.dumps(payload, indent=4, sort_keys=True)) - - # E.g. http://192.168.0.1:8082/api/jobs - url = "{}/api/jobs".format(self.deadline_url) - response = requests_post(url, json=payload) - if not response.ok: - raise Exception(response.text) - instance.data["deadlineSubmissionJob"] = response.json() - - def _get_maya_payload(self, data): - payload = copy.deepcopy(self.payload_skeleton) - - if not self.asset_dependencies: - job_info_ext = {} - - else: - job_info_ext = { - # Asset dependency to wait for at least the scene file to sync. - "AssetDependency0": data["filepath"], - } - - renderer = self._instance.data["renderer"] - - # This hack is here because of how Deadline handles Renderman version. - # it considers everything with `renderman` set as version older than - # Renderman 22, and so if we are using renderman > 21 we need to set - # renderer string on the job to `renderman22`. We will have to change - # this when Deadline releases new version handling this. - if self._instance.data["renderer"] == "renderman": - try: - from rfm2.config import cfg # noqa - except ImportError: - raise Exception("Cannot determine renderman version") - - rman_version = cfg().build_info.version() # type: str - if int(rman_version.split(".")[0]) > 22: - renderer = "renderman22" - - plugin_info = { - "SceneFile": data["filepath"], - # Output directory and filename - "OutputFilePath": data["dirname"].replace("\\", "/"), - "OutputFilePrefix": data["render_variables"]["filename_prefix"], # noqa: E501 - - # Only render layers are considered renderable in this pipeline - "UsingRenderLayers": True, - - # Render only this layer - "RenderLayer": data["renderlayer"], - - # Determine which renderer to use from the file itself - "Renderer": renderer, - - # Resolve relative references - "ProjectPath": data["workspace"], - } - payload["JobInfo"].update(job_info_ext) - payload["PluginInfo"].update(plugin_info) - return payload - - def _get_vray_export_payload(self, data): - payload = copy.deepcopy(self.payload_skeleton) - vray_settings = cmds.ls(type="VRaySettingsNode") - node = vray_settings[0] - template = cmds.getAttr("{}.vrscene_filename".format(node)) - scene, _ = os.path.splitext(data["filename"]) - first_file = self.format_vray_output_filename(scene, template) - first_file = "{}/{}".format(data["workspace"], first_file) - output = os.path.dirname(first_file) - job_info_ext = { - # Job name, as seen in Monitor - "Name": "Export {} [{}-{}]".format( - data["jobname"], - int(self._instance.data["frameStartHandle"]), - int(self._instance.data["frameEndHandle"])), - - "Plugin": self._instance.data.get( - "mayaRenderPlugin", "MayaPype"), - "FramesPerTask": self._instance.data.get("framesPerTask", 1) - } - - plugin_info_ext = { - # Renderer - "Renderer": "vray", - # Input - "SceneFile": data["filepath"], - "SkipExistingFrames": True, - "UsingRenderLayers": True, - "UseLegacyRenderLayers": True, - "RenderLayer": data["renderlayer"], - "ProjectPath": data["workspace"], - "OutputFilePath": output - } - - payload["JobInfo"].update(job_info_ext) - payload["PluginInfo"].update(plugin_info_ext) - return payload - - def _get_arnold_export_payload(self, data): - - try: - from openpype.scripts import export_maya_ass_job - except Exception: - raise AssertionError( - "Expected module 'export_maya_ass_job' to be available") - - module_path = export_maya_ass_job.__file__ - if module_path.endswith(".pyc"): - module_path = module_path[: -len(".pyc")] + ".py" - - script = os.path.normpath(module_path) - - payload = copy.deepcopy(self.payload_skeleton) - job_info_ext = { - # Job name, as seen in Monitor - "Name": "Export {} [{}-{}]".format( - data["jobname"], - int(self._instance.data["frameStartHandle"]), - int(self._instance.data["frameEndHandle"])), - - "Plugin": "Python", - "FramesPerTask": self._instance.data.get("framesPerTask", 1), - "Frames": 1 - } - - plugin_info_ext = { - "Version": "3.6", - "ScriptFile": script, - "Arguments": "", - "SingleFrameOnly": "True", - } - payload["JobInfo"].update(job_info_ext) - payload["PluginInfo"].update(plugin_info_ext) - - envs = [ - v - for k, v in payload["JobInfo"].items() - if k.startswith("EnvironmentKeyValue") - ] - - # add app name to environment - envs.append( - "AVALON_APP_NAME={}".format(os.environ.get("AVALON_APP_NAME"))) - envs.append( - "OPENPYPE_ASS_EXPORT_RENDER_LAYER={}".format(data["renderlayer"])) - envs.append( - "OPENPYPE_ASS_EXPORT_SCENE_FILE={}".format(data["filepath"])) - envs.append( - "OPENPYPE_ASS_EXPORT_OUTPUT={}".format( - payload['JobInfo']['OutputFilename0'])) - envs.append( - "OPENPYPE_ASS_EXPORT_START={}".format( - int(self._instance.data["frameStartHandle"]))) - envs.append( - "OPENPYPE_ASS_EXPORT_END={}".format( - int(self._instance.data["frameEndHandle"]))) - envs.append( - "OPENPYPE_ASS_EXPORT_STEP={}".format(1)) - - for i, e in enumerate(envs): - payload["JobInfo"]["EnvironmentKeyValue{}".format(i)] = e - return payload - - def _get_vray_render_payload(self, data): - payload = copy.deepcopy(self.payload_skeleton) - vray_settings = cmds.ls(type="VRaySettingsNode") - node = vray_settings[0] - template = cmds.getAttr("{}.vrscene_filename".format(node)) - # "vrayscene//_/" - - scene, _ = os.path.splitext(data["filename"]) - first_file = self.format_vray_output_filename(scene, template) - first_file = "{}/{}".format(data["workspace"], first_file) - job_info_ext = { - "Name": "Render {} [{}-{}]".format( - data["jobname"], - int(self._instance.data["frameStartHandle"]), - int(self._instance.data["frameEndHandle"])), - - "Plugin": "Vray", - "OverrideTaskExtraInfoNames": False, - } - - plugin_info = { - "InputFilename": first_file, - "SeparateFilesPerFrame": True, - "VRayEngine": "V-Ray", - - "Width": self._instance.data["resolutionWidth"], - "Height": self._instance.data["resolutionHeight"], - "OutputFilePath": payload["JobInfo"]["OutputDirectory0"], - "OutputFileName": payload["JobInfo"]["OutputFilename0"] - } - - payload["JobInfo"].update(job_info_ext) - payload["PluginInfo"].update(plugin_info) - return payload - - def _get_arnold_render_payload(self, data): - payload = copy.deepcopy(self.payload_skeleton) - ass_file, _ = os.path.splitext(data["output_filename_0"]) - first_file = ass_file + ".ass" - job_info_ext = { - "Name": "Render {} [{}-{}]".format( - data["jobname"], - int(self._instance.data["frameStartHandle"]), - int(self._instance.data["frameEndHandle"])), - - "Plugin": "Arnold", - "OverrideTaskExtraInfoNames": False, - } - - plugin_info = { - "ArnoldFile": first_file, - } - - payload["JobInfo"].update(job_info_ext) - payload["PluginInfo"].update(plugin_info) - return payload - - def _submit_export(self, data, format): - if format == "vray": - payload = self._get_vray_export_payload(data) - self.log.info("Submitting vrscene export job.") - elif format == "arnold": - payload = self._get_arnold_export_payload(data) - self.log.info("Submitting ass export job.") - - url = "{}/api/jobs".format(self.deadline_url) - response = requests_post(url, json=payload) - if not response.ok: - self.log.error("Submition failed!") - self.log.error(response.status_code) - self.log.error(response.content) - self.log.debug(payload) - raise RuntimeError(response.text) - - dependency = response.json() - return dependency["_id"] - - def preflight_check(self, instance): - """Ensure the startFrame, endFrame and byFrameStep are integers.""" - for key in ("frameStartHandle", "frameEndHandle", "byFrameStep"): - value = instance.data[key] - - if int(value) == value: - continue - - self.log.warning( - "%f=%d was rounded off to nearest integer" - % (value, int(value)) - ) - - def format_vray_output_filename(self, filename, template, dir=False): - """Format the expected output file of the Export job. - - Example: - /_/ - "shot010_v006/shot010_v006_CHARS/CHARS" - - Args: - instance: - filename(str): - dir(bool): - - Returns: - str - - """ - def smart_replace(string, key_values): - new_string = string - for key, value in key_values.items(): - new_string = new_string.replace(key, value) - return new_string - - # Ensure filename has no extension - file_name, _ = os.path.splitext(filename) - - layer = self._instance.data['setMembers'] - - # Reformat without tokens - output_path = smart_replace( - template, - {"": file_name, - "": layer}) - - if dir: - return output_path.replace("\\", "/") - - start_frame = int(self._instance.data["frameStartHandle"]) - filename_zero = "{}_{:04d}.vrscene".format(output_path, start_frame) - - result = filename_zero.replace("\\", "/") - - return result - - def _patch_workfile(self, file, patches): - # type: (str, dict) -> [str, None] - """Patch Maya scene. - - This will take list of patches (lines to add) and apply them to - *published* Maya scene file (that is used later for rendering). - - Patches are dict with following structure:: - { - "name": "Name of patch", - "regex": "regex of line before patch", - "line": "line to insert" - } - - Args: - file (str): File to patch. - patches (dict): Dictionary defining patches. - - Returns: - str: Patched file path or None - - """ - if os.path.splitext(file)[1].lower() != ".ma" or not patches: - return None - - compiled_regex = [re.compile(p["regex"]) for p in patches] - with open(file, "r+") as pf: - scene_data = pf.readlines() - for ln, line in enumerate(scene_data): - for i, r in enumerate(compiled_regex): - if re.match(r, line): - scene_data.insert(ln + 1, patches[i]["line"]) - pf.seek(0) - pf.writelines(scene_data) - pf.truncate() - self.log.info( - "Applied {} patch to scene.".format( - patches[i]["name"])) - return file diff --git a/openpype/modules/deadline/plugins/publish/submit_publish_job.py b/openpype/modules/deadline/plugins/publish/submit_publish_job.py index 2647dcf0cb..aba505b3c6 100644 --- a/openpype/modules/deadline/plugins/publish/submit_publish_job.py +++ b/openpype/modules/deadline/plugins/publish/submit_publish_job.py @@ -700,9 +700,6 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin): self.context = context self.anatomy = instance.context.data["anatomy"] - if hasattr(instance, "_log"): - data['_log'] = instance._log - asset = data.get("asset") or legacy_io.Session["AVALON_ASSET"] subset = data.get("subset") @@ -781,7 +778,9 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin): "resolutionHeight": data.get("resolutionHeight", 1080), "multipartExr": data.get("multipartExr", False), "jobBatchName": data.get("jobBatchName", ""), - "useSequenceForReview": data.get("useSequenceForReview", True) + "useSequenceForReview": data.get("useSequenceForReview", True), + # map inputVersions `ObjectId` -> `str` so json supports it + "inputVersions": list(map(str, data.get("inputVersions", []))) } # skip locking version if we are creating v01 diff --git a/openpype/modules/deadline/repository/custom/plugins/OpenPypeTileAssembler/OpenPypeTileAssembler.py b/openpype/modules/deadline/repository/custom/plugins/OpenPypeTileAssembler/OpenPypeTileAssembler.py index 9fca1b5391..625a3f1a28 100644 --- a/openpype/modules/deadline/repository/custom/plugins/OpenPypeTileAssembler/OpenPypeTileAssembler.py +++ b/openpype/modules/deadline/repository/custom/plugins/OpenPypeTileAssembler/OpenPypeTileAssembler.py @@ -56,7 +56,7 @@ def convert_value_by_type_name(value_type, value): return float(value) # Vectors will probably have more types - if value_type == "vec2f": + if value_type in ("vec2f", "float2"): return [float(item) for item in value.split(",")] # Matrix should be always have square size of element 3x3, 4x4 @@ -71,7 +71,7 @@ def convert_value_by_type_name(value_type, value): elif parts_len == 4: divisor = 2 elif parts_len == 9: - divisor == 3 + divisor = 3 elif parts_len == 16: divisor = 4 else: @@ -127,7 +127,7 @@ def convert_value_by_type_name(value_type, value): return output print(( - "MISSING IMPLEMENTATION:" + "Dev note (missing implementation):" " Unknown attrib type \"{}\". Value: {}" ).format(value_type, value)) return value @@ -183,7 +183,7 @@ def parse_oiio_xml_output(xml_string): else: value = child.text print(( - "MISSING IMPLEMENTATION:" + "Dev note (missing implementation):" " Unknown tag \"{}\". Value \"{}\"" ).format(tag_name, value)) @@ -453,7 +453,7 @@ class OpenPypeTileAssembler(DeadlinePlugin): # Swap to have input as foreground args.append("--swap") # Paste foreground to background - args.append("--paste +{}+{}".format(pos_x, pos_y)) + args.append("--paste {x:+d}{y:+d}".format(x=pos_x, y=pos_y)) args.append("-o") args.append(output_path) diff --git a/openpype/modules/ftrack/__init__.py b/openpype/modules/ftrack/__init__.py index 7261254c6f..e520f08337 100644 --- a/openpype/modules/ftrack/__init__.py +++ b/openpype/modules/ftrack/__init__.py @@ -1,9 +1,13 @@ from .ftrack_module import ( FtrackModule, - FTRACK_MODULE_DIR + FTRACK_MODULE_DIR, + + resolve_ftrack_url, ) __all__ = ( "FtrackModule", - "FTRACK_MODULE_DIR" + "FTRACK_MODULE_DIR", + + "resolve_ftrack_url", ) diff --git a/openpype/modules/ftrack/event_handlers_user/action_create_project_structure.py b/openpype/modules/ftrack/event_handlers_user/action_create_project_structure.py index df914de854..7c896570b1 100644 --- a/openpype/modules/ftrack/event_handlers_user/action_create_project_structure.py +++ b/openpype/modules/ftrack/event_handlers_user/action_create_project_structure.py @@ -1,7 +1,10 @@ import re +from openpype.pipeline.project_folders import ( + get_project_basic_paths, + create_project_folders, +) from openpype_modules.ftrack.lib import BaseAction, statics_icon -from openpype.api import get_project_basic_paths, create_project_folders class CreateProjectFolders(BaseAction): @@ -81,7 +84,7 @@ class CreateProjectFolders(BaseAction): } # Invoking OpenPype API to create the project folders - create_project_folders(basic_paths, project_name) + create_project_folders(project_name, basic_paths) self.create_ftrack_entities(basic_paths, project_entity) self.trigger_event( diff --git a/openpype/modules/ftrack/ftrack_module.py b/openpype/modules/ftrack/ftrack_module.py index cb4f204523..75ffd7f864 100644 --- a/openpype/modules/ftrack/ftrack_module.py +++ b/openpype/modules/ftrack/ftrack_module.py @@ -6,14 +6,16 @@ import platform import click from openpype.modules import OpenPypeModule -from openpype_interfaces import ( +from openpype.modules.interfaces import ( ITrayModule, IPluginPaths, ISettingsChangeListener ) from openpype.settings import SaveWarningExc +from openpype.lib import Logger FTRACK_MODULE_DIR = os.path.dirname(os.path.abspath(__file__)) +_URL_NOT_SET = object() class FtrackModule( @@ -28,17 +30,8 @@ class FtrackModule( ftrack_settings = settings[self.name] self.enabled = ftrack_settings["enabled"] - # Add http schema - ftrack_url = ftrack_settings["ftrack_server"].strip("/ ") - if ftrack_url: - if "http" not in ftrack_url: - ftrack_url = "https://" + ftrack_url - - # Check if "ftrack.app" is part os url - if "ftrackapp.com" not in ftrack_url: - ftrack_url = ftrack_url + ".ftrackapp.com" - - self.ftrack_url = ftrack_url + self._settings_ftrack_url = ftrack_settings["ftrack_server"] + self._ftrack_url = _URL_NOT_SET current_dir = os.path.dirname(os.path.abspath(__file__)) low_platform = platform.system().lower() @@ -70,6 +63,16 @@ class FtrackModule( self.timers_manager_connector = None self._timers_manager_module = None + def get_ftrack_url(self): + if self._ftrack_url is _URL_NOT_SET: + self._ftrack_url = resolve_ftrack_url( + self._settings_ftrack_url, + logger=self.log + ) + return self._ftrack_url + + ftrack_url = property(get_ftrack_url) + def get_global_environments(self): """Ftrack's global environments.""" return { @@ -479,6 +482,51 @@ class FtrackModule( click_group.add_command(cli_main) +def _check_ftrack_url(url): + import requests + + try: + result = requests.get(url, allow_redirects=False) + except requests.exceptions.RequestException: + return False + + if (result.status_code != 200 or "FTRACK_VERSION" not in result.headers): + return False + return True + + +def resolve_ftrack_url(url, logger=None): + """Checks if Ftrack server is responding.""" + + if logger is None: + logger = Logger.get_logger(__name__) + + url = url.strip("/ ") + if not url: + logger.error("Ftrack URL is not set!") + return None + + if not url.startswith("http"): + url = "https://" + url + + ftrack_url = None + if not url.endswith("ftrackapp.com"): + ftrackapp_url = url + ".ftrackapp.com" + if _check_ftrack_url(ftrackapp_url): + ftrack_url = ftrackapp_url + + if not ftrack_url and _check_ftrack_url(url): + ftrack_url = url + + if ftrack_url: + logger.debug("Ftrack server \"{}\" is accessible.".format(ftrack_url)) + + else: + logger.error("Ftrack server \"{}\" is not accessible!".format(url)) + + return ftrack_url + + @click.group(FtrackModule.name, help="Ftrack module related commands.") def cli_main(): pass diff --git a/openpype/modules/ftrack/ftrack_server/__init__.py b/openpype/modules/ftrack/ftrack_server/__init__.py index 9e3920b500..8e5f7c4c51 100644 --- a/openpype/modules/ftrack/ftrack_server/__init__.py +++ b/openpype/modules/ftrack/ftrack_server/__init__.py @@ -1,8 +1,6 @@ from .ftrack_server import FtrackServer -from .lib import check_ftrack_url __all__ = ( "FtrackServer", - "check_ftrack_url" ) diff --git a/openpype/modules/ftrack/ftrack_server/event_server_cli.py b/openpype/modules/ftrack/ftrack_server/event_server_cli.py index 3ef7c8270a..20c5ab24a8 100644 --- a/openpype/modules/ftrack/ftrack_server/event_server_cli.py +++ b/openpype/modules/ftrack/ftrack_server/event_server_cli.py @@ -20,9 +20,11 @@ from openpype.lib import ( get_openpype_version, get_build_version, ) -from openpype_modules.ftrack import FTRACK_MODULE_DIR +from openpype_modules.ftrack import ( + FTRACK_MODULE_DIR, + resolve_ftrack_url, +) from openpype_modules.ftrack.lib import credentials -from openpype_modules.ftrack.ftrack_server.lib import check_ftrack_url from openpype_modules.ftrack.ftrack_server import socket_thread @@ -114,7 +116,7 @@ def legacy_server(ftrack_url): while True: if not ftrack_accessible: - ftrack_accessible = check_ftrack_url(ftrack_url) + ftrack_accessible = resolve_ftrack_url(ftrack_url) # Run threads only if Ftrack is accessible if not ftrack_accessible and not printed_ftrack_error: @@ -257,7 +259,7 @@ def main_loop(ftrack_url): while True: # Check if accessible Ftrack and Mongo url if not ftrack_accessible: - ftrack_accessible = check_ftrack_url(ftrack_url) + ftrack_accessible = resolve_ftrack_url(ftrack_url) if not mongo_accessible: mongo_accessible = check_mongo_url(mongo_uri) @@ -441,7 +443,7 @@ def run_event_server( os.environ["CLOCKIFY_API_KEY"] = clockify_api_key # Check url regex and accessibility - ftrack_url = check_ftrack_url(ftrack_url) + ftrack_url = resolve_ftrack_url(ftrack_url) if not ftrack_url: print('Exiting! < Please enter Ftrack server url >') return 1 diff --git a/openpype/modules/ftrack/ftrack_server/lib.py b/openpype/modules/ftrack/ftrack_server/lib.py index 947dacf917..c8143f739c 100644 --- a/openpype/modules/ftrack/ftrack_server/lib.py +++ b/openpype/modules/ftrack/ftrack_server/lib.py @@ -26,45 +26,12 @@ except ImportError: from openpype_modules.ftrack.lib import get_ftrack_event_mongo_info from openpype.client import OpenPypeMongoConnection -from openpype.api import Logger +from openpype.lib import Logger TOPIC_STATUS_SERVER = "openpype.event.server.status" TOPIC_STATUS_SERVER_RESULT = "openpype.event.server.status.result" -def check_ftrack_url(url, log_errors=True, logger=None): - """Checks if Ftrack server is responding""" - if logger is None: - logger = Logger.get_logger(__name__) - - if not url: - logger.error("Ftrack URL is not set!") - return None - - url = url.strip('/ ') - - if 'http' not in url: - if url.endswith('ftrackapp.com'): - url = 'https://' + url - else: - url = 'https://{0}.ftrackapp.com'.format(url) - try: - result = requests.get(url, allow_redirects=False) - except requests.exceptions.RequestException: - if log_errors: - logger.error("Entered Ftrack URL is not accesible!") - return False - - if (result.status_code != 200 or 'FTRACK_VERSION' not in result.headers): - if log_errors: - logger.error("Entered Ftrack URL is not accesible!") - return False - - logger.debug("Ftrack server {} is accessible.".format(url)) - - return url - - class SocketBaseEventHub(ftrack_api.event.hub.EventHub): hearbeat_msg = b"hearbeat" diff --git a/openpype/modules/ftrack/lib/avalon_sync.py b/openpype/modules/ftrack/lib/avalon_sync.py index 72be6a8e9a..935d1e85c9 100644 --- a/openpype/modules/ftrack/lib/avalon_sync.py +++ b/openpype/modules/ftrack/lib/avalon_sync.py @@ -19,11 +19,8 @@ from openpype.client.operations import ( CURRENT_PROJECT_SCHEMA, CURRENT_PROJECT_CONFIG_SCHEMA, ) -from openpype.api import ( - Logger, - get_anatomy_settings -) -from openpype.lib import ApplicationManager +from openpype.settings import get_anatomy_settings +from openpype.lib import ApplicationManager, Logger from openpype.pipeline import AvalonMongoDB, schema from .constants import CUST_ATTR_ID_KEY, FPS_KEYS diff --git a/openpype/modules/ftrack/plugins/publish/collect_username.py b/openpype/modules/ftrack/plugins/publish/collect_username.py index a9b746ea51..798f3960a8 100644 --- a/openpype/modules/ftrack/plugins/publish/collect_username.py +++ b/openpype/modules/ftrack/plugins/publish/collect_username.py @@ -1,5 +1,8 @@ """Loads publishing context from json and continues in publish process. +Should run before 'CollectAnatomyContextData' so the user on context is +changed before it's stored to context anatomy data or instance anatomy data. + Requires: anatomy -> context["anatomy"] *(pyblish.api.CollectorOrder - 0.11) @@ -13,7 +16,7 @@ import os import pyblish.api -class CollectUsername(pyblish.api.ContextPlugin): +class CollectUsernameForWebpublish(pyblish.api.ContextPlugin): """ Translates user email to Ftrack username. @@ -32,10 +35,8 @@ class CollectUsername(pyblish.api.ContextPlugin): hosts = ["webpublisher", "photoshop"] targets = ["remotepublish", "filespublish", "tvpaint_worker"] - _context = None - def process(self, context): - self.log.info("CollectUsername") + self.log.info("{}".format(self.__class__.__name__)) os.environ["FTRACK_API_USER"] = os.environ["FTRACK_BOT_API_USER"] os.environ["FTRACK_API_KEY"] = os.environ["FTRACK_BOT_API_KEY"] @@ -54,12 +55,14 @@ class CollectUsername(pyblish.api.ContextPlugin): return session = ftrack_api.Session(auto_connect_event_hub=False) - user = session.query("User where email like '{}'".format(user_email)) + user = session.query( + "User where email like '{}'".format(user_email) + ).first() if not user: raise ValueError( "Couldn't find user with {} email".format(user_email)) - user = user[0] + username = user.get("username") self.log.debug("Resolved ftrack username:: {}".format(username)) os.environ["FTRACK_API_USER"] = username @@ -67,5 +70,4 @@ class CollectUsername(pyblish.api.ContextPlugin): burnin_name = username if '@' in burnin_name: burnin_name = burnin_name[:burnin_name.index('@')] - os.environ["WEBPUBLISH_OPENPYPE_USERNAME"] = burnin_name context.data["user"] = burnin_name diff --git a/openpype/modules/ftrack/plugins/publish/integrate_ftrack_instances.py b/openpype/modules/ftrack/plugins/publish/integrate_ftrack_instances.py index 6024781d87..5ff75e7060 100644 --- a/openpype/modules/ftrack/plugins/publish/integrate_ftrack_instances.py +++ b/openpype/modules/ftrack/plugins/publish/integrate_ftrack_instances.py @@ -35,7 +35,7 @@ class IntegrateFtrackInstance(pyblish.api.InstancePlugin): family_mapping = { "camera": "cam", "look": "look", - "mayaascii": "scene", + "mayaAscii": "scene", "model": "geo", "rig": "rig", "setdress": "setdress", @@ -74,11 +74,15 @@ class IntegrateFtrackInstance(pyblish.api.InstancePlugin): version_number = int(instance_version) family = instance.data["family"] - family_low = family.lower() + # Perform case-insensitive family mapping + family_low = family.lower() asset_type = instance.data.get("ftrackFamily") - if not asset_type and family_low in self.family_mapping: - asset_type = self.family_mapping[family_low] + if not asset_type: + for map_family, map_value in self.family_mapping.items(): + if map_family.lower() == family_low: + asset_type = map_value + break if not asset_type: asset_type = "upload" @@ -86,15 +90,6 @@ class IntegrateFtrackInstance(pyblish.api.InstancePlugin): self.log.debug( "Family: {}\nMapping: {}".format(family_low, self.family_mapping) ) - - # Ignore this instance if neither "ftrackFamily" or a family mapping is - # found. - if not asset_type: - self.log.info(( - "Family \"{}\" does not match any asset type mapping" - ).format(family)) - return - status_name = self._get_asset_version_status_name(instance) # Base of component item data diff --git a/openpype/modules/ftrack/tray/ftrack_tray.py b/openpype/modules/ftrack/tray/ftrack_tray.py index 501d837a4c..e3c6e30ead 100644 --- a/openpype/modules/ftrack/tray/ftrack_tray.py +++ b/openpype/modules/ftrack/tray/ftrack_tray.py @@ -6,22 +6,18 @@ import threading from Qt import QtCore, QtWidgets, QtGui import ftrack_api -from ..ftrack_server.lib import check_ftrack_url -from ..ftrack_server import socket_thread -from ..lib import credentials -from ..ftrack_module import FTRACK_MODULE_DIR -from . import login_dialog - from openpype import resources from openpype.lib import Logger - - -log = Logger.get_logger("FtrackModule") +from openpype_modules.ftrack import resolve_ftrack_url, FTRACK_MODULE_DIR +from openpype_modules.ftrack.ftrack_server import socket_thread +from openpype_modules.ftrack.lib import credentials +from . import login_dialog class FtrackTrayWrapper: def __init__(self, module): self.module = module + self.log = Logger.get_logger(self.__class__.__name__) self.thread_action_server = None self.thread_socket_server = None @@ -62,19 +58,19 @@ class FtrackTrayWrapper: if validation: self.widget_login.set_credentials(ft_user, ft_api_key) self.module.set_credentials_to_env(ft_user, ft_api_key) - log.info("Connected to Ftrack successfully") + self.log.info("Connected to Ftrack successfully") self.on_login_change() return validation if not validation and ft_user and ft_api_key: - log.warning( + self.log.warning( "Current Ftrack credentials are not valid. {}: {} - {}".format( str(os.environ.get("FTRACK_SERVER")), ft_user, ft_api_key ) ) - log.info("Please sign in to Ftrack") + self.log.info("Please sign in to Ftrack") self.bool_logged = False self.show_login_widget() self.set_menu_visibility() @@ -104,7 +100,7 @@ class FtrackTrayWrapper: self.action_credentials.setIcon(self.icon_not_logged) self.action_credentials.setToolTip("Logged out") - log.info("Logged out of Ftrack") + self.log.info("Logged out of Ftrack") self.bool_logged = False self.set_menu_visibility() @@ -126,10 +122,6 @@ class FtrackTrayWrapper: ftrack_url = self.module.ftrack_url os.environ["FTRACK_SERVER"] = ftrack_url - parent_file_path = os.path.dirname( - os.path.dirname(os.path.realpath(__file__)) - ) - min_fail_seconds = 5 max_fail_count = 3 wait_time_after_max_fail = 10 @@ -154,17 +146,19 @@ class FtrackTrayWrapper: # Main loop while True: if not self.bool_action_server_running: - log.debug("Action server was pushed to stop.") + self.log.debug("Action server was pushed to stop.") break # Check if accessible Ftrack and Mongo url if not ftrack_accessible: - ftrack_accessible = check_ftrack_url(ftrack_url) + ftrack_accessible = resolve_ftrack_url(ftrack_url) # Run threads only if Ftrack is accessible if not ftrack_accessible: if not printed_ftrack_error: - log.warning("Can't access Ftrack {}".format(ftrack_url)) + self.log.warning( + "Can't access Ftrack {}".format(ftrack_url) + ) if self.thread_socket_server is not None: self.thread_socket_server.stop() @@ -191,7 +185,7 @@ class FtrackTrayWrapper: self.set_menu_visibility() elif failed_count == max_fail_count: - log.warning(( + self.log.warning(( "Action server failed {} times." " I'll try to run again {}s later" ).format( @@ -243,10 +237,10 @@ class FtrackTrayWrapper: self.thread_action_server.join() self.thread_action_server = None - log.info("Ftrack action server was forced to stop") + self.log.info("Ftrack action server was forced to stop") except Exception: - log.warning( + self.log.warning( "Error has happened during Killing action server", exc_info=True ) @@ -343,7 +337,7 @@ class FtrackTrayWrapper: self.thread_timer = None except Exception as e: - log.error("During Killing Timer event server: {0}".format(e)) + self.log.error("During Killing Timer event server: {0}".format(e)) def changed_user(self): self.stop_action_server() diff --git a/openpype/modules/kitsu/utils/credentials.py b/openpype/modules/kitsu/utils/credentials.py index 0529380d6d..adcfb07cd5 100644 --- a/openpype/modules/kitsu/utils/credentials.py +++ b/openpype/modules/kitsu/utils/credentials.py @@ -5,6 +5,7 @@ from typing import Tuple import gazu from openpype.lib.local_settings import OpenPypeSecureRegistry +from openpype.lib import emit_event def validate_credentials( @@ -32,6 +33,8 @@ def validate_credentials( except gazu.exception.AuthFailedException: return False + emit_event("kitsu.user.logged", data={"username": login}, source="kitsu") + return True diff --git a/openpype/modules/kitsu/utils/update_op_with_zou.py b/openpype/modules/kitsu/utils/update_op_with_zou.py index 8d65591c0b..4a064f6a16 100644 --- a/openpype/modules/kitsu/utils/update_op_with_zou.py +++ b/openpype/modules/kitsu/utils/update_op_with_zou.py @@ -166,50 +166,21 @@ def update_op_assets( # Substitute item type for general classification (assets or shots) if item_type in ["Asset", "AssetType"]: - substitute_item_type = "assets" + entity_root_asset_name = "Assets" elif item_type in ["Episode", "Sequence"]: - substitute_item_type = "shots" - else: - substitute_item_type = f"{item_type.lower()}s" - entity_parent_folders = [ - f - for f in project_module_settings["entities_root"] - .get(substitute_item_type) - .split("/") - if f - ] + entity_root_asset_name = "Shots" # Root parent folder if exist visual_parent_doc_id = ( asset_doc_ids[parent_zou_id]["_id"] if parent_zou_id else None ) if visual_parent_doc_id is None: - # Find root folder docs - root_folder_docs = get_assets( + # Find root folder doc ("Assets" or "Shots") + root_folder_doc = get_asset_by_name( project_name, - asset_names=[entity_parent_folders[-1]], + asset_name=entity_root_asset_name, fields=["_id", "data.root_of"], ) - # NOTE: Not sure why it's checking for entity type? - # OP3 does not support multiple assets with same names so type - # filtering is irelevant. - # This way mimics previous implementation: - # ``` - # root_folder_doc = dbcon.find_one( - # { - # "type": "asset", - # "name": entity_parent_folders[-1], - # "data.root_of": substitute_item_type, - # }, - # ["_id"], - # ) - # ``` - root_folder_doc = None - for folder_doc in root_folder_docs: - root_of = folder_doc.get("data", {}).get("root_of") - if root_of == substitute_item_type: - root_folder_doc = folder_doc - break if root_folder_doc: visual_parent_doc_id = root_folder_doc["_id"] @@ -240,7 +211,7 @@ def update_op_assets( item_name = item["name"] # Set root folders parents - item_data["parents"] = entity_parent_folders + item_data["parents"] + item_data["parents"] = [entity_root_asset_name] + item_data["parents"] # Update 'data' different in zou DB updated_data = { @@ -318,13 +289,13 @@ def write_project_to_op(project: dict, dbcon: AvalonMongoDB) -> UpdateOne: ) -def sync_all_projects(login: str, password: str): +def sync_all_projects(login: str, password: str, ignore_projects: list = None): """Update all OP projects in DB with Zou data. Args: login (str): Kitsu user login password (str): Kitsu user password - + ignore_projects (list): List of unsynced project names Raises: gazu.exception.AuthFailedException: Wrong user login and/or password """ @@ -340,6 +311,8 @@ def sync_all_projects(login: str, password: str): dbcon.install() all_projects = gazu.project.all_open_projects() for project in all_projects: + if ignore_projects and project["name"] in ignore_projects: + continue sync_project_from_kitsu(dbcon, project) @@ -396,54 +369,30 @@ def sync_project_from_kitsu(dbcon: AvalonMongoDB, project: dict): zou_ids_and_asset_docs[project["id"]] = project_doc # Create entities root folders - project_module_settings = get_project_settings(project_name)["kitsu"] - for entity_type, root in project_module_settings["entities_root"].items(): - parent_folders = root.split("/") - direct_parent_doc = None - for i, folder in enumerate(parent_folders, 1): - parent_doc = get_asset_by_name( - project_name, folder, fields=["_id", "data.root_of"] - ) - # NOTE: Not sure why it's checking for entity type? - # OP3 does not support multiple assets with same names so type - # filtering is irelevant. - # Also all of the entities could find be queried at once using - # 'get_assets'. - # This way mimics previous implementation: - # ``` - # parent_doc = dbcon.find_one( - # {"type": "asset", "name": folder, "data.root_of": entity_type} - # ) - # ``` - if ( - parent_doc - and parent_doc.get("data", {}).get("root_of") != entity_type - ): - parent_doc = None - - if not parent_doc: - direct_parent_doc = dbcon.insert_one( - { - "name": folder, - "type": "asset", - "schema": "openpype:asset-3.0", - "data": { - "root_of": entity_type, - "parents": parent_folders[:i], - "visualParent": direct_parent_doc.inserted_id - if direct_parent_doc - else None, - "tasks": {}, - }, - } - ) + to_insert = [ + { + "name": r, + "type": "asset", + "schema": "openpype:asset-3.0", + "data": { + "root_of": r, + "tasks": {}, + }, + } + for r in ["Assets", "Shots"] + if not get_asset_by_name( + project_name, r, fields=["_id", "data.root_of"] + ) + ] # Create - to_insert = [ - create_op_asset(item) - for item in all_entities - if item["id"] not in zou_ids_and_asset_docs.keys() - ] + to_insert.extend( + [ + create_op_asset(item) + for item in all_entities + if item["id"] not in zou_ids_and_asset_docs.keys() + ] + ) if to_insert: # Insert doc in DB dbcon.insert_many(to_insert) diff --git a/openpype/modules/slack/plugins/publish/integrate_slack_api.py b/openpype/modules/slack/plugins/publish/integrate_slack_api.py index c3b288f0cd..643e55915b 100644 --- a/openpype/modules/slack/plugins/publish/integrate_slack_api.py +++ b/openpype/modules/slack/plugins/publish/integrate_slack_api.py @@ -95,13 +95,15 @@ class IntegrateSlackAPI(pyblish.api.InstancePlugin): Reviews might be large, so allow only adding link to message instead of uploading only. """ + fill_data = copy.deepcopy(instance.context.data["anatomyData"]) + username = fill_data.get("user") fill_pairs = [ ("asset", instance.data.get("asset", fill_data.get("asset"))), ("subset", instance.data.get("subset", fill_data.get("subset"))), - ("username", instance.data.get("username", - fill_data.get("username"))), + ("user", username), + ("username", username), ("app", instance.data.get("app", fill_data.get("app"))), ("family", instance.data.get("family", fill_data.get("family"))), ("version", str(instance.data.get("version", @@ -110,13 +112,19 @@ class IntegrateSlackAPI(pyblish.api.InstancePlugin): if review_path: fill_pairs.append(("review_filepath", review_path)) - task_data = instance.data.get("task") - if not task_data: - task_data = fill_data.get("task") - for key, value in task_data.items(): - fill_key = "task[{}]".format(key) - fill_pairs.append((fill_key, value)) - fill_pairs.append(("task", task_data["name"])) + task_data = fill_data.get("task") + if task_data: + if ( + "{task}" in message_templ + or "{Task}" in message_templ + or "{TASK}" in message_templ + ): + fill_pairs.append(("task", task_data["name"])) + + else: + for key, value in task_data.items(): + fill_key = "task[{}]".format(key) + fill_pairs.append((fill_key, value)) self.log.debug("fill_pairs ::{}".format(fill_pairs)) multiple_case_variants = prepare_template_data(fill_pairs) diff --git a/openpype/pipeline/create/context.py b/openpype/pipeline/create/context.py index eaaed39357..a7e43cb2f2 100644 --- a/openpype/pipeline/create/context.py +++ b/openpype/pipeline/create/context.py @@ -7,7 +7,11 @@ from uuid import uuid4 from contextlib import contextmanager from openpype.client import get_assets -from openpype.host import INewPublisher +from openpype.settings import ( + get_system_settings, + get_project_settings +) +from openpype.host import IPublishHost from openpype.pipeline import legacy_io from openpype.pipeline.mongodb import ( AvalonMongoDB, @@ -20,11 +24,6 @@ from .creator_plugins import ( discover_creator_plugins, ) -from openpype.api import ( - get_system_settings, - get_project_settings -) - UpdateData = collections.namedtuple("UpdateData", ["instance", "changes"]) @@ -402,8 +401,12 @@ class CreatedInstance: self.creator = creator # Instance members may have actions on them + # TODO implement members logic self._members = [] + # Data that can be used for lifetime of object + self._transient_data = {} + # Create a copy of passed data to avoid changing them on the fly data = copy.deepcopy(data or {}) # Store original value of passed data @@ -596,6 +599,26 @@ class CreatedInstance: return self + @property + def transient_data(self): + """Data stored for lifetime of instance object. + + These data are not stored to scene and will be lost on object + deletion. + + Can be used to store objects. In some host implementations is not + possible to reference to object in scene with some unique identifier + (e.g. node in Fusion.). In that case it is handy to store the object + here. Should be used that way only if instance data are stored on the + node itself. + + Returns: + Dict[str, Any]: Dictionary object where you can store data related + to instance for lifetime of instance object. + """ + + return self._transient_data + def changes(self): """Calculate and return changes.""" @@ -771,7 +794,7 @@ class CreateContext: """ missing = set( - INewPublisher.get_missing_publish_methods(host) + IPublishHost.get_missing_publish_methods(host) ) return missing diff --git a/openpype/pipeline/create/creator_plugins.py b/openpype/pipeline/create/creator_plugins.py index bf2fdd2c5f..5b0532c60a 100644 --- a/openpype/pipeline/create/creator_plugins.py +++ b/openpype/pipeline/create/creator_plugins.py @@ -81,6 +81,13 @@ class BaseCreator: # - we may use UI inside processing this attribute should be checked self.headless = headless + self.apply_settings(project_settings, system_settings) + + def apply_settings(self, project_settings, system_settings): + """Method called on initialization of plugin to apply settings.""" + + pass + @property def identifier(self): """Identifier of creator (must be unique). diff --git a/openpype/pipeline/load/__init__.py b/openpype/pipeline/load/__init__.py index bf38a0b3c8..e96f64f2a4 100644 --- a/openpype/pipeline/load/__init__.py +++ b/openpype/pipeline/load/__init__.py @@ -5,6 +5,7 @@ from .utils import ( InvalidRepresentationContext, get_repres_contexts, + get_contexts_for_repre_docs, get_subset_contexts, get_representation_context, @@ -54,6 +55,7 @@ __all__ = ( "InvalidRepresentationContext", "get_repres_contexts", + "get_contexts_for_repre_docs", "get_subset_contexts", "get_representation_context", diff --git a/openpype/pipeline/load/utils.py b/openpype/pipeline/load/utils.py index 83b904e4a7..784d4628f3 100644 --- a/openpype/pipeline/load/utils.py +++ b/openpype/pipeline/load/utils.py @@ -37,7 +37,7 @@ log = logging.getLogger(__name__) ContainersFilterResult = collections.namedtuple( "ContainersFilterResult", - ["latest", "outdated", "not_foud", "invalid"] + ["latest", "outdated", "not_found", "invalid"] ) @@ -87,13 +87,20 @@ def get_repres_contexts(representation_ids, dbcon=None): if not dbcon: dbcon = legacy_io - contexts = {} if not representation_ids: - return contexts + return {} project_name = dbcon.active_project() repre_docs = get_representations(project_name, representation_ids) + return get_contexts_for_repre_docs(project_name, repre_docs) + + +def get_contexts_for_repre_docs(project_name, repre_docs): + contexts = {} + if not repre_docs: + return contexts + repre_docs_by_id = {} version_ids = set() for repre_doc in repre_docs: @@ -808,7 +815,7 @@ def filter_containers(containers, project_name): Categories are 'latest', 'outdated', 'invalid' and 'not_found'. The 'lastest' containers are from last version, 'outdated' are not, - 'invalid' are invalid containers (invalid content) and 'not_foud' has + 'invalid' are invalid containers (invalid content) and 'not_found' has some missing entity in database. Args: diff --git a/openpype/pipeline/project_folders.py b/openpype/pipeline/project_folders.py new file mode 100644 index 0000000000..1bcba5c320 --- /dev/null +++ b/openpype/pipeline/project_folders.py @@ -0,0 +1,107 @@ +import os +import re +import json + +import six + +from openpype.settings import get_project_settings +from openpype.lib import Logger + +from .anatomy import Anatomy +from .template_data import get_project_template_data + + +def concatenate_splitted_paths(split_paths, anatomy): + log = Logger.get_logger("concatenate_splitted_paths") + pattern_array = re.compile(r"\[.*\]") + output = [] + for path_items in split_paths: + clean_items = [] + if isinstance(path_items, str): + path_items = [path_items] + + for path_item in path_items: + if not re.match(r"{.+}", path_item): + path_item = re.sub(pattern_array, "", path_item) + clean_items.append(path_item) + + # backward compatibility + if "__project_root__" in path_items: + for root, root_path in anatomy.roots.items(): + if not os.path.exists(str(root_path)): + log.debug("Root {} path path {} not exist on \ + computer!".format(root, root_path)) + continue + clean_items = ["{{root[{}]}}".format(root), + r"{project[name]}"] + clean_items[1:] + output.append(os.path.normpath(os.path.sep.join(clean_items))) + continue + + output.append(os.path.normpath(os.path.sep.join(clean_items))) + + return output + + +def fill_paths(path_list, anatomy): + format_data = get_project_template_data(project_name=anatomy.project_name) + format_data["root"] = anatomy.roots + filled_paths = [] + + for path in path_list: + new_path = path.format(**format_data) + filled_paths.append(new_path) + + return filled_paths + + +def create_project_folders(project_name, basic_paths=None): + log = Logger.get_logger("create_project_folders") + anatomy = Anatomy(project_name) + if basic_paths is None: + basic_paths = get_project_basic_paths(project_name) + + if not basic_paths: + return + + concat_paths = concatenate_splitted_paths(basic_paths, anatomy) + filled_paths = fill_paths(concat_paths, anatomy) + + # Create folders + for path in filled_paths: + if os.path.exists(path): + log.debug("Folder already exists: {}".format(path)) + else: + log.debug("Creating folder: {}".format(path)) + os.makedirs(path) + + +def _list_path_items(folder_structure): + output = [] + for key, value in folder_structure.items(): + if not value: + output.append(key) + continue + + paths = _list_path_items(value) + for path in paths: + if not isinstance(path, (list, tuple)): + path = [path] + + item = [key] + item.extend(path) + output.append(item) + + return output + + +def get_project_basic_paths(project_name): + project_settings = get_project_settings(project_name) + folder_structure = ( + project_settings["global"]["project_folder_structure"] + ) + if not folder_structure: + return [] + + if isinstance(folder_structure, six.string_types): + folder_structure = json.loads(folder_structure) + return _list_path_items(folder_structure) diff --git a/openpype/pipeline/publish/__init__.py b/openpype/pipeline/publish/__init__.py index 8ccfc32a6b..04b1a66f3a 100644 --- a/openpype/pipeline/publish/__init__.py +++ b/openpype/pipeline/publish/__init__.py @@ -22,6 +22,8 @@ from .publish_plugins import ( ) from .lib import ( + get_publish_template_name, + DiscoverResult, publish_plugins_discover, load_help_content_from_plugin, @@ -62,6 +64,8 @@ __all__ = ( "Extractor", + "get_publish_template_name", + "DiscoverResult", "publish_plugins_discover", "load_help_content_from_plugin", diff --git a/openpype/pipeline/publish/contants.py b/openpype/pipeline/publish/contants.py new file mode 100644 index 0000000000..169eca2e5c --- /dev/null +++ b/openpype/pipeline/publish/contants.py @@ -0,0 +1,2 @@ +DEFAULT_PUBLISH_TEMPLATE = "publish" +DEFAULT_HERO_PUBLISH_TEMPLATE = "hero" diff --git a/openpype/pipeline/publish/lib.py b/openpype/pipeline/publish/lib.py index e7a3ef73fe..c76671fa39 100644 --- a/openpype/pipeline/publish/lib.py +++ b/openpype/pipeline/publish/lib.py @@ -2,6 +2,7 @@ import os import sys import types import inspect +import copy import tempfile import xml.etree.ElementTree @@ -9,8 +10,190 @@ import six import pyblish.plugin import pyblish.api -from openpype.lib import Logger -from openpype.settings import get_project_settings, get_system_settings +from openpype.lib import Logger, filter_profiles +from openpype.settings import ( + get_project_settings, + get_system_settings, +) + +from .contants import ( + DEFAULT_PUBLISH_TEMPLATE, + DEFAULT_HERO_PUBLISH_TEMPLATE, +) + + +def get_template_name_profiles( + project_name, project_settings=None, logger=None +): + """Receive profiles for publish template keys. + + At least one of arguments must be passed. + + Args: + project_name (str): Name of project where to look for templates. + project_settings(Dic[str, Any]): Prepared project settings. + + Returns: + List[Dict[str, Any]]: Publish template profiles. + """ + + if not project_name and not project_settings: + raise ValueError(( + "Both project name and project settings are missing." + " At least one must be entered." + )) + + if not project_settings: + project_settings = get_project_settings(project_name) + + profiles = ( + project_settings + ["global"] + ["tools"] + ["publish"] + ["template_name_profiles"] + ) + if profiles: + return copy.deepcopy(profiles) + + # Use legacy approach for cases new settings are not filled yet for the + # project + legacy_profiles = ( + project_settings + ["global"] + ["publish"] + ["IntegrateAssetNew"] + ["template_name_profiles"] + ) + if legacy_profiles: + if not logger: + logger = Logger.get_logger("get_template_name_profiles") + + logger.warning(( + "Project \"{}\" is using legacy access to publish template." + " It is recommended to move settings to new location" + " 'project_settings/global/tools/publish/template_name_profiles'." + ).format(project_name)) + + # Replace "tasks" key with "task_names" + profiles = [] + for profile in copy.deepcopy(legacy_profiles): + profile["task_names"] = profile.pop("tasks", []) + profiles.append(profile) + return profiles + + +def get_hero_template_name_profiles( + project_name, project_settings=None, logger=None +): + """Receive profiles for hero publish template keys. + + At least one of arguments must be passed. + + Args: + project_name (str): Name of project where to look for templates. + project_settings(Dic[str, Any]): Prepared project settings. + + Returns: + List[Dict[str, Any]]: Publish template profiles. + """ + + if not project_name and not project_settings: + raise ValueError(( + "Both project name and project settings are missing." + " At least one must be entered." + )) + + if not project_settings: + project_settings = get_project_settings(project_name) + + profiles = ( + project_settings + ["global"] + ["tools"] + ["publish"] + ["hero_template_name_profiles"] + ) + if profiles: + return copy.deepcopy(profiles) + + # Use legacy approach for cases new settings are not filled yet for the + # project + legacy_profiles = copy.deepcopy( + project_settings + ["global"] + ["publish"] + ["IntegrateHeroVersion"] + ["template_name_profiles"] + ) + if legacy_profiles: + if not logger: + logger = Logger.get_logger("get_hero_template_name_profiles") + + logger.warning(( + "Project \"{}\" is using legacy access to hero publish template." + " It is recommended to move settings to new location" + " 'project_settings/global/tools/publish/" + "hero_template_name_profiles'." + ).format(project_name)) + return legacy_profiles + + +def get_publish_template_name( + project_name, + host_name, + family, + task_name, + task_type, + project_settings=None, + hero=False, + logger=None +): + """Get template name which should be used for passed context. + + Publish templates are filtered by host name, family, task name and + task type. + + Default template which is used at if profiles are not available or profile + has empty value is defined by 'DEFAULT_PUBLISH_TEMPLATE' constant. + + Args: + project_name (str): Name of project where to look for settings. + host_name (str): Name of host integration. + family (str): Family for which should be found template. + task_name (str): Task name on which is intance working. + task_type (str): Task type on which is intance working. + project_setting (Dict[str, Any]): Prepared project settings. + logger (logging.Logger): Custom logger used for 'filter_profiles' + function. + + Returns: + str: Template name which should be used for integration. + """ + + template = None + filter_criteria = { + "hosts": host_name, + "families": family, + "task_names": task_name, + "task_types": task_type, + } + if hero: + default_template = DEFAULT_HERO_PUBLISH_TEMPLATE + profiles = get_hero_template_name_profiles( + project_name, project_settings, logger + ) + + else: + profiles = get_template_name_profiles( + project_name, project_settings, logger + ) + default_template = DEFAULT_PUBLISH_TEMPLATE + + profile = filter_profiles(profiles, filter_criteria, logger=logger) + if profile: + template = profile["template_name"] + return template or default_template class DiscoverResult: diff --git a/openpype/pipeline/template_data.py b/openpype/pipeline/template_data.py index bab46a627d..627eba5c3d 100644 --- a/openpype/pipeline/template_data.py +++ b/openpype/pipeline/template_data.py @@ -53,7 +53,7 @@ def get_project_template_data(project_doc=None, project_name=None): project_name = project_doc["name"] if not project_doc: - project_code = get_project(project_name, fields=["data.code"]) + project_doc = get_project(project_name, fields=["data.code"]) project_code = project_doc.get("data", {}).get("code") return { diff --git a/openpype/pipeline/workfile/__init__.py b/openpype/pipeline/workfile/__init__.py index 0aad29b6f9..94ecc81bd6 100644 --- a/openpype/pipeline/workfile/__init__.py +++ b/openpype/pipeline/workfile/__init__.py @@ -9,6 +9,8 @@ from .path_resolving import ( get_custom_workfile_template, get_custom_workfile_template_by_string_context, + + create_workdir_extra_folders, ) from .build_workfile import BuildWorkfile @@ -26,5 +28,7 @@ __all__ = ( "get_custom_workfile_template", "get_custom_workfile_template_by_string_context", + "create_workdir_extra_folders", + "BuildWorkfile", ) diff --git a/openpype/pipeline/workfile/abstract_template_loader.py b/openpype/pipeline/workfile/abstract_template_loader.py deleted file mode 100644 index 370dcef20a..0000000000 --- a/openpype/pipeline/workfile/abstract_template_loader.py +++ /dev/null @@ -1,526 +0,0 @@ -import os -from abc import ABCMeta, abstractmethod - -import six -import logging -from functools import reduce - -from openpype.client import get_asset_by_name -from openpype.settings import get_project_settings -from openpype.lib import ( - StringTemplate, - Logger, - filter_profiles, - get_linked_assets, -) -from openpype.pipeline import legacy_io, Anatomy -from openpype.pipeline.load import ( - get_loaders_by_name, - get_representation_context, - load_with_repre_context, -) - -from .build_template_exceptions import ( - TemplateAlreadyImported, - TemplateLoadingFailed, - TemplateProfileNotFound, - TemplateNotFound -) - -log = logging.getLogger(__name__) - - -def update_representations(entities, entity): - if entity['context']['subset'] not in entities: - entities[entity['context']['subset']] = entity - else: - current = entities[entity['context']['subset']] - incomming = entity - entities[entity['context']['subset']] = max( - current, incomming, - key=lambda entity: entity["context"].get("version", -1)) - - return entities - - -def parse_loader_args(loader_args): - if not loader_args: - return dict() - try: - parsed_args = eval(loader_args) - if not isinstance(parsed_args, dict): - return dict() - else: - return parsed_args - except Exception as err: - print( - "Error while parsing loader arguments '{}'.\n{}: {}\n\n" - "Continuing with default arguments. . .".format( - loader_args, - err.__class__.__name__, - err)) - return dict() - - -@six.add_metaclass(ABCMeta) -class AbstractTemplateLoader: - """ - Abstraction of Template Loader. - Properties: - template_path : property to get current template path - Methods: - import_template : Abstract Method. Used to load template, - depending on current host - get_template_nodes : Abstract Method. Used to query nodes acting - as placeholders. Depending on current host - """ - - _log = None - - def __init__(self, placeholder_class): - # TODO template loader should expect host as and argument - # - host have all responsibility for most of code (also provide - # placeholder class) - # - also have responsibility for current context - # - this won't work in DCCs where multiple workfiles with - # different contexts can be opened at single time - # - template loader should have ability to change context - project_name = legacy_io.active_project() - asset_name = legacy_io.Session["AVALON_ASSET"] - - self.loaders_by_name = get_loaders_by_name() - self.current_asset = asset_name - self.project_name = project_name - self.host_name = legacy_io.Session["AVALON_APP"] - self.task_name = legacy_io.Session["AVALON_TASK"] - self.placeholder_class = placeholder_class - self.current_asset_doc = get_asset_by_name(project_name, asset_name) - self.task_type = ( - self.current_asset_doc - .get("data", {}) - .get("tasks", {}) - .get(self.task_name, {}) - .get("type") - ) - - self.log.info( - "BUILDING ASSET FROM TEMPLATE :\n" - "Starting templated build for {asset} in {project}\n\n" - "Asset : {asset}\n" - "Task : {task_name} ({task_type})\n" - "Host : {host}\n" - "Project : {project}\n".format( - asset=self.current_asset, - host=self.host_name, - project=self.project_name, - task_name=self.task_name, - task_type=self.task_type - )) - # Skip if there is no loader - if not self.loaders_by_name: - self.log.warning( - "There is no registered loaders. No assets will be loaded") - return - - @property - def log(self): - if self._log is None: - self._log = Logger.get_logger(self.__class__.__name__) - return self._log - - def template_already_imported(self, err_msg): - """In case template was already loaded. - Raise the error as a default action. - Override this method in your template loader implementation - to manage this case.""" - self.log.error("{}: {}".format( - err_msg.__class__.__name__, - err_msg)) - raise TemplateAlreadyImported(err_msg) - - def template_loading_failed(self, err_msg): - """In case template loading failed - Raise the error as a default action. - Override this method in your template loader implementation - to manage this case. - """ - self.log.error("{}: {}".format( - err_msg.__class__.__name__, - err_msg)) - raise TemplateLoadingFailed(err_msg) - - @property - def template_path(self): - """ - Property returning template path. Avoiding setter. - Getting template path from open pype settings based on current avalon - session and solving the path variables if needed. - Returns: - str: Solved template path - Raises: - TemplateProfileNotFound: No profile found from settings for - current avalon session - KeyError: Could not solve path because a key does not exists - in avalon context - TemplateNotFound: Solved path does not exists on current filesystem - """ - project_name = self.project_name - host_name = self.host_name - task_name = self.task_name - task_type = self.task_type - - anatomy = Anatomy(project_name) - project_settings = get_project_settings(project_name) - - build_info = project_settings[host_name]["templated_workfile_build"] - profile = filter_profiles( - build_info["profiles"], - { - "task_types": task_type, - "task_names": task_name - } - ) - - if not profile: - raise TemplateProfileNotFound( - "No matching profile found for task '{}' of type '{}' " - "with host '{}'".format(task_name, task_type, host_name) - ) - - path = profile["path"] - if not path: - raise TemplateLoadingFailed( - "Template path is not set.\n" - "Path need to be set in {}\\Template Workfile Build " - "Settings\\Profiles".format(host_name.title())) - - # Try fill path with environments and anatomy roots - fill_data = { - key: value - for key, value in os.environ.items() - } - fill_data["root"] = anatomy.roots - result = StringTemplate.format_template(path, fill_data) - if result.solved: - path = result.normalized() - - if path and os.path.exists(path): - self.log.info("Found template at: '{}'".format(path)) - return path - - solved_path = None - while True: - try: - solved_path = anatomy.path_remapper(path) - except KeyError as missing_key: - raise KeyError( - "Could not solve key '{}' in template path '{}'".format( - missing_key, path)) - - if solved_path is None: - solved_path = path - if solved_path == path: - break - path = solved_path - - solved_path = os.path.normpath(solved_path) - if not os.path.exists(solved_path): - raise TemplateNotFound( - "Template found in openPype settings for task '{}' with host " - "'{}' does not exists. (Not found : {})".format( - task_name, host_name, solved_path)) - - self.log.info("Found template at: '{}'".format(solved_path)) - - return solved_path - - def populate_template(self, ignored_ids=None): - """ - Use template placeholders to load assets and parent them in hierarchy - Arguments : - ignored_ids : - Returns: - None - """ - - loaders_by_name = self.loaders_by_name - current_asset_doc = self.current_asset_doc - linked_assets = get_linked_assets(current_asset_doc) - - ignored_ids = ignored_ids or [] - placeholders = self.get_placeholders() - self.log.debug("Placeholders found in template: {}".format( - [placeholder.name for placeholder in placeholders] - )) - for placeholder in placeholders: - self.log.debug("Start to processing placeholder {}".format( - placeholder.name - )) - placeholder_representations = self.get_placeholder_representations( - placeholder, - current_asset_doc, - linked_assets - ) - - if not placeholder_representations: - self.log.info( - "There's no representation for this placeholder: " - "{}".format(placeholder.name) - ) - continue - - for representation in placeholder_representations: - self.preload(placeholder, loaders_by_name, representation) - - if self.load_data_is_incorrect( - placeholder, - representation, - ignored_ids): - continue - - self.log.info( - "Loading {}_{} with loader {}\n" - "Loader arguments used : {}".format( - representation['context']['asset'], - representation['context']['subset'], - placeholder.loader_name, - placeholder.loader_args)) - - try: - container = self.load( - placeholder, loaders_by_name, representation) - except Exception: - self.load_failed(placeholder, representation) - else: - self.load_succeed(placeholder, container) - finally: - self.postload(placeholder) - - def get_placeholder_representations( - self, placeholder, current_asset_doc, linked_asset_docs - ): - placeholder_representations = placeholder.get_representations( - current_asset_doc, - linked_asset_docs - ) - for repre_doc in reduce( - update_representations, - placeholder_representations, - dict() - ).values(): - yield repre_doc - - def load_data_is_incorrect( - self, placeholder, last_representation, ignored_ids): - if not last_representation: - self.log.warning(placeholder.err_message()) - return True - if (str(last_representation['_id']) in ignored_ids): - print("Ignoring : ", last_representation['_id']) - return True - return False - - def preload(self, placeholder, loaders_by_name, last_representation): - pass - - def load(self, placeholder, loaders_by_name, last_representation): - repre = get_representation_context(last_representation) - return load_with_repre_context( - loaders_by_name[placeholder.loader_name], - repre, - options=parse_loader_args(placeholder.loader_args)) - - def load_succeed(self, placeholder, container): - placeholder.parent_in_hierarchy(container) - - def load_failed(self, placeholder, last_representation): - self.log.warning( - "Got error trying to load {}:{} with {}".format( - last_representation['context']['asset'], - last_representation['context']['subset'], - placeholder.loader_name - ), - exc_info=True - ) - - def postload(self, placeholder): - placeholder.clean() - - def update_missing_containers(self): - loaded_containers_ids = self.get_loaded_containers_by_id() - self.populate_template(ignored_ids=loaded_containers_ids) - - def get_placeholders(self): - placeholders = map(self.placeholder_class, self.get_template_nodes()) - valid_placeholders = filter( - lambda i: i.is_valid, - placeholders - ) - sorted_placeholders = list(sorted( - valid_placeholders, - key=lambda i: i.order - )) - return sorted_placeholders - - @abstractmethod - def get_loaded_containers_by_id(self): - """ - Collect already loaded containers for updating scene - Return: - dict (string, node): A dictionnary id as key - and containers as value - """ - pass - - @abstractmethod - def import_template(self, template_path): - """ - Import template in current host - Args: - template_path (str): fullpath to current task and - host's template file - Return: - None - """ - pass - - @abstractmethod - def get_template_nodes(self): - """ - Returning a list of nodes acting as host placeholders for - templating. The data representation is by user. - AbstractLoadTemplate (and LoadTemplate) won't directly manipulate nodes - Args : - None - Returns: - list(AnyNode): Solved template path - """ - pass - - -@six.add_metaclass(ABCMeta) -class AbstractPlaceholder: - """Abstraction of placeholders logic. - - Properties: - required_keys: A list of mandatory keys to decribe placeholder - and assets to load. - optional_keys: A list of optional keys to decribe - placeholder and assets to load - loader_name: Name of linked loader to use while loading assets - - Args: - identifier (str): Placeholder identifier. Should be possible to be - used as identifier in "a scene" (e.g. unique node name). - """ - - required_keys = { - "builder_type", - "family", - "representation", - "order", - "loader", - "loader_args" - } - optional_keys = {} - - def __init__(self, identifier): - self._log = None - self._name = identifier - self.get_data(identifier) - - @property - def log(self): - if self._log is None: - self._log = Logger.get_logger(repr(self)) - return self._log - - def __repr__(self): - return "< {} {} >".format(self.__class__.__name__, self.name) - - @property - def name(self): - return self._name - - @property - def loader_args(self): - return self.data["loader_args"] - - @property - def builder_type(self): - return self.data["builder_type"] - - @property - def order(self): - return self.data["order"] - - @property - def loader_name(self): - """Return placeholder loader name. - - Returns: - str: Loader name that will be used to load placeholder - representations. - """ - - return self.data["loader"] - - @property - def is_valid(self): - """Test validity of placeholder. - - i.e.: every required key exists in placeholder data - - Returns: - bool: True if every key is in data - """ - - if set(self.required_keys).issubset(self.data.keys()): - self.log.debug("Valid placeholder : {}".format(self.name)) - return True - self.log.info("Placeholder is not valid : {}".format(self.name)) - return False - - @abstractmethod - def parent_in_hierarchy(self, container): - """Place loaded container in correct hierarchy given by placeholder - - Args: - container (Dict[str, Any]): Loaded container created by loader. - """ - - pass - - @abstractmethod - def clean(self): - """Clean placeholder from hierarchy after loading assets.""" - - pass - - @abstractmethod - def get_representations(self, current_asset_doc, linked_asset_docs): - """Query representations based on placeholder data. - - Args: - current_asset_doc (Dict[str, Any]): Document of current - context asset. - linked_asset_docs (List[Dict[str, Any]]): Documents of assets - linked to current context asset. - - Returns: - Iterable[Dict[str, Any]]: Representations that are matching - placeholder filters. - """ - - pass - - @abstractmethod - def get_data(self, identifier): - """Collect information about placeholder by identifier. - - Args: - identifier (str): A unique placeholder identifier defined by - implementation. - """ - - pass diff --git a/openpype/pipeline/workfile/build_template.py b/openpype/pipeline/workfile/build_template.py deleted file mode 100644 index 3328dfbc9e..0000000000 --- a/openpype/pipeline/workfile/build_template.py +++ /dev/null @@ -1,72 +0,0 @@ -import os -from importlib import import_module -from openpype.lib import classes_from_module -from openpype.host import HostBase -from openpype.pipeline import registered_host - -from .abstract_template_loader import ( - AbstractPlaceholder, - AbstractTemplateLoader) - -from .build_template_exceptions import ( - TemplateLoadingFailed, - TemplateAlreadyImported, - MissingHostTemplateModule, - MissingTemplatePlaceholderClass, - MissingTemplateLoaderClass -) - -_module_path_format = 'openpype.hosts.{host}.api.template_loader' - - -def build_workfile_template(*args): - template_loader = build_template_loader() - try: - template_loader.import_template(template_loader.template_path) - except TemplateAlreadyImported as err: - template_loader.template_already_imported(err) - except TemplateLoadingFailed as err: - template_loader.template_loading_failed(err) - else: - template_loader.populate_template() - - -def update_workfile_template(*args): - template_loader = build_template_loader() - template_loader.update_missing_containers() - - -def build_template_loader(): - # TODO refactor to use advantage of 'HostBase' and don't import dynamically - # - hosts should have methods that gives option to return builders - host = registered_host() - if isinstance(host, HostBase): - host_name = host.name - else: - host_name = os.environ.get("AVALON_APP") - if not host_name: - host_name = host.__name__.split(".")[-2] - - module_path = _module_path_format.format(host=host_name) - module = import_module(module_path) - if not module: - raise MissingHostTemplateModule( - "No template loader found for host {}".format(host_name)) - - template_loader_class = classes_from_module( - AbstractTemplateLoader, - module - ) - template_placeholder_class = classes_from_module( - AbstractPlaceholder, - module - ) - - if not template_loader_class: - raise MissingTemplateLoaderClass() - template_loader_class = template_loader_class[0] - - if not template_placeholder_class: - raise MissingTemplatePlaceholderClass() - template_placeholder_class = template_placeholder_class[0] - return template_loader_class(template_placeholder_class) diff --git a/openpype/pipeline/workfile/build_template_exceptions.py b/openpype/pipeline/workfile/build_template_exceptions.py deleted file mode 100644 index 7a5075e3dc..0000000000 --- a/openpype/pipeline/workfile/build_template_exceptions.py +++ /dev/null @@ -1,35 +0,0 @@ -class MissingHostTemplateModule(Exception): - """Error raised when expected module does not exists""" - pass - - -class MissingTemplatePlaceholderClass(Exception): - """Error raised when module doesn't implement a placeholder class""" - pass - - -class MissingTemplateLoaderClass(Exception): - """Error raised when module doesn't implement a template loader class""" - pass - - -class TemplateNotFound(Exception): - """Exception raised when template does not exist.""" - pass - - -class TemplateProfileNotFound(Exception): - """Exception raised when current profile - doesn't match any template profile""" - pass - - -class TemplateAlreadyImported(Exception): - """Error raised when Template was already imported by host for - this session""" - pass - - -class TemplateLoadingFailed(Exception): - """Error raised whend Template loader was unable to load the template""" - pass diff --git a/openpype/pipeline/workfile/build_workfile.py b/openpype/pipeline/workfile/build_workfile.py index bb6fcb4189..87b9df158f 100644 --- a/openpype/pipeline/workfile/build_workfile.py +++ b/openpype/pipeline/workfile/build_workfile.py @@ -1,3 +1,14 @@ +"""Workfile build based on settings. + +Workfile builder will do stuff based on project settings. Advantage is that +it need only access to settings. Disadvantage is that it is hard to focus +build per context and being explicit about loaded content. + +For more explicit workfile build is recommended 'AbstractTemplateBuilder' +from '~/openpype/pipeline/workfile/workfile_template_builder'. Which gives +more abilities to define how build happens but require more code to achive it. +""" + import os import re import collections @@ -8,10 +19,10 @@ from openpype.client import ( get_subsets, get_last_versions, get_representations, + get_linked_assets, ) from openpype.settings import get_project_settings from openpype.lib import ( - get_linked_assets, filter_profiles, Logger, ) diff --git a/openpype/pipeline/workfile/lock_workfile.py b/openpype/pipeline/workfile/lock_workfile.py new file mode 100644 index 0000000000..fbec44247a --- /dev/null +++ b/openpype/pipeline/workfile/lock_workfile.py @@ -0,0 +1,82 @@ +import os +import json +from uuid import uuid4 +from openpype.lib import Logger, filter_profiles +from openpype.lib.pype_info import get_workstation_info +from openpype.settings import get_project_settings + + +def _read_lock_file(lock_filepath): + if not os.path.exists(lock_filepath): + log = Logger.get_logger("_read_lock_file") + log.debug("lock file is not created or readable as expected!") + with open(lock_filepath, "r") as stream: + data = json.load(stream) + return data + + +def _get_lock_file(filepath): + return filepath + ".oplock" + + +def is_workfile_locked(filepath): + lock_filepath = _get_lock_file(filepath) + if not os.path.exists(lock_filepath): + return False + return True + + +def get_workfile_lock_data(filepath): + lock_filepath = _get_lock_file(filepath) + return _read_lock_file(lock_filepath) + + +def is_workfile_locked_for_current_process(filepath): + if not is_workfile_locked(filepath): + return False + + lock_filepath = _get_lock_file(filepath) + data = _read_lock_file(lock_filepath) + return data["process_id"] == _get_process_id() + + +def delete_workfile_lock(filepath): + lock_filepath = _get_lock_file(filepath) + if os.path.exists(lock_filepath): + os.remove(lock_filepath) + + +def create_workfile_lock(filepath): + lock_filepath = _get_lock_file(filepath) + info = get_workstation_info() + info["process_id"] = _get_process_id() + with open(lock_filepath, "w") as stream: + json.dump(info, stream) + + +def remove_workfile_lock(filepath): + if is_workfile_locked_for_current_process(filepath): + delete_workfile_lock(filepath) + + +def _get_process_id(): + process_id = os.environ.get("OPENPYPE_PROCESS_ID") + if not process_id: + process_id = str(uuid4()) + os.environ["OPENPYPE_PROCESS_ID"] = process_id + return process_id + + +def is_workfile_lock_enabled(host_name, project_name, project_setting=None): + if project_setting is None: + project_setting = get_project_settings(project_name) + workfile_lock_profiles = ( + project_setting + ["global"] + ["tools"] + ["Workfiles"] + ["workfile_lock_profiles"]) + profile = filter_profiles(workfile_lock_profiles, {"host_name": host_name}) + if not profile: + return False + return profile["enabled"] diff --git a/openpype/pipeline/workfile/path_resolving.py b/openpype/pipeline/workfile/path_resolving.py index 6d9e72dbd2..1243e84148 100644 --- a/openpype/pipeline/workfile/path_resolving.py +++ b/openpype/pipeline/workfile/path_resolving.py @@ -467,3 +467,60 @@ def get_custom_workfile_template_by_string_context( return get_custom_workfile_template( project_doc, asset_doc, task_name, host_name, anatomy, project_settings ) + + +def create_workdir_extra_folders( + workdir, + host_name, + task_type, + task_name, + project_name, + project_settings=None +): + """Create extra folders in work directory based on context. + + Args: + workdir (str): Path to workdir where workfiles is stored. + host_name (str): Name of host implementation. + task_type (str): Type of task for which extra folders should be + created. + task_name (str): Name of task for which extra folders should be + created. + project_name (str): Name of project on which task is. + project_settings (dict): Prepared project settings. Are loaded if not + passed. + """ + + # Load project settings if not set + if not project_settings: + project_settings = get_project_settings(project_name) + + # Load extra folders profiles + extra_folders_profiles = ( + project_settings["global"]["tools"]["Workfiles"]["extra_folders"] + ) + # Skip if are empty + if not extra_folders_profiles: + return + + # Prepare profiles filters + filter_data = { + "task_types": task_type, + "task_names": task_name, + "hosts": host_name + } + profile = filter_profiles(extra_folders_profiles, filter_data) + if profile is None: + return + + for subfolder in profile["folders"]: + # Make sure backslashes are converted to forwards slashes + # and does not start with slash + subfolder = subfolder.replace("\\", "/").lstrip("/") + # Skip empty strings + if not subfolder: + continue + + fullpath = os.path.join(workdir, subfolder) + if not os.path.exists(fullpath): + os.makedirs(fullpath) diff --git a/openpype/pipeline/workfile/workfile_template_builder.py b/openpype/pipeline/workfile/workfile_template_builder.py new file mode 100644 index 0000000000..582657c735 --- /dev/null +++ b/openpype/pipeline/workfile/workfile_template_builder.py @@ -0,0 +1,1451 @@ +"""Workfile build mechanism using workfile templates. + +Build templates are manually prepared using plugin definitions which create +placeholders inside the template which are populated on import. + +This approach is very explicit to achive very specific build logic that can be +targeted by task types and names. + +Placeholders are created using placeholder plugins which should care about +logic and data of placeholder items. 'PlaceholderItem' is used to keep track +about it's progress. +""" + +import os +import re +import collections +import copy +from abc import ABCMeta, abstractmethod + +import six + +from openpype.client import ( + get_asset_by_name, + get_linked_assets, + get_representations, +) +from openpype.settings import ( + get_project_settings, + get_system_settings, +) +from openpype.host import HostBase +from openpype.lib import ( + Logger, + StringTemplate, + filter_profiles, + attribute_definitions, +) +from openpype.lib.attribute_definitions import get_attributes_keys +from openpype.pipeline import legacy_io, Anatomy +from openpype.pipeline.load import ( + get_loaders_by_name, + get_contexts_for_repre_docs, + load_with_repre_context, +) +from openpype.pipeline.create import get_legacy_creator_by_name + + +class TemplateNotFound(Exception): + """Exception raised when template does not exist.""" + pass + + +class TemplateProfileNotFound(Exception): + """Exception raised when current profile + doesn't match any template profile""" + pass + + +class TemplateAlreadyImported(Exception): + """Error raised when Template was already imported by host for + this session""" + pass + + +class TemplateLoadFailed(Exception): + """Error raised whend Template loader was unable to load the template""" + pass + + +@six.add_metaclass(ABCMeta) +class AbstractTemplateBuilder(object): + """Abstraction of Template Builder. + + Builder cares about context, shared data, cache, discovery of plugins + and trigger logic. Provides public api for host workfile build systen. + + Rest of logic is based on plugins that care about collection and creation + of placeholder items. + + Population of placeholders happens in loops. Each loop will collect all + available placeholders, skip already populated, and populate the rest. + + Builder item has 2 types of shared data. Refresh lifetime which are cleared + on refresh and populate lifetime which are cleared after loop of + placeholder population. + + Args: + host (Union[HostBase, ModuleType]): Implementation of host. + """ + + _log = None + + def __init__(self, host): + # Get host name + if isinstance(host, HostBase): + host_name = host.name + else: + host_name = os.environ.get("AVALON_APP") + + self._host = host + self._host_name = host_name + + # Shared data across placeholder plugins + self._shared_data = {} + self._shared_populate_data = {} + + # Where created objects of placeholder plugins will be stored + self._placeholder_plugins = None + self._loaders_by_name = None + self._creators_by_name = None + + self._system_settings = None + self._project_settings = None + + self._current_asset_doc = None + self._linked_asset_docs = None + self._task_type = None + + @property + def project_name(self): + return legacy_io.active_project() + + @property + def current_asset_name(self): + return legacy_io.Session["AVALON_ASSET"] + + @property + def current_task_name(self): + return legacy_io.Session["AVALON_TASK"] + + @property + def system_settings(self): + if self._system_settings is None: + self._system_settings = get_system_settings() + return self._system_settings + + @property + def project_settings(self): + if self._project_settings is None: + self._project_settings = get_project_settings(self.project_name) + return self._project_settings + + @property + def current_asset_doc(self): + if self._current_asset_doc is None: + self._current_asset_doc = get_asset_by_name( + self.project_name, self.current_asset_name + ) + return self._current_asset_doc + + @property + def linked_asset_docs(self): + if self._linked_asset_docs is None: + self._linked_asset_docs = get_linked_assets( + self.current_asset_doc + ) + return self._linked_asset_docs + + @property + def current_task_type(self): + asset_doc = self.current_asset_doc + if not asset_doc: + return None + return ( + asset_doc + .get("data", {}) + .get("tasks", {}) + .get(self.current_task_name, {}) + .get("type") + ) + + def get_placeholder_plugin_classes(self): + """Get placeholder plugin classes that can be used to build template. + + Default implementation looks for method + 'get_workfile_build_placeholder_plugins' on host. + + Returns: + List[PlaceholderPlugin]: Plugin classes available for host. + """ + + if hasattr(self._host, "get_workfile_build_placeholder_plugins"): + return self._host.get_workfile_build_placeholder_plugins() + return [] + + @property + def host(self): + """Access to host implementation. + + Returns: + Union[HostBase, ModuleType]: Implementation of host. + """ + + return self._host + + @property + def host_name(self): + """Name of 'host' implementation. + + Returns: + str: Host's name. + """ + + return self._host_name + + @property + def log(self): + """Dynamically created logger for the plugin.""" + + if self._log is None: + self._log = Logger.get_logger(repr(self)) + return self._log + + def refresh(self): + """Reset cached data.""" + + self._placeholder_plugins = None + self._loaders_by_name = None + self._creators_by_name = None + + self._current_asset_doc = None + self._linked_asset_docs = None + self._task_type = None + + self._system_settings = None + self._project_settings = None + + self.clear_shared_data() + self.clear_shared_populate_data() + + def get_loaders_by_name(self): + if self._loaders_by_name is None: + self._loaders_by_name = get_loaders_by_name() + return self._loaders_by_name + + def get_creators_by_name(self): + if self._creators_by_name is None: + self._creators_by_name = get_legacy_creator_by_name() + return self._creators_by_name + + def get_shared_data(self, key): + """Receive shared data across plugins and placeholders. + + This can be used to scroll scene only once to look for placeholder + items if the storing is unified but each placeholder plugin would have + to call it again. + + Args: + key (str): Key under which are shared data stored. + + Returns: + Union[None, Any]: None if key was not set. + """ + + return self._shared_data.get(key) + + def set_shared_data(self, key, value): + """Store share data across plugins and placeholders. + + Store data that can be afterwards accessed from any future call. It + is good practice to check if the same value is not already stored under + different key or if the key is not already used for something else. + + Key should be self explanatory to content. + - wrong: 'asset' + - good: 'asset_name' + + Args: + key (str): Key under which is key stored. + value (Any): Value that should be stored under the key. + """ + + self._shared_data[key] = value + + def clear_shared_data(self): + """Clear shared data. + + Method only clear shared data to default state. + """ + + self._shared_data = {} + + def clear_shared_populate_data(self): + """Receive shared data across plugins and placeholders. + + These data are cleared after each loop of populating of template. + + This can be used to scroll scene only once to look for placeholder + items if the storing is unified but each placeholder plugin would have + to call it again. + + Args: + key (str): Key under which are shared data stored. + + Returns: + Union[None, Any]: None if key was not set. + """ + + self._shared_populate_data = {} + + def get_shared_populate_data(self, key): + """Store share populate data across plugins and placeholders. + + These data are cleared after each loop of populating of template. + + Store data that can be afterwards accessed from any future call. It + is good practice to check if the same value is not already stored under + different key or if the key is not already used for something else. + + Key should be self explanatory to content. + - wrong: 'asset' + - good: 'asset_name' + + Args: + key (str): Key under which is key stored. + value (Any): Value that should be stored under the key. + """ + + return self._shared_populate_data.get(key) + + def set_shared_populate_data(self, key, value): + """Store share populate data across plugins and placeholders. + + These data are cleared after each loop of populating of template. + + Store data that can be afterwards accessed from any future call. It + is good practice to check if the same value is not already stored under + different key or if the key is not already used for something else. + + Key should be self explanatory to content. + - wrong: 'asset' + - good: 'asset_name' + + Args: + key (str): Key under which is key stored. + value (Any): Value that should be stored under the key. + """ + + self._shared_populate_data[key] = value + + @property + def placeholder_plugins(self): + """Access to initialized placeholder plugins. + + Returns: + List[PlaceholderPlugin]: Initialized plugins available for host. + """ + + if self._placeholder_plugins is None: + placeholder_plugins = {} + for cls in self.get_placeholder_plugin_classes(): + try: + plugin = cls(self) + placeholder_plugins[plugin.identifier] = plugin + + except Exception: + self.log.warning( + "Failed to initialize placeholder plugin {}".format( + cls.__name__ + ), + exc_info=True + ) + + self._placeholder_plugins = placeholder_plugins + return self._placeholder_plugins + + def create_placeholder(self, plugin_identifier, placeholder_data): + """Create new placeholder using plugin identifier and data. + + Args: + plugin_identifier (str): Identifier of plugin. That's how builder + know which plugin should be used. + placeholder_data (Dict[str, Any]): Placeholder item data. They + should match options required by the plugin. + + Returns: + PlaceholderItem: Created placeholder item. + """ + + plugin = self.placeholder_plugins[plugin_identifier] + return plugin.create_placeholder(placeholder_data) + + def get_placeholders(self): + """Collect placeholder items from scene. + + Each placeholder plugin can collect it's placeholders and return them. + This method does not use cached values but always go through the scene. + + Returns: + List[PlaceholderItem]: Sorted placeholder items. + """ + + placeholders = [] + for placeholder_plugin in self.placeholder_plugins.values(): + result = placeholder_plugin.collect_placeholders() + if result: + placeholders.extend(result) + + return list(sorted( + placeholders, + key=lambda i: i.order + )) + + def build_template(self, template_path=None, level_limit=None): + """Main callback for building workfile from template path. + + Todo: + Handle report of populated placeholders from + 'populate_scene_placeholders' to be shown to a user. + + Args: + template_path (str): Path to a template file with placeholders. + Template from settings 'get_template_path' used when not + passed. + level_limit (int): Limit of populate loops. Related to + 'populate_scene_placeholders' method. + """ + + if template_path is None: + template_path = self.get_template_path() + self.import_template(template_path) + self.populate_scene_placeholders(level_limit) + + def rebuild_template(self): + """Go through existing placeholders in scene and update them. + + This could not make sense for all plugin types so this is optional + logic for plugins. + + Note: + Logic is not importing the template again but using placeholders + that were already available. We should maybe change the method + name. + + Question: + Should this also handle subloops as it is possible that another + template is loaded during processing? + """ + + if not self.placeholder_plugins: + self.log.info("There are no placeholder plugins available.") + return + + placeholders = self.get_placeholders() + if not placeholders: + self.log.info("No placeholders were found.") + return + + for placeholder in placeholders: + plugin = placeholder.plugin + plugin.repopulate_placeholder(placeholder) + + self.clear_shared_populate_data() + + @abstractmethod + def import_template(self, template_path): + """ + Import template in current host. + + Should load the content of template into scene so + 'populate_scene_placeholders' can be started. + + Args: + template_path (str): Fullpath for current task and + host's template file. + """ + + pass + + def _prepare_placeholders(self, placeholders): + """Run preparation part for placeholders on plugins. + + Args: + placeholders (List[PlaceholderItem]): Placeholder items that will + be processed. + """ + + # Prepare placeholder items by plugin + plugins_by_identifier = {} + placeholders_by_plugin_id = collections.defaultdict(list) + for placeholder in placeholders: + plugin = placeholder.plugin + identifier = plugin.identifier + plugins_by_identifier[identifier] = plugin + placeholders_by_plugin_id[identifier].append(placeholder) + + # Plugin should prepare data for passed placeholders + for identifier, placeholders in placeholders_by_plugin_id.items(): + plugin = plugins_by_identifier[identifier] + plugin.prepare_placeholders(placeholders) + + def populate_scene_placeholders(self, level_limit=None): + """Find placeholders in scene using plugins and process them. + + This should happen after 'import_template'. + + Collect available placeholders from scene. All of them are processed + after that shared data are cleared. Placeholder items are collected + again and if there are any new the loop happens again. This is possible + to change with defying 'level_limit'. + + Placeholders are marked as processed so they're not re-processed. To + identify which placeholders were already processed is used + placeholder's 'scene_identifier'. + + Args: + level_limit (int): Level of loops that can happen. Default is 1000. + """ + + if not self.placeholder_plugins: + self.log.warning("There are no placeholder plugins available.") + return + + placeholders = self.get_placeholders() + if not placeholders: + self.log.warning("No placeholders were found.") + return + + # Avoid infinite loop + # - 1000 iterations of placeholders processing must be enough + if not level_limit: + level_limit = 1000 + + placeholder_by_scene_id = { + placeholder.scene_identifier: placeholder + for placeholder in placeholders + } + all_processed = len(placeholders) == 0 + # Counter is checked at the ned of a loop so the loop happens at least + # once. + iter_counter = 0 + while not all_processed: + filtered_placeholders = [] + for placeholder in placeholders: + if placeholder.finished: + continue + + if placeholder.in_progress: + self.log.warning(( + "Placeholder that should be processed" + " is already in progress." + )) + continue + filtered_placeholders.append(placeholder) + + self._prepare_placeholders(filtered_placeholders) + + for placeholder in filtered_placeholders: + placeholder.set_in_progress() + placeholder_plugin = placeholder.plugin + try: + placeholder_plugin.populate_placeholder(placeholder) + + except Exception as exc: + self.log.warning( + ( + "Failed to process placeholder {} with plugin {}" + ).format( + placeholder.scene_identifier, + placeholder_plugin.__class__.__name__ + ), + exc_info=True + ) + placeholder.set_failed(exc) + + placeholder.set_finished() + + # Clear shared data before getting new placeholders + self.clear_shared_populate_data() + + iter_counter += 1 + if iter_counter >= level_limit: + break + + all_processed = True + collected_placeholders = self.get_placeholders() + for placeholder in collected_placeholders: + identifier = placeholder.scene_identifier + if identifier in placeholder_by_scene_id: + continue + + all_processed = False + placeholder_by_scene_id[identifier] = placeholder + placeholders.append(placeholder) + + self.refresh() + + def _get_build_profiles(self): + """Get build profiles for workfile build template path. + + Returns: + List[Dict[str, Any]]: Profiles for template path resolving. + """ + + return ( + self.project_settings + [self.host_name] + ["templated_workfile_build"] + ["profiles"] + ) + + def get_template_path(self): + """Unified way how template path is received usign settings. + + Method is dependent on '_get_build_profiles' which should return filter + profiles to resolve path to a template. Default implementation looks + into host settings: + - 'project_settings/{host name}/templated_workfile_build/profiles' + + Returns: + str: Path to a template file with placeholders. + + Raises: + TemplateProfileNotFound: When profiles are not filled. + TemplateLoadFailed: Profile was found but path is not set. + TemplateNotFound: Path was set but file does not exists. + """ + + host_name = self.host_name + project_name = self.project_name + task_name = self.current_task_name + task_type = self.current_task_type + + build_profiles = self._get_build_profiles() + profile = filter_profiles( + build_profiles, + { + "task_types": task_type, + "task_names": task_name + } + ) + + if not profile: + raise TemplateProfileNotFound(( + "No matching profile found for task '{}' of type '{}' " + "with host '{}'" + ).format(task_name, task_type, host_name)) + + path = profile["path"] + if not path: + raise TemplateLoadFailed(( + "Template path is not set.\n" + "Path need to be set in {}\\Template Workfile Build " + "Settings\\Profiles" + ).format(host_name.title())) + + # Try fill path with environments and anatomy roots + anatomy = Anatomy(project_name) + fill_data = { + key: value + for key, value in os.environ.items() + } + fill_data["root"] = anatomy.roots + result = StringTemplate.format_template(path, fill_data) + if result.solved: + path = result.normalized() + + if path and os.path.exists(path): + self.log.info("Found template at: '{}'".format(path)) + return path + + solved_path = None + while True: + try: + solved_path = anatomy.path_remapper(path) + except KeyError as missing_key: + raise KeyError( + "Could not solve key '{}' in template path '{}'".format( + missing_key, path)) + + if solved_path is None: + solved_path = path + if solved_path == path: + break + path = solved_path + + solved_path = os.path.normpath(solved_path) + if not os.path.exists(solved_path): + raise TemplateNotFound( + "Template found in openPype settings for task '{}' with host " + "'{}' does not exists. (Not found : {})".format( + task_name, host_name, solved_path)) + + self.log.info("Found template at: '{}'".format(solved_path)) + + return solved_path + + +@six.add_metaclass(ABCMeta) +class PlaceholderPlugin(object): + """Plugin which care about handling of placeholder items logic. + + Plugin create and update placeholders in scene and populate them on + template import. Populating means that based on placeholder data happens + a logic in the scene. Most common logic is to load representation using + loaders or to create instances in scene. + """ + + label = None + _log = None + + def __init__(self, builder): + self._builder = builder + + @property + def builder(self): + """Access to builder which initialized the plugin. + + Returns: + AbstractTemplateBuilder: Loader of template build. + """ + + return self._builder + + @property + def project_name(self): + return self._builder.project_name + + @property + def log(self): + """Dynamically created logger for the plugin.""" + + if self._log is None: + self._log = Logger.get_logger(repr(self)) + return self._log + + @property + def identifier(self): + """Identifier which will be stored to placeholder. + + Default implementation uses class name. + + Returns: + str: Unique identifier of placeholder plugin. + """ + + return self.__class__.__name__ + + @abstractmethod + def create_placeholder(self, placeholder_data): + """Create new placeholder in scene and get it's item. + + It matters on the plugin implementation if placeholder will use + selection in scene or create new node. + + Args: + placeholder_data (Dict[str, Any]): Data that were created + based on attribute definitions from 'get_placeholder_options'. + + Returns: + PlaceholderItem: Created placeholder item. + """ + + pass + + @abstractmethod + def update_placeholder(self, placeholder_item, placeholder_data): + """Update placeholder item with new data. + + New data should be propagated to object of placeholder item itself + and also into the scene. + + Reason: + Some placeholder plugins may require some special way how the + updates should be propagated to object. + + Args: + placeholder_item (PlaceholderItem): Object of placeholder that + should be updated. + placeholder_data (Dict[str, Any]): Data related to placeholder. + Should match plugin options. + """ + + pass + + @abstractmethod + def collect_placeholders(self): + """Collect placeholders from scene. + + Returns: + List[PlaceholderItem]: Placeholder objects. + """ + + pass + + def get_placeholder_options(self, options=None): + """Placeholder options for data showed. + + Returns: + List[AbtractAttrDef]: Attribute definitions of placeholder options. + """ + + return [] + + def get_placeholder_keys(self): + """Get placeholder keys that are stored in scene. + + Returns: + Set[str]: Key of placeholder keys that are stored in scene. + """ + + option_keys = get_attributes_keys(self.get_placeholder_options()) + option_keys.add("plugin_identifier") + return option_keys + + def prepare_placeholders(self, placeholders): + """Preparation part of placeholders. + + Args: + placeholders (List[PlaceholderItem]): List of placeholders that + will be processed. + """ + + pass + + @abstractmethod + def populate_placeholder(self, placeholder): + """Process single placeholder item. + + Processing of placeholders is defined by their order thus can't be + processed in batch. + + Args: + placeholder (PlaceholderItem): Placeholder that should be + processed. + """ + + pass + + def repopulate_placeholder(self, placeholder): + """Update scene with current context for passed placeholder. + + Can be used to re-run placeholder logic (if it make sense). + """ + + pass + + def get_plugin_shared_data(self, key): + """Receive shared data across plugin and placeholders. + + Using shared data from builder but stored under plugin identifier. + + Args: + key (str): Key under which are shared data stored. + + Returns: + Union[None, Any]: None if key was not set. + """ + + plugin_data = self.builder.get_shared_data(self.identifier) + if plugin_data is None: + return None + return plugin_data.get(key) + + def set_plugin_shared_data(self, key, value): + """Store share data across plugin and placeholders. + + Using shared data from builder but stored under plugin identifier. + + Key should be self explanatory to content. + - wrong: 'asset' + - good: 'asset_name' + + Args: + key (str): Key under which is key stored. + value (Any): Value that should be stored under the key. + """ + + plugin_data = self.builder.get_shared_data(self.identifier) + if plugin_data is None: + plugin_data = {} + plugin_data[key] = value + self.builder.set_shared_data(self.identifier, plugin_data) + + def get_plugin_shared_populate_data(self, key): + """Receive shared data across plugin and placeholders. + + Using shared populate data from builder but stored under plugin + identifier. + + Shared populate data are cleaned up during populate while loop. + + Args: + key (str): Key under which are shared data stored. + + Returns: + Union[None, Any]: None if key was not set. + """ + + plugin_data = self.builder.get_shared_populate_data(self.identifier) + if plugin_data is None: + return None + return plugin_data.get(key) + + def set_plugin_shared_populate_data(self, key, value): + """Store share data across plugin and placeholders. + + Using shared data from builder but stored under plugin identifier. + + Key should be self explanatory to content. + - wrong: 'asset' + - good: 'asset_name' + + Shared populate data are cleaned up during populate while loop. + + Args: + key (str): Key under which is key stored. + value (Any): Value that should be stored under the key. + """ + + plugin_data = self.builder.get_shared_populate_data(self.identifier) + if plugin_data is None: + plugin_data = {} + plugin_data[key] = value + self.builder.set_shared_populate_data(self.identifier, plugin_data) + + +class PlaceholderItem(object): + """Item representing single item in scene that is a placeholder to process. + + Items are always created and updated by their plugins. Each plugin can use + modified class of 'PlacehoderItem' but only to add more options instead of + new other. + + Scene identifier is used to avoid processing of the palceholder item + multiple times so must be unique across whole workfile builder. + + Args: + scene_identifier (str): Unique scene identifier. If placeholder is + created from the same "node" it must have same identifier. + data (Dict[str, Any]): Data related to placeholder. They're defined + by plugin. + plugin (PlaceholderPlugin): Plugin which created the placeholder item. + """ + + default_order = 100 + + def __init__(self, scene_identifier, data, plugin): + self._log = None + self._scene_identifier = scene_identifier + self._data = data + self._plugin = plugin + + # Keep track about state of Placeholder process + self._state = 0 + + # Error messages to be shown in UI + # - all other messages should be logged + self._errors = [] # -> List[str] + + @property + def plugin(self): + """Access to plugin which created placeholder. + + Returns: + PlaceholderPlugin: Plugin object. + """ + + return self._plugin + + @property + def builder(self): + """Access to builder. + + Returns: + AbstractTemplateBuilder: Builder which is the top part of + placeholder. + """ + + return self.plugin.builder + + @property + def data(self): + """Placeholder data which can modify how placeholder is processed. + + Possible general keys + - order: Can define the order in which is palceholder processed. + Lower == earlier. + + Other keys are defined by placeholder and should validate them on item + creation. + + Returns: + Dict[str, Any]: Placeholder item data. + """ + + return self._data + + def to_dict(self): + """Create copy of item's data. + + Returns: + Dict[str, Any]: Placeholder data. + """ + + return copy.deepcopy(self.data) + + @property + def log(self): + if self._log is None: + self._log = Logger.get_logger(repr(self)) + return self._log + + def __repr__(self): + return "< {} {} >".format(self.__class__.__name__, self.name) + + @property + def order(self): + """Order of item processing.""" + + order = self._data.get("order") + if order is None: + return self.default_order + return order + + @property + def scene_identifier(self): + return self._scene_identifier + + @property + def finished(self): + """Item was already processed.""" + + return self._state == 2 + + @property + def in_progress(self): + """Processing is in progress.""" + + return self._state == 1 + + def set_in_progress(self): + """Change to in progress state.""" + + self._state = 1 + + def set_finished(self): + """Change to finished state.""" + + self._state = 2 + + def set_failed(self, exception): + self.add_error(str(exception)) + + def add_error(self, error): + """Set placeholder item as failed and mark it as finished.""" + + self._errors.append(error) + + def get_errors(self): + """Exception with which the placeholder process failed. + + Gives ability to access the exception. + """ + + return self._errors + + +class PlaceholderLoadMixin(object): + """Mixin prepared for loading placeholder plugins. + + Implementation prepares options for placeholders with + 'get_load_plugin_options'. + + For placeholder population is implemented 'populate_load_placeholder'. + + PlaceholderItem can have implemented methods: + - 'load_failed' - called when loading of one representation failed + - 'load_succeed' - called when loading of one representation succeeded + """ + + def get_load_plugin_options(self, options=None): + """Unified attribute definitions for load placeholder. + + Common function for placeholder plugins used for loading of + repsentations. Use it in 'get_placeholder_options'. + + Args: + plugin (PlaceholderPlugin): Plugin used for loading of + representations. + options (Dict[str, Any]): Already available options which are used + as defaults for attributes. + + Returns: + List[AbtractAttrDef]: Attribute definitions common for load + plugins. + """ + + loaders_by_name = self.builder.get_loaders_by_name() + loader_items = [ + (loader_name, loader.label or loader_name) + for loader_name, loader in loaders_by_name.items() + ] + + loader_items = list(sorted(loader_items, key=lambda i: i[1])) + options = options or {} + return [ + attribute_definitions.UISeparatorDef(), + attribute_definitions.UILabelDef("Main attributes"), + attribute_definitions.UISeparatorDef(), + + attribute_definitions.EnumDef( + "builder_type", + label="Asset Builder Type", + default=options.get("builder_type"), + items=[ + ("context_asset", "Current asset"), + ("linked_asset", "Linked assets"), + ("all_assets", "All assets") + ], + tooltip=( + "Asset Builder Type\n" + "\nBuilder type describe what template loader will look" + " for." + "\ncontext_asset : Template loader will look for subsets" + " of current context asset (Asset bob will find asset)" + "\nlinked_asset : Template loader will look for assets" + " linked to current context asset." + "\nLinked asset are looked in database under" + " field \"inputLinks\"" + ) + ), + attribute_definitions.TextDef( + "family", + label="Family", + default=options.get("family"), + placeholder="model, look, ..." + ), + attribute_definitions.TextDef( + "representation", + label="Representation name", + default=options.get("representation"), + placeholder="ma, abc, ..." + ), + attribute_definitions.EnumDef( + "loader", + label="Loader", + default=options.get("loader"), + items=loader_items, + tooltip=( + "Loader" + "\nDefines what OpenPype loader will be used to" + " load assets." + "\nUseable loader depends on current host's loader list." + "\nField is case sensitive." + ) + ), + attribute_definitions.TextDef( + "loader_args", + label="Loader Arguments", + default=options.get("loader_args"), + placeholder='{"camera":"persp", "lights":True}', + tooltip=( + "Loader" + "\nDefines a dictionnary of arguments used to load assets." + "\nUseable arguments depend on current placeholder Loader." + "\nField should be a valid python dict." + " Anything else will be ignored." + ) + ), + attribute_definitions.NumberDef( + "order", + label="Order", + default=options.get("order") or 0, + decimals=0, + minimum=0, + maximum=999, + tooltip=( + "Order" + "\nOrder defines asset loading priority (0 to 999)" + "\nPriority rule is : \"lowest is first to load\"." + ) + ), + attribute_definitions.UISeparatorDef(), + attribute_definitions.UILabelDef("Optional attributes"), + attribute_definitions.UISeparatorDef(), + attribute_definitions.TextDef( + "asset", + label="Asset filter", + default=options.get("asset"), + placeholder="regex filtering by asset name", + tooltip=( + "Filtering assets by matching field regex to asset's name" + ) + ), + attribute_definitions.TextDef( + "subset", + label="Subset filter", + default=options.get("subset"), + placeholder="regex filtering by subset name", + tooltip=( + "Filtering assets by matching field regex to subset's name" + ) + ), + attribute_definitions.TextDef( + "hierarchy", + label="Hierarchy filter", + default=options.get("hierarchy"), + placeholder="regex filtering by asset's hierarchy", + tooltip=( + "Filtering assets by matching field asset's hierarchy" + ) + ) + ] + + def parse_loader_args(self, loader_args): + """Helper function to parse string of loader arugments. + + Empty dictionary is returned if conversion fails. + + Args: + loader_args (str): Loader args filled by user. + + Returns: + Dict[str, Any]: Parsed arguments used as dictionary. + """ + + if not loader_args: + return {} + + try: + parsed_args = eval(loader_args) + if isinstance(parsed_args, dict): + return parsed_args + + except Exception as err: + print( + "Error while parsing loader arguments '{}'.\n{}: {}\n\n" + "Continuing with default arguments. . .".format( + loader_args, err.__class__.__name__, err)) + + return {} + + def _get_representations(self, placeholder): + """Prepared query of representations based on load options. + + This function is directly connected to options defined in + 'get_load_plugin_options'. + + Note: + This returns all representation documents from all versions of + matching subset. To filter for last version use + '_reduce_last_version_repre_docs'. + + Args: + placeholder (PlaceholderItem): Item which should be populated. + + Returns: + List[Dict[str, Any]]: Representation documents matching filters + from placeholder data. + """ + + project_name = self.builder.project_name + current_asset_doc = self.builder.current_asset_doc + linked_asset_docs = self.builder.linked_asset_docs + + builder_type = placeholder.data["builder_type"] + if builder_type == "context_asset": + context_filters = { + "asset": [current_asset_doc["name"]], + "subset": [re.compile(placeholder.data["subset"])], + "hierarchy": [re.compile(placeholder.data["hierarchy"])], + "representation": [placeholder.data["representation"]], + "family": [placeholder.data["family"]] + } + + elif builder_type != "linked_asset": + context_filters = { + "asset": [re.compile(placeholder.data["asset"])], + "subset": [re.compile(placeholder.data["subset"])], + "hierarchy": [re.compile(placeholder.data["hierarchy"])], + "representation": [placeholder.data["representation"]], + "family": [placeholder.data["family"]] + } + + else: + asset_regex = re.compile(placeholder.data["asset"]) + linked_asset_names = [] + for asset_doc in linked_asset_docs: + asset_name = asset_doc["name"] + if asset_regex.match(asset_name): + linked_asset_names.append(asset_name) + + context_filters = { + "asset": linked_asset_names, + "subset": [re.compile(placeholder.data["subset"])], + "hierarchy": [re.compile(placeholder.data["hierarchy"])], + "representation": [placeholder.data["representation"]], + "family": [placeholder.data["family"]], + } + + return list(get_representations( + project_name, + context_filters=context_filters + )) + + def _before_repre_load(self, placeholder, representation): + """Can be overriden. Is called before representation is loaded.""" + + pass + + def _reduce_last_version_repre_docs(self, representations): + """Reduce representations to last verison.""" + + mapping = {} + for repre_doc in representations: + repre_context = repre_doc["context"] + + asset_name = repre_context["asset"] + subset_name = repre_context["subset"] + version = repre_context.get("version", -1) + + if asset_name not in mapping: + mapping[asset_name] = {} + + subset_mapping = mapping[asset_name] + if subset_name not in subset_mapping: + subset_mapping[subset_name] = collections.defaultdict(list) + + version_mapping = subset_mapping[subset_name] + version_mapping[version].append(repre_doc) + + output = [] + for subset_mapping in mapping.values(): + for version_mapping in subset_mapping.values(): + last_version = tuple(sorted(version_mapping.keys()))[-1] + output.extend(version_mapping[last_version]) + return output + + def populate_load_placeholder(self, placeholder, ignore_repre_ids=None): + """Load placeholder is goind to load matching representations. + + Note: + Ignore repre ids is to avoid loading the same representation again + on load. But the representation can be loaded with different loader + and there could be published new version of matching subset for the + representation. We should maybe expect containers. + + Also import loaders don't have containers at all... + + Args: + placeholder (PlaceholderItem): Placeholder item with information + about requested representations. + ignore_repre_ids (Iterable[Union[str, ObjectId]]): Representation + ids that should be skipped. + """ + + if ignore_repre_ids is None: + ignore_repre_ids = set() + + # TODO check loader existence + loader_name = placeholder.data["loader"] + loader_args = placeholder.data["loader_args"] + + placeholder_representations = self._get_representations(placeholder) + + filtered_representations = [] + for representation in self._reduce_last_version_repre_docs( + placeholder_representations + ): + repre_id = str(representation["_id"]) + if repre_id not in ignore_repre_ids: + filtered_representations.append(representation) + + if not filtered_representations: + self.log.info(( + "There's no representation for this placeholder: {}" + ).format(placeholder.scene_identifier)) + return + + repre_load_contexts = get_contexts_for_repre_docs( + self.project_name, filtered_representations + ) + loaders_by_name = self.builder.get_loaders_by_name() + for repre_load_context in repre_load_contexts.values(): + representation = repre_load_context["representation"] + repre_context = representation["context"] + self._before_repre_load( + placeholder, representation + ) + self.log.info( + "Loading {} from {} with loader {}\n" + "Loader arguments used : {}".format( + repre_context["subset"], + repre_context["asset"], + loader_name, + loader_args + ) + ) + try: + container = load_with_repre_context( + loaders_by_name[loader_name], + repre_load_context, + options=self.parse_loader_args(loader_args) + ) + + except Exception: + failed = True + self.load_failed(placeholder, representation) + + else: + failed = False + self.load_succeed(placeholder, container) + self.cleanup_placeholder(placeholder, failed) + + def load_failed(self, placeholder, representation): + if hasattr(placeholder, "load_failed"): + placeholder.load_failed(representation) + + def load_succeed(self, placeholder, container): + if hasattr(placeholder, "load_succeed"): + placeholder.load_succeed(container) + + def cleanup_placeholder(self, placeholder, failed): + """Cleanup placeholder after load of single representation. + + Can be called multiple times during placeholder item populating and is + called even if loading failed. + + Args: + placeholder (PlaceholderItem): Item which was just used to load + representation. + failed (bool): Loading of representation failed. + """ + + pass + + +class LoadPlaceholderItem(PlaceholderItem): + """PlaceholderItem for plugin which is loading representations. + + Connected to 'PlaceholderLoadMixin'. + """ + + def __init__(self, *args, **kwargs): + super(LoadPlaceholderItem, self).__init__(*args, **kwargs) + self._failed_representations = [] + + def get_errors(self): + if not self._failed_representations: + return [] + message = ( + "Failed to load {} representations using Loader {}" + ).format( + len(self._failed_representations), + self.data["loader"] + ) + return [message] + + def load_failed(self, representation): + self._failed_representations.append(representation) diff --git a/openpype/plugins/load/add_site.py b/openpype/plugins/load/add_site.py index 55fda55d17..ac931e41db 100644 --- a/openpype/plugins/load/add_site.py +++ b/openpype/plugins/load/add_site.py @@ -1,6 +1,6 @@ +from openpype.client import get_linked_representation_id from openpype.modules import ModulesManager from openpype.pipeline import load -from openpype.lib.avalon_context import get_linked_ids_for_representations from openpype.modules.sync_server.utils import SiteAlreadyPresentError @@ -45,9 +45,11 @@ class AddSyncSite(load.LoaderPlugin): force=True) if family == "workfile": - links = get_linked_ids_for_representations(project_name, - [repre_id], - link_type="reference") + links = get_linked_representation_id( + project_name, + repre_id=repre_id, + link_type="reference" + ) for link_repre_id in links: try: self.sync_server.add_site(project_name, link_repre_id, diff --git a/openpype/plugins/publish/collect_audio.py b/openpype/plugins/publish/collect_audio.py new file mode 100644 index 0000000000..7d53b24e54 --- /dev/null +++ b/openpype/plugins/publish/collect_audio.py @@ -0,0 +1,105 @@ +import pyblish.api + +from openpype.client import ( + get_last_version_by_subset_name, + get_representations, +) +from openpype.pipeline import ( + legacy_io, + get_representation_path, +) + + +class CollectAudio(pyblish.api.InstancePlugin): + """Collect asset's last published audio. + + The audio subset name searched for is defined in: + project settings > Collect Audio + """ + label = "Collect Asset Audio" + order = pyblish.api.CollectorOrder + 0.1 + families = ["review"] + hosts = [ + "nuke", + "maya", + "shell", + "hiero", + "premiere", + "harmony", + "traypublisher", + "standalonepublisher", + "fusion", + "tvpaint", + "resolve", + "webpublisher", + "aftereffects", + "flame", + "unreal" + ] + + audio_subset_name = "audioMain" + + def process(self, instance): + if instance.data.get("audio"): + self.log.info( + "Skipping Audio collecion. It is already collected" + ) + return + + # Add audio to instance if exists. + self.log.info(( + "Searching for audio subset '{subset}'" + " in asset '{asset}'" + ).format( + subset=self.audio_subset_name, + asset=instance.data["asset"] + )) + + repre_doc = self._get_repre_doc(instance) + + # Add audio to instance if representation was found + if repre_doc: + instance.data["audio"] = [{ + "offset": 0, + "filename": get_representation_path(repre_doc) + }] + self.log.info("Audio Data added to instance ...") + + def _get_repre_doc(self, instance): + cache = instance.context.data.get("__cache_asset_audio") + if cache is None: + cache = {} + instance.context.data["__cache_asset_audio"] = cache + asset_name = instance.data["asset"] + + # first try to get it from cache + if asset_name in cache: + return cache[asset_name] + + project_name = legacy_io.active_project() + + # Find latest versions document + last_version_doc = get_last_version_by_subset_name( + project_name, + self.audio_subset_name, + asset_name=asset_name, + fields=["_id"] + ) + + repre_doc = None + if last_version_doc: + # Try to find it's representation (Expected there is only one) + repre_docs = list(get_representations( + project_name, version_ids=[last_version_doc["_id"]] + )) + if not repre_docs: + self.log.warning( + "Version document does not contain any representations" + ) + else: + repre_doc = repre_docs[0] + + # update cache + cache[asset_name] = repre_doc + + return repre_doc diff --git a/openpype/plugins/publish/collect_from_create_context.py b/openpype/plugins/publish/collect_from_create_context.py index 9236c698ed..fc0f97b187 100644 --- a/openpype/plugins/publish/collect_from_create_context.py +++ b/openpype/plugins/publish/collect_from_create_context.py @@ -25,7 +25,9 @@ class CollectFromCreateContext(pyblish.api.ContextPlugin): for created_instance in create_context.instances: instance_data = created_instance.data_to_store() if instance_data["active"]: - self.create_instance(context, instance_data) + self.create_instance( + context, instance_data, created_instance.transient_data + ) # Update global data to context context.data.update(create_context.context_data_to_store()) @@ -37,7 +39,7 @@ class CollectFromCreateContext(pyblish.api.ContextPlugin): legacy_io.Session[key] = value os.environ[key] = value - def create_instance(self, context, in_data): + def create_instance(self, context, in_data, transient_data): subset = in_data["subset"] # If instance data already contain families then use it instance_families = in_data.get("families") or [] @@ -56,5 +58,8 @@ class CollectFromCreateContext(pyblish.api.ContextPlugin): for key, value in in_data.items(): if key not in instance.data: instance.data[key] = value + + instance.data["transientData"] = transient_data + self.log.info("collected instance: {}".format(instance.data)) self.log.info("parsing data: {}".format(in_data)) diff --git a/openpype/plugins/publish/collect_input_representations_to_versions.py b/openpype/plugins/publish/collect_input_representations_to_versions.py new file mode 100644 index 0000000000..18a19bce80 --- /dev/null +++ b/openpype/plugins/publish/collect_input_representations_to_versions.py @@ -0,0 +1,47 @@ +import pyblish.api + +from bson.objectid import ObjectId + +from openpype.client import get_representations + + +class CollectInputRepresentationsToVersions(pyblish.api.ContextPlugin): + """Converts collected input representations to input versions. + + Any data in `instance.data["inputRepresentations"]` gets converted into + `instance.data["inputVersions"]` as supported in OpenPype v3. + + """ + # This is a ContextPlugin because then we can query the database only once + # for the conversion of representation ids to version ids (optimization) + label = "Input Representations to Versions" + order = pyblish.api.CollectorOrder + 0.499 + hosts = ["*"] + + def process(self, context): + # Query all version ids for representation ids from the database once + representations = set() + for instance in context: + inst_repre = instance.data.get("inputRepresentations", []) + representations.update(inst_repre) + + representations_docs = get_representations( + project_name=context.data["projectEntity"]["name"], + representation_ids=representations, + fields=["_id", "parent"]) + + representation_id_to_version_id = { + repre["_id"]: repre["parent"] for repre in representations_docs + } + + for instance in context: + inst_repre = instance.data.get("inputRepresentations", []) + if not inst_repre: + continue + + input_versions = instance.data.get("inputVersions", []) + for repre_id in inst_repre: + repre_id = ObjectId(repre_id) + version_id = representation_id_to_version_id[repre_id] + input_versions.append(version_id) + instance.data["inputVersions"] = input_versions diff --git a/openpype/plugins/publish/collect_otio_frame_ranges.py b/openpype/plugins/publish/collect_otio_frame_ranges.py index 40e89e29bc..9a68b6e43d 100644 --- a/openpype/plugins/publish/collect_otio_frame_ranges.py +++ b/openpype/plugins/publish/collect_otio_frame_ranges.py @@ -10,6 +10,7 @@ import opentimelineio as otio import pyblish.api from pprint import pformat from openpype.pipeline.editorial import ( + get_media_range_with_retimes, otio_range_to_frame_range, otio_range_with_handles ) @@ -29,6 +30,7 @@ class CollectOtioFrameRanges(pyblish.api.InstancePlugin): # get basic variables otio_clip = instance.data["otioClip"] workfile_start = instance.data["workfileFrameStart"] + workfile_source_duration = instance.data.get("shotDurationFromSource") # get ranges otio_tl_range = otio_clip.range_in_parent() @@ -54,6 +56,18 @@ class CollectOtioFrameRanges(pyblish.api.InstancePlugin): frame_end = frame_start + otio.opentime.to_frames( otio_tl_range.duration, otio_tl_range.duration.rate) - 1 + # in case of retimed clip and frame range should not be retimed + if workfile_source_duration: + # get available range trimmed with processed retimes + retimed_attributes = get_media_range_with_retimes( + otio_clip, 0, 0) + self.log.debug( + ">> retimed_attributes: {}".format(retimed_attributes)) + media_in = int(retimed_attributes["mediaIn"]) + media_out = int(retimed_attributes["mediaOut"]) + frame_end = frame_start + (media_out - media_in) + 1 + self.log.debug(frame_end) + data = { "frameStart": frame_start, "frameEnd": frame_end, diff --git a/openpype/plugins/publish/extract_burnin.py b/openpype/plugins/publish/extract_burnin.py index 8ddee162e3..4179199317 100644 --- a/openpype/plugins/publish/extract_burnin.py +++ b/openpype/plugins/publish/extract_burnin.py @@ -488,12 +488,6 @@ class ExtractBurnin(publish.Extractor): "frame_end_handle": frame_end_handle } - # use explicit username for webpublishes as rewriting - # OPENPYPE_USERNAME might have side effects - webpublish_user_name = os.environ.get("WEBPUBLISH_OPENPYPE_USERNAME") - if webpublish_user_name: - burnin_data["username"] = webpublish_user_name - self.log.debug( "Basic burnin_data: {}".format(json.dumps(burnin_data, indent=4)) ) diff --git a/openpype/plugins/publish/extract_otio_file.py b/openpype/plugins/publish/extract_otio_file.py index c692205d81..1a6a82117d 100644 --- a/openpype/plugins/publish/extract_otio_file.py +++ b/openpype/plugins/publish/extract_otio_file.py @@ -16,6 +16,8 @@ class ExtractOTIOFile(publish.Extractor): hosts = ["resolve", "hiero", "traypublisher"] def process(self, instance): + if not instance.context.data.get("otioTimeline"): + return # create representation data if "representations" not in instance.data: instance.data["representations"] = [] diff --git a/openpype/plugins/publish/integrate.py b/openpype/plugins/publish/integrate.py index f99c718f8a..8972e6ab70 100644 --- a/openpype/plugins/publish/integrate.py +++ b/openpype/plugins/publish/integrate.py @@ -5,6 +5,9 @@ import copy import clique import six +from bson.objectid import ObjectId +import pyblish.api + from openpype.client.operations import ( OperationsSession, new_subset_document, @@ -14,8 +17,6 @@ from openpype.client.operations import ( prepare_version_update_data, prepare_representation_update_data, ) -from bson.objectid import ObjectId -import pyblish.api from openpype.client import ( get_representations, @@ -23,10 +24,12 @@ from openpype.client import ( get_version_by_name, ) from openpype.lib import source_hash -from openpype.lib.profiles_filtering import filter_profiles from openpype.lib.file_transaction import FileTransaction from openpype.pipeline import legacy_io -from openpype.pipeline.publish import KnownPublishError +from openpype.pipeline.publish import ( + KnownPublishError, + get_publish_template_name, +) log = logging.getLogger(__name__) @@ -135,7 +138,7 @@ class IntegrateAsset(pyblish.api.InstancePlugin): # the database even if not used by the destination template db_representation_context_keys = [ "project", "asset", "task", "subset", "version", "representation", - "family", "hierarchy", "username", "output" + "family", "hierarchy", "username", "user", "output" ] skip_host_families = [] @@ -792,52 +795,26 @@ class IntegrateAsset(pyblish.api.InstancePlugin): def get_template_name(self, instance): """Return anatomy template name to use for integration""" - # Define publish template name from profiles - filter_criteria = self.get_profile_filter_criteria(instance) - template_name_profiles = self._get_template_name_profiles(instance) - profile = filter_profiles( - template_name_profiles, - filter_criteria, - logger=self.log - ) - - if profile: - return profile["template_name"] - return self.default_template_name - - def _get_template_name_profiles(self, instance): - """Receive profiles for publish template keys. - - Reuse template name profiles from legacy integrator. Goal is to move - the profile settings out of plugin settings but until that happens we - want to be able set it at one place and don't break backwards - compatibility (more then once). - """ - - return ( - instance.context.data["project_settings"] - ["global"] - ["publish"] - ["IntegrateAssetNew"] - ["template_name_profiles"] - ) - - def get_profile_filter_criteria(self, instance): - """Return filter criteria for `filter_profiles`""" # Anatomy data is pre-filled by Collectors - anatomy_data = instance.data["anatomyData"] + + project_name = legacy_io.active_project() # Task can be optional in anatomy data - task = anatomy_data.get("task", {}) + host_name = instance.context.data["hostName"] + anatomy_data = instance.data["anatomyData"] + family = anatomy_data["family"] + task_info = anatomy_data.get("task") or {} - # Return filter criteria - return { - "families": anatomy_data["family"], - "tasks": task.get("name"), - "task_types": task.get("type"), - "hosts": instance.context.data["hostName"], - } + return get_publish_template_name( + project_name, + host_name, + family, + task_name=task_info.get("name"), + task_type=task_info.get("type"), + project_settings=instance.context.data["project_settings"], + logger=self.log + ) def get_rootless_path(self, anatomy, path): """Returns, if possible, path without absolute portion from root diff --git a/openpype/plugins/publish/integrate_hero_version.py b/openpype/plugins/publish/integrate_hero_version.py index 7d698ff98d..c0760a5471 100644 --- a/openpype/plugins/publish/integrate_hero_version.py +++ b/openpype/plugins/publish/integrate_hero_version.py @@ -14,14 +14,12 @@ from openpype.client import ( get_archived_representations, get_representations, ) -from openpype.lib import ( - create_hard_link, - filter_profiles -) +from openpype.lib import create_hard_link from openpype.pipeline import ( schema, legacy_io, ) +from openpype.pipeline.publish import get_publish_template_name class IntegrateHeroVersion(pyblish.api.InstancePlugin): @@ -46,7 +44,7 @@ class IntegrateHeroVersion(pyblish.api.InstancePlugin): ignored_representation_names = [] db_representation_context_keys = [ "project", "asset", "task", "subset", "representation", - "family", "hierarchy", "task", "username" + "family", "hierarchy", "task", "username", "user" ] # QUESTION/TODO this process should happen on server if crashed due to # permissions error on files (files were used or user didn't have perms) @@ -68,10 +66,11 @@ class IntegrateHeroVersion(pyblish.api.InstancePlugin): ) return - template_key = self._get_template_key(instance) - anatomy = instance.context.data["anatomy"] project_name = anatomy.project_name + + template_key = self._get_template_key(project_name, instance) + if template_key not in anatomy.templates: self.log.warning(( "!!! Anatomy of project \"{}\" does not have set" @@ -527,30 +526,24 @@ class IntegrateHeroVersion(pyblish.api.InstancePlugin): return publish_folder - def _get_template_key(self, instance): + def _get_template_key(self, project_name, instance): anatomy_data = instance.data["anatomyData"] - task_data = anatomy_data.get("task") or {} - task_name = task_data.get("name") - task_type = task_data.get("type") + task_info = anatomy_data.get("task") or {} host_name = instance.context.data["hostName"] + # TODO raise error if Hero not set? family = self.main_family_from_instance(instance) - key_values = { - "families": family, - "task_names": task_name, - "task_types": task_type, - "hosts": host_name - } - profile = filter_profiles( - self.template_name_profiles, - key_values, + + return get_publish_template_name( + project_name, + host_name, + family, + task_info.get("name"), + task_info.get("type"), + project_settings=instance.context.data["project_settings"], + hero=True, logger=self.log ) - if profile: - template_name = profile["template_name"] - else: - template_name = self._default_template_name - return template_name def main_family_from_instance(self, instance): """Returns main family of entered instance.""" @@ -584,8 +577,11 @@ class IntegrateHeroVersion(pyblish.api.InstancePlugin): return except OSError as exc: - # re-raise exception if different than cross drive path - if exc.errno != errno.EXDEV: + # re-raise exception if different than + # EXDEV - cross drive path + # EINVAL - wrong format, must be NTFS + self.log.debug("Hardlink failed with errno:'{}'".format(exc.errno)) + if exc.errno not in [errno.EXDEV, errno.EINVAL]: raise shutil.copy(src_path, dst_path) diff --git a/openpype/plugins/publish/integrate_legacy.py b/openpype/plugins/publish/integrate_legacy.py index b90b61f587..536ab83f2c 100644 --- a/openpype/plugins/publish/integrate_legacy.py +++ b/openpype/plugins/publish/integrate_legacy.py @@ -15,7 +15,6 @@ from bson.objectid import ObjectId from pymongo import DeleteOne, InsertOne import pyblish.api -import openpype.api from openpype.client import ( get_asset_by_name, get_subset_by_id, @@ -25,14 +24,17 @@ from openpype.client import ( get_representations, get_archived_representations, ) -from openpype.lib.profiles_filtering import filter_profiles from openpype.lib import ( prepare_template_data, create_hard_link, StringTemplate, - TemplateUnsolved + TemplateUnsolved, + source_hash, + filter_profiles, + get_local_site_id, ) from openpype.pipeline import legacy_io +from openpype.pipeline.publish import get_publish_template_name # this is needed until speedcopy for linux is fixed if sys.platform == "win32": @@ -127,7 +129,7 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): exclude_families = ["render.farm"] db_representation_context_keys = [ "project", "asset", "task", "subset", "version", "representation", - "family", "hierarchy", "task", "username" + "family", "hierarchy", "task", "username", "user" ] default_template_name = "publish" @@ -138,7 +140,6 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): integrated_file_sizes = {} # Attributes set by settings - template_name_profiles = None subset_grouping_profiles = None def process(self, instance): @@ -388,22 +389,16 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): family = self.main_family_from_instance(instance) - key_values = { - "families": family, - "tasks": task_name, - "hosts": instance.context.data["hostName"], - "task_types": task_type - } - profile = filter_profiles( - self.template_name_profiles, - key_values, + template_name = get_publish_template_name( + project_name, + instance.context.data["hostName"], + family, + task_name=task_info.get("name"), + task_type=task_info.get("type"), + project_settings=instance.context.data["project_settings"], logger=self.log ) - template_name = "publish" - if profile: - template_name = profile["template_name"] - published_representations = {} for idx, repre in enumerate(repres): published_files = [] @@ -1058,7 +1053,7 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): for _src, dest in resources: path = self.get_rootless_path(anatomy, dest) dest = self.get_dest_temp_url(dest) - file_hash = openpype.api.source_hash(dest) + file_hash = source_hash(dest) if self.TMP_FILE_EXT and \ ',{}'.format(self.TMP_FILE_EXT) in file_hash: file_hash = file_hash.replace(',{}'.format(self.TMP_FILE_EXT), @@ -1168,7 +1163,7 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): def _get_sites(self, sync_project_presets): """Returns tuple (local_site, remote_site)""" - local_site_id = openpype.api.get_local_site_id() + local_site_id = get_local_site_id() local_site = sync_project_presets["config"]. \ get("active_site", "studio").strip() diff --git a/openpype/pype_commands.py b/openpype/pype_commands.py index 85561495fd..d08a812c61 100644 --- a/openpype/pype_commands.py +++ b/openpype/pype_commands.py @@ -187,7 +187,7 @@ class PypeCommands: (to choose validator for example) """ - from openpype.hosts.webpublisher.cli_functions import ( + from openpype.hosts.webpublisher.publish_functions import ( cli_publish_from_app ) @@ -218,7 +218,7 @@ class PypeCommands: RuntimeError: When there is no path to process. """ - from openpype.hosts.webpublisher.cli_functions import ( + from openpype.hosts.webpublisher.publish_functions import ( cli_publish ) diff --git a/openpype/settings/defaults/project_settings/blender.json b/openpype/settings/defaults/project_settings/blender.json index 2720e0286d..7acecfaae0 100644 --- a/openpype/settings/defaults/project_settings/blender.json +++ b/openpype/settings/defaults/project_settings/blender.json @@ -36,35 +36,35 @@ "layout" ] }, - "ExtractBlendAnimation": { - "enabled": true, - "optional": true, - "active": true - }, - "ExtractCamera": { - "enabled": true, - "optional": true, - "active": true - }, "ExtractFBX": { "enabled": true, "optional": true, "active": false }, - "ExtractAnimationFBX": { - "enabled": true, - "optional": true, - "active": false - }, "ExtractABC": { "enabled": true, "optional": true, "active": false }, + "ExtractBlendAnimation": { + "enabled": true, + "optional": true, + "active": true + }, + "ExtractAnimationFBX": { + "enabled": true, + "optional": true, + "active": false + }, + "ExtractCamera": { + "enabled": true, + "optional": true, + "active": true + }, "ExtractLayout": { "enabled": true, "optional": true, "active": false } } -} +} \ No newline at end of file diff --git a/openpype/settings/defaults/project_settings/flame.json b/openpype/settings/defaults/project_settings/flame.json index bfdc58d9ee..c90193fe13 100644 --- a/openpype/settings/defaults/project_settings/flame.json +++ b/openpype/settings/defaults/project_settings/flame.json @@ -17,7 +17,9 @@ "workfileFrameStart": 1001, "handleStart": 5, "handleEnd": 5, - "includeHandles": false + "includeHandles": false, + "retimedHandles": true, + "retimedFramerange": true } }, "publish": { diff --git a/openpype/settings/defaults/project_settings/ftrack.json b/openpype/settings/defaults/project_settings/ftrack.json index 09b194e21c..cdf861df4a 100644 --- a/openpype/settings/defaults/project_settings/ftrack.json +++ b/openpype/settings/defaults/project_settings/ftrack.json @@ -455,7 +455,7 @@ "family_mapping": { "camera": "cam", "look": "look", - "mayaascii": "scene", + "mayaAscii": "scene", "model": "geo", "rig": "rig", "setdress": "setdress", diff --git a/openpype/settings/defaults/project_settings/fusion.json b/openpype/settings/defaults/project_settings/fusion.json new file mode 100644 index 0000000000..1b4c4c55b5 --- /dev/null +++ b/openpype/settings/defaults/project_settings/fusion.json @@ -0,0 +1,12 @@ +{ + "imageio": { + "ocio": { + "enabled": false, + "configFilePath": { + "windows": [], + "darwin": [], + "linux": [] + } + } + } +} \ No newline at end of file diff --git a/openpype/settings/defaults/project_settings/global.json b/openpype/settings/defaults/project_settings/global.json index 0ff9363ba7..115a719995 100644 --- a/openpype/settings/defaults/project_settings/global.json +++ b/openpype/settings/defaults/project_settings/global.json @@ -3,6 +3,10 @@ "CollectAnatomyInstanceData": { "follow_workfile_version": false }, + "CollectAudio": { + "enabled": false, + "audio_subset_name": "audioMain" + }, "CollectSceneVersion": { "hosts": [ "aftereffects", @@ -403,7 +407,8 @@ "enabled": false } ], - "extra_folders": [] + "extra_folders": [], + "workfile_lock_profiles": [] }, "loader": { "family_filter_profiles": [ @@ -414,6 +419,10 @@ "filter_families": [] } ] + }, + "publish": { + "template_name_profiles": [], + "hero_template_name_profiles": [] } }, "project_folder_structure": "{\"__project_root__\": {\"prod\": {}, \"resources\": {\"footage\": {\"plates\": {}, \"offline\": {}}, \"audio\": {}, \"art_dept\": {}}, \"editorial\": {}, \"assets\": {\"characters\": {}, \"locations\": {}}, \"shots\": {}}}", diff --git a/openpype/settings/defaults/project_settings/houdini.json b/openpype/settings/defaults/project_settings/houdini.json index b7d2104ba1..cdf829db57 100644 --- a/openpype/settings/defaults/project_settings/houdini.json +++ b/openpype/settings/defaults/project_settings/houdini.json @@ -1,4 +1,15 @@ { + "shelves": [ + { + "shelf_set_name": "OpenPype Shelves", + "shelf_set_source_path": { + "windows": "", + "darwin": "", + "linux": "" + }, + "shelf_definition": [] + } + ], "create": { "CreateArnoldAss": { "enabled": true, diff --git a/openpype/settings/defaults/project_settings/kitsu.json b/openpype/settings/defaults/project_settings/kitsu.json index ba02d8d259..3a9723b9c0 100644 --- a/openpype/settings/defaults/project_settings/kitsu.json +++ b/openpype/settings/defaults/project_settings/kitsu.json @@ -1,8 +1,4 @@ { - "entities_root": { - "assets": "Assets", - "shots": "Shots" - }, "entities_naming_pattern": { "episode": "E##", "sequence": "SQ##", diff --git a/openpype/settings/defaults/project_settings/maya.json b/openpype/settings/defaults/project_settings/maya.json index 38063bc2c1..76ef0a7338 100644 --- a/openpype/settings/defaults/project_settings/maya.json +++ b/openpype/settings/defaults/project_settings/maya.json @@ -1,4 +1,5 @@ { + "mel_workspace": "workspace -fr \"shaders\" \"renderData/shaders\";\nworkspace -fr \"images\" \"renders\";\nworkspace -fr \"particles\" \"particles\";\nworkspace -fr \"mayaAscii\" \"\";\nworkspace -fr \"mayaBinary\" \"\";\nworkspace -fr \"scene\" \"\";\nworkspace -fr \"alembicCache\" \"cache/alembic\";\nworkspace -fr \"renderData\" \"renderData\";\nworkspace -fr \"sourceImages\" \"sourceimages\";\nworkspace -fr \"fileCache\" \"cache/nCache\";\n", "ext_mapping": { "model": "ma", "mayaAscii": "ma", @@ -48,7 +49,7 @@ "vray_renderer": { "image_prefix": "maya///", "engine": "1", - "image_format": "png", + "image_format": "exr", "aov_list": [], "additional_options": [] }, @@ -56,7 +57,7 @@ "image_prefix": "maya///", "primary_gi_engine": "0", "secondary_gi_engine": "0", - "image_format": "iff", + "image_format": "exr", "multilayer_exr": true, "force_combine": true, "aov_list": [], @@ -677,25 +678,20 @@ "isolate_view": true, "off_screen": true }, - "PanZoom": { - "pan_zoom": true - }, "Renderer": { "rendererName": "vp2Renderer" }, "Resolution": { "width": 1920, - "height": 1080, - "percent": 1.0, - "mode": "Custom" + "height": 1080 }, "Viewport Options": { "override_viewport_options": true, "displayLights": "default", + "displayTextures": true, "textureMaxResolution": 1024, "renderDepthOfField": true, "shadows": true, - "textures": true, "twoSidedLighting": true, "lineAAEnable": true, "multiSample": 8, @@ -718,7 +714,6 @@ "motionBlurShutterOpenFraction": 0.2, "cameras": false, "clipGhosts": false, - "controlVertices": false, "deformers": false, "dimensions": false, "dynamicConstraints": false, @@ -730,8 +725,7 @@ "grid": false, "hairSystems": true, "handles": false, - "hud": false, - "hulls": false, + "headsUpDisplay": false, "ikHandles": false, "imagePlane": true, "joints": false, @@ -742,7 +736,9 @@ "nCloths": false, "nParticles": false, "nRigids": false, + "controlVertices": false, "nurbsCurves": false, + "hulls": false, "nurbsSurfaces": false, "particleInstancers": false, "pivots": false, @@ -750,7 +746,8 @@ "pluginShapes": false, "polymeshes": true, "strokes": false, - "subdivSurfaces": false + "subdivSurfaces": false, + "textures": false }, "Camera Options": { "displayGateMask": false, diff --git a/openpype/settings/defaults/project_settings/photoshop.json b/openpype/settings/defaults/project_settings/photoshop.json index 552c2c9cad..feda219179 100644 --- a/openpype/settings/defaults/project_settings/photoshop.json +++ b/openpype/settings/defaults/project_settings/photoshop.json @@ -15,6 +15,12 @@ "CollectInstances": { "flatten_subset_template": "" }, + "CollectReview": { + "publish": true + }, + "CollectVersion": { + "enabled": false + }, "ValidateContainers": { "enabled": true, "optional": true, diff --git a/openpype/settings/defaults/system_settings/applications.json b/openpype/settings/defaults/system_settings/applications.json index 30b0a5cbe3..c37c3d299e 100644 --- a/openpype/settings/defaults/system_settings/applications.json +++ b/openpype/settings/defaults/system_settings/applications.json @@ -706,30 +706,28 @@ "icon": "{}/app_icons/fusion.png", "host_name": "fusion", "environment": { - "FUSION_UTILITY_SCRIPTS_SOURCE_DIR": [], - "FUSION_UTILITY_SCRIPTS_DIR": { - "windows": "{PROGRAMDATA}/Blackmagic Design/Fusion/Scripts/Comp", - "darwin": "/Library/Application Support/Blackmagic Design/Fusion/Scripts/Comp", - "linux": "/opt/Fusion/Scripts/Comp" - }, - "PYTHON36": { + "FUSION_PYTHON3_HOME": { "windows": "{LOCALAPPDATA}/Programs/Python/Python36", "darwin": "~/Library/Python/3.6/bin", "linux": "/opt/Python/3.6/bin" - }, - "PYTHONPATH": [ - "{PYTHON36}/Lib/site-packages", - "{VIRTUAL_ENV}/Lib/site-packages", - "{PYTHONPATH}" - ], - "PATH": [ - "{PYTHON36}", - "{PYTHON36}/Scripts", - "{PATH}" - ], - "OPENPYPE_LOG_NO_COLORS": "Yes" + } }, "variants": { + "18": { + "executables": { + "windows": [ + "C:\\Program Files\\Blackmagic Design\\Fusion 18\\Fusion.exe" + ], + "darwin": [], + "linux": [] + }, + "arguments": { + "windows": [], + "darwin": [], + "linux": [] + }, + "environment": {} + }, "17": { "executables": { "windows": [ diff --git a/openpype/settings/entities/schemas/projects_schema/schema_main.json b/openpype/settings/entities/schemas/projects_schema/schema_main.json index 80b1baad1b..0b9fbf7470 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_main.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_main.json @@ -90,6 +90,10 @@ "type": "schema", "name": "schema_project_nuke" }, + { + "type": "schema", + "name": "schema_project_fusion" + }, { "type": "schema", "name": "schema_project_hiero" diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_flame.json b/openpype/settings/entities/schemas/projects_schema/schema_project_flame.json index ca62679b3d..5f05bef0e1 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_flame.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_flame.json @@ -128,6 +128,16 @@ "type": "boolean", "key": "includeHandles", "label": "Enable handles including" + }, + { + "type": "boolean", + "key": "retimedHandles", + "label": "Enable retimed handles" + }, + { + "type": "boolean", + "key": "retimedFramerange", + "label": "Enable retimed shot frameranges" } ] } diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_fusion.json b/openpype/settings/entities/schemas/projects_schema/schema_project_fusion.json new file mode 100644 index 0000000000..8f98a8173f --- /dev/null +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_fusion.json @@ -0,0 +1,38 @@ +{ + "type": "dict", + "collapsible": true, + "key": "fusion", + "label": "Fusion", + "is_file": true, + "children": [ + { + "key": "imageio", + "type": "dict", + "label": "Color Management (ImageIO)", + "collapsible": true, + "children": [ + { + "key": "ocio", + "type": "dict", + "label": "OpenColorIO (OCIO)", + "collapsible": true, + "checkbox_key": "enabled", + "children": [ + { + "type": "boolean", + "key": "enabled", + "label": "Set OCIO variable for Fusion" + }, + { + "type": "path", + "key": "configFilePath", + "label": "OCIO Config File Path", + "multiplatform": true, + "multipath": true + } + ] + } + ] + } + ] +} diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_houdini.json b/openpype/settings/entities/schemas/projects_schema/schema_project_houdini.json index d8728c0f4b..808f154226 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_houdini.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_houdini.json @@ -5,6 +5,10 @@ "label": "Houdini", "is_file": true, "children": [ + { + "type": "schema", + "name": "schema_houdini_scriptshelf" + }, { "type": "schema", "name": "schema_houdini_create" @@ -14,4 +18,4 @@ "name": "schema_houdini_publish" } ] -} +} \ No newline at end of file diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_kitsu.json b/openpype/settings/entities/schemas/projects_schema/schema_project_kitsu.json index 014a1b7886..fb47670e74 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_kitsu.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_kitsu.json @@ -5,23 +5,6 @@ "collapsible": true, "is_file": true, "children": [ - { - "type": "dict", - "key": "entities_root", - "label": "Entities root folder", - "children": [ - { - "type": "text", - "key": "assets", - "label": "Assets:" - }, - { - "type": "text", - "key": "shots", - "label": "Shots (includes Episodes & Sequences if any):" - } - ] - }, { "type": "dict", "key": "entities_naming_pattern", diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_maya.json b/openpype/settings/entities/schemas/projects_schema/schema_project_maya.json index 816874779e..d7a2b086d9 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_maya.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_maya.json @@ -5,6 +5,13 @@ "label": "Maya", "is_file": true, "children": [ + { + "type": "text", + "multiline" : true, + "use_label_wrap": true, + "key": "mel_workspace", + "label": "Maya MEL Workspace" + }, { "type": "dict-modifiable", "key": "ext_mapping", diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json b/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json index 7aa49c99a4..b768db30ee 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json @@ -131,6 +131,35 @@ } ] }, + { + "type": "dict", + "collapsible": true, + "key": "CollectReview", + "label": "Collect Review", + "children": [ + { + "type": "boolean", + "key": "publish", + "label": "Active" + } + ] + }, + { + "type": "dict", + "key": "CollectVersion", + "label": "Collect Version", + "children": [ + { + "type": "label", + "label": "Synchronize version for image and review instances by workfile version." + }, + { + "type": "boolean", + "key": "enabled", + "label": "Enabled" + } + ] + }, { "type": "schema_template", "name": "template_publish_plugin", diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_publish.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_publish.json index e1aa230b49..297f96aa8c 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_publish.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_publish.json @@ -18,6 +18,27 @@ } ] }, + { + "type": "dict", + "collapsible": true, + "checkbox_key": "enabled", + "key": "CollectAudio", + "label": "Collect Audio", + "is_group": true, + "children": [ + { + "type": "boolean", + "key": "enabled", + "label": "Enabled" + }, + { + "key": "audio_subset_name", + "label": "Name of audio variant", + "type": "text", + "placeholder": "audioMain" + } + ] + }, { "type": "dict", "collapsible": true, @@ -642,10 +663,14 @@ ] } }, + { + "type": "label", + "label": "NOTE: Publish template profiles settings were moved to Tools/Publish/Template name profiles. Please move values there." + }, { "type": "list", "key": "template_name_profiles", - "label": "Template name profiles", + "label": "Template name profiles (DEPRECATED)", "use_label_wrap": true, "object_type": { "type": "dict", @@ -750,10 +775,14 @@ "type": "list", "object_type": "text" }, + { + "type": "label", + "label": "NOTE: Hero publish template profiles settings were moved to Tools/Publish/Hero template name profiles. Please move values there." + }, { "type": "list", "key": "template_name_profiles", - "label": "Template name profiles", + "label": "Template name profiles (DEPRECATED)", "use_label_wrap": true, "object_type": { "type": "dict", diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_tools.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_tools.json index f8c9482e5f..ba446135e2 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_tools.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_global_tools.json @@ -238,6 +238,31 @@ } ] } + }, + { + "type": "list", + "key": "workfile_lock_profiles", + "label": "Workfile lock profiles", + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "type": "hosts-enum", + "key": "host_name", + "label": "Hosts", + "multiselection": true + }, + { + "type": "splitter" + }, + { + "key": "enabled", + "label": "Enabled", + "type": "boolean" + } + ] + } } ] }, @@ -284,6 +309,102 @@ } } ] + }, + { + "type": "dict", + "key": "publish", + "label": "Publish", + "children": [ + { + "type": "label", + "label": "NOTE: For backwards compatibility can be value empty and in that case are used values from IntegrateAssetNew. This will change in future so please move all values here as soon as possible." + }, + { + "type": "list", + "key": "template_name_profiles", + "label": "Template name profiles", + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "key": "families", + "label": "Families", + "type": "list", + "object_type": "text" + }, + { + "type": "hosts-enum", + "key": "hosts", + "label": "Hosts", + "multiselection": true + }, + { + "key": "task_types", + "label": "Task types", + "type": "task-types-enum" + }, + { + "key": "task_names", + "label": "Task names", + "type": "list", + "object_type": "text" + }, + { + "type": "separator" + }, + { + "type": "text", + "key": "template_name", + "label": "Template name" + } + ] + } + }, + { + "type": "list", + "key": "hero_template_name_profiles", + "label": "Hero template name profiles", + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "key": "families", + "label": "Families", + "type": "list", + "object_type": "text" + }, + { + "type": "hosts-enum", + "key": "hosts", + "label": "Hosts", + "multiselection": true + }, + { + "key": "task_types", + "label": "Task types", + "type": "task-types-enum" + }, + { + "key": "task_names", + "label": "Task names", + "type": "list", + "object_type": "text" + }, + { + "type": "separator" + }, + { + "type": "text", + "key": "template_name", + "label": "Template name", + "tooltip": "Name of template from Anatomy templates" + } + ] + } + } + ] } ] } diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_houdini_scriptshelf.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_houdini_scriptshelf.json new file mode 100644 index 0000000000..bab9b604b4 --- /dev/null +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_houdini_scriptshelf.json @@ -0,0 +1,71 @@ +{ + "type": "list", + "key": "shelves", + "label": "Shelves Manager", + "is_group": true, + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "type": "text", + "key": "shelf_set_name", + "label": "Shelf Set Name" + }, + { + "type": "path", + "key": "shelf_set_source_path", + "label": "Shelf Set Path (optional)", + "multipath": false, + "multiplatform": true + }, + { + "type": "list", + "key": "shelf_definition", + "label": "Shelves", + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "type": "text", + "key": "shelf_name", + "label": "Shelf Name" + }, + { + "type": "list", + "key": "tools_list", + "label": "Tools", + "use_label_wrap": true, + "object_type": { + "type": "dict", + "children": [ + { + "type": "text", + "key": "label", + "label": "Name" + }, + { + "type": "path", + "key": "script", + "label": "Script" + }, + { + "type": "path", + "key": "icon", + "label": "Icon" + }, + { + "type": "text", + "key": "help", + "label": "Help" + } + ] + } + } + ] + } + } + ] + } +} \ No newline at end of file diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_capture.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_capture.json index 7a40f349cc..62c33f55fc 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_capture.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_capture.json @@ -94,18 +94,6 @@ } ] }, - - { - "type": "dict", - "key": "PanZoom", - "children": [ - { - "type": "boolean", - "key": "pan_zoom", - "label": " Pan Zoom" - } - ] - }, { "type": "splitter" }, @@ -153,19 +141,6 @@ "decimal": 0, "minimum": 0, "maximum": 99999 - }, - { - "type": "number", - "key": "percent", - "label": "percent", - "decimal": 1, - "minimum": 0, - "maximum": 200 - }, - { - "type": "text", - "key": "mode", - "label": "Mode" } ] }, @@ -195,6 +170,11 @@ { "nolights": "No Lights"} ] }, + { + "type": "boolean", + "key": "displayTextures", + "label": "Display Textures" + }, { "type": "number", "key": "textureMaxResolution", @@ -217,11 +197,6 @@ "key": "shadows", "label": "Display Shadows" }, - { - "type": "boolean", - "key": "textures", - "label": "Display Textures" - }, { "type": "boolean", "key": "twoSidedLighting", @@ -369,120 +344,114 @@ { "type": "splitter" }, + { + "type": "label", + "label": "Show" + }, { "type": "boolean", "key": "cameras", - "label": "cameras" + "label": "Cameras" }, { "type": "boolean", "key": "clipGhosts", - "label": "clipGhosts" - }, - { - "type": "boolean", - "key": "controlVertices", - "label": "controlVertices" + "label": "Clip Ghosts" }, { "type": "boolean", "key": "deformers", - "label": "deformers" + "label": "Deformers" }, { "type": "boolean", "key": "dimensions", - "label": "dimensions" + "label": "Dimensions" }, { "type": "boolean", "key": "dynamicConstraints", - "label": "dynamicConstraints" + "label": "Dynamic Constraints" }, { "type": "boolean", "key": "dynamics", - "label": "dynamics" + "label": "Dynamics" }, { "type": "boolean", "key": "fluids", - "label": "fluids" + "label": "Fluids" }, { "type": "boolean", "key": "follicles", - "label": "follicles" + "label": "Follicles" }, { "type": "boolean", "key": "gpuCacheDisplayFilter", - "label": "gpuCacheDisplayFilter" + "label": "GPU Cache" }, { "type": "boolean", "key": "greasePencils", - "label": "greasePencils" + "label": "Grease Pencil" }, { "type": "boolean", "key": "grid", - "label": "grid" + "label": "Grid" }, { "type": "boolean", "key": "hairSystems", - "label": "hairSystems" + "label": "Hair Systems" }, { "type": "boolean", "key": "handles", - "label": "handles" + "label": "Handles" }, { "type": "boolean", - "key": "hud", - "label": "hud" - }, - { - "type": "boolean", - "key": "hulls", - "label": "hulls" + "key": "headsUpDisplay", + "label": "HUD" }, { "type": "boolean", "key": "ikHandles", - "label": "ikHandles" + "label": "IK Handles" }, { "type": "boolean", "key": "imagePlane", - "label": "imagePlane" + "label": "Image Planes" }, { "type": "boolean", "key": "joints", - "label": "joints" + "label": "Joints" }, { "type": "boolean", "key": "lights", - "label": "lights" + "label": "Lights" }, { "type": "boolean", "key": "locators", - "label": "locators" + "label": "Locators" }, { "type": "boolean", "key": "manipulators", - "label": "manipulators" + "label": "Manipulators" }, { "type": "boolean", "key": "motionTrails", - "label": "motionTrails" + "label": "Motion Trails" }, { "type": "boolean", @@ -499,50 +468,65 @@ "key": "nRigids", "label": "nRigids" }, + { + "type": "boolean", + "key": "controlVertices", + "label": "NURBS CVs" + }, { "type": "boolean", "key": "nurbsCurves", - "label": "nurbsCurves" + "label": "NURBS Curves" + }, + { + "type": "boolean", + "key": "hulls", + "label": "NURBS Hulls" }, { "type": "boolean", "key": "nurbsSurfaces", - "label": "nurbsSurfaces" + "label": "NURBS Surfaces" }, { "type": "boolean", "key": "particleInstancers", - "label": "particleInstancers" + "label": "Particle Instancers" }, { "type": "boolean", "key": "pivots", - "label": "pivots" + "label": "Pivots" }, { "type": "boolean", "key": "planes", - "label": "planes" + "label": "Planes" }, { "type": "boolean", "key": "pluginShapes", - "label": "pluginShapes" + "label": "Plugin Shapes" }, { "type": "boolean", "key": "polymeshes", - "label": "polymeshes" + "label": "Polygons" }, { "type": "boolean", "key": "strokes", - "label": "strokes" + "label": "Strokes" }, { "type": "boolean", "key": "subdivSurfaces", - "label": "subdivSurfaces" + "label": "Subdiv Surfaces" + }, + { + "type": "boolean", + "key": "textures", + "label": "Texture Placements" } ] }, @@ -555,47 +539,47 @@ { "type": "boolean", "key": "displayGateMask", - "label": "displayGateMask" + "label": "Display Gate Mask" }, { "type": "boolean", "key": "displayResolution", - "label": "displayResolution" + "label": "Display Resolution" }, { "type": "boolean", "key": "displayFilmGate", - "label": "displayFilmGate" + "label": "Display Film Gate" }, { "type": "boolean", "key": "displayFieldChart", - "label": "displayFieldChart" + "label": "Display Field Chart" }, { "type": "boolean", "key": "displaySafeAction", - "label": "displaySafeAction" + "label": "Display Safe Action" }, { "type": "boolean", "key": "displaySafeTitle", - "label": "displaySafeTitle" + "label": "Display Safe Title" }, { "type": "boolean", "key": "displayFilmPivot", - "label": "displayFilmPivot" + "label": "Display Film Pivot" }, { "type": "boolean", "key": "displayFilmOrigin", - "label": "displayFilmOrigin" + "label": "Display Film Origin" }, { "type": "number", "key": "overscan", - "label": "overscan", + "label": "Overscan", "decimal": 1, "minimum": 0, "maximum": 10 diff --git a/openpype/style/data.json b/openpype/style/data.json index 15d9472e3e..adda49de23 100644 --- a/openpype/style/data.json +++ b/openpype/style/data.json @@ -20,7 +20,7 @@ "color": { "font": "#D3D8DE", "font-hover": "#F0F2F5", - "font-disabled": "#99A3B2", + "font-disabled": "#5b6779", "font-view-selection": "#ffffff", "font-view-hover": "#F0F2F5", diff --git a/openpype/tools/launcher/models.py b/openpype/tools/launcher/models.py index 6d40d21f96..6e3b531018 100644 --- a/openpype/tools/launcher/models.py +++ b/openpype/tools/launcher/models.py @@ -281,18 +281,25 @@ class ActionModel(QtGui.QStandardItemModel): if not action_item: return - action = action_item.data(ACTION_ROLE) - actual_data = self._prepare_compare_data(action) + actions = action_item.data(ACTION_ROLE) + if not isinstance(actions, list): + actions = [actions] + + action_actions_data = [ + self._prepare_compare_data(action) + for action in actions + ] stored = self.launcher_registry.get_item("force_not_open_workfile") - if is_checked: - stored.append(actual_data) - else: - final_values = [] - for config in stored: - if config != actual_data: - final_values.append(config) - stored = final_values + for actual_data in action_actions_data: + if is_checked: + stored.append(actual_data) + else: + final_values = [] + for config in stored: + if config != actual_data: + final_values.append(config) + stored = final_values self.launcher_registry.set_item("force_not_open_workfile", stored) self.launcher_registry._get_item.cache_clear() @@ -329,21 +336,24 @@ class ActionModel(QtGui.QStandardItemModel): item (QStandardItem) stored (list) of dict """ - action = item.data(ACTION_ROLE) - if not self.is_application_action(action): + + actions = item.data(ACTION_ROLE) + if not isinstance(actions, list): + actions = [actions] + + if not self.is_application_action(actions[0]): return False - actual_data = self._prepare_compare_data(action) + action_actions_data = [ + self._prepare_compare_data(action) + for action in actions + ] for config in stored: - if config == actual_data: + if config in action_actions_data: return True - return False def _prepare_compare_data(self, action): - if isinstance(action, list) and action: - action = action[0] - compare_data = {} if action and action.label: compare_data = { diff --git a/openpype/tools/launcher/widgets.py b/openpype/tools/launcher/widgets.py index 62599664fe..774ceb659d 100644 --- a/openpype/tools/launcher/widgets.py +++ b/openpype/tools/launcher/widgets.py @@ -312,11 +312,12 @@ class ActionBar(QtWidgets.QWidget): is_group = index.data(GROUP_ROLE) is_variant_group = index.data(VARIANT_GROUP_ROLE) + force_not_open_workfile = index.data(FORCE_NOT_OPEN_WORKFILE_ROLE) if not is_group and not is_variant_group: action = index.data(ACTION_ROLE) # Change data of application action if issubclass(action, ApplicationAction): - if index.data(FORCE_NOT_OPEN_WORKFILE_ROLE): + if force_not_open_workfile: action.data["start_last_workfile"] = False else: action.data.pop("start_last_workfile", None) @@ -385,10 +386,18 @@ class ActionBar(QtWidgets.QWidget): menu.addMenu(sub_menu) result = menu.exec_(QtGui.QCursor.pos()) - if result: - action = actions_mapping[result] - self._start_animation(index) - self.action_clicked.emit(action) + if not result: + return + + action = actions_mapping[result] + if issubclass(action, ApplicationAction): + if force_not_open_workfile: + action.data["start_last_workfile"] = False + else: + action.data.pop("start_last_workfile", None) + + self._start_animation(index) + self.action_clicked.emit(action) class ActionHistory(QtWidgets.QPushButton): diff --git a/openpype/tools/loader/delegates.py b/openpype/tools/loader/delegates.py new file mode 100644 index 0000000000..e6663d48f1 --- /dev/null +++ b/openpype/tools/loader/delegates.py @@ -0,0 +1,28 @@ +from Qt import QtWidgets, QtGui, QtCore + + +class LoadedInSceneDelegate(QtWidgets.QStyledItemDelegate): + """Delegate for Loaded in Scene state columns. + + Shows "yes" or "no" for True or False values + Colorizes green or dark grey based on True or False values + + """ + + def __init__(self, *args, **kwargs): + super(LoadedInSceneDelegate, self).__init__(*args, **kwargs) + self._colors = { + True: QtGui.QColor(80, 170, 80), + False: QtGui.QColor(90, 90, 90) + } + + def displayText(self, value, locale): + return "yes" if value else "no" + + def initStyleOption(self, option, index): + super(LoadedInSceneDelegate, self).initStyleOption(option, index) + + # Colorize based on value + value = index.data(QtCore.Qt.DisplayRole) + color = self._colors[bool(value)] + option.palette.setBrush(QtGui.QPalette.Text, color) diff --git a/openpype/tools/loader/model.py b/openpype/tools/loader/model.py index 929e497890..77a8669c46 100644 --- a/openpype/tools/loader/model.py +++ b/openpype/tools/loader/model.py @@ -17,6 +17,7 @@ from openpype.client import ( get_representations ) from openpype.pipeline import ( + registered_host, HeroVersionType, schema, ) @@ -24,6 +25,7 @@ from openpype.pipeline import ( from openpype.style import get_default_entity_icon_color from openpype.tools.utils.models import TreeModel, Item from openpype.tools.utils import lib +from openpype.host import ILoadHost from openpype.modules import ModulesManager from openpype.tools.utils.constants import ( @@ -136,6 +138,7 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): "duration", "handles", "step", + "loaded_in_scene", "repre_info" ] @@ -150,6 +153,7 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): "duration": "Duration", "handles": "Handles", "step": "Step", + "loaded_in_scene": "In scene", "repre_info": "Availability" } @@ -231,8 +235,14 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): self._doc_fetching_stop = False self._doc_payload = {} - self.doc_fetched.connect(self._on_doc_fetched) + self._host = registered_host() + self._loaded_representation_ids = set() + # Refresh loaded scene containers only every 3 seconds at most + self._host_loaded_refresh_timeout = 3 + self._host_loaded_refresh_time = 0 + + self.doc_fetched.connect(self._on_doc_fetched) self.refresh() def get_item_by_id(self, item_id): @@ -474,6 +484,27 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): last_versions_by_subset_id[subset_id] = hero_version + # Check loaded subsets + loaded_subset_ids = set() + ids = self._loaded_representation_ids + if ids: + if self._doc_fetching_stop: + return + + # Get subset ids from loaded representations in workfile + # todo: optimize with aggregation query to distinct subset id + representations = get_representations(project_name, + representation_ids=ids, + fields=["parent"]) + version_ids = set(repre["parent"] for repre in representations) + versions = get_versions(project_name, + version_ids=version_ids, + fields=["parent"]) + loaded_subset_ids = set(version["parent"] for version in versions) + + if self._doc_fetching_stop: + return + repre_info_by_version_id = {} if self.sync_server.enabled: versions_by_id = {} @@ -501,7 +532,8 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): "subset_docs_by_id": subset_docs_by_id, "subset_families": subset_families, "last_versions_by_subset_id": last_versions_by_subset_id, - "repre_info_by_version_id": repre_info_by_version_id + "repre_info_by_version_id": repre_info_by_version_id, + "subsets_loaded_by_id": loaded_subset_ids } self.doc_fetched.emit() @@ -533,6 +565,20 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): self.doc_fetched.emit() return + # Collect scene container representations to compare loaded state + # This runs in the main thread because it involves the host DCC + if self._host: + time_since_refresh = time.time() - self._host_loaded_refresh_time + if time_since_refresh > self._host_loaded_refresh_timeout: + if isinstance(self._host, ILoadHost): + containers = self._host.get_containers() + else: + containers = self._host.ls() + + repre_ids = {con.get("representation") for con in containers} + self._loaded_representation_ids = repre_ids + self._host_loaded_refresh_time = time.time() + self.fetch_subset_and_version() def _on_doc_fetched(self): @@ -554,6 +600,10 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): "repre_info_by_version_id" ) + subsets_loaded_by_id = self._doc_payload.get( + "subsets_loaded_by_id" + ) + if ( asset_docs_by_id is None or subset_docs_by_id is None @@ -568,7 +618,8 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): asset_docs_by_id, subset_docs_by_id, last_versions_by_subset_id, - repre_info_by_version_id + repre_info_by_version_id, + subsets_loaded_by_id ) self.endResetModel() self.refreshed.emit(True) @@ -596,8 +647,12 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): return merge_group def _fill_subset_items( - self, asset_docs_by_id, subset_docs_by_id, last_versions_by_subset_id, - repre_info_by_version_id + self, + asset_docs_by_id, + subset_docs_by_id, + last_versions_by_subset_id, + repre_info_by_version_id, + subsets_loaded_by_id ): _groups_tuple = self.groups_config.split_subsets_for_groups( subset_docs_by_id.values(), self._grouping @@ -621,6 +676,35 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): "index": self.index(group_item.row(), 0) } + def _add_subset_item(subset_doc, parent_item, parent_index): + last_version = last_versions_by_subset_id.get( + subset_doc["_id"] + ) + # do not show subset without version + if not last_version: + return + + data = copy.deepcopy(subset_doc) + data["subset"] = subset_doc["name"] + + asset_id = subset_doc["parent"] + data["asset"] = asset_docs_by_id[asset_id]["name"] + + data["last_version"] = last_version + data["loaded_in_scene"] = subset_doc["_id"] in subsets_loaded_by_id + + # Sync server data + data.update( + self._get_last_repre_info(repre_info_by_version_id, + last_version["_id"])) + + item = Item() + item.update(data) + self.add_child(item, parent_item) + + index = self.index(item.row(), 0, parent_index) + self.set_version(index, last_version) + subset_counter = 0 for group_name, subset_docs_by_name in subset_docs_by_group.items(): parent_item = group_item_by_name[group_name]["item"] @@ -643,31 +727,9 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): _parent_index = parent_index for subset_doc in subset_docs: - asset_id = subset_doc["parent"] - - data = copy.deepcopy(subset_doc) - data["subset"] = subset_name - data["asset"] = asset_docs_by_id[asset_id]["name"] - - last_version = last_versions_by_subset_id.get( - subset_doc["_id"] - ) - data["last_version"] = last_version - - # do not show subset without version - if not last_version: - continue - - data.update( - self._get_last_repre_info(repre_info_by_version_id, - last_version["_id"])) - - item = Item() - item.update(data) - self.add_child(item, _parent_item) - - index = self.index(item.row(), 0, _parent_index) - self.set_version(index, last_version) + _add_subset_item(subset_doc, + parent_item=_parent_item, + parent_index=_parent_index) for subset_name in sorted(subset_docs_without_group.keys()): subset_docs = subset_docs_without_group[subset_name] @@ -682,31 +744,9 @@ class SubsetsModel(TreeModel, BaseRepresentationModel): subset_counter += 1 for subset_doc in subset_docs: - asset_id = subset_doc["parent"] - - data = copy.deepcopy(subset_doc) - data["subset"] = subset_name - data["asset"] = asset_docs_by_id[asset_id]["name"] - - last_version = last_versions_by_subset_id.get( - subset_doc["_id"] - ) - data["last_version"] = last_version - - # do not show subset without version - if not last_version: - continue - - data.update( - self._get_last_repre_info(repre_info_by_version_id, - last_version["_id"])) - - item = Item() - item.update(data) - self.add_child(item, parent_item) - - index = self.index(item.row(), 0, parent_index) - self.set_version(index, last_version) + _add_subset_item(subset_doc, + parent_item=parent_item, + parent_index=parent_index) def data(self, index, role): if not index.isValid(): diff --git a/openpype/tools/loader/widgets.py b/openpype/tools/loader/widgets.py index cbf5720803..c028aa4174 100644 --- a/openpype/tools/loader/widgets.py +++ b/openpype/tools/loader/widgets.py @@ -58,6 +58,7 @@ from .model import ( ITEM_ID_ROLE ) from . import lib +from .delegates import LoadedInSceneDelegate from openpype.tools.utils.constants import ( LOCAL_PROVIDER_ROLE, @@ -169,6 +170,7 @@ class SubsetWidget(QtWidgets.QWidget): ("duration", 60), ("handles", 55), ("step", 10), + ("loaded_in_scene", 25), ("repre_info", 65) ) @@ -234,6 +236,10 @@ class SubsetWidget(QtWidgets.QWidget): column = model.Columns.index("repre_info") view.setItemDelegateForColumn(column, avail_delegate) + loaded_in_scene_delegate = LoadedInSceneDelegate(view) + column = model.Columns.index("loaded_in_scene") + view.setItemDelegateForColumn(column, loaded_in_scene_delegate) + layout = QtWidgets.QVBoxLayout(self) layout.setContentsMargins(0, 0, 0, 0) layout.addLayout(top_bar_layout) diff --git a/openpype/tools/project_manager/project_manager/window.py b/openpype/tools/project_manager/project_manager/window.py index c6ae0ff352..3b2dea8ca3 100644 --- a/openpype/tools/project_manager/project_manager/window.py +++ b/openpype/tools/project_manager/project_manager/window.py @@ -1,5 +1,12 @@ from Qt import QtWidgets, QtCore, QtGui +from openpype import resources +from openpype.style import load_stylesheet +from openpype.widgets import PasswordDialog +from openpype.lib import is_admin_password_required, Logger +from openpype.pipeline import AvalonMongoDB +from openpype.pipeline.project_folders import create_project_folders + from . import ( ProjectModel, ProjectProxyFilter, @@ -13,17 +20,6 @@ from . import ( ) from .widgets import ConfirmProjectDeletion from .style import ResourceCache -from openpype.style import load_stylesheet -from openpype.lib import is_admin_password_required -from openpype.widgets import PasswordDialog -from openpype.pipeline import AvalonMongoDB - -from openpype import resources -from openpype.api import ( - get_project_basic_paths, - create_project_folders, - Logger -) class ProjectManagerWindow(QtWidgets.QWidget): @@ -259,12 +255,8 @@ class ProjectManagerWindow(QtWidgets.QWidget): qm.Yes | qm.No) if ans == qm.Yes: try: - # Get paths based on presets - basic_paths = get_project_basic_paths(project_name) - if not basic_paths: - pass # Invoking OpenPype API to create the project folders - create_project_folders(basic_paths, project_name) + create_project_folders(project_name) except Exception as exc: self.log.warning( "Cannot create starting folders: {}".format(exc), diff --git a/openpype/tools/publisher/widgets/widgets.py b/openpype/tools/publisher/widgets/widgets.py index aa7e3be687..d1fa71343c 100644 --- a/openpype/tools/publisher/widgets/widgets.py +++ b/openpype/tools/publisher/widgets/widgets.py @@ -16,6 +16,7 @@ from openpype.tools.utils import ( BaseClickableFrame, set_style_property, ) +from openpype.style import get_objected_colors from openpype.pipeline.create import ( SUBSET_NAME_ALLOWED_SYMBOLS, TaskNotSetError, @@ -125,28 +126,21 @@ class PublishIconBtn(IconButton): def __init__(self, pixmap_path, *args, **kwargs): super(PublishIconBtn, self).__init__(*args, **kwargs) - loaded_image = QtGui.QImage(pixmap_path) + colors = get_objected_colors() + icon = self.generate_icon( + pixmap_path, + enabled_color=colors["font"].get_qcolor(), + disabled_color=colors["font-disabled"].get_qcolor()) + self.setIcon(icon) - pixmap = self.paint_image_with_color(loaded_image, QtCore.Qt.white) - - self._base_image = loaded_image - self._enabled_icon = QtGui.QIcon(pixmap) - self._disabled_icon = None - - self.setIcon(self._enabled_icon) - - def get_enabled_icon(self): - """Enabled icon.""" - return self._enabled_icon - - def get_disabled_icon(self): - """Disabled icon.""" - if self._disabled_icon is None: - pixmap = self.paint_image_with_color( - self._base_image, QtCore.Qt.gray - ) - self._disabled_icon = QtGui.QIcon(pixmap) - return self._disabled_icon + def generate_icon(self, pixmap_path, enabled_color, disabled_color): + icon = QtGui.QIcon() + image = QtGui.QImage(pixmap_path) + enabled_pixmap = self.paint_image_with_color(image, enabled_color) + icon.addPixmap(enabled_pixmap, icon.Normal) + disabled_pixmap = self.paint_image_with_color(image, disabled_color) + icon.addPixmap(disabled_pixmap, icon.Disabled) + return icon @staticmethod def paint_image_with_color(image, color): @@ -187,13 +181,6 @@ class PublishIconBtn(IconButton): return pixmap - def setEnabled(self, enabled): - super(PublishIconBtn, self).setEnabled(enabled) - if self.isEnabled(): - self.setIcon(self.get_enabled_icon()) - else: - self.setIcon(self.get_disabled_icon()) - class ResetBtn(PublishIconBtn): """Publish reset button.""" diff --git a/openpype/tools/publisher/window.py b/openpype/tools/publisher/window.py index 90a36b4f01..2a0e6e940a 100644 --- a/openpype/tools/publisher/window.py +++ b/openpype/tools/publisher/window.py @@ -30,8 +30,8 @@ from .widgets import ( class PublisherWindow(QtWidgets.QDialog): """Main window of publisher.""" - default_width = 1000 - default_height = 600 + default_width = 1200 + default_height = 700 def __init__(self, parent=None, reset_on_show=None): super(PublisherWindow, self).__init__(parent) diff --git a/openpype/tools/sceneinventory/model.py b/openpype/tools/sceneinventory/model.py index 63fbe04c5c..1a3b7c7055 100644 --- a/openpype/tools/sceneinventory/model.py +++ b/openpype/tools/sceneinventory/model.py @@ -34,7 +34,8 @@ from .lib import ( class InventoryModel(TreeModel): """The model for the inventory""" - Columns = ["Name", "version", "count", "family", "loader", "objectName"] + Columns = ["Name", "version", "count", "family", + "group", "loader", "objectName"] OUTDATED_COLOR = QtGui.QColor(235, 30, 30) CHILD_OUTDATED_COLOR = QtGui.QColor(200, 160, 30) @@ -157,8 +158,13 @@ class InventoryModel(TreeModel): # Family icon return item.get("familyIcon", None) + column_name = self.Columns[index.column()] + + if column_name == "group" and item.get("group"): + return qtawesome.icon("fa.object-group", + color=get_default_entity_icon_color()) + if item.get("isGroupNode"): - column_name = self.Columns[index.column()] if column_name == "active_site": provider = item.get("active_site_provider") return self._site_icons.get(provider) @@ -423,6 +429,7 @@ class InventoryModel(TreeModel): group_node["familyIcon"] = family_icon group_node["count"] = len(group_items) group_node["isGroupNode"] = True + group_node["group"] = subset["data"].get("subsetGroup") if self.sync_enabled: progress = get_progress_for_repre( diff --git a/openpype/tools/sceneinventory/window.py b/openpype/tools/sceneinventory/window.py index 463280b71c..578f47d1c0 100644 --- a/openpype/tools/sceneinventory/window.py +++ b/openpype/tools/sceneinventory/window.py @@ -89,7 +89,8 @@ class SceneInventoryWindow(QtWidgets.QDialog): view.setColumnWidth(1, 55) # version view.setColumnWidth(2, 55) # count view.setColumnWidth(3, 150) # family - view.setColumnWidth(4, 100) # namespace + view.setColumnWidth(4, 120) # group + view.setColumnWidth(5, 150) # loader # apply delegates version_delegate = VersionDelegate(legacy_io, self) diff --git a/openpype/tools/tray/pype_tray.py b/openpype/tools/tray/pype_tray.py index 85bc00ead6..348573a191 100644 --- a/openpype/tools/tray/pype_tray.py +++ b/openpype/tools/tray/pype_tray.py @@ -9,11 +9,11 @@ import platform from Qt import QtCore, QtGui, QtWidgets import openpype.version -from openpype.api import ( - resources, - get_system_settings +from openpype import resources, style +from openpype.lib import ( + get_openpype_execute_args, + Logger, ) -from openpype.lib import get_openpype_execute_args, Logger from openpype.lib.openpype_version import ( op_version_control_available, get_expected_version, @@ -25,8 +25,8 @@ from openpype.lib.openpype_version import ( get_openpype_version, ) from openpype.modules import TrayModulesManager -from openpype import style from openpype.settings import ( + get_system_settings, SystemSettings, ProjectSettings, DefaultsNotDefined @@ -774,10 +774,24 @@ class PypeTrayStarter(QtCore.QObject): def main(): + log = Logger.get_logger(__name__) app = QtWidgets.QApplication.instance() if not app: app = QtWidgets.QApplication([]) + for attr_name in ( + "AA_EnableHighDpiScaling", + "AA_UseHighDpiPixmaps" + ): + attr = getattr(QtCore.Qt, attr_name, None) + if attr is None: + log.debug(( + "Missing QtCore.Qt attribute \"{}\"." + " UI quality may be affected." + ).format(attr_name)) + else: + app.setAttribute(attr) + starter = PypeTrayStarter(app) # TODO remove when pype.exe will have an icon diff --git a/openpype/tools/traypublisher/window.py b/openpype/tools/traypublisher/window.py index cc33287091..930c27ca9c 100644 --- a/openpype/tools/traypublisher/window.py +++ b/openpype/tools/traypublisher/window.py @@ -7,6 +7,7 @@ publishing plugins. """ from Qt import QtWidgets, QtCore +import qtawesome from openpype.pipeline import ( install_host, @@ -20,6 +21,27 @@ from openpype.tools.utils.models import ( ProjectSortFilterProxy ) +from openpype.tools.utils import PlaceholderLineEdit +import appdirs +from openpype.lib import JSONSettingRegistry + + +class TrayPublisherRegistry(JSONSettingRegistry): + """Class handling OpenPype general settings registry. + + Attributes: + vendor (str): Name used for path construction. + product (str): Additional name used for path construction. + + """ + + def __init__(self): + self.vendor = "pypeclub" + self.product = "openpype" + name = "tray_publisher" + path = appdirs.user_data_dir(self.product, self.vendor) + super(TrayPublisherRegistry, self).__init__(name, path) + class StandaloneOverlayWidget(QtWidgets.QFrame): project_selected = QtCore.Signal(str) @@ -43,6 +65,7 @@ class StandaloneOverlayWidget(QtWidgets.QFrame): projects_model = ProjectModel(dbcon) projects_proxy = ProjectSortFilterProxy() projects_proxy.setSourceModel(projects_model) + projects_proxy.setFilterKeyColumn(0) projects_view = QtWidgets.QListView(content_widget) projects_view.setObjectName("ChooseProjectView") @@ -59,10 +82,17 @@ class StandaloneOverlayWidget(QtWidgets.QFrame): btns_layout.addWidget(cancel_btn, 0) btns_layout.addWidget(confirm_btn, 0) + txt_filter = PlaceholderLineEdit(content_widget) + txt_filter.setPlaceholderText("Quick filter projects..") + txt_filter.setClearButtonEnabled(True) + txt_filter.addAction(qtawesome.icon("fa.filter", color="gray"), + QtWidgets.QLineEdit.LeadingPosition) + content_layout = QtWidgets.QVBoxLayout(content_widget) content_layout.setContentsMargins(0, 0, 0, 0) content_layout.setSpacing(20) content_layout.addWidget(header_label, 0) + content_layout.addWidget(txt_filter, 0) content_layout.addWidget(projects_view, 1) content_layout.addLayout(btns_layout, 0) @@ -79,17 +109,40 @@ class StandaloneOverlayWidget(QtWidgets.QFrame): projects_view.doubleClicked.connect(self._on_double_click) confirm_btn.clicked.connect(self._on_confirm_click) cancel_btn.clicked.connect(self._on_cancel_click) + txt_filter.textChanged.connect(self._on_text_changed) self._projects_view = projects_view self._projects_model = projects_model + self._projects_proxy = projects_proxy self._cancel_btn = cancel_btn self._confirm_btn = confirm_btn + self._txt_filter = txt_filter self._publisher_window = publisher_window self._project_name = None def showEvent(self, event): self._projects_model.refresh() + # Sort projects after refresh + self._projects_proxy.sort(0) + + setting_registry = TrayPublisherRegistry() + try: + project_name = setting_registry.get_item("project_name") + except ValueError: + project_name = None + + if project_name: + index = None + src_index = self._projects_model.find_project(project_name) + if src_index is not None: + index = self._projects_proxy.mapFromSource(src_index) + if index: + mode = ( + QtCore.QItemSelectionModel.Select + | QtCore.QItemSelectionModel.Rows) + self._projects_view.selectionModel().select(index, mode) + self._cancel_btn.setVisible(self._project_name is not None) super(StandaloneOverlayWidget, self).showEvent(event) @@ -102,6 +155,10 @@ class StandaloneOverlayWidget(QtWidgets.QFrame): def _on_cancel_click(self): self._set_project(self._project_name) + def _on_text_changed(self): + self._projects_proxy.setFilterRegularExpression( + self._txt_filter.text()) + def set_selected_project(self): index = self._projects_view.currentIndex() @@ -119,6 +176,9 @@ class StandaloneOverlayWidget(QtWidgets.QFrame): self.setVisible(False) self.project_selected.emit(project_name) + setting_registry = TrayPublisherRegistry() + setting_registry.set_item("project_name", project_name) + class TrayPublishWindow(PublisherWindow): def __init__(self, *args, **kwargs): diff --git a/openpype/tools/utils/host_tools.py b/openpype/tools/utils/host_tools.py index 52d15a59f7..d2f05d3302 100644 --- a/openpype/tools/utils/host_tools.py +++ b/openpype/tools/utils/host_tools.py @@ -32,6 +32,7 @@ class HostToolsHelper: self._workfiles_tool = None self._loader_tool = None self._creator_tool = None + self._publisher_tool = None self._subset_manager_tool = None self._scene_inventory_tool = None self._library_loader_tool = None @@ -193,7 +194,6 @@ class HostToolsHelper: library_loader_tool.showNormal() library_loader_tool.refresh() - def show_publish(self, parent=None): """Try showing the most desirable publish GUI @@ -269,6 +269,31 @@ class HostToolsHelper: dialog.activateWindow() dialog.showNormal() + def get_publisher_tool(self, parent): + """Create, cache and return publisher window.""" + + if self._publisher_tool is None: + from openpype.tools.publisher import PublisherWindow + + host = registered_host() + ILoadHost.validate_load_methods(host) + + publisher_window = PublisherWindow( + parent=parent or self._parent + ) + self._publisher_tool = publisher_window + + return self._publisher_tool + + def show_publisher_tool(self, parent=None): + with qt_app_context(): + dialog = self.get_publisher_tool(parent) + + dialog.show() + dialog.raise_() + dialog.activateWindow() + dialog.showNormal() + def get_tool_by_name(self, tool_name, parent=None, *args, **kwargs): """Show tool by it's name. @@ -298,6 +323,10 @@ class HostToolsHelper: elif tool_name == "publish": self.log.info("Can't return publish tool window.") + # "new" publisher + elif tool_name == "publisher": + return self.get_publisher_tool(parent, *args, **kwargs) + elif tool_name == "experimental_tools": return self.get_experimental_tools_dialog(parent, *args, **kwargs) @@ -335,6 +364,9 @@ class HostToolsHelper: elif tool_name == "publish": self.show_publish(parent, *args, **kwargs) + elif tool_name == "publisher": + self.show_publisher_tool(parent, *args, **kwargs) + elif tool_name == "experimental_tools": self.show_experimental_tools_dialog(parent, *args, **kwargs) @@ -414,6 +446,10 @@ def show_publish(parent=None): _SingletonPoint.show_tool_by_name("publish", parent) +def show_publisher(parent=None): + _SingletonPoint.show_tool_by_name("publisher", parent) + + def show_experimental_tools_dialog(parent=None): _SingletonPoint.show_tool_by_name("experimental_tools", parent) diff --git a/openpype/tools/utils/models.py b/openpype/tools/utils/models.py index 1faccef4dd..2e93d94d7e 100644 --- a/openpype/tools/utils/models.py +++ b/openpype/tools/utils/models.py @@ -330,11 +330,26 @@ class ProjectModel(QtGui.QStandardItemModel): if new_items: root_item.appendRows(new_items) + def find_project(self, project_name): + """ + Get index of 'project_name' value. + + Args: + project_name (str): + Returns: + (QModelIndex) + """ + val = self._items_by_name.get(project_name) + if val: + return self.indexFromItem(val) + class ProjectSortFilterProxy(QtCore.QSortFilterProxyModel): def __init__(self, *args, **kwargs): super(ProjectSortFilterProxy, self).__init__(*args, **kwargs) self._filter_enabled = True + # Disable case sensitivity + self.setSortCaseSensitivity(QtCore.Qt.CaseInsensitive) def lessThan(self, left_index, right_index): if left_index.data(PROJECT_NAME_ROLE) is None: @@ -356,10 +371,14 @@ class ProjectSortFilterProxy(QtCore.QSortFilterProxyModel): def filterAcceptsRow(self, source_row, source_parent): index = self.sourceModel().index(source_row, 0, source_parent) + string_pattern = self.filterRegularExpression().pattern() if self._filter_enabled: result = self._custom_index_filter(index) if result is not None: - return result + project_name = index.data(PROJECT_NAME_ROLE) + if project_name is None: + return result + return string_pattern.lower() in project_name.lower() return super(ProjectSortFilterProxy, self).filterAcceptsRow( source_row, source_parent diff --git a/openpype/tools/workfile_template_build/__init__.py b/openpype/tools/workfile_template_build/__init__.py new file mode 100644 index 0000000000..82a22aea50 --- /dev/null +++ b/openpype/tools/workfile_template_build/__init__.py @@ -0,0 +1,5 @@ +from .window import WorkfileBuildPlaceholderDialog + +__all__ = ( + "WorkfileBuildPlaceholderDialog", +) diff --git a/openpype/tools/workfile_template_build/window.py b/openpype/tools/workfile_template_build/window.py new file mode 100644 index 0000000000..ea4e2fec5a --- /dev/null +++ b/openpype/tools/workfile_template_build/window.py @@ -0,0 +1,242 @@ +from Qt import QtWidgets + +from openpype import style +from openpype.lib import Logger +from openpype.pipeline import legacy_io +from openpype.widgets.attribute_defs import AttributeDefinitionsWidget + + +class WorkfileBuildPlaceholderDialog(QtWidgets.QDialog): + def __init__(self, host, builder, parent=None): + super(WorkfileBuildPlaceholderDialog, self).__init__(parent) + self.setWindowTitle("Workfile Placeholder Manager") + + self._log = None + + self._first_show = True + self._first_refreshed = False + + self._builder = builder + self._host = host + # Mode can be 0 (create) or 1 (update) + # TODO write it a little bit better + self._mode = 0 + self._update_item = None + self._last_selected_plugin = None + + host_name = getattr(self._host, "name", None) + if not host_name: + host_name = legacy_io.Session.get("AVALON_APP") or "NA" + self._host_name = host_name + + plugins_combo = QtWidgets.QComboBox(self) + + content_widget = QtWidgets.QWidget(self) + content_layout = QtWidgets.QVBoxLayout(content_widget) + content_layout.setContentsMargins(0, 0, 0, 0) + + btns_widget = QtWidgets.QWidget(self) + create_btn = QtWidgets.QPushButton("Create", btns_widget) + save_btn = QtWidgets.QPushButton("Save", btns_widget) + close_btn = QtWidgets.QPushButton("Close", btns_widget) + + create_btn.setVisible(False) + save_btn.setVisible(False) + + btns_layout = QtWidgets.QHBoxLayout(btns_widget) + btns_layout.addStretch(1) + btns_layout.addWidget(create_btn, 0) + btns_layout.addWidget(save_btn, 0) + btns_layout.addWidget(close_btn, 0) + + main_layout = QtWidgets.QVBoxLayout(self) + main_layout.addWidget(plugins_combo, 0) + main_layout.addWidget(content_widget, 1) + main_layout.addWidget(btns_widget, 0) + + create_btn.clicked.connect(self._on_create_click) + save_btn.clicked.connect(self._on_save_click) + close_btn.clicked.connect(self._on_close_click) + plugins_combo.currentIndexChanged.connect(self._on_plugin_change) + + self._attr_defs_widget = None + self._plugins_combo = plugins_combo + + self._content_widget = content_widget + self._content_layout = content_layout + + self._create_btn = create_btn + self._save_btn = save_btn + self._close_btn = close_btn + + @property + def log(self): + if self._log is None: + self._log = Logger.get_logger(self.__class__.__name__) + return self._log + + def _clear_content_widget(self): + while self._content_layout.count() > 0: + item = self._content_layout.takeAt(0) + widget = item.widget() + if widget: + widget.setVisible(False) + widget.deleteLater() + + def _add_message_to_content(self, message): + msg_label = QtWidgets.QLabel(message, self._content_widget) + self._content_layout.addWidget(msg_label, 0) + self._content_layout.addStretch(1) + + def refresh(self): + self._first_refreshed = True + + self._clear_content_widget() + + if not self._builder: + self._add_message_to_content(( + "Host \"{}\" does not have implemented logic" + " for template workfile build." + ).format(self._host_name)) + self._update_ui_visibility() + return + + placeholder_plugins = self._builder.placeholder_plugins + + if self._mode == 1: + self._last_selected_plugin + plugin = self._builder.placeholder_plugins.get( + self._last_selected_plugin + ) + self._create_option_widgets( + plugin, self._update_item.to_dict() + ) + self._update_ui_visibility() + return + + if not placeholder_plugins: + self._add_message_to_content(( + "Host \"{}\" does not have implemented plugins" + " for template workfile build." + ).format(self._host_name)) + self._update_ui_visibility() + return + + last_selected_plugin = self._last_selected_plugin + self._last_selected_plugin = None + self._plugins_combo.clear() + for identifier, plugin in placeholder_plugins.items(): + label = plugin.label or identifier + self._plugins_combo.addItem(label, identifier) + + index = self._plugins_combo.findData(last_selected_plugin) + if index < 0: + index = 0 + self._plugins_combo.setCurrentIndex(index) + self._on_plugin_change() + + self._update_ui_visibility() + + def set_create_mode(self): + if self._mode == 0: + return + + self._mode = 0 + self._update_item = None + self.refresh() + + def set_update_mode(self, update_item): + if self._mode == 1: + return + + self._mode = 1 + self._update_item = update_item + if update_item: + self._last_selected_plugin = update_item.plugin.identifier + self.refresh() + return + + self._clear_content_widget() + self._add_message_to_content(( + "Nothing to update." + " (You maybe don't have selected placeholder.)" + )) + self._update_ui_visibility() + + def _create_option_widgets(self, plugin, options=None): + self._clear_content_widget() + attr_defs = plugin.get_placeholder_options(options) + widget = AttributeDefinitionsWidget(attr_defs, self._content_widget) + self._content_layout.addWidget(widget, 0) + self._content_layout.addStretch(1) + self._attr_defs_widget = widget + self._last_selected_plugin = plugin.identifier + + def _update_ui_visibility(self): + create_mode = self._mode == 0 + self._plugins_combo.setVisible(create_mode) + + if not self._builder: + self._save_btn.setVisible(False) + self._create_btn.setVisible(False) + return + + save_enabled = not create_mode + if save_enabled: + save_enabled = self._update_item is not None + self._save_btn.setVisible(save_enabled) + self._create_btn.setVisible(create_mode) + + def _on_plugin_change(self): + index = self._plugins_combo.currentIndex() + plugin_identifier = self._plugins_combo.itemData(index) + if plugin_identifier == self._last_selected_plugin: + return + + plugin = self._builder.placeholder_plugins.get(plugin_identifier) + self._create_option_widgets(plugin) + + def _on_save_click(self): + options = self._attr_defs_widget.current_value() + plugin = self._builder.placeholder_plugins.get( + self._last_selected_plugin + ) + # TODO much better error handling + try: + plugin.update_placeholder(self._update_item, options) + self.accept() + except Exception: + self.log.warning("Something went wrong", exc_info=True) + dialog = QtWidgets.QMessageBox(self) + dialog.setWindowTitle("Something went wrong") + dialog.setText("Something went wrong") + dialog.exec_() + + def _on_create_click(self): + options = self._attr_defs_widget.current_value() + plugin = self._builder.placeholder_plugins.get( + self._last_selected_plugin + ) + # TODO much better error handling + try: + plugin.create_placeholder(options) + self.accept() + except Exception: + self.log.warning("Something went wrong", exc_info=True) + dialog = QtWidgets.QMessageBox(self) + dialog.setWindowTitle("Something went wrong") + dialog.setText("Something went wrong") + dialog.exec_() + + def _on_close_click(self): + self.reject() + + def showEvent(self, event): + super(WorkfileBuildPlaceholderDialog, self).showEvent(event) + if not self._first_refreshed: + self.refresh() + + if self._first_show: + self._first_show = False + self.setStyleSheet(style.load_stylesheet()) + self.resize(390, 450) diff --git a/openpype/tools/workfiles/files_widget.py b/openpype/tools/workfiles/files_widget.py index a5d5b14bb6..b7d31e4af4 100644 --- a/openpype/tools/workfiles/files_widget.py +++ b/openpype/tools/workfiles/files_widget.py @@ -8,12 +8,15 @@ from Qt import QtWidgets, QtCore from openpype.host import IWorkfileHost from openpype.client import get_asset_by_id +from openpype.pipeline.workfile.lock_workfile import ( + is_workfile_locked, + is_workfile_lock_enabled, + is_workfile_locked_for_current_process +) from openpype.tools.utils import PlaceholderLineEdit from openpype.tools.utils.delegates import PrettyTimeDelegate -from openpype.lib import ( - emit_event, - create_workdir_extra_folders, -) +from openpype.lib import emit_event +from openpype.tools.workfiles.lock_dialog import WorkfileLockDialog from openpype.pipeline import ( registered_host, legacy_io, @@ -23,7 +26,10 @@ from openpype.pipeline.context_tools import ( compute_session_changes, change_current_context ) -from openpype.pipeline.workfile import get_workfile_template_key +from openpype.pipeline.workfile import ( + get_workfile_template_key, + create_workdir_extra_folders, +) from .model import ( WorkAreaFilesModel, @@ -452,8 +458,19 @@ class FilesWidget(QtWidgets.QWidget): "host_name": self.host_name } + def _is_workfile_locked(self, filepath): + if not is_workfile_lock_enabled(self.host_name, self.project_name): + return False + if not is_workfile_locked(filepath): + return False + return not is_workfile_locked_for_current_process(filepath) + def open_file(self, filepath): host = self.host + if self._is_workfile_locked(filepath): + # add lockfile dialog + WorkfileLockDialog(filepath) + if isinstance(host, IWorkfileHost): has_unsaved_changes = host.workfile_has_unsaved_changes() else: @@ -561,7 +578,7 @@ class FilesWidget(QtWidgets.QWidget): src = self._get_selected_filepath() dst = os.path.join(self._workfiles_root, work_file) - shutil.copy(src, dst) + shutil.copyfile(src, dst) self.workfile_created.emit(dst) @@ -658,7 +675,7 @@ class FilesWidget(QtWidgets.QWidget): else: self.host.save_file(filepath) else: - shutil.copy(src_path, filepath) + shutil.copyfile(src_path, filepath) if isinstance(self.host, IWorkfileHost): self.host.open_workfile(filepath) else: diff --git a/openpype/tools/workfiles/lock_dialog.py b/openpype/tools/workfiles/lock_dialog.py new file mode 100644 index 0000000000..c574a74e32 --- /dev/null +++ b/openpype/tools/workfiles/lock_dialog.py @@ -0,0 +1,47 @@ +from Qt import QtWidgets, QtCore, QtGui +from openpype.style import load_stylesheet, get_app_icon_path + +from openpype.pipeline.workfile.lock_workfile import get_workfile_lock_data + + +class WorkfileLockDialog(QtWidgets.QDialog): + def __init__(self, workfile_path, parent=None): + super(WorkfileLockDialog, self).__init__(parent) + self.setWindowTitle("Warning") + icon = QtGui.QIcon(get_app_icon_path()) + self.setWindowIcon(icon) + + data = get_workfile_lock_data(workfile_path) + + message = "{} on {} machine is working on the same workfile.".format( + data["username"], + data["hostname"] + ) + + msg_label = QtWidgets.QLabel(message, self) + + btns_widget = QtWidgets.QWidget(self) + + cancel_btn = QtWidgets.QPushButton("Cancel", btns_widget) + ignore_btn = QtWidgets.QPushButton("Ignore lock", btns_widget) + + btns_layout = QtWidgets.QHBoxLayout(btns_widget) + btns_layout.setContentsMargins(0, 0, 0, 0) + btns_layout.setSpacing(10) + btns_layout.addStretch(1) + btns_layout.addWidget(cancel_btn, 0) + btns_layout.addWidget(ignore_btn, 0) + + main_layout = QtWidgets.QVBoxLayout(self) + main_layout.setContentsMargins(15, 15, 15, 15) + main_layout.addWidget(msg_label, 1, QtCore.Qt.AlignCenter), + main_layout.addSpacing(10) + main_layout.addWidget(btns_widget, 0) + + cancel_btn.clicked.connect(self.reject) + ignore_btn.clicked.connect(self.accept) + + def showEvent(self, event): + super(WorkfileLockDialog, self).showEvent(event) + + self.setStyleSheet(load_stylesheet()) diff --git a/openpype/version.py b/openpype/version.py index 0c114b6060..fd6e894fe2 100644 --- a/openpype/version.py +++ b/openpype/version.py @@ -1,3 +1,3 @@ # -*- coding: utf-8 -*- """Package declaring Pype version.""" -__version__ = "3.14.2-nightly.1" +__version__ = "3.14.3-nightly.4" diff --git a/openpype/widgets/attribute_defs/widgets.py b/openpype/widgets/attribute_defs/widgets.py index 60ae952553..dc697b08a6 100644 --- a/openpype/widgets/attribute_defs/widgets.py +++ b/openpype/widgets/attribute_defs/widgets.py @@ -108,10 +108,12 @@ class AttributeDefinitionsWidget(QtWidgets.QWidget): row = 0 for attr_def in attr_defs: - if attr_def.key in self._current_keys: - raise KeyError("Duplicated key \"{}\"".format(attr_def.key)) + if not isinstance(attr_def, UIDef): + if attr_def.key in self._current_keys: + raise KeyError( + "Duplicated key \"{}\"".format(attr_def.key)) - self._current_keys.add(attr_def.key) + self._current_keys.add(attr_def.key) widget = create_widget_for_attr_def(attr_def, self) expand_cols = 2 diff --git a/openpype/widgets/nice_checkbox.py b/openpype/widgets/nice_checkbox.py index ccd079c0fb..56e6d2ac24 100644 --- a/openpype/widgets/nice_checkbox.py +++ b/openpype/widgets/nice_checkbox.py @@ -111,14 +111,14 @@ class NiceCheckbox(QtWidgets.QFrame): return QtCore.QSize(width, height) def get_width_hint_by_height(self, height): - return ( - height / self._base_size.height() - ) * self._base_size.width() + return int(( + float(height) / self._base_size.height() + ) * self._base_size.width()) def get_height_hint_by_width(self, width): - return ( - width / self._base_size.width() - ) * self._base_size.height() + return int(( + float(width) / self._base_size.width() + ) * self._base_size.height()) def setFixedHeight(self, *args, **kwargs): self._fixed_height_set = True @@ -321,7 +321,7 @@ class NiceCheckbox(QtWidgets.QFrame): bg_color = self.unchecked_bg_color else: - offset_ratio = self._current_step / self._steps + offset_ratio = float(self._current_step) / self._steps # Animation bg bg_color = self.steped_color( self.checked_bg_color, @@ -332,7 +332,8 @@ class NiceCheckbox(QtWidgets.QFrame): margins_ratio = self._checker_margins_divider if margins_ratio > 0: size_without_margins = int( - (frame_rect.height() / margins_ratio) * (margins_ratio - 2) + (float(frame_rect.height()) / margins_ratio) + * (margins_ratio - 2) ) size_without_margins -= size_without_margins % 2 margin_size_c = ceil( @@ -434,21 +435,21 @@ class NiceCheckbox(QtWidgets.QFrame): def _get_enabled_icon_path( self, painter, checker_rect, step=None, half_steps=None ): - fifteenth = checker_rect.height() / 15 + fifteenth = float(checker_rect.height()) / 15 # Left point p1 = QtCore.QPoint( - checker_rect.x() + (5 * fifteenth), - checker_rect.y() + (9 * fifteenth) + int(checker_rect.x() + (5 * fifteenth)), + int(checker_rect.y() + (9 * fifteenth)) ) # Middle bottom point p2 = QtCore.QPoint( checker_rect.center().x(), - checker_rect.y() + (11 * fifteenth) + int(checker_rect.y() + (11 * fifteenth)) ) # Top right point p3 = QtCore.QPoint( - checker_rect.x() + (10 * fifteenth), - checker_rect.y() + (5 * fifteenth) + int(checker_rect.x() + (10 * fifteenth)), + int(checker_rect.y() + (5 * fifteenth)) ) if step is not None: multiplier = (half_steps - step) @@ -458,16 +459,16 @@ class NiceCheckbox(QtWidgets.QFrame): p3c = p3 - checker_rect.center() p1o = QtCore.QPoint( - (p1c.x() / half_steps) * multiplier, - (p1c.y() / half_steps) * multiplier + int((float(p1c.x()) / half_steps) * multiplier), + int((float(p1c.y()) / half_steps) * multiplier) ) p2o = QtCore.QPoint( - (p2c.x() / half_steps) * multiplier, - (p2c.y() / half_steps) * multiplier + int((float(p2c.x()) / half_steps) * multiplier), + int((float(p2c.y()) / half_steps) * multiplier) ) p3o = QtCore.QPoint( - (p3c.x() / half_steps) * multiplier, - (p3c.y() / half_steps) * multiplier + int((float(p3c.x()) / half_steps) * multiplier), + int((float(p3c.y()) / half_steps) * multiplier) ) p1 -= p1o @@ -484,11 +485,12 @@ class NiceCheckbox(QtWidgets.QFrame): self, painter, checker_rect, step=None, half_steps=None ): center_point = QtCore.QPointF( - checker_rect.width() / 2, checker_rect.height() / 2 + float(checker_rect.width()) / 2, + float(checker_rect.height()) / 2 ) - offset = ( + offset = float(( (center_point + QtCore.QPointF(0, 0)) / 2 - ).x() / 4 * 5 + ).x()) / 4 * 5 if step is not None: diff = center_point.x() - offset diff_offset = (diff / half_steps) * (half_steps - step) diff --git a/pyproject.toml b/pyproject.toml index 9ed1872eac..f74f40c561 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "OpenPype" -version = "3.14.2-nightly.1" # OpenPype +version = "3.14.2-nightly.2" # OpenPype description = "Open VFX and Animation pipeline with support." authors = ["OpenPype Team "] license = "MIT License" diff --git a/tests/lib/testing_classes.py b/tests/lib/testing_classes.py index 64676f62f4..85121f0d73 100644 --- a/tests/lib/testing_classes.py +++ b/tests/lib/testing_classes.py @@ -10,7 +10,7 @@ import glob import platform from tests.lib.db_handler import DBHandler -from tests.lib.file_handler import RemoteFileHandler +from distribution.file_handler import RemoteFileHandler class BaseTest: diff --git a/tools/ci_tools.py b/tools/ci_tools.py index 4c59cd6af6..c8f0cd48b4 100644 --- a/tools/ci_tools.py +++ b/tools/ci_tools.py @@ -101,7 +101,7 @@ def bump_file_versions(version): # bump pyproject.toml filename = "pyproject.toml" - regex = "version = \"(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(-((0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(\+([0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*))?\" # OpenPype" + regex = "version = \"(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(\+((0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*))?(\+([0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*))?\" # OpenPype" pyproject_version = f"version = \"{version}\" # OpenPype" file_regex_replace(filename, regex, pyproject_version) diff --git a/tools/create_env.ps1 b/tools/create_env.ps1 index 3f956e5c6a..cdb97d4942 100644 --- a/tools/create_env.ps1 +++ b/tools/create_env.ps1 @@ -68,6 +68,7 @@ function Install-Poetry() { } $env:POETRY_HOME="$openpype_root\.poetry" + $env:POETRY_VERSION="1.1.15" (Invoke-WebRequest -Uri https://install.python-poetry.org/ -UseBasicParsing).Content | & $($python) - } diff --git a/tools/create_env.sh b/tools/create_env.sh index fba3942b87..1ecd960fe1 100755 --- a/tools/create_env.sh +++ b/tools/create_env.sh @@ -109,6 +109,7 @@ detect_python () { install_poetry () { echo -e "${BIGreen}>>>${RST} Installing Poetry ..." export POETRY_HOME="$openpype_root/.poetry" + export POETRY_VERSION="1.1.15" command -v curl >/dev/null 2>&1 || { echo -e "${BIRed}!!!${RST}${BIYellow} Missing ${RST}${BIBlue}curl${BIYellow} command.${RST}"; return 1; } curl -sSL https://install.python-poetry.org/ | python - } diff --git a/tools/create_zip.ps1 b/tools/create_zip.ps1 index 7b852b7c54..e5ebc7678b 100644 --- a/tools/create_zip.ps1 +++ b/tools/create_zip.ps1 @@ -96,6 +96,7 @@ Write-Color -Text ">>> ", "Cleaning cache files ... " -Color Green, Gray -NoNewl Get-ChildItem $openpype_root -Filter "__pycache__" -Force -Recurse| Where-Object {( $_.FullName -inotmatch '\\build\\' ) -and ( $_.FullName -inotmatch '\\.venv' )} | Remove-Item -Force -Recurse Get-ChildItem $openpype_root -Filter "*.pyc" -Force -Recurse | Where-Object {( $_.FullName -inotmatch '\\build\\' ) -and ( $_.FullName -inotmatch '\\.venv' )} | Remove-Item -Force Get-ChildItem $openpype_root -Filter "*.pyo" -Force -Recurse | Where-Object {( $_.FullName -inotmatch '\\build\\' ) -and ( $_.FullName -inotmatch '\\.venv' )} | Remove-Item -Force +Write-Color -Text "OK" -Color Green Write-Color -Text ">>> ", "Generating zip from current sources ..." -Color Green, Gray $env:PYTHONPATH="$($openpype_root);$($env:PYTHONPATH)" diff --git a/tools/get_python_packages_info.py b/tools/get_python_packages_info.py new file mode 100644 index 0000000000..b4952840e6 --- /dev/null +++ b/tools/get_python_packages_info.py @@ -0,0 +1,83 @@ +# -*- coding: utf-8 -*- +"""Get version and license information on used Python packages. + +This is getting over all packages installed with Poetry and printing out +their name, version and available license information from PyPi in Markdown +table format. + +Usage: + ./.poetry/bin/poetry run python ./tools/get_python_packages_info.py + +""" + +import toml +import requests + + +packages = [] + +# define column headers +package_header = "Package" +version_header = "Version" +license_header = "License" + +name_col_width = len(package_header) +version_col_width = len(version_header) +license_col_width = len(license_header) + +# read lock file to get packages +with open("poetry.lock", "r") as fb: + lock_content = toml.load(fb) + + for package in lock_content["package"]: + # query pypi for license information + url = f"https://pypi.org/pypi/{package['name']}/json" + response = requests.get( + f"https://pypi.org/pypi/{package['name']}/json") + package_data = response.json() + version = package.get("version") or "N/A" + try: + package_license = package_data["info"].get("license") or "N/A" + except KeyError: + package_license = "N/A" + + if len(package_license) > 64: + package_license = f"{package_license[:32]}..." + packages.append( + ( + package["name"], + version, + package_license + ) + ) + + # update column width based on max string length + if len(package["name"]) > name_col_width: + name_col_width = len(package["name"]) + if len(version) > version_col_width: + version_col_width = len(version) + if len(package_license) > license_col_width: + license_col_width = len(package_license) + +# pad columns +name_col_width += 2 +version_col_width += 2 +license_col_width += 2 + +# print table header +print((f"|{package_header.center(name_col_width)}" + f"|{version_header.center(version_col_width)}" + f"|{license_header.center(license_col_width)}|")) + +print( + "|" + ("-" * len(package_header.center(name_col_width))) + + "|" + ("-" * len(version_header.center(version_col_width))) + + "|" + ("-" * len(license_header.center(license_col_width))) + "|") + +# print rest of the table +for package in packages: + print(( + f"|{package[0].center(name_col_width)}" + f"|{package[1].center(version_col_width)}" + f"|{package[2].center(license_col_width)}|" + )) diff --git a/website/docs/admin_hosts_houdini.md b/website/docs/admin_hosts_houdini.md new file mode 100644 index 0000000000..64c54db591 --- /dev/null +++ b/website/docs/admin_hosts_houdini.md @@ -0,0 +1,11 @@ +--- +id: admin_hosts_houdini +title: Houdini +sidebar_label: Houdini +--- + +## Shelves Manager +You can add your custom shelf set into Houdini by setting your shelf sets, shelves and tools in **Houdini -> Shelves Manager**. +![Custom menu definition](assets/houdini-admin_shelvesmanager.png) + +The Shelf Set Path is used to load a .shelf file to generate your shelf set. If the path is specified, you don't have to set the shelves and tools. \ No newline at end of file diff --git a/website/docs/assets/houdini-admin_shelvesmanager.png b/website/docs/assets/houdini-admin_shelvesmanager.png new file mode 100644 index 0000000000..ba2f15a6a5 Binary files /dev/null and b/website/docs/assets/houdini-admin_shelvesmanager.png differ diff --git a/website/sidebars.js b/website/sidebars.js index 920a3134f6..af64282d61 100644 --- a/website/sidebars.js +++ b/website/sidebars.js @@ -101,6 +101,7 @@ module.exports = { items: [ "admin_hosts_blender", "admin_hosts_hiero", + "admin_hosts_houdini", "admin_hosts_maya", "admin_hosts_nuke", "admin_hosts_resolve", @@ -142,7 +143,7 @@ module.exports = { ], }, ], - Dev: [ + Dev: [ "dev_introduction", "dev_requirements", "dev_build", @@ -157,5 +158,5 @@ module.exports = { "dev_publishing" ] } - ] + ] };