diff --git a/.dockerignore b/.dockerignore index 07c1c151ce..9c506b9964 100644 --- a/.dockerignore +++ b/.dockerignore @@ -87,7 +87,7 @@ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: -# .python-version +.python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. @@ -142,5 +142,6 @@ cython_debug/ .poetry/ .github/ vendor/bin/ +vendor/python/ docs/ website/ diff --git a/.gitignore b/.gitignore index 221a2f2241..fa3fae1ad2 100644 --- a/.gitignore +++ b/.gitignore @@ -39,6 +39,7 @@ Temporary Items /dist/ /vendor/bin/* +/vendor/python/* /.venv /venv/ diff --git a/CHANGELOG.md b/CHANGELOG.md index 32943834d0..95792f8a7a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,18 +1,29 @@ # Changelog -## [3.5.0-nightly.4](https://github.com/pypeclub/OpenPype/tree/HEAD) +## [3.5.0](https://github.com/pypeclub/OpenPype/tree/3.5.0) (2021-10-17) -[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.4.1...HEAD) +[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.4.1...3.5.0) + +**Deprecated:** + +- Maya: Change mayaAscii family to mayaScene [\#2106](https://github.com/pypeclub/OpenPype/pull/2106) **πŸ†• New features** +- Added project and task into context change message in Maya [\#2131](https://github.com/pypeclub/OpenPype/pull/2131) +- Add ExtractBurnin to photoshop review [\#2124](https://github.com/pypeclub/OpenPype/pull/2124) +- PYPE-1218 - changed namespace to contain subset name in Maya [\#2114](https://github.com/pypeclub/OpenPype/pull/2114) - Added running configurable disk mapping command before start of OP [\#2091](https://github.com/pypeclub/OpenPype/pull/2091) - SFTP provider [\#2073](https://github.com/pypeclub/OpenPype/pull/2073) - Maya: Validate setdress top group [\#2068](https://github.com/pypeclub/OpenPype/pull/2068) **πŸš€ Enhancements** +- Maya: make rig validators configurable in settings [\#2137](https://github.com/pypeclub/OpenPype/pull/2137) +- Settings: Updated readme for entity types in settings [\#2132](https://github.com/pypeclub/OpenPype/pull/2132) +- Nuke: unified clip loader [\#2128](https://github.com/pypeclub/OpenPype/pull/2128) - Settings UI: Project model refreshing and sorting [\#2104](https://github.com/pypeclub/OpenPype/pull/2104) +- Create Read From Rendered - Disable Relative paths by default [\#2093](https://github.com/pypeclub/OpenPype/pull/2093) - Added choosing different dirmap mapping if workfile synched locally [\#2088](https://github.com/pypeclub/OpenPype/pull/2088) - General: Remove IdleManager module [\#2084](https://github.com/pypeclub/OpenPype/pull/2084) - Tray UI: Message box about missing settings defaults [\#2080](https://github.com/pypeclub/OpenPype/pull/2080) @@ -23,26 +34,33 @@ - Nuke: Adding `still` image family workflow [\#2064](https://github.com/pypeclub/OpenPype/pull/2064) - Maya: validate authorized loaded plugins [\#2062](https://github.com/pypeclub/OpenPype/pull/2062) - Tools: add support for pyenv on windows [\#2051](https://github.com/pypeclub/OpenPype/pull/2051) -- SyncServer: Dropbox Provider [\#1979](https://github.com/pypeclub/OpenPype/pull/1979) **πŸ› Bug fixes** +- Maya: fix model publishing [\#2130](https://github.com/pypeclub/OpenPype/pull/2130) +- Fix - oiiotool wasn't recognized even if present [\#2129](https://github.com/pypeclub/OpenPype/pull/2129) +- General: Disk mapping group [\#2120](https://github.com/pypeclub/OpenPype/pull/2120) +- Hiero: publishing effect first time makes wrong resources path [\#2115](https://github.com/pypeclub/OpenPype/pull/2115) +- Add startup script for Houdini Core. [\#2110](https://github.com/pypeclub/OpenPype/pull/2110) - TVPaint: Behavior name of loop also accept repeat [\#2109](https://github.com/pypeclub/OpenPype/pull/2109) - Ftrack: Project settings save custom attributes skip unknown attributes [\#2103](https://github.com/pypeclub/OpenPype/pull/2103) +- Blender: Fix NoneType error when animation\_data is missing for a rig [\#2101](https://github.com/pypeclub/OpenPype/pull/2101) - Fix broken import in sftp provider [\#2100](https://github.com/pypeclub/OpenPype/pull/2100) - Global: Fix docstring on publish plugin extract review [\#2097](https://github.com/pypeclub/OpenPype/pull/2097) +- Delivery Action Files Sequence fix [\#2096](https://github.com/pypeclub/OpenPype/pull/2096) - General: Cloud mongo ca certificate issue [\#2095](https://github.com/pypeclub/OpenPype/pull/2095) - TVPaint: Creator use context from workfile [\#2087](https://github.com/pypeclub/OpenPype/pull/2087) - Blender: fix texture missing when publishing blend files [\#2085](https://github.com/pypeclub/OpenPype/pull/2085) - General: Startup validations oiio tool path fix on linux [\#2083](https://github.com/pypeclub/OpenPype/pull/2083) +- Deadline: Collect deadline server does not check existence of deadline key [\#2082](https://github.com/pypeclub/OpenPype/pull/2082) - Blender: fixed Curves with modifiers in Rigs [\#2081](https://github.com/pypeclub/OpenPype/pull/2081) +- Nuke UI scaling [\#2077](https://github.com/pypeclub/OpenPype/pull/2077) +- Maya: Fix multi-camera renders [\#2065](https://github.com/pypeclub/OpenPype/pull/2065) - Fix Sync Queue when project disabled [\#2063](https://github.com/pypeclub/OpenPype/pull/2063) **Merged pull requests:** -- Delivery Action Files Sequence fix [\#2096](https://github.com/pypeclub/OpenPype/pull/2096) - Bump pywin32 from 300 to 301 [\#2086](https://github.com/pypeclub/OpenPype/pull/2086) -- Nuke UI scaling [\#2077](https://github.com/pypeclub/OpenPype/pull/2077) ## [3.4.1](https://github.com/pypeclub/OpenPype/tree/3.4.1) (2021-09-23) @@ -60,7 +78,6 @@ - Settings UI: Deffered set value on entity [\#2044](https://github.com/pypeclub/OpenPype/pull/2044) - Loader: Families filtering [\#2043](https://github.com/pypeclub/OpenPype/pull/2043) - Settings UI: Project view enhancements [\#2042](https://github.com/pypeclub/OpenPype/pull/2042) -- Added possibility to configure of synchronization of workfile version… [\#2041](https://github.com/pypeclub/OpenPype/pull/2041) - Settings for Nuke IncrementScriptVersion [\#2039](https://github.com/pypeclub/OpenPype/pull/2039) - Loader & Library loader: Use tools from OpenPype [\#2038](https://github.com/pypeclub/OpenPype/pull/2038) - Adding predefined project folders creation in PM [\#2030](https://github.com/pypeclub/OpenPype/pull/2030) @@ -75,7 +92,6 @@ - Differentiate jpg sequences from thumbnail [\#2056](https://github.com/pypeclub/OpenPype/pull/2056) - FFmpeg: Split command to list does not work [\#2046](https://github.com/pypeclub/OpenPype/pull/2046) - Removed shell flag in subprocess call [\#2045](https://github.com/pypeclub/OpenPype/pull/2045) -- Hiero: Fix "none" named tags [\#2033](https://github.com/pypeclub/OpenPype/pull/2033) **Merged pull requests:** @@ -85,16 +101,13 @@ [Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.4.0-nightly.6...3.4.0) -### πŸ“– Documentation - -- Documentation: Ftrack launch argsuments update [\#2014](https://github.com/pypeclub/OpenPype/pull/2014) - **πŸ†• New features** - Nuke: Compatibility with Nuke 13 [\#2003](https://github.com/pypeclub/OpenPype/pull/2003) **πŸš€ Enhancements** +- Added possibility to configure of synchronization of workfile version… [\#2041](https://github.com/pypeclub/OpenPype/pull/2041) - General: Task types in profiles [\#2036](https://github.com/pypeclub/OpenPype/pull/2036) - Console interpreter: Handle invalid sizes on initialization [\#2022](https://github.com/pypeclub/OpenPype/pull/2022) - Ftrack: Show OpenPype versions in event server status [\#2019](https://github.com/pypeclub/OpenPype/pull/2019) @@ -103,28 +116,20 @@ - Modules: Connect method is not required [\#2009](https://github.com/pypeclub/OpenPype/pull/2009) - Settings UI: Number with configurable steps [\#2001](https://github.com/pypeclub/OpenPype/pull/2001) - Moving project folder structure creation out of ftrack module \#1989 [\#1996](https://github.com/pypeclub/OpenPype/pull/1996) -- Configurable items for providers without Settings [\#1987](https://github.com/pypeclub/OpenPype/pull/1987) -- Global: Example addons [\#1986](https://github.com/pypeclub/OpenPype/pull/1986) -- Standalone Publisher: Extract harmony zip handle workfile template [\#1982](https://github.com/pypeclub/OpenPype/pull/1982) -- Settings UI: Number sliders [\#1978](https://github.com/pypeclub/OpenPype/pull/1978) **πŸ› Bug fixes** - Workfiles tool: Task selection [\#2040](https://github.com/pypeclub/OpenPype/pull/2040) - Ftrack: Delete old versions missing settings key [\#2037](https://github.com/pypeclub/OpenPype/pull/2037) - Nuke: typo on a button [\#2034](https://github.com/pypeclub/OpenPype/pull/2034) +- Hiero: Fix "none" named tags [\#2033](https://github.com/pypeclub/OpenPype/pull/2033) - FFmpeg: Subprocess arguments as list [\#2032](https://github.com/pypeclub/OpenPype/pull/2032) - General: Fix Python 2 breaking line [\#2016](https://github.com/pypeclub/OpenPype/pull/2016) - Bugfix/webpublisher task type [\#2006](https://github.com/pypeclub/OpenPype/pull/2006) -- Nuke thumbnails generated from middle of the sequence [\#1992](https://github.com/pypeclub/OpenPype/pull/1992) -- Nuke: last version from path gets correct version [\#1990](https://github.com/pypeclub/OpenPype/pull/1990) -- nuke, resolve, hiero: precollector order lest then 0.5 [\#1984](https://github.com/pypeclub/OpenPype/pull/1984) -- Last workfile with multiple work templates [\#1981](https://github.com/pypeclub/OpenPype/pull/1981) -- Collectors order [\#1977](https://github.com/pypeclub/OpenPype/pull/1977) -- Stop timer was within validator order range. [\#1975](https://github.com/pypeclub/OpenPype/pull/1975) -- Ftrack: arrow submodule has https url source [\#1974](https://github.com/pypeclub/OpenPype/pull/1974) -- Ftrack: Fix hosts attribute in collect ftrack username [\#1972](https://github.com/pypeclub/OpenPype/pull/1972) -- Deadline: Houdini plugins in different hierarchy [\#1970](https://github.com/pypeclub/OpenPype/pull/1970) + +### πŸ“– Documentation + +- Documentation: Ftrack launch argsuments update [\#2014](https://github.com/pypeclub/OpenPype/pull/2014) ## [3.3.1](https://github.com/pypeclub/OpenPype/tree/3.3.1) (2021-08-20) diff --git a/Dockerfile b/Dockerfile index 2d8ed27b15..cef83b5811 100644 --- a/Dockerfile +++ b/Dockerfile @@ -1,7 +1,9 @@ # Build Pype docker image -FROM centos:7 AS builder -ARG OPENPYPE_PYTHON_VERSION=3.7.10 +FROM debian:bookworm-slim AS builder +ARG OPENPYPE_PYTHON_VERSION=3.7.12 +LABEL maintainer="info@openpype.io" +LABEL description="Docker Image to build and run OpenPype" LABEL org.opencontainers.image.name="pypeclub/openpype" LABEL org.opencontainers.image.title="OpenPype Docker Image" LABEL org.opencontainers.image.url="https://openpype.io/" @@ -9,56 +11,49 @@ LABEL org.opencontainers.image.source="https://github.com/pypeclub/pype" USER root -# update base -RUN yum -y install deltarpm \ - && yum -y update \ - && yum clean all +ARG DEBIAN_FRONTEND=noninteractive -# add tools we need -RUN yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \ - && yum -y install centos-release-scl \ - && yum -y install \ +# update base +RUN apt-get update \ + && apt-get install -y --no-install-recommends \ + ca-certificates \ bash \ - which \ git \ - devtoolset-7-gcc* \ - make \ cmake \ + make \ curl \ wget \ - gcc \ - zlib-devel \ - bzip2 \ - bzip2-devel \ - readline-devel \ - sqlite sqlite-devel \ - openssl-devel \ - tk-devel libffi-devel \ - qt5-qtbase-devel \ - patchelf \ - && yum clean all + build-essential \ + checkinstall \ + libssl-dev \ + zlib1g-dev \ + libbz2-dev \ + libreadline-dev \ + libsqlite3-dev \ + llvm \ + libncursesw5-dev \ + xz-utils \ + tk-dev \ + libxml2-dev \ + libxmlsec1-dev \ + libffi-dev \ + liblzma-dev \ + patchelf + +SHELL ["/bin/bash", "-c"] RUN mkdir /opt/openpype -# RUN useradd -m pype -# RUN chown pype /opt/openpype -# USER pype -RUN curl https://pyenv.run | bash -ENV PYTHON_CONFIGURE_OPTS --enable-shared - -RUN echo 'export PATH="$HOME/.pyenv/bin:$PATH"'>> $HOME/.bashrc \ +RUN curl https://pyenv.run | bash \ + && echo 'export PATH="$HOME/.pyenv/bin:$PATH"'>> $HOME/.bashrc \ && echo 'eval "$(pyenv init -)"' >> $HOME/.bashrc \ && echo 'eval "$(pyenv virtualenv-init -)"' >> $HOME/.bashrc \ - && echo 'eval "$(pyenv init --path)"' >> $HOME/.bashrc -RUN source $HOME/.bashrc && pyenv install ${OPENPYPE_PYTHON_VERSION} + && echo 'eval "$(pyenv init --path)"' >> $HOME/.bashrc \ + && source $HOME/.bashrc && pyenv install ${OPENPYPE_PYTHON_VERSION} COPY . /opt/openpype/ -RUN rm -rf /openpype/.poetry || echo "No Poetry installed yet." -# USER root -# RUN chown -R pype /opt/openpype -RUN chmod +x /opt/openpype/tools/create_env.sh && chmod +x /opt/openpype/tools/build.sh -# USER pype +RUN chmod +x /opt/openpype/tools/create_env.sh && chmod +x /opt/openpype/tools/build.sh WORKDIR /opt/openpype @@ -67,16 +62,8 @@ RUN cd /opt/openpype \ && pyenv local ${OPENPYPE_PYTHON_VERSION} RUN source $HOME/.bashrc \ - && ./tools/create_env.sh - -RUN source $HOME/.bashrc \ + && ./tools/create_env.sh \ && ./tools/fetch_thirdparty_libs.sh RUN source $HOME/.bashrc \ - && bash ./tools/build.sh \ - && cp /usr/lib64/libffi* ./build/exe.linux-x86_64-3.7/lib \ - && cp /usr/lib64/libssl* ./build/exe.linux-x86_64-3.7/lib \ - && cp /usr/lib64/libcrypto* ./build/exe.linux-x86_64-3.7/lib - -RUN cd /opt/openpype \ - rm -rf ./vendor/bin + && bash ./tools/build.sh diff --git a/Dockerfile.centos7 b/Dockerfile.centos7 new file mode 100644 index 0000000000..f3b257e66b --- /dev/null +++ b/Dockerfile.centos7 @@ -0,0 +1,98 @@ +# Build Pype docker image +FROM centos:7 AS builder +ARG OPENPYPE_PYTHON_VERSION=3.7.10 + +LABEL org.opencontainers.image.name="pypeclub/openpype" +LABEL org.opencontainers.image.title="OpenPype Docker Image" +LABEL org.opencontainers.image.url="https://openpype.io/" +LABEL org.opencontainers.image.source="https://github.com/pypeclub/pype" + +USER root + +# update base +RUN yum -y install deltarpm \ + && yum -y update \ + && yum clean all + +# add tools we need +RUN yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm \ + && yum -y install centos-release-scl \ + && yum -y install \ + bash \ + which \ + git \ + make \ + devtoolset-7 \ + cmake \ + curl \ + wget \ + gcc \ + zlib-devel \ + bzip2 \ + bzip2-devel \ + readline-devel \ + sqlite sqlite-devel \ + openssl-devel \ + openssl-libs \ + tk-devel libffi-devel \ + patchelf \ + automake \ + autoconf \ + ncurses \ + ncurses-devel \ + qt5-qtbase-devel \ + && yum clean all + +# we need to build our own patchelf +WORKDIR /temp-patchelf +RUN git clone https://github.com/NixOS/patchelf.git . \ + && source scl_source enable devtoolset-7 \ + && ./bootstrap.sh \ + && ./configure \ + && make \ + && make install + +RUN mkdir /opt/openpype +# RUN useradd -m pype +# RUN chown pype /opt/openpype +# USER pype + +RUN curl https://pyenv.run | bash +# ENV PYTHON_CONFIGURE_OPTS --enable-shared + +RUN echo 'export PATH="$HOME/.pyenv/bin:$PATH"'>> $HOME/.bashrc \ + && echo 'eval "$(pyenv init -)"' >> $HOME/.bashrc \ + && echo 'eval "$(pyenv virtualenv-init -)"' >> $HOME/.bashrc \ + && echo 'eval "$(pyenv init --path)"' >> $HOME/.bashrc +RUN source $HOME/.bashrc && pyenv install ${OPENPYPE_PYTHON_VERSION} + +COPY . /opt/openpype/ +RUN rm -rf /openpype/.poetry || echo "No Poetry installed yet." +# USER root +# RUN chown -R pype /opt/openpype +RUN chmod +x /opt/openpype/tools/create_env.sh && chmod +x /opt/openpype/tools/build.sh + +# USER pype + +WORKDIR /opt/openpype + +RUN cd /opt/openpype \ + && source $HOME/.bashrc \ + && pyenv local ${OPENPYPE_PYTHON_VERSION} + +RUN source $HOME/.bashrc \ + && ./tools/create_env.sh + +RUN source $HOME/.bashrc \ + && ./tools/fetch_thirdparty_libs.sh + +RUN source $HOME/.bashrc \ + && bash ./tools/build.sh + +RUN cp /usr/lib64/libffi* ./build/exe.linux-x86_64-3.7/lib \ + && cp /usr/lib64/libssl* ./build/exe.linux-x86_64-3.7/lib \ + && cp /usr/lib64/libcrypto* ./build/exe.linux-x86_64-3.7/lib \ + && cp /root/.pyenv/versions/${OPENPYPE_PYTHON_VERSION}/lib/libpython* ./build/exe.linux-x86_64-3.7/lib + +RUN cd /opt/openpype \ + rm -rf ./vendor/bin diff --git a/README.md b/README.md index 209af24c75..0e450fc48d 100644 --- a/README.md +++ b/README.md @@ -133,6 +133,12 @@ Easiest way to build OpenPype on Linux is using [Docker](https://www.docker.com/ sudo ./tools/docker_build.sh ``` +This will by default use Debian as base image. If you need to make Centos 7 compatible build, please run: + +```sh +sudo ./tools/docker_build.sh centos7 +``` + If all is successful, you'll find built OpenPype in `./build/` folder. #### Manual build @@ -158,6 +164,11 @@ you'll need also additional libraries for Qt5: ```sh sudo apt install qt5-default ``` +or if you are on Ubuntu > 20.04, there is no `qt5-default` packages so you need to install its content individually: + +```sh +sudo apt-get install qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools +```
diff --git a/openpype/__init__.py b/openpype/__init__.py index 9d55006a67..11b563ebfe 100644 --- a/openpype/__init__.py +++ b/openpype/__init__.py @@ -69,6 +69,7 @@ def install(): """Install Pype to Avalon.""" from pyblish.lib import MessageHandler from openpype.modules import load_modules + from avalon import pipeline # Make sure modules are loaded load_modules() @@ -117,7 +118,9 @@ def install(): # apply monkey patched discover to original one log.info("Patching discovery") + avalon.discover = patched_discover + pipeline.discover = patched_discover avalon.on("taskChanged", _on_task_change) diff --git a/openpype/cli.py b/openpype/cli.py index 18cc1c63cd..c69407e295 100644 --- a/openpype/cli.py +++ b/openpype/cli.py @@ -283,3 +283,18 @@ def run(script): args_string = " ".join(args[1:]) print(f"... running: {script} {args_string}") runpy.run_path(script, run_name="__main__", ) + + +@main.command() +@click.argument("folder", nargs=-1) +@click.option("-m", + "--mark", + help="Run tests marked by", + default=None) +@click.option("-p", + "--pyargs", + help="Run tests from package", + default=None) +def runtests(folder, mark, pyargs): + """Run all automatic tests after proper initialization via start.py""" + PypeCommands().run_tests(folder, mark, pyargs) diff --git a/openpype/hooks/pre_global_host_data.py b/openpype/hooks/pre_global_host_data.py index c669d91ad5..b32fb5e44a 100644 --- a/openpype/hooks/pre_global_host_data.py +++ b/openpype/hooks/pre_global_host_data.py @@ -43,6 +43,8 @@ class GlobalHostDataHook(PreLaunchHook): "env": self.launch_context.env, + "last_workfile_path": self.data.get("last_workfile_path"), + "log": self.log }) diff --git a/openpype/hosts/blender/plugins/load/load_rig.py b/openpype/hosts/blender/plugins/load/load_rig.py index c385dc237e..6062c293df 100644 --- a/openpype/hosts/blender/plugins/load/load_rig.py +++ b/openpype/hosts/blender/plugins/load/load_rig.py @@ -111,7 +111,8 @@ class BlendRigLoader(plugin.AssetLoader): if action is not None: local_obj.animation_data.action = action - elif local_obj.animation_data.action is not None: + elif (local_obj.animation_data and + local_obj.animation_data.action is not None): plugin.prepare_data( local_obj.animation_data.action, group_name) diff --git a/openpype/hosts/houdini/startup/scripts/houdinicore.py b/openpype/hosts/houdini/startup/scripts/houdinicore.py new file mode 100644 index 0000000000..4233d68c15 --- /dev/null +++ b/openpype/hosts/houdini/startup/scripts/houdinicore.py @@ -0,0 +1,9 @@ +from avalon import api, houdini + + +def main(): + print("Installing OpenPype ...") + api.install(houdini) + + +main() diff --git a/openpype/hosts/maya/api/__init__.py b/openpype/hosts/maya/api/__init__.py index 725a075417..13f8a4cb78 100644 --- a/openpype/hosts/maya/api/__init__.py +++ b/openpype/hosts/maya/api/__init__.py @@ -313,9 +313,15 @@ def on_task_changed(*args): lib.set_context_settings() lib.update_content_on_context_change() + msg = " project: {}\n asset: {}\n task:{}".format( + avalon.Session["AVALON_PROJECT"], + avalon.Session["AVALON_ASSET"], + avalon.Session["AVALON_TASK"] + ) + lib.show_message( "Context was changed", - ("Context was changed to {}".format(avalon.Session["AVALON_ASSET"])), + ("Context was changed to:\n{}".format(msg)), ) diff --git a/openpype/hosts/maya/api/lib_renderproducts.py b/openpype/hosts/maya/api/lib_renderproducts.py index fb99584c5d..4983109d58 100644 --- a/openpype/hosts/maya/api/lib_renderproducts.py +++ b/openpype/hosts/maya/api/lib_renderproducts.py @@ -114,6 +114,8 @@ class RenderProduct(object): aov = attr.ib(default=None) # source aov driver = attr.ib(default=None) # source driver multipart = attr.ib(default=False) # multichannel file + camera = attr.ib(default=None) # used only when rendering + # from multiple cameras def get(layer, render_instance=None): @@ -183,6 +185,16 @@ class ARenderProducts: self.layer_data = self._get_layer_data() self.layer_data.products = self.get_render_products() + def has_camera_token(self): + # type: () -> bool + """Check if camera token is in image prefix. + + Returns: + bool: True/False if camera token is present. + + """ + return "" in self.layer_data.filePrefix.lower() + @abstractmethod def get_render_products(self): """To be implemented by renderer class. @@ -307,7 +319,7 @@ class ARenderProducts: # Deadline allows submitting renders with a custom frame list # to support those cases we might want to allow 'custom frames' # to be overridden to `ExpectFiles` class? - layer_data = LayerMetadata( + return LayerMetadata( frameStart=int(self.get_render_attribute("startFrame")), frameEnd=int(self.get_render_attribute("endFrame")), frameStep=int(self.get_render_attribute("byFrameStep")), @@ -321,7 +333,6 @@ class ARenderProducts: defaultExt=self._get_attr("defaultRenderGlobals.imfPluginKey"), filePrefix=file_prefix ) - return layer_data def _generate_file_sequence( self, layer_data, @@ -330,7 +341,7 @@ class ARenderProducts: force_cameras=None): # type: (LayerMetadata, str, str, list) -> list expected_files = [] - cameras = force_cameras if force_cameras else layer_data.cameras + cameras = force_cameras or layer_data.cameras ext = force_ext or layer_data.defaultExt for cam in cameras: file_prefix = layer_data.filePrefix @@ -361,8 +372,8 @@ class ARenderProducts: ) return expected_files - def get_files(self, product, camera): - # type: (RenderProduct, str) -> list + def get_files(self, product): + # type: (RenderProduct) -> list """Return list of expected files. It will translate render token strings ('', etc.) to @@ -373,7 +384,6 @@ class ARenderProducts: Args: product (RenderProduct): Render product to be used for file generation. - camera (str): Camera name. Returns: List of files @@ -383,7 +393,7 @@ class ARenderProducts: self.layer_data, force_aov_name=product.productName, force_ext=product.ext, - force_cameras=[camera] + force_cameras=[product.camera] ) def get_renderable_cameras(self): @@ -460,15 +470,21 @@ class RenderProductsArnold(ARenderProducts): return prefix - def _get_aov_render_products(self, aov): + def _get_aov_render_products(self, aov, cameras=None): """Return all render products for the AOV""" - products = list() + products = [] aov_name = self._get_attr(aov, "name") ai_drivers = cmds.listConnections("{}.outputs".format(aov), source=True, destination=False, type="aiAOVDriver") or [] + if not cameras: + cameras = [ + self.sanitize_camera_name( + self.get_renderable_cameras()[0] + ) + ] for ai_driver in ai_drivers: # todo: check aiAOVDriver.prefix as it could have @@ -497,30 +513,37 @@ class RenderProductsArnold(ARenderProducts): name = "beauty" # Support Arnold light groups for AOVs - # Global AOV: When disabled the main layer is not written: `{pass}` + # Global AOV: When disabled the main layer is + # not written: `{pass}` # All Light Groups: When enabled, a `{pass}_lgroups` file is - # written and is always merged into a single file - # Light Groups List: When set, a product per light group is written + # written and is always merged into a + # single file + # Light Groups List: When set, a product per light + # group is written # e.g. {pass}_front, {pass}_rim global_aov = self._get_attr(aov, "globalAov") if global_aov: - product = RenderProduct(productName=name, - ext=ext, - aov=aov_name, - driver=ai_driver) - products.append(product) + for camera in cameras: + product = RenderProduct(productName=name, + ext=ext, + aov=aov_name, + driver=ai_driver, + camera=camera) + products.append(product) all_light_groups = self._get_attr(aov, "lightGroups") if all_light_groups: # All light groups is enabled. A single multipart # Render Product - product = RenderProduct(productName=name + "_lgroups", - ext=ext, - aov=aov_name, - driver=ai_driver, - # Always multichannel output - multipart=True) - products.append(product) + for camera in cameras: + product = RenderProduct(productName=name + "_lgroups", + ext=ext, + aov=aov_name, + driver=ai_driver, + # Always multichannel output + multipart=True, + camera=camera) + products.append(product) else: value = self._get_attr(aov, "lightGroupsList") if not value: @@ -529,11 +552,15 @@ class RenderProductsArnold(ARenderProducts): for light_group in selected_light_groups: # Render Product per selected light group aov_light_group_name = "{}_{}".format(name, light_group) - product = RenderProduct(productName=aov_light_group_name, - aov=aov_name, - driver=ai_driver, - ext=ext) - products.append(product) + for camera in cameras: + product = RenderProduct( + productName=aov_light_group_name, + aov=aov_name, + driver=ai_driver, + ext=ext, + camera=camera + ) + products.append(product) return products @@ -556,17 +583,26 @@ class RenderProductsArnold(ARenderProducts): # anyway. return [] - default_ext = self._get_attr("defaultRenderGlobals.imfPluginKey") - beauty_product = RenderProduct(productName="beauty", - ext=default_ext, - driver="defaultArnoldDriver") + # check if camera token is in prefix. If so, and we have list of + # renderable cameras, generate render product for each and every + # of them. + cameras = [ + self.sanitize_camera_name(c) + for c in self.get_renderable_cameras() + ] + default_ext = self._get_attr("defaultRenderGlobals.imfPluginKey") + beauty_products = [RenderProduct( + productName="beauty", + ext=default_ext, + driver="defaultArnoldDriver", + camera=camera) for camera in cameras] # AOVs > Legacy > Maya Render View > Mode aovs_enabled = bool( self._get_attr("defaultArnoldRenderOptions.aovMode") ) if not aovs_enabled: - return [beauty_product] + return beauty_products # Common > File Output > Merge AOVs or # We don't need to check for Merge AOVs due to overridden @@ -575,8 +611,9 @@ class RenderProductsArnold(ARenderProducts): "" in self.layer_data.filePrefix.lower() ) if not has_renderpass_token: - beauty_product.multipart = True - return [beauty_product] + for product in beauty_products: + product.multipart = True + return beauty_products # AOVs are set to be rendered separately. We should expect # token in path. @@ -598,14 +635,14 @@ class RenderProductsArnold(ARenderProducts): continue # For now stick to the legacy output format. - aov_products = self._get_aov_render_products(aov) + aov_products = self._get_aov_render_products(aov, cameras) products.extend(aov_products) - if not any(product.aov == "RGBA" for product in products): + if all(product.aov != "RGBA" for product in products): # Append default 'beauty' as this is arnolds default. # However, it is excluded whenever a RGBA pass is enabled. # For legibility add the beauty layer as first entry - products.insert(0, beauty_product) + products += beauty_products # TODO: Output Denoising AOVs? @@ -670,6 +707,11 @@ class RenderProductsVray(ARenderProducts): # anyway. return [] + cameras = [ + self.sanitize_camera_name(c) + for c in self.get_renderable_cameras() + ] + image_format_str = self._get_attr("vraySettings.imageFormatStr") default_ext = image_format_str if default_ext in {"exr (multichannel)", "exr (deep)"}: @@ -680,13 +722,21 @@ class RenderProductsVray(ARenderProducts): # add beauty as default when not disabled dont_save_rgb = self._get_attr("vraySettings.dontSaveRgbChannel") if not dont_save_rgb: - products.append(RenderProduct(productName="", ext=default_ext)) + for camera in cameras: + products.append( + RenderProduct(productName="", + ext=default_ext, + camera=camera)) # separate alpha file separate_alpha = self._get_attr("vraySettings.separateAlpha") if separate_alpha: - products.append(RenderProduct(productName="Alpha", - ext=default_ext)) + for camera in cameras: + products.append( + RenderProduct(productName="Alpha", + ext=default_ext, + camera=camera) + ) if image_format_str == "exr (multichannel)": # AOVs are merged in m-channel file, only main layer is rendered @@ -716,19 +766,23 @@ class RenderProductsVray(ARenderProducts): # instead seems to output multiple Render Products, # specifically "Self_Illumination" and "Environment" product_names = ["Self_Illumination", "Environment"] - for name in product_names: - product = RenderProduct(productName=name, - ext=default_ext, - aov=aov) - products.append(product) + for camera in cameras: + for name in product_names: + product = RenderProduct(productName=name, + ext=default_ext, + aov=aov, + camera=camera) + products.append(product) # Continue as we've processed this special case AOV continue aov_name = self._get_vray_aov_name(aov) - product = RenderProduct(productName=aov_name, - ext=default_ext, - aov=aov) - products.append(product) + for camera in cameras: + product = RenderProduct(productName=aov_name, + ext=default_ext, + aov=aov, + camera=camera) + products.append(product) return products @@ -875,6 +929,11 @@ class RenderProductsRedshift(ARenderProducts): # anyway. return [] + cameras = [ + self.sanitize_camera_name(c) + for c in self.get_renderable_cameras() + ] + # For Redshift we don't directly return upon forcing multilayer # due to some AOVs still being written into separate files, # like Cryptomatte. @@ -933,11 +992,14 @@ class RenderProductsRedshift(ARenderProducts): for light_group in light_groups: aov_light_group_name = "{}_{}".format(aov_name, light_group) - product = RenderProduct(productName=aov_light_group_name, - aov=aov_name, - ext=ext, - multipart=aov_multipart) - products.append(product) + for camera in cameras: + product = RenderProduct( + productName=aov_light_group_name, + aov=aov_name, + ext=ext, + multipart=aov_multipart, + camera=camera) + products.append(product) if light_groups: light_groups_enabled = True @@ -945,11 +1007,13 @@ class RenderProductsRedshift(ARenderProducts): # Redshift AOV Light Select always renders the global AOV # even when light groups are present so we don't need to # exclude it when light groups are active - product = RenderProduct(productName=aov_name, - aov=aov_name, - ext=ext, - multipart=aov_multipart) - products.append(product) + for camera in cameras: + product = RenderProduct(productName=aov_name, + aov=aov_name, + ext=ext, + multipart=aov_multipart, + camera=camera) + products.append(product) # When a Beauty AOV is added manually, it will be rendered as # 'Beauty_other' in file name and "standard" beauty will have @@ -959,10 +1023,12 @@ class RenderProductsRedshift(ARenderProducts): return products beauty_name = "Beauty_other" if has_beauty_aov else "" - products.insert(0, - RenderProduct(productName=beauty_name, - ext=ext, - multipart=multipart)) + for camera in cameras: + products.insert(0, + RenderProduct(productName=beauty_name, + ext=ext, + multipart=multipart, + camera=camera)) return products @@ -987,6 +1053,16 @@ class RenderProductsRenderman(ARenderProducts): :func:`ARenderProducts.get_render_products()` """ + cameras = [ + self.sanitize_camera_name(c) + for c in self.get_renderable_cameras() + ] + + if not cameras: + cameras = [ + self.sanitize_camera_name( + self.get_renderable_cameras()[0]) + ] products = [] default_ext = "exr" @@ -1000,9 +1076,11 @@ class RenderProductsRenderman(ARenderProducts): if aov_name == "rmanDefaultDisplay": aov_name = "beauty" - product = RenderProduct(productName=aov_name, - ext=default_ext) - products.append(product) + for camera in cameras: + product = RenderProduct(productName=aov_name, + ext=default_ext, + camera=camera) + products.append(product) return products diff --git a/openpype/hosts/maya/api/plugin.py b/openpype/hosts/maya/api/plugin.py index 448cb814d9..fdad0e0989 100644 --- a/openpype/hosts/maya/api/plugin.py +++ b/openpype/hosts/maya/api/plugin.py @@ -123,7 +123,7 @@ class ReferenceLoader(api.Loader): count = options.get("count") or 1 for c in range(0, count): namespace = namespace or lib.unique_namespace( - asset["name"] + "_", + "{}_{}_".format(asset["name"], context["subset"]["name"]), prefix="_" if asset["name"][0].isdigit() else "", suffix="_", ) diff --git a/openpype/hosts/maya/plugins/create/create_mayaascii.py b/openpype/hosts/maya/plugins/create/create_mayaascii.py index f51e126c00..8bbdf107c6 100644 --- a/openpype/hosts/maya/plugins/create/create_mayaascii.py +++ b/openpype/hosts/maya/plugins/create/create_mayaascii.py @@ -1,11 +1,11 @@ from openpype.hosts.maya.api import plugin -class CreateMayaAscii(plugin.Creator): - """Raw Maya Ascii file export""" +class CreateMayaScene(plugin.Creator): + """Raw Maya Scene file export""" - name = "mayaAscii" - label = "Maya Ascii" - family = "mayaAscii" + name = "mayaScene" + label = "Maya Scene" + family = "mayaScene" icon = "file-archive-o" defaults = ['Main'] diff --git a/openpype/hosts/maya/plugins/load/load_reference.py b/openpype/hosts/maya/plugins/load/load_reference.py index d5952ed267..cfe8149218 100644 --- a/openpype/hosts/maya/plugins/load/load_reference.py +++ b/openpype/hosts/maya/plugins/load/load_reference.py @@ -13,6 +13,7 @@ class ReferenceLoader(openpype.hosts.maya.api.plugin.ReferenceLoader): "pointcache", "animation", "mayaAscii", + "mayaScene", "setdress", "layout", "camera", @@ -40,14 +41,13 @@ class ReferenceLoader(openpype.hosts.maya.api.plugin.ReferenceLoader): family = "model" with maya.maintained_selection(): - - groupName = "{}:{}".format(namespace, name) + groupName = "{}:_GRP".format(namespace) cmds.loadPlugin("AbcImport.mll", quiet=True) nodes = cmds.file(self.fname, namespace=namespace, sharedReferenceFile=False, groupReference=True, - groupName="{}:{}".format(namespace, name), + groupName=groupName, reference=True, returnNewNodes=True) @@ -71,7 +71,7 @@ class ReferenceLoader(openpype.hosts.maya.api.plugin.ReferenceLoader): except: # noqa: E722 pass - if family not in ["layout", "setdress", "mayaAscii"]: + if family not in ["layout", "setdress", "mayaAscii", "mayaScene"]: for root in roots: root.setParent(world=True) diff --git a/openpype/hosts/maya/plugins/publish/collect_mayaascii.py b/openpype/hosts/maya/plugins/publish/collect_maya_scene.py similarity index 82% rename from openpype/hosts/maya/plugins/publish/collect_mayaascii.py rename to openpype/hosts/maya/plugins/publish/collect_maya_scene.py index b02f61b7c6..eb21b17989 100644 --- a/openpype/hosts/maya/plugins/publish/collect_mayaascii.py +++ b/openpype/hosts/maya/plugins/publish/collect_maya_scene.py @@ -3,14 +3,14 @@ from maya import cmds import pyblish.api -class CollectMayaAscii(pyblish.api.InstancePlugin): - """Collect May Ascii Data +class CollectMayaScene(pyblish.api.InstancePlugin): + """Collect Maya Scene Data """ order = pyblish.api.CollectorOrder + 0.2 label = 'Collect Model Data' - families = ["mayaAscii"] + families = ["mayaScene"] def process(self, instance): # Extract only current frame (override) diff --git a/openpype/hosts/maya/plugins/publish/collect_render.py b/openpype/hosts/maya/plugins/publish/collect_render.py index 5049647ff9..575cc2456b 100644 --- a/openpype/hosts/maya/plugins/publish/collect_render.py +++ b/openpype/hosts/maya/plugins/publish/collect_render.py @@ -174,10 +174,16 @@ class CollectMayaRender(pyblish.api.ContextPlugin): assert render_products, "no render products generated" exp_files = [] for product in render_products: - for camera in layer_render_products.layer_data.cameras: - exp_files.append( - {product.productName: layer_render_products.get_files( - product, camera)}) + product_name = product.productName + if product.camera and layer_render_products.has_camera_token(): + product_name = "{}{}".format( + product.camera, + "_" + product_name if product_name else "") + exp_files.append( + { + product_name: layer_render_products.get_files( + product) + }) self.log.info("multipart: {}".format( layer_render_products.multipart)) @@ -199,12 +205,14 @@ class CollectMayaRender(pyblish.api.ContextPlugin): # replace relative paths with absolute. Render products are # returned as list of dictionaries. + publish_meta_path = None for aov in exp_files: full_paths = [] for file in aov[aov.keys()[0]]: full_path = os.path.join(workspace, "renders", file) full_path = full_path.replace("\\", "/") full_paths.append(full_path) + publish_meta_path = os.path.dirname(full_path) aov_dict[aov.keys()[0]] = full_paths frame_start_render = int(self.get_render_attribute( @@ -230,6 +238,26 @@ class CollectMayaRender(pyblish.api.ContextPlugin): frame_end_handle = frame_end_render full_exp_files.append(aov_dict) + + # find common path to store metadata + # so if image prefix is branching to many directories + # metadata file will be located in top-most common + # directory. + # TODO: use `os.path.commonpath()` after switch to Python 3 + common_publish_meta_path = os.path.splitdrive( + publish_meta_path)[0] + if common_publish_meta_path: + common_publish_meta_path += os.path.sep + for part in publish_meta_path.split("/"): + common_publish_meta_path = os.path.join( + common_publish_meta_path, part) + if part == expected_layer_name: + break + common_publish_meta_path = common_publish_meta_path.replace( + "\\", "/") + self.log.info( + "Publish meta path: {}".format(common_publish_meta_path)) + self.log.info(full_exp_files) self.log.info("collecting layer: {}".format(layer_name)) # Get layer specific settings, might be overrides @@ -262,6 +290,7 @@ class CollectMayaRender(pyblish.api.ContextPlugin): # which was submitted originally "source": filepath, "expectedFiles": full_exp_files, + "publishRenderMetadataFolder": common_publish_meta_path, "resolutionWidth": cmds.getAttr("defaultResolution.width"), "resolutionHeight": cmds.getAttr("defaultResolution.height"), "pixelAspect": cmds.getAttr("defaultResolution.pixelAspect"), diff --git a/openpype/hosts/maya/plugins/publish/collect_scene.py b/openpype/hosts/maya/plugins/publish/collect_workfile.py similarity index 97% rename from openpype/hosts/maya/plugins/publish/collect_scene.py rename to openpype/hosts/maya/plugins/publish/collect_workfile.py index be2a294f26..ee676f50d0 100644 --- a/openpype/hosts/maya/plugins/publish/collect_scene.py +++ b/openpype/hosts/maya/plugins/publish/collect_workfile.py @@ -4,7 +4,7 @@ import os from maya import cmds -class CollectMayaScene(pyblish.api.ContextPlugin): +class CollectWorkfile(pyblish.api.ContextPlugin): """Inject the current working file into context""" order = pyblish.api.CollectorOrder - 0.01 diff --git a/openpype/hosts/maya/plugins/publish/extract_look.py b/openpype/hosts/maya/plugins/publish/extract_look.py index bbf25ebdc7..e0b85907e9 100644 --- a/openpype/hosts/maya/plugins/publish/extract_look.py +++ b/openpype/hosts/maya/plugins/publish/extract_look.py @@ -205,6 +205,9 @@ class ExtractLook(openpype.api.Extractor): lookdata = instance.data["lookData"] relationships = lookdata["relationships"] sets = relationships.keys() + if not sets: + self.log.info("No sets found") + return results = self.process_resources(instance, staging_dir=dir_path) transfers = results["fileTransfers"] diff --git a/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py b/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py index 3c2b70900d..e7fb5bc8cb 100644 --- a/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py +++ b/openpype/hosts/maya/plugins/publish/extract_maya_scene_raw.py @@ -17,6 +17,7 @@ class ExtractMayaSceneRaw(openpype.api.Extractor): label = "Maya Scene (Raw)" hosts = ["maya"] families = ["mayaAscii", + "mayaScene", "setdress", "layout", "camerarig", diff --git a/openpype/hosts/maya/plugins/publish/validate_rendersettings.py b/openpype/hosts/maya/plugins/publish/validate_rendersettings.py index 7c795db43d..65ddacfc57 100644 --- a/openpype/hosts/maya/plugins/publish/validate_rendersettings.py +++ b/openpype/hosts/maya/plugins/publish/validate_rendersettings.py @@ -76,7 +76,7 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): r'%a||', re.IGNORECASE) R_LAYER_TOKEN = re.compile( r'%l||', re.IGNORECASE) - R_CAMERA_TOKEN = re.compile(r'%c|', re.IGNORECASE) + R_CAMERA_TOKEN = re.compile(r'%c|Camera>') R_SCENE_TOKEN = re.compile(r'%s|', re.IGNORECASE) DEFAULT_PADDING = 4 @@ -126,7 +126,9 @@ class ValidateRenderSettings(pyblish.api.InstancePlugin): if len(cameras) > 1 and not re.search(cls.R_CAMERA_TOKEN, prefix): invalid = True cls.log.error("Wrong image prefix [ {} ] - " - "doesn't have: '' token".format(prefix)) + "doesn't have: '' token".format(prefix)) + cls.log.error( + "Note that to needs to have capital 'C' at the beginning") # renderer specific checks if renderer == "vray": diff --git a/openpype/hosts/nuke/api/lib.py b/openpype/hosts/nuke/api/lib.py index ab4c992719..6fe9af744b 100644 --- a/openpype/hosts/nuke/api/lib.py +++ b/openpype/hosts/nuke/api/lib.py @@ -288,7 +288,8 @@ def script_name(): def add_button_write_to_read(node): name = "createReadNode" label = "Create Read From Rendered" - value = "import write_to_read;write_to_read.write_to_read(nuke.thisNode())" + value = "import write_to_read;\ + write_to_read.write_to_read(nuke.thisNode(), allow_relative=False)" knob = nuke.PyScript_Knob(name, label, value) knob.clearFlag(nuke.STARTLINE) node.addKnob(knob) diff --git a/openpype/hosts/nuke/api/plugin.py b/openpype/hosts/nuke/api/plugin.py index 0ad98146b1..62eadecaf4 100644 --- a/openpype/hosts/nuke/api/plugin.py +++ b/openpype/hosts/nuke/api/plugin.py @@ -1,4 +1,10 @@ +import random +import string + import avalon.nuke +from avalon.nuke import lib as anlib +from avalon import api + from openpype.api import ( get_current_project_settings, PypeCreatorMixin @@ -23,3 +29,68 @@ class PypeCreator(PypeCreatorMixin, avalon.nuke.pipeline.Creator): self.log.error(msg + '\n\nPlease use other subset name!') raise NameError("`{0}: {1}".format(__name__, msg)) return + + +def get_review_presets_config(): + settings = get_current_project_settings() + review_profiles = ( + settings["global"] + ["publish"] + ["ExtractReview"] + ["profiles"] + ) + + outputs = {} + for profile in review_profiles: + outputs.update(profile.get("outputs", {})) + + return [str(name) for name, _prop in outputs.items()] + + +class NukeLoader(api.Loader): + container_id_knob = "containerId" + container_id = ''.join(random.choice( + string.ascii_uppercase + string.digits) for _ in range(10)) + + def get_container_id(self, node): + id_knob = node.knobs().get(self.container_id_knob) + return id_knob.value() if id_knob else None + + def get_members(self, source): + """Return nodes that has same 'containerId' as `source`""" + source_id = self.get_container_id(source) + return [node for node in nuke.allNodes(recurseGroups=True) + if self.get_container_id(node) == source_id + and node is not source] if source_id else [] + + def set_as_member(self, node): + source_id = self.get_container_id(node) + + if source_id: + node[self.container_id_knob].setValue(self.container_id) + else: + HIDEN_FLAG = 0x00040000 + _knob = anlib.Knobby( + "String_Knob", + self.container_id, + flags=[nuke.READ_ONLY, HIDEN_FLAG]) + knob = _knob.create(self.container_id_knob) + node.addKnob(knob) + + def clear_members(self, parent_node): + members = self.get_members(parent_node) + + dependent_nodes = None + for node in members: + _depndc = [n for n in node.dependent() if n not in members] + if not _depndc: + continue + + dependent_nodes = _depndc + break + + for member in members: + self.log.info("removing node: `{}".format(member.name())) + nuke.delete(member) + + return dependent_nodes diff --git a/openpype/hosts/nuke/plugins/inventory/repair_old_loaders.py b/openpype/hosts/nuke/plugins/inventory/repair_old_loaders.py new file mode 100644 index 0000000000..e7ae51fa86 --- /dev/null +++ b/openpype/hosts/nuke/plugins/inventory/repair_old_loaders.py @@ -0,0 +1,37 @@ +from avalon import api, style +from avalon.nuke import lib as anlib +from openpype.api import ( + Logger) + + +class RepairOldLoaders(api.InventoryAction): + + label = "Repair Old Loaders" + icon = "gears" + color = style.colors.alert + + log = Logger().get_logger(__name__) + + def process(self, containers): + import nuke + new_loader = "LoadClip" + + for cdata in containers: + orig_loader = cdata["loader"] + orig_name = cdata["objectName"] + if orig_loader not in ["LoadSequence", "LoadMov"]: + self.log.warning( + "This repair action is only working on " + "`LoadSequence` and `LoadMov` Loaders") + continue + + new_name = orig_name.replace(orig_loader, new_loader) + node = nuke.toNode(cdata["objectName"]) + + cdata.update({ + "loader": new_loader, + "objectName": new_name + }) + node["name"].setValue(new_name) + # get data from avalon knob + anlib.set_avalon_knob_data(node, cdata) diff --git a/openpype/hosts/nuke/plugins/inventory/select_containers.py b/openpype/hosts/nuke/plugins/inventory/select_containers.py index b420f53431..bd00983172 100644 --- a/openpype/hosts/nuke/plugins/inventory/select_containers.py +++ b/openpype/hosts/nuke/plugins/inventory/select_containers.py @@ -8,10 +8,10 @@ class SelectContainers(api.InventoryAction): color = "#d8d8d8" def process(self, containers): - + import nuke import avalon.nuke - nodes = [i["_node"] for i in containers] + nodes = [nuke.toNode(i["objectName"]) for i in containers] with avalon.nuke.viewer_update_and_undo_stop(): # clear previous_selection diff --git a/openpype/hosts/nuke/plugins/inventory/set_tool_color.py b/openpype/hosts/nuke/plugins/inventory/set_tool_color.py deleted file mode 100644 index 7a81444c90..0000000000 --- a/openpype/hosts/nuke/plugins/inventory/set_tool_color.py +++ /dev/null @@ -1,68 +0,0 @@ -# from avalon import api, style -# from avalon.vendor.Qt import QtGui, QtWidgets -# -# import avalon.fusion -# -# -# class FusionSetToolColor(api.InventoryAction): -# """Update the color of the selected tools""" -# -# label = "Set Tool Color" -# icon = "plus" -# color = "#d8d8d8" -# _fallback_color = QtGui.QColor(1.0, 1.0, 1.0) -# -# def process(self, containers): -# """Color all selected tools the selected colors""" -# -# result = [] -# comp = avalon.fusion.get_current_comp() -# -# # Get tool color -# first = containers[0] -# tool = first["_node"] -# color = tool.TileColor -# -# if color is not None: -# qcolor = QtGui.QColor().fromRgbF(color["R"], color["G"], color["B"]) -# else: -# qcolor = self._fallback_color -# -# # Launch pick color -# picked_color = self.get_color_picker(qcolor) -# if not picked_color: -# return -# -# with avalon.fusion.comp_lock_and_undo_chunk(comp): -# for container in containers: -# # Convert color to RGB 0-1 floats -# rgb_f = picked_color.getRgbF() -# rgb_f_table = {"R": rgb_f[0], "G": rgb_f[1], "B": rgb_f[2]} -# -# # Update tool -# tool = container["_node"] -# tool.TileColor = rgb_f_table -# -# result.append(container) -# -# return result -# -# def get_color_picker(self, color): -# """Launch color picker and return chosen color -# -# Args: -# color(QtGui.QColor): Start color to display -# -# Returns: -# QtGui.QColor -# -# """ -# -# color_dialog = QtWidgets.QColorDialog(color) -# color_dialog.setStyleSheet(style.load_stylesheet()) -# -# accepted = color_dialog.exec_() -# if not accepted: -# return -# -# return color_dialog.selectedColor() diff --git a/openpype/hosts/nuke/plugins/load/load_clip.py b/openpype/hosts/nuke/plugins/load/load_clip.py new file mode 100644 index 0000000000..f8fc5e3928 --- /dev/null +++ b/openpype/hosts/nuke/plugins/load/load_clip.py @@ -0,0 +1,371 @@ +import nuke +from avalon.vendor import qargparse +from avalon import api, io + +from openpype.hosts.nuke.api.lib import ( + get_imageio_input_colorspace +) +from avalon.nuke import ( + containerise, + update_container, + viewer_update_and_undo_stop, + maintained_selection +) +from openpype.hosts.nuke.api import plugin + + +class LoadClip(plugin.NukeLoader): + """Load clip into Nuke + + Either it is image sequence or video file. + """ + + families = [ + "source", + "plate", + "render", + "prerender", + "review" + ] + representations = [ + "exr", + "dpx", + "mov", + "review", + "mp4" + ] + + label = "Load Clip" + order = -20 + icon = "file-video-o" + color = "white" + + script_start = int(nuke.root()["first_frame"].value()) + + # option gui + defaults = { + "start_at_workfile": True + } + + options = [ + qargparse.Boolean( + "start_at_workfile", + help="Load at workfile start frame", + default=True + ) + ] + + node_name_template = "{class_name}_{ext}" + + @classmethod + def get_representations(cls): + return ( + cls.representations + + cls._representations + + plugin.get_review_presets_config() + ) + + def load(self, context, name, namespace, options): + + is_sequence = len(context["representation"]["files"]) > 1 + + file = self.fname.replace("\\", "/") + + start_at_workfile = options.get( + "start_at_workfile", self.defaults["start_at_workfile"]) + + version = context['version'] + version_data = version.get("data", {}) + repr_id = context["representation"]["_id"] + colorspace = version_data.get("colorspace") + iio_colorspace = get_imageio_input_colorspace(file) + repr_cont = context["representation"]["context"] + + self.log.info("version_data: {}\n".format(version_data)) + self.log.debug( + "Representation id `{}` ".format(repr_id)) + + self.handle_start = version_data.get("handleStart", 0) + self.handle_end = version_data.get("handleEnd", 0) + + first = version_data.get("frameStart", None) + last = version_data.get("frameEnd", None) + first -= self.handle_start + last += self.handle_end + + if not is_sequence: + duration = last - first + 1 + first = 1 + last = first + duration + elif "#" not in file: + frame = repr_cont.get("frame") + assert frame, "Representation is not sequence" + + padding = len(frame) + file = file.replace(frame, "#" * padding) + + # Fallback to asset name when namespace is None + if namespace is None: + namespace = context['asset']['name'] + + if not file: + self.log.warning( + "Representation id `{}` is failing to load".format(repr_id)) + return + + name_data = { + "asset": repr_cont["asset"], + "subset": repr_cont["subset"], + "representation": context["representation"]["name"], + "ext": repr_cont["representation"], + "id": context["representation"]["_id"], + "class_name": self.__class__.__name__ + } + + read_name = self.node_name_template.format(**name_data) + + # Create the Loader with the filename path set + read_node = nuke.createNode( + "Read", + "name {}".format(read_name)) + + # to avoid multiple undo steps for rest of process + # we will switch off undo-ing + with viewer_update_and_undo_stop(): + read_node["file"].setValue(file) + + # Set colorspace defined in version data + if colorspace: + read_node["colorspace"].setValue(str(colorspace)) + elif iio_colorspace is not None: + read_node["colorspace"].setValue(iio_colorspace) + + self.set_range_to_node(read_node, first, last, start_at_workfile) + + # add additional metadata from the version to imprint Avalon knob + add_keys = ["frameStart", "frameEnd", + "source", "colorspace", "author", "fps", "version", + "handleStart", "handleEnd"] + + data_imprint = {} + for k in add_keys: + if k == 'version': + data_imprint.update({k: context["version"]['name']}) + else: + data_imprint.update( + {k: context["version"]['data'].get(k, str(None))}) + + data_imprint.update({"objectName": read_name}) + + read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) + + container = containerise( + read_node, + name=name, + namespace=namespace, + context=context, + loader=self.__class__.__name__, + data=data_imprint) + + if version_data.get("retime", None): + self.make_retimes(read_node, version_data) + + self.set_as_member(read_node) + + return container + + def switch(self, container, representation): + self.update(container, representation) + + def update(self, container, representation): + """Update the Loader's path + + Nuke automatically tries to reset some variables when changing + the loader's path to a new file. These automatic changes are to its + inputs: + + """ + + is_sequence = len(representation["files"]) > 1 + + read_node = nuke.toNode(container['objectName']) + file = api.get_representation_path(representation).replace("\\", "/") + + start_at_workfile = bool("start at" in read_node['frame_mode'].value()) + + version = io.find_one({ + "type": "version", + "_id": representation["parent"] + }) + version_data = version.get("data", {}) + repr_id = representation["_id"] + colorspace = version_data.get("colorspace") + iio_colorspace = get_imageio_input_colorspace(file) + repr_cont = representation["context"] + + self.handle_start = version_data.get("handleStart", 0) + self.handle_end = version_data.get("handleEnd", 0) + + first = version_data.get("frameStart", None) + last = version_data.get("frameEnd", None) + first -= self.handle_start + last += self.handle_end + + if not is_sequence: + duration = last - first + 1 + first = 1 + last = first + duration + elif "#" not in file: + frame = repr_cont.get("frame") + assert frame, "Representation is not sequence" + + padding = len(frame) + file = file.replace(frame, "#" * padding) + + if not file: + self.log.warning( + "Representation id `{}` is failing to load".format(repr_id)) + return + + read_node["file"].setValue(file) + + # to avoid multiple undo steps for rest of process + # we will switch off undo-ing + with viewer_update_and_undo_stop(): + + # Set colorspace defined in version data + if colorspace: + read_node["colorspace"].setValue(str(colorspace)) + elif iio_colorspace is not None: + read_node["colorspace"].setValue(iio_colorspace) + + self.set_range_to_node(read_node, first, last, start_at_workfile) + + updated_dict = { + "representation": str(representation["_id"]), + "frameStart": str(first), + "frameEnd": str(last), + "version": str(version.get("name")), + "colorspace": colorspace, + "source": version_data.get("source"), + "handleStart": str(self.handle_start), + "handleEnd": str(self.handle_end), + "fps": str(version_data.get("fps")), + "author": version_data.get("author"), + "outputDir": version_data.get("outputDir"), + } + + # change color of read_node + # get all versions in list + versions = io.find({ + "type": "version", + "parent": version["parent"] + }).distinct('name') + + max_version = max(versions) + + if version.get("name") not in [max_version]: + read_node["tile_color"].setValue(int("0xd84f20ff", 16)) + else: + read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) + + # Update the imprinted representation + update_container( + read_node, + updated_dict + ) + self.log.info("udated to version: {}".format(version.get("name"))) + + if version_data.get("retime", None): + self.make_retimes(read_node, version_data) + else: + self.clear_members(read_node) + + self.set_as_member(read_node) + + def set_range_to_node(self, read_node, first, last, start_at_workfile): + read_node['origfirst'].setValue(int(first)) + read_node['first'].setValue(int(first)) + read_node['origlast'].setValue(int(last)) + read_node['last'].setValue(int(last)) + + # set start frame depending on workfile or version + self.loader_shift(read_node, start_at_workfile) + + def remove(self, container): + + from avalon.nuke import viewer_update_and_undo_stop + + read_node = nuke.toNode(container['objectName']) + assert read_node.Class() == "Read", "Must be Read" + + with viewer_update_and_undo_stop(): + members = self.get_members(read_node) + nuke.delete(read_node) + for member in members: + nuke.delete(member) + + def make_retimes(self, parent_node, version_data): + ''' Create all retime and timewarping nodes with coppied animation ''' + speed = version_data.get('speed', 1) + time_warp_nodes = version_data.get('timewarps', []) + last_node = None + source_id = self.get_container_id(parent_node) + self.log.info("__ source_id: {}".format(source_id)) + self.log.info("__ members: {}".format(self.get_members(parent_node))) + dependent_nodes = self.clear_members(parent_node) + + with maintained_selection(): + parent_node['selected'].setValue(True) + + if speed != 1: + rtn = nuke.createNode( + "Retime", + "speed {}".format(speed)) + + rtn["before"].setValue("continue") + rtn["after"].setValue("continue") + rtn["input.first_lock"].setValue(True) + rtn["input.first"].setValue( + self.script_start + ) + self.set_as_member(rtn) + last_node = rtn + + if time_warp_nodes != []: + start_anim = self.script_start + (self.handle_start / speed) + for timewarp in time_warp_nodes: + twn = nuke.createNode( + timewarp["Class"], + "name {}".format(timewarp["name"]) + ) + if isinstance(timewarp["lookup"], list): + # if array for animation + twn["lookup"].setAnimated() + for i, value in enumerate(timewarp["lookup"]): + twn["lookup"].setValueAt( + (start_anim + i) + value, + (start_anim + i)) + else: + # if static value `int` + twn["lookup"].setValue(timewarp["lookup"]) + + self.set_as_member(twn) + last_node = twn + + if dependent_nodes: + # connect to original inputs + for i, n in enumerate(dependent_nodes): + last_node.setInput(i, n) + + def loader_shift(self, read_node, workfile_start=False): + """ Set start frame of read node to a workfile start + + Args: + read_node (nuke.Node): The nuke's read node + workfile_start (bool): set workfile start frame if true + + """ + if workfile_start: + read_node['frame_mode'].setValue("start at") + read_node['frame'].setValue(str(self.script_start)) diff --git a/openpype/hosts/nuke/plugins/load/load_image.py b/openpype/hosts/nuke/plugins/load/load_image.py index 9b8bc43d12..2af44d6eba 100644 --- a/openpype/hosts/nuke/plugins/load/load_image.py +++ b/openpype/hosts/nuke/plugins/load/load_image.py @@ -12,7 +12,15 @@ from openpype.hosts.nuke.api.lib import ( class LoadImage(api.Loader): """Load still image into Nuke""" - families = ["render", "source", "plate", "review", "image"] + families = [ + "render2d", + "source", + "plate", + "render", + "prerender", + "review", + "image" + ] representations = ["exr", "dpx", "jpg", "jpeg", "png", "psd", "tiff"] label = "Load Image" @@ -33,6 +41,10 @@ class LoadImage(api.Loader): ) ] + @classmethod + def get_representations(cls): + return cls.representations + cls._representations + def load(self, context, name, namespace, options): from avalon.nuke import ( containerise, diff --git a/openpype/hosts/nuke/plugins/load/load_mov.py b/openpype/hosts/nuke/plugins/load/load_mov.py deleted file mode 100644 index f7523d0a6e..0000000000 --- a/openpype/hosts/nuke/plugins/load/load_mov.py +++ /dev/null @@ -1,347 +0,0 @@ -import nuke -from avalon.vendor import qargparse -from avalon import api, io -from openpype.api import get_current_project_settings -from openpype.hosts.nuke.api.lib import ( - get_imageio_input_colorspace -) - - -def add_review_presets_config(): - returning = { - "families": list(), - "representations": list() - } - settings = get_current_project_settings() - review_profiles = ( - settings["global"] - ["publish"] - ["ExtractReview"] - ["profiles"] - ) - - outputs = {} - for profile in review_profiles: - outputs.update(profile.get("outputs", {})) - - for output, properities in outputs.items(): - returning["representations"].append(output) - returning["families"] += properities.get("families", []) - - return returning - - -class LoadMov(api.Loader): - """Load mov file into Nuke""" - families = ["render", "source", "plate", "review"] - representations = ["mov", "review", "mp4"] - - label = "Load mov" - order = -10 - icon = "code-fork" - color = "orange" - - first_frame = nuke.root()["first_frame"].value() - - # options gui - defaults = { - "start_at_workfile": True - } - - options = [ - qargparse.Boolean( - "start_at_workfile", - help="Load at workfile start frame", - default=True - ) - ] - - node_name_template = "{class_name}_{ext}" - - def load(self, context, name, namespace, options): - from avalon.nuke import ( - containerise, - viewer_update_and_undo_stop - ) - - start_at_workfile = options.get( - "start_at_workfile", self.defaults["start_at_workfile"]) - - version = context['version'] - version_data = version.get("data", {}) - repr_id = context["representation"]["_id"] - - self.handle_start = version_data.get("handleStart", 0) - self.handle_end = version_data.get("handleEnd", 0) - - orig_first = version_data.get("frameStart") - orig_last = version_data.get("frameEnd") - diff = orig_first - 1 - - first = orig_first - diff - last = orig_last - diff - - colorspace = version_data.get("colorspace") - repr_cont = context["representation"]["context"] - - self.log.debug( - "Representation id `{}` ".format(repr_id)) - - context["representation"]["_id"] - # create handles offset (only to last, because of mov) - last += self.handle_start + self.handle_end - - # Fallback to asset name when namespace is None - if namespace is None: - namespace = context['asset']['name'] - - file = self.fname - - if not file: - self.log.warning( - "Representation id `{}` is failing to load".format(repr_id)) - return - - file = file.replace("\\", "/") - - name_data = { - "asset": repr_cont["asset"], - "subset": repr_cont["subset"], - "representation": context["representation"]["name"], - "ext": repr_cont["representation"], - "id": context["representation"]["_id"], - "class_name": self.__class__.__name__ - } - - read_name = self.node_name_template.format(**name_data) - - read_node = nuke.createNode( - "Read", - "name {}".format(read_name) - ) - - # to avoid multiple undo steps for rest of process - # we will switch off undo-ing - with viewer_update_and_undo_stop(): - read_node["file"].setValue(file) - - read_node["origfirst"].setValue(first) - read_node["first"].setValue(first) - read_node["origlast"].setValue(last) - read_node["last"].setValue(last) - read_node['frame_mode'].setValue("start at") - - if start_at_workfile: - # start at workfile start - read_node['frame'].setValue(str(self.first_frame)) - else: - # start at version frame start - read_node['frame'].setValue( - str(orig_first - self.handle_start)) - - if colorspace: - read_node["colorspace"].setValue(str(colorspace)) - - preset_clrsp = get_imageio_input_colorspace(file) - - if preset_clrsp is not None: - read_node["colorspace"].setValue(preset_clrsp) - - # add additional metadata from the version to imprint Avalon knob - add_keys = [ - "frameStart", "frameEnd", "handles", "source", "author", - "fps", "version", "handleStart", "handleEnd" - ] - - data_imprint = {} - for key in add_keys: - if key == 'version': - data_imprint.update({ - key: context["version"]['name'] - }) - else: - data_imprint.update({ - key: context["version"]['data'].get(key, str(None)) - }) - - data_imprint.update({"objectName": read_name}) - - read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) - - if version_data.get("retime", None): - speed = version_data.get("speed", 1) - time_warp_nodes = version_data.get("timewarps", []) - self.make_retimes(speed, time_warp_nodes) - - return containerise( - read_node, - name=name, - namespace=namespace, - context=context, - loader=self.__class__.__name__, - data=data_imprint - ) - - def switch(self, container, representation): - self.update(container, representation) - - def update(self, container, representation): - """Update the Loader's path - - Nuke automatically tries to reset some variables when changing - the loader's path to a new file. These automatic changes are to its - inputs: - - """ - - from avalon.nuke import ( - update_container - ) - - read_node = nuke.toNode(container['objectName']) - - assert read_node.Class() == "Read", "Must be Read" - - file = self.fname - - if not file: - repr_id = representation["_id"] - self.log.warning( - "Representation id `{}` is failing to load".format(repr_id)) - return - - file = file.replace("\\", "/") - - # Get start frame from version data - version = io.find_one({ - "type": "version", - "_id": representation["parent"] - }) - - # get all versions in list - versions = io.find({ - "type": "version", - "parent": version["parent"] - }).distinct('name') - - max_version = max(versions) - - version_data = version.get("data", {}) - - orig_first = version_data.get("frameStart") - orig_last = version_data.get("frameEnd") - diff = orig_first - 1 - - # set first to 1 - first = orig_first - diff - last = orig_last - diff - self.handle_start = version_data.get("handleStart", 0) - self.handle_end = version_data.get("handleEnd", 0) - colorspace = version_data.get("colorspace") - - if first is None: - self.log.warning(( - "Missing start frame for updated version" - "assuming starts at frame 0 for: " - "{} ({})").format( - read_node['name'].value(), representation)) - first = 0 - - # create handles offset (only to last, because of mov) - last += self.handle_start + self.handle_end - - read_node["file"].setValue(file) - - # Set the global in to the start frame of the sequence - read_node["origfirst"].setValue(first) - read_node["first"].setValue(first) - read_node["origlast"].setValue(last) - read_node["last"].setValue(last) - read_node['frame_mode'].setValue("start at") - - if int(float(self.first_frame)) == int( - float(read_node['frame'].value())): - # start at workfile start - read_node['frame'].setValue(str(self.first_frame)) - else: - # start at version frame start - read_node['frame'].setValue(str(orig_first - self.handle_start)) - - if colorspace: - read_node["colorspace"].setValue(str(colorspace)) - - preset_clrsp = get_imageio_input_colorspace(file) - - if preset_clrsp is not None: - read_node["colorspace"].setValue(preset_clrsp) - - updated_dict = {} - updated_dict.update({ - "representation": str(representation["_id"]), - "frameStart": str(first), - "frameEnd": str(last), - "version": str(version.get("name")), - "colorspace": version_data.get("colorspace"), - "source": version_data.get("source"), - "handleStart": str(self.handle_start), - "handleEnd": str(self.handle_end), - "fps": str(version_data.get("fps")), - "author": version_data.get("author"), - "outputDir": version_data.get("outputDir") - }) - - # change color of node - if version.get("name") not in [max_version]: - read_node["tile_color"].setValue(int("0xd84f20ff", 16)) - else: - read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) - - if version_data.get("retime", None): - speed = version_data.get("speed", 1) - time_warp_nodes = version_data.get("timewarps", []) - self.make_retimes(speed, time_warp_nodes) - - # Update the imprinted representation - update_container( - read_node, updated_dict - ) - self.log.info("udated to version: {}".format(version.get("name"))) - - def remove(self, container): - - from avalon.nuke import viewer_update_and_undo_stop - - read_node = nuke.toNode(container['objectName']) - assert read_node.Class() == "Read", "Must be Read" - - with viewer_update_and_undo_stop(): - nuke.delete(read_node) - - def make_retimes(self, speed, time_warp_nodes): - ''' Create all retime and timewarping nodes with coppied animation ''' - if speed != 1: - rtn = nuke.createNode( - "Retime", - "speed {}".format(speed)) - rtn["before"].setValue("continue") - rtn["after"].setValue("continue") - rtn["input.first_lock"].setValue(True) - rtn["input.first"].setValue( - self.first_frame - ) - - if time_warp_nodes != []: - start_anim = self.first_frame + (self.handle_start / speed) - for timewarp in time_warp_nodes: - twn = nuke.createNode(timewarp["Class"], - "name {}".format(timewarp["name"])) - if isinstance(timewarp["lookup"], list): - # if array for animation - twn["lookup"].setAnimated() - for i, value in enumerate(timewarp["lookup"]): - twn["lookup"].setValueAt( - (start_anim + i) + value, - (start_anim + i)) - else: - # if static value `int` - twn["lookup"].setValue(timewarp["lookup"]) diff --git a/openpype/hosts/nuke/plugins/load/load_sequence.py b/openpype/hosts/nuke/plugins/load/load_sequence.py deleted file mode 100644 index 003b406ee7..0000000000 --- a/openpype/hosts/nuke/plugins/load/load_sequence.py +++ /dev/null @@ -1,320 +0,0 @@ -import nuke -from avalon.vendor import qargparse -from avalon import api, io -from openpype.hosts.nuke.api.lib import ( - get_imageio_input_colorspace -) - - -class LoadSequence(api.Loader): - """Load image sequence into Nuke""" - - families = ["render", "source", "plate", "review"] - representations = ["exr", "dpx"] - - label = "Load Image Sequence" - order = -20 - icon = "file-video-o" - color = "white" - - script_start = nuke.root()["first_frame"].value() - - # option gui - defaults = { - "start_at_workfile": True - } - - options = [ - qargparse.Boolean( - "start_at_workfile", - help="Load at workfile start frame", - default=True - ) - ] - - node_name_template = "{class_name}_{ext}" - - def load(self, context, name, namespace, options): - from avalon.nuke import ( - containerise, - viewer_update_and_undo_stop - ) - - start_at_workfile = options.get( - "start_at_workfile", self.defaults["start_at_workfile"]) - - version = context['version'] - version_data = version.get("data", {}) - repr_id = context["representation"]["_id"] - - self.log.info("version_data: {}\n".format(version_data)) - self.log.debug( - "Representation id `{}` ".format(repr_id)) - - self.first_frame = int(nuke.root()["first_frame"].getValue()) - self.handle_start = version_data.get("handleStart", 0) - self.handle_end = version_data.get("handleEnd", 0) - - first = version_data.get("frameStart", None) - last = version_data.get("frameEnd", None) - - # Fallback to asset name when namespace is None - if namespace is None: - namespace = context['asset']['name'] - - first -= self.handle_start - last += self.handle_end - - file = self.fname - - if not file: - repr_id = context["representation"]["_id"] - self.log.warning( - "Representation id `{}` is failing to load".format(repr_id)) - return - - file = file.replace("\\", "/") - - repr_cont = context["representation"]["context"] - assert repr_cont.get("frame"), "Representation is not sequence" - - if "#" not in file: - frame = repr_cont.get("frame") - if frame: - padding = len(frame) - file = file.replace(frame, "#" * padding) - - name_data = { - "asset": repr_cont["asset"], - "subset": repr_cont["subset"], - "representation": context["representation"]["name"], - "ext": repr_cont["representation"], - "id": context["representation"]["_id"], - "class_name": self.__class__.__name__ - } - - read_name = self.node_name_template.format(**name_data) - - # Create the Loader with the filename path set - read_node = nuke.createNode( - "Read", - "name {}".format(read_name)) - - # to avoid multiple undo steps for rest of process - # we will switch off undo-ing - with viewer_update_and_undo_stop(): - read_node["file"].setValue(file) - - # Set colorspace defined in version data - colorspace = context["version"]["data"].get("colorspace") - if colorspace: - read_node["colorspace"].setValue(str(colorspace)) - - preset_clrsp = get_imageio_input_colorspace(file) - - if preset_clrsp is not None: - read_node["colorspace"].setValue(preset_clrsp) - - # set start frame depending on workfile or version - self.loader_shift(read_node, start_at_workfile) - read_node["origfirst"].setValue(int(first)) - read_node["first"].setValue(int(first)) - read_node["origlast"].setValue(int(last)) - read_node["last"].setValue(int(last)) - - # add additional metadata from the version to imprint Avalon knob - add_keys = ["frameStart", "frameEnd", - "source", "colorspace", "author", "fps", "version", - "handleStart", "handleEnd"] - - data_imprint = {} - for k in add_keys: - if k == 'version': - data_imprint.update({k: context["version"]['name']}) - else: - data_imprint.update( - {k: context["version"]['data'].get(k, str(None))}) - - data_imprint.update({"objectName": read_name}) - - read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) - - if version_data.get("retime", None): - speed = version_data.get("speed", 1) - time_warp_nodes = version_data.get("timewarps", []) - self.make_retimes(speed, time_warp_nodes) - - return containerise(read_node, - name=name, - namespace=namespace, - context=context, - loader=self.__class__.__name__, - data=data_imprint) - - def switch(self, container, representation): - self.update(container, representation) - - def update(self, container, representation): - """Update the Loader's path - - Nuke automatically tries to reset some variables when changing - the loader's path to a new file. These automatic changes are to its - inputs: - - """ - - from avalon.nuke import ( - update_container - ) - - read_node = nuke.toNode(container['objectName']) - - assert read_node.Class() == "Read", "Must be Read" - - repr_cont = representation["context"] - assert repr_cont.get("frame"), "Representation is not sequence" - - file = api.get_representation_path(representation) - - if not file: - repr_id = representation["_id"] - self.log.warning( - "Representation id `{}` is failing to load".format(repr_id)) - return - - file = file.replace("\\", "/") - - if "#" not in file: - frame = repr_cont.get("frame") - if frame: - padding = len(frame) - file = file.replace(frame, "#" * padding) - - # Get start frame from version data - version = io.find_one({ - "type": "version", - "_id": representation["parent"] - }) - - # get all versions in list - versions = io.find({ - "type": "version", - "parent": version["parent"] - }).distinct('name') - - max_version = max(versions) - - version_data = version.get("data", {}) - - self.first_frame = int(nuke.root()["first_frame"].getValue()) - self.handle_start = version_data.get("handleStart", 0) - self.handle_end = version_data.get("handleEnd", 0) - - first = version_data.get("frameStart") - last = version_data.get("frameEnd") - - if first is None: - self.log.warning( - "Missing start frame for updated version" - "assuming starts at frame 0 for: " - "{} ({})".format(read_node['name'].value(), representation)) - first = 0 - - first -= self.handle_start - last += self.handle_end - - read_node["file"].setValue(file) - - # set start frame depending on workfile or version - self.loader_shift( - read_node, - bool("start at" in read_node['frame_mode'].value())) - - read_node["origfirst"].setValue(int(first)) - read_node["first"].setValue(int(first)) - read_node["origlast"].setValue(int(last)) - read_node["last"].setValue(int(last)) - - updated_dict = {} - updated_dict.update({ - "representation": str(representation["_id"]), - "frameStart": str(first), - "frameEnd": str(last), - "version": str(version.get("name")), - "colorspace": version_data.get("colorspace"), - "source": version_data.get("source"), - "handleStart": str(self.handle_start), - "handleEnd": str(self.handle_end), - "fps": str(version_data.get("fps")), - "author": version_data.get("author"), - "outputDir": version_data.get("outputDir"), - }) - - # change color of read_node - if version.get("name") not in [max_version]: - read_node["tile_color"].setValue(int("0xd84f20ff", 16)) - else: - read_node["tile_color"].setValue(int("0x4ecd25ff", 16)) - - if version_data.get("retime", None): - speed = version_data.get("speed", 1) - time_warp_nodes = version_data.get("timewarps", []) - self.make_retimes(speed, time_warp_nodes) - - # Update the imprinted representation - update_container( - read_node, - updated_dict - ) - self.log.info("udated to version: {}".format(version.get("name"))) - - def remove(self, container): - - from avalon.nuke import viewer_update_and_undo_stop - - read_node = nuke.toNode(container['objectName']) - assert read_node.Class() == "Read", "Must be Read" - - with viewer_update_and_undo_stop(): - nuke.delete(read_node) - - def make_retimes(self, speed, time_warp_nodes): - ''' Create all retime and timewarping nodes with coppied animation ''' - if speed != 1: - rtn = nuke.createNode( - "Retime", - "speed {}".format(speed)) - rtn["before"].setValue("continue") - rtn["after"].setValue("continue") - rtn["input.first_lock"].setValue(True) - rtn["input.first"].setValue( - self.first_frame - ) - - if time_warp_nodes != []: - start_anim = self.first_frame + (self.handle_start / speed) - for timewarp in time_warp_nodes: - twn = nuke.createNode(timewarp["Class"], - "name {}".format(timewarp["name"])) - if isinstance(timewarp["lookup"], list): - # if array for animation - twn["lookup"].setAnimated() - for i, value in enumerate(timewarp["lookup"]): - twn["lookup"].setValueAt( - (start_anim + i) + value, - (start_anim + i)) - else: - # if static value `int` - twn["lookup"].setValue(timewarp["lookup"]) - - def loader_shift(self, read_node, workfile_start=False): - """ Set start frame of read node to a workfile start - - Args: - read_node (nuke.Node): The nuke's read node - workfile_start (bool): set workfile start frame if true - - """ - if workfile_start: - read_node['frame_mode'].setValue("start at") - read_node['frame'].setValue(str(self.script_start)) diff --git a/openpype/hosts/nuke/startup/write_to_read.py b/openpype/hosts/nuke/startup/write_to_read.py index 295a6e3c85..f5cf66b357 100644 --- a/openpype/hosts/nuke/startup/write_to_read.py +++ b/openpype/hosts/nuke/startup/write_to_read.py @@ -9,7 +9,9 @@ SINGLE_FILE_FORMATS = ['avi', 'mp4', 'mxf', 'mov', 'mpg', 'mpeg', 'wmv', 'm4v', 'm2v'] -def evaluate_filepath_new(k_value, k_eval, project_dir, first_frame): +def evaluate_filepath_new( + k_value, k_eval, project_dir, first_frame, allow_relative): + # get combined relative path combined_relative_path = None if k_eval is not None and project_dir is not None: @@ -26,8 +28,9 @@ def evaluate_filepath_new(k_value, k_eval, project_dir, first_frame): combined_relative_path = None try: - k_value = k_value % first_frame - if os.path.exists(k_value): + # k_value = k_value % first_frame + if os.path.isdir(os.path.basename(k_value)): + # doesn't check for file, only parent dir filepath = k_value elif os.path.exists(k_eval): filepath = k_eval @@ -37,10 +40,12 @@ def evaluate_filepath_new(k_value, k_eval, project_dir, first_frame): filepath = os.path.abspath(filepath) except Exception as E: - log.error("Cannot create Read node. Perhaps it needs to be rendered first :) Error: `{}`".format(E)) + log.error("Cannot create Read node. Perhaps it needs to be \ + rendered first :) Error: `{}`".format(E)) return None filepath = filepath.replace('\\', '/') + # assumes last number is a sequence counter current_frame = re.findall(r'\d+', filepath)[-1] padding = len(current_frame) basename = filepath[: filepath.rfind(current_frame)] @@ -51,11 +56,13 @@ def evaluate_filepath_new(k_value, k_eval, project_dir, first_frame): pass else: # Image sequence needs hashes + # to do still with no number not handled filepath = basename + '#' * padding + '.' + filetype # relative path? make it relative again - if not isinstance(project_dir, type(None)): - filepath = filepath.replace(project_dir, '.') + if allow_relative: + if (not isinstance(project_dir, type(None))) and project_dir != "": + filepath = filepath.replace(project_dir, '.') # get first and last frame from disk frames = [] @@ -95,41 +102,40 @@ def create_read_node(ndata, comp_start): return -def write_to_read(gn): +def write_to_read(gn, + allow_relative=False): + comp_start = nuke.Root().knob('first_frame').value() - comp_end = nuke.Root().knob('last_frame').value() project_dir = nuke.Root().knob('project_directory').getValue() if not os.path.exists(project_dir): project_dir = nuke.Root().knob('project_directory').evaluate() group_read_nodes = [] - with gn: height = gn.screenHeight() # get group height and position new_xpos = int(gn.knob('xpos').value()) new_ypos = int(gn.knob('ypos').value()) + height + 20 group_writes = [n for n in nuke.allNodes() if n.Class() == "Write"] - print("__ group_writes: {}".format(group_writes)) if group_writes != []: # there can be only 1 write node, taking first n = group_writes[0] if n.knob('file') is not None: - file_path_new = evaluate_filepath_new( + myfile, firstFrame, lastFrame = evaluate_filepath_new( n.knob('file').getValue(), n.knob('file').evaluate(), project_dir, - comp_start + comp_start, + allow_relative ) - if not file_path_new: + if not myfile: return - myfiletranslated, firstFrame, lastFrame = file_path_new # get node data ndata = { - 'filepath': myfiletranslated, - 'firstframe': firstFrame, - 'lastframe': lastFrame, + 'filepath': myfile, + 'firstframe': int(firstFrame), + 'lastframe': int(lastFrame), 'new_xpos': new_xpos, 'new_ypos': new_ypos, 'colorspace': n.knob('colorspace').getValue(), @@ -139,7 +145,6 @@ def write_to_read(gn): } group_read_nodes.append(ndata) - # create reads in one go for oneread in group_read_nodes: # create read node diff --git a/openpype/hosts/photoshop/plugins/publish/extract_review.py b/openpype/hosts/photoshop/plugins/publish/extract_review.py index 1c53c3a2ef..8c4d05b282 100644 --- a/openpype/hosts/photoshop/plugins/publish/extract_review.py +++ b/openpype/hosts/photoshop/plugins/publish/extract_review.py @@ -17,6 +17,10 @@ class ExtractReview(openpype.api.Extractor): hosts = ["photoshop"] families = ["review"] + # Extract Options + jpg_options = None + mov_options = None + def process(self, instance): staging_dir = self.staging_dir(instance) self.log.info("Outputting image to {}".format(staging_dir)) @@ -53,7 +57,8 @@ class ExtractReview(openpype.api.Extractor): "name": "jpg", "ext": "jpg", "files": output_image, - "stagingDir": staging_dir + "stagingDir": staging_dir, + "tags": self.jpg_options['tags'] }) instance.data["stagingDir"] = staging_dir @@ -97,7 +102,7 @@ class ExtractReview(openpype.api.Extractor): "frameEnd": 1, "fps": 25, "preview": True, - "tags": ["review", "ftrackreview"] + "tags": self.mov_options['tags'] }) # Required for extract_review plugin (L222 onwards). diff --git a/openpype/hosts/testhost/plugins/publish/collect_context.py b/openpype/hosts/testhost/plugins/publish/collect_context.py index 5aff76a42d..0ab98fb84b 100644 --- a/openpype/hosts/testhost/plugins/publish/collect_context.py +++ b/openpype/hosts/testhost/plugins/publish/collect_context.py @@ -1,5 +1,4 @@ import pyblish.api -from avalon import io from openpype.pipeline import ( OpenPypePyblishPluginMixin, diff --git a/openpype/hosts/testhost/run_publish.py b/openpype/hosts/testhost/run_publish.py index 1bb9c46806..44860a30e4 100644 --- a/openpype/hosts/testhost/run_publish.py +++ b/openpype/hosts/testhost/run_publish.py @@ -1,6 +1,6 @@ import os import sys -openpype_dir = "" + mongo_url = "" project_name = "" asset_name = "" @@ -9,9 +9,6 @@ ftrack_url = "" ftrack_username = "" ftrack_api_key = "" -host_name = "testhost" -current_file = os.path.abspath(__file__) - def multi_dirname(path, times=1): for _ in range(times): @@ -19,8 +16,12 @@ def multi_dirname(path, times=1): return path +host_name = "testhost" +current_file = os.path.abspath(__file__) +openpype_dir = multi_dirname(current_file, 4) + os.environ["OPENPYPE_MONGO"] = mongo_url -os.environ["OPENPYPE_ROOT"] = multi_dirname(current_file, 4) +os.environ["OPENPYPE_ROOT"] = openpype_dir os.environ["AVALON_MONGO"] = mongo_url os.environ["AVALON_PROJECT"] = project_name os.environ["AVALON_ASSET"] = asset_name @@ -42,7 +43,7 @@ for path in [ from Qt import QtWidgets, QtCore -from openpype.tools.new_publisher.window import PublisherWindow +from openpype.tools.publisher.window import PublisherWindow def main(): diff --git a/openpype/hosts/tvpaint/api/__init__.py b/openpype/hosts/tvpaint/api/__init__.py index 57a03d38b7..1c50987d6d 100644 --- a/openpype/hosts/tvpaint/api/__init__.py +++ b/openpype/hosts/tvpaint/api/__init__.py @@ -1,6 +1,8 @@ import os import logging +import requests + import avalon.api import pyblish.api from avalon.tvpaint import pipeline @@ -8,6 +10,7 @@ from avalon.tvpaint.communication_server import register_localization_file from .lib import set_context_settings from openpype.hosts import tvpaint +from openpype.api import get_current_project_settings log = logging.getLogger(__name__) @@ -51,6 +54,19 @@ def initial_launch(): set_context_settings() +def application_exit(): + data = get_current_project_settings() + stop_timer = data["tvpaint"]["stop_timer_on_application_exit"] + + if not stop_timer: + return + + # Stop application timer. + webserver_url = os.environ.get("OPENPYPE_WEBSERVER_URL") + rest_api_url = "{}/timers_manager/stop_timer".format(webserver_url) + requests.post(rest_api_url) + + def install(): log.info("OpenPype - Installing TVPaint integration") localization_file = os.path.join(HOST_DIR, "resources", "avalon.loc") @@ -67,6 +83,7 @@ def install(): pyblish.api.register_callback("instanceToggled", on_instance_toggle) avalon.api.on("application.launched", initial_launch) + avalon.api.on("application.exit", application_exit) def uninstall(): diff --git a/openpype/lib/applications.py b/openpype/lib/applications.py index 6676f6e79f..cc8cb8e7be 100644 --- a/openpype/lib/applications.py +++ b/openpype/lib/applications.py @@ -1353,23 +1353,23 @@ def _prepare_last_workfile(data, workdir, workfile_template_key): ) # Last workfile path - last_workfile_path = "" - extensions = avalon.api.HOST_WORKFILE_EXTENSIONS.get( - app.host_name - ) - if extensions: - anatomy = data["anatomy"] - # Find last workfile - file_template = anatomy.templates[workfile_template_key]["file"] - workdir_data.update({ - "version": 1, - "user": get_openpype_username(), - "ext": extensions[0] - }) + last_workfile_path = data.get("last_workfile_path") or "" + if not last_workfile_path: + extensions = avalon.api.HOST_WORKFILE_EXTENSIONS.get(app.host_name) - last_workfile_path = avalon.api.last_workfile( - workdir, file_template, workdir_data, extensions, True - ) + if extensions: + anatomy = data["anatomy"] + # Find last workfile + file_template = anatomy.templates["work"]["file"] + workdir_data.update({ + "version": 1, + "user": get_openpype_username(), + "ext": extensions[0] + }) + + last_workfile_path = avalon.api.last_workfile( + workdir, file_template, workdir_data, extensions, True + ) if os.path.exists(last_workfile_path): log.debug(( diff --git a/openpype/lib/plugin_tools.py b/openpype/lib/plugin_tools.py index 27031cd1ff..ac60f0291e 100644 --- a/openpype/lib/plugin_tools.py +++ b/openpype/lib/plugin_tools.py @@ -6,6 +6,7 @@ import logging import re import json import tempfile +import distutils from .execute import run_subprocess from .profiles_filtering import filter_profiles @@ -468,7 +469,7 @@ def oiio_supported(): """ Checks if oiiotool is configured for this platform. - Expects full path to executable. + Triggers simple subprocess, handles exception if fails. 'should_decompress' will throw exception if configured, but not present or not working. @@ -476,7 +477,10 @@ def oiio_supported(): (bool) """ oiio_path = get_oiio_tools_path() - if not oiio_path or not os.path.exists(oiio_path): + if oiio_path: + oiio_path = distutils.spawn.find_executable(oiio_path) + + if not oiio_path: log.debug("OIIOTool is not configured or not present at {}". format(oiio_path)) return False diff --git a/openpype/modules/base.py b/openpype/modules/base.py index 748c7857a9..7779fff6ec 100644 --- a/openpype/modules/base.py +++ b/openpype/modules/base.py @@ -48,7 +48,7 @@ class _ModuleClass(object): def __getattr__(self, attr_name): if attr_name not in self.__attributes__: - if attr_name in ("__path__"): + if attr_name in ("__path__", "__file__"): return None raise ImportError("No module named {}.{}".format( self.name, attr_name @@ -104,6 +104,9 @@ class _InterfacesClass(_ModuleClass): """ def __getattr__(self, attr_name): if attr_name not in self.__attributes__: + if attr_name in ("__path__", "__file__"): + return None + # Fake Interface if is not missing self.__attributes__[attr_name] = type( attr_name, diff --git a/openpype/modules/default_modules/deadline/plugins/publish/collect_deadline_server_from_instance.py b/openpype/modules/default_modules/deadline/plugins/publish/collect_deadline_server_from_instance.py index 784616615d..1bc4eaa067 100644 --- a/openpype/modules/default_modules/deadline/plugins/publish/collect_deadline_server_from_instance.py +++ b/openpype/modules/default_modules/deadline/plugins/publish/collect_deadline_server_from_instance.py @@ -11,7 +11,7 @@ import pyblish.api class CollectDeadlineServerFromInstance(pyblish.api.InstancePlugin): """Collect Deadline Webservice URL from instance.""" - order = pyblish.api.CollectorOrder + order = pyblish.api.CollectorOrder + 0.02 label = "Deadline Webservice from the Instance" families = ["rendering"] @@ -46,24 +46,25 @@ class CollectDeadlineServerFromInstance(pyblish.api.InstancePlugin): ["deadline"] ) - try: - default_servers = deadline_settings["deadline_urls"] - project_servers = ( - render_instance.context.data - ["project_settings"] - ["deadline"] - ["deadline_servers"] - ) - deadline_servers = { - k: default_servers[k] - for k in project_servers - if k in default_servers - } - - except AttributeError: - # Handle situation were we had only one url for deadline. - return render_instance.context.data["defaultDeadline"] + default_server = render_instance.context.data["defaultDeadline"] + instance_server = render_instance.data.get("deadlineServers") + if not instance_server: + return default_server + default_servers = deadline_settings["deadline_urls"] + project_servers = ( + render_instance.context.data + ["project_settings"] + ["deadline"] + ["deadline_servers"] + ) + deadline_servers = { + k: default_servers[k] + for k in project_servers + if k in default_servers + } + # This is Maya specific and may not reflect real selection of deadline + # url as dictionary keys in Python 2 are not ordered return deadline_servers[ list(deadline_servers.keys())[ int(render_instance.data.get("deadlineServers")) diff --git a/openpype/modules/default_modules/deadline/plugins/publish/submit_maya_deadline.py b/openpype/modules/default_modules/deadline/plugins/publish/submit_maya_deadline.py index 1ab3dc2554..2d43b0d085 100644 --- a/openpype/modules/default_modules/deadline/plugins/publish/submit_maya_deadline.py +++ b/openpype/modules/default_modules/deadline/plugins/publish/submit_maya_deadline.py @@ -351,6 +351,11 @@ class MayaSubmitDeadline(pyblish.api.InstancePlugin): f.replace(orig_scene, new_scene) ) instance.data["expectedFiles"] = [new_exp] + + if instance.data.get("publishRenderMetadataFolder"): + instance.data["publishRenderMetadataFolder"] = \ + instance.data["publishRenderMetadataFolder"].replace( + orig_scene, new_scene) self.log.info("Scene name was switched {} -> {}".format( orig_scene, new_scene )) diff --git a/openpype/modules/default_modules/deadline/plugins/publish/submit_publish_job.py b/openpype/modules/default_modules/deadline/plugins/publish/submit_publish_job.py index 19e3174384..6b07749819 100644 --- a/openpype/modules/default_modules/deadline/plugins/publish/submit_publish_job.py +++ b/openpype/modules/default_modules/deadline/plugins/publish/submit_publish_job.py @@ -385,6 +385,7 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin): """ task = os.environ["AVALON_TASK"] subset = instance_data["subset"] + cameras = instance_data.get("cameras", []) instances = [] # go through aovs in expected files for aov, files in exp_files[0].items(): @@ -410,7 +411,11 @@ class ProcessSubmittedJobOnFarm(pyblish.api.InstancePlugin): task[0].upper(), task[1:], subset[0].upper(), subset[1:]) - subset_name = '{}_{}'.format(group_name, aov) + cam = [c for c in cameras if c in col.head] + if cam: + subset_name = '{}_{}_{}'.format(group_name, cam, aov) + else: + subset_name = '{}_{}'.format(group_name, aov) if isinstance(col, (list, tuple)): staging = os.path.dirname(col[0]) diff --git a/openpype/modules/default_modules/ftrack/ftrack_module.py b/openpype/modules/default_modules/ftrack/ftrack_module.py index 9cbf979239..73a4dfee82 100644 --- a/openpype/modules/default_modules/ftrack/ftrack_module.py +++ b/openpype/modules/default_modules/ftrack/ftrack_module.py @@ -1,7 +1,6 @@ import os import json import collections -import openpype from openpype.modules import OpenPypeModule from openpype_interfaces import ( @@ -372,7 +371,7 @@ class FtrackModule( return self.tray_module.validate() def tray_exit(self): - return self.tray_module.stop_action_server() + self.tray_module.tray_exit() def set_credentials_to_env(self, username, api_key): os.environ["FTRACK_API_USER"] = username or "" @@ -397,3 +396,16 @@ class FtrackModule( def timer_stopped(self): if self._timers_manager_module is not None: self._timers_manager_module.timer_stopped(self.id) + + def get_task_time(self, project_name, asset_name, task_name): + session = self.create_ftrack_session() + query = ( + 'Task where name is "{}"' + ' and parent.name is "{}"' + ' and project.full_name is "{}"' + ).format(task_name, asset_name, project_name) + task_entity = session.query(query).first() + if not task_entity: + return 0 + hours_logged = (task_entity["time_logged"] / 60) / 60 + return hours_logged diff --git a/openpype/modules/default_modules/ftrack/tray/ftrack_tray.py b/openpype/modules/default_modules/ftrack/tray/ftrack_tray.py index 34e4646767..c6201a94f6 100644 --- a/openpype/modules/default_modules/ftrack/tray/ftrack_tray.py +++ b/openpype/modules/default_modules/ftrack/tray/ftrack_tray.py @@ -289,6 +289,10 @@ class FtrackTrayWrapper: parent_menu.addMenu(tray_menu) + def tray_exit(self): + self.stop_action_server() + self.stop_timer_thread() + # Definition of visibility of each menu actions def set_menu_visibility(self): self.tray_server_menu.menuAction().setVisible(self.bool_logged) diff --git a/openpype/modules/default_modules/timers_manager/rest_api.py b/openpype/modules/default_modules/timers_manager/rest_api.py index ac8d8b7b74..19b72d688b 100644 --- a/openpype/modules/default_modules/timers_manager/rest_api.py +++ b/openpype/modules/default_modules/timers_manager/rest_api.py @@ -1,3 +1,5 @@ +import json + from aiohttp.web_response import Response from openpype.api import Logger @@ -28,6 +30,11 @@ class TimersManagerModuleRestApi: self.prefix + "/stop_timer", self.stop_timer ) + self.server_manager.add_route( + "GET", + self.prefix + "/get_task_time", + self.get_task_time + ) async def start_timer(self, request): data = await request.json() @@ -48,3 +55,20 @@ class TimersManagerModuleRestApi: async def stop_timer(self, request): self.module.stop_timers() return Response(status=200) + + async def get_task_time(self, request): + data = await request.json() + try: + project_name = data['project_name'] + asset_name = data['asset_name'] + task_name = data['task_name'] + except KeyError: + message = ( + "Payload must contain fields 'project_name, 'asset_name'," + " 'task_name'" + ) + log.warning(message) + return Response(text=message, status=404) + + time = self.module.get_task_time(project_name, asset_name, task_name) + return Response(text=json.dumps(time)) diff --git a/openpype/modules/default_modules/timers_manager/timers_manager.py b/openpype/modules/default_modules/timers_manager/timers_manager.py index 6d08ad9dc5..7687d056f8 100644 --- a/openpype/modules/default_modules/timers_manager/timers_manager.py +++ b/openpype/modules/default_modules/timers_manager/timers_manager.py @@ -150,6 +150,7 @@ class TimersManager(OpenPypeModule, ITrayService): def tray_exit(self): if self._idle_manager: self._idle_manager.stop() + self._idle_manager.wait() def start_timer(self, project_name, asset_name, task_name, hierarchy): """ @@ -191,6 +192,16 @@ class TimersManager(OpenPypeModule, ITrayService): } self.timer_started(None, data) + def get_task_time(self, project_name, asset_name, task_name): + times = {} + for module_id, connector in self._connectors_by_module_id.items(): + if hasattr(connector, "get_task_time"): + module = self._modules_by_id[module_id] + times[module.name] = connector.get_task_time( + project_name, asset_name, task_name + ) + return times + def timer_started(self, source_id, data): for module_id, connector in self._connectors_by_module_id.items(): if module_id == source_id: diff --git a/openpype/plugins/publish/collect_resources_path.py b/openpype/plugins/publish/collect_resources_path.py index 98b59332da..fa181301ee 100644 --- a/openpype/plugins/publish/collect_resources_path.py +++ b/openpype/plugins/publish/collect_resources_path.py @@ -26,6 +26,7 @@ class CollectResourcesPath(pyblish.api.InstancePlugin): "animation", "model", "mayaAscii", + "mayaScene", "setdress", "layout", "ass", @@ -67,6 +68,12 @@ class CollectResourcesPath(pyblish.api.InstancePlugin): "representation": "TEMP" }) + # For the first time publish + if instance.data.get("hierarchy"): + template_data.update({ + "hierarchy": instance.data["hierarchy"] + }) + anatomy_filled = anatomy.format(template_data) if "folder" in anatomy.templates["publish"]: diff --git a/openpype/plugins/publish/extract_burnin.py b/openpype/plugins/publish/extract_burnin.py index e386f97551..06eb85c593 100644 --- a/openpype/plugins/publish/extract_burnin.py +++ b/openpype/plugins/publish/extract_burnin.py @@ -1,6 +1,5 @@ import os import re -import subprocess import json import copy import tempfile @@ -46,7 +45,8 @@ class ExtractBurnin(openpype.api.Extractor): "aftereffects", "tvpaint", "webpublisher", - "aftereffects" + "aftereffects", + "photoshop" # "resolve" ] optional = True @@ -158,6 +158,11 @@ class ExtractBurnin(openpype.api.Extractor): filled_anatomy = anatomy.format_all(burnin_data) burnin_data["anatomy"] = filled_anatomy.get_solved() + # Add context data burnin_data. + burnin_data["custom"] = ( + instance.data.get("custom_burnin_data") or {} + ) + # Add source camera name to burnin data camera_name = repre.get("camera_name") if camera_name: diff --git a/openpype/plugins/publish/integrate_new.py b/openpype/plugins/publish/integrate_new.py index 3bff3ff79c..451ea1d80d 100644 --- a/openpype/plugins/publish/integrate_new.py +++ b/openpype/plugins/publish/integrate_new.py @@ -63,6 +63,7 @@ class IntegrateAssetNew(pyblish.api.InstancePlugin): "animation", "model", "mayaAscii", + "mayaScene", "setdress", "layout", "ass", diff --git a/openpype/pype_commands.py b/openpype/pype_commands.py index c18fe36667..5288749e8b 100644 --- a/openpype/pype_commands.py +++ b/openpype/pype_commands.py @@ -257,3 +257,30 @@ class PypeCommands: def validate_jsons(self): pass + def run_tests(self, folder, mark, pyargs): + """ + Runs tests from 'folder' + + Args: + folder (str): relative path to folder with tests + mark (str): label to run tests marked by it (slow etc) + pyargs (str): package path to test + """ + print("run_tests") + import subprocess + + if folder: + folder = " ".join(list(folder)) + else: + folder = "../tests" + + mark_str = pyargs_str = '' + if mark: + mark_str = "-m {}".format(mark) + + if pyargs: + pyargs_str = "--pyargs {}".format(pyargs) + + cmd = "pytest {} {} {}".format(folder, mark_str, pyargs_str) + print("Running {}".format(cmd)) + subprocess.run(cmd) diff --git a/openpype/scripts/otio_burnin.py b/openpype/scripts/otio_burnin.py index 184d7689e3..206abfc0b4 100644 --- a/openpype/scripts/otio_burnin.py +++ b/openpype/scripts/otio_burnin.py @@ -109,9 +109,7 @@ def _prores_codec_args(ffprobe_data, source_ffmpeg_cmd): def _h264_codec_args(ffprobe_data, source_ffmpeg_cmd): - output = [] - - output.extend(["-codec:v", "h264"]) + output = ["-codec:v", "h264"] # Use arguments from source if are available source arguments if source_ffmpeg_cmd: @@ -137,6 +135,32 @@ def _h264_codec_args(ffprobe_data, source_ffmpeg_cmd): return output +def _dnxhd_codec_args(ffprobe_data, source_ffmpeg_cmd): + output = ["-codec:v", "dnxhd"] + + # Use source profile (profiles in metadata are not usable in args directly) + profile = ffprobe_data.get("profile") or "" + # Lower profile and replace space with underscore + cleaned_profile = profile.lower().replace(" ", "_") + dnx_profiles = { + "dnxhd", + "dnxhr_lb", + "dnxhr_sq", + "dnxhr_hq", + "dnxhr_hqx", + "dnxhr_444" + } + if cleaned_profile in dnx_profiles: + output.extend(["-profile:v", cleaned_profile]) + + pix_fmt = ffprobe_data.get("pix_fmt") + if pix_fmt: + output.extend(["-pix_fmt", pix_fmt]) + + output.extend(["-g", "1"]) + return output + + def get_codec_args(ffprobe_data, source_ffmpeg_cmd): codec_name = ffprobe_data.get("codec_name") # Codec "prores" @@ -147,6 +171,10 @@ def get_codec_args(ffprobe_data, source_ffmpeg_cmd): if codec_name == "h264": return _h264_codec_args(ffprobe_data, source_ffmpeg_cmd) + # Coded DNxHD + if codec_name == "dnxhd": + return _dnxhd_codec_args(ffprobe_data, source_ffmpeg_cmd) + output = [] if codec_name: output.extend(["-codec:v", codec_name]) diff --git a/openpype/settings/defaults/project_settings/global.json b/openpype/settings/defaults/project_settings/global.json index d311d9611e..45c1a59d17 100644 --- a/openpype/settings/defaults/project_settings/global.json +++ b/openpype/settings/defaults/project_settings/global.json @@ -24,7 +24,7 @@ "animation", "setdress", "layout", - "mayaAscii" + "mayaScene" ] }, "ExtractJpegEXR": { diff --git a/openpype/settings/defaults/project_settings/maya.json b/openpype/settings/defaults/project_settings/maya.json index bff376b374..f8f3432d0f 100644 --- a/openpype/settings/defaults/project_settings/maya.json +++ b/openpype/settings/defaults/project_settings/maya.json @@ -315,6 +315,21 @@ "optional": true, "active": true }, + "ValidateRigContents": { + "enabled": false, + "optional": true, + "active": true + }, + "ValidateRigJointsHidden": { + "enabled": false, + "optional": true, + "active": true + }, + "ValidateRigControllers": { + "enabled": false, + "optional": true, + "active": true + }, "ValidateCameraAttributes": { "enabled": false, "optional": true, @@ -489,6 +504,12 @@ 255, 255 ], + "mayaScene": [ + 67, + 174, + 255, + 255 + ], "setdress": [ 255, 250, diff --git a/openpype/settings/defaults/project_settings/nuke.json b/openpype/settings/defaults/project_settings/nuke.json index dd65df02e5..e3c7834e4a 100644 --- a/openpype/settings/defaults/project_settings/nuke.json +++ b/openpype/settings/defaults/project_settings/nuke.json @@ -117,16 +117,7 @@ "load": { "LoadImage": { "enabled": true, - "families": [ - "render2d", - "source", - "plate", - "render", - "prerender", - "review", - "image" - ], - "representations": [ + "_representations": [ "exr", "dpx", "jpg", @@ -137,39 +128,9 @@ ], "node_name_template": "{class_name}_{ext}" }, - "LoadMov": { + "LoadClip": { "enabled": true, - "families": [ - "source", - "plate", - "render", - "prerender", - "review" - ], - "representations": [ - "mov", - "review", - "mp4", - "h264" - ], - "node_name_template": "{class_name}_{ext}" - }, - "LoadSequence": { - "enabled": true, - "families": [ - "render2d", - "source", - "plate", - "render", - "prerender", - "review" - ], - "representations": [ - "exr", - "dpx", - "jpg", - "jpeg", - "png" + "_representations": [ ], "node_name_template": "{class_name}_{ext}" } diff --git a/openpype/settings/defaults/project_settings/photoshop.json b/openpype/settings/defaults/project_settings/photoshop.json index 4c36e4bd49..36c30bad6c 100644 --- a/openpype/settings/defaults/project_settings/photoshop.json +++ b/openpype/settings/defaults/project_settings/photoshop.json @@ -17,6 +17,18 @@ "png", "jpg" ] + }, + "ExtractReview": { + "jpg_options": { + "tags": [ + ] + }, + "mov_options": { + "tags": [ + "review", + "ftrackreview" + ] + } } }, "workfile_builder": { diff --git a/openpype/settings/defaults/project_settings/tvpaint.json b/openpype/settings/defaults/project_settings/tvpaint.json index 47f486aa98..528bf6de8e 100644 --- a/openpype/settings/defaults/project_settings/tvpaint.json +++ b/openpype/settings/defaults/project_settings/tvpaint.json @@ -1,4 +1,5 @@ { + "stop_timer_on_application_exit": false, "publish": { "ExtractSequence": { "review_bg": [ diff --git a/openpype/settings/entities/schemas/README.md b/openpype/settings/entities/schemas/README.md index c8432f0f2e..5258fef9ec 100644 --- a/openpype/settings/entities/schemas/README.md +++ b/openpype/settings/entities/schemas/README.md @@ -2,7 +2,7 @@ ## Basic rules - configurations does not define GUI, but GUI defines configurations! -- output is always json (yaml is not needed for anatomy templates anymore) +- output is always json serializable - GUI schema has multiple input types, all inputs are represented by a dictionary - each input may have "input modifiers" (keys in dictionary) that are required or optional - only required modifier for all input items is key `"type"` which says what type of item it is @@ -13,16 +13,16 @@ - `"is_group"` - define that all values under key in hierarchy will be overriden if any value is modified, this information is also stored to overrides - this keys is not allowed for all inputs as they may have not reason for that - key is validated, can be only once in hierarchy but is not required -- currently there are `system configurations` and `project configurations` +- currently there are `system settings` and `project settings` ## Inner schema - GUI schemas are huge json files, to be able to split whole configuration into multiple schema there's type `schema` -- system configuration schemas are stored in `~/tools/settings/settings/gui_schemas/system_schema/` and project configurations in `~/tools/settings/settings/gui_schemas/projects_schema/` +- system configuration schemas are stored in `~/openpype/settings/entities/schemas/system_schema/` and project configurations in `~/openpype/settings/entities/schemas/projects_schema/` - each schema name is filename of json file except extension (without ".json") - if content is dictionary content will be used as `schema` else will be used as `schema_template` ### schema -- can have only key `"children"` which is list of strings, each string should represent another schema (order matters) string represebts name of the schema +- can have only key `"children"` which is list of strings, each string should represent another schema (order matters) string represents name of the schema - will just paste schemas from other schema file in order of "children" list ``` @@ -32,8 +32,9 @@ } ``` -### schema_template +### template - allows to define schema "templates" to not duplicate same content multiple times +- legacy name is `schema_template` (still usable) ```javascript // EXAMPLE json file content (filename: example_template.json) [ @@ -59,11 +60,11 @@ // EXAMPLE usage of the template in schema { "type": "dict", - "key": "schema_template_examples", + "key": "template_examples", "label": "Schema template examples", "children": [ { - "type": "schema_template", + "type": "template", // filename of template (example_template.json) "name": "example_template", "template_data": { @@ -72,7 +73,7 @@ "multipath_executables": false } }, { - "type": "schema_template", + "type": "template", "name": "example_template", "template_data": { "host_label": "Maya 2020", @@ -98,8 +99,16 @@ ... } ``` -- Unfilled fields can be also used for non string values, in that case value must contain only one key and value for fill must contain right type. +- Unfilled fields can be also used for non string values(e.g. dictionary), in that case value must contain only one key and value for fill must contain right type. ```javascript +// Passed data +{ + "executable_multiplatform": { + "type": "schema", + "name": "my_multiplatform_schema" + } +} +// Template content { ... // Allowed @@ -121,32 +130,34 @@ "name": "project_settings/global" } ``` -- all valid `ModuleSettingsDef` classes where calling of `get_settings_schemas` +- all valid `BaseModuleSettingsDef` classes where calling of `get_settings_schemas` will return dictionary where is key "project_settings/global" with schemas will extend and replace this item -- works almost the same way as templates +- dynamic schemas work almost the same way as templates - one item can be replaced by multiple items (or by 0 items) - goal is to dynamically loaded settings of OpenPype addons without having their schemas or default values in main repository + - values of these schemas are saved using the `BaseModuleSettingsDef` methods +- easiest is to use `JsonFilesSettingsDef` which has full implementation of storing default values to json files all you have to implement is method `get_settings_root_path` which should return path to root directory where settings schema can be found and will be saved ## Basic Dictionary inputs - these inputs wraps another inputs into {key: value} relation ## dict -- this is another dictionary input wrapping more inputs but visually makes them different -- item may be used as widget (in `list` or `dict-modifiable`) +- this is dictionary type wrapping more inputs with keys defined in schema +- may be used as dynamic children (e.g. in `list` or `dict-modifiable`) - in that case the only key modifier is `children` which is list of it's keys - USAGE: e.g. List of dictionaries where each dictionary have same structure. -- item may be with or without `"label"` if is not used as widget - - required keys are `"key"` under which will be stored - - without label it is just wrap item holding `"key"` - - can't have `"is_group"` key set to True as it breaks visual override showing - - if `"label"` is entetered there which will be shown in GUI - - item with label can be collapsible - - that can be set with key `"collapsible"` as `True`/`False` (Default: `True`) - - with key `"collapsed"` as `True`/`False` can be set that is collapsed when GUI is opened (Default: `False`) - - it is possible to add darker background with `"highlight_content"` (Default: `False`) - - darker background has limits of maximum applies after 3-4 nested highlighted items there is not difference in the color +- if is not used as dynamic children then must have defined `"key"` under which are it's values stored +- may be with or without `"label"` (only for GUI) + - `"label"` must be set to be able mark item as group with `"is_group"` key set to True +- item with label can visually wrap it's children + - this option is enabled by default to turn off set `"use_label_wrap"` to `False` + - label wrap is by default collapsible + - that can be set with key `"collapsible"` to `True`/`False` + - with key `"collapsed"` as `True`/`False` can be set that is collapsed when GUI is opened (Default: `False`) + - it is possible to add lighter background with `"highlight_content"` (Default: `False`) + - lighter background has limits of maximum applies after 3-4 nested highlighted items there is not much difference in the color - output is dictionary `{the "key": children values}` ``` # Example @@ -198,8 +209,8 @@ ``` ## dict-conditional -- is similar to `dict` but has only one child entity that will be always available -- the one entity is enumerator of possible values and based on value of the entity are defined and used other children entities +- is similar to `dict` but has always available one enum entity + - the enum entity has single selection and it's value define other children entities - each value of enumerator have defined children that will be used - there is no way how to have shared entities across multiple enum items - value from enumerator is also stored next to other values @@ -207,22 +218,27 @@ - `enum_key` must match key regex and any enum item can't have children with same key - `enum_label` is label of the entity for UI purposes - enum items are define with `enum_children` - - it's a list where each item represents enum item + - it's a list where each item represents single item for the enum - all items in `enum_children` must have at least `key` key which represents value stored under `enum_key` - - items can define `label` for UI purposes + - enum items can define `label` for UI purposes - most important part is that item can define `children` key where are definitions of it's children (`children` value works the same way as in `dict`) - to set default value for `enum_key` set it with `enum_default` - entity must have defined `"label"` if is not used as widget -- is set as group if any parent is not group -- if `"label"` is entetered there which will be shown in GUI - - item with label can be collapsible - - that can be set with key `"collapsible"` as `True`/`False` (Default: `True`) - - with key `"collapsed"` as `True`/`False` can be set that is collapsed when GUI is opened (Default: `False`) - - it is possible to add darker background with `"highlight_content"` (Default: `False`) - - darker background has limits of maximum applies after 3-4 nested highlighted items there is not difference in the color - - output is dictionary `{the "key": children values}` +- is set as group if any parent is not group (can't have children as group) +- may be with or without `"label"` (only for GUI) + - `"label"` must be set to be able mark item as group with `"is_group"` key set to True +- item with label can visually wrap it's children + - this option is enabled by default to turn off set `"use_label_wrap"` to `False` + - label wrap is by default collapsible + - that can be set with key `"collapsible"` to `True`/`False` + - with key `"collapsed"` as `True`/`False` can be set that is collapsed when GUI is opened (Default: `False`) + - it is possible to add lighter background with `"highlight_content"` (Default: `False`) + - lighter background has limits of maximum applies after 3-4 nested highlighted items there is not much difference in the color - for UI porposes was added `enum_is_horizontal` which will make combobox appear next to children inputs instead of on top of them (Default: `False`) - this has extended ability of `enum_on_right` which will move combobox to right side next to children widgets (Default: `False`) +- output is dictionary `{the "key": children values}` +- using this type as template item for list type can be used to create infinite hierarchies + ``` # Example { @@ -298,8 +314,8 @@ How output of the schema could look like on save: ``` ## Inputs for setting any kind of value (`Pure` inputs) -- all these input must have defined `"key"` under which will be stored and `"label"` which will be shown next to input - - unless they are used in different types of inputs (later) "as widgets" in that case `"key"` and `"label"` are not required as there is not place where to set them +- all inputs must have defined `"key"` if are not used as dynamic item + - they can also have defined `"label"` ### boolean - simple checkbox, nothing more to set @@ -355,21 +371,15 @@ How output of the schema could look like on save: ``` ### path-input -- enhanced text input - - does not allow to enter backslash, is auto-converted to forward slash - - may be added another validations, like do not allow end path with slash - this input is implemented to add additional features to text input -- this is meant to be used in proxy input `path-widget` +- this is meant to be used in proxy input `path` - DO NOT USE this input in schema please ### raw-json - a little bit enhanced text input for raw json +- can store dictionary (`{}`) or list (`[]`) but not both + - by default stores dictionary to change it to list set `is_list` to `True` - has validations of json format - - empty value is invalid value, always must be json serializable - - valid value types are list `[]` and dictionary `{}` -- schema also defines valid value type - - by default it is dictionary - - to be able use list it is required to define `is_list` to `true` - output can be stored as string - this is to allow any keys in dictionary - set key `store_as_string` to `true` @@ -385,7 +395,7 @@ How output of the schema could look like on save: ``` ### enum -- returns value of single on multiple items from predefined values +- enumeration of values that are predefined in schema - multiselection can be allowed with setting key `"multiselection"` to `True` (Default: `False`) - values are defined under value of key `"enum_items"` as list - each item in list is simple dictionary where value is label and key is value which will be stored @@ -415,6 +425,8 @@ How output of the schema could look like on save: - have only single selection mode - it is possible to define default value `default` - `"work"` is used if default value is not specified +- enum values are not updated on the fly it is required to save templates and + reset settings to recache values ``` { "key": "host", @@ -449,6 +461,42 @@ How output of the schema could look like on save: } ``` +### apps-enum +- enumeration of available application and their variants from system settings + - applications without host name are excluded +- can be used only in project settings +- has only `multiselection` +- used only in project anatomy +``` +{ + "type": "apps-enum", + "key": "applications", + "label": "Applications" +} +``` + +### tools-enum +- enumeration of available tools and their variants from system settings +- can be used only in project settings +- has only `multiselection` +- used only in project anatomy +``` +{ + "type": "tools-enum", + "key": "tools_env", + "label": "Tools" +} +``` + +### task-types-enum +- enumeration of task types from current project +- enum values are not updated on the fly and modifications of task types on project require save and reset to be propagated to this enum +- has set `multiselection` to `True` but can be changed to `False` in schema + +### deadline_url-enum +- deadline module specific enumerator using deadline system settings to fill it's values +- TODO: move this type to deadline module + ## Inputs for setting value using Pure inputs - these inputs also have required `"key"` - attribute `"label"` is required in few conditions @@ -594,7 +642,7 @@ How output of the schema could look like on save: } ``` -### path-widget +### path - input for paths, use `path-input` internally - has 2 input modifiers `"multiplatform"` and `"multipath"` - `"multiplatform"` - adds `"windows"`, `"linux"` and `"darwin"` path inputs result is dictionary @@ -685,12 +733,13 @@ How output of the schema could look like on save: } ``` -### splitter -- visual splitter of items (more divider than splitter) +### separator +- legacy name is `splitter` (still usable) +- visual separator of items (more divider than separator) ``` { - "type": "splitter" + "type": "separator" } ``` diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json b/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json index 3b65f08ac4..6f5577650c 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_photoshop.json @@ -60,7 +60,39 @@ "object_type": "text" } ] - } + }, + { + "type": "dict", + "collapsible": true, + "key": "ExtractReview", + "label": "Extract Review", + "children": [ + { + "type": "dict", + "collapsible": false, + "key": "jpg_options", + "label": "Extracted jpg Options", + "children": [ + { + "type": "schema", + "name": "schema_representation_tags" + } + ] + }, + { + "type": "dict", + "collapsible": false, + "key": "mov_options", + "label": "Extracted mov Options", + "children": [ + { + "type": "schema", + "name": "schema_representation_tags" + } + ] + } + ] + } ] }, { diff --git a/openpype/settings/entities/schemas/projects_schema/schema_project_tvpaint.json b/openpype/settings/entities/schemas/projects_schema/schema_project_tvpaint.json index 368141813f..8286ed1193 100644 --- a/openpype/settings/entities/schemas/projects_schema/schema_project_tvpaint.json +++ b/openpype/settings/entities/schemas/projects_schema/schema_project_tvpaint.json @@ -5,6 +5,11 @@ "label": "TVPaint", "is_file": true, "children": [ + { + "type": "boolean", + "key": "stop_timer_on_application_exit", + "label": "Stop timer on application exit" + }, { "type": "dict", "collapsible": true, diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_load.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_load.json index 0b09d08700..7c87644817 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_load.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_load.json @@ -47,9 +47,14 @@ }, { "type": "color", - "label": "Maya Scene:", + "label": "Maya Ascii:", "key": "mayaAscii" }, + { + "type": "color", + "label": "Maya Scene:", + "key": "mayaScene" + }, { "type": "color", "label": "Set Dress:", diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_publish.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_publish.json index 26ebfb2bd7..cbacd12efa 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_publish.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_maya_publish.json @@ -166,7 +166,6 @@ } ] }, - { "type": "collapsible-wrap", "label": "Model", @@ -329,6 +328,30 @@ } ] }, + { + "type": "collapsible-wrap", + "label": "Rig", + "children": [ + { + "type": "schema_template", + "name": "template_publish_plugin", + "template_data": [ + { + "key": "ValidateRigContents", + "label": "Validate Rig Contents" + }, + { + "key": "ValidateRigJointsHidden", + "label": "Validate Rig Joints Hidden" + }, + { + "key": "ValidateRigControllers", + "label": "Validate Rig Controllers" + } + ] + } + ] + }, { "type": "schema_template", "name": "template_publish_plugin", diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/schema_nuke_load.json b/openpype/settings/entities/schemas/projects_schema/schemas/schema_nuke_load.json index 737843ad98..5bd8337e4c 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/schema_nuke_load.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/schema_nuke_load.json @@ -13,12 +13,8 @@ "label": "Image Loader" }, { - "key": "LoadMov", - "label": "Movie Loader" - }, - { - "key": "LoadSequence", - "label": "Image Sequence Loader" + "key": "LoadClip", + "label": "Clip Loader" } ] } diff --git a/openpype/settings/entities/schemas/projects_schema/schemas/template_loader_plugin_nuke.json b/openpype/settings/entities/schemas/projects_schema/schemas/template_loader_plugin_nuke.json index d01691ed5f..7ee8d0bda0 100644 --- a/openpype/settings/entities/schemas/projects_schema/schemas/template_loader_plugin_nuke.json +++ b/openpype/settings/entities/schemas/projects_schema/schemas/template_loader_plugin_nuke.json @@ -13,13 +13,7 @@ }, { "type": "list", - "key": "families", - "label": "Families", - "object_type": "text" - }, - { - "type": "list", - "key": "representations", + "key": "_representations", "label": "Representations", "object_type": "text" }, diff --git a/openpype/settings/entities/schemas/system_schema/example_schema.json b/openpype/settings/entities/schemas/system_schema/example_schema.json index af6a2d49f4..c30e1f6848 100644 --- a/openpype/settings/entities/schemas/system_schema/example_schema.json +++ b/openpype/settings/entities/schemas/system_schema/example_schema.json @@ -95,11 +95,11 @@ }, { "type": "dict", - "key": "schema_template_exaples", + "key": "template_exaples", "label": "Schema template examples", "children": [ { - "type": "schema_template", + "type": "template", "name": "example_template", "template_data": { "host_label": "Application 1", @@ -108,7 +108,7 @@ } }, { - "type": "schema_template", + "type": "template", "name": "example_template", "template_data": { "host_label": "Application 2", diff --git a/openpype/settings/entities/schemas/system_schema/schema_general.json b/openpype/settings/entities/schemas/system_schema/schema_general.json index 31cd997d14..51a58a6e27 100644 --- a/openpype/settings/entities/schemas/system_schema/schema_general.json +++ b/openpype/settings/entities/schemas/system_schema/schema_general.json @@ -44,6 +44,7 @@ "type": "dict", "key": "disk_mapping", "label": "Disk mapping", + "is_group": true, "use_label_wrap": false, "collapsible": false, "children": [ diff --git a/openpype/tools/new_publisher/__init__.py b/openpype/tools/new_publisher/__init__.py deleted file mode 100644 index 2d81de8fee..0000000000 --- a/openpype/tools/new_publisher/__init__.py +++ /dev/null @@ -1,5 +0,0 @@ -from .app import show - -__all__ = ( - "show", -) diff --git a/openpype/tools/publisher/__init__.py b/openpype/tools/publisher/__init__.py new file mode 100644 index 0000000000..a7b597eece --- /dev/null +++ b/openpype/tools/publisher/__init__.py @@ -0,0 +1,7 @@ +from .app import show +from .window import PublisherWindow + +__all__ = ( + "show", + "PublisherWindow" +) diff --git a/openpype/tools/new_publisher/app.py b/openpype/tools/publisher/app.py similarity index 100% rename from openpype/tools/new_publisher/app.py rename to openpype/tools/publisher/app.py diff --git a/openpype/tools/new_publisher/constants.py b/openpype/tools/publisher/constants.py similarity index 100% rename from openpype/tools/new_publisher/constants.py rename to openpype/tools/publisher/constants.py diff --git a/openpype/tools/new_publisher/control.py b/openpype/tools/publisher/control.py similarity index 100% rename from openpype/tools/new_publisher/control.py rename to openpype/tools/publisher/control.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/__init__.py b/openpype/tools/publisher/publish_report_viewer/__init__.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/__init__.py rename to openpype/tools/publisher/publish_report_viewer/__init__.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/constants.py b/openpype/tools/publisher/publish_report_viewer/constants.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/constants.py rename to openpype/tools/publisher/publish_report_viewer/constants.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/delegates.py b/openpype/tools/publisher/publish_report_viewer/delegates.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/delegates.py rename to openpype/tools/publisher/publish_report_viewer/delegates.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/model.py b/openpype/tools/publisher/publish_report_viewer/model.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/model.py rename to openpype/tools/publisher/publish_report_viewer/model.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/widgets.py b/openpype/tools/publisher/publish_report_viewer/widgets.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/widgets.py rename to openpype/tools/publisher/publish_report_viewer/widgets.py diff --git a/openpype/tools/new_publisher/publish_report_viewer/window.py b/openpype/tools/publisher/publish_report_viewer/window.py similarity index 100% rename from openpype/tools/new_publisher/publish_report_viewer/window.py rename to openpype/tools/publisher/publish_report_viewer/window.py diff --git a/openpype/tools/new_publisher/widgets/__init__.py b/openpype/tools/publisher/widgets/__init__.py similarity index 100% rename from openpype/tools/new_publisher/widgets/__init__.py rename to openpype/tools/publisher/widgets/__init__.py diff --git a/openpype/tools/new_publisher/widgets/border_label_widget.py b/openpype/tools/publisher/widgets/border_label_widget.py similarity index 100% rename from openpype/tools/new_publisher/widgets/border_label_widget.py rename to openpype/tools/publisher/widgets/border_label_widget.py diff --git a/openpype/tools/new_publisher/widgets/card_view_widgets.py b/openpype/tools/publisher/widgets/card_view_widgets.py similarity index 100% rename from openpype/tools/new_publisher/widgets/card_view_widgets.py rename to openpype/tools/publisher/widgets/card_view_widgets.py diff --git a/openpype/tools/new_publisher/widgets/create_dialog.py b/openpype/tools/publisher/widgets/create_dialog.py similarity index 100% rename from openpype/tools/new_publisher/widgets/create_dialog.py rename to openpype/tools/publisher/widgets/create_dialog.py diff --git a/openpype/tools/new_publisher/widgets/icons.py b/openpype/tools/publisher/widgets/icons.py similarity index 100% rename from openpype/tools/new_publisher/widgets/icons.py rename to openpype/tools/publisher/widgets/icons.py diff --git a/openpype/tools/new_publisher/widgets/images/add.png b/openpype/tools/publisher/widgets/images/add.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/add.png rename to openpype/tools/publisher/widgets/images/add.png diff --git a/openpype/tools/new_publisher/widgets/images/branch_closed.png b/openpype/tools/publisher/widgets/images/branch_closed.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/branch_closed.png rename to openpype/tools/publisher/widgets/images/branch_closed.png diff --git a/openpype/tools/new_publisher/widgets/images/branch_open.png b/openpype/tools/publisher/widgets/images/branch_open.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/branch_open.png rename to openpype/tools/publisher/widgets/images/branch_open.png diff --git a/openpype/tools/new_publisher/widgets/images/change_view.png b/openpype/tools/publisher/widgets/images/change_view.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/change_view.png rename to openpype/tools/publisher/widgets/images/change_view.png diff --git a/openpype/tools/new_publisher/widgets/images/delete.png b/openpype/tools/publisher/widgets/images/delete.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/delete.png rename to openpype/tools/publisher/widgets/images/delete.png diff --git a/openpype/tools/new_publisher/widgets/images/play.png b/openpype/tools/publisher/widgets/images/play.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/play.png rename to openpype/tools/publisher/widgets/images/play.png diff --git a/openpype/tools/new_publisher/widgets/images/refresh.png b/openpype/tools/publisher/widgets/images/refresh.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/refresh.png rename to openpype/tools/publisher/widgets/images/refresh.png diff --git a/openpype/tools/new_publisher/widgets/images/stop.png b/openpype/tools/publisher/widgets/images/stop.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/stop.png rename to openpype/tools/publisher/widgets/images/stop.png diff --git a/openpype/tools/new_publisher/widgets/images/thumbnail.png b/openpype/tools/publisher/widgets/images/thumbnail.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/thumbnail.png rename to openpype/tools/publisher/widgets/images/thumbnail.png diff --git a/openpype/tools/new_publisher/widgets/images/validate.png b/openpype/tools/publisher/widgets/images/validate.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/validate.png rename to openpype/tools/publisher/widgets/images/validate.png diff --git a/openpype/tools/new_publisher/widgets/images/warning.png b/openpype/tools/publisher/widgets/images/warning.png similarity index 100% rename from openpype/tools/new_publisher/widgets/images/warning.png rename to openpype/tools/publisher/widgets/images/warning.png diff --git a/openpype/tools/new_publisher/widgets/list_view_widgets.py b/openpype/tools/publisher/widgets/list_view_widgets.py similarity index 100% rename from openpype/tools/new_publisher/widgets/list_view_widgets.py rename to openpype/tools/publisher/widgets/list_view_widgets.py diff --git a/openpype/tools/new_publisher/widgets/models.py b/openpype/tools/publisher/widgets/models.py similarity index 100% rename from openpype/tools/new_publisher/widgets/models.py rename to openpype/tools/publisher/widgets/models.py diff --git a/openpype/tools/new_publisher/widgets/publish_widget.py b/openpype/tools/publisher/widgets/publish_widget.py similarity index 100% rename from openpype/tools/new_publisher/widgets/publish_widget.py rename to openpype/tools/publisher/widgets/publish_widget.py diff --git a/openpype/tools/new_publisher/widgets/validations_widget.py b/openpype/tools/publisher/widgets/validations_widget.py similarity index 100% rename from openpype/tools/new_publisher/widgets/validations_widget.py rename to openpype/tools/publisher/widgets/validations_widget.py diff --git a/openpype/tools/new_publisher/widgets/widgets.py b/openpype/tools/publisher/widgets/widgets.py similarity index 100% rename from openpype/tools/new_publisher/widgets/widgets.py rename to openpype/tools/publisher/widgets/widgets.py diff --git a/openpype/tools/new_publisher/window.py b/openpype/tools/publisher/window.py similarity index 100% rename from openpype/tools/new_publisher/window.py rename to openpype/tools/publisher/window.py diff --git a/openpype/tools/settings/settings/base.py b/openpype/tools/settings/settings/base.py index ab6b27bdaf..92fffe6f9c 100644 --- a/openpype/tools/settings/settings/base.py +++ b/openpype/tools/settings/settings/base.py @@ -214,7 +214,8 @@ class BaseWidget(QtWidgets.QWidget): def _paste_value_actions(self, menu): output = [] # Allow paste of value only if were copied from this UI - mime_data = QtWidgets.QApplication.clipboard().mimeData() + clipboard = QtWidgets.QApplication.clipboard() + mime_data = clipboard.mimeData() mime_value = mime_data.data("application/copy_settings_value") # Skip if there is nothing to do if not mime_value: diff --git a/openpype/tools/settings/settings/categories.py b/openpype/tools/settings/settings/categories.py index be2264340b..5f9051344d 100644 --- a/openpype/tools/settings/settings/categories.py +++ b/openpype/tools/settings/settings/categories.py @@ -508,7 +508,7 @@ class SettingsCategoryWidget(QtWidgets.QWidget): first_invalid_item = invalid_items[0] self.scroll_widget.ensureWidgetVisible(first_invalid_item) if first_invalid_item.isVisible(): - first_invalid_item.setFocus(True) + first_invalid_item.setFocus() return False def on_saved(self, saved_tab_widget): diff --git a/openpype/tools/settings/settings/dict_mutable_widget.py b/openpype/tools/settings/settings/dict_mutable_widget.py index cfb9d4a4b1..9afce7259e 100644 --- a/openpype/tools/settings/settings/dict_mutable_widget.py +++ b/openpype/tools/settings/settings/dict_mutable_widget.py @@ -128,9 +128,9 @@ class ModifiableDictEmptyItem(QtWidgets.QWidget): def add_new_item(self, key=None, label=None): input_field = self.entity_widget.add_new_key(key, label) if self.collapsible_key: - self.key_input.setFocus(True) + self.key_input.setFocus() else: - input_field.key_input.setFocus(True) + input_field.key_input.setFocus() return input_field def _on_add_clicked(self): @@ -563,7 +563,7 @@ class ModifiableDictItem(QtWidgets.QWidget): def on_add_clicked(self): widget = self.entity_widget.add_new_key(None, None) - widget.key_input.setFocus(True) + widget.key_input.setFocus() def on_edit_pressed(self): if not self.key_input.isVisible(): diff --git a/openpype/tools/settings/settings/list_item_widget.py b/openpype/tools/settings/settings/list_item_widget.py index 17412a30b9..128af92631 100644 --- a/openpype/tools/settings/settings/list_item_widget.py +++ b/openpype/tools/settings/settings/list_item_widget.py @@ -357,7 +357,7 @@ class ListWidget(InputWidget): new_entity = self.entity.add_new_item(row) input_field = self._input_fields_by_entity_id.get(new_entity.id) if input_field is not None: - input_field.input_field.setFocus(True) + input_field.input_field.setFocus() return new_entity def add_row(self, child_entity, row=None): diff --git a/openpype/tools/tray/pype_tray.py b/openpype/tools/tray/pype_tray.py index 3050e206ce..0f817d7130 100644 --- a/openpype/tools/tray/pype_tray.py +++ b/openpype/tools/tray/pype_tray.py @@ -268,7 +268,6 @@ class SystemTrayIcon(QtWidgets.QSystemTrayIcon): # Set modules self.tray_man = TrayManager(self, self.parent) - self.tray_man.initialize_modules() # Add menu to Context of SystemTrayIcon self.setContextMenu(self.menu) @@ -291,6 +290,17 @@ class SystemTrayIcon(QtWidgets.QSystemTrayIcon): self._doubleclick = False self._click_pos = None + self._initializing_modules = False + + @property + def initializing_modules(self): + return self._initializing_modules + + def initialize_modules(self): + self._initializing_modules = True + self.tray_man.initialize_modules() + self._initializing_modules = False + def _click_timer_timeout(self): self._click_timer.stop() doubleclick = self._doubleclick @@ -334,38 +344,48 @@ class SystemTrayIcon(QtWidgets.QSystemTrayIcon): QtCore.QCoreApplication.exit() -class TrayMainWindow(QtWidgets.QMainWindow): - """ TrayMainWindow is base of Pype application. - - Every widget should have set this window as parent because - QSystemTrayIcon widget is not allowed to be a parent of any widget. - """ - +class PypeTrayStarter(QtCore.QObject): def __init__(self, app): - super(TrayMainWindow, self).__init__() - self.app = app + app.setQuitOnLastWindowClosed(False) + self._app = app + self._splash = None - self.tray_widget = SystemTrayIcon(self) - self.tray_widget.show() + main_window = QtWidgets.QMainWindow() + tray_widget = SystemTrayIcon(main_window) + start_timer = QtCore.QTimer() + start_timer.setInterval(100) + start_timer.start() -class PypeTrayApplication(QtWidgets.QApplication): - """Qt application manages application's control flow.""" + start_timer.timeout.connect(self._on_start_timer) - def __init__(self): - super(PypeTrayApplication, self).__init__(sys.argv) - # Allows to close widgets without exiting app - self.setQuitOnLastWindowClosed(False) + self._main_window = main_window + self._tray_widget = tray_widget + self._timer_counter = 0 + self._start_timer = start_timer - # Sets up splash - splash_widget = self.set_splash() + def _on_start_timer(self): + if self._timer_counter == 0: + self._timer_counter += 1 + splash = self._get_splash() + splash.show() + self._tray_widget.show() - splash_widget.show() - self.processEvents() - self.main_window = TrayMainWindow(self) - splash_widget.hide() + elif self._timer_counter == 1: + self._timer_counter += 1 + self._tray_widget.initialize_modules() - def set_splash(self): + elif not self._tray_widget.initializing_modules: + splash = self._get_splash() + splash.hide() + self._start_timer.stop() + + def _get_splash(self): + if self._splash is None: + self._splash = self._create_splash() + return self._splash + + def _create_splash(self): splash_pix = QtGui.QPixmap(resources.get_openpype_splash_filepath()) splash = QtWidgets.QSplashScreen(splash_pix) splash.setMask(splash_pix.mask()) @@ -377,7 +397,12 @@ class PypeTrayApplication(QtWidgets.QApplication): def main(): - app = PypeTrayApplication() + app = QtWidgets.QApplication.instance() + if not app: + app = QtWidgets.QApplication([]) + + starter = PypeTrayStarter(app) + # TODO remove when pype.exe will have an icon if os.name == "nt": import ctypes diff --git a/openpype/version.py b/openpype/version.py index 99ba3fd543..d88d79b995 100644 --- a/openpype/version.py +++ b/openpype/version.py @@ -1,3 +1,3 @@ # -*- coding: utf-8 -*- """Package declaring Pype version.""" -__version__ = "3.5.0-nightly.4" +__version__ = "3.5.0" diff --git a/poetry.lock b/poetry.lock index 10b049cd0a..e5f5919a01 100644 --- a/poetry.lock +++ b/poetry.lock @@ -288,15 +288,20 @@ test = ["pytest (>=6.0)", "pytest-cov", "pytest-subtests", "pytest-xdist", "pret [[package]] name = "cx-freeze" -version = "6.6" +version = "6.7" description = "Create standalone executables from Python scripts" category = "dev" optional = false python-versions = ">=3.6" [package.dependencies] -cx-Logging = {version = ">=3.0", markers = "sys_platform == \"win32\""} -importlib-metadata = ">=3.1.1" +cx-logging = {version = ">=3.0", markers = "sys_platform == \"win32\""} +importlib-metadata = ">=4.3.1" + +[package.source] +type = "legacy" +url = "https://distribute.openpype.io/wheels" +reference = "openpype" [[package]] name = "cx-logging" @@ -1013,34 +1018,6 @@ category = "main" optional = false python-versions = ">=2.6, !=3.0.*, !=3.1.*, !=3.2.*" -[[package]] -name = "pyqt5" -version = "5.15.4" -description = "Python bindings for the Qt cross platform application toolkit" -category = "main" -optional = false -python-versions = ">=3.6" - -[package.dependencies] -PyQt5-Qt5 = ">=5.15" -PyQt5-sip = ">=12.8,<13" - -[[package]] -name = "pyqt5-qt5" -version = "5.15.2" -description = "The subset of a Qt installation needed by PyQt5." -category = "main" -optional = false -python-versions = "*" - -[[package]] -name = "pyqt5-sip" -version = "12.9.0" -description = "The sip module support for PyQt5" -category = "main" -optional = false -python-versions = ">=3.5" - [[package]] name = "pyrsistent" version = "0.17.3" @@ -1814,17 +1791,7 @@ cryptography = [ {file = "cryptography-3.4.7-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:bf40af59ca2465b24e54f671b2de2c59257ddc4f7e5706dbd6930e26823668d3"}, {file = "cryptography-3.4.7.tar.gz", hash = "sha256:3d10de8116d25649631977cb37da6cbdd2d6fa0e0281d014a5b7d337255ca713"}, ] -cx-freeze = [ - {file = "cx_Freeze-6.6-cp36-cp36m-win32.whl", hash = "sha256:b3d3a6bcd1a07c50b4e1c907f14842642156110e63a99cd5c73b8a24751e9b97"}, - {file = "cx_Freeze-6.6-cp36-cp36m-win_amd64.whl", hash = "sha256:1935266ec644ea4f7e584985f44cefc0622a449a09980d990833a1a2afcadac8"}, - {file = "cx_Freeze-6.6-cp37-cp37m-win32.whl", hash = "sha256:1eac2b0f254319cc641ce25bd83337effd7936092562fde701f3ffb40e0274ec"}, - {file = "cx_Freeze-6.6-cp37-cp37m-win_amd64.whl", hash = "sha256:2bc46ef6d510811b6002f34a3ae4cbfdea44e18644febd2a404d3ee8e48a9fc4"}, - {file = "cx_Freeze-6.6-cp38-cp38-win32.whl", hash = "sha256:46eb50ebc46f7ae236d16c6a52671ab0f7bb479bea668da19f4b6de3cc413e9e"}, - {file = "cx_Freeze-6.6-cp38-cp38-win_amd64.whl", hash = "sha256:8c3b00476ce385bb58595bffce55aed031e5a6e16ab6e14d8bee9d1d569e46c3"}, - {file = "cx_Freeze-6.6-cp39-cp39-win32.whl", hash = "sha256:6e9340cbcf52d4836980ecc83ddba4f7704ff6654dd41168c146b74f512977ce"}, - {file = "cx_Freeze-6.6-cp39-cp39-win_amd64.whl", hash = "sha256:2fcf1c8b77ae5c06f45be3a9aff79e1dd808c0d624e97561f840dec5ea9b214a"}, - {file = "cx_Freeze-6.6.tar.gz", hash = "sha256:c4af8ad3f7e7d71e291c1dec5d0fb26bbe92df834b098ed35434c901fbd6762f"}, -] +cx-freeze = [] cx-logging = [ {file = "cx_Logging-3.0-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:9fcd297e5c51470521c47eff0f86ba844aeca6be97e13c3e2114ebdf03fa3c96"}, {file = "cx_Logging-3.0-cp36-cp36m-win32.whl", hash = "sha256:0df4be47c5022cc54316949e283403214568ef599817ced0c0972183d6d4fabb"}, @@ -2355,38 +2322,6 @@ pyparsing = [ {file = "pyparsing-2.4.7-py2.py3-none-any.whl", hash = "sha256:ef9d7589ef3c200abe66653d3f1ab1033c3c419ae9b9bdb1240a85b024efc88b"}, {file = "pyparsing-2.4.7.tar.gz", hash = "sha256:c203ec8783bf771a155b207279b9bccb8dea02d8f0c9e5f8ead507bc3246ecc1"}, ] -pyqt5 = [ - {file = "PyQt5-5.15.4-cp36.cp37.cp38.cp39-abi3-macosx_10_13_intel.whl", hash = "sha256:8c0848ba790a895801d5bfd171da31cad3e551dbcc4e59677a3b622de2ceca98"}, - {file = "PyQt5-5.15.4-cp36.cp37.cp38.cp39-abi3-manylinux2014_x86_64.whl", hash = "sha256:883a549382fc22d29a0568f3ef20b38c8e7ab633a59498ac4eb63a3bf36d3fd3"}, - {file = "PyQt5-5.15.4-cp36.cp37.cp38.cp39-none-win32.whl", hash = "sha256:a88526a271e846e44779bb9ad7a738c6d3c4a9d01e15a128ecfc6dd4696393b7"}, - {file = "PyQt5-5.15.4-cp36.cp37.cp38.cp39-none-win_amd64.whl", hash = "sha256:213bebd51821ed89b4d5b35bb10dbe67564228b3568f463a351a08e8b1677025"}, - {file = "PyQt5-5.15.4.tar.gz", hash = "sha256:2a69597e0dd11caabe75fae133feca66387819fc9bc050f547e5551bce97e5be"}, -] -pyqt5-qt5 = [ - {file = "PyQt5_Qt5-5.15.2-py3-none-macosx_10_13_intel.whl", hash = "sha256:76980cd3d7ae87e3c7a33bfebfaee84448fd650bad6840471d6cae199b56e154"}, - {file = "PyQt5_Qt5-5.15.2-py3-none-manylinux2014_x86_64.whl", hash = "sha256:1988f364ec8caf87a6ee5d5a3a5210d57539988bf8e84714c7d60972692e2f4a"}, - {file = "PyQt5_Qt5-5.15.2-py3-none-win32.whl", hash = "sha256:9cc7a768b1921f4b982ebc00a318ccb38578e44e45316c7a4a850e953e1dd327"}, - {file = "PyQt5_Qt5-5.15.2-py3-none-win_amd64.whl", hash = "sha256:750b78e4dba6bdf1607febedc08738e318ea09e9b10aea9ff0d73073f11f6962"}, -] -pyqt5-sip = [ - {file = "PyQt5_sip-12.9.0-cp36-cp36m-macosx_10_6_intel.whl", hash = "sha256:d85002238b5180bce4b245c13d6face848faa1a7a9e5c6e292025004f2fd619a"}, - {file = "PyQt5_sip-12.9.0-cp36-cp36m-manylinux1_x86_64.whl", hash = "sha256:83c3220b1ca36eb8623ba2eb3766637b19eb0ce9f42336ad8253656d32750c0a"}, - {file = "PyQt5_sip-12.9.0-cp36-cp36m-win32.whl", hash = "sha256:d8b2bdff7bbf45bc975c113a03b14fd669dc0c73e1327f02706666a7dd51a197"}, - {file = "PyQt5_sip-12.9.0-cp36-cp36m-win_amd64.whl", hash = "sha256:69a3ad4259172e2b1aa9060de211efac39ddd734a517b1924d9c6c0cc4f55f96"}, - {file = "PyQt5_sip-12.9.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:42274a501ab4806d2c31659170db14c282b8313d2255458064666d9e70d96206"}, - {file = "PyQt5_sip-12.9.0-cp37-cp37m-manylinux1_x86_64.whl", hash = "sha256:6a8701892a01a5a2a4720872361197cc80fdd5f49c8482d488ddf38c9c84f055"}, - {file = "PyQt5_sip-12.9.0-cp37-cp37m-win32.whl", hash = "sha256:ac57d796c78117eb39edd1d1d1aea90354651efac9d3590aac67fa4983f99f1f"}, - {file = "PyQt5_sip-12.9.0-cp37-cp37m-win_amd64.whl", hash = "sha256:4347bd81d30c8e3181e553b3734f91658cfbdd8f1a19f254777f906870974e6d"}, - {file = "PyQt5_sip-12.9.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:c446971c360a0a1030282a69375a08c78e8a61d568bfd6dab3dcc5cf8817f644"}, - {file = "PyQt5_sip-12.9.0-cp38-cp38-manylinux1_x86_64.whl", hash = "sha256:fc43f2d7c438517ee33e929e8ae77132749c15909afab6aeece5fcf4147ffdb5"}, - {file = "PyQt5_sip-12.9.0-cp38-cp38-win32.whl", hash = "sha256:055581c6fed44ba4302b70eeb82e979ff70400037358908f251cd85cbb3dbd93"}, - {file = "PyQt5_sip-12.9.0-cp38-cp38-win_amd64.whl", hash = "sha256:c5216403d4d8d857ec4a61f631d3945e44fa248aa2415e9ee9369ab7c8a4d0c7"}, - {file = "PyQt5_sip-12.9.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:a25b9843c7da6a1608f310879c38e6434331aab1dc2fe6cb65c14f1ecf33780e"}, - {file = "PyQt5_sip-12.9.0-cp39-cp39-manylinux1_x86_64.whl", hash = "sha256:dd05c768c2b55ffe56a9d49ce6cc77cdf3d53dbfad935258a9e347cbfd9a5850"}, - {file = "PyQt5_sip-12.9.0-cp39-cp39-win32.whl", hash = "sha256:4f8e05fe01d54275877c59018d8e82dcdd0bc5696053a8b830eecea3ce806121"}, - {file = "PyQt5_sip-12.9.0-cp39-cp39-win_amd64.whl", hash = "sha256:b09f4cd36a4831229fb77c424d89635fa937d97765ec90685e2f257e56a2685a"}, - {file = "PyQt5_sip-12.9.0.tar.gz", hash = "sha256:d3e4489d7c2b0ece9d203ae66e573939f7f60d4d29e089c9f11daa17cfeaae32"}, -] pyrsistent = [ {file = "pyrsistent-0.17.3.tar.gz", hash = "sha256:2e636185d9eb976a18a8a8e96efce62f2905fea90041958d8cc2a189756ebf3e"}, ] diff --git a/pyproject.toml b/pyproject.toml index fb47c7eff6..085538d306 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -49,7 +49,6 @@ Pillow = "^8.1" # only used for slates prototype pyblish-base = "^1.8.8" pynput = "^1.7.2" # idle manager in tray pymongo = "^3.11.2" -pyqt5 = "^5.12.2" # ideally should be replaced with PySide2 "Qt.py" = "^1.3.3" speedcopy = "^2.1" six = "^1.15" @@ -63,14 +62,16 @@ jinxed = [ python3-xlib = { version="*", markers = "sys_platform == 'linux'"} enlighten = "^1.9.0" slack-sdk = "^3.6.0" +requests = "2.25.1" pysftp = "^0.2.9" dropbox = "^11.20.0" + [tool.poetry.dev-dependencies] flake8 = "^3.7" autopep8 = "^1.4" coverage = "*" -cx_freeze = "^6.6" +cx_freeze = { version = "6.7", source = "openpype" } GitPython = "^3.1.17" jedi = "^0.13" Jinja2 = "^2.11" @@ -89,7 +90,6 @@ wheel = "*" enlighten = "*" # cool terminal progress bars toml = "^0.10.2" # for parsing pyproject.toml - [tool.poetry.urls] "Bug Tracker" = "https://github.com/pypeclub/openpype/issues" "Discussions" = "https://github.com/pypeclub/openpype/discussions" @@ -104,6 +104,12 @@ build-backend = "poetry.core.masonry.api" [openpype] +[openpype.pyside2] +# note: in here we can use pip version specifiers as this is installed with pip until +# Poetry will support custom location (-t flag for pip) +# https://pip.pypa.io/en/stable/cli/pip_install/#requirement-specifiers +version = "==5.15.2" + [openpype.thirdparty.ffmpeg.windows] url = "https://distribute.openpype.io/thirdparty/ffmpeg-4.4-windows.zip" hash = "dd51ba29d64ee238e7c4c3c7301b19754c3f0ee2e2a729c20a0e2789e72db925" diff --git a/repos/avalon-core b/repos/avalon-core index 4b80f81e66..7e5efd6885 160000 --- a/repos/avalon-core +++ b/repos/avalon-core @@ -1 +1 @@ -Subproject commit 4b80f81e66aca593784be8b299110a0b6541276f +Subproject commit 7e5efd6885330d84bb8495975bcab84df49bfa3d diff --git a/setup.py b/setup.py index 55098cb0b4..cd3ed4f82c 100644 --- a/setup.py +++ b/setup.py @@ -59,13 +59,14 @@ includes = [] excludes = [ "openpype" ] -bin_includes = [] +bin_includes = [ + "vendor" +] include_files = [ "igniter", "openpype", "repos", "schema", - "vendor", "LICENSE", "README.md" ] diff --git a/start.py b/start.py index ada613b4eb..29b4921d19 100644 --- a/start.py +++ b/start.py @@ -124,6 +124,10 @@ else: paths.append(frozen_libs) os.environ["PYTHONPATH"] = os.pathsep.join(paths) +# Vendored python modules that must not be in PYTHONPATH environment but +# are required for OpenPype processes +vendor_python_path = os.path.join(OPENPYPE_ROOT, "vendor", "python") +sys.path.insert(0, vendor_python_path) import blessed # noqa: E402 import certifi # noqa: E402 @@ -287,9 +291,8 @@ def run_disk_mapping_commands(mongo_url): if not disk_mapping: return - for mapping in disk_mapping.get(low_platform): - source, destination = mapping - + mappings = disk_mapping.get(low_platform) or [] + for source, destination in mappings: args = ["subst", destination.rstrip('/'), source.rstrip('/')] _print("disk mapping args:: {}".format(args)) try: diff --git a/tests/README.md b/tests/README.md index e69de29bb2..6317b2ab3c 100644 --- a/tests/README.md +++ b/tests/README.md @@ -0,0 +1,25 @@ +Automatic tests for OpenPype +============================ +Structure: +- integration - end to end tests, slow (see README.md in the integration folder for more info) + - openpype/modules/MODULE_NAME - structure follow directory structure in code base + - fixture - sample data `(MongoDB dumps, test files etc.)` + - `tests.py` - single or more pytest files for MODULE_NAME +- unit - quick unit test + - MODULE_NAME + - fixture + - `tests.py` + +How to run: +---------- +- single test class could be run by PyCharm and its pytest runner directly +- OR +- use Openpype command 'runtests' from command line +-- `${OPENPYPE_ROOT}/start.py runtests` + +By default, this command will run all tests in ${OPENPYPE_ROOT}/tests. + +Specific location could be provided to this command as an argument, either as absolute path, or relative path to ${OPENPYPE_ROOT}. +(eg. `${OPENPYPE_ROOT}/start.py runtests ../tests/integration`) will trigger only tests in `integration` folder. + +See `${OPENPYPE_ROOT}/cli.py:runtests` for other arguments. diff --git a/tests/__init__.py b/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/integration/README.md b/tests/integration/README.md new file mode 100644 index 0000000000..81c07ec50c --- /dev/null +++ b/tests/integration/README.md @@ -0,0 +1,37 @@ +Integration test for OpenPype +============================= +Contains end-to-end tests for automatic testing of OP. + +Should run headless publish on all hosts to check basic publish use cases automatically +to limit regression issues. + +How to create test for publishing from host +------------------------------------------ +- Extend PublishTest +- Use `resources\test_data.zip` skeleton file as a template for testing input data +- Put workfile into `test_data.zip/input/workfile` +- If you require other than base DB dumps provide them to `test_data.zip/input/dumps` +-- (Check commented code in `db_handler.py` how to dump specific DB. Currently all collections will be dumped.) +- Implement `last_workfile_path` +- `startup_scripts` - must contain pointing host to startup script saved into `test_data.zip/input/startup` + -- Script must contain something like +``` +import openpype +from avalon import api, HOST + +api.install(HOST) +pyblish.util.publish() + +EXIT_APP (command to exit host) +``` +(Install and publish methods must be triggered only AFTER host app is fully initialized!) +- Zip `test_data.zip`, named it with descriptive name, upload it to Google Drive, right click - `Get link`, copy hash id +- Put this hash id and zip file name into TEST_FILES [(HASH_ID, FILE_NAME, MD5_OPTIONAL)]. If you want to check MD5 of downloaded +file, provide md5 value of zipped file. +- Implement any assert checks you need in extended class +- Run test class manually (via Pycharm or pytest runner (TODO)) +- If you want test to compare expected files to published one, set PERSIST to True, run test manually + -- Locate temporary `publish` subfolder of temporary folder (found in debugging console log) + -- Copy whole folder content into .zip file into `expected` subfolder + -- By default tests are comparing only structure of `expected` and published format (eg. if you want to save space, replace published files with empty files, but with expected names!) + -- Zip and upload again, change PERSIST to False \ No newline at end of file diff --git a/tests/integration/hosts/maya/test_publish_in_maya.py b/tests/integration/hosts/maya/test_publish_in_maya.py new file mode 100644 index 0000000000..1babf30029 --- /dev/null +++ b/tests/integration/hosts/maya/test_publish_in_maya.py @@ -0,0 +1,103 @@ +import pytest +import os +import shutil + +from tests.lib.testing_classes import PublishTest + + +class TestPublishInMaya(PublishTest): + """Basic test case for publishing in Maya + + Shouldnt be running standalone only via 'runtests' pype command! (??) + + Uses generic TestCase to prepare fixtures for test data, testing DBs, + env vars. + + Opens Maya, run publish on prepared workile. + + Then checks content of DB (if subset, version, representations were + created. + Checks tmp folder if all expected files were published. + + """ + PERSIST = True + + TEST_FILES = [ + ("1pOwjA_VVBc6ooTZyFxtAwLS2KZHaBlkY", "test_maya_publish.zip", "") + ] + + APP = "maya" + APP_VARIANT = "2019" + + APP_NAME = "{}/{}".format(APP, APP_VARIANT) + + TIMEOUT = 120 # publish timeout + + @pytest.fixture(scope="module") + def last_workfile_path(self, download_test_data): + """Get last_workfile_path from source data. + + Maya expects workfile in proper folder, so copy is done first. + """ + src_path = os.path.join(download_test_data, + "input", + "workfile", + "test_project_test_asset_TestTask_v001.mb") + dest_folder = os.path.join(download_test_data, + self.PROJECT, + self.ASSET, + "work", + self.TASK) + os.makedirs(dest_folder) + dest_path = os.path.join(dest_folder, + "test_project_test_asset_TestTask_v001.mb") + shutil.copy(src_path, dest_path) + + yield dest_path + + @pytest.fixture(scope="module") + def startup_scripts(self, monkeypatch_session, download_test_data): + """Points Maya to userSetup file from input data""" + startup_path = os.path.join(download_test_data, + "input", + "startup") + original_pythonpath = os.environ.get("PYTHONPATH") + monkeypatch_session.setenv("PYTHONPATH", + "{}{}{}".format(startup_path, + os.pathsep, + original_pythonpath)) + + def test_db_asserts(self, dbcon, publish_finished): + """Host and input data dependent expected results in DB.""" + print("test_db_asserts") + assert 5 == dbcon.count_documents({"type": "version"}), \ + "Not expected no of versions" + + assert 0 == dbcon.count_documents({"type": "version", + "name": {"$ne": 1}}), \ + "Only versions with 1 expected" + + assert 1 == dbcon.count_documents({"type": "subset", + "name": "modelMain"}), \ + "modelMain subset must be present" + + assert 1 == dbcon.count_documents({"type": "subset", + "name": "workfileTest_task"}), \ + "workfileTest_task subset must be present" + + assert 11 == dbcon.count_documents({"type": "representation"}), \ + "Not expected no of representations" + + assert 2 == dbcon.count_documents({"type": "representation", + "context.subset": "modelMain", + "context.ext": "abc"}), \ + "Not expected no of representations with ext 'abc'" + + assert 2 == dbcon.count_documents({"type": "representation", + "context.subset": "modelMain", + "context.ext": "ma"}), \ + "Not expected no of representations with ext 'abc'" + + +if __name__ == "__main__": + test_case = TestPublishInMaya() diff --git a/tests/lib/README.md b/tests/lib/README.md new file mode 100644 index 0000000000..0384cd2ff0 --- /dev/null +++ b/tests/lib/README.md @@ -0,0 +1,46 @@ +Automatic testing +----------------- +Folder for libs and tooling for automatic testing. + +- db_handler.py - class for preparation of test DB + - dumps DB(s) to BSON (mongodump) + - loads dump(s) to new DB (mongorestore) + - loads sql file(s) to DB (mongoimport) + - deletes test DB + +- file_handler.py - class to download test data from GDrive + - downloads data from (list) of files from GDrive + - check file integrity with MD5 hash + - unzips if zip + +- testing_wrapper.py - base class to use for testing + - all env var necessary for running (OPENPYPE_MONGO ...) + - implements reusable fixtures to: + - load test data (uses `file_handler`) + - prepare DB (uses `db_handler`) + - modify temporarily env vars for testing + + Should be used as a skeleton to create new test cases. + + +Test data +--------- +Each class implementing `TestCase` can provide test file(s) by adding them to +TEST_FILES ('GDRIVE_FILE_ID', 'ACTUAL_FILE_NAME', 'MD5HASH') + +GDRIVE_FILE_ID can be pulled from shareable link from Google Drive app. + +Currently it is expected that test file will be zip file with structure: +- expected - expected files (not implemented yet) +- input + - data - test data (workfiles, images etc) + - dumps - folder for BSON dumps from (`mongodump`) + - env_vars + env_vars.json - dictionary with environment variables {key:value} + + - json - json files to load with `mongoimport` (human readable) + + +Example +------- +See `tests\unit\openpype\modules\sync_server\test_site_operations.py` for example usage of implemented classes. \ No newline at end of file diff --git a/tests/lib/__init__.py b/tests/lib/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/tests/lib/db_handler.py b/tests/lib/db_handler.py new file mode 100644 index 0000000000..9be70895da --- /dev/null +++ b/tests/lib/db_handler.py @@ -0,0 +1,231 @@ +""" + Helper class for automatic testing, provides dump and restore via command + line utilities. + + Expect mongodump, mongoimport and mongorestore present at PATH +""" +import os +import pymongo +import subprocess + + +class DBHandler: + + def __init__(self, uri=None, host=None, port=None, + user=None, password=None): + """'uri' or rest of separate credentials""" + if uri: + self.uri = uri + if host: + if all([user, password]): + host = "{}:{}@{}".format(user, password, host) + self.uri = 'mongodb://{}:{}'.format(host, port or 27017) + + assert self.uri, "Must have uri to MongoDB" + self.client = pymongo.MongoClient(uri) + self.db = None + + def setup_empty(self, name): + # not much sense + self.db = self.client[name] + + def setup_from_sql(self, db_name, sql_dir, collection=None, + drop=True, mode=None): + """ + Restores 'db_name' from 'sql_url'. + + Works with directory with .json files, + if 'collection' arg is empty, name + of .json file is used as name of target collection. + + Args: + db_name (str): source DB name + sql_dir (str): folder with json files + collection (str): if all sql files are meant for single coll. + drop (bool): True if drop whole collection + mode (str): "insert" - fails on duplicates + "upsert" - modifies existing + "merge" - updates existing + "delete" - removes in DB present if file + """ + if not os.path.exists(sql_dir): + raise RuntimeError( + "Backup folder {} doesn't exist".format(sql_dir)) + + for (dirpath, _dirnames, filenames) in os.walk(sql_dir): + for file_name in filenames: + sql_url = os.path.join(dirpath, file_name) + query = self._import_query(self.uri, sql_url, + db_name=db_name, + collection=collection, + drop=drop, + mode=mode) + + print("mongoimport query:: {}".format(query)) + subprocess.run(query) + + def setup_from_sql_file(self, db_name, sql_url, + collection=None, drop=True, mode=None): + """ + Restores 'db_name' from 'sql_url'. + + Works with single .json file. + If 'collection' arg is empty, name + of .json file is used as name of target collection. + + Args: + db_name (str): source DB name + sql_file (str): folder with json files + collection (str): name of target collection + drop (bool): True if drop collection + mode (str): "insert" - fails on duplicates + "upsert" - modifies existing + "merge" - updates existing + "delete" - removes in DB present if file + """ + if not os.path.exists(sql_url): + raise RuntimeError( + "Sql file {} doesn't exist".format(sql_url)) + + query = self._import_query(self.uri, sql_url, + db_name=db_name, + collection=collection, + drop=drop, + mode=mode) + + print("mongoimport query:: {}".format(query)) + subprocess.run(query) + + def setup_from_dump(self, db_name, dump_dir, overwrite=False, + collection=None, db_name_out=None): + """ + Restores 'db_name' from 'dump_dir'. + + Works with BSON folders exported by mongodump + + Args: + db_name (str): source DB name + dump_dir (str): folder with dumped subfolders + overwrite (bool): True if overwrite target + collection (str): name of source project + db_name_out (str): name of target DB, if empty restores to + source 'db_name' + """ + db_name_out = db_name_out or db_name + if self._db_exists(db_name) and not overwrite: + raise RuntimeError("DB {} already exists".format(db_name_out) + + "Run with overwrite=True") + + dir_path = os.path.join(dump_dir, db_name) + if not os.path.exists(dir_path): + raise RuntimeError( + "Backup folder {} doesn't exist".format(dir_path)) + + query = self._restore_query(self.uri, dump_dir, + db_name=db_name, db_name_out=db_name_out, + collection=collection) + print("mongorestore query:: {}".format(query)) + subprocess.run(query) + + def teardown(self, db_name): + """Drops 'db_name' if exists.""" + if not self._db_exists(db_name): + print("{} doesn't exist".format(db_name)) + return + + print("Dropping {} database".format(db_name)) + self.client.drop_database(db_name) + + def backup_to_dump(self, db_name, dump_dir, overwrite=False): + """ + Helper method for running mongodump for specific 'db_name' + """ + if not self._db_exists(db_name) and not overwrite: + raise RuntimeError("DB {} doesn't exists".format(db_name)) + + dir_path = os.path.join(dump_dir, db_name) + if os.path.exists(dir_path) and not overwrite: + raise RuntimeError("Backup already exists, " + "run with overwrite=True") + + query = self._dump_query(self.uri, dump_dir, db_name=db_name) + print("Mongodump query:: {}".format(query)) + subprocess.run(query) + + def _db_exists(self, db_name): + return db_name in self.client.list_database_names() + + def _dump_query(self, uri, output_path, db_name=None, collection=None): + """Prepares dump query based on 'db_name' or 'collection'.""" + db_part = coll_part = "" + if db_name: + db_part = "--db={}".format(db_name) + if collection: + if not db_name: + raise ValueError("db_name must be present") + coll_part = "--nsInclude={}.{}".format(db_name, collection) + query = "\"{}\" --uri=\"{}\" --out={} {} {}".format( + "mongodump", uri, output_path, db_part, coll_part + ) + + return query + + def _restore_query(self, uri, dump_dir, + db_name=None, db_name_out=None, + collection=None, drop=True): + """Prepares query for mongorestore base on arguments""" + db_part = coll_part = drop_part = "" + if db_name: + db_part = "--nsInclude={}.* --nsFrom={}.*".format(db_name, db_name) + if collection: + assert db_name, "Must provide db name too" + db_part = "--nsInclude={}.{} --nsFrom={}.{}".format(db_name, + collection, + db_name, + collection) + if drop: + drop_part = "--drop" + + if db_name_out: + db_part += " --nsTo={}.*".format(db_name_out) + + query = "\"{}\" --uri=\"{}\" --dir=\"{}\" {} {} {}".format( + "mongorestore", uri, dump_dir, db_part, coll_part, drop_part + ) + + return query + + def _import_query(self, uri, sql_url, + db_name=None, + collection=None, drop=True, mode=None): + + db_part = coll_part = drop_part = mode_part = "" + if db_name: + db_part = "--db {}".format(db_name) + if collection: + assert db_name, "Must provide db name too" + coll_part = "--collection {}".format(collection) + if drop: + drop_part = "--drop" + if mode: + mode_part = "--mode {}".format(mode) + + query = \ + "\"{}\" --legacy --uri=\"{}\" --file=\"{}\" {} {} {} {}".format( + "mongoimport", uri, sql_url, + db_part, coll_part, drop_part, mode_part) + + return query + +# handler = DBHandler(uri="mongodb://localhost:27017") +# +# backup_dir = "c:\\projects\\dumps" +# # +# handler.backup_to_dump("openpype", backup_dir, True) +# # handler.setup_from_dump("test_db", backup_dir, True) +# # handler.setup_from_sql_file("test_db", "c:\\projects\\sql\\item.sql", +# # collection="test_project", +# # drop=False, mode="upsert") +# handler.setup_from_sql("test_db", "c:\\projects\\sql", +# collection="test_project", +# drop=False, mode="upsert") diff --git a/tests/lib/file_handler.py b/tests/lib/file_handler.py new file mode 100644 index 0000000000..ee3abc6ecb --- /dev/null +++ b/tests/lib/file_handler.py @@ -0,0 +1,260 @@ +import enlighten +import os +import re +import urllib +from urllib.parse import urlparse +import urllib.request +import urllib.error +import itertools +import hashlib +import tarfile +import zipfile + + +USER_AGENT = "openpype" + + +class RemoteFileHandler: + """Download file from url, might be GDrive shareable link""" + + IMPLEMENTED_ZIP_FORMATS = ['zip', 'tar', 'tgz', + 'tar.gz', 'tar.xz', 'tar.bz2'] + + @staticmethod + def calculate_md5(fpath, chunk_size): + md5 = hashlib.md5() + with open(fpath, 'rb') as f: + for chunk in iter(lambda: f.read(chunk_size), b''): + md5.update(chunk) + return md5.hexdigest() + + @staticmethod + def check_md5(fpath, md5, **kwargs): + return md5 == RemoteFileHandler.calculate_md5(fpath, **kwargs) + + @staticmethod + def check_integrity(fpath, md5=None): + if not os.path.isfile(fpath): + return False + if md5 is None: + return True + return RemoteFileHandler.check_md5(fpath, md5) + + @staticmethod + def download_url( + url, root, filename=None, + md5=None, max_redirect_hops=3 + ): + """Download a file from a url and place it in root. + Args: + url (str): URL to download file from + root (str): Directory to place downloaded file in + filename (str, optional): Name to save the file under. + If None, use the basename of the URL + md5 (str, optional): MD5 checksum of the download. + If None, do not check + max_redirect_hops (int, optional): Maximum number of redirect + hops allowed + """ + root = os.path.expanduser(root) + if not filename: + filename = os.path.basename(url) + fpath = os.path.join(root, filename) + + os.makedirs(root, exist_ok=True) + + # check if file is already present locally + if RemoteFileHandler.check_integrity(fpath, md5): + print('Using downloaded and verified file: ' + fpath) + return + + # expand redirect chain if needed + url = RemoteFileHandler._get_redirect_url(url, + max_hops=max_redirect_hops) + + # check if file is located on Google Drive + file_id = RemoteFileHandler._get_google_drive_file_id(url) + if file_id is not None: + return RemoteFileHandler.download_file_from_google_drive( + file_id, root, filename, md5) + + # download the file + try: + print('Downloading ' + url + ' to ' + fpath) + RemoteFileHandler._urlretrieve(url, fpath) + except (urllib.error.URLError, IOError) as e: + if url[:5] == 'https': + url = url.replace('https:', 'http:') + print('Failed download. Trying https -> http instead.' + ' Downloading ' + url + ' to ' + fpath) + RemoteFileHandler._urlretrieve(url, fpath) + else: + raise e + + # check integrity of downloaded file + if not RemoteFileHandler.check_integrity(fpath, md5): + raise RuntimeError("File not found or corrupted.") + + @staticmethod + def download_file_from_google_drive(file_id, root, + filename=None, + md5=None): + """Download a Google Drive file from and place it in root. + Args: + file_id (str): id of file to be downloaded + root (str): Directory to place downloaded file in + filename (str, optional): Name to save the file under. + If None, use the id of the file. + md5 (str, optional): MD5 checksum of the download. + If None, do not check + """ + # Based on https://stackoverflow.com/questions/38511444/python-download-files-from-google-drive-using-url # noqa + import requests + url = "https://docs.google.com/uc?export=download" + + root = os.path.expanduser(root) + if not filename: + filename = file_id + fpath = os.path.join(root, filename) + + os.makedirs(root, exist_ok=True) + + if os.path.isfile(fpath) and RemoteFileHandler.check_integrity(fpath, + md5): + print('Using downloaded and verified file: ' + fpath) + else: + session = requests.Session() + + response = session.get(url, params={'id': file_id}, stream=True) + token = RemoteFileHandler._get_confirm_token(response) + + if token: + params = {'id': file_id, 'confirm': token} + response = session.get(url, params=params, stream=True) + + response_content_generator = response.iter_content(32768) + first_chunk = None + while not first_chunk: # filter out keep-alive new chunks + first_chunk = next(response_content_generator) + + if RemoteFileHandler._quota_exceeded(first_chunk): + msg = ( + f"The daily quota of the file {filename} is exceeded and " + f"it can't be downloaded. This is a limitation of " + f"Google Drive and can only be overcome by trying " + f"again later." + ) + raise RuntimeError(msg) + + RemoteFileHandler._save_response_content( + itertools.chain((first_chunk, ), + response_content_generator), fpath) + response.close() + + @staticmethod + def unzip(path, destination_path=None): + if not destination_path: + destination_path = os.path.dirname(path) + + _, archive_type = os.path.splitext(path) + archive_type = archive_type.lstrip('.') + + if archive_type in ['zip']: + print("Unzipping {}->{}".format(path, destination_path)) + zip_file = zipfile.ZipFile(path) + zip_file.extractall(destination_path) + zip_file.close() + + elif archive_type in [ + 'tar', 'tgz', 'tar.gz', 'tar.xz', 'tar.bz2' + ]: + print("Unzipping {}->{}".format(path, destination_path)) + if archive_type == 'tar': + tar_type = 'r:' + elif archive_type.endswith('xz'): + tar_type = 'r:xz' + elif archive_type.endswith('gz'): + tar_type = 'r:gz' + elif archive_type.endswith('bz2'): + tar_type = 'r:bz2' + else: + tar_type = 'r:*' + try: + tar_file = tarfile.open(path, tar_type) + except tarfile.ReadError: + raise SystemExit("corrupted archive") + tar_file.extractall(destination_path) + tar_file.close() + + @staticmethod + def _urlretrieve(url, filename, chunk_size): + with open(filename, "wb") as fh: + with urllib.request.urlopen( + urllib.request.Request(url, + headers={"User-Agent": USER_AGENT})) \ + as response: + for chunk in iter(lambda: response.read(chunk_size), ""): + if not chunk: + break + fh.write(chunk) + + @staticmethod + def _get_redirect_url(url, max_hops): + initial_url = url + headers = {"Method": "HEAD", "User-Agent": USER_AGENT} + + for _ in range(max_hops + 1): + with urllib.request.urlopen( + urllib.request.Request(url, headers=headers)) as response: + if response.url == url or response.url is None: + return url + + url = response.url + else: + raise RecursionError( + f"Request to {initial_url} exceeded {max_hops} redirects. " + f"The last redirect points to {url}." + ) + + @staticmethod + def _get_confirm_token(response): + for key, value in response.cookies.items(): + if key.startswith('download_warning'): + return value + + return None + + @staticmethod + def _save_response_content( + response_gen, destination, + ): + with open(destination, "wb") as f: + pbar = enlighten.Counter( + total=None, desc="Save content", units="%", color="green") + progress = 0 + for chunk in response_gen: + if chunk: # filter out keep-alive new chunks + f.write(chunk) + progress += len(chunk) + + pbar.close() + + @staticmethod + def _quota_exceeded(first_chunk): + try: + return "Google Drive - Quota exceeded" in first_chunk.decode() + except UnicodeDecodeError: + return False + + @staticmethod + def _get_google_drive_file_id(url): + parts = urlparse(url) + + if re.match(r"(drive|docs)[.]google[.]com", parts.netloc) is None: + return None + + match = re.match(r"/file/d/(?P[^/]*)", parts.path) + if match is None: + return None + + return match.group("id") diff --git a/tests/lib/testing_classes.py b/tests/lib/testing_classes.py new file mode 100644 index 0000000000..1832efb7ed --- /dev/null +++ b/tests/lib/testing_classes.py @@ -0,0 +1,263 @@ +"""Testing classes for module testing and publishing in hosts.""" +import os +import sys +import six +import json +import pytest +import tempfile +import shutil +import glob + +from tests.lib.db_handler import DBHandler +from tests.lib.file_handler import RemoteFileHandler + + +class BaseTest: + """Empty base test class""" + + +class ModuleUnitTest(BaseTest): + """Generic test class for testing modules + + Use PERSIST==True to keep temporary folder and DB prepared for + debugging or preparation of test files. + + Implemented fixtures: + monkeypatch_session - fixture for env vars with session scope + download_test_data - tmp folder with extracted data from GDrive + env_var - sets env vars from input file + db_setup - prepares avalon AND openpype DBs for testing from + binary dumps from input data + dbcon - returns DBConnection to AvalonDB + dbcon_openpype - returns DBConnection for OpenpypeMongoDB + + """ + PERSIST = False # True to not purge temporary folder nor test DB + + TEST_OPENPYPE_MONGO = "mongodb://localhost:27017" + TEST_DB_NAME = "test_db" + TEST_PROJECT_NAME = "test_project" + TEST_OPENPYPE_NAME = "test_openpype" + + TEST_FILES = [] + + PROJECT = "test_project" + ASSET = "test_asset" + TASK = "test_task" + + @pytest.fixture(scope='session') + def monkeypatch_session(self): + """Monkeypatch couldn't be used with module or session fixtures.""" + from _pytest.monkeypatch import MonkeyPatch + m = MonkeyPatch() + yield m + m.undo() + + @pytest.fixture(scope="module") + def download_test_data(self): + tmpdir = tempfile.mkdtemp() + for test_file in self.TEST_FILES: + file_id, file_name, md5 = test_file + + f_name, ext = os.path.splitext(file_name) + + RemoteFileHandler.download_file_from_google_drive(file_id, + str(tmpdir), + file_name) + + if ext.lstrip('.') in RemoteFileHandler.IMPLEMENTED_ZIP_FORMATS: + RemoteFileHandler.unzip(os.path.join(tmpdir, file_name)) + print("Temporary folder created:: {}".format(tmpdir)) + yield tmpdir + + if not self.PERSIST: + print("Removing {}".format(tmpdir)) + shutil.rmtree(tmpdir) + + @pytest.fixture(scope="module") + def env_var(self, monkeypatch_session, download_test_data): + """Sets temporary env vars from json file.""" + env_url = os.path.join(download_test_data, "input", + "env_vars", "env_var.json") + if not os.path.exists(env_url): + raise ValueError("Env variable file {} doesn't exist". + format(env_url)) + + env_dict = {} + try: + with open(env_url) as json_file: + env_dict = json.load(json_file) + except ValueError: + print("{} doesn't contain valid JSON") + six.reraise(*sys.exc_info()) + + for key, value in env_dict.items(): + all_vars = globals() + all_vars.update(vars(ModuleUnitTest)) # TODO check + value = value.format(**all_vars) + print("Setting {}:{}".format(key, value)) + monkeypatch_session.setenv(key, str(value)) + import openpype + + openpype_root = os.path.dirname(os.path.dirname(openpype.__file__)) + # ?? why 2 of those + monkeypatch_session.setenv("OPENPYPE_ROOT", openpype_root) + monkeypatch_session.setenv("OPENPYPE_REPOS_ROOT", openpype_root) + + @pytest.fixture(scope="module") + def db_setup(self, download_test_data, env_var, monkeypatch_session): + """Restore prepared MongoDB dumps into selected DB.""" + backup_dir = os.path.join(download_test_data, "input", "dumps") + + uri = os.environ.get("OPENPYPE_MONGO") + db_handler = DBHandler(uri) + db_handler.setup_from_dump(self.TEST_DB_NAME, backup_dir, True, + db_name_out=self.TEST_DB_NAME) + + db_handler.setup_from_dump("openpype", backup_dir, True, + db_name_out=self.TEST_OPENPYPE_NAME) + + yield db_handler + + if not self.PERSIST: + db_handler.teardown(self.TEST_DB_NAME) + db_handler.teardown(self.TEST_OPENPYPE_NAME) + + @pytest.fixture(scope="module") + def dbcon(self, db_setup): + """Provide test database connection. + + Database prepared from dumps with 'db_setup' fixture. + """ + from avalon.api import AvalonMongoDB + dbcon = AvalonMongoDB() + dbcon.Session["AVALON_PROJECT"] = self.TEST_PROJECT_NAME + yield dbcon + + @pytest.fixture(scope="module") + def dbcon_openpype(self, db_setup): + """Provide test database connection for OP settings. + + Database prepared from dumps with 'db_setup' fixture. + """ + from openpype.lib import OpenPypeMongoConnection + mongo_client = OpenPypeMongoConnection.get_mongo_client() + yield mongo_client[self.TEST_OPENPYPE_NAME]["settings"] + + +class PublishTest(ModuleUnitTest): + """Test class for publishing in hosts. + + Implemented fixtures: + launched_app - launches APP with last_workfile_path + publish_finished - waits until publish is finished, host must + kill its process when finished publishing. Includes timeout + which raises ValueError + + Not implemented: + last_workfile_path - returns path to testing workfile + startup_scripts - provide script for setup in host + + Implemented tests: + test_folder_structure_same - compares published and expected + subfolders if they contain same files. Compares only on file + presence + + TODO: implement test on file size, file content + """ + + APP = "" + APP_VARIANT = "" + + APP_NAME = "{}/{}".format(APP, APP_VARIANT) + + TIMEOUT = 120 # publish timeout + + @pytest.fixture(scope="module") + def last_workfile_path(self, download_test_data): + raise NotImplementedError + + @pytest.fixture(scope="module") + def startup_scripts(self, monkeypatch_session, download_test_data): + raise NotImplementedError + + @pytest.fixture(scope="module") + def launched_app(self, dbcon, download_test_data, last_workfile_path, + startup_scripts): + """Launch host app""" + # set publishing folders + root_key = "config.roots.work.{}".format("windows") # TEMP + dbcon.update_one( + {"type": "project"}, + {"$set": + { + root_key: download_test_data + }} + ) + + # set schema - for integrate_new + from openpype import PACKAGE_DIR + # Path to OpenPype's schema + schema_path = os.path.join( + os.path.dirname(PACKAGE_DIR), + "schema" + ) + os.environ["AVALON_SCHEMA"] = schema_path + + import openpype + openpype.install() + os.environ["OPENPYPE_EXECUTABLE"] = sys.executable + from openpype.lib import ApplicationManager + + application_manager = ApplicationManager() + data = { + "last_workfile_path": last_workfile_path, + "start_last_workfile": True, + "project_name": self.PROJECT, + "asset_name": self.ASSET, + "task_name": self.TASK + } + + yield application_manager.launch(self.APP_NAME, **data) + + @pytest.fixture(scope="module") + def publish_finished(self, dbcon, launched_app, download_test_data): + """Dummy fixture waiting for publish to finish""" + import time + time_start = time.time() + while launched_app.poll() is None: + time.sleep(0.5) + if time.time() - time_start > self.TIMEOUT: + raise ValueError("Timeout reached") + + # some clean exit test possible? + print("Publish finished") + yield True + + def test_folder_structure_same(self, dbcon, publish_finished, + download_test_data): + """Check if expected and published subfolders contain same files. + + Compares only presence, not size nor content! + """ + published_dir_base = download_test_data + published_dir = os.path.join(published_dir_base, + self.PROJECT, + self.TASK, + "**") + expected_dir_base = os.path.join(published_dir_base, + "expected") + expected_dir = os.path.join(expected_dir_base, + self.PROJECT, + self.TASK, + "**") + + published = set(f.replace(published_dir_base, '') for f in + glob.glob(published_dir, recursive=True) if + f != published_dir_base and os.path.exists(f)) + expected = set(f.replace(expected_dir_base, '') for f in + glob.glob(expected_dir, recursive=True) if + f != expected_dir_base and os.path.exists(f)) + + not_matched = expected.difference(published) + assert not not_matched, "Missing {} files".format(not_matched) diff --git a/tests/resources/test_data.zip b/tests/resources/test_data.zip new file mode 100644 index 0000000000..0faab86b37 Binary files /dev/null and b/tests/resources/test_data.zip differ diff --git a/tests/igniter/test_bootstrap_repos.py b/tests/unit/igniter/test_bootstrap_repos.py similarity index 100% rename from tests/igniter/test_bootstrap_repos.py rename to tests/unit/igniter/test_bootstrap_repos.py diff --git a/tests/igniter/test_tools.py b/tests/unit/igniter/test_tools.py similarity index 100% rename from tests/igniter/test_tools.py rename to tests/unit/igniter/test_tools.py diff --git a/tests/openpype/lib/test_user_settings.py b/tests/unit/openpype/lib/test_user_settings.py similarity index 100% rename from tests/openpype/lib/test_user_settings.py rename to tests/unit/openpype/lib/test_user_settings.py diff --git a/tests/unit/openpype/modules/sync_server/test_site_operations.py b/tests/unit/openpype/modules/sync_server/test_site_operations.py new file mode 100644 index 0000000000..6a861100a4 --- /dev/null +++ b/tests/unit/openpype/modules/sync_server/test_site_operations.py @@ -0,0 +1,134 @@ +"""Test file for Sync Server, tests site operations add_site, remove_site. + + File: + creates temporary directory and downloads .zip file from GDrive + unzips .zip file + uses content of .zip file (MongoDB's dumps) to import to new databases + with use of 'monkeypatch_session' modifies required env vars + temporarily + runs battery of tests checking that site operation for Sync Server + module are working + removes temporary folder + removes temporary databases (?) +""" +import pytest + +from tests.lib.testing_classes import ModuleUnitTest +from bson.objectid import ObjectId + + +class TestSiteOperation(ModuleUnitTest): + + REPRESENTATION_ID = "60e578d0c987036c6a7b741d" + + TEST_FILES = [("1eCwPljuJeOI8A3aisfOIBKKjcmIycTEt", + "test_site_operations.zip", '')] + + @pytest.fixture(scope="module") + def setup_sync_server_module(self, dbcon): + """Get sync_server_module from ModulesManager""" + from openpype.modules import ModulesManager + + manager = ModulesManager() + sync_server = manager.modules_by_name["sync_server"] + yield sync_server + + @pytest.mark.usefixtures("dbcon") + def test_project_created(self, dbcon): + assert ['test_project'] == dbcon.database.collection_names(False) + + @pytest.mark.usefixtures("dbcon") + def test_objects_imported(self, dbcon): + count_obj = len(list(dbcon.database[self.TEST_PROJECT_NAME].find({}))) + assert 15 == count_obj + + @pytest.mark.usefixtures("setup_sync_server_module") + def test_add_site(self, dbcon, setup_sync_server_module): + """Adds 'test_site', checks that added, + checks that doesn't duplicate.""" + query = { + "_id": ObjectId(self.REPRESENTATION_ID) + } + + ret = dbcon.database[self.TEST_PROJECT_NAME].find(query) + + assert 1 == len(list(ret)), \ + "Single {} must be in DB".format(self.REPRESENTATION_ID) + + setup_sync_server_module.add_site(self.TEST_PROJECT_NAME, + self.REPRESENTATION_ID, + site_name='test_site') + + ret = list(dbcon.database[self.TEST_PROJECT_NAME].find(query)) + + assert 1 == len(ret), \ + "Single {} must be in DB".format(self.REPRESENTATION_ID) + + ret = ret.pop() + site_names = [site["name"] for site in ret["files"][0]["sites"]] + assert 'test_site' in site_names, "Site name wasn't added" + + @pytest.mark.usefixtures("setup_sync_server_module") + def test_add_site_again(self, dbcon, setup_sync_server_module): + """Depends on test_add_site, must throw exception.""" + with pytest.raises(ValueError): + setup_sync_server_module.add_site(self.TEST_PROJECT_NAME, + self.REPRESENTATION_ID, + site_name='test_site') + + @pytest.mark.usefixtures("setup_sync_server_module") + def test_add_site_again_force(self, dbcon, setup_sync_server_module): + """Depends on test_add_site, must not throw exception.""" + setup_sync_server_module.add_site(self.TEST_PROJECT_NAME, + self.REPRESENTATION_ID, + site_name='test_site', force=True) + + query = { + "_id": ObjectId(self.REPRESENTATION_ID) + } + + ret = list(dbcon.database[self.TEST_PROJECT_NAME].find(query)) + + assert 1 == len(ret), \ + "Single {} must be in DB".format(self.REPRESENTATION_ID) + + @pytest.mark.usefixtures("setup_sync_server_module") + def test_remove_site(self, dbcon, setup_sync_server_module): + """Depends on test_add_site, must remove 'test_site'.""" + setup_sync_server_module.remove_site(self.TEST_PROJECT_NAME, + self.REPRESENTATION_ID, + site_name='test_site') + + query = { + "_id": ObjectId(self.REPRESENTATION_ID) + } + + ret = list(dbcon.database[self.TEST_PROJECT_NAME].find(query)) + + assert 1 == len(ret), \ + "Single {} must be in DB".format(self.REPRESENTATION_ID) + + ret = ret.pop() + site_names = [site["name"] for site in ret["files"][0]["sites"]] + + assert 'test_site' not in site_names, "Site name wasn't removed" + + @pytest.mark.usefixtures("setup_sync_server_module") + def test_remove_site_again(self, dbcon, setup_sync_server_module): + """Depends on test_add_site, must trow exception""" + with pytest.raises(ValueError): + setup_sync_server_module.remove_site(self.TEST_PROJECT_NAME, + self.REPRESENTATION_ID, + site_name='test_site') + + query = { + "_id": ObjectId(self.REPRESENTATION_ID) + } + + ret = list(dbcon.database[self.TEST_PROJECT_NAME].find(query)) + + assert 1 == len(ret), \ + "Single {} must be in DB".format(self.REPRESENTATION_ID) + + +test_case = TestSiteOperation() diff --git a/tools/build.sh b/tools/build.sh index c44e7157af..bc79f03db7 100755 --- a/tools/build.sh +++ b/tools/build.sh @@ -58,7 +58,7 @@ BICyan='\033[1;96m' # Cyan BIWhite='\033[1;97m' # White args=$@ -disable_submodule_update = 0 +disable_submodule_update=0 while :; do case $1 in --no-submodule-update) @@ -90,6 +90,7 @@ done ############################################################################### detect_python () { echo -e "${BIGreen}>>>${RST} Using python \c" + command -v python >/dev/null 2>&1 || { echo -e "${BIRed}- NOT FOUND${RST} ${BIYellow}You need Python 3.7 installed to continue.${RST}"; return 1; } local version_command version_command="import sys;print('{0}.{1}'.format(sys.version_info[0], sys.version_info[1]))" local python_version @@ -122,7 +123,7 @@ clean_pyc () { local path path=$openpype_root echo -e "${BIGreen}>>>${RST} Cleaning pyc at [ ${BIWhite}$path${RST} ] ... \c" - find "$path" -path ./build -prune -o -regex '^.*\(__pycache__\|\.py[co]\)$' -delete + find "$path" -path ./build -o -regex '^.*\(__pycache__\|\.py[co]\)$' -delete echo -e "${BIGreen}DONE${RST}" } @@ -173,7 +174,7 @@ main () { else echo -e "${BIYellow}NOT FOUND${RST}" echo -e "${BIYellow}***${RST} We need to install Poetry and virtual env ..." - . "$openpype_root/tools/create_env.sh" || { echo -e "${BIRed}!!!${RST} Poetry installation failed"; return; } + . "$openpype_root/tools/create_env.sh" || { echo -e "${BIRed}!!!${RST} Poetry installation failed"; return 1; } fi if [ "$disable_submodule_update" == 1 ]; then @@ -184,9 +185,9 @@ if [ "$disable_submodule_update" == 1 ]; then fi echo -e "${BIGreen}>>>${RST} Building ..." if [[ "$OSTYPE" == "linux-gnu"* ]]; then - "$POETRY_HOME/bin/poetry" run python "$openpype_root/setup.py" build > "$openpype_root/build/build.log" || { echo -e "${BIRed}!!!${RST} Build failed, see the build log."; return; } + "$POETRY_HOME/bin/poetry" run python "$openpype_root/setup.py" build &> "$openpype_root/build/build.log" || { echo -e "${BIRed}!!!${RST} Build failed, see the build log."; return 1; } elif [[ "$OSTYPE" == "darwin"* ]]; then - "$POETRY_HOME/bin/poetry" run python "$openpype_root/setup.py" bdist_mac > "$openpype_root/build/build.log" || { echo -e "${BIRed}!!!${RST} Build failed, see the build log."; return; } + "$POETRY_HOME/bin/poetry" run python "$openpype_root/setup.py" bdist_mac &> "$openpype_root/build/build.log" || { echo -e "${BIRed}!!!${RST} Build failed, see the build log."; return 1; } fi "$POETRY_HOME/bin/poetry" run python "$openpype_root/tools/build_dependencies.py" @@ -210,4 +211,6 @@ if [ "$disable_submodule_update" == 1 ]; then echo -e "${BIWhite}$openpype_root/build${RST} directory." } -main +return_code=0 +main || return_code=$? +exit $return_code diff --git a/tools/build_dependencies.py b/tools/build_dependencies.py index e5a430e220..8eaf9d87e3 100644 --- a/tools/build_dependencies.py +++ b/tools/build_dependencies.py @@ -23,6 +23,7 @@ import sys import site from distutils.util import get_platform import platform +import subprocess from pathlib import Path import shutil import blessed @@ -93,18 +94,18 @@ _print("Getting venv site-packages ...") assert site_pkg, "No venv site-packages are found." _print(f"Working with: {site_pkg}", 2) - -build_dir = "exe.{}-{}".format(get_platform(), sys.version[0:3]) +openpype_root = Path(os.path.dirname(__file__)).parent # create full path if platform.system().lower() == "darwin": - build_dir = Path(os.path.dirname(__file__)).parent.joinpath( + build_dir = openpype_root.joinpath( "build", "OpenPype.app", "Contents", "MacOS") else: - build_dir = Path(os.path.dirname(__file__)).parent / "build" / build_dir + build_subdir = "exe.{}-{}".format(get_platform(), sys.version[0:3]) + build_dir = openpype_root / "build" / build_subdir _print(f"Using build at {build_dir}", 2) if not build_dir.exists(): @@ -112,7 +113,28 @@ if not build_dir.exists(): _print("Probably freezing of code failed. Check ./build/build.log", 3) sys.exit(1) + +def _progress(_base, _names): + progress_bar.update() + return [] + + deps_dir = build_dir / "dependencies" +vendor_dir = build_dir / "vendor" +vendor_src = openpype_root / "vendor" + +# copy vendor files +_print("Copying vendor files ...") + +total_files = count_folders(vendor_src) +progress_bar = enlighten.Counter( + total=total_files, desc="Copying vendor files ...", + units="%", color=(64, 128, 222)) + +shutil.copytree(vendor_src.as_posix(), + vendor_dir.as_posix(), + ignore=_progress) +progress_bar.close() # copy all files _print("Copying dependencies ...") @@ -122,12 +144,6 @@ progress_bar = enlighten.Counter( total=total_files, desc="Processing Dependencies", units="%", color=(53, 178, 202)) - -def _progress(_base, _names): - progress_bar.update() - return [] - - shutil.copytree(site_pkg.as_posix(), deps_dir.as_posix(), ignore=_progress) @@ -135,15 +151,29 @@ progress_bar.close() # iterate over frozen libs and create list to delete libs_dir = build_dir / "lib" -# On Windows "python3.dll" is needed for PyQt5 from the build. -if platform.system().lower() == "windows": - src = Path(libs_dir / "PyQt5" / "python3.dll") - dst = Path(deps_dir / "PyQt5" / "python3.dll") - if src.exists(): - shutil.copyfile(src, dst) - else: - _print("Could not find {}".format(src), 1) - sys.exit(1) + +# On Linux use rpath from source libraries in destination libraries +if platform.system().lower() == "linux": + src_pyside_dir = openpype_root / "vendor" / "python" / "PySide2" + dst_pyside_dir = build_dir / "vendor" / "python" / "PySide2" + src_rpath_per_so_file = {} + for filepath in src_pyside_dir.glob("*.so"): + filename = filepath.name + rpath = ( + subprocess.check_output(["patchelf", "--print-rpath", filepath]) + .decode("utf-8") + .strip() + ) + src_rpath_per_so_file[filename] = rpath + + for filepath in dst_pyside_dir.glob("*.so"): + filename = filepath.name + if filename not in src_rpath_per_so_file: + continue + src_rpath = src_rpath_per_so_file[filename] + subprocess.check_call( + ["patchelf", "--set-rpath", src_rpath, filepath] + ) to_delete = [] # _print("Finding duplicates ...") diff --git a/tools/create_env.sh b/tools/create_env.sh index cc9eddc317..917ddc36ba 100755 --- a/tools/create_env.sh +++ b/tools/create_env.sh @@ -88,6 +88,7 @@ done ############################################################################### detect_python () { echo -e "${BIGreen}>>>${RST} Using python \c" + command -v python >/dev/null 2>&1 || { echo -e "${BIRed}- NOT FOUND${RST} ${BIYellow}You need Python 3.7 installed to continue.${RST}"; return 1; } local version_command="import sys;print('{0}.{1}'.format(sys.version_info[0], sys.version_info[1]))" local python_version="$(python <<< ${version_command})" oIFS="$IFS" @@ -125,7 +126,7 @@ clean_pyc () { local path path=$openpype_root echo -e "${BIGreen}>>>${RST} Cleaning pyc at [ ${BIWhite}$path${RST} ] ... \c" - find "$path" -path ./build -prune -o -regex '^.*\(__pycache__\|\.py[co]\)$' -delete + find "$path" -path ./build -o -regex '^.*\(__pycache__\|\.py[co]\)$' -delete echo -e "${BIGreen}DONE${RST}" } @@ -166,7 +167,7 @@ main () { echo -e "${BIGreen}OK${RST}" else echo -e "${BIYellow}NOT FOUND${RST}" - install_poetry || { echo -e "${BIRed}!!!${RST} Poetry installation failed"; return; } + install_poetry || { echo -e "${BIRed}!!!${RST} Poetry installation failed"; return 1; } fi if [ -f "$openpype_root/poetry.lock" ]; then @@ -175,7 +176,11 @@ main () { echo -e "${BIGreen}>>>${RST} Installing dependencies ..." fi - "$POETRY_HOME/bin/poetry" install --no-root $poetry_verbosity || { echo -e "${BIRed}!!!${RST} Poetry environment installation failed"; return; } + "$POETRY_HOME/bin/poetry" install --no-root $poetry_verbosity || { echo -e "${BIRed}!!!${RST} Poetry environment installation failed"; return 1; } + if [ $? -ne 0 ] ; then + echo -e "${BIRed}!!!${RST} Virtual environment creation failed." + return 1 + fi echo -e "${BIGreen}>>>${RST} Cleaning cache files ..." clean_pyc @@ -183,11 +188,15 @@ main () { # reinstall these because of bug in poetry? or cx_freeze? # cx_freeze will crash on missing __pychache__ on these but # reinstalling them solves the problem. - echo -e "${BIGreen}>>>${RST} Fixing pycache bug ..." - "$POETRY_HOME/bin/poetry" run python -m pip install --force-reinstall pip - "$POETRY_HOME/bin/poetry" run pip install --force-reinstall setuptools - "$POETRY_HOME/bin/poetry" run pip install --force-reinstall wheel - "$POETRY_HOME/bin/poetry" run python -m pip install --force-reinstall pip + echo -e "${BIGreen}>>>${RST} Post-venv creation fixes ..." + local openpype_index=$("$POETRY_HOME/bin/poetry" run python "$openpype_root/tools/parse_pyproject.py" tool.poetry.source.0.url) + echo -e "${BIGreen}- ${RST} Using index: ${BIWhite}$openpype_index${RST}" + "$POETRY_HOME/bin/poetry" run pip install setuptools==49.6.0 + "$POETRY_HOME/bin/poetry" run pip install --disable-pip-version-check --force-reinstall wheel + "$POETRY_HOME/bin/poetry" run python -m pip install --disable-pip-version-check --force-reinstall pip + "$POETRY_HOME/bin/poetry" run pip install --disable-pip-version-check --force-reinstall cx_freeze -i $openpype_index --extra-index-url https://pypi.org/simple } -main -3 +return_code=0 +main || return_code=$? +exit $return_code diff --git a/tools/docker_build.sh b/tools/docker_build.sh index 7600fe044b..04c26424eb 100755 --- a/tools/docker_build.sh +++ b/tools/docker_build.sh @@ -20,6 +20,41 @@ realpath () { echo $(cd $(dirname "$1"); pwd)/$(basename "$1") } +create_container () { + if [ ! -f "$openpype_root/build/docker-image.id" ]; then + echo -e "${BIRed}!!!${RST} Docker command failed, cannot find image id." + exit 1 + fi + local id=$(<"$openpype_root/build/docker-image.id") + echo -e "${BIYellow}---${RST} Creating container from $id ..." + cid="$(docker create $id bash)" + if [ $? -ne 0 ] ; then + echo -e "${BIRed}!!!${RST} Cannot create container." + exit 1 + fi +} + +retrieve_build_log () { + create_container + echo -e "${BIYellow}***${RST} Copying build log to ${BIWhite}$openpype_root/build/build.log${RST}" + docker cp "$cid:/opt/openpype/build/build.log" "$openpype_root/build" +} + +openpype_root=$(realpath $(dirname $(dirname "${BASH_SOURCE[0]}"))) + + +if [ -z $1 ]; then + dockerfile="Dockerfile" +else + dockerfile="Dockerfile.$1" + if [ ! -f "$openpype_root/$dockerfile" ]; then + echo -e "${BIRed}!!!${RST} Dockerfile for specifed platform ${BIWhite}$1${RST} doesn't exist." + exit 1 + else + echo -e "${BIGreen}>>>${RST} Using Dockerfile for ${BIWhite}$1${RST} ..." + fi +fi + # Main main () { openpype_root=$(realpath $(dirname $(dirname "${BASH_SOURCE[0]}"))) @@ -28,36 +63,35 @@ main () { echo -e "${BIYellow}---${RST} Cleaning build directory ..." rm -rf "$openpype_root/build" && mkdir "$openpype_root/build" > /dev/null - version_command="import os;exec(open(os.path.join('$openpype_root', 'openpype', 'version.py')).read());print(__version__);" - openpype_version="$(python3 <<< ${version_command})" + local version_command="import os;exec(open(os.path.join('$openpype_root', 'openpype', 'version.py')).read());print(__version__);" + local openpype_version="$(python3 <<< ${version_command})" echo -e "${BIGreen}>>>${RST} Running docker build ..." - docker build --pull --no-cache -t pypeclub/openpype:$openpype_version . + # docker build --pull --no-cache -t pypeclub/openpype:$openpype_version . + docker build --pull --iidfile $openpype_root/build/docker-image.id -t pypeclub/openpype:$openpype_version -f $dockerfile . if [ $? -ne 0 ] ; then + echo $? echo -e "${BIRed}!!!${RST} Docker build failed." + retrieve_build_log return 1 fi echo -e "${BIGreen}>>>${RST} Copying build from container ..." - echo -e "${BIYellow}---${RST} Creating container from pypeclub/openpype:$openpype_version ..." - id="$(docker create -ti pypeclub/openpype:$openpype_version bash)" - if [ $? -ne 0 ] ; then - echo -e "${BIRed}!!!${RST} Cannot create just built container." - return 1 - fi + create_container echo -e "${BIYellow}---${RST} Copying ..." - docker cp "$id:/opt/openpype/build/exe.linux-x86_64-3.7" "$openpype_root/build" + docker cp "$cid:/opt/openpype/build/exe.linux-x86_64-3.7" "$openpype_root/build" + docker cp "$cid:/opt/openpype/build/build.log" "$openpype_root/build" if [ $? -ne 0 ] ; then echo -e "${BIRed}!!!${RST} Copying failed." return 1 fi echo -e "${BIGreen}>>>${RST} Fixing user ownership ..." - username="$(logname)" + local username="$(logname)" chown -R $username ./build echo -e "${BIGreen}>>>${RST} All done, you can delete container:" - echo -e "${BIYellow}$id${RST}" + echo -e "${BIYellow}$cid${RST}" } return_code=0 diff --git a/tools/fetch_thirdparty_libs.py b/tools/fetch_thirdparty_libs.py index 75ee052950..b616beab27 100644 --- a/tools/fetch_thirdparty_libs.py +++ b/tools/fetch_thirdparty_libs.py @@ -20,6 +20,7 @@ import hashlib import tarfile import zipfile import time +import subprocess term = blessed.Terminal() @@ -65,12 +66,40 @@ def _print(msg: str, message_type: int = 0) -> None: print("{}{}".format(header, msg)) - -_print("Processing third-party dependencies ...") start_time = time.time_ns() openpype_root = Path(os.path.dirname(__file__)).parent pyproject = toml.load(openpype_root / "pyproject.toml") +_print("Handling PySide2 Qt framework ...") +pyside2_version = None +try: + pyside2_version = pyproject["openpype"]["pyside2"]["version"] + _print("We'll install PySide2{}".format(pyside2_version)) +except AttributeError: + _print("No PySide2 version was specified, using latest available.", 2) + +pyside2_arg = "PySide2" if not pyside2_version else "PySide2{}".format(pyside2_version) # noqa: E501 +python_vendor_dir = openpype_root / "vendor" / "python" +try: + subprocess.run( + [sys.executable, "-m", "pip", "install", "--upgrade", + pyside2_arg, "-t", str(python_vendor_dir)], + check=True, stdout=subprocess.DEVNULL) +except subprocess.CalledProcessError as e: + _print("Error during PySide2 installation.", 1) + _print(str(e), 1) + sys.exit(1) + +# Remove libraries for QtSql which don't have available libraries +# by default and Postgre library would require to modify rpath of dependency platform_name = platform.system().lower() +if platform_name == "darwin": + pyside2_sqldrivers_dir = ( + python_vendor_dir / "PySide2" / "Qt" / "plugins" / "sqldrivers" + ) + for filepath in pyside2_sqldrivers_dir.iterdir(): + os.remove(str(filepath)) + +_print("Processing third-party dependencies ...") try: thirdparty = pyproject["openpype"]["thirdparty"] diff --git a/tools/fetch_thirdparty_libs.sh b/tools/fetch_thirdparty_libs.sh index 93d0674965..940af76ed6 100755 --- a/tools/fetch_thirdparty_libs.sh +++ b/tools/fetch_thirdparty_libs.sh @@ -98,4 +98,4 @@ main () { "$POETRY_HOME/bin/poetry" run python "$openpype_root/tools/fetch_thirdparty_libs.py" } -main \ No newline at end of file +main diff --git a/tools/parse_pyproject.py b/tools/parse_pyproject.py new file mode 100644 index 0000000000..dacecd88d0 --- /dev/null +++ b/tools/parse_pyproject.py @@ -0,0 +1,44 @@ +# -*- coding: utf-8 -*- +"""Parse pyproject.toml and return its values. + +Useful for shell scripts to know more about OpenPype build. +""" +import sys +import os +import toml +from pathlib import Path +import click + + +@click.command() +@click.argument("keys", nargs=-1, type=click.STRING) +def main(keys): + """Get values from `pyproject.toml`. + + You can specify dot separated keys from `pyproject.toml` + as arguments and this script will return them on separate + lines. If key doesn't exists, None is returned. + + """ + openpype_root = Path(os.path.dirname(__file__)).parent + py_project = toml.load(openpype_root / "pyproject.toml") + for q in keys: + query = q.split(".") + data = py_project + + for k in query: + if isinstance(data, list): + try: + data = data[int(k)] + except IndexError: + print("None") + sys.exit() + continue + + if isinstance(data, dict): + data = data.get(k) + print(data) + + +if __name__ == "__main__": + main() diff --git a/website/docs/dev_build.md b/website/docs/dev_build.md index b3e0c24fc2..bf606ae7c0 100644 --- a/website/docs/dev_build.md +++ b/website/docs/dev_build.md @@ -19,6 +19,7 @@ We use [CX_Freeze](https://cx-freeze.readthedocs.io/en/latest) to freeze the cod This is outline of build steps. Most of them are done automatically via scripts: - Virtual environment is created using **Poetry** in `.venv` +- Necessary python modules outside of `.venv` are stored to `./vendor/python` (like `PySide2`) - Necessary third-party tools (like [ffmpeg](https://www.ffmpeg.org/), [OpenImageIO](https://github.com/OpenImageIO/oiio) and [usd libraries](https://developer.nvidia.com/usd)) are downloaded to `./vendor/bin` - OpenPype code is frozen with **cx_freeze** to `./build` @@ -55,14 +56,14 @@ For development purposes it is possible to run OpenPype directly from the source To start OpenPype from source you need to 1. Run `.\tools\create_env.ps1` to create virtual environment in `.venv` -2. Run `.\tools\fetch_thirdparty_libs.ps1` to get **ffmpeg**, **oiio** and other tools needed. +2. Run `.\tools\fetch_thirdparty_libs.ps1` to get **PySide2**, **ffmpeg**, **oiio** and other tools needed. 3. Run `.\tools\run_tray.ps1` if you have all required dependencies on your machine you should be greeted with OpenPype igniter window and once you give it your Mongo URL, with OpenPype icon in the system tray. Step 1 and 2 needs to be run only once (or when something was changed). #### To build OpenPype: 1. Run `.\tools\create_env.ps1` to create virtual environment in `.venv` -2. Run `.\tools\fetch_thirdparty_libs.ps1` to get **ffmpeg**, **oiio** and other tools needed. +2. Run `.\tools\fetch_thirdparty_libs.ps1` to get **PySide2**, **ffmpeg**, **oiio** and other tools needed. 3. `.\tools\build.ps1` to build OpenPype to `.\build` @@ -84,6 +85,13 @@ You can use Docker to build OpenPype. Just run: ```shell $ sudo ./tools/docker_build.sh ``` + +This will by default use Debian as base image. If you need to make Centos 7 compatible build, please run: + +```sh +sudo ./tools/docker_build.sh centos7 +``` + and you should have built OpenPype in `build` directory. It is using **Centos 7** as a base image. @@ -178,7 +186,7 @@ For more information about setting your build environment please refer to [pyenv #### To build Pype: 1. Run `./tools/create_env.sh` to create virtual environment in `./venv` -2. Run `./tools/fetch_thirdparty_libs.sh` to get **ffmpeg**, **oiio** and other tools needed. +2. Run `./tools/fetch_thirdparty_libs.sh` to get **PySide2**, **ffmpeg**, **oiio** and other tools needed. 3. Run `./tools/build.sh` to build pype executables in `.\build\` @@ -266,6 +274,19 @@ pywin32 = { version = "300", markers = "sys_platform == 'win32'" } For more information see [Poetry documentation](https://python-poetry.org/docs/dependency-specification/). +### Python modules as thirdparty +There are some python modules that can be available only in OpenPype and should not be propagated to any subprocess. +Best example is **PySide2** which is required to run OpenPype but can be used only in OpenPype and should not be in PYTHONPATH for most of host applications. +We've decided to separate these breaking dependencies to be able run OpenPype from code and from build the same way. + +:::warning +**PySide2** has handled special cases related to it's build process. +### Linux +- We're fixing rpath of shared objects on linux which is modified during cx freeze processing. +### MacOS +- **QtSql** libraries are removed on MacOS because their dependencies are not available and would require to modify rpath of Postgre library. +::: + ### Binary dependencies To add some binary tool or something that doesn't fit standard Python distribution methods, you can use [fetch_thirdparty_libs](#fetch_thirdparty_libs) script. It will take things defined in @@ -323,14 +344,18 @@ Same as: poetry run python ./tools/create_zip.py ``` -### docker_build.sh +### docker_build.sh *[variant]* Script to build OpenPype on [Docker](https://www.docker.com/) enabled systems - usually Linux and Windows with [Docker Desktop](https://docs.docker.com/docker-for-windows/install/) and [Windows Subsystem for Linux](https://docs.microsoft.com/en-us/windows/wsl/about) (WSL) installed. It must be run with administrative privileges - `sudo ./docker_build.sh`. -It will use **Centos 7** base image to build OpenPype. You'll see your build in `./build` folder. +It will use latest **Debian** base image to build OpenPype. If you need to build OpenPype for +older systems like Centos 7, use `centos7` as argument. This will use another Dockerfile to build +OpenPype with **Centos 7** as base image. + +You'll see your build in `./build` folder. ### fetch_thirdparty_libs This script will download necessary tools for OpenPype defined in `pyproject.toml` like FFMpeg, diff --git a/website/docs/pype2/admin_presets_plugins.md b/website/docs/pype2/admin_presets_plugins.md index 797995d2b7..9c838d4a64 100644 --- a/website/docs/pype2/admin_presets_plugins.md +++ b/website/docs/pype2/admin_presets_plugins.md @@ -469,6 +469,7 @@ maya outliner colours for various families "camera": [0.447, 0.312, 1.0], "fbx": [1.0, 0.931, 0.312], "mayaAscii": [0.312, 1.0, 0.747], + "mayaScene": [0.312, 1.0, 0.747], "setdress": [0.312, 1.0, 0.747], "layout": [0.312, 1.0, 0.747], "vdbcache": [0.312, 1.0, 0.428],