Merge branch 'develop' of https://github.com/pypeclub/OpenPype into feature/multiverse

This commit is contained in:
Bo Zhou 2022-03-11 21:25:08 +09:00
commit 3f8103f758
13 changed files with 362 additions and 89 deletions

View file

@ -1,6 +1,6 @@
# Changelog
## [3.9.0-nightly.7](https://github.com/pypeclub/OpenPype/tree/HEAD)
## [3.9.0-nightly.8](https://github.com/pypeclub/OpenPype/tree/HEAD)
[Full Changelog](https://github.com/pypeclub/OpenPype/compare/3.8.2...HEAD)
@ -9,15 +9,13 @@
- AssetCreator: Remove the tool [\#2845](https://github.com/pypeclub/OpenPype/pull/2845)
- Houdini: Remove unused code [\#2779](https://github.com/pypeclub/OpenPype/pull/2779)
### 📖 Documentation
- Documentation: fixed broken links [\#2799](https://github.com/pypeclub/OpenPype/pull/2799)
- Documentation: broken link fix [\#2785](https://github.com/pypeclub/OpenPype/pull/2785)
- Various testing updates [\#2726](https://github.com/pypeclub/OpenPype/pull/2726)
**🚀 Enhancements**
- NewPublisher: Descriptions and Icons in creator dialog [\#2867](https://github.com/pypeclub/OpenPype/pull/2867)
- NewPublisher: Changing task on publishing instance [\#2863](https://github.com/pypeclub/OpenPype/pull/2863)
- TrayPublisher: Choose project widget is more clear [\#2859](https://github.com/pypeclub/OpenPype/pull/2859)
- New: Validation exceptions [\#2841](https://github.com/pypeclub/OpenPype/pull/2841)
- Maya: add loaded containers to published instance [\#2837](https://github.com/pypeclub/OpenPype/pull/2837)
- Ftrack: Can sync fps as string [\#2836](https://github.com/pypeclub/OpenPype/pull/2836)
- General: Custom function for find executable [\#2822](https://github.com/pypeclub/OpenPype/pull/2822)
- General: Color dialog UI fixes [\#2817](https://github.com/pypeclub/OpenPype/pull/2817)
@ -25,11 +23,16 @@
- Nuke: adding Reformat to baking mov plugin [\#2811](https://github.com/pypeclub/OpenPype/pull/2811)
- Manager: Update all to latest button [\#2805](https://github.com/pypeclub/OpenPype/pull/2805)
- General: Set context environments for non host applications [\#2803](https://github.com/pypeclub/OpenPype/pull/2803)
- Tray publisher: New Tray Publisher host \(beta\) [\#2778](https://github.com/pypeclub/OpenPype/pull/2778)
- Flame: use Shot Name on segment for asset name [\#2751](https://github.com/pypeclub/OpenPype/pull/2751)
**🐛 Bug fixes**
- Deadline: Fix plugin name for tile assemble [\#2868](https://github.com/pypeclub/OpenPype/pull/2868)
- General: Fix hardlink for windows [\#2864](https://github.com/pypeclub/OpenPype/pull/2864)
- General: ffmpeg was crashing on slate merge [\#2860](https://github.com/pypeclub/OpenPype/pull/2860)
- WebPublisher: Video file was published with one too many frame [\#2858](https://github.com/pypeclub/OpenPype/pull/2858)
- New Publisher: Error dialog got right styles [\#2857](https://github.com/pypeclub/OpenPype/pull/2857)
- General: Fix getattr clalback on dynamic modules [\#2855](https://github.com/pypeclub/OpenPype/pull/2855)
- Nuke: slate resolution to input video resolution [\#2853](https://github.com/pypeclub/OpenPype/pull/2853)
- WebPublisher: Fix username stored in DB [\#2852](https://github.com/pypeclub/OpenPype/pull/2852)
- WebPublisher: Fix wrong number of frames for video file [\#2851](https://github.com/pypeclub/OpenPype/pull/2851)
- Nuke: fix multiple baking profile farm publishing [\#2842](https://github.com/pypeclub/OpenPype/pull/2842)
@ -42,26 +45,25 @@
- General: Host name was formed from obsolete code [\#2821](https://github.com/pypeclub/OpenPype/pull/2821)
- Settings UI: Fix "Apply from" action [\#2820](https://github.com/pypeclub/OpenPype/pull/2820)
- Ftrack: Job killer with missing user [\#2819](https://github.com/pypeclub/OpenPype/pull/2819)
- Nuke: Use AVALON\_APP to get value for "app" key [\#2818](https://github.com/pypeclub/OpenPype/pull/2818)
- StandalonePublisher: use dynamic groups in subset names [\#2816](https://github.com/pypeclub/OpenPype/pull/2816)
- Settings UI: Search case sensitivity [\#2810](https://github.com/pypeclub/OpenPype/pull/2810)
- Flame Babypublisher optimalization [\#2806](https://github.com/pypeclub/OpenPype/pull/2806)
- resolve: fixing fusion module loading [\#2802](https://github.com/pypeclub/OpenPype/pull/2802)
- Ftrack: Unset task ids from asset versions before tasks are removed [\#2800](https://github.com/pypeclub/OpenPype/pull/2800)
- Slack: fail gracefully if slack exception [\#2798](https://github.com/pypeclub/OpenPype/pull/2798)
- Flame: Fix version string in default settings [\#2783](https://github.com/pypeclub/OpenPype/pull/2783)
**Merged pull requests:**
**🔀 Refactored code**
- General: Move create logic from avalon to OpenPype [\#2854](https://github.com/pypeclub/OpenPype/pull/2854)
- General: Add vendors from avalon [\#2848](https://github.com/pypeclub/OpenPype/pull/2848)
- General: Move change context functions [\#2839](https://github.com/pypeclub/OpenPype/pull/2839)
- Tools: Don't use avalon tools code [\#2829](https://github.com/pypeclub/OpenPype/pull/2829)
- Move Unreal Implementation to OpenPype [\#2823](https://github.com/pypeclub/OpenPype/pull/2823)
- Nuke: Use AVALON\_APP to get value for "app" key [\#2818](https://github.com/pypeclub/OpenPype/pull/2818)
- Ftrack: Moved module one hierarchy level higher [\#2792](https://github.com/pypeclub/OpenPype/pull/2792)
- SyncServer: Moved module one hierarchy level higher [\#2791](https://github.com/pypeclub/OpenPype/pull/2791)
- Royal render: Move module one hierarchy level higher [\#2790](https://github.com/pypeclub/OpenPype/pull/2790)
- Deadline: Move module one hierarchy level higher [\#2789](https://github.com/pypeclub/OpenPype/pull/2789)
- General: Extract template formatting from anatomy [\#2766](https://github.com/pypeclub/OpenPype/pull/2766)
**Merged pull requests:**
- Nuke: gizmo precollect fix [\#2866](https://github.com/pypeclub/OpenPype/pull/2866)
- Nuke: Fix family test in validate\_write\_legacy to work with stillImage [\#2847](https://github.com/pypeclub/OpenPype/pull/2847)
## [3.8.2](https://github.com/pypeclub/OpenPype/tree/3.8.2) (2022-02-07)
[Full Changelog](https://github.com/pypeclub/OpenPype/compare/CI/3.8.2-nightly.3...3.8.2)

View file

@ -5,6 +5,7 @@ from maya import cmds
import qargparse
from avalon import api
from avalon.pipeline import AVALON_CONTAINER_ID
from openpype.pipeline import LegacyCreator
from .pipeline import containerise
@ -169,16 +170,18 @@ class ReferenceLoader(Loader):
return
ref_node = get_reference_node(nodes, self.log)
loaded_containers.append(containerise(
container = containerise(
name=name,
namespace=namespace,
nodes=[ref_node],
context=context,
loader=self.__class__.__name__
))
)
loaded_containers.append(container)
self._organize_containers([ref_node], container)
c += 1
namespace = None
return loaded_containers
def process_reference(self, context, name, namespace, data):
@ -311,3 +314,13 @@ class ReferenceLoader(Loader):
deleteNamespaceContent=True)
except RuntimeError:
pass
@staticmethod
def _organize_containers(nodes, container):
# type: (list, str) -> None
for node in nodes:
id_attr = "{}.id".format(node)
if not cmds.attributeQuery("id", node=node, exists=True):
continue
if cmds.getAttr(id_attr) == AVALON_CONTAINER_ID:
cmds.sets(node, forceElement=container)

View file

@ -1,6 +1,7 @@
import os
from maya import cmds
from avalon import api
from openpype.api import get_project_settings
from openpype.lib import get_creator_by_name
from openpype.pipeline import legacy_create
@ -126,6 +127,12 @@ class ReferenceLoader(openpype.hosts.maya.api.plugin.ReferenceLoader):
return new_nodes
def load(self, context, name=None, namespace=None, options=None):
container = super(ReferenceLoader, self).load(
context, name, namespace, options)
# clean containers if present to AVALON_CONTAINERS
self._organize_containers(self[:], container[0])
def switch(self, container, representation):
self.update(container, representation)

View file

@ -6,6 +6,7 @@ from maya import cmds
import openpype.api
from openpype.hosts.maya.api.lib import maintained_selection
from avalon.pipeline import AVALON_CONTAINER_ID
class ExtractMayaSceneRaw(openpype.api.Extractor):
@ -57,10 +58,21 @@ class ExtractMayaSceneRaw(openpype.api.Extractor):
else:
members = instance[:]
loaded_containers = None
if set(self.add_for_families).intersection(
set(instance.data.get("families")),
set(instance.data.get("family").lower())):
loaded_containers = self._add_loaded_containers(members)
selection = members
if loaded_containers:
self.log.info(loaded_containers)
selection += loaded_containers
# Perform extraction
self.log.info("Performing extraction ...")
with maintained_selection():
cmds.select(members, noExpand=True)
cmds.select(selection, noExpand=True)
cmds.file(path,
force=True,
typ="mayaAscii" if self.scene_type == "ma" else "mayaBinary", # noqa: E501
@ -83,3 +95,33 @@ class ExtractMayaSceneRaw(openpype.api.Extractor):
instance.data["representations"].append(representation)
self.log.info("Extracted instance '%s' to: %s" % (instance.name, path))
@staticmethod
def _add_loaded_containers(members):
# type: (list) -> list
refs_to_include = [
cmds.referenceQuery(ref, referenceNode=True)
for ref in members
if cmds.referenceQuery(ref, isNodeReferenced=True)
]
refs_to_include = set(refs_to_include)
obj_sets = cmds.ls("*.id", long=True, type="objectSet", recursive=True,
objectsOnly=True)
loaded_containers = []
for obj_set in obj_sets:
if not cmds.attributeQuery("id", node=obj_set, exists=True):
continue
id_attr = "{}.id".format(obj_set)
if cmds.getAttr(id_attr) != AVALON_CONTAINER_ID:
continue
set_content = set(cmds.sets(obj_set, query=True))
if set_content.intersection(refs_to_include):
loaded_containers.append(obj_set)
return loaded_containers

View file

@ -105,17 +105,18 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
task_dir, task_data["files"], tags
)
file_url = os.path.join(task_dir, task_data["files"][0])
duration = self._get_duration(file_url)
if duration:
no_of_frames = self._get_number_of_frames(file_url)
if no_of_frames:
try:
frame_end = int(frame_start) + math.ceil(duration)
instance.data["frameEnd"] = math.ceil(frame_end)
frame_end = int(frame_start) + math.ceil(no_of_frames)
instance.data["frameEnd"] = math.ceil(frame_end) - 1
self.log.debug("frameEnd:: {}".format(
instance.data["frameEnd"]))
except ValueError:
self.log.warning("Unable to count frames "
"duration {}".format(duration))
"duration {}".format(no_of_frames))
# raise ValueError("STOP")
instance.data["handleStart"] = asset_doc["data"]["handleStart"]
instance.data["handleEnd"] = asset_doc["data"]["handleEnd"]
@ -261,7 +262,7 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
else:
return 0
def _get_duration(self, file_url):
def _get_number_of_frames(self, file_url):
"""Return duration in frames"""
try:
streams = ffprobe_streams(file_url, self.log)
@ -288,7 +289,7 @@ class CollectPublishedFiles(pyblish.api.ContextPlugin):
duration = stream.get("duration")
frame_rate = get_fps(stream.get("r_frame_rate", '0/0'))
self.log.debu("duration:: {} frame_rate:: {}".format(
self.log.debug("duration:: {} frame_rate:: {}".format(
duration, frame_rate))
try:
return float(duration) * float(frame_rate)

View file

@ -90,7 +90,7 @@ class RationalToInt:
if len(parts) != 1:
bottom = float(parts[1])
self._value = top / bottom
self._value = float(top) / float(bottom)
self._string_value = string_value
@property
@ -170,6 +170,23 @@ def convert_value_by_type_name(value_type, value, logger=None):
if value_type == "rational2i":
return RationalToInt(value)
if value_type == "vector":
parts = [part.strip() for part in value.split(",")]
output = []
for part in parts:
if part == "-nan":
output.append(None)
continue
try:
part = float(part)
except ValueError:
pass
output.append(part)
return output
if value_type == "timecode":
return value
# Array of other types is converted to list
re_result = ARRAY_TYPE_REGEX.findall(value_type)
if re_result:

View file

@ -5,8 +5,9 @@ Todo:
Currently we support only EXRs with their data window set.
"""
import os
import re
import subprocess
from xml.dom import minidom
import xml.etree.ElementTree
from System.IO import Path
@ -15,17 +16,220 @@ from Deadline.Scripting import (
FileUtils, RepositoryUtils, SystemUtils)
INT_KEYS = {
"x", "y", "height", "width", "full_x", "full_y",
"full_width", "full_height", "full_depth", "full_z",
"tile_width", "tile_height", "tile_depth", "deep", "depth",
"nchannels", "z_channel", "alpha_channel", "subimages"
STRING_TAGS = {
"format"
}
LIST_KEYS = {
"channelnames"
INT_TAGS = {
"x", "y", "z",
"width", "height", "depth",
"full_x", "full_y", "full_z",
"full_width", "full_height", "full_depth",
"tile_width", "tile_height", "tile_depth",
"nchannels",
"alpha_channel",
"z_channel",
"deep",
"subimages",
}
XML_CHAR_REF_REGEX_HEX = re.compile(r"&#x?[0-9a-fA-F]+;")
# Regex to parse array attributes
ARRAY_TYPE_REGEX = re.compile(r"^(int|float|string)\[\d+\]$")
def convert_value_by_type_name(value_type, value):
"""Convert value to proper type based on type name.
In some cases value types have custom python class.
"""
# Simple types
if value_type == "string":
return value
if value_type == "int":
return int(value)
if value_type == "float":
return float(value)
# Vectors will probably have more types
if value_type == "vec2f":
return [float(item) for item in value.split(",")]
# Matrix should be always have square size of element 3x3, 4x4
# - are returned as list of lists
if value_type == "matrix":
output = []
current_index = -1
parts = value.split(",")
parts_len = len(parts)
if parts_len == 1:
divisor = 1
elif parts_len == 4:
divisor = 2
elif parts_len == 9:
divisor == 3
elif parts_len == 16:
divisor = 4
else:
print("Unknown matrix resolution {}. Value: \"{}\"".format(
parts_len, value
))
for part in parts:
output.append(float(part))
return output
for idx, item in enumerate(parts):
list_index = idx % divisor
if list_index > current_index:
current_index = list_index
output.append([])
output[list_index].append(float(item))
return output
if value_type == "rational2i":
parts = value.split("/")
top = float(parts[0])
bottom = 1.0
if len(parts) != 1:
bottom = float(parts[1])
return float(top) / float(bottom)
if value_type == "vector":
parts = [part.strip() for part in value.split(",")]
output = []
for part in parts:
if part == "-nan":
output.append(None)
continue
try:
part = float(part)
except ValueError:
pass
output.append(part)
return output
if value_type == "timecode":
return value
# Array of other types is converted to list
re_result = ARRAY_TYPE_REGEX.findall(value_type)
if re_result:
array_type = re_result[0]
output = []
for item in value.split(","):
output.append(
convert_value_by_type_name(array_type, item)
)
return output
print((
"MISSING IMPLEMENTATION:"
" Unknown attrib type \"{}\". Value: {}"
).format(value_type, value))
return value
def parse_oiio_xml_output(xml_string):
"""Parse xml output from OIIO info command."""
output = {}
if not xml_string:
return output
# Fix values with ampresand (lazy fix)
# - oiiotool exports invalid xml which ElementTree can't handle
# e.g. ""
# WARNING: this will affect even valid character entities. If you need
# those values correctly, this must take care of valid character ranges.
# See https://github.com/pypeclub/OpenPype/pull/2729
matches = XML_CHAR_REF_REGEX_HEX.findall(xml_string)
for match in matches:
new_value = match.replace("&", "&")
xml_string = xml_string.replace(match, new_value)
tree = xml.etree.ElementTree.fromstring(xml_string)
attribs = {}
output["attribs"] = attribs
for child in tree:
tag_name = child.tag
if tag_name == "attrib":
attrib_def = child.attrib
value = convert_value_by_type_name(
attrib_def["type"], child.text
)
attribs[attrib_def["name"]] = value
continue
# Channels are stored as tex on each child
if tag_name == "channelnames":
value = []
for channel in child:
value.append(channel.text)
# Convert known integer type tags to int
elif tag_name in INT_TAGS:
value = int(child.text)
# Keep value of known string tags
elif tag_name in STRING_TAGS:
value = child.text
# Keep value as text for unknown tags
# - feel free to add more tags
else:
value = child.text
print((
"MISSING IMPLEMENTATION:"
" Unknown tag \"{}\". Value \"{}\""
).format(tag_name, value))
output[child.tag] = value
return output
def info_about_input(oiiotool_path, filepath):
args = [
oiiotool_path,
"--info",
"-v",
"-i:infoformat=xml",
filepath
]
popen = subprocess.Popen(args, stdout=subprocess.PIPE)
_stdout, _stderr = popen.communicate()
output = ""
if _stdout:
output += _stdout.decode("utf-8")
if _stderr:
output += _stderr.decode("utf-8")
output = output.replace("\r\n", "\n")
xml_started = False
lines = []
for line in output.split("\n"):
if not xml_started:
if not line.startswith("<"):
continue
xml_started = True
if xml_started:
lines.append(line)
if not xml_started:
raise ValueError(
"Failed to read input file \"{}\".\nOutput:\n{}".format(
filepath, output
)
)
xml_text = "\n".join(lines)
return parse_oiio_xml_output(xml_text)
def GetDeadlinePlugin(): # noqa: N802
"""Helper."""
return OpenPypeTileAssembler()
@ -218,8 +422,9 @@ class OpenPypeTileAssembler(DeadlinePlugin):
# Create new image with output resolution, and with same type and
# channels as input
oiiotool_path = self.render_executable()
first_tile_path = tile_info[0]["filepath"]
first_tile_info = self.info_about_input(first_tile_path)
first_tile_info = info_about_input(oiiotool_path, first_tile_path)
create_arg_template = "--create{} {}x{} {}"
image_type = ""
@ -236,7 +441,7 @@ class OpenPypeTileAssembler(DeadlinePlugin):
for tile in tile_info:
path = tile["filepath"]
pos_x = tile["pos_x"]
tile_height = self.info_about_input(path)["height"]
tile_height = info_about_input(oiiotool_path, path)["height"]
if self.renderer == "vray":
pos_y = tile["pos_y"]
else:
@ -326,47 +531,3 @@ class OpenPypeTileAssembler(DeadlinePlugin):
ffmpeg_args.append("\"{}\"".format(output_path))
return ffmpeg_args
def info_about_input(self, input_path):
args = [self.render_executable(), "--info:format=xml", input_path]
popen = subprocess.Popen(
" ".join(args),
shell=True,
stdout=subprocess.PIPE
)
popen_output = popen.communicate()[0].replace(b"\r\n", b"")
xmldoc = minidom.parseString(popen_output)
image_spec = None
for main_child in xmldoc.childNodes:
if main_child.nodeName.lower() == "imagespec":
image_spec = main_child
break
info = {}
if not image_spec:
return info
def child_check(node):
if len(node.childNodes) != 1:
self.FailRender((
"Implementation BUG. Node {} has more children than 1"
).format(node.nodeName))
for child in image_spec.childNodes:
if child.nodeName in LIST_KEYS:
values = []
for node in child.childNodes:
child_check(node)
values.append(node.childNodes[0].nodeValue)
info[child.nodeName] = values
elif child.nodeName in INT_KEYS:
child_check(child)
info[child.nodeName] = int(child.childNodes[0].nodeValue)
else:
child_check(child)
info[child.nodeName] = child.childNodes[0].nodeValue
return info

View file

@ -46,7 +46,7 @@
"enabled": true,
"optional": false,
"active": true,
"tile_assembler_plugin": "oiio",
"tile_assembler_plugin": "OpenPypeTileAssembler",
"use_published": true,
"asset_dependencies": true,
"group": "none",

View file

@ -502,6 +502,12 @@
}
}
},
"ExtractMayaSceneRaw": {
"enabled": true,
"add_for_families": [
"layout"
]
},
"ExtractCameraAlembic": {
"enabled": true,
"optional": true,

View file

@ -103,7 +103,7 @@
"DraftTileAssembler": "Draft Tile Assembler"
},
{
"oiio": "Open Image IO"
"OpenPypeTileAssembler": "Open Image IO"
}
]
},

View file

@ -547,6 +547,30 @@
"type": "schema",
"name": "schema_maya_capture"
},
{
"type": "dict",
"collapsible": true,
"key": "ExtractMayaSceneRaw",
"label": "Maya Scene (Raw)",
"checkbox_key": "enabled",
"children": [
{
"type": "boolean",
"key": "enabled",
"label": "Enabled"
},
{
"type": "label",
"label": "Add loaded instances to those published families:"
},
{
"key": "add_for_families",
"label": "Families",
"type": "list",
"object_type": "text"
}
]
},
{
"type": "dict",
"collapsible": true,

View file

@ -1,3 +1,3 @@
# -*- coding: utf-8 -*-
"""Package declaring Pype version."""
__version__ = "3.9.0-nightly.7"
__version__ = "3.9.0-nightly.8"

View file

@ -1,6 +1,6 @@
[tool.poetry]
name = "OpenPype"
version = "3.9.0-nightly.7" # OpenPype
version = "3.9.0-nightly.8" # OpenPype
description = "Open VFX and Animation pipeline with support."
authors = ["OpenPype Team <info@openpype.io>"]
license = "MIT License"