ayon-core/ayon_start.py
Jakub Trllo 47473a8a23 General: Connect to AYON server (base) (#3924)
* implemented 'get_workfile_info' in entities

* removed 'prepare_asset_update_data' which is not used

* disable settings and project manager if in v4 mode

* prepared conversion helper functions for v4 entities

* prepared conversion functions for hero versions

* fix hero versions

* implemented get_archived_representations

* fix get latest versions

* return prepared changes

* handle archived representation

* raise exception on failed json conversion

* map archived to active properly

* make sure default fields are added

* fix conversion of hero version entity

* fix conversion of archived representations

* fix some conversions of representations and versions

* changed active behavior in queries

* fixed hero versions

* implemented basic thumbnail caching

* added raw variants of crud methods

* implemented methods to get and create thumbnail

* fix from flat dict

* implemented some basic folder conversion for updates

* fix thumbnail updates for version

* implemented v4 thumbnail integrator

* simplified data mapping

* 'get_thumbnail' function also expect entity type and entity id for which is the thumbnail received

* implemented 'get_thumbnail' for server

* fix how thumbnail id is received from entity

* removed unnecessary method 'get_thumbnail_id_from_source'

* implemented thumbnail resolver for v4

* removed unnecessary print

* move create and delete project directly to server api

* disable local settings action too on v4

* OP-3521 - added method to check and download updated addons from v4 server

* OP-3521 - added more descriptive error message for missing source

* OP-3521 - added default implementation of addon downloader to import

* OP-3521 - added check for dependency package zips

WIP - server doesn't contain required endpoint. Testing only with mockup data for now.

* OP-3521 - fixed parsing of DependencyItem

Added Server Url type and ServerAddonDownloader - v4 server doesn't know its own DNS for static files so it is sending unique name and url must be created during runtime.

* OP-3521 - fixed creation of targed directories

* change nev keys to look for and don't set them automatically

* fix task type conversion

* implemented base of loading v4 addons in v3

* Refactored argument name in Downloaders

* Updated parsing to DependencyItem according to current schema

* Implemented downloading of package from server

* Updated resolving of failures

Uses Enum items.

* Introduced passing of authorization token

Better to inject it than to have it from env var.

* Remove weird parsing of server_url

Not necessary, endpoints have same prefix.

* Fix doubling asset version name in addons folder

Zip file should already contain `addonName_addonVersion` as first subfolder

* Fix doubling asset version name in addons folder

Zip file should already contain `addonName_addonVersion` as first subfolder

* Made server_endpoint optional

Argument should be better for testing, but for calling from separate methods it would be better to encapsulate it.

Removed unwanted temporary productionPackage value

* Use existing method to pull addon info from Server to load v4 version of addon

* Raise exception when server doesn't have any production dependency package

* added ability to specify v3 alias of addon name

* expect v3_alias as uppered constant

* Re-implemented method to get addon info

Previous implementation wouldn't work in Python2 hosts.
Will be refactored in the future.

* fix '__getattr__'

* added ayon api to pyproject.toml and lock file

* use ayon api in common connection

* added mapping for label

* use ayon_api in client codebase

* separated clearing cache of url and username

* bump ayon api version

* rename env 'OP4_TEST' to 'USE_AYON_SERVER'

* Move and renamend get_addons_info to get_addons_info_as_dict in addon_distribution

Should be moved to ayon_api later

* Replaced requests calls with ayon_api

* Replaced OP4_TEST_ENABLED with AYON_SERVER_ENABLED

fixed endpoints

* Hound

* Hound

* OP-3521 - fix wrong key in get_representation_parents

parents overloads parents

* OP-3521 - changes for v4 of SiteSync addon

* OP-3521 - fix names

* OP-3521 - remove storing project_name

It should be safer to go thorug self.dbcon apparently

* OP-3521 - remove unwanted

"context["folder"]" can be only in dummy test data

* OP-3521 - move site sync loaders to addon

* Use only project instead of self.project

* OP-3521 - added missed get_progress_for_repre

* base of settings conversion script

* simplified ayon functions in start.py

* added loading of settings from ayon server

* added a note about colors

* fix global and local settings functions

* AvalonMongoDB is not using mongo connection on ayon server enabled

* 'get_dynamic_modules_dirs' is not checking system settings for paths in setting

* log viewer is disabled when ayon server is enabled

* basic logic of enabling/disabled addons

* don't use mongo logging if ayon server is enabled

* update ayon api

* bump ayon api again

* use ayon_api to get addons info in modules/base

* update ayon api

* moved helper functions to get addons and dependencies dir to common functions

* Initialization of AddonInfo is not crashing on unkonwn sources

* renamed 'DependencyDownloader' to 'AyonServerDownloader'

* renamed function 'default_addon_downloader' to 'get_default_addon_downloader'

* Added ability to convert 'WebAddonSource' to 'ServerResourceSorce'

* missing dependency package on server won't cause crash

* data sent to downloaders don't contain ayon specific headers

* modified addon distribution to not duplicate 'ayon_api' functionality

* fix doubled function defintioin

* unzip client file to addon destination

* formatting - unify quotes

* disable usage of mongo connection if in ayon mode

* renamed window.py to login_window.py

* added webpublisher settings conversion

* added maya conversion function

* reuse variable

* reuse variable (similar to previous commit)

* fix ayon addons loading

* fix typo 'AyonSettingsCahe' -> 'AyonSettingsCache'

* fix enabled state changes

* fix rr_path in royal render conversion

* avoid mongo calls in AYON state

* implemented custom AYON start script

* fix formatting (after black)

* ayon_start cleanup

* 'get_addons_dir' and 'get_dependencies_dir' store value to environment variable

* add docstrings to local dir functions

* addon info has full name

* fix modules enabled states

* removed unused 'run_disk_mapping_commands'

* removed ayon logic from 'start.py'

* fix warning message

* renamed 'openpype_common' to 'ayon_common'

* removed unused import

* don't import igniter

* removed startup validations of third parties

* change what's shown in version info

* fix which keys are applied from ayon values

* fix method name

* get applications from attribs

* Implemented UI basics to be able change user or logout

* merged server.py and credentials.py

* add more metadata to urls

* implemented change token

* implemented change user ui functionality

* implemented change user ui

* modify window to handle username and token value

* pass username to add server

* fix show UI cases

* added loggin action to tray

* update ayon api

* added missing dependency

* convert applications to config in a right way

* initial implementation of 'nuke' settings conversion

* removed few nuke comments

* implemented hiero conversion

* added imageio conversion

* added run ayon tray script

* fix few settings conversions

* Renamed class of source classes as they are not just for addons

* implemented objec to track source transfer progress

* Implemented distribution item with multiple sources

* Implemented ayon distribution wrapper to care about multiple things during distribution

* added 'cleanup' method for downlaoders

* download gets tranfer progress object

* Change UploadState enum

* added missing imports

* use AyonDistribution in ayon_start.py

* removed unused functions

* removed implemented TODOs

* fix import

* fix key used for Web source

* removed temp development fix

* formatting fix

* keep information if source require distribution

* handle 'require_distribution' attribute in distribution process

* added path attribute to server source

* added option to pass addons infor to ayon distribution

* fix tests

* fix formatting

* Fix typo

* Fix typo

* remove '_try_convert_to_server_source'

* renamed attributes and methods to match their content

* it is possible to pass dependency package info to AyonDistribution

* fix called methods in tests

* added public properties for error message and error detail

* Added filename to WebSourceInfo

Useful for GDrive sharable links where target file name is unknown/unparsable, it should be provided explicitly.

* unify source conversion by adding 'convert_source' function

* Fix error message

Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>

* added docstring for 'transfer_progress'

* don't create metadata file on read

* added few docstrings

* add default folder fields to folder/task queries

* fix generators

* add dependencies when runnign from code

* add sys paths from distribution to pythonpath env

* fix missing applications

* added missing conversions for maya renderers

* fix formatting

* update ayon api

* fix hashes in lock file

* Use better exception

Co-authored-by: Ondřej Samohel <33513211+antirotor@users.noreply.github.com>

* Use Python 3 syntax

Co-authored-by: Ondřej Samohel <33513211+antirotor@users.noreply.github.com>

* apply some of sugested changes in ayon_start

* added some docstrings and suggested modifications

* copy create env from develop

* fix rendersettings conversion

* change code by suggestions

* added missing args to docstring

* added missing docstrings

* separated downloader and download factory

* fix ayon settings

* added some basic file docstring to ayon_settings

* join else conditions

* fix project settings conversion

* fix created at conversion

* fix workfile info query

* fix publisher UI

* added utils function 'get_ayon_appdirs'

* fix 'get_all_current_info'

* fix server url assignment when url is set

* updated ayon api

* added utils functions to create local site id for ayon

* added helper functions to create global connection

* create global connection in ayon start to start use site id

* use ayon site id in ayon mode

* formatting cleanup

* added header docstring

* fixes after ayon_api update

* load addons from ynput appdirs

* fix function call

* added docstring

* update ayon pyton api

* fix settings access

* use ayon_api to get root overrides in Anatomy

* bumbayon version to 0.1.13

* nuke: fixing settings keys from settings

* fix burnins definitions

* change v4 to AYON in thumbnail integrate

* fix one more v4 information

* Fixes after rebase

* fix extract burnin conversion

* additional fix of extract burnin

* SiteSync:added missed loaders or v3 compatibility (#4587)

* Added site sync loaders for v3 compatibility

* Fix get_progress_for_repre

* use 'files.name' instead of 'files.baseName'

* update ayon api to 0.1.14

* add common to include files

* change arguments for hero version creation

* skip shotgrid settings conversion if different ayon addon is used

* added ayon icons

* fix labels of application variants

* added option to show login window always on top

* login window on invalid credentials is always on top

* update ayon api

* update ayon api

* add entityType to project and folders

* AYON: Editorial hierarchy creation (#4699)

* disable extract hierarchy avalon when ayon mode is enabled

* implemented extract hierarchy to AYON

---------

Co-authored-by: Petr Kalis <petr.kalis@gmail.com>
Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>
Co-authored-by: Ondřej Samohel <33513211+antirotor@users.noreply.github.com>
Co-authored-by: Jakub Jezek <jakubjezek001@gmail.com>
2023-07-11 17:50:50 +02:00

376 lines
11 KiB
Python

# -*- coding: utf-8 -*-
"""Main entry point for AYON command.
Bootstrapping process of AYON.
"""
import os
import sys
import site
import traceback
import contextlib
# Enabled logging debug mode when "--debug" is passed
if "--verbose" in sys.argv:
expected_values = (
"Expected: notset, debug, info, warning, error, critical"
" or integer [0-50]."
)
idx = sys.argv.index("--verbose")
sys.argv.pop(idx)
if idx < len(sys.argv):
value = sys.argv.pop(idx)
else:
raise RuntimeError((
f"Expect value after \"--verbose\" argument. {expected_values}"
))
log_level = None
low_value = value.lower()
if low_value.isdigit():
log_level = int(low_value)
elif low_value == "notset":
log_level = 0
elif low_value == "debug":
log_level = 10
elif low_value == "info":
log_level = 20
elif low_value == "warning":
log_level = 30
elif low_value == "error":
log_level = 40
elif low_value == "critical":
log_level = 50
if log_level is None:
raise ValueError((
"Unexpected value after \"--verbose\" "
f"argument \"{value}\". {expected_values}"
))
os.environ["OPENPYPE_LOG_LEVEL"] = str(log_level)
# Enable debug mode, may affect log level if log level is not defined
if "--debug" in sys.argv:
sys.argv.remove("--debug")
os.environ["OPENPYPE_DEBUG"] = "1"
if "--automatic-tests" in sys.argv:
sys.argv.remove("--automatic-tests")
os.environ["IS_TEST"] = "1"
if "--use-staging" in sys.argv:
sys.argv.remove("--use-staging")
os.environ["OPENPYPE_USE_STAGING"] = "1"
_silent_commands = {
"run",
"standalonepublisher",
"extractenvironments",
"version"
}
if "--headless" in sys.argv:
os.environ["OPENPYPE_HEADLESS_MODE"] = "1"
sys.argv.remove("--headless")
elif os.getenv("OPENPYPE_HEADLESS_MODE") != "1":
os.environ.pop("OPENPYPE_HEADLESS_MODE", None)
IS_BUILT_APPLICATION = getattr(sys, "frozen", False)
HEADLESS_MODE_ENABLED = os.environ.get("OPENPYPE_HEADLESS_MODE") == "1"
SILENT_MODE_ENABLED = any(arg in _silent_commands for arg in sys.argv)
_pythonpath = os.getenv("PYTHONPATH", "")
_python_paths = _pythonpath.split(os.pathsep)
if not IS_BUILT_APPLICATION:
# Code root defined by `start.py` directory
AYON_ROOT = os.path.dirname(os.path.abspath(__file__))
_dependencies_path = site.getsitepackages()[-1]
else:
AYON_ROOT = os.path.dirname(sys.executable)
# add dependencies folder to sys.pat for frozen code
_dependencies_path = os.path.normpath(
os.path.join(AYON_ROOT, "dependencies")
)
# add stuff from `<frozen>/dependencies` to PYTHONPATH.
sys.path.append(_dependencies_path)
_python_paths.append(_dependencies_path)
# Vendored python modules that must not be in PYTHONPATH environment but
# are required for OpenPype processes
sys.path.insert(0, os.path.join(AYON_ROOT, "vendor", "python"))
# Add common package to sys path
# - common contains common code for bootstraping and OpenPype processes
sys.path.insert(0, os.path.join(AYON_ROOT, "common"))
# This is content of 'core' addon which is ATM part of build
common_python_vendor = os.path.join(
AYON_ROOT,
"openpype",
"vendor",
"python",
"common"
)
# Add tools dir to sys path for pyblish UI discovery
tools_dir = os.path.join(AYON_ROOT, "openpype", "tools")
for path in (AYON_ROOT, common_python_vendor, tools_dir):
while path in _python_paths:
_python_paths.remove(path)
while path in sys.path:
sys.path.remove(path)
_python_paths.insert(0, path)
sys.path.insert(0, path)
os.environ["PYTHONPATH"] = os.pathsep.join(_python_paths)
# enabled AYON state
os.environ["USE_AYON_SERVER"] = "1"
# Set this to point either to `python` from venv in case of live code
# or to `ayon` or `ayon_console` in case of frozen code
os.environ["OPENPYPE_EXECUTABLE"] = sys.executable
os.environ["AYON_ROOT"] = AYON_ROOT
os.environ["OPENPYPE_ROOT"] = AYON_ROOT
os.environ["OPENPYPE_REPOS_ROOT"] = AYON_ROOT
os.environ["AVALON_LABEL"] = "AYON"
# Set name of pyblish UI import
os.environ["PYBLISH_GUI"] = "pyblish_pype"
import blessed # noqa: E402
import certifi # noqa: E402
if sys.__stdout__:
term = blessed.Terminal()
def _print(message: str):
if SILENT_MODE_ENABLED:
pass
elif message.startswith("!!! "):
print(f'{term.orangered2("!!! ")}{message[4:]}')
elif message.startswith(">>> "):
print(f'{term.aquamarine3(">>> ")}{message[4:]}')
elif message.startswith("--- "):
print(f'{term.darkolivegreen3("--- ")}{message[4:]}')
elif message.startswith("*** "):
print(f'{term.gold("*** ")}{message[4:]}')
elif message.startswith(" - "):
print(f'{term.wheat(" - ")}{message[4:]}')
elif message.startswith(" . "):
print(f'{term.tan(" . ")}{message[4:]}')
elif message.startswith(" - "):
print(f'{term.seagreen3(" - ")}{message[7:]}')
elif message.startswith(" ! "):
print(f'{term.goldenrod(" ! ")}{message[7:]}')
elif message.startswith(" * "):
print(f'{term.aquamarine1(" * ")}{message[7:]}')
elif message.startswith(" "):
print(f'{term.darkseagreen3(" ")}{message[4:]}')
else:
print(message)
else:
def _print(message: str):
if not SILENT_MODE_ENABLED:
print(message)
# if SSL_CERT_FILE is not set prior to OpenPype launch, we set it to point
# to certifi bundle to make sure we have reasonably new CA certificates.
if not os.getenv("SSL_CERT_FILE"):
os.environ["SSL_CERT_FILE"] = certifi.where()
elif os.getenv("SSL_CERT_FILE") != certifi.where():
_print("--- your system is set to use custom CA certificate bundle.")
from ayon_common.connection.credentials import (
ask_to_login_ui,
add_server,
need_server_or_login,
load_environments,
set_environments,
create_global_connection,
confirm_server_login,
)
from ayon_common.distribution.addon_distribution import AyonDistribution
def set_global_environments() -> None:
"""Set global OpenPype's environments."""
import acre
from openpype.settings import get_general_environments
general_env = get_general_environments()
# first resolve general environment because merge doesn't expect
# values to be list.
# TODO: switch to OpenPype environment functions
merged_env = acre.merge(
acre.compute(acre.parse(general_env), cleanup=False),
dict(os.environ)
)
env = acre.compute(
merged_env,
cleanup=False
)
os.environ.clear()
os.environ.update(env)
# Hardcoded default values
os.environ["PYBLISH_GUI"] = "pyblish_pype"
# Change scale factor only if is not set
if "QT_AUTO_SCREEN_SCALE_FACTOR" not in os.environ:
os.environ["QT_AUTO_SCREEN_SCALE_FACTOR"] = "1"
def set_addons_environments():
"""Set global environments for OpenPype modules.
This requires to have OpenPype in `sys.path`.
"""
import acre
from openpype.modules import ModulesManager
modules_manager = ModulesManager()
# Merge environments with current environments and update values
if module_envs := modules_manager.collect_global_environments():
parsed_envs = acre.parse(module_envs)
env = acre.merge(parsed_envs, dict(os.environ))
os.environ.clear()
os.environ.update(env)
def _connect_to_ayon_server():
load_environments()
if not need_server_or_login():
create_global_connection()
return
if HEADLESS_MODE_ENABLED:
_print("!!! Cannot open v4 Login dialog in headless mode.")
_print((
"!!! Please use `AYON_SERVER_URL` to specify server address"
" and 'AYON_TOKEN' to specify user's token."
))
sys.exit(1)
current_url = os.environ.get("AYON_SERVER_URL")
url, token, username = ask_to_login_ui(current_url, always_on_top=True)
if url is not None and token is not None:
confirm_server_login(url, token, username)
return
if url is not None:
add_server(url, username)
_print("!!! Login was not successful.")
sys.exit(0)
def _check_and_update_from_ayon_server():
"""Gets addon info from v4, compares with local folder and updates it.
Raises:
RuntimeError
"""
distribution = AyonDistribution()
distribution.distribute()
distribution.validate_distribution()
python_paths = [
path
for path in os.getenv("PYTHONPATH", "").split(os.pathsep)
if path
]
for path in distribution.get_sys_paths():
sys.path.insert(0, path)
if path not in python_paths:
python_paths.append(path)
os.environ["PYTHONPATH"] = os.pathsep.join(python_paths)
def boot():
"""Bootstrap OpenPype."""
from openpype.version import __version__
# TODO load version
os.environ["OPENPYPE_VERSION"] = __version__
os.environ["AYON_VERSION"] = __version__
use_staging = os.environ.get("OPENPYPE_USE_STAGING") == "1"
_connect_to_ayon_server()
_check_and_update_from_ayon_server()
# delete OpenPype module and it's submodules from cache so it is used from
# specific version
modules_to_del = [
sys.modules.pop(module_name)
for module_name in tuple(sys.modules)
if module_name == "openpype" or module_name.startswith("openpype.")
]
for module_name in modules_to_del:
with contextlib.suppress(AttributeError, KeyError):
del sys.modules[module_name]
from openpype import cli
from openpype.lib import terminal as t
_print(">>> loading environments ...")
_print(" - global AYON ...")
set_global_environments()
_print(" - for addons ...")
set_addons_environments()
# print info when not running scripts defined in 'silent commands'
if not SILENT_MODE_ENABLED:
info = get_info(use_staging)
info.insert(0, f">>> Using AYON from [ {AYON_ROOT} ]")
t_width = 20
with contextlib.suppress(ValueError, OSError):
t_width = os.get_terminal_size().columns - 2
_header = f"*** AYON [{__version__}] "
info.insert(0, _header + "-" * (t_width - len(_header)))
for i in info:
t.echo(i)
try:
cli.main(obj={}, prog_name="openpype")
except Exception: # noqa
exc_info = sys.exc_info()
_print("!!! OpenPype crashed:")
traceback.print_exception(*exc_info)
sys.exit(1)
def get_info(use_staging=None) -> list:
"""Print additional information to console."""
inf = []
if use_staging:
inf.append(("AYON variant", "staging"))
else:
inf.append(("AYON variant", "production"))
# NOTE add addons information
maximum = max(len(i[0]) for i in inf)
formatted = []
for info in inf:
padding = (maximum - len(info[0])) + 1
formatted.append(f'... {info[0]}:{" " * padding}[ {info[1]} ]')
return formatted
if __name__ == "__main__":
boot()