AYON: OpenPype as server addon (#5199)

* added openpype server addon creation logic

* Better readme

Co-authored-by: Ondřej Samohel <33513211+antirotor@users.noreply.github.com>

* added docstring to 'find_files_in_subdir'

* added some type notations

* add whitespace around assignment

* move pyproject toml to client subfolder and fix its location in private dir

* Fix whitespace

---------

Co-authored-by: Ondřej Samohel <33513211+antirotor@users.noreply.github.com>

.gitignore

@@ -37,6 +37,7 @@ Temporary Items
 ###########
 /build
 /dist/
+/server_addon/package/*
 
 /vendor/bin/*
 /vendor/python/*

server_addon/README.md

@@ -0,0 +1,21 @@
+# OpenPype addon for AYON server
+Convert openpype into AYON addon which can be installed on AYON server. The versioning of the addon is following versioning of OpenPype.
+
+## Intro
+OpenPype is transitioning to AYON, a dedicated server with its own database, moving away from MongoDB. During this transition period, OpenPype will remain compatible with both MongoDB and AYON. However, we will gradually update the codebase to align with AYON's data structure and separate individual components into addons.
+
+Currently, OpenPype has an AYON mode, which means it utilizes the AYON server instead of MongoDB through conversion utilities. Initially, we added the AYON executable alongside the OpenPype executables to enable AYON mode. While this approach worked, updating to new code versions would require a complete reinstallation. To address this, we have decided to create a new repository specifically for the base desktop application logic, which we currently refer to as the AYON Launcher. This Launcher will replace the executables generated by the OpenPype build and convert the OpenPype code into a server addon, resulting in smaller updates.
+
+Since the implementation of the AYON Launcher is not yet fully completed, we will maintain both methods of starting AYON mode for now. Once the AYON Launcher is finished, we will remove the AYON executables from the OpenPype codebase entirely.
+
+During this transitional period, the AYON Launcher addon will be a requirement as the entry point for using the AYON Launcher.
+
+## How to start
+There is a `create_ayon_addon.py` python file which contains logic how to create server addon from OpenPype codebase. Just run the code.
+```shell
+./.poetry/bin/poetry run python ./server_addon/create_ayon_addon.py
+```
+
+It will create directory `./package/openpype/<OpenPype version>/*` folder with all files necessary for AYON server. You can then copy `./package/openpype/` to server addons, or zip the folder and upload it to AYON server. Restart server to update addons information, add the addon version to server bundle and set the bundle for production or staging usage.
+
+Once addon is on server and is enabled, you can just run AYON launcher. Content will be downloaded and used automatically.

server_addon/client/pyproject.toml

@@ -0,0 +1,24 @@
+[project]
+name="openpype"
+description="OpenPype addon for AYON server."
+
+[tool.poetry.dependencies]
+python = ">=3.9.1,<3.10"
+aiohttp_json_rpc = "*" # TVPaint server
+aiohttp-middlewares = "^2.0.0"
+wsrpc_aiohttp = "^3.1.1" # websocket server
+clique = "1.6.*"
+shotgun_api3 = {git = "https://github.com/shotgunsoftware/python-api.git", rev = "v3.3.3"}
+gazu = "^0.9.3"
+google-api-python-client = "^1.12.8" # sync server google support (should be separate?)
+jsonschema = "^2.6.0"
+log4mongo = "^1.7"
+pathlib2= "^2.3.5" # deadline submit publish job only (single place, maybe not needed?)
+pyblish-base = "^1.8.11"
+pynput = "^1.7.2" # Timers manager - TODO replace
+"Qt.py" = "^1.3.3"
+qtawesome = "0.7.3"
+speedcopy = "^2.1"
+slack-sdk = "^3.6.0"
+pysftp = "^0.2.9"
+dropbox = "^11.20.0"

server_addon/create_ayon_addon.py

@@ -0,0 +1,140 @@
+import os
+import re
+import shutil
+import zipfile
+import collections
+from pathlib import Path
+from typing import Any, Optional, Iterable
+
+# Patterns of directories to be skipped for server part of addon
+IGNORE_DIR_PATTERNS: list[re.Pattern] = [
+    re.compile(pattern)
+    for pattern in {
+        # Skip directories starting with '.'
+        r"^\.",
+        # Skip any pycache folders
+        "^__pycache__$"
+    }
+]
+
+# Patterns of files to be skipped for server part of addon
+IGNORE_FILE_PATTERNS: list[re.Pattern] = [
+    re.compile(pattern)
+    for pattern in {
+        # Skip files starting with '.'
+        # NOTE this could be an issue in some cases
+        r"^\.",
+        # Skip '.pyc' files
+        r"\.pyc$"
+    }
+]
+
+
+def _value_match_regexes(value: str, regexes: Iterable[re.Pattern]) -> bool:
+    return any(
+        regex.search(value)
+        for regex in regexes
+    )
+
+
+def find_files_in_subdir(
+    src_path: str,
+    ignore_file_patterns: Optional[list[re.Pattern]] = None,
+    ignore_dir_patterns: Optional[list[re.Pattern]] = None
+):
+    """Find all files to copy in subdirectories of given path.
+
+    All files that match any of the patterns in 'ignore_file_patterns' will
+        be skipped and any directories that match any of the patterns in
+        'ignore_dir_patterns' will be skipped with all subfiles.
+
+    Args:
+        src_path (str): Path to directory to search in.
+        ignore_file_patterns (Optional[list[re.Pattern]]): List of regexes
+            to match files to ignore.
+        ignore_dir_patterns (Optional[list[re.Pattern]]): List of regexes
+            to match directories to ignore.
+
+    Returns:
+        list[tuple[str, str]]: List of tuples with path to file and parent
+            directories relative to 'src_path'.
+    """
+
+    if ignore_file_patterns is None:
+        ignore_file_patterns = IGNORE_FILE_PATTERNS
+
+    if ignore_dir_patterns is None:
+        ignore_dir_patterns = IGNORE_DIR_PATTERNS
+    output: list[tuple[str, str]] = []
+
+    hierarchy_queue = collections.deque()
+    hierarchy_queue.append((src_path, []))
+    while hierarchy_queue:
+        item: tuple[str, str] = hierarchy_queue.popleft()
+        dirpath, parents = item
+        for name in os.listdir(dirpath):
+            path = os.path.join(dirpath, name)
+            if os.path.isfile(path):
+                if not _value_match_regexes(name, ignore_file_patterns):
+                    items = list(parents)
+                    items.append(name)
+                    output.append((path, os.path.sep.join(items)))
+                continue
+
+            if not _value_match_regexes(name, ignore_dir_patterns):
+                items = list(parents)
+                items.append(name)
+                hierarchy_queue.append((path, items))
+
+    return output
+
+
+def main():
+    openpype_addon_dir = Path(os.path.dirname(os.path.abspath(__file__)))
+    server_dir = openpype_addon_dir / "server"
+    package_root = openpype_addon_dir / "package"
+    pyproject_path = openpype_addon_dir / "client" / "pyproject.toml"
+
+    root_dir = openpype_addon_dir.parent
+    openpype_dir = root_dir / "openpype"
+    version_path = openpype_dir / "version.py"
+
+    # Read version
+    version_content: dict[str, Any] = {}
+    with open(str(version_path), "r") as stream:
+        exec(stream.read(), version_content)
+    addon_version: str = version_content["__version__"]
+
+    output_dir = package_root / "openpype" / addon_version
+    private_dir = output_dir / "private"
+
+    # Make sure package dir is empty
+    if package_root.exists():
+        shutil.rmtree(str(package_root))
+    # Make sure output dir is created
+    output_dir.mkdir(parents=True)
+
+    # Copy version
+    shutil.copy(str(version_path), str(output_dir))
+    for subitem in server_dir.iterdir():
+        shutil.copy(str(subitem), str(output_dir / subitem.name))
+
+    # Make sure private dir exists
+    private_dir.mkdir(parents=True)
+
+    # Copy pyproject.toml
+    shutil.copy(
+        str(pyproject_path),
+        (private_dir / pyproject_path.name)
+    )
+
+    # Zip client
+    zip_filepath = private_dir / "client.zip"
+    with zipfile.ZipFile(zip_filepath, "w", zipfile.ZIP_DEFLATED) as zipf:
+        # Add client code content to zip
+        for path, sub_path in find_files_in_subdir(str(openpype_dir)):
+            zipf.write(path, f"{openpype_dir.name}/{sub_path}")
+
+
+if __name__ == "__main__":
+    main()

server_addon/server/__init__.py

@@ -0,0 +1,9 @@
+from ayon_server.addons import BaseServerAddon
+
+from .version import __version__
+
+
+class OpenPypeAddon(BaseServerAddon):
+    name = "openpype"
+    title = "OpenPype"
+    version = __version__