Merge branch 'main' into develop

CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at info@pype.club. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

CONTRIBUTING.md

@@ -0,0 +1,53 @@
+## How to contribute to Pype
+
+#### **Did you find a bug?**
+
+1. Check in the issues and our [bug triage[(https://github.com/pypeclub/pype/projects/2) to make sure it wasn't reported already.
+2. Ask on our [discord](http://pype.community/chat) Often, what appears as a bug, might be the intended behaviour for someone else.
+3. Create a new issue.
+4. Use the issue template for you PR please.
+
+
+#### **Did you write a patch that fixes a bug?**
+
+- Open a new GitHub pull request with the patch.
+- Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable.
+
+
+#### **Do you intend to add a new feature or change an existing one?**
+
+- Open a new thread in the [github discussions](https://github.com/pypeclub/pype/discussions/new)
+- Do not open issue untill the suggestion is discussed. We will convert accepted suggestions into backlog and point them to the relevant discussion thread to keep the context.
+
+#### **Do you have questions about the source code?**
+
+Open a new question on [github discussions](https://github.com/pypeclub/pype/discussions/new)
+
+## Branching Strategy
+
+As we move to 3.x as the primary supported version of pype and only keep 2.15 on bug bugfixes and client sponsored feature requests, we need to be very careful with merging strategy.
+
+We also use this opportunity to switch the branch naming. 3.0 production branch will no longer be called MASTER,  but will be renamed to MAIN. Develop will stay as it is.
+
+A few important notes about 2.x and 3.x development:
+
+- 3.x features are not backported to 2.x unless specifically requested
+- 3.x bugs and hotfixes can be ported to 2.x if they are relevant or severe
+- 2.x features and bugs MUST be ported to 3.x at the same time
+
+## Pull Requests
+
+- Each 2.x PR MUST have a corresponding 3.x PR in github. Without 3.x PR, 2.x features will not be merged! Luckily most of the code is compatible, albeit sometimes in a different place after refactor. Porting from 2.x to 3.x should be really easy.
+- Please keep the corresponding 2 and 3 PR names the same so they can be easily identified from the PR list page.
+- Each 2.x PR should be labeled with `2.x-dev` label.
+
+Inside each PR, put a link to the corresponding PR
+
+Of course if you want to contribute, feel free to make a PR to only 2.x/develop or develop, based on what you are using. While reviewing the PRs, we might convert the code to corresponding PR for the other release ourselves. 
+
+We might also change the target of you PR to and intermediate branch, rather than `develop` if we feel it requires some extra work on our end. That way, we preserve all your commits so you don't loos out on the contribution credits.
+
+
+
+
+If a PR is targeted at 2.x release it must be labelled with 2x-dev label in Github.   

pype/modules/sync_server/README.md

@@ -62,7 +62,7 @@ Needed configuration:
  - `"local_id": "local_0",` -- identifier of user pype
  - `"retry_cnt": 3,`        -- how many times try to synch file in case of error
  - `"loop_delay": 60,`      -- how many seconds between sync loops
- - `"active_site": "studio",` -- which site user current, 'studio' by default, 
+ - `"publish_site": "studio",` -- which site user current, 'studio' by default, 
                               could by same as 'local_id' if user is working
                               from home without connection to studio 
                               infrastructure
@@ -71,7 +71,7 @@ Needed configuration:
                              Used in IntegrateNew to prepare skeleton for 
                              syncing in the representation record.
                              Leave empty if no syncing is wanted.
-  This is a general configuration, 'local_id', 'active_site' and 'remote_site'
+  This is a general configuration, 'local_id', 'publish_site' and 'remote_site'
   will be set and changed by some GUI in the future.                           
   
 `pype/settings/defaults/project_settings/global.json`.`sync_server`.`sites`:

pype/modules/sync_server/providers/abstract_provider.py

@@ -3,6 +3,13 @@ from abc import ABCMeta, abstractmethod
 
 class AbstractProvider(metaclass=ABCMeta):
 
+    def __init__(self, site_name, tree=None, presets=None):
+        self.presets = None
+        self.active = False
+        self.site_name = site_name
+
+        self.presets = presets
+
     @abstractmethod
     def is_active(self):
         """
@@ -27,13 +34,14 @@ class AbstractProvider(metaclass=ABCMeta):
         pass
 
     @abstractmethod
-    def download_file(self, source_path, local_path):
+    def download_file(self, source_path, local_path, overwrite=True):
         """
             Download file from provider into local system
 
         Args:
             source_path (string): absolute path on provider
             local_path (string): absolute path on local
+            overwrite (bool): default set to True
         Returns:
             None
         """

pype/modules/sync_server/providers/gdrive.py

@@ -351,6 +351,10 @@ class GDriveHandler(AbstractProvider):
             last_tick = status = response = None
             status_val = 0
             while response is None:
+                if server.is_representation_paused(representation['_id'],
+                                                   check_parents=True,
+                                                   project_name=collection):
+                    raise ValueError("Paused during process, please redo.")
                 if status:
                     status_val = float(status.progress())
                 if not last_tick or \
@@ -433,6 +437,10 @@ class GDriveHandler(AbstractProvider):
             last_tick = status = response = None
             status_val = 0
             while response is None:
+                if server.is_representation_paused(representation['_id'],
+                                                   check_parents=True,
+                                                   project_name=collection):
+                    raise ValueError("Paused during process, please redo.")
                 if status:
                     status_val = float(status.progress())
                 if not last_tick or \

pype/modules/sync_server/providers/lib.py

@@ -1,10 +1,6 @@
 from enum import Enum
 from .gdrive import GDriveHandler
-
-
-class Providers(Enum):
-    LOCAL = 'studio'
-    GDRIVE = 'gdrive'
+from .local_drive import LocalDriveHandler
 
 
 class ProviderFactory:
@@ -94,3 +90,4 @@ factory = ProviderFactory()
 # 7 denotes number of files that could be synced in single loop - learned by
 # trial and error
 factory.register_provider('gdrive', GDriveHandler, 7)
+factory.register_provider('local_drive', LocalDriveHandler, 10)

pype/modules/sync_server/providers/local_drive.py

@@ -0,0 +1,59 @@
+from __future__ import print_function
+import os.path
+import shutil
+
+from pype.api import Logger
+from .abstract_provider import AbstractProvider
+
+log = Logger().get_logger("SyncServer")
+
+
+class LocalDriveHandler(AbstractProvider):
+    """ Handles required operations on mounted disks with OS """
+    def is_active(self):
+        return True
+
+    def upload_file(self, source_path, target_path, overwrite=True):
+        """
+            Copies file from 'source_path' to 'target_path'
+        """
+        if os.path.exists(source_path):
+            if overwrite:
+                shutil.copy(source_path, target_path)
+            else:
+                if os.path.exists(target_path):
+                    raise ValueError("File {} exists, set overwrite".
+                                     format(target_path))
+
+    def download_file(self, source_path, local_path, overwrite=True):
+        """
+            Download a file form 'source_path' to 'local_path'
+        """
+        if os.path.exists(source_path):
+            if overwrite:
+                shutil.copy(source_path, local_path)
+            else:
+                if os.path.exists(local_path):
+                    raise ValueError("File {} exists, set overwrite".
+                                     format(local_path))
+
+    def delete_file(self, path):
+        """
+            Deletes a file at 'path'
+        """
+        if os.path.exists(path):
+            os.remove(path)
+
+    def list_folder(self, folder_path):
+        """
+            Returns list of files and subfolder in a 'folder_path'. Non recurs
+        """
+        lst = []
+        if os.path.isdir(folder_path):
+            for (dir_path, dir_names, file_names) in os.walk(folder_path):
+                for name in file_names:
+                    lst.append(os.path.join(dir_path, name))
+                for name in dir_names:
+                    lst.append(os.path.join(dir_path, name))
+
+        return lst

pype/plugins/publish/integrate_master_version.py

@@ -298,6 +298,62 @@ class IntegrateMasterVersion(pyblish.api.InstancePlugin):
                 repre["data"] = repre_data
                 repre.pop("_id", None)
 
+                # Prepare paths of source and destination files
+                if len(published_files) == 1:
+                    src_to_dst_file_paths.append(
+                        (published_files[0], template_filled)
+                    )
+                else:
+                    collections, remainders = clique.assemble(published_files)
+                    if remainders or not collections or len(collections) > 1:
+                        raise Exception((
+                                            "Integrity error. Files of published representation "
+                                            "is combination of frame collections and single files."
+                                            "Collections: `{}` Single files: `{}`"
+                                        ).format(str(collections),
+                                                 str(remainders)))
+
+                    src_col = collections[0]
+
+                    # Get head and tail for collection
+                    frame_splitter = "_-_FRAME_SPLIT_-_"
+                    anatomy_data["frame"] = frame_splitter
+                    _anatomy_filled = anatomy.format(anatomy_data)
+                    _template_filled = _anatomy_filled["master"]["path"]
+                    head, tail = _template_filled.split(frame_splitter)
+                    padding = int(
+                        anatomy.templates["render"].get(
+                            "frame_padding",
+                            anatomy.templates["render"].get("padding")
+                        )
+                    )
+
+                    dst_col = clique.Collection(
+                        head=head, padding=padding, tail=tail
+                    )
+                    dst_col.indexes.clear()
+                    dst_col.indexes.update(src_col.indexes)
+                    for src_file, dst_file in zip(src_col, dst_col):
+                        src_to_dst_file_paths.append(
+                            (src_file, dst_file)
+                        )
+
+                # replace original file name with master name in repre doc
+                for index in range(len(repre.get("files"))):
+                    file = repre.get("files")[index]
+                    file_name = os.path.basename(file.get('path'))
+                    for src_file, dst_file in src_to_dst_file_paths:
+                        src_file_name = os.path.basename(src_file)
+                        if src_file_name == file_name:
+                            repre["files"][index]["path"] = self._update_path(
+                                anatomy, repre["files"][index]["path"],
+                                src_file, dst_file)
+
+                            repre["files"][index]["hash"] = self._update_hash(
+                                repre["files"][index]["hash"],
+                                src_file_name, dst_file
+                            )
+
                 schema.validate(repre)
 
                 repre_name_low = repre["name"].lower()
@@ -333,46 +389,6 @@ class IntegrateMasterVersion(pyblish.api.InstancePlugin):
                         InsertOne(repre)
                     )
 
-                # Prepare paths of source and destination files
-                if len(published_files) == 1:
-                    src_to_dst_file_paths.append(
-                        (published_files[0], template_filled)
-                    )
-                    continue
-
-                collections, remainders = clique.assemble(published_files)
-                if remainders or not collections or len(collections) > 1:
-                    raise Exception((
-                        "Integrity error. Files of published representation "
-                        "is combination of frame collections and single files."
-                        "Collections: `{}` Single files: `{}`"
-                    ).format(str(collections), str(remainders)))
-
-                src_col = collections[0]
-
-                # Get head and tail for collection
-                frame_splitter = "_-_FRAME_SPLIT_-_"
-                anatomy_data["frame"] = frame_splitter
-                _anatomy_filled = anatomy.format(anatomy_data)
-                _template_filled = _anatomy_filled["master"]["path"]
-                head, tail = _template_filled.split(frame_splitter)
-                padding = int(
-                    anatomy.templates["render"].get(
-                        "frame_padding",
-                        anatomy.templates["render"].get("padding")
-                    )
-                )
-
-                dst_col = clique.Collection(
-                    head=head, padding=padding, tail=tail
-                )
-                dst_col.indexes.clear()
-                dst_col.indexes.update(src_col.indexes)
-                for src_file, dst_file in zip(src_col, dst_col):
-                    src_to_dst_file_paths.append(
-                        (src_file, dst_file)
-                    )
-
             self.path_checks = []
 
             # Copy(hardlink) paths of source and destination files
@@ -533,3 +549,39 @@ class IntegrateMasterVersion(pyblish.api.InstancePlugin):
             "type": "representation"
         }))
         return (master_version, master_repres)
+
+    def _update_path(self, anatomy, path, src_file, dst_file):
+        """
+            Replaces source path with new master path
+
+            'path' contains original path with version, must be replaced with
+            'master' path (with 'master' label and without version)
+
+            Args:
+                anatomy (Anatomy) - to get rootless style of path
+                path (string) - path from DB
+                src_file (string) - original file path
+                dst_file (string) - master file path
+        """
+        _, rootless = anatomy.find_root_template_from_path(
+            dst_file
+        )
+        _, rtls_src = anatomy.find_root_template_from_path(
+            src_file
+        )
+        return path.replace(rtls_src, rootless)
+
+    def _update_hash(self, hash, src_file_name, dst_file):
+        """
+            Updates hash value with proper master name
+        """
+        src_file_name = self._get_name_without_ext(
+            src_file_name)
+        master_file_name = self._get_name_without_ext(
+            dst_file)
+        return hash.replace(src_file_name, master_file_name)
+
+    def _get_name_without_ext(self, value):
+        file_name = os.path.basename(value)
+        file_name, _ = os.path.splitext(file_name)
+        return file_name

pype/settings/defaults/project_settings/global.json

@@ -180,7 +180,7 @@
         }
     },
     "sync_server": {
-        "enabled": false,
+        "enabled": true,
         "config": {
             "local_id": "local_0",
             "retry_cnt": "3",
@@ -192,7 +192,23 @@
             "gdrive": {
                 "provider": "gdrive",
                 "credentials_url": "",
-                "root": "/sync_testing/test"
+                "root": {
+                    "work": ""
+                }
+            },
+            "studio": {
+                "provider": "local_drive",
+                "credentials_url": "",
+                "root": {
+                    "work": ""
+                }
+            },
+            "local_0": {
+                "provider": "local_drive",
+                "credentials_url": "",
+                "root": {
+                    "work": ""
+                }
             }
         }
     }

pype/settings/entities/schemas/projects_schema/schema_project_syncserver.json

@@ -66,10 +66,14 @@
                 "label": "Credentials url"
             },
             {
-                "type": "text",
+                "type": "dict-modifiable",
                 "key": "root",
-                "label": "Root"
-            }]
+                "label": "Roots",
+                "collapsable": false,
+                "collapsable_key": false,
+                "object_type": "text"
+            }
+            ]
         }
     }
     ]