mirror of
https://github.com/ynput/ayon-core.git
synced 2025-12-26 05:42:15 +01:00
9607db50f2
* Ingest input workfile * Ingest input workfile * Ingested expected files, workfile Implemented LocalFileHandler. Test name added to structure to separate files for each test. Removed superfluous `files` to keep other Maya test working * Missing time import * Hound * Skip directories when checking folder structure. * Update tests/lib/testing_classes.py Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com> * Updated integration tests documentation * Ingested test files for Deadline test in maya * Removed unneeded files * Refactored name --------- Co-authored-by: kalisp <petr.kalis@gmail.com> Co-authored-by: Roy Nieterau <roy_nieterau@hotmail.com>
102 lines
5 KiB
Markdown
102 lines
5 KiB
Markdown
Integration test for OpenPype
|
|
=============================
|
|
Contains end-to-end tests for automatic testing of OP.
|
|
|
|
Should run headless publish on all hosts to check basic publish use cases automatically
|
|
to limit regression issues.
|
|
|
|
Uses env var `HEADLESS_PUBLISH` (set in test data zip files) to differentiate between regular publish
|
|
and "automated" one.
|
|
|
|
How to run
|
|
----------
|
|
- activate `{OPENPYPE_ROOT}/.venv`
|
|
- run in cmd
|
|
`{OPENPYPE_ROOT}/.venv/Scripts/python.exe {OPENPYPE_ROOT}/start.py runtests {OPENPYPE_ROOT}/tests/integration`
|
|
- add `hosts/APP_NAME` after integration part to limit only on specific app (eg. `{OPENPYPE_ROOT}/tests/integration/hosts/maya`)
|
|
|
|
OR can use built executables
|
|
`openpype_console runtests {ABS_PATH}/tests/integration`
|
|
|
|
Command line arguments
|
|
----------------------
|
|
- "--mark" - "Run tests marked by",
|
|
- "--pyargs" - "Run tests from package",
|
|
- "--test_data_folder" - "Unzipped directory path of test file",
|
|
- "--persist" - "Persist test DB and published files after test end",
|
|
- "--app_variant" - "Provide specific app variant for test, empty for latest",
|
|
- "--app_group" - "Provide specific app group for test, empty for default",
|
|
- "--timeout" - "Provide specific timeout value for test case",
|
|
- "--setup_only" - "Only create dbs, do not run tests",
|
|
- "--mongo_url" - "MongoDB for testing.",
|
|
|
|
Run Tray for test
|
|
-----------------
|
|
In case of failed test you might want to run it manually and visually debug what happened.
|
|
For that:
|
|
- run tests that is failing
|
|
- add environment variables (to command line process or your IDE)
|
|
- OPENPYPE_DATABASE_NAME = openpype_tests
|
|
- AVALON_DB = avalon_tests
|
|
- run tray as usual
|
|
- `{OPENPYPE_ROOT}/.venv/Scripts/python.exe {OPENPYPE_ROOT}/start.py run tray --debug`
|
|
|
|
You should see only test asset and state of databases for that particular use case.
|
|
|
|
How to check logs/errors from app
|
|
--------------------------------
|
|
Keep PERSIST to True in the class and check `test_openpype.logs` collection.
|
|
|
|
How to create test for publishing from host
|
|
------------------------------------------
|
|
- Extend PublishTest in `tests/lib/testing_classes.py`
|
|
- Use `resources\test_data.zip` skeleton file as a template for testing input data
|
|
- Create subfolder `test_data` with matching name to your test file containing you test class
|
|
- (see `tests/integration/hosts/maya/test_publish_in_maya` and `test_publish_in_maya.py`)
|
|
- Put this subfolder name into TEST_FILES [(HASH_ID, FILE_NAME, MD5_OPTIONAL)]
|
|
- at first position, all others may be ""
|
|
- Put workfile into `test_data/input/workfile`
|
|
- If you require other than base DB dumps provide them to `test_data/input/dumps`
|
|
-- (Check commented code in `db_handler.py` how to dump specific DB. Currently all collections will be dumped.)
|
|
- Implement `last_workfile_path`
|
|
- `startup_scripts` - must contain pointing host to startup script saved into `test_data/input/startup`
|
|
-- Script must contain something like (pseudocode)
|
|
```
|
|
import openpype
|
|
from avalon import api, HOST
|
|
|
|
from openpype.api import Logger
|
|
|
|
log = Logger().get_logger(__name__)
|
|
|
|
api.install(HOST)
|
|
log_lines = []
|
|
for result in pyblish.util.publish_iter():
|
|
for record in result["records"]: # for logging to test_openpype DB
|
|
log_lines.append("{}: {}".format(
|
|
result["plugin"].label, record.msg))
|
|
|
|
if result["error"]:
|
|
err_fmt = "Failed {plugin.__name__}: {error} -- {error.traceback}"
|
|
log.error(err_fmt.format(**result))
|
|
|
|
EXIT_APP (command to exit host)
|
|
```
|
|
(Install and publish methods must be triggered only AFTER host app is fully initialized!)
|
|
- If you would like add any command line arguments for your host app add it to `test_data/input/app_args/app_args.json` (as a json list)
|
|
- Provide any required environment variables to `test_data/input/env_vars/env_vars.json` (as a json dictionary)
|
|
- Implement any assert checks you need in extended class
|
|
- Run test class manually (via Pycharm or pytest runner (TODO))
|
|
- If you want test to visually compare expected files to published one, set PERSIST to True, run test manually
|
|
-- Locate temporary `publish` subfolder of temporary folder (found in debugging console log)
|
|
-- Copy whole folder content into .zip file into `expected` subfolder
|
|
-- By default tests are comparing only structure of `expected` and published format (eg. if you want to save space, replace published files with empty files, but with expected names!)
|
|
-- Zip and upload again, change PERSIST to False
|
|
|
|
- Use `TEST_DATA_FOLDER` variable in your class to reuse existing downloaded and unzipped test data (for faster creation of tests)
|
|
- Keep `APP_VARIANT` empty if you want to trigger test on latest version of app, or provide explicit value (as '2022' for Photoshop for example)
|
|
|
|
For storing test zip files on Google Drive:
|
|
- Zip `test_data.zip`, named it with descriptive name, upload it to Google Drive, right click - `Get link`, copy hash id (file must be accessible to anyone with a link!)
|
|
- Put this hash id and zip file name into TEST_FILES [(HASH_ID, FILE_NAME, MD5_OPTIONAL)]. If you want to check MD5 of downloaded
|
|
file, provide md5 value of zipped file.
|