From dd136858f1ea40ad3c94191d647487fa4f31926c Mon Sep 17 00:00:00 2001 From: Daniel Baumann Date: Fri, 18 Oct 2024 20:33:49 +0200 Subject: Adding upstream version 9.0.0. Signed-off-by: Daniel Baumann --- tests/e2e/README.md | 220 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 220 insertions(+) create mode 100644 tests/e2e/README.md (limited to 'tests/e2e/README.md') diff --git a/tests/e2e/README.md b/tests/e2e/README.md new file mode 100644 index 0000000..6552011 --- /dev/null +++ b/tests/e2e/README.md @@ -0,0 +1,220 @@ +# End to end tests + +Thank you for your effort to provide good software tests for Forgejo. +Please also read the general testing instructions in the +[Forgejo contributor documentation](https://forgejo.org/docs/next/contributor/testing/) +and make sure to also check the +[Playwright documentation](https://playwright.dev/docs/intro) +for further information. + +This file is meant to provide specific information for the integration tests +as well as some tips and tricks you should know. + +Feel free to extend this file with more instructions if you feel like you have something to share! + + +## How to run the tests? + +Before running any tests, please ensure you perform a clean frontend build: + +``` +make clean frontend +``` + +Whenever you modify frontend code (i.e. JavaScript and CSS files), +you need to create a new frontend build. + +For tests that require interactive Git repos, +you also need to ensure a Forgejo binary is ready to be used by Git hooks. +For this, you additionally need to run + +~~~ +make TAGS="sqlite sqlite_unlock_notify" backend +~~~ + +### Install dependencies + +Browsertesting is performed by playwright. +You need certain system libraries and playwright will download required browsers. +Playwright takes care of this when you run: + +``` +npx playwright install-deps +``` + +> **Note** +> On some operating systems, the installation of missing libraries can complicate testing certain browsers. +> It is often not necessary to test with all browsers locally. +> Choosing either Firefox or Chromium is fine. + + +### Run all tests + +If you want to run the full test suite, you can use + +``` +make test-e2e-sqlite +``` + +### Interactive testing + +We recommend that you use interactive testing for the development. +After you performed the required builds, +you should use one shell to start the debugserver (and leave it running): + +``` +make test-e2e-debugserver +``` + +It allows you to explore the test data in your local browser, +and playwright to perform tests on it. + +> **Note** +> The modifications persist while the debugserver is running. +> If you modified things, it might be useful to restart it to get back to a fresh state. +> While writing playwright tests, you either +> need to ensure they are resilient against repeated runs +> (e.g. when only creating new content), +> or that they restore the initial state for the next browser run. + +#### With the playwright UI: + +Playwright ships with an integrated UI mode which allows you to +run individual tests and to debug them by seeing detailed traces of what playwright does. +Launch it with: + +``` +npx playwright test --ui +``` + +#### Running individual tests + +``` +npx playwright test actions.test.e2e.js:9 +``` + +First, specify the complete test filename, +and after the colon you can put the linenumber where the test is defined. + + +#### With VSCodium or VSCode + +To debug a test, you can also use "Playwright Test" for +[VScodium](https://open-vsx.org/extension/ms-playwright/playwright) +or [VSCode](https://marketplace.visualstudio.com/items?itemName=ms-playwright.playwright). + + +### Run all tests via local act_runner + +If you have a [forgejo runner](https://code.forgejo.org/forgejo/runner/), +you can use it to run the test jobs: + +``` +forgejo-runner exec -W .forgejo/workflows/e2e.yml --event=pull_request +``` + +### Run e2e tests with another database + +This approach is not currently used, +neither in the CI/CD nor by core contributors on their lcoal machines. +It is still documented for the sake of completeness: +You can also perform e2e tests using MariaDB/MySQL or PostgreSQL if you want. + +Setup a MySQL database inside docker +``` +docker run -e "MYSQL_DATABASE=test" -e "MYSQL_ALLOW_EMPTY_PASSWORD=yes" -p 3306:3306 --rm --name mysql mysql:latest #(just ctrl-c to stop db and clean the container) +docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" --rm --name elasticsearch elasticsearch:7.6.0 #(in a second terminal, just ctrl-c to stop db and clean the container) +``` +Start tests based on the database container +``` +TEST_MYSQL_HOST=localhost:3306 TEST_MYSQL_DBNAME=test TEST_MYSQL_USERNAME=root TEST_MYSQL_PASSWORD='' make test-e2e-mysql +``` + +Setup a pgsql database inside docker +``` +docker run -e "POSTGRES_DB=test" -p 5432:5432 --rm --name pgsql postgres:latest #(just ctrl-c to stop db and clean the container) +``` +Start tests based on the database container +``` +TEST_PGSQL_HOST=localhost:5432 TEST_PGSQL_DBNAME=test TEST_PGSQL_USERNAME=postgres TEST_PGSQL_PASSWORD=postgres make test-e2e-pgsql +``` + +### Running individual tests + +Example command to run `example.test.e2e.js` test file: + +> **Note** +> Unlike integration tests, this filtering is at the file level, not function + +For SQLite: + +``` +make test-e2e-sqlite#example +``` + +### Visual testing + +> **Warning** +> This is not currently used by most Forgejo contributors. +> Your help to improve the situation and allow for visual testing is appreciated. + +Although the main goal of e2e is assertion testing, we have added a framework for visual regress testing. If you are working on front-end features, please use the following: + - Check out `main`, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1` to generate outputs. This will initially fail, as no screenshots exist. You can run the e2e tests again to assert it passes. + - Check out your branch, `make clean frontend`, and run e2e tests with `VISUAL_TEST=1`. You should be able to assert you front-end changes don't break any other tests unintentionally. + +VISUAL_TEST=1 will create screenshots in tests/e2e/test-snapshots. The test will fail the first time this is enabled (until we get visual test image persistence figured out), because it will be testing against an empty screenshot folder. + +ACCEPT_VISUAL=1 will overwrite the snapshot images with new images. + + +## Tips and tricks + +If you know noteworthy tests that can act as an inspiration for new tests, +please add some details here. + +### Run tests very selectively + +Browser testing can take some time. +If you want to iterate fast, +save your time and only run very selected tests. +Use only one browser. + +### Skip Safari if it doesn't work + +Many contributors have issues getting Safari (webkit) +and especially Safari Mobile to work. + +At the top of your test function, you can use: + +~~~javascript +test.skip(workerInfo.project.name === 'Mobile Safari', 'Unable to get tests working on Safari Mobile.'); +~~~ + +### Don't forget the formatting. + +When writing tests without modifying other frontend code, +it is easy to forget that the JavaScript test files also need formatting. + +Run `make lint-frontend-fix`. + +### Define new repos + +Take a look at `declare_repos_test.go` to see how to add your repositories. +Feel free to improve the logic used there if you need more advanced functionality +(it is a simplified version of the code used in the integration tests). + +### Accessibility testing + +If you can, perform automated accessibility testing using +[AxeCore](https://github.com/dequelabs/axe-core-npm/blob/develop/packages/playwright/README.md). + +Take a look at `shared/forms.js` and some other places for inspiration. + +### List related files coverage + +If you think your playwright tests covers an important aspect of some template, CSS or backend files, +consider adding the paths to `.forgejo/workflows/e2e.yml` in the path filter. + +It ensures that future modifications to this file will be tested as well. + +Currently, we do not run the e2e tests on all changes. -- cgit v1.2.3