Skip to content

Contributing

This project is a community effort and lives off your contributions, be it bug reports, feature requests, new plugins, discussions, documentation or code. It follows the contributing guide of ELIXIR On-Cloud, which covers getting started, issues, branches and pull requests, commit messages and pull request titles, the code of conduct and the security policy. This page covers what is specific to this repository; for a new plugin, see Writing a plugin.

Development

Development needs uv (which installs a suitable Python, 3.11 or later, if needed) and Git. The repository root is a uv workspace: one environment (.venv/) with every package installed in editable mode, plus the development tools (dependency group dev).

git clone https://github.com/elixir-cloud-aai/foca-plugins.git
cd foca-plugins
uv sync  # all packages and the dev tools, as locked in `uv.lock`
uv run pre-commit install  # run the checks on every commit
uv run pre-commit run --all-files  # ruff, mypy, semgrep, ast-grep, conventions
uv run pre-commit run --all-files --hook-stage manual gitleaks-all  # secrets
uv run pytest  # all tests: the plugins', the scripts' and the docs hooks'

Run pre-commit in the environment (uv run): the mypy, ast-grep and conventions hooks use the installed packages. The first run sets up the other hooks, which takes a few minutes.

CI tests each plugin on its own, in an environment with only that plugin and the dev tools (uv sync --package <distribution> --group dev), so that a plugin can't rely on another's dependencies; the commands of one plugin's checks are in Writing a plugin. Its change detection (python -m scripts.packages changed <base> <head>) runs the tests of the plugins that a pull request affects: changes of a plugin's directory affect that plugin; those of the meta distribution and of shared files (the root pyproject.toml, uv.lock, workflows, scripts, rules) affect all plugins; those of the docs affect none. The gate CI passed is the only required check.

When you change dependencies, run uv lock (or uv sync) and commit uv.lock; CI fails if it is not up to date. Project-specific rules are in .semgrep/ (security) and .ast-grep/ (e.g., no framework imports in plugins); suppress a false positive with a comment with the reason on its line: # noqa: <code> (ruff), # type: ignore[<code>] (mypy), # nosemgrep: <rule-id>, # ast-grep-ignore: <rule-id> or # gitleaks:allow.

Dependabot updates the dependencies, the GitHub Actions and the pre-commit hooks. Update these pinned tools by hand from time to time, as it doesn't:

  • the uv version (UV_VERSION in ci.yml and release.yml; Dependabot bumps only the action astral-sh/setup-uv);
  • the build backend (build-constraint-dependencies in the root pyproject.toml);
  • cffconvert (ci.yml, job lint).

twine, the release workflow's other tool, is locked in uv.lock (dependency group release), so Dependabot updates it.

End-to-end tests

tes.data.crypt4gh changes what a TES backend runs, so it also has an end-to-end test against a real backend (plugins/tes/data/crypt4gh/e2e/check.sh, see the plugin's README): proTES, from a release image with the plugin of the checkout installed, sends tasks with Crypt4GH inputs, served over HTTP, to Funnel, which runs their executors as containers through the Docker socket of the host, in Docker Compose. CI runs it in the job e2e ("End-to-end test") whenever the plugin is affected; it needs no credentials, so it runs for pull requests from forks, too. To run it locally, you need Docker with Compose, uv, curl and jq:

plugins/tes/data/crypt4gh/e2e/check.sh

It builds the decryption image and the plugin's wheel, starts the stack, runs the tasks, and removes the stack, the task containers and its work directory when it ends, but not the images; if something remains, a test that passed fails. Only proTES is published (on 127.0.0.1): the other services, Funnel's API among them, which has no authentication, are on an internal Compose network that the host can't reach. Settings (environment variables):

  • E2E_PORT: the port of proTES on 127.0.0.1 (default 18080). It also names the Compose project (foca-plugins-e2e-<port>), so that runs of the same checkout with different ports and work directories can run side by side. Runs of different checkouts can't, as they share the tags of the images that they build.
  • E2E_WORK_DIR: the work directory, a normalized absolute path that the Docker daemon can mount (default /tmp/foca-plugins-e2e). The test creates it and removes it at the end. It refuses an existing directory that an earlier run didn't create (without its marker file, of another user, or a symbolic link), a directory inside the work directory of another run, and paths with . or .. components or :.
  • E2E_WAIT: the seconds to wait for each task (default 180).

Pass an image to test it instead of building one (e.g., check.sh crypt4gh-decrypt:dev). With a bad work directory, or without Docker, Docker Compose, uv, curl or jq, it exits with status 2 before starting anything.

Dependabot updates the base images of the test's Dockerfiles (proTES, uv, Funnel, the Docker CLI), and the minor and patch versions of the images of its compose.yaml (MongoDB, RabbitMQ; their major versions by hand). The image of the tasks' own executors (alpine, in check.sh) is updated by hand.

Documentation

This documentation is built with MkDocs and Material for MkDocs, from docs/ and mkdocs.yml, by Read the Docs (.readthedocs.yaml) and in CI; warnings, e.g., of broken links, fail the build. The home page includes sections of README.md, and the plugin catalogue is generated from the plugins' READMEs (docs/hooks.py).

uv sync --group docs
uv run mkdocs build --strict  # into `site/`, ignored by Git
uv run mkdocs serve

Commit messages and pull request titles

  • The scope of a change of one plugin is its path below plugins/, e.g., feat(tes/routing/distance): ...; that of the meta distribution is meta.

Use feat, fix, perf, refactor and style only for changes of what is published: the plugins (plugins/) and the meta distribution (meta/).

Citation metadata

The repository is cited with the metadata in CITATION.cff (Citation File Format), which GitHub ("Cite this repository") uses. It has no version, as each distribution has its own. Its authors follow the citation metadata rule of the contributing guide. CI validates the file (cffconvert --validate).

Python guidelines

Plugins support the Python versions that FOCA supports (3.11 and later; requires-python), which CI tests (3.11, 3.13 and 3.15). Most of the following is checked by pre-commit and CI; the rest is checked in reviews.

  • Packaging: each package's metadata and dependencies are in its pyproject.toml; the tool configuration and the development dependencies (dependency groups dev and docs) are in the root pyproject.toml. Don't add setup.py or requirements files.
  • Formatting and linting: ruff formats the code (ruff format, compatible with Black, 88 characters per line) and lints it (ruff check) with the rule set shared by FOCA and FOCA apps: pycodestyle, Pyflakes, isort, flake8-bugbear, flake8-comprehensions, flake8-simplify, Pylint, Ruff's own rules, pyupgrade, bandit (security), logging format and pydocstyle, also for the tests (root pyproject.toml, where each ignored rule says why). Prefer explicit checks to suppressions.
  • Docstrings: Google-style docstrings at all levels: packages, modules, classes, methods and functions, also private ones and tests, with a one-line summary and, where applicable, the sections Args, Returns (or Yields), Raises and, for classes, Attributes. The first line of a plugin class's docstring is its title for clients, the rest its description.
  • Type hints: complete type hints for all functions and methods (also of the tests), and generic types with their parameters (e.g., dict[str, Any]). mypy checks them (disallow_untyped_defs, disallow_any_generics), through python -m scripts.packages mypy, which puts the plugins' src/ directories on the search path.
  • Tests: pytest, with coverage measured by coverage (branches included): keep it at 100% of each plugin's src/ and of scripts/ and docs/hooks.py, which is enforced (fail_under in the root pyproject.toml, and Codecov's checks, codecov.yml). Add tests that fail without your change.
  • Further checks: ast-grep (rules in .ast-grep/, with tests), semgrep (security rules in .semgrep/), gitleaks (secrets), actionlint (workflows), pip-audit (known vulnerabilities of each plugin's dependencies, in CI), and the package conventions (python -m scripts.checks check).