Q2Developer Pipeline

The Q2Developer pipeline runs on code.q2developer.com and provides CI/CD for SDK projects. The one documented here is the primary Q2 Developer Pipeline and its derivative Q2 Developer Pipeline Child. It automates building, testing, scanning, and deploying SDK projects, enforcing code quality standards, security checks, and license compliance before code reaches production.

Gitlab Pipelines are broken into two types: Merge and Branch. Merge is kicked off when a Merge Request is created. Branch is kicked off when code is changed in a branch. As such, the flow is often to make a merge request (which will run the Merge pipeline), then after the code merge is complete, the branch pipeline will run on the new destination branch.

Note

We are moving clients in batches. If you have not yet moved, here’s a reference to the Legacy Pipeline Steps

Note

For updates over time, check out our Pipeline Changelog

Merge Pipeline

Stages

Build
- q2 check
Test
- q2 test
- q2 lint
Code Review
Manual Override
- Review buddy (pass)
- Review buddy (fail)

Branch Pipeline

Overrides

We currenly only have a single way to to override the pipeline. More planned for the future. These configurations happen in your repo’s pyproject.toml

In pyproject.toml:

[tool.q2.pipeline]
allow_lint_failure = false

Stage Descriptions

Build

q2 check

Checks built into the SDK version. (Runnable locally with q2 check). For advanced use, check out Adding Custom Checks.

Test

q2 test

Required To Pass: True

Unit tests. (Runnable locally with q2 test). Check out our guide to Testing Code for ideas.

q2 lint

Required To Pass: False

Code linting. (Runnable locally with q2 lint). Check out our guide to Linting for ideas.

Code Review

bandit-security-scan

Required To Pass: False

Bandit is an open source security linter for Python. It scans Python code for common security issues such as hardcoded passwords, SQL injection, and insecure function calls.

Details in the Bandit Security Scan doc.

js-license-scan-restricted

Required To Pass: True

See License Scanning for details

js-license-scan-warnings

Required To Pass: False

See License Scanning for details

pip-license-scan-restricted

Required To Pass: True

See License Scanning for details

pip-license-scan-warnings

Required To Pass: False

See License Scanning for details

review buddy

Required To Pass: True

Runs the review_buddy static application security testing (SAST) tool written specifically for testing SDK repositories. More details called out in the Review Buddy doc.

sql-fluff-scan

Required To Pass: False

SQLFluff is an open source SQL linter. The pipeline scan focuses on syntax issues that would prevent installation, rather than style enforcement.

Details in the SQLFluff doc

Trigger Review Buddy Gate Tasks

This exists because of how gitlab pipelines must be built. Purely convenience.

Manual Override

Review Buddy (pass)

If Review Buddy passes, this step will run to mark it, but no action is necessary.

Review Buddy (fail)

If Review Buddy fails, you can either correct the error or request that we (Q2 team) allow it anyway. To correct the errors, the output of review_buddy itself should be helpful in the pipeline step.

Mirror

Code.q2developer.com hosts your code and is visible outside of the Q2 infrastructure. However, most of our deployment tools and APIs are inside the firewall. This mirrors the code to a place we can see it internally. However, know that the version on code.q2developer.com is the source of truth.

Dockerize

To run your service and all its endpoints within the Q2 hosting network, we first turn the code repository into a docker image. This effectively lays your code on top of a recent Ubuntu build, sprinkles in a few Q2 certificate goodies, and marks it as a deployable artifact. This will run on any branch contained in the CI/CD variable $BRANCHES_TO_DOCKERIZE. By default, this is master, main, production, but can be run on other branches per request (please enter a support ticket).

Scan

Many scans happen in the above Code Review step, this is a final check at the moment before deployment that there are 0 fixable critical vulnerabilities and <5 high vulnerabilities in the built image.

Deploy

This turns the container on and points it to the environment specified by the $VAULT_KEY CI/CD variable.

This will run on any branch contained in the CI/CD variable $BRANCHES_TO_DEPLOY. By default, this is production.