Continuous Integration (CI)

When you submit a pull request, some automated checks are ran for your code by the ClickHouse continuous integration (CI) system. This happens after a repository maintainer (someone from ClickHouse team) has screened your code and added the can be tested label to your pull request. The results of the checks are listed on the GitHub pull request page as described in the GitHub checks documentation. If a check is failing, you might be required to fix it. This page gives an overview of checks you may encounter, and what you can do to fix them. If it looks like the check failure is not related to your changes, it may be some transient failure or an infrastructure problem. Push an empty commit to the pull request to restart the CI checks:

git commit --allow-empty
git push

If you are not sure what to do, ask a maintainer for help.

Merge with master

Verifies that the PR can be merged to master. If not, it will fail with a message Cannot fetch mergecommit. To fix this check, resolve the conflict as described in the GitHub documentation, or merge the master branch to your pull request branch using git.

Docs check (Mintlify)

Validates the Mintlify documentation, internal links and anchors, redirects, snippet imports, and changelogs. External-link failures are reported as warnings. It also rejects direct edits to generated regions and read-only documentation copies. Update the structured documentation in the source registration instead; intentional generated updates must carry the pr-autogenerated-docs label. If the check fails after a documentation change, open its report and look for ERROR and WARNING messages.

Description check

Check that the description of your pull request conforms to the template PULL_REQUEST_TEMPLATE.md. You have to specify a changelog category for your change (e.g., Bug Fix), and write a user-readable message describing the change for CHANGELOG.md

Docker image

Builds the ClickHouse server and keeper Docker images to verify that they build correctly.

Official docker library tests

Runs the tests from the official Docker library to verify that the clickhouse/clickhouse-server Docker image works correctly. To add new tests, create a directory ci/jobs/scripts/docker_server/tests/$test_name and the script run.sh there. Additional details about the tests can be found in the CI jobs scripts documentation.

Marker check

This check means that the CI system started to process the pull request. When it has ‘pending’ status, it means that not all checks have been started yet. After all checks have been started, it changes status to ‘success’.

Style check

Performs various style checks on the code base. Each sub-check below corresponds to a testname in ci/jobs/check_style.py and can be run individually with --test <name> (see below). Regex-based C++ style checks via check_cpp.sh. If it fails, fix the issues according to the code style guide. Flags double spaces after commas in C++ that are not part of column alignment. Forbids catch (...) outside of destructors, main, and fuzzer entry points where swallowing an unknown exception is unsafe. Lints YAML workflow files under .github/ using .yamllint. Validates XML files under tests/ and programs/. Checks stateless tests: queries filtering on event_date must use >= yesterday() rather than today() (to avoid flakiness around midnight), and test file names must not contain fail. Flags large gaps in stateless test numbering (tests/queries/0_stateless/<NNNNN>_*). Detects broken symlinks in the repository. Miscellaneous repository checks via various_checks.sh: queries on system.query_log / system.parts / etc. must filter by currentDatabase, Replicated*MergeTree ZooKeeper paths must include a per-test prefix, integration test directories must have __init__.py, no UTF BOMs, no executable bits on source/data files, no :latest tags on third-party docker-compose images, and more.

Running the style check job locally

The entire Style Check job can be run locally in a Docker container with:

python -m ci.praktika run "Style check"

To run a specific check (e.g., cpp check):

python -m ci.praktika run "Style check" --test cpp

These commands pull the clickhouse/style-test Docker image and run the job in a containerized environment. No dependencies other than Python 3 and Docker are required.

Running stateless tests

A locally installed ClickHouse with default settings may work for specific test cases, but cannot run all test queries correctly. In CI, each job installs a specific ClickHouse configuration (e.g., S3 storage, Parallel Replicas) which can be cumbersome to reproduce manually. To avoid this, you can reproduce any CI job locally using the same orchestration as CI — no manual configuration needed.

Prerequisites

Python 3 (standard library only)
Docker

Install Docker on Ubuntu if needed and re-login:

sudo apt-get update
sudo apt-get install docker.io
sudo usermod -aG docker "$USER"
sudo tee /etc/docker/daemon.json <<'EOF'
{
  "ipv6": true,
  "ip6tables": true
}
EOF
sudo systemctl restart docker

Run a CI Job Locally

Pick any job name from a CI report and run it locally:

python -m ci.praktika run "<JOB_NAME>"

Always quote the job name exactly as it appears in the CI report (it may contain spaces and commas), e.g.: "Stateless tests (amd_debug, parallel)". This sets up the same ClickHouse configuration and runs the same tests as in CI.
The architecture and build type in the job name (e.g., amd_debug) are CI-specific labels. When running locally, they have no effect — the job will use whatever binary you provide, on whatever architecture you are running. The job name only determines the ClickHouse configuration and the test set (unless overridden with --test).
In CI, functional tests are split into batches for better resource utilization. For example, "Stateless tests (amd_debug, parallel)" and "Stateless tests (amd_debug, sequential)" together cover the entire scope: parallel-safe tests run concurrently, and the rest run sequentially. The split reduces total CI time by maximizing parallelism where possible. To reproduce the full test scope locally, run both batches.
There is also a "Fast test" CI job that runs a limited scope of functional tests to verify basic ClickHouse functionality — it uses a build without all optional modules and is the quickest way to catch regressions. You can run it locally the same way. Place your ClickHouse binary in one of the default search paths (./ci/tmp/clickhouse, ./build/programs/clickhouse, or ./clickhouse) — otherwise the job will attempt to build ClickHouse first:
python -m ci.praktika run "Fast test"

Run Specific Tests Within a CI Job

With --test, the job prepares an identical ClickHouse setup as used in CI but runs only the selected tests:

python -m ci.praktika run "Stateless tests (amd_debug, parallel)" \
  --test 00001_select1

You can pass multiple test names:

python -m ci.praktika run "Stateless tests (amd_debug, parallel)" \
  --test 00001_select1 00002_log_and_exception_messages_formatting

Tip: If any ClickHouse configuration is acceptable and you just need to run specific tests, use the alias functional instead of the full job name:
python -m ci.praktika run functional --test 00001_select1

Additional Customization Options

--path PATH — custom path to the ClickHouse binary. By default, the runner searches in order: ./ci/tmp/clickhouse, ./build/programs/clickhouse, ./clickhouse.
--count N — repeat each test N times.
--workers N — override the automatic calculation of parallel workers derived from machine capacity.

Build check

Builds ClickHouse in various configurations for use in further steps.

Running Builds Locally

The build can be run locally in a CI-like environment using:

python -m ci.praktika run "<BUILD_JOB_NAME>"

No dependencies other than Python 3 and Docker are required.

Available Build Jobs

The build job names are exactly as they appear in the CI Report: AMD64 Builds:

Build (amd_debug) - Debug build with symbols
Build (amd_release) - Optimized release build
Build (amd_asan) - Address Sanitizer build
Build (amd_tsan) - Thread Sanitizer build
Build (amd_msan) - Memory Sanitizer build
Build (amd_ubsan) - Undefined Behavior Sanitizer build
Build (amd_binary) - Quick release build without Thin LTO
Build (amd_compat) - Compatibility build for older systems
Build (amd_musl) - Build with musl libc
Build (amd_darwin) - macOS build
Build (amd_freebsd) - FreeBSD build

ARM64 Builds:

Build (arm_release) - ARM64 optimized release build
Build (arm_asan) - ARM64 Address Sanitizer build
Build (arm_coverage) - ARM64 build with coverage instrumentation
Build (arm_binary) - ARM64 Quick release build without Thin LTO
Build (arm_darwin) - macOS ARM64 build
Build (arm_v80compat) - ARMv8.0 compatibility build

Other Architectures:

Build (ppc64le) - PowerPC 64-bit Little Endian
Build (riscv64) - RISC-V 64-bit
Build (s390x) - IBM System/390 64-bit
Build (loongarch64) - LoongArch 64-bit

If the job succeeds, build results will be available in the <repo_root>/ci/tmp/build directory. Note: For builds not in the “Other Architectures” category (which use cross-compilation), your local machine architecture must match the build type to produce the build as requested by BUILD_JOB_NAME.

Example

To run a local debug build:

python -m ci.praktika run "Build (amd_debug)"

If the above approach does not work for you, use the cmake options from the build log and follow the general build process.

Functional stateless tests

Runs stateless functional tests for ClickHouse binaries built in various configurations — release, debug, with sanitizers, etc. Look at the report to see which tests fail, then reproduce the failure locally as described here. Note that you have to use the correct build configuration to reproduce — a test might fail under AddressSanitizer but pass in Debug. Download the binary from CI build checks page, or build it locally.

Integration tests

Runs integration tests.

Bugfix validate check

Checks that either a new test (functional or integration) or there some changed tests that fail with the binary built on master branch. This check is triggered when pull request has “pr-bugfix” label.

Stress test

Runs stateless functional tests concurrently from several clients to detect concurrency-related errors. If it fails:

Fix all other test failures first;
Look at the report to find the server logs and check them for possible causes of error.

Compatibility check

Checks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help.

AST fuzzer

Runs randomly generated queries to catch program errors. If it fails, ask a maintainer for help.

Performance tests

Measure changes in query performance. This is the longest check that takes just below 6 hours to run. The performance test report is described in detail here.

Revert CI regressions

This is not a check on your pull request: it runs on master every hour, and it may revert a pull request that has already been merged. The job takes the failing tests the CI database recorded for master over the last 24 hours and groups them by test name, across every check the test failed in. The same test failing in the debug and in the tsan build is one failure with one cause to look for, and the checks it appeared in go to the investigation as evidence: a change that breaks a test usually breaks it in several builds at once. Failures that are not attributed to any test, such as a build failure or a job that ran out of time, are left out: “why does this check fail” has no single answer to revert on. Rows the test harness writes about the whole script under a test-like name, such as Test script failed or Server died, are rejected the same way. A test that failed on more than one master commit is handed to an AI agent, which is given the repository with the full history of master and read-only access to the CI database, and answers a single question: was this failure introduced by a recently merged pull request, and which one. The agent holds no GitHub credential and no way to mint one — it runs as an unprivileged user of its own, with an empty environment and the cloud credential endpoints firewalled off for that user — and it works in a disposable clone of the repository rather than in the job’s own checkout, so nothing it concludes — and nothing it could leave behind — can reach GitHub except through the checks below. The threshold counts commits rather than failing rows, so one bad commit that fails in three builds is still a single occurrence and is not acted on. It also counts them per failure mode: the recorded outputs are fingerprinted with the volatile parts (addresses, timestamps, random database names) normalized away, and a test whose name spans two different causes — a regression on one commit and an unrelated flake on another — is not a repeated failure, so nothing is investigated until one cause repeats on its own. Only an unambiguous answer leads to an action. When the agent reports a regression with high confidence, and the named pull request passes the safety checks (merged into master within the last three days, not a revert itself, not already reverted, and the revert applies cleanly), the job reverts it, merges the revert immediately without waiting for checks, and opens a draft pull request titled Reapply "..." that reintroduces the change. A regression verdict has to name both the pull request and the master commit it came in on, and the two have to agree: the job checks the number against GitHub’s record of which merge commit that pull request produced, and acts on neither if they disagree. Nothing is reverted once the failure is gone: a failure stays in the observation window for a whole day after it stopped, so right before reverting the job asks the CI database again, and a failure absent from the newest master commits that every affected check exercised is recorded as already fixed and left alone. The newest commits by the branch’s own history, not by when their checks ran — an old commit whose check started late must not be read as fresh green evidence. Absence rather than a pass, because most of what this job investigates has no passing row to find: a logical error or a hung check is recorded under the text of the failure itself, and only when it happens. A check counts as having exercised a commit only when a run of it got through its tests: a run that aborted partway — recorded by the harness as Test script failed or Server died next to the test rows it did produce — ran some tests, not necessarily this one, and its silence about the failure is not evidence, while a rerun of the same check that completed on the same commit is. How much absence counts depends on how often the failure hits — a couple of clean commits mean nothing for something that fails one run in a hundred, so the requirement is more than the longest the failure is on record for going quiet between its own occurrences. When the question cannot be answered at all — a check the failure was seen in no longer reports under that name, or the commit history since the failure started is longer than the query returns — that is recorded too and nothing is reverted. At most two pull requests are reverted per run. If your pull request was reverted:

The revert pull request explains what fails and why the change was blamed. If the attribution is wrong, say so there and bring the change back.
The Reapply "..." draft pull request holds your change unchanged. Fix the failure on that branch, mark it ready for review, and let it go through normal CI.

Every investigation is recorded in the checks_investigated table of the CI database, including the ones that revert nothing. The values are carried over from checks as they were recorded there, so the two tables join back together — on test_name directly, with has(check_names, check_name) and has(commit_shas, commit_sha) for the columns that collect several checks rows into an array, and with offending_pull_request_number = pull_request_number for the blamed pull request — and the history of what the job looked at, what it concluded and what it did is queryable on play.clickhouse.com:

SELECT investigation_time, test_name, check_names, failure_count, commit_count, verdict, confidence, action, explanation
FROM checks_investigated
WHERE investigation_time >= now() - INTERVAL 7 DAY
ORDER BY investigation_time DESC;

The job is implemented in ci/jobs/revert_ci_regressions.py and runs as part of the Hourly workflow. Running it with --dry-run investigates and evaluates every guard but changes nothing: no table, no rows, no branch, no pull request, no merge; the rows it would have written are printed instead. A separate workflow, .github/workflows/revert_broken_prs.yml, reverts merges that landed while their own CI was red; both use the same revert-<pull request number> branch name, so a pull request is never reverted twice. A revert started by hand counts too: the job stands down when the revert is already on master, when a branch named revert-<pull request number> or revert-<pull request number>-<branch> (what the Revert button on GitHub creates) exists, or when a pull request from such a branch is open or merged.

Introduction

Building

Native protocol

Contributing

Integration development

Roadmap

Merge with master

Docs check (Mintlify)

Description check

Docker image

Official docker library tests

Marker check

Style check

Running the style check job locally

Running stateless tests

Prerequisites

Run a CI Job Locally

Run Specific Tests Within a CI Job

Additional Customization Options

Build check

Running Builds Locally

Available Build Jobs

Example

Functional stateless tests

Integration tests

Bugfix validate check

Stress test

Compatibility check

AST fuzzer

Performance tests

Revert CI regressions

​Merge with master

​Docs check (Mintlify)

​Description check

​Docker image

​Official docker library tests

​Marker check

​Style check

​Running the style check job locally

​Running stateless tests

​Prerequisites

​Run a CI Job Locally

​Run Specific Tests Within a CI Job

​Additional Customization Options

​Build check

​Running Builds Locally

​Available Build Jobs

​Example

​Functional stateless tests

​Integration tests

​Bugfix validate check

​Stress test

​Compatibility check

​AST fuzzer

​Performance tests

​Revert CI regressions

Merge with master

Docs check (Mintlify)

Description check

Docker image

Official docker library tests

Marker check

Style check

Running the style check job locally

Running stateless tests

Prerequisites

Run a CI Job Locally

Run Specific Tests Within a CI Job

Additional Customization Options

Build check

Running Builds Locally

Available Build Jobs

Example

Functional stateless tests

Integration tests

Bugfix validate check

Stress test

Compatibility check

AST fuzzer

Performance tests

Revert CI regressions