> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-revert-104359-revert-104251-parquet-single.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Managed services

> Create, watch, and retire ClickHouse services that ClickHouse Cloud operates in your cluster through the connector executor

export const Image = ({img, alt, size = "lg", background}) => {
  const normalizedSize = ["sm", "md", "lg"].includes(size) ? size : "lg";
  const backgroundColor = background === "white" ? "white" : background === "black" ? "rgb(31 31 28)" : undefined;
  return <div className={`ch-image-${normalizedSize}`}>
      <Frame>
        <img src={img} alt={alt} style={{
    backgroundColor
  }} />
      </Frame>
    </div>;
};

export const PrivatePreviewBadge = () => {
  return <div className="privatePreviewBadge">
            <div className="privatePreviewIcon">
            <svg width="16" height="16" viewBox="0 0 16 16" fill="none" xmlns="http://www.w3.org/2000/svg">
                <path d="M5.33301 6.66667V4.66667V4.66667C5.33301 3.194 6.52701 2 7.99967 2V2C9.47234 2 10.6663 3.194 10.6663 4.66667V4.66667V6.66667" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path d="M8.00033 9.33337V11.3334" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
                <path fillRule="evenodd" clipRule="evenodd" d="M11.333 14H4.66634C3.92967 14 3.33301 13.4033 3.33301 12.6666V7.99996C3.33301 7.26329 3.92967 6.66663 4.66634 6.66663H11.333C12.0697 6.66663 12.6663 7.26329 12.6663 7.99996V12.6666C12.6663 13.4033 12.0697 14 11.333 14Z" stroke="currentColor" strokeLinecap="round" strokeLinejoin="round" />
            </svg>
        </div>
            {'Private preview'}
        </div>;
};

<PrivatePreviewBadge />

In managed mode the connector runs a third component, the executor. ClickHouse Cloud operates ClickHouse services in your Kubernetes cluster through it. This page covers creating a service, reading its status, what ClickHouse Cloud does with it, and how it is retired.

<h2 id="what-managed-mode-does">
  What managed mode does
</h2>

The executor is a daemon in the same `clicklink` binary as the scraper and the troubleshooter. It holds an outbound WebSocket channel to your connector endpoint and receives lifecycle commands from ClickHouse Cloud. It applies each command to the one Kubernetes cluster it is configured for and reports the outcome over that channel. Like the other components it makes only outbound connections: ClickHouse Cloud never connects into your cluster, and the executor's local API binds to loopback.

Managed mode is available on Amazon EKS with S3 storage during the private preview. Your account team enables it when registering your environment. `init` refuses a kube context that does not point at an EKS cluster.

Each service needs three cloud resources you own: a data bucket, a backup bucket, and an IAM role the ClickHouse pods assume to reach them. You create them with your own credentials before the service exists and remove them after it is gone. The executor holds no credentials for your buckets or IAM and never deletes data. Its only cloud calls go to Amazon ECR and STS. It logs in to the registry when a platform sync or create pulls a chart, and checks the images under the read-only pull role during a platform update. It makes no S3 or IAM calls. For each service it creates a Kubernetes `Service` of type LoadBalancer, which your cluster's load-balancer controller realizes as an internal NLB in your account.

On a VM the executor runs as the `clicklink-executor` systemd unit next to the other two. On Kubernetes it is a single-replica Deployment in the connector namespace. It serves health and metrics on port 8086 and its local API on `127.0.0.1:9999`.

<h2 id="enable-managed-mode">
  Enable managed mode
</h2>

You choose managed mode when you enroll: pass `--managed` to `clicklink clctl init`, or answer the prompt on a terminal. On a VM, `init` takes the cluster from this host's kubeconfig (`--cluster-name` when it holds more than one EKS cluster) and grants the executor's cluster-wide access inline. On Kubernetes, pass `--egress-cidrs` with the CIDRs of your connector endpoint so the chart stages its default-deny NetworkPolicy enabled.

The install itself is unchanged. See [onboarding](/products/bring-your-own-cloud/connector/onboarding#install-and-enroll) for the full flow and the [CLI reference](/products/bring-your-own-cloud/connector/reference/cli#init) for the flags.

<h2 id="create-a-service">
  Create a service
</h2>

Service creation through the connector endpoint is enabled per environment; check with your account team before you start.

Creating a service takes two commands. `prepare` creates everything on your side; `instances create` submits the create to ClickHouse Cloud through your connector endpoint. ClickHouse Cloud renders the service definition and sends the create to the executor over its outbound channel. The executor applies it to your cluster and reports back.

Only `prepare` needs AWS credentials: it creates the S3 buckets and the IAM role. `instances create` needs the connector's configuration and credentials, and reaches the executor's local API only for `--wait`. On a VM, run both as root on the connector host. On Kubernetes, run both from a workstation with a kube context for the managed cluster:

* `prepare` needs a copy of the connector configuration (`--config`), a `kubectl port-forward` to the executor's local API, and a writable `--output-dir`. It checks the service name against the executor and refuses to run when it cannot reach it.
* When this host lacks the credential files the configuration names, `instances create` reads them from the `clicklink-hmac` and `clicklink-mtls` Secrets and announces each read. Add `--connector-namespace` when the connector namespace is not `clicklink`. Only `instances create` has this fallback.

<Image img="https://mintcdn.com/private-7c7dfe99-revert-104359-revert-104251-parquet-single/FjaG6PmLNcoIR7ck/images/cloud/reference/byoc-connector-service-lifecycle.svg?fit=max&auto=format&n=FjaG6PmLNcoIR7ck&q=85&s=622304f6db9845d3e868ea67e1521414" size="lg" alt="ClickHouse Connector service lifecycle in managed mode" width="1320" height="870" data-path="images/cloud/reference/byoc-connector-service-lifecycle.svg" />

<Steps>
  <Step title="Prepare the service" id="prepare">
    <Tabs>
      <Tab title="Kubernetes" id="prepare-kubernetes">
        ```bash theme={null}
        CONNECTOR_NAMESPACE='clicklink'   # the connector namespace you chose at init
        kubectl -n "${CONNECTOR_NAMESPACE}" get configmap clicklink-connector-executor \
          -o jsonpath='{.data.config\.yaml}' > clicklink-config.yaml
        kubectl -n "${CONNECTOR_NAMESPACE}" port-forward deployment/clicklink-connector-executor 9999:9999 &
        clicklink clctl executor prepare --config clicklink-config.yaml --output-dir ./clicklink-access
        ```

        Keep the output directory: it holds the create body and the name record that `instances create` and any retry read.
      </Tab>

      <Tab title="Linux VM" id="prepare-vm">
        ```bash theme={null}
        sudo clicklink clctl executor prepare --config /etc/clicklink/config.yaml
        ```
      </Tab>
    </Tabs>

    The command runs four steps in order and stops at the first failure:

    1. **Name.** Picks a service name, or validates the one you pass with `--instance <name>`. A generated name is recorded in `<output-dir>/_prepare/<eks-cluster-name>.name` and resumed by the next run, so a retry reuses the first run's buckets and role. `--new-name` picks another. The command refuses a name the executor still holds, or whose namespace already contains a ClickHouse cluster.
    2. **Storage.** Creates the data and backup buckets and the IAM role `CH-S3-<name>-<region>-00-Role`. The default bucket names are `<cluster>-clickhouse-data-<rand>` and `<cluster>-clickhouse-backup-<rand>`; `--data-bucket` and `--backup-bucket` override them. With `--role-arn` it verifies a role you bring and writes no IAM.
    3. **Grant and apply.** On a VM, renders the executor's access bundle for the service namespace (`ns-<name>`), applies its RBAC, and registers the service in the executor's registry. On Kubernetes this step is skipped: the executor runs as its pod ServiceAccount and registers the service itself when the create arrives.
    4. **Summary.** Mints the `default` user's password and writes the create body to `<output-dir>/_prepare/<name>.create.json` (mode `0600`; it carries only the password's hashes). Prints the next command on a `Next:` line.

    <Warning>
      `prepare` prints the `default` user's password once, on stderr. Neither the create body nor `--output json` contains it, and ClickHouse Cloud receives only its hashes. Store it before moving on. A re-run that finds the create body keeps the hashes already written, and says so.
    </Warning>

    Pass `--context <name>` when your kubeconfig holds several contexts; the context must point at the EKS cluster named in `executor.cluster` in the config. `--dry-run` runs every step without creating or writing anything. The hashing step uses SHA-1, which Go refuses under `GODEBUG=fips140=only`; run `prepare` on a host without that setting.
  </Step>

  <Step title="Submit the create" id="create">
    <Tabs>
      <Tab title="Kubernetes" id="create-kubernetes">
        ```bash theme={null}
        clicklink clctl instances create \
          --from-prepare ./clicklink-access/_prepare/<name>.create.json \
          --config clicklink-config.yaml \
          --wait
        ```

        Leave the port-forward from `prepare` open: `--wait` polls the executor through it.
      </Tab>

      <Tab title="Linux VM" id="create-vm">
        ```bash theme={null}
        sudo clicklink clctl instances create \
          --from-prepare /etc/clicklink/access/executor/_prepare/<name>.create.json \
          --config /etc/clicklink/config.yaml \
          --wait
        ```
      </Tab>
    </Tabs>

    The command posts the prepared body to your connector endpoint with the connector's own credentials. It prints `created <spoken-name> (state provisioning)` and a `watch:` hint. `<spoken-name>` is the name ClickHouse Cloud assigned (for example `amberaws-kq-42`), not the service name you prepared. The `watch:` hint and every `clctl` command use your service name.

    A retry with the same inputs is safe. The idempotency key is derived by default from your environment and the service name, so resubmitting returns the first create.

    `--wait` polls the executor's local API every 10 seconds until the service is `running`, for up to `--wait-timeout` (default `30m`). It fails fast, with the recorded error, when the executor records a failed create for the name or the service turns `terminating`, `terminated`, or `stale`.
  </Step>

  <Step title="Verify" id="verify">
    ```bash theme={null}
    clicklink clctl instances get --name <name> --cluster <eks-cluster-name>
    ```

    The service is ready when its `status` is `running`. Its `default` user takes the password `prepare` printed. `--cluster` can also come from the `CLCTL_CLUSTER` environment variable.
  </Step>
</Steps>

<h2 id="status">
  Status
</h2>

The executor answers status questions from its local API, which binds to `127.0.0.1:9999` and carries no authentication of its own. On a VM, run the commands on the host. On Kubernetes, open a port-forward first and point the commands at it:

```bash theme={null}
kubectl -n <connector-namespace> port-forward deployment/clicklink-connector-executor 9999:9999
clicklink clctl instances list --endpoint http://127.0.0.1:9999
```

`clicklink clctl instances list` prints every service the executor knows as JSON. `clicklink clctl instances get --name <name> --cluster <eks-cluster-name>` prints one, with the storage it was created with. The executor derives status from the service's `ClickHouseCluster` resource (ready versus expected server replicas) and refreshes it every `sync_interval` (30 seconds by default):

* `provisioning`: no server replica is ready yet, or the `ClickHouseCluster` does not exist yet
* `running`: every expected server replica is ready
* `degraded`: some, but not all, server replicas are ready
* `terminating`: the executor is uninstalling the service
* `terminated`: the service's namespace is gone
* `stale`: the service was dropped from the executor's registry without a delete; `--wait` and `teardown` treat it like `terminated`

A terminated service stays listed, with its storage, until [teardown](#delete-a-service) removes its cloud resources.

The executor records every command ClickHouse Cloud sends it. `clicklink clctl commands list` prints them, filtered with `--status` (`pending`, `running`, `completed`, `failed`), `--action` (for example `create_instance`), or `--cluster`. `clicklink clctl commands get <id>` prints one command with its stage and result. A failed command's `result` holds the error: why a create did not converge, or which [platform update](/products/bring-your-own-cloud/connector/platform-updates) is waiting for your approval.

<h2 id="service-lifecycle">
  Service lifecycle
</h2>

Once a service exists, ClickHouse Cloud operates it through the executor. The commands it sends are:

* **Scale.** ClickHouse Cloud sets a fixed replica count, capped by a ClickHouse Cloud setting (20 in the default configuration). There is no autoscaling.
* **Stop and start.** Stopping scales the servers to zero and keeps Keeper; starting restores the replica counts. Data stays in your buckets throughout.
* **Restart.** The whole service, its Keeper, or a single pod.
* **Backups.** ClickHouse Cloud triggers them; they land in your backup bucket, and backup deletion removes them.
* **Version upgrades and configuration changes.** ClickHouse Cloud re-renders the service definition with the new version or setting. It arrives as a `create_instance` command, so `commands list --action create_instance` also shows upgrades. The executor applies it and waits for the replicas to become ready again.
* **Delete.** Described in [delete a service](#delete-a-service).

<Warning>
  Version upgrades and configuration changes have no customer approval step. ClickHouse Cloud applies them to a managed service the same way it applies a scale or a restart: it re-sends the definition and the executor converges the cluster on it.
</Warning>

Creates, scales, and starts complete asynchronously. The executor reports the command as running as soon as it applies the definition and sends a progress report every 2 minutes. It reports the final outcome once the server replicas are ready. If they are not ready within 2 hours it reports the command as failed; if they come up later, it moves the service back to `running`.

The `instances scale`, `instances patch`, and `instances delete` subcommands post commands straight to the executor's local API, bypassing ClickHouse Cloud. Run them only when your account team asks; the [CLI reference](/products/bring-your-own-cloud/connector/reference/cli#instances-scale) describes each.

<h2 id="support-sessions">
  Support sessions for a managed service
</h2>

The executor wires the scraper into each service it creates. It does not provision the troubleshooter, so a [support session](/products/bring-your-own-cloud/connector/support-sessions) cannot run diagnostics on a managed service until you do. Provision it once per service, with the service namespace `ns-<name>`, the same way as for an instance you registered yourself:

<Tabs>
  <Tab title="Kubernetes" id="troubleshooter-kubernetes">
    ```bash theme={null}
    printf '%s\n' "$CH_DEFAULT_PASSWORD" | clicklink clctl troubleshoot access provision --target helm \
      --target-namespace "${CONNECTOR_NAMESPACE}" \
      --instance <name> --instance-namespace ns-<name> \
      --server <kubernetes-api-server-url> --ch-admin-password-stdin \
      --apply-ch-grants --ch-pod <clickhouse-pod-or-label-selector> --ch-pod-namespace ns-<name>
    ```

    `$CH_DEFAULT_PASSWORD` is the `default` user's password that `prepare` printed. The command authenticates with it to apply the SQL grants and does not prompt for it. Then add the Secret and ServiceAccount pair to `troubleshooter.accessBundles` and run `helm upgrade`, as shown under [adding ClickHouse instances](/products/bring-your-own-cloud/connector/configuration#clickhouse-instances).
  </Tab>

  <Tab title="Linux VM" id="troubleshooter-vm">
    ```bash theme={null}
    printf '%s\n' "$CH_DEFAULT_PASSWORD" | sudo clicklink clctl troubleshoot access provision --provider local \
      --instance <name> --instance-namespace ns-<name> \
      --server <kubernetes-api-server-url> --ch-admin-password-stdin
    ```

    `$CH_DEFAULT_PASSWORD` is the `default` user's password that `prepare` printed.
  </Tab>
</Tabs>

Grant the ClickHouse user with SQL, as above. A user added with `--ch-user-via cr` would not survive: ClickHouse Cloud owns the service definition and re-applies it. When the service is deleted, the executor removes the troubleshooter's bundle on a VM. On Kubernetes the bundle Secret and ServiceAccount stay until you delete them, as listed under [uninstall](/products/bring-your-own-cloud/connector/operations#uninstall-kubernetes).

<h2 id="delete-a-service">
  Delete a service
</h2>

There is no customer delete command through your connector endpoint: ask your account team to delete the service. ClickHouse Cloud terminates it, and the executor deletes the workload and its namespace (`terminating`, then `terminated`). Nothing in AWS is touched: the buckets, their data, and the IAM role remain until you remove them.

Once the service reports `terminated`, tear down what `prepare` created. Run `teardown` where you ran `prepare`, with the same AWS credentials. On Kubernetes that means the same workstation, the same configuration copy and output directory, and an open port-forward to the executor:

<Tabs>
  <Tab title="Kubernetes" id="teardown-kubernetes">
    ```bash theme={null}
    kubectl -n "${CONNECTOR_NAMESPACE}" port-forward deployment/clicklink-connector-executor 9999:9999 &
    clicklink clctl executor teardown --instance <name> --config clicklink-config.yaml --output-dir ./clicklink-access
    ```
  </Tab>

  <Tab title="Linux VM" id="teardown-vm">
    ```bash theme={null}
    sudo clicklink clctl executor teardown --instance <name>
    ```
  </Tab>
</Tabs>

The command reads the executor's record of the service: where its data lives and which role it had. It deletes the IAM role that `prepare` created and makes the executor forget the service, which frees the name. On a VM it also removes the service's cluster-wide RBAC objects, the local access bundle, and the registry entry.

By default it keeps the service's data and backups. It re-tags a kept bucket from `clicklink:deployed-name` to `clicklink:retained-from=<name>`, so a re-created service with the same name never inherits it, and the summary says where the data is. To delete them, add `--delete-data --delete-backups --yes` to the same command.

<Warning>
  `--delete-data` and `--delete-backups` empty and delete the buckets they name; there is no recovery afterwards. Confirm the bucket names in the record step before adding `--yes`.
</Warning>

Without `--yes` the run stops after the record step and names the buckets it would empty. `--dry-run` reads everything and writes nothing. A role you brought with `--role-arn`, or a bucket `prepare` did not create, is reported as kept and never touched. The run refuses while the service's namespace still holds a ClickHouse cluster. It reads everything before it deletes anything, so a refused run changes nothing.

<h2 id="when-the-connector-is-offline">
  When the connector is offline
</h2>

Running services do not depend on the executor. The ClickHouse operator in your cluster keeps them running, and a disconnected connector interrupts nothing that is already serving queries.

While no executor is connected, ClickHouse Cloud cannot hand it new work. A create, delete, or platform sync it has already accepted is kept and retried. A create is retried for 30 minutes from its last progress report, a delete for 2 hours, a platform sync for up to 10 attempts. Past that budget your connector endpoint marks the command failed. Every other lifecycle command (scale, stop, start, restart, backup, backup deletion) is refused rather than queued. A create or delete you submit while no executor is connected is refused the same way; only commands already accepted are retried.

A command the executor has already accepted runs to completion; a result it could not report is sent on its next connection.

`instances create` needs your connector endpoint reachable from the host it runs on. `--wait`, `instances list`, `instances get`, and `commands list` need the executor's local API, so the executor must be running. A platform approval's token expires on its own clock, regardless of connectivity; see [the approval window](/products/bring-your-own-cloud/connector/platform-updates#the-approval-window).