Creating a ClickHouse Cluster

This article describes how to provision a ClickHouse cluster on the Instaclustr platform using the Instaclustr Console, Instaclustr API, and Terraform Provider. A basic ClickHouse cluster is used for provisioning demonstration and any add-on features and options will be covered separately in later sections.

Provision Using the Instaclustr Console 

1. To get started, after setting up your user account and logging into the console, click the Create Cluster button. This will take you to the wizard for creating a new cluster  
2. Under the Application page, enter an appropriate name for your cluster and choose the SLA tier you require. Select ClickHouse as the base application and choose the desired infrastructure provider (AWS is supported right now, with support for GCP and Azure coming soon). More details on creating a Private Network Cluster is available below. 
3. Under the ClickHouse Setup page, select ClickHouse version from the drop-down list. If you want to access the cluster directly from the IP address where this cluster is provisioned, simply check the “Add your current IP …” checkbox. Otherwise, you could do this after the cluster has been provisioned as well by going to the “Firewall Rules” tab. Choose the number of Replicas and Shards you require for the cluster. Note that the product of Replicas and Shards is the total number of ClickHouse server nodes in the cluster. Both number shards and replicas can be increased, but not decreased, after a cluster is provisioned by horizontal scaling. More details on creating a cluster with Dedicated ClickHouse Keeper, Storage Tiering, and Load Balancer is available below.
4. Under the Data Centre Options page, first select your provider account. This can be either INSTACLUSTR (for RIIA), or your own provider account (for RIYOA). Then choose your data centre region and cluster network address block. Refer to our support article on Network Address Allocation to understand how we divide up the specified network range to determine the node IP address.  If the data centre for your cluster is running on Amazon’s EBS infrastructure, you can choose to encrypt data with an AWS KMS encryption key. The number of ClickHouse nodes is decided by your Replicas and Shards selection from the previous page and is immutable on this page. Choose the node size based on your requirements. 
5. The Confirmation page displays a brief summary of your cluster configurations and pricing details. Click the terms of service link to open the Instaclustr Terms and Conditions. After going through the document, select the checkbox to accept the Terms and Conditions. Once you are happy with the cluster configuration and have accepted the terms and conditions, click the Create Cluster button to start provisioning of your cluster.
6. Provisioning a cluster can take some minutes depending on the responsiveness of the underlying cloud provider. There are separate status indicators for the overall cluster provisioning state. Refer to our support article on The Provisioning Lifecycle to understand each of the different stages in more detail. 
7. Congratulations! You have now finished creating your cluster on the Instaclustr Console. All your clusters will be listed on the left sidebar of the console. You can view your cluster’s configuration details by navigating to the Details tab (public and private IPs will be available, only redacted on the figure below).

Provision Using Instaclustr API

1. To provision a ClickHouse cluster using the Instaclustr API, you’ll need to obtain the API keys for your account. Go to the Account Settings tab on the top right corner of the Instaclustr Console.
2. Go to the API Keys tab on the page and add a Provisioning key for your account user by clicking the Add button on that row (User Name has been redacted in the figure below).
3. After adding the key, it will be displayed once on the Instaclustr Console. Please take your time to copy and store it safely on your end before it is removed from the Instaclustr Console.
4. With the username and API Key obtained, you can now send POST requests to the Instaclustr API (https://api.instaclustr.com/cluster-management/v2/resources/applications/clickhouse/clusters/v2) with request body template below to create a ClickHouse cluster. Simply fill in each field with your required values and send the request. For a complete reference check out our API Documentation. 
   1. { "clickhouseVersion": "24.8.2, "name": "test-clickhouse-cluster", "slaTier": "NON_PRODUCTION", "privateNetworkCluster": false, "pciComplianceMode": false, "dataCentres": [{ "name": "AWS_VPC_US_EAST_1", "region": "US_EAST_1", "cloudProvider": "AWS_VPC", "network": "10.0.0.0/16", "shards": “1”, "replicas": "3", "nodeSize": "CLK-DEV-t3.medium-50" }] }

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

      { "clickhouseVersion": "24.8.2, "name": "test-clickhouse-cluster", "slaTier": "NON_PRODUCTION", "privateNetworkCluster": false, "pciComplianceMode": false, "dataCentres": [{ "name": "AWS_VPC_US_EAST_1", "region": "US_EAST_1", "cloudProvider": "AWS_VPC", "network": "10.0.0.0/16", "shards": “1”, "replicas": "3", "nodeSize": "CLK-DEV-t3.medium-50" }] }

5. After sending the request, an API response with cluster provisioning config is expected to mark the success of your request. You’ll also be able to view the status of the provisioning on Instaclustr Console.  

Provision Using Terraform 

1. Similar to provisioning with the Instaclustr API, you’ll need to first obtain your provisioning API Key from the Instaclustr Console (See steps 1-3 in the Provision Using Instaclustr API section).  
2. Create a main.tf file with content below and fill in values according to your requirement following the Instaclustr API Documentation
   1. terraform { required_providers { instaclustr = { source = "instaclustr/instaclustr" version = ">= 2.0.0, < 3.0.0" } } } provider "instaclustr" { terraform_key = "Instaclustr-Terraform <username>:<provisioning-API-Key>" } resource "instaclustr_clickhouse_cluster_v2" "MyClickHouseCluster" { data_centre { cloud_provider = "AWS_VPC” name = "AWS_VPC_US_EAST_1" network = "10.0.0.0/16" node_size = "CLK-DEV-t3.medium-50" replicas = 2 shards = 2 region = "US_EAST_1" } private_network_cluster = false clickhouse_version = "24.8.2" name = "test-clickhouse-cluster" sla_tier = "NON_PRODUCTION" }

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

      terraform { required_providers { instaclustr = { source = "instaclustr/instaclustr" version = ">= 2.0.0, < 3.0.0" } } }   provider "instaclustr" { terraform_key = "Instaclustr-Terraform <username>:<provisioning-API-Key>" }   resource "instaclustr_clickhouse_cluster_v2" "MyClickHouseCluster" { data_centre { cloud_provider = "AWS_VPC” name = "AWS_VPC_US_EAST_1" network = "10.0.0.0/16" node_size = "CLK-DEV-t3.medium-50" replicas = 2 shards = 2 region = "US_EAST_1" }   private_network_cluster = false clickhouse_version = "24.8.2" name = "test-clickhouse-cluster" sla_tier = "NON_PRODUCTION" }

3. Run terraform init to initialise terraform from the repo.  
4. Run terraform plan to generate the resource plan as laid out in the main.tf file. You’re expected to receive terraform response similar to the following:
   1. Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create Terraform will perform the following actions: # instaclustr_clickhouse_cluster_v2.MyClickHouseCluster will be created + resource "instaclustr_clickhouse_cluster_v2" "MyClickHouseCluster" { + clickhouse_version = "24.8.2" + current_cluster_operation_status = (known after apply) + description = (known after apply) + id = (known after apply) + name = "test-clickhouse-cluster" + private_network_cluster = false + sla_tier = "NON_PRODUCTION" + status = (known after apply) + data_centre { + cloud_provider = "AWS_VPC" + id = (known after apply) + load_balancer_domain = (known after apply) + load_balancer_enabled = false + name = "AWS_VPC_US_EAST_1" + network = "10.0.0.0/16" + node_size = "CLK-DEV-t3.medium-50" + provider_account_name = (known after apply) + region = "US_EAST_1" + replicas = 3 + shards = 1 + status = (known after apply) } } Plan: 1 to add, 0 to change, 0 to destroy.

      1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

      Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols: + create   Terraform will perform the following actions:   # instaclustr_clickhouse_cluster_v2.MyClickHouseCluster will be created + resource "instaclustr_clickhouse_cluster_v2" "MyClickHouseCluster" { + clickhouse_version = "24.8.2" + current_cluster_operation_status = (known after apply) + description = (known after apply) + id = (known after apply) + name = "test-clickhouse-cluster" + private_network_cluster = false + sla_tier = "NON_PRODUCTION" + status = (known after apply)   + data_centre { + cloud_provider = "AWS_VPC" + id = (known after apply) + load_balancer_domain = (known after apply) + load_balancer_enabled = false + name = "AWS_VPC_US_EAST_1" + network = "10.0.0.0/16" + node_size = "CLK-DEV-t3.medium-50" + provider_account_name = (known after apply) + region = "US_EAST_1" + replicas = 3 + shards = 1 + status = (known after apply) } }   Plan: 1 to add, 0 to change, 0 to destroy.

5. Run terraform apply from the same repository to and confirm the actions (i.e., entering “yes” to the prompt) to start provisioning the cluster. The provisioning will take a while depending on the responsiveness of the cloud providers. Once finished, you should see logs printing out a success message with the cluster ID:

Optional Features 

Instaclustr supports several features that you can add to your vanilla ClickHouse cluster based on your requirements. They are only available at cluster creation time as of now, and cannot be altered afterwards. This includes: 

• Private Network Clusters 
• Dedicated ClickHouse Keeper 
• Storage Tiering 
• Load Balancer 

Instructions on setting up the add-ons and features are provided below.  

1. Private Network Clusters: Private Network Clusters build on the security available in Standard Network Clusters by allowing clusters to be created without publicly routable IPs allocated to cluster nodes.  All inter-node communication occurs within a private network. Details on Private Network Clusters is available on Instaclustr’s documentation on Private Network Clusters
   • Console: Check the “Private Network Cluster” checkbox on the Application page.
   • API: Setting the “privateNetworkCluster” parameter in the request body to “true”.  
   • Terraform: Setting the “privateNetworkCluster” parameter in the terraform main.tf file to “true”. 
2. Dedicated ClickHouse Keeper nodes: Dedicated ClickHouseKeeper nodes provide customer benefits in situations that require high performance and reliability by adding extra nodes to the cluster with a dedicated ClickHouse Keeper functionality. Details on Dedicated ClickHouse Keeper is available in Instaclustr’s documentation on Dedicated ClickHouse Keeper
   • Console: Check the “Use Dedicated ClickHouse Keeper nodes” checkbox on the “ClickHouse Setup” page.
     • Under the data centre section, you can now choose a node size for ClickHouse and dedicated ClickHouse Keeper separately. 
     • Once you have set up the other ClickHouse related settings and created your cluster, you can see the ClickHouse Keeper is running on separate instances identified by the Dedicated ClickHouse Keeper role. 
   • API: Edit the “dataCentres” section of the request body to include information about the desired dedicated ClickHouse Keeper node configuration.
     • "dataCentres": [ { "dedicatedClickHouseKeeper": [ { "nodeCount": 3, "nodeSize": "DCK-DEV-t3.small-30"}] ... // other dataCentre properties }]

       1	"dataCentres": [ { "dedicatedClickHouseKeeper": [ { "nodeCount": 3, "nodeSize": "DCK-DEV-t3.small-30"}] ... // other dataCentre properties }]

   • Terraform: Edit the “dataCentre” section of the main.tf file to include information about the desired dedicated ClickHouse Keeper node configuration.
     1. dedicated_click_house_keeper { node_count = 3 node_size = "DCK-DEV-t3.small-30"}

        1	dedicated_click_house_keeper { node_count = 3 node_size = "DCK-DEV-t3.small-30"}

3. Storage Tiering: ClickHouse allows for more than one block device to be designated for data storage. This not only allows using additional storage types, such as object storage like AWS S3, but also introduces support for storage tiering as found in typical hot-cold storage setups. Details on Storage Tiering for ClickHouse clusters is available in Instaclustr’s documentation on Storage Tiering for ClickHouse Clusters
   • Console: Enable Storage Tiering by checking the “Enable Storage Tiering” checkbox on the ClickHouse Setup page and putting appropriate bucket name and an optional prefix into the text input field below
   • API: Edit the request body “dataCentres” section to include information about the desired storage tiering configuration
     • "dataCentres": [ { "tieredStorage": [ { "s3Settings": [ { "s3BucketName": "my-test-bucket", "prefix": "some-prefix"} ... // other dataCentres properties ]}]}]

       1	"dataCentres": [ { "tieredStorage": [ { "s3Settings": [ { "s3BucketName": "my-test-bucket", "prefix": "some-prefix"} ... // other dataCentres properties ]}]}]

   • Terraform: edit the main.tf file data_centre section to include information about the desired storage tiering configuration 
     1. tiered_storage { s3_settings { s3_bucket_name = "my-test-bucket" prefix = "some-prefix" }}

        1	tiered_storage { s3_settings { s3_bucket_name = "my-test-bucket" prefix = "some-prefix" }}

4. Load Balancer: Load Balancer for a ClickHouse cluster is a Network Load Balancer that has the functionality of directing client requests to individual nodes in the cluster, and performing health checks on nodes to ensure connectivity. Details on Load Balancer are available on Instaclustr’s documentation on Load Balancer for ClickHouse clusters
   • Console: Check the “Load Balancer” checkbox on the ClickHouse Setup page
     • Instaclustr provides a DNS record that can be used to connect to the Load Balancer which can be found on the connection info page.

     

   • API: Edit the “dataCentres” section of the request body to enable load balancer on the data centre 
     • "dataCentres": [ { "loadBalancerEnabled": false ... // other dataCentres properties }]

       1 2 3 4 5

       "dataCentres": [ { "loadBalancerEnabled": false ... // other dataCentres properties }]

   • Terraform: Edit the “data_centre” section of the main.tf file to enable load balancer on the data centre 
     • data_centre { load_balancer_enabled = true ... // other data_centre properties }

       1 2 3 4

       data_centre { load_balancer_enabled = true ... // other data_centre properties }