Task cluster API

In Hadoop Eco, when a cluster completes a specific job, its status changes to Terminated. In this state, the cluster is no longer usable. However, using the task cluster API, you can repeatedly schedule jobs on a previously created cluster that is in the Terminated state.

The task cluster API improves cluster reusability, allowing tasks to be executed repeatedly on an existing cluster without having to create a new one each time, enabling efficient cluster management.

The task cluster API works as follows:

Select a cluster: Choose the cluster to reprocess from the console. It must be an existing cluster in the Terminated state.
Issue an Open API key: To use the task cluster API, issue an Open API key from the KakaoCloud Console for the corresponding cluster. This key is required to access and control the cluster via API calls.
- When the Open API key is issued, a security group dedicated to the Open API cluster is automatically created. If the API key is deleted, the associated security group is also removed.
Call the task cluster API: Use the issued Open API key to invoke the task cluster API. This enables new job scheduling and execution on a cluster in the Terminated state.

Prepare API usage

To call the task cluster API, you must:

Issue access key credentials
Issue an Open API key for the Hadoop Eco cluster from the console

Create task cluster

Repeatedly schedules jobs on a cluster currently in the Terminated state.

Request

Request syntax

Cluster creation request syntax
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
          "workerCnt": 2,
          "workerVolumeSize": 50,
          "configHdfsReplication": 2,
          "configHdfsBlockSize": 128,
          "userTaskDeployMode": "cluster",
          "userTaskExecOpts": "--driver-cores 1 --driver-memory 1024m --executor-cores 1 --num-executors 4 --executor-memory 2048m",
          "userTaskExecParams": "-sparkApplicationParams=testParams"
    }'

API endpoint

Method	Request URL
POST	`https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}`

Path parameters

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	Cluster ID - Available in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster > Cluster details

Request header

Header	Type	Description
`{credential-id}`*	String	User’s access key ID - Available in the KakaoCloud Console > Top right profile > Credentials > IAM access key
`{credential-secret}`*	String	User’s secret access key - Only viewable at time of creation - If lost, reissue the IAM access key
`{hadoop-eco-api-key}`*	String	Open API key - Available in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster Select cluster > Top right Cluster actions > Issue Open API key - See Open API key guide for details

Request elements

These fields are optional and can be used to override the original cluster settings if needed.

Field	Type	Description
workerCnt	Integer	Number of Hadoop Eco worker nodes - Range: 1 to 1,000
workerVolumeSize	Integer	Block storage size of Hadoop Eco worker nodes - Range: 50 to 5120 GB
configHdfsReplication	Integer	HDFS replication factor - Range: 1 to 500
configHdfsBlockSize	Integer	HDFS block size in MB - Range: 1 to 1024
userTaskDeployMode	String	Spark job deploy mode - Options: `client`, `cluster`
userTaskExecOpts	String	Hive or Spark job execution options
userTaskExecParams	String	Spark application parameters

Response

Response syntax

Cluster creation response syntax
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "test-cluster",
  "requestId": "req-f02cf9abc130410ab365e77511db4318",
  "clusterMainSettingValues": {
    "workerCnt": 1,
    "workerVolumeSize": 50,
    "configHdfsReplication": 1,
    "configHdfsBlockSize": 128,
    "jobType": "hive",
    "userTaskDeployMode": true,
    "userTaskExecOpts": "",
    "userTaskExecParams": ""
  }
}

Response elements

Field	Description
clusterId	ID of the created cluster
clusterName	Name of the created cluster
requestId	ID of the job request
clusterMainSettingValues ▼	Cluster configuration details
workerCnt	Number of worker nodes
workerVolumeSize	Block storage size of worker nodes
configHdfsReplication	HDFS replication factor
configHdfsBlockSize	HDFS block size
jobType	Type of job (`Spark`, `Hive`)
userTaskDeployMode	Spark deploy mode
userTaskExecOpts	Hive/Spark job execution options
userTaskExecParams	Spark application parameters

Status codes

HTTP Status	Description
`200`	Success
`202`	Cluster job creation in progress
`400`	Invalid request
`401`, `403`	Authentication failed or unauthorized
`404`	Cluster not found

Retrieve task cluster details

Check the status of a cluster using the issued Open API key.

Request

Request syntax

Cluster status request (basic)
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json'

Cluster status request (verbose)
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}?verbose=true' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json'

API endpoint

Method	URL
GET	`https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}`

Path parameters

Parameter	Type	Description
`{cluster-id}`*	String	Cluster ID - Viewable in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster > Cluster details
`{request-id}`*	String	The `requestId` value returned from Create task cluster

Query parameters

Parameter	Type	Description
verbose	Boolean	If set to `true`, returns detailed information including master/worker node status

Request header

Header	Type	Description
`{credential-id}`*	String	Access key ID
`{credential-secret}`*	String	Secret access key
`{hadoop-eco-api-key}`*	String	Open API key

Response

Response (verbose = false)

Cluster status (verbose=false)
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "peb-hadoop-ga-test",
  "requestId": "req-ac2aad4c128d4486bbc34fe4862737cb",
  "requestStatus": "Terminate",
  "requestResult": "Success"
}

Response (verbose = true)

Cluster status (verbose=true)
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "peb-hadoop-ga-test",
  "requestId": "req-ac2aad4c128d4486bbc34fe4862737cb",
  "requestStatus": "Terminate",
  "requestResult": "Failure",
  "clusterMeta": {
    "cluster_version": "hde-1.1.0",
    "service_check_timeout": 10,
    "socket_timeout": 5,
    "global_timeout": 1500,
    "monitoring": false,
    "cluster_id": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
    "cluster_name": "peb-hadoop-ga-test",
    "cluster_type": "hadoop_single",
    "project": {
      "id": "353094e122d6493d9899f0f2523d4cc1",
      "name": "bigdata",
      "domain": {
        "id": "982fc392699d4385b0b1a9d10b9f2393",
        "name": "kakaoenterprise"
      }
    },
    "master_vm": {
      "host_list": ["HadoopMST-peb-hadoop-ga-test-1"],
      "flavor_id": "9b944f27-12fd-4542-8073-e477931c9365",
      "flavor_name": "a1-4-std",
      "key_pair": "bigdata-peb",
      "network": {
        "uuid": "3541a58c-3930-4dcb-8785-0032ff926b80",
        "subnet_id": "be864742-c658-4f7f-8077-9b9138b86df1",
        "security_group_id_list": [
          "73e64d4f-cf71-4be5-8cd5-42e5cb5ccceb",
          "1a21f01b-0437-40ab-a26b-979552555b50"
        ]
      },
      "block_device": {
        "source_type": "image",
        "uuid": "99612215-f9c2-4914-9c76-c8567285675f",
        "volume_size": 50
      },
      "master_active": "HadoopMST-peb-hadoop-ga-test-1",
      "master_standby": "HadoopMST-peb-hadoop-ga-test-1",
      "master_etc": "HadoopMST-peb-hadoop-ga-test-1"
    },
    "worker_vm": {
      "host_list": [],
      "flavor_id": "9b944f27-12fd-4542-8073-e477931c9365",
      "flavor_name": "a1-4-std",
      "key_pair": "bigdata-peb",
      "network": {
        "uuid": "3541a58c-3930-4dcb-8785-0032ff926b80",
        "subnet_id": "be864742-c658-4f7f-8077-9b9138b86df1",
        "security_group_id_list": [
          "73e64d4f-cf71-4be5-8cd5-42e5cb5ccceb",
          "1a21f01b-0437-40ab-a26b-979552555b50"
        ]
      },
      "block_device": {
        "source_type": "image",
        "uuid": "99612215-f9c2-4914-9c76-c8567285675f",
        "volume_size": 50
      }
    },
    "user_cluster_exec_command": {
      "type": "hive",
      "termination_policy": "always",
      "hive_query": "show databases;",
      "exec_opts": "",
      "exec_params": ""
    }
  }
}

Response elements

Field	Description
clusterId	ID of the created cluster
clusterName	Name of the created cluster
requestId	ID of the request
requestStatus	Status of the request
requestResult	Result of the request
clusterMeta ▼	Metadata about the cluster
cluster_version	Version of the created cluster
service_check_timeout	Service timeout duration (seconds)
socket_timeout	Socket timeout duration (seconds)
global_timeout	Global timeout duration (seconds)
monitoring	Whether monitoring is enabled
cluster_id	ID of the cluster
cluster_name	Name of the cluster
cluster_type	Cluster type
project	Project information
id	Project ID
name	Project name
project (domain)	Domain information
id(domain)	ID of the domain the project belongs to
name(domain)	Name of the domain the project belongs to
master_vm ▼	Master node information
host_list	Hostnames of the master nodes
flavor_id	Flavor ID of the master node
flavor_name	Flavor name of the master node
key_pair	Key pair name used by the master node
network ▼	Network settings
uuid(network)	UUID of the network
subnet_id(network)	Subnet ID
security_group_id_list	List of security group IDs
block_device ▼	Block storage settings
source_type	Source type of the block storage
uuid	UUID of the block storage
volume_size	Size of the block storage (GB)
master_active	Name of the active master node
master_standby	Name of the standby master node
master_etc	Name of the etc (other role) master node
worker_vm ▼	Worker node information
host_list	Hostnames of the worker nodes
flavor_id	Flavor ID of the worker nodes
flavor_name	Flavor name of the worker nodes
key_pair	Key pair name used by the worker nodes
network	Network settings for the worker nodes
uuid(network)	UUID of the network
subnet_id(network)	Subnet ID
security_group_id_list	List of security group IDs
user_cluster_exec_command ▼	User-defined task execution details
type	Type of job - `Spark`, `Hive`
termination_policy	Post-execution behavior - `always`, `onSuccess`, `never`
hive_query	Hive query string
exec_opts	Hive or Spark execution options
exec_params	Parameters for the Spark application

Status codes

HTTP Status	Description
`200`	Success
`400`	Invalid request
`401`, `403`	Authentication failed or insufficient permissions
`404`	Cluster not found

Prepare API usage​

Create task cluster​

Request​

Request syntax​

API endpoint​

Path parameters​

Request header​

Request elements​

Response​

Response syntax​

Response elements​

Status codes​

Retrieve task cluster details​

Request​

Request syntax​

API endpoint​

Path parameters​

Query parameters​

Request header​

Response​

Response (verbose = false)​

Response (verbose = true)​

Response elements​

Status codes​

Prepare API usage

Create task cluster

Request

Request syntax

API endpoint

Path parameters

Request header

Request elements

Response

Response syntax

Response elements

Status codes

Retrieve task cluster details

Request

Request syntax

API endpoint

Path parameters

Query parameters

Request header

Response

Response (verbose = false)

Response (verbose = true)

Response elements

Status codes