Job cluster API

In Hadoop Eco, clusters typically switch to the Terminated state once a specific task is completed, making them unavailable for further use. However, with the job cluster API, you can repeatedly schedule tasks on clusters already in the Terminated state.

The job cluster API enhances cluster reusability, allowing repeated task execution on existing clusters without creating new clusters each time, enabling efficient cluster management.

The job cluster API operates through the following process:

Cluster selection: Select a cluster for rework from the console. This must be an existing cluster in the Terminated state.
Issue Open API Key: To use the job cluster API, you must issue an Open API Key from the KakaoCloud console for the respective cluster. This key is essential for accessing and controlling the cluster through API calls.
- When issuing an Open API Key, a security group exclusive to the Open API cluster is automatically created. If the API Key is deleted, the security group is also removed.
Call the job cluster API: Use the issued Open API Key to call the job cluster API. This allows you to schedule and execute new tasks on a Terminated cluster.

Prepare API usage

To call the job cluster API, you must get an access key and obtain the Open API Key for the Hadoop Eco cluster from the console.

Create job cluster

Repeatedly execute task scheduling on a cluster currently in the Terminated state.

Request

Request Syntax

(kr-central-2) region cluster creation request syntax
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
          "workerCnt": 2,
          "workerVolumeSize": 50,
          "configHdfsReplication": 2,
          "configHdfsBlockSize": 128,
          "userTaskDeployMode": "cluster",
          "userTaskExecOpts": "--driver-cores 1 --driver-memory 1024m --executor-cores 1 --num-executors 4 --executor-memory 2048m",
          "userTaskExecParams": "-sparkApplicationParams=testParams"
    }'

API call method

Method	Request URL
POST (kr-central-2)	`https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}`

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information

Request header

Request	Type	Description
`{credential-id}`*	String	User's access key ID - Available from KakaoCloud console > Top-right profile > Access key
`{credential-secret}`*	String	User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key
`{hadoop-eco-api-key}`*	String	Open API key - Issuable from KakaoCloud console > Analytics > Hadoop Eco > Left menu Select a cluster > Top-right menu Cluster tasks > Issue Open API Key - Refer to Issue Open API Key for details

Request elements

These are not mandatory values but can be used when changes to the existing cluster configuration are needed.

Request	Type	Description
workerCnt	Integer	Number of Hadoop Eco worker nodes - Range: 1 ~ 1,000
workerVolumeSize	Integer	Block storage size for Hadoop Eco worker nodes - Range: 50 ~ 5120GB
configHdfsReplication	Integer	Number of Hadoop Eco HDFS replications - Range: 1 ~ 500
configHdfsBlockSize	Integer	Block size for Hadoop Eco HDFS - Range: 1 ~ 1024MB
userTaskDeployMode	String	Hadoop Eco Spark Job Deploy mode - Modes: `client`, `cluster`
userTaskExecOpts	String	Hadoop Eco Hive configuration parameters
userTaskExecParams	String	Hadoop Eco Job application parameters

Response

Response Syntax

Create cluster Response Syntax
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "test-cluster",
  "requestId": "req-f02cf9abc130410ab365e77511db4318",
  "clusterMainSettingValues": {
    "workerCnt": 1,
    "workerVolumeSize": 50,
    "configHdfsReplication": 1,
    "configHdfsBlockSize": 128,
    "jobType": "hive",
    "userTaskDeployMode": true,
    "userTaskExecOpts": "",
    "userTaskExecParams": ""
  }
}

Response Elements

Response	Description
clusterId	ID of the created cluster
clusterName	Name of the created cluster
requestId	Request ID
clusterMainSettingValues▼	Configuration details
workerCnt	Number of Hadoop Eco worker nodes
workerVolumeSize	Block storage size for Hadoop Eco worker nodes
configHdfsReplication	Number of Hadoop Eco HDFS replications
configHdfsBlockSize	Block size for Hadoop Eco HDFS
jobType	Job type - Types: `Spark`, `Hive`
userTaskDeployMode	Hadoop Eco Spark Job Deploy mode
userTaskExecOpts	Hadoop Eco Spark or Hive Job configuration parameters
userTaskExecParams	Hadoop Eco Spark Application parameters

Status codes

HTTP Status	Description
`200`	Success
`202`	Cluster task (creation) in progress
`400`	Request information error
`401`, `403`	Authentication failed, forbidden
`404`	Cluster not found

Retrieve job cluster details

You can retrieve the status of a cluster issued with an Open API Key.

Request

Request Syntax

(kr-central-2) region cluster status retrieval request syntax
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json'
 
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}?verbose=true' \
    --header 'Hadoop-Eco-Api-Key: {hadoop-eco-api-key}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json'

API call method

Method	Request URL
GET (Kr-central-1)	`https://hadoop-eco.kr-central-1.kakaoi.io/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}`
GET (Kr-central-2)	`https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/requests/{request-id}`

Path	Type	Description
`{cluster-id}`*	String	Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information
`{request-id}`*	String	`requestId` value received in the response after Job cluster creation

Query parameters

Request	Type	Description
verbose	Boolean	If the `verbose` option is enabled, the status of master/worker nodes can also be retrieved - Options: `true`, `false`

Request Header

Request	Type	Description
`{credential-id}`*	String	User's access key ID - Available from KakaoCloud console > Top-right profile > Access key
`{credential-secret}`*	String	User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key
`{hadoop-eco-api-key}`*	String	Open API key - Issuable from KakaoCloud console > Analytics > Hadoop Eco > Left menu Select a cluster > Top-right menu Cluster tasks > Issue Open API Key - Refer to Issue Open API Key for details

Response

Response syntax

Response syntax when verbose=false
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "peb-hadoop-ga-test",
  "requestId": "req-ac2aad4c128d4486bbc34fe4862737cb",
  "requestStatus": "Terminate",
  "requestResult": "Success"
}

Response syntax when verbose=true
{
  "clusterId": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
  "clusterName": "peb-hadoop-ga-test",
  "requestId": "req-ac2aad4c128d4486bbc34fe4862737cb",
  "requestStatus": "Terminate",
  "requestResult": "Failure",
  "clusterMeta": {
    "cluster_version": "hde-1.1.0",
    "service_check_timeout": 10,
    "socket_timeout": 5,
    "global_timeout": 1500,
    "monitoring": false,
    "cluster_id": "58ceed05-e4f5-4a85-b9a6-e9b79ae8dcdf",
    "cluster_name": "peb-hadoop-ga-test",
    "cluster_type": "hadoop_single",
    "project": {
      "id": "353094e122d6493d9899f0f2523d4cc1",
      "name": "bigdata",
      "domain": {
        "id": "982fc392699d4385b0b1a9d10b9f2393",
        "name": "kakaoenterprise"
      }
    },
    "master_vm": {
      "host_list": [
        "HadoopMST-peb-hadoop-ga-test-1"
      ],
      "flavor_id": "9b944f27-12fd-4542-8073-e477931c9365",
      "flavor_name": "a1-4-std",
      "key_pair": "bigdata-peb",
      "network": {
        "uuid": "3541a58c-3930-4dcb-8785-0032ff926b80",
        "subnet_id": "be864742-c658-4f7f-8077-9b9138b86df1",
        "security_group_id_list": [
          "73e64d4f-cf71-4be5-8cd5-42e5cb5ccceb",
          "1a21f01b-0437-40ab-a26b-979552555b50"
        ]
      },
      "block_device": {
        "source_type": "image",
        "uuid": "99612215-f9c2-4914-9c76-c8567285675f",
        "volume_size": 50
      },
      "master_active": "HadoopMST-peb-hadoop-ga-test-1",
      "master_standby": "HadoopMST-peb-hadoop-ga-test-1",
      "master_etc": "HadoopMST-peb-hadoop-ga-test-1"
    },
    "worker_vm": {
      "host_list": [],
      "flavor_id": "9b944f27-12fd-4542-8073-e477931c9365",
      "flavor_name": "a1-4-std",
      "key_pair": "bigdata-peb",
      "network": {
        "uuid": "3541a58c-3930-4dcb-8785-0032ff926b80",
        "subnet_id": "be864742-c658-4f7f-8077-9b9138b86df1",
        "security_group_id_list": [
          "73e64d4f-cf71-4be5-8cd5-42e5cb5ccceb",
          "1a21f01b-0437-40ab-a26b-979552555b50"
        ]
      },
      "block_device": {
        "source_type": "image",
        "uuid": "99612215-f9c2-4914-9c76-c8567285675f",
        "volume_size": 50
      }
    },
    "user_cluster_exec_command": {
      "type": "hive",
      "termination_policy": "always",
      "hive_query": "show databases;",
      "exec_opts": "",
      "exec_params": ""
    }
  }
}

Response Elements

Response	Description
clusterId	ID of the created cluster
clusterName	Name of the created cluster
requestId	Request ID
requestStatus	Status of the request
requestResult	Result of the request
clusterMeta▼	Cluster metadata information
cluster_version	Version of the created cluster
service_check_timeout	Service timeout duration
socket_timeout	Socket timeout duration
global_timeout	Global timeout duration
monitoring	Monitoring enabled status
cluster_id	Cluster ID
cluster_name	Cluster name
cluster_type	Cluster type
project	Project information
id	Project ID
name	Project name
project	Domain information
id(domain)	Domain ID the project belongs to
name(domain)	Domain name the project belongs to
master_vm▼	Master node information
host_list	List of host names
flavor_id	Flavor ID
flavor_name	Flavor name
key_pair	Key pair name
network▼	Network information
uuid(network)	Network UUID
subnet_id(network)	Subnet ID
security_group_id_list	Security group IDs
block_device▼	Block storage information
source_type	Block storage source type
uuid	UUID of the block storage
volume_size	Size of the block storage
master_active	Active server name of the master node
master_standby	Standby server name of the master node
master_etc	Etc server name of the master node
worker_vm▼	Worker node information
host_list	List of host names
flavor_id	Flavor ID
flavor_name	Flavor name
key_pair	Key pair name
network	Network information
uuid(network)	Network UUID
subnet_id(network)	Subnet ID
security_group_id_list	Security group IDs
user_cluster_exec_command▼	User-submitted task information
type	Task type - `Spark`, `Hive`
termination_policy	Task termination policy - `always`, `onSuccess`, `never`
hive_query	Hive task query
exec_opts	Hive, Spark configuration values
exec_params	Spark application parameter values

Status codes

HTTP Status	Description
`200`	Success
`400`	Request information error
`401`, `403`	Authentication failed, forbidden
`404`	Cluster not found

Prepare API usage​

Create job cluster​

Request​

Request Syntax​

API call method​

Request header​

Request elements​

Response​

Response Syntax​

Response Elements​

Status codes​

Retrieve job cluster details​

Request​

Request Syntax​

API call method​

Query parameters​

Request Header​

Response​

Response syntax​

Response Elements​

Status codes​

Prepare API usage

Create job cluster

Request

Request Syntax

API call method

Request header

Request elements

Response

Response Syntax

Response Elements

Status codes

Retrieve job cluster details

Request

Request Syntax

API call method

Query parameters

Request Header

Response

Response syntax

Response Elements

Status codes