Skip to main content

Hadoop Eco Open API

Before getting started

Hadoop Eco Open API performs general cluster creation and management tasks, such as creating new clusters, querying clusters, deleting clusters, and scaling them. Open API key issuance is not required when using the Hadoop Eco Open API.

The usage method for Hadoop Eco Open API is as follows.

Prepare API usage

To call the Hadoop Eco Open API, you must get an access key.

API endpoint

The Hadoop Eco API endpoint URL for API requests is as follows:

kr-central-2 region API endpoint URL format
https://hadoop-eco.kr-central-2.kakaocloud.com

Base64 encoding method

When using the Open API, certain values must be encoded in Base64 before being input. The encoding method is as follows:

Base64 encoding 'hello' string in terminal
1. Linux: echo "hello" | base64

2. OS X: echo "hello" | base64

3. Windows: echo hello > input.txt
certutil -encode "input.txt" "output.txt"
type output.txt

Create cluster

Create a cluster using the Open API. The endpoint is as follows:

Create cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'

Request

Request Header

KeyValue
Content-Type*Fixed as application/json
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, get a new access key

Request body

Enter basic information

Enter basic information such as cluster name, type, version, network details, and availability settings.

Request Body
{
"name": "string",
"clusterType": "string",
"clusterVersion": "string",
"isHa": true | false,
"keypairName": "string",
"masterFlavorName": "string",
"masterVolumeSize": integer,
"workerCnt": integer,
"workerFlavorName": "string",
"workerVolumeSize": integer,
"monitoring": true,
"securityGroupIds": ["string"],
"securityGroupName": "string",
"vpcId": "string",
"subnetId": "string",
"adminInfo": {
"userId": "string",
"userPw": "string"
},
"config": {
"hdfsReplication": integer,
"hdfsBlockSize": integer,
"configText": "string",
"fileUrl": "string",
"userScript": "string"
}
}
CategoryTypeDescription
name*StringCluster name
clusterType*StringCluster type
clusterVersion*StringCluster version
isHa*BooleanHigh availability
- true: High availability enabled
- false: High availability disabled
keypairName*StringKey pair name used by virtual machines composing the cluster
masterFlavorName*StringMaster node flavor type name
- ex) m2a.xlarge
masterVolumeSize*IntegerMaster node volume size (GB)
workerCnt*IntegerNumber of worker nodes, must be greater than or equal to config.hdfsReplication
workerFlavorName*StringWorker node flavor type name
- ex) m2a.xlarge
workerVolumeSize*IntegerWorker node volume size (GB)
monitoring*BooleanWhether to install monitoring agent
securityGroupIds,
securityGroupName
StringSecurity group ID and security group name; one of these fields is required
- Enter the security group ID in securityGroupIds to reuse an existing security group
- Enter a security group name in securityGroupName to automatically create one during cluster creation
vpcId*StringVPC ID where the cluster will be installed
subnetId*StringSubnet ID where the cluster will be installed
adminInfo.userIdStringAccount information (ID) for Hue and Superset
- If not entered, the IAM account ID is used
adminInfo.userPwStringAccount information (PW) for Hue and Superset
- If not entered, the IAM account password is used
- Enter a Base64-encoded string for custom settings
config.hdfsReplication*IntegerReplication count
- Must be less than or equal to workerCnt
config.BlockSize*IntegerBlock size (MB)
config.configTextStringCluster configuration settings to inject
- Write the settings as a JSON string, then encode them as Base64
- Refer to fileUrl and configText injection example
config.fileUrlStringURL of the file in Object Storage
- Base64 encoding is not required
Example: https://objectstorage.{region}.kakaocloud.com/v1/<project ID>/<bucket>/<path>/<file>
- Refer to fileUrl and configText injection example
config.userScriptStringUser script to run when virtual machines are created
- Enter the script encoded in Base64
- Refer to userScript injection example
fileUrl and configText injection example

You can input cluster configuration settings using fileUrl or configText as a JSON string or from a file in Object Storage.
For example, if you want to inject the following configuration settings, first write them in JSON format and refer to the Configuration settings injection section.

core-site.xml
<configuration>
... (omitted) ...
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
... (omitted) ...
</configuration>
hive-site.xml
<configuration>
... (omitted) ...
<property>
<name>mapred.max.split.size</name>
<value>128000000</value>
</property>
... (omitted) ...
</configuration>
hadoop-env.sh
... (omitted) ...
export HADOOP_HEAPSIZE="3001"
export HADOOP_NAMENODE_INIT_HEAPSIZE="-Xmx3002m"
... (omitted) ...
spark-defaults.conf
... (omitted) ...
spark.driver.memory 4000M
spark.network.timeout 800s
... (omitted) ...

Inject configuration settings

Once the JSON is prepared, there are two ways to inject the configuration settings during cluster creation.

First method

The first method involves encoding the JSON in Base64 and injecting it into the configText field.

  1. Run the following command to encode the JSON configuration file in Base64:

    echo | cat config.json | base64
  2. Enter the Base64 string into the configText field.

    ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAiaGl2ZS1zaXRlIiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgIm1hcHJlZC5tYXguc3BsaXQuc2l6ZSI6ICIxMjgwMDAwMDAiCiAgICAgIH0KICAgIH0sCgl7CiAgICAgICJjbGFzc2lmaWNhdGlvbiI6ICJoYWRvb3AtZW52IiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgImhhZG9vcF9lbnZfaGFkb29wX2hlYXBzaXplIjogMzAwMSwKICAgICAgICAiaGFkb29wX2Vudl9oYWRvb3BfbmFtZW5vZGVfaGVhcHNpemUiOiAiLVhteDMwMDJtIgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAic3BhcmstZGVmYXVsdHMiLAogICAgICAicHJvcGVydGllcyI6IHsKICAgICAgICAic3BhcmsuZHJpdmVyLm1lbW9yeSI6ICI0MDAwTSIsCiAgICAgICAgInNwYXJrLm5ldHdvcmsudGltZW91dCI6ICI4MDBzIgogICAgICB9CiAgICB9CiAgXQp9Cg==

Second method

The second method involves uploading the file to Object Storage and injecting the file's object URL into fileUrl.

  1. Upload the JSON file to Object Storage.
  2. Click the [More] button on the uploaded file and select 'Copy object URL'.
  3. If copied correctly, you will see a URL in the following format:
    • https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{file}
  4. Inject the copied URL into the fileUrl field.
Example of injecting a userScript
  1. Prepare a user script file.

    User script example
    #!/bin/bash

    function print(){
    msg=$1
    echo "${msg}"
    }

    print "hello world!"
  2. Run the following command to encode the script content in Base64:

    echo | cat script.sh | base64
  3. Enter the Base64 string into the userScript field.

    IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK

(Optional) If you want to run task scheduling after cluster installation, enter userTask as shown below.

userTask
"userTask": {
"type": "hive|spark",
"terminationPolicy": "never|onSuccess|always",
"fileUrl": "string",
"hiveQuery": "string",
"deployMode": "cluster|client",
"execOpts": "string",
"execParams": "string",
"logBucketName": "string"
}
CategoryDescription
type*Enter the task type
- Supported types: hive, spark
terminationPolicy*Specify the cluster behavior after task scheduling ends
- never: Do not terminate the cluster
- onSuccess: Terminate the cluster only if the task succeeds
- always: Always terminate the cluster
hiveQuery(For Hive tasks) Enter the Hive query string to execute
fileUrl(For Hive tasks) Enter the Object Storage URL where the Hive query file is located
(For Spark tasks) Enter the Object Storage URL where the JAR file is located
deployMode(For Spark tasks) Specify the Spark task deployment mode: cluster or client
execOpts(For Hive tasks) Enter Hive options as a string for task execution
(For Spark tasks) Enter Spark options as a string for task execution
execParams(For Spark tasks) Enter parameters as a string to pass during task execution
logBucketNameEnter the Object Storage bucket name to store task scheduling logs
If logBucketName is set, logs will be stored under <logBucketName>/HadoopEco/<cluster name>

(Optional) If you want to integrate MySQL with the Hive metastore, include the hiveDbInfo object as shown below.

MySQL integration
"hiveDbInfo": {
"objectId": "string",
"dbName": "string",
"userId": "string",
"userPw": "string"
}
CategoryDescription
objectId*Enter the object ID of the MySQL to integrate
dbName*, userId*, userPw*Enter the MySQL database name, user ID, and password for access
- The password must be a Base64-encoded string

(Optional) If you want to integrate a data catalog, include the dataCatalogInfo object as shown below.

Data catalog integration

"dataCatalogInfo": {
"catalogId": "string"
}
CategoryDescription
catalogId*Enter the object ID of the data catalog to integrate

(Optional) If you want to integrate Redis, include the object ID as shown below.

Redis integration
"redisInfo": {
"objectId": "string",
"dbIdSupersetCelery": integer,
"dbIdSupersetResults": integer
}
CategoryDescription
objectId*Enter the object ID of the Redis to integrate
dbIdSupersetCeleryAn integer between 0 and 15 (if not set, defaults to 0, 1)
dbIdSupersetResultsAn integer between 0 and 15 (if not set, defaults to 0, 1)

Response

Response
{
"id": "48fd271f-01f8-47bd-8e42-8c872fb6bf3a",
"name": "hive-job-cluster",
"status": "Initializing"
}
CategoryDescription
idID of the created cluster
nameName of the created cluster
statusStatus of the cluster

Status codes

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden
409Duplicate cluster name

Cluster creation example 1

Cluster creation conditions

  • Core Hadoop type with version hde-2.0.1
  • No high availability applied, with 3 worker nodes
  • Security group automatically created
  • Cluster configuration override: Set the io.file.buffer.size field in HDFS core-site.xml to 65536
  1. Write the JSON for applying the configuration.

    JSON for applying configuration
    {
    "configurations": [
    {
    "classification": "core-site",
    "properties": {
    "io.file.buffer.size": "65536"
    }
    }
    ]
    }
  2. Run the following command to encode the JSON configuration file in Base64:

    echo | cat config.json | base64
  3. Enter the Base64 string into the configText field.

    ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg==
  • Apply Hive task scheduling and keep the cluster running after the task is complete.
  • Integrate with a data catalog:
    • The data catalog ID is required.

    • You can retrieve the ID from KakaoCloud Console > Analytics > Data Catalog.

      Cluster creation example 1
      curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
      --header 'Credential-ID: {credential-id}' \
      --header 'Credential-Secret: {credential-secret}' \
      --header 'Content-Type: application/json' \
      --data-raw '{
      "name": "hive-job-cluster",
      "clusterType": "hadoop",
      "clusterVersion": "hde-2.0.1",
      "isHa": false,
      "workerCnt": 3,
      "vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
      "subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
      "securityGroupName": "test-security-group1",
      "keypairName": "keypairname",
      "monitoring": true,
      "masterFlavorName": "m2a.xlarge",
      "workerFlavorName": "m2a.xlarge",
      "masterVolumeSize": 50,
      "workerVolumeSize": 100,
      "adminInfo": {
      "userId": "admin_user",
      "userPw": "<base_64_string>"
      },
      "config": {
      "hdfsReplication": 3,
      "hdfsBlockSize": 128,
      "configText": "ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg=="
      },
      "userTask": {
      "type": "hive",
      "terminationPolicy": "never",
      "hiveQuery": "create table if not exists t1 (col1 string); insert into table t1 values ('a'), ('b'), ('c');",
      "execOpts": "--hiveconf hive.tez.container.size=1024 --hiveconf hive.tez.java.opts=-Xmx1600m",
      "logBucketName": "user-bucket"
      },
      "dataCatalog": {
      "catalogId": "e1ebae48-daba-a4c5-56da-fbb2b684ae07"
      }
      }'

Cluster creation example 2

Cluster creation conditions

  • Core Hadoop type with version hde-2.0.1
  • High availability enabled, with 5 worker nodes
  • Apply user script via a file:
    • Upload the user script file to Object Storage.
    • Click the [More] icon on the uploaded file and select 'Copy object URL'.
    • If copied correctly, you will see a URL in the following format:
      https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{file}
    • Inject the copied URL into the userScriptfileUrl field.
  • Reuse an existing security group
  • Apply Spark task scheduling and automatically terminate the cluster upon successful task completion.
Cluster creation example 2
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "spark-job-cluster",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isHa": true,
"workerCnt": 5,
"vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
"subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
"securityGroupIds": ["56aa3028-b163-b114-baac-gg1dtfa1bec1", "76163abf-j11a-fbbd-a123-d1a4bedb123c"],
"keypairName": "keypairname",
"monitoring": true,
"masterFlavorName": "m2a.xlarge",
"workerFlavorName": "m2a.xlarge",
"masterVolumeSize": 50,
"workerVolumeSize": 100,
"config": {
"hdfsReplication": 3,
"hdfsBlockSize": 128,
"userScriptFileUrl": "https://objectstorage.{리전명}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/user_script.sh"
},
"userTask": {
"type": "spark",
"terminationPolicy": "onSuccess",
"fileUrl": "https://objectstorage.{리전명}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/sparkjob/spark_example.jar",
"deployMode": "cluster",
"execOpts": "--class org.apache.spark.examples.SparkPi --master yarn",
"logBucketName": "user-bucket"
}
}'

Cluster creation example 3

Cluster creation conditions

  • Core Hadoop type with version hde-2.0.1

  • No high availability applied, with 3 worker nodes

  • Use IAM account as the administrator account

  • Security group automatically created

  • Use a user script

    User script file script.sh
    #!/bin/bash

    function print(){
    msg=$1
    echo "${msg}"
    }

    print "hello world!"
    1. Run the following command to encode the script content in Base64:
    echo | cat script.sh | base64
    1. Enter the Base64 string into the userScript field.

      IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK
  • Apply environment variables using an Object Storage file

    • When applying the following environment variables for Ubuntu users:
    Apply environment variables for Ubuntu users
    export JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}
    export CUSTOM_HOME=/etc/custom
    1. Write the JSON for applying environment variables:
    Write JSON for applying environment variables
      {
    "configurations": [
    {
    "classification": "user-env:ubuntu",
    "properties": {
    "env": "export JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}\nexport CUSTOM_HOME=/etc/custom"
    }
    }
    ]
    }
    1. Upload the created JSON file (config.json) to Object Storage, copy the object URL, and enter it into the fileUrl field.
  • MySQL integration

    • You must pre-create a database to be used as the metadata database in MySQL as follows:
    CREATE DATABASE meta_db;
    • Prepare the object ID of the MySQL instance:

      • Go to the MySQL menu in the console → Navigate to the detailed screen of the target MySQL instance.
      • Extract the UUID-style object ID from Developer Tools or the address bar.
      • Viewing the MySQL object ID directly from the details screen will be improved in future updates.
    • Prepare accessible MySQL account credentials (ID, password).

      • The password must be Base64-encoded.
Cluster creation example 3
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "test-cluster",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isHa": false,
"workerCnt": 3,
"vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
"subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
"securityGroupName": "security-group-test",
"keypairName": "keypairname",
"monitoring": true,
"masterFlavorName": "m2a.xlarge",
"workerFlavorName": "m2a.xlarge",
"masterVolumeSize": 50,
"workerVolumeSize": 100,
"config": {
"hdfsReplication": 3,
"hdfsBlockSize": 128,
"configFileUrl": "https://objectstorage.{regionName}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/config.json",
"userScript": "IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK"
},
"hiveDbInfo": {
"objectId": "cfdbd7cd-f0ce-bbfb-de5b-e73bdbbcb0cf",
"dbName": "meta_db",
"userId": "mysql_user",
"userPw": "<base_64_string>"
}
}'

Retrieve cluster

Retrieve cluster information using the Open API. The endpoint is as follows:

kr-central-2 retrieve cluster
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Request

Path Parameter

TypeParameterData typeDescription
URL{cluster-id}*StringCluster ID
- Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information

Request Header

KeyValue
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key
Response Example
Response
{
"id": "string",
"name": "string",
"owner": "string",
"clusterType": "string",
"clusterVersion": "string",
"isHa": true,
"isScalable": true,
"status": "string",
"apiKeyStatus": "impossible|possible|applied",
"masterCnt": integer,
"workerCnt": integer,
"nodeCnt": integer,
"vpcId": "string",
"subnetId": "string",
"securityGroupIds": ["string"],
"keypairName": "string",
"monitoring": true|false,
"imageName": "string",
"nameNodeUrl": "string",
"resourceManagerUrl": "string",
"hueUrl": "string",
"hmasterUrl": "string",
"trinoUrl": "string",
"druidUrl": "string",
"supersetUrl": "string",
"createdAt": integer,
"runningAt": integer,
"elapsedTime": integer,
"totalVmUptime": integer,
"masterInfo": [
{
"flavorId": "string",
"flavorName": "string",
"volumeSize": integer
}
],
"workerInfo": [
{
"flavorId": "string",
"flavorName": "string",
"volumeSize": integer
}
],
"adminInfo": {
"userId": "string"
},
"config": {
"hdfsBlockSize": integer,
"hdfsReplication": integer,
"configText": "string",
"configFileUrl": "string"
},
"userTask": {
"type": "string",
"terminationPolicy": "string",
"fileUrl": "string",
"hiveQuery": "string",
"deployMode": "string",
"execOpts": "string",
"execParams": "string",
"logBucketName": "string",
"logUrl": "string"
},
"hiveDbInfo": {
"objectId": "string",
"subnetId": "string",
"name": "string",
"host": "string",
"port": integer,
"dbName": "string",
"userId": "string"
},
"dataCatalogInfo": {
"metastoreUri": "string",
"catalogId": "string",
"catalogName": "string",
"vpcId": "string",
"vpcName": "string",
"subnetId": "string",
"subnetName": "string"
},
"redisInfo": {
"objectId": "string",
"name": "string",
"subnetId": "string",
"primaryEndpoint": "string",
"readEndpoint": "string",
"clusterEnabled": true,
"port": integer,
"dbIdSupersetCelery": integer,
"dbIdSupersetResults": integer
}
}

Response

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden
404Cluster not found

Retrieve cluster list

Retrieve a list of clusters using the Open API. The endpoint is as follows:

kr-central-2 retrieve cluster list
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Request

Request Header

KeyValue
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key

Query parameter

You can filter the cluster list using the following conditions. When using multiple conditions, connect them with &.

KeyData typeDescriptionRequiredDefault
sizeintegerPage size, indicating the number of clusters displayed per page.Optional20
pageintegerPage number to retrieve.
The available page numbers depend on the page size.
Optional0
includeTerminatedbooleanWhether to include terminated clusters in the response list.Optionaltrue
sortstringField and sort order for sorting results.

Specify the field and order (asc or desc) separated by a comma.
For example, to sort the name field in descending order, use sort=name,desc.
For multiple sort conditions, priority is applied in the specified order.

Sortable fields:
id, name, status, owner, clusterType, clusterVersion, apiKeyStatus, masterCnt,
workerCnt, createdAt, runningAt, elapsedTime
OptionalcreatedAt,desc
namestringCluster name for like search. If not specified, all clusters are retrieved.Optional
apiKeyStatusstringOpen API Key status for like search.

Search values: possible, impossible, applied.
They represent applicable, not applicable, and in-progress, respectively. If not specified, all values are retrieved.
Optional

Response

Response Example

Response
{
"content": [
{
"id": "string",
"name": "string",
"status": "string",
"owner": "string",
"clusterType": "string",
"clusterVersion": "string",
"isScalable": boolean,
"apiKeyStatus": "string",
"masterCnt": integer,
"workerCnt": integer,
"nodeCnt": integer,
"createdAt": long,
"runningAt": long,
"elapsedTime": long
}
],
"pageable": {
"pageNumber": integer,
"pageSize": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"offset": integer,
"paged": boolean,
"unpaged": boolean
},
"totalPages": integer,
"totalElements": integer,
"last": boolean,
"numberOfElements": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"size": integer,
"first": boolean,
"number": integer,
"empty": boolean
}
CategoryDescription
contentList containing the data
idCluster ID
nameCluster name
statusCluster status
ownerCluster owner
clusterTypeCluster type
clusterVersionCluster version
isScalableWhether the cluster is scalable
apiKeyStatusOpen API Key status of the cluster
masterCntNumber of master nodes
workerCntNumber of worker nodes
nodeCntTotal number of nodes
createdAtUnix timestamp when the cluster was created
runningAtUnix timestamp when the cluster entered the Running state
elapsedTimeUptime (ms)
pageablePaging-related information
pageNumberCurrent page number
pageSizeCurrent page size
sortSorting information
sortedWhether sorting is applied
unsortedWhether sorting is not applied
emptyWhether the sorting conditions are empty
offsetStarting point of the page
pagedWhether paging is applied
unpagedWhether paging is not applied
totalPagesTotal number of pages
totalElementsTotal number of data entries
lastWhether the current page is the last page
sizeCurrent page size
firstWhether the current page is the first page
numberCurrent page number
emptyWhether the current page is empty

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden

Cluster list retrieval example 1

List retrieval conditions

  • Page size is set to 3, and page number is set to 2
  • Include terminated clusters
  • Apply sorting by name in descending order and status in ascending order
kr-central-2 retrieve cluster list 1
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=0&size=3&includeTerminated=true&sort=name,desc&sort=status,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Response example 1

Cluster list retrieval example 1 response
{
"content": [
{
"id": "fa70ff9f-98a2-446c-9f33-75d7dce7c160",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1716359078000,
"runningAt": 1716359422659,
"elapsedTime": 12108694
},
{
"id": "3ec42a8d-a5a3-42ef-9c34-e270db0354b1",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1716536420000,
"runningAt": 1716536754352,
"elapsedTime": 265434374
},
{
"id": "d66c3ff5-29f3-4f2d-a611-4d343573cc17",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1717049297000,
"runningAt": 1717049871150,
"elapsedTime": 727811
}
],
"pageable": {
"pageNumber": 2,
"pageSize": 3,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 6,
"unpaged": false,
"paged": true
},
"totalPages": 32,
"totalElements": 96,
"last": false,
"size": 3,
"number": 2,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 3,
"first": false,
"empty": false
}

Cluster list retrieval example 2

List retrieval conditions

  • Page size is set to 5, and page number is set to 0
  • Exclude terminated clusters
  • Filter clusters with test in their name
  • Apply sorting by createdAt in ascending order
  • Filter clusters with apiKeyStatus set to impossible
kr-central-2 retrieve cluster list 2
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=0&size=5&includeTerminated=false&name=test&sort=createdAt,asc&apiKeyStatus=impossibl' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Response example 2

Cluster list retrieval example 2 response
{
"content": [
{
"id": "90878bcd-0f38-4338-810e-40d6fbfd8b45",
"name": "andy-cluster-test-01",
"status": "Running",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": true,
"apiKeyStatus": "impossible",
"masterCnt": 1,
"workerCnt": 1,
"nodeCnt": 2,
"createdAt": 1720797464322,
"runningAt": 1720797856006,
"elapsedTime": 827416064
},
{
"id": "eb9a91ad-d1b6-462c-8e20-0e8b2c0aa074",
"name": "test",
"status": "Running",
"owner": "user@kakaoenterprise.com",
"clusterType": "dataflow",
"clusterVersion": "hde-2.0.1",
"isScalable": true,
"apiKeyStatus": "impossible",
"masterCnt": 1,
"workerCnt": 2,
"nodeCnt": 3,
"createdAt": 1721131126322,
"runningAt": 1721131467376,
"elapsedTime": 493804694
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 5,
"sort": {
"empty": false,
"unsorted": false,
"sorted": true
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 2,
"totalPages": 1,
"last": true,
"size": 5,
"number": 0,
"sort": {
"empty": false,
"unsorted": false,
"sorted": true
},
"numberOfElements": 2,
"first": true,
"empty": false
}

Retrieve VM list of cluster

Retrieve the list of VMs in a cluster using the Open API. The endpoint is as follows:

kr-central-2 retrieve VM list of a cluster
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/vms' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Request

Request header

KeyValue
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key

Query parameter

You can filter the VM list using the following conditions. When using multiple conditions, connect them with &.

KeyData typeDescriptionRequiredDefault
sizeintegerPage size, indicating the number of VMs displayed per page.Optional20
pageintegerPage number to retrieve.
The available page numbers depend on the page size.
Optional0
ipbooleanPrivate IP for like search. If not specified, all VMs are retrieved.Optional
namestringName for like search. If not specified, all VMs are retrieved.Optional
instanceIdstringInstance ID for like search. If not specified, all VMs are retrieved.Optional
sortstringField and sort order for sorting results. Specify the field and order (asc or desc) separated by a comma.
For example, to sort the name field in descending order, use sort=name,desc.
For multiple sort conditions, priority is applied in the specified order.

Sortable fields:
name, instanceId, isMaster, flavorName, ip, floatingIp, createdAt
OptionalcreatedAt,desc

Response

Response Example

Response
{
"content": [
{
"name": "string",
"instanceId": "string",
"index": integer,
"isMaster": boolean,
"status": "string",
"flavorName": "string",
"ip": "string",
"floatingIp": "string",
"createdAt": long,
}
],
"pageable": {
"pageNumber": integer,
"pageSize": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"offset": integer,
"paged": boolean,
"unpaged": boolean
},
"totalPages": integer,
"totalElements": integer,
"last": boolean,
"numberOfElements": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"size": integer,
"first": boolean,
"number": integer,
"empty": boolean
}
CategoryDescription
contentList containing the data
nameCluster name
instanceIdVM instance ID
indexVM index
For example, if HadoopMST-test-1, it indicates master VM 1, with an index value of 1
isMasterWhether the VM is a master node
statusVM status
flavorNameVM flavor name
ipVM private IP
floatingIpVM public IP
createdAtTimestamp when the VM was created
pageablePaging-related information
pageNumberCurrent page number
pageSizeCurrent page size
sortSorting information
sortedWhether sorting is applied
unsortedWhether sorting is not applied
emptyWhether the sorting conditions are empty
offsetStarting point of the page
pagedWhether paging is applied
unpagedWhether paging is not applied
totalPagesTotal number of pages
totalElementsTotal number of data entries
lastWhether the current page is the last page
sizeCurrent page size
firstWhether the current page is the first page
numberCurrent page number
emptyWhether the current page is empty

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden

Cluster VM list retrieval example 1

VM list retrieval conditions

  • Retrieve VM list for cluster ID 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
  • Page size is set to 10, and page number is set to 0
  • Apply sorting by name in ascending order
kr-central-2 retrieve VM list of a cluster 1
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/bbe8ce14-3497-476e-894c-142c48c9c469/vms?sort=name,asc&page=0&size=10' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Response example 1

Cluster VM list retrieval example 1 response
{
"content": [
{
"name": "HadoopMST-test-cluster-1",
"instanceId": "23948ac6-86e0-42f6-acdb-d214cdd200c0",
"index": 1,
"isMaster": true,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.129.243",
"createdAt": 1719452777733
},
{
"name": "HadoopWRK-test-cluster-1",
"instanceId": "8606bbb2-52bf-4154-82f5-647c9da4ac2d",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.132.127",
"createdAt": 1719452777733
},
{
"name": "HadoopWRK-test-cluster-2",
"instanceId": "7db47633-5676-42ad-bf34-c6e5da5c30c7",
"index": 2,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.131.196",
"createdAt": 1719413846446
},
{
"name": "HadoopWRK-test-cluster-3",
"instanceId": "2fdda50e-cf40-4976-a394-7ead7e32f042",
"index": 3,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.128.201",
"createdAt": 1719413114832
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 10,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"unpaged": false,
"paged": true
},
"totalPages": 1,
"totalElements": 4,
"last": true,
"size": 10,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 4,
"first": true,
"empty": false
}

Cluster VM list retrieval example 2

VM list retrieval conditions

  • Retrieve VM list for cluster ID 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
  • Page size is set to 10, and page number is set to 0
  • Apply a filter to include WRK in name
  • Apply sorting by name in descending order and status in ascending order
kr-central-2 retrieve VM list of a cluster 2
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=WRK&page=0&size=10&sort=name,desc&sort=status,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Response example 2

Cluster VM list retrieval example 2 response
{
"content": [
{
"name": "HadoopWRK-hailey13-2",
"instanceId": "10c5dfab-cae7-4b8c-a975-7b5d285482aa",
"index": 2,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.2.159",
"createdAt": 1721202811333
},
{
"name": "HadoopWRK-hailey13-1",
"instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.3.96",
"createdAt": 1721202811333
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 10,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 2,
"totalPages": 1,
"last": true,
"size": 10,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 2,
"first": true,
"empty": false
}

Cluster VM list retrieval example 3

VM list retrieval conditions

  • Retrieve VM list for cluster ID 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
  • Page size is set to 5, and page number is set to 0
  • Apply a filter to include hailey13 in name and -4c3b in instanceId
  • Apply sorting by isMaster in descending order and ip in ascending order
kr-central-2 retrieve VM list of a cluster 3
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=hailey13&instanceId=-4c3b&page=0&size=5&sort=isMaster,desc&sort=ip,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Response example 3

Cluster VM list retrieval example 3 response
{
"content": [
{
"name": "HadoopWRK-hailey13-1",
"instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.3.96",
"createdAt": 1721202811333
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 5,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 1,
"totalPages": 1,
"last": true,
"size": 5,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 1,
"first": true,
"empty": false
}

Delete cluster

Delete a cluster using the Open API. Deleted clusters cannot be recovered. The endpoint is as follows:

kr-central-2 delete cluster
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Request

Path Parameter

TypeParameterData typeDescription
URL{cluster-id}*StringCluster ID
- Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information

Request header

KeyValue
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key

Response

Response Example
Response
{
"id": "string",
"name": "string",
"status": "string"
}

Cluster deletion example

Deletion conditions

To delete a cluster with the ID 90ac14f2-3837-11ef-b1da-72300678fa60, make the following request:

kr-central-2 delete cluster example
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden
404Cluster not found

Scale out cluster

Scale out worker nodes in a cluster using the Open API. The endpoint is as follows:

kr-central-2 scale out cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-out' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'

Request

Path Parameter

TypeParameterData typeDescription
URL{cluster-id}*StringCluster ID
- Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information

Request header

KeyValue
Content-Type*Fixed as application/json
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key

Request body

Enter the number of nodes to scale out.

증설할 노드 개수를 입력합니다.

Response
{
"scaleCnt": integer
}

Response

Response Example
Response
{
"id": "string",
"name": "string",
"status": "string"
}

Cluster scale-out example

Scale-out conditions

To scale out the cluster with ID 90ac14f2-3837-11ef-b1da-72300678fa60 by adding 3 nodes, make the following request:

kr-central-2 scale out cluster example
curl -X POST ‘https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-out’ \
--header ‘Credential-ID: {credential-id}\
--header ‘Credential-Secret: {credential-secret}\
--header ‘Content-Type: application/json’ \
--data-raw '
{
“scaleCnt”: 3
}'

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden
404Cluster not found

Scale in cluster

Scale in worker nodes in a cluster using the Open API. The endpoint is as follows:

kr-central-2 scale in cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-in' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'

Request

Path Parameter

종류파라미터유형설명
URL{cluster-id}*String클러스터의 ID
- 카카오클라우드 콘솔 > Analytics > Hadoop Eco > 좌측 Cluster 메뉴 > 클러스터 정보에서 확인 가능

Request Header

TypeParameterData typeDescription
URL{cluster-id}*StringCluster ID
- Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information

Request header

KeyDescription
Content-Type*Fixed as application/json
{credential-id}*User's access key ID
- Available from KakaoCloud console > Top-right profile > Access key
{credential-secret}*User's secret access key
- Only available when issuing the access key
- If the secret access key is lost, issue a new access key

Request body

Enter the number of nodes to scale in.

  • The remaining number of nodes after scaling in must be greater than or equal to the cluster's hdfs.replication.
Request Body
{
"scaleCnt": integer
}

Response

Response Example
Response
{
"id": "string",
"name": "string",
"status": "string"
}

Cluster scale-in example

Scale-in conditions

  • To scale in the cluster with ID 90ac14f2-3837-11ef-b1da-72300678fa60 by removing 5 nodes, make the following request.
  • The remaining number of worker nodes after scaling in must not be less than the HDFS replication count set during cluster creation.
kr-central-2 scale in cluster example
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-in' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '
{
"scaleCnt": 5
}'

Status Code

CodeDescription
200Success
400Bad request
401Credential authentication failed
403Forbidden
404Cluster not found