Hadoop Eco Open API

Before getting started

The Hadoop Eco Open API supports general cluster operations such as creating new clusters, retrieving cluster information, deleting clusters, and scaling. When using the Hadoop Eco Open API, issuing an Open API key is not required.

The usage flow for the Hadoop Eco Open API is as follows.

Prepare API usage

To call the Hadoop Eco Open API, you must issue access key credentials.

API endpoint

The Hadoop Eco API endpoint URL for requests is as follows:

API endpoint URL format

https://hadoop-eco.kr-central-2.kakaocloud.com

Base64 encoding method

Some values must be encoded in Base64 before being used in Open API requests. Encoding methods are as follows:

How to encode the string 'hello' in Base64 using a terminal
1. Linux: echo "hello" | base64

2. OS X: echo "hello" | base64

3. Windows: echo hello > input.txt
            certutil -encode "input.txt" "output.txt"
            type output.txt

Create cluster

Creates a cluster using the Open API. The endpoint is:

Create cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{Body}'

Request

Request header

Key	Value
Content-Type*	Fixed to `application/json`
`{credential-id}`*	User’s access key ID - Can be found at KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User’s secret access key - Only available at time of key creation - If lost, a new IAM access key must be issued

Request body

Enter basic information

Provide basic information such as the cluster name, type, version, network, and availability.

Request body
{
  "name": "string",
  "clusterType": "string",
  "clusterVersion": "string",
  "isHa": true | false,
  "keypairName": "string",
  "masterFlavorName": "string",
  "masterVolumeSize": integer,
  "workerCnt": integer,
  "workerFlavorName": "string",
  "workerVolumeSize": integer,
  "monitoring": true,
  "securityGroupIds": ["string"],
  "securityGroupName": "string",
  "vpcId": "string",
  "subnetId": "string",
  "adminInfo": {
    "userId": "string",
    "userPw": "string"
  },
  "config": {
    "hdfsReplication": integer,
    "hdfsBlockSize": integer,
    "configText": "string",
    "fileUrl": "string",
    "userScript": "string"
  }
}

Field	Type	Description
name*	String	Cluster name
clusterType*	String	Cluster type
clusterVersion*	String	Cluster version
isHa*	Boolean	High availability setting - `true`: enabled - `false`: disabled
keypairName*	String	Name of the key pair used for virtual machines in the cluster
masterFlavorName*	String	Flavor type name for the master node - e.g., m2a.xlarge
masterVolumeSize*	Integer	Volume size (GB) for the master node
workerCnt*	Integer	Number of worker nodes; must be equal to or greater than `config.hdfsReplication`
workerFlavorName*	String	Flavor type name for worker nodes - e.g., m2a.xlarge
workerVolumeSize*	Integer	Volume size (GB) for worker nodes
monitoring*	Boolean	Whether to install a monitoring agent
securityGroupIds, securityGroupName	String	Either `securityGroupIds` or `securityGroupName` must be provided - Use `securityGroupIds` to reuse an existing security group - Use `securityGroupName` to auto-create a new security group during cluster creation
vpcId*	String	VPC ID where the cluster will be deployed
subnetId*	String	Subnet ID where the cluster will be deployed
adminInfo.userId	String	User ID for Hue or Superset - If omitted, defaults to IAM account ID
adminInfo.userPw	String	Password for Hue or Superset - If omitted, defaults to IAM password - To set a custom password, input a Base64-encoded string
config.hdfsReplication*	Integer	Number of HDFS replications - Must be equal to or less than `workerCnt`
config.BlockSize*	Integer	Block size in MB
config.configText	String	Cluster configuration data - Must be JSON stringified and Base64-encoded - See Configuration injection example
config.fileUrl	String	Object Storage file URL - No Base64 encoding required Example: `https://objectstorage.{region}.kakaocloud.com/v1/<project ID>/<bucket>/<path>/<filename>`
config.userScript	String	User script to be executed when virtual machines are created - Input the script string as Base64-encoded Bash shell commands

Configuration injection example using `fileUrl` or `configText`

You can inject configuration values via either a JSON string or a file stored in Object Storage.
For instance, if you want to inject the settings shown below, first convert them into JSON format as described in Configuration injection.

Configuration examples
Write configuration as JSON

core-site.xml
<configuration>
... (omitted) ...
    <property>
        <name>io.file.buffer.size</name>
        <value>65536</value>
    </property>
... (omitted) ...
</configuration>

hive-site.xml
<configuration>
... (omitted) ...
    <property>
        <name>mapred.max.split.size</name>
        <value>128000000</value>
    </property>
... (omitted) ...
</configuration>

hadoop-env.sh
... (omitted) ...
export HADOOP_HEAPSIZE="3001"
export HADOOP_NAMENODE_INIT_HEAPSIZE="-Xmx3002m"
... (omitted) ...

spark-defaults.conf
... (omitted) ...
spark.driver.memory                     4000M
spark.network.timeout                   800s
... (omitted) ...

To input the configuration values, write them in JSON format as follows:

config.json
{
  "configurations": [
    {
      "classification": "core-site",
      "properties": {
        "io.file.buffer.size": "65536"
      }
    },
    {
      "classification": "hive-site",
      "properties": {
        "mapred.max.split.size": "128000000"
      }
    },
    {
      "classification": "hadoop-env",
      "properties": {
        "hadoop_env_hadoop_heapsize": 3001,
        "hadoop_env_hadoop_namenode_heapsize": "-Xmx3002m"
      }
    },
    {
      "classification": "spark-defaults",
      "properties": {
        "spark.driver.memory": "4000M",
        "spark.network.timeout": "800s"
      }
    }
  ]
}

Inject configuration settings

Once your JSON configuration is ready, there are two ways to inject it when creating a cluster:

Method 1

The first method is to Base64-encode the JSON and inject it into the configText field.

To Base64-encode the JSON configuration file, run the following command:
```
echo | cat config.json | base64
```

Enter the Base64-encoded string into the configText field.

ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAiaGl2ZS1zaXRlIiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgIm1hcHJlZC5tYXguc3BsaXQuc2l6ZSI6ICIxMjgwMDAwMDAiCiAgICAgIH0KICAgIH0sCgl7CiAgICAgICJjbGFzc2lmaWNhdGlvbiI6ICJoYWRvb3AtZW52IiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgImhhZG9vcF9lbnZfaGFkb29wX2hlYXBzaXplIjogMzAwMSwKICAgICAgICAiaGFkb29wX2Vudl9oYWRvb3BfbmFtZW5vZGVfaGVhcHNpemUiOiAiLVhteDMwMDJtIgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAic3BhcmstZGVmYXVsdHMiLAogICAgICAicHJvcGVydGllcyI6IHsKICAgICAgICAic3BhcmsuZHJpdmVyLm1lbW9yeSI6ICI0MDAwTSIsCiAgICAgICAgInNwYXJrLm5ldHdvcmsudGltZW91dCI6ICI4MDBzIgogICAgICB9CiAgICB9CiAgXQp9Cg==

Method 2

The second method is to upload the configuration file to Object Storage and inject the object's URL into the fileUrl field.

Upload the JSON file to Object Storage.
Click the More icon next to the uploaded file and select Copy object URL.
The copied URL should resemble:
- https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{filename}
Paste the URL into the fileUrl field.

Example: Inject script into `userScript`

Prepare your user script file:

User script
#!/bin/bash

function print(){
    msg=$1
    echo "${msg}"
}

print "hello world!"

Base64-encode the script using:
```
echo | cat script.sh | base64
```

Paste the encoded string into the userScript field:

IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK

(Optional) To run job scheduling after cluster installation, add the userTask object:

userTask
"userTask": {
  "type": "hive|spark",
  "terminationPolicy": "never|onSuccess|always",
  "fileUrl": "string",
  "hiveQuery": "string",
  "deployMode": "cluster|client",
  "execOpts": "string",
  "execParams": "string",
  "logBucketName": "string"
}

Field	Description
type*	Type of job to run - Supported values: `hive`, `spark`
terminationPolicy*	Cluster behavior after job completion - `never`: do not terminate - `onSuccess`: terminate only on success - `always`: always terminate
hiveQuery	(For Hive) Hive query string
fileUrl	(For Hive) Object Storage URL of Hive query file (For Spark) Object Storage URL of JAR file
deployMode	(For Spark) Spark job deploy mode: `cluster` or `client`
execOpts	Job execution options as a string - Hive or Spark options
execParams	(For Spark) Parameters to pass to the application
logBucketName	Object Storage bucket name where logs will be saved Logs will be saved under `<logBucketName>/HadoopEco/<cluster name>`

(Optional) To integrate MySQL as the Hive metastore, add the hiveDbInfo object:

MySQL integration
"hiveDbInfo": {
  "objectId": "string",
  "dbName": "string",
  "userId": "string",
  "userPw": "string"
}

Field	Description
objectId*	Object ID of the MySQL instance
dbName, userId, userPw*	MySQL DB name, user ID, and Base64-encoded password

(Optional) To integrate a data catalog, add the dataCatalogInfo object:

Data catalog integration
"dataCatalogInfo": {
  "catalogId": "string"
}

Field	Description
catalogId*	Object ID of the data catalog to integrate

(Optional) To integrate Redis, add the following object:

Redis integration
"redisInfo": {
  "objectId": "string",
  "dbIdSupersetCelery": integer,
  "dbIdSupersetResults": integer
}

Field	Description
objectId*	Object ID of the Redis instance
dbIdSupersetCelery	Integer from 0–15 (defaults to `0`, `1` if not set)
dbIdSupersetResults	Integer from 0–15 (defaults to `0`, `1` if not set)

Response

Example response
{
  "id": "48fd271f-01f8-47bd-8e42-8c872fb6bf3a",
  "name": "hive-job-cluster",
  "status": "Initializing"
}

Field	Description
id	ID of the created cluster
name	Name of the created cluster
status	Cluster status

Status codes

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized
`409`	Cluster name already exists

Cluster creation example 1

Cluster specification

Core Hadoop, version hde-2.0.1
No high availability
3 worker nodes
Auto-created security group
Override io.file.buffer.size in HDFS core-site.xml to 65536

Write JSON for the configuration override:

JSON for configuration override
{
  "configurations": [
    {
      "classification": "core-site",
      "properties": {
        "io.file.buffer.size": "65536"
      }
    }
  ]
}

Base64-encode the JSON:
```
echo | cat config.json | base64
```
Insert the Base64 string into the configText field:
```
ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg==
```
- Hive job scheduling is configured and the cluster remains after the job completes
- Data catalog integration requires a valid catalog ID
  - You can find it in KakaoCloud Console > Analytics > Data Catalog

Cluster creation example 1
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
  "name": "hive-job-cluster",
  "clusterType": "hadoop",
  "clusterVersion": "hde-2.0.1",
  "isHa": false,
  "workerCnt": 3,
  "vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
  "subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
  "securityGroupName": "test-security-group1",
  "keypairName": "keypairname",
  "monitoring": true,
  "masterFlavorName": "m2a.xlarge",
  "workerFlavorName": "m2a.xlarge",
  "masterVolumeSize": 50,
  "workerVolumeSize": 100,
  "adminInfo": {
    "userId": "admin_user",
    "userPw": "<base_64_string>"
  },
  "config": {
    "hdfsReplication": 3,
    "hdfsBlockSize": 128,
    "configText": "ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg=="
  },
  "userTask": {
    "type": "hive",
    "terminationPolicy": "never",
    "hiveQuery": "create table if not exists t1 (col1 string); insert into table t1 values ('a'), ('b'), ('c');",
    "execOpts": "--hiveconf hive.tez.container.size=1024 --hiveconf hive.tez.java.opts=-Xmx1600m",
    "logBucketName": "user-bucket"
  },
  "dataCatalog": {
    "catalogId": "e1ebae48-daba-a4c5-56da-fbb2b684ae07"
  }
}'

Cluster creation example 2

Cluster specifications

Core Hadoop, version hde-2.0.1
High availability enabled, 5 worker nodes
Apply a user script from a file
- Upload the script to Object Storage
- Click the More icon for the uploaded file and select Copy object URL
- A properly copied URL will look like:
  https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{filename}
- Inject this URL into the userScriptFileUrl field
Reuse existing security groups
Configure Spark job scheduling with automatic cluster termination on success

Cluster creation example 2
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
  "name": "spark-job-cluster",
  "clusterType": "hadoop",
  "clusterVersion": "hde-2.0.1",
  "isHa": true,
  "workerCnt": 5,
  "vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
  "subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
  "securityGroupIds": ["56aa3028-b163-b114-baac-gg1dtfa1bec1", "76163abf-j11a-fbbd-a123-d1a4bedb123c"],
  "keypairName": "keypairname",
  "monitoring": true,
  "masterFlavorName": "m2a.xlarge",
  "workerFlavorName": "m2a.xlarge",
  "masterVolumeSize": 50,
  "workerVolumeSize": 100,
  "config": {
    "hdfsReplication": 3,
    "hdfsBlockSize": 128,
    "userScriptFileUrl": "https://objectstorage.{region}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/user_script.sh"
  },
  "userTask": {
    "type": "spark",
    "terminationPolicy": "onSuccess",
    "fileUrl": "https://objectstorage.{region}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/sparkjob/spark_example.jar",
    "deployMode": "cluster",
    "execOpts": "--class org.apache.spark.examples.SparkPi --master yarn",
    "logBucketName": "user-bucket"
  }
}'

Cluster creation example 3

Cluster specifications

Core Hadoop, version hde-2.0.1
No high availability, 3 worker nodes
Use IAM account as the administrator
Automatically create security group
Use inline user script:

User script file: script.sh
#!/bin/bash

function print(){
    msg=$1
    echo "${msg}"
}

print "hello world!"

Base64-encode the script:

echo | cat script.sh | base64

Paste the encoded result into the userScript field:

IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK

Apply environment variables via Object Storage file for Ubuntu users:

Environment variable script for Ubuntu
export JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}
export CUSTOM_HOME=/etc/custom

Create the following configuration JSON:

Environment variable configuration JSON
{
  "configurations": [
    {
      "classification": "user-env:ubuntu",
      "properties": {
        "env": "export JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}\nexport CUSTOM_HOME=/etc/custom"
      }
    }
  ]
}

Upload this config.json to Object Storage and copy its URL into the configFileUrl field.

Integrate with MySQL (for Hive metastore)
- You must pre-create the metadata database:

CREATE DATABASE meta_db;

Extract the MySQL object ID from the console's developer tools or URL
Prepare MySQL credentials (password must be Base64-encoded)

Cluster creation example 3
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{
  "name": "test-cluster",
  "clusterType": "hadoop",
  "clusterVersion": "hde-2.0.1",
  "isHa": false,
  "workerCnt": 3,
  "vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
  "subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
  "securityGroupName": "security-group-test",
  "keypairName": "keypairname",
  "monitoring": true,
  "masterFlavorName": "m2a.xlarge",
  "workerFlavorName": "m2a.xlarge",
  "masterVolumeSize": 50,
  "workerVolumeSize": 100,
  "config": {
    "hdfsReplication": 3,
    "hdfsBlockSize": 128,
    "configFileUrl": "https://objectstorage.{region}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/config.json",
    "userScript": "IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK"
  },
  "hiveDbInfo": {
    "objectId": "cfdbd7cd-f0ce-bbfb-de5b-e73bdbbcb0cf",
    "dbName": "meta_db",
    "userId": "mysql_user",
    "userPw": "<base_64_string>"
  }
}'

Retrieve cluster

Retrieve cluster information via the Open API using the following endpoint:

Retrieve cluster
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Request

Path parameter

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	Cluster ID - Can be found in KakaoCloud Console > Analytics > Hadoop Eco > Cluster menu

Request header

Key	Value
`{credential-id}`*	User's access key ID - Available from KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User's secret access key - Only viewable at key creation time - If lost, Get IAM access key

Response example

Example response
{
  "id": "string",
  "name": "string",
  "owner": "string",
  "clusterType": "string",
  "clusterVersion": "string",
  "isHa": true,
  "isScalable": true,
  "status": "string",
  "apiKeyStatus": "impossible|possible|applied",
  "masterCnt": integer,
  "workerCnt": integer,
  "nodeCnt": integer,
  "vpcId": "string",
  "subnetId": "string",
  "securityGroupIds": ["string"],
  "keypairName": "string",
  "monitoring": true|false,
  "imageName": "string",
  "nameNodeUrl": "string",
  "resourceManagerUrl": "string",
  "hueUrl": "string",
  "hmasterUrl": "string",
  "trinoUrl": "string",
  "druidUrl": "string",
  "supersetUrl": "string",
  "createdAt": integer,
  "runningAt": integer,
  "elapsedTime": integer,
  "totalVmUptime": integer,
  "masterInfo": [
    {
      "flavorId": "string",
      "flavorName": "string",
      "volumeSize": integer
    }
  ],
  "workerInfo": [
    {
      "flavorId": "string",
      "flavorName": "string",
      "volumeSize": integer
    }
  ],
  "adminInfo": {
    "userId": "string"
  },
  "config": {
    "hdfsBlockSize": integer,
    "hdfsReplication": integer,
    "configText": "string",
    "configFileUrl": "string"
  },
  "userTask": {
    "type": "string",
    "terminationPolicy": "string",
    "fileUrl": "string",
    "hiveQuery": "string",
    "deployMode": "string",
    "execOpts": "string",
    "execParams": "string",
    "logBucketName": "string",
    "logUrl": "string"
  },
  "hiveDbInfo": {
    "objectId": "string",
    "subnetId": "string",
    "name": "string",
    "host": "string",
    "port": integer,
    "dbName": "string",
    "userId": "string"
  },
  "dataCatalogInfo": {
    "metastoreUri": "string",
    "catalogId": "string",
    "catalogName": "string",
    "vpcId": "string",
    "vpcName": "string",
    "subnetId": "string",
    "subnetName": "string"
  },
  "redisInfo": {
    "objectId": "string",
    "name": "string",
    "subnetId": "string",
    "primaryEndpoint": "string",
    "readEndpoint": "string",
    "clusterEnabled": true,
    "port": integer,
    "dbIdSupersetCelery": integer,
    "dbIdSupersetResults": integer
  }
}

Response

Status codes

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized
`404`	Cluster not found

Retrieve cluster list

Use the Open API to retrieve a list of clusters. The endpoint is as follows:

Retrieve cluster list
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Request

Request header

Key	Value
`{credential-id}`*	User's access key ID - Available from KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User's secret access key - Only viewable at time of key creation - If lost, Get IAM access key

Query parameters

You can filter the cluster list using the following optional parameters. Use & to chain multiple conditions.

Key	Type	Description	Required	Default
size	integer	Number of clusters per page	optional	20
page	integer	Page number to retrieve	optional	0
includeTerminated	boolean	Whether to include terminated clusters in the response	optional	true
sort	string	Sorting field and order Format: `field,asc	desc`<br/> For multiple fields, separate with`&sort=`<br/>Example:`sort=name,desc&sort=status,asc` Sortable fields: id, name, status, owner, clusterType, clusterVersion, apiKeyStatus, masterCnt, workerCnt, createdAt, runningAt, elapsedTime	optional
name	string	Filter clusters where name contains this string	optional
apiKeyStatus	string	Filter by Open API key status: `possible`, `impossible`, or `applied`	optional

Response

Response example

Response example
{
  "content": [
    {
      "id": "string",
      "name": "string",
      "status": "string",
      "owner": "string",
      "clusterType": "string",
      "clusterVersion": "string",
      "isScalable": boolean,
      "apiKeyStatus": "string",
      "masterCnt": integer,
      "workerCnt": integer,
      "nodeCnt": integer,
      "createdAt": long,
      "runningAt": long,
      "elapsedTime": long
    }
  ],
  "pageable": {
    "pageNumber": integer,
    "pageSize": integer,
    "sort": {
      "sorted": boolean,
      "unsorted": boolean,
      "empty": boolean
    },
    "offset": integer,
    "paged": boolean,
    "unpaged": boolean
  },
  "totalPages": integer,
  "totalElements": integer,
  "last": boolean,
  "numberOfElements": integer,
  "sort": {
    "sorted": boolean,
    "unsorted": boolean,
    "empty": boolean
  },
  "size": integer,
  "first": boolean,
  "number": integer,
  "empty": boolean
}

Field	Description
content	List of cluster entries
id	Cluster ID
name	Cluster name
status	Cluster status
owner	Cluster owner
clusterType	Cluster type
clusterVersion	Cluster version
isScalable	Whether the cluster is scalable
apiKeyStatus	Open API key status
masterCnt	Number of master nodes
workerCnt	Number of worker nodes
nodeCnt	Total number of nodes
createdAt	Unix timestamp of cluster creation
runningAt	Unix timestamp when the cluster entered `Running` state
elapsedTime	Uptime in milliseconds
pageable, sort, ...	Pagination and sorting metadata

Status codes

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized

Cluster list retrieval example 1

Conditions

Page size: 3, Page number: 2
Include terminated clusters
Sort by name descending, then status ascending

Cluster list retrieval example 1
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=2&size=3&includeTerminated=true&sort=name,desc&sort=status,asc' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Example 1 response

Example 1 response
{
  "content": [
    {
      "id": "fa70ff9f-98a2-446c-9f33-75d7dce7c160",
      "name": "sally-kbctest",
      "status": "Terminated(User)",
      "owner": "user@kakaoenterprise.com",
      "clusterType": "hadoop",
      "clusterVersion": "hde-2.0.1",
      "isScalable": false,
      "apiKeyStatus": "impossible",
      "masterCnt": 0,
      "workerCnt": 0,
      "nodeCnt": 0,
      "createdAt": 1716359078000,
      "runningAt": 1716359422659,
      "elapsedTime": 12108694
    },
    {
      "id": "3ec42a8d-a5a3-42ef-9c34-e270db0354b1",
      "name": "sally-kbctest",
      "status": "Terminated(User)",
      "owner": "user@kakaoenterprise.com",
      "clusterType": "hadoop",
      "clusterVersion": "hde-2.0.1",
      "isScalable": false,
      "apiKeyStatus": "impossible",
      "masterCnt": 0,
      "workerCnt": 0,
      "nodeCnt": 0,
      "createdAt": 1716536420000,
      "runningAt": 1716536754352,
      "elapsedTime": 265434374
    },
    {
      "id": "d66c3ff5-29f3-4f2d-a611-4d343573cc17",
      "name": "sally-kbctest",
      "status": "Terminated(User)",
      "owner": "user@kakaoenterprise.com",
      "clusterType": "hadoop",
      "clusterVersion": "hde-2.0.1",
      "isScalable": false,
      "apiKeyStatus": "impossible",
      "masterCnt": 0,
      "workerCnt": 0,
      "nodeCnt": 0,
      "createdAt": 1717049297000,
      "runningAt": 1717049871150,
      "elapsedTime": 727811
    }
  ],
  "pageable": {
    "pageNumber": 2,
    "pageSize": 3,
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 6,
    "unpaged": false,
    "paged": true
  },
  "totalPages": 32,
  "totalElements": 96,
  "last": false,
  "size": 3,
  "number": 2,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "numberOfElements": 3,
  "first": false,
  "empty": false
}

Cluster list retrieval example 2

Conditions

Page size: 5, Page number: 0
Exclude terminated clusters
Filter by cluster names containing test
Sort by createdAt ascending
Filter by apiKeyStatus=impossible

Cluster list retrieval example 2
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=0&size=5&includeTerminated=false&name=test&sort=createdAt,asc&apiKeyStatus=impossible' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Example 2 response

Example 2 response
{
  "content": [
    {
      "id": "90878bcd-0f38-4338-810e-40d6fbfd8b45",
      "name": "andy-cluster-test-01",
      "status": "Running",
      "owner": "user@kakaoenterprise.com",
      "clusterType": "hadoop",
      "clusterVersion": "hde-2.0.1",
      "isScalable": true,
      "apiKeyStatus": "impossible",
      "masterCnt": 1,
      "workerCnt": 1,
      "nodeCnt": 2,
      "createdAt": 1720797464322,
      "runningAt": 1720797856006,
      "elapsedTime": 827416064
    },
    {
      "id": "eb9a91ad-d1b6-462c-8e20-0e8b2c0aa074",
      "name": "test",
      "status": "Running",
      "owner": "user@kakaoenterprise.com",
      "clusterType": "dataflow",
      "clusterVersion": "hde-2.0.1",
      "isScalable": true,
      "apiKeyStatus": "impossible",
      "masterCnt": 1,
      "workerCnt": 2,
      "nodeCnt": 3,
      "createdAt": 1721131126322,
      "runningAt": 1721131467376,
      "elapsedTime": 493804694
    }
  ],
  "pageable": {
    "pageNumber": 0,
    "pageSize": 5,
    "sort": {
      "empty": false,
      "unsorted": false,
      "sorted": true
    },
    "offset": 0,
    "paged": true,
    "unpaged": false
  },
  "totalElements": 2,
  "totalPages": 1,
  "last": true,
  "size": 5,
  "number": 0,
  "sort": {
    "empty": false,
    "unsorted": false,
    "sorted": true
  },
  "numberOfElements": 2,
  "first": true,
  "empty": false
}

Retrieve VM list of cluster

Use the Open API to retrieve the list of VMs in a cluster. The endpoint is as follows:

Retrieve VM list of cluster
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/vms' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Request

Request header

Key	Value
`{credential-id}`*	User’s access key ID - Available from KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User’s secret access key - Only viewable at the time of key creation - If lost, Get IAM access key

Query parameters

When retrieving a list of VMs in a cluster, the following optional query parameters can be used. Multiple conditions can be chained using &.

Key	Type	Description	Required	Default
size	integer	Number of VMs per page	optional	20
page	integer	Page number to retrieve	optional	0
ip	boolean	Filters by matching private IP (like query)	optional
name	string	Filters by VM name (like query)	optional
instanceId	string	Filters by instance ID (like query)	optional
sort	string	Field and order to sort by. Format: `field,asc	desc`. <br/>Example:` sort=name,desc`. Supports multiple sort fields in priority order. Sortable fields: name, instanceId, isMaster, flavorName, ip, floatingIp, createdAt	optional

Response

Response example

Response example
{
  "content": [
    {
      "name": "string",
      "instanceId": "string",
      "index": integer,
      "isMaster": boolean,
      "status": "string",
      "flavorName": "string",
      "ip": "string",
      "floatingIp": "string",
      "createdAt": long
    }
  ],
  "pageable": {
    "pageNumber": integer,
    "pageSize": integer,
    "sort": {
      "sorted": boolean,
      "unsorted": boolean,
      "empty": boolean
    },
    "offset": integer,
    "paged": boolean,
    "unpaged": boolean
  },
  "totalPages": integer,
  "totalElements": integer,
  "last": boolean,
  "numberOfElements": integer,
  "sort": {
    "sorted": boolean,
    "unsorted": boolean,
    "empty": boolean
  },
  "size": integer,
  "first": boolean,
  "number": integer,
  "empty": boolean
}

Field	Description
content	List of VM entries
name	VM name
instanceId	VM instance ID
index	Index of the VM e.g., `HadoopMST-test-1` indicates the first master VM, and index is `1`
isMaster	Whether the VM is a master node
status	VM status
flavorName	VM flavor name
ip	Private IP of the VM
floatingIp	Public IP of the VM
createdAt	Creation timestamp (Unix time)
pageable, sort, ...	Pagination and sorting metadata

Status codes

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized

Cluster VM list retrieval example 1

Conditions

Cluster ID: 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
Page size: 10, Page number: 0
Sorted by name in ascending order

VM list retrieval example 1
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/bbe8ce14-3497-476e-894c-142c48c9c469/vms?sort=name,asc&page=0&size=10' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Example 1 response

VM list retrieval example 1 response
{
  "content": [
    {
      "name": "HadoopMST-test-cluster-1",
      "instanceId": "23948ac6-86e0-42f6-acdb-d214cdd200c0",
      "index": 1,
      "isMaster": true,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.129.243",
      "createdAt": 1719452777733
    },
    {
      "name": "HadoopWRK-test-cluster-1",
      "instanceId": "8606bbb2-52bf-4154-82f5-647c9da4ac2d",
      "index": 1,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.132.127",
      "createdAt": 1719452777733
    },
    {
      "name": "HadoopWRK-test-cluster-2",
      "instanceId": "7db47633-5676-42ad-bf34-c6e5da5c30c7",
      "index": 2,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.131.196",
      "createdAt": 1719413846446
    },
    {
      "name": "HadoopWRK-test-cluster-3",
      "instanceId": "2fdda50e-cf40-4976-a394-7ead7e32f042",
      "index": 3,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.128.201",
      "createdAt": 1719413114832
    }
  ],
  "pageable": {
    "pageNumber": 0,
    "pageSize": 10,
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "unpaged": false,
    "paged": true
  },
  "totalPages": 1,
  "totalElements": 4,
  "last": true,
  "size": 10,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "numberOfElements": 4,
  "first": true,
  "empty": false
}

Cluster VM list retrieval example 2

Conditions

Retrieve VMs for cluster ID 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
Page size: 10, Page number: 0
Filter by name containing WRK
Sort by name descending and status ascending

VM list retrieval example 2
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=WRK&page=0&size=10&sort=name,desc&sort=status,asc' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Response example 2
{
  "content": [
    {
      "name": "HadoopWRK-hailey13-2",
      "instanceId": "10c5dfab-cae7-4b8c-a975-7b5d285482aa",
      "index": 2,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.2.159",
      "createdAt": 1721202811333
    },
    {
      "name": "HadoopWRK-hailey13-1",
      "instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
      "index": 1,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.3.96",
      "createdAt": 1721202811333
    }
  ],
  "pageable": {
    "pageNumber": 0,
    "pageSize": 10,
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "paged": true,
    "unpaged": false
  },
  "totalElements": 2,
  "totalPages": 1,
  "last": true,
  "size": 10,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "numberOfElements": 2,
  "first": true,
  "empty": false
}

Cluster VM list retrieval example 3

Conditions

Retrieve VMs for cluster ID 31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
Page size: 5, Page number: 0
Filter by name containing hailey13 and instanceId containing -4c3b
Sort by isMaster descending and ip ascending

VM list retrieval example 3
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=hailey13&instanceId=-4c3b&page=0&size=5&sort=isMaster,desc&sort=ip,asc' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Response example 3
{
  "content": [
    {
      "name": "HadoopWRK-hailey13-1",
      "instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
      "index": 1,
      "isMaster": false,
      "status": "Running",
      "flavorName": "m2a.xlarge",
      "ip": "10.0.3.96",
      "createdAt": 1721202811333
    }
  ],
  "pageable": {
    "pageNumber": 0,
    "pageSize": 5,
    "sort": {
      "empty": false,
      "sorted": true,
      "unsorted": false
    },
    "offset": 0,
    "paged": true,
    "unpaged": false
  },
  "totalElements": 1,
  "totalPages": 1,
  "last": true,
  "size": 5,
  "number": 0,
  "sort": {
    "empty": false,
    "sorted": true,
    "unsorted": false
  },
  "numberOfElements": 1,
  "first": true,
  "empty": false
}

Delete cluster

Deletes a cluster using the Open API. ⚠️ Deleted clusters cannot be recovered. The endpoint is as follows:

Delete cluster
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Request

Path parameter

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	Cluster ID - Can be found in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster menu

Request header

Key	Value
`{credential-id}`*	User's access key ID - Available from the KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User's secret access key - Only viewable at the time of creation - If lost, Get IAM access key

Response

Response example
{
  "id": "string",
  "name": "string",
  "status": "string"
}

Cluster deletion example

Condition

To delete a cluster with ID 90ac14f2-3837-11ef-b1da-72300678fa60, send the following request:

Cluster deletion example
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}'

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized
`404`	Cluster not found

Scale out cluster

Scales out (adds) worker nodes to a cluster using the Open API. The endpoint is:

Scale out cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-out' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{Body}'

Request

Path parameter

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	Cluster ID - Available in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster menu (left panel) > Cluster details

Request header

Key	Value
Content-Type*	Must be set to `application/json`
`{credential-id}`*	User's access key ID - Available from the KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User's secret access key - Only available at the time of key creation - If lost, you must Get IAM access key

Request body

Specify the number of worker nodes to add.

Request body
{
  "scaleCnt": integer
}

Response

Response example
{
  "id": "string",
  "name": "string",
  "status": "string"
}

Scale-out example

Condition

To add 3 worker nodes to the cluster with ID 90ac14f2-3837-11ef-b1da-72300678fa60:

Scale-out example
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-out' \
  --header 'Credential-ID: {credential-id}' \
  --header 'Credential-Secret: {credential-secret}' \
  --header 'Content-Type: application/json' \
  --data-raw '
{
  "scaleCnt": 3
}'

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized
`404`	Cluster not found

Scale in cluster

Scales in (removes) worker nodes from a cluster using the Open API. The endpoint is:

Scale in cluster
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-in' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '{Body}'

Request

Path parameter

Type	Parameter	Data type	Description
URL	`{cluster-id}`*	String	ID of the cluster - Can be found in the KakaoCloud Console > Analytics > Hadoop Eco > Cluster menu (left panel) > Cluster details

Request header

Key	Description
Content-Type*	Must be set to `application/json`
`{credential-id}`*	User's access key ID - Available in the KakaoCloud Console > Profile (top right) > Credentials > IAM access key
`{credential-secret}`*	User's secret access key - Only viewable at the time of creation - If lost, Get IAM access key

Request body

Specify the number of worker nodes to remove.

⚠️ The number of remaining worker nodes must not be less than the HDFS replication count set during cluster creation.

Request body
{
  "scaleCnt": integer
}

Response

Response example
{
  "id": "string",
  "name": "string",
  "status": "string"
}

Scale-in example

Condition

To reduce the number of nodes by 5 for cluster ID 90ac14f2-3837-11ef-b1da-72300678fa60:

Scale-in example
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-in' \
    --header 'Credential-ID: {credential-id}' \
    --header 'Credential-Secret: {credential-secret}' \
    --header 'Content-Type: application/json' \
    --data-raw '
{
  "scaleCnt": 5
}'

Code	Description
`200`	Success
`400`	Invalid request
`401`	Credential authentication failed
`403`	Unauthorized
`404`	Cluster not found

Before getting started​

Prepare API usage​

API endpoint​

Base64 encoding method​

Create cluster​

Request​

Request header​

Request body​

Enter basic information​

Configuration injection example using fileUrl or configText​

Inject configuration settings​

Example: Inject script into userScript​

Response​

Status codes​

Cluster creation example 1​

Cluster creation example 2​

Cluster creation example 3​

Retrieve cluster​

Request​

Path parameter​

Request header​

Response example​

Response​

Status codes​

Retrieve cluster list​

Request​

Request header​

Query parameters​

Response​

Response example​

Status codes​

Cluster list retrieval example 1​

Cluster list retrieval example 2​

Retrieve VM list of cluster​

Request​

Request header​

Query parameters​

Response​

Response example​

Status codes​

Cluster VM list retrieval example 1​

Cluster VM list retrieval example 2​

Cluster VM list retrieval example 3​

Delete cluster​

Request​

Path parameter​

Request header​

Response​

Cluster deletion example​

Scale out cluster​

Request​

Path parameter​

Request header​

Request body​

Response​

Scale-out example​

Scale in cluster​

Request​

Path parameter​

Request header​

Request body​

Response​

Scale-in example​

Before getting started

Prepare API usage

API endpoint

Base64 encoding method

Create cluster

Request

Request header

Request body

Enter basic information

Configuration injection example using `fileUrl` or `configText`

Inject configuration settings

Example: Inject script into `userScript`

Response

Status codes

Cluster creation example 1

Cluster creation example 2

Cluster creation example 3

Retrieve cluster

Request

Path parameter

Request header

Response example

Response

Status codes

Retrieve cluster list

Request

Request header

Query parameters

Response

Response example

Status codes

Cluster list retrieval example 1

Cluster list retrieval example 2

Retrieve VM list of cluster

Request

Request header

Query parameters

Response

Response example

Status codes

Cluster VM list retrieval example 1

Cluster VM list retrieval example 2

Cluster VM list retrieval example 3

Delete cluster

Request

Path parameter

Request header

Response

Cluster deletion example

Scale out cluster

Request

Path parameter

Request header

Request body

Response

Scale-out example

Scale in cluster

Request

Path parameter

Request header

Request body

Response

Scale-in example