Hadoop Eco Open API
Before getting started
Hadoop Eco Open API performs general cluster creation and management tasks, such as creating new clusters, querying clusters, deleting clusters, and scaling them. Open API key issuance is not required when using the Hadoop Eco Open API.
The usage method for Hadoop Eco Open API is as follows.
Prepare API usage
To call the Hadoop Eco Open API, you must get an access key.
API endpoint
The Hadoop Eco API endpoint URL for API requests is as follows:
https://hadoop-eco.kr-central-2.kakaocloud.com
Base64 encoding method
When using the Open API, certain values must be encoded in Base64 before being input. The encoding method is as follows:
1. Linux: echo "hello" | base64
2. OS X: echo "hello" | base64
3. Windows: echo hello > input.txt
certutil -encode "input.txt" "output.txt"
type output.txt
Create cluster
Create a cluster using the Open API. The endpoint is as follows:
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'
Request
Request Header
Key | Value |
---|---|
Content-Type* | Fixed as application/json |
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, get a new access key |
Request body
Enter basic information
Enter basic information such as cluster name, type, version, network details, and availability settings.
{
"name": "string",
"clusterType": "string",
"clusterVersion": "string",
"isHa": true | false,
"keypairName": "string",
"masterFlavorName": "string",
"masterVolumeSize": integer,
"workerCnt": integer,
"workerFlavorName": "string",
"workerVolumeSize": integer,
"monitoring": true,
"securityGroupIds": ["string"],
"securityGroupName": "string",
"vpcId": "string",
"subnetId": "string",
"adminInfo": {
"userId": "string",
"userPw": "string"
},
"config": {
"hdfsReplication": integer,
"hdfsBlockSize": integer,
"configText": "string",
"fileUrl": "string",
"userScript": "string"
}
}
Category | Type | Description |
---|---|---|
name* | String | Cluster name |
clusterType* | String | Cluster type |
clusterVersion* | String | Cluster version |
isHa* | Boolean | High availability - true : High availability enabled - false : High availability disabled |
keypairName* | String | Key pair name used by virtual machines composing the cluster |
masterFlavorName* | String | Master node flavor type name - ex) m2a.xlarge |
masterVolumeSize* | Integer | Master node volume size (GB) |
workerCnt* | Integer | Number of worker nodes, must be greater than or equal to config.hdfsReplication |
workerFlavorName* | String | Worker node flavor type name - ex) m2a.xlarge |
workerVolumeSize* | Integer | Worker node volume size (GB) |
monitoring* | Boolean | Whether to install monitoring agent |
securityGroupIds, securityGroupName | String | Security group ID and security group name; one of these fields is required - Enter the security group ID in securityGroupIds to reuse an existing security group - Enter a security group name in securityGroupName to automatically create one during cluster creation |
vpcId* | String | VPC ID where the cluster will be installed |
subnetId* | String | Subnet ID where the cluster will be installed |
adminInfo.userId | String | Account information (ID) for Hue and Superset - If not entered, the IAM account ID is used |
adminInfo.userPw | String | Account information (PW) for Hue and Superset - If not entered, the IAM account password is used - Enter a Base64-encoded string for custom settings |
config.hdfsReplication* | Integer | Replication count - Must be less than or equal to workerCnt |
config.BlockSize* | Integer | Block size (MB) |
config.configText | String | Cluster configuration settings to inject - Write the settings as a JSON string, then encode them as Base64 - Refer to fileUrl and configText injection example |
config.fileUrl | String | URL of the file in Object Storage - Base64 encoding is not required Example: https://objectstorage.{region}.kakaocloud.com/v1/<project ID>/<bucket>/<path>/<file> - Refer to fileUrl and configText injection example |
config.userScript | String | User script to run when virtual machines are created - Enter the script encoded in Base64 - Refer to userScript injection example |
fileUrl
and configText
injection example
You can input cluster configuration settings using fileUrl
or configText
as a JSON string or from a file in Object Storage.
For example, if you want to inject the following configuration settings, first write them in JSON format and refer to the Configuration settings injection section.
- Configuration example
- Write configuration JSON
<configuration>
... (omitted) ...
<property>
<name>io.file.buffer.size</name>
<value>65536</value>
</property>
... (omitted) ...
</configuration>
<configuration>
... (omitted) ...
<property>
<name>mapred.max.split.size</name>
<value>128000000</value>
</property>
... (omitted) ...
</configuration>
... (omitted) ...
export HADOOP_HEAPSIZE="3001"
export HADOOP_NAMENODE_INIT_HEAPSIZE="-Xmx3002m"
... (omitted) ...
... (omitted) ...
spark.driver.memory 4000M
spark.network.timeout 800s
... (omitted) ...
Write the desired configuration settings in JSON format as shown below.
{
"configurations": [
{
"classification": "core-site",
"properties": {
"io.file.buffer.size": "65536"
}
},
{
"classification": "hive-site",
"properties": {
"mapred.max.split.size": "128000000"
}
},
{
"classification": "hadoop-env",
"properties": {
"hadoop_env_hadoop_heapsize": 3001,
"hadoop_env_hadoop_namenode_heapsize": "-Xmx3002m"
}
},
{
"classification": "spark-defaults",
"properties": {
"spark.driver.memory": "4000M",
"spark.network.timeout": "800s"
}
}
]
}
Inject configuration settings
Once the JSON is prepared, there are two ways to inject the configuration settings during cluster creation.
First method
The first method involves encoding the JSON in Base64 and injecting it into the configText
field.
-
Run the following command to encode the JSON configuration file in Base64:
echo | cat config.json | base64
-
Enter the Base64 string into the
configText
field.ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAiaGl2ZS1zaXRlIiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgIm1hcHJlZC5tYXguc3BsaXQuc2l6ZSI6ICIxMjgwMDAwMDAiCiAgICAgIH0KICAgIH0sCgl7CiAgICAgICJjbGFzc2lmaWNhdGlvbiI6ICJoYWRvb3AtZW52IiwKICAgICAgInByb3BlcnRpZXMiOiB7CiAgICAgICAgImhhZG9vcF9lbnZfaGFkb29wX2hlYXBzaXplIjogMzAwMSwKICAgICAgICAiaGFkb29wX2Vudl9oYWRvb3BfbmFtZW5vZGVfaGVhcHNpemUiOiAiLVhteDMwMDJtIgogICAgICB9CiAgICB9LAogICAgewogICAgICAiY2xhc3NpZmljYXRpb24iOiAic3BhcmstZGVmYXVsdHMiLAogICAgICAicHJvcGVydGllcyI6IHsKICAgICAgICAic3BhcmsuZHJpdmVyLm1lbW9yeSI6ICI0MDAwTSIsCiAgICAgICAgInNwYXJrLm5ldHdvcmsudGltZW91dCI6ICI4MDBzIgogICAgICB9CiAgICB9CiAgXQp9Cg==
Second method
The second method involves uploading the file to Object Storage and injecting the file's object URL into fileUrl
.
- Upload the JSON file to Object Storage.
- Click the [More] button on the uploaded file and select 'Copy object URL'.
- If copied correctly, you will see a URL in the following format:
https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{file}
- Inject the copied URL into the
fileUrl
field.
Example of injecting a userScript
-
Prepare a user script file.
User script example#!/bin/bash
function print(){
msg=$1
echo "${msg}"
}
print "hello world!" -
Run the following command to encode the script content in Base64:
echo | cat script.sh | base64
-
Enter the Base64 string into the
userScript
field.IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK
(Optional) If you want to run task scheduling after cluster installation, enter userTask
as shown below.
"userTask": {
"type": "hive|spark",
"terminationPolicy": "never|onSuccess|always",
"fileUrl": "string",
"hiveQuery": "string",
"deployMode": "cluster|client",
"execOpts": "string",
"execParams": "string",
"logBucketName": "string"
}
Category | Description |
---|---|
type* | Enter the task type - Supported types: hive , spark |
terminationPolicy* | Specify the cluster behavior after task scheduling ends - never : Do not terminate the cluster - onSuccess : Terminate the cluster only if the task succeeds - always : Always terminate the cluster |
hiveQuery | (For Hive tasks) Enter the Hive query string to execute |
fileUrl | (For Hive tasks) Enter the Object Storage URL where the Hive query file is located (For Spark tasks) Enter the Object Storage URL where the JAR file is located |
deployMode | (For Spark tasks) Specify the Spark task deployment mode: cluster or client |
execOpts | (For Hive tasks) Enter Hive options as a string for task execution (For Spark tasks) Enter Spark options as a string for task execution |
execParams | (For Spark tasks) Enter parameters as a string to pass during task execution |
logBucketName | Enter the Object Storage bucket name to store task scheduling logs If logBucketName is set, logs will be stored under <logBucketName>/HadoopEco/<cluster name> |
(Optional) If you want to integrate MySQL with the Hive metastore, include the hiveDbInfo
object as shown below.
"hiveDbInfo": {
"objectId": "string",
"dbName": "string",
"userId": "string",
"userPw": "string"
}
Category | Description |
---|---|
objectId* | Enter the object ID of the MySQL to integrate |
dbName*, userId*, userPw* | Enter the MySQL database name, user ID, and password for access - The password must be a Base64-encoded string |
(Optional) If you want to integrate a data catalog, include the dataCatalogInfo
object as shown below.
"dataCatalogInfo": {
"catalogId": "string"
}
Category | Description |
---|---|
catalogId* | Enter the object ID of the data catalog to integrate |
(Optional) If you want to integrate Redis, include the object ID as shown below.
"redisInfo": {
"objectId": "string",
"dbIdSupersetCelery": integer,
"dbIdSupersetResults": integer
}
Category | Description |
---|---|
objectId* | Enter the object ID of the Redis to integrate |
dbIdSupersetCelery | An integer between 0 and 15 (if not set, defaults to 0 , 1 ) |
dbIdSupersetResults | An integer between 0 and 15 (if not set, defaults to 0 , 1 ) |
Response
{
"id": "48fd271f-01f8-47bd-8e42-8c872fb6bf3a",
"name": "hive-job-cluster",
"status": "Initializing"
}
Category | Description |
---|---|
id | ID of the created cluster |
name | Name of the created cluster |
status | Status of the cluster |
Status codes
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
409 | Duplicate cluster name |
Cluster creation example 1
Cluster creation conditions
- Core Hadoop type with version
hde-2.0.1
- No high availability applied, with 3 worker nodes
- Security group automatically created
- Cluster configuration override: Set the
io.file.buffer.size
field in HDFScore-site.xml
to65536
-
Write the JSON for applying the configuration.
JSON for applying configuration{
"configurations": [
{
"classification": "core-site",
"properties": {
"io.file.buffer.size": "65536"
}
}
]
} -
Run the following command to encode the JSON configuration file in Base64:
echo | cat config.json | base64
-
Enter the Base64 string into the
configText
field.ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg==
- Apply Hive task scheduling and keep the cluster running after the task is complete.
- Integrate with a data catalog:
-
The data catalog ID is required.
-
You can retrieve the ID from KakaoCloud Console > Analytics > Data Catalog.
Cluster creation example 1curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "hive-job-cluster",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isHa": false,
"workerCnt": 3,
"vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
"subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
"securityGroupName": "test-security-group1",
"keypairName": "keypairname",
"monitoring": true,
"masterFlavorName": "m2a.xlarge",
"workerFlavorName": "m2a.xlarge",
"masterVolumeSize": 50,
"workerVolumeSize": 100,
"adminInfo": {
"userId": "admin_user",
"userPw": "<base_64_string>"
},
"config": {
"hdfsReplication": 3,
"hdfsBlockSize": 128,
"configText": "ewogICJjb25maWd1cmF0aW9ucyI6IFsKICAgIHsKICAgICAgImNsYXNzaWZpY2F0aW9uIjogImNvcmUtc2l0ZSIsCiAgICAgICJwcm9wZXJ0aWVzIjogewogICAgICAgICJpby5maWxlLmJ1ZmZlci5zaXplIjogIjY1NTM2IgogICAgICB9CiAgICB9CiAgXQp9Cg=="
},
"userTask": {
"type": "hive",
"terminationPolicy": "never",
"hiveQuery": "create table if not exists t1 (col1 string); insert into table t1 values ('a'), ('b'), ('c');",
"execOpts": "--hiveconf hive.tez.container.size=1024 --hiveconf hive.tez.java.opts=-Xmx1600m",
"logBucketName": "user-bucket"
},
"dataCatalog": {
"catalogId": "e1ebae48-daba-a4c5-56da-fbb2b684ae07"
}
}'
-
Cluster creation example 2
Cluster creation conditions
- Core Hadoop type with version
hde-2.0.1
- High availability enabled, with 5 worker nodes
- Apply user script via a file:
- Upload the user script file to Object Storage.
- Click the [More] icon on the uploaded file and select 'Copy object URL'.
- If copied correctly, you will see a URL in the following format:
https://objectstorage.{region}.kakaocloud.com/v1/{project ID}/{bucket}/{path}/{file}
- Inject the copied URL into the
userScriptfileUrl
field.
- Reuse an existing security group
- Apply Spark task scheduling and automatically terminate the cluster upon successful task completion.
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "spark-job-cluster",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isHa": true,
"workerCnt": 5,
"vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
"subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
"securityGroupIds": ["56aa3028-b163-b114-baac-gg1dtfa1bec1", "76163abf-j11a-fbbd-a123-d1a4bedb123c"],
"keypairName": "keypairname",
"monitoring": true,
"masterFlavorName": "m2a.xlarge",
"workerFlavorName": "m2a.xlarge",
"masterVolumeSize": 50,
"workerVolumeSize": 100,
"config": {
"hdfsReplication": 3,
"hdfsBlockSize": 128,
"userScriptFileUrl": "https://objectstorage.{리전명}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/user_script.sh"
},
"userTask": {
"type": "spark",
"terminationPolicy": "onSuccess",
"fileUrl": "https://objectstorage.{리전명}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/sparkjob/spark_example.jar",
"deployMode": "cluster",
"execOpts": "--class org.apache.spark.examples.SparkPi --master yarn",
"logBucketName": "user-bucket"
}
}'
Cluster creation example 3
Cluster creation conditions
-
Core Hadoop type with version
hde-2.0.1
-
No high availability applied, with 3 worker nodes
-
Use IAM account as the administrator account
-
Security group automatically created
-
Use a user script
User script file script.sh#!/bin/bash
function print(){
msg=$1
echo "${msg}"
}
print "hello world!"- Run the following command to encode the script content in Base64:
echo | cat script.sh | base64
-
Enter the Base64 string into the
userScript
field.IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK
-
Apply environment variables using an Object Storage file
- When applying the following environment variables for Ubuntu users:
Apply environment variables for Ubuntu usersexport JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}
export CUSTOM_HOME=/etc/custom- Write the JSON for applying environment variables:
Write JSON for applying environment variables{
"configurations": [
{
"classification": "user-env:ubuntu",
"properties": {
"env": "export JAVA_HOME=${JAVA_HOME:-/usr/lib/jdk}\nexport CUSTOM_HOME=/etc/custom"
}
}
]
}- Upload the created JSON file (
config.json
) to Object Storage, copy the object URL, and enter it into thefileUrl
field.
-
MySQL integration
- You must pre-create a database to be used as the metadata database in MySQL as follows:
CREATE DATABASE meta_db;
-
Prepare the object ID of the MySQL instance:
- Go to the MySQL menu in the console → Navigate to the detailed screen of the target MySQL instance.
- Extract the UUID-style object ID from Developer Tools or the address bar.
- Viewing the MySQL object ID directly from the details screen will be improved in future updates.
-
Prepare accessible MySQL account credentials (ID, password).
- The password must be Base64-encoded.
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{
"name": "test-cluster",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isHa": false,
"workerCnt": 3,
"vpcId": "3ebb3020-d416-f1a4-534c-d3ad0fa1bec0",
"subnetId": "5c1632bf-a7da-fdbf-a594-e6a7bedb328c",
"securityGroupName": "security-group-test",
"keypairName": "keypairname",
"monitoring": true,
"masterFlavorName": "m2a.xlarge",
"workerFlavorName": "m2a.xlarge",
"masterVolumeSize": 50,
"workerVolumeSize": 100,
"config": {
"hdfsReplication": 3,
"hdfsBlockSize": 128,
"configFileUrl": "https://objectstorage.{regionName}.kakaocloud.com/v1/111c5b646a194bf09ac123566a39d123/user-bucket/config.json",
"userScript": "IyEvYmluL2Jhc2gKCmZ1bmN0aW9uIHByaW50KCl7CiAgICBtc2c9JDEKICAgIGVjaG8gIiR7bXNnfSIKfQoKcHJpbnQgImhlbGxvIHdvcmxkISIK"
},
"hiveDbInfo": {
"objectId": "cfdbd7cd-f0ce-bbfb-de5b-e73bdbbcb0cf",
"dbName": "meta_db",
"userId": "mysql_user",
"userPw": "<base_64_string>"
}
}'
Retrieve cluster
Retrieve cluster information using the Open API. The endpoint is as follows:
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Request
Path Parameter
Type | Parameter | Data type | Description |
---|---|---|---|
URL | {cluster-id} * | String | Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information |
Request Header
Key | Value |
---|---|
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Response Example
{
"id": "string",
"name": "string",
"owner": "string",
"clusterType": "string",
"clusterVersion": "string",
"isHa": true,
"isScalable": true,
"status": "string",
"apiKeyStatus": "impossible|possible|applied",
"masterCnt": integer,
"workerCnt": integer,
"nodeCnt": integer,
"vpcId": "string",
"subnetId": "string",
"securityGroupIds": ["string"],
"keypairName": "string",
"monitoring": true|false,
"imageName": "string",
"nameNodeUrl": "string",
"resourceManagerUrl": "string",
"hueUrl": "string",
"hmasterUrl": "string",
"trinoUrl": "string",
"druidUrl": "string",
"supersetUrl": "string",
"createdAt": integer,
"runningAt": integer,
"elapsedTime": integer,
"totalVmUptime": integer,
"masterInfo": [
{
"flavorId": "string",
"flavorName": "string",
"volumeSize": integer
}
],
"workerInfo": [
{
"flavorId": "string",
"flavorName": "string",
"volumeSize": integer
}
],
"adminInfo": {
"userId": "string"
},
"config": {
"hdfsBlockSize": integer,
"hdfsReplication": integer,
"configText": "string",
"configFileUrl": "string"
},
"userTask": {
"type": "string",
"terminationPolicy": "string",
"fileUrl": "string",
"hiveQuery": "string",
"deployMode": "string",
"execOpts": "string",
"execParams": "string",
"logBucketName": "string",
"logUrl": "string"
},
"hiveDbInfo": {
"objectId": "string",
"subnetId": "string",
"name": "string",
"host": "string",
"port": integer,
"dbName": "string",
"userId": "string"
},
"dataCatalogInfo": {
"metastoreUri": "string",
"catalogId": "string",
"catalogName": "string",
"vpcId": "string",
"vpcName": "string",
"subnetId": "string",
"subnetName": "string"
},
"redisInfo": {
"objectId": "string",
"name": "string",
"subnetId": "string",
"primaryEndpoint": "string",
"readEndpoint": "string",
"clusterEnabled": true,
"port": integer,
"dbIdSupersetCelery": integer,
"dbIdSupersetResults": integer
}
}
Response
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
404 | Cluster not found |
Retrieve cluster list
Retrieve a list of clusters using the Open API. The endpoint is as follows:
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Request
Request Header
Key | Value |
---|---|
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Query parameter
You can filter the cluster list using the following conditions. When using multiple conditions, connect them with &
.
Key | Data type | Description | Required | Default |
---|---|---|---|---|
size | integer | Page size, indicating the number of clusters displayed per page. | Optional | 20 |
page | integer | Page number to retrieve. The available page numbers depend on the page size. | Optional | 0 |
includeTerminated | boolean | Whether to include terminated clusters in the response list. | Optional | true |
sort | string | Field and sort order for sorting results. Specify the field and order ( asc or desc ) separated by a comma. For example, to sort the name field in descending order, use sort=name,desc . For multiple sort conditions, priority is applied in the specified order. Sortable fields: id, name, status, owner, clusterType, clusterVersion, apiKeyStatus, masterCnt, workerCnt, createdAt, runningAt, elapsedTime | Optional | createdAt,desc |
name | string | Cluster name for like search. If not specified, all clusters are retrieved. | Optional | |
apiKeyStatus | string | Open API Key status for like search. Search values: possible , impossible , applied . They represent applicable, not applicable, and in-progress, respectively. If not specified, all values are retrieved. | Optional |
Response
Response Example
{
"content": [
{
"id": "string",
"name": "string",
"status": "string",
"owner": "string",
"clusterType": "string",
"clusterVersion": "string",
"isScalable": boolean,
"apiKeyStatus": "string",
"masterCnt": integer,
"workerCnt": integer,
"nodeCnt": integer,
"createdAt": long,
"runningAt": long,
"elapsedTime": long
}
],
"pageable": {
"pageNumber": integer,
"pageSize": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"offset": integer,
"paged": boolean,
"unpaged": boolean
},
"totalPages": integer,
"totalElements": integer,
"last": boolean,
"numberOfElements": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"size": integer,
"first": boolean,
"number": integer,
"empty": boolean
}
Category | Description |
---|---|
content | List containing the data |
id | Cluster ID |
name | Cluster name |
status | Cluster status |
owner | Cluster owner |
clusterType | Cluster type |
clusterVersion | Cluster version |
isScalable | Whether the cluster is scalable |
apiKeyStatus | Open API Key status of the cluster |
masterCnt | Number of master nodes |
workerCnt | Number of worker nodes |
nodeCnt | Total number of nodes |
createdAt | Unix timestamp when the cluster was created |
runningAt | Unix timestamp when the cluster entered the Running state |
elapsedTime | Uptime (ms) |
pageable | Paging-related information |
pageNumber | Current page number |
pageSize | Current page size |
sort | Sorting information |
sorted | Whether sorting is applied |
unsorted | Whether sorting is not applied |
empty | Whether the sorting conditions are empty |
offset | Starting point of the page |
paged | Whether paging is applied |
unpaged | Whether paging is not applied |
totalPages | Total number of pages |
totalElements | Total number of data entries |
last | Whether the current page is the last page |
size | Current page size |
first | Whether the current page is the first page |
number | Current page number |
empty | Whether the current page is empty |
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
Cluster list retrieval example 1
List retrieval conditions
- Page size is set to 3, and page number is set to 2
- Include terminated clusters
- Apply sorting by
name
in descending order andstatus
in ascending order
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=0&size=3&includeTerminated=true&sort=name,desc&sort=status,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Response example 1
{
"content": [
{
"id": "fa70ff9f-98a2-446c-9f33-75d7dce7c160",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1716359078000,
"runningAt": 1716359422659,
"elapsedTime": 12108694
},
{
"id": "3ec42a8d-a5a3-42ef-9c34-e270db0354b1",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1716536420000,
"runningAt": 1716536754352,
"elapsedTime": 265434374
},
{
"id": "d66c3ff5-29f3-4f2d-a611-4d343573cc17",
"name": "sally-kbctest",
"status": "Terminated(User)",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": false,
"apiKeyStatus": "impossible",
"masterCnt": 0,
"workerCnt": 0,
"nodeCnt": 0,
"createdAt": 1717049297000,
"runningAt": 1717049871150,
"elapsedTime": 727811
}
],
"pageable": {
"pageNumber": 2,
"pageSize": 3,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 6,
"unpaged": false,
"paged": true
},
"totalPages": 32,
"totalElements": 96,
"last": false,
"size": 3,
"number": 2,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 3,
"first": false,
"empty": false
}
Cluster list retrieval example 2
List retrieval conditions
- Page size is set to 5, and page number is set to 0
- Exclude terminated clusters
- Filter clusters with
test
in their name - Apply sorting by
createdAt
in ascending order - Filter clusters with
apiKeyStatus
set toimpossible
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters?page=0&size=5&includeTerminated=false&name=test&sort=createdAt,asc&apiKeyStatus=impossibl' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Response example 2
{
"content": [
{
"id": "90878bcd-0f38-4338-810e-40d6fbfd8b45",
"name": "andy-cluster-test-01",
"status": "Running",
"owner": "user@kakaoenterprise.com",
"clusterType": "hadoop",
"clusterVersion": "hde-2.0.1",
"isScalable": true,
"apiKeyStatus": "impossible",
"masterCnt": 1,
"workerCnt": 1,
"nodeCnt": 2,
"createdAt": 1720797464322,
"runningAt": 1720797856006,
"elapsedTime": 827416064
},
{
"id": "eb9a91ad-d1b6-462c-8e20-0e8b2c0aa074",
"name": "test",
"status": "Running",
"owner": "user@kakaoenterprise.com",
"clusterType": "dataflow",
"clusterVersion": "hde-2.0.1",
"isScalable": true,
"apiKeyStatus": "impossible",
"masterCnt": 1,
"workerCnt": 2,
"nodeCnt": 3,
"createdAt": 1721131126322,
"runningAt": 1721131467376,
"elapsedTime": 493804694
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 5,
"sort": {
"empty": false,
"unsorted": false,
"sorted": true
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 2,
"totalPages": 1,
"last": true,
"size": 5,
"number": 0,
"sort": {
"empty": false,
"unsorted": false,
"sorted": true
},
"numberOfElements": 2,
"first": true,
"empty": false
}
Retrieve VM list of cluster
Retrieve the list of VMs in a cluster using the Open API. The endpoint is as follows:
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/vms' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Request
Request header
Key | Value |
---|---|
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Query parameter
You can filter the VM list using the following conditions. When using multiple conditions, connect them with &
.
Key | Data type | Description | Required | Default |
---|---|---|---|---|
size | integer | Page size, indicating the number of VMs displayed per page. | Optional | 20 |
page | integer | Page number to retrieve. The available page numbers depend on the page size. | Optional | 0 |
ip | boolean | Private IP for like search. If not specified, all VMs are retrieved. | Optional | |
name | string | Name for like search. If not specified, all VMs are retrieved. | Optional | |
instanceId | string | Instance ID for like search. If not specified, all VMs are retrieved. | Optional | |
sort | string | Field and sort order for sorting results. Specify the field and order (asc or desc ) separated by a comma. For example, to sort the name field in descending order, use sort=name,desc . For multiple sort conditions, priority is applied in the specified order. Sortable fields: name, instanceId, isMaster, flavorName, ip, floatingIp, createdAt | Optional | createdAt,desc |
Response
Response Example
{
"content": [
{
"name": "string",
"instanceId": "string",
"index": integer,
"isMaster": boolean,
"status": "string",
"flavorName": "string",
"ip": "string",
"floatingIp": "string",
"createdAt": long,
}
],
"pageable": {
"pageNumber": integer,
"pageSize": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"offset": integer,
"paged": boolean,
"unpaged": boolean
},
"totalPages": integer,
"totalElements": integer,
"last": boolean,
"numberOfElements": integer,
"sort": {
"sorted": boolean,
"unsorted": boolean,
"empty": boolean
},
"size": integer,
"first": boolean,
"number": integer,
"empty": boolean
}
Category | Description |
---|---|
content | List containing the data |
name | Cluster name |
instanceId | VM instance ID |
index | VM index For example, if HadoopMST-test-1 , it indicates master VM 1, with an index value of 1 |
isMaster | Whether the VM is a master node |
status | VM status |
flavorName | VM flavor name |
ip | VM private IP |
floatingIp | VM public IP |
createdAt | Timestamp when the VM was created |
pageable | Paging-related information |
pageNumber | Current page number |
pageSize | Current page size |
sort | Sorting information |
sorted | Whether sorting is applied |
unsorted | Whether sorting is not applied |
empty | Whether the sorting conditions are empty |
offset | Starting point of the page |
paged | Whether paging is applied |
unpaged | Whether paging is not applied |
totalPages | Total number of pages |
totalElements | Total number of data entries |
last | Whether the current page is the last page |
size | Current page size |
first | Whether the current page is the first page |
number | Current page number |
empty | Whether the current page is empty |
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
Cluster VM list retrieval example 1
VM list retrieval conditions
- Retrieve VM list for cluster ID
31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
- Page size is set to 10, and page number is set to 0
- Apply sorting by
name
in ascending order
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/bbe8ce14-3497-476e-894c-142c48c9c469/vms?sort=name,asc&page=0&size=10' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Response example 1
{
"content": [
{
"name": "HadoopMST-test-cluster-1",
"instanceId": "23948ac6-86e0-42f6-acdb-d214cdd200c0",
"index": 1,
"isMaster": true,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.129.243",
"createdAt": 1719452777733
},
{
"name": "HadoopWRK-test-cluster-1",
"instanceId": "8606bbb2-52bf-4154-82f5-647c9da4ac2d",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.132.127",
"createdAt": 1719452777733
},
{
"name": "HadoopWRK-test-cluster-2",
"instanceId": "7db47633-5676-42ad-bf34-c6e5da5c30c7",
"index": 2,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.131.196",
"createdAt": 1719413846446
},
{
"name": "HadoopWRK-test-cluster-3",
"instanceId": "2fdda50e-cf40-4976-a394-7ead7e32f042",
"index": 3,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.128.201",
"createdAt": 1719413114832
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 10,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"unpaged": false,
"paged": true
},
"totalPages": 1,
"totalElements": 4,
"last": true,
"size": 10,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 4,
"first": true,
"empty": false
}
Cluster VM list retrieval example 2
VM list retrieval conditions
- Retrieve VM list for cluster ID
31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
- Page size is set to 10, and page number is set to 0
- Apply a filter to include
WRK
inname
- Apply sorting by
name
in descending order andstatus
in ascending order
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=WRK&page=0&size=10&sort=name,desc&sort=status,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Response example 2
{
"content": [
{
"name": "HadoopWRK-hailey13-2",
"instanceId": "10c5dfab-cae7-4b8c-a975-7b5d285482aa",
"index": 2,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.2.159",
"createdAt": 1721202811333
},
{
"name": "HadoopWRK-hailey13-1",
"instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.3.96",
"createdAt": 1721202811333
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 10,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 2,
"totalPages": 1,
"last": true,
"size": 10,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 2,
"first": true,
"empty": false
}
Cluster VM list retrieval example 3
VM list retrieval conditions
- Retrieve VM list for cluster ID
31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e
- Page size is set to 5, and page number is set to 0
- Apply a filter to include
hailey13
inname
and-4c3b
ininstanceId
- Apply sorting by
isMaster
in descending order andip
in ascending order
curl -X GET 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/31ca7ea4-93f3-4c76-9f9e-c0b3d947c17e/vms?name=hailey13&instanceId=-4c3b&page=0&size=5&sort=isMaster,desc&sort=ip,asc' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Response example 3
{
"content": [
{
"name": "HadoopWRK-hailey13-1",
"instanceId": "e204678e-bf9d-4c3b-81ce-812ca09a78c3",
"index": 1,
"isMaster": false,
"status": "Running",
"flavorName": "m2a.xlarge",
"ip": "10.0.3.96",
"createdAt": 1721202811333
}
],
"pageable": {
"pageNumber": 0,
"pageSize": 5,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"offset": 0,
"paged": true,
"unpaged": false
},
"totalElements": 1,
"totalPages": 1,
"last": true,
"size": 5,
"number": 0,
"sort": {
"empty": false,
"sorted": true,
"unsorted": false
},
"numberOfElements": 1,
"first": true,
"empty": false
}
Delete cluster
Delete a cluster using the Open API. Deleted clusters cannot be recovered. The endpoint is as follows:
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Request
Path Parameter
Type | Parameter | Data type | Description |
---|---|---|---|
URL | {cluster-id} * | String | Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information |
Request header
Key | Value |
---|---|
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Response
Response Example
{
"id": "string",
"name": "string",
"status": "string"
}
Cluster deletion example
Deletion conditions
To delete a cluster with the ID 90ac14f2-3837-11ef-b1da-72300678fa60
, make the following request:
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}'
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
404 | Cluster not found |
Scale out cluster
Scale out worker nodes in a cluster using the Open API. The endpoint is as follows:
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-out' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'
Request
Path Parameter
Type | Parameter | Data type | Description |
---|---|---|---|
URL | {cluster-id} * | String | Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information |
Request header
Key | Value |
---|---|
Content-Type* | Fixed as application/json |
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Request body
Enter the number of nodes to scale out.
증설할 노드 개수를 입력합니다.
{
"scaleCnt": integer
}
Response
Response Example
{
"id": "string",
"name": "string",
"status": "string"
}
Cluster scale-out example
Scale-out conditions
To scale out the cluster with ID 90ac14f2-3837-11ef-b1da-72300678fa60
by adding 3 nodes, make the following request:
curl -X POST ‘https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-out’ \
--header ‘Credential-ID: {credential-id}’ \
--header ‘Credential-Secret: {credential-secret}’ \
--header ‘Content-Type: application/json’ \
--data-raw '
{
“scaleCnt”: 3
}'
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
404 | Cluster not found |
Scale in cluster
Scale in worker nodes in a cluster using the Open API. The endpoint is as follows:
curl -X POST 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{cluster-id}/scale-in' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '{Body}'
Request
Path Parameter
종류 | 파라미터 | 유형 | 설명 |
---|---|---|---|
URL | {cluster-id} * | String | 클러스터의 ID - 카카오클라우드 콘솔 > Analytics > Hadoop Eco > 좌측 Cluster 메뉴 > 클러스터 정보에서 확인 가능 |
Request Header
Type | Parameter | Data type | Description |
---|---|---|---|
URL | {cluster-id} * | String | Cluster ID - Available from KakaoCloud console > Analytics > Hadoop Eco > Left menu > Cluster information |
Request header
Key | Description |
---|---|
Content-Type* | Fixed as application/json |
{credential-id} * | User's access key ID - Available from KakaoCloud console > Top-right profile > Access key |
{credential-secret} * | User's secret access key - Only available when issuing the access key - If the secret access key is lost, issue a new access key |
Request body
Enter the number of nodes to scale in.
- The remaining number of nodes after scaling in must be greater than or equal to the cluster's
hdfs.replication
.
{
"scaleCnt": integer
}
Response
Response Example
{
"id": "string",
"name": "string",
"status": "string"
}
Cluster scale-in example
Scale-in conditions
- To scale in the cluster with ID
90ac14f2-3837-11ef-b1da-72300678fa60
by removing 5 nodes, make the following request. - The remaining number of worker nodes after scaling in must not be less than the HDFS replication count set during cluster creation.
curl -X DELETE 'https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/90ac14f2-3837-11ef-b1da-72300678fa60/scale-in' \
--header 'Credential-ID: {credential-id}' \
--header 'Credential-Secret: {credential-secret}' \
--header 'Content-Type: application/json' \
--data-raw '
{
"scaleCnt": 5
}'
Status Code
Code | Description |
---|---|
200 | Success |
400 | Bad request |
401 | Credential authentication failed |
403 | Forbidden |
404 | Cluster not found |