Manage cluster

A cluster is a collection of nodes provisioned using Virtual Machines. The following describes how to manage a cluster in the Hadoop Eco service.

View cluster list

You can view the list of currently created clusters and their basic information.

1. Go to the KakaoCloud Console > Analytics > Hadoop Eco menu.

2. In the Cluster menu, view the list of clusters.

   Item	Description
   Cluster filter	Filter to search specific clusters or search by keyword. - Selected items work with an AND condition, and general keyword searches work with an OR condition.
   Name	The name of the cluster specified by the user.
   Status	The status of the cluster. - If the cluster status is Terminated (including clusters deleted due to errors), it will be displayed for 90 days. - Detailed explanations of each status value are available in Cluster lifecycle.
   Type	Core Hadoop, HBase, Trino, Dataflow
   Node count	The total number of instances used in the cluster configuration.
   Open api key status	Status of the Open API key
   Creation date	The date the cluster was created
   Uptime	The time the cluster has been operational
   More actions	- Clone cluster: Clone the cluster regardless of its status. - Open api key issuance: Issue an API Key (visible when Open API is applicable). - Adjust worker node count: Increase or decrease the number of worker nodes. - Delete cluster: Delete the cluster.

View cluster details

You can view detailed information about the cluster, node details, node list, action logs, and monitoring information.

1. From the Cluster menu, select the cluster for which you want to view details.

2. View the information on the cluster's detail page.

   Item	Description
   Cluster status	The status of the cluster
   Total instance uptime	The total uptime of the instances
   Creator	The requester who created the cluster
   Creation date	The date the cluster was created

Check cluster information

You can view cluster tasks, task scheduling, and service integration information.

Item	Description
Cluster information	All information related to the cluster
Cluster configuration (optional)	HDFS replication count HDFS block size Cluster configuration settings
Service integration (optional)	Service integration details - Monitoring agent installation status - Data Catalog integration - MySQL database name - MySQL database ID
Task scheduling (optional)	Task scheduling details - Task type - Task completion action - Executable file - Scheduling log file storage

Node information

You can check information about master and worker nodes.

Item	Description
Node type	The type of node (master or worker)
Instance name	Click the instance name to navigate to the detailed page of the corresponding Virtual Machine instance and perform actions.
Instance id	The ID of the node instance
Type	The type of the node instance
Private ip	The private IP of the node instance
Public ip	The public IP of the node instance
Status	The status of the node instance

Check monitoring

You can check detailed monitoring metrics for HDFS, YARN, Namenode, Resource Manager, Nodes, HBase, and other nodes of the Hadoop Eco cluster.

info

Monitoring for Trino-type clusters will be supported in the future.

1. From the Cluster menu, select the cluster for which you want to check monitoring metrics.

2. Click the Monitoring tab and view the monitoring metrics for that cluster.

   Item	Description
   Data period	The period for which monitoring metrics will be retrieved - Period options: 1 hour (default), 3 hours, 12 hours, 1 day, 7 days
   Query items	The monitoring metric items to query - Default items (if monitoring agent is not installed): HDFS / YARN / Namenode / ResourceManager / HBase / Kafka - If monitoring agent is installed: Default items + Nodes - By default, all items are queried, and multiple items can be selected.
   Auto-refresh interval	Set the auto-refresh interval - Interval options: No auto-refresh (default), 10 seconds, 30 seconds, 1 minute, 5 minutes
   Manual refresh	Click to manually refresh the monitoring results

   Monitoring data provided

   Item	Monitoring data	Description
   HDFS	HDFS usage (%)	The usage of HDFS
   	Running datanodes (count)	The number of running datanodes
   YARN	YARN memory usage (Bytes)	The total available and used memory
   	YARN cpu usage (%)	The total vCore usage and used vCore
   	Running nodemanager (count)	The number of running nodemanager
   	Running applications (count)	The number of running applications
   Namenode	Heap size (Bytes)	The total available and used heap memory
   ResourceManager	Heap size (Bytes)	The total available and used heap memory
   Nodes	Node cpu usage (%)	The CPU usage of node instances - Available when monitoring agent is installed
   	Node memory usage (%)	The memory usage of node instances - Available when monitoring agent is installed
   HBase	Hmaster heap size (Bytes)	The total available and used heap memory - Available only when the cluster type is HBase
   	Running region servers (count)	The number of running region servers - Available only when the cluster type is HBase
   Kafka	Active brokers (count)	The number of active Kafka brokers - Available only when the cluster type is Dataflow
   	Created topics (count)	The number of created topics - Available only when the cluster type is Dataflow

Check action logs

If you have issued an Open API Key and created a Hadoop Eco cluster, you can check the action logs.

1. From the Cluster menu, select the cluster for which you want to view action logs.

2. Click the Action logs tab and check the action log entries.

   Item	Description
   Action log filter	Select filter attributes or enter a keyword to search for action logs matching the conditions
   Request id	The request id for the cluster
   Status	The status of the cluster
   Task result	The result of the cluster task
   User	The email address of the user for the cluster
   Uptime/instance uptime	The time the cluster has been operational - Hover over the time to see when the cluster was requested
   Action log details	Click the icon to view detailed information for each action log - Worker node count: The number of worker nodes set - Worker volume size: The size of the worker volume in the action log - HDFS replication count: The replication count in HDFS - HDFS block size: The block size in HDFS - User task options: The options the user set during the task

Clone cluster

You can clone a cluster with the same configuration.

info

You can clone a cluster regardless of its status, but only clusters that have information retained for 90 days after creation can be cloned.
(Note: Clusters of unsupported versions cannot be cloned.)

1. From the Cluster menu, select the [More] icon for the cluster you want to clone > Clone cluster.

2. In the Clone cluster popup, check the cluster information and select whether to clone the task scheduling settings, then click [Confirm].

   • Task scheduling settings are only visible if the original cluster is Core Hadoop.

   Item	Description
   Cluster configuration	Cluster version/type/availability
   Cluster availability	Provides two availability types for operational stability: Standard and High availability
   Master node settings	Node instance type, disk volume, hostname
   Worker node settings	Node instance type, disk volume

Delete cluster

You can delete a cluster that is no longer in use.

caution

Once a deleted cluster's resources are terminated, they are fully reclaimed and cannot be restored. The HDFS data will also be deleted along with the cluster's resources and cannot be recovered.

1. From the Cluster menu, select the [More] icon for the cluster you want to delete > Delete cluster.
2. In the Delete cluster popup, confirm the cluster to delete, enter permanent deletion, and click [Delete].

Api key

Issue new api key

You can issue an API key for the Hadoop Eco cluster when the cluster type is Core Hadoop and the following conditions are met.

Conditions for issuing an api key for Core Hadoop type clusters

• Task scheduling is enabled for Hive or Spark.
• The cluster status is Terminated(User Command) or Terminated(User).

info

When Open API is enabled, two cluster status values are displayed:

• Pending: The state when Open API is enabled and Hadoop Eco creation requests are possible.
• Processing: The state when Open API is enabled and Hadoop Eco creation and job scheduling are in progress.

1. From the Cluster menu, select the cluster for which you want to issue an API key.
   You can search for clusters by filtering the Open API status or by searching for the cluster for which you want to issue the API key.

   Open api status	Description
   Not applicable	Open API authentication is not exposed
   Available	Open API authentication is exposed
   In progress	Open API authentication is unavailable

2. Click the [More] icon for the cluster > Open api authentication.

3. In the Open api authentication popup, click [Issue].

4. Copy the issued API key and save it separately.

info

The API key cannot be confirmed again, so please copy it from the Open api authentication popup and save it separately. If lost, click [Reissue] to issue it again.

• If a cluster with the same name exists, the Open API key will not be issued.
  
• If you no longer need to create clusters through Open API, click [Delete] to delete the API key.

Reissue api key

You can reissue the API key for Hadoop Eco service.
When reissuing the Open API key, the old API key will no longer work, and the newly issued Open API key must be used for API calls.

1. From the Cluster menu, select the cluster for which you want to reissue an API key > [More] icon > Open api authentication.
2. In the Open api authentication popup, click [Reissue].
3. In the Reissue Open api key popup, click [Reissue].

Delete api key

You can delete an API key. When deleting the API key, the Action logs tab of the cluster details page will also be deleted.

1. From the Cluster menu, select the cluster for which you want to delete the API key > [More] icon > Open api authentication.
2. In the Open api authentication popup, click [Delete].
3. In the Delete Open api key popup, click [Delete].

Adjust worker node count

You can change the number of worker nodes.

info

You can only change the node count when the cluster status is Running.

1. From the Cluster menu, select the cluster for which you want to change the worker node count > [More] icon > Adjust worker node count.
2. In the Adjust worker node count popup, check the current instance type and count, then enter a number greater or smaller than the current node count and click [Save].