Automating Hadoop cluster auto-scaling

This document describes the process of automating the auto-scaling of a Hadoop cluster using KakaoCloud Monitoring Flow and Hadoop Eco.

Prerequisites

Prepare Monitoring Flow environment

Before using Monitoring Flow, you need to create a VPC and subnet.

1. Go to KakaoCloud Console > Beyond Networking Service > VPC > VPC.
2. Refer to the Create VPC documentation to create a VPC.
3. Refer to the Create subnet documentation to create a subnet.

Monitoring Flow scenario example

You can use Monitoring Flow to adjust the size of a Hadoop cluster.

Scenario overview

This scenario uses API steps to cover the following:

• Create a Hadoop cluster using KakaoCloud Hadoop Eco.
• Create a Monitoring Flow scenario to schedule Hadoop cluster node scale in/out operations.
• Use Alert Center integration to check the results of the scenario.

Basic info

• Estimated time: 20 minutes
• Region: kr-central-2
• Prerequisites:
• Access Key
• Reference document:
• Create cluster

1. Create Hadoop Eco cluster

This exercise uses Hadoop Eco. You need to have an active cluster, or you can create a cluster if none exists.

1. Go to KakaoCloud Console > Analytics > Hadoop Eco > Cluster.
2. Confirm the Cluster ID of an active cluster.

• This information will be required when setting up the Monitoring Flow steps.

Cluster details

2. Issue access key

An access key (Access Key ID and Secret Access Key) is required to issue API authentication tokens. The permissions of the access key are based on the IAM role assigned when creating the project.

1. Go to KakaoCloud Console > User Settings > Access Key.

2. Refer to the Create access key documentation to create an access key.

3. After creation, copy the Access Key ID and Secret Access Key.
   - These will be used when setting up the Monitoring Flow steps.

   Create access key

3. Create scenario

Create a scenario and configure the schedule and scenario steps.

1. Go to KakaoCloud Console > Management > Monitoring Flow > Scenario.

2. Click the [Create scenario] button.

3. Set the schedule without selecting a flow connection.
   - For this scheduling, no communication between subnets is needed, so you can create the scenario without a flow connection.

   Create scenario

4. Add scenario step: API

Combine steps to complete the scenario. Refer to the following example and enter the values.

1. Go to KakaoCloud Console > Management > Monitoring Flow > Scenario.

2. Click the scenario name in the Scenario menu.

3. In the Details tab, click [Add scenario step] at the top right.

4. Refer to the following table to configure the items. Enter all three examples to save the header values.

   Item	Example 1	Example 2	Example 3	Description
   Type	API			Select API
   Step name	API-1			Enter the step name
   Expected code	200			Enter the expected status code for the URL - Defaults to 200 if not entered
   Method	POST			Select the API request method
   URL	Refer to the example - Enter /scale-out or /scale-in			Enter the API URL - Example: https://hadoop-eco.kr-central-2.kakaocloud.com/v2/hadoop-eco/clusters/{Cluster-ID}/scale-out - You can check the cluster ID in Analytics > Hadoop Eco > Cluster
   Header Key	Content-type	credential-id	credential-secret	Enter the header key - Refer to Create cluster
   Header Value	application/json	Access Key ID	Secret Access Key	Enter the header value - Refer to the Create cluster
   Body	{"scaleCnt": 1 }			Enter the body value - Variables can be used

   Scale out

5. Test scenario

You can test the scenario steps to ensure they work correctly.

1. After entering the required fields for the scenario steps, click the [Test] button at the top right.

2. Review the test results.

   Success result

3. If there are any errors, check the error details at the bottom of the right panel.

   info

   An error will occur if you try to access an inactive cluster. Ensure that the correct IP address is entered.

4. After confirming the test results, click [Close] at the top right to return to the scenario step editing screen.

5. In the scenario step editing screen, click [Save] at the top right to save the scenario. The saved scenario will be executed according to the set schedule.

6. For execution results, refer to 6. Check execution results below.

6. Check execution results

Review the execution results and times for completed scenarios.

1. Go to KakaoCloud Console > Management > Monitoring Flow > Scenario.

2. Select the scenario from the Scenario menu.

3. In the scenario details screen, click the Execution results tab.

   Scenario execution results list

4. Click the event in the execution results list to view the detailed results.

   Scenario execution result details

7. Create Alert Center alarm policy

You can receive alarms about Monitoring Flow scenario results (success/failure) by configuring an Alert Center alarm policy.

1. Go to KakaoCloud Console > Management > Alert Center > Alarm policy. Refer to the Create alarm policy documentation.

2. Refer to the following table for configuration. You can create multiple alarms by clicking Add condition.

   Item	Configuration
   Condition type	Metric
   Service	Monitoring Flow
   Condition	Set up success/failure alarms for metrics - By setting Scenario Success Count and Scenario Fail Count for each scenario, you can receive all metric alarms - To receive only failure alarms, set Scenario Fail Count only
   Resource item	Select the scenario to receive alerts for
   Threshold	More than 1 count
   Duration	1 minute

3. Click the [Next] button at the bottom to complete the alarm policy creation.

   Create alarm policy

4. Verify that the alarm policy is working by checking whether alarms for success or failure are received for Monitoring Flow scenario results.