Using Spark History Server

With Hadoop Eco, you can monitor Spark jobs using the Spark History Server. The Spark History Server provides a visual representation of the logs of Spark jobs executed by users through the Spark Web UI, allowing you to check the job's execution status and results.

caution

There may be data discrepancies between Spark History Servers within the cluster due to the log upload interval.

Spark history server features

The Spark History Server provides the following features:

• Monitor Spark jobs within and outside the cluster
• View events and container logs of running Spark applications
• View the history of Spark jobs in terminated clusters

Collect Spark history server logs

To use the Spark History Server, Hadoop Eco collects and stores the following two types of user logs:

Log Type	Description
Spark Apps Logs	Event logs collected when Spark jobs are executed, containing job execution time, task status, etc.
YARN Container Logs	Logs generated by Spark executors during the execution of Spark jobs, with stdout and stderr stored in YARN container logs.

These logs are stored in the location specified by the user when creating the cluster.
(Configure log storage can be set when creating the cluster, and settings cannot be changed after the cluster creation.)

However, the logs may not be collected properly under the following conditions.

Spark History Server execution conditions

1. Hadoop Eco cluster status

• If the cluster is not in the Running state, the Spark History Server may not run properly.
• However, it is possible to run the Spark History Server in clusters that are in the Terminated state as long as the logs have been properly collected.

2. Log collection settings

• Logs may not be collected properly if the following settings are changed by the user.

  • Spark's spark-defaults.conf configuration

  spark.eventLog.enabled                  true
  spark.eventLog.dir                      hdfs:///var/log/spark/apps
  • YARN's yarn-site.xml configuration
  How to install Kerberos and Ranger

  <configuration>
      <property>
          <name>yarn.log-aggregation-enable</name>
          <value>true</value>
      </property>
  
      <property>
          <name>yarn.nodemanager.log-dirs</name>
          <value>/hadoop/yarn/log</value>
      </property>
  
      <property>
          <name>yarn.nodemanager.log.retain-seconds</name>
          <value>604800</value>
      </property>
  </configuration>

caution

Setting the log retention period too short may make it difficult to check the logs.

Run Spark History Server

1. Go to the KakaoCloud Console > Analytics > Hadoop Eco menu.

2. In the Clusters menu, click the [Create Cluster] button located at the top-right.

3. In Step 3: Detailed Settings > Cluster Detailed Settings, change the log storage setting to [Enabled].

4. Select an Object Storage bucket, specify the desired path, and click the [Create] button.

5. Once the created Hadoop Eco cluster's status changes to Running, select the cluster and go to the cluster detail page.

6. In the detail page, click the Run Spark History Server button at the top-right.

   • The server will start within 5 minutes after clicking the button, and once it finishes, the button will change to Open Spark History Server, and the server will automatically run.
   • The Spark History Server will remain running for a certain time if there is a request; otherwise, it will automatically shut down.
   • Even if the server is shut down, the user logs are not deleted, and the server can be restarted by clicking the Run Spark History Server button.

caution

• If the log files are large, the Spark History Server may terminate due to memory issues.
• If the stored logs are deleted, the Spark History Server may not function properly.
• If security features are enabled, log collection errors may occur depending on the settings.