Check Hadoop Eco Information from CLI
You can check Hadoop Eco information in a CLI environment as follows.
Using WebHDFS
curl http://$(hostname -f):50070/webhdfs/v1/user/?op=LISTSTATUS
Check HDFS usage
You can check the usage of each HDFS data node.
Check HDFS usage
hdfs dfsadmin -report
Check HDFS usage
| Code | Description |
|---|---|
| Present Capacity | Total available HDFS capacity |
| DFS Remaining | Remaining HDFS capacity |
| DFS Used | Used HDFS capacity |
| Live datanodes (N) | N data nodes are running - Detailed usage per data node can be viewed |
Use WebHDFS REST API
You can use the curl command to access HDFS via REST API. Refer to the WebHDFS documentation for complete details.
Use WebHDFS REST API
curl -s http://$(hostname -f):50070/webhdfs/v1/user/?op=LISTSTATUS
Using WebHDFS REST API
YARN
| Command | Description |
|---|---|
| yarn node -list | View the list of NodeManagers |
| yarn node -list -showDetails | View detailed information of NodeManagers |
| yarn top | Check the status of jobs |
| yarn application -list | View the list of applications |
| yarn application -list -appStates RUNNING | View the list of applications filtered by their status |
| yarn application -status < Application ID > | Check the status of a specific application by < Application ID > |
| yarn application -kill < Application ID > | Terminate the application with < Application ID > |
| yarn logs -applicationId < Application ID > | View the logs of the application with < Application ID > |
| mapred queue -list | View the list of job queues |
| yarn queue -status < Queue Name > | View the status of a specific queue by < Queue Name > |
Check NodeManager status
You can check the status of NodeManagers.
Check NodeManager status
yarn node -list -showDetails
Check NodeManager status
Check job status
You can check the current state of the resource manager, including vCore and memory usage.
Check job status
yarn top
Check job status
View application list
You can view the list of currently running applications.
View application list
yarn application -list
View application list
| Code | Description |
|---|---|
| Application-Id | Application ID |
| Application-Name | Application name |
| Application-Type | Type of application |
| User | User |
| Queue | Queue name |
| State | Current state |
| Final-State | Final execution state |
| Progress | Completion progress |
| Tracking-URL | Application Master URL |
View application logs
You can check logs of currently running applications.
View application logs
yarn logs -applicationId <Application Id>
View application logs
View all queue list
You can view the list of all queues.
View all queue list
mapred queue -list
View all queue list
| Code | Description |
|---|---|
| Capacity | Allocated queue capacity |
| MaximumCapacity | Maximum capacity |
| CurrentCapacity | Current usage |
Check queue status
You can check the status of a specific queue.
Check queue status
yarn queue -status <Queue Name>
| Code | Description |
|---|---|
| Capacity | Allocated queue capacity |
| CurrentCapacity | Current usage |
| MaximumCapacity | Maximum capacity |
Check queue status