HDFS DataNode

A DataNode stores data in HadoopFileSystem. A functional filesystem has more than one DataNode, with data replicated across them.

Prerequisite

For Virtual Machines, install the Linux Agent.

Configuring the credentials

Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml

hdfs-datanode:
- name: hdfs-datanode
  user: <username>
  pwd: <Password>
  encoding-type: plain
  labels:
    key1: val1
    key2: val2

Configuring the application

Virtual machine

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml

- name: hdfs-datanode
  instance-checks:
    process-check:
      - datanode
    port-check:
      - 50075
      - 9864
    uri-check:
      - "jmx?qry=Hadoop:service=DataNode,name=FSDatasetState*"
      - "jmx?qry=java.lang:type=OperatingSystem"

Docker environment

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Kubernetes environment

Configure the application in config.yaml

- name: hdfs-datanode
  container-checks:
    image-check:
      - datanode
    port-check:
      - 50075
      - 9864

Validate

Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.

Metrics


OpsRamp Metric	Metric Display Name	Unit	Description
hdfs_datanode_dfs_remaining	Dfs Remaining	bytes	The remaining disk space left
hdfs_datanode_dfs_remaining_percent	Dfs Remaining Percent	percent	The remaining disk space left
hdfs_datanode_dfs_capacity	Dfs Capacity	bytes	Capacity of the disk
hdfs_datanode_dfs_used	Dfs Used	bytes	Disk usage
hdfs_datanode_dfs_used_percent	Dfs Used Percent	percent	Disk usage
hdfs_datanode_cache_capacity	Cache Capacity	bytes	Capacity of the cache
hdfs_datanode_cache_used	Cache Used	bytes	Cache used
hdfs_datanode_num_failed_volumes	Num Failed Volumes		Number of failed volumes
hdfs_datanode_last_volume_failure_date	Last Volume Failure Date	milliseconds	Date/time of the last volume failure since epoch
hdfs_datanode_estimated_capacity_lost_total	Estimated Capacity Lost Total	bytes	Estimated capacity lost
hdfs_datanode_num_blocks_cached	Num Blocks Cached	Blocks	Number of blocks cached
hdfs_datanode_num_blocks_failed_to_cache	Num Blocks Failed To Cache	Blocks	Number of blocks that failed to cache
hdfs_datanode_num_blocks_failed_to_uncache	Num Blocks Failed To Uncache	Blocks	Number of failed blocks to remove from cache
hdfs_datanode_process_cpu_load	Process Cpu Load	Load	CPU Load of the process