Introduction
Cloud Dataproc is a managed Apache Spark and Apache Hadoop service that lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning.
Cloud Dataproc automation helps you create clusters quickly, manage them easily, and save money by turning clusters off when you do not need them. With less time and money spent on administration, you can focus on your jobs and your data.
Setup
To set up the OpsRamp Google integration and discover the Google service,
go to Google Integration Discovery Profile and select GOOGLE/Dataproc Cluster.
Metrics
| OpsRamp Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|
| google_dataproc_cluster_hdfs_datanodes | Cluster Hdfs Datanodes | Count | Average | Indicates the number of HDFS DataNodes that are running inside a cluster. |
| google_dataproc_cluster_hdfs_storage_capacity | Cluster Hdfs Storage Capacity | Count | Average | Indicates capacity of HDFS system running on cluster in GB. |
| google_dataproc_cluster_hdfs_storage_utilization | Cluster Hdfs Storage Utilization | Count | Average | The percentage of HDFS storage currently used. |
| google_dataproc_cluster_hdfs_unhealthy_blocks | Cluster Hdfs Unhealthy Blocks | Count | Average | Indicates the number of unhealthy blocks inside the cluster. |
| google_dataproc_cluster_job_completion_time | Cluster Job Completion Time | Count | Average | The time jobs took to complete from the time the user submits a job to the time Dataproc reports it is completed. |
| google_dataproc_cluster_job_duration | Cluster Job Duration | Count | Average | The time jobs have spent in a given state. |
| google_dataproc_cluster_job_failed_count | Cluster Job Failed Count | Count | Average | Indicates the number of jobs that have failed on a cluster. |
| google_dataproc_cluster_job_running_count | Cluster Job Running Count | Count | Average | Indicates the number of jobs that are running on a cluster. |
| google_dataproc_cluster_job_submitted_count | Cluster Job Submitted Count | Count | Average | Indicates the number of jobs that have been submitted to a cluster. |
| google_dataproc_cluster_operation_completion_time | Cluster Operation Completion Time | Count | Average | The time operations took to complete from the time the user submits a operation to the time Dataproc reports it is completed. |
| google_dataproc_cluster_operation_duration | Cluster Operation Duration | Count | Average | The time operations have spent in a given state. |
| google_dataproc_cluster_operation_failed_count | Cluster Operation Failed Count | Count | Average | Indicates the number of operations that have failed on a cluster. |
| google_dataproc_cluster_operation_running_count | Cluster Operation Running Count | Count | Average | Indicates the number of operations that are running on a cluster. |
| google_dataproc_cluster_operation_submitted_count | Cluster Operation Submitted Count | Count | Average | Indicates the number of operations that have been submitted to a cluster. |
| google_dataproc_cluster_yarn_allocated_memory_percentage | Cluster Yarn Allocated Memory Percentage | Count | Average | The percentage of YARN memory is allocated. |
| google_dataproc_cluster_yarn_apps | Cluster Yarn Apps | Count | Average | Indicates the number of active YARN applications. |
| google_dataproc_cluster_yarn_containers | Cluster Yarn Containers | Count | Average | Indicates the number of YARN containers. |
| google_dataproc_cluster_yarn_memory_size | Cluster Yarn Memory Size | Count | Average | Indicates the YARN memory size in GB. |
| google_dataproc_cluster_yarn_nodemanagers | Cluster Yarn Nodemanagers | Count | Average | Indicates the number of YARN NodeManagers running inside cluster. |
| google_dataproc_cluster_yarn_pending_memory_size | Cluster Yarn Pending Memory Size | Count | Average | The current memory request, in GB, that is pending to be fulfilled by the scheduler. |
| google_dataproc_cluster_yarn_virtual_cores | Cluster Yarn Virtual Cores | Count | Average | Indicates the number of virtual cores in YARN. |
Event support
- Not supported