Apache Kafka is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java. The project aims to provide a unified, high-throughput, low-latency platform for handling real-time data feeds.

Prerequisite

  1. Install Kafka and configure the JMX Port using the environment variable.
export JMX_PORT=5199 && ./bin/kafka-server-start.sh -daemon config/server.properties
  1. For Virtual Machines, install the Linux Agent.

Configuring the credentials

Configure the credentials in the directory /opt/opsramp/agent/conf/app.d/creds.yaml

kafka:
- name: kafka
  user: <username>
  pwd: <Password>
  encoding-type: plain
  labels:
    key1: val1
    key2: val2

Configuring the application

Virtual machine

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-detection.yaml

- name: kafka
  instance-checks:
    service-check:
      - kafka
    process-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Docker environment

Configure the application in the directory /opt/opsramp/agent/conf/app/discovery/auto-container-detection.yaml

- name: kafka
  container-checks:
    image-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Kubernetes environment

Configure the application in config.yaml

- name: kafka
  container-checks:
    image-check:
      - kafka
    port-check:
      - 9092
  mon-type: "jmx"
  misc:
    jmx-port: "5199"

Validate

Go to Resources under the Infrastructure tab to check if your resources are onboarded and the metrics are collected.

Metrics

OpsRamp MetricMetric Display NameUnitDescription
kafka_channel_request_queue_sizeRequest Queue SizeRequestsNumber of queued requests
kafka_channel_response_queue_sizeResponse Queue SizeResponsesNumber of queued responses
kafka_controller_active_controller_countActive Controller CountControllersNumber of active controller on broker
kafka_fetch_requests_delayedFetch Delayed RequestsRequestsRequests delayed in the fetch purgatory
kafka_fetch_requests_purgatory_delayedFetch Delayed RequestsRequestsRequests delayed in the fetch purgatory
kafka_fetch_requests_purgatory_sizeFetch Purgatory SizeRequestsRequests waiting in the fetch purgatory
kafka_fetch_requests_sizeFetch Purgatory SizeRequestsRequests waiting in the fetch purgatory
kafka_jvm_gc_collection_countKafka JVM GC collection_countObjects collectedNumber of garbage objects collected
kafka_jvm_gc_collection_timeKafka JVM GC collection_timeSecondsTime taken for collection of the garbage objects
kafka_jvm_mem_heap_committedKafka JVM Mem heap_committedmegabytesHeap memory committed for the server
kafka_jvm_mem_heap_usedKafka JVM Mem heap_usedmegabytesHeap memory usage of the server
kafka_jvm_mem_non_heap_committedKafka JVM Mem non_heap_committedmegabytesNon-heap memory committed for the server
kafka_jvm_mem_non_heap_usedKafka JVM Mem non_heap_usedmegabytesNon-heap memory usage of the server
kafka_jvm_open_fdsKafka JVM OpenFDsOpen FDsNumber of Open file descriptors of the server
kafka_jvm_threadsKafka JVM ThreadsThreadsNumber of threads
kafka_jvm_uptimeUptimeMinutesUptime of the server
kafka_log_flush_rateLogFlush Rate And TimeLog flushesLog flush rate and time
kafka_metrics_controlled_shutdown_req_queue_timeControlled Shutdown Request Queue TimeRequest time,msRefers to the time for which the request is waiting in queue
kafka_metrics_controlled_shutdown_requestsControlled Shutdown RequestsRequests / secRequest rate
kafka_metrics_controlled_shutdown_resp_send_timeControlled Shutdown Response Send TimeRequest time,msTime to send the response
kafka_metrics_controlled_shutdown_total_timeControlled Shutdown Total TimeTotal time,msRequest total time
kafka_metrics_fetch_consumer_req_queue_timeFetch Consumer Request Queue TimemillisecondsRefers to the time for which the request is waiting in the request queue
kafka_metrics_fetch_consumer_resp_send_timeFetch Consumer Response Send TimemillisecondsTime to send the response
kafka_metrics_fetch_consumer_total_timeFetch Consumer Total TimemillisecondsRequest total time
kafka_metrics_fetch_follower_local_timeFetch Follower Local TimemillisecondsTime taken to process the request at the leader
kafka_metrics_fetch_follower_resp_queue_timeFetch Follower Response Queue TimemillisecondsTime taken by the request to wait in the response queue
kafka_metrics_fetch_requestsFetch RequestsRequests / secondRequest rate
kafka_metrics_leader_isr_local_timeLeader And Isr Local TimemillisecondsTime taken to process the request at the leader
kafka_metrics_leader_isr_remote_timeLeader And Isr Remote TimemillisecondsTime taken by the request to wait for the follower
kafka_metrics_metadata_req_queue_timeMetadata Request Queue TimemillisecondsTime taken by the request to wait in the request queue
kafka_metrics_offset_commit_remote_timeOffset Commit Remote TimemillisecondsTime taken by the request to wait for the follower
kafka_metrics_offset_commit_resp_queue_timeOffset Commit Response Queue TimemillisecondsTime taken by the request to wait in the response queue
kafka_metrics_offsets_req_queue_timeOffsets Request Queue TimemillisecondsTime taken by the request to wait in the request queue
kafka_metrics_offsets_resp_queue_timeOffsets Response Queue TimemillisecondsTime taken by the request to wait in the response queue
kafka_metrics_produce_remote_timeProducer Remote TimemillisecondsTime taken by the request to wait for the follower
kafka_metrics_stop_replica_total_timeStop Replica Total TimemillisecondsRequest total time
kafka_metrics_update_metadata_remote_timeUpdate Metadata Remote TimemillisecondsTime taken by the request to wait for the follower
kafka_metrics_update_metadata_requestsUpdate Metadata RequestsRequests / secondRequest rate
kafka_net_bytes_inBytes Inbytes / secondBytes in rate
kafka_net_bytes_outBytes Outbytes / secondBytes out rate
kafka_net_bytes_rejectedBytes Rejectedbytes / secondBytes Rejected
kafka_net_messages_inMessages InMessages / secondMessages in rate
kafka_producer_requests_delayedProducer Delayed RequestsRequestsRequests delayed in the producer purgatory
kafka_producer_requests_purgatory_delayedProducer Delayed RequestsRequestsRequests delayed in the producer purgatory
kafka_producer_requests_purgatory_sizeProducer Purgatory SizeRequestsRequests waiting in the producer purgatory
kafka_producer_requests_sizeProducer Purgatory SizeRequestsRequests waiting in the producer purgatory
kafka_replication_isr_expandsISR ExpandsExapands / secondISR expansion rate
kafka_replication_isr_shrinksISR ShrinksShrinks / secondISR shrink rate
kafka_replication_leader_countLeader CountTotal LeadersLeader replica counts
kafka_replication_leader_electionsLeader Election Rate And TimeLeader electionsLeader election rate
kafka_replication_max_lagReplication Max LagMessagesMaximum lag in messages between follower and leader replicas
kafka_replication_partitionsPartition CountTotal PartitionsPartition counts
kafka_replication_unclean_leader_electionsUnclean Leader ElectionsElections / secondUnclean leader election rate
kafka_replication_under_replicated_partitionsUnder Replicated PartitionsUnder Replicated PartitionsNumber of under replicated partitions(|ISR| < |all replicas|)
kafka_request_fetch_failedFailed Fetch RequestsRequests / secondFailed fetch requests rate
kafka_request_fetch_time_99percentileFetch Total Time 99percentileRequests / secondTime for produce requests for 99th percentile
kafka_request_fetch_time_avgFetch Total TimemillisecondsRequest total time
kafka_request_handler_avg_idle_pctRequest Handler Threads Idle TimeFractionAverage fraction of time when the request handler threads are idle
kafka_request_metadata_time_99percentileMetadata 99percentile TimemillisecondsTime for metadata requests for 99th percentile
kafka_request_metadata_time_avgMetadata Total TimemillisecondsRequest total time
kafka_request_offsets_time_99percentileOffsets Request Time 99percentilemillisecondsTime for offset requests for 99th percentile
kafka_request_offsets_time_avgOffsets Request TimemillisecondsAverage time for an offset request
kafka_request_produce_failedFailed Produce RequestsRequests / secondFailed producer requests rate
kafka_request_produce_time_99percentileProduce Request Time 99percentileRequest / secondTime for produce requests for 99th percentile
kafka_request_produce_time_avgProduce Request TimeRequest / secondAverage time for a produce request
kafka_request_update_metadata_time_99percentileUpdate Metadata 99percentile TimemillisecondsTime for update metadata requests for 99th percentile
kafka_request_update_metadata_time_avgUpdate Metadata Total TimemillisecondsRequest total time