Introduction
Amazon Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to destinations such as Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elasticsearch Service (Amazon ES), and Splunk. Kinesis Data Firehose is part of the Kinesis streaming data platform, along with Kinesis Data Streams, Kinesis Video Streams, and Amazon Kinesis Data Analytics. With Kinesis Data Firehose, you do not need to write applications or manage resources. You configure your data producers to send data to Kinesis Data Firehose, and it automatically delivers the data to the destination that you specified. You can also configure Kinesis Data Firehose to transform your data before delivering it.
Note
Use the OpsRamp AWS public cloud integration to discover and collect metrics against the AWS service.Setup
To set up the OpsRamp AWS integration and discover the AWS service,
go to AWS Integration Discovery Profile and select Firehose.
Metrics
| OpsRamp Metric | Metric Display Name | Unit | Aggregation Type | Description |
|---|---|---|---|---|
| aws_firehose_PutRecord_Bytes | PutRecord.Bytes | Bytes | Average | Number of bytes put to the Amazon Kinesis stream using PutRecord over the specified time period. |
| aws_firehose_PutRecord_Latency | PutRecord.Latency | ms | Average | Time taken per PutRecord operation, measured over the specified time period. |
| aws_firehose_IncomingBytes | IncomingBytes | Bytes | Average | Number of bytes successfully put to the Amazon Kinesis stream over the specified time period. This metric includes bytes from PutRecord and PutRecords. |
| aws_firehose_IncomingRecords | IncomingRecords | Count | Average | Number of records sucessfully put to the Amazon Kinesis stream over the specified time period. This metric includes record counts from PutRecord and PutRecords. |
| aws_firehose_DeliveryToElasticsearch_Bytes | DeliveryToElasticsearch.Bytes.firehose | Bytes | Average | Number of bytes indexed to Amazon ES over the specified time period. |
| aws_firehose_DeliveryToElasticsearch_Records | DeliveryToElasticsearch.Records.firehose | Count | Average | Number of records indexed to Amazon ES over the specified time period. |
| aws_firehose_DeliveryToElasticsearch_Success | DeliveryToElasticsearch.Success.firehose | Count | Average | Sum of the successfully indexed records over the sum of records that were attempted. |
| aws_firehose_DeliveryToRedshift_Bytes | DeliveryToRedshift.Bytes.firehose | Bytes | Average | Number of bytes copied to Amazon Redshift over the specified time period. |
| aws_firehose_DeliveryToRedshift_Records | DeliveryToRedshift.Records.firehose | Count | Average | Number of records copied to Amazon Redshift over the specified time period. |
| aws_firehose_DeliveryToRedshift_Success | DeliveryToRedshift.Success.firehose | Count | Average | Sum of successful Amazon Redshift COPY commands over the sum of all Amazon Redshift COPY commands. |
| aws_firehose_DeliveryToRedshift_Success | DeliveryToS3.Bytes.firehose | Bytes | Average | Number of bytes delivered to Amazon S3 over the specified time period. |
| aws_firehose_DeliveryToS3_DataFreshness | DeliveryToS3.DataFreshness.firehose | Seconds | Average | Age (from getting into Kinesis Firehose to now) of the oldest record in Kinesis Firehose. Any record older than this age has been delivered to the S3 bucket. |
| aws_firehose_DeliveryToS3_Records | DeliveryToS3.Records.firehose | Count | Average | Number of records delivered to Amazon S3 over the specified time period. |
| aws_firehose_DeliveryToS3_Success | DeliveryToS3.Success.firehose | Count | Average | Sum of successful Amazon S3 put commands over the sum of all Amazon S3 put commands. |
| aws_firehose_DescribeDeliveryStream_Latency | DescribeDeliveryStream.Latency.firehose | Milliseconds | Average | Time taken per DescribeDeliveryStream operation, measured over the specified time period. |
| aws_firehose_DescribeDeliveryStream_Requests | DescribeDeliveryStream.Requests.firehose | Count | Average | Total number of DescribeDeliveryStream requests. |
| aws_firehose_ListDeliveryStreams_Latency | ListDeliveryStreams.Latency.firehose | Milliseconds | Average | Time taken per ListDeliveryStream operation, measured over the specified time period. |
| aws_firehose_ListDeliveryStreams_Requests | ListDeliveryStreams.Requests.firehose | Count | Average | Total number of ListFirehose requests. |
| aws_firehose_PutRecord_Requests | PutRecord.Requests.firehose | Count | Average | Total number of PutRecord requests, which is equal to total number of records from PutRecord operations. |
| aws_firehose_PutRecordBatch_Bytes | PutRecordBatch.Bytes.firehose | Bytes | Average | Number of bytes put to the Kinesis Firehose delivery stream using PutRecordBatch over the specified time period. |
| aws_firehose_PutRecordBatch_Latency | PutRecordBatch.Latency.firehose | Milliseconds | Average | Time taken per PutRecordBatch operation, measured over the specified time period. |
| aws_firehose_PutRecordBatch_Records | PutRecordBatch.Records.firehose | Count | Average | Total number of records from PutRecordBatch operations. |
| aws_firehose_PutRecordBatch_Requests | PutRecordBatch.Requests.firehose | Count | Average | Total number of PutRecordBatch requests. |
| aws_firehose_UpdateDeliveryStream_Requests | UpdateDeliveryStream.Requests.firehose | Count | Average | Total number of UpdateDeliveryStream requests. |
| aws_firehose_UpdateDeliveryStream_Latency | UpdateDeliveryStream.Latency.firehose | Milliseconds | Average | Time taken per UpdateDeliveryStream operation, measured over the specified time period. |
| aws_firehose_BackupToS3_Bytes | BackupToS3 Bytes | Bytes | Average | Number of bytes delivered to Amazon S3 for backup over the specified time period. |
| aws_firehose_BackupToS3_DataFreshness | BackupToS3 DataFreshness | Seconds | Average | Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to the Amazon S3 bucket for backup. Kinesis Data Firehose emits this metric when data transformation is enabled for Amazon S3 or Amazon Redshift destinations. |
| aws_firehose_BackupToS3_Records | BackupToS3 Records | Count | Average | Number of records delivered to Amazon S3 for backup over the specified time period. |
| aws_firehose_BytesPerSecondLimit | Bytes Per Second Limit | Bytes | Average | The current maximum number of bytes per second that a delivery stream can ingest before throttling. |
| aws_firehose_DataReadFromKinesisStream_Bytes | Data Read From Kinesis Stream Bytes | Bytes | Average | When the data source is a Kinesis data stream, this metric indicates the number of bytes read from that data stream. |
| aws_firehose_DataReadFromKinesisStream_Records | Data Read From Kinesis Stream Records | Count | Average | When the data source is a Kinesis data stream, this metric indicates the number of records read from that data stream. |
| aws_firehose_DeliveryToElasticsearch_DataFreshness | Delivery To Elasticsearch Data Freshness | Seconds | Average | Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. |
| aws_firehose_DeliveryToSplunk_Bytes | Delivery To Splunk Bytes | Bytes | Average | Number of bytes delivered to Splunk over the specified time period. |
| aws_firehose_DeliveryToSplunk_DataAckLatency | Delivery To Splunk Data Ack Latency | Seconds | Average | The approximate duration it takes to receive an acknowledgement from Splunk after Kinesis Data Firehose sends it data. |
| aws_firehose_DeliveryToSplunk_DataFreshness | Delivery To Splunk Data Freshness | Seconds | Average | Age (from getting into Kinesis Data Firehose to now) of the oldest record in Kinesis Data Firehose. Any record older than this age has been delivered to Splunk. |
| aws_firehose_KinesisMillisBehindLatest | Kinesis Millis Behind Latest | Millisecond | Average | When the data source is a Kinesis data stream, this metric indicates the number of milliseconds that the last read record is behind the newest record in the Kinesis data stream. |
| aws_firehose_RecordsPerSecondLimit | Records Per Second Limit | Bytes | Average | Current maximum number of records per second that a delivery stream can ingest before throttling. |
| aws_firehose_ThrottledRecords | Throttled Records | Count | Average | Number of records that were throttled because data ingestion exceeded one of the delivery stream limits. |
| aws_firehose_IncomingPutRequests | Incoming Put Requests | Count | Average | Number of PutRecord and PutRecordBatch requests over the specified period of time. |
| aws_firehose_PutRequestsPerSecondLimit | Put Requests Per Second Limit | Count | Average | Maximum number of put requests per second that a delivery stream can handle before throttling. |
| aws_firehose_ThrottledDescribeStream | Throttled Describe Stream | Count | Average | Total number of times the DescribeStream operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_ThrottledGetRecords | Throttled Get Records | Count | Average | Total number of times the GetRecords operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_ThrottledGetShardIterator | Throttled Get Shard Iterator | Count | Average | Total number of times the GetShardIterator operation is throttled when the data source is a Kinesis data stream. |
| aws_firehose_KMSKeyAccessDenied | KMS Key Access Denied | Count | Average | Number of times the service encounters a KMSAccessDeniedException for the delivery stream. |
| aws_firehose_KMSKeyDisabled | KMS Key Disabled | Count | Average | Number of times the service encounters a KMSDisabledException for the delivery stream. |
| aws_firehose_KMSKeyInvalidState | KMS Key Invalid State | Count | Average | Number of times the service encounters a KMSInvalidStateException for the delivery stream. |
| aws_firehose_KMSKeyNotFound | KMS Key Not Found | Count | Average | Number of times the service encounters a KMSNotFoundException for the delivery stream. |
Event support
CloudTrail event support
- Supported (CreateDeliveryStream, DeleteDeliveryStream)
- Configurable in OpsRamp AWS Integration Discovery Profile.
CloudWatch alarm support
- Supported
- Configurable in OpsRamp AWS Integration Discovery Profile.