Day 10: 𝐊𝐢𝐧𝐞𝐬𝐢𝐬 𝐃𝐚𝐭𝐚 𝐒𝐭𝐫𝐞𝐚𝐦𝐬

👉 Producers > Kinesis Data Streams > Consumers, VPC Endpoints available

👉 Partition Key, Sequence no, Data Blob, 1 day to 365 days of Retention 👉 Provisioned mode/ On-demand mode, Supports replay capability

👉 Security (IAM policies, HTTPS endpoints, KMS, encryption on client side)

👉 Monitor API calls using CloudTrail

1.Producers > Kinesis Data Streams > Consumers, VPC Endpoints available:

Producers are data sources that push data into Kinesis Data Streams.

• Kinesis Data Streams is the service that collects and manages real-time data streams.
• Consumers are applications or services that read and process data from the streams.
• VPC (Virtual Private Cloud) Endpoints are available, which allow you to securely access Kinesis Data Streams from within your VPC, enhancing data isolation and security.

2.Partition Key, Sequence no, Data Blob, 1 day to 365 days of Retention:

• Partition Key: Data is divided into shards based on a partition key. Data with the same partition key goes to the same shard, allowing for parallel processing.
• Sequence Number: It is a unique identifier for records within a shard and helps maintain the order of records.
• Data Blob: The actual data in a record is referred to as the data blob.
• Kinesis Data Streams supports data retention ranging from 1 day to 365 days, meaning data can be stored for a specified duration.

3.Provisioned mode/On-demand mode, Supports replay capability:

• Provisioned mode allows you to pre-configure the number of shards to accommodate your expected data volume.
• On-demand mode dynamically adjusts the number of shards based on the incoming data rate.
• Supports replay capability: You can reprocess or replay data from the stream, which is useful for debugging, reprocessing, or reanalysing data.

4.Security (IAM policies, HTTPS endpoints, KMS, encryption on client side):

• IAM (Identity and Access Management) policies are used to control access to Kinesis Data Streams.
• HTTPS endpoints ensure data transmitted to and from Kinesis is encrypted over the network.
• KMS (Key Management Service) can be used to encrypt the data at rest in Kinesis Streams.
• Client-side encryption allows data producers to encrypt the data before sending it to Kinesis Streams.

5.Monitor API calls using CloudTrail:

• AWS CloudTrail is a service for logging API calls made on your AWS account.
• You can use CloudTrail to monitor and audit API calls related to Kinesis Data Streams, which helps in tracking and ensuring security and compliance.

Amazon Kinesis Data Streams is a powerful tool for ingesting, processing, and analysing streaming data, making it a valuable resource for a wide range of real-time data use cases, including log and event data, IoT (Internet of Things) data, and more.