How AWS Kinesis and Lambda Work Together to Send Emails Instantly
Recently, I was working on a new project — a clone of the company’s original product — and was assigned a ticket regarding an email sending issue on a high-priority feature. While working on the ticket, I came across the AWS service Kinesis. I added all necessary constants, configurations, and conditions for the new project into the code and ran a self-call (I work with NodeJS). However, I encountered the following error: ResourceNotFoundException: Stream l*l_microservice_backend under account 2********1 not found. Reason: The Kinesis data stream l***l_microservice_backend, originally present in our product instance, had not yet been created on the new project’s AWS instance. Solution: I created the Kinesis data stream in the new instance with the help of my mentor. What is a stream, and where does Kinesis come in? Amazon Kinesis is a real-time data streaming service. It is used to process and stream large volumes of data efficiently. Kinesis is commonly used for: Events Clickstreams IoT sensor data Video streams Logs Kinesis has 4 Core components: Kinesis Data Streams (KDS) Kinesis Data Firehose Kinesis Data Analytics Kinesis Video streams We implement our email sending service using KDS or Data Streams in Kinesis. We can create data-processing applications, known as Kinesis Data Streams applications. A typical Kinesis Data Streams application reads data from a data stream as data records. When we say we created a new "data stream," we are actually referring to setting up a data stream and a Kinesis Data Streams application to process a specific set of data records. The delay between the time a record is put into the stream and the time it can be retrieved (put-to-get delay) can be less than 1 second. We can scale the stream up or down based on the application’s requirements, so we never lose any data. How we used Kinesis? In Kinesis, there are two major components: Producer: Anything that puts data into the stream. Consumer: Anything that reads data from the stream and processes it. In our case, the producer is the service that I was testing, and a subsequent function that communicates directly with the KDS. In a sequence of service calls, we prepared some data and used that data further in an email template. Then we sent the email template along with some other details to the KDS as the producer. The consumer of the KDS in our case is a Lambda. The Lambda contains code to process the output of the data stream application to perform the needed actions. Our Lambda decodes the response from KDS into a usable form. Then the Lambda code uses the business logic to take the required action— here sending emails. Using Kinesis makes it possible to send bulk emails instantly. Summary What Kinesis does: It's an AWS service that handles real-time data like events, logs, or sensor data super fast. It has different parts, but we mainly use Kinesis Data Streams (KDS) for sending emails in our app. How we use it: One of our services acts like a producer — it prepares the email details and sends them into the stream. A Lambda function reads from the stream, processes the info, and sends out the emails. Using Kinesis helps us send bulk emails almost instantly and handle scaling easily.
Recently, I was working on a new project — a clone of the company’s original product — and was assigned a ticket regarding an email sending issue on a high-priority feature.
While working on the ticket, I came across the AWS service Kinesis.
I added all necessary constants, configurations, and conditions for the new project into the code and ran a self-call (I work with NodeJS). However, I encountered the following error:
ResourceNotFoundException: Stream l*l_microservice_backend under account 2********1 not found.
Reason:
The Kinesis data stream l***l_microservice_backend, originally present in our product instance, had not yet been created on the new project’s AWS instance.
Solution:
I created the Kinesis data stream in the new instance with the help of my mentor.
What is a stream, and where does Kinesis come in?
Amazon Kinesis is a real-time data streaming service.
It is used to process and stream large volumes of data efficiently.
Kinesis is commonly used for:
- Events
- Clickstreams
- IoT sensor data
- Video streams
- Logs
Kinesis has 4 Core components:
- Kinesis Data Streams (KDS)
- Kinesis Data Firehose
- Kinesis Data Analytics
- Kinesis Video streams
We implement our email sending service using KDS or Data Streams in Kinesis.
We can create data-processing applications, known as Kinesis Data Streams applications.
A typical Kinesis Data Streams application reads data from a data stream as data records.
When we say we created a new "data stream," we are actually referring to setting up a data stream and a Kinesis Data Streams application to process a specific set of data records.
The delay between the time a record is put into the stream and the time it can be retrieved (put-to-get delay) can be less than 1 second.
We can scale the stream up or down based on the application’s requirements, so we never lose any data.
How we used Kinesis?
In Kinesis, there are two major components:
- Producer: Anything that puts data into the stream.
- Consumer: Anything that reads data from the stream and processes it.
In our case, the producer is the service that I was testing, and a subsequent function that communicates directly with the KDS.
In a sequence of service calls, we prepared some data and used that data further in an email template.
Then we sent the email template along with some other details to the KDS as the producer.
The consumer of the KDS in our case is a Lambda.
The Lambda contains code to process the output of the data stream application to perform the needed actions.
Our Lambda decodes the response from KDS into a usable form.
Then the Lambda code uses the business logic to take the required action— here sending emails.
Using Kinesis makes it possible to send bulk emails instantly.
Summary
What Kinesis does:
It's an AWS service that handles real-time data like events, logs, or sensor data super fast.
It has different parts, but we mainly use Kinesis Data Streams (KDS) for sending emails in our app.
How we use it:
One of our services acts like a producer — it prepares the email details and sends them into the stream.
A Lambda function reads from the stream, processes the info, and sends out the emails.
Using Kinesis helps us send bulk emails almost instantly and handle scaling easily.
