Design a Notification System
A notification is more than just mobile push notification. Three types of notification formats are: mobile push notification, SMS message, and Email.
Understand the problem and establish design scope
- Types of notifications: push notification, SMS message, and email
- Real-time system: soft real-time. We want users to receive notifications as soon as possible. However, a slight delay is acceptable
- Supported devices: iOS, android, laptop/desktop
- What triggers notifications: Client applications. They can also be schedueld on the server-side
- Can users opt-out?: Yes
- How many notifications are sent out each day?: 10 million push notifications, 1 million SMS messages, and 5 million emails
Propose high-level design and get buy-in
Different types of notifications
Contant info gathering flow
To send notifications, we need to gather mobile device tokens, phone numbers, or email addresses. As shown below, when a user installs our app or signs up for the first time, API servers collect user contact info and store it in the database:
Figure 10-8 shows simplified database tables to store contact info. Email addresses and phone numbers are stored in the user table, whereas device tokens are stored in the device table. A user can have multiple devices, indicating that a push notification can be sent to all the user devices.
Notification sending/receiving
We will first present the initial design; then, identify possible problems for some optimizations:
Three problems are identified in this design:
- Single point of failure (SPOF): A single notification server means SPOF.
- Hard to scale: The notification system handles everything related to push notifications in one server. It is challenging to scale databases, caches, and different notification processing components independently.
- Performance bottleneck: Processing and sending notifications can be resource intensive. For example, constructing HTML pages and waiting for responses from third party services could take time. Handling everything in one system can result in the system overload, especially during peak hours.
After enumerating challenges in the initial design, we improve the design as listed below:
- Move the database and cache out of the notification server.
- Add more notification servers and set up automatic horizontal scaling.
- Introduce message queues to decouple the system components.
Service 1 to N: They represent different services that send notifications via APIs provided by notification servers.
Notification servers: They provide the following functionalities:
- Provide APIs for services to send notifications. Those APIs are only accessible internally or by verified clients to prevent spams.
- Carry out basic validations to verify emails, phone numbers, etc.
- Query the database or cache to fetch data needed to render a notification.
- Put notification data to message queues for parallel processing.
Cache: User info, device info, notification templates are cached.
DB: It stores data about user, notification, settings, etc.
Message queues: They remove dependencies between components. Message queues serve as buffers when high volumes of notifications are to be sent out. Each notification type is assigned with a distinct message queue so an outage in one third-party service will not affect other notification types.
Workers: Workers are a list of servers that pull notification events from message queues and send them to the corresponding third-party services.
Next, let us examine how every component works together to send a notification:
- A service calls APIs provided by notification servers to send notifications.
- Notification servers fetch metadata such as user info, device token, and notification setting from the cache or database.
- A notification event is sent to the corresponding queue for processing. For instance, an iOS push notification event is sent to the iOS PN queue.
- Workers pull notification events from message queues.
- Workers send notifications to third party services.
- Third-party services send notifications to user devices.
Design deep dive
We will explore the following in deep dive:
- Reliability.
- Additional component and considerations: notification template, notification settings, rate limiting, retry mechanism, security in push notifications, monitor queued notifications and event tracking.
- Updated design.
Reliability
How to prevent data lost?
Notifications can usually be delayed or re-ordered, but never lost. To satisfy this requirement, the notification system persists notification data in a database and implements a retry mechanism. The notification log database is included for data persistence:
Will recipients receive a notification exactly once?
You cannot have exactly once delivery
Additional components and consideration
We have discussed how to collect user contact info, send, and receive a notification. A notification system is a lot more than that. Here we discuss additional components including template reusing, notification settings, event tracking, system monitoring, rate limiting.
Notification template
The benefits of using notification templates include maintaining a consistent format, reducing the margin error, and saving time.
Notification setting
many websites and apps give users fine-grained control over notification settings. This information is stored in the notification setting table, with the following fields:
user_id bigInt
channel varchar
opt_in boolean
Before any notification is sent to a user, we first check if a user is opted-in to receive this type of notification.
Rate limiting
To avoid overwhelming users with too many notifications, we can limit the number of notifications a user can receive.
Retry mechanism
When a third-party service fails to send a notification, the notification will be added to the message queue for retrying. If the problem persists, an alert will be sent out to developers.
Security in push notifications
For iOS or Android apps, appKey and appSecret are used to secure push notification APIs. Only authenticated or verified clients are allowed to send push notifications using our APIs.
Monitor queued notifications
A key metric to monitor is the total number of queued notifications. If the number is large, the notification events are not processed fast enough by workers. To avoid delay in the notification delivery, more workers are needed.
Events tracking
Notification metrics, such as open rate, click rate, and engagement are important in understanding customer behaviors. Analytics service implements events tracking. Integration between the notification system and the analytics service is usually required.
Updated design
In this design, many new components are added in comparison with the previous design.
- The notification servers are equipped with two more critical features: authentication and rate-limiting.
- We also add a retry mechanism to handle notification failures. If the system fails to send notifications, they are put back in the messaging queue and the workers will retry for a predefined number of times.
- Furthermore, notification templates provide a consistent and efficient notification creation process.
- Finally, monitoring and tracking systems are added for system health checks and future improvements.
Wrap up
Nan