rauhan.sheikh
← Back to Work Directory
PROFESSIONAL[BULLMQ]2024–2025

Bulk Email Engine (Redis / BullMQ)

Asynchronous high-volume email delivery architecture replacing fragile browser-dependent API loops with resilient Redis/BullMQ worker queues.

< 0.05%
Failure Rate
From ~30% browser drops down to near zero
1,500+ / campaign
Batch Scale
Autonomous asynchronous background batches
100% Freed
Operator Time
Fire-and-forget submission with real-time job status

Engineered With

Node.jsBullMQRedisTypeScriptExpressDocker
Architecture Transformation · Background Queues

Why I Moved Bulk Email to BullMQ

The problem wasn't sending emails. The problem was where the work lived. Transitioning from client-tied loops to resilient Redis background workers.

SYSTEM TOPOLOGY// FRAGILE CLIENT-DEPENDENT
Browser Client
Executes for-loop for 1,500 rows
↓ Sequential HTTP requests
API Endpoint
Direct SMTP invoke
↓ Dependent on browser tab remaining open
⚠️ If coordinator shuts laptop at row 480: process dies, state is lost, resending triggers duplicate emails.

Every campaign forced the employee to babysit their browser. A network hiccup, laptop sleep mode, or single timeout permanently halted the dispatch.

Interactive Simulator1,500 Emails Batch
Dispatch Progress0 / 1,500

Click “Simulate Dispatch” to test how this architecture handles real-world batch workloads.

Failure rate dropped from ~30% to <0.05%Read Technical Essay

The Problem & Context

The original implementation made HTTP requests sequentially from the browser client in a loop: Browser -> API -> Email Service. If the employee closed their laptop, had a network hiccup, or hit rate limits on email #480, the campaign died midway. Resending resulted in duplicate emails to the first 480 recipients.

The Messy Part (Behind the Build)

Database state management during partial queue runs. When a worker dies or an email bounces, the recipient status in the database needed atomic locking to avoid race conditions between parallel worker threads.

Architecture & Design Decisions

Browser pushes batch payload to API -> API validates and enqueues jobs to Redis via BullMQ -> Decoupled background workers consume jobs with rate limiting and exponential retry strategies -> Worker publishes completion events via WebSockets to admin dashboard.

The Problem Wasn't Sending Emails. The Problem Was Where the Work Lived.

If an operation requires processing 1,500 records and someone tells the user:

"Please do not close this browser tab until the progress bar reaches 100%"

...that is not an operation. That is an architectural flaw.

Before: The Client-Tied Loop

Browser (Loop) ──HTTP──> API ──SMTP──> Provider
      │
(User closes laptop at 32%)
      │
      ▼
Campaign halts. 480 sent, 1020 lost. Re-run causes duplicates.

After: Decoupled Queue Architecture

Browser ──HTTP (Single Batch)──> API ──Push Job──> Redis (BullMQ)
                                                         │
                                               ┌─────────┴─────────┐
                                               ▼                   ▼
                                           Worker 1            Worker 2
                                               │                   │
                                               ▼                   ▼
                                            Provider            Provider

With workers processing asynchronously in Docker containers, the user can immediately close their laptop, take a train, or shut down their workstation. The work continues reliably, failures retry with exponential backoff, and idempotency guarantees that no student receives duplicate correspondence.

What I'd Do Differently Today

Add dead-letter queue (DLQ) automated replay UI directly into the admin dashboard from day one.

Related Work & Systems

PERSONAL

BusinessOS

Commercial-grade financial operating system for SMEs with ledger-first accounting, immutable transaction journals, integer minor-unit money calculations, and multi-tenant audit logs.

PROFESSIONAL

Microsoft Teams & Calendar Engine

Hiding deep Microsoft Graph API complexity behind an automated calendar & meeting sync engine for 20K+ EdTech users.