TL;DRWe built a distributed timer service capable of handling 100,000 timer creations per second with high precision and at least once delivery guarantees. The architecture separates concerns between a stateless Timer Service API (for CRUD operations) and horizontally scalable Timer Processors (for expiration handling). Workers scan their partitions for soon-to-expire timers (2-3 minute look-ahead window), load them into in-memory data structures for precise firing, then publish notifications to Kafka. ZooKeeper coordinates partition ownership among workers, preventing duplicate processing through ephemeral nodes and automatic rebalancing. DynamoDB provides the storage layer with a clever GSI design using time-bucketing and worker assignment for efficient scanning. Key innovations include temporal partitioning via time buckets, a two-stage scan-and-fire mechanism, ZooKeeper-based coordination, checkpoint-based recovery, and at-least-once delivery semantics.Tech Stack: DynamoDB, Kafka, ZooKeeper\The Problem: Why We Need a Generic Timer ServiceIn today's microservices landscape, countless applications need to schedule delayed actions: sending reminder emails, expiring user sessions, triggering scheduled workflows, or managing SLA-based notifications. Yet despite this universal need, most teams either build bespoke solutions or rely on heavyweight job schedulers that aren't optimized for high-throughput timer management.What if we could build a generic, horizontally scalable timer service that handles 100,000 timer creations per second while maintaining high precision and reliability? Let's dive into the architecture.\Functional RequirementsOur timer service needs to support four core operations:Create Timer: Allow users to schedule a timer with custom expiration times and notification metadataRetrieve Timer: Query existing timer details by IDDelete Timer: Cancel timers before they fireNotify on Expiration: Reliably deliver notifications when timers expire\Non-Functional RequirementsThe real challenge lies in the non-functional requirements:High Throughput: Support ~100,000 timer creations per secondPrecision: Maintain accuracy in timer expiration (minimize drift)Scalability: Handle burst scenarios where thousands of timers fire simultaneouslyAvailability: Ensure the timer creation service remains highly available\Architecture OverviewThe system consists of four main components working in concert:\1. Timer Service (API Layer)The Timer Service exposes a RESTful API for timer management:Create TimerPOST /createTimer{ "UserDrivenTimerID": "user-defined-id", "Namespace": "payment-reminders", "timerExpiration": "2025-11-10T18:00:00Z", "notificationChannelMetadata": { "topic": "payment-notifications", "context": {"orderId": "12345"} }}Retrieve TimerGET /timer?timerId=Delete TimerDELETE /timer?timerId=The API layer sits behind a load balancer, distributing requests across multiple service instances for horizontal scalability.2. Database Layer (DynamoDB)We use DynamoDB for its ability to handle high write throughput with predictable performance. The table is structured for our access patterns:Timers TablePrimary Key: namespace:UserDrivenTimerID:uuidThis composite key ensures even distribution across partitions while allowing user-defined identifiers.Key Attributes:expiration_timestamp: Human-readable expiration timetime_bucket: Temporal partitioning for efficient scanningworkerId: Worker assignment for load distributionMessageMetadata: JSON containing Kafka topic and context dataGlobal Secondary Index (GSI): timers_scan_gsiPartition Key: time_bucket:workerIdSort Key: expiration_timestampThis GSI is the secret sauce enabling efficient timer scanning. By combining time buckets with worker IDs, we achieve:Temporal partitioning (preventing hot partitions)Worker-level isolation (each processor scans its assigned partition)Ordered retrieval (sort by expiration enables sequential processing)Checkpoint TablePrimary Key: worker_idEach timer processor maintains a checkpoint containing:{ "time_bucket": "2025-11-10-18", "expiration_time": "2025-11-10T18:30:45Z"}This enables crash recovery and prevents duplicate processing.3. ZooKeeper (Coordination Layer)Before processors can scan partitions, they need to coordinate who owns what. This is where ZooKeeper comes in.ZooKeeper manages partition ownership to ensure each partition is processed by exactly one worker at any time, preventing duplicate processing and wasted resources.How it works:Worker Registration: When a Timer Processor starts, it registers itself as an ephemeral node in ZooKeeper (e.g., /workers/worker-1)Partition Assignment: Workers watch the /workers path and participate in partition rebalancing when:A new worker joins (scale up)A worker crashes (ephemeral node disappears)A worker gracefully shuts downLeader Election: ZooKeeper handles leader election for partition assignment coordinationOwnership Tracking: Each worker maintains a lock on its assigned partitions (e.g., /partitions/partition-5/owner → worker-2)Rebalancing Example:Initial: 10 partitions, 2 workers- Worker-1: partitions [0,1,2,3,4]- Worker-2: partitions [5,6,7,8,9]Worker-3 joins → Rebalance triggered- Worker-1: partitions [0,1,2,3]- Worker-2: partitions [4,5,6]- Worker-3: partitions [7,8,9]Benefits:No duplicate work: Only one worker processes each partitionAutomatic failover: If a worker crashes, its partitions are reassignedDynamic scaling: Add/remove workers without downtimeConsistent view: All workers see the same partition assignments4. Timer Processors (Consumer Workers)Timer processors are the workhorses of the system. Each processor follows a two-stage approach: scan and schedule, then fire and notify.Stage 1: Scan and Schedule (every 30-60 seconds)Claims partitions via ZooKeeper coordination (ensuring exclusive ownership)Scans for soon-to-expire timers using the GSI, looking ahead by a configurable window (typically 2-3 minutes): // DynamoDB Query using the timers_scan_gsi { TableName: "Timers", IndexName: "timers_scan_gsi", KeyConditionExpression: "time_bucket_worker = :tbw AND expiration_timestamp BETWEEN :checkpoint AND :lookahead", ExpressionAttributeValues: { ":tbw": "2025-11-10-18:worker-1", ":checkpoint": last_checkpoint_time, // e.g., "2025-11-10T18:42:00Z" ":lookahead": current_time + 3_minutes // e.g., "2025-11-10T18:48:00Z" } }Creates in-memory timers for each retrieved timer using a data structure like a priority queue or timing wheel: InMemoryTimer { timerId: "abc-123" expirationTime: "2025-11-10T18:45:30Z" messageMetadata: {...} }Updates checkpoint to track scan progress, preventing re-processing of the same timersStage 2: Fire and Notify (continuous)Monitors in-memory timers - When a timer expires:Extract the notification metadataPublish message to Kafka with the configured topic and contextMark timer for deletionDeletes processed timers from DynamoDB (asynchronously, in batches for efficiency)Maintains heartbeats with ZooKeeper to retain partition ownershipAt-Least-Once Delivery GuaranteeThe system guarantees at-least-once delivery through several mechanisms:Checkpoint lag: Checkpoints are updated after creating notification. If a worker crashes after notifying, the next scan will re-fetch those timers.Timer deletion delay: Timers are deleted from DynamoDB only after successful Kafka publish, but the deletion happens asynchronouslyKafka durability: Messages are persisted in Kafka before acknowledgmentRetry on failure: If Kafka publish fails, the timer remains in memory for retryExample Timeline:T+0s: Scan finds timer expiring at T+120sT+0s: Create in-memory timer, update checkpointT+120s: In-memory timer firesT+120s: Publish to KafkaT+121s: Async delete from DynamoDB (batch)If the worker crashes at T+90s, the replacement worker will:Read checkpoint (T+0s)Re-scan the partitionRe-fetch the same timer (it's still in DynamoDB)Create a new in-memory timerFire it at T+120s (might be slightly delayed due to crash recovery)The processors run continuously, scanning their assigned partitions at regular intervals (30-60 seconds) while the in-memory timers fire with millisecond precision.5. Kafka + Consumer LayerProcessed timers are published to Kafka topics, where user-owned consumers can subscribe and handle notifications according to their business logic. This decoupling provides:Flexibility: Users define their own notification handlersReliability: Kafka's durability ensures messages aren't lostScalability: Consumer groups can scale independently\Design Deep DiveSeparation of ConcernsThe architecture deliberately separates the Timer Service (write path) from Timer Processors (read/process path). This separation enables:Independent scaling: Scale writers during creation bursts, scale processors during expiration burstsAvailability isolation: Timer creation remains available even if processors face issues (notifications may be delayed but timers are persisted)Operational flexibility: Deploy, upgrade, and maintain components independentlyTime Bucketing StrategyTime buckets are crucial for managing scan efficiency. Consider bucketing by hour:Timer expiring at 2025-11-10T18:45:00Z → bucket 2025-11-10-18Timer expiring at 2025-11-10T19:15:00Z → bucket 2025-11-10-19Benefits:Limited scan scope: Processors only scan current and near-future bucketsPredictable load: Each bucket's size is bounded by timers created for that hourEasy archival: Old buckets can be archived or deletedWorker Assignment and Load DistributionThe combination of workerId field and ZooKeeper coordination enables robust horizontal scaling:During Timer Creation:Timer Service assigns a worker using consistent hashing: workerId = hash(namespace:UserDrivenTimerID) % worker_countThis ensures even distribution across workersThe workerId is stored with the timer for routingDuring Timer Processing:Workers register with ZooKeeper and participate in partition assignmentEach worker claims ownership of specific partition ranges via ZooKeeper locksWorkers only scan partitions they own in the GSI: time_bucket:workerIdZooKeeper ensures no two workers process the same partition simultaneouslyPreventing Duplicate Work:Ephemeral nodes: When a worker crashes, its ZooKeeper node disappearsAutomatic rebalancing: Remaining workers redistribute the orphaned partitionsGraceful shutdown: Workers release partitions before terminating\This design eliminates race conditions and ensures exactly-once processing per timer.Handling Scale100K writes/second across DynamoDB:With 1KB average timer size, that's ~100MB/sDynamoDB's WCU (Write Capacity Units) can easily handle this with proper partitioningThe composite partition key ensures writes distribute evenlySimultaneous expiration handling:Workers scan ahead and load timers into memory before expirationIn-memory data structures (priority queues/timing wheels) fire timers with high precisionMultiple processors work in parallel on different partitionsEach processor can handle thousands of in-memory timers concurrentlyKafka provides the backpressure management for downstream consumersMemory considerations:With a 3-minute look-ahead window at 100K creates/sec: ~18M timers in memory across all workersAt 1KB per timer: ~18GB total memory footprintDistributed across 10 workers: ~1.8GB per worker (manageable)Can adjust look-ahead window based on memory constraintsAt-least-once delivery impact:Duplicate notifications are rare (only on worker crashes during the look-ahead window)Consumers can implement idempotency using timer IDs\Trade-offs and ConsiderationsEventual ConsistencyThere's a small window between timer creation and processor visibility (DynamoDB GSI replication lag, typically milliseconds). For most use cases, this is acceptable.Precision vs. ThroughputThe two-stage approach (scan → in-memory → fire) creates interesting trade-offs:Scan Interval (30-60 seconds):Determines how quickly new timers become visible to processorsLonger intervals → lower database load, higher risk of missing timers if workers crashShorter intervals → more database queries, faster recovery from failuresLook-ahead Window (2-3 minutes):Too short → risk of missing timers if scan is delayedToo long → more in-memory timers, higher memory usageBalances memory footprint with reliabilityIn-Memory Timer Precision:Once loaded in memory, timers fire with millisecond precisionUses efficient data structures (timing wheels or priority queues)End-to-end latency: database polling interval + Kafka publish time\Alternative ApproachesWhy Not Redis with Sorted Sets?Redis with sorted sets (using expiration timestamp as score) is a popular alternative. However:Memory constraints limit scalePersistence and durability require careful configurationWhy Not Kafka with Timestamp-based Topics?Using Kafka's timestamp-based retention is interesting but:Requires custom consumer logic for time-based processingDoesn't support easy retrieval and deletion of pending timersRetention policies may conflict with timer expiration times\ConclusionBuilding a distributed timer service that handles 100,000 operations per second requires careful consideration of data modeling, partitioning strategies, and component separation. By leveraging DynamoDB's scalability, implementing smart time-bucketing, and separating concerns between creation and processing, we can build a robust, horizontally scalable timer service.The architecture described here provides a solid foundation that can be adapted to various use cases: from simple reminder systems to complex workflow orchestration engines. The key is understanding your specific requirements around precision, throughput, and consistency, then tuning the system accordingly.What timer-based challenges are you solving in your systems? How would you extend this architecture for your use case?\