Chapter 3 stopped at “the tweet is in DynamoDB and we returned 201.” Now: how does it get from there into the home timelines of all the followers we tallied in chapter 5? Two strategies on the table — pull at read time, or push at write time.
Architecture
flowchart TD
TS["Tweet Service
(Fargate)"]
DB[("DynamoDB
— tweets —")]
DS["DynamoDB Streams"]
Bridge["MSK Connect"]
Kafka["Kafka (MSK)"]
FW["Fanout Worker
(Fargate)"]
Redis[("Redis
home_timeline:{user_id}")]
DR[("DynamoDB
— followers —")]
TS -->|"PutItem"| DB
DB --> DS
DS --> Bridge
Bridge -->|"TweetCreated event"| Kafka
Kafka --> FW
FW -->|"reads follower list"| DR
FW -->|"prepend + trim"| Redis
Pull (fanout-on-read)
When you load your home timeline, the Timeline Service:
- Looks up everyone you follow (the
followingtable from chapter 5). - Queries each of their tweet stores for the latest N tweets.
- Merges, sorts by time, returns the top 20.
Pros: Trivial writes. Posting a tweet is one DB insert. No precomputed state.
Cons: Reads are expensive. 500K reads/s × 500 follows = 250M downstream reads/s. Latency is unpredictable (slowest shard wins). Falls apart at scale.
Good for: a system where users follow few people, or read timelines rarely. Not Twitter.
Push (fanout-on-write)
When you post a tweet, a worker pushes the tweet ID into a precomputed list — your home timeline cache — for each of your followers.
- The tweet has been written to DynamoDB (chapter 3).
- A
TweetCreatedevent reaches the fanout worker. - Worker reads
followers(followee_id=author)(chapter 5). - Worker does N writes — one per follower’s
home_timeline:{user_id}Redis list: prepend the tweet ID and trim the list to the latest 800 entries.
Reads become trivial: read the first 20 entries of home_timeline:{me} — a single in-memory list lookup. (Read path covered in chapter 7.)
Why 800 entries per user
The cap bounds storage and keeps each list small enough to read in one round-trip. 800 IDs at 8 bytes is ~6 KB per user; across 300M users that’s ~2 TB total — fits in a Redis cluster. Without a cap, a follower of 500 prolific accounts would accumulate millions of IDs they’ll never scroll to.
800 is also well past where real users stop scrolling. At 20 tweets per page that’s 40 pages of pull-to-refresh. Anyone who scrolls deeper falls off the cache and onto a slower path (chapter 7).
Pros: Read latency is constant and tiny. The 100:1 read:write ratio means doing more work on the rare write to save it on the common read is the right trade.
Cons: Write amplification. Storage waste — most followers won’t read most tweets you push to them.
The choice
Push wins. The 100:1 read-to-write ratio means doing the work once on write and never again on read is the right trade. The rest of this chapter walks the push path.
How the worker actually gets triggered
The Tweet Service finished its job at 201. Something needs to wake the fanout worker up. The naive option is for the Tweet Service to also publish to a queue after the DB write — but that’s a dual write, and the two operations can’t be made atomic:
- DB commits, queue publish fails → tweet exists, no fanout. Silent data loss downstream.
- Queue publishes, DB commit fails (or service crashes between) → phantom event for a tweet that doesn’t exist.
The fix is to derive the event from the committed write itself. DynamoDB Streams emits a change record for every row that lands in the tweets table. A bridge process reads the stream and produces a TweetCreated event onto Kafka. If the write didn’t commit, there’s no stream record. If it did, the stream record is delivered at least once. Standard CDC / transactional outbox pattern.
How to actually wire DynamoDB Streams to Kafka
MSK Connect with a source connector. A Kafka Connect cluster with a DynamoDB CDC connector tails the stream and writes directly to topics. The contract is: one stream record → one Kafka event, keyed by tweet ID, with at-least-once delivery. The fanout worker must be idempotent — re-pushing a tweet ID into a Redis list is safe (dedupe on read, or strip duplicates after the fact).