The first seven chapters walked the happy path: post, follow, fan out, read. Each one quietly assumed load is roughly evenly distributed across users. It isn’t. Taylor Swift has ~95M followers; the median user has ~200. Every action that touches a celebrity concentrates load on a single key or partition.
This chapter and the next three address that — same root cause, surfacing in different actions (posting, reading, counting). All three need the same cheap, local check on the hot path: given a user_id, are they a celebrity? A DynamoDB lookup per tweet write or per timeline read defeats the point. Before any of the per-action fixes, we need the plumbing that answers that question.
Architecture
flowchart TD
FT[("DynamoDB
— followers —")]
CA["Count Aggregator
(Lambda)"]
DU[("DynamoDB
— users —")]
UC[("Redis
user_counts:{user_id}
{follower_count, is_celebrity}")]
K["Kafka
(CelebrityFlagChanged)"]
FT -.->|"DynamoDB Stream"| CA
CA -->|"UpdateItem ADD ±1"| DU
CA -->|"increment / set field"| UC
CA -->|"on threshold cross"| K
What “celebrity” means here
A celebrity, for this system, is a user whose follower count is large enough that fanning their tweets out at write time costs more than letting their followers pull at read time. A round number for the threshold: > 1M followers. Where the cutoff really sits is a tuning question — chapter 9 picks that up.
The user_counts cache
Chapter 5 maintains follower_count and following_count on the DynamoDB users item, updated by a Count Aggregator off the followers stream. That’s fine for a profile page, where one GetItem per page load is nothing. The fanout worker (chapter 6) and the home-timeline reader (chapter 7) are different — they hit this lookup on every tweet and every timeline fetch, so the cost is QPS and RCU load on the users table, not the latency of any single call. Promote the counters to a Redis hash:
Key: user_counts:{user_id}
Type: Redis HASH
Value: { follower_count, following_count, is_celebrity }Extend the Count Aggregator: every time it processes a follow or unfollow event from the followers stream and bumps the counter in DynamoDB, have it bump the matching field in the Redis hash by the same amount. DynamoDB stays the source of truth; Redis is a cache that can be rebuilt from the users row on miss.
Setting the flag
Same processor. After applying the delta, compare the new follower_count against the threshold:
- Crossing above: set
is_celebrity = trueonuser_counts:{user_id}, publishCelebrityFlagChanged{user_id, promoted: true}. - Crossing below: same with
false.
Why publish an event? When a user crosses the threshold, the fanout worker needs to stop (or start) fanning their tweets out, and every one of their followers needs that user added to (or removed from) a “celebrities I follow” set that the home-timeline reader uses at read time. The Count Aggregator shouldn’t do that work itself — it’s a per-event processor, not something that should iterate over millions of followers. It publishes the change and lets separate consumers handle the follow-up; chapter 10 wires up the follower-set consumer. Threshold crossings are rare, so the event stream is low-volume by definition.
On cache miss
If user_counts:{author_id} is missing when the fanout worker reads it, fall back to the DynamoDB users row, recompute is_celebrity from follower_count, and populate Redis with a long TTL. Misses are rare in steady state — the cache is written on every follow event for the user, so any actively-followed account stays warm.