That is the wrong approach.

When designing systems like LeetCode, start with a simple question:

“What exactly are we trying to build?”

System design is not about throwing fancy technologies into a diagram.
It is about solving a problem step-by-step.

So let us build this like a story.

Step 1 — Understanding the Product

Before touching architecture, define the features.

For a LeetCode-like platform, the core requirements are:

• Authentication

• Problems / Problem List

• Submissions

• Leaderboard

• Discussions

Now let us expand what “Submissions” actually means.

A submission is not just code.

It contains:

• Problem statement reference

• User code

• Programming language

• Test cases

• Submission status

• Execution result

• Memory usage

• Runtime

• Timestamps

This becomes the heart of the entire system.

Everything else is secondary.

Step 2 — The Most Important Rule

Whenever a user submits code:

Save it immediately to the database.

Do not first execute the code.

Do not first validate it.

Do not first send it somewhere else.

Persist it first.

Why?

Because systems fail.

Workers crash. Containers die. Servers restart. Queues get delayed.

But if the submission is already stored safely in the database, you never lose user data.

That is why in the architecture:

• Client sends submission

• Primary server receives it

• Entry is created in PostgreSQL immediately

Only after that does further processing begin.

This is an extremely important system design principle.

Step 3 — Thinking About Scale

Now let us talk about the “fancy words”.

People hear terms like:

CAP Theorem, Horizontal Scaling, Throughput, Latency, Fault Tolerance, Availability

And immediately try to force them everywhere.

Do not do that.

Use concepts only where they actually matter.

For LeetCode, the biggest challenge is not authentication.

It is code execution.

Because executing user code is:

• CPU intensive

• Dangerous

• Slow

• Unpredictable

• Resource heavy

One user might submit:

while True: 
    pass

Another user may try:

rm -rf /

Another might try memory bombs.

So the entire architecture revolves around handling code execution safely and at scale.

Step 4 — High-Level Flow

Here is the overall flow of the system:

1. Client submits code

2. Primary server receives submission

3. Submission is stored in PostgreSQL

4. Job is pushed into AWS SQS

5. Worker picks job from queue

6. Worker sends code to execution engine

7. Code runs inside isolated runtime containers

8. Result gets stored back in database

9. Client polls for status updates

Simple.

Do not overcomplicate it.

Step 5 - Why We Use a Queue

The queue is one of the most important parts of the architecture.

In this design, AWS SQS acts as a buffer between:

• API servers

• Code execution system

Without queues, your primary server becomes overloaded very quickly.

Imagine:

• 10,000 users submitting code simultaneously

• Each execution taking several seconds

• Some submissions hanging forever

If the API server directly executes code:

• Requests block

• Server threads get exhausted

• Latency increases

• Entire application slows down

Instead:

• Save submission

• Push job into queue

• Return response immediately

This makes the system responsive.

The user gets:

{
    "submissionId": "123"
}

And later polls for results.

Step 6 - Polling vs WebSockets

In the architecture diagram, the client continuously asks:

GET /submission/:id

This is polling.

Why polling first?

Because it is simpler.

Many beginners immediately jump to WebSockets.

But system design is about choosing the simplest thing that works.

Polling works perfectly fine for:

• Hackathons

• MVPs

• Early-stage systems

Later, if needed, you can move to:

• WebSockets

• Server-Sent Events (SSE)

• Realtime updates

Start simple.

Step 7 — The Worker / Orchestrator

Now comes the interesting part.

The worker continuously reads jobs from SQS.

Its responsibilities are:

• Fetch submission data

• Determine programming language

• Send code to correct runtime

• Track execution status

• Handle retries

• Store final results

Think of the orchestrator as a traffic controller.

It does not execute code itself.

It delegates execution to isolated runtimes.

Step 8 — Code Execution Engine

This is the core of the system.

The execution engine contains isolated runtimes for:

• Python

• JavaScript

• Java

Each runtime runs inside containers.

Why containers?

Because user code cannot be trusted.

Containers provide isolation.

This prevents:

• File system access

• Network abuse

• Infinite resource consumption

• Attacks on host machine

Step 9 - Security Layer

This is where most online judge systems become difficult.

Running user code is dangerous.

That is why the architecture includes:

• Docker isolation

• No network calls

• Restricted system access

• CPU limits Memory limits

For example:

• Limit execution time to 2 seconds

• Limit memory to 256 MB

• Disable internet access

• Prevent root access

If a submission exceeds limits:

• Kill the container

• Mark submission as failed

Never trust user code.

Step 10 — Why Separate Runtimes?

Different languages behave differently.

Python execution is different from Java.

Java needs:

• Compilation step

• JVM startup

JavaScript behaves differently from compiled languages.

Separating runtimes gives:

• Better isolation

• Easier debugging

• Independent scaling

• Cleaner deployments

You can scale Python workers independently if Python traffic becomes high.

Step 11 — AWS Fargate for Scaling

The execution engine is deployed on AWS Fargate.

Why?

Because execution load is unpredictable.

Some days:

• 100 submissions

Some days:

• 1 million submissions

Fargate helps automatically:

• Add containers

• Remove containers

• Scale based on load

This prevents idle infrastructure costs.

You only pay for what you use.

Step 12 — Database Design Thoughts

PostgreSQL is enough initially.

You do not need Cassandra, DynamoDB, or distributed databases on day one.

Start simple.

Possible tables:

Users:

users
- id
- username
- email

Problems:

problems
- id
- title
- difficulty
- statement

Submissions:

submissions
- id
- user_id
- problem_id
- language
- code
- status
- runtime
- memory

This is enough to start.

Do not prematurely optimize.

Step 13 — Where CAP Theorem Actually Matters

Now we can finally mention CAP theorem properly.

In distributed systems, you usually balance between:

• Consistency

• Availability

• Partition Tolerance

For LeetCode:

• Availability matters

• Consistency matters for submissions

• Partition tolerance matters because systems are distributed

But honestly:

You do not need deep CAP discussions for an MVP.

Most beginners force these terms unnecessarily.

Mention them only when scaling becomes genuinely complex.

The biggest mistake beginners make in system design is trying to build Google on day one.

Good system design is iterative.

Start with:

• Simple API

• Simple database

• Queue

• Worker

• Isolated execution

Then improve step-by-step.

The architecture shown here already teaches several real-world concepts:

• Async processing

• Queue-based systems

• Distributed execution

• Container isolation

• Fault tolerance

• Horizontal scaling

And most importantly:

Always store the submission before processing it.

That single decision saves systems in production more than people realise.

You can find the complete diagram here:

Consider a simple example of a user creating an application which has the following steps:

1. User sign’s up

2. Insert User in DB

3. Send a Welcome Email to the User

4. Create Token

5. Return Response

Now this process is synchronous, meaning each step gets executed one after the other.

This is a perfectly working system for a single user. What if the number of users are now x1,00,000. Let us look at the services which are used here:

• Authentication APIs: These APIs are the ones which are built by the user and does not depend on any external dependency.

• Database: The database can be hosted by the user on their own server.

• Email Service: The email service will be a third party service. This is an external dependency.

Now, what if the email service goes down ? This would result in the signup process to go down as it is dependent on the email service. When 1,00,000 users would be constantly signing up on the service, this would result in the email service thinking that the “Users are SPAMMING” which would result in rate limiting. This is a typical problem which comes with a Synchronous Architecture.

Asynchronous Architecture

Asynchronous system design involves components or services interacting in a manner where the sender does not wait (or block) for the receiver’s response to continue processing. When designing an asynchronous architecture, you must be able do differentiate between which of the services can be synchronous and which can be asynchronous. In our example, the process remains the same till the signup.

When we come to the task of send mail, we add the sendWelcomeEmail on to the Queue. Remember, we are not sending the mail, just adding the task in the queue. Let us now get into the depth of understanding what is a Message Queue.

Message Queue

The Message Queue is a queue just like the queue we stand on when purchasing tickets for a movie where the messages are kept one after the other. It consists of 2 things: Producer & Consumer. Producer is the one who produces messages into the queue and Consumer is the one which consumes the messages produced by the producer.

Now what can be done here is that, we can have our server on post the queue where the Consumer task is to take the message, process it and then send the email. Since the email service (say we are using Amazon SQS) has a rate limiting of 30 emails / min, we can put the same condition on the server. This will now help the server to be available and the emails to be sent one after the other without any additional load on the server. We can even add a condition saying that once 60 minutes are crossed, server can take a break for a minute and then continue with the next set of emails.

This is called Asynchronous Processing.

Till now we completed with something known as (HLD) High Level Design. It is a high level overview of the approach.

Low Level Design of Queues Architecture

There are different types of queues here:

1. Push Based Mechanism:

1. In a push-based mechanism, the queue takes the responsibility of sending the payload to the processing functions. The consumer functions do not know what is inside the queue until the queue pushes a message to them. RabbitMQ is a classic example of a system that utilises this mechanism**.** This is how it works:

   1. Registeration: Consumer functions (workers) do not have direct access to pull from queue. Instead, they must interact with the queue’s API to register themselves, essentially telling the broker, “I am available for work”.

   2. Broker Management: The Broker stores this registration in an internal database, mapping worker IDs, and then decides which worker receives which message. It handles balancing across multiple workers.

   3. Heartbeats: To ensure workers are still active, the broker requires a continuous “heartbeat”. For example, workers might send an “I am alive” signal every 30 seconds. If a worker fails to send a heartbeat within a specific timeframe, the broker marks it as dead and stops sending it messages.

2. Pull Based Mechanism:

1. In a pull-based mechanism, the broker is more passive. There is no central broker managing worker registrations; instead, the control is entirely in the hands of the consumer. AWS SQS (Simple Queue Service) and BullMQQ are prominent examples of pull-based systems.

   How it Works

   1. Polling: The workers must actively ask the queue for data. This is typically done using polling mechanisms, such as setting a setInterval or running a continuous while(true) loop to call the queue’s “pull” API.

   2. Manual Processing: The worker pulls a batch of messages, processes them, and then is responsible for explicitly discarding them or putting them back into the queue if they fail.

The Myth of Strict FIFO

A massive misconception in system design is that queues always guarantee a strict First-In-First-Out (FIFO) sequence. While pushing into a queue is sequential, the processing is often not.

When you introduce parallel processors (multiple workers with different processing speeds), the order breaks down. For instance, Worker A might pick up Message 1, but a faster Worker B might pick up Message 2 and finish processing it before Worker A is done. This can lead to illogical outcomes, like sending a user a product upgrade prompt before sending their welcome email.

The Solution: To solve this sequence issue in parallel environments, tools like Apache Kafka introduce the concept of partitions and keys. Kafka assigns messages with the same key (e.g., the same User ID) to the same specific worker within a consumer group. This ensures parallelism across different users while maintaining strict, sequential processing for individual user journeys.

When it comes to System Design, everything is a trade-off. Neither Push nor Pull is universally “better” or “worse”.

• Choose Push-based (RabbitMQ) if you want a simple architecture with built-in deduplication and routing.

• Choose Pull-based (AWS SQS) if you need granular control over the consumption rate and don’t mind the engineering overhead of building back-off and lock mechanisms.

• You can even mix both in the same system! For example, using a push queue for immediate notifications and a pull queue for heavy background tasks like video processing.

Understanding these LLD concepts is crucial for mastering asynchronous patterns and succeeding in advanced System Design interviews.

Why Basic TTL caching Fails?

Basic TTL (Time to Live) is like a ticking time bomb. If you set 1 million keys to expire in exactly 3600 seconds, you are essentially scheduling a system crash for 1 hour from now. Imagine a “Flash Sale” on an e-commerce site:

1. You cache the “Product Price” for 60 seconds.

2. 50,000 people are looking at that price.

3. At the 60th second, the cache disappears.

4. All 50,000 users see a “Cache Miss” and hit the Database at the exact same millisecond.

The Roadmap to a Bulletproof Cache:

Step 1: Advanced Expiry (Jitter & PER)

TTL Jitter: The Power of Randomness

Instead of TTL = 60s , we use TTL = 60s + random(0,10s). Now, some user’s cache expire at 60s, some at 63s, some at 70s. This turns a “Thundering Herd” into a “Predictable Stream”. It’s like a staggered release times for a crowded theatre.

Probabilistic Early Re-computation (PER)

This sounds fancy, but the logic is simple: The close we get to the expiration time, the higher the “chance” that a request will decide to refresh the cache early.

• At 10 seconds left, there is a 1% chance a request will refresh it.

• At 1 second left, there’s an 80% chance.

This ensures that one lucky user refreshes the data before it every actually hits zero and disappears

Step 2: The “Wait or Serve Old” Strategies

1. Mutex / Cache Locking (The “Old Producer” Rule)

Imagine the cache for “Current Score” expires, 100 requests arrive.

• Without a lock: All 100 go to the DB.

• With a Mutex: The request “locks” the key. It says, “I’m, going to the DB to get the new score. Everyone else, stay here and wait”.

• The other 99 requests, “wait” for a few milliseconds, until the first one returns and updates the cache.

2. Stale-While-Revalidate (SWR)

This is the “Netflix/CDN favourite.” Instead of making users wait for the new data, you give them the old (stale) data one last time while you fetch the new data in the background.

• Request arrives: “Is the data expired?”

• System: “Yes, it expired 2 seconds ago. But here is the old version so you don’t have to wait. I’ll go grab the new one now for the next person.”

• Result: Zero latency for the user!

Step 3: Proactive Measures (”The Chef’s Preparation”)

Cache Warming / Pre-Warming

Think of an IPL final or a Netflix release. You know a million people are going to ask for the “Match Summary” or “Stranger Things Episode 1” at exactly 8:00 PM. Don’t wait for them to ask! Your system should “warm up” the cache by fetching that data and putting it in Redis at 7:55 PM. By the time the crowd arrives, the “oven” is already hot.

The Big Tradeoffs: Freshness vs. Latency

In engineering, there is no free lunch.

• SWR: Amazing speed (low latency), but some users see “stale” (old) data for a few seconds.

• Mutex: Perfect data (high consistency), but some users have to “wait” (higher latency) while the lock is held.

Choosing Your Weapon

• Use Jitter ALWAYS: It’s the “low-hanging fruit.” There is almost no reason not to add a little randomness to your expiries to prevent synchronized spikes.

• Use SWR for UX: If your app feels “snappy,” users stay longer. If a user sees a post from 10 seconds ago instead of 1 second ago, they usually won’t notice. This is why CDNs (like Cloudflare) love SWR.

• Use Cache Warming for “The Big Event”: If you know 10 million people are coming at 8:00 PM, waiting for the first user to “trigger” the cache is a mistake. You want that data waiting for them.

That’s the Thundering Herd problem: a massive, synchronized surge of requests that crashes a system because everyone tried to access the same resource at the exact same moment.

What is the Thundering Herd Problem ?

In technical terms, it happens when a large number of processes or clients are waiting for an event (like a lock being released, or a cache being updated). When that event finally happens, all of them wake up and try to handle it simultaneously.

Instead of one person “fixing” the problem, everyone tries to fix it, which ends up breaking the system entirely.

Think of a standard web app setup:

1. The Goldilocks Zone: Usually, the App asks the Cache for data. The Cache has it, and the DB stays “chilling” and relaxed.

2. The Expiry: The data in the Cache has a “Time to Live (TTL)”. Eventually, it expires.

3. The Stampede: If 10,000 users ask for that data, the millisecond after it expires, the Cache says “I don’t have it!” to all 10,000 users. All 10,000 requests then “thunder” straight to your database to fetch the data.

Normal Spikes v/s Thundering Herds

Before moving forward with the actual system design, let us first go with the fundamentals of understanding the difference between normal spikes and thundering herds.

A Normal Spike is like a gold rush hour. You know more people are coming because it’s 5:00 PM. You can prepare by adding more lanes.

A Thundering Herd is a “system-induced” spike. It’s often caused by your owned code!

For example:

• The IPL Example: A million people hit “Refresh” at the exact moment a match starts.

• The Cache Expiry: You set your cache to expire every 10 minutes. At exactly 10:01, the system itself creates a massive hole that everyone falls into

In distributed systems, if one database gets crushed, or goes down, the “HERD” doesn’t go away - it just moves to the next available server, knocking it down too. This is called Cascading Failure.

Quick Scenario Check: Imagine you run a site for concert tickets.

• Scenario A: Users slowly trickle in over an hour.

• Scenario B: You send a “Buy Now” push notification to 1 million users at once, and your cache for the “Ticket Price” expires at that exact second.

Consider another example:

Imagine you are the Lead Engineer for Netflix. A massive new season of Stranger Things is about to drop at exactly midnight. You have two situations:

1. Situation A: You have a “Coming Soon” page that updates every minute. At 11:59:59, the cache for that page expires for everyone at the same time.

2. Situation B: People are already on the site, browsing different categories (Horror, Comedy, Sci-Fi) throughout the evening.

Which will be a Thundering Herd Situation ? Obviously the situation A. The “event” that triggered the stampede was the Cache Expiry (TTL hitting zero) at 11:59:59. Because every single user was looking at the same page, and that page disappeared from the cache for everyone at once, they all rushed the database together.

The System Heart Attack

When that herd hits, it’s not just “slow” - it can be fatal for your infrastructure. Here’s what happens under the hood:

• Database (DB): Imagine a librarian who can handle 10 people at a time. Suddenly, 10,000 people scream the same question at her. She trips, her glasses break, and she stops answering everyone. Similarly, the DB connection pool fills up, and it stops responding.

• CPU: Your servers are sweating. They spend all their energy just trying to manage the “waiting line” (context switching) rather than actually fetching data.

• Cache: Ironically, the cache stays empty as the DB is too busy to send it the update!

• Latency: This is the “spinning wheel of death” for users. Requests that usually take 50ms now take 30 seconds... or just time out.

Why it’s Dangerous in Distributed Systems

In a big system like Netflix, services depend on each other. If the “Movie Service”, dies due Thundering Herd, the “Recommendation Service” might start failing too because it’s waiting for the Movie Service. This is a Cascading Failure - one fallen domino knocks over the whole data centre.

The Fixes (How to Tame the Herd)

There are 3 main “weapons” we use here:

Request Coalescing (The “Group Order”)

Imagine 100 people want a coffee. Instead of the barista making 100 individual trips to the kitchen for beans, the first person says, “Wait! Everyone wants the same thing, I’ll go get the beans, and rest of you just wait here. “

The system only sends one request to the database. When the data comes back, it shares with all waiting user.

Cache Locking / Mutex

When the cache expires, the first request to arrive “grabs a lock”. It tells the system, “I am currently updating the cache! Nobody else go to the Database.” Other requests see the lock and either wait or server slightly “stale” (old) data for a few more seconds.

Staggered Expiry (Adding “Jitter”)

This is a clever one. Instead of setting every cache to expire in exactly 600 seconds, you add a bit of randomness:

• User A’s cache expires in 600s.

• User B’s cache expires in 605s.

• User C’s cache expires in 597s.

This spreads the “thunder” out over several seconds so it sounds more like a gentle rain.

Exponential Backoff

If a request fails (because the herd already knocked the server down), don’t let the client try again immediately! If 1,000,000 apps all retry at the exact same second, they just knock the server down again.

The Fix: Each time a request fails, the client waits a bit longer before trying again (1s, then 2s, then 4s, then 8s...). It gives the “Librarian” time to stand back up.

In system design, we can’t always stop the crowd from showing up, but by using tools like jitter and coalescing, we make sure our servers aren’t the ones getting trampled. Remember: a healthy system doesn’t just handle traffic; it survives the thunder.

Understanding how UPI works requires first looking back at the traditional infrastructure it replaced.

Part 1: The Traditional Approach to Payments in India

Before UPI, the Indian payment system was heavily bank-centric and relied on multiple layers of detail and processing time.

The Problem: Complexity and Latency

In the traditional system, every transaction was rooted in the bank accounts of the sender and receiver. Users needed to possess a bank account with any regulated financial institution (like HDFC, ICICI, SBI, HSBC, etc.) to participate.

The key regulatory body overseeing all these banks is the RBI (Reserve Bank of India), as illustrated in the flow below:

To make a payment, the sender required a multitude of unique details from the receiver:

• Account Number

• Bank Name

• Branch Code

• IFSC Code (Indian Financial System Code)

The Old Transfer Methods

Depending on the transaction amount and urgency, different services were used, often introducing delays:

IMPS - Immediate Payment Service

Small Transaction Range like ₹1,000 - ₹2,000 with Real-time processing

NEFT - National Electronic Funds Transfer

Medium Transaction Range like 50,000 - ₹1,00,000 with Batch Processing (Can take hours)

RTGS - Real-time Gross Settlement

High Transaction Range like 5,00,000 and above with Real-time processing

The major drawback, particularly with NEFT, was the latency. Users often had to take screenshots of their payment confirmations to prove a payment was initiated, even if the amount hadn’t yet reflected in the receiver’s account. The system lacked true, consistent, real-time settlement across all tiers.

Part 2: How UPI Revolutionized Payments

The Unified Payments Interface (UPI) was designed to solve the complexity and latency issues of the traditional model. Developed by the NPCI (National Payments Corporation of India), UPI is an instant, real-time payment system that acts as a simplifying layer above the existing bank infrastructure.

The core idea was to say: “Let’s keep the banks for holding the money, but simplify the process of moving it.”

The Key Players in the UPI System Design

1. NPCI (National Payments Corporation of India)

NPCI acts as the Payment Infrastructure and the central switch. It is the core hub that connects all participating banks. Critically, the APIs provided by NPCI are not public—only a list of trusted, regulated banks (known as Partner Banks) can communicate with the NPCI network.

2. PSP (Payment Service Provider)

Since the public cannot directly communicate with NPCI, the PSP steps in. A PSP is a third-party entity that provides the user-facing application (the front-end) to facilitate electronic payment transactions.

• Customer PSPs: GooglePay, PhonePe, Paytm, etc.

• The PSPs have direct tie-ups with the Partner Banks to route the transaction requests.

3. VPA (Virtual Payment Address)

This is the game-changer. The Virtual Payment Address is a unique identifier that abstracts (hides) all the complex bank details (Account No., IFSC, etc.) of the user. It works like an email address for money transfers, typically in the format: username@handle (e.g., aditya@hdfc, anil@icici).

Part 3: The Step-by-Step UPI Transaction Flow

The system design of a UPI transaction uses the PSP and VPA to facilitate a rapid request-response cycle coordinated by NPCI.

UPI Transaction Flow Diagram

Let’s walk through the transaction: Aditya (@hdfc) wants to pay Anil (@icici) ₹5,000.

Step 1: User Initiates Payment via PSP
Aditya opens a Customer PSP App (e.g., GooglePay) and scans Anil’s QR code, which contains Anil’s VPA (anil@icici). The user interface creates an Intent for the transaction:

• From: aditya@hdfc

• To: anil@icici

• Amount: ₹5,000

Step 2: PSP Sends Request to Sender’s Bank
The PSP App does not talk to NPCI directly. Instead, it communicates the payment Intent to the Sender’s Bank (HDFC), with which the PSP has a tie-up.

Step 3: Sender’s Bank Communicates with NPCI
HDFC Bank receives the request and forwards it to the central NPCI network, asking for permission to debit Aditya’s account.

Step 4: NPCI Coordinates Debit and Credit

• NPCI first verifies the existence of the VPAs and checks the balance in the sender’s account. This is the stage where the user is prompted to enter their secure UPI PIN.

• Upon successful PIN verification, NPCI instructs the Sender’s Bank (HDFC) to debit the amount from Aditya’s account.

• Simultaneously, NPCI sends a Credit Request to the Receiver’s Bank (ICICI), asking it to credit the amount to Anil’s account associated with the VPA.

  • Note: If the receiver’s bank server is temporarily overloaded or busy, the transaction may fail at this point, resulting in a “Receiver’s server is busy” notification.

Step 5: Transaction Completion and Notification
Once the Receiver’s Bank confirms the credit, NPCI confirms the transaction. A success notification is instantly relayed back to both the sender and the receiver via their respective PSP apps.

Conclusion: The Power of Abstraction

UPI’s brilliance lies in its system design, which successfully abstracts away the complexity of traditional banking details using the VPA and uses NPCI as a highly efficient, real-time central switch.

By leveraging a trusted third-party layer (PSPs) and a secure, central network (NPCI), the UPI system turned a multi-step, multi-hour process into a single, instant interaction, solidifying its place as one of the world’s most successful real-time payment systems.