Concurrency & Race Conditions in Production: From Mutexes to Database Concurrency

Concurrency bugs are some of the hardest bugs to reason about in production.

The code often looks completely reasonable:

if stock > 0 {    stock--}

Yet two requests arriving at the same time can produce a result that neither request intended.

The usual reaction is:

“Let’s add a mutex.”

Sometimes that’s exactly right.

Sometimes it’s completely wrong.

A mutex may protect process memory while the actual shared state lives in PostgreSQL. A database transaction may make multiple operations atomic while still leaving a concurrency bug in the business logic. And once external systems enter the picture, neither a mutex nor a database transaction can give you the guarantees you need.

The key is to stop asking:

“What synchronization primitive should I use?”

and start asking:

“What shared state and invariant am I trying to protect, and where does that state live?”

This article builds that mental model from the ground up.

1. Concurrency Is Not the Same as a Race Condition

First, separate three concepts.

Concurrency

Multiple executions can make progress during overlapping periods.

Imagine two requests:

Request A: A1 → A2 → A3Request B: B1 → B2 → B3

A concurrent scheduler might execute:

A1B1A2B2B3A3

The exact order is not guaranteed.

Parallelism

Parallelism means work is literally executing at the same time, for example on different CPU cores:

Core 1 → Request ACore 2 → Request B

Concurrency does not require parallelism.

Race condition

A race condition occurs when correctness depends on the ordering of concurrent operations.

For example:

Execution 1 → correct resultExecution 2 → incorrect result

with the same logical input.

A data race is a more specific case involving unsynchronized concurrent access to the same memory location, where at least one access is a write.

Not every concurrency bug is a data race.

A database race between two HTTP requests, for example, doesn’t necessarily involve two goroutines directly accessing the same Go variable.

2. The First Question: Is There Shared Mutable State?

Concurrency by itself isn’t a problem.

This is usually fine:

go calculateReport(userA)go calculateReport(userB)

If each goroutine operates on independent state, there may be nothing to synchronize.

The problem starts when multiple executions interact with shared mutable state.

For example:

counter := 0

go func() {    counter++}()go func() {    counter++}()

We have:

shared   → countermutable  → counter changesmultiple concurrent executions

Now we have a potential race.

3. Why counter++ Isn't Atomic

This:

counter++

looks like one operation, but conceptually it is closer to:

READ counterADD 1WRITE counter

Suppose:

counter = 0

Two goroutines can interleave like this:

G1: READ  → 0G2: READ  → 0

G1: ADD   → 1G2: ADD   → 1G1: WRITE → 1G2: WRITE → 1

Final value:

counter = 1

Expected:

counter = 2

This is a lost update.

The important lesson is:

A multi-step operation is not automatically atomic just because it is written on one line.

4. Invariants: What Must Never Become False?

Before choosing a synchronization mechanism, define the invariant.

An invariant is a condition that must remain true for the system to be correct.

For an inventory system:

stock >= 0

For a bank account:

balance >= 0

For a booking system:

successful bookings <= available capacity

For an idempotent payment API:

one logical payment request → one payment effect

This is one of the most important habits in concurrency engineering.

Don’t start with:

“Should I use a mutex?”

Start with:

“What invariant can concurrent execution violate?”

5. Critical Sections: It’s Often More Than One Line

Consider:

if balance >= amount {    balance -= amount}

The critical section isn’t necessarily just:

balance -= amount

The business operation is:

CHECK  ↓ACT

The check and the action are logically coupled.

If two executions can both pass the check before either performs the action, you have a race.

This pattern appears everywhere:

if stock > 0 → decrementif balance >= amount → withdrawif seat available → reserveif username available → createif job not claimed → process

These are all variations of check-then-act.

6. Atomic Operations

When the state transition itself is simple enough, an atomic operation can be the right tool.

For example:

var requestCount atomic.Int64

requestCount.Add(1)

Instead of:

requestCount++

the increment is performed as an atomic operation.

When are atomics appropriate?

They are a good fit when:

• the state is small and simple,
• the operation is well-defined,
• you don’t need to protect a larger multi-step critical section.

Typical examples:

countersflagssequence numbersreference countssimple state transitions

The important limitation

Atomicity of one operation does not make an entire business operation atomic.

Imagine:

if stock > 0 {    atomic decrement(stock)}

The decrement might be atomic, but the check + decrement still needs correct concurrency semantics.

So:

Atomic does not mean “my whole business operation is safe.”

It means a particular operation is indivisible with respect to the relevant concurrency mechanism.

7. Mutexes

A mutex is useful when multiple goroutines need to operate on shared mutable state in process memory.

For example:

type Cache struct {    mu    sync.Mutex    items map[string]Item}

A critical section might be:

mu.Lock()defer mu.Unlock()

if balance >= amount {    balance -= amount}

The idea is:

G1 → LOCK → critical section → UNLOCKG2 → WAIT ----------------------→ LOCK → critical section

Only one goroutine enters the protected region at a time.

Why use a mutex?

Because the operation you’re protecting may contain multiple steps:

readcheckmodifywrite

and all of them need to behave as one critical section.

But mutexes have a boundary

A Go mutex protects memory within the process.

Consider:

Load Balancer                /     |     \               /      |      \           Server A Server B Server C               \      |      /                \     |     /                    DB

If Server A has:

mu.Lock()

that lock does not stop Server B.

So if the shared state is in PostgreSQL:

mutex A ≠ database lock

This is a critical production distinction.

8. Channels and Ownership

Locks aren’t the only way to deal with shared state.

Another approach is to give the state a single owner and communicate with it through messages.

For example:

Worker A ──┐Worker B ──┼──→ channel ──→ state ownerWorker C ──┘

Instead of allowing every goroutine to mutate the state directly, one goroutine owns it.

This can make certain systems easier to reason about because the invariant is maintained by a single owner.

When does this make sense?

• The state has a natural owner.
• Work is naturally message-oriented.
• You want to serialize mutations rather than protect them with locks.

But channels are not automatically better than mutexes.

For a simple shared map, a mutex may be much clearer.

9. Now Move to the Database

Here’s where concurrency gets more interesting.

Imagine a web application with multiple instances:

Load Balancer              /    |    \             /     |     \          App A  App B  App C             \     |     /              \    |    /                  DB

Suppose inventory lives in the database:

products

id | stock---+------42 | 1

A process-local mutex cannot protect this state across all application instances.

Now we need database concurrency control.

10. The Naive Inventory Race

A developer might write:

SELECT stockFROM productsWHERE id = 42;

Then in application code:

if stock > 0 {    // decrement}

Then:

UPDATE productsSET stock = stock - 1WHERE id = 42;

Looks reasonable.

But now two requests arrive:

A: SELECT → 1B: SELECT → 1

Both requests believe inventory is available.

Then:

A: UPDATEB: UPDATE

The check happened separately from the state transition.

This is the database version of check-then-act.

11. Conditional UPDATE: Push the Invariant Into the Database

Instead, we can write:

UPDATE productsSET stock = stock - 1WHERE id = 42  AND stock > 0;

Now the condition and the state transition are part of the same SQL statement.

Conceptually:

CHECK stock > 0        +DECREMENT stock        ↓one database statement

With:

stock = 1

and two concurrent buyers:

A → UPDATE succeeds → stock = 0B → waits / re-evaluates → stock > 0 is false

The application can inspect the number of affected rows:

RowsAffected = 1    → purchase succeeded

RowsAffected = 0    → purchase could not decrement inventory

This is a powerful pattern.

You don’t need an application-level mutex to protect the database row.

You don’t necessarily need SELECT ... FOR UPDATE.

The database is enforcing the state transition.

12. But Atomic SQL Is Not the Same as a Transaction

Now suppose buying a product means:

1. decrement stock2. create order

We need both operations to succeed or both to fail.

Without a transaction:

UPDATE stock       → successINSERT order       → failure

We could end up with:

stock = 0order = missing

That’s a business consistency bug.

So we use:

BEGIN;UPDATE productsSET stock = stock - 1WHERE id = 42  AND stock > 0;INSERT INTO orders (...);COMMIT;

If something fails:

ROLLBACK;

Now:

UPDATE stock+INSERT order

form one atomic database transaction.

What does a transaction actually guarantee?

A transaction gives us an atomic commit/rollback boundary:

These database changes are treated as one unit.

It does not mean:

Nobody else can concurrently access these rows.

That’s where isolation and locking come in.

13. Transaction ≠ Isolation ≠ Lock

These concepts are often mixed together.

Transaction

“Commit or roll back these database operations as one unit.”

Isolation

“What can concurrent transactions see, and how do their executions interact?”

Lock

“How are conflicting accesses to a database resource coordinated?”

They work together, but they are different concepts.

A transaction can use locks.

An isolation level can influence visibility and concurrency behavior.

But:

transaction ≠ locktransaction ≠ isolation

14. MVCC: Why Doesn’t B See A’s Uncommitted Change?

PostgreSQL uses MVCC — Multi-Version Concurrency Control.

Suppose:

stock = 1

Transaction A:

BEGIN;UPDATE productsSET stock = 0WHERE id = 42;-- not committed yet

Then transaction B:

BEGIN;SELECT stockFROM productsWHERE id = 42;

Under PostgreSQL’s default READ COMMITTED isolation, B does not see A's uncommitted update.

B sees:

stock = 1

Conceptually:

Committed versionstock = 1       ↑       │       └── visible to BA's uncommitted versionstock = 0       ↑       │       └── not yet visible to B

This is one reason database concurrency is different from ordinary in-memory concurrency.

15. But Writes Behave Differently From Reads

Here’s the subtle part.

Suppose:

stock = 1

A:

UPDATE productsSET stock = stock - 1WHERE id = 42;

A is modifying the row.

B can still perform:

SELECT stock ...

and see the appropriate visible version.

But if B also tries:

UPDATE productsSET stock = stock - 1WHERE id = 42;

B is now trying to perform a conflicting write.

The database must coordinate the writes.

Conceptually:

A                         BUPDATE🔒 row 42                          UPDATE                          ⏳ WAITCOMMIT🔓                          continue

This is where row-level locking becomes important.

16. SELECT ... FOR UPDATE

Suppose your application needs to:

1. read the row,
2. inspect several fields,
3. make a business decision,
4. update the row.

A conditional UPDATE may no longer be expressive enough.

You can use:

SELECT stockFROM productsWHERE id = 42FOR UPDATE;

This means, roughly:

“Read this row and acquire a lock suitable for an upcoming update.”

Now:

Transaction A              Transaction BSELECT FOR UPDATE🔒 row 42                           SELECT FOR UPDATE                           ⏳ WAITCHECKUPDATEINSERT ORDERCOMMIT🔓                           continue                           read current state

This is a form of pessimistic concurrency control.

You assume a conflict may happen, so you lock the resource before making the decision.

17. Conditional UPDATE vs. FOR UPDATE

These solve related but different problems.

Conditional UPDATE

UPDATE productsSET stock = stock - 1WHERE id = 42  AND stock > 0;

Think:

“If this invariant currently holds, perform this state transition.”

Excellent for simple state transitions.

SELECT ... FOR UPDATE

SELECT ...FROM ...WHERE ...FOR UPDATE;

Think:

“I need to inspect this row, make a more complex decision, and then modify it without another transaction changing it underneath me.”

Excellent for more complex check-then-act logic.

18. Isolation Levels

Isolation determines how concurrent transactions interact from the application’s perspective.

Common levels include:

READ COMMITTEDREPEATABLE READSERIALIZABLE

READ COMMITTED

PostgreSQL’s default.

A useful mental model:

Each statement sees an appropriate committed snapshot.

Therefore, don’t assume:

“I saw value X once, so every later SELECT in this transaction must see X.”

That’s not how READ COMMITTED works.

REPEATABLE READ

Provides a stronger transaction-level snapshot model.

A transaction gets a more stable view of data, but conflicts can result in transaction failures that the application may need to retry.

SERIALIZABLE

Provides the strongest general isolation guarantee.

The database attempts to make the result equivalent to some serial ordering of transactions.

The cost can include:

• more contention,
• serialization failures,
• retries,
• lower concurrency in some workloads.

The highest isolation level isn’t automatically the best choice.

Isolation should be selected based on the correctness requirements and workload.

19. Optimistic Concurrency Control

Pessimistic locking says:

“Lock first because a conflict might happen.”

Optimistic concurrency says:

“Assume conflicts are uncommon. Detect them when writing.”

Suppose:

productstock   = 10version = 7

Application reads:

version = 7

Then updates:

UPDATE productsSET stock = stock - 1,    version = version + 1WHERE id = 42  AND version = 7;

If:

RowsAffected = 1

the update succeeded.

If:

RowsAffected = 0

another transaction changed the row first.

The application can then retry or report a conflict.

This is commonly called optimistic concurrency control.

It works especially well when conflicts are relatively rare and retrying is acceptable.

20. Database Constraints: Let the Database Enforce the Invariant

Some invariants don’t need application-level locking at all.

Suppose:

A user can only create one registration for an event.

Instead of:

SELECT existing registrationif not exists:    INSERT

which has a race, enforce it with a unique constraint.

Conceptually:

UNIQUE(user_id, event_id)

Now:

App A → INSERT → successApp B → INSERT → constraint violation

The database itself guarantees the invariant.

This is a powerful production principle:

If an invariant can be enforced directly by the database, consider enforcing it there.

Typical tools include:

UNIQUECHECKPRIMARY KEYFOREIGN KEY

21. A More Complete Production Example: Withdraw Money

Suppose:

balance = 100

and two concurrent requests both try to withdraw 100.

A simple and strong approach is:

UPDATE accountsSET balance = balance - 100WHERE id = 1  AND balance >= 100;

With two concurrent requests:

A → succeeds → balance = 0B → waits / re-evaluates → balance >= 100 is false

Therefore:

RowsAffected = 1 → one successful withdrawalRowsAffected = 0 → failed withdrawal

Notice what we did not need:

application mutexSELECT firstSELECT FOR UPDATE

for this particular state transition.

But if a successful withdrawal also requires:

1. update balance2. insert ledger record

then we likely want:

BEGIN;UPDATE accountsSET balance = balance - 100WHERE id = 1  AND balance >= 100;-- check affected rowsINSERT INTO ledger (...);COMMIT;

Now two concerns are handled independently:

Conditional UPDATE→ concurrency-safe balance transitionTransaction→ atomic balance + ledger update

This distinction is fundamental.

22. Where Database Transactions Stop

Now the problem becomes much more interesting.

Suppose a payment operation does:

1. update database2. call payment provider3. send email4. publish event

A database transaction can cover:

database

It cannot magically make this atomic:

PostgreSQL+Stripe/payment provider+email service+Kafka

There is no ordinary local transaction spanning all of them.

This is where distributed systems patterns become necessary.

23. Idempotency

Consider:

Client   ↓POST /payments   ↓Server   ↓Payment provider   ↓SUCCESS

But the response gets lost:

Payment succeeded       ↓Response lost       ↓Client timeout

The client retries.

Now:

POST /paymentsPOST /payments

might produce two payments.

The solution is often an idempotency key:

Idempotency-Key: abc-123

The server records the logical operation.

First request:

abc-123 → create payment

Retry:

abc-123 → return existing result

The key idea:

Repeating the same logical operation should not produce another unintended side effect.

Idempotency is essential because retries are normal in distributed systems.

24. Message Queues

Suppose you need to send an email after an order is created.

Instead of doing everything synchronously:

HTTP request   ↓DB   ↓Email provider

you can separate the work:

DB transaction    ↓event    ↓Queue    ↓Worker    ↓Email provider

The worker can retry failed delivery.

Queues can also provide:

• decoupling,
• buffering,
• backpressure,
• asynchronous processing,
• retry mechanisms.

But queues introduce their own concurrency concerns, especially duplicate delivery.

That’s why idempotent consumers matter.

25. Transactional Outbox

There is a classic failure mode when writing to a database and publishing an event.

Suppose:

BEGINUPDATE orderINSERT eventCOMMITpublish event

The database commit succeeds.

Then:

publish event 💥

Now the database says the order changed, but the event never reached the message broker.

The reverse order isn’t safe either:

publish eventDB COMMIT💥

The event exists but the database change doesn’t.

The transactional outbox pattern

Store the event in the same database transaction:

BEGIN;UPDATE ordersSET status = 'PAID';INSERT INTO outbox_events (...);COMMIT;

Now:

order state+event intent

are committed atomically.

A separate publisher reads the outbox:

outbox table      ↓publisher      ↓message broker      ↓consumer

If publishing fails, the publisher can retry.

This pattern connects database atomicity with asynchronous distributed processing.

26. Exactly-Once Is Usually an Effect, Not a Delivery Guarantee

Distributed systems often operate with:

at-least-once delivery

For example:

message deliveredmessage processedACK lostmessage delivered again

The consumer may receive the same event twice.

Rather than trying to make the entire transport literally exactly-once, a common strategy is:

at-least-once delivery+idempotent consumer

For example:

event_id = abc

Consumer checks whether abc has already been processed.

First delivery:

abc → process

Duplicate delivery:

abc → already processed → ignore/replay result

The system can then provide an exactly-once effect even though the underlying delivery is at-least-once.

27. The Production Hierarchy

At this point, the progression should look like this:

CONCURRENCY                         │                         ▼              Shared mutable state?                  /            \                No              Yes                                │                                ▼                         Where is the state?                       /        |          \                      /         |           \                 Memory      Database    Distributed                   │             │            │                   ▼             ▼            ▼                Atomic      Atomic SQL    Idempotency                   │        / Transaction      │                Mutex       /       \          Queue                   │       /         \          │              Ownership   Lock     Isolation   Outbox                   │       │           │         │                   └───────┴───────────┴─────────┘

The actual decision process is more useful than memorizing the tree.

28. A Practical Decision Framework

When you encounter a concurrency problem, walk through these questions.

1. What is the shared state?

Examples:

countercachestockbalanceseatorderpayment

2. Where does it live?

process memorydatabasemultiple servicesexternal provider

3. What is the invariant?

Examples:

stock >= 0balance >= 0one booking per seatone payment per idempotency key

4. What concurrent interleaving breaks it?

Write the execution explicitly:

A1B1A2B2

If you cannot demonstrate the bad interleaving, you probably don’t fully understand the bug yet.

5. What is the atomic business operation?

Maybe:

increment counter

Or:

check stock + decrement

Or:

decrement stock + create order

Or:

create payment + publish event

The boundary determines the mechanism.

6. What is the smallest correct primitive?

Use the least powerful mechanism that correctly protects the invariant:

simple memory transition    → atomic

compound in-memory state    → mutex / ownershipsingle DB state transition    → conditional UPDATEmultiple DB operations    → transactioncomplex DB check-then-act    → row lock / SELECT FOR UPDATElow-conflict versioned state    → optimistic concurrencydatabase invariant    → constraintDB + external systems    → idempotency / outbox / messaging

29. The Most Important Mental Model

Don’t think:

“Concurrency means everything needs a mutex.”

And don’t think:

“If I use transactions, I’m safe.”

Instead:

Concurrent execution        ↓Shared state        ↓Invariant        ↓Possible bad interleaving        ↓Atomic business boundary        ↓Correct synchronization mechanism

The goal isn’t to eliminate concurrency.

The goal is to make all unacceptable concurrent outcomes impossible or safely recoverable.

30. Cheat Sheet

Problem

Typical solution

Why

counter++

Atomic

One simple state transition

Shared in-memory map

Mutex

Protect compound operations

Single state owner

Channel / ownership

Serialize mutations through one owner

balance >= 100 → decrement

Conditional UPDATE

Keep condition and transition together

UPDATE + INSERT

Transaction

Atomic DB business operation

Read → complex decision → write

SELECT FOR UPDATE

Pessimistic row locking

Low-conflict versioned updates

Optimistic concurrency

Detect conflicts without holding locks

Unique business identity

DB constraint

Enforce invariant at the database boundary

Retryable API

Idempotency key

Prevent duplicate side effects

DB + event publishing

Transactional outbox

Atomically persist state and event intent

Async processing

Queue

Decouple work and enable retries

Duplicate message delivery

Idempotent consumer

Make retries safe

31. Final Takeaway

Concurrency engineering is fundamentally about invariants and boundaries.

A mutex isn’t “the concurrency solution.”

A transaction isn’t “the concurrency solution.”

A lock isn’t “the concurrency solution.”

They are tools for different concurrency domains.

The right question is always:

What state am I protecting, what invariant must remain true, and where is the atomic boundary of this business operation?

Once you can answer those three questions, the technology choice becomes much easier:

Process memory    → atomic / mutex / ownershipDatabase    → atomic SQL / transaction / locks / isolation / constraintsDistributed system    → idempotency / queues / outbox / retries

And that’s the real progression from understanding counter++ races to designing production-grade concurrent systems.