4 Concurrency Mistakes That Crash Go Apps and How to Fix Them

Why Concurrency Bugs Are the #1 Cause of Go Production Crashes

Go's concurrency model is elegant, but it's deceptively easy to introduce bugs that only manifest under load. In my years of reviewing production incidents, I've found that four patterns account for the majority of crashes: data races, goroutine leaks, deadlocks, and channel misuse. These aren't academic problems—they cause real outages, data corruption, and hard-to-reproduce failures. The challenge is that Go's race detector can catch some issues, but it's not a silver bullet. Many concurrency bugs are timing-dependent and may not trigger during testing.

Consider a typical web service handling thousands of requests per second. A single unprotected map access can cause a panic that brings down the entire process. Or a forgotten goroutine that blocks forever, slowly leaking memory until the OOM killer intervenes. These scenarios are common, and they're often introduced by well-intentioned developers who are new to Go's concurrency primitives.

The Cost of Concurrency Mistakes

In production, a concurrency-related crash can mean minutes of downtime, lost revenue, and damaged reputation. For example, a trading platform I worked with experienced intermittent crashes due to a data race on a shared order book. The race only occurred during high-frequency trading windows, making it nearly impossible to reproduce in staging. The fix—adding a simple mutex—took minutes, but the debugging process took weeks. This pattern repeats across industries: the root cause is often a small oversight that compounds under load.

Why Go's Built-in Race Detector Isn't Enough

Go's -race flag is a powerful tool, but it only detects races that occur during execution. If a race condition requires a specific interleaving of goroutines, it may not trigger in your test environment. Moreover, the race detector adds overhead and is often disabled in production. Relying solely on it is risky. The better approach is to design concurrent code that is provably safe, using patterns that minimize shared state and enforce clear ownership.

This guide will walk you through the four most common concurrency mistakes, explain why they happen, and provide concrete, tested solutions. Each section includes a real-world scenario, a step-by-step fix, and preventive measures. By the end, you'll have a mental checklist to avoid these pitfalls in your own projects.

1. Unprotected Shared State: The Data Race Trap

The most frequent concurrency mistake in Go is accessing shared memory from multiple goroutines without synchronization. A data race occurs when two or more goroutines access the same variable concurrently, and at least one of them is writing. The result is undefined behavior—the program may crash, produce incorrect results, or silently corrupt data. Go's memory model guarantees that a single goroutine sees its own writes consistently, but without explicit synchronization, there's no guarantee across goroutines.

Scenario: A Concurrent Counter

Imagine a web server that tracks the number of active connections using a shared integer. Multiple handler goroutines increment and decrement this counter. Without a mutex, the counter's value can become inconsistent. For example, two goroutines might read the same value, increment it, and write it back, resulting in only one increment instead of two. This is a classic race condition. In production, this could lead to resource leaks or incorrect metrics.

The Fix: Using sync.Mutex

The standard fix is to protect the shared variable with a sync.Mutex. Acquire the lock before reading or writing, and release it afterward. Go's sync package provides Mutex and RWMutex for this purpose. For the counter example, a simple mutex ensures atomic access:

type Server struct { mu sync.Mutex counter int } func (s *Server) Increment() { s.mu.Lock() s.counter++ s.mu.Unlock() }

This pattern is straightforward, but it's easy to forget to unlock, especially in error paths. A safer approach is to use defer s.mu.Unlock() immediately after locking, which ensures the lock is released even if the function panics.

Comparison of Synchronization Options

Choosing the right primitive depends on your use case. For most cases, a mutex is the safest default. However, if you have a read-heavy scenario, an RWMutex can improve performance. For simple counters, atomic operations are faster and avoid lock contention entirely. The key is to be deliberate and test with the race detector enabled.

Preventive Patterns

To avoid data races from the start, follow these guidelines: (1) Minimize shared state—design goroutines to communicate via channels rather than sharing memory. (2) Encapsulate shared state behind a single goroutine that serializes access (actor model). (3) Use the race detector in all tests and CI pipelines. (4) Consider using go-racedetect as a pre-commit hook. These habits reduce the likelihood of races reaching production.

2. Goroutine Leaks: The Silent Resource Drain

Goroutines are lightweight, but they're not free. Each goroutine consumes stack memory (initially 2 KB, but can grow) and holds references to heap objects. A goroutine leak occurs when goroutines are created but never exit, accumulating over time and eventually exhausting memory. This is one of the most insidious concurrency bugs because it doesn't cause an immediate crash—performance degrades slowly until the system becomes unresponsive.

Scenario: A Blocked Worker Pool

Consider a worker pool where tasks are submitted via a channel. If the worker exits unexpectedly (e.g., due to a panic) without draining the channel, other workers may block indefinitely waiting for tasks. Or, if the main goroutine stops sending tasks but doesn't close the channel, workers will block on receive. In a long-running service, leaked goroutines can accumulate until the process runs out of memory.

The Fix: Context Cancellation and WaitGroups

The standard solution is to use context.Context to signal goroutines to stop, and sync.WaitGroup to wait for their completion. When the context is cancelled, goroutines should check ctx.Done() and exit cleanly. For example:

func worker(ctx context.Context, tasks <-chan Task, wg *sync.WaitGroup) { defer wg.Done() for { select { case task, ok := <-tasks: if !ok { return // channel closed } process(task) case <-ctx.Done(): return // cancelled } } }

This pattern ensures that workers exit promptly when the context is cancelled or the channel is closed. The main goroutine can cancel the context to signal all workers to stop, then call wg.Wait() to ensure they have finished before shutting down. This prevents leaks and ensures clean termination.

Detecting Goroutine Leaks

In production, monitor the number of goroutines via runtime metrics (runtime.NumGoroutine()). A steadily increasing count is a red flag. Use the pprof package to capture goroutine profiles and identify which goroutines are blocked. For unit tests, use the leaktest library (like go.uber.org/goleak) to verify that no goroutines leak after a test completes. Integrate this into your CI to catch leaks early.

Preventive Patterns

To prevent goroutine leaks: (1) Always have a plan for how a goroutine will stop—use contexts or close channels explicitly. (2) Use sync.WaitGroup to track goroutines and wait for them to finish. (3) Avoid blocking operations without timeouts; use select with time.After to prevent indefinite blocking. (4) In tests, assert that goroutine counts return to baseline. These practices ensure that goroutines are ephemeral and don't accumulate.

3. Deadlocks: When Goroutines Wait Forever

A deadlock occurs when two or more goroutines are blocked waiting for each other to release a resource, and none can proceed. In Go, deadlocks often involve mutexes, channels, or a combination of both. The runtime can detect some deadlocks (all goroutines blocked) and crash the program, but partial deadlocks (where only a subset of goroutines are stuck) can go unnoticed, causing application hang.

Scenario: Circular Lock Dependency

Imagine two goroutines that need to acquire locks A and B in different orders. Goroutine 1 locks A then tries to lock B, while Goroutine 2 locks B then tries to lock A. If the timing is right, they end up waiting for each other indefinitely. This is a classic deadlock pattern. In a more complex system, deadlocks can involve multiple goroutines and resources, making them hard to reproduce.

The Fix: Consistent Lock Ordering

The primary fix is to establish a global ordering for locks and always acquire them in that order. For example, if you have two resources, always lock A before B. This prevents circular dependencies. Document the ordering and enforce it with tooling (e.g., custom linting rules). If you need to hold multiple locks, consider using a single mutex that protects all related state, or use a hierarchical locking scheme.

Another approach is to use timeouts when acquiring locks. Go's sync.Mutex doesn't support timeouts natively, but you can use channels to implement a try-lock pattern. For example, a goroutine can attempt to acquire a lock via a buffered channel of size 1, with a select that includes time.After. If the lock isn't acquired within the timeout, the goroutine can back off and retry, reducing the chance of deadlock.

Detecting Deadlocks

Deadlocks are often detected via monitoring: if a service stops responding to health checks, it may be deadlocked. Use pprof to capture goroutine stacks and look for mutexes locked by other goroutines. The Go runtime's deadlock detector only catches cases where all goroutines are blocked; partial deadlocks require manual inspection. Tools like go-deadlock (from Uber) can instrument mutexes to detect potential deadlocks during development.

Preventive Patterns

To avoid deadlocks: (1) Minimize the number of locks in your code; prefer channels for communication. (2) Use lock ordering and document it. (3) Keep critical sections short to reduce contention. (4) Consider using lock-free data structures where possible. (5) Use the go-deadlock package in tests to catch potential deadlocks before they reach production. These habits reduce the complexity of your concurrency model.

4. Channel Misuse: Blocking and Panicking

Channels are Go's primary communication mechanism, but they have sharp edges. Two common mistakes are sending to an unbuffered channel without a receiver (causing a deadlock) and closing a channel while goroutines are still sending to it (causing a panic). These errors are often introduced when refactoring code or adding error handling paths.

Scenario: Unbuffered Channel Without Receiver

Consider a function that sends a result to an unbuffered channel but the receiver hasn't started yet. The send blocks forever, causing a goroutine leak or deadlock. This often happens when the sender and receiver are in different goroutines and the startup order is not synchronized. For example, a worker goroutine that sends to a channel before the main goroutine starts reading from it will block.

The Fix: Buffered Channels or Synchronization

There are two common fixes. First, use a buffered channel with a capacity that matches the number of expected sends. This allows sends to proceed without a receiver, up to the buffer size. Second, ensure that the receiver is ready before the sender sends, using a sync.WaitGroup or a startup signal. For example, the sender can wait for the receiver to signal readiness via a separate channel.

// Using buffered channel ch := make(chan Result, 1) // buffer size 1 ch <- result // non-blocking if buffer not full // Using synchronization start := make(chan struct{}) go func() { <-start // wait for signal ch <- result }() close(start) // signal to start value := <-ch

Both approaches prevent blocking. Choose buffered channels when the number of sends is known in advance, and use synchronization when you need precise control over ordering.

Scenario: Closing a Channel While Senders Are Active

Another common mistake is closing a channel while other goroutines are still sending to it. This causes a panic with the message "send on closed channel". This often happens when one goroutine detects an error and closes the channel to signal termination, but another goroutine is still trying to send results. The fix is to ensure that only one goroutine closes the channel, and that no sends happen after the close.

The Fix: Use a Done Channel

A robust pattern is to use a separate "done" channel to signal cancellation, and leave the data channel open until all senders have finished. Alternatively, use a sync.WaitGroup to wait for all senders to complete before closing the channel. For example:

var wg sync.WaitGroup for i := 0; i < numWorkers; i++ { wg.Add(1) go func() { defer wg.Done() // send to ch }() } go func() { wg.Wait() close(ch) }()

This ensures that the channel is closed only after all senders have finished. The receiver can then safely range over the channel until it's closed.

Preventive Patterns

To avoid channel misuse: (1) Prefer buffered channels when the number of sends is bounded. (2) Use a dedicated "done" channel for cancellation instead of closing the data channel. (3) Clearly document who owns the channel and who is responsible for closing it. (4) Use select with default to handle non-blocking sends or receives when appropriate. (5) In complex pipelines, use the fan-out/fan-in pattern with WaitGroups to coordinate senders and receivers.

5. Real-World Case Studies: Lessons from Production

To illustrate the impact of these mistakes, here are two anonymized case studies from production environments. These examples highlight how seemingly small concurrency errors can escalate into major incidents.

Case Study 1: The Data Race That Caused Silent Data Loss

A financial services company had a service that aggregated trade data from multiple exchanges. Each exchange's data arrived in a separate goroutine, and they all updated a shared map of order books. The developers knew about data races but thought the map was only read after all goroutines finished. However, a race condition caused intermittent corruption: occasionally, a write would be lost, leading to incorrect pricing. The bug went unnoticed for weeks because it only occurred under peak load. When discovered, the fix was to replace the map with a concurrent map (e.g., sync.Map) or use a mutex. The lesson: never assume exclusive access without synchronization, even if the timing seems safe.

Case Study 2: The Goroutine Leak That Brought Down a SaaS Platform

A SaaS platform used a worker pool to process incoming webhooks. Each webhook spawned a goroutine that performed I/O operations. The code had a bug: if the I/O operation timed out, the goroutine would exit without releasing its resources, but the main loop would still try to send to a channel that the goroutine was supposed to read from. Over time, the number of blocked goroutines grew, consuming memory until the process was killed by the OOM killer. The fix involved using a context with timeout and ensuring that all goroutines checked for cancellation. The platform now monitors goroutine counts and alerts if they exceed a threshold.

Key Takeaways

These case studies underscore the importance of defensive concurrency programming. Both issues were preventable with proper synchronization and lifecycle management. The cost of fixing them after deployment was significantly higher than if they had been caught during development. Invest in code reviews, race detector runs, and monitoring from day one.

6. Tools and Techniques for Safe Concurrency

Beyond fixing the four mistakes, there are tools and techniques that can help you write correct concurrent code from the start. This section covers the most effective ones, from linting to runtime analysis.

Static Analysis and Linters

Several static analysis tools can catch concurrency issues before runtime. go vet includes checks for common mistakes, like passing a mutex by value. golangci-lint integrates multiple linters, including misspell, staticcheck, and ineffassign. For concurrency-specific checks, tools like go-deadlock instrument mutexes to detect potential deadlocks at runtime (in tests). Integrating these into your CI pipeline provides a safety net.

Race Detector

Go's built-in race detector is invaluable. Run your tests with -race flag to detect data races. However, remember that it only catches races that occur during execution; it's not exhaustive. Use it in combination with stress testing (e.g., running tests with GOMAXPROCS set to a high value) to increase the likelihood of triggering races.

pprof for Runtime Profiling

The net/http/pprof package provides runtime profiling endpoints. Use it to capture goroutine stacks, heap profiles, and mutex contention. In production, enable it securely (e.g., on an internal port) to debug issues. A sudden increase in blocked goroutines or mutex wait times is a strong signal of a concurrency problem.

Testing Patterns

Write concurrency tests that simulate high load and edge cases. Use sync.WaitGroup to coordinate goroutines in tests. The leaktest library (e.g., go.uber.org/goleak) can verify that no goroutines leak after a test. For deadlock detection, use go-deadlock in tests. Combine these with table-driven tests to cover multiple scenarios.

Design Patterns

Adopt design patterns that reduce concurrency complexity: (1) Actor model: encapsulate state within a single goroutine that receives messages via channels. (2) Pipeline pattern: use channels to connect stages, with each stage running in its own goroutine. (3) Fan-out/fan-in: distribute work across multiple goroutines and collect results. These patterns promote safe communication and minimize shared state.

7. Frequently Asked Questions

This section answers common questions about Go concurrency, drawing from real developer experiences and best practices.

Q: Should I use sync.Mutex or channels?

There's no one-size-fits-all answer. Use mutexes when you need to protect a small amount of shared state (like a counter or a map) and the critical section is short. Use channels when you need to communicate data between goroutines, especially when the data flow is sequential or requires buffering. A good rule of thumb: "Share memory by communicating, don't communicate by sharing memory." Prefer channels for passing ownership of data, and mutexes for guarding access to shared data.

Q: How do I choose between buffered and unbuffered channels?

Unbuffered channels provide synchronous communication: the sender blocks until the receiver is ready. This is useful for coordination and ensuring that events are processed in order. Buffered channels allow asynchronous communication: the sender can proceed as long as the buffer isn't full. Use buffered channels when you want to decouple the sender and receiver (e.g., in a producer-consumer pattern), but be careful about choosing the buffer size. A buffer that's too small can cause blocking, while one that's too large can hide backpressure issues. Start with a small buffer and increase based on profiling.

Q: What's the best way to handle errors in concurrent code?

Errors from goroutines should be propagated back to the caller. A common pattern is to use a result struct that includes both the value and an error. Send this over a channel. Alternatively, use an error group (golang.org/x/sync/errgroup) which waits for all goroutines and returns the first error. For more complex error handling, consider using a separate error channel. Always ensure that goroutines don't leak when errors occur—use contexts with cancellation to abort remaining work.

Q: How do I test concurrency code effectively?

Test concurrency code with the race detector enabled. Use stress tests that run the code under high concurrency (e.g., multiple goroutines calling the same function). Use t.Parallel() in tests to increase interleaving. For deterministic testing, consider using a mock time or a scheduler that controls goroutine ordering. Tools like goleak help catch goroutine leaks. Also, write integration tests that simulate production load.

Q: My app panics with "send on closed channel" - what went wrong?

This panic occurs when a goroutine tries to send on a channel that has been closed. The fix is to ensure that only one goroutine closes the channel, and that no sends happen after the close. Use a sync.WaitGroup to wait for all senders to finish before closing. Alternatively, use a separate done channel to signal cancellation, and close the data channel only when all senders have completed. Check for the closed state before sending by using a select with a default case or by using a separate boolean flag protected by a mutex.

8. Synthesis and Next Actions

Concurrency bugs are among the most challenging to debug, but they are also preventable with the right practices. This guide has covered the four most common mistakes: unprotected shared state, goroutine leaks, deadlocks, and channel misuse. Each section provided a concrete scenario, a fix, and preventive measures. Now, it's time to apply these lessons to your own codebase.

Start by auditing your current projects. Run the race detector on your tests. Check for goroutine leaks by monitoring runtime.NumGoroutine(). Review your mutex usage for potential deadlocks (do you always lock in the same order?). Examine your channel usage: are buffered channels sized appropriately? Are channels closed safely? Create a checklist based on the four mistakes and go through it systematically.

Next, integrate concurrency safety into your development workflow. Add -race to your test commands. Add golangci-lint to your CI pipeline. Use goleak in tests to detect leaks. Consider using go-deadlock during development. These tools will catch many issues before they reach production.

Finally, invest in learning advanced concurrency patterns. Explore the golang.org/x/sync package for tools like errgroup, semaphore, and singleflight. Read about lock-free data structures and the actor model. Practice writing concurrent code with a focus on correctness first, then optimize for performance. Remember, it's better to have a slightly slower but correct program than a fast one that crashes.

Concurrency is a powerful tool in Go, but it demands respect. By understanding these four mistakes and how to avoid them, you'll be well on your way to building robust, crash-resistant applications. Keep learning, keep testing, and keep iterating. Your future self—and your users—will thank you.