Low Level Concurrency

Introduction

Threads are smaller units of a process that allow executing tasks in parallel, sharing memory space. Threads are particularly useful for I/O bound tasks, where much of the time is spent waiting for external resources.

Limit Concurrent Operations

Applies when you have operations that can run concurrently, but you need to cap how many can run at once, e.g., integrating with an external API that allows at most 10 concurrent requests per account, limiting number of image processing jobs to avoid CPU saturation, etc.

Semaphores solve this directly, e.g.,

public class APIClient
{
  // Limit to at most 5 concurrent requests.
  private readonly SemaphoreSlim _requestPermits = new SemaphoreSlim(5);

  public async Task<Response> MakeRequestAsync(string endpoint)
  {
    await _requestPermits.WaitAsync(); // Block if 5 requests are in-flight

    try { return await _httpClient.GetAsync(endpoint); }
    finally { _requestPermits.Release(); }
  }
}

The Problem

Some work should not be ran in the mainline as it takes too long, and makes your mainline unresponsive, e.g., on user signup, sending a welcome email should be done out of line lest your application cannot process new requests. Conceptually, API handlers produce tasks, workers consume tasks, and something sits in between to coordinate the handoff.

Three things need solving in this scenario:

• Efficient waiting: Consumers should sleep when there’s no work, and wake up immediately when new work arrives.
• Backpressure: Producers should slow down when consumers can’t keep up, preventing memory exhaustion which would take down the whole application.
• Thread safety: The coordination mechanism itself must handle concurrent access without corruption.

Bug Patterns

Check-Then-Act

Involves checking a condition and then making a decision based on that check, e.g., in a ticket booking system, checking that a seat is available and then booking it if so.

Consider this buggy code:

public class RateLimiterBroken
{
  private readonly Dictionary<string, int> _requestCounts = [];
  private readonly int _maxRequests = 100;

  public bool ShouldAllowRequest(string userId)
  {
    _requestCounts.TryGetValue(userId, out int count);

    if (count >= _maxRequests) { return false; }

    _requestCounts[userId] = count + 1;
    return true;
  }
}

… where thread A reads count as 99, but before it updates the map, thread B also reads count as 99. Both threads A and B increment the count from 99 to 100 even though the user has made 101 requests.