Latch Troubleshooting

Lock acquisition times out (408)

Why? The lock holder has not released within your timeout, or contention is high.

Fix:

  • Increase the timeout value on your acquire request.
  • Check whether the lock holder is releasing promptly. Use GET /v1/latch/:key to inspect TTL remaining.
  • If the holder crashed, wait for the TTL to expire — it will auto-release.

Lock not releasing

Why? The process crashed before calling release, the token is wrong, or the lock already expired and was re-acquired by someone else.

Fix:

  • If the process crashed, the lock auto-releases after TTL. This is expected behavior.
  • Verify you are passing the exact token returned from acquire.
  • Always release in a finally block to handle crashes.
  • If your lock expired while you were still working, your TTL is too short.

Getting 403 Forbidden on release

Why? The token you sent does not match the current lock holder.

Fix:

  • The lock may have auto-released (TTL expired) and another client acquired it. Your token is now stale.
  • Store the token immediately after acquire and use it exactly once for release.
  • Check if your TTL is long enough for your operation.

Getting 429 on acquire (queue full)

Why? The lock key already has 100 waiters queued.

Fix:

  • This is separate from account-level rate limiting.
  • Reduce contention by sharding work across multiple lock keys.
  • Set a shorter timeout so waiters drain faster.

Lock expires mid-operation

Why? Your TTL is shorter than the work duration.

Fix:

  • Set TTL to at least 2x your expected operation time.
  • For variable-length work, use reentrant locking to refresh the TTL periodically. Re-acquire with the same token and timeout: 0.

Deadlock between processes

Why? Process A holds lock X and waits for lock Y. Process B holds lock Y and waits for lock X.

Fix:

  • Always acquire locks in a consistent order across all processes.
  • Use a single lock with a broader scope instead of multiple fine-grained locks when possible.
  • Set a timeout on every acquire so deadlocked requests eventually return 408.

408 after storage eviction

Why? The stateful storage instance holding the waiter queue was evicted from memory.

Fix:

  • Treat 408 the same as any timeout — retry the acquire.
  • This is rare for active locks. No special handling needed beyond normal timeout retry logic.