Idempotency

The Problem

In distributed systems, operations may be executed multiple times due to: - Network retries - Client retries on timeout - Message queue redelivery - Load balancer failover - Duplicate requests from users (double-click)

Without idempotency: - Double charges on payments - Duplicate orders - Multiple emails sent - Incorrect inventory counts - Data corruption

Definition: An operation is idempotent if executing it multiple times has the same effect as executing it once.

---

Naturally Idempotent Operations

Some operations are inherently idempotent:

// Idempotent: Setting absolute value
user.setEmail("[email protected]");
UPDATE users SET email = 'john@example.com' WHERE id = 123;

// Idempotent: DELETE
DELETE FROM orders WHERE id = 456;

// Idempotent: GET requests
GET /api/users/123

// NOT Idempotent: Relative operations
user.incrementBalance(100); // Each call adds 100
INSERT INTO orders (...);   // Each call creates new row
POST /api/orders            // Each call creates new order

---

Options & Trade-offs

1. Idempotency Keys

Philosophy: "Client provides unique key, server ensures single execution"

Request 1: POST /api/payments
           Idempotency-Key: abc123
           → Process payment, store result with key

Request 2: POST /api/payments (retry)
           Idempotency-Key: abc123
           → Return stored result, don't reprocess

Implementation:

@RestController
public class PaymentController {

    @PostMapping("/api/payments")
    public ResponseEntity<PaymentResult> createPayment(
            @RequestHeader("Idempotency-Key") String idempotencyKey,
            @RequestBody PaymentRequest request) {

        // Check for existing result
        Optional<IdempotencyRecord> existing =
            idempotencyRepository.findByKey(idempotencyKey);

        if (existing.isPresent()) {
            IdempotencyRecord record = existing.get();

            // Still processing
            if (record.getStatus() == Status.PROCESSING) {
                return ResponseEntity.status(HttpStatus.CONFLICT)
                    .header("Retry-After", "5")
                    .build();
            }

            // Return cached result
            return ResponseEntity
                .status(record.getResponseStatus())
                .body(deserialize(record.getResponseBody()));
        }

        // Create processing record
        IdempotencyRecord record = IdempotencyRecord.builder()
            .key(idempotencyKey)
            .status(Status.PROCESSING)
            .createdAt(Instant.now())
            .build();

        try {
            idempotencyRepository.save(record);
        } catch (DuplicateKeyException e) {
            // Race condition - another request got there first
            return ResponseEntity.status(HttpStatus.CONFLICT).build();
        }

        try {
            // Process the payment
            PaymentResult result = paymentService.process(request);

            // Store successful result
            record.setStatus(Status.COMPLETED);
            record.setResponseStatus(HttpStatus.OK.value());
            record.setResponseBody(serialize(result));
            idempotencyRepository.save(record);

            return ResponseEntity.ok(result);

        } catch (Exception e) {
            // Store error result
            record.setStatus(Status.FAILED);
            record.setResponseStatus(HttpStatus.INTERNAL_SERVER_ERROR.value());
            record.setResponseBody(serialize(new ErrorResponse(e.getMessage())));
            idempotencyRepository.save(record);

            throw e;
        }
    }
}

@Entity
@Table(name = "idempotency_records",
       indexes = @Index(columnList = "idempotency_key", unique = true))
public class IdempotencyRecord {
    @Id
    private String id;

    @Column(name = "idempotency_key", unique = true)
    private String key;

    @Enumerated(EnumType.STRING)
    private Status status;

    private Integer responseStatus;

    @Lob
    private String responseBody;

    private Instant createdAt;
    private Instant expiresAt;
}

Cleanup:

@Scheduled(cron = "0 0 * * * *") // Every hour
public void cleanupOldRecords() {
    Instant threshold = Instant.now().minus(Duration.ofDays(7));
    idempotencyRepository.deleteByCreatedAtBefore(threshold);
}

Pros	Cons
Works for any operation	Client must generate keys
Stateless application tier	Storage overhead
Complete response caching	Expiration policy needed
Works across retries	Key generation responsibility

---

2. Request Deduplication

Philosophy: "Detect and reject duplicate requests"

@Component
public class DeduplicationFilter implements Filter {

    private final Cache<String, Boolean> recentRequests =
        Caffeine.newBuilder()
            .expireAfterWrite(Duration.ofMinutes(5))
            .maximumSize(100_000)
            .build();

    @Override
    public void doFilter(ServletRequest request, ServletResponse response,
                         FilterChain chain) throws IOException, ServletException {

        HttpServletRequest httpRequest = (HttpServletRequest) request;

        if ("POST".equals(httpRequest.getMethod())) {
            String requestHash = computeHash(httpRequest);

            Boolean exists = recentRequests.getIfPresent(requestHash);
            if (exists != null) {
                HttpServletResponse httpResponse = (HttpServletResponse) response;
                httpResponse.setStatus(HttpStatus.TOO_MANY_REQUESTS.value());
                httpResponse.getWriter().write("Duplicate request detected");
                return;
            }

            recentRequests.put(requestHash, true);
        }

        chain.doFilter(request, response);
    }

    private String computeHash(HttpServletRequest request) {
        // Hash of: user ID + endpoint + request body
        return DigestUtils.sha256Hex(
            request.getUserPrincipal().getName() +
            request.getRequestURI() +
            getRequestBody(request)
        );
    }
}

Pros	Cons
Transparent to clients	Doesn't return original response
Simple implementation	Time-window based
No client changes needed	May reject legitimate requests

---

3. Database Constraints

Philosophy: "Use database to enforce uniqueness"

-- Unique constraint on business key
CREATE TABLE orders (
    id BIGSERIAL PRIMARY KEY,
    order_number VARCHAR(50) UNIQUE NOT NULL,
    customer_id BIGINT NOT NULL,
    amount DECIMAL(10, 2) NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Unique constraint on composite key
CREATE TABLE user_actions (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL,
    action_type VARCHAR(50) NOT NULL,
    reference_id VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    UNIQUE(user_id, action_type, reference_id)
);

@Transactional
public Order createOrder(OrderRequest request) {
    String orderNumber = generateOrderNumber(request);

    try {
        Order order = Order.builder()
            .orderNumber(orderNumber) // Unique
            .customerId(request.getCustomerId())
            .amount(request.getAmount())
            .build();

        return orderRepository.save(order);

    } catch (DataIntegrityViolationException e) {
        // Duplicate - return existing order
        return orderRepository.findByOrderNumber(orderNumber)
            .orElseThrow(() -> new IllegalStateException("Order not found after conflict"));
    }
}

private String generateOrderNumber(OrderRequest request) {
    // Deterministic based on request content
    return DigestUtils.sha256Hex(
        request.getCustomerId() +
        request.getCartId() +
        request.getTimestamp()
    ).substring(0, 16);
}

Pros	Cons
Database enforced	Requires unique business key
No additional storage	Constraint violations to handle
ACID guaranteed	May not return original response

---

4. Optimistic Locking with Version

Philosophy: "Prevent concurrent modifications with version checking"

@Entity
public class Account {
    @Id
    private Long id;

    private BigDecimal balance;

    @Version
    private Long version;
}

@Transactional
public void transfer(Long fromId, Long toId, BigDecimal amount) {
    Account from = accountRepository.findById(fromId).orElseThrow();
    Account to = accountRepository.findById(toId).orElseThrow();

    from.setBalance(from.getBalance().subtract(amount));
    to.setBalance(to.getBalance().add(amount));

    // Version check happens on save
    try {
        accountRepository.save(from);
        accountRepository.save(to);
    } catch (OptimisticLockingFailureException e) {
        // Another transaction modified - retry or fail
        throw new ConcurrentModificationException("Account was modified, please retry");
    }
}

Pros	Cons
Prevents lost updates	Doesn't prevent duplicate operations
Built into JPA	Requires retry logic
No additional tables

---

5. Operation Log / Event Sourcing

Philosophy: "Check if operation was already performed"

@Service
public class TransferService {

    @Transactional
    public TransferResult transfer(TransferRequest request) {
        // Check if already processed
        Optional<TransferEvent> existing = eventStore.findByTransferId(request.getTransferId());
        if (existing.isPresent()) {
            return existing.get().getResult();
        }

        // Perform transfer
        Account from = accountRepository.findById(request.getFromAccountId()).orElseThrow();
        Account to = accountRepository.findById(request.getToAccountId()).orElseThrow();

        from.debit(request.getAmount());
        to.credit(request.getAmount());

        // Record operation
        TransferEvent event = TransferEvent.builder()
            .transferId(request.getTransferId())
            .fromAccountId(request.getFromAccountId())
            .toAccountId(request.getToAccountId())
            .amount(request.getAmount())
            .result(TransferResult.SUCCESS)
            .timestamp(Instant.now())
            .build();

        eventStore.save(event);

        return TransferResult.SUCCESS;
    }
}

---

6. Token-Based Deduplication

Philosophy: "Issue tokens for sensitive operations"

1. Client: GET /api/orders/prepare
   Server: Returns token "tok_abc123" (valid for 5 minutes)

2. Client: POST /api/orders
           Token: tok_abc123
   Server: Create order, invalidate token

3. Client: POST /api/orders (retry)
           Token: tok_abc123
   Server: Token already used, reject

Implementation:

@RestController
public class OrderController {

    // Step 1: Get a token
    @GetMapping("/api/orders/prepare")
    public PrepareResponse prepareOrder() {
        String token = UUID.randomUUID().toString();
        tokenStore.save(token, Duration.ofMinutes(5));
        return new PrepareResponse(token);
    }

    // Step 2: Use token to create order
    @PostMapping("/api/orders")
    public ResponseEntity<Order> createOrder(
            @RequestHeader("X-Order-Token") String token,
            @RequestBody OrderRequest request) {

        // Atomically check and consume token
        boolean consumed = tokenStore.consume(token);
        if (!consumed) {
            return ResponseEntity.status(HttpStatus.CONFLICT)
                .body(null); // Token already used or expired
        }

        Order order = orderService.create(request);
        return ResponseEntity.ok(order);
    }
}

@Component
public class TokenStore {
    private final StringRedisTemplate redis;

    public void save(String token, Duration ttl) {
        redis.opsForValue().set("token:" + token, "valid", ttl);
    }

    public boolean consume(String token) {
        // Atomic delete, returns true if existed
        return redis.delete("token:" + token);
    }
}

Pros	Cons
Simple and effective	Extra round trip for token
Works for any operation	Token management
Time-limited by default	Client must store token

---

7. Conditional Requests (ETags)

Philosophy: "Only modify if unchanged since last read"

# Read resource
GET /api/documents/123
Response:
  ETag: "abc123"
  Body: { ... }

# Update only if unchanged
PUT /api/documents/123
If-Match: "abc123"
Body: { updated content }

Response (if changed by someone else):
  409 Conflict

Response (if unchanged):
  200 OK
  ETag: "def456"

@GetMapping("/api/documents/{id}")
public ResponseEntity<Document> getDocument(@PathVariable String id) {
    Document doc = documentRepository.findById(id).orElseThrow();

    return ResponseEntity.ok()
        .eTag(doc.getVersion().toString())
        .body(doc);
}

@PutMapping("/api/documents/{id}")
public ResponseEntity<Document> updateDocument(
        @PathVariable String id,
        @RequestHeader("If-Match") String ifMatch,
        @RequestBody Document update) {

    Document existing = documentRepository.findById(id).orElseThrow();

    // Compare versions
    if (!existing.getVersion().toString().equals(ifMatch.replace("\"", ""))) {
        return ResponseEntity.status(HttpStatus.PRECONDITION_FAILED).build();
    }

    existing.setContent(update.getContent());
    existing.setVersion(existing.getVersion() + 1);
    documentRepository.save(existing);

    return ResponseEntity.ok()
        .eTag(existing.getVersion().toString())
        .body(existing);
}

---

8. Message Queue Idempotency

Philosophy: "Deduplicate at consumer level"

Kafka Consumer:

@KafkaListener(topics = "orders")
public void processOrder(ConsumerRecord<String, OrderEvent> record) {
    String messageId = record.key();

    // Check if already processed
    if (processedMessageRepository.existsByMessageId(messageId)) {
        log.info("Skipping duplicate message: {}", messageId);
        return;
    }

    // Process order
    orderService.process(record.value());

    // Mark as processed
    processedMessageRepository.save(new ProcessedMessage(
        messageId,
        Instant.now()
    ));
}

With Transactional Outbox:

@Transactional
public void processOrder(OrderEvent event) {
    String eventId = event.getId();

    // Idempotent check in same transaction
    if (processedEventRepository.existsById(eventId)) {
        return;
    }

    // Business logic
    Order order = orderService.create(event);

    // Mark processed in same transaction
    processedEventRepository.save(new ProcessedEvent(eventId, Instant.now()));
}

RabbitMQ with Deduplication Plugin:

// Set message ID for deduplication
rabbitTemplate.convertAndSend(exchange, routingKey, message, m -> {
    m.getMessageProperties().setMessageId(UUID.randomUUID().toString());
    m.getMessageProperties().setHeader("x-deduplication-header", deduplicationKey);
    return m;
});

---

Comparison Matrix

Approach	Complexity	Returns Original?	Storage	Best For
Idempotency Keys	Medium	Yes	Yes	APIs with retries
Deduplication Filter	Low	No	Minimal	Simple duplicate prevention
DB Constraints	Low	Maybe	No extra	Natural business keys
Optimistic Locking	Low	N/A	No extra	Update conflicts
Operation Log	Medium	Yes	Yes	Audit + idempotency
Token-Based	Medium	No	Minimal	Sensitive operations
ETags	Low	N/A	No extra	Resource updates
MQ Deduplication	Medium	N/A	Yes	Event processing

---

Decision Tree

---

Architecture Example

---

Best Practices

Idempotency Key Generation (Client)

// Good: Deterministic based on intent
const idempotencyKey = sha256(`
    ${userId}
    ${action}
    ${JSON.stringify(sortedPayload)}
    ${timestamp.toISOString().slice(0, 16)} // 1-minute granularity
`);

// Or: UUID stored with the request
const idempotencyKey = localStorage.getItem('pendingOrderKey')
    || localStorage.setItem('pendingOrderKey', uuid());

Retry Strategies (Client)

async function requestWithRetry(url, options, idempotencyKey) {
    const maxRetries = 3;
    const baseDelay = 1000;

    for (let attempt = 0; attempt < maxRetries; attempt++) {
        try {
            const response = await fetch(url, {
                ...options,
                headers: {
                    ...options.headers,
                    'Idempotency-Key': idempotencyKey
                }
            });

            if (response.status === 409) { // Conflict - still processing
                await delay(baseDelay * Math.pow(2, attempt));
                continue;
            }

            return response;
        } catch (error) {
            if (attempt === maxRetries - 1) throw error;
            await delay(baseDelay * Math.pow(2, attempt));
        }
    }
}

---

Common Pitfalls

1. Not returning cached response - Client retries get different results
2. Too short TTL - Retries after expiration create duplicates
3. Not handling "in progress" - Parallel retries cause issues
4. Mutable idempotency keys - Different key = different operation
5. Not cleaning up old records - Storage grows unbounded
6. Side effects not idempotent - Email sent twice, webhook called twice

---

Key Takeaways

1. Make all mutating endpoints idempotent - Assume retries will happen
2. Idempotency keys for critical operations - Payments, orders, transfers
3. Database constraints as safety net - Unique business keys
4. Return cached responses - Same input = same output
5. Handle concurrent requests - "Processing" state is important
6. Clean up old records - Storage management
7. Client-side key generation - Deterministic or stored
8. Document expected behavior - Clients need to know retry semantics