ChargeCustomerJob.perform_later(order) looks like a function call and behaves like a distributed system. It runs on another machine, at an unknown time, possibly more than once, possibly never. Most job bugs come from writing the first thing and reasoning about it as if it were the second. Getting retries and idempotency right is what separates a queue that quietly heals from one that silently double-charges people.

At-Least-Once Is the Only Guarantee

Every queue backend — Solid Queue, Sidekiq, SQS — delivers at least once. A worker can pick up a job, complete the side effect, and die before acknowledging. The job comes back. Your code runs twice.

This is not a rare edge case. It happens on every deploy that terminates workers mid-job, every OOM kill, every network partition.

So the design rule is unavoidable: a job must produce the same outcome whether it runs once or five times.

Retry Configuration

Example:

class ChargeCustomerJob < ApplicationJob
  queue_as :payments

  retry_on Net::OpenTimeout, Net::ReadTimeout,
           wait: :polynomially_longer,
           attempts: 5

  retry_on PaymentGateway::RateLimited,
           wait: ->(executions) { executions * 30.seconds },
           attempts: 10

  discard_on ActiveJob::DeserializationError
  discard_on PaymentGateway::CardDeclined

  def perform(order_id)
    order = Order.find(order_id)
    # ...
  end
end

Three distinct decisions here, and they’re the whole design:

retry_on for transient failures — timeouts, rate limits, temporary unavailability. These succeed later.

discard_on for permanent failures. A declined card will be declined on every retry; retrying it five times just delays the notification and burns rate limit. DeserializationError means the record was deleted — retrying can never help.

Everything else falls through to the queue backend’s default retry, then to the dead set. That’s correct: an unexpected exception should not be silently swallowed.

Backoff strategy

wait: :polynomially_longer (formerly :exponentially_longer) waits roughly executions ** 4 seconds — about 1s, 16s, 81s, 256s, 625s. Rails adds jitter automatically, which matters: without it, a thousand jobs failing on the same outage all retry at the same instant and knock the recovering service back over.

For a rate-limited API, respect its Retry-After instead of guessing:

retry_on PaymentGateway::RateLimited do |job, error|
  job.class.set(wait: error.retry_after.seconds).perform_later(*job.arguments)
end

Pass IDs, Not Objects

Example:

# Fragile
ChargeCustomerJob.perform_later(order)

# Correct
ChargeCustomerJob.perform_later(order.id)

GlobalID serialization looks convenient and creates two problems. If the record is deleted between enqueue and execution, the job raises DeserializationError — which you can discard, but you’ve lost the ability to handle the deletion meaningfully. And the record is re-fetched at execution time anyway, so passing the object bought nothing.

Pass the id and any values that must be captured at enqueue time:

ChargeCustomerJob.perform_later(order.id, amount_cents: order.total_cents)

That second argument matters if the order total can change — you want to charge what was agreed, not what the record says three minutes later.

Making Jobs Idempotent

Four techniques, roughly in order of how often you’ll reach for them.

1. Guard on state

The cheapest and most common. Check whether the work is already done.

def perform(order_id)
  order = Order.find(order_id)
  return if order.charged?

  gateway.charge(order)
  order.update!(charged_at: Time.current)
end

The gap between the check and the update is a race. Two workers can both pass the guard. Close it with a lock or a database constraint.

2. Idempotency key on the external call

The right answer whenever the external service supports it, because it moves the deduplication to the system that actually knows whether the charge happened.

def perform(order_id)
  order = Order.find(order_id)

  gateway.charge(
    amount_cents: order.total_cents,
    idempotency_key: "order-charge-#{order.id}"
  )
end

Stripe, and most payment APIs, return the original response for a repeated key rather than charging again. Your job can run ten times and the customer is charged once.

3. Unique database constraint

Let the database enforce it.

# add_index :charges, :order_id, unique: true

def perform(order_id)
  Charge.create!(order_id:, amount_cents: ...)
rescue ActiveRecord::RecordNotUnique
  Rails.logger.info("Charge for order #{order_id} already exists — skipping")
end

Rescuing the uniqueness violation is not a hack; it’s the correct pattern. The database is the only component that can make this decision atomically.

4. Advisory lock for non-transactional work

def perform(order_id)
  Order.with_advisory_lock("charge-order-#{order_id}", timeout_seconds: 0) do
    order = Order.find(order_id)
    return if order.charged?

    gateway.charge(order)
    order.update!(charged_at: Time.current)
  end
end

timeout_seconds: 0 means a second worker fails to acquire the lock and moves on rather than queueing behind the first. That’s usually what you want for a duplicate.

Transactions and Enqueueing

A subtle and very common bug:

Example:

# Broken
ActiveRecord::Base.transaction do
  order = Order.create!(params)
  ChargeCustomerJob.perform_later(order.id)   # may run before COMMIT
end

The job can be picked up by a worker before the transaction commits, and Order.find raises RecordNotFound. Worse, if the transaction rolls back, you’ve enqueued a job for a record that will never exist.

Two fixes:

# Enqueue after commit
ActiveRecord::Base.transaction do
  order = Order.create!(params)
  order.create_charge_record!
end
ChargeCustomerJob.perform_later(order.id)

# Or, from a model callback
class Order < ApplicationRecord
  after_commit :enqueue_charge, on: :create
end

Solid Queue avoids this entirely when it shares the database — the job insert is in the same transaction, so it commits or rolls back atomically. That’s a genuine and underappreciated advantage of a database-backed queue.

Dead Jobs Are a Signal, Not Garbage

After the retries are exhausted, the job lands in a dead set. Most teams never look at it, which means failures accumulate invisibly for months.

Example:

class ApplicationJob < ActiveJob::Base
  around_perform do |job, block|
    block.call
  rescue StandardError => e
    if job.executions >= 5
      FailedJob.create!(
        job_class: job.class.name,
        arguments: job.arguments,
        error_class: e.class.name,
        error_message: e.message,
        backtrace: e.backtrace&.first(20)&.join("\n"),
        failed_at: Time.current
      )
      Sentry.capture_exception(e, extra: { job: job.class.name, args: job.arguments })
    end
    raise
  end
end

Recording failures in your own table gives you something you can query, group, and retry selectively — and it survives the queue backend being flushed.

Alert on dead-set growth rate, not absolute count. A dead set that gained forty jobs in an hour is an incident; one holding forty jobs from three months ago is cleanup.

Pro-Tip: Set a job timeout, because the failure mode nobody plans for isn’t a job that errors — it’s one that hangs. An HTTP call with no timeout against a service that accepts the connection and never responds will occupy a worker thread indefinitely. Five of those and your entire worker pool is gone, the queue backs up, and nothing in your monitoring says “error” because nothing errored. Set explicit timeouts on every external client (Faraday open_timeout and timeout, both — the default for many clients is nil), and wrap the job body in a Timeout guard as a backstop. Then alert on queue latency — the age of the oldest unstarted job — rather than only on depth, because a stalled pool shows up in latency long before depth looks unusual.

Queue Separation

One queue means a slow bulk-import job blocks a password reset email.

class ChargeCustomerJob < ApplicationJob
  queue_as :critical
end

class GenerateReportJob < ApplicationJob
  queue_as :bulk
end

Separate by latency requirement, not by feature area. Three queues covers most applications: critical (user is waiting), default (should happen soon), bulk (whenever). Give them separate worker pools so a backlog in one can’t starve another.

Conclusion

Treat every job as if it will run twice, because eventually it will. Pass ids rather than objects, use retry_on for transient failures and discard_on for permanent ones, and make the work idempotent with the cheapest mechanism that fits — a state guard, an idempotency key on the external call, or a unique index. Enqueue after commit unless your queue shares the database transaction. Set explicit timeouts so a hung call can’t consume the worker pool, alert on queue latency rather than depth alone, and actually look at the dead set — it’s the highest-signal, least-read source of information about what’s broken.

FAQs

Q1: What’s the difference between retry_on and the backend’s own retries?
retry_on is ActiveJob-level and re-enqueues the job. Sidekiq’s retry mechanism is separate and operates on its own schedule. Configuring both means multiplied attempts — pick one layer and be explicit about it.

Q2: How many retry attempts is right?
Enough to outlast a typical transient outage, which for most APIs is five attempts over roughly fifteen minutes. More than that and you’re delaying a human’s awareness that something is broken.

Q3: Should I use perform_now in tests?
Use perform_enqueued_jobs or the :test adapter’s assertions so you’re testing that the job gets enqueued with the right arguments and that it works. perform_now everywhere hides serialization bugs.

Q4: How do I handle a job whose arguments are too large?
Don’t pass large payloads. Write them to storage or a database row and pass the identifier. Queue backends have size limits, and a large argument is serialized and deserialized on every retry.

Q5: Is discard_on dangerous?
It can be — a discarded job is gone with no record unless you log it. Always log inside a discard_on block so a discarded failure is still visible.