Someone on the team claims map.compact is faster than filter_map. Someone else says the opposite. Both have benchmarks. Both benchmarks are wrong, in different ways — one measured a hundred iterations on a cold VM, the other measured an empty array. Micro-benchmarking Ruby is easy to do and surprisingly hard to do correctly, and the difference between the two is mostly about what you leave out of the measured block.

Why Benchmark.measure Isn’t Enough

The standard library’s Benchmark reports elapsed time for a fixed number of iterations. That’s the wrong unit for comparing implementations.

Example:

require "benchmark"

n = 100_000
Benchmark.bm do |x|
  x.report("map.compact")  { n.times { data.map { |i| i if i.even? }.compact } }
  x.report("filter_map")   { n.times { data.filter_map { |i| i if i.even? } }}
end

Two problems. You had to pick n yourself, and if you picked badly the whole run finishes before the JIT and the GC settle into steady state. And the output is seconds, so a 4% difference is buried in noise you have no way to quantify.

benchmark-ips inverts it: fix the time, measure the iterations, and report the variance.

The Basic Shape

Setup:

gem install benchmark-ips

Example:

require "benchmark/ips"

data = Array.new(1_000) { rand(100) }

Benchmark.ips do |x|
  x.report("map.compact") { data.map { |i| i if i.even? }.compact }
  x.report("filter_map")  { data.filter_map { |i| i if i.even? } }
  x.report("select.map")  { data.select(&:even?).map { |i| i } }

  x.compare!
end

Output:

Warming up --------------------------------------
         map.compact     4.312k i/100ms
          filter_map     5.891k i/100ms
          select.map     4.508k i/100ms
Calculating -------------------------------------
         map.compact     43.201k (± 1.8%) i/s  (23.15 μs/i) -    216.  in   5.002047s
          filter_map     58.904k (± 1.4%) i/s  (16.98 μs/i) -    295.  in   5.008321s
          select.map     45.117k (± 2.1%) i/s  (22.16 μs/i) -    225.  in   5.001884s

Comparison:
          filter_map:    58904.2 i/s
          select.map:    45117.3 i/s - 1.31x  slower
         map.compact:    43201.1 i/s - 1.36x  slower

Three things to read:

• i/s — iterations per second, higher is better. This is the number.
• ± percentage — standard deviation. Above roughly 5% and the run is too noisy to trust; something else on your machine is competing for CPU.
• x.compare! — prints the ratios so you don’t do arithmetic in your head.

The warmup phase exists so the VM reaches steady state before measurement begins. That alone eliminates the most common source of bogus results.

What to Put Inside the Block — and What Not To

The block runs millions of times. Anything inside it is measured, including setup you didn’t mean to measure.

Example:

# Wrong — measuring array construction, not the method under test
Benchmark.ips do |x|
  x.report("filter_map") do
    data = Array.new(1_000) { rand(100) }     # ← dominates the measurement
    data.filter_map { |i| i if i.even? }
  end
end

Example:

# Right — build the fixture once, outside
data = Array.new(1_000) { rand(100) }

Benchmark.ips do |x|
  x.report("filter_map") { data.filter_map { |i| i if i.even? } }
end

When each iteration genuinely needs fresh state, use x.report with a per-iteration setup you accept as constant overhead across all variants — the comparison stays valid even if the absolute numbers include the setup, as long as every variant pays the same cost.

The Fixture Determines the Answer

This is the failure that produces the most confident wrong conclusions. Method performance depends on the data, and a benchmark on unrepresentative data measures nothing useful.

Example:

require "benchmark/ips"

def bench(label, data)
  puts "\n=== #{label} (#{data.size} elements) ==="
  Benchmark.ips do |x|
    x.report("include?") { data.include?(999_999) }
    x.report("Set#include?") do
      @set ||= Set.new(data)
      @set.include?(999_999)
    end
    x.compare!
  end
end

bench("tiny",  (1..10).to_a)
bench("small", (1..1_000).to_a)
bench("large", (1..1_000_000).to_a)

On ten elements, Array#include? wins — building a Set costs more than a linear scan. On a million, Set#include? is thousands of times faster. Either result, published without the size, is misleading.

Benchmark with the data shape you actually have. Empty arrays, single-element arrays, and the worst case all belong in the run.

Measuring Allocations, Not Just Time

Speed and memory are different questions, and in a long-running Rails process allocation pressure often matters more than raw i/s because it drives GC.

Example:

require "benchmark/memory"

data = Array.new(1_000) { rand(100) }

Benchmark.memory do |x|
  x.report("map.compact") { data.map { |i| i if i.even? }.compact }
  x.report("filter_map")  { data.filter_map { |i| i if i.even? } }
  x.compare!
end

Output:

Comparison:
          filter_map:      8264 allocated
         map.compact:     16472 allocated - 1.99x more

map.compact allocates two arrays; filter_map allocates one. That’s the actual reason for the speed difference, and knowing the mechanism is more transferable than knowing the ratio.

For a quick check without another gem:

require "objspace"

before = GC.stat(:total_allocated_objects)
1_000.times { data.filter_map { |i| i if i.even? } }
puts GC.stat(:total_allocated_objects) - before

Controlling for GC

A garbage collection during a measurement window skews that window. benchmark-ips runs long enough that GC averages out, but for tight comparisons you can suppress it:

Example:

Benchmark.ips do |x|
  x.config(time: 5, warmup: 2)

  x.report("candidate") do
    GC.disable
    result = expensive_operation
    GC.enable
    result
  end
end

Be careful with this. Disabling GC measures the algorithm in isolation, which is sometimes what you want and sometimes hides the very cost that matters. If one candidate allocates ten times more than another, the GC pressure is the difference — suppressing it produces a benchmark that says the wrong thing about production.

Pro-Tip: Before optimizing anything based on a micro-benchmark, work out how many times the code actually runs. A method that takes 23μs instead of 17μs is 6μs slower — call it a thousand times per request and you’ve saved 6 milliseconds, which is invisible next to a single 40ms database query. Micro-benchmarks are for choosing between two equally readable options, not for finding what to optimize. To find that, use a sampling profiler on the real workload: stackprof with mode: :wall over a hundred requests will point at the one method consuming 30% of the time, and it is essentially never the one you’d have guessed. Profile to decide where, benchmark to decide how.

A Benchmark You Can Commit

For a change where performance is the point, put the benchmark in the repo so the next person can verify the claim.

Example:

# benchmarks/serialization.rb
require_relative "../config/environment"
require "benchmark/ips"

ACCOUNT = Account.includes(orders: :line_items).first

Benchmark.ips do |x|
  x.config(time: 10, warmup: 3)

  x.report("as_json")        { ACCOUNT.as_json(include: { orders: { include: :line_items } }) }
  x.report("AccountSerializer") { AccountSerializer.new(ACCOUNT).to_h }

  x.compare!
  x.hold! "tmp/serialization_benchmark.json"
end

hold! persists results across runs, so you can benchmark one implementation, switch branches, benchmark the other, and get a comparison between them — which is otherwise awkward when the two versions can’t coexist in one process.

Conclusion

benchmark-ips gets the methodology right by default: it warms up, it fixes time rather than iteration count, and it reports variance so you know whether to believe the number. The remaining discipline is yours — keep fixture construction out of the measured block, benchmark the data shape you actually have including the degenerate cases, check allocations alongside time, and confirm the code is hot before you optimize it at all. A 1.36× win on a method called twice per request is a rounding error; the same win on a method called ten thousand times is worth the pull request.

FAQs

Q1: How long should a benchmark run?
The defaults — 2 seconds warmup, 5 seconds measurement — are fine for most comparisons. Increase them via x.config(time:, warmup:) when the standard deviation is above 5% or when the operation itself takes milliseconds.

Q2: Why do I get different results on my laptop than in CI?
Different CPU, different Ruby build, thermal throttling, and noisy neighbours on shared runners. Absolute numbers aren’t portable; ratios between candidates measured in the same run usually are.

Q3: Does YJIT change benchmark results?
Substantially, and not uniformly — JIT-friendly code can improve several times over while allocation-bound code barely moves. Benchmark with the same JIT configuration you run in production, and state which one you used when sharing results.

Q4: What’s the difference between i/s and μs/i?
They’re reciprocals of the same measurement. Iterations per second is easier to compare across candidates; microseconds per iteration is easier to reason about when estimating the impact on a request.

Q5: Should benchmarks live in the test suite?
Not as assertions — they’re too machine-dependent to gate a build reliably. Keep them in a benchmarks/ directory as scripts anyone can run, and reserve CI performance checks for end-to-end timings with generous thresholds.