instance method `in_batches`

Ruby on Rails 7.0.10

Since v5.2.8.1

Available in: v5.2.8.1 v6.0.6 v6.1.7.10 v7.0.10 v7.1.6 v7.2.3 v8.0.4 v8.1.2

Compare with:

in_batches(of: 1000, start: nil, finish: nil, load: false, error_on_ignore: nil, order: :asc)

Yields ActiveRecord::Relation objects to work with a batch of records.

Person.where("age > 21").in_batches do |relation|
  relation.delete_all
  sleep(10) # Throttle the delete queries
end

If you do not provide a block to #in_batches, it will return a BatchEnumerator which is enumerable.

Person.in_batches.each_with_index do |relation, batch_index|
  puts "Processing relation ##{batch_index}"
  relation.delete_all
end

Examples of calling methods on the returned BatchEnumerator object:

Person.in_batches.delete_all
Person.in_batches.update_all(awesome: true)
Person.in_batches.each_record(&:party_all_night!)

Options

:of - Specifies the size of the batch. Defaults to 1000.
:load - Specifies if the relation should be loaded. Defaults to false.
:start - Specifies the primary key value to start from, inclusive of the value.
:finish - Specifies the primary key value to end at, inclusive of the value.
:error_on_ignore - Overrides the application config to specify if an error should be raised when an order is present in the relation.
:order - Specifies the primary key order (can be :asc or :desc). Defaults to :asc.

Limits are honored, and if present there is no requirement for the batch size, it can be less than, equal, or greater than the limit.

The options start and finish are especially useful if you want multiple workers dealing with the same processing queue. You can make worker 1 handle all the records between id 1 and 9999 and worker 2 handle from 10000 and beyond by setting the :start and :finish option on each worker.

# Let's process from record 10_000 on.
Person.in_batches(start: 10_000).update_all(awesome: true)

An example of calling where query method on the relation:

Person.in_batches.each do |relation|
  relation.update_all('age = age + 1')
  relation.where('age > 21').update_all(should_party: true)
  relation.where('age <= 21').delete_all
end

NOTE: If you are going to iterate through each record, you should call #each_record on the yielded BatchEnumerator:

Person.in_batches.each_record(&:party_all_night!)

NOTE: Order can be ascending (:asc) or descending (:desc). It is automatically set to ascending on the primary key (“id ASC”). This also means that this method only works when the primary key is orderable (e.g. an integer or string).

NOTE: By its nature, batch processing is subject to race conditions if other processes are modifying the database.

Parameters

of key = 1000
start key = nil
finish key = nil
load key = false
error_on_ignore key = nil
order key = :asc

Source

# File activerecord/lib/active_record/relation/batches.rb, line 204
    def in_batches(of: 1000, start: nil, finish: nil, load: false, error_on_ignore: nil, order: :asc)
      relation = self
      unless block_given?
        return BatchEnumerator.new(of: of, start: start, finish: finish, relation: self)
      end

      unless [:asc, :desc].include?(order)
        raise ArgumentError, ":order must be :asc or :desc, got #{order.inspect}"
      end

      if arel.orders.present?
        act_on_ignored_order(error_on_ignore)
      end

      batch_limit = of
      if limit_value
        remaining   = limit_value
        batch_limit = remaining if remaining < batch_limit
      end

      relation = relation.reorder(batch_order(order)).limit(batch_limit)
      relation = apply_limits(relation, start, finish, order)
      relation.skip_query_cache! # Retaining the results in the query cache would undermine the point of batching
      batch_relation = relation

      loop do
        if load
          records = batch_relation.records
          ids = records.map(&:id)
          yielded_relation = where(primary_key => ids)
          yielded_relation.load_records(records)
        else
          ids = batch_relation.pluck(primary_key)
          yielded_relation = where(primary_key => ids)
        end

        break if ids.empty?

        primary_key_offset = ids.last
        raise ArgumentError.new("Primary key not included in the custom select clause") unless primary_key_offset

        yield yielded_relation

        break if ids.length < batch_limit

        if limit_value
          remaining -= ids.length

          if remaining == 0
            # Saves a useless iteration when the limit is a multiple of the
            # batch size.
            break
          elsif remaining < batch_limit
            relation = relation.limit(remaining)
          end
        end

        batch_relation = relation.where(
          predicate_builder[primary_key, primary_key_offset, order == :desc ? :lt : :gt]
        )
      end
    end

Defined in activerecord/lib/active_record/relation/batches.rb line 204 · View on GitHub · Improve this page · Find usages on GitHub

Defined in ActiveRecord::Batches

Signature

Options

Parameters