Building the DataSaudi ETL pipeline taught me more about data engineering in six months than any course could. Here's what actually matters when your pipeline needs to run reliably at scale.
The naive approach breaks fast
Everyone starts with something like this:
def import_records(records)
records.each do |record|
MyModel.create!(transform(record))
end
end
It works in development. It fails spectacularly in production.
External APIs timeout. Records arrive malformed. Your database has a bad moment. And now you have a half-imported dataset with no way to tell which half made it.
Idempotency is not optional
Every import operation needs to be safely re-runnable. If you import the same record twice, you should end up with one record — not two.
def upsert_record(external_id, attributes)
MyModel.upsert(
attributes.merge(external_id: external_id),
unique_by: :external_id,
update_only: [:name, :value, :updated_at]
)
end
With upsert, re-running the same import is safe. You can retry on failure without fear.
Background jobs with proper retry logic
Never run ETL synchronously. Use Sidekiq (or similar) with exponential backoff:
class ImportJob < ApplicationJob
retry_on StandardError, wait: :polynomially_longer, attempts: 5
discard_on ActiveRecord::RecordInvalid
def perform(batch_id)
batch = ImportBatch.find(batch_id)
batch.records.each { |r| process_record(r) }
batch.mark_complete!
end
end
The polynomially_longer wait strategy means transient failures (API timeouts, DB hiccups) heal themselves without manual intervention.
Monitoring that actually tells you something
A pipeline that silently fails is worse than one that crashes loudly. Track:
- Records ingested vs. expected
- Transformation error rate
- Job queue depth
- Time since last successful run
We pushed all of this to a custom admin dashboard. When the queue depth spiked, we knew immediately — usually before any user noticed.
What I'd do differently
More aggressive batching. Individual INSERT statements for every record will destroy your DB under load. Batch inserts with insert_all are an order of magnitude faster.
Schema validation at ingestion. By the time a malformed record hits your transformation logic, you've wasted compute. Validate at the boundary.
Separate concerns properly. extract, transform, and load should be distinct, testable units. If your transform logic is mixed into your load logic, debugging becomes archaeology.
This post is part of a series on production Rails patterns. Next: background job architectures and what "reliable" actually means.