CodingSaves ~2 hours

Write a data pipeline with validation

Build a robust, observable data pipeline with per-record error handling and dead-letter logging.

The prompt

You are a data engineer building a batch processing pipeline. Write a [LANGUAGE] data pipeline that processes the following input data through multiple transformation stages. The pipeline must: read from [SOURCE e.g. CSV file / S3 / database / API], validate each record against a schema at ingestion (reject invalid records to a dead-letter sink), apply transformations in configurable stages, handle errors per-record without stopping the pipeline, output successful records to [DESTINATION], output failed records with error reason to [ERROR SINK], emit metrics (records processed, failed, throughput), support checkpoint/resume for large datasets, and be testable by injecting mock sources and sinks.

Input format: [FORMAT]
Transformations: [LIST TRANSFORMATION STEPS]
Expected volume: [RECORDS PER RUN]
Language: [LANGUAGE]

Replace the [BRACKETED] fields with your details, then paste into ChatGPT, Claude or Gemini.

Want AI to fill this in for you?

Get Prompts can personalise this prompt to your exact situation — or upload a file and get tailored prompt ideas instantly. 3 free edits, no sign-up.

Try it free →

More coding prompts