A Key Pipeline Setting – Batch Timeout In Minutes

Recently I came across an issue with Pipelines where if a job was taking longer than around 30 mins in a pipeline, the pipeline would move onto the next job before letting the job finish. The job would also show as "RUNNING" in the pipeline email notification, and the email would say the pipeline has failed, even though when we check the job console, none of the jobs in the pipeline had failed.

For example below, the pipeline moves on even though the aggregation is still running which then means the remaining jobs are not running in the correct order and the pipeline status shows as FAILED:

After asking on Customer Connect, a helpful user (thank you Ija Vidaurri!) gave us the solution, which is to set the Batch Timeout In Minutes setting to a higher value (by default this is empty so I’m not sure of the exact default timeout).

This can be done by going to Data Exchange -> System Settings and entering a minute value into the Batch Timeout In Minutes setting. In our case, we set it to 120 minutes, to ensure all jobs have enough time to finish before the pipeline moves on:

Hopefully this will help if you’re experiencing a similar issue! Going forward I’d recommend updating this setting whenever implementing Pipelines, to avoid any potential problems with timeout.