A Key Pipeline Setting – Batch Timeout In Minutes
Recently I came across an issue with Pipelines where if a job was taking longer than around 30 mins in a pipeline, the pipeline would move onto the next job before letting the job finish. The job would also show as "RUNNING" in the pipeline email notification, and the email would say the pipeline has failed, even though when we check the job console, none of the jobs in the pipeline had failed.
For example below, the pipeline moves on even though the aggregation is still running which then means the remaining jobs are not running in the correct order and the pipeline status shows as FAILED:
After asking on Customer Connect, a helpful user (thank you Ija Vidaurri!) gave us the solution, which is to set the Batch Timeout In Minutes setting to a higher value (by default this is empty so I’m not sure of the exact default timeout).
This can be done by going to Data Exchange -> System Settings and entering a minute value into the Batch Timeout In Minutes setting. In our case, we set it to 120 minutes, to ensure all jobs have enough time to finish before the pipeline moves on:
This comment has been removed by the author.
ReplyDeleteThis is extremely helpful. It's simple and direct, but I don't know if you have any idea how many people with Oracle on the community hemmed and hawed over what the problem might be when something was timing out. All that to say that it's hard to get direct and helpful information on any part of this system, so thanks!
ReplyDelete