Many articles on implementing a loop in PDI suggest to put a ‘Wait For’ step and join the output hop back to a previous step.
Something as suggested by below screen shot:
But I would like to warn about this implementation of looping that this causes recursive stack allocation by JVM during job execution and the system may run out of memory after a high number of iterations (depending the system available available memory).
In a test run on my laptop in which I continuously ran the above job with a ‘Wait For’ configuration of 1 second, within minutes the JVM crashed. I also noticed a continuous increase in JVM memory usage before it crashed.
Therefore to conclude it is ok to run few iterations (like retrying on error condition) with above mechanism but make sure you do so for a finite number of iteration (may be lesser than 100), but not infinite looping.
So what is the suggested approach for infinite looping – One of the possible way is to use the settings of ‘Start’ step. Set the ‘Repeat’ flag and add interval configuration. This cause the job to be re-initialize completely as a new instance and does not cause any memory issue.
