I am implementing spring batch job for processing millions of records in a DB table using partition approach as follows -
Fetch a unique partitioning codes from table in a partitioner and set the same in execution context.
Create a chunk step with reader,processor and writer to process records based on particular partition code.
Is this approach is proper or is there any better approach for situation like this? As some partition codes can have more number of records than others it will more time to process.
Is it possible to create partition/thread to process like thread1 process 1-1000,thread2 process 1001-2000 etc ?
How do I control number of threads getting created as partition codes can be around 100, I would like to create only 20 thread and process in 5 iteration?
What happens if one partition fails, will all processing stop and reverted back?
Thanks
Aucun commentaire:
Enregistrer un commentaire