Quick Tip – C# Batch Parallel Processing

There’s a fair few creative solutions around for executing data processing tasks in parallel in SQL and SSIS. As good as they are they’re not really necessary if you use Parallel.ForEach in the C# library System.Threading.Tasks. This is a parallel loop that can be used to iterate items in a collection and run some code for each item. All the thread allocation and loop tracking control is handled internally without having to do anything.

The only other thing you could do is pass it a parameter to specify how many concurrent executions you want making it adjustable and easy to configure at runtime:

int maxP = 5;

Parallel.ForEach(
collectionOfProcesses,

new ParallelOptions { MaxDegreeOfParallelism = maxP},
(currentProcess) =>
{

//Excecute SSIS Package
//Execute SQL Proc
//Azure Data Lake Analytics Job

  Execute(currentProcess);
});