Azure Durable Functions: Fan-Out/Fan-In Pattern
When working with Azure Functions, one common scenario is importing a file for processing. In our specific case, we encounter an Excel file with thousands of lines that need to be processed, checking the accuracy of values using multiple rules and data sources, followed by sending out notifications. Initially, with a small number of lines, the processing time was acceptable. However, as the size of the Excel file grew, reaching tens of thousands of lines, the processing time became unbearable. In order to overcome this challenge, we will explore the capabilities of Durable Functions and leverage their power to address this issue effectively.
The “fan-out/fan-in” pattern is a technique used to parallelize and consolidate the execution of multiple tasks. During the fan-out phase, tasks are initiated in parallel to perform different activities concurrently. Then, during the fan-in phase, the function waits for all the parallel tasks to complete and combines their results. This pattern allows for efficient parallel execution, reduces processing time, and enables the consolidation of data or results obtained from parallel tasks. It is a scalable approach that maximizes resource utilization and enhances the overall efficiency of Durable Azure Functions.
[FunctionName(“ImportExcel”)]
public async Task ImportExcel(
[DurableClient] IDurableOrchestrationClient orchestrationClient,
[BlobTrigger(“storage/{name}”)]
Stream file,
string name,
IDictionary metaData,
ILogger log)
{
var data = await ParseExcel(file);
var dataToBeProcessed = new DataToBeProcessed { Data = data, FileName = name, Email = metaData[“email”]};
var id = await orchestrationClient.StartNewAsync(“DataOrchestrator”, dataToBeProcessed);
log.LogInformation(“End ImportExcel. Orchestrator instance id: {Id}”, id);
}
ProcessData([ActivityTrigger] PartialData inputs…)