Improved handling of Async Tasks?

General
1

I’m working on my first F# project in an attempt to learn the language. My application involves making calls to a data api to retrieve batches of data. I’m allowed to make up to 10 concurrent calls to the API at a time. I’ve built a crude F# program which processes multiple batches of parallel requests but the code is more verbose than I would like, and I feel there must be a better way. My main program basically does the following:

// Create an async workflow which retrieves data from an API and processes the result.
let makeRequest offset = async {
let! result = // async API request
… process result and post to a mailbox processor …
}

// Define a bunch of batches that will be sent to the API. The requests in each batch can be run in parallel, but the batches themselves must be run sequentially. Each batch retrieves a window of results starting at some offset.

let batch1 = [0…1000…6000] |> List.map makeRequest
let batch2 = [7000…1000…12000] |> List.map makeRequest
…
let batchN = [nn…1000…mm] |> List.map makeRequest

// Take a batch of requests and run them in parallel
let processBatch batch = async {
return! batch |> Async.Parallel
}

// Manually run each of the defined batches in sequence, waiting for the previous to finish
let readData = async {
let! responses1 = processBatch batch1
let! responses2 = processBatch batch2
…
return ()
}

// Execute the program
readData
|> Async.RunSynchronously
|> ignore

Conceptually what I want to do is create a list of batches and iterate over them sequentially. Ideally I would create the list of batches from a function that would define the total number of results expected, the maximum number of batches that can be run in parallel, and the size of the result of each individual batch.

I feel like there must be a clean way to do this, but my inexperience with the language is creating a mental block at the moment.

Any help would be appreciated.

Thanks!

2

There are some stuff in FSharpX.Async: is this what you are after: http://fsprojects.github.io/FSharpx.Async/reference/fsharpx-control-async.html ?

3

Thanks, I hadn’t see that one. I found something else just today that works for me here - https://fsprojects.github.io/FSharpPlus/. I was able to use it like this:

let processAllBatches = async {
    let processBatch batch = async {
        return! batch |> Async.Parallel
    }
    return! batches 
            |> List.map processBatch
            |> Async.Sequence
}

processAllBatches
|> Async.RunSynchronously
|> ignore

I’ll also check out your reference since the parallel with throttling looks interesting.

Thanks!

4

F# 4.7 actually already has support for this - there’s an overload for Async.Parallel which allows you to specify the max degree of parallelism (in your case e.g. 10) BUT there is currently a bug in FSharp.Core that means it doesn’t work! There are a number of sample implementations that you could use until a new version of FSharp.Core is rolled out linked here: https://github.com/fsharp/fslang-suggestions/issues/706. However, the actual version of FSharp.Core that is ready to be released is more efficient than a single “batch” operation so once that’s out, I’d recommend moving to it.

5

Thanks! I’ll check it out.