I’m working on my first F# project in an attempt to learn the language. My application involves making calls to a data api to retrieve batches of data. I’m allowed to make up to 10 concurrent calls to the API at a time. I’ve built a crude F# program which processes multiple batches of parallel requests but the code is more verbose than I would like, and I feel there must be a better way. My main program basically does the following:
// Create an async workflow which retrieves data from an API and processes the result.
let makeRequest offset = async {
let! result = // async API request
… process result and post to a mailbox processor …
}
// Define a bunch of batches that will be sent to the API. The requests in each batch can be run in parallel, but the batches themselves must be run sequentially. Each batch retrieves a window of results starting at some offset.
let batch1 = [0…1000…6000] |> List.map makeRequest
let batch2 = [7000…1000…12000] |> List.map makeRequest
…
let batchN = [nn…1000…mm] |> List.map makeRequest
// Take a batch of requests and run them in parallel
let processBatch batch = async {
return! batch |> Async.Parallel
}
// Manually run each of the defined batches in sequence, waiting for the previous to finish
let readData = async {
let! responses1 = processBatch batch1
let! responses2 = processBatch batch2
…
return ()
}
// Execute the program
readData
|> Async.RunSynchronously
|> ignore
Conceptually what I want to do is create a list of batches and iterate over them sequentially. Ideally I would create the list of batches from a function that would define the total number of results expected, the maximum number of batches that can be run in parallel, and the size of the result of each individual batch.
I feel like there must be a clean way to do this, but my inexperience with the language is creating a mental block at the moment.
Any help would be appreciated.
Thanks!