Simple Javascript Promise pool

Build one for yourself instead of using a library

The problem

Recently at work, I received a task to collect data using our REST APIs. I received a list of product IDs and I had to fetch the detailes from an endpoint. After that, I needed to pick the category field and invoke another API to receive the cost center. Very simple task.

I implemented my first solution, but I needed to improve it over time. I tried the following approaches:

• Wait for each request to finish: very simple solution but slow
• Use Promise.all for asynchronous processing: very fast but it may overflow the server with requests and that could cause timeouts
• Use a pool of Promises: the most efficient way as it guarantees the number of concurrent requests to the server

Using a library

Naturally, I went googling for a suitable library. There are many options to choose from but not one solution seemed to be simple enough for my needs.

A few libraries I found that other people seem to be using:

• ES6 Promise Pool: https://github.com/timdp/es6-promise-pool/blob/master/es6-promise-pool.js
• Supercharge Promise pool: https://superchargejs.com/docs/master/promise-pool
• Bottleneck: https://github.com/SGrondin/bottleneck
• Node rate limiter: https://github.com/jhurliman/node-rate-limiter

Implement it yourself

It turns out that implementing it yourself is not that hard. You just need to use an appropriate data structure to model your pool. In my case, I used a simple object, where each key would be an ID to process and the value would be the Promise fulfilling that request.

The other catch was to use Promise.race() to find the first Promise that has finished.