Simplify Asynchronous Control Flow in Javascript - Part 1

In the coming weeks, Rakit will be offering trainings on various development courses, including fundamental topics on languages like Javascript. We'll be writing series of blog posts to complement the trainings, such as this one. We are developers ourselves, actively engaging in various projects. We will share our techniques and tools based on our experience.

This post is meant to help Javascript newcomers understanding asynchronous workflow and keeping such code manageable and readable. Consider the following example to asynchronously download the latest movie titles at every hour. And because this hypothetical movieApi is inefficient, you would then need to download each synopsis individually!. After each pair of title and synopsis is downloaded, it will be rendered into the UI.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
// 3rd party API client interface, see at the bottom of this post for stub implementation
var movieApi = {
  // Returns an array of titles
  fetchTitles: function (cb) { /*implementation*/ },
  // Returns a string
  fetchSynopsis: function (title, cb) { /*implementation*/ }
}

// Our code
var interval = 60 * 60 * 1000;
setInterval(function () {
  movieApi.fetchTitles(function (err, titles) {
    notify('Fetching latest titles')

    if (err) {
      notify('>> ERROR fetching titles!')
    } else {
      var fetchSynopsisCount = 0
      var synopsisFetched = 0
      var totalTitles = titles.length

      titles.forEach(function (title) {
        movieApi.fetchSynopsis(title, function (err, synopsis) {
          ++fetchSynopsisCount

          if (err) {
            notify('>> ERROR fetching synopsis!. Reason: ' + err.message)
          } else {
            notify('Fetched synopsis ' + synopsis)
            // Update UI
            renderListItem(title, synopsis)
            ++synopsisFetched

            if (totalTitles === fetchSynopsisCount) {
              if (totalTitles === synopsisFetched) {
                notify('All synopsis fetch!')
              } else {
                notify('Could not fetched some synopsis!')
              }
            }
          }
        })
      })
    }
  })
}, interval)

// Example output when no errors:
// ------------------------------
// Fetching latest titles
// Fetched synopsis I – The Phantom Menace (1999) jeng jeng jeng jeng
// Fetched synopsis II – Attack of the Clones (2002) jeng jeng jeng jeng
// Fetched synopsis V – The Empire Strikes Back (1980) jeng jeng jeng jeng
// Fetched synopsis IV – A New Hope (1977) jeng jeng jeng jeng
// Fetched synopsis III – Revenge of the Sith (2005) jeng jeng jeng jeng
// Fetched synopsis VI – Return of the Jedi (1983) jeng jeng jeng jeng
// Fetched synopsis VII – The Force Awakens (2015) jeng jeng jeng jeng
// All synopsis fetch!

// Example output when error occurs:
// ---------------------------------
// Fetching latest titles
// Fetched synopsis I – The Phantom Menace (1999) jeng jeng jeng jeng
// Fetched synopsis VI – Return of the Jedi (1983) jeng jeng jeng jeng
// Fetched synopsis VII – The Force Awakens (2015) jeng jeng jeng jeng
// Fetched synopsis IV – A New Hope (1977) jeng jeng jeng jeng
// >> ERROR fetching synopsis!. Reason: VIII – The Last Jedi (2017) is in the making!
// Fetched synopsis III – Revenge of the Sith (2005) jeng jeng jeng jeng
// Fetched synopsis V – The Empire Strikes Back (1980) jeng jeng jeng jeng
// Fetched synopsis II – Attack of the Clones (2002) jeng jeng jeng jeng
// Could not fetched some synopsis!

I think you can pretty much tell at a first glance, that is one hell of a mess. There are only three asynchronous operations in the code: setInterval(), movieApi.fetchTitles() and movieApi.fetchSynopsis, yet they are enough obscure what the whole thing actually does. Let’s identify some problems with the code:

  1. It is absolutely unclear as to what the code does.
  2. setInterval invokes movieApi.fetchTitles at intervals without waiting previous invocations to finish. It is unlikely, but more than one movieApi.fetchTitles could be running at the same time.
  3. movieApi.fetchSynopsis is called immediately one after another in the forEach block. The order of the result is undetermined as in the example output.

Let’s start refactoring the code with #3. It is the most inner function so it makes sense to take it out first. First let’s make movie.fetchSynopsis execute in series:

1
2
3
4
5
6
7
8
9
function fetchEachSynopsis (titles, synopsisCb, doneCb) {
  var title = titles.shift()
  if (!title) return doneCb()

  movieApi.fetchSynopsis(title, function (err, synopsis) {
    synopsisCb(err, title, synopsis)
    setTimeout(fetchEachSynopsis.bind(null, titles, synopsisCb, doneCb), 0)
  })
}

The function now takes two callbacks. synopsisCb is called whenever fetching synopsis is done (regardless of error). doneCb is called when there is nothing more to fetch. You can think of synopsisCb as a block in a for-each loop for example, and doneCb is called after that block.

Take note in line 3, where I use early return to avoid indenting if-else statements. This technique is especially useful in languages like Javascript as we tend to use lots of callbacks, hence nested blocks.

Another thing to notice is how the function does things is absolutely implementation detail, and the fact it calls itself recursively (to serially fetching synopsis) should not matter to its client. I use recursive function to iterate through titles, shifting it after in each iteration. And if you wonder why I wrap fetchEachSynopsis call in with setTimeout in line 7, check out this stackoverflow question.

Let’s introduce another function, fetchEachMovie:

1
2
3
4
5
6
function fetchEachMovie (movieCb, doneCb) {
  movieApi.fetchTitles(function (err, titles) {
    if (err) return doneCb(err)
    fetchEachSynopsis(titles, movieCb, doneCb)
  })
}

It also takes two callbacks with the same purposes like fetchEachSynopsis. Do you see the pattern now?. Technically, all movie titles are fetch at once but like I said, the implementation detail should not matter to its client.

The client can expect its movieCb callback to be called with a movie title and a synopsis (or error) for each movie.

And the final part, I call the function updateUIAtInterval because it does exactly that. It fetches a list of movies and their synopsis at intervals, and renders the UI individually. I also replace setInterval with setTimeout to fix #2. So at each interval, the whole process must finish before it can start again. The function is now the main entry point of the application.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
function updateUIAtInterval (interval) {
  notify('Fetching latest titles')
  fetchEachMovie(
    function each (err, title, synopsis) {
      if (err) return notify('Error fetching synopsis!. Reason: ' + err.message)
      notify('Fetched synopsis ' + synopsis)
      // Update UI
      renderListItem(title, synopsis)
    },
    function done (err) {
      notify('Done fetching all synopsis, error: ' + (err || 'none'))
      setTimeout(updateUIAtInterval.bind(null, interval), interval)
    }
  )
}

// Start the application
updateUIAtInterval(interval)

Doesn’t that look better now?

Recap

So, what to look for when refactoring asynchronous workflow?

  1. Use early return instead of if-else when possible to reduce indentations needed.
  2. Take out anonymous functions and give them descriptive names to describe what it does. How it does is implementation detail.
  3. Make sure not to break callback chain. Always call callbacks passed by the caller (see how fetchEachSynopsis and fetchEachMovie call doneCb).

Full code & stub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
// Stub Implementation
// ===================

function notify (msg) { console.log(msg) }
function renderListItem () { /* UI update goes here */ }

var movieApi = {
  titles: [
    'IV – A New Hope (1977)',
    'V – The Empire Strikes Back (1980)',
    'VI – Return of the Jedi (1983)',
    'I – The Phantom Menace (1999)',
    'II – Attack of the Clones (2002)',
    'III – Revenge of the Sith (2005)',
    'VII – The Force Awakens (2015)',
    'VIII – The Last Jedi (2017)'
  ],

  fetchTitles: function (cb) {
    // Pretending to download titles...
    var titles = this.titles
    setTimeout(function () {
      // Remove the last one 80% of the time, then shuffle the list a bit
      titles = titles.slice(0, titles.length - (Math.ceil(Math.random() * 9) > 7 ? 0 : 1))
      titles.sort(function () { return 1 - (Math.ceil(Math.random() * 3) - 1) })
      cb(null, titles)
    }, 1000)
  },

  fetchSynopsis: function (title, cb) {
    // Pretending to download the synopsis
    var titles = this.titles
    setTimeout(function () {
      if (title === titles[titles.length - 1]) {
        cb(new Error(title + ' is in the making!'))
      } else {
        cb(null, title + ' jeng'.repeat(4))
      }
    }, Math.random() * 200)
  }
}

// The application
// ===============

function fetchEachSynopsis (titles, synopsisCb, doneCb) {
  var title = titles.shift()
  if (!title) return doneCb()

  movieApi.fetchSynopsis(title, function (err, synopsis) {
    synopsisCb(err, title, synopsis)
    setTimeout(fetchEachSynopsis.bind(null, titles, synopsisCb, doneCb), 0)
  })
}

function fetchEachMovie (movieCb, doneCb) {
  movieApi.fetchTitles(function (err, titles) {
    if (err) return doneCb(err)
    fetchEachSynopsis(titles, movieCb, doneCb)
  })
}

function updateUIAtInterval (interval) {
  notify('Fetching latest titles')
  fetchEachMovie(
    function each (err, title, synopsis) {
      if (err) return notify('Error fetching synopsis!. Reason: ' + err.message)
      notify('Fetched synopsis ' + synopsis)
      // Update UI
      renderListItem(title, synopsis)
    },
    function done (err) {
      notify('Done fetching all synopsis, error: ' + (err || 'none'))
      setTimeout(updateUIAtInterval.bind(null, interval), interval)
    }
  )
}

// Start the program
var interval = 60 * 60 * 1000;
updateUIAtInterval(interval)

In the next part of this series, I’m thinking of refactoring fetchEachSynopsis to fetch synopsis in parallel. Probably we can use some notable Javascript workflow libraries to replace the current implementation as well!.