
In this post I would like to share my recent experience of working with DataLoader, with a step-by-step example.

I actually have been thinking about writing an article about DataLoader for few years, since I first tried it. I even had [a talk](https://youtu.be/Iy1DUcBjXcQ) (in russian) about it once.
And finally I have some really simple and yet meaningful example of its usage outside of GraphQL ecosystem.

## Problem space

An app I have recently been working on was a _contentful_ extension which gathered some metrics about content usage/completeness.
One particular feature was to find links to the pages from other ones. E.g. we wanted to know how many offer pages are associated with a city page.

To give you an idea, _contentful_ has a very powerful mechanism to build site hierarchy (not only) - reference fields. Where one entry can have link/s to others.
With a help of it we've been able to create structure like this:

TODO

On which we heavily rely to automate a lot of content sections throughout our website: main navigation, breadcrumbs, thumbnails with parent/child/sibling pages and basically the whole gatsby (yep, we use SSG) build process relies on it.

## Limitation of existing API

And obviously we'd like to use this info to gather our metrics, moreover _contentful_ provides a way to do so - [links_to_entry](https://www.contentful.com/developers/docs/references/content-delivery-api/#/reference/search-parameters/links-to-entry) query param.
It returns a list of entries which refer to the one you are interested in. E.g. using js sdk:

```js
let { items: entries } = await space.getEntries({
  links_to_entry: 'id-of-our-city',
})
```

But there are few issues with this approach:

- it works for 1 entry, we can't get links to multiple cities;
- _contentful_ docs suggest: "it is more performant to query based on a content type's reference field";

And in my case, I really need to work with hundreds of pages, so both of them sound rather concerning.
Here is an example:

```js
let cityIds = [1, 2, 3, ...hundredMore]

// make 103 parallel heavy requests
// at least contentful sdk handles rate limit responses for us (retries requests)
let citiesWithLinks = await Promise.all(
  cityIds.map((id) => space.getEntries({ links_to_entry: id }))
)
```

Luckily _contentful_ provides an alternative:

```js
space.getEntries({
  // now we need to know
  content_type: 'offer', // referencing content type
  'fields.city.sys.id': 42, // and field
})
```

It also allows using operators, so we actually can make a single request for all cities at once:

```js
let offersReferencingToAllCities = space.getEntries({
  content_type: 'offer',
  'fields.city.sys.id[in]': cityIds,
})
```

Now we have to group this flat array of offers by the city they belong to:

```js
// lodash have a handy method for that
let offersByCityId = _.groupBy(
  offersReferencingToAllCities,
  (offer) => offer.fields.city.sys.id
)
```

Finally, we can get same result we had in the beginning with fewer (just 1 in our case) api calls. Here is full example:

```js
let cityIds = [1, 2, 3, ...hundredMore]

let offersReferencingToAllCities = space.getEntries({
  content_type: 'offer',
  'fields.city.sys.id[in]': cityIds,
})

let offersByCityId = _.groupBy(
  offersReferencingToAllCities,
  (offer) => offer.fields.city.sys.id
)

// ta-da!
cityIds.map((id) => offersByCityId[id] ?? [])
```

Well great, but it's it confusing that we achieved what we wanted to, but didn't use _DataLoader_ as I promised?

Right, there is a couple of issues with our final solution actually:

- it is obviously more complicated than the one we started with;
- contentful has a limit on number of returned entries, so it is very likely that we'd wanna batch in a smart way (and complicate solution even further);
- our abstractions not always allow us to operate on all entities, so that we can batch calls explicitly (e.g. graphql resolvers);

So wouldn't it be cool if we had an api similar to our initial solution, and performance of the last one? Easy! That's what we need _DataLoader_ for!

```js
/**
 * We'll have a data loader per content type and field,
 * cause that's what contentful allows us to batch
 */
let createLinksLoader = (contentType, field) =>
  // we should provide a batch loader
  // which gets an array of keys (in our case entry ids)
  // and returns array of resuls, respecting the original order
  new DataLoader(async (ids) => {
    // pretty much same code we had,
    // except now it is more generic/reusable
    let referencingEntries = await space.getEntries({
      content_type: contentType,
      [`fields.${field}.sys.id[in]`]: cityIds,
    })

    let groupedEntries = _.groupBy(
      referencingEntries,
      (offer) => offer.fields[field].sys.id
    )

    return ids.map((id) => groupedEntries[id] ?? [])
  })
```

And now we can use our loader:

```js
let cityIds = [1, 2, 3, ...hundredMore]

let offerToCityLinksLoader = createLinksLoader('offer', 'city')

let citiesWithLinks = await Promise.all(
  cityIds.map((id) => offerToCityLinksLoader.load(id))
)
```

Neat, isn't it? Now we have a performant, reusable links loader and even more:

- _DataLoader_ also deduplicate and cache calls;
- with a help of `maxBatchSize` option, we can prevent hitting the limit for number of returned entries I mentioned above;

Hope you enjoyed the article!


Power of DataLoader

In this article I will tell how to build "Related Reads" section for a [gatsby](https://www.gatsbyjs.org/) blog, like the one you can find at the bottom of this page.

As a prerequisite I expect, that you are already familiar with gatsby or even have an app with some source `[-plugin]`where you pull your articles from.

I use [netlify cms](https://www.netlifycms.org/) and store all my articles next to the source code [in git](https://github.com/kitos/kitos.github.io/tree/develop/src/_content/blog) as markdown files. So `gatsby-source-filesystem` and `gatsby-transformer-remark` plugins are responsible for making them available in graphql. E.g. here is how I can query all articles:

```graphql
query BlogPostList {
  allMarkdownRemark {
    nodes {
      frontmatter {
        title
      }
    }
  }
}
```

But enough about me, let's get back to business.

## Setup local plugin

First of all, I want to implement this feature as a local plugin (instead of root `gatsby-node.js`). I find this approach more idiomatic and clear. Creating one is a piece of cake and gatsby has [a decent documentation about it](https://www.gatsbyjs.org/docs/creating-a-local-plugin/), so check it out if you haven't yet.

As a result, we should have `/plugins/related-reads/gatsby-node.js` along with empty `package.json` and name (equal to name of the folder!) of our plugin listed in `gatsby-config.js`.

## Extend graphql

The next step would be extending graphql schema, generated by `gatsby-transformer-remark`. So along with a content of an article, we can also request a list of articles about similar topic[s]. Like this:

```graphql
query BlogPostList {
  allMarkdownRemark {
    nodes {
      frontmatter {
        title
      }

      // highlight-start
      relatedReads {
        slug
        title
      }
      // highlight-end
    }
  }
}
```

[A while ego](https://www.gatsbyjs.org/blog/2019-03-04-new-schema-customization/) gatsby team provided very nice way to achieve this - [schema customization API](https://www.gatsbyjs.org/docs/schema-customization). And we are going to leverage it, particularly `createResolvers`:

```js
exports.createResolvers = ({ createResolvers }) =>
  createResolvers({
    // node type we want to extend
    MarkdownRemark: {
      // name of the field we want to add
      relatedReads: {
        // we return an array of other articles
        type: '[MarkdownRemark!]',
        // let's allow to specify number of articles returned
        args: { limit: 'Int' },

        async resolve(
          source, // the node we extend
          args,
          // collection of helpers
          // including access to internal store - nodeModel
          ctx

        ) {
          return []
        },
      },
    },
  })
```

No we can implement the last bit - the logic of _Relative Reads_. In my blog each post have a list of tags, so I gonna use it to measure similarity.

```js
const intersection = require('lodash.intersection')

// I've extracted the resolve function just to get rid of extra noise
let resolve = async (source, args, ctx) {
  let { frontmatter: { slug, lang, tags } } = source
  let { limit = Number.MAX_SAFE_INTEGER } = args

  // request all other posts
  let otherPosts = await ctx.nodeModel.runQuery({
    firstOnly: false, // we want to get an array
    type: 'MarkdownRemark',
    query: {
      filter: {
        frontmatter: {
          slug: { ne: slug }, // not current article or translation
          lang: { eq: lang }, // same lang
        },
      },
    },
  })

  return otherPosts
    .map((p) => ({
      ...p,
      // measure the similarity by the size of tags intersection
      similarity: intersection(p.frontmatter.tags, tags).length,
    }))
    .filter(({ similarity }) => similarity !== 0)
    .sort(
      (a, b) =>
        b.similarity - a.similarity ||
        b.frontmatter.date.localeCompare(a.frontmatter.date)
    )
    .slice(0, limit)
}
```

That is it! Pretty easy implementation and zero runtime cost - everything is requested ahead of time and build into your html pages, that is the strongest point if gatsby and JAMStack.

Did you like this post? Want to hear more stories about gatby? Please leave comment about things you'd like to know about.


Build "Related Reads" for your gatsby blog

It is hard to underestimate an importance  of [AST](https://en.wikipedia.org/wiki/Abstract_syntax_tree) transformation tools in the frontend now:

* we use `babel`

  * to transform the most recent *ECMAScript* versions to work with older versions of browsers
  * to [optimize our code](https://github.com/jamiebuilds/babel-react-optimize)
  * to get [new features](https://emotion.sh/docs/css-prop)
* we use `eslint` to enforce best practices or avoid mistakes/bugs
* we use `prettier` to format the code

But today I wanna tell a short story about using *AST* transformation to refactor the code. Like some codemods you might used.

## Backstory

At [tourlane](https://www.tourlane.de/) we use [styled-components](https://styled-components.com/) (probably, the most popular *CSS-IN-JS* solution in react) along with [styled-system](https://styled-system.com/) (awesome utility belt for writing responsive styles based on scales from global theme), so we can build components like this:

```jsx
let User = ({ avatar, name }) => (
  <Flex flexDirection={['column', 'row']}>
    <Box as="img" mb={[1, 0]} mr={[0, 1]} src={avatar} />
    <Box p={1}>{name}</Box>
  </Flex>
)
```

While some of you might find this style of writing components pretty controversial, this is not the point of this article. But I still can recommend a couple articles to [](https://jxnblk.com/blog/two-steps-forward/)get more reasoning behind it:

* [Two steps forward, one step back](https://jxnblk.com/blog/two-steps-forward/)
* [Styles and Naming](<* [https://www.christopherbiscardi.com/post/styles-and-naming](https://www.christopherbiscardi.com/post/styles-and-naming/)/>)
* [Old and new ideas in React UI](https://react-ui.dev/core-concepts/ideas)

The point of the code block from above is the usage of responsive CSS values: `jsx±flexDirection={['column', 'row']}`(pretty handy isn't it?). Under the hood it will use media breakpoints provided in the theme to compile the responsive styles like:

```css
.some-generated-class {
  flex-direction: column;
}

@media screen and (min-width: 64em) {
  .some-generated-class {
    flex-direction: row;
  }
}
```

## The problem

When we started one of our projects we didn't have any styles specific to tablets, let's say we had something similar to this:

```jsx
let breakpoints = [
  // no need to define lower breakpoint
  // since we use mobile-first
  // (styles applied without media are considered to be mobile)
  
  '64em', // desktop
  '80em', // wide
]

let Layout = ({ children }) => (
  <ThemeProvider theme={{ breakpoints }}>
    <Header />
    {children}
    <Footer />
  </ThemeProvider>
)
```

But lately we got a new page where the mobile and table designs are quite different. But our scale system doesn't support it 😧: we can not use responsive attributes that we are used to 😨. Adding one more breakpoint will break all existing components 😰.

I bet you've been in a situation like this. E.g. you might used some library api which was deprecated. Luckily in cases like this, library authors usually prepare codemods which will update your codebase for you. So you don't even have to understand what it does under the hood.

But in our case there was no one to rely on, nobody to blame... but... me? 🥺

## jscodeshift to the rescue

Even though we could write tablet specific styles *"by hand"* - using *normal* `styled-component`s syntax. It wasn't the way we wanted to proceed - we really like the api provided by `styled-system`, moreover we didn't want to introduce  ambiguity in writing responsive styles. And obviously we weren’t too keen to go through the whole codebase and change all usages of responsive props.

Sounds like a perfect task for codemode!

I have never worked with them, but I've heard that [jscodeshift](https://github.com/facebook/jscodeshift) is a cool tool to build them, e.g. react team [uses it.](https://github.com/reactjs/react-codemod) All you have to do is create `js/ts` file and export the *transform* function which will take care of required AST transformations.

Another important part is, of course, a general understanding of AST: what it is, what it consists of and how we can alter it. I found this [babel handbook](https://github.com/jamiebuilds/babel-handbook) to be super useful intro. Another must-have tool is [AST explorer](https://astexplorer.net/). With help of this tool you can write some code to see its AST representation and write transform functions with immediate results!

Also I can hardly imagine writhing AST transformation without `typescript` (you can get info about nodes and their properties, right during typing intead of switching between editor and bable docs all the time), so I installed its typings (`@types/jscodeshift`) along with a library itself.

And after about an hour of playing with it I built this:

```typescript
import { Transform } from 'jscodeshift';

let directions = ['', 't', 'r', 'b', 'l', 'x', 'y'];
// these are responsive attrs provided by styled-system
let spaceAttributes = [
  ...directions.map(d => `m${d}`),
  ...directions.map(d => `p${d}`),
  'flexDirection', 'justifyContent', // ...
];

let transform: Transform = (fileInfo, { j }) =>
  j(fileInfo.source)
    .find(j.JSXAttribute)
    .forEach(({ node }) => {
      let attributeName =
        typeof node.name.name === 'string' ? node.name.name : `¯\\_(ツ)_/¯`;
      let { value } = node;

      if (
        spaceAttributes.includes(attributeName) &&
        // we are interested only array expressions which has more then 1 value
        // like <Box m={[8, 16, 24]}>
        value?.type === 'JSXExpressionContainer' &&
        value.expression.type === 'ArrayExpression' &&
        value.expression.elements.length > 1
      ) {
        let [xs, ...otherMedias] = value.expression.elements;

        // null in styled-system means - do not introduce new media query,
        // so thanks to mobile first approach we'll have values defined in xs.
        // I could also write [xs, xs, ...otherMedias],
        // it will just result in bigger css output
        value.expression.elements = [xs, j.identifier('null'), ...otherMedias];
      }
    })
    .toSource();

export default transform;
```

You can also play with live example in [AST explorer](https://astexplorer.net/#/gist/d76e9a0c6e5f0cea12c039bc1b3f0d4c/4a5c9afcfda6967a4d0205ba8093e2b0c363ac4c).

As you can see the transform is not that big and yet very descriptive. Obviously it doesn't cover all possible cases, e.g. we could use ternary expressions in jsx attributes or use variables referring to arrays and etc. But it does explain the idea, at least I hope so.

The cool thing is you can run your codemode ➡️rollback using git ➡️improve 🔁1000 times, until you are happy with the result.

## Make codemods part of your toolbelt

That is all I wanted to share today. I hope after reading this article you will consider codemods to be not just powerful tool, but also a thing that is easy to learn, as I did. If you have any questions or suggestion feel free to use comments section or rich me directly in [twitter](https://twitter.com/kitos_kirsanov). Thanks!


Hi, I'm Nikita!

Recent posts