Using the Expand Operator to Fetch Recursively

The last key feature we need to implement is to actually make use of the perPage value supplied by the user, but there is some hidden complexity here. Let’s consider how we would actually implement this.

At the moment, we fetch GIFs from this URL:

        `https://www.reddit.com/r/${subreddit}/${sort}/.json?limit=100` +
          (after ? `&after=${after}` : '')

Notice that we supply a limit=100 to tell the Reddit API how many posts we want to return. It then might seem sensible to implement a perPage limit like this:

        `https://www.reddit.com/r/${subreddit}/${sort}/.json?limit=${perPage}` +
          (after ? `&after=${after}` : '')

However, that won’t quite work for us. At the moment this is what we do:

1. Fetch 100 posts from Reddit
2. Filter those to only include valid GIFs
3. Display those GIFs

The actual number of GIFs we end up displaying in our application won’t necessarily be 100 or whatever limit we supply, it could be anywhere from 0 to 100. This is where things get tricky.

We will continue to discuss this as we implement the solution, but the general idea that we want to implement is (assuming a perPage value of 10):

1. Fetch 100 posts from Reddit
2. Filter those for valid GIFs
3. If there are more than 10 valid GIFs, trim them to just 10
4. If there are less than 10 valid GIFs, keep hitting the API until we do have 10 valid GIFs

The tricky part is that last step, and it is where the expand operator will become extremely useful.

Update the fetchFromReddit method

Update the fetchFromReddit method to reflect the following:

  private fetchFromReddit(
    subreddit: string,
    sort: string,
    after: string | null,
    gifsRequired: number
  ) {
    return this.http
      .get<RedditResponse>(
        `https://www.reddit.com/r/${subreddit}/${sort}/.json?limit=100` +
          (after ? `&after=${after}` : '')
      )
      .pipe(
        // If there is an error, just return an empty observable
        // This prevents the stream from breaking
        catchError(() => EMPTY),

        // Convert response into the gif format we need
        // AND keep track of how many gifs we want from the API
        map((res) => ({
          gifs: this.convertRedditPostsToGifs(res.data.children),
          gifsRequired,
        }))
      );
  }

We have added one additional parameter here to support the functionality we are about to add. We now pass a gifsRequired value to indicate how many GIFs we need to return. For example, if our perPage is 10 and it is the first request the gifsRequired would be 10. However, if we only got 3 on the first request and we are now making our second request, the gifsRequired would be 7.

An important thing to note here is that we don’t actually use the gifsRequired value in fetchFromReddit. We are just making sure that value is available in the stream that is returned and we will make use of it later. The fetchFromReddit method will now return a stream that will emit an object containing two properties:

• gifs that will contain all of the GIFs returned
• gifsRequired which will be out gifsRequired value

Implement the expand operator

Ok, time for the big one. Have a read through of this code but don’t worry if you can’t make much sense of it, we will talk through the important parts below.

Modify the combineLatest stream in the getGifs method to reflect the following:

    return combineLatest([subreddit$, this.settings$]).pipe(
      switchMap(([subreddit, settings]) => {
        // Fetch Gifs
        const gifsForCurrentPage$ = this.pagination$.pipe(
          concatMap((pagination) =>
            this.fetchFromReddit(
              subreddit,
              settings.sort,
              pagination.after,
              settings.perPage
            ).pipe(
              // Keep retrying until we have enough valid gifs to fill a page
              // 'expand' will keep repeating itself as long as it returns
              // a non-empty observable
              expand((res, index) => {
                const validGifs = res.gifs.filter((gif) => gif.src !== null);
                const gifsRequired = res.gifsRequired - validGifs.length;
                const maxAttempts = 10;

                // Keep trying if all criteria is met
                // - we need more gifs to fill the page
                // - we got at least one gif back from the API
                // - we haven't exceeded the max retries
                const shouldKeepTrying =
                  gifsRequired > 0 && res.gifs.length && index < maxAttempts;

                if (!shouldKeepTrying) {
                  pagination.infiniteScroll?.complete();
                }

                return shouldKeepTrying
                  ? this.fetchFromReddit(
                      subreddit,
                      settings.sort,
                      res.gifs[res.gifs.length - 1].name,
                      gifsRequired
                    )
                  : EMPTY; // Return an empty observable to stop retrying
              })
            )
          ),
          // Filter out any gifs without a src, and don't return more than the amount required
          // NOTE: Even though expand will keep repeating, each result of expand will be passed
          // here immediately without waiting for all expand calls to complete
          map((res) =>
            res.gifs
              .filter((gif) => gif.src !== null)
              .slice(0, res.gifsRequired)
          )
        );

        // Every time we get a new batch of gifs, add it to the cached gifs
        const allGifs$ = gifsForCurrentPage$.pipe(
          scan((previousGifs, currentGifs) => [...previousGifs, ...currentGifs])
        );

        return allGifs$;
      })
    );