Depp: A fast unused and duplicate dependency checker

Technical

11/3/2021

··

Rahul Tarak

Recently we ran into a couple of bugs on our main fairly large monorepo which were caused my the same package existing twice but having different versions. This made me search for some dependency checking tools, and while there are existing ones none of them really had good monorepo support.

So I decided to make one, this is a brief about how I made it and also how to use it.

If you want to just try the tool without hearing ramble about it, feel free to checkout the repo

Basic Usage and Features

Install

npm install -g depp-installer
depp --help  # Will should all avaliable options

Usage

Default Config

depp

Will check only typescript (.ts, .tsx) files in your root folder and all its children. It will also read the packages from then root package.json but also any package.json inside child folder (It supports mono repositories by default)

It will show unused packages, unused @type packages and duplicate packages with different versions

It will also generate a temporary html report file and open it in your browser. This file will look something like the following

Report for - /home/cryogenicplanet/modfy/editor

There are a total of 89 packages and 19 are the unused [] @babel/core [] @modfy/videocontext [] redux [] @visx/axis [] graphql-scalars [] graphql-fields [] mobx [] @magic-ext/oauth [] hash-stream-validation [] @sentry/node [] @visx/scale [] class-validator [] ws [] workbox-webpack-plugin [] worker-plugin [] graphql [] lodash.debounce [] magic-sdk [] apollo

https://cryogenicplanet.github.io/depp/static/htmlReport.html

Some Major Flags

--js Will enable checking js files

--dev Will enable checking dev dependencies (this is not very accurate)

--report Will save the report to .depp/report.md

--externals Can use this to external certain packages, useful if the build fails by default

--ignore-namespace Can ignore namspaced internal packages using thing, good for ignore @monorepo packages

--show-versions Will explicitly print the versions of duplicate packages in console

This is not an exhaustive list of all flags, for that run depp --help

Example advance usage is

depp  -e mobx -e magic-sdk -e domain -e @daybrush/utils -e yjs -e constants -e ws -v -in @editor -in @server -j -e perf_hooks --report

How it was built

Just briefly wanted to talk about how this built and how it works, of course the entire codebase is open-source so feel free to poke around yourself

So this package works by wrapping around the esbuild go api to add this extra functionality, so yes technically we built your whole code base to parse what dependencies you are using. This might sound incredibly stupid, but considering esbuild speed and flexibility, it is still significantly quicker than most solutions written in javascript

I realize this is not a great benchmark but considering most other tools don't have good monorepo support it was the best I can do. I compared depp (this tool) vs depcheck (a popular javascript tool with almost 3k stars) on a previous project https://github.com/wei/socialify

depp  0.42s user 0.36s system 135% cpu 0.576 total
depcheck  1.69s user 0.13s system 116% cpu 1.567 total

This tool was about 2.7x faster than depcheck while practically building the entire codebase

Using esbuild like this also allows me to build this tool, without writing an entire javascript/typescript parser

Setup

First we need to get the paths of all the package.json files and then all the source files,

I did this using something I built for an older project which is a modification of go standard library's Glob This modification allow me to easily find all nested package.json and source files

Now we can read the package.json files and get all the packages in the repository, we can store these as a hash map or dictionary to compare against in the future.

Finally, we can pass the source files to esbuild as entry points and have it do its magic

Callbacks

First we use Resolve and Load callbacks from the esbuild api to check for packages as they are being imported.

There are some caveats here tho, we don't want to go down a rabbit hole of every import. Any import from node_module we don't care about, these are generally nested imports of modules importing other modules. So we can mark these files as external

Another optimization we can add is, any non .ts, .tsx, .js,.jsx file can be completely ignored. This is a bit painful in go because it does not support regex lookaheads (?)

Now with any actual import, we can check the kind of resolve it is, whether it is a import-statement , require-call or something else we don't care about. Finally we can check if the imported path contains any of our packages

Metafile

Additionally we use the esbuild meta file to do a second pass on any import we might have missed or that slip through or resolve callbacks. The esbuild meta file contains a bunch of information about inputs and their imports which we can use to see if a given package has been used or not

Duplicates

While the process of finding unused packages is quite involved, finding duplicates is quite easy on the other hand. It is just about correctly inputting the parsed package.jsons and checking for duplicates and storing their versions. There is a bit more work for check unused @types packages but it mostly similar to duplicates

Not going to go into it much here, that said it probably constitutes its own post of how to bundle go binaries with an npm package and how to release them

All that said this was a fairly fun project to make in the 8 or so hours it took, and it has already been fairly useful internally. Hope you find some use in the tool itself or maybe how I built it