Tentacle Labs Blog

I recently wrote about my indicators that I use while choosing dependencies in the NPM ecosystem. After writing the post I thought, can you automate these steps to provide a quick view on package health? One tool that does that is the Snyk Advisor. I tried the advisor and want to summarize my experience.

Snyk is a company that provides a dev-centric security platform. One of their products is the Snyk Advisor. The Snyk Advisor rates packages in the NPM ecosystem based on a package health score, giving an overview whether the package is a good choice. Besides NPM, they also support Docker Hub, PyPi, and plan to expand to other ecosystems.

Each NPM package is ranked with a package health score, a score between 0 and 100 points. The higher the score, the better. Ranking is based on four categories popularity, maintenance, security, and community — similar to the indicators I listed in my post. It’s not fully transparent how the score is calculated based on these categories, but the report lists some things that they take into account. In the popularity category, packages can range from “small” to “key ecosystem project” for packages that are downloaded and depend on a lot. The maintenance category seems to be based on issues, PRs, releases and commits, ranging in a ranking from “inactive” over “sustainable” and “healthy”, up to “maintained”. I’m actually not sure what the exact difference between these rankings is, the tool doesn’t make this clear. As being a primary focus of Snyk, the security category is sourced from other Snyk products, collecting all kinds of security and licensing relevant data, like CVEs. For the community category, a package can be ranked between “limited” and “active”, based on the project setup and number of maintainers.

Note that the following section describes the state at the time this post was written and might have changed. First, the tool looks great. Especially if you want to get a fast overview about a package, or to compare two packages. Let’s take a look at some packages with a similar feature set: classnames and clsx. Both packages can be used to conditionally combine CSS class names when working with react. When doing the audit yourself, there is a big tendency towards classnames being the better choice, as it has more stars, usage, more recent commits, releases, and way more contributors. The Snyk Advisor gives a similar picture, by ranking clsx with 70/100 points and classnames with 91/100 points. That looks like a great result.

But not everything is great with Snyk Advisor: It only analyzes the requested package, without the indirect dependencies. I prefer to assess the whole ecosystem of packages that a dependency brings. The dependency tree might contain non-obvious risks. Depending on the size of the ecosystem, doing this manual can be a lot of work. Also keep in mind that not only the dependencies, but also their version constraints should be checked. You don’t want to use a package that is still working with an old and unsupported indirect dependency. Here, an automated tool would be really useful. Let’s have a look at an example. I really like the idea of @react-pdf/renderer, a PDF renderer based on JSX. Using JSX to define the PDF documents gives a very sweet developer experience. The project is well maintained and Snyk gives it a matching score of 85/100. But it has some problematic dependencies like yoga-layout which are not that well maintained. Even worse, the dependency on Facebook’s yoga-layout isn’t detected here, because it’s using a pre-build versioning without having a direct dependency on it. But that is not something to blame Snyk Advisor for, no tool would be able to detect that. But let’s follow the dependency tree a bit more down. yoga-layout itself is rated as 60/100, but depends on packages like nbind which is unmaintained and has a rating of just 49/100. It doesn’t feel like that this situation is part of the rating of yoga-layout. Due to these dependencies, @react-pdf/renderer has issues with running without unsafe-eval which make the package way less attractive. But sadly, this situation will be hard to resolve. I would prefer that Snyk Advisor shows (an additional) ecosystem score, which is limited to the lowest rated indirect package. They should have all the data for this.

But there is another issue with Snyk Advisor. Let’s come back to nbind. It’s a package that I would consider as unmaintained and dead — nobody should use it. But it still gets a score of 49/100 points. It’s around half of the points, is that good or bad? I looked at multiple discontinued projects and most of them end up around this score. For some of them, Snyk Advisor even shows that they are archived on GitHub. I think the problem is the weighting of the categories. Sure, nbind has “no known security issues”, but for a project that has a “limited” usage, “limited” community, and which maintenance is “inactive” you probably don’t know about security issue. Which doesn’t mean that it might not have some. Who should spot them and where should they be reported to? I would expect this package to have score near to zero. Properly weighting the health score would give better results. As the rating is not transparent, it’s just a guess that the security category has a big influence on the result.

I think overall the Snyk Advisor is a nice start for evaluating a dependency. However, it shouldn’t be your only source. You should also have a look at the indirect dependencies, for example by running the advisor on each of them. Take the health score with care, I think it’s best to set a target score of more than 80, maybe even more than 90, for each dependency. But as always: It depends. Consider the risk that the new package introduces — is it a package that is easy to replace or maintain by yourself, or is it a central part of your architecture? If it is easy to replace, a score between 50 and 80 might still be acceptable. If it is a central framework that influences your whole architecture, go for a score of 80 or more. Snyk Advisor is still in beta, let’s see how it evolves in the future.