Javascript

Web Scraping with TypeScript and Node.js

Apr 21, 2022

6 min read

Sometimes you'll find yourself wanting to use a set of data from a website, but the data won't be available as an API or in a downloadable format like CSV. In these cases, you may have to write a web scraper to download individual web pages and extract the data you want from within their HTML. This guide will teach you the basics of writing a web scraper using TypeScript and Node.js, and will note several of the obstacles you might encounter during web scraping.

If you want to skip straight to the finish code example, check it out on GitHub.

Setup

First things first: we need to initialize our project and install the base dependencies. We'll be writing our web scraper in TypeScript and running it as Node.js scripts using ts-node. For simplicity, we'll create an index.ts file in the project root to work from. From the command line, run the following to get started:

mkdir my-web-scraper && cd my-web-scraper # create project directory
git init # initialize new git repository
echo "node_modules" >> .gitignore # do not track node_modules in git
npm init -y # initialize Node.js project
# install dependencies
npm install typescript ts-node
npm install --save-dev @types/node
touch index.ts # create an empty TypeScript file

Node.js doesn't run TypeScript files natively. Rather than use the TypeScript compiler to output new JavaScript files whenever we want to run the script, we'll use ts-node to run the TypeScript files directly. We'll go ahead and add this to our new package.json file as an npm script.

  "scripts": {
    "scrape": "ts-node ./index.ts"
  }

Now, we'll be able to run our scraper from index.ts with the command npm run scrape.

Fetching Websites

In our examples, we'll be using Axios to make our http requests. If you'd prefer something else, like Node Fetch to match the Fetch API until it's ready in Node.js, that's fine too.

npm install axios

Let's create our first function for fetching a given URL and returning the HTML from that page.

import axios from 'axios';

function fetchPage(url: string): Promise<string | undefined> {
  const HTMLData = axios
    .get(url)
    .then(res => res.data)
    .catch((error: AxiosError) => {
      console.error(`There was an error with ${error.config.url}.`);
      console.error(error.toJSON());
    });

  return HTMLData;
}

This function will use Axios to create a promise to fetch a given URL, and return the HTML it gets back as a string. If there's an error, it will log that error to the console, and return undefined instead. Since you're probably going to be running this scraper from your command line throughout development, a healthy number of console.logs will help you make sure the script is running as expected.

Caching Scraped Pages

In the event that you're trying to scrape many, many static web pages in a single script, you might want to cache the pages locally as you download them. This will save you time and headache as you work on your scraper. You're much less likely to annoy the website you're scraping with high traffic and the bandwidth costs associated with it, and your scripts will probably run faster if they aren't limited by your Internet connection.

Let's go ahead and create a .cache folder in the project root. You probably won't want to keep cached files in your git history, so we'll want to add this folder to your .gitignore file.

mkdir .cache
echo ".cache" >> .gitignore

To cache our results, we'll first check if a cached version of the given page already exists. If so, we'll use that. If not, we'll fetch the page and save it to the .cache folder. For filenames, we're just going to base-64 encode the page's URL. If you prefer some other way to generate a unique filename, that's fine too. I've chosen the base-64 encoded URLs because it's easy and very obviously a temporary sort of file. We also have an optional function argument ignoreCache, in case you've built up your cache but want to scrape fresh data anyway.

import { existsSync, mkdirSync } from 'fs';
import { readFile, writeFile } from 'fs/promises';
import { resolve } from 'path';

async function fetchFromWebOrCache(url: string, ignoreCache = false) {
  // If the cache folder doesn't exist, create it
  if (!existsSync(resolve(__dirname, '.cache'))) {
    mkdirSync('.cache');
  }
  console.log(`Getting data for ${url}...`);
  if (
    !ignoreCache &&
    existsSync(
      resolve(__dirname, `.cache/${Buffer.from(url).toString('base64')}.html`),
    )
  ) {
    console.log(`I read ${url} from cache`);
    const HTMLData = await readFile(
      resolve(__dirname, `.cache/${Buffer.from(url).toString('base64')}.html`),
      { encoding: 'utf8' },
    );
    return HTMLData;
  } else {
    console.log(`I fetched ${url} fresh`);
    const HTMLData = await fetchPage(url);
    if (!ignoreCache && HTMLData) {
      writeFile(
        resolve(
          __dirname,
          `.cache/${Buffer.from(url).toString('base64')}.html`,
        ),
        HTMLData,
        { encoding: 'utf8' },
      );
    }
    return HTMLData;
  }
}

Extracting Data with jsdom

Now that we have HTML to work with, we want to extract the relevant data from it. To do this, we will use jsdom, a JavaScript implementation of the DOM. This will let us interact with the downloaded HTML in the exact same way as if we were working in a browser's console, giving access to methods like querySelector.

(If you prefer a syntax more like jQuery's, Cheerio is also a popular option.)

npm install jsdom
npm install --save-dev @types/jsdom

Now let's import jsdom and use it to return the Document object of our HTML string. Just modify the previous fetchFromWebOrCache to turn HTMLData into a DOM object, and return its window.document.

import { JSDOM } from 'jsdom';

async function fetchFromWebOrCache(url: string, ignoreCache = false) {
  // Get the HTMLData from fetching or from cache
  const HTMLData = '<html>...</html>'
  const dom = new JSDOM(HTMLData);
  return dom.window.document;
}

Now that we're working with a Document instead of a string, we've got access to everything we'd have if we were working in the browser console. This makes it much easier to write code that extracts the pieces of a page that we want! For example, let's scrape whatever is on the front page of Hacker News right now. We'll write a function that accepts the Document of the Hacker News front page, finds all of the links, and gives us back the link text and URL as a JavaScript object.

Using your browser's developer tools, you can easily inspect an element on the page with desired data to figure out a selector path. In our example, we can right-click a link and choose Inspect to view it in DevTools. Then we right-click the DOM element, and choose "Copy > Copy selector" in Chrome or "Copy > CSS Selector" in Firefox, for example.

A copied selector will give you a string of text that selects only the element you copied it from in DevTools. And often that is useful! Just throw your selector into document.querySelector('selector'), and you're good to go. But in our case, we want all of the front page links. So we need a broader selector than copy-pasting from DevTools will give us. This is where you'll have to actually read through the HTML, classes, ids, etc., to figure out how to craft the right selector.

Fortunately for us in this example, all of the links on the Hacker News feed have a unique class: titlelink. So we can use document.querySelectorAll('a.titlelink') to get all of them.

// Pass the scraped Document from news.ycombinator.com to this
// function to extract data about front page links.
function extractData(document: Document) {
  const writingLinks: HTMLAnchorElement[] = Array.from(
    document.querySelectorAll('a.titlelink'),
  );
  return writingLinks.map(link => {
    return {
      title: link.text,
      url: link.href,
    };
  });
}

This function is only a simple example, and would be different depending on what you want to get out of a page. When working with jsdom, remember that you're not working with arrays and objects but with NodeLists and Elements. To get useful data out of your selections, you'll often have to do things like convert a NodeList into an array as shown above.

Sometimes you'll have to get creative with your selections. I recently tried to scrape the information from an HTML table on a page with varying numbers of tables and no classes. Because the number of tables was always different, I couldn't reliably select from a list of tables by which number table it was. I had to select every table present on a page, then filter them by the text in the first cell to get precisely the one table I needed!

// Sometimes, web scraping is just hard...
const table: HTMLTableElement = Array.from(
    data.querySelectorAll('table'),
  ).filter(t =>
    t.children[0].children[0].children[0].innerHTML.match(
      /Unique Text in First Cell which IDs the Table/,
    ),
  )[0];

Extracting Data with Regular Expressions

Unfortunately for us, not all pages on the Internet are well-structured and ready for scraping. Sometimes, they don't even try to use HTML tags properly. In these sad cases, you may need to turn to regular expressions (regex) to extract what you need. We won't need to resort to such extreme measures in our example of scraping Hacker News, but it's worth knowing that you might need to do this.

I'll give you a contrived example where you would need some regex, based on another site I recently scraped. Imagine the following badly-done HTML:

<div class="pokemon">
  Name: Pikachu<br />
  Number: 25<br />
  Type: Electric<br />
  Weakness: Ground
</div>

The various data attributes we care about aren't wrapped by their own HTML elements! Everything is just inside a div with some br tags to create line breaks. If I wanted to extract the data from this, I could use regex to find and match the text and patterns I expect to find. This can require trial and error, and I recommend using a tool like regex101 to test the regular expressions you come up with. In this example, we might write the following code:

const rawPokemonHTML = document.querySelector('.pokemon');
const name = rawPokemonHTML.match(/Name: (\w+)/)[0];
const num = rawPokemonHTML.match(/Number: (\d+)/)[0];
// etc...

Saving Data

Once we've extracted our data from the HTML, we'll want to save it. This is basically the same as when we created a cache for the downloaded HTML files.

import { existsSync, mkdirSync } from 'fs';
import { writeFile } from 'fs/promises';
import { resolve } from 'path';

function saveData(filename: string, data: any) {
  if (!existsSync(resolve(__dirname, 'data'))) {
    mkdirSync('data');
  }
  writeFile(resolve(__dirname, `data/${filename}.json`), JSON.stringify(data), {
    encoding: 'utf8',
  });
}

Putting It All Together

Now that we've got all the necessary pieces, we're ready to build our JSON file of Hacker News front page stories. To see all of our code in one piece, check it out on GitHub.

async function getData() {
  const document = await fetchFromWebOrCache(
    'https://news.ycombinator.com/',
    true, // Hacker News is always changing, so ignore the cache!
  );
  const data = extractData(document);
  saveData('hacker-news-links', data);
}

getData();

When we run our script from the command line, it will execute getData(). That function will fetch the HTML from Hacker News' front page, extract all of the links and their titles, and then save it to data/hacker-news-links.json. And while you probably don't need a list of links from Hacker News, this information should be enough to get you started with collecting some data from the web which you do care about.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

Tom VanAntwerp

@tvanantwerp @tvanantwerp

JSR - The cross-platform package manager for ESM

JSR - The cross-platform package manager for ESM Move over NPM, there’s a new package manager in town. JSR is a new open-source package registry for JavaScript and TypeScript. This particular release has caught my attention, and I’m excited to explore if and where JSR fits into our overflowing ecosystem. Setting the stage The JSR website has a page called Why JSR? I want to explore the _why_ a little bit further. I think that the _why_ points to JSR potentially being the future of the JavaScript ecosystem. NPM and JavaScript modules If you’ve found yourself confused about publishing and consuming packages and modules in the modern ESM era, you’re not alone. When NodeJS was created, there was no standard for JS modules. Node introduced CommonJS modules, and NPM provided a way to publish and consume these modules in our applications. It’s been years now since ES Modules became a standard and we are still in a weird module purgatory with no clear end in sight. JSR represents a fresh start, free from the baggage of the Node and NPM ecosystems. JavaScript runtimes and interoperability New JavaScript runtimes are popping up every year. WinterCG (_Web-interoperable Runtimes Community Group)_ was started to improve interoperability for the various runtimes with some standard APIs. Node has made steps towards this goal, but it might never get all the way there and NPM is coupled pretty tightly to that ecosystem. JSR supports several runtimes (depending on the package). This is a huge opportunity for it to become the package manager for the ecosystem at large. See cross-runtime support section for more details. NPM Compatibility JSR doesn’t end up being a complete departure from NPM since it includes an NPM compatibility layer. The docs highlight some limitations and provide deeper technical details on how this piece works. The simple overview is that you can install and use packages from JSR to a Node/NPM based project. The packages will get added to your package.json and node_modules directory. About the same as any package installed from NPM. JSR Overview At a high level, JSR is very similar to NPM. You can search for packages on the website. You can publish packages to their registry, and you can download packages from it as well. We touched on the two major features that set it apart from NPM: It only supports ESModules, and it has cross-runtime support. In the rest of the article we’ll explore the essentials (package management) and also some other really great features that feel like are bringing the JS package management ecosystem up to speed with our modern workflows. Native TypeScript I have to say I’m not surprised by this feature, considering JSR was developed by the same people behind Deno (a runtime that supports TS natively). It’s a great feature. If you haven’t figured it out yet TypeScript is a big deal and doesn’t seem to be fading away anytime soon. So what exactly does TS support look like in a package manager, though? If your runtime supports TypeScript natively (like Deno) - it can consume the TS files in your package directly. For environments that don’t, it will automatically compile your code to JavaScript and package it with type declaration files (.d.ts). Pretty, stinking, cool. The docs include a page called “about slow types”. It is probably worth a read on its own ,but the TLDR is that JSR will analyze your TS code and penalize you in certain ways if you have some type code that it considers to be slow. On the page, it outlines exactly what these penalties are, as well as what it considers slow types to be. Slow types will also come into play when we get into package scoring later in the article. Packages The docs have a page dedicated to packages. It covers some important stuff like deleting packages, versioning, documentation, and publishing. We’ll touch on some of it with an emphasis on the things that are done differently or better than with NPM. jsr.json file jsr.json is a configuration file that specifies a name, version, and exports of a package. It doesn’t include a list of dependencies like you would have in a package.json file. This is a configuration file for a package to be published to JSR. JSR dependencies are defined in an import map file or a package.json file, depending on the runtime. Adding packages to a project Adding packages to your project will vary depending on your target runtime. Here’s an example of installing it for Deno and NPM using various package managers. For Deno, an import map entry is created. For NPM, it gets installed as an NPM package using the JSR NPM compatibility layer. ` If the runtime has native JSR support, you don’t need to explicitly install packages. You can important them using the jsr: scheme. ` As you can see semantic versioning is supported and works similarly to NPM. Publishing packages If you want to publish a package to JSR, there are many important rules you need to be aware of. I recommend reading “Publishing packages” before moving forward with this. The main things to note are: ESModules only, npm packages are supported, JSR packages are supported, and node APIs are supported. So there aren’t many constraints. You just need to understand how exactly to do these things. IE: how do I use NPM dependencies, how do the imports work, etc? Once you’ve got all that down, you will be ready to publish your first package to JSR 😃 Documentation JSR emphasizes quality package documentation. It’s one of the factors in their package scoring algorithm, which we will discuss shortly. The important parts of a package’s documentation include: * a README.md file * symbol documentation - functions, interfaces, classes, etc * module documentation - docs for each exported module in a package The symbol and module documentation are written using JSDoc. The JSR website generates some nice documentation based on these 3 pieces on your package page. This means you can understand a package's entire public API without ever leaving jsr.io. Browser support The cross-runtime support is amazing, but I was curious where the original JavaScript runtime fit (the browser). According to the documentation: “You can use JSR with any tool that supports ES modules, and either has native support for JSR or supports npm packages using node_modules.” Since modern web browsers support ES modules, I will count this as JSR support. From what I can tell, though, there is one caveat. To include a module on a web page, we need a URL to download it from. Can we publish our package to JSR and use it to load a module via a script tag? According to the JSR usage policy, no. The “Unacceptable use” section states: “It is not acceptable to use JSR as a CDN for serving assets directly to web applications in a browser, except for development purposes.” So, it seems that this is a no-go for now. We will probably need to wait a bit for CDN services like jsdelivr to pull packages from the JSR registry, similar to what they do with NPM currently. Other notable features We’ve covered the most important bits about package management, TS support, NPM compatibility, etc. But there are still a few unique features that add an extra bit of sparkle to JSR. Scoring We mentioned earlier that documentation plays a part in a scoring algorithm. JSR analyzes all of the packages published to its registry and assigns them a score based on certain criteria: documentation, best practices, discoverability, and compatibility. This score is shown with every package listed on the website. You can also add a shiny badge to your README if you’re particularly proud of something. It’s still early on, so it’s likely this will evolve as time goes on. It’s an interesting gimmick, at the very least! Summary I am excited to have a new cross-platform compatible package manager that only supports ESM. As useful as NPM has been all these years, it carries a lot of baggage from “the olden days”. JSR’s native TypeScript support and reliance on modern standards is a breath of fresh air. It feels like I can publish a simple package without much ceremony. I’m excited to release my first package to JSR soon :)...

Apr 12, 2024

6 mins

JSRJavaScript

Understanding Vue.js's <Suspense> and Async Components

In this blog post, we will delve into how and async components work, their benefits, and practical implementation strategies to make your Vue.js applications more efficient and user-friendly. Without further ado, let’s get started! Suspense Let's kick off by explaining what Suspense components are. They are a new component that helps manage how your application handles components that need to await for some async resource to resolve, like fetching data from a server, waiting for images to load, or any other task that might take some time to complete before they can be properly rendered. Imagine you're building a web page that needs to load data from a server, and you have 2 components that fetch the data you need as they will show different things. Typically, you might see a loading spinner or a skeleton while the data is being fetched. Suspense components make it easier to handle these scenarios. Instead of manually managing loading states and error messages for each component that needs to fetch data, Suspense components let you wrap all these components together. Inside this wrapper, you can define: 1. What to show while the data is loading (like a loading spinner). 2. The actual content that should be displayed once the data is successfully fetched. This way, Vue Suspense simplifies the process of handling asynchronous operations (like data fetching) and improves the user (and the developer) experience by providing a more seamless and integrated way to show loading states and handle errors. There are two types of async dependencies that can wait on: - Components with an async setup() hook. This includes components using with top-level await expressions. *Note: These can only be used within a component.* - Async Components. Async components Vue's asynchronous components are like a smart loading system for your web app. Imagine your app as a big puzzle. Normally, you'd put together all the pieces at once, which can take time. But what if some pieces aren't needed right away? Asynchronous components help with this. Here's how they work: - Load Only What's Needed: Just like only picking up puzzle pieces you need right now, asynchronous components let your app load only the parts that are immediately necessary. Other parts can be loaded later, as needed. - Faster Start: Your app starts up faster because it doesn't have to load everything at once. It's like quickly starting with the border of a puzzle and filling in the rest later. - Save Resources: It uses your web resources (like internet data) more wisely, only grabbing what’s essential when it's essential. In short, asynchronous components make your app quicker to start and more efficient, improving the overall experience for your users. Example: ` Combining Async Components and Suspense Let's explore how combining asynchronous components with Vue's Suspense feature can enhance your application. When asynchronous components are used with Vue's Suspense, they form a powerful combination. The key point is that async components are "suspensable" by default. This means they can be easily integrated with Suspense to improve how your app handles loading and rendering components. When used together, you can do the following things: - Centralized Loading and Error Handling: With Suspense, you don't have to handle loading and error states individually for each async component. Instead, you can define a single loading indicator or error message within the Suspense component. This unified approach simplifies your code and ensures consistency across different parts of your app. - Flexible and Clean Code Structure: By combining async components with Suspense, your code becomes more organized and easier to maintain. An asynchronous component has the flexibility to operate independently of Suspense's oversight. By setting suspensible: false in its options, the component takes charge of its own loading behavior. This means that instead of relying on Suspense to manage when it appears, the component itself dictates its loading state and presentation. This option is particularly useful for components that have specific loading logic or visuals they need to maintain, separate from the broader Suspense-driven loading strategy in the application. In practice, this combo allows you to create a user interface that feels responsive and cohesive. Users see a well-timed loading indicator while the necessary components are being fetched, and if something goes wrong, a single, well-crafted error message is displayed. It's like ensuring that the entire puzzle is either revealed in its completed form or not at all rather than showing disjointed parts at different times. How it works When a component inside the boundary is waiting for something asynchronous, shows fallback content. This fallback content can be anything you choose, such as a loading spinner or a message indicating that data is being loaded. Example Usage Let’s use a simple example: In the visual example provided, imagine we have two Vue components: one showcasing a selected Pokémon, Eevee, and a carousel showcasing a variety of other Pokémon. Both components are designed to fetch data asynchronously. Without , while the data is being fetched, we would typically see two separate loading indicators: one for the Eevee Pokemon that is selected and another for the carousel. This can make the page look disjointed and be a less-than-ideal user experience. We could display a single, cohesive loading indicator by wrapping both components inside a boundary. This unified loading state would persist until all the data for both components—the single Pokémon display and the carousel—has been fetched and is ready to be rendered. Here's how you might structure the code for such a scenario: ` Here, is the component that's performing asynchronous operations. While loading, the text 'Loading...' is displayed to the user. Great! But what about when things don't go as planned and an error occurs? Currently, Vue's doesn't directly handle errors within its boundary. However, there's a neat workaround. You can use the onErrorCaptured() hook in the parent component of to catch and manage errors. Here's how it works: ` If we run this code, and let’s say that we had an error selecting our Pokemon, this is how it is going to display to the user: The error message is specifically tied to the component where the issue occurred, ensuring that it's the only part of your application that shows an error notification. Meanwhile, the rest of your components will continue to operate and display as intended, maintaining the overall user experience without widespread disruption. This targeted error handling keeps the application's functionality intact while indicating where the problem lies. Conclusion stands out as a formidable feature in Vue.js, transforming the management of asynchronous operations into a more streamlined and user-centric process. It not only elevates the user experience by ensuring smoother interactions during data loading phases but also enhances code maintainability and application performance. I hope you found this blog post enlightening and that it adds value to your Vue.js projects. As always, happy coding and continue to explore the vast possibilities Vue.js offers to make your applications more efficient and engaging!...

Apr 24, 2024

6 mins

VueWeb PerformanceUXJavaScript

Linting, Formatting, and Type Checking Commits in an Nx Monorepo with Husky and lint-staged

One way to keep your codebase clean is to enforce linting, formatting, and type checking on every commit. This is made very easy with pre-commit hooks. Using Husky, you can run arbitrary commands before a commit is made. This can be combined with lint-staged, which allows you to run commands on only the files that have been staged for commit. This is useful because you don't want to run linting, formatting, and type checking on every file in your project, but only on the ones that have been changed. But if you're using an Nx monorepo for your project, things can get a little more complicated. Rather than have you use eslint or prettier directly, Nx has its own scripts for linting and formatting. And type checking is complicated by the use of specific tsconfig.json files for each app or library. Setting up pre-commit hooks with Nx isn't as straightforward as in a simpler repository. This guide will show you how to set up pre-commit hooks to run linting, formatting, and type checking in an Nx monorepo. Configure Formatting Nx comes with a command, nx format:write for applying formatting to affected files which we can give directly to lint-staged. This command uses Prettier under the hood, so it will abide by whatever rules you have in your root-level .prettierrc file. Just install Prettier, and add your preferred configuration. ` Then add a .prettierrc file to the root of your project with your preferred configuration. For example, if you want to use single quotes and trailing commas, you can add the following: ` Configure Linting Nx has its own plugin that uses ESLint to lint projects in your monorepo. It also has a plugin with sensible ESLint defaults for your linter commands to use, including ones specific to Nx. To install them, run the following command: ` Then, we can create a default .eslintrc.json file in the root of our project: ` The above ESLint configuration will, by default, apply Nx's module boundary rules to any TypeScript or JavaScript files in your project. It also applies its recommended rules for JavaScript and TypeScript respectively, and gives you room to add your own. You can also have ESLint configurations specific to your apps and libraries. For example, if you have a React app, you can add a .eslintrc.json file to the root of your app directory with the following contents: ` Set Up Type Checking Type checking with tsc is normally a very straightforward process. You can just run tsc --noEmit to check your code for type errors. But things are more complicated in Nx with lint-staged. There are a two tricky things about type checking with lint-staged in an Nx monorepo. First, different apps and libraries can have their own tsconfig.json files. When type checking each app or library, we need to make sure we're using that specific configuration. The second wrinkle comes from the fact that lint-staged passes a list of staged files to commands it runs by default. And tsc will only accept either a specific tsconfig file, or a list of files to check. We do want to use the specific tsconfig.json files, and we also only want to run type checking against apps and libraries with changes. To do this, we're going to create some Nx run commands within our apps and libraries and run those instead of calling tsc directly. Within each app or library you want type checked, open the project.json file, and add a new run command like this one: ` Inside commands is our type-checking command, using the local tsconfig.json file for that specific Nx app. The cwd option tells Nx where to run the command from. The forwardAllArgs option tells Nx to ignore any arguments passed to the command. This is important because tsc will fail if you pass both a tsconfig.json and a list of files from lint-staged. Now if we ran nx affected --target=typecheck from the command line, we would be able to type check all affected apps and libraries that have a typecheck target in their project.json. Next we'll have lint-staged handle this for us. Installing Husky and lint-staged Finally, we'll install and configure Husky and lint-staged. These are the two packages that will allow us to run commands on staged files before a commit is made. ` In your package.json file, add the prepare script to run Husky's install command: ` Then, run your prepare script to set up git hooks in your repository. This will create a .husky directory in your project root with the necessary file system permissions. ` The next step is to create our pre-commit hook. We can do this from the command line: ` It's important to use Husky's CLI to create our hooks, because it handles file system permissions for us. Creating files manually could cause problems when we actually want to use the git hooks. After running the command, we will now have a file at .husky/pre-commit that looks like this: ` Now whenever we try to commit, Husky will run the lint-staged command. We've given it some extra options. First, --concurrent false to make sure attempts to write fixes with formatting and linting don't conflict with simultaneous attempts at type checking. Second is --relative, because our Nx commands for formatting and linting expect a list of file paths relative to the repo root, but lint-staged would otherwise pass the full path by default. We've got our pre-commit command ready, but we haven't actually configured lint-staged yet. Let's do that next. Configuring lint-staged In a simpler repository, it would be easy to add some lint-staged configuration to our package.json file. But because we're trying to check a complex monorepo in Nx, we need to add a separate configuration file. We'll call it lint-staged.config.js and put it in the root of our project. Here is what our configuration file will look like: ` Within our module.exports object, we've defined two globs: one that will match any TypeScript files in our apps, libraries, and tools directories, and another that also matches JavaScript and JSON files in those directories. We only need to run type checking for the TypeScript files, which is why that one is broken out and narrowed down to only those files. These globs defining our directories can be passed a single command, or an array of commands. It's common with lint-staged to just pass a string like tsc --noEmit or eslint --fix. But we're going to pass a function instead to combine the list of files provided by lint-staged with the desired Nx commands. The nx affected and nx format:write commands both accept a --files option. And remember that lint-staged always passes in a list of staged files. That array of file paths becomes the argument to our functions, and we concatenate our list of files from lint-staged into a comma-delimitted string and interpolate that into the desired Nx command's --files option. This will override Nx's normal behavior to explicitly tell it to only run the commands on the files that have changed and any other files affected by those changes. Testing It Out Now that we've got everything set up, let's try it out. Make a change to a TypeScript file in one of your apps or libraries. Then try to commit that change. You should see the following in your terminal as lint-staged runs: ` Now, whenever you try to commit changes to files that match the globs defined in lint-staged.config.js, the defined commands will run first, and verify that the files contain no type errors, linting errors, or formatting errors. If any of those commands fail, the commit will be aborted, and you'll have to fix the errors before you can commit. Conclusion We've now set up a monorepo with Nx and configured it to run type checking, linting, and formatting on staged files before a commit is made. This will help us catch errors before they make it into our codebase, and it will also help us keep our codebase consistent and readable. To see an example Nx monorepo with these configurations, check out this repo....

May 10, 2023

6 mins

Transforming Auth in an AI World with Rod Boothby

In today's digital landscape, the need for secure identity verification has become paramount. With the increasing risks of personal data exposure and the rise of sophisticated cyber threats aided by Artificial Intelligence, it is crucial to adopt robust verification processes to protect individuals and organizations alike. In a recent discussion with Rod Boothby, CEO of ID Partner Systems, the significance of trusted institutions for identity verification was emphasized, particularly the efficiency of bank-based ID verification over traditional methods. One of the key takeaways from the conversation was the importance of continuous authentication. As technology advances, so do the methods employed by cybercriminals. Deepfake technology, for instance, poses a significant threat to identity verification systems. To combat this, tighter security measures and continuous authentication are essential. By constantly verifying and validating user identities, organizations can stay one step ahead of potential fraudsters. Rod Boothby also highlighted the need for a developer-focused approach to identity verification. By providing developers with the tools and resources they need, companies like ID Partner Systems aim to streamline the verification process and enhance security. This approach not only ensures a more efficient experience for users but also allows for the integration of behavioral biometrics, which can further strengthen the verification process. Download this episode here....

Apr 22, 2024

1 min

Engineering Leadership

Web Scraping with TypeScript and Node.js

Setup

Fetching Websites

Caching Scraped Pages

Extracting Data with jsdom

Extracting Data with Regular Expressions

Saving Data

Putting It All Together

Tom VanAntwerp

You might also like

JSR - The cross-platform package manager for ESM

Understanding Vue.js's <Suspense> and Async Components

Linting, Formatting, and Type Checking Commits in an Nx Monorepo with Husky and lint-staged

Transforming Auth in an AI World with Rod Boothby

You might also like

JSR - The cross-platform package manager for ESM

Understanding Vue.js's <Suspense> and Async Components

Linting, Formatting, and Type Checking Commits in an Nx Monorepo with Husky and lint-staged

Transforming Auth in an AI World with Rod Boothby