Javascript

Best Practices for Managing RxJS Subscriptions

Published Dec 14, 2020

Updated Feb 14, 2023

5 min read

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

When we use RxJS, it's standard practice to subscribe to Observables. By doing so, we create a Subscription. This object provides us with some methods that will aid in managing these subscriptions. This is very important, and is something that should not be overlooked!

Why do we care about subscription management?

If we do not put some thought into how we manage and clean up the subscriptions we create, we can cause an array of problems in our applications. This is due to how the Observer Pattern is implemented.

When an Observable emits a new value, its Observers execute code that was set up during the subscription. For example:

obs$.subscribe(data => doSomethingWithDataReceived(data));

If we do not manage this subscription, every time obs$ emits a new value doSomethingWithDataReceived will be called.

Let's say this code is set up on the Home View of our App. It should only ever be run when the user is on the Home View. Without managing this subscription correctly when the user navigates to a new view in the App, doSomethingWithDataReceived could still be called, potentially causing unexpected results, errors or even hard-to-track bugs.

So what do we mean by Subscription Management?

Essentially, subscription management revolves around knowing when to complete or unsubscribe from an Observable, to prevent incorrect code from being executed, especially when we would not expect it to be executed.

We can refer to this management of subscriptions as cleaning up active subscriptions.

How can we clean up subscriptions?

So, now that we know that managing subscriptions are an essential part of working with RxJS, what methods are available for us to manage them?

Unsubscribing Manually

One method we can use, is to unsubscribe manually from active subscriptions when we no longer require them. RxJS provides us with a convenient method to do this. It lives on the Subscription object and is simply called .unsubscribe().

If we take the example we had above; we can see how easy it is to unsubscribe when we need to:

let homeViewSubscription = null;

function onEnterView() {
  homeViewSubscription = obs$.subscribe(data => doSomethingWithDataReceived(data));
}

function onLeaveView() {
  homeViewSubscription.unsubscribe();
}

We create a variable to store the subscription.
We store the subscription in a variable when we enter the view.
We unsubscribe from the subscription when we leave the view preventing doSomethingWithDataReceived() from being executed when we don't need it.

This is great; however, when working with RxJS, you will likely have more than one subscription. Calling unsubscribe for each of them could get tedious. A solution I have seen many codebases employ is to store an array of active subscriptions, loop through this array, unsubscribing from each when required.

Let's modify the example above to see how we could do this:

const homeViewSubscriptions = [];

function onEnterView() {
  homeViewSubscriptions.push(
      obs$.subscribe(data => doSomethingWithDataReceived(data)),
      anotherObs$.subscribe(user => updateUserData(user))
    );
}

function onLeaveView() {
  homeViewSubscriptions.forEach(subscription => subscription.unsubscribe());
}

We create an array to store the subscriptions.
We add each subscription to the array when we enter the view.
We loop through and unsubscribe from the subscriptions in the array.

These are both valid methods of managing subscriptions and can and should be employed when necessary. There are other options however, that can add a bit more resilience to your management of subscriptions.

Using Operators

RxJS provides us with some operators that will clean up the subscription automatically when a condition is met, meaning we do not need to worry about setting up a variable to track our subscriptions.

Let's take a look at some of these!

`first`

The first operator will take only the first value emitted, or the first value that meets the specified criteria. Then it will complete, meaning we do not have to worry about manually unsubscribing. Let's see how we would use this with our example above:

function onEnterView() {
    obs$.pipe(first())
        .subscribe(data => doSomethingWithDataReceived(data))
}

When obs$ emits a value, first() will pass the value to doSomethingWithDataReceived and then unsubscribe!

`take`

The take operator allows us to specify how many values we want to receive from the Observable before we unsubscribe. This means that when we receive the specified number of values, take will automatically unsubscribe!

function onEnterView() {
    obs$.pipe(take(5))
        .subscribe(data => doSomethingWithDataReceived(data))
}

Once obs$ has emitted five values, take will unsubscribe automatically!

`takeUntil`

The takeUntil operator provides us with an option to continue to receive values from an Observable until a different, notifier Observable emits a new value.

Let's see it in action:


const notifier$ = new Subject();

function onEnterView() {
      obs$.pipe(takeUntil(notifier$)).subscribe(data => doSomethingWithDataReceived(data))
}

function onLeaveView() {
  notifier$.next();
  notifier$.complete();
}

We create a notifier$ Observable using a Subject. (You can learn more about Creating Observables here.)
We use takeUntil to state that we want to receive values until notifier$ emits a value
We tell notifier$ to emit a value and complete _(we need to clean notifer$ up ourselves) when we leave the view, allowing our original subscription to be unsubscribed.

`takeWhile`

Another option is the takeWhile operator. It allows us to continue receiving values whilst a specified condition remains true. Once it becomes false, it will unsubscribe automatically.

function onEnterView() {
      obs$
        .pipe(takeWhile(data => data.finished === false))
        .subscribe(data => doSomethingWithDataReceived(data))
}

In the example above we can see that whilst the property finished on the data emitted is false we will continue to receive values. When it turns to true, takeWhile will unsubscribe!

BONUS: With Angular

RxJS and Angular go hand-in-hand, even if the Angular team has tried to make the framework as agnostic as possible. From this, we usually find ourselves having to manage subscriptions in some manner.

`async` Pipe

Angular itself provides one option for us to manage subscriptions, the async pipe. This pipe will subscribe to an Observable in the template, and when the template is destroyed, it will unsubscribe from the Observable automatically. It's very simple to use:

<div *ngIf="obs$ | async as data">
  {{ data | json }}
</div>

By using the as data, we set the value emitted from the Observable to a template variable called data, allowing us to use it elsewhere in the children nodes to the div node.

When the template is destroyed, Angular will handle the cleanup!

`untilDestroyed`

Another option comes from a third-party library developed by Netanel Basal. It's called until-destroyed, and it provides us with multiple options for cleaning up subscriptions in Angular when Angular destroys a Component.

We can use it similarly to takeUntil:

import { UntilDestroy, untilDestroyed } from '@ngneat/until-destroy';

@UntilDestroy()
@Component({
    selector: 'home'
})
export class HomeComponent implements OnInit {
  ngOnInit() {
    obs$
      .pipe(untilDestroyed(this))
      .subscribe(data => this.doSoemthingWithDataReceived(data));
  }
}

It can also find which properties in your component are Subscription objects and automatically unsubscribe from them:

@UntilDestroy({ checkProperties: true })
@Component({
    selector: 'home'
})
export class HomeComponent {

  subscription = obs$
      .pipe(untilDestroyed(this))
      .subscribe(data => this.doSoemthingWithDataReceived(data));
}

This little library can be beneficial for managing subscriptions for Angular!

When should we employ one of these methods?

The simple answer to this question would be:

When we no longer want to execute code when the Observable emits a new value

But that doesn't give an example use-case.

We have covered one example use case in this article: when you navigate away from a view in your SPA.
In Angular, you'd want to use it when you destroy Components.
Combined with State Management, you could use it only to select a slice of state once that you do not expect to change over the lifecycle of the application.
Generally, you'd want to do it when a condition is met. This condition could be anything from the first click a user makes to when a certain length of time has passed.

Next time you're working with RxJS and subscriptions, think about when you no longer want to receive values from an Observable, and ensure you have code that will allow this to happen!

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

About the author(s)

Colum Ferry
JavaScript Developer and Enthusiast. Really enjoy working with Angular. Married with 2 amazing kids. Powered by Coffee.
@FerryColum @https://github.com/coly010

Introduction to Zod for Data Validation

As web developers, we're often working with data from external sources like APIs we don't control or user inputs submitted to our backends. We can't always rely on this data to take the form we expect, and we can encounter unexpected errors when it deviates from expectations. But with the Zod library, we can define what our data ought to look like and parse the incoming data against those defined schemas. This lets us work with that data confidently, or to quickly throw an error when it isn't correct. Why use Zod? TypeScript is great for letting us define the shape of our data in our code. It helps us write more correct code the first time around by warning us if we are doing something we shouldn't. But TypeScript can't do everything for us. For example, we can define a variable as a string or a number, but we can't say "a string that starts with user_id_ and is 20 characters long" or "an integer between 1 and 5". There are limits to how much TypeScript can narrow down our data for us. Also, TypeScript is a tool for us developers. When we compile our code, our types are not available to the vanilla JavaScript. JavaScript can't validate that the data we actually use in our code matches what we thought we'd get when we wrote our TypeScript types unless you're willing to manually write code to perform those checks. This is where we can reach for a tool like Zod. With Zod, we can write data schemas. These schemas, in the simplest scenarios, look very much like TypeScript types. But we can do more with Zod than we can with TypeScript alone. Zod schemas let us create additional rules for data parsing and validation. A 20-character string that starts with user_id_? It's z.string().startsWith('user_id_').length(20). An integer between 1 and 5 inclusive? It's z.number().int().gte(1).lte(5). Zod's primitives give us many extra functions to be more specific about *exactly* what data we expect. Unlike TypeScript, Zod schemas aren't removed on compilation to JavaScript—we still have access to them! If our app receives some data, we can verify that it matches the expected shape by passing it to your Zod schema's parse function. You'll either get back your data in exactly the shape you defined in your schema, or Zod will give you an error to tell you what was wrong. Zod schemas aren't a replacement for TypeScript; rather, they are an excellent complement. Once we've defined our Zod schema, it's simple to derive a TypeScript type from it and to use that type as we normally would. But when we really need to be sure our data conforms to the schema, we can always parse the data with our schema for that extra confidence. Defining Data Schemas Zod schemas are the variables that define our expectations for the shape of our data, validate those expectations, and transform the data if necessary to match our desired shape. It's easy to start with simple schemas, and to add complexity as required. Zod provides different functions that represent data structures and related validation options, which can be combined to create larger schemas. In many cases, you'll probably be building a schema for a data object with properties of some primitive type. For example, here's a schema that would validate a JavaScript object representing an order for a pizza: ` Zod provides a number of primitives for defining schemas that line up with JavaScript primitives: string, number, bigint, boolean, date, symbol, undefined, and null. It also includes primitives void, any, unknown, and never for additional typing information. In addition to basic primitives, Zod can define object, array, and other native data structure schemas, as well as schemas for data structures not natively part of JavaScript like tuple and enum. The documentation contains considerable detail on the available data structures and how to use them. Parsing and Validating Data with Schemas With Zod schemas, you're not only telling your program what data should look like; you're also creating the tools to easily verify that the incoming data matches the schema definitions. This is where Zod really shines, as it greatly simplifies the process of validating data like user inputs or third party API responses. Let's say you're writing a website form to register new users. At a minimum, you'll need to make sure the new user's email address is a valid email address. For a password, we'll ask for something at least 8 characters long and including one letter, one number, and one special character. (Yes, this is not really the best way to write strong passwords; but for the sake of showing off how Zod works, we're going with it.) We'll also ask the user to confirm their password by typing it twice. First, let's create a Zod schema to model these inputs: ` So far, this schema is pretty basic. It's only making sure that whatever the user types as an email is an email, and it's checking that the password is at least 8 characters long. But it is *not* checking if password and confirmPassword match, nor checking for the complexity requirements. Let's enhance our schema to fix that! ` By adding refine with a custom validation function, we have been able to verify that the passwords match. If they don't, parsing will give us an error to let us know that the data was invalid. We can also chain refine functions to add checks for our password complexity rules: ` Here we've chained multiple refine functions. You could alternatively use superRefine, which gives you even more fine grained control. Now that we've built out our schema and added refinements for extra validation, we can parse some user inputs. Let's see two test cases: one that's bound to fail, and one that will succeed. ` There are two main ways we can use our schema to validate our data: parse and safeParse. The main difference is that parse will throw an error if validation fails, while safeParse will return an object with a success property of either true or false, and either a data property with your parsed data or an error property with the details of a ZodError explaining why the parsing failed. In the case of our example data, userInput2 will parse just fine and return the data for you to use. But userInput1 will create a ZodError listing all of the ways it has failed validation. ` ` We can use these error messages to communicate to the user how they need to fix their form inputs if validation fails. Each error in the list describes the validation failure and gives us a human readable message to go with it. You'll notice that the validation errors for checking for a valid email and for checking password length have a lot of details, but we've got three items at the end of the error list that don't really tell us anything useful: just a custom error of Invalid input. The first is from our refine checking if the passwords match, and the second two are from our refine functions checking for password complexity (numbers and special characters). Let's modify our refine functions so that these errors are useful! We'll add our own error parameters to customize the message we get back and the path to the data that failed validation. ` Now, our error messages from failures in refine are informative! You can figure out which form fields aren't validating from the path, and then display the messages next to form fields to let the user know how to remedy the error. ` By giving our refine checks a custom path and message, we can make better use of the returned errors. In this case, we can highlight specific problem form fields for the user and give them the message about what is wrong. Integrating with TypeScript Integrating Zod with TypeScript is very easy. Using z.infer<typeof YourSchema> will allow you to avoid writing extra TypeScript types that merely reflect the intent of your Zod schemas. You can create a type from any Zod schema like so: ` Using a TypeScript type derived from a Zod schema does *not* give you any extra level of data validation at the type level beyond what TypeScript is capable of. If you create a type from z.string.min(3).max(20), the TypeScript type will still just be string. And when compiled to JavaScript, even that will be gone! That's why you still need to use parse/safeParse on incoming data to validate it before proceeding as if it really does match your requirements. A common pattern with inferring types from Zod schemas is to use the same name for both. Because the schema is a variable, there's no name conflict if the type uses the same name. However, I find that this can lead to confusing situations when trying to import one or the other—my personal preference is to name the Zod schema with Schema at the end to make it clear which is which. Conclusion Zod is an excellent tool for easily and confidently asserting that the data you're working with is exactly the sort of data you were expecting. It gives us the ability to assert at runtime that we've got what we wanted, and allows us to then craft strategies to handle what happens if that data is wrong. Combined with the ability to infer TypeScript types from Zod schemas, it lets us write and run more reliable code with greater confidence....

Nov 15, 2024

7 mins

JavaScriptZod

D1 SQLite: Writing queries with the D1 Client API

Writing queries with the D1 Client API In the previous post we defined our database schema, got up and running with migrations, and loaded some seed data into our database. In this post we will be working with our new database and seed data. If you want to participate, make sure to follow the steps in the first post. We’ve been taking a minimal approach so far by using only wrangler and sql scripts for our workflow. The D1 Client API has a small surface area. Thanks to the power of SQL, we will have everything we need to construct all types of queries. Before we start writing our queries, let's touch on some important concepts. Prepared statements and parameter binding This is the first section of the docs and it highlights two different ways to write our SQL statements using the client API: prepared and static statements. Best practice is to use prepared statements because they are more performant and prevent SQL injection attacks. So we will write our queries using prepared statements. We need to use parameter binding to build our queries with prepared statements. This is pretty straightforward and there are two variations. By default we add ? ’s to our statement to represent a value to be filled in. The bind method will bind the parameters to each question mark by their index. The first ? is tied to the first parameter in bind, 2nd, etc. I would stick with this most of the time to avoid any confusion. ` I like this second method less as it feels like something I can imagine messing up very innocently. You can add a number directly after a question mark to indicate which number parameter it should be bound to. In this exampl, we reverse the previous binding. ` Reusing prepared statements If we take the first example above and not bind any values we have a statement that can be reused: ` Querying For the purposes of this post we will just build example queries by writing them out directly in our Worker fetch handler. If you are building an app I would recommend building functions or some other abstraction around your queries. select queries Let's write our first query against our data set to get our feet wet. Here’s the initial worker code and a query for all authors: ` We pass our SQL statement into prepare and use the all method to get all the rows. Notice that we are able to pass our types to a generic parameter in all. This allows us to get a fully typed response from our query. We can run our worker with npm run dev and access it at http://localhost:8787 by default. We’ll keep this simple workflow of writing queries and passing them as a json response for inspection in the browser. Opening the page we get our author results. joins Not using an ORM means we have full control over our own destiny. Like anything else though, this has tradeoffs. Let’s look at a query to fetch the list of posts that includes author and tags information. ` Let’s walk through each part of the query and highlight some pros and cons. ` * The query selects all columns from the posts table. * It also selects the name column from the authors table and renames it to author_name. * It aggregates the name column from the tags table into a JSON array. If there are no tags, it returns an empty JSON array. This aggregated result is renamed to tags. ` * The query starts by selecting data from the posts table. * It then joins the authors table to include author information for each post, matching posts to authors using the author_id column in posts and the id column in authors. * Next, it left joins the posts_tags table to include tag associations for each post, ensuring that all posts are included even if they have no tags. * Next, it left joins the tags table to include tag names, matching tags to posts using the tag_id column in posts_tags and the id column in tags. * Finally, group the results by the post id so that all rows with the same post id are combined in a single row SQL provides a lot of power to query our data in interesting ways. JOIN ’s will typically be more performant than performing additional queries.You could just as easily write a simpler version of this query that uses subqueries to fetch post tags and join all the data by hand with JavaScript. This is the nice thing about writing SQL, you’re free to fetch and handle your data how you please. Our results should look similar to this: ` This brings us to our next topic. Marshaling / coercing result data A couple of things we notice about the format of the result data our query provides: Rows are flat. We join the author directly onto the post and prefix its column names with author. ` Using an ORM we might get the data back as a child object: ` Another thing is that our tags data is a JSON string and not a JavaScript array. This means that we will need to parse it ourselves. ` This isn’t the end of the world but it is some more work on our end to coerce the result data into the format that we actually want. This problem is handled in most ORM’s and is their main selling point in my opinion. insert / update / delete Next, let’s write a function that will add a new post to our database. ` There’s a few queries involved in our create post function: * first we create the new post * next we run through the tags and either create or return an existing tag * finally, we add entries to our post_tags join table to associate our new post with the tags assigned We can test our new function by providing post content in query params on our index page and formatting them for our function. ` I gave it a run like this: http://localhost:8787authorId=1&tags=Food%2CReview&title=A+review+of+my+favorite+Italian+restaurant&content=I+got+the+sausage+orchette+and+it+was+amazing.+I+wish+that+instead+of+baby+broccoli+they+used+rapini.+Otherwise+it+was+a+perfect+dish+and+the+vibes+were+great And got a new post with the id 11. UPDATE and DELETE operations are pretty similar to what we’ve seen so far. Most complexity in your queries will be similar to what we’ve seen in the posts query where we want to JOIN or GROUP BY data in various ways. To update the post we can write a query that looks like this: ` COALESCE acts similarly to if we had written a ?? b in JavaScript. If the binded value that we provide is null it will fall back to the default. We can delete our new post with a simple DELETE query: ` Transactions / Batching One thing to note with D1 is that I don’t think the traditional style of SQLite transactions are supported. You can use the db.batch API to achieve similar functionality though. According to the docs: Batched statements are SQL transactions ↗. If a statement in the sequence fails, then an error is returned for that specific statement, and it aborts or rolls back the entire sequence. ` Summary In this post, we've taken a hands-on approach to exploring the D1 Client API, starting with defining our database schema and loading seed data. We then dove into writing queries, covering the basics of prepared statements and parameter binding, before moving on to more complex topics like joins and transactions. We saw how to construct and execute queries to fetch data from our database, including how to handle relationships between tables and marshal result data into a usable format. We also touched on inserting, updating, and deleting data, and how to use transactions to ensure data consistency. By working through these examples, we've gained a solid understanding of how to use the D1 Client API to interact with our database and build robust, data-driven applications....

Dec 23, 2024

6 mins

JavaScript

NgRx Facade Pattern

NgRx Facade Pattern The NgRx Facade Pattern was first introduced by Thomas Burleson in 2018 and has drawn a lot of attention in recent years. In this article, we will discuss the pattern, how to implement it in Angular and discuss whether or not we _should_ implement it. What is NgRx? First, what is NgRx? NgRx is a state management solution for Angular built on top of RxJS which adheres to the redux pattern. It contains an immutable centralized store where the state of our application gets stored. - We select slices of state from the Store using Selectors, which we can then render in our components. - We dispatch Actions to our Store. - Our Store redirects our Action to our Reducers to recalculate our state and replaces the state within our Store. See the diagram below for an illustrated example: This provides us with a tried and tested pattern for managing the state of our application. What is the Facade Pattern? Now that we know what NgRx is, what is the Facade Pattern? Well, what _are_ Facades? Facades are a pattern that provides a simple public interface to mask more complex usage. As we use NgRx more and more in our application, we add more actions and more selectors that our components must use and track. This increases the coupling between our component and the actions and selectors themselves. The Facade pattern wants to simplify this approach by wrapping the NgRx interactions in one place, allowing the Component to only ever interact with the Facade. This means you are free to refactor the NgRx artefacts without worrying about breaking your Components. In Angular, NgRx Facades are simply services. They inject the NgRx Store allowing you to contain your interactions with the Store in the service. How do we implement it? To begin with, let's show a Component that uses NgRx directly: ` As we can see, this depends a lot on interactions with the Store and has made our component fairly complex and coupled to NgRx. Let's create a Facade that will encapsulate this interaction with NgRx: ` It's essentially everything we had in the component, except now in a service. We then inject this service into our Component: ` By implementing the Facade and using it in our Component, our component no longer depends on NgRx and we do not have to import all actions and selectors. The Facade hides those implementation details, keeping our Component cleaner and easier tested. Pros What are some advantages of using Facades? - It adds a single abstraction of a section of the Store. - This service can be used by any component that needs to interact with this section of the store. For example, if another component needs to access the TodoListState from our example above, they do not have to reimplement the action dispatch or state selector code. It's all readily available in the Facade. - Facades are scalable - As Facades are just services, we can compose them within other Facades allowing us to maintain the encapsulation and hide complex logic that interacts directly with NgRx, leaving us with an API that our developers can consume. Cons - Facades lead to reusing Actions. - Mike Ryan gave a talk at ng-conf 2018 on Good Action Hygiene which promotes creating as many actions as possible that dictate how your user is using your app and allowing NgRx to update the state of the application from your user's interactions. - Facades force actions to be reused. This becomes a problem as we no longer update state based on the user's interactions. Instead, we create a coupling between our actions and various component areas within our application. - Therefore, by changing one action and one accompanying reducer, we could be impacting a significant portion of our application. - We lose indirection - Indirection is when a portion of our app is responsible for certain logic, and the other pieces of our app (the view layer etc.) communicate with it via messages. - In NgRx, this means that our Effects or Reducers do not know what told them to work; they just know they have to. - With Facades, we hide this indirection as only the service knows about how the state is being updated. - Knowledge Cost - It becomes more difficult for junior developers to understand how to interact, update and work with NgRx if their only interactions with the state management solution are through Facades. - It also becomes more difficult for them to write new Actions, Reducers and Selectors as they may not have been exposed to them before. Conclusion Hopefully, this gives you an introduction to NgRx Facades and the pros and cons of using them. This should help you evaluate whether to use them or not....

Feb 9, 2021

4 mins

AngularNgRxJavaScriptTypeScript

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging) In the first part of my series on the importance of a scientific mindset in software engineering, we explored how the principles of the scientific method can help us evaluate sources and make informed decisions. Now, we will focus on how these principles can help us tackle one of the most crucial and challenging tasks in software engineering: debugging. In software engineering, debugging is often viewed as an art - an intuitive skill honed through experience and trial and error. In a way, it is - the same as a GP, even a very evidence-based one, will likely diagnose most of their patients based on their experience and intuition and not research scientific literature every time; a software engineer will often rely on their experience and intuition to identify and fix common bugs. However, an internist faced with a complex case will likely not be able to rely on their intuition alone and must apply the scientific method to diagnose the patient. Similarly, a software engineer can benefit from using the scientific method to identify and fix the problem when faced with a complex bug. From that perspective, treating engineering challenges like scientific inquiries can transform the way we tackle problems. Rather than resorting to guesswork or gut feelings, we can apply the principles of the scientific method—forming hypotheses, designing controlled experiments, gathering and evaluating evidence—to identify and eliminate bugs systematically. This approach, sometimes referred to as "scientific debugging," reframes debugging from a haphazard process into a structured, disciplined practice. It encourages us to be skeptical, methodical, and transparent in our reasoning. For instance, as Andreas Zeller notes in the book _Why Programs Fail_, the key aspect of scientific debugging is its explicitness: Using the scientific method, you make your assumptions and reasoning explicit, allowing you to understand your assumptions and often reveals hidden clues that can lead to the root cause of the problem on hand. Note: If you'd like to read an excerpt from the book, you can find it on Embedded.com. Scientific Debugging At its core, scientific debugging applies the principles of the scientific method to the process of finding and fixing software defects. Rather than attempting random fixes or relying on intuition, it encourages engineers to move systematically, guided by data, hypotheses, and controlled experimentation. By adopting debugging as a rigorous inquiry, we can reduce guesswork, speed up the resolution process, and ensure that our fixes are based on solid evidence. Just as a scientist begins with a well-defined research question, a software engineer starts by identifying the specific symptom or error condition. For instance, if our users report inconsistencies in the data they see across different parts of the application, our research question could be: _"Under what conditions does the application display outdated or incorrect user data?"_ From there, we can follow a structured debugging process that mirrors the scientific method: - 1. Observe and Define the Problem: First, we need to clearly state the bug's symptoms and the environment in which it occurs. We should isolate whether the issue is deterministic or intermittent and identify any known triggers if possible. Such a structured definition serves as the groundwork for further investigation. - 2. Formulate a Hypothesis: A hypothesis in debugging is a testable explanation for the observed behavior. For instance, you might hypothesize: _"The data inconsistency occurs because a caching layer is serving stale data when certain user profiles are updated."_ The key is that this explanation must be falsifiable; if experiments don't support the hypothesis, it must be refined or discarded. - 3. Collect Evidence and Data: Evidence often includes logs, system metrics, error messages, and runtime traces. Similar to reviewing primary sources in academic research, treat your raw debugging data as crucial evidence. Evaluating these data points can reveal patterns. In our example, such patterns could be whether the bug correlates with specific caching mechanisms, increased memory usage, or database query latency. During this step, it's essential to approach data critically, just as you would analyze the quality and credibility of sources in a research literature review. Don't forget that even logs can be misleading, incomplete, or even incorrect, so cross-referencing multiple sources is key. - 4. Design and Run Experiments: Design minimal, controlled tests to confirm or refute your hypothesis. In our example, you may try disabling or shortening the cache's time-to-live (TTL) to see if more recent data is displayed correctly. By manipulating one variable at a time - such as cache invalidation intervals - you gain clearer insights into causation. Tools such as profilers, debuggers, or specialized test harnesses can help isolate factors and gather precise measurements. - 5. Analyze Results and Refine Hypotheses: If the experiment's outcome doesn't align with your hypothesis, treat it as a stepping stone, not a dead end. Adjust your explanation, form a new hypothesis, or consider additional variables (for example, whether certain API calls bypass caching). Each iteration should bring you closer to a better understanding of the bug's root cause. Remember, the goal is not to prove an initial guess right but to arrive at a verifiable explanation. - 6. Implement and Verify the Fix: Once you're confident in the identified cause, you can implement the fix. Verification doesn't stop at deployment - re-test under the same conditions and, if possible, beyond them. By confirming the fix in a controlled manner, you ensure that the solution is backed by evidence rather than wishful thinking. - Personally, I consider implementing end-to-end tests (e.g., with Playwright) that reproduce the bug and verify the fix to be a crucial part of this step. This both ensures that the bug doesn't reappear in the future due to changes in the codebase and avoids possible imprecisions of manual testing. Now, we can explore these steps in more detail, highlighting how the scientific method can guide us through the debugging process. Establishing Clear Debugging Questions (Formulating a Hypothesis) A hypothesis is a proposed explanation for a phenomenon that can be tested through experimentation. In a debugging context, that phenomenon is the bug or issue you're trying to resolve. Having a clear, falsifiable statement that you can prove or disprove ensures that you stay focused on the real problem rather than jumping haphazardly between possible causes. A properly formulated hypothesis lets you design precise experiments to evaluate whether your explanation holds true. To formulate a hypothesis effectively, you can follow these steps: 1. Clearly Identify the Symptom(s) Before forming any hypothesis, pin down the specific issue users are experiencing. For instance: - "Users intermittently see outdated profile information after updating their accounts." - "Some newly created user profiles don't reflect changes in certain parts of the application." Having a well-defined problem statement keeps your hypothesis focused on the actual issue. Just like a research question in science, the clarity of your symptom definition directly influences the quality of your hypothesis. 2. Draft a Tentative Explanation Next, convert your symptom into a statement that describes a _possible root cause_, such as: - "Data inconsistency occurs because the caching layer isn't invalidating or refreshing user data properly when profiles are updated." - "Stale data is displayed because the cache timeout is too long under certain load conditions." This step makes your assumption about the root cause explicit. As with the scientific method, your hypothesis should be something you can test and either confirm or refute with data or experimentation. 3. Ensure Falsifiability A valid hypothesis must be falsifiable - meaning it can be proven _wrong_. You'll struggle to design meaningful experiments if a hypothesis is too vague or broad. For example: - Not Falsifiable: "Occasionally, the application just shows weird data." - Falsifiable: "Users see stale data when the cache is not invalidated within 30 seconds of profile updates." Making your hypothesis specific enough to fail a test will pave the way for more precise debugging. 4. Align with Available Evidence Match your hypothesis to what you already know - logs, stack traces, metrics, and user reports. For example: - If logs reveal that cache invalidation events aren't firing, form a hypothesis explaining why those events fail or never occur. - If metrics show that data served from the cache is older than the configured TTL, hypothesize about how or why the TTL is being ignored. If your current explanation contradicts existing data, refine your hypothesis until it fits. 5. Plan for Controlled Tests Once you have a testable hypothesis, figure out how you'll attempt to _disprove_ it. This might involve: - Reproducing the environment: Set up a staging/local system that closely mimics production. For instance with the same cache layer configurations. - Varying one condition at a time: For example, only adjust cache invalidation policies or TTLs and then observe how data freshness changes. - Monitoring metrics: In our example, such monitoring would involve tracking user profile updates, cache hits/misses, and response times. These metrics should lead to confirming or rejecting your explanation. These plans become your blueprint for experiments in further debugging stages. Collecting and Evaluating Evidence After formulating a clear, testable hypothesis, the next crucial step is to gather data that can either support or refute it. This mirrors how scientists collect observations in a literature review or initial experiments. 1. Identify "Primary Sources" (Logs, Stack Traces, Code History): - Logs and Stack Traces: These are your direct pieces of evidence - treat them like raw experimental data. For instance, look closely at timestamps, caching-related events (e.g., invalidation triggers), and any error messages related to stale reads. - Code History: Look for related changes in your source control, e.g. using Git bisect. In our example, we would look for changes to caching mechanisms or references to cache libraries in commits, which could pinpoint when the inconsistency was introduced. Sometimes, reverting a commit that altered cache settings helps confirm whether the bug originated there. 2. Corroborate with "Secondary Sources" (Documentation, Q&A Forums): - Documentation: Check official docs for known behavior or configuration details that might differ from your assumptions. - Community Knowledge: Similar issues reported on GitHub or StackOverflow may reveal known pitfalls in a library you're using. 3. Assess Data Quality and Relevance: - Look for Patterns: For instance, does stale data appear only after certain update frequencies or at specific times of day? - Check Environmental Factors: For instance, does the bug happen only with particular deployment setups, container configurations, or memory constraints? - Watch Out for Biases: Avoid seeking only the data that confirms your hypothesis. Look for contradictory logs or metrics that might point to other root causes. You keep your hypothesis grounded in real-world system behavior by treating logs, stack traces, and code history as primary data - akin to raw experimental results. This evidence-first approach reduces guesswork and guides more precise experiments. Designing and Running Experiments With a hypothesis in hand and evidence gathered, it's time to test it through controlled experiments - much like scientists isolate variables to verify or debunk an explanation. 1. Set Up a Reproducible Environment: - Testing Environments: Replicate production conditions as closely as possible. In our example, that would involve ensuring the same caching configuration, library versions, and relevant data sets are in place. - Version Control Branches: Use a dedicated branch to experiment with different settings or configuration, e.g., cache invalidation strategies. This streamlines reverting changes if needed. 2. Control Variables One at a Time: - For instance, if you suspect data inconsistency is tied to cache invalidation events, first adjust only the invalidation timeout and re-test. - Or, if concurrency could be a factor (e.g., multiple requests updating user data simultaneously), test different concurrency levels to see if stale data issues become more pronounced. 3. Measure and Record Outcomes: - Automated Tests: Tests provide a great way to formalize and verify your assumptions. For instance, you could develop tests that intentionally update user profiles and check if the displayed data matches the latest state. - Monitoring Tools: Monitor relevant metrics before, during, and after each experiment. In our example, we might want to track cache hit rates, TTL durations, and query times. - Repeat Trials: Consistency across multiple runs boosts confidence in your findings. 4. Validate Against a Baseline: - If baseline tests manifest normal behavior, but your experimental changes manifest the bug, you've isolated the variable causing the issue. E.g. if the baseline tests show that data is consistently fresh under normal caching conditions but your experimental changes cause stale data. - Conversely, if your change eliminates the buggy behavior, it supports your hypothesis - e.g. that the cache configuration was the root cause. Each experiment outcome is a data point supporting or contradicting your hypothesis. Over time, these data points guide you toward the true cause. Analyzing Results and Iterating In scientific debugging, an unexpected result isn't a failure - it's valuable feedback that brings you closer to the right explanation. 1. Compare Outcomes to the hypothesis. For instance: - Did user data stay consistent after you reduced the cache TTL or fixed invalidation logic? - Did logs show caching events firing as expected, or did they reveal unexpected errors? - Are there only partial improvements that suggest multiple overlapping issues? 2. Incorporate Unexpected Observations: - Sometimes, debugging uncovers side effects - e.g. performance bottlenecks exposed by more frequent cache invalidations. Note these for future work. - If your hypothesis is disproven, revise it. For example, the cache may only be part of the problem, and a separate load balancer setting also needs attention. 3. Avoid Confirmation Bias: - Don't dismiss contrary data. For instance, if you see evidence that updates are fresh in some modules but stale in others, you may have found a more nuanced root cause (e.g., partial cache invalidation). - Consider other credible explanations if your teammates propose them. Test those with the same rigor. 4. Decide If You Need More Data: - If results aren't conclusive, add deeper instrumentation or enable debug modes to capture more detailed logs. - For production-only issues, implement distributed tracing or sampling logs to diagnose real-world usage patterns. 5. Document Each Iteration: - Record the results of each experiment, including any unexpected findings or new hypotheses that arise. - Through iterative experimentation and analysis, each cycle refines your understanding. By letting evidence shape your hypothesis, you ensure that your final conclusion aligns with reality. Implementing and Verifying the Fix Once you've identified the likely culprit - say, a misconfigured or missing cache invalidation policy - the next step is to implement a fix and verify its resilience. 1. Implementing the Change: - Scoped Changes: Adjust just the component pinpointed in your experiments. Avoid large-scale refactoring that might introduce other issues. - Code Reviews: Peer reviews can catch overlooked logic gaps or confirm that your changes align with best practices. 2. Regression Testing: - Re-run the same experiments that initially exposed the issue. In our stale data example, confirm that the data remains fresh under various conditions. - Conduct broader tests - like integration or end-to-end tests - to ensure no new bugs are introduced. 3. Monitoring in Production: - Even with positive test results, real-world scenarios can differ. Monitor logs and metrics (e.g. cache hit rates, user error reports) closely post-deployment. - If the buggy behavior reappears, revisit your hypothesis or consider additional factors, such as unpredicted user behavior. 4. Benchmarking and Performance Checks (If Relevant): - When making changes that affect the frequency of certain processes - such as how often a cache is refreshed - be sure to measure the performance impact. Verify you meet any latency or resource usage requirements. - Keep an eye on the trade-offs: For instance, more frequent cache invalidations might solve stale data but could also raise system load. By systematically verifying your fix - similar to confirming experimental results in research - you ensure that you've addressed the true cause and maintained overall software stability. Documenting the Debugging Process Good science relies on transparency, and so does effective debugging. Thorough documentation guarantees your findings are reproducible and valuable to future team members. 1. Record Your Hypothesis and Experiments: - Keep a concise log of your main hypothesis, the tests you performed, and the outcomes. - A simple markdown file within the repo can capture critical insights without being cumbersome. 2. Highlight Key Evidence and Observations: - Note the logs or metrics that were most instrumental - e.g., seeing repeated stale cache hits 10 minutes after updates. - Document any edge cases discovered along the way. 3. List Follow-Up Actions or Potential Risks: - If you discover additional issues - like memory spikes from more frequent invalidation - note them for future sprints. - Identify parts of the code that might need deeper testing or refactoring to prevent similar issues. 4. Share with Your Team: - Publish your debugging report on an internal wiki or ticket system. A well-documented troubleshooting narrative helps educate other developers. - Encouraging open discussion of the debugging process fosters a culture of continuous learning and collaboration. By paralleling scientific publication practices in your documentation, you establish a knowledge base to guide future debugging efforts and accelerate collective problem-solving. Conclusion Debugging can be as much a rigorous, methodical exercise as an art shaped by intuition and experience. By adopting the principles of scientific inquiry - forming hypotheses, designing controlled experiments, gathering evidence, and transparently documenting your process - you make your debugging approach both systematic and repeatable. The explicitness and structure of scientific debugging offer several benefits: - Better Root-Cause Discovery: Structured, hypothesis-driven debugging sheds light on the _true_ underlying factors causing defects rather than simply masking symptoms. - Informed Decisions: Data and evidence lead the way, minimizing guesswork and reducing the chance of reintroducing similar issues. - Knowledge Sharing: As in scientific research, detailed documentation of methods and outcomes helps others learn from your process and fosters a collaborative culture. Ultimately, whether you are diagnosing an intermittent crash or chasing elusive performance bottlenecks, scientific debugging brings clarity and objectivity to your workflow. By aligning your debugging practices with the scientific method, you build confidence in your solutions and empower your team to tackle complex software challenges with precision and reliability. But most importantly, do not get discouraged by the number of rigorous steps outlined above or by the fact you won't always manage to follow them all religiously. Debugging is a complex and often frustrating process, and it's okay to rely on your intuition and experience when needed. Feel free to adapt the debugging process to your needs and constraints, and as long as you keep the scientific mindset at heart, you'll be on the right track....

May 9, 2025

13 mins

JavaScript

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Best Practices for Managing RxJS Subscriptions

Why do we care about subscription management?

So what do we mean by Subscription Management?

How can we clean up subscriptions?

Unsubscribing Manually

Using Operators

`first`

`take`

`takeUntil`

`takeWhile`

BONUS: With Angular

`async` Pipe

`untilDestroyed`

When should we employ one of these methods?

Colum Ferry

You might also like

Introduction to Zod for Data Validation

D1 SQLite: Writing queries with the D1 Client API

NgRx Facade Pattern

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

Let's innovate together!

You might also like

Introduction to Zod for Data Validation

D1 SQLite: Writing queries with the D1 Client API

NgRx Facade Pattern

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

Why do we care about subscription management?

So what do we mean by Subscription Management?

How can we clean up subscriptions?

Unsubscribing Manually

Using Operators

first

take

takeUntil

takeWhile

BONUS: With Angular

async Pipe

untilDestroyed

When should we employ one of these methods?

Colum Ferry

Introduction to Zod for Data Validation

D1 SQLite: Writing queries with the D1 Client API

NgRx Facade Pattern

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

Let's innovate together!

Introduction to Zod for Data Validation

D1 SQLite: Writing queries with the D1 Client API

NgRx Facade Pattern

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

`first`

`take`

`takeUntil`

`takeWhile`

`async` Pipe

`untilDestroyed`