Skip to content

Getting Authenticated Images in Angular

This article was written over 18 months ago and may contain information that is out of date. Some content may be relevant but please refer to the relevant official documentation or available resources for the latest information.

Getting authenticated images in Angular

Let's imagine the following situation: We are working on an application which handles contracts between clients and sales representatives from a particular company.

For example, an insurance company could require us to develop a secure application where damage reports can be uploaded, and then the insurance agent, who has access rights to deal with such reports, can check the uploaded photos. These uploaded photos can be photos of damaged cars where the license plate is, and photos of the car's registration papers. These photos can contain sensitive data.

So the application is almost ready. One of the last features is to display these uploaded photos to everybody who has access to that particular insurance report. The application is set up in a way that allows us to download the images from a specific endpoint: /api/reports/{reportId}/{imageId}. We have set up our layout, have generated URLs for all the images that we need to request, and put them into the img tags.

However, they don't show up, because when they are fetched, they never hit our HttpInterceptor that sets the Authorization header on the request.

Introducing @this-dot/ng-utils UseHttpImageSource pipe

We came up with the idea of using a pipe to solve the above problem. We decided to include it as the first element in a collection of useful utilities for Angular. We called it @this-dot/ng-utils.

How to solve the problem?

If we send our request using Angular's HttpClient, it will hit our HttpInterceptor, which will, in turn, attach the Authorization header to the request. To avoid putting this logic into our services and/or components, we are going to implement a pipe.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false,
})
export class UseHttpImageSourcePipe implements PipeTransform {

  constructor(private httpClient: HttpClient,
              private domSanitizer: DomSanitizer,
              private cdr: ChangeDetectorRef) {
  }

  transform(imagePath: string): string | SafeUrl {
    // our logic will come here
    return imagePath;
  }

}

We immediately know that we are going to need the HttpClient for getting the image, the DomSanitizer, so we will be able to use the bypassSecurityTrustUrl method to allow the returned blob to be displayed.

We do know that the endpoints we call will return safe images, so that is why we trust the returned values. We will also need the ChangeDetectorRef because this process is async, and we are going to trigger change detection manually. The pipe itself is not pure, because the returned value will change eventually, so we need to get the value from the transform on every change detection cycle.

We also need to keep in mind that whenever the input value changes, a new request needs to be sent out. That is why we are going to use a BehaviorSubject as the base of our async subscription. Let's also use the ngOnDestroy lifecycle hook to tear down the subscription when the component that has our pipe instance destroys.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false,
})
export class UseHttpImageSourcePipe implements PipeTransform, OnDestroy {
  private subscription = new Subscription();
  private transformValue = new BehaviorSubject<string>('');

  private latestValue!: string | SafeUrl;

  constructor(private httpClient: HttpClient,
              private domSanitizer: DomSanitizer,
              private cdr: ChangeDetectorRef) {
    // every pipe instance will set up their subscription
    this.setUpSubscription();
  }

  // ...

  transform(imagePath: string): string | SafeUrl {
    // we emit a new value
    this.transformValue.next(imagePath);

    // we always return the latest value
    return this.latestValue;
  }

  ngOnDestroy(): void {
    this.subscription.unsubscribe();
  }

  private setUpSubscription(): void {
    const transformSubscription = this.transformValue
      .asObservable()
      .pipe(
        // we filter out empty strings and falsy values
        filter((v): v is string => !!v),
        // we don't emit if the input hasn't changed
        distinctUntilChanged(),
        // Our HttpClient logic will come here
        // 
        tap((imagePath: string | SafeUrl) => {
        // we set the latestValue property of the pipe
        this.latestValue = imagePath;
        // and we mark the DOM changed, so the pipe's transform method is called again
        this.cdr.markForCheck();
      })
      )
      .subscribe();
    this.subscription.add(transformSubscription);
  }
}

Let's walk through what happens. When the pipe is initialised, it sets up the subscription to our transformValue, BehaviorSubject. This subscription will only be initialised once per pipe instance, and we unsubscribe from it in the ngOnDestroy() lifecycle hook.

The logic will happen mainly inside our setUpSubscription() method, where we filter out falsy values. We use the distinctUntilChanged() operator to check if the current transformValue is different from the previous one. If it is, it emits the new value. And finally, the transform method returns the latestValue, which is stored on the pipe.

We are going to implement our HttpClient logic there. Right now it just sets the latestValue property and triggers change detection. Let's get our images using the HttpClient.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false,
})
export class UseHttpImageSourcePipe implements PipeTransform, OnDestroy {
  // ...
  private setUpSubscription(): void {
    const transformSubscription = this.transformValue
      .asObservable()
      .pipe(
        filter((v): v is string => !!v),
        distinctUntilChanged(),
        // we use switchMap, so the previous subscription gets torn down 
        switchMap((imagePath: string) => this.httpClient
          // we get the imagePath, observing the response and getting it as a 'blob'
          .get(imagePath, { observe: 'response', responseType: 'blob' })
          .pipe(
            // we map our blob into an ObjectURL
            map((response: HttpResponse<Blob>) => URL.createObjectURL(response.body)),
            // we bypass Angular's security mechanisms
            map((unsafeBlobUrl: string) => this.domSanitizer.bypassSecurityTrustUrl(unsafeBlobUrl)),
            // we trigger it only when there is a change in the result
            filter((blobUrl) => blobUrl !== this.latestValue),
          )
        ),
        tap((imagePath: string | SafeUrl) => {
          this.latestValue = imagePath;
          this.cdr.markForCheck();
        })
      )
      .subscribe();
    this.subscription.add(transformSubscription);
  }
}

We subscribe to our httpClient.get() method inside a switchMap operator, so we unsubscribe from the previous subscription if the image path passed into our transform method changes. We set up our get request to return a blob which we then convert into an ObjectURL, and we bypass the safety mechanisms of Angular. When the change detection cycle is triggered, this sanitised blob will be returned as the latestValue, and the image gets displayed. And when we check the network tab of the dev tools, we can see that our Authorization header is set on the image request.

Our pipe in our template:

  <img width="200px" [src]="'assets/images/success.png' | useHttpImgSrc" />
With authorization header

Some user experience improvements

Although our images now display on the page, we got a requirement to make them stateful. A loading image should be displayed before the actual image is loaded, and if the request has an error, another image should be displayed. Let's set up the loading image logic in our pipe, using an optional parameter in the transform() method.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false,
})
export class UseHttpImageSourcePipe implements PipeTransform, OnDestroy {
  private subscription = new Subscription();
  private loadingImagePath!: string;
  private latestValue!: string | SafeUrl;
  private transformValue = new BehaviorSubject<string>('');

  // ...

  transform(
    imagePath: string,
    loadingImagePath?: string
  ): string | SafeUrl {
    this.setLoadingImagePath(loadingImagePath);
    // ...

    this.transformValue.next(imagePath);
    // return the loading image while there is no value present
    return this.latestValue || this.loadingImagePath;
  }

  // ...

  private setLoadingImagePath(
    loadingImagePath?: string
  ): void {
    // if it is already set we do nothing
    if (this.loadingImagePath) {
      return;
    }
    this.loadingImagePath = loadingImagePath;
  }

}

The latestValue property is only falsy before the actual image arrives. We created the setLoadingImagePath() method, which if provided, sets up a path for the loading placeholder image. Let's set up our template for two images, one which would surely fail, the other one will be loaded, but both of them will display the loading screen.

  <img width="200px" [src]="'assets/images/success.png' | useHttpImgSrc:'assets/images/loading.png'" />
  // ...
  <img width="200px" [src]="'assets/images/notfound.png' | useHttpImgSrc:'assets/images/loading.png'" />
Images are loading

Let's make the error image working as well. We are going to pass it as a second optional parameter. Let's update our template first.

  <img width="200px" [src]="'assets/images/success.png' | useHttpImgSrc:'assets/images/loading.png':'assets/images/error.png'" />
  // ...
  <img width="200px" [src]="'assets/images/notfound.png' | useHttpImgSrc:'assets/images/loading.png':'assets/images/error.png'" />

Let's update our pipe's implementations as well.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false,
})
export class UseHttpImageSourcePipe implements PipeTransform, OnDestroy {
  private subscription = new Subscription();
  private loadingImagePath!: string;
  private errorImagePath!: string;
  private latestValue!: string | SafeUrl;
  private transformValue = new BehaviorSubject<string>('');

  // ...

  transform(
    imagePath: string,
    loadingImagePath?: string,
    errorImagePath?: string
  ): string | SafeUrl {
    this.setLoadingAndErrorImagePaths(loadingImagePath, errorImagePath);
    if (!imagePath) {
      return this.errorImagePath;
    }

    this.transformValue.next(imagePath);
    return this.latestValue || this.loadingImagePath;
  }

  // ...

  private setUpSubscription(): void {
    const transformSubscription = this.transformValue
      .asObservable()
      .pipe(
        filter((v): v is string => !!v),
        switchMap((imagePath: string) => this.httpClient
          .get(imagePath, { observe: 'response', responseType: 'blob' })
          .pipe(
            map((response: HttpResponse<Blob>) => URL.createObjectURL(response.body)),
            map((unsafeBlobUrl: string) => this.domSanitizer.bypassSecurityTrustUrl(unsafeBlobUrl)),
            filter((blobUrl) => blobUrl !== this.latestValue),
            // if the request errors out we return the error image's path value
            catchError(() => of(this.errorImagePath))
          )
        ),
        tap((imagePath: string | SafeUrl) => {
          this.latestValue = imagePath;
          this.cdr.markForCheck();
        })
      )
      .subscribe();
    this.subscription.add(transformSubscription);
  }

  // ...

  private setLoadingAndErrorImagePaths(
    loadingImagePath: string,
    errorImagePath: string
  ): void {
    if (this.loadingImagePath && this.errorImagePath) {
      return;
    }
    this.loadingImagePath = loadingImagePath;
    this.errorImagePath = errorImagePath;
  }

}

Finally, when the image is not loaded, our error image will be displayed.

When the image loads and an error happens

But what about developer experience?

Adding the loading and error image paths becomes a tedious job when you have more than one place where you need to set those up. We would like to keep that functionality if we ever need to override default values on another page. Let's set up our pipe's container module in a way that we can set default values in the app's root module.

First, we create the injectors:

import { InjectionToken } from '@angular/core';

export const THIS_DOT_LOADING_IMAGE_PATH = new InjectionToken<string>('THIS_DOT_LOADING_IMAGE_PATH');
export const THIS_DOT_ERROR_IMAGE_PATH = new InjectionToken<string>('THIS_DOT_ERROR_IMAGE_PATH');

Then, we create our forRoot method:

@NgModule({
  imports: [CommonModule],
  declarations: [UseHttpImageSourcePipe],
  exports: [UseHttpImageSourcePipe],
})
export class UseHttpImageSourcePipeModule {
  static forRoot(
    config: { loadingImagePath?: string; errorImagePath?: string } = {}
  ): ModuleWithProviders<UseHttpImageSourcePipeModule> {
    return {
      ngModule: UseHttpImageSourcePipeModule,
      providers: [
        // set up the providers
        {
          provide: THIS_DOT_LOADING_IMAGE_PATH,
          useValue: config.loadingImagePath || null,
        },
        {
          provide: THIS_DOT_ERROR_IMAGE_PATH,
          useValue: config.errorImagePath || null,
        },
      ],
    };
  }
}

And in our app.module.ts file:

@NgModule({
  // ...
  imports: [
    BrowserModule,
    HttpClientModule,
    UseHttpImageSourcePipeModule.forRoot({
      loadingImagePath: 'assets/images/loading.png',
      errorImagePath: 'assets/images/error.png',
    }),
    // ...
  ],
  // ...
})
export class AppModule {}

With this setup, we can inject the THIS_DOT_LOADING_IMAGE_PATH and the THIS_DOT_ERROR_IMAGE_PATH in our pipe, and update the logic so it defaults to that.

@Pipe({
  name: 'useHttpImgSrc',
  pure: false
})
export class UseHttpImageSourcePipe implements PipeTransform, OnDestroy {
  // ...
  constructor(
    private httpClient: HttpClient,
    private domSanitizer: DomSanitizer,
    private cdr: ChangeDetectorRef,
    @Inject(THIS_DOT_LOADING_IMAGE_PATH) private defaultLoadingImagePath: string,
    @Inject(THIS_DOT_ERROR_IMAGE_PATH) private defaultErrorImagePath: string
  ) {
  }

  transform(
    imagePath: string,
    loadingImagePath?: string,
    errorImagePath?: string
  ): string | SafeUrl {
    this.setLoadingAndErrorImagePaths(loadingImagePath, errorImagePath);
    // ...
  }

  // ...

  private setLoadingAndErrorImagePaths(
    loadingImagePath: string = this.defaultLoadingImagePath,
    errorImagePath: string = this.defaultErrorImagePath
  ): void {
    if (this.loadingImagePath && this.errorImagePath) {
      return;
    }
    this.loadingImagePath = loadingImagePath;
    this.errorImagePath = errorImagePath;
  }
}

With that, we can update our templates.

  <img width="200px" [src]="'assets/images/success.png' | useHttpImgSrc" />
  // ...
  <img width="200px" [src]="'assets/images/notfound.png' | useHttpImgSrc" />

And everything works as before. We can still override the loading and error images by passing properties to the pipe, but with Angular's dependency injection, we made it simpler to use default values. Using the injection tokens we can also override the values on component level if we ever need that.

This Dot is a consultancy dedicated to guiding companies through their modernization and digital transformation journeys. Specializing in replatforming, modernizing, and launching new initiatives, we stand out by taking true ownership of your engineering projects.

We love helping teams with projects that have missed their deadlines or helping keep your strategic digital initiatives on course. Check out our case studies and our clients that trust us with their engineering.

You might also like

Angular 18 Announced: Zoneless Change Detection and More cover image

Angular 18 Announced: Zoneless Change Detection and More

Angular 18 Announced: Zoneless Change Detection and More Angular 18 has officially landed, and yet again, the Angular team has proven that they are listening to the community and are committed to continuing the framework's renaissance. The release polishes existing features, addresses common developer requests, and introduces experimental zoneless change detection. Let's examine the major updates and enhancements that Angular 18 offers. 1. Zoneless Angular One of the most exciting features of Angular 18 is the introduction of experimental support for zoneless change detection. Historically, Zone.js has been responsible for triggering Angular's change detection. However, this approach has downsides, especially regarding performance and debugging complexity. The Angular team has been working towards making Zone.js optional, and Angular 18 marks a significant milestone in this journey. Key Features of Zoneless Angular 1. Hybrid Change Detection: In Angular 18, a new hybrid change detection mode is enabled by default. This mode allows Angular to listen to changes from signals and other notifications regarding changes that occur either inside an Angular zone or not. That effectively means you can write (library) code that works regardless of whether Zone.js is being used, paving the way to fully zoneless apps without compromising backward compatibility. For new applications, Angular enables zone coalescing by default, which removes the number of change detection cycles and improves performance. For existing applications, you can enable zone coalescing by configuring your NgZone provider in bootstrapApplication: ` 2. Experimental API for Zoneless Mode: Angular 18 introduces an experimental API to disable Zone.js entirely. This API allows developers to run applications in a fully zoneless mode, paving the way for improved performance and simpler debugging. The zoneless change detection requires an entirely new scheduler which relies on notifications from the Angular core APIs, such as ChangeDetectorRef.markForCheck (called automatically by the AsyncPipe), ComponentRef.setInput, signal updates, host listener updates, or attaching a view that was marked dirty. 3. Improved Composability and Interoperability: Zoneless change detection enhances composability for micro-frontends and interoperability with other frameworks. It also offers faster initial render and runtime, smaller bundle sizes, more readable stack traces, and simpler debugging. How to Try Zoneless Angular To experiment with zoneless change detection, you can add the provideExperimentalZonelessChangeDetection provider to your application bootstrap: ` After adding the provider, remove Zone.js from your polyfills in angular.json. You can read more about the experimental zoneless change detection in the official documentation. By the way, angular.dev is now considered the official home for Angular developers! 2. Server-side Rendering and Hydration Enhancements A feature I'm particularly excited about is the improvements to Angular's server-side rendering (SSR) and hydration in terms of developer experience and debugging: 1. Enhanced DevTools Support: Angular DevTools now includes overlays and detailed error breakdowns to help visualize and debug hydration issues directly in the browser. This refreshing focus on developer experience shows that the Angular team wants to make the framework more approachable and user-friendly. 2. Hydration Support for Angular Material: All Angular Material components now support client hydration and are no longer skipped, which enhances performance and user experience. 3. Event Replay: Available in developer preview, event replay captures user interactions during SSR and replays them once the application is hydrated, ensuring a seamless user experience before complete hydration. It is powered by the same library as Google Search. 4. i18n Hydration Support: Up to v17, Angular skipped hydration for components with i18n blocks. From v18, hydration support for i18n blocks is in developer preview, allowing developers to use client hydration in internationalized applications. 3. Stable Material Design 3 After introducing experimental support for Material Design 3 in Angular 17, Angular 18 now includes stable support. The key features of Material Design 3 in Angular 18 include: 1. Simplified Theme Styles: Based on CSS variables, the new theming styles offer more granular customization and a flexible API for applying color variants to components. 2. Theming Generation Schematics: Using the Ng CLI, you can generate Material 3 themes. 3. Sass APIs: New Sass APIs allow developers to read colors, typography, and other properties from the Material 3 theme, making it easier to create custom components. How to use Material Design 3 in Angular 18 To use Material Design 3 in Angular 18, you can define a theme in your application's styles.scss file using the mat.defineTheme function: ` Or generate a Material 3 theme using the Ng CLI: ` You can then apply the theme to your application using the mat.theme mixin: ` Head to the official guide for a more detailed guide. You'll also notice they have refreshed the docs with the new themes and further documentation. 4. Signal-Based APIs The path to fully signal-based components includes new signal inputs, model inputs, and signal query APIs. We already wrote about them as they were in developer-preview in v17, but they have been further refined in v18. These APIs offer a type-safe, reactive way to manage state changes and interactions within components: 1. Signal Input API: Signal inputs allow values to be bound from parent to child components. Those values are exposed using a signal and can change during the component's life cycle. 2. Model Input API: Model inputs are a special input type that enables a component to propagate new values back to the parent component. That allows developers to keep the parent component in sync with the child component with two-way binding. 3. Signal Query API: This was a particularly requested feature from the community. The signal query APIs work the same way as ViewChild and ContentChild under the hood, but they return signals, providing more predictable timing and type safety. 5. Fallback Content For ng-content A very requested feature from the community, Angular 18 introduces a new ng-content directive that allows developers to define fallback content when no content is projected into a component. This feature is particularly useful for creating reusable components with default content. Here's an example of using the new ng-content directive. Using the following component ` like this ` will render Howdy World. 6. Other Improvements In addition to the major updates mentioned above, Angular 18 also includes several other improvements and updates: 1. TypeScript 5.4: Angular 18 now supports TypeScript 5.4, which lets you take advantage of new features such as preserved narrowing in closures following last assignments. 2. Global Observable in Angular Forms: Angular 18 introduces a global events observable in Angular forms, which allows you to track all changes around any abstract control and its children, including the touched or dirty in a single observable. Here's an example of how you can use the global observable: ` 3. Stable Deferrable views: Deferrable views are now stable in Angular 18. 4. Stable Control Flow: The built-in control flow is now stable in Angular 18! It is more performant than its predecessor. It also received improved type checking, including guardrails for certain performance-related anti-patterns. 5. Route Redirects as Functions: For added flexibility in managing redirects, Angular v18 now lets you use a function for redirectTo that returns a string, which allows you to create more sophisticated redirection logic based on runtime conditions. For example: ` Conclusion Angular 18 is a significant release that brings many new features, enhancements, and experimental APIs to the Angular ecosystem. The introduction of zoneless change detection, improvements to server-side rendering and hydration, stable Material Design 3 support, signal-based APIs, and fallback content for ng-content are just a few of the highlights of this release. The Angular team has again demonstrated its commitment to improving the framework's developer experience, performance, and flexibility. It also demonstrated a clear vision for Angular's future. If you're curious about what's next, you can check out the Angular roadmap....

Overview of the New Signal APIs in Angular cover image

Overview of the New Signal APIs in Angular

Overview of the New Signal APIs in Angular Google's Minko Gechev and Jeremy Elbourn announced many exciting things at NG Conf 2024. Among them is the addition of several new signal-based APIs. They are already in developer preview, so we can play around with them. Let's dig into it, starting with signal-based inputs and the new matching outputs API. Signal Based Inputs Discussions about signal-based inputs have been taking place in the Angular community for some time now, and they are finally here. Until now, you used the @Input() decorator to define inputs. This is what you'd have to write in your component to declare an optional and a required input: ` With the new signal-based inputs, you can write much less boilerplate code. Here is how you can define the same inputs using the new syntax: ` It's not only less boilerplate, but because the values are signals, you can also use them directly in computed signals and effects. That, effectively, means you get to avoid computing combined values in ngOnChanges or using setters for your inputs to be able to compute derived values. In addition, input signals are read-only. The New Output I intentionally avoid calling them signal-based outputs because they are not. They still work the same way as the old outputs. The Angular team has just introduced a new output API that is more consistent with the latest inputs API and allows you to write less boilerplate, similar to the new input API. Here is how you would define an output until now: ` Here is how you can define the same output using the new syntax: ` The thing I like about the new output API is that sometimes it happens to me that I forget to instantiate the EventEmitter because I do this instead: ` You won't forget to instantiate the output with the new syntax because the output function does it for you. Signal Queries I am sure most readers know the @ViewChild, @ViewChildren, @ContentChild, and @ContentChildren decorators very well and have experienced the pain of triggering the infamous ExpressionChangedAfterItHasBeenCheckedError or having the values unavailable when needed. Here is a refresher on how you would use these decorators until now: ` With the new signal queries, similar to the new input API, the values are signals, and you can use them directly in computed signals and effects. You can define the same queries using the new syntax: ` Jeremy Elbourn mentioned that the new signal queries have better type inference and are more consistent with the new input and output APIs. He also showcased a brand new feature not available with the old queries. You can now define a query as required, and the Angular compiler will throw an error if the query has no result, guaranteeing that the value won't be undefined. Here is how you can define a required query: ` Model Inputs Jeremy and Minko announced the last new feature is the model inputs. The name is vague, but the feature is cool—it simplifies the definition of two-way bindings. Until now, to achieve two-way binding, you would have to define an input and an output following a given naming convention. @Input and @Output had to be defined with the same name (followed by "Change" in the case of the output). Then, you could use the template's [()] syntax. ` ` That way, you could keep the value in sync between the parent and the child component. With the new model inputs, you can define a two-way binding with a single line of code. Here is how you can define the same two-way binding using the new syntax: ` The html template stays the same: ` The model function returns a writable signal that can be updated directly. The value will be propagated back to any two-way bindings. Conclusion The new signal-based APIs are a great addition to Angular. They allow you to write less boilerplate code and make your components more reactive. The new APIs are already in developer preview, so you can start playing around with them today. I look forward to seeing how the community will adopt these new features and what other exciting things the Angular team has in store for us, such as zoneless apps by default....

Introducing the express-typeorm-postgres Starter Kit cover image

Introducing the express-typeorm-postgres Starter Kit

Here at This Dot, we've been working with ExpressJS APIs for a while, and we've created a starter.dev kit for ExpressJS that you can use to scaffold your next backend project. The starter kit uses many well-known npm packages, such as TypeORM or BullMQ and integrates with databases such as PostgreSQL and Redis. Kit contents The express-typeorm-postgres starter kit provides you with infrastructure for development, and integrations with these infrastructures. It comes with a working Redis instance for caching and a second Redis instance for queues. It also starts up a Postgres instance for you, which you can seed with TypeORM. The infrastructure runs on docker using docker-compose. The generated project comes with prettier and eslint set-up, so you only need to spend time on configuration if you want to tweak or change the existing rules. Unit testing is set up using Jest, and there are some example tests provided with the example controllers. How to initialise API development usually requires more infrastructure than front-end development. Before you start, please make sure you have docker and docker-compose installed on your machine. To initialize a project with the express-typeorm-postgres kit, run the following: 1. Run npx @this-dot/create-starter to run the scaffolding tool 2. Select the Express.js, TypeORM, and PostgreSQL kit from the CLI library options 3. Name your project 4. cd into your project directory, and install dependencies using the tool of your choice (npm, yarn or pnpm) 5. copy the contents of the .env.example file into a .env file With this setup, you have a working starter kit that you can modify to your needs. TypeORM and Database When we started developing the kit, we decided to use PostgreSQL as the database, because it is a powerful, open-source object-relational database system that is widely used for storing and manipulating data. It has a strong reputation for reliability, performance, and feature richness, making it a great choice for a wide range of applications. It can also handle high levels of concurrency and large amounts of data, and supports complex queries and data types. Postgres is also highly extensible because it allows developers to add custom functions and data types to the database. It has a large and active community of developers and users who contribute to its ongoing development and support. The kit uses TypeORM to connect to the database instance. We chose TypeORM because it makes it easy to manage database connections and perform common database operations, such as querying, inserting, updating and deleting data. It supports TypeScript and a wide range of databases, such as PostgreSQL, MySQL, SQLite and MongoDB, therefore if you want to be able to switch between databases, it makes it easier. TypeORM also includes features such as database migrations, which help manage changes to database schema over time, and an entity model that allows you to define your database schema using classes and decorators. Overall, TypeORM is a useful tool for improving the efficiency and reliability of database-related code, and it can be a valuable addition to any TypeScript or JavaScript project that needs to interact with a database. To seed an initial set of data into your database, run the following commands: 1. npm run infrastructure:start - this starts up the database instance 2. npm run db:seed - this leverages TypeORM to seed the database. The seed command runs the src/db/run-seeders.ts file, where you can introduce your seeders for your own needs. The kit uses TypeORM-extension for seeding. Please refer to the src/db/seeding/technology-seeder.ts file for an example. Caching Storing response data in caches allows subsequent requests for the same data to be served more quickly. This can improve the performance and user experience of an API by reducing the amount of time it takes to retrieve data from the server. It can reduce the load on the database or mitigate rate limiting on third-party APIs called from your back-end. It also improves the reliability of an application by providing a fallback mechanism in case the database or the server is unavailable or slow to respond. There is a Redis instance set up in the kit to be used for caching data. Under the hood, we use the cachified library to store cached data in Redis. The kit has a useCache method exported from src/cache/cache.ts, which requires a key and a callback function to be called to fetch the data. ` When you need to invalidate cache entries, you can use the clearCacheEntry method by supplying a key string to it. It will remove the cached data from Redis, and the next request that fetches from the database will cache the new values. ` Under the src/modules/technology folder, you can see a complete example of a basic CRUD REST endpoint with caching enabled. Feel free to use those handlers as examples for your development needs. Queue A message queue allows different parts of an application, or different applications, to communicate with each other asynchronously by sending and receiving messages. This can be useful in a variety of situations, such as when one part of the application needs to perform a task that could take a long time, or when different parts of the application need to be decoupled from each other for flexibility and scalability. We chose BullMQ because it is a fast, reliable, and feature-rich message queue system. It is built on top of the popular Redis in-memory data store, which makes it very performant and scalable. It has support for parallel processing, rate limiting, retries, and a variety of other features that make it well-suited for a wide range of use cases. BullMQ has a straightforward API and good documentation. The kit has a second Redis instance set up to be used with BullMQ, and there is a queue set up out of the box, so resource-intensive tasks can be offloaded to a background process. The src/queue/ folder contains all the configuration and setup steps for the queue. Both the queue and its worker is set up in the queue.ts file. The job-processor.ts file contains the function that will process the data. To run int in a separate thread, we must pass the path to this file into the worker: ` When to use this kit This kit is most optimal when you: - want to build back-end services that can be consumed by other applications and services using ExpressJS - need a flexible and scalable way to build server-side applications - need to deal with CPU-intense operations on the server and you need a messaging queue - need to build an API with relational data - would like to just jump right into API development with ExpressJS using TypeORM and Postgres Conclusion The express-typeorm-postgres starter kit can help you kickstart your development by providing you with a working preset. It has testing configured, and it comes with a complete infrastructure orchestrated by docker-compose....

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging) cover image

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging)

The Importance of a Scientific Mindset in Software Engineering: Part 2 (Debugging) In the first part of my series on the importance of a scientific mindset in software engineering, we explored how the principles of the scientific method can help us evaluate sources and make informed decisions. Now, we will focus on how these principles can help us tackle one of the most crucial and challenging tasks in software engineering: debugging. In software engineering, debugging is often viewed as an art - an intuitive skill honed through experience and trial and error. In a way, it is - the same as a GP, even a very evidence-based one, will likely diagnose most of their patients based on their experience and intuition and not research scientific literature every time; a software engineer will often rely on their experience and intuition to identify and fix common bugs. However, an internist faced with a complex case will likely not be able to rely on their intuition alone and must apply the scientific method to diagnose the patient. Similarly, a software engineer can benefit from using the scientific method to identify and fix the problem when faced with a complex bug. From that perspective, treating engineering challenges like scientific inquiries can transform the way we tackle problems. Rather than resorting to guesswork or gut feelings, we can apply the principles of the scientific method—forming hypotheses, designing controlled experiments, gathering and evaluating evidence—to identify and eliminate bugs systematically. This approach, sometimes referred to as "scientific debugging," reframes debugging from a haphazard process into a structured, disciplined practice. It encourages us to be skeptical, methodical, and transparent in our reasoning. For instance, as Andreas Zeller notes in the book _Why Programs Fail_, the key aspect of scientific debugging is its explicitness: Using the scientific method, you make your assumptions and reasoning explicit, allowing you to understand your assumptions and often reveals hidden clues that can lead to the root cause of the problem on hand. Note: If you'd like to read an excerpt from the book, you can find it on Embedded.com. Scientific Debugging At its core, scientific debugging applies the principles of the scientific method to the process of finding and fixing software defects. Rather than attempting random fixes or relying on intuition, it encourages engineers to move systematically, guided by data, hypotheses, and controlled experimentation. By adopting debugging as a rigorous inquiry, we can reduce guesswork, speed up the resolution process, and ensure that our fixes are based on solid evidence. Just as a scientist begins with a well-defined research question, a software engineer starts by identifying the specific symptom or error condition. For instance, if our users report inconsistencies in the data they see across different parts of the application, our research question could be: _"Under what conditions does the application display outdated or incorrect user data?"_ From there, we can follow a structured debugging process that mirrors the scientific method: - 1. Observe and Define the Problem: First, we need to clearly state the bug's symptoms and the environment in which it occurs. We should isolate whether the issue is deterministic or intermittent and identify any known triggers if possible. Such a structured definition serves as the groundwork for further investigation. - 2. Formulate a Hypothesis: A hypothesis in debugging is a testable explanation for the observed behavior. For instance, you might hypothesize: _"The data inconsistency occurs because a caching layer is serving stale data when certain user profiles are updated."_ The key is that this explanation must be falsifiable; if experiments don't support the hypothesis, it must be refined or discarded. - 3. Collect Evidence and Data: Evidence often includes logs, system metrics, error messages, and runtime traces. Similar to reviewing primary sources in academic research, treat your raw debugging data as crucial evidence. Evaluating these data points can reveal patterns. In our example, such patterns could be whether the bug correlates with specific caching mechanisms, increased memory usage, or database query latency. During this step, it's essential to approach data critically, just as you would analyze the quality and credibility of sources in a research literature review. Don't forget that even logs can be misleading, incomplete, or even incorrect, so cross-referencing multiple sources is key. - 4. Design and Run Experiments: Design minimal, controlled tests to confirm or refute your hypothesis. In our example, you may try disabling or shortening the cache's time-to-live (TTL) to see if more recent data is displayed correctly. By manipulating one variable at a time - such as cache invalidation intervals - you gain clearer insights into causation. Tools such as profilers, debuggers, or specialized test harnesses can help isolate factors and gather precise measurements. - 5. Analyze Results and Refine Hypotheses: If the experiment's outcome doesn't align with your hypothesis, treat it as a stepping stone, not a dead end. Adjust your explanation, form a new hypothesis, or consider additional variables (for example, whether certain API calls bypass caching). Each iteration should bring you closer to a better understanding of the bug's root cause. Remember, the goal is not to prove an initial guess right but to arrive at a verifiable explanation. - 6. Implement and Verify the Fix: Once you're confident in the identified cause, you can implement the fix. Verification doesn't stop at deployment - re-test under the same conditions and, if possible, beyond them. By confirming the fix in a controlled manner, you ensure that the solution is backed by evidence rather than wishful thinking. - Personally, I consider implementing end-to-end tests (e.g., with Playwright) that reproduce the bug and verify the fix to be a crucial part of this step. This both ensures that the bug doesn't reappear in the future due to changes in the codebase and avoids possible imprecisions of manual testing. Now, we can explore these steps in more detail, highlighting how the scientific method can guide us through the debugging process. Establishing Clear Debugging Questions (Formulating a Hypothesis) A hypothesis is a proposed explanation for a phenomenon that can be tested through experimentation. In a debugging context, that phenomenon is the bug or issue you're trying to resolve. Having a clear, falsifiable statement that you can prove or disprove ensures that you stay focused on the real problem rather than jumping haphazardly between possible causes. A properly formulated hypothesis lets you design precise experiments to evaluate whether your explanation holds true. To formulate a hypothesis effectively, you can follow these steps: 1. Clearly Identify the Symptom(s) Before forming any hypothesis, pin down the specific issue users are experiencing. For instance: - "Users intermittently see outdated profile information after updating their accounts." - "Some newly created user profiles don't reflect changes in certain parts of the application." Having a well-defined problem statement keeps your hypothesis focused on the actual issue. Just like a research question in science, the clarity of your symptom definition directly influences the quality of your hypothesis. 2. Draft a Tentative Explanation Next, convert your symptom into a statement that describes a _possible root cause_, such as: - "Data inconsistency occurs because the caching layer isn't invalidating or refreshing user data properly when profiles are updated." - "Stale data is displayed because the cache timeout is too long under certain load conditions." This step makes your assumption about the root cause explicit. As with the scientific method, your hypothesis should be something you can test and either confirm or refute with data or experimentation. 3. Ensure Falsifiability A valid hypothesis must be falsifiable - meaning it can be proven _wrong_. You'll struggle to design meaningful experiments if a hypothesis is too vague or broad. For example: - Not Falsifiable: "Occasionally, the application just shows weird data." - Falsifiable: "Users see stale data when the cache is not invalidated within 30 seconds of profile updates." Making your hypothesis specific enough to fail a test will pave the way for more precise debugging. 4. Align with Available Evidence Match your hypothesis to what you already know - logs, stack traces, metrics, and user reports. For example: - If logs reveal that cache invalidation events aren't firing, form a hypothesis explaining why those events fail or never occur. - If metrics show that data served from the cache is older than the configured TTL, hypothesize about how or why the TTL is being ignored. If your current explanation contradicts existing data, refine your hypothesis until it fits. 5. Plan for Controlled Tests Once you have a testable hypothesis, figure out how you'll attempt to _disprove_ it. This might involve: - Reproducing the environment: Set up a staging/local system that closely mimics production. For instance with the same cache layer configurations. - Varying one condition at a time: For example, only adjust cache invalidation policies or TTLs and then observe how data freshness changes. - Monitoring metrics: In our example, such monitoring would involve tracking user profile updates, cache hits/misses, and response times. These metrics should lead to confirming or rejecting your explanation. These plans become your blueprint for experiments in further debugging stages. Collecting and Evaluating Evidence After formulating a clear, testable hypothesis, the next crucial step is to gather data that can either support or refute it. This mirrors how scientists collect observations in a literature review or initial experiments. 1. Identify "Primary Sources" (Logs, Stack Traces, Code History): - Logs and Stack Traces: These are your direct pieces of evidence - treat them like raw experimental data. For instance, look closely at timestamps, caching-related events (e.g., invalidation triggers), and any error messages related to stale reads. - Code History: Look for related changes in your source control, e.g. using Git bisect. In our example, we would look for changes to caching mechanisms or references to cache libraries in commits, which could pinpoint when the inconsistency was introduced. Sometimes, reverting a commit that altered cache settings helps confirm whether the bug originated there. 2. Corroborate with "Secondary Sources" (Documentation, Q&A Forums): - Documentation: Check official docs for known behavior or configuration details that might differ from your assumptions. - Community Knowledge: Similar issues reported on GitHub or StackOverflow may reveal known pitfalls in a library you're using. 3. Assess Data Quality and Relevance: - Look for Patterns: For instance, does stale data appear only after certain update frequencies or at specific times of day? - Check Environmental Factors: For instance, does the bug happen only with particular deployment setups, container configurations, or memory constraints? - Watch Out for Biases: Avoid seeking only the data that confirms your hypothesis. Look for contradictory logs or metrics that might point to other root causes. You keep your hypothesis grounded in real-world system behavior by treating logs, stack traces, and code history as primary data - akin to raw experimental results. This evidence-first approach reduces guesswork and guides more precise experiments. Designing and Running Experiments With a hypothesis in hand and evidence gathered, it's time to test it through controlled experiments - much like scientists isolate variables to verify or debunk an explanation. 1. Set Up a Reproducible Environment: - Testing Environments: Replicate production conditions as closely as possible. In our example, that would involve ensuring the same caching configuration, library versions, and relevant data sets are in place. - Version Control Branches: Use a dedicated branch to experiment with different settings or configuration, e.g., cache invalidation strategies. This streamlines reverting changes if needed. 2. Control Variables One at a Time: - For instance, if you suspect data inconsistency is tied to cache invalidation events, first adjust only the invalidation timeout and re-test. - Or, if concurrency could be a factor (e.g., multiple requests updating user data simultaneously), test different concurrency levels to see if stale data issues become more pronounced. 3. Measure and Record Outcomes: - Automated Tests: Tests provide a great way to formalize and verify your assumptions. For instance, you could develop tests that intentionally update user profiles and check if the displayed data matches the latest state. - Monitoring Tools: Monitor relevant metrics before, during, and after each experiment. In our example, we might want to track cache hit rates, TTL durations, and query times. - Repeat Trials: Consistency across multiple runs boosts confidence in your findings. 4. Validate Against a Baseline: - If baseline tests manifest normal behavior, but your experimental changes manifest the bug, you've isolated the variable causing the issue. E.g. if the baseline tests show that data is consistently fresh under normal caching conditions but your experimental changes cause stale data. - Conversely, if your change eliminates the buggy behavior, it supports your hypothesis - e.g. that the cache configuration was the root cause. Each experiment outcome is a data point supporting or contradicting your hypothesis. Over time, these data points guide you toward the true cause. Analyzing Results and Iterating In scientific debugging, an unexpected result isn't a failure - it's valuable feedback that brings you closer to the right explanation. 1. Compare Outcomes to the hypothesis. For instance: - Did user data stay consistent after you reduced the cache TTL or fixed invalidation logic? - Did logs show caching events firing as expected, or did they reveal unexpected errors? - Are there only partial improvements that suggest multiple overlapping issues? 2. Incorporate Unexpected Observations: - Sometimes, debugging uncovers side effects - e.g. performance bottlenecks exposed by more frequent cache invalidations. Note these for future work. - If your hypothesis is disproven, revise it. For example, the cache may only be part of the problem, and a separate load balancer setting also needs attention. 3. Avoid Confirmation Bias: - Don't dismiss contrary data. For instance, if you see evidence that updates are fresh in some modules but stale in others, you may have found a more nuanced root cause (e.g., partial cache invalidation). - Consider other credible explanations if your teammates propose them. Test those with the same rigor. 4. Decide If You Need More Data: - If results aren't conclusive, add deeper instrumentation or enable debug modes to capture more detailed logs. - For production-only issues, implement distributed tracing or sampling logs to diagnose real-world usage patterns. 5. Document Each Iteration: - Record the results of each experiment, including any unexpected findings or new hypotheses that arise. - Through iterative experimentation and analysis, each cycle refines your understanding. By letting evidence shape your hypothesis, you ensure that your final conclusion aligns with reality. Implementing and Verifying the Fix Once you've identified the likely culprit - say, a misconfigured or missing cache invalidation policy - the next step is to implement a fix and verify its resilience. 1. Implementing the Change: - Scoped Changes: Adjust just the component pinpointed in your experiments. Avoid large-scale refactoring that might introduce other issues. - Code Reviews: Peer reviews can catch overlooked logic gaps or confirm that your changes align with best practices. 2. Regression Testing: - Re-run the same experiments that initially exposed the issue. In our stale data example, confirm that the data remains fresh under various conditions. - Conduct broader tests - like integration or end-to-end tests - to ensure no new bugs are introduced. 3. Monitoring in Production: - Even with positive test results, real-world scenarios can differ. Monitor logs and metrics (e.g. cache hit rates, user error reports) closely post-deployment. - If the buggy behavior reappears, revisit your hypothesis or consider additional factors, such as unpredicted user behavior. 4. Benchmarking and Performance Checks (If Relevant): - When making changes that affect the frequency of certain processes - such as how often a cache is refreshed - be sure to measure the performance impact. Verify you meet any latency or resource usage requirements. - Keep an eye on the trade-offs: For instance, more frequent cache invalidations might solve stale data but could also raise system load. By systematically verifying your fix - similar to confirming experimental results in research - you ensure that you've addressed the true cause and maintained overall software stability. Documenting the Debugging Process Good science relies on transparency, and so does effective debugging. Thorough documentation guarantees your findings are reproducible and valuable to future team members. 1. Record Your Hypothesis and Experiments: - Keep a concise log of your main hypothesis, the tests you performed, and the outcomes. - A simple markdown file within the repo can capture critical insights without being cumbersome. 2. Highlight Key Evidence and Observations: - Note the logs or metrics that were most instrumental - e.g., seeing repeated stale cache hits 10 minutes after updates. - Document any edge cases discovered along the way. 3. List Follow-Up Actions or Potential Risks: - If you discover additional issues - like memory spikes from more frequent invalidation - note them for future sprints. - Identify parts of the code that might need deeper testing or refactoring to prevent similar issues. 4. Share with Your Team: - Publish your debugging report on an internal wiki or ticket system. A well-documented troubleshooting narrative helps educate other developers. - Encouraging open discussion of the debugging process fosters a culture of continuous learning and collaboration. By paralleling scientific publication practices in your documentation, you establish a knowledge base to guide future debugging efforts and accelerate collective problem-solving. Conclusion Debugging can be as much a rigorous, methodical exercise as an art shaped by intuition and experience. By adopting the principles of scientific inquiry - forming hypotheses, designing controlled experiments, gathering evidence, and transparently documenting your process - you make your debugging approach both systematic and repeatable. The explicitness and structure of scientific debugging offer several benefits: - Better Root-Cause Discovery: Structured, hypothesis-driven debugging sheds light on the _true_ underlying factors causing defects rather than simply masking symptoms. - Informed Decisions: Data and evidence lead the way, minimizing guesswork and reducing the chance of reintroducing similar issues. - Knowledge Sharing: As in scientific research, detailed documentation of methods and outcomes helps others learn from your process and fosters a collaborative culture. Ultimately, whether you are diagnosing an intermittent crash or chasing elusive performance bottlenecks, scientific debugging brings clarity and objectivity to your workflow. By aligning your debugging practices with the scientific method, you build confidence in your solutions and empower your team to tackle complex software challenges with precision and reliability. But most importantly, do not get discouraged by the number of rigorous steps outlined above or by the fact you won't always manage to follow them all religiously. Debugging is a complex and often frustrating process, and it's okay to rely on your intuition and experience when needed. Feel free to adapt the debugging process to your needs and constraints, and as long as you keep the scientific mindset at heart, you'll be on the right track....

Let's innovate together!

We're ready to be your trusted technical partners in your digital innovation journey.

Whether it's modernization or custom software solutions, our team of experts can guide you through best practices and how to build scalable, performant software that lasts.

Prefer email? hi@thisdot.co