In the previous article we created the foundations to implement a database engine, not matter what type of engine, the IRepository set the direction of how we can interact with the technologies and database engine. We implement the PostgreSQL but we could have used MySQL, and the differences would have been minimal, in this opportunity we will integrate the NoSQL approach to demonstrate that not only can we switch to a db engine to another, also that good practices, separations of concerns and software engineering fundamentals play an important role in the technical decisions can be awful at the beginning but it save a lot of time in the future.

Common Criticisms of NoSQL

NoSQL is the next replacement of SQL

Not really, NoSQL comes to play to start thinking the projects in terms of scalability and as an extension of Relational Model capabilities.

NoSQL work well with Node.js only because the asynchronous model

This myth was spread a couple years before, that was the beggining of MERN (MongoDB, Express, React, Node.js) as a definitive stack and the facilities between the technologies since all have in common the same language javascript, today we can found multiple languages like python and java are implementing mongodb.

NoSQL is slow

Poor queries and bad data modeling in NoSQL is an indicator of bad performance. Also if the modeling is not taking in consideration the metrics and real needs of the business, that will be painful.

NoSQL means No SQL anymore

The real meaning of NoSQL is Not only SQL or Not only relational, which mean that NoSQL comes to fill the existing gaps in the relational model, bringing flexibility, and design using the metrics of the application to set the business rules in order to optimize, store, retrieval and updates.

Free Resources to Refresh Your NoSQL(MongoDB) Knowledge

At the time this article is written mongodb.com company launched their own FREE course catalog to learn mongodb called MongoDB University I taken a few of their courses and I don’t regret it, I really recommend them if you want to learn seriously.

Is MongoDB use an ORM?

In the previous article we introduce the concept of ORM, in the case of MongoDB exists libraries and ORMs in node.js like mongoose. My opinion is the same, sometimes ORM are overkill, in particular when the mongodb node.js driver has everything we need.

Installing MongoDB using Docker compose

In this article I will use the MongoDB official database to start creating our CRUD of Note entity, the docker image package is on the mongodb_registry. The docker-containers will be managed with Docker Compose, which requires creating a compose.yml file. MongoDB docker image requires a SSL to start the database engine to keep everything Encrypted at Rest so we will extend their image using a docker file. To keep everything separated in the root folder lets create a folder containers/mongodb and the Dockerfile for mongodb extension.

The Dockerfile contains the next content:

1
FROM mongo:latest
2
RUN openssl rand -base64 756 > /etc/mongo-keyfile
3
RUN chmod 400 /etc/mongo-keyfile
4
RUN chown mongodb:mongodb /etc/mongo-keyfile

Which run openssl command to generate the key that will use mongodb engine to start. Then we are able to use that docker image in the compose.yml

1
services:
2
  mongodb:
3
    build:
4
      context: ./containers/mongodb
5
      dockerfile: Dockerfile
6
    restart: always
7
    environment:
8
       MONGO_INITDB_ROOT_USERNAME: mongo
9
       MONGO_INITDB_ROOT_PASSWORD: mongo
10
       MONGO_INITDB_DATABASE: mongo
11
    ports:
12
      - 27017:27017
13
    command: --replSet rs0 --keyFile /etc/mongo-keyfile --bind_ip_all --port 27017
14
    healthcheck:
15
      test: echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'127.0.0.1:27017'}]}) }" | mongosh --port 27017 -u mongo -p mongo --authenticationDatabase admin
16
      interval: 5s
17
      timeout: 15s
18
      start_period: 15s
19
      retries: 10
20
    volumes:
21
      - "mongo_data:/data/db"
22
      - "mongo_config:/data/configdb"
23
    networks:
24
      - adaptable_backend_nodejs
25

26
volumes:
27
  mongo_data:
28
  mongo_config:

We are defining the service mongodb which use the latest version of mongodb, set the variables MONGO_INITDB_ for credentials. Added networks to keep the application isolated, in case you have running other local services, volumes to keep the persistency even if we restart the machine, ports to forward to host machine (ours) and healthcheck a command that is quite complex, but it’s the only one I found to check health on mongodb.

Running the mongodb service

Running the command:

1
npm run prepare:dev

Check in the docker desktop if the container mongodb is running, then you are ready for start building.

Integrate MongoDB to our node.js project

Following the guidelines of the project, the database nosql stuff will reside inside core/database/nosql folder with the purpose of handling all persistence layer there. To begin, install the official connector library for mongodb, I will use mongodb driver official, is good maintained and provide the majority of tools required.

In the previous article we introduce the structure and contract of persistence by using interfaces and base classes, we will create the MongoDB adaptation.

The MongoDB Repository

Implementing the IRepository using the mongodb client and follow the contract will guarantee the interoperability in the application.

Lets add the NoSQLRepository:

1
import {
2
  Collection, Filter,
3
  MongoClient, ObjectId, OptionalUnlessRequiredId, UpdateFilter,
4
} from "mongodb";
5

6
import {BaseEntity} from "@core/database/BaseEntity";
7
import { CreatePayload, IRepository, UpdatePayload} from "@core/database/IRepository";
8
import { config } from "src/core/configuration/configuration";
9

10
export class NoSQLRepository<Entity extends BaseEntity> implements IRepository<Entity> {
11
  private readonly tableName: string;
12
  private client: MongoClient;
13
  protected collection: Collection<Entity>;
14

15
  constructor(tableName: string) {
16
    this.tableName = tableName;
17
    this.client = new MongoClient(config.get("DATABASE_URL"));
18
    this.collection = this.client.db().collection<Entity>(this.tableName);
19
    this.connect();
20
  }
21
  // shortened for brevity
22
}

We are importing all toys from mongodb driver library to start integrating to the CRUD, then create the NoSQLRepository to implement the IRepository contract. In the constructor we are receiving the tableName or CollectionName in the MongoDB world, having as attributes the client who will manage all connection things and collection will perform the queries. I implemented the private connect() to verify the connection.

1
  private async connect(): Promise<void> {
2
    try {
3
      await this.client.connect();
4
      this.client.on("error", (error) => {
5
        console.error("MongoDB connection error:", error);
6
      });
7
    } catch (error) {
8
      console.error("Failed to connect to MongoDB:", error);
9
      process.exit(1);
10
    }
11
  }

In the constructor we call private method connect(), if the connection cannot be established the application finish with error, the database is a very important dependency we can’t risk start the app if the database connection is not stablish.

Implementing the create() method

1
  async create(data: CreatePayload<Entity>): Promise<Entity> {
2
    const now = new Date();
3
    const documentToInsert = {
4
      ...data,
5
      createdAt: now,
6
      updatedAt: now,
7
    };
8

9
    const result = await this.collection.insertOne(
10
      documentToInsert as OptionalUnlessRequiredId<Entity>
11
    );
12

13
    return {
14
      id: result.insertedId.toHexString(),
15
      ...documentToInsert,
16
    } as Entity;
17
  }

Due to limitations of mongodb with autogenerated fields, we will add the date manually, convert the generic Entity into OptionalUnlessRequiredId and return the inserted item.

Implementing the find or Read methods

We perform the collection find method to retrieve all data available in the database and map the result to be compliant with the BaseEntity.

1
  async findAll(): Promise<Entity[]> {
2
    const result = await this.collection.find<Entity>({}).toArray();
3
    return result.map((item) => {
4
      const { _id, ...rest } = item as any;
5
      return { id: _id.toHexString(), ...rest } as Entity;
6
    });
7
  }

Mongodb provide Filter class that allow add criteria of filter similar to WHERE clause of SQL, but just using an object. Also, we need to convert explicitilythe id into ObjectId since in mongodb are using a string hash instead an integer sequence by default.

1
  async findById(id: string | number): Promise<Entity | null> {
2
    const query: Filter<Entity> = { _id: new ObjectId(id) } as Filter<Entity>;
3
    const result = await this.collection.findOne(query);
4

5
    if (!result) {
6
      return null;
7
    }
8
    const { _id, ...rest } = result as any;
9
    return { id: _id.toHexString(), ...rest } as Entity;
10
  }

Implementing the Update

Using the Filter class to prepare the criteria, UpdateFilter to build the document to update, and using the findOneAndUpdate() method that mongodb provides to check if the document exists first and then updates it.

1
  async update(id: string | number, data: UpdatePayload<Entity>): Promise<Entity | null> {
2
    const query: Filter<Entity> = { _id: new ObjectId(id) } as Filter<Entity>;
3
    const updateDocument: UpdateFilter<Entity> = {
4
      $set: {
5
        ...data,
6
        updatedAt: new Date(),
7
      } as Partial<Entity>,
8
    };
9

10
    const result = await this.collection.findOneAndUpdate(
11
      query,
12
      updateDocument,
13
      { returnDocument: 'after' }
14
    );
15

16
    if (!result) {
17
      return null;
18
    }
19

20
    const { _id, ...rest } = result as any;
21
    return { id: _id.toHexString(), ...rest } as Entity;
22
  }

Implementing the delete

The deletion is self-explanatory and follow the same patterns of previous implementation

1
  async delete(id: string | number): Promise<boolean> {
2
    const query: Filter<Entity> = { _id: new ObjectId(id) } as Filter<Entity>;
3
    const result = await this.collection.deleteOne(query);
4
    return result.deletedCount > 0;
5
  }

Adding the database mongodb environment variables

As we did in compose.yml file, we need to update our .env to load that configuration, comment out the SQL engine, add the variables:

1
# Database SQL
2
#DATABASE_URL=postgresql://postgres:postgres@localhost:5432/postgres
3
#DATABASE_ENGINE=postgres
4

5
# Database NoSQL
6
#DATABASE_URL=mongodb://mongo:mongo@localhost:27017/mongo?authSource=admin&replicaSet=rs0
7
#DATABASE_ENGINE=mongodb

Create the NoSQL setup

MongoDB and the NoSQL movement facilitate the schema building and provide flexibility giving some agility in the process to start buidling, in our case we don’t require create schema migration since we will use the agnostic entities in the features folder.

Creating the seeders

Similarly we did for [SQL](previous article) we will replicate the the same idea.

1
const now = new Date();
2

3
export const notes = {
4
  collection: "notes",
5
  data: [
6
    {
7
      content: "The only limit to our realization of tomorrow is our doubts of today.",
8
      timesSent: 0,
9
      createdAt: now,
10
      updatedAt: now,
11
    },
12
    {
13
      content: "Do what you can, with what you have, where you are.",
14
      timesSent: 0,
15
      createdAt: now,
16
      updatedAt: now,
17
    },
18
    {
19
      content: "The best way to predict the future is to invent it.",
20
      timesSent: 0,
21
      createdAt: now,
22
      updatedAt: now,
23
    }
24
  ],
25
};

We will create the script that will run all seeders:

1
import { MongoClient } from "mongodb";
2

3
import { notes } from "@core/database/nosql/seeders/notes";
4
import { config } from "src/core/configuration/configuration";
5

6
export class Seeder {
7
  private static readonly seeders = [
8
    notes,
9
  ];
10

11
  static async run() {
12
    const dbConnection = new MongoClient(config.get("DATABASE_URL"));
13
    console.log("Running seeders... ");
14

15
    try {
16

17
      await dbConnection.connect();
18

19
      for (const seeder of Seeder.seeders) {
20
        const collection = dbConnection.db().collection(seeder.collection);
21
        console.log(`Running seeder: ${seeder.collection}`);
22
        await collection.insertMany(seeder.data);
23
      }
24
    } catch (error) {
25
      console.error("error seeding: ", error);
26
    } finally {
27
      await dbConnection.close();
28
    }
29
  }
30
}
31

32
Seeder.run();

then lets update the script to simplify the process of seeding

1
"scripts": {
2
    ...
3
  "seeder": "NODE_OPTIONS=--no-warnings npx ts-node -r tsconfig-paths/register src/core/database/sql/seeder.ts",
4
 "seederNoSQL": "NODE_OPTIONS=--no-warnings npx ts-node -r tsconfig-paths/register src/core/database/sql/seeder.ts"
5
},

Make sure the docker services are running performing npm run prepare:dev first.

Running npm run seederNoSQL the note items are stored in database, using the Compass mongoDB client, you can see the data created:

Updating the NoteRepository

After creating all the infrastructure for NoSQL, now we need to update our previous strategy in the constructor to choose a database engine depending of the environment variables.

1
export class NoteRepository implements IRepository<Note> {
2
  private DB_COLLECTION_NAME = "notes";
3
  private repository: IRepository<Note>;
4

5
  constructor() {
6
    const dbEngine = config.get("DATABASE_ENGINE");
7

8
    if (dbEngine === DatabaseEngine.SQL) {
9
      this.repository = new SQLRepository<Note>(this.DB_COLLECTION_NAME);
10
    } else if (dbEngine === DatabaseEngine.NOSQL) {
11
     // To be implemented
12
     this.repository = new NoSQLRepository<Note>(this.DB_COLLECTION_NAME);
13
    }
14
  }
15
// shorten for brevity...
16
}

Updating the NoteController

Not really, thanks to the separation of concerns and modularity that were applied, there is no need to update anything, same apply for NoteRestController.

Conclusion

In this example switching between SQL to NoSQL approach just by using a few design patterns and good practices. The REAL need to switch between a technology to another resides in the correct interpretation of the needs of the business, some organizations tend to use both database engines, other require a huge refactoring session to integrate a new database engine because at upfront did not apply good practices. In the inference and AI world a type of database can coexist in the same backend application, we need to deep dive into the needs of the project and take strategic desicions that make the backend easy to change in the long-term and adaptable to the persistency needs. I recommend to read this wonderful hellointerview.com - sql vs nosql article explaining the key differences between them. As usual all the code will be uploaded in the Github Repo for your replication and follow up.

Have you ever faced similar issues when required to implement a database engine at your Back-end applications? - Please let me know in the comments.