Integrate Protobuf with Rabbitmq in NodeJS - Part I
Often, the task at hand involves building a system that facilitates communication between services using asynchronous methods. There are several ways to achieve this, but for our purposes, we’ll be using a queue system.
Additionally, we aim to include a protocol in our list of service contracts that allows any service to benefit from it and generate a client based on it.
This journey is divided into three parts. In the first part, we’ll focus on creating protobuf contracts and generating TypeScript definitions from them.
Let’s begin by defining an example for our use case. Imagine we have a metrics-gathering system that collects various types of information, such as GPS data and device information.
We will also use proto definitions to define the structure of the messages.
Project Setup
Let’s start by setting up the project directory
mkdir nodejs-proto-rabbit
cd nodejs-proto-rabbit
npm init -y
mkdir src
mkdir proto
Installing required dependencies
npm install -D typescript @types/node
Let’s initialise ts-config.json file by running:
npx tsc --init
This will add a ts-config.json file to the root of the project with all defaults.
Implementation
For GPS data we will define the following structure:
// proto/gps.message.proto
syntax = "proto3";
import "google/protobuf/timestamp.proto";
message GPSMessage {
google.protobuf.Timestamp time = 1;
string latitude = 2;
string longitude = 3;
}
For Device Information data we will define the following structure:
// proto/device-information.message.proto
syntax = "proto3";
import "google/protobuf/timestamp.proto";
message DeviceInformationMessage {
google.protobuf.Timestamp time = 1;
string mac = 2;
string name = 3;
}
After defining these proto definitions, the next step is to find a library that can generate TypeScript definitions. My preferred choice is @bufbuild/protoc-gen-es, which is a code generator plugin for Protocol Buffers tailored for ECMAScript.
Following their guidelines how to install, we run the following commands
npm install --save-dev @bufbuild/protoc-gen-es
npm install @bufbuild/protobuf
In order to generate we will use the simple approach with buf. In order to proceed with that we have to define a file such buf.gen.yaml where we store all the configuration for the generation
// buf.gen.yaml
version: v2
plugins:
# This will invoke protoc-gen-es and write output to gen
- local: protoc-gen-es
out: gen
opt:
# Add more plugin options here
- target=ts
To generate code for all Protobuf files within our project, we run the following command:
npm install --save-dev @bufbuild/buf
npx buf generate
In your files there should be two new files, gps.message_pb.ts & device-information.message_pb.ts, with content as the example below:
// gen/proto/gps.message_pb.ts
// @generated by protoc-gen-es v2.0.0
// with parameter "target=ts"
// @generated from file
// proto/gps.message.proto (syntax proto3)
/* eslint-disable */
import type {
GenFile,
GenMessage
} from "@bufbuild/protobuf/codegenv1";
import {
fileDesc,
messageDesc
} from "@bufbuild/protobuf/codegenv1";
import type {
Timestamp
} from "@bufbuild/protobuf/wkt";
import {
file_google_protobuf_timestamp
} from "@bufbuild/protobuf/wkt";
import type { Message } from "@bufbuild/protobuf";
/**
* Describes the file proto/gps.message.proto.
*/
export const file_proto_gps_message: GenFile
= fileDesc("Chdwcm90by9ncHM ....(continues)");
/**
* @generated from message GPSMessage
*/
export type GPSMessage = Message<"GPSMessage"> & {
/**
* @generated from field:
* google.protobuf.Timestamp time = 1;
*/
time?: Timestamp;
/**
* @generated from field: string latitude = 2;
*/
latitude: string;
/**
* @generated from field: string longitude = 3;
*/
longitude: string;
};
/**
* Describes the message GPSMessage.
* Use `create(GPSMessageSchema)` to create a new message.
*/
export const GPSMessageSchema: GenMessage<GPSMessage>
= messageDesc(file_proto_gps_message, 0);
// gen/proto/device-information.message_pb.ts
// @generated by protoc-gen-es v2.0.0
// with parameter "target=ts"
// @generated from file
// proto/device-information.message.proto
/* eslint-disable */
import type {
GenFile,
GenMessage
} from "@bufbuild/protobuf/codegenv1";
import {
fileDesc,
messageDesc
} from "@bufbuild/protobuf/codegenv1";
import type {
Timestamp
} from "@bufbuild/protobuf/wkt";
import {
file_google_protobuf_timestamp
} from "@bufbuild/protobuf/wkt";
import type { Message } from "@bufbuild/protobuf";
/**
* Describes the file
* proto/device-information.message.proto.
*/
export const file_proto_device_information_message ...;
/**
* @generated from message DeviceInformationMessage
*/
export type DeviceInformationMessage
= Message<"DeviceInformationMessage"> & {
/**
* @generated from field:
* google.protobuf.Timestamp time = 1;
*/
time?: Timestamp;
/**
* @generated from field: string mac = 2;
*/
mac: string;
/**
* @generated from field: string name = 3;
*/
name: string;
};
/**
* Describes the message DeviceInformationMessage.
* Use `create(DeviceInformationMessageSchema)`
* to create a new message.
*/
export const DeviceInformationMessageSchema: ...;
To generate a message, we need to write the following lines of code:
import { create } from '@bufbuild/protobuf';
import {
DeviceInformationMessageSchema
} from 'gen/proto/device-information.message_pb';
const message = create(DeviceInformationMessageSchema, {
mac: '00:11:22:33:44:55',
name: 'device',
});
console.log(message);
//{
// '$typeName': 'DeviceInformationMessage',
// mac: '00:11:22:33:44:55',
// name: 'device'
//}
In the second part of this journey, we will focus on building a generic solution for RabbitMQ that can be used for every message. The third part will involve diving into the configuration of all these components to work together seamlessly in a simple project.
Keep pushing forward and savor every step of your coding journey.