14 October 2021/ miscellaneous

service definitions with protocol buffers

We’ve all had those days when we defined our API contracts in a swagger.json file and shared it with the client folks. This approach to sharing the signature or the contract has its own inherent sorts of issues due to the way json is interpreted. Now alongside these issues, the problems get amplified when we have a set of services which are communicating with each other over HTTP + JSON in a microservices based architecture.

For example let’s consider an imaginary system where we have the following set of services

User Service (Taking care of managing user information)
Auth Service (Taking care of managing user access tokens)
BFF (Orchestrator)

Let’s imagine a simple login flow which results in the orchestrator talking to user and auth services

Call to BFF 

POST ${bff_url}/v1/login
{
  "email": "abc@xyz.com",
  "password":"super-secret"
}

Response
{
  "access_token": "asdfghjkinl==",
  "refresh_token": "rgnmoijhdfghj==",
  "expires_in": 7200
}

Call to User Service
PUT ${user_svc_url}/v1/users
{
  "email": "abc@xyz.com",
  "password":"super-secret"
}

Response
{
  "first_name": "foo",
  "last_name": "bar",
  "deleted": false,
  "status": "active"
  "external_id": "aabbc-ddeeff-gghhh",
  "id": 1
} 

Call to Auth Service
PUT ${auth_svc_url}/v1/auth-tokens
{
  "first_name": "foo",
  "last_name": "bar",
  "deleted": false,
  "status": "active"
  "external_id": "aabbc-ddeeff-gghhh",
  "id": 1  
}

Response 
{
  "access_token": "asdfghjkinl==",
  "refresh_token": "rgnmoijhdfghj==",
  "expires_in": 7200
}

Orchestration

Ignoring the optimizations which can be done, focussing only on the functionality here, the BFF service is going to have two gateway classes (UserSvcGateway, AuthSvcGateway) which takes care of handling the communication b/w itself and the corresponding services.

If you are one of those developers like me who hates writing gateway classes because most of the code there is going be repetitive, welcome to the club! :P Reason for hatred being, you’ll have to define classes or structs which will correspond to the request and the responses, assuming that there’ll be a fair amount of error scenarios, you’ll have to define corresponding classes or structs for these as well. This will violate the DRY principle because the same set of definitions will be present in the owner services as well.

Solution

Protocol Buffers

Protocol buffers are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages.

How does solve this pain point?

With protocol buffers we will define our service definitions with the contracts in a file called, protocol buffer files, which will be acted upon by special tools like protoc, buf etc to generate source code in a number of supported languages (java, python, javascript, golang etc). What this means is that, all the request response related classes or struct files will be auto generated.

If we carefully package all the service definitions in a common git repository, we can have CI/CD pipelines which will take the sources(protocol buffer files) in the repository and generate source code in the language of choice. The generated source code can be packaged up into packages(java/golang/js/python) and pushed into either artifactory/npm or any of the package managers of choice.

Using this approach, you get the following benefits

Central repository or single source of truth for all the service definitions and their associated contracts.
Source code generation in any supported language of choice.
Version controlling the APIs out of the box.

Example

syntax = "proto3";

package users.v1;

import "commons/v1/commons.proto";
import "google/protobuf/timestamp.proto";
import "google/api/annotations.proto";

message UserDto {
  string first_name = 1;
  string last_name = 2;
  commons.v1.Gender gender = 4;
  google.protobuf.Timestamp dob = 5;
  string external_id = 6;
  commons.v1.Relation relation = 7;
}

message IdentityDto {
  commons.v1.IdentityType type = 1;
  string value = 2;
  string external_id = 3;
}

message AddressDto {
  string line_1 = 1;
  string line_2 = 2;
  string city = 3;
  string province = 4;
  string country = 5;
  string zipcode = 6;
  double latitude = 7;
  double longitude = 8;
  string external_id = 9;
}

message CreateUserRequest {
  UserDto request = 1;
}

message CreateUserResponse {
  UserDto response = 1;
}

message UpdateUserRequest {
  string user_id = 1;
  UserDto request = 2;
}

message UpdateUserResponse {
  UserDto response = 1;
}

message GetUserByIdRequest {
  string user_id = 1;
}

message GetUserByIdResponse {
  UserDto response = 1;
}

message BlockUserRequest {
  string user_id = 1;
}

message BlockUserResponse {
  bool status = 1;
}

message CreateUserIdentityRequest{
  string user_id = 1;
  IdentityDto request = 2;
}

message CreateUserIdentityResponse {
  IdentityDto response = 1;
}

message UpdateUserIdentityRequest {
  string user_id = 1;
  string identity_id = 2;
  IdentityDto request = 3;
}

message UpdateUserIdentityResponse {
  IdentityDto response = 1;
}

message CreateUserRelationRequest {
  string primary_user_id = 1;
  UserDto request = 2;
}

message CreateUserRelationResponse {
  UserDto response = 1;
}

message DeleteUserRelationRequest {
  string primary_user_id = 1;
  string relation_id = 2;
}

message DeleteUserRelationResponse {
  bool status = 1;
}

message GetUserRelationsRequest {
  string user_id = 1;
}

message GetUserRelationsResponse {
  repeated UserDto response = 1;
}

message GetUserIdentitiesRequest {
  string user_id = 1;
}

message GetUserIdentitiesResponse {
  repeated IdentityDto response = 1;
}

message CreateUserAddressRequest {
  AddressDto request = 1;
  string user_id = 2;
}

message CreateUserAddressResponse {
  AddressDto response = 1;
}

message UpdateUserAddressRequest {
  AddressDto request = 1;
  string address_id = 2;
  string user_id = 3;
}

message UpdateUserAddressResponse {
  AddressDto response = 1;
}

message GetUserAddressesRequest {
  string user_id = 1;
}

message GetUserAddressesResponse {
  repeated AddressDto response = 1;
}

service UserService {
  rpc CreateUser(CreateUserRequest) returns (CreateUserResponse) {
    option(google.api.http) = {
      post: "/v1/users"
      body: "*"
    };
  }

  rpc UpdateUser(UpdateUserRequest) returns (UpdateUserResponse){
    option(google.api.http) = {
      patch:"/v1/users/{user_id}"
      body: "request"
    };
  }

  rpc GetUser(GetUserByIdRequest) returns (GetUserByIdResponse){
    option(google.api.http) = {
      get:"/v1/users/{user_id}"
    };
  }

  rpc BlockUser(BlockUserRequest) returns (BlockUserResponse){
    option(google.api.http) = {
      put: "/v1/users/{user_id}/block"
      body: "*"
    };
  }

  rpc CreateUserIdentity(CreateUserIdentityRequest) returns (CreateUserIdentityResponse){
    option(google.api.http) = {
      post:"/v1/users/{user_id}/identities"
      body:"request"
    };
  }

  rpc UpdateUserIdentity(UpdateUserIdentityRequest) returns (UpdateUserIdentityResponse){
    option(google.api.http) = {
      patch:"/v1/users/{user_id}/identities/{identity_id}"
      body:"request"
    };
  }

  rpc GetUserIdentities(GetUserIdentitiesRequest) returns (GetUserIdentitiesResponse){
    option(google.api.http) = {
      get: "/v1/users/{user_id}/identities"
    };
  }

  rpc CreateUserRelation(CreateUserRelationRequest) returns (CreateUserRelationResponse){
    option(google.api.http) = {
      post: "/v1/users/{primary_user_id}/relations"
      body:"request"
    };
  }

  rpc DeleteUserRelation(DeleteUserRelationRequest) returns (DeleteUserRelationResponse){
    option(google.api.http) = {
      delete:"/v1/users/{primary_user_id}/relations/{relation_id}"
    };
  }

  rpc CreateUserAddress(CreateUserAddressRequest) returns (CreateUserAddressResponse){
    option(google.api.http) = {
      post:"/v1/users/{user_id}/addresses"
      body:"request"
    };
  }

  rpc UpdateUserAddress(UpdateUserAddressRequest) returns (UpdateUserAddressResponse){
    option(google.api.http) = {
      patch:"/v1/users/{user_id}/addresses/{address_id}"
      body:"request"
    };
  }

  rpc GetUserAddresses(GetUserAddressesRequest)returns(GetUserAddressesResponse){
    option(google.api.http) = {
      get:"/v1/users/{user_id}/addresses"
    };
  }
}

The above example uses buf for generating golang source code files, which is done with github actions, the generated source contains both the server and the client interfaces which can be independently imported and implemented, that giving us the flexibility of switching b/w languages based on the need.

CI/CD definitions and the setup will be shared in another blog post, please feel to reach out to me if the above stuff seems interesting to you!!!

Happy Coding!

service definitions with protocol buffers

Solution

Protocol Buffers

How does solve this pain point?

Continuous integration with argo-events

object allocations in golang