Beginner
Intermediate
Advance
Blockchain
Comparison
VersionsUpdates

Building Microservices in Golang - Basic to Advance

Building Microservices in Golang's picture

We love Golang and want to contribute to the Golang community. After researching for a Microservices blog I could find blogs where the basic difference between monolithic and microservices is explained or the use of gin and gorilla framework for the microservices is available.

Therefore, we have taken the task of writing a long series of blogs to completely set up or build a live project using microservices. It will be our longest blog series with all practical things where we will complete a project from scratch using microservices. We will have 21 parts to this blog tutorial. Here are the details:

  1. Project brief and creating data sender
  2. Setting up Kafka
  3. Producing Kafka and logging middleware
  4. Implementing distance service
  5. Implementing Invoice aggregator
  6. Aggregator HTTP client
  7. Aggregate Invoice API handler
  8. Implementing GRPC and Protobuf
  9. Aggregate GRPC client
  10. Internal Service communication
  11. Custom HTTP Gateway
  12. Running and Testing Gateway with all services
  13. Metric layer with Prometheus
  14. Grafana
  15. DevOps setup for the project
  16. Custom Error Handling
  17. Request Tracing
  18. Building services with Gokit- setup
  19. Setting up circuit breaking and rate limiting
  20. Finishing aggregator services with Go kit


Get ready to start the first part and launch yourself knowing the best of golang.

Project Brief:

What is microservices in Golang?

Microservice is a small service that does one activity at a time. Say if it is storing value, or receiving value. It is just that it is doing one task and we have several microservice to complete a project.

What is microservices in Golang

In this program, there are multiple trucks we call on-board units driving in a state. So these onboard units send GPS signals to one of our services. Let’s name it receivers. The job of receivers is to get latitude and longitude GPS coordinates from obs unit. And put that in the Kafka queue.

The reason to use Kafka is supposedly because our distance calculator is not working and we miss getting the GPS coordinates, so we can not invoice based on the limited information. But with the help of Kafka, we will receive all coordinates in Kafka and if the service is down we can replay Kafka data so that the right value is invoiced.

Distance calculator pulls the coordinates from Kafka and calculates the distance based on latitude and longitude.

Then the distance is passed to the invoice generator to get the invoice for the customers.
The invoice generator will call the service invoice calculator to calculate the invoice. This is also used as a standalone so customers can put the coordinates and see the proposed invoice bill before using the services.

The invoice generator will save the invoice to the Database. The invoice generator will be connected to API gateway to get invoice or invoice data back.

I hope you will love making this project now. It is one of its kind which will be long and complex and we will do all this together. This will make your Golang journey from beginner to advanced Golang developers.

Project setup for obu


Let us setup the project and name it toll-calculator and in that create obu folder and in that create main.go

In this, we will make something that simulates an onboard unit which will be sending GPS connection in regular intervals so we will be creating a web socket connection that will be sending messages to the web socket connection. And from web socket our receiver microservice will receive the GPS coordinates

In main.go file

Let’s first create obu data struct

1 2 3 4 5 type OBUDATA struct { OBUID int `json:"obuid"` Lat float64 `json:"lat"` Long float64 `json:"long"` }

We will be binding data there using JSON. Obuid will help us identify the truck and Lat and Lang will store the coordinates.

Now we will generate the coordinates:

1 2 3 4 5 func genCoord() float64{ n:=float64(rand.Intn(100)+1) f :=rand.Float64() return n+f }

rand.Intn(100) will give us a random number between 0 and 100 so we added one so that we never receive 0 as a value and added that to another random number generated in the float.

Now, create a function that will generate coordinates calling genCoordtwice

1 2 3 4 5 func genLatLong() (float64, float64) { return genCoord(), genCoord() }

Just to make sure we have random data we will create an init function that will initialize at the start of the program and seed the data.

1 2 3 func init() { rand.Seed(time.Now().UnixNano()) }

We need to generate random OBUs

1 2 3 4 5 6 7 func generateOBUIDS(n int)[]int{ ids:= make([]int,n) for i:=0; i<n; i++{ ids[i]=rand.Intn(math.MaxInt) } return ids }

Here n will be the number of ids generated
In function main we will assign the number of Obusid to be 20 and every second these 20 Obus will have different coordinates.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 func main() { obuIDs :=generateOBUIDS(20) for { for i := 0; i < len(obuIDs); i++ { lat, long := genLatLong() data:= OBUDATA{ OBUID: obuIDs[i], Lat:lat, Long:long, } fmt.Printf("%+v/n",data) } time.Sleep(sendtimeinterval) } }

Here is the sample output generated every second

{OBUID:8989538902329231495 Lat:35.14136459291441 Long:29.221480201177027}/n{OBUID:9186940219822894195 Lat:11.66887547645881 Long:67.53295376047541}/n{OBUID:3702423557580638999 Lat:56.00944857983324 Long:12.188696783609902}/n{OBUID:203522087024529392 Lat:37.41741275915713 Long:20.04137169529667}/n{OBUID:4497805496821290525 Lat:38.543587662860986 Long:86.28845854687073}/n{OBUID:619357277175535044 Lat:38.02944330808388 Long:74.84902546593072}/n{OBUID:4682882019870889782 Lat:21.69251102824321 Long:15.911887139335414}/n{OBUID:6340217492303384951 Lat:23.260380867845587 Long:61.86451028273996}/n{OBUID:7656946716200206197 Lat:54.62625610272676 Long:77.64384071560322}/n{OBUID:180849369364738802 Lat:14.147730732999143 Long:75.2152050839426}/n{OBUID:768599825226656255 Lat:57.84511739599838 Long:11.760496647184384}/n{OBUID:1491583917833820328 Lat:48.95522513040361 Long:15.716889329637887}/n{OBUID:1319931517003611052 Lat:34.45919303574331 Long:92.96466495206326}/n{OBUID:1282939034134693823 Lat:99.22337262965765 Long:11.251294545579196}/n{OBUID:534115152870930757 Lat:25.513741743884616 Long:38.381013835842936}/n{OBUID:2202089226941032807 Lat:10.93014754209808 Long:63.84129437379379}/n{OBUID:3748183091786525994 Lat:32.937385461773246 Long:28.230709634371983}/n{OBUID:6365895452718625076 Lat:5.510889372924315 Long:50.26332215161264}/n{OBUID:6391781997162461253 Lat:42.92106934825001 Long:5.417316013420223}/n{OBUID:667864433690811737 Lat:51.286656469973295 Long:45.514217203661914}/n{OBUID:8989538902329231495 Lat:91.32769974123478 Long:39.46841855495135}/n{OBUID:9186940219822894195 Lat:29.94267816692366 Long:3.157410435515387}/n{OBUID:3702423557580638999 Lat:57.15487839581777 Long:56.658243373786696}/n{OBUID:203522087024529392 Lat:42.65800705028815 Long:50.48347810764381}/n{OBUID:4497805496821290525 Lat:92.3589964373187 Long:22.21345449208416}/n{OBUID:619357277175535044 Lat:69.90711263514308 Long:44.137284407172}/n{OBUID:4682882019870889782 Lat:96.90964606491374 Long:5.152749028461344}/n{OBUID:6340217492303384951 Lat:89.80765314671203 Long:69.21480817588936}/n{OBUID:7656946716200206197 Lat:25.079860877143524 Long:40.694920418445946}/n{OBUID:180849369364738802 Lat:43.14041489160797 Long:29.266328906894294}/n{OBUID:768599825226656255 Lat:53.64496536787735 Long:65.90180703898292}/n{OBUID:1491583917833820328 Lat:22.178246765654755 Long:93.21912359224306}/n{OBUID:1319931517003611052 Lat:68.97848181418102 Long:16.274152146461827}/n{OBUID:1282939034134693823 Lat:52.16875707748394 Long:77.05708536777648}/n{OBUID:534115152870930757 Lat:99.7804605516221 Long:39.580944431634535}/n{OBUID:2202089226941032807 Lat:46.63383180682249 Long:11.544324099862726}/n{OBUID:3748183091786525994 Lat:73.27855348563918 Long:75.94378970390535}/n{OBUID:6365895452718625076 Lat:47.2254208055999 Long:22.136547985464524}/n{OBUID:6391781997162461253 Lat:91.73709848220896 Long:77.58881731776391}/n{OBUID:667864433690811737 Lat:79.19801567756036 Long:27.503930292658826}/n{OBUID:8989538902329231495 Lat:26.27389617183455 Long:45.16252100512445}/n{OBUID:9186940219822894195 Lat:2.883956113112139 Long:81.74954697588433}/n{OBUID:3702423557580638999 Lat:56.077366741302214 Long:19.73765279275143}/n{OBUID:203522087024529392 Lat:90.5953289565079 Long:15.196401465139552}/n{OBUID:4497805496821290525

Lat:40.69004394172066 Long:16.189759170831074}/n{OBUID:619357277175535044 Lat:44.72000412305799 Long:48.10479865434942}/n{OBUID:4682882019870889782 Lat:40.169963820565 Long:90.07188668474606}/n{OBUID:6340217492303384951 Lat:9.295398742642094 Long:3.688452129745553}/n{OBUID:7656946716200206197 Lat:70.36144158649455 Long:10.00919082588468}/n{OBUID:180849369364738802 Lat:72.86387330048478 Long:21.31881822846235}/n{OBUID:768599825226656255 Lat:61.959187256591285 Long:76.64086436722893}/n{OBUID:1491583917833820328 Lat:26.841289722295596 Long:65.53175959019445}/n{OBUID:1319931517003611052 Lat:27.5177001147005 Long:92.23576004043993}/n{OBUID:1282939034134693823 Lat:72.02252866311294 Long:80.52823917632158}/n{OBUID:534115152870930757 Lat:65.12513935427165 Long:50.15980193458817}/n{OBUID:2202089226941032807 Lat:57.03274550689483 Long:10.936072863384188}/n{OBUID:3748183091786525994 Lat:18.179209897665924 Long:96.58773925810384}/

Build Make File

Let's make a Makefile in our root so that we can bootstrap it later on.

1 2 3 4 5 6 7 obu: @go build -o bin/obu obu/main.go @./bin/obu .PHONY: obu

Those, who don’t understand what is makefile can refer to our other links where we have mentioned in detail about makefile.

Once we have created the Obuids as discussed before we will create a WebSocket to send the data to the receiver service. For WebSocket, we will use gorilla/websocket

Sending Obudata

First, we will create an endpoint then we will create a dialer in main.go

1 2 3 4 5 6 const wsEndpoint = "ws://127.0.0.1:30000/ws" conn, _, err := websocket.DefaultDialer.Dial(wsEndpoint, nil) if err != nil { log.Fatal(err) }

Then we write the JSON data

1 2 3 4 if err := conn.WriteJSON(data); err != nil { log.Fatal(err) } }

Now, it is time to create an OBU receiver service. Let's create a new folder in our project and name it data_reciever and create a new main.go file

Ok, we are creating microservices we need something to share between the two services so let's share the obu data and create a new folder types with types.go file and move the data struct in there.

Code in types.go

1 2 3 4 5 6 7 8 package types type OBUDATA struct { OBUID int `json:"obuid"` Lat float64 `json:"lat"` Long float64 `json:"long"` }

Let’s open data_reciever main.go and create a channel with the type OBUDATA

1 2 3 4 type DataReceiver struct { msgch chan types.OBUDATA conn *websocket.Conn }


Here msg will be a channel that will receive OBUDATA and conn to close the channels. The concept of channel is covered earlier you can look there for more details.

Let me first share the receiver signal main.go code and explain the same.

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 package main import ( "fmt" "log" "net/http" "github.com/anurag4way/go-concurrency/types" "github.com/gorilla/websocket" ) func main() { recv := NewDataReceiver() http.HandleFunc("/ws", recv.handleWS) http.ListenAndServe(":30000", nil) fmt.Println("data receiver working fine") } type DataReceiver struct { msgch chan types.OBUDATA conn *websocket.Conn } func NewDataReceiver() *DataReceiver { return &DataReceiver{ msgch: make(chan types.OBUDATA, 128), } } func (dr *DataReceiver) handleWS(w http.ResponseWriter, r *http.Request) { u := websocket.Upgrader{ ReadBufferSize: 1028, WriteBufferSize: 1028, } conn, err := u.Upgrade(w, r, nil) if err != nil { log.Fatal(err) } dr.conn = conn go dr.wsRecieveLoop() } func (dr *DataReceiver) wsRecieveLoop() { fmt.Println("new obu client connected") for { var data types.OBUDATA if err := dr.conn.ReadJSON(&data); err != nil { log.Println("read error:", err) continue } fmt.Printf("received OBU data from [%d]:: <lat %.2f, Long %.2f>\n", data.OBUID, data.Lat, data.Long) } } recv := NewDataReceiver()


Here we will create a channel msgch which is a buffered channel with 128 bytes size. Then we will call handleWS function which will handle the response from OBU unit and read the data coming from OBU unit at 30000 port and save the data as pointers in OBU data struct which is printed at the bottom.

Next, we will send the data received from services to the kafka.

Build Your Golang Team