index on OyunDev

Deploy go fiber webapp to azure cloud from private docker hub container repo

Sun, 29 Aug 2021 00:00:00 +0000

Azure WebApp running a docker container from your private docker hub repository

Microsoft Azure Cloud app services allows customers to run web applications which can be deployed using docker containers residing in Azure Container Registry, Docker Hub (public or private) and other custom docker repositories.

Just like other big-cloud vendors Microsoft Azure offers a limited time free plan and an always free tier within certain limitations. You can find list of free service offers and information about limits at this azure portal free services page . You can find source code for this article on github here

Synopsis and prerequisites

I wanted to create a gofiber webapp, running inside of a docker from scratch container, deploy and run it using azure app services and secure it using azure active directory authentication provider goth/azureadv2 . Prerequisites to do this are:

Go compiler: https://golang.org/dl/
Docker and a free docker hub account to store my containers: https://hub.docker.com
Microsoft Azure Cloud account: https://azure.microsoft.com free tier would suffice.

End result will be a simple web page that only your default Azure AD users can login:

Go webapp with gofiber and goth/azureadv2

Go code for this web app fairly simple. I was getting “Request Header Fields Too Large” error with default gofiber config. Increasing ReadBufferSize to 16K in fiber config constructor fixed this issue.

Also goth/azureadv2 provider has a known bug getting authenticated users email address info so I used a little workaround for it. You can find full source code on github here .

func main() {
	var testparam string
	flag.Parse()
	if len(strings.TrimSpace(*flagkey)) > 1 {
		testparam = strings.TrimSpace(*flagkey)
	}

	options := azureadv2.ProviderOptions{
		Scopes: []azureadv2.ScopeType{azureadv2.UserReadScope},
		Tenant: azureadv2.TenantType(tenantID),
	}
	goth.UseProviders(
		azureadv2.New(applicationID, secret, redirectUri, options),
	)

	app := fiber.New(fiber.Config{
		ReadBufferSize: 16384,
	})

	app.Get("/auth/:provider/callback", func(ctx *fiber.Ctx) error {
		user, err := gf.CompleteUserAuth(ctx)
		if err != nil {
			return err
		}

		//PROVIDER-BUG:https://github.com/markbates/goth/issues/289
		var userPrincipalName string
		for k, v := range dumpMap(user.RawData) {
			if k == "userPrincipalName" {
				userPrincipalName = v
			}
		}

		var b strings.Builder
		b.WriteString(htmlheadsrc)
		b.WriteString(`<body><ul><li><a href="/logout/azureadv2">Logout</a></li></ul><br/><table><thead><tr><th>Name</th><th>Value</th></tr></thead><tbody>`)
		if len(user.Email) < 2 {
			b.WriteString(`<tr><td nowrap>Email</td><td>` + userPrincipalName + `</td></tr>`)
		} else {
			b.WriteString(`<tr><td nowrap>Email</td><td>` + user.Email + `</td></tr>`)
		}
		b.WriteString(`<tr><td nowrap>Name</td><td>` + user.Name + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>FirstName</td><td>` + user.FirstName + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>LastName</td><td>` + user.LastName + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>NickName</td><td>` + user.NickName + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>Description</td><td>` + user.Description + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>UserID</td><td>` + user.UserID + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>AvatarURL</td><td>` + user.AvatarURL + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>Location</td><td>` + user.Location + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>AccessToken</td><td>` + user.AccessToken + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>AccessTokenSecret</td><td>` + user.AccessTokenSecret + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>RefreshToken</td><td>` + user.RefreshToken + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>ExpiresAt</td><td>` + user.ExpiresAt.String() + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>IDToken</td><td>` + user.IDToken + `</td></tr>`)
		b.WriteString(`<tr><td nowrap>userPrincipalName</td><td>` + user.RawData["userPrincipalName"].(string) + `</td></tr>`)
		b.WriteString(`</tbody></table><br/>`)
		b.WriteString(`<p>` + os.Getenv("WEBSITE_HOSTNAME") + `</p>`)
		b.WriteString(`<p>` + testparam + `</p>`)
		b.WriteString(`</body></html>`)

		ctx.Set("Content-Type", "text/html")
		ctx.Send([]byte(b.String()))
		return nil
	})

	app.Get("/logout/:provider", func(ctx *fiber.Ctx) error {
		gf.Logout(ctx)
		ctx.Redirect("/")
		return nil
	})

	app.Get("/auth/:provider", func(ctx *fiber.Ctx) error {
		if authUser, err := gf.CompleteUserAuth(ctx); err == nil {
			ctx.JSON(authUser)
		} else {
			gf.BeginAuthHandler(ctx)
		}
		return nil
	})
	
	app.Get("/favicon.ico", func(ctx *fiber.Ctx) error {
		buf := []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A, 0x00, 0x00, 0x00, 0x0D, 0x49, 0x48, 0x44, 0x52, 0x00, 0x00, 0x00, 0x14, 0x00, 0x00, 0x00, 0x16, 0x08, 0x03, 0x00, 0x00, 0x00, 0xF7, 0x9F, 0x4C, 0x34, 0x00, 0x00, 0x00, 0x12, 0x50, 0x4C, 0x54, 0x45, 0xFF, 0xFF, 0xFF, 0xCC, 0xFF, 0xFF, 0x99, 0x99, 0x99, 0x66, 0x66, 0x66, 0x33, 0x33, 0x33, 0x00, 0x00, 0x00, 0x9E, 0x8B, 0x9A, 0xE7, 0x00, 0x00, 0x00, 0x02, 0x74, 0x52, 0x4E, 0x53, 0xFF, 0x00, 0xE5, 0xB7, 0x30, 0x4A, 0x00, 0x00, 0x00, 0x3D, 0x49, 0x44, 0x41, 0x54, 0x78, 0xDA, 0xCD, 0xD0, 0x31, 0x0E, 0x00, 0x20, 0x0C, 0x02, 0x40, 0xA0, 0xF2, 0xFF, 0x2F, 0x8B, 0x23, 0x89, 0xBB, 0x76, 0xB9, 0x00, 0x5B, 0xC1, 0xCB, 0x3D, 0x2F, 0xED, 0x58, 0x80, 0x96, 0x62, 0x01, 0x0B, 0x88, 0x05, 0xC2, 0x19, 0x8B, 0x94, 0xD2, 0x98, 0x05, 0x96, 0x67, 0xBC, 0x58, 0x20, 0x26, 0xB0, 0x40, 0x3C, 0xA1, 0xF8, 0xF0, 0x75, 0x1B, 0x75, 0xA3, 0x02, 0xBE, 0x47, 0x80, 0xD3, 0xF4, 0x00, 0x00, 0x00, 0x00, 0x49, 0x45, 0x4E, 0x44, 0xAE, 0x42, 0x60, 0x82}
		ctx.Set("Content-Type", "image/x-icon")
		ctx.Send(buf)
		return nil
	})

	app.Get("/", func(ctx *fiber.Ctx) error {
		var b strings.Builder
		b.WriteString(htmlheadsrc)
		b.WriteString(`<body><ul><li><a href="/auth/azureadv2">Login</a></li><li><a>Logout</a></li></ul>`)
		b.WriteString(`<p>` + os.Getenv("WEBSITE_HOSTNAME") + `</p>`)
		b.WriteString(`<p>` + testparam + `</p>`)
		b.WriteString(`</body></html>`)

		ctx.Set("Content-Type", "text/html")
		ctx.Send([]byte(b.String()))
		return nil
	})

	log.Fatal(app.Listen(":80"))
}

Dockerfile and preparing container

Because we are using a scratch container we need to provide ssl certificates manually otherwise authentication will fail. To do this we use a docker multi stage build:

FROM alpine:latest as certs
RUN apk --update add ca-certificates
# https://medium.com/on-docker/use-multi-stage-builds-to-inject-ca-certs-ad1e8f01de1b

FROM scratch
EXPOSE 80
ENV PATH=/bin
COPY --from=certs /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
COPY ./tweb1 /
CMD ["/tweb1"]

Deploy container to Azure Cloud using Portal

At this stage we should have our app container inside our docker hub private repo. I think how to use docker and docker hub is out of scope of this post so I assume we have built our container and pushed it to our private hub repository already.

Creating a web app at Azure Portal using web based UI is very easy. This is just a wizard type UI that you fill out some required values and click next to continue on next step until final page where you review all info and create the webapp. After that we need to configure authentication and create Azure AD app registration for our webapp. Here steps how to to so:

Login to Azure Portal and create a new webapp in app services.
- Bear in mind free-tier may not be available in some regions, you can change the region if necessary.
- Choose docker hub for deployment source and fill out your docker credentials.
After creating webapp, go to authentication section and click to add identity provider.
- This step will create active directory app registration and create access token secret for our app.
- It is important to allow anonymous access to users so they can click login button to start the authentication process.
Go to webapp configuration section and create tree new values for:
- APPLICATION_ID, AD_TENANT_ID, REDIRECT_URI.
- These are necessary values to authenticate users with Azure AD provider.
- You should also see settings value for MICROSOFT_PROVIDER_AUTHENTICATION_SECRET already created for us when we added identity provider in previous step.

Deploy using Azure CLI:

Azure CLI is very powerful tool to manage azure resources. We can use it to automate deployment with CI/CD pipelines. I think the easiest way to run azure cli is using docker. Run it with the command: ~$ docker run –rm -it mcr.microsoft.com/azure-cli

This script below shows how to use cli to create webapp with container, configure environment and set up authentication:

# https://docs.microsoft.com/en-us/cli/azure/install-azure-cli
# https://docs.microsoft.com/en-us/cli/azure/run-azure-cli-docker
# Using Docker: docker run -it mcr.microsoft.com/azure-cli
# ~$ sudo docker run --rm -it mcr.microsoft.com/azure-cli
# Unable to find image 'mcr.microsoft.com/azure-cli:latest' locally
# latest: Pulling from azure-cli
# 540db60ca938: Pull complete
# a7ad1a75a999: Pull complete
# 8d9a9d314bf5: Pull complete 
# Digest: sha256:2090963629cc9c595bbad24354bb6879112d8900000000000009
# Status: Downloaded newer image for mcr.microsoft.com/azure-cli:latest
# bash-5.1# az
# 
# Welcome to Azure CLI!
# ---------------------
# Use `az -h` to see available commands or go to https://aka.ms/cli.
# 
# Telemetry
# ---------
# The Azure CLI collects usage data in order to improve your experience.
# The data is anonymous and does not include commandline argument values.
# The data is collected by Microsoft.
# 
# You can change your telemetry settings with `az configure`.
# 
# 
#      /\
#     /  \    _____   _ _  ___ _
#    / /\ \  |_  / | | | \'__/ _\
#   / ____ \  / /| |_| | | |  __/
#  /_/    \_\/___|\__,_|_|  \___|
# 
# 
# Welcome to the cool new Azure CLI!
#
# this optional step, if you have more than one subscription, 
# you should set the default subscription, 
# which will be used for the remaining Azure CLI commands.
# ~$ az account set --subscription XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
#
export tenantId='XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
export webAppName='tweb1app'
export resGroup='tweb1-resgrp'
export location='northeurope'
export servicePlan='tweb1-freeplan'
export tier='F1'
export containerImg='DOCKER|dockeruser/myrepo:tweb1'
export dockerUsr='dockeruser'
export dockerPass='xxxxxxxxxxx' 
#
az login --tenant $tenantId 
# bash-5.1# az login --tenant $tenantId
# Not able to launch a browser to log you in, falling back to device code...
# To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code DA5EFUFVL to authenticate.
#
# create new resource group
az group create --name $resGroup --location $location
# create new app service plan Free tier (sku:F1)
az appservice plan create -n $servicePlan -g $resGroup --sku $tier --is-linux
# create new web app (Note: as of 2021-08-28 this command can not configure private-docker-hub-conatiner image deployments so after this command you need to issue another config-container-set command to reach desired configuration.)
az webapp create -n $webAppName -g $resGroup -p $servicePlan -i $containerImg -s $dockerUsr -w $dockerPass --tags Lifecycle=Test
az webapp stop --name $webAppName --resource-group $resGroup
#
az webapp config container set --name $webAppName --resource-group $resGroup --docker-custom-image-name 'DOCKER|dockeruser/myrepo:tweb1' --docker-registry-server-url 'https://index.docker.io/v1' --docker-registry-server-user 'dockeruser' --docker-registry-server-password 'xxxxxxxxxxx'
#
az ad app create --display-name $webAppName --available-to-other-tenants false --key-type Password --password '111xxxxxxxxxxxx11_1111xxxx1x' --reply-urls 'https://tweb1app.azurewebsites.net/auth/azureadv2/callback'
#
az webapp stop --name $webAppName --resource-group $resGroup
# a custom startup command for docker container
az webapp config set --resource-group $resGroup --name $webAppName --startup-file '/tweb1 -k sample_param-value,1714'
# configure web app settings, get the value for APPLICATION_ID which you can find inside of resulted json output from `az ad create` command like: "appId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX"  
az webapp config appsettings set --resource-group $resGroup --name $webAppName --settings APPLICATION_ID='XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX'
az webapp config appsettings set --resource-group $resGroup --name $webAppName --settings AD_TENANT_ID=$tenantId
az webapp config appsettings set --resource-group $resGroup --name $webAppName --settings REDIRECT_URI='https://tweb1app.azurewebsites.net/auth/azureadv2/callback'
az webapp config appsettings set --resource-group $resGroup --name $webAppName --settings MICROSOFT_PROVIDER_AUTHENTICATION_SECRET='111xxxxxxxxxxxx11_1111xxxx1x'
#
az webapp start --name $webAppName --resource-group $resGroup

Faster Directory Size Calculator

Sat, 22 May 2021 00:00:00 +0000

A faster version of Linux/Unix “du -hs” command implemented in Rust/Go/C#

Find out fastest Rust/Go/C# code calculating total disk usage of a directory under Windows and Linux.

This is an experiment inspired by https://github.com/artempyanykh/rdu project implementing a fast directory disk usage calculator, similar to unix/linux du tool but using rust-lang. Measurements are taken with Hyperfine v1.11.0: https://github.com/sharkdp/hyperfine. All measurements are with warm disk cache.

Linux: Debian GNU/Linux 9 (stretch) under WSL2 (Linux 4.19.128-microsoft-standard x86_64 GNU/Linux)
Windows 10 x64: Microsoft Windows [Version 10.0.19042.450]

Disk Info:

Sequential write [4MB] block        Avg: 819,08MB/s
Sequential read [4MB] block         Avg: 2412,23MB/s
Random write [4KB] block            Avg: 120,39MB/s
Random read [4KB] block             Avg: 323,64MB/s
Memory copy [4MB] block             Avg: 3703,64MB/s

Target Directory:

Files:        5403
Directories:  813
Size:         1.227.079.350 bytes

Contenders	Programming Language	Size Bytes Linux	Size Bytes Win10	Repo
du/du64.exe	C/C++	105.640	465.784	Linux: internal /usr/bin/du, Windows: https://docs.microsoft.com/en-us/sysinternals/downloads/du
rdu-async-par	Rust	1.243.632	1.126.400	https://github.com/artempyanykh/rdu
diskus	Rust	801.088	759.808	https://github.com/sharkdp/diskus
du-rs	Rust	649.568	459.776	https://github.com/oyundev/du-rs
du-go	Go	1.585.304	1.735.168	https://github.com/oyundev/du-go
du-cs	C#	5.172.232	4.855.296	https://github.com/oyundev/du-cs

Notes:

x86-64 Platform: amd64
Rust version: 1.52
Go version: go1.16.3
.Net 5.0 C# code compiled to native binary using experimental .net native AOT compiler at https://github.com/dotnet/runtimelab/tree/feature/NativeAOT/samples/HelloWorld

Linux Debian 9 (WSL2):

~$ hyperfine -L exe 'du -hs',./rdu-async-par,'./du-rs -p 64 -t 64','./du-cs 64',./du-go,'./diskus --threads 64' '{exe} /mnt/r/test'
Benchmark #1: du -hs /mnt/r/test

  Time (mean ± σ):     10.058 s ±  0.375 s    [User: 183.9 ms, System: 3104.2 ms]
  Range (min … max):    9.562 s … 10.570 s    10 runs

Benchmark #2: ./rdu-async-par /mnt/r/test
  Time (mean ± σ):      5.426 s ±  0.030 s    [User: 723.1 ms, System: 6757.7 ms]
  Range (min … max):    5.390 s …  5.494 s    10 runs

Benchmark #3: ./du-rs -p 64 -t 64 /mnt/r/test
  Time (mean ± σ):      3.717 s ±  0.016 s    [User: 235.6 ms, System: 5625.0 ms]
  Range (min … max):    3.687 s …  3.735 s    10 runs

Benchmark #4: ./du-cs 64 /mnt/r/test
  Time (mean ± σ):      4.566 s ±  0.040 s    [User: 682.7 ms, System: 6599.2 ms]
  Range (min … max):    4.515 s …  4.642 s    10 runs

Benchmark #5: ./du-go /mnt/r/test
  Time (mean ± σ):      3.210 s ±  0.016 s    [User: 404.6 ms, System: 4832.7 ms]
  Range (min … max):    3.192 s …  3.231 s    10 runs

Benchmark #6: ./diskus --threads 64 /mnt/r/test
  Time (mean ± σ):      4.552 s ±  0.141 s    [User: 1.723 s, System: 7.016 s]
  Range (min … max):    4.305 s …  4.732 s    10 runs

Summary
  './du-go /mnt/r/test' ran
    1.16 ± 0.01 times faster than './du-rs -p 64 -t 64 /mnt/r/test'
    1.42 ± 0.04 times faster than './diskus --threads 64 /mnt/r/test'
    1.42 ± 0.01 times faster than './du-cs 64 /mnt/r/test'
    1.69 ± 0.01 times faster than './rdu-async-par /mnt/r/test'
    3.13 ± 0.12 times faster than 'du -hs /mnt/r/test'

Command	Mean [s]	Min [s]	Max [s]	Relative
`du -hs /mnt/r/test`	10.058 ± 0.375	9.562	10.570	3.13 ± 0.12
`./rdu-async-par /mnt/r/test`	5.426 ± 0.030	5.390	5.494	1.69 ± 0.01
`./du-rs -p 64 -t 64 /mnt/r/test`	3.717 ± 0.016	3.687	3.735	1.16 ± 0.01
`./du-cs 64 /mnt/r/test`	4.566 ± 0.040	4.515	4.642	1.42 ± 0.01
`./du-go /mnt/r/test`	3.210 ± 0.016	3.192	3.231	1.00
`./diskus --threads 64 /mnt/r/test`	4.552 ± 0.141	4.305	4.732	1.42 ± 0.04

Windows 10 (x64):

PS> .\hyperfine -L exe "du64.exe -nobanner",rdu-async-par.exe,du-rs.exe,"du-cs.exe -t 64",du-go.exe,"diskus.exe --threads 64" "{exe} r:\test"
Benchmark #1: du64.exe -nobanner r:\test

  Time (mean ± σ):     703.1 ms ±   5.3 ms    [User: 1.4 ms, System: 6.2 ms]
  Range (min … max):   692.4 ms … 709.2 ms    10 runs

Benchmark #2: rdu-async-par.exe r:\test
  Time (mean ± σ):     344.9 ms ±  24.3 ms    [User: 4.1 ms, System: 5.4 ms]
  Range (min … max):   325.1 ms … 411.0 ms    10 runs

Benchmark #3: du-rs.exe r:\test
  Time (mean ± σ):      72.4 ms ±   5.1 ms    [User: 3.5 ms, System: 5.8 ms]
  Range (min … max):    66.8 ms …  93.3 ms    28 runs

Benchmark #4: du-cs.exe -t 64 r:\test
  Time (mean ± σ):      86.4 ms ±  10.5 ms    [User: 1.9 ms, System: 4.6 ms]
  Range (min … max):    71.3 ms … 109.5 ms    29 runs

Benchmark #5: du-go.exe r:\test
  Time (mean ± σ):      57.9 ms ±   2.4 ms    [User: 0.6 ms, System: 2.6 ms]
  Range (min … max):    54.1 ms …  63.5 ms    42 runs

Benchmark #6: diskus.exe --threads 64 r:\test
  Time (mean ± σ):     156.8 ms ±   4.5 ms    [User: 2.5 ms, System: 4.4 ms]
  Range (min … max):   150.4 ms … 169.4 ms    16 runs

Summary
  'du-go.exe r:\test' ran
    1.25 ± 0.10 times faster than 'du-rs.exe r:\test'
    1.49 ± 0.19 times faster than 'du-cs.exe -t 64 r:\test'
    2.71 ± 0.14 times faster than 'diskus.exe --threads 64 r:\test'
    5.96 ± 0.49 times faster than 'rdu-async-par.exe r:\test'
   12.14 ± 0.51 times faster than 'du64.exe -nobanner r:\test'

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
`du64.exe -nobanner r:\test`	703.1 ± 5.3	692.4	709.2	12.14 ± 0.51
`rdu-async-par.exe r:\test`	344.9 ± 24.3	325.1	411.0	5.96 ± 0.49
`du-rs.exe r:\test`	72.4 ± 5.1	66.8	93.3	1.25 ± 0.10
`du-cs.exe -t 64 r:\test`	86.4 ± 10.5	71.3	109.5	1.49 ± 0.19
`du-go.exe r:\test`	57.9 ± 2.4	54.1	63.5	1.00
`diskus.exe --threads 64 r:\test`	156.8 ± 4.5	150.4	169.4	2.71 ± 0.14

Who is the winner ?

Go is fastest of all in both platforms. I tried a lot of different approaches in Rust (without unsafe hacks) to speed it up but coroutine tasks in Go-lang are realy fast and efficient.
Sadly none of my attemts using Rust tokio-main with multithreaded runtime flavor, async-std, walkdir, rayon etc. helped to speed up rust. Only using Xudong-Huang’s coroutines implementation (https://github.com/Xudong-Huang/may) almost helped to match the speed of Go implementation.
Diskus when using 64 threads is very fast and beats standard solid implementation of rdu-async-par. At some point though increasing thread count begins to degrade performance.
There is an interesting comment in diskus source code about how performance is effected by context switching in multithreading and I/O Queue.

Tue, 28 May 2019 00:00:00 +0000

About

Thu, 28 Feb 2019 00:00:00 +0000

Written in Go, Hugo is an open source static site generator available under the Apache Licence 2.0. Hugo supports TOML, YAML and JSON data file types, Markdown and HTML content files and uses shortcodes to add rich content. Other notable features are taxonomies, multilingual mode, image processing, custom output formats, HTML/CSS/JS minification and support for Sass SCSS workflows.

Hugo makes use of a variety of open source projects including:

Hugo is ideal for blogs, corporate websites, creative portfolios, online magazines, single page applications or even a website with thousands of pages.

Hugo is for people who want to hand code their own website without worrying about setting up complicated runtimes, dependencies and databases.

Websites built with Hugo are extremelly fast, secure and can be deployed anywhere including, AWS, GitHub Pages, Heroku, Netlify and any other hosting provider.

Learn more and contribute on GitHub .