MB

▄▄▄·▪ ▐ ▄ ▄▄▄ . ▐█ ▄███ •█▌▐█▀▄.▀· ██▀·▐█·▐█▐▐▌▐▀▀▪▄ ▐█▪·•▐█▌██▐█▌▐█▄▄▌ .▀ ▀▀▀▀▀ █▪ ▀▀▀

DO $$ DECLARE cols text; query text; BEGIN SELECT string_agg(quote_ident(name) || ' text', ', ') INTO cols FROM ( SELECT name FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o ORDER BY priority DESC ) AS o; BEGIN EXECUTE 'DROP MATERIALIZED VIEW IF EXISTS device_analysis_summary'; query := format(' CREATE MATERIALIZED VIEW device_analysis_summary AS SELECT * FROM crosstab( ''SELECT d.deveui, da.name, da.value FROM devices d LEFT JOIN device_analyses da ON d.deveui = da.device_id ORDER BY d.deveui, da.name'', ''SELECT name FROM (SELECT DISTINCT name, priority FROM device_analyses) AS o ORDER BY priority DESC'' ) AS ct(deveui text, %s); ', cols); EXECUTE query; EXCEPTION WHEN OTHERS THEN RAISE NOTICE 'Error creating materialized view: %', SQLERRM; ROLLBACK; RETURN; END; END $$;

Finally added cache busting to mb. I should have done this long before but I only learned this technique recently. For some reason I never wondered how bigger sites did this.

For those who don’t know what cache busting is. It’s a technique used to force a web browser to download the latest version of a static file (css, img, …) instead of using a cached version. This is done by making every version of a file have a different and unique name, so the browser won’t recognize the updated file and will fetch it from the server. I have done this by adding a hash of the file to the filename, for example main-ad5d104a4a86.css.

With every version being a unique file we can update the Cache-Control header.

Cache-Control: public, max-age=31536000, immutable

This the most aggressive caching that you can configure, it tells the browser that this file will never change and that it will never need to refetch it from the server. This could be a problem for normal files but this is perfect for this usecase because we have control of the clients cache from the server.

Normally these hashes would be added at build/bundle time. But Golang doesn’t have these options so I did it at start-up of the server. Luckily Golang is very fast so it didn’t add any time to the start up (1-2 ms). The implementation works better then expected.

The perfectionist within me did try it with code-gen and it did work, see commit. But it didn’t feel clean or maintainable and was drifting from the goal of simplicity of this project, so I decided to scrap it.

<label> Choose a flavor: <input list="ice-cream-flavors" name="ice-cream-choice" /> </label> <datalist id="ice-cream-flavors"> <option value="Chocolate"></option> <option value="Coconut"></option> <option value="Mint"></option> <option value="Strawberry"></option> <option value="Vanilla"></option> </datalist>

<a href="/auth" accesskey="a" aria-hidden="true" tabindex="-1"></a> <a href="/media" accesskey="m" aria-hidden="true" tabindex="-1"></a> <a href="/backup" accesskey="b" aria-hidden="true" tabindex="-1"></a>

Docker vs Native binary

Tested the backend of this blog on small VM on my desktop with both a native binary and Docker to compare performance. My desktop could have a lot of variance so this test is not that accurate. The VM has 2GB of ram and 4 cores allocated. Memory is not limiting factor as the webserver uses 20MB at most. All 4 cores are utilised but the one core is always more used then the other, I assume that this is the case because the webserver is database limited for the most part. SQLite is in some ways single threaded.

  1  [|||||||||||||||||||||||||||||||   78.5%] 
  2  [||||||||||||||||||||||||||||||||  80.2%] 
  3  [||||||||||||||||||||||||||||||||  80.3%]
  4  [||||||||||||||||||||||||||||||||||95.7%]

I used the latest version at the time of writing.

I did 3 runs of each, alternating between the two version to limit variation. I used a tool called bombardier with 20 concurrent connections for every 10 second run. The requests are made from the host machine to give the webserver the whole VM to itself. This does add extra overhead but it is the same for both versions, so for sake of comparison this is fine.

These are the averaged results for both versions.

Binary

Statistics        Avg      Stdev        Max
  Reqs/sec       262.03     152.02     913.95
  Latency       75.22ms    53.44ms      0.93s
  Latency Distribution
     50%    71.05ms
     75%   104.18ms
     90%   127.14ms
     95%   143.13ms
     99%   188.12ms
  HTTP codes:
    1xx - 0, 2xx - 2643, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     3.15MB/s

Docker

Statistics        Avg      Stdev        Max
  Reqs/sec       260.70     150.30     962.85
  Latency       76.59ms    53.97ms      0.95s
  Latency Distribution
     50%    70.67ms
     75%   107.55ms
     90%   135.69ms
     95%   151.88ms
     99%   197.22ms
  HTTP codes:
    1xx - 0, 2xx - 2622, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     3.12MB/s

These results are really close with a very slight edge for the binary. Only 0.5% more requests per second and 1.78% lower latency on average. This is within the margin of error for this unstable setup. But even if we would take these results as 100% accurate I still think it is worth the small hit in performance. This small hit is probably even smaller when you are using a bigger server on more native hardware.

func BuildingSSE(c echo.Context) error { c.Response().Header().Set(echo.HeaderCacheControl, "no-cache") c.Response().Header().Set(echo.HeaderConnection, "keep-alive") c.Response().Header().Set(echo.HeaderContentType, "text/event-stream") queue := queue.GetBuildQueue() ctx := c.Request().Context() for { select { case result := <-queue.BuildLogsChannel: fmt.Fprint(c.Response(), buildSSE("message", result)) c.Response().Flush() case <-ctx.Done(): return nil } } } func buildSSE(event, context string) string { var result string if len(event) != 0 { result = result + "event: " + event + "\n" } if len(context) != 0 { result = result + "data: " + context + "\n" } result = result + "\n" return result }

Write your own auth!!

Docker vs Native binary

Binary

Docker