Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

• HTTP Response Snippets for Go
  Alex Edwards · Oct 19, 2013

  Taking inspiration from the Rails layouts and rendering guide, I thought it'd be a nice idea to build a snippet collection illustrating some common HTTP responses for Go web applications. Sending Headers Only Rendering Plain Text Rendering JSON Rendering XML Serving a File Rendering a HTML Template Rendering a HTML Template to a String Using Layouts and Nested Templates Sending Headers Only File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "A Go Web Server") w.WriteHeader(200) } $ curl -i localhost:3000 HTTP/1.1 200 OK Server: A Go Web Server Content-Type: text/plain; charset=utf-8 Content-Length: 0 Rendering Plain Text File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 2 OK Rendering JSON File: main.go package main import ( "encoding/json" "net/http" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} js, err := json.Marshal(profile) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/json") w.Write(js) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/json Content-Length: 56 {"Name":"Alex",Hobbies":["snowboarding","programming"]} Rendering XML File: main.go package main import ( "encoding/xml" "net/http" ) type Profile struct { Name string Hobbies []string `xml:"Hobbies>Hobby"` } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} x, err := xml.MarshalIndent(profile, "", " ") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/xml") w.Write(x) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/xml Content-Length: 128 Alex snowboarding programming Serving a File File: main.go package main import ( "net/http" "path" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { // Assuming you want to serve a photo at 'images/foo.png' fp := path.Join("images", "foo.png") http.ServeFile(w, r, fp) } $ curl -I localhost:3000 HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 236717 Content-Type: image/png Last-Modified: Thu, 10 Oct 2013 22:23:26 GMT Rendering a HTML Template File: templates/index.html Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} fp := path.Join("templates", "index.html") tmpl, err := template.ParseFiles(fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 84 Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. Rendering a HTML Template to a String Instead of passing in the http.ResponseWriter when executing your template (like in the above snippet) use a buffer instead: File: main.go ... buf := new(bytes.Buffer) if err := tmpl.Execute(buf, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } templateString := buf.String() ... Using Layouts and Nested Templates File: templates/layout.html {{ template "title" . }} {{ template "content" . }} File: templates/index.html {{ define "title" }}An example layout{{ end }} {{ define "content" }} Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. {{ end }} File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} lp := path.Join("templates", "layout.html") fp := path.Join("templates", "index.html") // Note that the layout file must be the first parameter in ParseFiles tmpl, err := template.ParseFiles(lp, fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 180 An example layout Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. If you found this post useful, you might like to subscribe to my RSS feed.

• Understanding Mutexes
  Alex Edwards · Oct 04, 2013

  For anyone new to building web applications with Go, it's important to realise that all incoming HTTP requests are served in their own Goroutine. This means that any code in or called by your application handlers will be running concurrently, and there is a risk of race conditions occurring. In case you're new to concurrent programming, I'll quickly explain the problem. Race conditions occur when two or more Goroutines try to use a piece of shared data at the same time, but the result of their operations is dependent on the exact order that the scheduler executes their instructions. As an illustration, here's an example where two Goroutines try to add money to a shared bank balance at the same time: InstructionGoroutine 1Goroutine 2Bank Balance 1Read balance ⇐ £50£50 2Read balance ⇐ £50£50 3Add £100 to balance£50 4Add £50 to balance£50 5Write balance ⇒ £150£150 6Write balance ⇒ £100£100 Despite making two separate deposits, only the second one is reflected in the final balance because the two Goroutines were racing each other to make the change. The Go blog describes the downsides: Race conditions are among the most insidious and elusive programming errors. They typically cause erratic and mysterious failures, often long after the code has been deployed to production. While Go's concurrency mechanisms make it easy to write clean concurrent code, they don't prevent race conditions. Care, diligence, and testing are required. Go provides a number of tools to help us avoid data races. These include Channels for communicating data between Goroutines, a Race Detector for monitoring unsynchronized access to memory at runtime, and a variety of 'locking' features in the Atomic and Sync packages. One of these features are Mutual Exclusion locks, or mutexes, which we'll be looking at in the rest of this post. Creating a Basic Mutex Let's create some toy code to mimic the bank balance example: import "strconv" var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { // This is racy c.amount += i } func (c *currency) Display() string { // This is racy return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } We know that if there are multiple Goroutines using this code and calling Balance.Add() and Balance.Display(), then at some point a race condition is likely to occur. One way we could prevent a data race is to ensure that if one Goroutine is using the Balance variable, then all other Goroutines are prevented (or mutually excluded) from using it at the same time. We can do this by creating a Mutex and setting a lock around particular lines of code with it. While one Goroutine holds the lock, all other Goroutines are prevented from executing any lines of code protected by the same mutex, and are forced to wait until the lock is yielded before they can proceed. In practice, it's more simple than it sounds: import ( "strconv" "sync" ) var mu = &sync.Mutex{} var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { mu.Lock() c.amount += i mu.Unlock() } func (c *currency) Display() string { mu.Lock() amt := c.amount mu.Unlock() return strconv.FormatFloat(amt, 'f', 2, 64) + " " + c.code } Here we've created a new mutex and assigned it to mu. We then use mu.Lock() to create a lock immediately before both racy parts of the code, and mu.Unlock() to yield the lock immediately after. There's a couple of things to note: The same mutex variable can be used in multiple places throughout your code. So long as it's the same mutex (in our case mu) then none of the chunks of code protected by it can be executed at the same time. Holding a mutex lock doesn't 'protect' a memory location from being read or updated. A non-mutex-locked line of code could still access it at any time and create a race condition. Therefore you need to be careful to make sure all points in your code which are potentially racy are protected. Let's tidy up the example a bit: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.Mutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.Lock() defer c.Unlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } So what's changed here? Because our mutex is only being used in the context of a currency object, it makes sense to anonymously embed it in the currency struct (an idea borrowed from Andrew Gerrard's excellent 10 things you (probably) don't know about Go slideshow). If you look at a larger codebase with lots of mutexes, like Go's HTTP Server, you can see how this approach helps to keep locking rules nice and clear. We've also made use of the defer statement, which ensures that the mutex gets unlocked immediately before a function returns. This is common practice for functions that contain multiple return statements, or where the return statement itself is racy. Read Write Mutexes In our bank balance example, having a full mutex lock on the Display() function isn't strictly necessary. It would be OK for us to have multiple reads of Balance happening at the same time, so long as nothing is being written. We can achieve this using RWMutex, a reader/writer mutual exclusion lock which allows any number of readers to hold the lock or one writer. Depending on the nature of your application and ratio of reads to writes, this may be more efficient than using a full mutex. Reader locks can be opened and closed with RLock() and RUnlock() like so: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.RWMutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.RLock() defer c.RUnlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } If you found this post useful, you might like to subscribe to my RSS feed.

• Automatic code reloading in Go
  Alex Edwards · Sep 20, 2013

  I wrote a short Bash script to automatically reload Go programs. The script acts as a light wrapper around go run, stopping and restarting it whenever a .go file in your current directory or $GOPATH/src folder is saved. I've been using it mainly when developing web applications, in the same way that I use Shotgun or Guard when working with Ruby. You can grab this from the Github repository. File: go-reload #!/bin/bash # Watch all *.go files in the specified directory # Call the restart function when they are saved function monitor() { inotifywait -q -m -r -e close_write --exclude '[^g][^o]$' $1 | while read line; do restart done } # Terminate and rerun the main Go program function restart { if [ "$(pidof $PROCESS_NAME)" ]; then killall -q -w -9 $PROCESS_NAME fi echo ">> Reloading..." go run $FILE_PATH $ARGS & } # Make sure all background processes get terminated function close { killall -q -w -9 inotifywait exit 0 } trap close INT echo "== Go-reload" echo ">> Watching directories, CTRL+C to stop" FILE_PATH=$1 FILE_NAME=$(basename $FILE_PATH) PROCESS_NAME=${FILE_NAME%%.*} shift ARGS=$@ # Start the main Go program go run $FILE_PATH $ARGS & # Monitor the /src directories in all directories on the GOPATH OIFS="$IFS" IFS=':' for path in $GOPATH do monitor $path/src & done IFS="$OIFS" # Monitor the current directory monitor . Usage The only dependency for this script is inotify-tools, which is used to monitor the filesystem for changes. $ sudo apt-get install inotify-tools Once you've downloaded (or copy-pasted) the script, you'll need to make it executable and move it to /usr/local/bin or another directory on your system path: $ wget https://raw.github.com/alexedwards/go-reload/master/go-reload $ chmod +x go-reload $ sudo mv go-reload /usr/local/bin/ You should then be able to use the go-reload command in place of go run: $ go-reload main.go == Go-reload >> Watching directories, CTRL+C to stop If you found this post useful, you might like to subscribe to my RSS feed.

• An introduction to Handlers and Servemuxes in Go
  Alex Edwards · Sep 12, 2013

  Processing HTTP requests with Go is primarily about two things: handlers and servemuxes. If you’re coming from an MVC-background, you can think of handlers as being a bit like controllers. Generally speaking, they're responsible for carrying out your application logic and writing response headers and bodies. Whereas a servemux (also known as a router) stores a mapping between the predefined URL paths for your application and the corresponding handlers. Usually you have one servemux for your application containing all your routes. Go's net/http package ships with the simple but effective http.ServeMux servemux, plus a few functions to generate common handlers including http.FileServer(), http.NotFoundHandler() and http.RedirectHandler(). Let's take a look at a simple (but slightly contrived!) example which uses these: $ mkdir example $ cd example $ go mod init example.com $ touch main.go File: main.go package main import ( "log" "net/http" ) func main() { // Use the http.NewServeMux() function to create an empty servemux. mux := http.NewServeMux() // Use the http.RedirectHandler() function to create a handler which 307 // redirects all requests it receives to http://example.org. rh := http.RedirectHandler("http://example.org", 307) // Next we use the mux.Handle() function to register this with our new // servemux, so it acts as the handler for all incoming requests with the URL // path /foo. mux.Handle("/foo", rh) log.Print("Listening...") // Then we create a new server and start listening for incoming requests // with the http.ListenAndServe() function, passing in our servemux for it to // match requests against as the second parameter. http.ListenAndServe(":3000", mux) } Go ahead and run the application: $ go run main.go 2021/12/06 15:09:43 Listening... And if you make a request to http://localhost:3000/foo you should find that it gets successfully redirected like so: $ curl -IL localhost:3000/foo HTTP/1.1 307 Temporary Redirect Content-Type: text/html; charset=utf-8 Location: http://example.org Date: Mon, 06 Dec 2021 14:10:18 GMT HTTP/1.1 200 OK Content-Encoding: gzip Accept-Ranges: bytes Age: 254488 Cache-Control: max-age=604800 Content-Type: text/html; charset=UTF-8 Date: Mon, 06 Dec 2021 14:10:18 GMT Etag: "3147526947+gzip" Expires: Mon, 13 Dec 2021 14:10:18 GMT Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT Server: ECS (dcb/7EEF) X-Cache: HIT Content-Length: 648 Whereas all other requests should be met with a 404 Not Found error response. $ curl -IL localhost:3000/bar HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Mon, 06 Dec 2021 14:22:51 GMT Content-Length: 19 Custom handlers The handlers that ship with net/http are useful, but most of the time when building a web application you'll want to use your own custom handlers instead. So how do you do that? The first thing to explain is that anything in Go can be a handler so long as it satisfies the http.Handler interface, which looks like this: type Handler interface { ServeHTTP(ResponseWriter, *Request) } If you're not familiar with interfaces in Go I've written an explanation here, but in simple terms all it means is that a handler must have a ServeHTTP() method with the following signature: ServeHTTP(http.ResponseWriter, *http.Request) To help demonstrate, let's create a custom handler which responds with the current time in a specific format. Like this: type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } The exact code here isn't too important. All that really matters is that we have an object (in this case it's a timeHandler struct, but it could equally be a string or function or anything else), and we've implemented a method with the signature ServeHTTP(http.ResponseWriter, *http.Request) on it. That's all we need to make a handler. Let's try this out in a concrete example: File: main.go package main import ( "log" "net/http" "time" ) type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Initialise the timeHandler in exactly the same way we would any normal // struct. th := timeHandler{format: time.RFC1123} // Like the previous example, we use the mux.Handle() fnction to register // this with our ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Run the application, then go ahead and try making a request to http://localhost:3000/time. You should get a response containing the current time, similar to this: $ curl localhost:3000/time The time is: Mon, 06 Dec 2021 15:33:21 CET Let's step through what's happening here: When our Go server receives an incoming HTTP request it hands it off to our servemux (the one that we passed to the http.ListenAndServe() function). The servemux then looks up the appropriate handler based on the request path (in this case, the /time path maps to our timeHandler handler). The serve mux then calls the ServeHTTP() method of the handler, which in turn writes out the HTTP response. The eagle-eyed of you might have also noticed something interesting: the signature for the http.ListenAndServe() function is ListenAndServe(addr string, handler Handler), but we passed a servemux as the second parameter. We were able to do this because the http.ServeMux type has a ServeHTTP() method, meaning that it too satisfies the http.Handler interface. For me it simplifies things to think of http.ServeMux as just being a special kind of handler, which instead of providing a response itself passes the request on to a second handler. This isn't as much of a leap as it first sounds — chaining handlers together is very commonplace in Go. Functions as handlers For simple cases (like the example above) defining new a custom type just to make a handler feels a bit verbose. Fortunately, we can rewrite the handler as a simple function instead: func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } Now, if you've been following along, you're probably looking at that and wondering: How can that be a handler? It doesn't have a ServeHTTP() method. And you'd be correct. This function itself is not a handler. But we can coerce it into being a handler by converting it to a http.HandlerFunc type. Basically, any function which has the signature func(http.ResponseWriter, *http.Request) can be converted into a http.HandlerFunc type. This is useful because http.HandlerFunc objects come with an inbuilt ServeHTTP() method which — rather cleverly and conveniently — executes the content of the original function. If that sounds confusing, try taking a look at the relevant source code. You'll see that it's a very succinct way of making a function satisfy the http.Handler interface. Let's reproduce the our application using this technique: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Convert the timeHandler function to a http.HandlerFunc type. th := http.HandlerFunc(timeHandler) // And add it to the ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } In fact, converting a function to a http.HandlerFunc type and then adding it to a servemux like this is so common that Go provides a shortcut: the mux.HandleFunc() method. You can use this like so: func main() { mux := http.NewServeMux() mux.HandleFunc("/time", timeHandler) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Passing variables to handlers Most of the time using a function as a handler like this works well. But there is a bit of a limitation when things start getting more complex. You've probably noticed that, unlike the method before, we've had to hardcode the time format in the timeHandler function. What happens when you want to pass information or variables from main() to a handler? A neat approach is to put our handler logic into a closure, and close over the variables we want to use, like this: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { mux := http.NewServeMux() th := timeHandler(time.RFC1123) mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } The timeHandler() function now has a subtly different role. Instead of coercing the function into a handler (like we did previously), we are now using it to return a handler. There's two key elements to making this work. First it creates fn, an anonymous function which accesses — or closes over — the format variable forming a closure. Regardless of what we do with the closure it will always be able to access the variables that are local to the scope it was created in — which in this case means it'll always have access to the format variable. Secondly our closure has the signature func(http.ResponseWriter, *http.Request). As you may remember from a moment ago, this means that we can convert it into a http.HandlerFunc type (so that it satisfies the http.Handler interface). Our timeHandler() function then returns this converted closure. In this example we've just been passing a simple string to a handler. But in a real-world application you could use this method to pass database connection, template map, or any other application-level context. It's a good alternative to using global variables, and has the added benefit of making neat self-contained handlers for testing. You might also see this same pattern written as: func timeHandler(format string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) }) } Or using an implicit conversion to the http.HandlerFunc type on return: func timeHandler(format string) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } } The default servemux You've probably seen the default servemux mentioned in a lot of places, from the simplest Hello World examples to the Go source code. It took me a long time to realise it isn't anything special. The default servemux is just a plain ol' servemux like we've already been using, which gets instantiated by default when the net/http package is used and is stored in a global variable. Here's the relevant line from the Go source: var DefaultServeMux = NewServeMux() Generally speaking, I recommended against using the default servemux because it makes your code less clear and explicit and it poses a security risk. Because it's stored in a global variable, any package is able to access it and register a route — including any third-party packages that your application imports. If one of those third-party packages is compromised, they could use the default servemux to expose a malicious handler to the web. Instead it's better to use your own locally-scoped servemux, like we have been so far. But if you do decide to use the default servemux... The net/http package provides a couple of shortcuts for registering routes with the default servemux: http.Handle() and http.HandleFunc(). These do exactly the same as their namesake functions we've already looked at, with the difference that they add handlers to the default servemux instead of one that you've created. Additionally, http.ListenAndServe() will fall back to using the default servemux if no other handler is provided (that is, the second parameter is set to nil). So as a final step, let's demonstrate how to use the default servemux in our application instead: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { // Note that we skip creating the ServeMux... var format string = time.RFC1123 th := timeHandler(format) // We use http.Handle instead of mux.Handle... http.Handle("/time", th) log.Print("Listening...") // And pass nil as the handler to ListenAndServe. http.ListenAndServe(":3000", nil) }

• Serving static sites with Go
  Alex Edwards · Aug 24, 2013

  I've recently moved the site you're reading right now from a Sinatra/Ruby application to an (almost) static site served by Go. So while it's fresh in my head, here's an explanation of principles behind creating and serving static sites with Go. Let's begin with a simple but real-world example: serving vanilla HTML and CSS files from a particular location on disk. Start by creating a directory to hold the project: $ mkdir static-site $ cd static-site And then add a main.go file to hold our code, and some simple HTML and CSS files in a static directory. $ touch main.go $ mkdir -p static/stylesheets $ touch static/example.html static/stylesheets/main.css File: static/example.html A static page Hello from a static page File: static/stylesheets/main.css body {color: #c0392b} Once those files are created, the code we need to get up and running is wonderfully compact: File: main.go package main import ( "log" "net/http" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/", fs) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Let's step through this. First we use the http.FileServer() function to create a handler which responds to all HTTP requests with the contents of a given file system. For our file system we're using the static directory relative to our application, but you could use any other directory on your machine (or indeed any object that implements the http.FileSystem interface). Next we use the http.Handle() function to register the file server as the handler for all requests, and launch the server listening on port 3000. It's worth pointing out that in Go the pattern "/" matches all request paths, rather than just the empty path. Go ahead and run the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the HTML page we made with a big red heading. Almost-Static Sites If you're creating a lot of static HTML files by hand, it can be tedious to keep repeating boilerplate content. Let's explore using Go's html/template package to put shared markup in a layout file. At the moment all requests are being handled by our file server. Let's make a slight adjustment to our application so the file server only handles request paths that begin with the pattern /static/ instead. File: main.go ... func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Notice that because our static directory is set as the root of the file system, we need to strip off the /static/ prefix from the request path before searching the file system for the given file. We do this using the http.StripPrefix() function. If you restart the application, you should find the CSS file we made earlier available at localhost:3000/static/stylesheets/main.css. Now let's create a templates directory, containing a layout.html file with shared markup, and an example.html file with some page-specific content. $ mkdir templates $ touch templates/layout.html templates/example.html File: templates/layout.html {{define "layout"}} {{template "title"}} {{template "body"}} Made with Go {{end}} File: templates/example.html {{define "title"}}A templated page{{end}} {{define "body"}} Hello from a templated page {{end}} If you've used templating in other web frameworks or languages before, this should hopefully feel familiar. Go templates – in the way we're using them here – are essentially just named text blocks surrounded by {{define}} and {{end}} tags. Templates can be embedded into each other using the {{template}} tag, like we do above where the layout template embeds both the title and body templates. Let's update the application code to use these: File: main.go package main import ( "html/template" "log" "net/http" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) tmpl, _ := template.ParseFiles(lp, fp) tmpl.ExecuteTemplate(w, "layout", nil) } So what's changed here? First we've added the html/template and path packages to the import statement. Then we've specified that all the requests not picked up by the static file server should be handled with a new serveTemplate function (if you were wondering, Go matches patterns based on length, with longer patterns take precedence over shorter ones). In the serveTemplate function, we build paths to the layout file and the template file corresponding with the request. Rather than manual concatenation we use filepath.Join(), which has the advantage joining paths using the correct separator for your OS. Importantly, because the URL path is untrusted user input, we use filepath.Clean() to sanitise the URL path before using it. (Note that even though filepath.Join() automatically runs the joined path through filepath.Clean(), to help prevent directory traversal attacks you need to manually sanitise any untrusted inputs before joining them.) We then use the template.ParseFiles() function to bundle the requested template and layout into a template set. Finally, we use the template.ExecuteTemplate() function to render a named template in the set, in our case the layout template. Restart the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the markup from all the templates merged together like so: If you use web developer tools to inspect the HTTP response, you'll also see that Go automatically sets the correct Content-Type and Content-Length headers for us. Lastly, let's make the code a bit more robust. We should: Send a 404 response if the requested template doesn't exist. Send a 404 response if the requested template path is a directory. Send a 500 response if the template.ParseFiles() or template.ExecuteTemplate() functions throw an error, and log the detailed error message. File: main.go package main import ( "html/template" "log" "net/http" "os" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) // Return a 404 if the template doesn't exist info, err := os.Stat(fp) if err != nil { if os.IsNotExist(err) { http.NotFound(w, r) return } } // Return a 404 if the request is for a directory if info.IsDir() { http.NotFound(w, r) return } tmpl, err := template.ParseFiles(lp, fp) if err != nil { // Log the detailed error log.Print(err.Error()) // Return a generic "Internal Server Error" message http.Error(w, http.StatusText(500), 500) return } err = tmpl.ExecuteTemplate(w, "layout", nil) if err != nil { log.Print(err.Error()) http.Error(w, http.StatusText(500), 500) } }

• A fresh start
  Alex Edwards · Aug 17, 2013

  I've never really known what to do with my personal site. Over the years it's been a dumping ground for links to different projects, and played host to various half-hearted attempts at blogging. But it's never really had much in the way of an actual purpose. I decided to start afresh and relaunch this site with more of a focus. After speaking to the guys from Techzing, I'm going to hunker down and focus my efforts on learning Go really well, with the aim of possibly doing some consultancy work around it in the future. So over the coming months and maybe even years, I hope to create a lot of useful content for anyone else doing the same. Because it's also full redesign of the site, I'll do a little colophon. The site is now just static content, although I use Sass for stylesheets and Markdown for writing blog posts (both of which are compiled on my local machine before publication). Some custom Go code handles the routing and templating, and it's all hosted on Heroku. For development I used Ubuntu as my operating system, Sublime Text as my editor, Git for version control, and Dropbox for real-time backups. So with the mandatory first new post out of the way, I'm looking forward to doing a lot more with this site in the future!

• Go Experiments Explained
  Alex Edwards

  Go often ships with experimental features as part of a release. These experimental features can take different forms: sometimes they're completely new packages in the standard library, sometimes they're changes to the compiler or runtime, or – very occasionally – they can be breaking changes to Go's behavior. Most of the time, the purpose of experimental features is to get real-world feedback from users before something graduates to general availability and becomes a permanent part of Go. If the feature causes regressions, or gets negative feedback from the community, it can be changed before it is finalized – or even abandoned entirely. Some examples Let's look at a few recent examples to illustrate the type of things that Go experiments can cover. Go 1.24 shipped with experimental support for a new testing/synctest package (which provides support for testing concurrent code). After feedback, the package API was adjusted slightly and it graduated to general availability in Go 1.25. Go 1.25 shipped with experimental support for a new garbage collector design with better performance. After incorporating feedback, the new garbage collector became the default in Go 1.26. Go 1.21 shipped with an experimental behavioral change to loop variable semantics. This change closed off a previously common bug with Go code, but was technically a breaking change to the language. Shipping the change as an experiment gave people a chance to test their code before the new behavior became the default in Go 1.22. Experiment lifecycle There isn’t a single fixed lifecycle for experiments, but there are some common patterns. Most experiments initially ship as off-by-default. You explicitly opt-in to try out the feature, usually by setting the GOEXPERIMENT environment value (which we'll talk about more in a moment). If things go well, one or two releases later the experimental feature is finalized, graduates to general availability, and becomes on-by-default. If an experiment affects the behavior of something, then after it graduates to general availability there is sometimes – but not always – a transitional grace period where it's possible to temporarily disable it and use the old behavior. For example, in Go 1.26 the new garbage collector design (which we briefly mentioned above) graduated to general availability and is on-by-default, but it's still possible to disable it and use the old garbage collector if you need to. So that's the most common pattern, but sometimes things take longer or work out differently. For example: Go 1.22 shipped with an experimental implementation of the compiler's inlining logic, which is still off-by-default and under evaluation more than two years later. The same release also shipped with a memory arenas experiment. After negative feedback and concerns from users, it remains off-by-default, is on indefinite hold, and may eventually be removed completely. Or finally, when the Go team is confident in a change, they might skip the feedback stage and go straight to general availability... but there may still be a transitional grace period where it's possible to disable it. A good example of this is when Go 1.24 changed its map implementation to use Swiss tables. The Go team was confident enough in the implementation and its performance benefits for this to go straight to general availability and become on-by-default, but – at least for now – it's still possible to opt out and use the old map implementation if you want to. So in practice there are really three broad experiment states: Off-by-default and under evaluation Off-by-default and on hold/dormant On-by-default with a temporary opt-out Permanent experiments Go also has a handful of experimental features that aren't really “experiments” in the normal sense. These are features that are off-by-default, but they're not under evaluation, not seeking feedback, and there's no expectation that they will ever graduate to general availability and become on-by-default. Although they are controlled by the GOEXPERIMENT environment setting in the same way as other experiments, really they are more like optional Go features that you might want to use in specialist situations. I'll refer to these as "permanent experiments" in the rest of this post. For example there is a field tracking diagnostic feature that tracks which struct fields are accessed. It's been available for a decade, and there's no intention for it to ever graduate to general availability. Or there is a static lock ranking feature, which is a diagnostic for finding potential deadlocks in the Go runtime. What experiments are available right now? It's surprisingly difficult to find out what experimental features are currently available and what their status is. Unfortunately, there isn't a page in the official Go documentation or Go Wiki that tracks experiment status, and for this post I've had to piece together the information from various places. If you want to do the same: You can get a list of all available experiments by running $ go doc goexperiment.Flags. You can figure out which experiments are on-by-default by reading the source code of src/internal/buildcfg/exp.go – specifically looking at the baseline variable declaration in the ParseGOEXPERIMENT() function. You can cross-reference the experiment names with the Go release notes and search through GitHub issues to try to figure out the current status. As far as I can tell, as of Go 1.26 here are the available permanent experiments: Experiment name Description Status FieldTrack Diagnostic to track which struct fields are accessed Off-by-default and permanent fixture StaticLockRanking Diagnostic to validate lock acquisition order to catch deadlocks Off-by-default and permanent fixture CgoCheck2 Diagnostic to check cgo pointer passing rules; too expensive to run by default Off-by-default and permanent fixture BoringCrypto Replaces Go's crypto with FIPS-validated BoringSSL; no longer relevant since Go 1.24 Off-by-default and permanent fixture but will be removed soon PreemptibleLoops Allows scheduler to preempt goroutines at loop back-edges; generally not relevant since Go 1.14, but still may be useful on platforms where preemption is otherwise unsupported Off-by-default and permanent fixture Here are the current off-by-default experiments and their status: Experiment name Description Status HeapMinimum512KiB Reduces minimum heap size from 4MB to 512KiB; may be useful for constrained environments Off-by-default and likely dormant Arenas Memory arena implementation Off-by-default and on hold following negative feedback NewInliner Rewritten compiler inliner with better call-site heuristics Off-by-default and under evaluation (available since Go 1.22) JSONv2 New encoding/json/v2 package with improved JSON encoding/decoding functions Off-by-default and under evaluation (available since Go 1.25) RuntimeSecret New runtime/secret package with functions for zeroing out memory; available on Linux amd64/arm64 only Off-by-default and under evaluation (available since Go 1.26) GoroutineLeakProfile Adds a goroutineleak pprof profile type Off-by-default and under evaluation (available since Go 1.26) SIMD New simd/archsimd package providing access to architecture-specific SIMD operations; only available on amd64 Off-by-default and under evaluation (available since Go 1.26) RuntimeFreegc Allows immediate reuse of memory without waiting for a GC cycle when safe to do so Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) SizeSpecializedMalloc Enables malloc implementations that are specialized per size class Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) And here are the currently on-by-default experiments: Experiment name Description Status LoopVar Per-iteration loop variable scoping On-by-default since Go 1.22, but opt-out kept for edge cases Dwarf5 DWARF 5 debug info generation; reduces binary size On-by-default with a temporary opt-out (opt-out may be removed in a future release) RandomizedHeapBase64 Randomizes the heap base address at startup as a security measure On-by-default with a temporary opt-out (opt-out expected to be removed in a future release) GreenTeaGC New garbage collector with improved performance; unavailable on darwin/ios/aix On-by-default with a temporary opt-out (opt-out expected to be removed in Go 1.27) RegabiWrappers ABI wrappers for calling between ABI0 and ABIInternal functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 RegabiArgs Enables register arguments/results in all compiled Go functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 How do you enable and disable experiments? Experiments are controlled using the GOEXPERIMENT environment setting. If there are some off-by-default experiments you want to try, you should include the experiment names as comma-separated lowercase values in GOEXPERIMENT. For example, if you wanted to build your application with the JSONv2 and GoroutineLeakProfile experiments enabled, you would do so like this: $ GOEXPERIMENT=jsonv2,goroutineleakprofile go build ./... If there is an on-by-default experiment that you want to turn off, you do so by prefixing the lowercase experiment name with no. For example, if you want to build your application with the GreenTeaGC and RandomizedHeapBase64 experiments turned off, you would do so like this: $ GOEXPERIMENT=nogreenteagc,norandomizedheapbase64 go build ./... It's totally fine to mix enabled and disabled experiments: $ GOEXPERIMENT=jsonv2,nogreenteagc go build ./... Note that if you build the same package with different GOEXPERIMENT values, Go treats them as different builds and stores separate entries in the build cache. I've used go build in the examples above, but you can use exactly the same pattern when using go run or go test too. If you want to try it yourself, try creating the following program which uses the experimental encoding/json/v2 package: package main import ( "encoding/json/v2" "fmt" ) type Person struct { Name string `json:"name"` Age int `json:"age"` City string `json:"city"` } func main() { p := Person{Name: "Ada", Age: 36, City: "Vienna"} data, _ := json.Marshal(p, json.StringifyNumbers(true)) fmt.Println(string(data)) } If you run this normally, the program won't compile and you'll get an error message similar to this: $ go run main.go package command-line-arguments imports encoding/json/v2: build constraints exclude all Go files in /usr/local/go/src/encoding/json/v2 But if you enable the JSONv2 experiment, the program will run as expected: $ GOEXPERIMENT=jsonv2 go run main.go {"name":"Ada","age":"36","city":"Vienna"} Which experiments should you actually care about? If you're a run-of-the-mill Gopher like me, who mainly uses Go to write programs rather than working on Go itself, most of the available experiments probably won't be very relevant to you. The most interesting and relevant ones probably are: GreenTeaGC – If you're using Go 1.26, you're already using this by default. But if you notice any performance or behavior problems, it's worth being aware that you can still disable it (and you should also file an issue). Dwarf5 – Again, if you're using Go 1.25 or later then you're already using this by default. But if you run into any problems, it's useful to know that you can still disable it. JSONv2 – I don't recommend switching to this until it graduates to general availability, but if you write a lot of code that deals with JSON, it's worth experimenting with the new encoding/json/v2 package, familiarizing yourself with what's coming, and giving feedback if you notice any problems. GoroutineLeakProfile – This one is immediately useful and worth enabling if you suspect you have a goroutine leak and need to debug it. RuntimeSecret – Worth experimenting with and giving feedback on if you write cryptographic code or need to handle sensitive data. RuntimeFreegc – If you have an application that leans heavily on the garbage collector, it may be worth benchmarking your code with this enabled to see if it improves performance, and giving feedback if you notice any issues. Finally, it's worth emphasizing that experimental features are not covered by the Go compatibility promise. Their APIs, behavior, and performance characteristics may all change, so it's generally a good idea to avoid adopting too early and depending on experimental features before they are finalized. But experimental features often act as a preview to some of the biggest changes in Go. If you know that an experiment is likely to affect you or your code once it eventually becomes generally available and on-by-default, it's a good idea to try it out, run benchmarks where appropriate, and give feedback if you find issues. If you want to keep track of what experiments are available and their status, the Go release notes have recently started doing a much better job of documenting experimental features and how to use them. Between this blog post and browsing the release notes when there's a new Go release, you should have a decent idea of what's going on.

• Demystifying function parameters in Go
  Alex Edwards

  In this post we're going to talk about how (and why!) different types of function parameters behave differently in Go. If you're new (or even not-so-new) to Go, this can be a common source of confusion and questions. Why do functions generally mutate maps and slices, but not other data types? Why isn't my slice being mutated when I append to it in a function? Why doesn't assigning a new value to a pointer parameter have any effect outside the function? Once you understand how functions and the different Go types work, the answers to these kind of questions becomes clearer. You'll discover that Go's behavior consistently follows a few fairly straightforward rules, which I'll aim to highlight in this post. (If you just want the actionable takeaways, you can skip to the summary.) Note: In this post we'll be talking a lot about pointers, so if you're not 100% sure what pointers are, or the terms reference operator and dereference operator don't mean anything to you, then I recommend reading my gentle introduction to pointers tutorial before continuing with this one. Parameters and arguments Before we dive into this post, I'd like to quickly explain the difference between parameters and arguments. People sometimes use these terms interchangeably – but for this tutorial it's important that we're precise on the terminology. Parameters are the variables that you define in a function declaration. Arguments are the values that get passed to the function for execution. (A neat way to remember this is arguments = actual values.) Functions operate on copies of the arguments It's important to understand that when you call a function in Go, the function always operates on a copy of the arguments. That is, the parameters contain a copy of the argument values. We can illustrate this with the following short example: package main import "fmt" func incrementScore(s int) { s += 10 } func main() { score := 20 incrementScore(score) fmt.Println("The score is", score) // Prints: "The score is 20" } When you run this program it will print "The score is 20", not "The score is 30". That's because the parameter s in incrementScore() contains a copy of the score argument, and when we increment the value with s += 10 we are updating this copy, not the original score variable in the main() function. We can confirm this behavior by using the reference operator & to get the memory addresses of the score argument and s parameter, like so: package main import "fmt" func incrementScore(s int) { fmt.Println("has address", &s) // Prints: "has address 0xc000012040" s += 10 } func main() { score := 20 fmt.Println("has address", &score) // Prints: "has address 0xc000012028" incrementScore(score) } If you run this, you'll see that the printed memory addresses are different – in my case 0xc000012028 for the score argument and 0xc000012040 for the parameter s. That confirms that they are truly different variables, with their values stored at different locations in memory. With that in mind, it's not surprising that changing one doesn't change the other. Just to hammer home the point one more time: in Go, functions always operate on a copy of the arguments. There are no exceptions to this. Pointer parameters So, what can we do if we want incrementScore() to actually change the score variable? The answer is to change the signature of incrementScore() so that the parameter s is a pointer, like func incrementScore(s *int). Let's take a look at a working example and then talk it through. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 *s = newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 30" } In this code: We declare the score variable normally in main() with the line score := 20. Then in the line incrementScore(&score) we use the reference operator & to get a pointer to the score variable, and pass this pointer as the argument to incrementScore(). Remember, a pointer just contains a memory address – in this case it's the memory address of the score variable. When incrementScore() is executed, the parameter s contains a copy of this pointer. But this copy still holds the same memory address – the memory address of the score variable. In the line newScore := *s + 10 we use the dereference operator *s to 'read through' and get the underlying value at that memory address, and add ten to it. Then in the next line *s = newScore we use the dereference operator again to 'write through' and set newScore as the value at that memory address. The end result is that we've mutated the value at the memory address of the score variable. So when the program executes the final line of code, we get the output "The score is 30". I should point out that I made the code here a bit more verbose than it needs to be. You can simplify incrementScore() to use the += operator like so: func incrementScore(s *int) { *s += 10 } Write-though vs reassignment In the example above, we used the deference operator *s to read-through and then write-through to the underlying memory address. But what would happen if we didn't write-through, and assigned a completely new pointer value to s instead? Let's take a look. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 s = &newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 20" } If you run this, you'll see we're back to the situation where the score value isn't being mutated, and the program is printing "The score is 20" again. The only thing that's changed here is the body of the incrementScore() function. In this code: The line newScore := *s + 10 is exactly the same as before. It reads through to get the underlying score value from the s parameter, adds ten to it, and assigns the result to the newScore variable. But the line s = &newScore is different. Here we use the reference operator &newScore to get a pointer to the newScore variable, and assign this to s. This means that the variable s no longer contains the memory address of the score variable from main() – it now contains the memory address of the newScore variable. So, in this example, incrementScore() doesn't ever 'write-through' and change anything at the memory address of the score variable. All it does is replace s with a completely different pointer, which is then discarded when the function returns. This is just one example of a more general rule. Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. Automatic dereferencing Let's continue with the same example, but update the incrementScore() function so that it accepts a pointer to a custom player struct, containing the player's name and score. package main import "fmt" type player struct { name string score int } // Make the parameter a pointer to a player struct. func incrementScore(p *player) { p.score += 10 } func main() { // Initialize a player struct and assign it to the variable p1. p1 := player{name: "Alice", score: 20} // Pass a pointer to p1 to incrementScore(). incrementScore(&p1) fmt.Printf("The score for %s is %d", p1.name, p1.score) // Prints: "The score for Alice is 30" } So as you might expect, because the parameter p in incrementScore() is a pointer, the changes that we make to p affect the data at the underlying memory address of p1 and the program prints "The score for Alice is 30". But the most interesting part here is the line of code p.score += 10 in the incrementScore() function. p is a pointer, but we appears that we don't have to dereference it using the * operator in order to write-through the new value. You could – if you wanted to – change this line to be (*p).score += 10. That's perfectly valid and will compile fine. But it's not necessary. If you have a pointer to a struct (which is what the p parameter is here), then Go will automatically dereference the pointer for you when you use the dot operator . on it to access a field or call a method. You can also use index expressions on a pointer to an array without dereferencing it. (Note that this will only work on arrays, not slices). For example: a := &[3]string{"a", "b", "c"} fmt.Println(a[1]) // Instead of having to write (*a)[1] "Reference types" Everything we've illustrated in this tutorial so far is true when the parameter type is a basic type, a struct, an array, a function, or a pointer to any of those things. However, the behavior that you get when a parameter is a map, slice or channel type needs some further discussion. Once you realize how these types are implemented at runtime behind the scenes, you'll see that their behavior actually follows the same rules as the other Go types – but nonetheless it can be a bit confusing at first. If you've been programming for a while, you might be familiar with the terms pass-by-value and pass-by-reference from other languages. You might have also heard or read people in the Go world saying things like "maps, slices and channels are reference types", or "maps, slices and channels are passed by reference". Well... the sentiment there is sort of right, but the wording isn't correct and needs tightening up. Firstly, Go does not support pass-by-reference behavior. I've probably banged this drum enough already now, but parameters are always a copy of the arguments. That is, they are always passed by value. Even pointers are passed by value; a pointer parameter will contain a copy of the pointer. Strictly speaking, there's also no such group of things in Go known as "reference types". To be fair, the Go spec did use "reference types" as an umbrella term for maps, slices and channels in one sentence, but this was removed over a decade ago (with the commit message Go has no 'reference types'). Basically, I recommend forgetting hearing the term "reference types" in relation to Go, and replacing it with an understanding of how maps, channels and slices are actually implemented. Maps and Channels The important thing to understand is that behind-the-scenes when your code is running, the Go runtime implements a map as a pointer to a runtime.hmap struct, and a channel as a pointer to a runtime.hchan struct. This means that map and channel parameters behave in a similar way to regular pointer parameters. The parameter will contain a copy of the map or channel, but this copy will still point to the same underlying memory location that holds the runtime.hmap or runtime.hchan struct. In turn, that means that any changes you make to a map or channel parameter will also mutate the argument. Let's look at an example, where we create a scores map containing the names and scores for multiple players like map[string]int{"Alice": 20, "Bob": 160}, and then pass it to a function that increments the score by ten for all players. package main import "fmt" func incrementAllScores(sm map[string]int) { for name := range sm { sm[name] += 10 } } func main() { scores := map[string]int{"Alice": 20, "Bob": 160} incrementAllScores(scores) fmt.Println(scores) // Prints: map[Alice:30 Bob:170] } When you run this, you'll see that incrementAllScores() mutates the scores argument and the program prints map[Alice:30 Bob:170] as the output. Because of this behavior, you normally won't need to use a pointer to a map or channel as a function parameter. On the other hand, if you don't want a function to mutate a map, you can use the maps.Clone() function to create a clone that points to a different memory location, and work on the clone instead. func example(m map[string]int) { cm := maps.Clone(sm) // ... do something with the cloned map. } Note: Although the Go runtime implements maps and channels as pointers to internal structures, this is a runtime implementation detail. Maps and channels are their own concrete types as far as the compiler is concerned, they are not pointers, and you can't do things like dereferencing a map or channel type like you would a pointer. Slices How slices work behind the scenes in Go can be pretty difficult to grok, and if you'd like a detailed explanation I recommend reading the Go Slices: usage and internals post on the official blog. But as a high-level summary, the Go runtime implements slices as a runtime.slice struct. This struct wraps a pointer to a (fixed-size) array that actually stores the slice data. You can think of a slice as being a bit like a 'window through' to a segment of this underlying array. So when you have a function with a slice parameter, the parameter will contain a copy of the slice argument you pass it. Effectively, it will have a copy of the runtime.slice struct. But the pointer in this copy of runtime.slice will still point to the same underlying array, meaning that any changes you make to a slice parameter will also mutate the argument. To demonstrate this, let's say that we have a slice containing some player scores, and pass it to a addBonus() function that adds fifty to each score in the slice. package main import "fmt" func addBonus(s []int) { for i := range s { s[i] += 50 } } func main() { scores := []int{10, 20, 30} addBonus(scores) fmt.Println(scores) // Prints: [60 70 80] } When you run this code it will print [60 70 80], demonstrating that the changes made in addBonus() mutated the elements in the scores slice. If you don't want a function to mutate a slice, you can make a clone of it using slices.Clone() and work on that instead. func example(s []int) { cs := slices.Clone(s) // ... do something with the cloned slice. } So far, so good. Slices generally behave pretty much like maps and channels, in the sense that changing a slice parameter will mutate the argument. If you don't want that, you can make a clone at the start of the function and use the clone instead. But using append() on a slice parameter can sometimes be a source of confusion. Consider the following code, where we create a variadic addScores() function that appends some new values to a scores slice. package main import "fmt" func addScores(s []int, values ...int) { s = append(s, values...) } func main() { scores := []int{10, 20, 30} addScores(scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30] } (Yes, this is a bit of a silly example, but it illustrates the point in a simple way.) When you run this program, it will print out [10 20 30] – demonstrating that the append() operation in addScores() has not affected the scores argument. This actually makes sense and is consistent with the other behavior we've seen in this post. Earlier on I said: Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. The code s = append(s, values...) is no different. We're replacing the s parameter with a new value, and this operation doesn't touch the argument in any way. Note: As a slight aside, the slice returned by append() may or may not point to the same underlying array as the original slice that you're appending too. It all depends on whether the underlying array has enough capacity to store the new values or not. If a new underlying array needs to be reallocated, the pointer in the slice returned by append() will be different. If it doesn't, then the pointer will remain the same and point to the same underlying array. When it comes to slices as a function parameter, this means that if you change the elements in a slice parameter after a call to append(), the change may or may not mutate the argument. It all depends on whether the append() operation resulted in a new underlying array being allocated or not. You can see an example of this behavior here. So what about when you want an append() operation in a function to mutate the argument? The answer here is to make the parameter a pointer to a slice, like so: package main import "fmt" func addScores(s *[]int, values ...int) { *s = append(*s, values...) } func main() { scores := []int{10, 20, 30} addScores(&scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30 40 50 60] } Now with the line *s = append(*s, values...), whatever is returned by the append() function will be 'written-through' to the memory address of the scores argument. Exactly the same logic applies for operations to 'reslice' a slice and assign the result back to the parameter, like s = s[0:1]. If you want this operation to mutate the argument, you should make the parameter a pointer and dereference it like *s = (*s)[0:1]. Summary We've covered a lot of ground in this post, so I'll try to summarize everything into a handful of take-away points. Parameters always contain a copy of the argument. Go doesn't have "reference types" or support pass-by-reference semantics. For the basic Go types, as well as structs, arrays and functions, changing the value of a parameter in the function body won’t change the value of the argument. But if you do want to mutate the argument, you can use a pointer parameter instead and dereference it inside the function to ‘write-through’ a new value to the argument's memory address. For common operations on structs and arrays, Go will automatically dereference the pointer for you. Because of the way that they're implemented by the Go runtime, changes you make to map, slice, channel parameters in a function will mutate the argument. If you don't want this, make a clone at the start of the function and use that instead. Using the = operator to assign a new value to a parameter does not affect the argument (unless you are manually-or-automatically dereferencing a pointer and 'writing-through' a new value). So for slices, if you want a function to perform an append or reslice operation that mutates the argument, you should use a pointer to a slice as the function parameter and dereference it as necessary.

• A time-saving Makefile for your Go projects
  Alex Edwards

  Whenever I start a new Go project, one of the first things I do is create a Makefile in the root of my project directory. This Makefile serves two purposes. The first is to automate common admin tasks (like running tests, checking for vulnerabilities, pushing changes to a remote repository, and deploying to production), and the second is to provide short aliases for Go commands that are long or difficult to remember. I find that it's a simple way to save time and mental overhead, as well as helping me to catch potential problems early and keep my codebases in good shape. While the exact contents of the Makefile changes from project to project, in this post, I want to share the boilerplate that I'm currently using as a starting point. It's generic enough that you should be able to use it as-is for almost all projects. Note: You can also find the Makefile code in this Gist. File: Makefile # Change these variables as necessary. main_package_path = ./cmd/example binary_name = example # ==================================================================================== # # HELPERS # ==================================================================================== # ## help: print this help message .PHONY: help help: @echo 'Usage:' @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /' .PHONY: confirm confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] .PHONY: no-dirty no-dirty: @test -z "$(shell git status --porcelain)" # ==================================================================================== # # QUALITY CONTROL # ==================================================================================== # ## audit: run quality control checks .PHONY: audit audit: test go mod tidy -diff go mod verify test -z "$(shell gofmt -l .)" go vet ./... go run honnef.co/go/tools/cmd/staticcheck@latest -checks=all,-ST1000,-U1000 ./... go run golang.org/x/vuln/cmd/govulncheck@latest ./... ## test: run all tests .PHONY: test test: go test -v -race -buildvcs ./... ## test/cover: run all tests and display coverage .PHONY: test/cover test/cover: go test -v -race -buildvcs -coverprofile=/tmp/coverage.out ./... go tool cover -html=/tmp/coverage.out ## upgradeable: list direct dependencies that have upgrades available .PHONY: upgradeable upgradeable: @go run github.com/oligot/go-mod-upgrade@latest # ==================================================================================== # # DEVELOPMENT # ==================================================================================== # ## tidy: tidy modfiles and format .go files .PHONY: tidy tidy: go mod tidy -v go fmt ./... ## build: build the application .PHONY: build build: # Include additional build steps, like TypeScript, SCSS or Tailwind compilation here... go build -o=/tmp/bin/${binary_name} ${main_package_path} ## run: run the application .PHONY: run run: build /tmp/bin/${binary_name} ## run/live: run the application with reloading on file changes .PHONY: run/live run/live: go run github.com/cosmtrek/air@v1.43.0 \ --build.cmd "make build" --build.bin "/tmp/bin/${binary_name}" --build.delay "100" \ --build.exclude_dir "" \ --build.include_ext "go, tpl, tmpl, html, css, scss, js, ts, sql, jpeg, jpg, gif, png, bmp, svg, webp, ico" \ --misc.clean_on_exit "true" # ==================================================================================== # # OPERATIONS # ==================================================================================== # ## push: push changes to the remote Git repository .PHONY: push push: confirm audit no-dirty git push ## production/deploy: deploy the application to production .PHONY: production/deploy production/deploy: confirm audit no-dirty GOOS=linux GOARCH=amd64 go build -ldflags='-s' -o=/tmp/bin/linux_amd64/${binary_name} ${main_package_path} upx -5 /tmp/bin/linux_amd64/${binary_name} # Include additional deployment steps here... The Makefile is organized into several sections, each with its own set of targets: 1. HELPERS help: Prints a help message for the Makefile, including a list of available targets and their descriptions. confirm: Prompts the user to confirm an action with a "y/N" prompt. no-dirty: Checks that there there are no untracked files or uncommitted changes to the tracked files in the current git repository. 2. QUALITY CONTROL audit: Runs quality control checks on the codebase, including using go mod tidy -diff to check that the go.mod and go.sum files are up-to-date and correctly formatted, verifying the dependencies with go mod verify, running test -z "$(shell gofmt -l .)" to check that all .go files are correctly formatted, running static analysis with go vet and staticcheck, checking for vulnerabilities using govulncheck, and running all tests. Note that it uses go run to execute the latest versions of the remote staticcheck and govulncheck packages, meaning that you don't need to install these tools first. I've written more about this pattern in a previous post. test: Runs all tests. Note that we enable the race detector and embed build info in the test binary. test/cover: Runs all tests and outputs a coverage report in HTML format. upgradeable: List all direct module dependencies that have a newer version available, using the oligot/go-mod-upgrade tool. 3. DEVELOPMENT tidy: Updates the dependencies and formats the go.mod and go.sum using go mod tidy, and formats all .go files using go fmt. build: Builds the package at main_package_path and outputs a binary at /tmp/bin/{binary_name}. run: Calls the build target and then runs the binary. Note that my main reason for not using go run here is that go run doesn't embed build info in the binary. run/live: Use the air tool to run the application with live reloading enabled. When changes are made to any files with the specified extensions, the application is rebuilt and the binary is re-run. Depending on the project I often add more to this section, such as targets for connecting to a development database instance and managing SQL migrations. Here's an example. 4. OPERATIONS push: Push changes to the remote Git repository. This asks for y/N confirmation first, and automatically runs the audit and no-dirty targets to make sure that all audit checks are passing and there are no uncommitted changes in the repository before the push is executed. production/deploy: Builds the a binary for linux/amd64 architecture, compress it using upx, and then run any deployment steps. Note that this target asks for y/N confirmation before anything is executed, and also runs the audit and no-dirty checks too. Depending on the project I often add more to this section too. For example, a staging/deploy rule for deploying to a staging server, production/connect for SSHing into a production server, production/log for viewing production logs, production/db for connecting to the production database, and production/upgrade for updating and upgrading software on a production server. Usage Each of these targets can be executed by running make followed by the target name in your terminal. For example: $ make tidy go mod tidy -v go fmt ./... If you run make help (or the naked make command without specifiying a target) then you'll get a description of the available targets. $ make help Usage: help print this help message tidy tidy modfiles and format .go files audit run quality control checks test run all tests test/cover run all tests and display coverage build build the application run run the application run/live run the application with reloading on file changes push push changes to the remote Git repository production/deploy deploy the application to production

• How to use the http.ResponseController type
  Alex Edwards

  One of my favorite things about the recent Go 1.20 release is the new http.ResponseController type, which brings with it three nice benefits: You can now override your server-wide read and write deadlines on a per request basis. The pattern for using the http.Flusher and http.Hijacker interfaces is clearer and feels less hacky. No more type assertions necessary! It makes it easier and safer to create and use custom http.ResponseWriter implementations. The first two benefits are mentioned in the release notes, but the third one seems to have gone under the radar a bit... which is a shame, because it's very helpful! Let's dive in a take a look. Per-request deadlines Go's http.Server has ReadTimeout and WriteTimeout settings, which you can use to automatically close a HTTP connection if reading a request or writing response takes longer than a fixed amount of time. These settings are server-wide and apply to all requests, irrespective of the handler or URL. With http.ResponseController you can now use the SetReadDeadline() and SetWriteDeadline() methods to relax or tighten these settings on a per-request basis if you need too. For example: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) // Set a write deadline in 5 seconds time. err := rc.SetWriteDeadline(time.Now().Add(5 * time.Second)) if err != nil { // Handle error } // Do something... // Write the response as normal. w.Write([]byte("Done!")) } This is particularly helpful in an application where you have a small number of handlers that need longer deadlines than all the others, for things like processing a file upload or carrying out a long-running operation. A few other details to mention: If you set a short server-wide deadline, and that deadline is hit before you call SetWriteDeadline() or SetReadDeadline() then they will have no effect. The server-wide deadline wins. If your underlying http.ResponseWriter doesn't support setting per-request deadlines, then calling SetWriteDeadline() or SetReadDeadline() will return a http.ErrNotSupported error. You can effectively remove the server-wide deadline on a per-request basis by passing a zero-valued time.Time struct to SetWriteDeadline() or SetReadDeadline(). For example: rc := http.NewResponseController(w) err := rc.SetWriteDeadline(time.Time{}) if err != nil { // Handle error } Flusher and Hijacker interfaces The http.ResponseController type also makes it slightly nicer to use the 'optional' http.Flusher and http.Hijacker interfaces. For example, before Go 1.20 you would use a code pattern like this this to flush response data to the client: func exampleHandler(w http.ResponseWriter, r *http.Request) { f, ok := w.(http.Flusher) if !ok { // Handle error } for i := 0; i Now you can do this: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) for i := 0; i The pattern for hijacking a connection is similar: func (app *application) home(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) conn, bufrw, err := rc.Hijack() if err != nil { // Handle error } defer conn.Close() // Do something... } Again, if your underlying http.ResponseWriter doesn't support support flushing or hijacking, then calling Flush() or Hijack() on a http.ResponseController will also return an http.ErrNotSupported error. Custom http.ResponseWriters It's now also easier and safer to create and use custom http.ResponseWriter implementations that still support flushing and hijacking. It's probably easiest to explain how this works with an example, so let's look at the code for a custom http.ResponseWriter implementation that records the HTTP status code of a response. type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } So here we've defined a custom statusResponseWriter type, which embeds an existing http.ResponseWriter and implements custom WriteHeader() and Write() methods to support the recording of the HTTP response status code. But the important thing to notice here is the Unwrap() method at the end, which returns the original embedded http.ResponseWriter. When you use the new http.ResponseController type to to flush, hijack or set a deadline, it will call this Unwrap() method to access the original http.ResponseWriter. This is done recursively if necessary, so you can potentially layer multiple custom http.ResponseWriter implementations on top of each other. Let's look at a complete example, where we use this statusResponseWriter in conjunction with some middleware to log response status codes, along with a handler that sends a 'normal' response and another that uses the new http.ResponseController type to send a flushed response. package main import ( "log" "net/http" "time" ) type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } func main() { mux := http.NewServeMux() mux.HandleFunc("/normal", normalHandler) mux.HandleFunc("/flushed", flushedHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", logResponse(mux)) if err != nil { log.Fatal(err) } } func logResponse(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { sw := newstatusResponseWriter(w) next.ServeHTTP(sw, r) log.Printf("%s %s: status %d\n", r.Method, r.URL.Path, sw.statusCode) }) } func normalHandler(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusTeapot) w.Write([]byte("OK")) } func flushedHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) w.Write([]byte("Write A....")) err := rc.Flush() if err != nil { log.Println(err) return } time.Sleep(time.Second) w.Write([]byte("Write B....")) err = rc.Flush() if err != nil { log.Println(err) } } If you want, you can run this and try making requests to the /normal and /flushed endpoints: $ curl http://localhost:3000/normal OK $ curl --no-buffer http://localhost:3000/flushed Write A....Write B.... You should see the response from the flushedHandler in two parts, first the Write A... part, then followed a second later by the Write B... part. And you should see that the statusResponseWriter and logResponse middleware have successfully written log messages, including the correct HTTP status code for each response. $ go run main.go 2023/03/06 21:41:21 Listening... 2023/03/06 21:41:32 GET /normal: status 418 2023/03/06 21:41:44 GET /flushed: status 200

• An introduction to Packages, Imports and Modules in Go
  Alex Edwards

  This tutorial is written for anyone who is new to Go. In it we'll explain what packages, import statements and modules are in Go, how they work and relate to each other and — hopefully — clear up any questions that you have. We'll start at a high level, then work down to the details later. There's quite a lot of content in this tutorial, so I've broken it down into the following eight sections: Packages The main package Importing and using standard library packages Unused and missing imports Exported vs unexported Modules Using multiple packages in your code Importing and using third-party packages Organizing import statements To help illustrate things throughout this post we'll build a small CLI (command-line interface) application which generates and prints out a random 'lucky number'. If you'd like to follow along, run the following commands: $ mkdir lucky-number $ cd lucky-number $ touch main.go Then add the following code to the main.go file: File: main.go package main import ( "fmt" "math/rand" ) func main() { // Get a random number between 0 and 99 inclusive. n := rand.Intn(100) // Print it out. fmt.Printf("Your lucky number is %d!\n", n) } At this point you should be able to run the application and see some output like this: $ go run main.go Your lucky number is 81! Packages A package in Go is essentially a named collection of one or more related .go files. In Go, the primary purpose of packages is to help you isolate and reuse code. Every .go file that you write should begin with a package {name} statement which indicates the name of the package that the file is a part of. For example, in the 'lucky number' code above, the package main line declares that the main.go file is part of a package named main. At the moment: Our 'lucky number' application consists of one package, with the package name main. The main package is made up of one file, with the filename main.go. It's important to explain that code in a package can access and use all types, constants, variables and functions within that package — even if they are declared in a different .go file. Let's illustrate this by splitting our 'lucky number' code across two files. Go ahead and add an additional random.go file: $ touch random.go Then update the two files so that the main() function calls a new randomNumber() function, like so: File: random.go package main import ( "math/rand" ) func randomNumber() int { return rand.Intn(100) } File: main.go package main import ( "fmt" ) func main() { fmt.Printf("Your lucky number is %d!\n", randomNumber()) } So now: Our 'lucky number' application consists of two .go files. Both files are part of the main package (because they both start with a package main statement). If you re-run the application using the two files, you should see the same output. $ go run *.go Your lucky number is 81! Note: If your terminal doesn't support wildcard expansion, you'll need to list the files explicitly and run the command $ go run main.go random.go instead. This example is a bit contrived but it illustrates the point nicely — our main() function is able to call our randomNumber() function because they are part of the same package — despite being in separate .go files. It's totally OK to have quite a lot of .go files in the same package. Having 5, 10 or even 20 files — and thousands of lines of code — in the same package is not uncommon or an anti-pattern in Go. The main package In Go, main is actually a special package name which indicates that the package contains the code for an executable application. That is, it indicates that the package contains code that can be built into a binary and run. Any package with the name main must also contain a main() function somewhere in the package which acts as the entry point for the program. If it doesn't, and you try to run it, you will get this error: $ go run *.go function main is undeclared in the main package It's conventional for your main() function to live in a file with the filename main.go. Technically it doesn't have to, but following this convention makes the application entry point easier to find for anyone reading your code in the future. As an aside, if you try to build or run a non-main package it will also result in an error. For example, if you changed the 'lucky number' code so that the package name is foo instead of main and try to run it, you will get the following (somewhat confusing) error: $ go run *.go package command-line-arguments is not a main package Importing and using standard library packages I'm sure you know this already, but individual .go files can import and use exported types, constants, variables and functions from other packages — including the packages in the Go standard library. The complete tree of Go standard library packages is available here. In our 'lucky number' code we've imported and used the math/rand and fmt packages from the standard library to help us generate a random number and print a message. For example, in the random.go file: File: random.go package main import ( "math/rand" // Import the math/rand package. ) func randomNumber() int { return rand.Intn(100) // Call the Intn() function from the math/rand package. } When importing a package from the standard library you need to use the full path to the package in the standard library tree, not just the name of the package. For example: import ( "fmt" "math/rand" // Not "rand" "net/http" // Not "http" "net/http/httptest" // Not "httptest" ) Once imported, the package name becomes an accessor for the contents of that package. Conveniently, all the packages in the Go standard library have a package name which is the same as the final element of their import path. That means we can use the Intn() function from math/rand by calling rand.Intn(), or the Printf() function from fmt by calling fmt.Printf(). As well as importing packages from the standard library it's possible to import your own packages or third-party packages too. We'll get to that shortly. Unused and missing imports If you import a package but don't actually use it in your code, it will result in a compile-time error. For example, if you import the os package but don't use it you will get an error like: "os" imported and not used Similarly, you'll also get a compile-time error if a package is referenced in your code but not imported. For example, if you try to use the strconv package without importing it you'll get an error like this: undefined: strconv When you're developing rapidly it can sometimes be annoying to keep editing your import statements, but ultimately this behavior helps to keep your code correct and your import list clean and accurate. Tip: You can use the goimports tool to automatically add and remove import statements in your .go files. It's also possible to integrate this with many popular text editors (including VSCode, Emacs and Sublime), so that import statements are updated for you whenever you save a file. But you should be careful if you are using one of the rand or template packages in your code — these standard library package names are ambiguous and you should always check that goimports has added the one that you want (for example, that it has imported html/template instead of text/template, or crypto/rand instead of math/rand). Exported vs unexported Earlier in this tutorial I said: Individual .go files can import and use exported types, constants, variables, functions and methods from other packages — including the packages in the Go standard library. So what does exported mean? Essentially, something in Go code is exported if its name starts with a capital letter. Otherwise it is unexported. For example: var fooBaz string // This is an unexported variable. var FooBar string // This is an exported variable. func fooBaz() {...} // This is an unexported function. func FooBar() {...} // This is an exported function. type fooBaz struct {...} // This is an unexported type. type FooBar struct {...} // This is an exported type. The difference between them is: Unexported things are 'private' to the package that they are declared in. They are only visible to code in the same package. In contrast, exported things in a package are 'public' and are visible to any code that that imports the package. In other words: when you import a package you get to use its exported things, but not its unexported things. Depending on your programming background, capitalization might seem like a funny way to control visibility. But once you get used to it, it has some positives. It's simple, doesn't require you to remember any additional syntax, and it's trivial to see at a glance whether something is exported or not — even when that thing is being used far away from where it is declared. Tips: Generally don't export things unless you actually have a reason to (i.e. don't capitalize a name just because it looks nicer!). Additionally, a main package should never normally be imported by anything, so it probably shouldn't have any exported things in it. Modules If you have a small application which only imports packages from the standard library, then what we've done so far works just fine. But if you want to import and use a third-party package — or structure your code so it's split into multiple packages — then you first need to turn your code into a Go module. The Go Wiki defines modules like this: A module is... a tree of Go source files with a go.mod file in the tree's root directory. In our example the lucky-number directory already contains our two .go files, so all we need to do is add a valid go.mod file to the directory to make it a module. The easiest way to do this is by running the go mod init command and passing in a module path as the final argument, like so: $ go mod init lucky-number.alexedwards.net go: creating new go.mod: module lucky-number.alexedwards.net go: to add module requirements and sums: go mod tidy Before we go further, let's talk about module paths. The module path act as a canonical identifier for a module. Ideally it should be unique and something that is unlikely to be used by anyone else, in any other project. In the command above I've used lucky-number.alexedwards.net as the module path, but it could be (almost) any string value. In the Go community it's conventional to base your module path on a URL that you own or control. So, for this example, a good module path would be something like lucky-number.alexedwards.net or github.com/alexedwards/lucky-number. Important: In most cases, your module path doesn't need to be a 'real' functioning URL with something hosted at it. It's really just an arbitrary string which acts as a unique identifier for your module. But… if you plan to make your code available for reuse (e.g. as an open source package) then your module path must be the location that the code will be fetchable from. So, for example, if you're planning to host the code at github.com/example/package the module path should also be github.com/example/package. OK, let's take a look at the go.mod file that was generated for us: File: go.mod module lucky-number.alexedwards.net go 1.19 We can see that (for now) all this does is declare the module path, along with the version of Go that you are using. We'll revisit this file again later when we talk about using third-party Go packages. Tip: If you're ever looking at some code and want to know what it's module path is, just take a look in its go.mod file. So at this point in the tutorial: The code in the lucky-number directory is now a Go module. The Go module has the module path lucky-number.alexedwards.net. The module contains one main package, which is made up of our main.go and random.go files. Using multiple packages in your code Let's make our 'lucky number' application structure a bit more complex and split up the code into two packages. Before we get started on this change there are a couple of rules and conventions to be aware of: In Go, one package == one directory. That is, all .go files for a package should be contained in the same directory, and a directory should contain the .go files for one package only. You shouldn't ever have .go files with different package names in the same directory. For all non-main packages, the directory name that the code lives in should be the same as the package name. When choosing a name you should pick something that is short, descriptive, lower case and ideally one word. The Go blog has a helpful post with additional guidance and some examples of good and bad names. With those things in mind, let's restructure our 'lucky number' application so that the code for generating the random number is isolated in a new, separate, package called random. $ rm random.go $ mkdir random $ touch random/number.go The file tree for the lucky-number directory should now look like this: $ tree --dirsfirst . ├── random │ └── number.go ├── go.mod └── main.go The important thing to point out is that all the .go files are still part of the same module — they are all part of a file tree with a single go.mod file in the root directory of the tree. OK, let's go ahead and add the following code to the new random/number.go file: File: random/number.go package random import ( "math/rand" ) func Number() int { return rand.Intn(100) } There are four things I'd like to quickly highlight and re-iterate here: The number.go file is part of the random package (notice the statement in the first line). The Number() function is exported (i.e. its name begins with a capital letter). This means it will be visible to any code which imports the random package. The directory name that the code lives in is exactly the same as the package name (random) . The random package is part of the lucky-number.alexedwards.net module. Next let's update our main.go file to import and use the new package. Like so: File: main.go package main import ( "fmt" // Import the random package. "lucky-number.alexedwards.net/random" ) func main() { // Call the random.Number() function to get the random number. Notice that // we use the package name as the accessor, just like we do for the standard // library packages. fmt.Printf("Your lucky number is %d!\n", random.Number()) } The most interesting thing about this is the import path for our new package. When you are importing packages that are part of the same module as your current .go file, the import statement should take the form: import {module path}/{path to the package relative to your go.mod file} So in this case, the module path is lucky-number.alexedwards.net and the path within the module for the package is random, giving us an import path of lucky-number.alexedwards.net/random. Before we go further, I'd like to point out that I'm making this code structure more complicated than it needs to be (just to illustrate things for teaching purposes). There's no real reason here to have split the code into two packages. In fact, overusing packages is a common mistake that newcomers to Go make. Generally you should only split code into additional packages if you have a demonstrable reason to, such as: You want a convenient way to reuse it the code, or to make it available for reuse. You want to isolate or enforce some boundary between the package code and the rest of your codebase. You have some complex code that acts as a 'black box' and moving it to a standalone package will reduce cognitive overhead when working with the rest of your code. A more complex structure Let's tweak the directory structure of our 'lucky number' code a bit more. We'll: Move the main package files into a new cmd/cli directory. Move the random package files into a new internal/random directory. (Again, this is just for teaching purposes. This structure isn't actually necessary for such a small and simple application.) $ mkdir -p cmd/cli internal $ mv main.go cmd/cli/ $ mv random internal/ $ tree --dirsfirst . ├── cmd │ └── cli │ └── main.go ├── internal │ └── random │ └── number.go └── go.mod Once that's done, let's update the cmd/cli/main.go file so that the random package is imported from its new location. Like so: File: cmd/cli/main.go package main import ( "fmt" // Import the random package using the new location under the // `internal` directory. "lucky-number.alexedwards.net/internal/random" ) func main() { fmt.Printf("Your lucky number is %d!\n", random.Number()) } You should now be able to run the application by calling go run with the path to the main package. Like this: $ go run ./cmd/cli Your lucky number is 81! This change helps to illustrate a couple of things: It's not necessary for a main package to live in the module root. It can be anywhere. In fact, it's totally OK for a module to contain multiple main packages. For example, in a larger project you could have a cmd/cli directory with the main package for a CLI tool, and a cmd/web directory with the main package for a web application in the same module. Your non-main packages don't need to be a direct child of the module root either. They can be anywhere in an arbitrarily deep directory structure within the module. Note: The directory name internal has a special behavior in Go. Any packages which live under a directory called internal can only be imported by code inside the parent of the internal directory. In this example, it means that any packages nested under internal can only be imported by code inside our lucky-number directory. Or, looking at it the other way, any packages under internal cannot be imported by code outside of the lucky-number directory. This is useful because it prevents other codebases from importing and relying on the (potentially unversioned and unsupported) packages in an internal directory — even if the code is publicly available somewhere like GitHub. Importing and using third-party packages Let's quickly explore how to import and use a third-party packages. As an example, we'll import the github.com/fatih/color package and use it change the color of the message that our application prints out — but the general process is exactly the same for most other third-party packages too. First you need to download the third-party code from its public repository to your local machine, which you can do with go get: $ go get github.com/fatih/color@latest go: added github.com/fatih/color v1.14.1 go: added github.com/mattn/go-colorable v0.1.13 go: added github.com/mattn/go-isatty v0.0.17 go: added golang.org/x/sys v0.3.0 Notice that go get will recursively download any dependencies that the code has too. Then using the third-party package in your code is fairly straightforward. You'll need to import the third-party package using its module path (which should normally be the same as the repository location that you used when running go get), and then access its exported things via its package name (which in most cases should be the same as the final element of the import path… if it is not, the documentation for the package should make that clear). Let's head to our main.go file and update the code to print a colored message using github.com/fatih/color. File: cmd/cli/main.go package main import ( "lucky-number.alexedwards.net/internal/random" // Import the color package. "github.com/fatih/color" ) func main() { // Use it to print the message in green. green := color.New(color.FgGreen) green.Printf("Your lucky number is %d!\n", random.Number()) } If you run the application again now, you should see a colorized message similar to this: $ go run ./cmd/cli Your lucky number is 81! The go.mod file should have been updated to include the dependencies that the lucky-number.alexedwards.net module has too, along with their exact version numbers. It should look similar to this: File: go.mod module lucky-number.alexedwards.net go 1.19 require github.com/fatih/color v1.14.1 require ( github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.17 // indirect golang.org/x/sys v0.3.0 // indirect ) We can see that github.com/fatih/color is listed as a direct dependency of the lucky-number.alexedwards.net module, and the other dependencies are indirect (that is, our code doesn't import them directly, but they are imported by a package that our code imports). Note: If your go.mod file looks different to this, try running the $ go mod tidy command to format it. go mod tidy also ensures that the listed dependencies match the source code in your module, so it's a good idea to run this command fairly often… especially before committing a change that adds or removes a third-party package import statement in your code. Also note: If you are using a Go version older than 1.17, indirect dependencies are handled differently and not all of them will necessarily be listed in your go.mod file. As an aside, if you are ever looking at a go.mod file and wondering why something is listed as a dependency you can use the $ go mod why command. For example, if you wanted to find out why golang.org/x/sys is an indirect dependency for the lucky-number.alexedwards.net module you could run: $ go mod why -m golang.org/x/sys # golang.org/x/sys lucky-number.alexedwards.net/cmd/cli github.com/fatih/color github.com/mattn/go-isatty golang.org/x/sys/unix We can see from the output that our cmd/cli package imports github.com/fatih/color, which in turn imports github.com/mattn/go-isatty, which in turn imports golang.org/x/sys/unix. Version 2+ packages Sometimes the third-party packages that you want to use will be in modules with a major version number greater than 1 (like v2.0.0, v3.4.5 etc). In Go, it is conventional for modules with a major version number greater than 1 to append the major version number to their module path. A few popular real-life examples are: github.com/go-chi/chi/v5 github.com/jackc/pgx/v5 github.com/go-playground/validator/v10 Typically you will need to go get these version 2+ packages using the full module path including the version number. Like this: $ go get github.com/go-chi/chi/v5 go: added github.com/go-chi/chi/v5 v5.0.8 And then you need to also import them in your .go files using the full module path (including the version number), but reference their exported things using the package name (which will normally now be the second-to-last element in the import path). For example: import ( "github.com/go-chi/chi/v5" ) func main() { router := chi.NewRouter() ... } Organizing import statements Lastly, there's no right or wrong way to organize your import statements in Go. No single convention has really emerged in the Go community, so I recommend just picking something that works for you and being consistent with it. Personally, I like to separate imports into four groups separated by an empty line. Like this: import ( {standard library packages} {packages from the current module} {third-party packages} {aliased packages} ) Within each group, go fmt will automatically sort the imports alphabetically for you. I like having aliased imports as a final standalone group because it helps to draws attention to them and highlight to the reader that 'there is something a little bit unusual going on here' with them. As an illustration, here's an example from the main.go file of a web application I was recently working on: import ( "fmt" "net/http" "os" "runtime/debug" "sync" "example.com/internal/logger" "example.com/internal/smtp" "github.com/go-playground/form/v4" "github.com/spf13/pflag" _ "github.com/mattn/go-sqlite3" )

• The 'fat service' pattern for Go web applications
  Alex Edwards

  In this post I'd like to talk about one of my favorite architectural patterns for building web applications and APIs in Go. It's kind of a mix between the service object and fat model patterns — so I mentally refer to it as the 'fat service' pattern, but it might have a more formal name that I'm not aware of 🙃 It's certainly not a perfect pattern (we'll discuss some of the pros and cons later) — but it is (deliberately) simple, pragmatic, and I find it often works well for small-to-medium sized projects. Note: Before we start I'd like to emphasize that there's no single 'correct' way to structure your project in Go. Different architectures suit different projects and teams, and this is just one option to consider. At a high-level, the fat service pattern splits your project code into two distinct 'layers': The application layer. This contains your code related to reading and writing HTTP requests and responses, authenticating/authorizing requests, session management, etc. The service layer. This contains your business logic, defines your core data types, and is also responsible for interacting with any persistent data stores. A fat service example Let's illustrate how this pattern works with an example of a JSON API. Specifically, let's say that we want to build an API with a POST /register endpoint which is used to register a new user. When a client makes a request to this endpoint, let's pretend we want to take the following actions: Parse the JSON input into a Go struct so we can work with it easily. Carry out some validation checks on the data (and return an error response to the client if any of them fail). Create a hash of the new user's password. Insert a record for the user into a database. Send a notification to a Slack channel to say that a new user has registered. Return a 204 No Content response to the client if everything worked successfully. Using the fat service pattern, we could structure our project so that the directory and file layout looks like this: . ├── cmd │ └── api | ├── handlers.go │ └── main.go └── internal └── service ├── service.go └── users.go The cmd/api package will contain the application layer code, and the internal/service package will contain the service layer code. Then, very roughly, the code in our service layer might look something like this: File: internal/service/service.go package service import ( "database/sql" "errors" ) var ErrFailedValidation = errors.New("failed validation") type Service struct { DB *sql.DB SlackWebhookURL string } File: internal/service/users.go package service import ( "github.com/slack-go/slack" "golang.org/x/crypto/bcrypt" ) type RegisterUserInput struct { Username string `json:"username"` Password string `json:"password"` ValidationErrors map[string]string `json:"-"` } func (s *Service) RegisterUser(input *RegisterUserInput) error { input.ValidationErrors = make(map[string]string) if input.Username == "" { input.ValidationErrors["username"] = "must be provided" } // And any other validation checks... if len(input.ValidationErrors) > 0 { return ErrFailedValidation } hashedPassword, err := bcrypt.GenerateFromPassword([]byte(input.Password), 12) if err != nil { return err } _, err = s.DB.Exec("INSERT INTO (username, hashed_password) VALUES ($1, $2)", input.Username, string(hashedPassword)) if err != nil { return err } msg := slack.WebhookMessage{ Username: "robot", Channel: "#general", Text: "A new user has signed up!", } return slack.PostWebhook(s.SlackWebhookURL, &msg) } And the code in our application layer might look like this (I've omitted the helper functions for brevity): File: cmd/api/main.go package main import ( "database/sql" "flag" "log" "net/http" "os" "example.com/internal/service" "github.com/alexedwards/flow" _ "github.com/mattn/go-sqlite3" ) type application struct { logger *log.Logger service *service.Service } func main() { dsn := flag.String("dsn", "./db.sqlite", "sqlite3 DSN") slackWebhookURL := flag.String("slack-webhook-url", "https://hooks.slack.com/services/example", "slack webhook URL for notifications") flag.Parse() logger := log.New(os.Stdout, "", log.LstdFlags|log.Llongfile) db, err := sql.Open("sqlite3", *dsn) if err != nil { logger.Fatal(err) } defer db.Close() app := &application{ logger: logger, service: &service.Service{DB: db, SlackWebhookURL: *slackWebhookURL}, } mux := flow.New() mux.HandleFunc("/register", app.registerUserHandler, "POST") logger.Print("starting server on :3000") err = http.ListenAndServe(":3000", mux) logger.Fatal(err) } File: cmd/api/handlers.go package main import ( "errors" "net/http" "example.com/internal/service" ) func (app *application) registerUserHandler(w http.ResponseWriter, r *http.Request) { var input service.RegisterUserInput err := app.decodeJSON(r.Body, &input) if err != nil { app.badRequest(w, r, err) return } err = app.service.RegisterUser(&input) if err != nil { if errors.Is(err, service.ErrFailedValidation) { app.failedValidation(w, r, input.ValidationErrors) } else { app.serverError(w, r, err) } return } w.WriteHeader(http.StatusNoContent) } Hopefully you get the rough idea. Essentially, our service layer contains a Service.RegisterUser() method which executes all the validation checks, business logic and SQL queries related to registering a user. The expected input to this method is the simple, standard, service.RegisterUserInput Go struct. And in our application layer's registerUserHandler() handler we can decode the JSON request body directly into that struct and pass it on the the service layer, handling any returned errors as necessary. The pros and cons In terms of benefits, there are quite a lot of nice things about this pattern: It's fairly simple. The number of mental hoops to jump through when reading the code is relatively low. You don't have to dig through lots of packages and functions to follow what the code is doing — meaning it's relatively easy for newcomers to your project to understand (or even yourself after a long break). The separation of concerns keeps our registerUserHandler() code primarily focused on reading and writing HTTP requests and responses. For applications with more than a few endpoints, I find that not trying to do everything in your handlers helps to make your codebase easier to navigate and reason about. The code in the service layer can be reused by other applications. For example, we could create a CLI application under cmd/cli with a task that also calls the Service.RegisterUser() method. This one is more personal, but I find it easier to reason about my business logic and write the code for it when the input is a well-defined Go struct with the correct types (rather than a more 'messy' input like a JSON string or HTML-encoded form data). It's really practical for APIs and web applications. You can parse JSON or HTML form data from a request body directly into the service.RegisterUserInput struct in your handlers, and then pass that struct to the service layer for processing. You don't need to create interim types in your handlers to hold the decoded request data, or copy data from one struct to another. Methods in the service layer can potentially return validation errors from multiple points in the code, and you can deal with them all just once in your handler. For example, if our user INSERT failed because we tried to insert a record with a duplicate username, then we could return a "username is already taken" validation error from our service layer in addition to the pre-INSERT validation checks. Working with database transactions is easy. If we wanted to execute multiple SQL statements as part of registering a user in a single transaction, we could initialize the sql.TX, execute all the necessary statements, and commit the transaction all within our Service.RegisterUser() method. We don't need to pass the sql.TX around to a bunch of different places in our codebase. If you want to test only your application layer logic only, this pattern lends itself nicely to creating an interface type that describes the methods on the service.Service struct, which you can then satisfy with a mock implementation. But it's not perfect, and there are also a few downsides: When you are looking at the code for your handlers, you can't immediately see what the expected inputs are. You have to navigate to the service package and look at the fields of the service.RegisterUserInput struct. With most modern text editors this is just one click away, but it still introduces a bit of 'obscurity' and feels less than ideal to me. Not having a separate abstraction for the database logic makes it harder to swap out one database for another in the future (say moving from SQLite to PostgreSQL). You can't easily mock the database calls during tests. Personally I tend to prefer using a test instance of an actual database for testing, so I don't find this too much of a drawback most of the time. But if you need to mock the database (i.e. to speed up test runtime, or because it's a hard requirement from a client) then this pattern doesn't really suit that. Lastly, SQL queries which use database/sql and the Query() method to return multiple rows of data are quite verbose. These queries can take up a lot of visual space and add clutter to the service layer methods — which ultimately starts to reduce the scannability of the code. Using jmoiron/sqlx or blockloop/scan can be a big help here. But overall — so long as you don't need to mock your database calls — I like this pattern. I've used it a lot over the past 3-4 years and have found that the relative simplicity and practical benefits comfortably outweigh any downsides.

• Easy test assertions with Go generics
  Alex Edwards

  Now that Go 1.18 has been released with support for generics, it's easier than ever to create helper functions for your test assertions. Using helpers for your test assertions can help to: Make your test functions clean and clear; Keep test failure messages consistent; And reduce the potential for errors in your code due to typos. To illustrate this, let's say that you have a simple greet() function that you want to test: package main import "fmt" func greet(name string) (string, int) { greeting := fmt.Sprintf("Hello %s", name) // Return the greeting and its length (in bytes). return greeting, len(greeting) } In the past, your test for the greet() function would probably look something like this: package main import "testing" func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") // Test assertion to check the returned greeting string. if greeting != "Hello Alice" { t.Errorf("want: %s; got: %s", "Hello Alice", greeting) } // Test assertion to check the returned greeting length. if greetingLength != 11 { t.Errorf("want: %d; got: %d", 11, greetingLength) } } With Go 1.18, we can use generics and the comparable constraint to create an Equal() helper function which carries out our test assertions. Personally, I like to put this in a reusable assert package. Like so: package assert import "testing" func Equal[T comparable](t *testing.T, expected, actual T) { t.Helper() if expected != actual { t.Errorf("want: %v; got: %v", expected, actual) } } Note: The t.Helper() function indicates to the Go test runner that our Equal() function is a test helper. This means that when t.Errorf() is called from our Equal() function, the Go test runner will report the filename and line number of the code which called our Equal() function in the output. And with that in place, the TestGreet() test can be simplified like so: package main import ( "testing" "your.module.path/assert" // Import your assert package. ) func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") assert.Equal(t, "Hello Alice", greeting) assert.Equal(t, 11, greetingLength) }

• Continuous integration with Go and GitHub Actions
  Alex Edwards

  In this post we're going to walk through how to use GitHub Actions to create a continuous integration (CI) pipeline that automatically tests, vets and lints your Go code. For solo projects I usually create a pre-commit Git hook to carry out these kinds of checks, but for team projects or open-source work — where you don't have control over everyone's development environment — using a CI workflow is a great way to flag up potential problems and help catch bugs before they make it into production or a versioned release. And if you're already using GitHub to host your repository, it's nice and easy to use their built-in functionality to do this without any need for additional third-party tools or services. To demonstrate how it works, let's run through a step-by-step example. If you'd like to follow along, please create a new repository and clone it to your local machine. For the purpose of this post I'm going to use the private repository alexedwards/example. $ git clone git@github.com:alexedwards/example.git Cloning into 'example'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done. Then let's scaffold a simple Go application along with a (failing) test like so: $ cd example/ $ touch main.go main_test.go $ go mod init github.com/alexedwards/example File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { return fmt.Sprintf("Hi %s", name) } File: main_test.go package main import "testing" func Test_sayHello(t *testing.T) { name := "Bob" want := "Hello Bob" if got := sayHello(name); got != want { t.Errorf("hello() = %q, want %q", got, want) } } If you run this application it should compile correctly and print "Hi Alice", but executing go test . will result in a failure. Similar to this: $ go test . --- FAIL: Test_sayHello (0.00s) main_test.go:10: hello() = "Hi Bob", want "Hello Bob" FAIL FAIL github.com/alexedwards/example 0.002s FAIL Creating a workflow file The next thing that we want to do is create a workflow file which describes what we want to do in our CI checks, and when we want them to run. By convention this file should be stored in a .github/workflow directory in the root of your repository and should be in YAML format. Let's create this directory along with an audit.yml workflow file. $ mkdir -p .github/workflows $ touch .github/workflows/audit.yml There's an excellent introduction to the workflow file syntax here, and there's also a collection of templates for different languages and frameworks that you can use as a starting point. But for now, let's jump in and update the workflow file so that it looks like this: File: .github/workflows/audit.yml name: Audit on: push: branches: [main] pull_request: branches: [main] jobs: audit: runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.17 - name: Verify dependencies run: go mod verify - name: Build run: go build -v ./... - name: Run go vet run: go vet ./... - name: Install staticcheck run: go install honnef.co/go/tools/cmd/staticcheck@latest - name: Run staticcheck run: staticcheck ./... - name: Install golint run: go install golang.org/x/lint/golint@latest - name: Run golint run: golint ./... - name: Run tests run: go test -race -vet=off ./... Let's quickly step through this and explain what the different parts of the file do. First we use the on keyword to define when we want the workflow to run. In this case, I've configured the workflow so that it runs when a new commit is made to the main branch, or a pull request is submitted. Then we use the jobs keyword to define a list of the jobs that are to be run. At the moment our workflow only contains one job called audit, but you can specify multiple jobs if you want and (by default) they will be executed in parallel. An independent runner will be spun up for each job. This is essentially a virtual machine that will execute the steps for the job. In the file above we use the runs-on keyword to specify that we want the runner to use Ubuntu 20.04 as a base OS, but others operating systems are available. It's also worth noting that the runner has a lot of useful software and tooling pre-installed. In the first step for our audit job we use the uses keyword to execute the community action actions/checkout@v2. This action will checkout our project repository to the runner so that the following steps access the code. Then we use the actions/setup-go@v2 action to install Go version 1.17 on the runner. Once that's done, in the remaining steps we use the run keyword to execute specific commands on the runner. In this case we build our code and then audit it using the standard go build|vet|test commands and the additional golint and staticcheck tools. Important: If you're following along, please run $ git branch --show-current to check the name of your branch before continuing. In certain cases, your branch may have the name master instead of main, in which case please edit the on directive in your audit.yml file accordingly. Now that's in place, let's commit everything and push the changes to your repository: $ git add . $ git commit -m "Initial commit" $ git push Once the push has completed, head to your repository and select the Actions tab. You should see that the CI 'Audit' workflow is running, similar to the screenshot below. You can click through on the workflow name to see more details while it's running, and after a minute or two you should see that the workflow is terminated due to our failing test. Additionally, as the owner of the repository, you should also get an email notification to tell you that the workflow failed, and everyone who browses the repository will see a red cross symbol next to the commit in the Git history. Fixing the code Let's fix our codebase by updating the sayHello() function to return the correct output, like so: File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { // Change this to "Hello %s" instead of "Hi %s". return fmt.Sprintf("Hello %s", name) } If you want, you can commit this change and push it… $ git add . $ git commit -m "Fix sayHello() to return the correct value" $ git push … and you should see that the 'Audit' job in our workflow file now completes successfully and everything has a nice green check mark next to it. Great! That's working really well and, from now on, any time someone makes a push or pull request to the main branch, the tests and vetting and linter checks will be automatically run. From here, you can extend the workflow to carry out more checks or send additional notifications if you want to — or even expand it to act as a continuous deployment (CD) pipeline that builds and deploys your binaries. To give you some ideas, here are a couple of slightly more complicated workflows from my own projects: Run integration tests against a PostgreSQL database Run audit checks, then build a binary and deploy it to a remote server using Ansible

• Which Go router should I use?
  Alex Edwards

  Note: This post has been fully updated to reflect the new http.ServeMux features released in Go 1.22. When you start to build web applications with Go, one of the first questions you'll probably ask is "which router should I use?". It's not an easy question to answer, either. You've got http.ServeMux in the Go standard library, and probably more than 100 different third-party routers also available — all with distinct APIs, features, and behaviors. Is http.ServeMux going to be sufficient? Or will you need to use a different router? And if so, which one is the right choice? For this blog post, I've evaluated 30 of the most popular third-party routers on GitHub (along with http.ServeMux), created a shortlist of the best options, and made a comparison table you can use to help make your choice. If you want, you can skip to the comparison table and summary. Shortlisted routers There are five routers which make the shortlist and that I recommend using. They are http.ServeMux, httprouter, chi, flow, and gorilla/mux. All the shortlisted routers are well-tested, well-documented, and actively maintained. They have stable APIs, and are compatible with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. There are a few common features that all five of these routers support: Method matching: All let you register routes that require a matching HTTP method (GET, POST etc). Path segment wildcards: All let you declare routes like /movies/{id}/edit where {id} is a dynamic segment in the URL path. Automatic sending of 404 responses: All automatically send plaintext 404 responses when a matching route cannot be found. Automatic sending of 405 responses: All automatically send 405 responses when a route is found with a matching URL pattern, but not a matching HTTP method. Please note though that gorilla/mux does not automatically include an Allow header in 405 responses, and chi will potentially include duplicate values in the Allow header (there is an open issue about this here). In terms of speed, all five routers are fast enough for (almost) every application. Unless you have profiling that confirms your router is a bottleneck in your application, I recommend choosing between them based on the specific features that you need rather than performance. I've personally used all five routers in production applications at different times and have been happy with them. Note: One downside of httprouter is that the API and documentation is a bit confusing. The package was first published prior to the introduction of request context in Go 1.7, and lot of the current API still exists in order to support these older versions of Go. Nowadays, you can write your handlers using regular http.Handler and http.HandlerFunc signatures and all you need is the router.Handler() and router.HandlerFunc() methods to register them, like this. So with that out of the way, I'll start by saying… Use the standard library if you can If you can use http.ServeMux, you probably should. As part of the Go standard library, it's very battle tested and well documented. Using it means that you don't need to import any third-party dependencies, and most other Go developers will also be familiar with how it works. The Go compatibility promise also means that you should be able to rely on http.ServeMux working the same way in the long-term. All of those things are big positives in terms of application maintenance. It also has some really nice features that don't always appear in the third-party routers. Hostname matching: http.ServeMux lets you register routes that require a matching hostname, like example.com/post/{id} and example.org/post/{id}. Hostname matching is also supported by gorilla/mux, and chi supports it via the additional hostrouter package. URL path sanitization: http.ServeMux will automatically sanitize request URL paths and redirect the client if necessary. For example, if a client makes a request to /foo/bar/..//baz they will automatically be sent a 301 redirect to /foo/baz. URL sanitization is also done by gorilla/mux and httprouter in the same way. Automatic handling of HEAD requests: http.ServeMux automatically handles HEAD requests and sends the appropriate headers in the response. This is also supported by flow in the same way. Overlapping routes: If you register the routes /post/edit and /post/{id}, they overlap because a request to /post/edit matches both route patterns. The way that http.ServeMux matches overlapping wildcard routes is smart — the most specific matching route pattern wins and /post/edit is more specific than /post/{id}. This is nice because it means you can register patterns in any order and it won’t affect how http.ServeMux behaves. chi behaves in a similar-ish way to http.ServeMux and will prioritize non-wildcard matches. In contrast, gorilla/mux and flow will dispatch requests to the first matching route, and httprouter simply disallows overlapping routes and will panic if you try to register them. If you need additional features While I recommend using http.ServeMux as your go-to router, there may be times where you need a feature or a behavior that http.ServeMux doesn't provide or easily support. These include: Subsegment wildcards: chi is the only shortlisted router to support more than one wildcard within a single URL path segment, like /articles/{month}-{year}-{day}/{id}. Regexp wildcards: gorilla/mux, chi and flow support regexp wildcards, like /movies/{[a-z-]+}, where [a-z-]+ is a required regexp pattern in the URL path. Header matching: gorilla/mux is the only shortlisted router to easily support routing to different handlers based on the value of a request header (like Authorization or Content-Type). Custom matching rules: gorilla/mux is the only shortlisted router to support custom rules for matching requests (such as routing to different handlers based on IP address). Custom 404 responses: With http.ServeMux it's simple to implement a 'catch all' route "/" which will send a custom 404 response, but doing this will inhibit the automatic sending of 405 responses. There's an open issue about this, and hopefully it will get resolved soon. In contrast, httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 404 response without this problem. Custom 405 responses: With http.ServeMux there is no simple way to send custom 405 responses. Whereas httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 405 responses. But be aware that both chi and gorilla/mux will not automatically set an Allow header if you are using a custom 405 handler. Automatic handling of OPTIONS requests: Both httprouter and flow automatically send correct responses for OPTIONS requests. One route, multiple methods: Both gorilla/mux and flow support matching multiple HTTP methods in a single route declaration. Middleware groups: Both chi and flow provide 'grouping' functionality that lets you batch routes into groups that use specific middleware. Note that you can also wrap http.ServeMux to do this (and I've written about how to do that here). Route reversing: gorilla/mux is the only shortlisted router to support route reversing (like you get in Django, Rails, and Laravel). Subrouters: Both chi and gorilla/mux allow the creation of 'subrouters' that can be assigned to handle a subset of your application routes. Case sensitivity: All shortlisted routers require a case-sensitive match on non-wildcard parts of a route, apart from httprouter which is case-insensitive. Trailing slashes: All shortlisted routers treat trailing slashes as significant (i.e. /foo is a different route to /foo/). But... gorilla/mux has an optional StrictSlash setting where requests to /foo can automatically be redirected to /foo/. In contrast, chi has an optional RedirectSlashes middleware which will automatically redirect requests from /foo/ to /foo. And httprouter will automatically redirect requests from /foo to /foo/ and vice-versa if a matching route exists — this can be disabled via the RedirectTrailingSlash setting. Comparison table Summary If the comparison table is too overwhelming, or you don't yet know what your full requirements will be, I suggest falling back to the following guidelines: If you know you're going to have routes with complex matching requirements (i.e. more than just simple method, wildcard segment, and hostname matching), then opt for gorilla/mux or chi. If you're building something that requires custom 404 and 405 responses, and it's important that it adheres correctly to the HTTP specs (such as a JSON API for public use), opt for httprouter or flow. If you know that your application will require a lot of route-specific middleware, opt for chi or flow because of their middleware grouping functionality. Otherwise, start with http.ServeMux, and refactor to use a third-party router only if there is a specific feature or behavior that you need. Other routers For completeness, the other routers that I evaluated are listed below, along with a short note to explain why they didn't made the shortlist. Note: I used the question "does the repository contain a go.mod file?" as a proxy measure for whether a codebase is currently maintained or not. This seems reasonable — if the maintainer is still engaged with the Go world and caring for the code, my guess is that they would have updated the repository to use modules at some point. Repository Notes celrenheit/lion Currently unmaintained. claygod/Bxog Currently unmaintained. clevergo/clevergo Uses custom handler signature (not http.Handler or http.HandlerFunc). dimfeld/httptreemux Doesn’t fully support http.Handler. Requires middleware for setting custom 404/405 handlers. donutloop/mux Currently unmaintained. gernest/alien Currently unmaintained. go-ozzo/ozzo-routing Uses custom handler signature (not http.Handler or http.HandlerFunc). go-playground/lars Currently unmaintained. go-zoo/bone Good, but has similar use case to chi (which offers more). Incomplete tests. go101/tinyrouter Verbose route declarations. Doesn’t automatically send 405 responses. gocraft/web Currently unmaintained. goji/goji Slightly unusual, but flexible, API which supports custom matchers. Requires middleware for setting custom 404/405 handlers. Good, but I think gorilla/mux offers similar features and is easier to use. goroute/route Uses custom handler signature (not http.Handler or http.HandlerFunc). gowww/router Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. GuilhermeCaruso/bellt No way to set custom 404 or 405 handlers. husobee/vestigo Currently unmaintained. Only supports http.HandlerFunc. naoina/denco Currently unmaintained. nbari/violetear Good, but has similar use case to chi (which offers more). Wraps http.ResponseWriter with own custom type, which may cause problems in some cases. nbio/hitch Lacking documentation. nissy/bon Currently unmaintained. razonyang/fastrouter Currently unmaintained. rs/xmux Currently unmaintained. Uses custom handler signature (not http.Handler or http.HandlerFunc). takama/router Currently unmaintained. vardius/gorouter Good, but has similar use case to chi (which offers more). Four major versions in 5 years suggests the API may not be reliable. VividCortex/siesta Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. xujiajun/gorouter Currently unmaintained.

• Custom command-line flags with flag.Func
  Alex Edwards

  One of my favorite things about the recent Go 1.16 release is a small — but very welcome — addition to the flag package: the flag.Func() function. This makes it much easier to define and use custom command-line flags in your application. For example, if you want to parse a flag like --pause=10s directly into a time.Duration type, or parse --urls="http://example.com http://example.org" directly into a []string slice, then previously you had two options. You could either create a custom type to implement the flag.Value interface, or use a third-party package like pflag. But now the flag.Func() function gives you a simple and lightweight alternative. In this short post we're going to take a look at a few examples of how you can use it in your own code. Parsing custom flag types To demonstrate how this works, let's start with the two examples I gave above and create a sample application which accepts a list of URLs and then prints them out with a pause between them. Similar to this: $ go run . --pause=3s --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:16:04 http://example.com 2021/03/08 08:16:07 http://example.org 2021/03/08 08:16:10 http://example.net To make this work, we'll need to do two things: Convert the --pause flag value from a 'human-readable' string like 200ms, 5s or 10m into a native Go time.Duration type. We can do this using the time.ParseDuration() function. Split the values in the --urls flag into a slice, so we can loop through them. The strings.Fields function is a good fit for this task. We can use those together with flag.Func() like so: package main import ( "flag" "log" "strings" "time" ) func main() { // First we need to declare variables to hold the values from the // command-line flags. Notice that we also need to set any defaults, // which will be used if the relevant flag is not provided at runtime. var ( urls []string // Default of the empty slice pause time.Duration = time.Second // Default of one second ) // The flag.Func() function takes three parameters: the flag name, // descriptive help text, and a function with the signature // `func(string) error` which is called to process the string value // from the command-line flag at runtime and assign it to the necessary // variable. In this case, we use strings.Fields() to split the string // based on whitespace and store the resulting slice in the urls // variable that we declared above. We then return nil from the // function to indicate that the flag was parsed without any errors. flag.Func("urls", "List of URLs to print", func(flagValue string) error { urls = strings.Fields(flagValue) return nil }) // Likewise we can do the same thing to parse the pause duration. The // time.ParseDuration() function may throw an error here, so we make // sure to return that from our function. flag.Func("pause", "Duration to pause between printing URLs", func(flagValue string) error { var err error pause, err = time.ParseDuration(flagValue) return err }) // Importantly, call flag.Parse() to trigger actual parsing of the // flags. flag.Parse() // Print out the URLs, pausing between each iteration. for _, u := range urls { log.Print(u) time.Sleep(pause) } } If you try to run this application, you should find that the flags are parsed and work just like you would expect. For example: $ go run . --pause=500ms --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:22:33 http://example.com 2021/03/08 08:22:34 http://example.org 2021/03/08 08:22:34 http://example.net Whereas if you provide an invalid flag value that triggers an error in one of the flag.Func() functions, Go will automatically display the corresponding error message and exit. For example: $ go run . --pause=500xx --urls="http://example.com http://example.org http://example.net" invalid value "500xx" for flag -pause: time: unknown unit "xx" in duration "500xx" Usage of /tmp/go-build3141872390/b001/exe/example.text: -pause value Duration to pause between printing URLs -urls value List of URLs to print exit status 2 It's really important to point out here that if a flag isn't provided, the corresponding flag.Func() function will not be called at all. This means that you cannot set a default value inside a flag.Func() function, so trying to do something like this won't work: flag.Func("pause", "Duration to pause between printing URLs (default 1s)", func(flagValue string) error { // DON'T DO THIS! This function wont' be called if the flag value is "". if flagValue == "" { pause = time.Second return nil } var err error pause, err = time.ParseDuration(flagValue) return err }) On the plus side though, there are no restrictions on the code that can be contained in a flag.Func() function, so if you want, you could get even fancier with this and parse the URLs into a []*url.URL slice instead of a []string. Like so: var ( urls []*url.URL pause time.Duration = time.Second ) flag.Func("urls", "List of URLs to print", func(flagValue string) error { for _, u := range strings.Fields(flagValue) { parsedURL, err := url.Parse(u) if err != nil { return err } urls = append(urls, parsedURL) } return nil }) Validating flag values The flag.Func() function also opens up some new opportunities for validating input data from command-line flags. For example, let's say that your application has an --environment flag and you want to restrict the possible values to development, staging or production. To do that, you can implement a flag.Func() function similar to this: package main import ( "errors" "flag" "fmt" ) func main() { var ( environment string = "development" ) flag.Func("environment", "Operating environment", func(flagValue string) error { for _, allowedValue := range []string{"development", "staging", "production"} { if flagValue == allowedValue { environment = flagValue return nil } } return errors.New(`must be one of "development", "staging" or "production"`) }) flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } Making reusable helpers If you find yourself repeating the same code in your flag.Func() functions, or the logic is getting too complex, it's possible to break it out into a reusable helper. For example, we could rewrite the example above to process our --environment flag via a generic enumFlag() function, like so: package main import ( "flag" "fmt" ) func main() { var ( environment string = "development" ) enumFlag(&environment, "environment", []string{"development", "staging", "production"}, "Operating environment") flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } func enumFlag(target *string, name string, safelist []string, usage string) { flag.Func(name, usage, func(flagValue string) error { for _, allowedValue := range safelist { if flagValue == allowedValue { *target = flagValue return nil } } return fmt.Errorf("must be one of %v", safelist) }) }

• Golang Interfaces explained
  Alex Edwards

  For the past few months I've been running a survey which asks people what they're finding difficult about learning Go. And something that keeps coming up in the responses is the concept of interfaces. I get that. Go was the first language I ever used that had interfaces, and I remember at the time that the whole concept felt pretty confusing. So in this tutorial I want to do a few things: Provide a plain-English explanation of what interfaces are; Explain why they are useful and how you might want to use them in your code; Talk about what interface{} (the empty interface) is; And run through some of the helpful interface types that you'll find in the standard library. So what is an interface? An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. One example of an interface type from the standard library is the fmt.Stringer interface, which looks like this: type Stringer interface { String() string } We say that something satisfies this interface (or implements this interface) if it has a method with the exact signature String() string. For example, the following Book type satisfies the interface because it has a String() string method: type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } It's not really important what this Book type is or does. The only thing that matters is that is has a method called String() which returns a string value. Or, as another example, the following Count type also satisfies the fmt.Stringer interface — again because it has a method with the exact signature String() string. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } The important thing to grasp is that we have two different types, Book and Count, which do different things. But the thing they have in common is that they both satisfy the fmt.Stringer interface. You can think of this the other way around too. If you know that an object satisfies the fmt.Stringer interface, you can rely on it having a method with the exact signature String() string that you can call. Now for the important part. Wherever you see declaration in Go (such as a variable, function parameter or struct field) which has an interface type, you can use an object of any type so long as it satisfies the interface. For example, let's say that you have the following function: func WriteLog(s fmt.Stringer) { log.Print(s.String()) } Because this WriteLog() function uses the fmt.Stringer interface type in its parameter declaration, we can pass in any object that satisfies the fmt.Stringer interface. For example, we could pass either of the Book and Count types that we made earlier to the WriteLog() method, and the code would work OK. Additionally, because the object being passed in satisfies the fmt.Stringer interface, we know that it has a String() string method that the WriteLog() function can safely call. Let's put this together in an example, which gives us a peek into the power of interfaces. package main import ( "fmt" "strconv" "log" ) // Declare a Book type which satisfies the fmt.Stringer interface. type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } // Declare a Count type which satisfies the fmt.Stringer interface. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } // Declare a WriteLog() function which takes any object that satisfies // the fmt.Stringer interface as a parameter. func WriteLog(s fmt.Stringer) { log.Print(s.String()) } func main() { // Initialize a Count object and pass it to WriteLog(). book := Book{"Alice in Wonderland", "Lewis Carrol"} WriteLog(book) // Initialize a Count object and pass it to WriteLog(). count := Count(3) WriteLog(count) } This is pretty cool. In the main function we've created different Book and Count types, but passed both of them to the same WriteLog() function. In turn, that calls their relevant String() functions and logs the result. If you run the code, you should get some output which looks like this: 2009/11/10 23:00:00 Book: Alice in Wonderland - Lewis Carrol 2009/11/10 23:00:00 3 I don't want to labor the point here too much. But the key thing to take away is that by using a interface type in our WriteLog() function declaration, we have made the function agnostic (or flexible) about the exact type of object it receives. All that matters is what methods it has. Why are they useful? There are all sorts of reasons that you might end up using a interface in Go, but in my experience the three most common are: To help reduce duplication or boilerplate code. To make it easier to use mocks instead of real objects in unit tests. As an architectural tool, to help enforce decoupling between parts of your codebase. Let's step through these three use-cases and explore them in a bit more detail. Reducing boilerplate code OK, imagine that we have a Customer struct containing some data about a customer. In one part of our codebase we want to write the customer information to a bytes.Buffer, and in another part of our codebase we want to write the customer information to an os.File on disk. But in both cases, we want to serialize the customer struct to JSON first. This is a scenario where we can use Go's interfaces to help reduce boilerplate code. The first thing you need to know is that Go has an io.Writer interface type which looks like this: type Writer interface { Write(p []byte) (n int, err error) } And we can leverage the fact that both bytes.Buffer and the os.File type satisfy this interface, due to them having the bytes.Buffer.Write() and os.File.Write() methods respectively. Let's take a look at a simple implementation: package main import ( "bytes" "encoding/json" "io" "log" "os" ) // Create a Customer type type Customer struct { Name string Age int } // Implement a WriteJSON method that takes an io.Writer as the parameter. // It marshals the customer struct to JSON, and if the marshal worked // successfully, then calls the relevant io.Writer's Write() method. func (c *Customer) WriteJSON(w io.Writer) error { js, err := json.Marshal(c) if err != nil { return err } _, err = w.Write(js) return err } func main() { // Initialize a customer struct. c := &Customer{Name: "Alice", Age: 21} // We can then call the WriteJSON method using a buffer... var buf bytes.Buffer err := c.WriteJSON(&buf) if err != nil { log.Fatal(err) } // Or using a file. f, err := os.Create("/tmp/customer") if err != nil { log.Fatal(err) } defer f.Close() err = c.WriteJSON(f) if err != nil { log.Fatal(err) } } Of course, this is just a toy example (and there are other ways we could structure the code to achieve the same end result). But it nicely illustrates the benefit of using an interface — we can create the Customer.WriteJSON() method once, and we can call that method any time that we want to write to something that satisfies the io.Writer interface. But if you're new to Go, this still begs a couple of questions: How do you know that the io.Writer interface even exists? And how do you know in advance that bytes.Buffer and os.File both satisfy it? There's no easy shortcut here I'm afraid — you simply need to build up experience and familiarity with the interfaces and different types in the standard library. Spending time thoroughly reading the standard library documentation, and looking at other people's code will help here. But as a quick-start I've included a list of some of the most useful interface types at the end of this post. But even if you don't use the interfaces from the standard library, there's nothing to stop you from creating and using your own interface types. We'll cover how to do that next. Unit testing and mocking To help illustrate how interfaces can be used to assist in unit testing, let's take a look at a slightly more complex example. Let's say you run a shop, and you store information about the number of customers and sales in a PostgreSQL database. You want to write some code that calculates the sales rate (i.e. sales per customer) for the past 24 hours, rounded to 2 decimal places. A minimal implementation of the code for that could look something like this: // File: main.go package main import ( "fmt" "log" "time" "database/sql" _ "github.com/lib/pq" ) type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } func calculateSalesRate(sdb *ShopDB) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sdb.CountSales(since) if err != nil { return "", err } customers, err := sdb.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } Now, what if we want to create a unit test for the calculateSalesRate() function to make sure that the math logic in it is working correctly? Currently this is a bit of a pain. We would need to set up a test instance of our PostgreSQL database, along with setup and teardown scripts to scaffold the database with dummy data. That's quite lot of work when all we really want to do is test our math logic. So what can we do? You guessed it — interfaces to the rescue! A solution here is to create our own interface type which describes the CountSales() and CountCustomers() methods that the calculateSalesRate() function relies on. Then we can update the signature of calculateSalesRate() to use this custom interface type as a parameter, instead of the concrete *ShopDB type. Like so: // File: main.go package main import ( "database/sql" "fmt" "log" "time" _ "github.com/lib/pq" ) // Create our own custom ShopModel interface. Notice that it is perfectly // fine for an interface to describe multiple methods, and that it should // describe input parameter types as well as return value types. type ShopModel interface { CountCustomers(time.Time) (int, error) CountSales(time.Time) (int, error) } // The ShopDB type satisfies our new custom ShopModel interface, because it // has the two necessary methods -- CountCustomers() and CountSales(). type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } // Swap this to use the ShopModel interface type as the parameter, instead of the // concrete *ShopDB type. func calculateSalesRate(sm ShopModel) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sm.CountSales(since) if err != nil { return "", err } customers, err := sm.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } With that done, it's straightforward for us to create a mock which satisfies our ShopModel interface. We can then use that mock during unit tests to test that the math logic in our calculateSalesRate() function works correctly. Like so: // File: main_test.go package main import ( "testing" "time" ) type MockShopDB struct{} func (m *MockShopDB) CountCustomers(_ time.Time) (int, error) { return 1000, nil } func (m *MockShopDB) CountSales(_ time.Time) (int, error) { return 333, nil } func TestCalculateSalesRate(t *testing.T) { // Initialize the mock. m := &MockShopDB{} // Pass the mock to the calculateSalesRate() function. sr, err := calculateSalesRate(m) if err != nil { t.Fatal(err) } // Check that the return value is as expected, based on the mocked // inputs. exp := "0.33" if sr != exp { t.Fatalf("got %v; expected %v", sr, exp) } } You could run that test now, everything should work fine. Application architecture In the previous examples, we've seen how interfaces can be used to decouple certain parts of your code from relying on concrete types. For instance, the calculateSalesRate() function is totally flexible about what you pass to it — the only thing that matters is that it satisfies the ShopModel interface. You can extend this idea to create decoupled 'layers' in larger projects. Let's say that you are building a web application which interacts with a database. If you create an interface that describes the exact methods for interacting with the database, you can refer to the interface throughout your HTTP handlers instead of a concrete type. Because the HTTP handlers only refer to an interface, this helps to decouple the HTTP layer and database-interaction layer. It makes it easier to work on the layers independently, and to swap out one layer in the future without affecting the other. I've written about this pattern in this previous blog post, which goes into more detail and provides some practical example code. What is the empty interface? If you've been programming with Go for a while, you've probably come across the empty interface type: interface{}. This can be a bit confusing, but I'll try to explain it here. At the start of this blog post I said: An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. The empty interface type essentially describes no methods. It has no rules. And because of that, it follows that any and every object satisfies the empty interface. Or to put it in a more plain-English way, the empty interface type interface{} is kind of like a wildcard. Wherever you see it in a declaration (such as a variable, function parameter or struct field) you can use an object of any type. Take a look at the following code: package main import "fmt" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 fmt.Printf("%+v", person) } In this code snippet we initialize a person map, which uses the string type for keys and the empty interface type interface{} for values. We've assigned three different types as the map values (a string, int and float32) — and that's OK. Because objects of any and every type satisfy the empty interface, the code will work just fine. You can give it a try here, and when you run it you should see some output which looks like this: map[age:21 height:167.64 name:Alice] But there's an important thing to point out when it comes to retrieving and using a value from this map. For example, let's say that we want to get the "age" value and increment it by 1. If you write something like the following code, it will fail to compile: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 person["age"] = person["age"] + 1 fmt.Printf("%+v", person) } And you'll get the following error message: invalid operation: person["age"] + 1 (mismatched types interface {} and int) This happens because the value stored in the map takes on the type interface{}, and ceases to have it's original, underlying, type of int. Because it's no longer an int type we cannot add 1 to it. To get around this this, you need to type assert the value back to an int before using it. Like so: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 age, ok := person["age"].(int) if !ok { log.Fatal("could not assert value to int") return } person["age"] = age + 1 log.Printf("%+v", person) } If you run this now, everything should work as expected: 2009/11/10 23:00:00 map[age:22 height:167.64 name:Alice] So when should you use the empty interface type in your own code? The answer is probably not that often. If you find yourself reaching for it, pause and consider whether using interface{} is really the right option. As a general rule it's clearer, safer and more performant to use concrete types — or non-empty interface types — instead. In the code snippet above, it would have been more appropriate to define a Person struct with relevant typed fields similar to this: type Person struct { Name string Age int Height float32 } But that said, the empty interface is useful in situations where you need to accept and work with unpredictable or user-defined types. You'll see it used in a a number of places throughout the standard library for that exact reason, such as in the gob.Encode, fmt.Print and template.Execute functions. Comman and useful types Lastly, here's a short list of some of the most common and useful interfaces in the standard library. If you're not familiar with them already, then I recommend taking out a bit of time to look at the relevant documentation for them. builtin.Error fmt.Stringer io.Reader io.Writer io.ReadWriteCloser http.ResponseWriter http.Handler There is also a longer and more comprehensive listing of standard libraries available in this gist.

• Streamline your Sublime Text + Go workflow
  Alex Edwards

  For the past couple of years I've used Sublime Text as my primary code editor, along with the GoSublime plugin to provide some extra IDE-like features. But I've recently swapped GoSublime for a more modular plugin setup and have been really happy with the way it's worked out. Although it took a while to configure, it's resulted in a coding environment that feels clearer to use and more streamlined than before. I've opted for: Tooling integration with the official sublime-build plugin. Automatic formatting with the Gofmt plugin and goimports. Code linting with the SublimeLinter plugin and gometalinter. Autocompletion with the gocode package. Code navigation with the GoGuru plugin. Snippet management with Sublime Text's inbuilt tool and the PackageResourceViewer plugin. In this post I'm going to run through the process of setting these up. If you haven't come across these plugins before, I recommend giving them a try! Prerequisites To work correctly some of these Sublime Text plugins need an explicit $GOPATH environment variable to be set. And if you're following along, you should also make sure that your workspace's bin directory is on your system path. Accordingly my bash ~/.profile configuration includes these lines: ... export GOPATH=/home/alex/Code/go export PATH=$PATH:$GOPATH/bin You'll also need to install Package Control, if you haven't already. In the latest version of Sublime Text the easiest way to do that by going to Tools > Install Package Control…. Tooling integration The official sublime-build plugin provides integrations so you can execute common go commands (like go run, go test and go get) without leaving your editor. You can install it like so: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Golang Build and hit Enter to install the package. After installation should see a bunch of new tools in your command palette. Their names are pretty self explanatory: Build With: Go Build With: Go - Clean Build With: Go - Install Build With: Go - Run Build With: Go - Test Build With: Go - Cross-Compile Go: Get Go: Open Terminal When you run these commands they will open and execute in a panel within Sublime Text. As an example, here's a screenshot of output from the Build With: Go - Test command: Automatic formatting For automatic formatting of .go files I've been using the Gofmt plugin. You can install it as follows: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Gofmt and hit Enter to install the package. By default this will run go fmt -s -e on the current file each time it is saved. I've customised this further to use the goimports tool. If you're not already familiar with goimports, it runs go fmt and fixes your import lines — adding missing packages and removing unreferenced ones as necessary. To set this up you'll need to install goimports and make sure it's available on your system path: $ go get golang.org/x/tools/cmd/goimports $ which goimports /home/alex/Code/go/bin/goimports When that's installed, you'll then need to change the Gofmt plugin settings in Sublime Text by opening Preferences > Package Settings > Gofmt > Settings - User and adding the following configuration settings: { "cmds": [ ["goimports"] ], "format_on_save": true } (You'll probably need to restart Sublime Text for this to take effect.) Each time you now save a .go file, you'll find that it gets automatically formatted and the import packages are updated. No more "imported and not used" errors! Code linting For linting of source code I'm using the SublimeLinter plugin. This plugin isn't a linter itself, but provides a framework for running linters and displaying error messages. You can install it like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter and hit Enter to install the package. The next step is to install an actual linter. I'm using gometalinter, which acts as a wrapper around a bunch of different linters and picks up more potential problems and inefficiencies than using go vet and golint alone. You can install it with the commands: $ go get github.com/alecthomas/gometalinter $ which gometalinter /home/alex/Code/go/bin/gometalinter $ gometalinter --install Once that's done, you'll need to install the SublimeLinter-contrib-gometalinter plugin. This acts as the bridge between SublimeLinter and gometalinter. Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter-contrib-gometalinter and hit Enter to install the package. By default the linter will run in the background as you type, and errors will be shown in the Sublime Text status bar at the bottom of the screen. But I've found suits me more to only lint when saving a file and to display all errors at once in a panel. If you want to do the same, go to Preferences > Package Settings > SublimeLinter > Settings and add the following settings to the SublimeLinter Settings - User file: { "show_panel_on_save": "window", "lint_mode": "save", } I should mention that the SublimeLinter-contrib-gometalinter plugin only executes the 'fast' linters included in gometalinter. You can see exactly which ones are run by checking the source code. Autocompletion For autocompletion I'm using the gocode package, which provides a deamon for code completion. You can install it like so: $ go get github.com/mdempsky/gocode $ which gocode /home/alex/Code/go/bin/gocode There isn't currently a gocode plugin available via Sublime Text package control (I might add one soon!)… but there is a plugin included in the subl3 directory within the gocode source itself. You should be able to copy it into your Sublime Text Packages directory with the following command: $ cp -r $GOPATH/src/github.com/mdempsky/gocode/subl3 ~/.config/sublime-text-3/Packages/gocode If you open the command palette and run Package Control: List Packages you should see a gocode entry in the list. By default Sublime Text will make autocomplete suggestions whenever a letter is pressed. But when working with Go I like also to display potential method names whenever I hit the . character. You can make that happen by going to Preferences > Settings and adding a new trigger in the Preferences.sublime-settings - User file: { ... "auto_complete_triggers": [ {"selector": "text.html", "characters": " You'll need to then restart Sublime Text for the settings to take effect. Once you have, you should have autocomplete working nicely and looking something like this: Code navigation To help with navigating code I use the guru tool, which you can install with the following command: $ go get golang.org/x/tools/cmd/guru $ which guru /home/alex/Code/go/bin/guru To integrate this with Sublime Text you'll also need to install the GoGuru plugin like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type GoGuru and hit Enter to install the package. To use the GoGuru tool, first place your cursor over the piece of code you're interested in. Then if you open the command palette and type the GoGuru prefix you'll see a list of available commands, including: GoGuru: callees – Show possible targets of selected function call GoGuru: callers – Show possible callers of selected function GoGuru: callstack – Show path from callgraph root to selected function GoGuru: definition – Show declaration of selected identifier GoGuru: describe – Describe selected syntax: definition, methods, etc GoGuru: freevars – Show free variables of selection GoGuru: implements – Show 'implements' relation for selected type or method GoGuru: jump to definition – Open the file at the declaration of selected identifier GoGuru: peers – Show send/receive corresponding to selected channel op GoGuru: pointsto – Show variables the selected pointer may point to GoGuru: referrers – Show all refs to thing denoted by selected identifier GoGuru: what – Show basic information about the selected syntax node GoGuru: whicherrs – Show possible values of the selected error variable You can find a detailed description these commands and their behaviour in this GoogleDoc. I don't use the GoGuru plugin as often as the others, but when working on a unfamiliar codebase it definitely makes navigating code and building up a mental map of how things work easier. I find the GoGuru: jump to definition and GoGuru: callers commands particularly useful, and easier to use than grepping or running Ctrl+F on the repository. As an illustration, here's a screenshot of running GoGuru: callers on the Sum function: Snippets Sublime Text ships with a pretty good workflow for creating and using custom snippets. If you're not already familiar with this Jimmy Zhang has written a great in-depth guide that I recommend reading. My most frequently-used snippet is probably this one for creating a HTTP handler function: hf source.go One thing that bugged me for a while was the built-in snippets for Go that Sublime Text ships with. In particular I didn't like the way that the main() snippet kept triggering whenever I wrote out "package main". If, like me, you want to edit these built-in snippets the easiest way is probably with the PackageResourceViewer plugin. You can install this as follows: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type PackageResourceViewer and hit Enter to install the package. Once installed you can open the command palette and run PackageResourceViewer: Open Resource which will list all packages on your system. If you navigate through Go > Snippets/ you should see a list of all the built-in snippets and you can open and edit them as you wish. Hint: You can also use PackageResourceViewer to edit your own custom snippets without leaving SublimeText. If – for example – your custom snippets are saved under your Packages/User directory, you can open them by running PackageResourceViewer: Open Resource and navigating to the User folder.

• Configuring sql.DB for better performance
  Alex Edwards

  There are a lot of good tutorials which talk about Go's sql.DB type and how to use it to execute SQL database queries and statements. But most of them gloss over the SetMaxOpenConns(), SetMaxIdleConns() and SetConnMaxLifetime() methods — which you can use to configure the behavior of sql.DB and alter its performance. In this post I'd like to explain exactly what these settings do and demonstrate the (positive and negative) impact that they can have. Open and idle connections I'll begin with a little background. A sql.DB object is a pool of many database connections which contains both 'in-use' and 'idle' connections. A connection is marked as in-use when you are using it to perform a database task, such as executing a SQL statement or querying rows. When the task is complete the connection is marked as idle. When you instruct sql.DB to perform a database task, it will first check if any idle connections are already available in the pool. If one is available then Go will reuse this existing connection and mark it as in-use for the duration of the task. If there are no idle connections in the pool when you need one, then Go will create an additional new additional connection. The SetMaxOpenConns method By default there's no limit on the number of open connections (in-use + idle) at the same time. But you can implement your own limit via the SetMaxOpenConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently open connections (in-use + idle) // to 5. Setting this to less than or equal to 0 will mean there is no // maximum limit (which is also the default setting). db.SetMaxOpenConns(5) In this example code the pool now has a maximum limit of 5 concurrently open connections. If all 5 connections are already marked as in-use and another new connection is needed, then the application will be forced to wait until one of the 5 connections is freed up and becomes idle. To illustrate the impact of changing MaxOpenConns I ran a benchmark test with the maximum open connections set to 1, 2, 5, 10 and unlimited. The benchmark executes parallel INSERT statements on a PostgreSQL database and you can find the code in this gist. Here's the results: BenchmarkMaxOpenConns1-8 500 3129633 ns/op 478 B/op 10 allocs/op BenchmarkMaxOpenConns2-8 1000 2181641 ns/op 470 B/op 10 allocs/op BenchmarkMaxOpenConns5-8 2000 859654 ns/op 493 B/op 10 allocs/op BenchmarkMaxOpenConns10-8 2000 545394 ns/op 510 B/op 10 allocs/op BenchmarkMaxOpenConnsUnlimited-8 2000 531030 ns/op 479 B/op 9 allocs/op PASS Edit: To make clear, the purpose of this benchmark is not to simulate 'real-life' behaviour of an application. It's solely to help illustrate how sql.DB behaves behind the scenes and the impact of changing MaxOpenConns on that behaviour. For this benchmark we can see that the more open connections that are allowed, the less time is taken to perform the INSERT on the database (3129633 ns/op with 1 open connection compared to 531030 ns/op for unlimited connections — about 6 times quicker). This is because the more open connections that are permitted, the more database queries can be performed concurrently. The SetMaxIdleConns method By default sql.DB allows a maximum of 2 idle connections to be retained in the connection pool. You can change this via the SetMaxIdleConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently idle connections to 5. Setting this // to less than or equal to 0 will mean that no idle connections are retained. db.SetMaxIdleConns(5) In theory, allowing a higher number of idle connections in the pool will improve performance because it makes it less likely that a new connection will need to be established from scratch — therefore helping to save resources. Lets take a look at the same benchmark with the maximum idle connections is set to none, 1, 2, 5 and 10 (and the number of open connections is unlimited): BenchmarkMaxIdleConnsNone-8 300 4567245 ns/op 58174 B/op 625 allocs/op BenchmarkMaxIdleConns1-8 2000 568765 ns/op 2596 B/op 32 allocs/op BenchmarkMaxIdleConns2-8 2000 529359 ns/op 596 B/op 11 allocs/op BenchmarkMaxIdleConns5-8 2000 506207 ns/op 451 B/op 9 allocs/op BenchmarkMaxIdleConns10-8 2000 501639 ns/op 450 B/op 9 allocs/op PASS When MaxIdleConns is set to none, a new connection has to be created from scratch for each INSERT and we can see from the benchmarks that the average runtime and memory usage is comparatively high. Allowing just 1 idle connection to be retained and reused makes a massive difference to this particular benchmark — it cuts the average runtime by about 8 times and reduces memory usage by about 20 times. Going on to increase the size of the idle connection pool makes the performance even better, although the improvements are less pronounced. So should you maintain a large idle connection pool? The answer is it depends on the application. It's important to realise that keeping an idle connection alive comes at a cost — it takes up memory which can otherwise be used for both your application and the database. It's also possible that if a connection is idle for too long then it may become unusable. For example, MySQL's wait_timeout setting will automatically close any connections that haven't been used for 8 hours (by default). When this happens sql.DB handles it gracefully. Bad connections will automatically be retried twice before giving up, at which point Go will remove the connection from the pool and create a new one. So setting MaxIdleConns too high may actually result in connections becoming unusable and more resources being used than if you had a smaller idle connection pool (with fewer connections that are used more frequently). So really you only want to keep a connection idle if you're likely to be using it again soon. One last thing to point out is that MaxIdleConns should always be less than or equal to MaxOpenConns. Go enforces this and will automatically reduce MaxIdleConns if necessary. The SetConnMaxLifetime method Let's now take a look at the SetConnMaxLifetime() method which sets the maximum length of time that a connection can be reused for. This can be useful if your SQL database also implements a maximum connection lifetime or if — for example — you want to facilitate gracefully swapping databases behind a load balancer. You use it like this: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum lifetime of a connection to 1 hour. Setting it to 0 // means that there is no maximum lifetime and the connection is reused // forever (which is the default behavior). db.SetConnMaxLifetime(time.Hour) In this example all our connections will 'expire' 1 hour after they were first created, and cannot be reused after they've expired. But note: This doesn't guarantee that a connection will exist in the pool for a whole hour; it's quite possible that the connection will have become unusable for some reason and been automatically closed before then. A connection can still be in use more than one hour after being created — it just cannot start to be reused after that time. This isn't an idle timeout. The connection will expire 1 hour after it was first created — not 1 hour after it last became idle. Once every second a cleanup operation is automatically run to remove 'expired' connections from the pool. In theory, the shorter ConnMaxLifetime is the more often connections will expire — and consequently — the more often they will need to be created from scratch. To illustrate this I ran the benchmarks with ConnMaxLifetime set to 100ms, 200ms, 500ms, 1000ms and unlimited (reused forever), with the default settings of unlimited open connections and 2 idle connections. These time periods are obviously much, much shorter than you'd use in most applications but they help illustrate the behaviour well. BenchmarkConnMaxLifetime100-8 2000 637902 ns/op 2770 B/op 34 allocs/op BenchmarkConnMaxLifetime200-8 2000 576053 ns/op 1612 B/op 21 allocs/op BenchmarkConnMaxLifetime500-8 2000 558297 ns/op 913 B/op 14 allocs/op BenchmarkConnMaxLifetime1000-8 2000 543601 ns/op 740 B/op 12 allocs/op BenchmarkConnMaxLifetimeUnlimited-8 3000 532789 ns/op 412 B/op 9 allocs/op PASS In these particular benchmarks we can see that memory usage was more than 3 times greater with a 100ms lifetime compared to an unlimited lifetime, and the average runtime for each INSERT was also slightly longer. If you do set ConnMaxLifetime in your code, it is important to bear in mind the frequency at which connections will expire (and subsequently be recreated). For example, if you have 100 total connections and a ConnMaxLifetime of 1 minute, then your application can potentially kill and recreate up to 1.67 connections (on average) every second. You don't want this frequency to be so great that it ultimately hinders performance, rather than helping it. Exceeding connection limits Lastly, this article wouldn't be complete without mentioning what happens if you exceed a hard limit on the number of database connections. As an illustration, I'll change my postgresql.conf file so only a total of 5 connections are permitted (the default is 100)... max_connections = 5 And then rerun the benchmark test with unlimited open connections... BenchmarkMaxOpenConnsUnlimited-8 --- FAIL: BenchmarkMaxOpenConnsUnlimited-8 main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already FAIL As soon as the hard limit of 5 connections is hit my database driver (pq) immediately returns a sorry, too many clients already error message instead of completing the INSERT. To prevent this error we need to set the total maximum of open connections (in-use + idle) in sql.DB to comfortably below 5. Like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the number of open connections (in-use + idle) to a maximum total of 3. db.SetMaxOpenConns(3) Now there will only ever be a maximum of 3 connections created by sql.DB at any moment in time, and the benchmark should run without any errors. But doing this comes with a big caveat: when the open connection limit is reached, any new database tasks that your application needs to execute will be forced to wait until a connection becomes free. In the context of a web application, for example, the user's HTTP request would appear to 'hang' and could potentially even timeout while waiting for the database task to be run. To mitigate this you should always pass in a context.Context object with a fixed, fast, timeout when making database calls, using the context-enabled methods like ExecContext(). An example can be seen in the gist here. Summary As a rule of thumb, you should explicitly set a MaxOpenConns value. This should be comfortably below any hard limits on the number of connections imposed by your database and infrastructure. In general, higher MaxOpenConns and MaxIdleConns values will lead to better performance. But the returns are diminishing, and you should be aware that having a too-large idle connection pool (with connections that are not re-used and eventually go bad) can actually lead to reduced performance. To mitigate the risk from point 2 above, you may want to set a relatively short ConnMaxLifetime. But you don't want this to be so short that leads to connections being killed and recreated unnecessarily often. MaxIdleConns should always be less than or equal to MaxOpenConns. For small-to-medium web applications I typically use the following settings as a starting point, and then optimize from there depending on the results of load-testing with real-life levels of throughput. db.SetMaxOpenConns(25) db.SetMaxIdleConns(25) db.SetConnMaxLifetime(5*time.Minute)