Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

• This is why I can’t have conversations using Twitter
  Antirez · Oct 29, 2014

  Yesterday Stripe engineers wrote a detailed report of why they had an issue with Redis. This is very appreciated. In the Hacker News thread I explained that because now we have diskless replication (http://antirez.com/news/81) now persistence is no longer mandatory for people having a master-slaves replicas set. This changes the design constraints: now that we can have diskless replicas synchronization, it is worth it to better support the Stripe (ex?) use case of replicas set with persistence turned down, in a more safe way. This is a work in progress effort. In the same post Stripe engineers said that they are going to switch to PostgreSQL for the use case where they have issues with Redis, which is a great database indeed, and many times if you can go with the SQL data model and an on-disk database, it is better to use that instead of Redis which is designed for when you really want to scale to a lot of complex operations per second. Stripe engineers also said that they measured the 99th percentile and it was better with PostgreSQL compared to Redis, so in a tweet @aphyr wrote: “Note that *synchronous* Postgres replication *between AZs* delivers lower 99th latencies than asynchronous Redis” And I replied: “It could be useful to look at average latency to better understand what is going on, since I believe the 99% percentile is very affected by the latency spikes that Redis can have running on EC2.” Which means, if you have also the average, you can tell if the 99th percentile is ruined (or not) by latency spikes, that many times can be solved. Usually it is as simple as that: if you have a very low average, but the 99th percentile is bad, likely it is not that Redis is running slow because, for example, operations performed are very time consuming or blocking, but instead a subset of queries are served slow because of the usual issues in EC2: fork time in certain instances, remote disks I/O, and so forth. Stuff that you can likely address, since for example, there are instance types without the fork latency issue. For half the Twitter IT community, my statement was to promote the average latency as the right metric over 99th percentiles: "averages are the worst possible metric for latency. No latency I've ever seen falls on a bell curve. Averages give nonsense." "You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here." “indeed; the problem is that averages are not robust in the presence of outliers” Ehm, who said that average is a good metric? I proposed it to *detect* if there are or not big outliers. So during what was supposed to be a normal exchange, I find after 10 minutes my Twitter completely full of people that tell me that I’m an idiot to endorse averages as The New Metric For Latency in the world. Once you get the first retweets, you got more and more. Even a notable builder of other NoSQL database finds the time to lecture me a few things via Twitter: I reply saying that clearly what I wrote was that if you have 99th + avg you have a better picture of the curve and can understand if the problem is the Redis spikes on EC2, but magically the original tweet gets removed, so my tweets are now more out of context. My three tweets: 1. “may point was, even if in the internet noise I'm not sure if it is still useful, that avg helps to understand why (…)” 2. “the 99% percentile is bad. If avg is very good but 99% percentile is bad, you can suspect a few very bad samples” 3. “this is useful with Redis, since with proper config sometimes you can improve the bad latency samples a lot.” Guess what? There is even somebody that isolated tweet #2 that was the continuation of “to understand why the 99% percentile is bad” (bad as in, is not providing good figures), and just read it out of context: “the 99% percentile is bad”. Once upon a time, people used to argue for days on usenet, but at least there was, most of the times, an argument against a new argument and so forth, with enough text and context to have a normal condition. This instead is just amplification of hate and engineering rules 101 together. 99th latency is the right metric and average is a poor one? Make sure to don’t talk about averages even in a context where it makes sense otherwise you get 10000 shitty replies. What to do with that? Now a good thing about me is that I’m not much affected by all this personally, but it is also clear that because I use Twitter for a matter of work, in order to inform people of what is happening with Redis, this is not a viable working environment. For example, latency: I care a lot about latency, so many efforts were done during the years in order to improve it (including diskless replication). We have monitoring as well in order to understand if and why there are latency spikes, Redis can provide you an human readable report of what is happening inside of it by monitoring different execution paths. After all this work, what you get instead is the wrong message retweeted one million times, which does not help. Most people will not follow the tweets to make an idea themselves, the reality is, at this point, rewritten: I said that average percentile is good and I don’t realize that you should look at the long tail. Next time I’ll talk about latency, for many people, I’ll be the one that has a few non clear ideas about it, so who knows what I’m talking about or what I’m doing? At the same time Twitter is RSS for humans, it is extremely useful to keep many people updated about what I love to do, which is, to work to my open source project that so far I tried to develop with care. So I’m trying to think about what a viable setup can be. Maybe I can just blog more, and use the Redis mailing list more, and use Twitter just to link stuff so that interested people can read, and interested people can argue and have real and useful discussions. I’ve a lot of things to do about Redis, for the users that have a good time with it, and a lot of things to do for the users that are experiencing problems. I feel like my time is best spent hacking instead of having non-conversations on Twitter. I love to argue, but this is just a futile exercise. Comments

• Diskless replication: a few design notes.
  Antirez · Oct 27, 2014

  Almost a month ago a number of people interested in Redis development met in London for the first Redis developers meeting. We identified together a number of features that are urgent (and are now listed in a Github issue here: https://github.com/antirez/redis/issues/2045), and among the identified issues, there was one that was mentioned multiple times in the course of the day: diskless replication. The feature is not exactly a new idea, it was proposed several times, especially by EC2 users that know that sometimes it is not trivial for a master to provide good performances during slaves synchronization. However there are a number of use cases where you don’t want to touch disks, even running on physical servers, and especially when Redis is used as a cache. Redis replication was, in short, forcing users to use disk even when they don’t need or want disk durability. When I returned back home I wanted to provide a quick feedback to the developers that attended the meeting, so the first thing I did was to focus on implementing the feature that seemed the most important and non-trivial among the list of identified issues. In the next weeks the attention will be moved to the Redis development process as well: the way issues are handled, how new ideas can be proposed to the Redis project, and so forth. Sorry for the delay about these other important things, for now what you can get is, some code at least ;-) Diskless replication provided a few design challenges. It looks trivial but it is not, so since I want to blog more, I thought about documenting how the internals of this feature work. I’m sure that a blog post may make the understanding and adoption of the new feature simpler. How replication used to work === Newer versions of Redis are able, when the connection with the master is lost, to reconnect with the master, and continue the replication process in an incremental way just fetching the differences accumulated so far. However when a slave is disconnected for a long time, or restarted, or it is a new slave, Redis requires it to perform what is called a “full resynchronization”. It is a trivial concept, and means: in order to setup this slave, let’s transfer *all* the master data set to the slave. It will flush away its old data, and reload the new data from scratch, making sure it is running an exact copy of master’s data. Once the slave is an exact copy of the master, successive changes are streamed as a normal Redis commands, in an incremental way, as the master data set itself gets modified because of write commands sent by clients. The problem was the way this initial “bulk transfer” needed for a full resynchronization was performed. Basically a child process was created by the master, in order to generate an RDB file. When the child was done with the RDB file generation, the file was sent to slaves, using non blocking I/O from the parent process. Finally when the transfer was complete, slaves could reload the RDB file and go online, receiving the incremental stream of new writes. However this means that from the master point of view, in order to perform a full sync, we need: 1) To write the RDB on disk. 2) To load back the RDB from disk in order to send it to slaves. “2” is not great but “1” is much worse. If AOF is active at the same time, for example, the AOF fsync() can be delayed a lot by the child writing to the disk as fast as possible. With the wrong setup, especially with non-local disks, but sometimes even because of a non perfect kernel parameters tuning, the disk pressure was cause of latency spikes that are hard to deal with. Partial resynchronizations introduced with Redis 2.8 mitigated this problem a bit, but from time to time you have to restart your slaves, or they go offline for too much time, so it is impossible to avoid full resynchronizations. At the same time, this process had a few advantages. The RDB saving code was reused for replication as well, making the replication code simpler. Moreover while the child was producing the RDB file, new slaves could attach, and put in a queue: when the RDB was ready, we could feed multiple slaves at the same time. All in all in many setups it works great and allows to synchronize a number of slaves at the same time. Also many users run with RDB persistence enabled in the master side, but not AOF, so anyway to persist on disk was happening from time to time. Most bare-metal users don’t have any latency at all while Redis is persisting, moreover disks, especially local disks, have easy to predict performances: once the child starts to save, you don’t really need to check for timeouts or if it is taking too much time, it will end eventually, and usually within a reasonable amount time. For this reasons, disk-backed replication is *still* the default replication strategy, and there are no plans to remove it so far, but now we have an alternative in order to serve the use cases where it was not great. So what is diskless replication? It is the idea that you can write directly from the child process to the slaves, via socket, without any intermediate step. Sockets are not disks === The obvious problem about diskless replication is that writing to disks is different than writing to sockets. To start the API is different, since the RDB code used to write to C FILE pointers, while to write to sockets is a matter of writing to file descriptors. Moreover disk writes don’t fail if not for hard I/O errors (for example if the disk is full), so when a write fails, you can consider the process aborted. For sockets it is different since writes can be delayed since the receiver is slow and the local kernel buffer is full. Another interesting issue is that there is to deal with timeouts: what about the receiving side to experience a failure so that it stops reading from us? Or just the TCP connection is dead but we don’t get resets, and so forth. We can’t take the child sending the RDB file to slaves active forever, there must be a way to detect timeouts. Fortunately modifying the RDB code to write to file descriptors was trivial, because for an entirely different problem (MIGRATE/RESTORE for Redis Cluster) the code was already using an abstraction called “rio” (redis I/O), that abstracts the serialization and deserialization of Redis values in RDB format, so you can write a value to the disk, or to an in memory buffer. What I did was to support a new “rio” target, called fdset: a set of file descriptors. This is because as I’ll write later, we need to write to multiple file descriptors at the same time. However this was not enough. One of the main design tradeoffs was to understand if the in memory RDB transfer would happen in one of the following two ways: 1) Way #1: produce a full RDB file in memory inside a buffer, than transfer it. 2) Way #2: directly write to slaves sockets, incrementally, as the RDB was created. Way #1 is a lot simpler since it is basically like the on-disk writing stuff, but in a kind of RAM disk. However the obvious risk is using too much memory. Way #2 is a bit more risky, because you have to transfer while the child producing the RDB file is active. However the essence of the feature was to target environments with slow disks perhaps, but *with fast networks*, without requiring too much additional memory, otherwise the feature risks to be useless. So Way #2 was selected. However if you stream an RDB file like this, there is a new problem to solve… how will the slave understand that EOF is reached? We don’t know, when we start the transfer, how big the transfer will be. With on-disk replication instead the size was known, so the transfer happened using just a Redis protocol “bulk” string, with prefixed length. Something like: $92384923423\r\n … data follows … I was too lazy to implement some complex chunked protocol to announce incremental blocks sizes, so went for a more brute force approach. The master generates an unguessable and unlikely to collide 160 bits random string, and sends something like that to the slave: $EOF:796f255829a040e80168f94c9fe7eda16b35e5df\r\n … data follows … 796f255829a040e80168f94c9fe7eda16b35e5df So basically this string, which is guaranteed (just because of infinitesimal probability) to never collide with anything inside the file, is used as the end of file mark. Trivial but works very well, and is simple. For timeouts, since it is a blocking write process (since we are in the context of the saving child process), I just used the SO_SNDTIMEO socket option. This way we are sure that we need to make progresses, otherwise the replication process is aborted. So for now there is no way to have an hard time limit for the child lifespan, and there are in theory pathological conditions where the slave would accept just one byte every timeout-1 seconds, to create a very slow transfer setup. Probably in the future the child will monitor the transfer rate, and if it drops under a reasonable figure, will exit with an error. Serving multiple slaves at the same time === Another goal of this implementation was to be able to serve multiple slaves at the same time. At first this looks impossible since once the RDB transfer starts, new slaves can’t attach, but need to wait for the current child to stop and a new one to start. However there is a very simple trick that covers a lot of use cases, which is, once the first slave want to replicate, we wait a few seconds for others to arrive as well. This covers the obvious case of a mass resync from multiple slaves for example. Because of this, the I/O code was designed in order to write to multiple file descriptors at the same time. Moreover in order to parallelize the transfer even if blocking I/O is used, the code tries to write a small amount of data to each fd in a loop, so that the kernel will send the packets in the background to multiple slaves at the same time. Probably the code itself is pretty easy to understand: while(len) { size_t count = len int broken = 0; for (j = 0; j io.fdset.numfds; j++) { … error checking removed … /* Make sure to write 'count' bytes to the socket regardless * of short writes. */ size_t nwritten = 0; while(nwritten != count) { retval = write(r->io.fdset.fds[j],p+nwritten,count-nwritten); if (retval … error checkign removed … } nwritten += retval; } } p += count; len -= count; r->io.fdset.pos += count; … more error checking removed … } Note that writes are bufferized by the rio.c write target, since we want to write only when a given amount of data is available, otherwise we risk to send TCP packets with 5 bytes of data inside. Handling partial failures === Handling multiple slaves is not just writing to multiple FDs, which is quite simple. A big part of the story is actually to handle a few slaves failing without requiring to block the process for all the other slaves. File descriptors in error are marked with the related error code, and no attempt is made to write to them again. Also the code detects if all the FDs are in error, and abort the process at all. However when the RDB writing is terminated, the child needs to report what are the slaves that received the RDB and can continue the replication process. For this task, a unix pipe is used between the processes. The child returns an array of slave IDs and associated error state, so that the parent can do a decent job at logging errors as well. How this changes Redis is a more deep way I thought === Diskless replication finally allows for a totally disk-free experience in Redis master-slaves sets. This means we need to support this use case better. Currently replication is dangerous to run with persistence disabled, since I thought there was not a case for turning off persistence when anyway replication was going to trigger it. But now this changed… and as a result, there are already plans to support better replication in a non-disk backed environment. The same will be applied to Redis Cluster as well… which is also a good candidate for diskless operations, especially for caching use cases, where replicas can do a good job to provide data redundancy, but where it may not be too critical if crash-restart of multiple instances cause data loss of a subset of hash slots in the cluster. ETA === The code is already available in beta here: https://github.com/antirez/redis/commits/memsync It will be merged into unstable in the next days, but the plan is to wait a bit for feedbacks and bug reports, and later merge into 3.0 and 2.8 as well. The feature is very useful and it has little interactions with the rest of the Redis core when it is turned off. The plan is to just back port it everywhere and release it as “experimental” for some time. Comments

• Making and using HTTP Middleware in Go
  Alex Edwards · Oct 21, 2014

  When you're building a web application, there's probably some shared functionality that you want to run for many (or even all) HTTP requests. You might want to log every request, gzip every response, or check that a user is authenticated before sending them any content. One way of organizing this shared functionality is to set it up as middleware — essentially a self-contained block of code that independently acts on a request, before or after your normal application handlers. In this post I'll explain how to create and use your own middleware, how to chain multiple middlewares together, and finish up with some practical real-world examples and tips. The standard pattern Before we talk about middleware, take a moment to consider the structure of the messageHandler function in the following code: func messageHandler(message string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(message)) }) } func main() { mux := http.NewServeMux() mux.Handle("GET /", messageHandler("Hello world!")) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } In this code we put our messageHandler logic — which is just a call to w.Write() — in an anonymous function which 'closes over' the message variable to form a closure. We then convert the closure to an http.Handler with the http.HandlerFunc() adapter, and then return it. Note: If this pattern is confusing or unfamiliar to you, before you go any further I recommend reading this primer which explains it in more detail. We can use this same general pattern to help us create a middleware function. Instead of passing a string into the closure (like above), you can pass another http.Handler as a parameter, and then transfer control to this handler by calling its ServeHTTP() method. Like so: func exampleMiddleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Your middleware logic goes here... next.ServeHTTP(w, r) }) } Essentially, the exampleMiddleware function accepts a next handler as a parameter, and it returns a closure which is also a handler. When this closure is executed, any code in the closure will be run and then the next handler will be called. Using middleware on specific routes If any of that sounds confusing, don't worry! In practice you can copy and paste that code pattern if you need to, and beyond that, making and using middleware is actually fairly straightforward. Let's start by looking at an example of how to use middleware on specific routes in your application. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } There is quite a lot going on in this code, so let's take a moment to unpack some of it: We've created a middleware function called middlewareOne, which uses the standard pattern that we talked about above. The middleware logs a message, calls the next handler, and then logs another message. We've made two normal handler functions, fooHandler and barHandler, which both log a message and send a 200 OK response. In the route mux.Handle("GET /foo", http.HandlerFunc(fooHandler)), we use the http.HandlerFunc() function to convert fooHandler to a http.Handler, and use it as normal with no middleware. In the route mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))), we use the http.HandlerFunc() function to convert barHandler to a http.Handler, and then pass it to the middlewareOne function as the next argument. Or in simpler terms — we wrap barHandler with the middlewareOne middleware function. If you run this application and make a request to http://localhost:3000/foo, you should see some log output containing only the message from fooHandler: $ go run main.go 2025/07/05 19:00:56 listening on :3000... 2025/07/05 19:01:09 /foo executing fooHandler In contrast, if you make a request to http://localhost:3000/bar, you should also see the log messages from middlewareOne, demonstrating that the middleware is successfully being used on that route. ... 2025/07/05 19:02:43 /bar executing middlewareOne 2025/07/05 19:02:43 /bar executing barHandler 2025/07/05 19:02:43 /bar executing middlewareOne again This log output also nicely illustrates the flow of control through the application code. We can see that any code in middlewareOne which comes before next.ServeHTTP(w, r) runs before barHandler is executed — and any code which comes after next.ServeHTTP(w, r) runs after barHandler has returned. So the flow of control through the application for the GET /bar route looks like this: http.ServeMux → middlewareOne → barHandler → middlewareOne → http.ServeMux Using middleware on all routes In the previous example, we used our middleware to wrap a specific handler in a specific route. But if you want your middleware to act on all routes, you can wrap http.ServeMux itself so that the flow of control looks like this: middlewareOne → http.ServeMux → fooHandler/barHandler → http.ServeMux → middlewareOne This works because Go's http.ServeMux implements the http.Handler interface — it has the necessary ServeHTTP() method. And as a result, we can directly pass an http.ServeMux into a middleware function as the next parameter. Let's update our example code to do this: main.go package main ... func main() { mux := http.NewServeMux() // We don't use any middleware on the individual routes. mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", http.HandlerFunc(fooHandler)) log.Println("listening on :3000...") // Wrap the http.ServeMux with the middlewareOne function. err := http.ListenAndServe(":3000", middlewareOne(mux)) log.Fatal(err) } And if you run the application and make the same requests to /foo and /bar again, you should see from the log output that middlewareOne is now being used on all routes. $ go run main.go 2025/07/05 19:04:48 listening on :3000... 2025/07/05 19:04:54 /foo executing middlewareOne 2025/07/05 19:04:54 /foo executing fooHandler 2025/07/05 19:04:54 /foo executing middlewareOne again 2025/07/05 19:04:58 /bar executing middlewareOne 2025/07/05 19:04:58 /bar executing fooHandler 2025/07/05 19:04:58 /bar executing middlewareOne again Chaining middleware Because the standard middleware function pattern accepts a http.Handler as a parameter, and it returns a http.Handler, that makes it possible to easily create arbitrarily long chains of middleware. Put simply, one middleware function can wrap another middleware function. To illustrate this, let's add some more middleware functions to our example and chain them together. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func middlewareTwo(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareTwo") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareTwo again") }) } func middlewareThree(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareThree") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareThree again") }) } func middlewareFour(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFour") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFour again") }) } func middlewareFive(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFive") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFive again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() // Apply middlewareThree and middlewareFour to GET /foo mux.Handle("GET /foo", middlewareThree(middlewareFour(http.HandlerFunc(fooHandler)))) // Apply middlewareFour and middlewareFive to GET /bar mux.Handle("GET /bar", middlewareFour(middlewareFive(http.HandlerFunc(barHandler)))) log.Println("listening on :3000...") // Apply middlewareOne and middlewareTwo to the entire http.ServeMux err := http.ListenAndServe(":3000", middlewareOne(middlewareTwo(mux))) log.Fatal(err) } In this code we are now wrapping the http.ServeMux with middlewares One and Two, on the GET /foo route we're using middlewares Three and Four, and on the GET /bar route we're using middlewares Four and Five. Again, if you run the application and make the same requests to /foo and /bar you should now see log output that demonstrates the middleware functions being chained together and the flow of control through them. Like so: 2025/07/05 19:06:25 /foo executing middlewareOne 2025/07/05 19:06:25 /foo executing middlewareTwo 2025/07/05 19:06:25 /foo executing middlewareThree 2025/07/05 19:06:25 /foo executing middlewareFour 2025/07/05 19:06:25 /foo executing fooHandler 2025/07/05 19:06:25 /foo executing middlewareFour again 2025/07/05 19:06:25 /foo executing middlewareThree again 2025/07/05 19:06:25 /foo executing middlewareTwo again 2025/07/05 19:06:25 /foo executing middlewareOne again 2025/07/05 19:06:43 /bar executing middlewareOne 2025/07/05 19:06:43 /bar executing middlewareTwo 2025/07/05 19:06:43 /bar executing middlewareFour 2025/07/05 19:06:43 /bar executing middlewareFive 2025/07/05 19:06:43 /bar executing barHandler 2025/07/05 19:06:43 /bar executing middlewareFive again 2025/07/05 19:06:43 /bar executing middlewareFour again 2025/07/05 19:06:43 /bar executing middlewareTwo again 2025/07/05 19:06:43 /bar executing middlewareOne again Early returns One of the useful things about middleware is that you can use it as a 'guard' to prevent downstream middleware and handlers in the chain from being executed unless certain conditions are met. For example, you can use middleware to check if a user is authenticated, or that a request contains the correct Content-Type header, or that the client hasn't hit a rate-limiter ceiling before doing any further processing. For example, you could create a middleware function to ensure that the request Content-Type header exactly matches application/json by returning early from the middleware, before calling next.ServeHTTP(w, r). Like this: func requireJSON(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { contentType := r.Header.Get("Content-Type") // If the content type is not application/json, send an error message and // return from the middleware. By returning before next.ServeHTTP(w, r) // is called, it means that the next handler in the chain is never executed. if contentType != "application/json" { http.Error(w, "Content-Type header must be application/json", http.StatusUnsupportedMediaType) return } // Otherwise, if the content type is application/json, call the next handler // in the chain as normal. next.ServeHTTP(w, r) }) } A more realistic example Now that we've covered the theory, let's look at a more practical example to give you a taste for using middleware in a real application. In this code, we'll create two middleware functions that we want to use on all routes: A serverHeader middleware that adds the Server: Go header to HTTP responses. A logRequest middleware that uses the log/slog package to log the details of the current request. And we'll also create a GET /admin route that is guarded by a requireBasicAuthentication middleware function, which requires the client to authenticate via HTTP basic authentication. This is another example where we will use the 'early return' pattern that we just talked about. main.go package main import ( "log/slog" "net/http" "os" ) func serverHeader(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "Go") next.ServeHTTP(w, r) }) } func logRequest(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { var ( ip = r.RemoteAddr method = r.Method url = r.URL.String() proto = r.Proto ) userAttrs := slog.Group("user", "ip", ip) requestAttrs := slog.Group("request", "method", method, "url", url, "proto", proto) slog.Info("request received", userAttrs, requestAttrs) next.ServeHTTP(w, r) }) } func requireBasicAuthentication(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { validUsername := "admin" validPassword := "secret" username, password, ok := r.BasicAuth() if !ok || username != validUsername || password != validPassword { w.Header().Set("WWW-Authenticate", `Basic realm="protected"`) http.Error(w, "401 Unauthorized", http.StatusUnauthorized) return } next.ServeHTTP(w, r) }) } func home(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Welcome to the home page!")) } func admin(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Admin dashboard - you are authenticated!")) } func main() { mux := http.NewServeMux() mux.HandleFunc("GET /{$}", home) // Use the requireBasicAuthentication middleware on the GET /admin route only. mux.Handle("GET /admin", requireBasicAuthentication(http.HandlerFunc(admin))) slog.Info("listening on :3000...") // Use the serverHeader and logRequest middleware on all routes. err := http.ListenAndServe(":3000", serverHeader(logRequest(mux))) if err != nil { slog.Error(err.Error()) os.Exit(1) } } Please note: I've made the requireBasicAuthentication code deliberately simple for this example, and while it works correctly, there is the tiny but theoretical risk of it being vulnerable to a timing attack. If this is something you're concerned about, I've written about how to mitigate that risk in this blog post. Go ahead and run this application, then open a second terminal window and use curl to make a request to GET /, and unauthenticated and authenticated requests to GET /admin. The responses should look similar to this: $ curl -i localhost:3000 HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:19:24 GMT Content-Length: 25 Content-Type: text/plain; charset=utf-8 Welcome to the home page! $ curl -i localhost:3000/admin HTTP/1.1 401 Unauthorized Content-Type: text/plain; charset=utf-8 Server: Go Www-Authenticate: Basic realm="protected" X-Content-Type-Options: nosniff Date: Sat, 05 Jul 2025 12:19:32 GMT Content-Length: 17 401 Unauthorized $ curl -i -u admin:secret localhost:3000/admin HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:26:53 GMT Content-Length: 40 Content-Type: text/plain; charset=utf-8 Admin dashboard - you are authenticated! We can see from these responses that our serverHeader middleware is setting the Server: Go header on all responses, and that the requireBasicAuthentication middleware is correctly protecting our GET /admin route. And if you head back to your original terminal window, you should see the corresponding log entries courtesy of the logRequest middleware. Similar to this: $ go run main.go 2025/07/05 14:18:44 INFO listening on :3000... 2025/07/05 14:19:24 INFO request received user.ip=127.0.0.1:41966 request.method=GET request.url=/ request.proto=HTTP/1.1 2025/07/05 14:19:32 INFO request received user.ip=127.0.0.1:59244 request.method=GET request.url=/admin request.proto=HTTP/1.1 2025/07/05 14:26:53 INFO request received user.ip=127.0.0.1:57670 request.method=GET request.url=/admin request.proto=HTTP/1.1 Managing and organizing middleware Lastly, a couple of tips. If you have an application with lots of routes and lots of middleware, you can potentially end up with very long route declarations and a lot of duplication in those declarations, which isn't ideal for easy-reading or maintainability. One of the tools that I've used for a long time to help manage this is justinas/alice, which is a small package that makes it easy to create reusable chains of handlers. At it's most basic, it let's you rewrite code that looks like this: mux.Handle("GET /foo", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(fooHandler))))) mux.Handle("GET /bar", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(barHandler))))) As this: stdChain := alice.New(middlewareOne, middlewareTwo, middlewareThree) mux.Handle("/foo", stdChain.Then(fooHandler)) mux.Handle("/bar", stdChain.Then(barHandler)) More recently, I've been rolling my own custom chain type instead of using justinas/alice, or wrapping http.ServeMux so that it supports 'groups' of routes which use specific middleware. If you're interested in this, I've written a more about it in the post "Organize your Go middleware without dependencies", and it's probably a good follow-on read from this post.

• A few arguments about Redis Sentinel properties and fail scenarios.
  Antirez · Oct 21, 2014

  Yesterday distributed systems expert Aphyr, posted a tweet about a Redis Sentinel issue experienced by an unknown company (that wishes to remain anonymous): “OH on Redis Sentinel "They kill -9'd the master, which caused a split brain..." “then the old master popped up with no data and replicated the lack of data to all the other nodes. Literally had to restore from backups." OMG we have some nasty bug I thought. However I tried to get more information from Kyle, and he replied that the users actually disabled disk persistence at all from the master process. Yep: the master was configured on purpose to restart with a wiped data set. Guess what? A Twitter drama immediately started. People were deeply worried for Redis users. Poor Redis users! Always in danger. However while to be very worried is a trait of sure wisdom, I want to take the other path: providing some more information. Moreover this blog post is interesting to write since actually Kyle, while reporting the problem with little context, a few tweets later was able to, IMHO, isolate what is the true aspect that I believe could be improved in Redis Sentinel, which is not related to the described incident, and is in my TODO list for a long time now. But before, let’s check a bit more closely the behavior of Redis / Sentinel about the drama-incident. Welcome to the crash recovery system model === Most real world distributed systems must be designed to be resilient to the fact that processes can restart at random. Note that this is very different from the problem of being partitioned away, which is, the inability to exchange messages with other processes. It is, instead, a matter of losing state. To be more accurate about this problem, we could say that if a distributed algorithm is designed so that a process must guarantee to preserve the state after a restart, and fails to do this, it is technically experiencing a bizantine failure: the state is corrupted, and the process is no longer reliable. Now in a distributed system composed of Redis instances, and Redis Sentinel instances, it is fundamental that rebooted instances are able to restart with the old data set. Starting with a wiped data set is a byzantine failure, and Redis Sentinel is not able to recover from this problem. But let’s do a step backward. Actually Redis Sentinel may not be directly involved in an incident like that. The typical example is what happens if a misconfigured master restarts fast enough so that no failure is detected at all by Sentinels. 1. Node A is the master. 2. Node A is restarted, with persistence disabled. 3. Sentinels (may) see that Node A is not reachable… but not enough to reach the configured timeout. 4. Node A is available again, except it restarted with a totally empty data set. 5. All the slave nodes B, C, D, ... will happily synchronize an empty data set form it. Everything wiped from the master, as per configuration, after all. And everything wiped from the slaves, that are replicating from what is believed to be the current source of truth for the data set. Let’s remove Sentinel from the equation, which is, point “3” of the above time line, since Sentinel did not acted at all in the example scenario. This is what you get. You have a Redis master replicating with N slaves. The master is restarted, configured to start with a fresh (empty) data set. Salves replicate again from it (an empty data set). I think this is not a big news for Redis users, this is how Redis replication works: slaves will always try to be the exact copy of their masters. However let’s consider alternative models. For example Redis instances could have a Node ID which is persisted in the RDB / AOF file. Every time the node restarts, it loads its Node ID. If the Node ID is wrong, slaves wont replicate from the master at all. Much safer right? Only marginally, actually. The master could have a different misconfiguration, so after a restart, it could reload a data set which is weeks old since snapshotting failed for some reason. So after a bad restart, we still have the right Node ID, but the dataset is so old to be basically, the same as being wiped more or less, just more subtle do detect. However at the cost of making things only marginally more secure we now have a system that may be more complex to operate, and slaves that are in danger of not replicating from the master since the ID does not match, because of operational errors similar to disabling persistence, except, a lot less obvious than that. So, let’s change topic, and see a failure mode where Sentinel is *actually* involved, and that can be improved. Not all replicas are the same === Technically Redis Sentinel offers a very limited set of simple to understand guarantees. 1) All the Sentinels will agree about the configuration as soon as they can communicate. Actually each sub-partition will always agree. 2) Sentinels can’t start a failover without an authorization from the majority of Sentinel processes. 3) Failovers are strictly ordered: if a failover happened later in time, it has a greater configuration “number” (config epoch in Sentinel slang), that will always win over older configurations. 4) Eventually the Redis instances are configured to map with the winning logical configuration (the one with the greater config epoch). This means that the dataset semantics is “last failover wins”. However the missing information here is, during a failover, what slave is picked to replace the master? This is, all in all, a fundamental property. For example if Redis Sentinel fails by picking a wiped slave (that just restarted with a wrong configuration), *that* is a problem with Sentinel. Sentinel should make sure that, even within the limits of the fact that Redis is an asynchronously replicated system, it will try to make the best interest of the user by picking the best slave around, and refusing to failover at all if there is no viable slave reachable. This is a place where improvements are possible, and this is what happens today to select a slave when the master is failing: 1) If a slaves was restarted, and never was connected with the master after the restart, performing a successful synchronization (data transfer), it is skipped. 2) If the slave is disconnected from its master for more than 10 times the configured timeout (the time a master should be not reachable for the set of Sentinels to detect a master as failing), it is considered to be non elegible. 3) Out of the remaining slaves, Sentinel picks the one with the best “replication offset”. The replication offset is a number that Redis master-slave replication uses to take a count of the amount of bytes sent via the replication channel. it is useful in many ways, not just for failovers. For example in partial resynchronizations after a net split, slaves will ask the master, give me data starting from offset X, which is the last byte I received, and so forth. However this replication number has two issues when used in the context of picking the best slave to promote. 1) It is reset after restarts. This sounds harmless at first, since we want to pick slaves with the higher number, and anyway, after a restart if a slave can’t connect, it is skipped. However it is not harmless at all, read more. 2) It is just a number: it does not imply that a Redis slave replicated from *a given* master. Also note that when a slave is promoted to master, it inherits the master’s replication offset. So modulo restarts, the number keeps increasing. Why “1” and/or “2” are suboptimal choices and can be improved? Imagine this setup. We have nodes A B C D E. D is the current master, and is partitioned away with E in a minority partition. E still replicates from D, everything is fine from their POV. However in the majority partition, A B C can exchange messages, and A is elected master. Later A restarts, resetting its offset. B and C replicate from it, starting again with lower offsets. After some time A fails, and, at the same time, E rejoins the majority partition. E has a data set that is less updated compared to the B and C data set, however its replication offset is higher. Not just that, actually E can claim it was recently connected to its master. To improve upon this is easy. Each Redis instance has a “runid”, an unique ID that changes for each new Redis run. This is useful in partial resynchronizations in order to avoid getting an incremental stream from a wrong master. Slaves should publish what is the last master run id they replicated successful from, and Sentinel failover should make sure to only pick slaves that replicated from the master they are actually failing over. Once you tight the replication offset to a given runid, what you get is an *absolute* meter of how much updated a slave is. If two slaves are available, and both can claim continuity with the old master, the one with the higher replication offset is guaranteed to be the best pick. However this also creates availability concerns in all the cases where data is not very important but availability is. For example if when A crashes, only E becomes available, even if it used to replicate from D, it is still better than nothing. I would say that when you need an highly available cache and consistency is not a big issue, to use a Redis cluster ala-memcached (client side consistent hashing among N masters) is the way to go. Note that even without checking the runid, to make the replication offsets durable after a restart, already improves the behavior considerably. In the above example E would be picked only if when isolated in a minority partition with the slave, received more writes than the other slaves in the majority side. TLDR: we have to fix this. It is not related to restarting masters without a dataset, but is useful to have a more correct implementation. However this will limit only a class of very-hard-to-trigger issues. This is in my TODO list for some time now, and congrats to Aphyr for identifying a real implementation issue in a matter of a few tweets exchanged. About the failure reported by Aphyr from the unknown company, I don’t think it is currently viable to try to protect against serious misconfigurations, however it is a clear sign that we need better Sentinel docs which are more incremental compared to the ones we have now, that try to describe how the system works. A wiser approach could be to start with a common sane configuration, and “don’t do” list like, don’t turn persistence off, unless you are ok with wiped instances. Comments

• Redis cluster, no longer vaporware.
  Antirez · Oct 09, 2014

  The first commit I can find in my git history about Redis Cluster is dated March 29 2011, but it is a “copy and commit” merge: the history of the cluster branch was destroyed since it was a total mess of work-in-progress commits, just to shape the initial idea of API and interactions with the rest of the system. Basically it is a roughly 4 years old project. This is about two thirds the whole history of the Redis project. Yet, it is only today, that I’m releasing a Release Candidate, the first one, of Redis 3.0.0, which is the first version with Cluster support. An erratic run — To understand why it took so long is straightforward: I started the cluster project with a lot of rush, in a moment where it looked like Redis was going to be totally useless without an automatic way to scale. It was not the right moment to start the Cluster project, simply because Redis itself was too immature, so we didn't yet have a solid “single instance” story to tell. While I did the error of starting a project with the wrong timing, at least I didn’t fell in the trap of ignoring the requests arriving from the community, so the project was stopped and stopped an infinite number of times in order to provide more bandwidth to other fundamental features. Persistence, replication, latency, introspection, received a lot more care than cluster, simply because they were more important for the user base. Another limit of the project was that, when I started it, I had no clue whatsoever about distributed programming. I did a first design that was horrible, and managed to capture well only what were the “products” requirement: low latency, linear scalability and small overhead for small clusters. However all the details were wrong, and it was far more complex than it had to be, the algorithms used were unsafe, and so forth. While I was doing small progresses I started to study the basics of distributed programming, redesigned Redis Cluster, and applied the same ideas to the new version of Sentinel. The distributed programming algorithms used by both systems are still primitive since they are asynchronous replicated, eventually consistent systems, so I had no need to deal with consensus and other non trivial problems. However even when you are addressing a simple problem, compared to writing a CP store at least, you need to understand what you are doing otherwise the resulting system can be totally wrong. Despite all this problems, I continued to work at the project, trying to fix it, fix the implementation, and bring it to maturity, because there was this simple fact, like an elephant into a small room, permeating all the Redis Community, which is: people were doing again and again, with their efforts, and many times in a totally broken way, two things: 1) Sharding the dataset among N nodes. 2) A responsive failover procedure in order to survive certain failures. Problem “2” was so bad that at some point I decided to start the Redis Sentinel project before Cluster was finished in order to provide an HA system ASAP, and one that was more suitable than Redis Cluster for the majority of use cases that required just “2” and not “1”. Finally I’m starting to see the first real-world result of this efforts, and now we have a release candidate that is the fundamental milestone required to get adoption, fix the remaining bugs, and improve the system in a more incremental way. What it actually does? — Redis Cluster is basically a data sharding strategy, with the ability to reshard keys from one node to another while the cluster is running, together with a failover procedure that makes sure the system is able to survive certain kinds of failures. From the point of view of distributed databases, Redis Cluster provides a limited amount of availability during partitions, and a weak form of consistency. Basically it is neither a CP nor an AP system. In other words, Redis Cluster does not achieve the theoretical limits of what is possible with distributed systems, in order to gain certain real world properties. The consistency model is the famous “eventual consistency” model. Basically if nodes get desynchronized because of partitions, it is guaranteed that when the partition heals, all the nodes serving a given key will agree about its value. However the merge strategy is “last failover wins”, so writes received during network partitions can be lost. A common example is what happens if a master is partitioned into a minority partition with clients trying to write to it. If when the partition heals, in the majority side of the partition a slave was promoted to replace this master, the writes received by the old master are lost. This in turn means that Redis Cluster does not have to take meta data in the data structures in order to attempt a value merge, and that the fancy commands and data structures supported by Redis are also supported by Redis Cluster. So no additional memory overhead, no API limits, no limits in the amount of elements a value can contain, but less safety during partitions. It is trivial to understand that in a system designed like Redis Cluster is, nodes diverging are not good, so the system tries to mitigate its shortcomings by trying to limit the probability of two nodes diverging (and the amount of divergence). This is achieved in a few ways: 1) The minority side of a partition becomes not available. 2) The replication is designed so that usually the reply to the client, and the replication stream to slaves, is sent at the same time. 3) When multiple slaves are available to failover a master, the system will try to pick the one that appears to be less diverging from the failed master. This strategies don’t change the theoretical properties of the system, but add some more real-world protection for the common Redis Clusters failure modes. For the Redis API and use case, I believe this design makes sense, but in the past many disagreed. However my opinion is that each designer is free to design a system as she or he wishes, there is just one rule: say the truth, so Redis Cluster documents its limits and failure modes clearly in the official documentation. It’s the user, and the use case at hand, that will make a system useful or not. My feeling is that after six years users continued to use Redis even without any clustering support at all, because the use case made this possible, and Redis offers certain specific features and performances that made it very suitable to address certain problems. My hope is that Redis Cluster will improve the life of many of those users. The road ahead — Finally we have a minimum viable product to ship, which is stable enough for users to seriously start testing and in certain cases adopt it already. The more adoption, the more we improve it. I know this from Redis and Sentinel: now there is the incremental process that moves a software forward from usable to mature. Listening to users, fixing bugs, covering more code in tests, … At the same time, I’m starting to think at the next version of Redis Cluster, improving v1 with many useful things that was not possible to add right now, like multi data center support, more write safety in the minority partition using commands replay, automatic nodes balancing (now there is to reshard manually if certain nodes are too empty and other too full), and many more things. Moreover, I believe Redis Cluster could benefit from a special execution mode specifically designed for caching, where nodes accept writes to hash slots they are not in charge for, in order to stay available in a minority partition. There is always time to improve and fix our implementation and designs, but focusing too much on how we would like some software to be, has the risk of putting it in the vaporware category for far longer than needed. It’s time to let it go. Enjoy Redis Cluster! Redis Cluster RC1 is available both as '3.0.0-rc1' tag at Github, or as a tarball in the Redis.io download page at http://redis.io/download Comments

• Queues and databases
  Antirez · Jul 14, 2014

  Queues are an incredibly useful tool in modern computing, they are often used in order to perform some possibly slow computation at a latter time in web applications. Basically queues allow to split a computation in two times, the time the computation is scheduled, and the time the computation is executed. A “producer”, will put a task to be executed into a queue, and a “consumer” or “worker” will get tasks from the queue to execute them. For example once a new user completes the registration process in a web application, the web application will add a new task to the queue in order to send an email with the activation link. The actual process of sending an email, that may require retrying if there are transient network failures or other errors, is up to the worker. Technically speaking we can think at queues as a form of inter-process messaging primitive, where the receiving process needs to acknowledge the reception of the message. Messages can not be fire-and-forget, since the queue needs to understand if the message can be removed from the queue, so some form of acknowledgement is strictly required. When receiving a message triggers the execution of a task, like it happens in the kind of queues we are talking about, the moment the message reception is acknowledged changes the semantic of the queue. When the worker process acknowledges the reception of the message *before* processing the message, if the worker fails the message can be lost before the task is performed at all. If the acknowledge is sent only *after* the message gets processed, if the worker fails or because of network partitions the queue may re-deliver the message again. This happens whatever the queue consistency properties are, so, even if the queue is modeled using a system providing strong consistency, the indetermination still holds true: * If messages are acknowledged before processing, the queue will have an at-most-once delivery property. This means messages can be processed zero or one time. * If messages are acknowledged after processing, the queue will have an at-least-once delivery property. This means messages can be processed from 1 to infinite number of times. While both of this cases are not perfect, in the real world the second behavior is often preferred, since it is usually much simpler to cope with the case of multiple delivery of the message (triggering multiple executions of the task) than a system that from time to time does not execute a given task at all. An example of at-least-once delivery system is Amazon SQS (Simple Queue Service). There is also a fundamental reason why at-least-once delivery systems are to be preferred, that has to do with distributed systems: the other semantics (at-most-once delivery) requires the queue to be strongly consistent: once the message is acknowledged no other worker must be able to acknowledge the same message, which is a strong property. Once we move our focus to at-least-once delivery systems, we may notice that to model the queue with a CP system is a waste, and also a disadvantage: * Anyway, we can’t guarantee more than at-least-once delivery. * Our queue lose the ability to work into a minority side of a network partition. * Because of the consistency requirements the queue needs agreement, so we are burning performances and adding latency without any good reason. Since messages may be delivered multiple times, what we want conceptually is a commutative data structure and an eventually consistent system. Messages can be stored into a set data structure replicated into N nodes, with the merge function being the union among the sets. Acknowledges, received by workers after execution of messages, are also conceptually elements of the set, marking a given element as processed. This is a trivial example which is not particularly practical for a real world system, but shows how a given kind of queue is well modeled by a given set of properties of a distributed system. Practically speaking there are other useful things our queue may try to provide: * Guaranteed delivery to a single worker at least for a window of time: while multiple delivery is allowed, we want to avoid it as much as possible. * Best-effort checks to avoid to re-delivery a message after a timeout if the message was already processed. Again, we can’t guarantee this property, but we may try hard to reduce re-issuing a message which was actually already processed. * Enough internal state to handle, during normal operations, messages as a FIFO, so that messages arriving first are processed first. * Auto cleanup of the internal data structures. On top of this we need to retain messages during network partitions, so that conceptually (even if practically we could use different data structures) the set of messages to deliver are the union of all the messages of all the nodes. Unfortunately while many Redis based queues implementations exist, no one try to use N Redis independent nodes and the offered primitives as a building block for a distributed system with such characteristics. Using Redis data structures and performances, and algorithms providing certain useful guarantees, may provide a queue system which is very practical to use, easy to administer and scale, while providing excellent performances (messages / second) per node. Because I find the topic is interesting and this is an excellent use case for Redis, I’m very slowly working at a design for such a Redis based queue system. I hope to show something during the next weeks, time permitting. Comments

• A proposal for more reliable locks using Redis
  Antirez · May 16, 2014

  ----------------- UPDATE: The algorithm is now described in the Redis documentation here => http://redis.io/topics/distlock. The article is left here in its older version, the updates will go into the Redis documentation instead. ----------------- Many people use Redis to implement distributed locks. Many believe that this is a great use case, and that Redis worked great to solve an otherwise hard to solve problem. Others believe that this is totally broken, unsafe, and wrong use case for Redis. Both are right, basically. Distributed locks are not trivial if we want them to be safe, and at the same time we demand high availability, so that Redis nodes can go down and still clients are able to acquire and release locks. At the same time a fast lock manager can solve tons of problems which are otherwise hard to solve in practice, and sometimes even a far from perfect solution is better than a very slow solution. Can we have a fast and reliable system at the same time based on Redis? This blog post is an exploration in this area. I’ll try to describe a proposal for a simple algorithm to use N Redis instances for distributed and reliable locks, in the hope that the community may help me analyze and comment the algorithm to see if this is a valid candidate. # What we really want? Talking about a distributed system without stating the safety and liveness properties we want is mostly useless, because only when those two requirements are specified it is possible to check if a design is correct, and for people to analyze and find bugs in the design. We are going to model our design with just three properties, that are what I believe the minimum guarantees you need to use distributed locks in an effective way. 1) Safety property: Mutual exclusion. At any given moment, only one client can hold a lock. 2) Liveness property A: Deadlocks free. Eventually it is always possible to acquire a lock, even if the client that locked a resource crashed or gets partitioned. 3) Liveness property B: Fault tolerance. As long as the majority of Redis nodes are up, clients are able to acquire and release locks. # Distributed locks, the naive way. To understand what we want to improve, let’s analyze the current state of affairs. The simple way to use Redis to lock a resource is to create a key into an instance. The key is usually created with a limited time to live, using Redis expires feature, so that eventually it gets released one way or the other (property 2 in our list). When the client needs to release the resource, it deletes the key. Superficially this works well, but there is a problem: this is a single point of failure in our architecture. What happens if the Redis master goes down? Well, let’s add a slave! And use it if the master is unavailable. This is unfortunately not viable. By doing so we can’t implement our safety property of the mutual exclusion, because Redis replication is asynchronous. This is an obvious race condition with this model: 1) Client A acquires the lock into the master. 2) The master crashes before the write to the key is transmitted to the slave. 3) The slave gets promoted to master. 4) Client B acquires the lock to the same resource A already holds a lock for. Sometimes it is perfectly fine that under special circumstances, like during a failure, multiple clients can hold the lock at the same time. If this is the case, stop reading and enjoy your replication based solution. Otherwise keep reading for a hopefully safer way to implement it. # First, let’s do it correctly with one instance. Before to try to overcome the limitation of the single instance setup described above, let’s check how to do it correctly in this simple case, since this is actually a viable solution in applications where a race condition from time to time is acceptable, and because locking into a single instance is the foundation we’ll use for the distributed algorithm described here. To acquire the lock, the way to go is the following: SET resource_name my_random_value NX PX 30000 The command will set the key only if it does not already exist (NX option), with an expire of 30000 milliseconds (PX option). The key is set to a value “my_random_value”. This value requires to be unique across all the clients and all the locks requests. Basically the random value is used in order to release the lock in a safe way, with a script that tells Redis: remove the key only if exists and the value stored at the key is exactly the one I expect to be. This is accomplished by the following Lua script: if redis.call("get",KEYS[1]) == ARGV[1] then return redis.call("del",KEYS[1]) else return 0 end This is important in order to avoid removing a lock that was created by another client. For example a client may acquire the lock, get blocked into some operation for longer than the lock validity time (the time at which the key will expire), and later remove the lock, that was already acquired by some other client. Using just DEL is not safe as a client may remove the lock of another client. With the above script instead every lock is “signed” with a random string, so the lock will be removed only if it is still the one that was set by the client trying to remove it. What this random string should be? I assume it’s 20 bytes from /dev/urandom, but you can find cheaper ways to make it unique enough for your tasks. For example a safe pick is to seed RC4 with /dev/urandom, and generate a pseudo random stream from that. A simpler solution is to use a combination of unix time with microseconds resolution, concatenating it with a client ID, it is not as safe, but probably up to the task in most environments. The time we use as the key time to live, is called the “lock validity time”. It is both the auto release time, and the time the client has in order to perform the operation required before another client may be able to acquire the lock again, without technically violating the mutual exclusion guarantee, which is only limited to a given window of time from the moment the lock is acquired. So now we have a good way to acquire and release the lock. The system, reasoning about a non-distrubited system which is composed of a single instance, always available, is safe. Let’s extend the concept to a distributed system where we don’t have such guarantees. # Distributed version In the distributed version of the algorithm we assume to have N Redis masters. Those nodes are totally independent, so we don’t use replication or any other implicit coordination system. We already described how to acquire and release the lock safely in a single instance. We give for granted that the algorithm will use this method to acquire and release the lock in a single instance. In our examples we set N=5, which is a reasonable value, so we need to run 5 Redis masters in different computers or virtual machines in order to ensure that they’ll fail in a mostly independent way. In order to acquire the lock, the client performs the following operations: Step 1) It gets the current time in milliseconds. Step 2) It tries to acquire the lock in all the N instances sequentially, using the same key name and random value in all the instances. During the step 2, when setting the lock in each instance, the client uses a timeout which is small compared to the total lock auto-release time in order to acquire it. For example if the auto-release time is 10 seconds, the timeout could be in the ~ 5-50 milliseconds range. This prevents the client to remain blocked for a long time trying to talk with a Redis node which is down: if an instance is not available, we should try to talk with the next instance ASAP. Step 3) The client computes how much time elapsed in order to acquire the lock, by subtracting to the current time the timestamp obtained in step 1. If and only if the client was able to acquire the lock in the majority of the instances (at least 3), and the total time elapsed to acquire the lock is less than lock validity time, the lock is considered to be acquired. Step 4) If the lock was acquired, its validity time is considered to be the initial validity time minus the time elapsed, as computed in step 3. Step 5) If the client failed to acquire the lock for some reason (either it was not able to lock N/2+1 instances or the validity time is negative), it will try to unlock all the instances (even the instances it believe it was not able to lock). # Synchronous or not? Basically the algorithm is partially synchronous: it relies on the assumption that while there is no synchronized clock across the processes, still the local time in every process flows approximately at the same rate, with an error which is small compared to the auto-release time of the lock. This assumption closely resembles a real-world computer: every computer has a local clock and we can usually rely on different computers to have a clock drift which is small. Moreover we need to refine our mutual exclusion rule: it is guaranteed only as long as the client holding the lock will terminate its work within the lock validity time (as obtained in step 3), minus some time (just a few milliseconds in order to compensate for clock drift between processes). # Retry When a client is not able to acquire the lock, it should try again after a random delay in order to try to desynchronize multiple clients trying to acquire the lock, for the same resource, at the same time (this may result in a split brain condition where nobody wins). Also the faster a client will try to acquire the lock in the majority of Redis instances, the less window for a split brain condition (and the need for a retry), so ideally the client should try to send the SET commands to the N instances at the same time using multiplexing. It is worth to stress how important is for the clients that failed to acquire the majority of locks, to release the (partially) acquired locks ASAP, so that there is no need to wait for keys expiry in order for the lock to be acquired again (however if a network partition happens and the client is no longer able to communicate with the Redis instances, there is to pay an availability penalty and wait for the expires). # Releasing the lock Releasing the lock is simple and involves just to release the lock in all the instances, regardless of the fact the client believe it was able to successfully lock a given instance. # Safety arguments Is this system safe? We can try to understand what happens in different scenarios. To start let’s assume that a client is able to acquire the lock in the majority of instances. All the instances will contain a key with the same time to live. However the key was set at different times, so the keys will also expire at different times. However if the first key was set at worst at time T1 (the time we sample before contacting the first server) and the last key was set at worst at time T2 (the time we obtained the reply from the last server), we are sure that the first key to expire in the set will exist for at least MIN_VALIDITY=TTL-(T2-T1)-CLOCK_DRIFT. All the other keys will expire later, so we are sure that the keys will be simultaneously set for at least this time. During the time the majority of keys are set, another client will not be able to acquire the lock, since N/2+1 SET NX operations can’t succeed if N/2+1 keys already exist. So if a lock was acquired, it is not possible to re-acquire it at the same time (violating the mutual exclusion property). However we want to also make sure that multiple clients trying to acquire the lock at the same time can’t simultaneously succeed. If a client locked the majority of instances using a time near, or greater, than the lock maximum validity time (the TTL we use for SET basically), it will consider the lock invalid and will unlock the instances, so we only need to consider the case where a client was able to lock the majority of instances in a time which is less than the validity time. In this case for the argument already expressed above, for MIN_VALIDITY no client should be able to re-acquire the lock. So multiple clients will be albe to lock N/2+1 instances at the same time (with “time" being the end of Step 2) only when the time to lock the majority was greater than the TTL time, making the lock invalid. Are you able to provide a formal proof of safety, or to find a bug? That would be very appreciated. # Liveness arguments The system liveness is based on three main features: 1) The auto release of the lock (since keys expire): eventually keys are available again to be locked. 2) The fact that clients, usually, will cooperate removing the locks when the lock was not acquired, or when the lock was acquired and the work terminated, making it likely that we don’t have to wait for keys to expire to re-acquire the lock. 3) The fact that when a client needs to retry a lock, it waits a time which is comparable greater to the time needed to acquire the majority of locks, in order to probabilistically make split brain conditions during resource contention unlikely. However there is at least a scenario where a very special network partition/rejoin pattern, repeated indefinitely, may violate the system availability. For example with N=5, two clients A and B may try to lock the same resource at the same time, nobody will be able to acquire the majority of locks, but they may be able to lock the majority of nodes if we sum the locks of A and B (for example client A locked 2 instances, client B just one instance). Then the clients are partitioned away before they can unlock the locked instances. This will leave the resource not lockable for a time roughly equal to the auto release time. Then when the keys expire, the two clients A and B join again the partition repeating the same pattern, and so forth indefinitely. Another point of view to see the problem above, is that we pay an availability penalty equal to “TTL” time on network partitions, so if there are continuous partitions, we can pay this penalty indefinitely. I can’t find a simple way to have guaranteed liveness (but did not tried very hard honestly), but the worst case appears to be hard to trigger. Basically it means that we can only provide, using this algorithm, an approximation of Property number 2. # Performance, crash-recovery and fsync Many users using Redis as a lock server need high performance in terms of both latency to acquire and release a lock, and number of acquire / release operations that it is possible to perform per second. In order to meet this requirement, the strategy to talk with the N Redis servers to reduce latency is definitely multiplexing (or poor’s man multiplexing, which is, putting the socket in non-blocking mode, send all the commands, and read all the commands later, assuming that the RTT between the client and each instance is similar). However there is another consideration to do about persistence if we want to target a crash-recovery system model. Basically to see the problem here, let’s assume we configure Redis without persistence at all. A client acquires the lock in 3 of 5 instances. One of the instances where the client was able to acquire the lock is restarted, at this point there are again 3 instances that we can lock for the same resource, and another client can lock it again, violating the safety property of exclusivity of lock. If we enable AOF persistence, things will improve quite a bit. For example we can upgrade a server by sending SHUTDOWN and restarting it. Because Redis expires are semantically implemented so that virtually the time still elapses when the server is off, all our requirements are fine. However everything is fine as long as it is a clean shutdown. What about a power outage? If Redis is configured, as by default, to fsync on disk every second, it is possible that after a restart our key is missing. Long story short if we want to guarantee the lock safety in the face of any kind of instance restart, we need to enable fsync=always in the persistence setting. This in turn will totally ruin performances to the same level of CP systems that are traditionally used to implement distributed locks in a safe way. The good news is that because in our algorithm we don’t stop to acquire locks as soon as we reach the majority of the servers, the actual probability of safety violation is small, because most of the times the lock will be hold in all the 5 servers, so even if one restarts without a key, it is practically unlikely (but not impossible) that an actual safety violation happens. Long story short, this is an user pick, and a big trade off. Given the small probability for a race condition, if it is acceptable that with an extremely small probability, after a crash-recovery event, the lock may be acquired at the same time by multiple clients, the fsync at every operation can (and should) be avoided. # Reference implementation I wrote a simple reference implementation in Ruby, backed by redis-rb, here: http://github.com/antirez/redlock-rb # Want to help? If you are into distributed systems, it would be great to have your opinion / analysis. Also reference implementations in other languages could be great. Thanks in advance! EDIT: I received feedbacks in this blog post comment and via Hacker News that's worth to incorporate in this blog post. 1) As Steven Benjamin notes in the comments below, if after restarting an instance we can make it unavailable for enough time for all the locks that used this instance to expire, we don't need fsync. Actually we don't need any persistence at all, so our safety guarantee can be provided with a pure in-memory configuration. An example: previously we described the example race condition where a lock is obtained in 3 servers out of 5, and one of the servers where the lock was obtained restarts empty: another client may acquire the same lock by locking this server and the other two that were not locked by the previous client. However if the restarted server is not available for queries enough time for all the locks that were obtained with it to expire, we are guaranteed this race is no longer possible. 2) The Hacker News user eurleif noticed how it is possible to reacquire the lock as a strategy if the client notices it is taking too much time in order to complete the operation. This can be done by just extending an existing lock, sending a script that extends the expire of the value stored at the key is the expected one. If there are no new partitions, and we try to extend the lock enough in advance so that the keys will not expire, there is the guarantee that the lock will be extended. 3) The Hacker News user mjb noted how the term "skew" is not correct to describe the difference of the rate at which different clocks increment their local time, and I'm actually talking about "Drift". I'm replacing the word "skew" with "drift" to use the correct term. Thanks for the very useful feedbacks. Comments

• Using Heartbleed as a starting point
  Antirez · Apr 10, 2014

  The strong reactions about the recent OpenSSL bug are understandable: it is not fun when suddenly all the internet needs to be patched. Moreover for me personally how trivial the bug is, is disturbing. I don’t want to point the finger to the OpenSSL developers, but you just usually think at those class of issues as a bit more subtle, in the case of a software like OpenSSL. Usually you fail to do sanity checks *correctly*, as opposed to this bug where there is a total *lack* of bound checks in the memcpy() call. However sometimes in the morning I read the code I wrote the night before and I’m deeply embarrassed. Programmers sometimes fail, I for sure do often, so my guess is that what is needed is a different process, and not a different OpenSSL team. There is who proposes a different language safer than C, and who proposes that the specification is broken because it is too complex. Probably there is some truth in both arguments, however it is unlikely that we move to a different specification or system language soon, so the real question is, what we can do now to improve system software security? 1) Throw money at it. Making system code safer is simple if there are investments. If different companies hire security experts to do code auditings in the OpenSSL code base, what happens is that the probability of discovering a bug like heartbleed is greater. I’ve seen very complex bugs that are triggered by a set of non-trivial conditions being discovered by serious code auditing efforts. A memcpy() without bound checks is something that if you analyze the code security-wise, will stand out in the first read. And guess how heartbleed was discovered? Via security auditings performed at Google. Probably the time to consider open source something that mostly we take from is over. Many companies should follow the example of Google and other companies, using workforce for OSS software development and security. 2) Static and dynamic checks. Static code analysis is, as a side effect, a semi-automated way to do code auditings. In critical system code like OpenSSL even to do some source code annotation or use a set of rules to make static analysis more effective is definitely acceptable. Static tools today are not a total solution, but the output of a static analysis if carefully inspected by an expert programmer can provide some value. Another great help comes from dynamic checks like Valgrind. Every system software written in C should be tested using Valgrind automatically at every new commit. 3) Abstract C with libraries. C is low level and has no built in safety in the language. However something good about C is that it is a language that allows to build layers on top of its rawness. A sane dynamic string library prevents a lot of buffer overflow issues, and today almost every decent project is using one. However there is more you can do about it. For example for security critical code where memory can contain things like private keys, you can augment your dynamic string library with memory copy primitives that only copy from one buffer to the other performing implicit sanity checks. Moreover if a buffer contains critical data, you can set logical permissions so that trying to copy from this area aborts the program. There are other less-portable ways using memory management to protect important memory pages in an even more effective ways, however an higher C-level protection can be much simpler in the real-world because of portability / predictability concerns. In general many things can be explored to avoid using C without protections, creating a library that abstracts on top of it to make programming safer. 4) Randomized tests. Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. In my experience having a great deal of randomized tests together with an environment where the same tests are ran again and again with the program running over Valgrind, can discover a number of real-world bugs that gets otherwise unnoticed. There are many models to explore, usually you want something that injects totally random data, and intermediate models where valid packets are corrupted in different random ways. A typical example of this technique is the old DNS compression infinite-loop bug. Trow a few random packets to a naive implementation and you’ll find it in a matter of minutes. 5) Change of mentality about security vs performance. It is interesting that OpenSSL is doing its own allocation caching stuff because in some systems malloc/free is slow. This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in *some* system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system. In general it does not help the fact that the system that is the de facto standard in today’s servers infrastructure, that is, Linux, has had, and still has, one of the worst allocators you will find around, mostly for licensing concerns, since the better allocators are not GPL but BSD licensed. Probably yet another area where big corps should contribute, by providing significant improvements to glibc malloc. Glibc malloc is, even if better alternatives are available, what many real-world system softwares are going to use anyway. I would love to see the discussion about heartbleed to take a more pragmatic approach, because one thing is guaranteed: to blame here or there will not change the actual level of the security of OpenSSL or anything else, and there are new challenges in the future. For example the implementation of HTTP/2.0 may be a very delicate moment security wise. EDIT: Actually I was not right and the malloc implementation inside the Glibc is BSD licensed, so it is not a license issue. I don't know why the Glibc is not using Jemalloc instead that is very good and actively developed allocator. Comments

• Redis new data structure: the HyperLogLog
  Antirez · Apr 01, 2014

  Generally speaking, I love randomized algorithms, but there is one I love particularly since even after you understand how it works, it still remains magical from a programmer point of view. It accomplishes something that is almost illogical given how little it asks for in terms of time or space. This algorithm is called HyperLogLog, and today it is introduced as a new data structure for Redis. Counting unique things === Usually counting unique things, for example the number of unique IPs that connected today to your web site, or the number of unique searches that your users performed, requires to remember all the unique elements encountered so far, in order to match the next element with the set of already seen elements, and increment a counter only if the new element was never seen before. This requires an amount of memory proportional to the cardinality (number of items) in the set we are counting, which is, often absolutely prohibitive. There is a class of algorithms that use randomization in order to provide an approximation of the number of unique elements in a set using just a constant, and small, amount of memory. The best of such algorithms currently known is called HyperLogLog, and is due to Philippe Flajolet. HyperLogLog is remarkable as it provides a very good approximation of the cardinality of a set even using a very small amount of memory. In the Redis implementation it only uses 12kbytes per key to count with a standard error of 0.81%, and there is no limit to the number of items you can count, unless you approach 2^64 items (which seems quite unlikely). The algorithm is documented in the original paper [1], and its practical implementation and variants were covered in depth by a 2013 paper from Google [2]. [1] http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf [2] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf How it works? === There are plenty of wonderful resources to learn more about HyperLogLog, such as [3]. [3] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/ Here I’ll cover only the basic idea using a very clever example found at [3]. Imagine you tell me you spent your day flipping a coin, counting how many times you encountered a non interrupted run of heads. If you tell me that the maximum run was of 3 heads, I can imagine that you did not really flipped the coin a lot of times. If instead your longest run was 13, you probably spent a lot of time flipping the coin. However if you get lucky and the first time you get 10 heads, an event that is unlikely but possible, and then stop flipping your coin, I’ll provide you a very wrong approximation of the time you spent flipping the coin. So I may ask you to repeat the experiment, but this time using 10 coins, and 10 different piece of papers, one per coin, where you record the longest run of heads. This time since I can observe more data, my estimation will be better. Long story short this is what HyperLogLog does: it hashes every new element you observe. Part of the hash is used to index a register (the coin+paper pair, in our previous example. Basically we are splitting the original set into m subsets). The other part of the hash is used to count the longest run of leading zeroes in the hash (our run of heads). The probability of a run of N+1 zeroes is half the probability of a run of length N, so observing the value of the different registers, that are set to the maximum run of zeroes observed so far for a given subset, HyperLogLog is able to provide a very good approximated cardinality. The Redis implementation === The standard error of HyperLogLog is 1.04/sqrt(m), where “m” is the number of registers used. Redis uses 16384 registers, so the standard error is 0.81%. Since the hash function used in the Redis implementation has a 64 bit output, and we use 14 bits of the hash output in order to address our 16k registers, we are left with 50 bits, so the longest run of zeroes we can encounter will fit a 6 bit register. This is why a Redis HyperLogLog value only uses 12k bytes for 16k registers. Because of the use of a 64 bit output function, which is one of the modifications of the algorithm that Google presented in [2], there are no practical limits to the cardinality of the sets we can count. Moreover it is worth to note that the error for very small cardinalities tend to be very small. The following graph shows a run of the algorithm against two different large sets. The cardinality of the set is shown in the x axis, while the relative error (in percentage) in the y axis. img://antirez.com/misc/hll_1.png The red and green lines are two different runs with two totally unrelated sets. It shows how the error is consistent as the cardinality increases. However for much smaller cardinalities, you can enjoy a much smaller error: img://antirez.com/misc/hll_2.png The green line shows the error of a single run up to cardinality 100, while the red line is the maximum error found in 100 runs. Up to a cardinality of a few hundreds the algorithm is very likely to make a very small error or to provide the exact answer. This is very valuable when the computed value is shown to an user that can visually match if the answer is correct. The source code of the Redis implementation is available at Github: https://github.com/antirez/redis/blob/unstable/src/hyperloglog.c The API === From the point of view of Redis an HyperLogLog is just a string, that happens to be exactly 12k + 8 bytes in length (12296 bytes to be precise). All the HyperLogLog commands will happily run if called with a String value exactly of this size, or will report an error. However all the calls are safe whatever is stored in the string: you can store garbage and still ask for an estimation of the cardinality. In no case this will make the server crash. Also everything in the representation is endian neutral and is not affected by the processor word size, so a 32 bit big endian processor can read the HLL of a 64 bit little endian processor. The fact that HyperLogLogs are strings avoided the introduction of an actual type at RDB level. This allows the work to be back ported into Redis 2.8 in the next days, so you’ll be able to use HyperLogLogs ASAP. Moreover the format is automatically serialized, and can be retrieved and restored easily. The API is constituted of three new commands: PFADD var element element … element PFCOUNT var PFMERGE dst src src src … src The commands prefix is “PF” in honor of Philippe Flajolet [4]. [4] http://en.wikipedia.org/wiki/Philippe_Flajolet PFADD adds elements to the HLL stored at “var”. If the variable does not exist, an empty HLL is automatically created as it happens always with Redis API calls. The command is variadic, so allows for very aggressive pipelining and mass insertion. The command returns 1 if the underlying HyperLogLog was modified, otherwise 0 is returned. This is interesting for the user since as we add elements the probability of an element actually modifying some register decreases. The fact that the API is able to provide hints about the fact that a new cardinality is available allows for programs that continuously add elements and retrieve the approximated cardinality only when a new one is available. PFCOUNT returns the estimated cardinality, which is zero if the key does not exist. Finallly PFMERGE can merge N different HLL values into one. The resulting HLL will report an estimated cardinality that is the cardinality of the union of the different sets that we counted with the different HLL values. This seems magical but works because HLL while randomized is fully deterministic, so PFMERGE just takes, for every register, the maximum value available across the N HLL values. A given element hashes to the same register with the same run of zeroes always, so the merge performed in this way will only add the count of the elements that are not common to the different HLLs. As you can see HyperLogLog is fully parallelizable, since it is possible to split a set into N subsets counted independently to later merge the values and obtain the total cardinality approximation. The fact that HLLs in Redis are just strings helps to move HLL values across instances. First make it correct, then make it fast === Redis HHLs are composed of 16k registers packed into 6 bit integers. This creates several performance issues that must be solved in order to provide an API of commands that can be called without thinking too much. One problem is that accessing to registers require accessing multiple bytes, shifting, and masking in order to retrieve the correct 6 bit value. This is not a big problem for PFADD that only touches a register for every element, but PFCOUNT needs to perform a computation using all the 16k registers, so if there are non trivial constant times to access every single register, the command risks to be slow. Moreover, while accessing the registers, we need to compute the sum of pow(2,-register) which involves floating point math. One may feel the temptation of using full bytes instead of 6 bit integers in order to speedup the computation, however this would be a shame since every HLL would use 16k instead of 12k that is a non trivial difference, so this route was discarded at the beginning. The command was optimized for a speedup of about 3 times compared to the initial implementation by doing the following changes: * For m=16k which is the Redis default (the implementation is more generic and could theoretically work with different values) the implementation selects a fast-path with unrolled loops accessing 16 register at every time. The registers are accessed using fixed offsets / shifts / masks (via some pointer that is incremented 12 bytes at the next iteration). * The floating point computation was modified in order to allow for multiple operations to be performed in parallel when possible. This was just a matter of adding parens. Floating point math is not commutative, but in this case there was no loss of precision. * The pow(2,-register) term was precomputed in a lookup table. With the 3x speedup provided by the above changes the command was able to perform about 60k calls per second in a fast hardware. However this is still far from the hundreds thousands calls possible with commands that are, from the user point of view, conceptually similar, like SCARD. Instead of optimizing the computation of the approximated cardinality further, there was a simpler solution. Basically the output of the algorithm only changes if some register changes. However as already observed above, most of the PFADD calls don’t result in any register changed. This basically means that it is possible to cache the last output and recompute it only if some register changes. So our data structure has an additional tail of 8 bytes representing a 64bit unsigned integer in little endian format. If the most significant bit is set, then the precomputed value is stale and requires to be recomputed, otherwise PFCOUNT can use it as it is. PFADD just turns on the “invalid cache” bit when some register is modified. After this change even trying to add elements at maximum speed using a pipeline of 32 elements with 50 simultaneous clients, PFCOUNT was able to perform as well as any other O(1) command with very small constant times. Bias correction using polynomial regression === The HLL algorithm, in order to be practical, must work equally well in any cardinality range. Unfortunately the raw estimation performed by the algorithm is not very good for cardinalities less than m*2.5 (around 40000 elements for m=16384) since in this range the algorithm outputs biased or even results with larger errors depending on the exact range. The original HLL paper [1] suggests switching to Linear Counting [5] when the raw cardinality estimated by the first part of the HLL algorithm is less than m*2.5. [5] http://dblab.kaist.ac.kr/Publication/pdf/ACM90_TODS_v15n2.pdf Linear counting is a different cardinality estimator that uses a simple concept. We have a bitmap of N bits. Every time a new element must be counted, it is hashed, and the hash is used in order to index a random bit inside the bitmap, that is turned to 1. The number of unset bits in the bitmap gives an idea of how many elements we added so far using the following formula: cardinality = m*log(m/ez); Where ‘ez’ is the number of zero bits and m is the total number of bits in the bitmap. Linear counting does not work well for large cardinalities compared to HyperLogLog, but works very well for small cardinalities. Since the HLL registers as a side effect also work as a linear counting bitmap, counting the number of zero registers it is possible to apply linear counting for the range where HLL does not perform well. Note that this is possible because when we update the registers, we don’t really use the longest run of zeroes, but the longest run of zeroes plus one. This means that if an element is added and it is addressing a register that was never addressed, the register will turn from 0 to a different value (at least 1). The problem with linear counting is that as the cardinality gets bigger, its output error gets larger, so we need to switch to HLL ASAP. However when we switch at 2.5m, HLL is still biased. In the following image the same cardinality was tested with 1000 different sets, and the error of each run is reported as a point: img://antirez.com/misc/hll_3.png The blu line is the average of the error. As you can see before a cardinality of 40k, where linear counting is used, the more we go towards greater cardinalities, the more the points “beam” gets larger (bigger errors). When we switch to HLL raw estimate the error is smaller, but there is a bias: the algorithm overestimates the cardinality in the range 40k-80k. Google engineers studied this problem extensively [2] in order to correct the bias. Their solution was to create an empirical table of cardinality values and the corresponding biases. Their modified algorithm uses the table and interpolation in order to get the bias in a given range, and correct accordingly. I used a different approach: you can see that the bias is not random but looks like a very smooth curve, so I calculated a few cardinality-bias samples and performed polynomial regression in order to find a polynomial approximating the curve. Currently I’m using a four order polynomial to correct in the range 40960-72000, and the following is the result after the bias correction: img://antirez.com/misc/hll_4.png While there is still some bias at the switching point between the two algorithms, the result is quite satisfying compared to the vanilla HLL algorithm, however it is probably possible to use a curve that fits better the bias curve. I had no time to investigate this further. It is worth to note that during my investigations I found that, when no bias correction is used, and at least for m=16384, the best value to switch from linear counting to raw HLL estimate is actually near 3 and not 2.5 as mentioned in [1], since a value of 3 both improves bias and error. Values larger than 3 will improve the bias (a value of 4 completely corrects it) but will have bad effects on the error. The original HLL algorithm also corrects for values towards 2^32 [1][2] since once we approach very large values collisions in the hash function starts to be an issue. We don’t need such correction since we use a 64 bit hash function and 6 bits counters, which is one of the modifications proposed by Google engineers [2] and adopted by the Redis implementation. Future work === Intuitively it seems like it is possible to improve the error of the algorithm output when linear counting is used by exploiting the additional informations we have. In the standard linear counting algorithm the registers are just 1 bit wide, so we have only two informations: if an element so far hashed to this bit or not. Still the HLL algorithm as proposed initially [1] and as modified at Google [2], when reverting to linear counting still only use the number of zero registers as the input of the algorithm. It is possible that also using the information stored in the registers could improve the output. For example in standard linear counting, assuming we have 10 bits, I may add 5 elements that all happen to address the same bit. This is an odd case that the algorithm has no way to correct, and the estimation provided will likely be smaller than the actual cardinality. However in the linear counting algorithm used by HLL in a similar situation we may found that the value at the only register set is an hint about multiple elements colliding there, allowing a correction of the output. Conclusion === HyperLogLog is an amazing data structure. My hope is that the Redis implementation, that will be available in a stable release in a matter of days (Redis 2.8.9 will include it), will provide this tool in a ready to use form to many programmers. The HN post is here: https://news.ycombinator.com/item?id=7506774 Comments

• Fascinating little programs
  Antirez · Mar 13, 2014

  Yesterday and today I managed to spend some time with linenoise (http://github.com/antirez/linenoise), a minimal line-editing library designed to be a simple and small replacement for readline. I was trying to merge a few pull requests, to fix issues, and doing some refactoring at the same time. It was some kind of nirvana I was feeling: a complete control of small, self-contained, and useful code. There is something special in simple code. Here I’m not referring to simplicity to fight complexity or over engineering, but to simplicity per se, auto referential, without goals if not beauty, understandability and elegance. After all the programming world has always been fascinated with small programs. For decades programmers challenged in 1k or 4k contexts, from the 6502 assembler to today’s javascript contests. Even the obfuscated C contest, after all, has a big component in the minimalism. Why is it so great to hack a small piece of code? Yes is small and simple, those are two good points. It can be totally understood, dominated. You can use smartness since little code is the only place of the world where coding smartness will pay off, since in large projects obviousness is far better in the long run. However I believe there is more than that, and is that small programs can be perfect. As perfect as a sonnet composed of a few words. The limits in size and in scope, constitute an intellectual stratagem to avoid the “it may be better" trap, when this better is not actually measurable and evident. Under these strict limits, what the program does is far more interesting than what it does not. Actually the constraints are the more fertile ground for creativity of the solutions, otherwise likely useless: at scale there is always a more correct, understood, canonical way to do everything. There is an interview of Bill Gates in the first years of the Microsoft experience where he describes this feeling when writing the famous Microsoft BASIC interpreter. The limits were the same we self impose today to ourselves for fun, in the contests, or just for the sake of it. There was a generation of programmers that was able to experience perfection in their creations, where it was obvious to measure and understand if a change actually lead to an improvement of the program or not, in a territory where space and time were so scarse. There was no room for wastes and not needed complexity. Today’s software is in some way the triumph of the other reality of software: layers of complexities that gave use incredible devices or infrastructure technologies that in the hands of non experts leverage a number of possibilities. However maybe there is still something to preserve from the ancient times where software could be perfect, the feeling that what you are creating has a structure and is not just a pile of code that works. If you zoom out enough, you’ll see your large program is actually quite small again, and at least at this scale, it should resemble perfection, or at least, aim at it. Comments

• What is performance?
  Antirez · Feb 28, 2014

  The title of this blog post is an apparently trivial to answer question, however it is worth to consider a bit better what performance really means: it is easy to get confused between scalability and performance, and to decompose performance, in the specific case of database systems, in its different main components, may not be trivial. In this short blog post I’ll try to write down my current idea of what performance is in the context of database systems. A good starting point is probably the first slide I use lately in my talks about Redis. This first slide is indeed about performance, and says that performance is mainly three different things. 1) Latency: the amount of time I need to get the reply for a query. 2) Operations per unit of time per core: how many queries (operations) the system is able to reply per second, in a given reference computational unit? 3) Quality of operations: how much work those operations are able to accomplish? Latency — This is probably the simplest component of performance. In many applications it is desirable that the time needed to get a reply from the system is small. However while the average time is important, another concern is the predictability of the latency figure, and how much difference there is between the average case and the worst case. When used well, in-memory systems are able to provide very good latency characteristics, and are also able to provide a consistent latency over time. Operations per second per core — The second component I’m enumerating is what makes the difference between raw performance and scalability. We are interested in the amount of work the system is able to do, in a given unit of time, for a given reference computational unit. Linearly scalable systems can reach a big number of operations per second by using a number of nodes, however this means they are scalable, and not necessarily performant. Operations per second per core is also usually bound to the amount of queries you can perform per watt, so to the energy efficiency of the system. Quality of operations — The last point, while probably not as stressed among developers as throughput and latency, is really important in certain kind of systems, especially in-memory systems. A system that is able to perform 100 operations per second, but with operations of “poor quality” (for example just GET and SET in Redis terms) has a lower performance compared to a system that is also able to perform an INCR operation with the same latency and OPS characteristics. For instance, if the problem at hand is to increment counters, the former system will require two operations to increment a counter (we are not considering race conditions in this context), while the system providing INCR is able to use a single operation. As a result it is actually able to provide twice the performance of the former system. As you can see the quality of operations is not an absolute meter, but depends on the kind of problem to solve. The same two systems if we want to cache HTML fragments are equivalent since the INCR operation would be useless. The quality of operations is particularly important in in-memory systems, since usually the computation itself is negligible compared to the time needed to receive, dispatch the command, and create a reply, so systems like Redis with a rich set of operations are able to provide better performance in many contexts almost for free, just allowing the user to do more with a single operation. The “do more” part can actually mean a lot of things: either provide a reply to a more complex question, like for example the ZRANK command of Redis, or simply being able to provide a more *selective* reply, like HMGET command that is able to provide information just for a subset of the fields composing an Hash value, reducing the amount of bandwidth required between the server and its clients. In general quality of operations don't only affect performances because they give less or more value to the operations per second the system is able to perform: operations quality also directly affect latency, since more complex operations are able to avoid back and forth data transfer between clients and servers required to mount multiple simpler operations into a more complex computation. Conclusion — I hope that this short exploration of what performance is uncovered some of the complexities involved in the process of evaluating the capabilities of a database system from this specific point of view. There is a lot more to say about it, but I found that the above three components of the performance are among the most interesting and important when evaluating a system and when there is to understand how to evolve an existing system to improve its performance characteristics. Thanks to Yiftach Shoolman for feedbacks about this topic. Comments

• Happy birthday Redis!
  Antirez · Feb 26, 2014

  Today Redis is 5 years old, at least if we count starting from the initial HN announcement [1], that’s actually a good starting point. After all an open source project really exists as soon as it is public. I’m a bit shocked I worked for five years straight to the same thing. The opportunities for learning new things I had because of the directions where Redis pushed me, and the opportunities to learn new things that I missed because I had almost consistently no time for random hacking, are huge. My feeling today is that the Redis project was possible because of the great coders I encountered in my journey: they made Redis popular adopting it in its infancy, since great coders don’t follow the hype. Great coders provided outstanding additions to Redis in the form of patches and ideas that were able to surpass my instinct to be conservative when the topic was to extend the system or accept external contributions. More great coders made possible to sponsor Redis when it was in its infancy, recognizing that there was something interesting about it, and more great coders applied it in the right way to solve problems in the course of many years, wrote an incredible ecosystem of client libraries and tools, and helped other coders to apply it when it was not clear what was the best way to solve a given problem. The Redis community is outstanding because in some way it managed to attract a number of great coders. I learned that in the future, whatever I’ll do more coding or I’ll be in a team to build something great in a different role, my top priority will be to stay with great coders, and I learned that they are not easy to recognize at first: their abilities don’t correlate with the number of followers on Twitter nor with the number of Github repositories. You have to discover great coders one after the other, and the biggest gift that Redis provided to me, was to get exposed to many of them. In the course of five years there was also time, for me, to evolve my idea of what Redis is. The idea I’ve of Redis today is that its contribution should be to try to explore corner designs and bizzarre ideas. After all there are large teams of people much smarter than me trying to work on the hard problems applying the best technologies available. Redis will continue to be a small research in more obscure places of the design space. After all I’ve the feeling that it helped to popularize certain non obvious ideas, like using data structures as data model for key value stores and caches, or that it is possible to apply scripting to database systems in a different way than stored procedures. However for Redis to be able to do this research, I should be ready to be opinionated and change development direction when something is weak. This was done in the past, deprecating swap and diskstore, but should be done even more in the future. Moreover Redis should be able to purse different goals at the same time: once Redis 3.0 will be stable, the design of Redis Cluster is conceived in order to leave my hands free about changes in the data model, without too much limits or compromises. This will result in a Redis 3.2 release that will focus again on the API, stressing one of the initial and fundamental aspects of Redis: caching, data model and computation. It is entirely not obvious to me, after five years, to consider the Redis journey still ongoing, and I’m happy about it, because my motivations are not investors or shares, nor that I’m particularly in love with Redis as a project. If something new appears tomorrow that marginalizes Redis and makes it totally useless I’ll be very happy to start some new gig, after all this is how technology works: for cycles. And, after all, starting from scratch with something new is always exciting. However currently I believe there is more to do about Redis, and I’ll be happy to continue my work on it in the next weeks. [1] https://news.ycombinator.com/item?id=494649 Comments

• A simple distributed algorithm for small idempotent information
  Antirez · Feb 21, 2014

  In this blog post I’m going to describe a very simple distributed algorithm that is useful in different programming scenarios. The algorithm is useful when you need to take some kind of information synchronized among a number of processes. The information can be everything as long as it is composed of a small number of bytes, and as long as it is idempotent, that is, the current value of the information does not depend on the previous value, and we can just replace an old value, with the new one. The size of the information is important because for the way the algorithm works, the information should be small enough that every node can broadcast it from time to time to some other random node, so it should fit the size of an “heartbeat” packet. Let’s say that up to a few kbytes everything is fine. This algorithm is no new in any way, it is basically just a trivial way to put together obvious ideas found in other distributed algorithms in a simple way. However the algorithm is very useful in many real-world contexts, and is extremely simple to implement. The algorithm is mostly borrowed from Raft, however because of the premises it uses only a subset of Raft that is trivial to implement. An example scenario === To understand better the algorithm, it is much better to have an example of problem that we want to solve in our distributed system. Let’s say that we have N processes connected with two kind of wifi networks: a very reliable but slow wireless network, that is only suitable to send “control” packets like heartbeats or other low bandwidth data, and a very fast wireless network. Now let’s imagine that while the slow network works at a fixed frequency, the high speed wireless network requires to adapt to changing conditions and noises in a given frequency, and is able to hop to a different frequency as soon as too much noise is detected. We need a way to make sure that all the processes use the same frequency to communicate with the high speed network, and we also need a way to switch frequency when the currently used frequency has issues. We need this system to work if there are network partitions in the slow network, as long as the majority of the processes are able to communicate. Note that this problem has the properties stated above: 1) The information is idempotent, if the high speed network switched to an different frequency, the new frequency does not depend on the old frequency. A process receiving the new frequency can just update frequency regardless of the fact that its old frequency was updated, or an older one (because for some reason it did not received some update). 2) The information is small, it is possible to propagate it easily across nodes in a small data packet. In this case it is actually extremely small, for example the frequency may be encoded in a 64 bit integer. Epochs and update messages === The basic idea of this algorithm is that there is an artificial notion of time across the processes, that is used to order events or informations without to resort to the system time of the process, that is hard to synchronize between them. This artificial time is called the “epoch”. Every process has the notion of currentEpoch, that is, initialized at zero at startup. Every time a process sees an epoch that is greater that its current epoch, it updates its epoch to match the observed epoch. Every process has also the notion of the frequencyEpoch, that is, the version of the currently used frequency. In order to propagate the information, every process periodically sends an update message to some other process. For example every 5 seconds every process picks a random process, and sends to it an update message containing: the current frequency in use, the epoch of the frequency used, and the currentEpoch of the process sending the update message. The first time a process is created its frequency is set to -1: this means that there is no frequency currently in use from the point of view of a given process, and that another one must be picked. Updating the frequency === When a process receives an update message from another process containing a frequency with a frequencyEpoch that is greater than its local frequencyEpoch, it updates its frequency to the received value, and sets the frequencyEpoch to the received value as well. In general when the currentEpoch or the frequency and frequencyEpoch are modified, the process writes this change to the disk or other permanente storage, so that when the process is restarted it will use the latest known information. Choosing a frequency === A process requires to choose a frequency in two different scenarios: 1) When the current frequency is detected to be noisy. 2) When the current frequency is set to -1 (at startup). In order to choose a frequency, a process requires to win an election. This is how it works: 1) The process increments its own currentEpoch, and writes it to permanent storage. It also selects a suitable new frequency. 2) The process sends to all the other processes a ELECT_ME packet to get the vote of the other processes. The ELECT_ME packet contains the currentEpoch of the sending process. 3) The other processes will reply with YOU_HAVE_MY_VOTE packet only if their currentEpoch is not greater compared to the one of the process requesting the vote (it can’t be smaller, since the reception of the ELECT_ME packet will cause an older currentEpoch to be updated to match the one of the incoming packet). The YOU_HAVE_MY_VOTE packet contains the currentEpoch of the voting process. 4) A given process only votes a single time for a given epoch, so it takes a variable called lastVoteEpoch, and will only provide its vote if the currentEpoch in the request for the vote is greater (>) than lastVoteEpoch. When the vote is provided, lastVoteEpoch is updated (and stored on disk *before* the vote is provided, so that a crash and restart will not cause this process to vote again for the same epoch). 5) YOU_HAVE_MY_VOTE messages with a currentEpoch smaller than the currentEpoch of the process that requested the vote are discarded. 6) The process requesting the vote will consider itself elected only if it receives the majority of the votes from the other processes (it will count itself as a voter and will vote for itself when the election starts). If the process is elected it will updated its frequencyEpoch and frequency variables. The frequencyEpoch that will be used is the epoch the process requested the vote with, that is, its currentEpoch at the time it sent the ELECT_ME packets, just after the increment. Given that a process requires to be elected to change the frequency, and that every process votes a single time in a given epoch, there must be only a single winner for a given epoch. If a given process is not able to get elected as the majority is not reached, it will try again after a random delay. This delay must be greater compared to the latency of the slow network that is used to exchange these messages (see the Raft paper for more information about this). Every process will consider the election aborted after some time that is smaller than the retry time. Propagating the new information === When a process wins an election, it updates its frequency value and frequencyEpoch to the new one, so by sending UPDATE messages, eventually all the other processes will receive the update as well and will switch to the new frequency. If some process is partitioned away, it will receive the update as soon as the partition heals. However it is a good idea to broadcast an UPDATE message to all the processes ASAP as soon as a process changes the frequency, so that all the other nodes will switch ASAP. Improving the algorithm with a simple change === The ELECT_ME packet can be improved by adding the value of the frequencyEpoch, so that other processes will refuse to vote if the process has a stale information. This may happen when, for example, the process was partitioned away for some time with an old frequency that does not work well as there is too much noise. So in a minority partition, it may try to get elected again and again. The majority probably already switched to a newer frequency. When the partition heals, the process may get elected and change the frequency to something else before having the chance to receive the updated frequency, causing a useless frequency switch. By adding the frequencyEpoch in the ELECT_ME packet and by making other processes checking that the info is updated before providing the vote, we avoid this problem. Other improvements === Another improvement may be to only provide the vote if the current frequency, from the point of view of the receiving node, is *really* noisy. This avoids that a single node having hardware issues in the high bandwidth radio will be able to continuously switch to new frequencies since every frequency will be detected as noisy. In this way a frequency switch can happen only if the majority of the nodes are detecting an issue with the current frequency. Similarly the node sending ELECT_ME messages to get elected may include the frequency it want to switch to, and the receiving node may vote only if the selected frequency passes some test and is considered a good pick, however this may affect the liveness of the algorithm: different nodes may believe that different frequencies are not a good pick so that majority can’t be reached. Conclusions === What I did in this blog post is just to take Raft, that is able to handle the complex problem of replicating a state machine across different processes in a consistent way, and simplify it in order to use it in a subset of problems where the state is a single value (or a set of values) that can just be updated in an idempotent way. The resulting algorithm is trivial to implement in a robust way, and is good enough for a non trivial set of real world problems. --------------------------------------- EDIT: I received some feedback via Twitter, and I think it is better to clarify what is the meaning of the above algorithm. The idea is to retain some safety under the specified scenario, where a replicated state machine is not needed, but still have a reasonable way to take a set of values synchronized across different processes. The goal is to provide, in exchange for the lack of functionality compared to Raft or Paxos, an algorithm that can be recalled by memory only without even reading a document. In this spirit my aim is to further simplify the above description of the algorithm without impacting the functionality. Moreover as somebody that is trying to understand more about distributed programming, I see that while it is very simple without even being aware of it, to get involved in something that is a distributed system, as a normal programmer (given that everything is networked today), descriptions of trivial algorithms may be a way to get somewhat exposed to basic distributed concepts. The next step is to learn a formal analysis tool and try to analyze the algorithm to provide a proof of safety / liveness. Comments

• Simple 'flash' messages in Go
  Alex Edwards · Nov 19, 2013

  Often in web applications you need to temporarily store data in-between requests, such as an error or success message during the Post-Redirect-Get process for a form submission. Frameworks such as Rails and Django have the concept of transient single-use flash messages to help with this. In this post I'm going to look at a way to create your own cookie-based flash messages in Go. We'll start by creating a directory for the project, along with a flash.go file for our code and a main.go file for an example application. $ mkdir flash-example $ cd flash-example $ touch flash.go main.go In order to keep our request handlers nice and clean, we'll create our primary SetFlash() and GetFlash() helper functions in the flash.go file. File: flash.go package main import ( "encoding/base64" "net/http" "time" ) func SetFlash(w http.ResponseWriter, name string, value []byte) { c := &http.Cookie{Name: name, Value: encode(value)} http.SetCookie(w, c) } func GetFlash(w http.ResponseWriter, r *http.Request, name string) ([]byte, error) { c, err := r.Cookie(name) if err != nil { switch err { case http.ErrNoCookie: return nil, nil default: return nil, err } } value, err := decode(c.Value) if err != nil { return nil, err } dc := &http.Cookie{Name: name, MaxAge: -1, Expires: time.Unix(1, 0)} http.SetCookie(w, dc) return value, nil } // ------------------------- func encode(src []byte) string { return base64.URLEncoding.EncodeToString(src) } func decode(src string) ([]byte, error) { return base64.URLEncoding.DecodeString(src) } Our SetFlash() function is pretty succinct. It creates a new Cookie, containing the name of the flash message and the content. You'll notice that we're encoding the content – this is because RFC 6265 is quite strict about the characters cookie values can contain, and encoding to base64 ensures our value satisfies the permitted character set. We then use the SetCookie function to write the cookie to the response. In the GetFlash() helper we use the request.Cookie method to load up the cookie containing the flash message – returning nil if it doesn't exist – and then decode the value from base64 back into a byte array. Because we want a flash message to only be available once, we need to instruct clients to not resend the cookie with future requests. We can do this by setting a new cookie with exactly the same name, with MaxAge set to a negative number and Expiry set to a historical time (to cater for old versions of IE). You should note that Go will only set an expiry time on a cookie if it is after the Unix epoch, so we've set ours for 1 second after that. Let's use these helper functions in a short example: File: main.go package main import ( "fmt" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } func set(w http.ResponseWriter, r *http.Request) { fm := []byte("This is a flashed message!") SetFlash(w, "message", fm) } func get(w http.ResponseWriter, r *http.Request) { fm, err := GetFlash(w, r, "message") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if fm == nil { fmt.Fprint(w, "No flash messages") return } fmt.Fprintf(w, "%s", fm) } Run the application: $ go run main.go flash.go Listening... And make some requests against it using cURL: $ curl -i --cookie-jar cj localhost:3000/set HTTP/1.1 200 OK Set-Cookie: message=VGhpcyBpcyBhIGZsYXNoZWQgbWVzc2FnZSE= Content-Type: text/plain; charset=utf-8 Content-Length: 0 $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Set-Cookie: message=; Expires=Thu, 01 Jan 1970 00:00:01 UTC; Max-Age=0 Content-Type: text/plain; charset=utf-8 Content-Length: 26 This is a flashed message! $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 17 No flash messages You can see our flash message being set, retrieved, and then not passed with subsequent requests as expected. Additional Tools If you don't want to roll your own helpers for flash messages, or need them to be 'signed' to prevent tampering, then the Gorilla Sessions package is a good option. Here's the previous example implemented with Gorilla instead: package main import ( "fmt" "github.com/gorilla/sessions" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } var store = sessions.NewCookieStore([]byte("a-secret-string")) func set(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } session.AddFlash("This is a flashed message!", "message") session.Save(r, w) } func get(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } fm := session.Flashes("message") if fm == nil { fmt.Fprint(w, "No flash messages") return } session.Save(r, w) fmt.Fprintf(w, "%v", fm[0]) } If you found this post useful, you might like to subscribe to my RSS feed.

• Form validation and processing in Go
  Alex Edwards · Nov 01, 2013

  In this post I want to outline a sensible pattern that you can use for validating and processing HTML forms in Go web applications. Over the years I've tried out a number of different approaches, but this is the basic pattern that I always keep coming back to. It's clear and uncomplicated, but also flexible and extensible enough to work well in a wide variety of projects and scenarios. To illustrate the pattern, I'll run through the start-to-finish build of a simple online contact form. So let's begin by creating a new directory for the application, along with a main.go file for our code and a couple of vanilla HTML templates: $ mkdir -p contact-form/templates $ cd contact-form $ touch main.go templates/home.html templates/confirmation.html File: templates/home.html Contact Your email: Your message: File: templates/confirmation.html Confirmation Your message has been sent! If you're following along you'll also need to enable modules in the application root by running the go mod init command like so: $ go mod init contact-form.example.com go: creating new go.mod: module contact-form.example.com Once that's done, your directory structure should look like this: . ├── templates │ ├── confirmation.html │ └── home.html ├── go.mod └── main.go Displaying the Form Our application is going to provide three routes: Method URL Path Handler Description GET / home Display the contact form POST / send Submit the contact form GET /confirmation confirmation Display a confirmation message after successful submission To handle the routing of requests we're going to use bmizerany/pat – but if you want to use an alternative Go router please feel free. Let's go ahead and create a skeleton for the application: File: main.go package main import ( "html/template" "log" "net/http" "github.com/bmizerany/pat" ) func main() { mux := pat.New() mux.Get("/", http.HandlerFunc(home)) mux.Post("/", http.HandlerFunc(send)) mux.Get("/confirmation", http.HandlerFunc(confirmation)) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func home(w http.ResponseWriter, r *http.Request) { render(w, "templates/home.html", nil) } func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form // Step 2: Send message in an email // Step 3: Redirect to confirmation page } func confirmation(w http.ResponseWriter, r *http.Request) { render(w, "templates/confirmation.html", nil) } func render(w http.ResponseWriter, filename string, data interface{}) { tmpl, err := template.ParseFiles(filename) if err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } if err := tmpl.Execute(w, data); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } } This is fairly straightforward stuff so far. The only real point of note is that we've put the template handling into a render function to cut down on boilerplate code. If you run the application: $ go run . 2020/03/30 06:41:42 Listening... And then visit localhost:3000 in your browser you should see the contact form being displayed (although it doesn't do anything yet!). Validating the Form Now for the interesting part. Let's add some validation rules to this contact form, display the validation errors if there are any, and make sure that the form values get presented back if there's an error so the user doesn't need to retype them. We could add the code for this inline in our send handler, but personally I find it cleaner and neater to break out the logic into a separate message.go file: $ touch message.go File: message.go package main import ( "regexp" "strings" ) var rxEmail = regexp.MustCompile(".+@.+\\..+") type Message struct { Email string Content string Errors map[string]string } func (msg *Message) Validate() bool { msg.Errors = make(map[string]string) match := rxEmail.Match([]byte(msg.Email)) if match == false { msg.Errors["Email"] = "Please enter a valid email address" } if strings.TrimSpace(msg.Content) == "" { msg.Errors["Content"] = "Please enter a message" } return len(msg.Errors) == 0 } So what's going on here? We've started by defining a rxEmail variable, containing a simple regular expression for validating the format of the email address in the form. Then we define a Message struct, consisting of Email and Content fields (which will hold the data from the submitted form), along with an Errors map to hold any validation error messages. We then created a Validate() method that acts on a given Message, which checks the format of the email address and makes sure that the content isn't blank. In the event of any errors we add them to the Errors map, and finally return a true or false value to indicate whether validation passed successful or not. In a large project you might want to break the validation checks into helper functions to reduce duplication. This approach means that we can keep the code in our send handler fantastically light. All we need it to do is retrieve the form values from the POST request, create a new Message instance containing them, and call Validate(). If the validation fails we can re-render the contact form, passing back the relevant Message struct. Like so: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send message in an email // Step 3: Redirect to confirmation page } ... As a side note, in the code above we're using the PostFormValue() method on the request to access the POST data. This is a helper method which parses the form data in the request body (using ParseForm()) and returns the value for a specific field. If no matching field exists in the request body, it will return the empty string "". For large request bodies, you might also want to consider using the Gorilla Schema package to automatically decode the form values in to a struct, instead of assigning them manually like we have done in the code above. Anyway, let's now update our home.html template so it displays the validation errors (if they exist) above the relevant fields, and repopulate the form inputs with any information that the user previously typed in: File: templates/home.html .error {color: red;} Contact {{ with .Errors.Email }} {{ . }} {{ end }} Your email: {{ with .Errors.Content }} {{ . }} {{ end }} Your message: {{ .Content }} Let's try this out. Go ahead and run the application: $ go run . 2020/03/30 08:41:42 Listening... And try submitting an invalid form. You should find that the form is redisplayed along with the relevant data and validation errors like so: Sending the Contact Form Message Great! That's now working nicely, but our contact form isn't very useful unless we actually do something with it. Let's add a Deliver() method to our Message which sends the contact form message to a particular email address. In the code below I'm using the go-mail/mail package and a mailtrap.io account for email sending, but the same thing should work with any other SMTP server. File: message.go package main import ( "regexp" "strings" "github.com/go-mail/mail" ) ... func (msg *Message) Deliver() error { email := mail.NewMessage() email.SetHeader("To", "admin@example.com") email.SetHeader("From", "server@example.com") email.SetHeader("Reply-To", msg.Email) email.SetHeader("Subject", "New message via Contact Form") email.SetBody("text/plain", msg.Content) username := "your_username" password := "your_password" return mail.NewDialer("smtp.mailtrap.io", 25, username, password).DialAndSend(email) } The final step is to head back to our main.go file, add some code to call Deliver(), and issue a 303 See Other redirect to the confirmation page that we made earlier: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send contact form message in an email if err := msg.Deliver(); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) return } // Step 3: Redirect to confirmation page http.Redirect(w, r, "/confirmation", http.StatusSeeOther) } ... So long as your SMTP server account credentials are set up correctly, you should now be able to successfully submit the contact form and you should see the confirmation message below in your browser.

• HTTP Response Snippets for Go
  Alex Edwards · Oct 19, 2013

  Taking inspiration from the Rails layouts and rendering guide, I thought it'd be a nice idea to build a snippet collection illustrating some common HTTP responses for Go web applications. Sending Headers Only Rendering Plain Text Rendering JSON Rendering XML Serving a File Rendering a HTML Template Rendering a HTML Template to a String Using Layouts and Nested Templates Sending Headers Only File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "A Go Web Server") w.WriteHeader(200) } $ curl -i localhost:3000 HTTP/1.1 200 OK Server: A Go Web Server Content-Type: text/plain; charset=utf-8 Content-Length: 0 Rendering Plain Text File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 2 OK Rendering JSON File: main.go package main import ( "encoding/json" "net/http" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} js, err := json.Marshal(profile) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/json") w.Write(js) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/json Content-Length: 56 {"Name":"Alex",Hobbies":["snowboarding","programming"]} Rendering XML File: main.go package main import ( "encoding/xml" "net/http" ) type Profile struct { Name string Hobbies []string `xml:"Hobbies>Hobby"` } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} x, err := xml.MarshalIndent(profile, "", " ") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/xml") w.Write(x) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/xml Content-Length: 128 Alex snowboarding programming Serving a File File: main.go package main import ( "net/http" "path" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { // Assuming you want to serve a photo at 'images/foo.png' fp := path.Join("images", "foo.png") http.ServeFile(w, r, fp) } $ curl -I localhost:3000 HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 236717 Content-Type: image/png Last-Modified: Thu, 10 Oct 2013 22:23:26 GMT Rendering a HTML Template File: templates/index.html Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} fp := path.Join("templates", "index.html") tmpl, err := template.ParseFiles(fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 84 Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. Rendering a HTML Template to a String Instead of passing in the http.ResponseWriter when executing your template (like in the above snippet) use a buffer instead: File: main.go ... buf := new(bytes.Buffer) if err := tmpl.Execute(buf, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } templateString := buf.String() ... Using Layouts and Nested Templates File: templates/layout.html {{ template "title" . }} {{ template "content" . }} File: templates/index.html {{ define "title" }}An example layout{{ end }} {{ define "content" }} Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. {{ end }} File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} lp := path.Join("templates", "layout.html") fp := path.Join("templates", "index.html") // Note that the layout file must be the first parameter in ParseFiles tmpl, err := template.ParseFiles(lp, fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 180 An example layout Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. If you found this post useful, you might like to subscribe to my RSS feed.

• Understanding Mutexes
  Alex Edwards · Oct 04, 2013

  For anyone new to building web applications with Go, it's important to realise that all incoming HTTP requests are served in their own Goroutine. This means that any code in or called by your application handlers will be running concurrently, and there is a risk of race conditions occurring. In case you're new to concurrent programming, I'll quickly explain the problem. Race conditions occur when two or more Goroutines try to use a piece of shared data at the same time, but the result of their operations is dependent on the exact order that the scheduler executes their instructions. As an illustration, here's an example where two Goroutines try to add money to a shared bank balance at the same time: InstructionGoroutine 1Goroutine 2Bank Balance 1Read balance ⇐ £50£50 2Read balance ⇐ £50£50 3Add £100 to balance£50 4Add £50 to balance£50 5Write balance ⇒ £150£150 6Write balance ⇒ £100£100 Despite making two separate deposits, only the second one is reflected in the final balance because the two Goroutines were racing each other to make the change. The Go blog describes the downsides: Race conditions are among the most insidious and elusive programming errors. They typically cause erratic and mysterious failures, often long after the code has been deployed to production. While Go's concurrency mechanisms make it easy to write clean concurrent code, they don't prevent race conditions. Care, diligence, and testing are required. Go provides a number of tools to help us avoid data races. These include Channels for communicating data between Goroutines, a Race Detector for monitoring unsynchronized access to memory at runtime, and a variety of 'locking' features in the Atomic and Sync packages. One of these features are Mutual Exclusion locks, or mutexes, which we'll be looking at in the rest of this post. Creating a Basic Mutex Let's create some toy code to mimic the bank balance example: import "strconv" var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { // This is racy c.amount += i } func (c *currency) Display() string { // This is racy return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } We know that if there are multiple Goroutines using this code and calling Balance.Add() and Balance.Display(), then at some point a race condition is likely to occur. One way we could prevent a data race is to ensure that if one Goroutine is using the Balance variable, then all other Goroutines are prevented (or mutually excluded) from using it at the same time. We can do this by creating a Mutex and setting a lock around particular lines of code with it. While one Goroutine holds the lock, all other Goroutines are prevented from executing any lines of code protected by the same mutex, and are forced to wait until the lock is yielded before they can proceed. In practice, it's more simple than it sounds: import ( "strconv" "sync" ) var mu = &sync.Mutex{} var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { mu.Lock() c.amount += i mu.Unlock() } func (c *currency) Display() string { mu.Lock() amt := c.amount mu.Unlock() return strconv.FormatFloat(amt, 'f', 2, 64) + " " + c.code } Here we've created a new mutex and assigned it to mu. We then use mu.Lock() to create a lock immediately before both racy parts of the code, and mu.Unlock() to yield the lock immediately after. There's a couple of things to note: The same mutex variable can be used in multiple places throughout your code. So long as it's the same mutex (in our case mu) then none of the chunks of code protected by it can be executed at the same time. Holding a mutex lock doesn't 'protect' a memory location from being read or updated. A non-mutex-locked line of code could still access it at any time and create a race condition. Therefore you need to be careful to make sure all points in your code which are potentially racy are protected. Let's tidy up the example a bit: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.Mutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.Lock() defer c.Unlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } So what's changed here? Because our mutex is only being used in the context of a currency object, it makes sense to anonymously embed it in the currency struct (an idea borrowed from Andrew Gerrard's excellent 10 things you (probably) don't know about Go slideshow). If you look at a larger codebase with lots of mutexes, like Go's HTTP Server, you can see how this approach helps to keep locking rules nice and clear. We've also made use of the defer statement, which ensures that the mutex gets unlocked immediately before a function returns. This is common practice for functions that contain multiple return statements, or where the return statement itself is racy. Read Write Mutexes In our bank balance example, having a full mutex lock on the Display() function isn't strictly necessary. It would be OK for us to have multiple reads of Balance happening at the same time, so long as nothing is being written. We can achieve this using RWMutex, a reader/writer mutual exclusion lock which allows any number of readers to hold the lock or one writer. Depending on the nature of your application and ratio of reads to writes, this may be more efficient than using a full mutex. Reader locks can be opened and closed with RLock() and RUnlock() like so: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.RWMutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.RLock() defer c.RUnlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } If you found this post useful, you might like to subscribe to my RSS feed.

• Automatic code reloading in Go
  Alex Edwards · Sep 20, 2013

  I wrote a short Bash script to automatically reload Go programs. The script acts as a light wrapper around go run, stopping and restarting it whenever a .go file in your current directory or $GOPATH/src folder is saved. I've been using it mainly when developing web applications, in the same way that I use Shotgun or Guard when working with Ruby. You can grab this from the Github repository. File: go-reload #!/bin/bash # Watch all *.go files in the specified directory # Call the restart function when they are saved function monitor() { inotifywait -q -m -r -e close_write --exclude '[^g][^o]$' $1 | while read line; do restart done } # Terminate and rerun the main Go program function restart { if [ "$(pidof $PROCESS_NAME)" ]; then killall -q -w -9 $PROCESS_NAME fi echo ">> Reloading..." go run $FILE_PATH $ARGS & } # Make sure all background processes get terminated function close { killall -q -w -9 inotifywait exit 0 } trap close INT echo "== Go-reload" echo ">> Watching directories, CTRL+C to stop" FILE_PATH=$1 FILE_NAME=$(basename $FILE_PATH) PROCESS_NAME=${FILE_NAME%%.*} shift ARGS=$@ # Start the main Go program go run $FILE_PATH $ARGS & # Monitor the /src directories in all directories on the GOPATH OIFS="$IFS" IFS=':' for path in $GOPATH do monitor $path/src & done IFS="$OIFS" # Monitor the current directory monitor . Usage The only dependency for this script is inotify-tools, which is used to monitor the filesystem for changes. $ sudo apt-get install inotify-tools Once you've downloaded (or copy-pasted) the script, you'll need to make it executable and move it to /usr/local/bin or another directory on your system path: $ wget https://raw.github.com/alexedwards/go-reload/master/go-reload $ chmod +x go-reload $ sudo mv go-reload /usr/local/bin/ You should then be able to use the go-reload command in place of go run: $ go-reload main.go == Go-reload >> Watching directories, CTRL+C to stop If you found this post useful, you might like to subscribe to my RSS feed.

• An introduction to Handlers and Servemuxes in Go
  Alex Edwards · Sep 12, 2013

  Processing HTTP requests with Go is primarily about two things: handlers and servemuxes. If you’re coming from an MVC-background, you can think of handlers as being a bit like controllers. Generally speaking, they're responsible for carrying out your application logic and writing response headers and bodies. Whereas a servemux (also known as a router) stores a mapping between the predefined URL paths for your application and the corresponding handlers. Usually you have one servemux for your application containing all your routes. Go's net/http package ships with the simple but effective http.ServeMux servemux, plus a few functions to generate common handlers including http.FileServer(), http.NotFoundHandler() and http.RedirectHandler(). Let's take a look at a simple (but slightly contrived!) example which uses these: $ mkdir example $ cd example $ go mod init example.com $ touch main.go File: main.go package main import ( "log" "net/http" ) func main() { // Use the http.NewServeMux() function to create an empty servemux. mux := http.NewServeMux() // Use the http.RedirectHandler() function to create a handler which 307 // redirects all requests it receives to http://example.org. rh := http.RedirectHandler("http://example.org", 307) // Next we use the mux.Handle() function to register this with our new // servemux, so it acts as the handler for all incoming requests with the URL // path /foo. mux.Handle("/foo", rh) log.Print("Listening...") // Then we create a new server and start listening for incoming requests // with the http.ListenAndServe() function, passing in our servemux for it to // match requests against as the second parameter. http.ListenAndServe(":3000", mux) } Go ahead and run the application: $ go run main.go 2021/12/06 15:09:43 Listening... And if you make a request to http://localhost:3000/foo you should find that it gets successfully redirected like so: $ curl -IL localhost:3000/foo HTTP/1.1 307 Temporary Redirect Content-Type: text/html; charset=utf-8 Location: http://example.org Date: Mon, 06 Dec 2021 14:10:18 GMT HTTP/1.1 200 OK Content-Encoding: gzip Accept-Ranges: bytes Age: 254488 Cache-Control: max-age=604800 Content-Type: text/html; charset=UTF-8 Date: Mon, 06 Dec 2021 14:10:18 GMT Etag: "3147526947+gzip" Expires: Mon, 13 Dec 2021 14:10:18 GMT Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT Server: ECS (dcb/7EEF) X-Cache: HIT Content-Length: 648 Whereas all other requests should be met with a 404 Not Found error response. $ curl -IL localhost:3000/bar HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Mon, 06 Dec 2021 14:22:51 GMT Content-Length: 19 Custom handlers The handlers that ship with net/http are useful, but most of the time when building a web application you'll want to use your own custom handlers instead. So how do you do that? The first thing to explain is that anything in Go can be a handler so long as it satisfies the http.Handler interface, which looks like this: type Handler interface { ServeHTTP(ResponseWriter, *Request) } If you're not familiar with interfaces in Go I've written an explanation here, but in simple terms all it means is that a handler must have a ServeHTTP() method with the following signature: ServeHTTP(http.ResponseWriter, *http.Request) To help demonstrate, let's create a custom handler which responds with the current time in a specific format. Like this: type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } The exact code here isn't too important. All that really matters is that we have an object (in this case it's a timeHandler struct, but it could equally be a string or function or anything else), and we've implemented a method with the signature ServeHTTP(http.ResponseWriter, *http.Request) on it. That's all we need to make a handler. Let's try this out in a concrete example: File: main.go package main import ( "log" "net/http" "time" ) type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Initialise the timeHandler in exactly the same way we would any normal // struct. th := timeHandler{format: time.RFC1123} // Like the previous example, we use the mux.Handle() fnction to register // this with our ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Run the application, then go ahead and try making a request to http://localhost:3000/time. You should get a response containing the current time, similar to this: $ curl localhost:3000/time The time is: Mon, 06 Dec 2021 15:33:21 CET Let's step through what's happening here: When our Go server receives an incoming HTTP request it hands it off to our servemux (the one that we passed to the http.ListenAndServe() function). The servemux then looks up the appropriate handler based on the request path (in this case, the /time path maps to our timeHandler handler). The serve mux then calls the ServeHTTP() method of the handler, which in turn writes out the HTTP response. The eagle-eyed of you might have also noticed something interesting: the signature for the http.ListenAndServe() function is ListenAndServe(addr string, handler Handler), but we passed a servemux as the second parameter. We were able to do this because the http.ServeMux type has a ServeHTTP() method, meaning that it too satisfies the http.Handler interface. For me it simplifies things to think of http.ServeMux as just being a special kind of handler, which instead of providing a response itself passes the request on to a second handler. This isn't as much of a leap as it first sounds — chaining handlers together is very commonplace in Go. Functions as handlers For simple cases (like the example above) defining new a custom type just to make a handler feels a bit verbose. Fortunately, we can rewrite the handler as a simple function instead: func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } Now, if you've been following along, you're probably looking at that and wondering: How can that be a handler? It doesn't have a ServeHTTP() method. And you'd be correct. This function itself is not a handler. But we can coerce it into being a handler by converting it to a http.HandlerFunc type. Basically, any function which has the signature func(http.ResponseWriter, *http.Request) can be converted into a http.HandlerFunc type. This is useful because http.HandlerFunc objects come with an inbuilt ServeHTTP() method which — rather cleverly and conveniently — executes the content of the original function. If that sounds confusing, try taking a look at the relevant source code. You'll see that it's a very succinct way of making a function satisfy the http.Handler interface. Let's reproduce the our application using this technique: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Convert the timeHandler function to a http.HandlerFunc type. th := http.HandlerFunc(timeHandler) // And add it to the ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } In fact, converting a function to a http.HandlerFunc type and then adding it to a servemux like this is so common that Go provides a shortcut: the mux.HandleFunc() method. You can use this like so: func main() { mux := http.NewServeMux() mux.HandleFunc("/time", timeHandler) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Passing variables to handlers Most of the time using a function as a handler like this works well. But there is a bit of a limitation when things start getting more complex. You've probably noticed that, unlike the method before, we've had to hardcode the time format in the timeHandler function. What happens when you want to pass information or variables from main() to a handler? A neat approach is to put our handler logic into a closure, and close over the variables we want to use, like this: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { mux := http.NewServeMux() th := timeHandler(time.RFC1123) mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } The timeHandler() function now has a subtly different role. Instead of coercing the function into a handler (like we did previously), we are now using it to return a handler. There's two key elements to making this work. First it creates fn, an anonymous function which accesses — or closes over — the format variable forming a closure. Regardless of what we do with the closure it will always be able to access the variables that are local to the scope it was created in — which in this case means it'll always have access to the format variable. Secondly our closure has the signature func(http.ResponseWriter, *http.Request). As you may remember from a moment ago, this means that we can convert it into a http.HandlerFunc type (so that it satisfies the http.Handler interface). Our timeHandler() function then returns this converted closure. In this example we've just been passing a simple string to a handler. But in a real-world application you could use this method to pass database connection, template map, or any other application-level context. It's a good alternative to using global variables, and has the added benefit of making neat self-contained handlers for testing. You might also see this same pattern written as: func timeHandler(format string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) }) } Or using an implicit conversion to the http.HandlerFunc type on return: func timeHandler(format string) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } } The default servemux You've probably seen the default servemux mentioned in a lot of places, from the simplest Hello World examples to the Go source code. It took me a long time to realise it isn't anything special. The default servemux is just a plain ol' servemux like we've already been using, which gets instantiated by default when the net/http package is used and is stored in a global variable. Here's the relevant line from the Go source: var DefaultServeMux = NewServeMux() Generally speaking, I recommended against using the default servemux because it makes your code less clear and explicit and it poses a security risk. Because it's stored in a global variable, any package is able to access it and register a route — including any third-party packages that your application imports. If one of those third-party packages is compromised, they could use the default servemux to expose a malicious handler to the web. Instead it's better to use your own locally-scoped servemux, like we have been so far. But if you do decide to use the default servemux... The net/http package provides a couple of shortcuts for registering routes with the default servemux: http.Handle() and http.HandleFunc(). These do exactly the same as their namesake functions we've already looked at, with the difference that they add handlers to the default servemux instead of one that you've created. Additionally, http.ListenAndServe() will fall back to using the default servemux if no other handler is provided (that is, the second parameter is set to nil). So as a final step, let's demonstrate how to use the default servemux in our application instead: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { // Note that we skip creating the ServeMux... var format string = time.RFC1123 th := timeHandler(format) // We use http.Handle instead of mux.Handle... http.Handle("/time", th) log.Print("Listening...") // And pass nil as the handler to ListenAndServe. http.ListenAndServe(":3000", nil) }

• Serving static sites with Go
  Alex Edwards · Aug 24, 2013

  I've recently moved the site you're reading right now from a Sinatra/Ruby application to an (almost) static site served by Go. So while it's fresh in my head, here's an explanation of principles behind creating and serving static sites with Go. Let's begin with a simple but real-world example: serving vanilla HTML and CSS files from a particular location on disk. Start by creating a directory to hold the project: $ mkdir static-site $ cd static-site And then add a main.go file to hold our code, and some simple HTML and CSS files in a static directory. $ touch main.go $ mkdir -p static/stylesheets $ touch static/example.html static/stylesheets/main.css File: static/example.html A static page Hello from a static page File: static/stylesheets/main.css body {color: #c0392b} Once those files are created, the code we need to get up and running is wonderfully compact: File: main.go package main import ( "log" "net/http" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/", fs) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Let's step through this. First we use the http.FileServer() function to create a handler which responds to all HTTP requests with the contents of a given file system. For our file system we're using the static directory relative to our application, but you could use any other directory on your machine (or indeed any object that implements the http.FileSystem interface). Next we use the http.Handle() function to register the file server as the handler for all requests, and launch the server listening on port 3000. It's worth pointing out that in Go the pattern "/" matches all request paths, rather than just the empty path. Go ahead and run the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the HTML page we made with a big red heading. Almost-Static Sites If you're creating a lot of static HTML files by hand, it can be tedious to keep repeating boilerplate content. Let's explore using Go's html/template package to put shared markup in a layout file. At the moment all requests are being handled by our file server. Let's make a slight adjustment to our application so the file server only handles request paths that begin with the pattern /static/ instead. File: main.go ... func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Notice that because our static directory is set as the root of the file system, we need to strip off the /static/ prefix from the request path before searching the file system for the given file. We do this using the http.StripPrefix() function. If you restart the application, you should find the CSS file we made earlier available at localhost:3000/static/stylesheets/main.css. Now let's create a templates directory, containing a layout.html file with shared markup, and an example.html file with some page-specific content. $ mkdir templates $ touch templates/layout.html templates/example.html File: templates/layout.html {{define "layout"}} {{template "title"}} {{template "body"}} Made with Go {{end}} File: templates/example.html {{define "title"}}A templated page{{end}} {{define "body"}} Hello from a templated page {{end}} If you've used templating in other web frameworks or languages before, this should hopefully feel familiar. Go templates – in the way we're using them here – are essentially just named text blocks surrounded by {{define}} and {{end}} tags. Templates can be embedded into each other using the {{template}} tag, like we do above where the layout template embeds both the title and body templates. Let's update the application code to use these: File: main.go package main import ( "html/template" "log" "net/http" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) tmpl, _ := template.ParseFiles(lp, fp) tmpl.ExecuteTemplate(w, "layout", nil) } So what's changed here? First we've added the html/template and path packages to the import statement. Then we've specified that all the requests not picked up by the static file server should be handled with a new serveTemplate function (if you were wondering, Go matches patterns based on length, with longer patterns take precedence over shorter ones). In the serveTemplate function, we build paths to the layout file and the template file corresponding with the request. Rather than manual concatenation we use filepath.Join(), which has the advantage joining paths using the correct separator for your OS. Importantly, because the URL path is untrusted user input, we use filepath.Clean() to sanitise the URL path before using it. (Note that even though filepath.Join() automatically runs the joined path through filepath.Clean(), to help prevent directory traversal attacks you need to manually sanitise any untrusted inputs before joining them.) We then use the template.ParseFiles() function to bundle the requested template and layout into a template set. Finally, we use the template.ExecuteTemplate() function to render a named template in the set, in our case the layout template. Restart the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the markup from all the templates merged together like so: If you use web developer tools to inspect the HTTP response, you'll also see that Go automatically sets the correct Content-Type and Content-Length headers for us. Lastly, let's make the code a bit more robust. We should: Send a 404 response if the requested template doesn't exist. Send a 404 response if the requested template path is a directory. Send a 500 response if the template.ParseFiles() or template.ExecuteTemplate() functions throw an error, and log the detailed error message. File: main.go package main import ( "html/template" "log" "net/http" "os" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) // Return a 404 if the template doesn't exist info, err := os.Stat(fp) if err != nil { if os.IsNotExist(err) { http.NotFound(w, r) return } } // Return a 404 if the request is for a directory if info.IsDir() { http.NotFound(w, r) return } tmpl, err := template.ParseFiles(lp, fp) if err != nil { // Log the detailed error log.Print(err.Error()) // Return a generic "Internal Server Error" message http.Error(w, http.StatusText(500), 500) return } err = tmpl.ExecuteTemplate(w, "layout", nil) if err != nil { log.Print(err.Error()) http.Error(w, http.StatusText(500), 500) } }

• A fresh start
  Alex Edwards · Aug 17, 2013

  I've never really known what to do with my personal site. Over the years it's been a dumping ground for links to different projects, and played host to various half-hearted attempts at blogging. But it's never really had much in the way of an actual purpose. I decided to start afresh and relaunch this site with more of a focus. After speaking to the guys from Techzing, I'm going to hunker down and focus my efforts on learning Go really well, with the aim of possibly doing some consultancy work around it in the future. So over the coming months and maybe even years, I hope to create a lot of useful content for anyone else doing the same. Because it's also full redesign of the site, I'll do a little colophon. The site is now just static content, although I use Sass for stylesheets and Markdown for writing blog posts (both of which are compiled on my local machine before publication). Some custom Go code handles the routing and templating, and it's all hosted on Heroku. For development I used Ubuntu as my operating system, Sublime Text as my editor, Git for version control, and Dropbox for real-time backups. So with the mandatory first new post out of the way, I'm looking forward to doing a lot more with this site in the future!

• Go Experiments Explained
  Alex Edwards

  Go often ships with experimental features as part of a release. These experimental features can take different forms: sometimes they're completely new packages in the standard library, sometimes they're changes to the compiler or runtime, or – very occasionally – they can be breaking changes to Go's behavior. Most of the time, the purpose of experimental features is to get real-world feedback from users before something graduates to general availability and becomes a permanent part of Go. If the feature causes regressions, or gets negative feedback from the community, it can be changed before it is finalized – or even abandoned entirely. Some examples Let's look at a few recent examples to illustrate the type of things that Go experiments can cover. Go 1.24 shipped with experimental support for a new testing/synctest package (which provides support for testing concurrent code). After feedback, the package API was adjusted slightly and it graduated to general availability in Go 1.25. Go 1.25 shipped with experimental support for a new garbage collector design with better performance. After incorporating feedback, the new garbage collector became the default in Go 1.26. Go 1.21 shipped with an experimental behavioral change to loop variable semantics. This change closed off a previously common bug with Go code, but was technically a breaking change to the language. Shipping the change as an experiment gave people a chance to test their code before the new behavior became the default in Go 1.22. Experiment lifecycle There isn’t a single fixed lifecycle for experiments, but there are some common patterns. Most experiments initially ship as off-by-default. You explicitly opt-in to try out the feature, usually by setting the GOEXPERIMENT environment value (which we'll talk about more in a moment). If things go well, one or two releases later the experimental feature is finalized, graduates to general availability, and becomes on-by-default. If an experiment affects the behavior of something, then after it graduates to general availability there is sometimes – but not always – a transitional grace period where it's possible to temporarily disable it and use the old behavior. For example, in Go 1.26 the new garbage collector design (which we briefly mentioned above) graduated to general availability and is on-by-default, but it's still possible to disable it and use the old garbage collector if you need to. So that's the most common pattern, but sometimes things take longer or work out differently. For example: Go 1.22 shipped with an experimental implementation of the compiler's inlining logic, which is still off-by-default and under evaluation more than two years later. The same release also shipped with a memory arenas experiment. After negative feedback and concerns from users, it remains off-by-default, is on indefinite hold, and may eventually be removed completely. Or finally, when the Go team is confident in a change, they might skip the feedback stage and go straight to general availability... but there may still be a transitional grace period where it's possible to disable it. A good example of this is when Go 1.24 changed its map implementation to use Swiss tables. The Go team was confident enough in the implementation and its performance benefits for this to go straight to general availability and become on-by-default, but – at least for now – it's still possible to opt out and use the old map implementation if you want to. So in practice there are really three broad experiment states: Off-by-default and under evaluation Off-by-default and on hold/dormant On-by-default with a temporary opt-out Permanent experiments Go also has a handful of experimental features that aren't really “experiments” in the normal sense. These are features that are off-by-default, but they're not under evaluation, not seeking feedback, and there's no expectation that they will ever graduate to general availability and become on-by-default. Although they are controlled by the GOEXPERIMENT environment setting in the same way as other experiments, really they are more like optional Go features that you might want to use in specialist situations. I'll refer to these as "permanent experiments" in the rest of this post. For example there is a field tracking diagnostic feature that tracks which struct fields are accessed. It's been available for a decade, and there's no intention for it to ever graduate to general availability. Or there is a static lock ranking feature, which is a diagnostic for finding potential deadlocks in the Go runtime. What experiments are available right now? It's surprisingly difficult to find out what experimental features are currently available and what their status is. Unfortunately, there isn't a page in the official Go documentation or Go Wiki that tracks experiment status, and for this post I've had to piece together the information from various places. If you want to do the same: You can get a list of all available experiments by running $ go doc goexperiment.Flags. You can figure out which experiments are on-by-default by reading the source code of src/internal/buildcfg/exp.go – specifically looking at the baseline variable declaration in the ParseGOEXPERIMENT() function. You can cross-reference the experiment names with the Go release notes and search through GitHub issues to try to figure out the current status. As far as I can tell, as of Go 1.26 here are the available permanent experiments: Experiment name Description Status FieldTrack Diagnostic to track which struct fields are accessed Off-by-default and permanent fixture StaticLockRanking Diagnostic to validate lock acquisition order to catch deadlocks Off-by-default and permanent fixture CgoCheck2 Diagnostic to check cgo pointer passing rules; too expensive to run by default Off-by-default and permanent fixture BoringCrypto Replaces Go's crypto with FIPS-validated BoringSSL; no longer relevant since Go 1.24 Off-by-default and permanent fixture but will be removed soon PreemptibleLoops Allows scheduler to preempt goroutines at loop back-edges; generally not relevant since Go 1.14, but still may be useful on platforms where preemption is otherwise unsupported Off-by-default and permanent fixture Here are the current off-by-default experiments and their status: Experiment name Description Status HeapMinimum512KiB Reduces minimum heap size from 4MB to 512KiB; may be useful for constrained environments Off-by-default and likely dormant Arenas Memory arena implementation Off-by-default and on hold following negative feedback NewInliner Rewritten compiler inliner with better call-site heuristics Off-by-default and under evaluation (available since Go 1.22) JSONv2 New encoding/json/v2 package with improved JSON encoding/decoding functions Off-by-default and under evaluation (available since Go 1.25) RuntimeSecret New runtime/secret package with functions for zeroing out memory; available on Linux amd64/arm64 only Off-by-default and under evaluation (available since Go 1.26) GoroutineLeakProfile Adds a goroutineleak pprof profile type Off-by-default and under evaluation (available since Go 1.26) SIMD New simd/archsimd package providing access to architecture-specific SIMD operations; only available on amd64 Off-by-default and under evaluation (available since Go 1.26) RuntimeFreegc Allows immediate reuse of memory without waiting for a GC cycle when safe to do so Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) SizeSpecializedMalloc Enables malloc implementations that are specialized per size class Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) And here are the currently on-by-default experiments: Experiment name Description Status LoopVar Per-iteration loop variable scoping On-by-default since Go 1.22, but opt-out kept for edge cases Dwarf5 DWARF 5 debug info generation; reduces binary size On-by-default with a temporary opt-out (opt-out may be removed in a future release) RandomizedHeapBase64 Randomizes the heap base address at startup as a security measure On-by-default with a temporary opt-out (opt-out expected to be removed in a future release) GreenTeaGC New garbage collector with improved performance; unavailable on darwin/ios/aix On-by-default with a temporary opt-out (opt-out expected to be removed in Go 1.27) RegabiWrappers ABI wrappers for calling between ABI0 and ABIInternal functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 RegabiArgs Enables register arguments/results in all compiled Go functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 How do you enable and disable experiments? Experiments are controlled using the GOEXPERIMENT environment setting. If there are some off-by-default experiments you want to try, you should include the experiment names as comma-separated lowercase values in GOEXPERIMENT. For example, if you wanted to build your application with the JSONv2 and GoroutineLeakProfile experiments enabled, you would do so like this: $ GOEXPERIMENT=jsonv2,goroutineleakprofile go build ./... If there is an on-by-default experiment that you want to turn off, you do so by prefixing the lowercase experiment name with no. For example, if you want to build your application with the GreenTeaGC and RandomizedHeapBase64 experiments turned off, you would do so like this: $ GOEXPERIMENT=nogreenteagc,norandomizedheapbase64 go build ./... It's totally fine to mix enabled and disabled experiments: $ GOEXPERIMENT=jsonv2,nogreenteagc go build ./... Note that if you build the same package with different GOEXPERIMENT values, Go treats them as different builds and stores separate entries in the build cache. I've used go build in the examples above, but you can use exactly the same pattern when using go run or go test too. If you want to try it yourself, try creating the following program which uses the experimental encoding/json/v2 package: package main import ( "encoding/json/v2" "fmt" ) type Person struct { Name string `json:"name"` Age int `json:"age"` City string `json:"city"` } func main() { p := Person{Name: "Ada", Age: 36, City: "Vienna"} data, _ := json.Marshal(p, json.StringifyNumbers(true)) fmt.Println(string(data)) } If you run this normally, the program won't compile and you'll get an error message similar to this: $ go run main.go package command-line-arguments imports encoding/json/v2: build constraints exclude all Go files in /usr/local/go/src/encoding/json/v2 But if you enable the JSONv2 experiment, the program will run as expected: $ GOEXPERIMENT=jsonv2 go run main.go {"name":"Ada","age":"36","city":"Vienna"} Which experiments should you actually care about? If you're a run-of-the-mill Gopher like me, who mainly uses Go to write programs rather than working on Go itself, most of the available experiments probably won't be very relevant to you. The most interesting and relevant ones probably are: GreenTeaGC – If you're using Go 1.26, you're already using this by default. But if you notice any performance or behavior problems, it's worth being aware that you can still disable it (and you should also file an issue). Dwarf5 – Again, if you're using Go 1.25 or later then you're already using this by default. But if you run into any problems, it's useful to know that you can still disable it. JSONv2 – I don't recommend switching to this until it graduates to general availability, but if you write a lot of code that deals with JSON, it's worth experimenting with the new encoding/json/v2 package, familiarizing yourself with what's coming, and giving feedback if you notice any problems. GoroutineLeakProfile – This one is immediately useful and worth enabling if you suspect you have a goroutine leak and need to debug it. RuntimeSecret – Worth experimenting with and giving feedback on if you write cryptographic code or need to handle sensitive data. RuntimeFreegc – If you have an application that leans heavily on the garbage collector, it may be worth benchmarking your code with this enabled to see if it improves performance, and giving feedback if you notice any issues. Finally, it's worth emphasizing that experimental features are not covered by the Go compatibility promise. Their APIs, behavior, and performance characteristics may all change, so it's generally a good idea to avoid adopting too early and depending on experimental features before they are finalized. But experimental features often act as a preview to some of the biggest changes in Go. If you know that an experiment is likely to affect you or your code once it eventually becomes generally available and on-by-default, it's a good idea to try it out, run benchmarks where appropriate, and give feedback if you find issues. If you want to keep track of what experiments are available and their status, the Go release notes have recently started doing a much better job of documenting experimental features and how to use them. Between this blog post and browsing the release notes when there's a new Go release, you should have a decent idea of what's going on.

• Demystifying function parameters in Go
  Alex Edwards

  In this post we're going to talk about how (and why!) different types of function parameters behave differently in Go. If you're new (or even not-so-new) to Go, this can be a common source of confusion and questions. Why do functions generally mutate maps and slices, but not other data types? Why isn't my slice being mutated when I append to it in a function? Why doesn't assigning a new value to a pointer parameter have any effect outside the function? Once you understand how functions and the different Go types work, the answers to these kind of questions becomes clearer. You'll discover that Go's behavior consistently follows a few fairly straightforward rules, which I'll aim to highlight in this post. (If you just want the actionable takeaways, you can skip to the summary.) Note: In this post we'll be talking a lot about pointers, so if you're not 100% sure what pointers are, or the terms reference operator and dereference operator don't mean anything to you, then I recommend reading my gentle introduction to pointers tutorial before continuing with this one. Parameters and arguments Before we dive into this post, I'd like to quickly explain the difference between parameters and arguments. People sometimes use these terms interchangeably – but for this tutorial it's important that we're precise on the terminology. Parameters are the variables that you define in a function declaration. Arguments are the values that get passed to the function for execution. (A neat way to remember this is arguments = actual values.) Functions operate on copies of the arguments It's important to understand that when you call a function in Go, the function always operates on a copy of the arguments. That is, the parameters contain a copy of the argument values. We can illustrate this with the following short example: package main import "fmt" func incrementScore(s int) { s += 10 } func main() { score := 20 incrementScore(score) fmt.Println("The score is", score) // Prints: "The score is 20" } When you run this program it will print "The score is 20", not "The score is 30". That's because the parameter s in incrementScore() contains a copy of the score argument, and when we increment the value with s += 10 we are updating this copy, not the original score variable in the main() function. We can confirm this behavior by using the reference operator & to get the memory addresses of the score argument and s parameter, like so: package main import "fmt" func incrementScore(s int) { fmt.Println("has address", &s) // Prints: "has address 0xc000012040" s += 10 } func main() { score := 20 fmt.Println("has address", &score) // Prints: "has address 0xc000012028" incrementScore(score) } If you run this, you'll see that the printed memory addresses are different – in my case 0xc000012028 for the score argument and 0xc000012040 for the parameter s. That confirms that they are truly different variables, with their values stored at different locations in memory. With that in mind, it's not surprising that changing one doesn't change the other. Just to hammer home the point one more time: in Go, functions always operate on a copy of the arguments. There are no exceptions to this. Pointer parameters So, what can we do if we want incrementScore() to actually change the score variable? The answer is to change the signature of incrementScore() so that the parameter s is a pointer, like func incrementScore(s *int). Let's take a look at a working example and then talk it through. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 *s = newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 30" } In this code: We declare the score variable normally in main() with the line score := 20. Then in the line incrementScore(&score) we use the reference operator & to get a pointer to the score variable, and pass this pointer as the argument to incrementScore(). Remember, a pointer just contains a memory address – in this case it's the memory address of the score variable. When incrementScore() is executed, the parameter s contains a copy of this pointer. But this copy still holds the same memory address – the memory address of the score variable. In the line newScore := *s + 10 we use the dereference operator *s to 'read through' and get the underlying value at that memory address, and add ten to it. Then in the next line *s = newScore we use the dereference operator again to 'write through' and set newScore as the value at that memory address. The end result is that we've mutated the value at the memory address of the score variable. So when the program executes the final line of code, we get the output "The score is 30". I should point out that I made the code here a bit more verbose than it needs to be. You can simplify incrementScore() to use the += operator like so: func incrementScore(s *int) { *s += 10 } Write-though vs reassignment In the example above, we used the deference operator *s to read-through and then write-through to the underlying memory address. But what would happen if we didn't write-through, and assigned a completely new pointer value to s instead? Let's take a look. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 s = &newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 20" } If you run this, you'll see we're back to the situation where the score value isn't being mutated, and the program is printing "The score is 20" again. The only thing that's changed here is the body of the incrementScore() function. In this code: The line newScore := *s + 10 is exactly the same as before. It reads through to get the underlying score value from the s parameter, adds ten to it, and assigns the result to the newScore variable. But the line s = &newScore is different. Here we use the reference operator &newScore to get a pointer to the newScore variable, and assign this to s. This means that the variable s no longer contains the memory address of the score variable from main() – it now contains the memory address of the newScore variable. So, in this example, incrementScore() doesn't ever 'write-through' and change anything at the memory address of the score variable. All it does is replace s with a completely different pointer, which is then discarded when the function returns. This is just one example of a more general rule. Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. Automatic dereferencing Let's continue with the same example, but update the incrementScore() function so that it accepts a pointer to a custom player struct, containing the player's name and score. package main import "fmt" type player struct { name string score int } // Make the parameter a pointer to a player struct. func incrementScore(p *player) { p.score += 10 } func main() { // Initialize a player struct and assign it to the variable p1. p1 := player{name: "Alice", score: 20} // Pass a pointer to p1 to incrementScore(). incrementScore(&p1) fmt.Printf("The score for %s is %d", p1.name, p1.score) // Prints: "The score for Alice is 30" } So as you might expect, because the parameter p in incrementScore() is a pointer, the changes that we make to p affect the data at the underlying memory address of p1 and the program prints "The score for Alice is 30". But the most interesting part here is the line of code p.score += 10 in the incrementScore() function. p is a pointer, but we appears that we don't have to dereference it using the * operator in order to write-through the new value. You could – if you wanted to – change this line to be (*p).score += 10. That's perfectly valid and will compile fine. But it's not necessary. If you have a pointer to a struct (which is what the p parameter is here), then Go will automatically dereference the pointer for you when you use the dot operator . on it to access a field or call a method. You can also use index expressions on a pointer to an array without dereferencing it. (Note that this will only work on arrays, not slices). For example: a := &[3]string{"a", "b", "c"} fmt.Println(a[1]) // Instead of having to write (*a)[1] "Reference types" Everything we've illustrated in this tutorial so far is true when the parameter type is a basic type, a struct, an array, a function, or a pointer to any of those things. However, the behavior that you get when a parameter is a map, slice or channel type needs some further discussion. Once you realize how these types are implemented at runtime behind the scenes, you'll see that their behavior actually follows the same rules as the other Go types – but nonetheless it can be a bit confusing at first. If you've been programming for a while, you might be familiar with the terms pass-by-value and pass-by-reference from other languages. You might have also heard or read people in the Go world saying things like "maps, slices and channels are reference types", or "maps, slices and channels are passed by reference". Well... the sentiment there is sort of right, but the wording isn't correct and needs tightening up. Firstly, Go does not support pass-by-reference behavior. I've probably banged this drum enough already now, but parameters are always a copy of the arguments. That is, they are always passed by value. Even pointers are passed by value; a pointer parameter will contain a copy of the pointer. Strictly speaking, there's also no such group of things in Go known as "reference types". To be fair, the Go spec did use "reference types" as an umbrella term for maps, slices and channels in one sentence, but this was removed over a decade ago (with the commit message Go has no 'reference types'). Basically, I recommend forgetting hearing the term "reference types" in relation to Go, and replacing it with an understanding of how maps, channels and slices are actually implemented. Maps and Channels The important thing to understand is that behind-the-scenes when your code is running, the Go runtime implements a map as a pointer to a runtime.hmap struct, and a channel as a pointer to a runtime.hchan struct. This means that map and channel parameters behave in a similar way to regular pointer parameters. The parameter will contain a copy of the map or channel, but this copy will still point to the same underlying memory location that holds the runtime.hmap or runtime.hchan struct. In turn, that means that any changes you make to a map or channel parameter will also mutate the argument. Let's look at an example, where we create a scores map containing the names and scores for multiple players like map[string]int{"Alice": 20, "Bob": 160}, and then pass it to a function that increments the score by ten for all players. package main import "fmt" func incrementAllScores(sm map[string]int) { for name := range sm { sm[name] += 10 } } func main() { scores := map[string]int{"Alice": 20, "Bob": 160} incrementAllScores(scores) fmt.Println(scores) // Prints: map[Alice:30 Bob:170] } When you run this, you'll see that incrementAllScores() mutates the scores argument and the program prints map[Alice:30 Bob:170] as the output. Because of this behavior, you normally won't need to use a pointer to a map or channel as a function parameter. On the other hand, if you don't want a function to mutate a map, you can use the maps.Clone() function to create a clone that points to a different memory location, and work on the clone instead. func example(m map[string]int) { cm := maps.Clone(sm) // ... do something with the cloned map. } Note: Although the Go runtime implements maps and channels as pointers to internal structures, this is a runtime implementation detail. Maps and channels are their own concrete types as far as the compiler is concerned, they are not pointers, and you can't do things like dereferencing a map or channel type like you would a pointer. Slices How slices work behind the scenes in Go can be pretty difficult to grok, and if you'd like a detailed explanation I recommend reading the Go Slices: usage and internals post on the official blog. But as a high-level summary, the Go runtime implements slices as a runtime.slice struct. This struct wraps a pointer to a (fixed-size) array that actually stores the slice data. You can think of a slice as being a bit like a 'window through' to a segment of this underlying array. So when you have a function with a slice parameter, the parameter will contain a copy of the slice argument you pass it. Effectively, it will have a copy of the runtime.slice struct. But the pointer in this copy of runtime.slice will still point to the same underlying array, meaning that any changes you make to a slice parameter will also mutate the argument. To demonstrate this, let's say that we have a slice containing some player scores, and pass it to a addBonus() function that adds fifty to each score in the slice. package main import "fmt" func addBonus(s []int) { for i := range s { s[i] += 50 } } func main() { scores := []int{10, 20, 30} addBonus(scores) fmt.Println(scores) // Prints: [60 70 80] } When you run this code it will print [60 70 80], demonstrating that the changes made in addBonus() mutated the elements in the scores slice. If you don't want a function to mutate a slice, you can make a clone of it using slices.Clone() and work on that instead. func example(s []int) { cs := slices.Clone(s) // ... do something with the cloned slice. } So far, so good. Slices generally behave pretty much like maps and channels, in the sense that changing a slice parameter will mutate the argument. If you don't want that, you can make a clone at the start of the function and use the clone instead. But using append() on a slice parameter can sometimes be a source of confusion. Consider the following code, where we create a variadic addScores() function that appends some new values to a scores slice. package main import "fmt" func addScores(s []int, values ...int) { s = append(s, values...) } func main() { scores := []int{10, 20, 30} addScores(scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30] } (Yes, this is a bit of a silly example, but it illustrates the point in a simple way.) When you run this program, it will print out [10 20 30] – demonstrating that the append() operation in addScores() has not affected the scores argument. This actually makes sense and is consistent with the other behavior we've seen in this post. Earlier on I said: Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. The code s = append(s, values...) is no different. We're replacing the s parameter with a new value, and this operation doesn't touch the argument in any way. Note: As a slight aside, the slice returned by append() may or may not point to the same underlying array as the original slice that you're appending too. It all depends on whether the underlying array has enough capacity to store the new values or not. If a new underlying array needs to be reallocated, the pointer in the slice returned by append() will be different. If it doesn't, then the pointer will remain the same and point to the same underlying array. When it comes to slices as a function parameter, this means that if you change the elements in a slice parameter after a call to append(), the change may or may not mutate the argument. It all depends on whether the append() operation resulted in a new underlying array being allocated or not. You can see an example of this behavior here. So what about when you want an append() operation in a function to mutate the argument? The answer here is to make the parameter a pointer to a slice, like so: package main import "fmt" func addScores(s *[]int, values ...int) { *s = append(*s, values...) } func main() { scores := []int{10, 20, 30} addScores(&scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30 40 50 60] } Now with the line *s = append(*s, values...), whatever is returned by the append() function will be 'written-through' to the memory address of the scores argument. Exactly the same logic applies for operations to 'reslice' a slice and assign the result back to the parameter, like s = s[0:1]. If you want this operation to mutate the argument, you should make the parameter a pointer and dereference it like *s = (*s)[0:1]. Summary We've covered a lot of ground in this post, so I'll try to summarize everything into a handful of take-away points. Parameters always contain a copy of the argument. Go doesn't have "reference types" or support pass-by-reference semantics. For the basic Go types, as well as structs, arrays and functions, changing the value of a parameter in the function body won’t change the value of the argument. But if you do want to mutate the argument, you can use a pointer parameter instead and dereference it inside the function to ‘write-through’ a new value to the argument's memory address. For common operations on structs and arrays, Go will automatically dereference the pointer for you. Because of the way that they're implemented by the Go runtime, changes you make to map, slice, channel parameters in a function will mutate the argument. If you don't want this, make a clone at the start of the function and use that instead. Using the = operator to assign a new value to a parameter does not affect the argument (unless you are manually-or-automatically dereferencing a pointer and 'writing-through' a new value). So for slices, if you want a function to perform an append or reslice operation that mutates the argument, you should use a pointer to a slice as the function parameter and dereference it as necessary.

• A time-saving Makefile for your Go projects
  Alex Edwards

  Whenever I start a new Go project, one of the first things I do is create a Makefile in the root of my project directory. This Makefile serves two purposes. The first is to automate common admin tasks (like running tests, checking for vulnerabilities, pushing changes to a remote repository, and deploying to production), and the second is to provide short aliases for Go commands that are long or difficult to remember. I find that it's a simple way to save time and mental overhead, as well as helping me to catch potential problems early and keep my codebases in good shape. While the exact contents of the Makefile changes from project to project, in this post, I want to share the boilerplate that I'm currently using as a starting point. It's generic enough that you should be able to use it as-is for almost all projects. Note: You can also find the Makefile code in this Gist. File: Makefile # Change these variables as necessary. main_package_path = ./cmd/example binary_name = example # ==================================================================================== # # HELPERS # ==================================================================================== # ## help: print this help message .PHONY: help help: @echo 'Usage:' @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /' .PHONY: confirm confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] .PHONY: no-dirty no-dirty: @test -z "$(shell git status --porcelain)" # ==================================================================================== # # QUALITY CONTROL # ==================================================================================== # ## audit: run quality control checks .PHONY: audit audit: test go mod tidy -diff go mod verify test -z "$(shell gofmt -l .)" go vet ./... go run honnef.co/go/tools/cmd/staticcheck@latest -checks=all,-ST1000,-U1000 ./... go run golang.org/x/vuln/cmd/govulncheck@latest ./... ## test: run all tests .PHONY: test test: go test -v -race -buildvcs ./... ## test/cover: run all tests and display coverage .PHONY: test/cover test/cover: go test -v -race -buildvcs -coverprofile=/tmp/coverage.out ./... go tool cover -html=/tmp/coverage.out ## upgradeable: list direct dependencies that have upgrades available .PHONY: upgradeable upgradeable: @go run github.com/oligot/go-mod-upgrade@latest # ==================================================================================== # # DEVELOPMENT # ==================================================================================== # ## tidy: tidy modfiles and format .go files .PHONY: tidy tidy: go mod tidy -v go fmt ./... ## build: build the application .PHONY: build build: # Include additional build steps, like TypeScript, SCSS or Tailwind compilation here... go build -o=/tmp/bin/${binary_name} ${main_package_path} ## run: run the application .PHONY: run run: build /tmp/bin/${binary_name} ## run/live: run the application with reloading on file changes .PHONY: run/live run/live: go run github.com/cosmtrek/air@v1.43.0 \ --build.cmd "make build" --build.bin "/tmp/bin/${binary_name}" --build.delay "100" \ --build.exclude_dir "" \ --build.include_ext "go, tpl, tmpl, html, css, scss, js, ts, sql, jpeg, jpg, gif, png, bmp, svg, webp, ico" \ --misc.clean_on_exit "true" # ==================================================================================== # # OPERATIONS # ==================================================================================== # ## push: push changes to the remote Git repository .PHONY: push push: confirm audit no-dirty git push ## production/deploy: deploy the application to production .PHONY: production/deploy production/deploy: confirm audit no-dirty GOOS=linux GOARCH=amd64 go build -ldflags='-s' -o=/tmp/bin/linux_amd64/${binary_name} ${main_package_path} upx -5 /tmp/bin/linux_amd64/${binary_name} # Include additional deployment steps here... The Makefile is organized into several sections, each with its own set of targets: 1. HELPERS help: Prints a help message for the Makefile, including a list of available targets and their descriptions. confirm: Prompts the user to confirm an action with a "y/N" prompt. no-dirty: Checks that there there are no untracked files or uncommitted changes to the tracked files in the current git repository. 2. QUALITY CONTROL audit: Runs quality control checks on the codebase, including using go mod tidy -diff to check that the go.mod and go.sum files are up-to-date and correctly formatted, verifying the dependencies with go mod verify, running test -z "$(shell gofmt -l .)" to check that all .go files are correctly formatted, running static analysis with go vet and staticcheck, checking for vulnerabilities using govulncheck, and running all tests. Note that it uses go run to execute the latest versions of the remote staticcheck and govulncheck packages, meaning that you don't need to install these tools first. I've written more about this pattern in a previous post. test: Runs all tests. Note that we enable the race detector and embed build info in the test binary. test/cover: Runs all tests and outputs a coverage report in HTML format. upgradeable: List all direct module dependencies that have a newer version available, using the oligot/go-mod-upgrade tool. 3. DEVELOPMENT tidy: Updates the dependencies and formats the go.mod and go.sum using go mod tidy, and formats all .go files using go fmt. build: Builds the package at main_package_path and outputs a binary at /tmp/bin/{binary_name}. run: Calls the build target and then runs the binary. Note that my main reason for not using go run here is that go run doesn't embed build info in the binary. run/live: Use the air tool to run the application with live reloading enabled. When changes are made to any files with the specified extensions, the application is rebuilt and the binary is re-run. Depending on the project I often add more to this section, such as targets for connecting to a development database instance and managing SQL migrations. Here's an example. 4. OPERATIONS push: Push changes to the remote Git repository. This asks for y/N confirmation first, and automatically runs the audit and no-dirty targets to make sure that all audit checks are passing and there are no uncommitted changes in the repository before the push is executed. production/deploy: Builds the a binary for linux/amd64 architecture, compress it using upx, and then run any deployment steps. Note that this target asks for y/N confirmation before anything is executed, and also runs the audit and no-dirty checks too. Depending on the project I often add more to this section too. For example, a staging/deploy rule for deploying to a staging server, production/connect for SSHing into a production server, production/log for viewing production logs, production/db for connecting to the production database, and production/upgrade for updating and upgrading software on a production server. Usage Each of these targets can be executed by running make followed by the target name in your terminal. For example: $ make tidy go mod tidy -v go fmt ./... If you run make help (or the naked make command without specifiying a target) then you'll get a description of the available targets. $ make help Usage: help print this help message tidy tidy modfiles and format .go files audit run quality control checks test run all tests test/cover run all tests and display coverage build build the application run run the application run/live run the application with reloading on file changes push push changes to the remote Git repository production/deploy deploy the application to production

• How to use the http.ResponseController type
  Alex Edwards

  One of my favorite things about the recent Go 1.20 release is the new http.ResponseController type, which brings with it three nice benefits: You can now override your server-wide read and write deadlines on a per request basis. The pattern for using the http.Flusher and http.Hijacker interfaces is clearer and feels less hacky. No more type assertions necessary! It makes it easier and safer to create and use custom http.ResponseWriter implementations. The first two benefits are mentioned in the release notes, but the third one seems to have gone under the radar a bit... which is a shame, because it's very helpful! Let's dive in a take a look. Per-request deadlines Go's http.Server has ReadTimeout and WriteTimeout settings, which you can use to automatically close a HTTP connection if reading a request or writing response takes longer than a fixed amount of time. These settings are server-wide and apply to all requests, irrespective of the handler or URL. With http.ResponseController you can now use the SetReadDeadline() and SetWriteDeadline() methods to relax or tighten these settings on a per-request basis if you need too. For example: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) // Set a write deadline in 5 seconds time. err := rc.SetWriteDeadline(time.Now().Add(5 * time.Second)) if err != nil { // Handle error } // Do something... // Write the response as normal. w.Write([]byte("Done!")) } This is particularly helpful in an application where you have a small number of handlers that need longer deadlines than all the others, for things like processing a file upload or carrying out a long-running operation. A few other details to mention: If you set a short server-wide deadline, and that deadline is hit before you call SetWriteDeadline() or SetReadDeadline() then they will have no effect. The server-wide deadline wins. If your underlying http.ResponseWriter doesn't support setting per-request deadlines, then calling SetWriteDeadline() or SetReadDeadline() will return a http.ErrNotSupported error. You can effectively remove the server-wide deadline on a per-request basis by passing a zero-valued time.Time struct to SetWriteDeadline() or SetReadDeadline(). For example: rc := http.NewResponseController(w) err := rc.SetWriteDeadline(time.Time{}) if err != nil { // Handle error } Flusher and Hijacker interfaces The http.ResponseController type also makes it slightly nicer to use the 'optional' http.Flusher and http.Hijacker interfaces. For example, before Go 1.20 you would use a code pattern like this this to flush response data to the client: func exampleHandler(w http.ResponseWriter, r *http.Request) { f, ok := w.(http.Flusher) if !ok { // Handle error } for i := 0; i Now you can do this: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) for i := 0; i The pattern for hijacking a connection is similar: func (app *application) home(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) conn, bufrw, err := rc.Hijack() if err != nil { // Handle error } defer conn.Close() // Do something... } Again, if your underlying http.ResponseWriter doesn't support support flushing or hijacking, then calling Flush() or Hijack() on a http.ResponseController will also return an http.ErrNotSupported error. Custom http.ResponseWriters It's now also easier and safer to create and use custom http.ResponseWriter implementations that still support flushing and hijacking. It's probably easiest to explain how this works with an example, so let's look at the code for a custom http.ResponseWriter implementation that records the HTTP status code of a response. type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } So here we've defined a custom statusResponseWriter type, which embeds an existing http.ResponseWriter and implements custom WriteHeader() and Write() methods to support the recording of the HTTP response status code. But the important thing to notice here is the Unwrap() method at the end, which returns the original embedded http.ResponseWriter. When you use the new http.ResponseController type to to flush, hijack or set a deadline, it will call this Unwrap() method to access the original http.ResponseWriter. This is done recursively if necessary, so you can potentially layer multiple custom http.ResponseWriter implementations on top of each other. Let's look at a complete example, where we use this statusResponseWriter in conjunction with some middleware to log response status codes, along with a handler that sends a 'normal' response and another that uses the new http.ResponseController type to send a flushed response. package main import ( "log" "net/http" "time" ) type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } func main() { mux := http.NewServeMux() mux.HandleFunc("/normal", normalHandler) mux.HandleFunc("/flushed", flushedHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", logResponse(mux)) if err != nil { log.Fatal(err) } } func logResponse(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { sw := newstatusResponseWriter(w) next.ServeHTTP(sw, r) log.Printf("%s %s: status %d\n", r.Method, r.URL.Path, sw.statusCode) }) } func normalHandler(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusTeapot) w.Write([]byte("OK")) } func flushedHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) w.Write([]byte("Write A....")) err := rc.Flush() if err != nil { log.Println(err) return } time.Sleep(time.Second) w.Write([]byte("Write B....")) err = rc.Flush() if err != nil { log.Println(err) } } If you want, you can run this and try making requests to the /normal and /flushed endpoints: $ curl http://localhost:3000/normal OK $ curl --no-buffer http://localhost:3000/flushed Write A....Write B.... You should see the response from the flushedHandler in two parts, first the Write A... part, then followed a second later by the Write B... part. And you should see that the statusResponseWriter and logResponse middleware have successfully written log messages, including the correct HTTP status code for each response. $ go run main.go 2023/03/06 21:41:21 Listening... 2023/03/06 21:41:32 GET /normal: status 418 2023/03/06 21:41:44 GET /flushed: status 200

• An introduction to Packages, Imports and Modules in Go
  Alex Edwards

  This tutorial is written for anyone who is new to Go. In it we'll explain what packages, import statements and modules are in Go, how they work and relate to each other and — hopefully — clear up any questions that you have. We'll start at a high level, then work down to the details later. There's quite a lot of content in this tutorial, so I've broken it down into the following eight sections: Packages The main package Importing and using standard library packages Unused and missing imports Exported vs unexported Modules Using multiple packages in your code Importing and using third-party packages Organizing import statements To help illustrate things throughout this post we'll build a small CLI (command-line interface) application which generates and prints out a random 'lucky number'. If you'd like to follow along, run the following commands: $ mkdir lucky-number $ cd lucky-number $ touch main.go Then add the following code to the main.go file: File: main.go package main import ( "fmt" "math/rand" ) func main() { // Get a random number between 0 and 99 inclusive. n := rand.Intn(100) // Print it out. fmt.Printf("Your lucky number is %d!\n", n) } At this point you should be able to run the application and see some output like this: $ go run main.go Your lucky number is 81! Packages A package in Go is essentially a named collection of one or more related .go files. In Go, the primary purpose of packages is to help you isolate and reuse code. Every .go file that you write should begin with a package {name} statement which indicates the name of the package that the file is a part of. For example, in the 'lucky number' code above, the package main line declares that the main.go file is part of a package named main. At the moment: Our 'lucky number' application consists of one package, with the package name main. The main package is made up of one file, with the filename main.go. It's important to explain that code in a package can access and use all types, constants, variables and functions within that package — even if they are declared in a different .go file. Let's illustrate this by splitting our 'lucky number' code across two files. Go ahead and add an additional random.go file: $ touch random.go Then update the two files so that the main() function calls a new randomNumber() function, like so: File: random.go package main import ( "math/rand" ) func randomNumber() int { return rand.Intn(100) } File: main.go package main import ( "fmt" ) func main() { fmt.Printf("Your lucky number is %d!\n", randomNumber()) } So now: Our 'lucky number' application consists of two .go files. Both files are part of the main package (because they both start with a package main statement). If you re-run the application using the two files, you should see the same output. $ go run *.go Your lucky number is 81! Note: If your terminal doesn't support wildcard expansion, you'll need to list the files explicitly and run the command $ go run main.go random.go instead. This example is a bit contrived but it illustrates the point nicely — our main() function is able to call our randomNumber() function because they are part of the same package — despite being in separate .go files. It's totally OK to have quite a lot of .go files in the same package. Having 5, 10 or even 20 files — and thousands of lines of code — in the same package is not uncommon or an anti-pattern in Go. The main package In Go, main is actually a special package name which indicates that the package contains the code for an executable application. That is, it indicates that the package contains code that can be built into a binary and run. Any package with the name main must also contain a main() function somewhere in the package which acts as the entry point for the program. If it doesn't, and you try to run it, you will get this error: $ go run *.go function main is undeclared in the main package It's conventional for your main() function to live in a file with the filename main.go. Technically it doesn't have to, but following this convention makes the application entry point easier to find for anyone reading your code in the future. As an aside, if you try to build or run a non-main package it will also result in an error. For example, if you changed the 'lucky number' code so that the package name is foo instead of main and try to run it, you will get the following (somewhat confusing) error: $ go run *.go package command-line-arguments is not a main package Importing and using standard library packages I'm sure you know this already, but individual .go files can import and use exported types, constants, variables and functions from other packages — including the packages in the Go standard library. The complete tree of Go standard library packages is available here. In our 'lucky number' code we've imported and used the math/rand and fmt packages from the standard library to help us generate a random number and print a message. For example, in the random.go file: File: random.go package main import ( "math/rand" // Import the math/rand package. ) func randomNumber() int { return rand.Intn(100) // Call the Intn() function from the math/rand package. } When importing a package from the standard library you need to use the full path to the package in the standard library tree, not just the name of the package. For example: import ( "fmt" "math/rand" // Not "rand" "net/http" // Not "http" "net/http/httptest" // Not "httptest" ) Once imported, the package name becomes an accessor for the contents of that package. Conveniently, all the packages in the Go standard library have a package name which is the same as the final element of their import path. That means we can use the Intn() function from math/rand by calling rand.Intn(), or the Printf() function from fmt by calling fmt.Printf(). As well as importing packages from the standard library it's possible to import your own packages or third-party packages too. We'll get to that shortly. Unused and missing imports If you import a package but don't actually use it in your code, it will result in a compile-time error. For example, if you import the os package but don't use it you will get an error like: "os" imported and not used Similarly, you'll also get a compile-time error if a package is referenced in your code but not imported. For example, if you try to use the strconv package without importing it you'll get an error like this: undefined: strconv When you're developing rapidly it can sometimes be annoying to keep editing your import statements, but ultimately this behavior helps to keep your code correct and your import list clean and accurate. Tip: You can use the goimports tool to automatically add and remove import statements in your .go files. It's also possible to integrate this with many popular text editors (including VSCode, Emacs and Sublime), so that import statements are updated for you whenever you save a file. But you should be careful if you are using one of the rand or template packages in your code — these standard library package names are ambiguous and you should always check that goimports has added the one that you want (for example, that it has imported html/template instead of text/template, or crypto/rand instead of math/rand). Exported vs unexported Earlier in this tutorial I said: Individual .go files can import and use exported types, constants, variables, functions and methods from other packages — including the packages in the Go standard library. So what does exported mean? Essentially, something in Go code is exported if its name starts with a capital letter. Otherwise it is unexported. For example: var fooBaz string // This is an unexported variable. var FooBar string // This is an exported variable. func fooBaz() {...} // This is an unexported function. func FooBar() {...} // This is an exported function. type fooBaz struct {...} // This is an unexported type. type FooBar struct {...} // This is an exported type. The difference between them is: Unexported things are 'private' to the package that they are declared in. They are only visible to code in the same package. In contrast, exported things in a package are 'public' and are visible to any code that that imports the package. In other words: when you import a package you get to use its exported things, but not its unexported things. Depending on your programming background, capitalization might seem like a funny way to control visibility. But once you get used to it, it has some positives. It's simple, doesn't require you to remember any additional syntax, and it's trivial to see at a glance whether something is exported or not — even when that thing is being used far away from where it is declared. Tips: Generally don't export things unless you actually have a reason to (i.e. don't capitalize a name just because it looks nicer!). Additionally, a main package should never normally be imported by anything, so it probably shouldn't have any exported things in it. Modules If you have a small application which only imports packages from the standard library, then what we've done so far works just fine. But if you want to import and use a third-party package — or structure your code so it's split into multiple packages — then you first need to turn your code into a Go module. The Go Wiki defines modules like this: A module is... a tree of Go source files with a go.mod file in the tree's root directory. In our example the lucky-number directory already contains our two .go files, so all we need to do is add a valid go.mod file to the directory to make it a module. The easiest way to do this is by running the go mod init command and passing in a module path as the final argument, like so: $ go mod init lucky-number.alexedwards.net go: creating new go.mod: module lucky-number.alexedwards.net go: to add module requirements and sums: go mod tidy Before we go further, let's talk about module paths. The module path act as a canonical identifier for a module. Ideally it should be unique and something that is unlikely to be used by anyone else, in any other project. In the command above I've used lucky-number.alexedwards.net as the module path, but it could be (almost) any string value. In the Go community it's conventional to base your module path on a URL that you own or control. So, for this example, a good module path would be something like lucky-number.alexedwards.net or github.com/alexedwards/lucky-number. Important: In most cases, your module path doesn't need to be a 'real' functioning URL with something hosted at it. It's really just an arbitrary string which acts as a unique identifier for your module. But… if you plan to make your code available for reuse (e.g. as an open source package) then your module path must be the location that the code will be fetchable from. So, for example, if you're planning to host the code at github.com/example/package the module path should also be github.com/example/package. OK, let's take a look at the go.mod file that was generated for us: File: go.mod module lucky-number.alexedwards.net go 1.19 We can see that (for now) all this does is declare the module path, along with the version of Go that you are using. We'll revisit this file again later when we talk about using third-party Go packages. Tip: If you're ever looking at some code and want to know what it's module path is, just take a look in its go.mod file. So at this point in the tutorial: The code in the lucky-number directory is now a Go module. The Go module has the module path lucky-number.alexedwards.net. The module contains one main package, which is made up of our main.go and random.go files. Using multiple packages in your code Let's make our 'lucky number' application structure a bit more complex and split up the code into two packages. Before we get started on this change there are a couple of rules and conventions to be aware of: In Go, one package == one directory. That is, all .go files for a package should be contained in the same directory, and a directory should contain the .go files for one package only. You shouldn't ever have .go files with different package names in the same directory. For all non-main packages, the directory name that the code lives in should be the same as the package name. When choosing a name you should pick something that is short, descriptive, lower case and ideally one word. The Go blog has a helpful post with additional guidance and some examples of good and bad names. With those things in mind, let's restructure our 'lucky number' application so that the code for generating the random number is isolated in a new, separate, package called random. $ rm random.go $ mkdir random $ touch random/number.go The file tree for the lucky-number directory should now look like this: $ tree --dirsfirst . ├── random │ └── number.go ├── go.mod └── main.go The important thing to point out is that all the .go files are still part of the same module — they are all part of a file tree with a single go.mod file in the root directory of the tree. OK, let's go ahead and add the following code to the new random/number.go file: File: random/number.go package random import ( "math/rand" ) func Number() int { return rand.Intn(100) } There are four things I'd like to quickly highlight and re-iterate here: The number.go file is part of the random package (notice the statement in the first line). The Number() function is exported (i.e. its name begins with a capital letter). This means it will be visible to any code which imports the random package. The directory name that the code lives in is exactly the same as the package name (random) . The random package is part of the lucky-number.alexedwards.net module. Next let's update our main.go file to import and use the new package. Like so: File: main.go package main import ( "fmt" // Import the random package. "lucky-number.alexedwards.net/random" ) func main() { // Call the random.Number() function to get the random number. Notice that // we use the package name as the accessor, just like we do for the standard // library packages. fmt.Printf("Your lucky number is %d!\n", random.Number()) } The most interesting thing about this is the import path for our new package. When you are importing packages that are part of the same module as your current .go file, the import statement should take the form: import {module path}/{path to the package relative to your go.mod file} So in this case, the module path is lucky-number.alexedwards.net and the path within the module for the package is random, giving us an import path of lucky-number.alexedwards.net/random. Before we go further, I'd like to point out that I'm making this code structure more complicated than it needs to be (just to illustrate things for teaching purposes). There's no real reason here to have split the code into two packages. In fact, overusing packages is a common mistake that newcomers to Go make. Generally you should only split code into additional packages if you have a demonstrable reason to, such as: You want a convenient way to reuse it the code, or to make it available for reuse. You want to isolate or enforce some boundary between the package code and the rest of your codebase. You have some complex code that acts as a 'black box' and moving it to a standalone package will reduce cognitive overhead when working with the rest of your code. A more complex structure Let's tweak the directory structure of our 'lucky number' code a bit more. We'll: Move the main package files into a new cmd/cli directory. Move the random package files into a new internal/random directory. (Again, this is just for teaching purposes. This structure isn't actually necessary for such a small and simple application.) $ mkdir -p cmd/cli internal $ mv main.go cmd/cli/ $ mv random internal/ $ tree --dirsfirst . ├── cmd │ └── cli │ └── main.go ├── internal │ └── random │ └── number.go └── go.mod Once that's done, let's update the cmd/cli/main.go file so that the random package is imported from its new location. Like so: File: cmd/cli/main.go package main import ( "fmt" // Import the random package using the new location under the // `internal` directory. "lucky-number.alexedwards.net/internal/random" ) func main() { fmt.Printf("Your lucky number is %d!\n", random.Number()) } You should now be able to run the application by calling go run with the path to the main package. Like this: $ go run ./cmd/cli Your lucky number is 81! This change helps to illustrate a couple of things: It's not necessary for a main package to live in the module root. It can be anywhere. In fact, it's totally OK for a module to contain multiple main packages. For example, in a larger project you could have a cmd/cli directory with the main package for a CLI tool, and a cmd/web directory with the main package for a web application in the same module. Your non-main packages don't need to be a direct child of the module root either. They can be anywhere in an arbitrarily deep directory structure within the module. Note: The directory name internal has a special behavior in Go. Any packages which live under a directory called internal can only be imported by code inside the parent of the internal directory. In this example, it means that any packages nested under internal can only be imported by code inside our lucky-number directory. Or, looking at it the other way, any packages under internal cannot be imported by code outside of the lucky-number directory. This is useful because it prevents other codebases from importing and relying on the (potentially unversioned and unsupported) packages in an internal directory — even if the code is publicly available somewhere like GitHub. Importing and using third-party packages Let's quickly explore how to import and use a third-party packages. As an example, we'll import the github.com/fatih/color package and use it change the color of the message that our application prints out — but the general process is exactly the same for most other third-party packages too. First you need to download the third-party code from its public repository to your local machine, which you can do with go get: $ go get github.com/fatih/color@latest go: added github.com/fatih/color v1.14.1 go: added github.com/mattn/go-colorable v0.1.13 go: added github.com/mattn/go-isatty v0.0.17 go: added golang.org/x/sys v0.3.0 Notice that go get will recursively download any dependencies that the code has too. Then using the third-party package in your code is fairly straightforward. You'll need to import the third-party package using its module path (which should normally be the same as the repository location that you used when running go get), and then access its exported things via its package name (which in most cases should be the same as the final element of the import path… if it is not, the documentation for the package should make that clear). Let's head to our main.go file and update the code to print a colored message using github.com/fatih/color. File: cmd/cli/main.go package main import ( "lucky-number.alexedwards.net/internal/random" // Import the color package. "github.com/fatih/color" ) func main() { // Use it to print the message in green. green := color.New(color.FgGreen) green.Printf("Your lucky number is %d!\n", random.Number()) } If you run the application again now, you should see a colorized message similar to this: $ go run ./cmd/cli Your lucky number is 81! The go.mod file should have been updated to include the dependencies that the lucky-number.alexedwards.net module has too, along with their exact version numbers. It should look similar to this: File: go.mod module lucky-number.alexedwards.net go 1.19 require github.com/fatih/color v1.14.1 require ( github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.17 // indirect golang.org/x/sys v0.3.0 // indirect ) We can see that github.com/fatih/color is listed as a direct dependency of the lucky-number.alexedwards.net module, and the other dependencies are indirect (that is, our code doesn't import them directly, but they are imported by a package that our code imports). Note: If your go.mod file looks different to this, try running the $ go mod tidy command to format it. go mod tidy also ensures that the listed dependencies match the source code in your module, so it's a good idea to run this command fairly often… especially before committing a change that adds or removes a third-party package import statement in your code. Also note: If you are using a Go version older than 1.17, indirect dependencies are handled differently and not all of them will necessarily be listed in your go.mod file. As an aside, if you are ever looking at a go.mod file and wondering why something is listed as a dependency you can use the $ go mod why command. For example, if you wanted to find out why golang.org/x/sys is an indirect dependency for the lucky-number.alexedwards.net module you could run: $ go mod why -m golang.org/x/sys # golang.org/x/sys lucky-number.alexedwards.net/cmd/cli github.com/fatih/color github.com/mattn/go-isatty golang.org/x/sys/unix We can see from the output that our cmd/cli package imports github.com/fatih/color, which in turn imports github.com/mattn/go-isatty, which in turn imports golang.org/x/sys/unix. Version 2+ packages Sometimes the third-party packages that you want to use will be in modules with a major version number greater than 1 (like v2.0.0, v3.4.5 etc). In Go, it is conventional for modules with a major version number greater than 1 to append the major version number to their module path. A few popular real-life examples are: github.com/go-chi/chi/v5 github.com/jackc/pgx/v5 github.com/go-playground/validator/v10 Typically you will need to go get these version 2+ packages using the full module path including the version number. Like this: $ go get github.com/go-chi/chi/v5 go: added github.com/go-chi/chi/v5 v5.0.8 And then you need to also import them in your .go files using the full module path (including the version number), but reference their exported things using the package name (which will normally now be the second-to-last element in the import path). For example: import ( "github.com/go-chi/chi/v5" ) func main() { router := chi.NewRouter() ... } Organizing import statements Lastly, there's no right or wrong way to organize your import statements in Go. No single convention has really emerged in the Go community, so I recommend just picking something that works for you and being consistent with it. Personally, I like to separate imports into four groups separated by an empty line. Like this: import ( {standard library packages} {packages from the current module} {third-party packages} {aliased packages} ) Within each group, go fmt will automatically sort the imports alphabetically for you. I like having aliased imports as a final standalone group because it helps to draws attention to them and highlight to the reader that 'there is something a little bit unusual going on here' with them. As an illustration, here's an example from the main.go file of a web application I was recently working on: import ( "fmt" "net/http" "os" "runtime/debug" "sync" "example.com/internal/logger" "example.com/internal/smtp" "github.com/go-playground/form/v4" "github.com/spf13/pflag" _ "github.com/mattn/go-sqlite3" )

• The 'fat service' pattern for Go web applications
  Alex Edwards

  In this post I'd like to talk about one of my favorite architectural patterns for building web applications and APIs in Go. It's kind of a mix between the service object and fat model patterns — so I mentally refer to it as the 'fat service' pattern, but it might have a more formal name that I'm not aware of 🙃 It's certainly not a perfect pattern (we'll discuss some of the pros and cons later) — but it is (deliberately) simple, pragmatic, and I find it often works well for small-to-medium sized projects. Note: Before we start I'd like to emphasize that there's no single 'correct' way to structure your project in Go. Different architectures suit different projects and teams, and this is just one option to consider. At a high-level, the fat service pattern splits your project code into two distinct 'layers': The application layer. This contains your code related to reading and writing HTTP requests and responses, authenticating/authorizing requests, session management, etc. The service layer. This contains your business logic, defines your core data types, and is also responsible for interacting with any persistent data stores. A fat service example Let's illustrate how this pattern works with an example of a JSON API. Specifically, let's say that we want to build an API with a POST /register endpoint which is used to register a new user. When a client makes a request to this endpoint, let's pretend we want to take the following actions: Parse the JSON input into a Go struct so we can work with it easily. Carry out some validation checks on the data (and return an error response to the client if any of them fail). Create a hash of the new user's password. Insert a record for the user into a database. Send a notification to a Slack channel to say that a new user has registered. Return a 204 No Content response to the client if everything worked successfully. Using the fat service pattern, we could structure our project so that the directory and file layout looks like this: . ├── cmd │ └── api | ├── handlers.go │ └── main.go └── internal └── service ├── service.go └── users.go The cmd/api package will contain the application layer code, and the internal/service package will contain the service layer code. Then, very roughly, the code in our service layer might look something like this: File: internal/service/service.go package service import ( "database/sql" "errors" ) var ErrFailedValidation = errors.New("failed validation") type Service struct { DB *sql.DB SlackWebhookURL string } File: internal/service/users.go package service import ( "github.com/slack-go/slack" "golang.org/x/crypto/bcrypt" ) type RegisterUserInput struct { Username string `json:"username"` Password string `json:"password"` ValidationErrors map[string]string `json:"-"` } func (s *Service) RegisterUser(input *RegisterUserInput) error { input.ValidationErrors = make(map[string]string) if input.Username == "" { input.ValidationErrors["username"] = "must be provided" } // And any other validation checks... if len(input.ValidationErrors) > 0 { return ErrFailedValidation } hashedPassword, err := bcrypt.GenerateFromPassword([]byte(input.Password), 12) if err != nil { return err } _, err = s.DB.Exec("INSERT INTO (username, hashed_password) VALUES ($1, $2)", input.Username, string(hashedPassword)) if err != nil { return err } msg := slack.WebhookMessage{ Username: "robot", Channel: "#general", Text: "A new user has signed up!", } return slack.PostWebhook(s.SlackWebhookURL, &msg) } And the code in our application layer might look like this (I've omitted the helper functions for brevity): File: cmd/api/main.go package main import ( "database/sql" "flag" "log" "net/http" "os" "example.com/internal/service" "github.com/alexedwards/flow" _ "github.com/mattn/go-sqlite3" ) type application struct { logger *log.Logger service *service.Service } func main() { dsn := flag.String("dsn", "./db.sqlite", "sqlite3 DSN") slackWebhookURL := flag.String("slack-webhook-url", "https://hooks.slack.com/services/example", "slack webhook URL for notifications") flag.Parse() logger := log.New(os.Stdout, "", log.LstdFlags|log.Llongfile) db, err := sql.Open("sqlite3", *dsn) if err != nil { logger.Fatal(err) } defer db.Close() app := &application{ logger: logger, service: &service.Service{DB: db, SlackWebhookURL: *slackWebhookURL}, } mux := flow.New() mux.HandleFunc("/register", app.registerUserHandler, "POST") logger.Print("starting server on :3000") err = http.ListenAndServe(":3000", mux) logger.Fatal(err) } File: cmd/api/handlers.go package main import ( "errors" "net/http" "example.com/internal/service" ) func (app *application) registerUserHandler(w http.ResponseWriter, r *http.Request) { var input service.RegisterUserInput err := app.decodeJSON(r.Body, &input) if err != nil { app.badRequest(w, r, err) return } err = app.service.RegisterUser(&input) if err != nil { if errors.Is(err, service.ErrFailedValidation) { app.failedValidation(w, r, input.ValidationErrors) } else { app.serverError(w, r, err) } return } w.WriteHeader(http.StatusNoContent) } Hopefully you get the rough idea. Essentially, our service layer contains a Service.RegisterUser() method which executes all the validation checks, business logic and SQL queries related to registering a user. The expected input to this method is the simple, standard, service.RegisterUserInput Go struct. And in our application layer's registerUserHandler() handler we can decode the JSON request body directly into that struct and pass it on the the service layer, handling any returned errors as necessary. The pros and cons In terms of benefits, there are quite a lot of nice things about this pattern: It's fairly simple. The number of mental hoops to jump through when reading the code is relatively low. You don't have to dig through lots of packages and functions to follow what the code is doing — meaning it's relatively easy for newcomers to your project to understand (or even yourself after a long break). The separation of concerns keeps our registerUserHandler() code primarily focused on reading and writing HTTP requests and responses. For applications with more than a few endpoints, I find that not trying to do everything in your handlers helps to make your codebase easier to navigate and reason about. The code in the service layer can be reused by other applications. For example, we could create a CLI application under cmd/cli with a task that also calls the Service.RegisterUser() method. This one is more personal, but I find it easier to reason about my business logic and write the code for it when the input is a well-defined Go struct with the correct types (rather than a more 'messy' input like a JSON string or HTML-encoded form data). It's really practical for APIs and web applications. You can parse JSON or HTML form data from a request body directly into the service.RegisterUserInput struct in your handlers, and then pass that struct to the service layer for processing. You don't need to create interim types in your handlers to hold the decoded request data, or copy data from one struct to another. Methods in the service layer can potentially return validation errors from multiple points in the code, and you can deal with them all just once in your handler. For example, if our user INSERT failed because we tried to insert a record with a duplicate username, then we could return a "username is already taken" validation error from our service layer in addition to the pre-INSERT validation checks. Working with database transactions is easy. If we wanted to execute multiple SQL statements as part of registering a user in a single transaction, we could initialize the sql.TX, execute all the necessary statements, and commit the transaction all within our Service.RegisterUser() method. We don't need to pass the sql.TX around to a bunch of different places in our codebase. If you want to test only your application layer logic only, this pattern lends itself nicely to creating an interface type that describes the methods on the service.Service struct, which you can then satisfy with a mock implementation. But it's not perfect, and there are also a few downsides: When you are looking at the code for your handlers, you can't immediately see what the expected inputs are. You have to navigate to the service package and look at the fields of the service.RegisterUserInput struct. With most modern text editors this is just one click away, but it still introduces a bit of 'obscurity' and feels less than ideal to me. Not having a separate abstraction for the database logic makes it harder to swap out one database for another in the future (say moving from SQLite to PostgreSQL). You can't easily mock the database calls during tests. Personally I tend to prefer using a test instance of an actual database for testing, so I don't find this too much of a drawback most of the time. But if you need to mock the database (i.e. to speed up test runtime, or because it's a hard requirement from a client) then this pattern doesn't really suit that. Lastly, SQL queries which use database/sql and the Query() method to return multiple rows of data are quite verbose. These queries can take up a lot of visual space and add clutter to the service layer methods — which ultimately starts to reduce the scannability of the code. Using jmoiron/sqlx or blockloop/scan can be a big help here. But overall — so long as you don't need to mock your database calls — I like this pattern. I've used it a lot over the past 3-4 years and have found that the relative simplicity and practical benefits comfortably outweigh any downsides.

• Easy test assertions with Go generics
  Alex Edwards

  Now that Go 1.18 has been released with support for generics, it's easier than ever to create helper functions for your test assertions. Using helpers for your test assertions can help to: Make your test functions clean and clear; Keep test failure messages consistent; And reduce the potential for errors in your code due to typos. To illustrate this, let's say that you have a simple greet() function that you want to test: package main import "fmt" func greet(name string) (string, int) { greeting := fmt.Sprintf("Hello %s", name) // Return the greeting and its length (in bytes). return greeting, len(greeting) } In the past, your test for the greet() function would probably look something like this: package main import "testing" func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") // Test assertion to check the returned greeting string. if greeting != "Hello Alice" { t.Errorf("want: %s; got: %s", "Hello Alice", greeting) } // Test assertion to check the returned greeting length. if greetingLength != 11 { t.Errorf("want: %d; got: %d", 11, greetingLength) } } With Go 1.18, we can use generics and the comparable constraint to create an Equal() helper function which carries out our test assertions. Personally, I like to put this in a reusable assert package. Like so: package assert import "testing" func Equal[T comparable](t *testing.T, expected, actual T) { t.Helper() if expected != actual { t.Errorf("want: %v; got: %v", expected, actual) } } Note: The t.Helper() function indicates to the Go test runner that our Equal() function is a test helper. This means that when t.Errorf() is called from our Equal() function, the Go test runner will report the filename and line number of the code which called our Equal() function in the output. And with that in place, the TestGreet() test can be simplified like so: package main import ( "testing" "your.module.path/assert" // Import your assert package. ) func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") assert.Equal(t, "Hello Alice", greeting) assert.Equal(t, 11, greetingLength) }

• Continuous integration with Go and GitHub Actions
  Alex Edwards

  In this post we're going to walk through how to use GitHub Actions to create a continuous integration (CI) pipeline that automatically tests, vets and lints your Go code. For solo projects I usually create a pre-commit Git hook to carry out these kinds of checks, but for team projects or open-source work — where you don't have control over everyone's development environment — using a CI workflow is a great way to flag up potential problems and help catch bugs before they make it into production or a versioned release. And if you're already using GitHub to host your repository, it's nice and easy to use their built-in functionality to do this without any need for additional third-party tools or services. To demonstrate how it works, let's run through a step-by-step example. If you'd like to follow along, please create a new repository and clone it to your local machine. For the purpose of this post I'm going to use the private repository alexedwards/example. $ git clone git@github.com:alexedwards/example.git Cloning into 'example'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done. Then let's scaffold a simple Go application along with a (failing) test like so: $ cd example/ $ touch main.go main_test.go $ go mod init github.com/alexedwards/example File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { return fmt.Sprintf("Hi %s", name) } File: main_test.go package main import "testing" func Test_sayHello(t *testing.T) { name := "Bob" want := "Hello Bob" if got := sayHello(name); got != want { t.Errorf("hello() = %q, want %q", got, want) } } If you run this application it should compile correctly and print "Hi Alice", but executing go test . will result in a failure. Similar to this: $ go test . --- FAIL: Test_sayHello (0.00s) main_test.go:10: hello() = "Hi Bob", want "Hello Bob" FAIL FAIL github.com/alexedwards/example 0.002s FAIL Creating a workflow file The next thing that we want to do is create a workflow file which describes what we want to do in our CI checks, and when we want them to run. By convention this file should be stored in a .github/workflow directory in the root of your repository and should be in YAML format. Let's create this directory along with an audit.yml workflow file. $ mkdir -p .github/workflows $ touch .github/workflows/audit.yml There's an excellent introduction to the workflow file syntax here, and there's also a collection of templates for different languages and frameworks that you can use as a starting point. But for now, let's jump in and update the workflow file so that it looks like this: File: .github/workflows/audit.yml name: Audit on: push: branches: [main] pull_request: branches: [main] jobs: audit: runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.17 - name: Verify dependencies run: go mod verify - name: Build run: go build -v ./... - name: Run go vet run: go vet ./... - name: Install staticcheck run: go install honnef.co/go/tools/cmd/staticcheck@latest - name: Run staticcheck run: staticcheck ./... - name: Install golint run: go install golang.org/x/lint/golint@latest - name: Run golint run: golint ./... - name: Run tests run: go test -race -vet=off ./... Let's quickly step through this and explain what the different parts of the file do. First we use the on keyword to define when we want the workflow to run. In this case, I've configured the workflow so that it runs when a new commit is made to the main branch, or a pull request is submitted. Then we use the jobs keyword to define a list of the jobs that are to be run. At the moment our workflow only contains one job called audit, but you can specify multiple jobs if you want and (by default) they will be executed in parallel. An independent runner will be spun up for each job. This is essentially a virtual machine that will execute the steps for the job. In the file above we use the runs-on keyword to specify that we want the runner to use Ubuntu 20.04 as a base OS, but others operating systems are available. It's also worth noting that the runner has a lot of useful software and tooling pre-installed. In the first step for our audit job we use the uses keyword to execute the community action actions/checkout@v2. This action will checkout our project repository to the runner so that the following steps access the code. Then we use the actions/setup-go@v2 action to install Go version 1.17 on the runner. Once that's done, in the remaining steps we use the run keyword to execute specific commands on the runner. In this case we build our code and then audit it using the standard go build|vet|test commands and the additional golint and staticcheck tools. Important: If you're following along, please run $ git branch --show-current to check the name of your branch before continuing. In certain cases, your branch may have the name master instead of main, in which case please edit the on directive in your audit.yml file accordingly. Now that's in place, let's commit everything and push the changes to your repository: $ git add . $ git commit -m "Initial commit" $ git push Once the push has completed, head to your repository and select the Actions tab. You should see that the CI 'Audit' workflow is running, similar to the screenshot below. You can click through on the workflow name to see more details while it's running, and after a minute or two you should see that the workflow is terminated due to our failing test. Additionally, as the owner of the repository, you should also get an email notification to tell you that the workflow failed, and everyone who browses the repository will see a red cross symbol next to the commit in the Git history. Fixing the code Let's fix our codebase by updating the sayHello() function to return the correct output, like so: File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { // Change this to "Hello %s" instead of "Hi %s". return fmt.Sprintf("Hello %s", name) } If you want, you can commit this change and push it… $ git add . $ git commit -m "Fix sayHello() to return the correct value" $ git push … and you should see that the 'Audit' job in our workflow file now completes successfully and everything has a nice green check mark next to it. Great! That's working really well and, from now on, any time someone makes a push or pull request to the main branch, the tests and vetting and linter checks will be automatically run. From here, you can extend the workflow to carry out more checks or send additional notifications if you want to — or even expand it to act as a continuous deployment (CD) pipeline that builds and deploys your binaries. To give you some ideas, here are a couple of slightly more complicated workflows from my own projects: Run integration tests against a PostgreSQL database Run audit checks, then build a binary and deploy it to a remote server using Ansible

• Which Go router should I use?
  Alex Edwards

  Note: This post has been fully updated to reflect the new http.ServeMux features released in Go 1.22. When you start to build web applications with Go, one of the first questions you'll probably ask is "which router should I use?". It's not an easy question to answer, either. You've got http.ServeMux in the Go standard library, and probably more than 100 different third-party routers also available — all with distinct APIs, features, and behaviors. Is http.ServeMux going to be sufficient? Or will you need to use a different router? And if so, which one is the right choice? For this blog post, I've evaluated 30 of the most popular third-party routers on GitHub (along with http.ServeMux), created a shortlist of the best options, and made a comparison table you can use to help make your choice. If you want, you can skip to the comparison table and summary. Shortlisted routers There are five routers which make the shortlist and that I recommend using. They are http.ServeMux, httprouter, chi, flow, and gorilla/mux. All the shortlisted routers are well-tested, well-documented, and actively maintained. They have stable APIs, and are compatible with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. There are a few common features that all five of these routers support: Method matching: All let you register routes that require a matching HTTP method (GET, POST etc). Path segment wildcards: All let you declare routes like /movies/{id}/edit where {id} is a dynamic segment in the URL path. Automatic sending of 404 responses: All automatically send plaintext 404 responses when a matching route cannot be found. Automatic sending of 405 responses: All automatically send 405 responses when a route is found with a matching URL pattern, but not a matching HTTP method. Please note though that gorilla/mux does not automatically include an Allow header in 405 responses, and chi will potentially include duplicate values in the Allow header (there is an open issue about this here). In terms of speed, all five routers are fast enough for (almost) every application. Unless you have profiling that confirms your router is a bottleneck in your application, I recommend choosing between them based on the specific features that you need rather than performance. I've personally used all five routers in production applications at different times and have been happy with them. Note: One downside of httprouter is that the API and documentation is a bit confusing. The package was first published prior to the introduction of request context in Go 1.7, and lot of the current API still exists in order to support these older versions of Go. Nowadays, you can write your handlers using regular http.Handler and http.HandlerFunc signatures and all you need is the router.Handler() and router.HandlerFunc() methods to register them, like this. So with that out of the way, I'll start by saying… Use the standard library if you can If you can use http.ServeMux, you probably should. As part of the Go standard library, it's very battle tested and well documented. Using it means that you don't need to import any third-party dependencies, and most other Go developers will also be familiar with how it works. The Go compatibility promise also means that you should be able to rely on http.ServeMux working the same way in the long-term. All of those things are big positives in terms of application maintenance. It also has some really nice features that don't always appear in the third-party routers. Hostname matching: http.ServeMux lets you register routes that require a matching hostname, like example.com/post/{id} and example.org/post/{id}. Hostname matching is also supported by gorilla/mux, and chi supports it via the additional hostrouter package. URL path sanitization: http.ServeMux will automatically sanitize request URL paths and redirect the client if necessary. For example, if a client makes a request to /foo/bar/..//baz they will automatically be sent a 301 redirect to /foo/baz. URL sanitization is also done by gorilla/mux and httprouter in the same way. Automatic handling of HEAD requests: http.ServeMux automatically handles HEAD requests and sends the appropriate headers in the response. This is also supported by flow in the same way. Overlapping routes: If you register the routes /post/edit and /post/{id}, they overlap because a request to /post/edit matches both route patterns. The way that http.ServeMux matches overlapping wildcard routes is smart — the most specific matching route pattern wins and /post/edit is more specific than /post/{id}. This is nice because it means you can register patterns in any order and it won’t affect how http.ServeMux behaves. chi behaves in a similar-ish way to http.ServeMux and will prioritize non-wildcard matches. In contrast, gorilla/mux and flow will dispatch requests to the first matching route, and httprouter simply disallows overlapping routes and will panic if you try to register them. If you need additional features While I recommend using http.ServeMux as your go-to router, there may be times where you need a feature or a behavior that http.ServeMux doesn't provide or easily support. These include: Subsegment wildcards: chi is the only shortlisted router to support more than one wildcard within a single URL path segment, like /articles/{month}-{year}-{day}/{id}. Regexp wildcards: gorilla/mux, chi and flow support regexp wildcards, like /movies/{[a-z-]+}, where [a-z-]+ is a required regexp pattern in the URL path. Header matching: gorilla/mux is the only shortlisted router to easily support routing to different handlers based on the value of a request header (like Authorization or Content-Type). Custom matching rules: gorilla/mux is the only shortlisted router to support custom rules for matching requests (such as routing to different handlers based on IP address). Custom 404 responses: With http.ServeMux it's simple to implement a 'catch all' route "/" which will send a custom 404 response, but doing this will inhibit the automatic sending of 405 responses. There's an open issue about this, and hopefully it will get resolved soon. In contrast, httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 404 response without this problem. Custom 405 responses: With http.ServeMux there is no simple way to send custom 405 responses. Whereas httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 405 responses. But be aware that both chi and gorilla/mux will not automatically set an Allow header if you are using a custom 405 handler. Automatic handling of OPTIONS requests: Both httprouter and flow automatically send correct responses for OPTIONS requests. One route, multiple methods: Both gorilla/mux and flow support matching multiple HTTP methods in a single route declaration. Middleware groups: Both chi and flow provide 'grouping' functionality that lets you batch routes into groups that use specific middleware. Note that you can also wrap http.ServeMux to do this (and I've written about how to do that here). Route reversing: gorilla/mux is the only shortlisted router to support route reversing (like you get in Django, Rails, and Laravel). Subrouters: Both chi and gorilla/mux allow the creation of 'subrouters' that can be assigned to handle a subset of your application routes. Case sensitivity: All shortlisted routers require a case-sensitive match on non-wildcard parts of a route, apart from httprouter which is case-insensitive. Trailing slashes: All shortlisted routers treat trailing slashes as significant (i.e. /foo is a different route to /foo/). But... gorilla/mux has an optional StrictSlash setting where requests to /foo can automatically be redirected to /foo/. In contrast, chi has an optional RedirectSlashes middleware which will automatically redirect requests from /foo/ to /foo. And httprouter will automatically redirect requests from /foo to /foo/ and vice-versa if a matching route exists — this can be disabled via the RedirectTrailingSlash setting. Comparison table Summary If the comparison table is too overwhelming, or you don't yet know what your full requirements will be, I suggest falling back to the following guidelines: If you know you're going to have routes with complex matching requirements (i.e. more than just simple method, wildcard segment, and hostname matching), then opt for gorilla/mux or chi. If you're building something that requires custom 404 and 405 responses, and it's important that it adheres correctly to the HTTP specs (such as a JSON API for public use), opt for httprouter or flow. If you know that your application will require a lot of route-specific middleware, opt for chi or flow because of their middleware grouping functionality. Otherwise, start with http.ServeMux, and refactor to use a third-party router only if there is a specific feature or behavior that you need. Other routers For completeness, the other routers that I evaluated are listed below, along with a short note to explain why they didn't made the shortlist. Note: I used the question "does the repository contain a go.mod file?" as a proxy measure for whether a codebase is currently maintained or not. This seems reasonable — if the maintainer is still engaged with the Go world and caring for the code, my guess is that they would have updated the repository to use modules at some point. Repository Notes celrenheit/lion Currently unmaintained. claygod/Bxog Currently unmaintained. clevergo/clevergo Uses custom handler signature (not http.Handler or http.HandlerFunc). dimfeld/httptreemux Doesn’t fully support http.Handler. Requires middleware for setting custom 404/405 handlers. donutloop/mux Currently unmaintained. gernest/alien Currently unmaintained. go-ozzo/ozzo-routing Uses custom handler signature (not http.Handler or http.HandlerFunc). go-playground/lars Currently unmaintained. go-zoo/bone Good, but has similar use case to chi (which offers more). Incomplete tests. go101/tinyrouter Verbose route declarations. Doesn’t automatically send 405 responses. gocraft/web Currently unmaintained. goji/goji Slightly unusual, but flexible, API which supports custom matchers. Requires middleware for setting custom 404/405 handlers. Good, but I think gorilla/mux offers similar features and is easier to use. goroute/route Uses custom handler signature (not http.Handler or http.HandlerFunc). gowww/router Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. GuilhermeCaruso/bellt No way to set custom 404 or 405 handlers. husobee/vestigo Currently unmaintained. Only supports http.HandlerFunc. naoina/denco Currently unmaintained. nbari/violetear Good, but has similar use case to chi (which offers more). Wraps http.ResponseWriter with own custom type, which may cause problems in some cases. nbio/hitch Lacking documentation. nissy/bon Currently unmaintained. razonyang/fastrouter Currently unmaintained. rs/xmux Currently unmaintained. Uses custom handler signature (not http.Handler or http.HandlerFunc). takama/router Currently unmaintained. vardius/gorouter Good, but has similar use case to chi (which offers more). Four major versions in 5 years suggests the API may not be reliable. VividCortex/siesta Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. xujiajun/gorouter Currently unmaintained.

• Custom command-line flags with flag.Func
  Alex Edwards

  One of my favorite things about the recent Go 1.16 release is a small — but very welcome — addition to the flag package: the flag.Func() function. This makes it much easier to define and use custom command-line flags in your application. For example, if you want to parse a flag like --pause=10s directly into a time.Duration type, or parse --urls="http://example.com http://example.org" directly into a []string slice, then previously you had two options. You could either create a custom type to implement the flag.Value interface, or use a third-party package like pflag. But now the flag.Func() function gives you a simple and lightweight alternative. In this short post we're going to take a look at a few examples of how you can use it in your own code. Parsing custom flag types To demonstrate how this works, let's start with the two examples I gave above and create a sample application which accepts a list of URLs and then prints them out with a pause between them. Similar to this: $ go run . --pause=3s --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:16:04 http://example.com 2021/03/08 08:16:07 http://example.org 2021/03/08 08:16:10 http://example.net To make this work, we'll need to do two things: Convert the --pause flag value from a 'human-readable' string like 200ms, 5s or 10m into a native Go time.Duration type. We can do this using the time.ParseDuration() function. Split the values in the --urls flag into a slice, so we can loop through them. The strings.Fields function is a good fit for this task. We can use those together with flag.Func() like so: package main import ( "flag" "log" "strings" "time" ) func main() { // First we need to declare variables to hold the values from the // command-line flags. Notice that we also need to set any defaults, // which will be used if the relevant flag is not provided at runtime. var ( urls []string // Default of the empty slice pause time.Duration = time.Second // Default of one second ) // The flag.Func() function takes three parameters: the flag name, // descriptive help text, and a function with the signature // `func(string) error` which is called to process the string value // from the command-line flag at runtime and assign it to the necessary // variable. In this case, we use strings.Fields() to split the string // based on whitespace and store the resulting slice in the urls // variable that we declared above. We then return nil from the // function to indicate that the flag was parsed without any errors. flag.Func("urls", "List of URLs to print", func(flagValue string) error { urls = strings.Fields(flagValue) return nil }) // Likewise we can do the same thing to parse the pause duration. The // time.ParseDuration() function may throw an error here, so we make // sure to return that from our function. flag.Func("pause", "Duration to pause between printing URLs", func(flagValue string) error { var err error pause, err = time.ParseDuration(flagValue) return err }) // Importantly, call flag.Parse() to trigger actual parsing of the // flags. flag.Parse() // Print out the URLs, pausing between each iteration. for _, u := range urls { log.Print(u) time.Sleep(pause) } } If you try to run this application, you should find that the flags are parsed and work just like you would expect. For example: $ go run . --pause=500ms --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:22:33 http://example.com 2021/03/08 08:22:34 http://example.org 2021/03/08 08:22:34 http://example.net Whereas if you provide an invalid flag value that triggers an error in one of the flag.Func() functions, Go will automatically display the corresponding error message and exit. For example: $ go run . --pause=500xx --urls="http://example.com http://example.org http://example.net" invalid value "500xx" for flag -pause: time: unknown unit "xx" in duration "500xx" Usage of /tmp/go-build3141872390/b001/exe/example.text: -pause value Duration to pause between printing URLs -urls value List of URLs to print exit status 2 It's really important to point out here that if a flag isn't provided, the corresponding flag.Func() function will not be called at all. This means that you cannot set a default value inside a flag.Func() function, so trying to do something like this won't work: flag.Func("pause", "Duration to pause between printing URLs (default 1s)", func(flagValue string) error { // DON'T DO THIS! This function wont' be called if the flag value is "". if flagValue == "" { pause = time.Second return nil } var err error pause, err = time.ParseDuration(flagValue) return err }) On the plus side though, there are no restrictions on the code that can be contained in a flag.Func() function, so if you want, you could get even fancier with this and parse the URLs into a []*url.URL slice instead of a []string. Like so: var ( urls []*url.URL pause time.Duration = time.Second ) flag.Func("urls", "List of URLs to print", func(flagValue string) error { for _, u := range strings.Fields(flagValue) { parsedURL, err := url.Parse(u) if err != nil { return err } urls = append(urls, parsedURL) } return nil }) Validating flag values The flag.Func() function also opens up some new opportunities for validating input data from command-line flags. For example, let's say that your application has an --environment flag and you want to restrict the possible values to development, staging or production. To do that, you can implement a flag.Func() function similar to this: package main import ( "errors" "flag" "fmt" ) func main() { var ( environment string = "development" ) flag.Func("environment", "Operating environment", func(flagValue string) error { for _, allowedValue := range []string{"development", "staging", "production"} { if flagValue == allowedValue { environment = flagValue return nil } } return errors.New(`must be one of "development", "staging" or "production"`) }) flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } Making reusable helpers If you find yourself repeating the same code in your flag.Func() functions, or the logic is getting too complex, it's possible to break it out into a reusable helper. For example, we could rewrite the example above to process our --environment flag via a generic enumFlag() function, like so: package main import ( "flag" "fmt" ) func main() { var ( environment string = "development" ) enumFlag(&environment, "environment", []string{"development", "staging", "production"}, "Operating environment") flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } func enumFlag(target *string, name string, safelist []string, usage string) { flag.Func(name, usage, func(flagValue string) error { for _, allowedValue := range safelist { if flagValue == allowedValue { *target = flagValue return nil } } return fmt.Errorf("must be one of %v", safelist) }) }

• Golang Interfaces explained
  Alex Edwards

  For the past few months I've been running a survey which asks people what they're finding difficult about learning Go. And something that keeps coming up in the responses is the concept of interfaces. I get that. Go was the first language I ever used that had interfaces, and I remember at the time that the whole concept felt pretty confusing. So in this tutorial I want to do a few things: Provide a plain-English explanation of what interfaces are; Explain why they are useful and how you might want to use them in your code; Talk about what interface{} (the empty interface) is; And run through some of the helpful interface types that you'll find in the standard library. So what is an interface? An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. One example of an interface type from the standard library is the fmt.Stringer interface, which looks like this: type Stringer interface { String() string } We say that something satisfies this interface (or implements this interface) if it has a method with the exact signature String() string. For example, the following Book type satisfies the interface because it has a String() string method: type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } It's not really important what this Book type is or does. The only thing that matters is that is has a method called String() which returns a string value. Or, as another example, the following Count type also satisfies the fmt.Stringer interface — again because it has a method with the exact signature String() string. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } The important thing to grasp is that we have two different types, Book and Count, which do different things. But the thing they have in common is that they both satisfy the fmt.Stringer interface. You can think of this the other way around too. If you know that an object satisfies the fmt.Stringer interface, you can rely on it having a method with the exact signature String() string that you can call. Now for the important part. Wherever you see declaration in Go (such as a variable, function parameter or struct field) which has an interface type, you can use an object of any type so long as it satisfies the interface. For example, let's say that you have the following function: func WriteLog(s fmt.Stringer) { log.Print(s.String()) } Because this WriteLog() function uses the fmt.Stringer interface type in its parameter declaration, we can pass in any object that satisfies the fmt.Stringer interface. For example, we could pass either of the Book and Count types that we made earlier to the WriteLog() method, and the code would work OK. Additionally, because the object being passed in satisfies the fmt.Stringer interface, we know that it has a String() string method that the WriteLog() function can safely call. Let's put this together in an example, which gives us a peek into the power of interfaces. package main import ( "fmt" "strconv" "log" ) // Declare a Book type which satisfies the fmt.Stringer interface. type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } // Declare a Count type which satisfies the fmt.Stringer interface. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } // Declare a WriteLog() function which takes any object that satisfies // the fmt.Stringer interface as a parameter. func WriteLog(s fmt.Stringer) { log.Print(s.String()) } func main() { // Initialize a Count object and pass it to WriteLog(). book := Book{"Alice in Wonderland", "Lewis Carrol"} WriteLog(book) // Initialize a Count object and pass it to WriteLog(). count := Count(3) WriteLog(count) } This is pretty cool. In the main function we've created different Book and Count types, but passed both of them to the same WriteLog() function. In turn, that calls their relevant String() functions and logs the result. If you run the code, you should get some output which looks like this: 2009/11/10 23:00:00 Book: Alice in Wonderland - Lewis Carrol 2009/11/10 23:00:00 3 I don't want to labor the point here too much. But the key thing to take away is that by using a interface type in our WriteLog() function declaration, we have made the function agnostic (or flexible) about the exact type of object it receives. All that matters is what methods it has. Why are they useful? There are all sorts of reasons that you might end up using a interface in Go, but in my experience the three most common are: To help reduce duplication or boilerplate code. To make it easier to use mocks instead of real objects in unit tests. As an architectural tool, to help enforce decoupling between parts of your codebase. Let's step through these three use-cases and explore them in a bit more detail. Reducing boilerplate code OK, imagine that we have a Customer struct containing some data about a customer. In one part of our codebase we want to write the customer information to a bytes.Buffer, and in another part of our codebase we want to write the customer information to an os.File on disk. But in both cases, we want to serialize the customer struct to JSON first. This is a scenario where we can use Go's interfaces to help reduce boilerplate code. The first thing you need to know is that Go has an io.Writer interface type which looks like this: type Writer interface { Write(p []byte) (n int, err error) } And we can leverage the fact that both bytes.Buffer and the os.File type satisfy this interface, due to them having the bytes.Buffer.Write() and os.File.Write() methods respectively. Let's take a look at a simple implementation: package main import ( "bytes" "encoding/json" "io" "log" "os" ) // Create a Customer type type Customer struct { Name string Age int } // Implement a WriteJSON method that takes an io.Writer as the parameter. // It marshals the customer struct to JSON, and if the marshal worked // successfully, then calls the relevant io.Writer's Write() method. func (c *Customer) WriteJSON(w io.Writer) error { js, err := json.Marshal(c) if err != nil { return err } _, err = w.Write(js) return err } func main() { // Initialize a customer struct. c := &Customer{Name: "Alice", Age: 21} // We can then call the WriteJSON method using a buffer... var buf bytes.Buffer err := c.WriteJSON(&buf) if err != nil { log.Fatal(err) } // Or using a file. f, err := os.Create("/tmp/customer") if err != nil { log.Fatal(err) } defer f.Close() err = c.WriteJSON(f) if err != nil { log.Fatal(err) } } Of course, this is just a toy example (and there are other ways we could structure the code to achieve the same end result). But it nicely illustrates the benefit of using an interface — we can create the Customer.WriteJSON() method once, and we can call that method any time that we want to write to something that satisfies the io.Writer interface. But if you're new to Go, this still begs a couple of questions: How do you know that the io.Writer interface even exists? And how do you know in advance that bytes.Buffer and os.File both satisfy it? There's no easy shortcut here I'm afraid — you simply need to build up experience and familiarity with the interfaces and different types in the standard library. Spending time thoroughly reading the standard library documentation, and looking at other people's code will help here. But as a quick-start I've included a list of some of the most useful interface types at the end of this post. But even if you don't use the interfaces from the standard library, there's nothing to stop you from creating and using your own interface types. We'll cover how to do that next. Unit testing and mocking To help illustrate how interfaces can be used to assist in unit testing, let's take a look at a slightly more complex example. Let's say you run a shop, and you store information about the number of customers and sales in a PostgreSQL database. You want to write some code that calculates the sales rate (i.e. sales per customer) for the past 24 hours, rounded to 2 decimal places. A minimal implementation of the code for that could look something like this: // File: main.go package main import ( "fmt" "log" "time" "database/sql" _ "github.com/lib/pq" ) type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } func calculateSalesRate(sdb *ShopDB) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sdb.CountSales(since) if err != nil { return "", err } customers, err := sdb.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } Now, what if we want to create a unit test for the calculateSalesRate() function to make sure that the math logic in it is working correctly? Currently this is a bit of a pain. We would need to set up a test instance of our PostgreSQL database, along with setup and teardown scripts to scaffold the database with dummy data. That's quite lot of work when all we really want to do is test our math logic. So what can we do? You guessed it — interfaces to the rescue! A solution here is to create our own interface type which describes the CountSales() and CountCustomers() methods that the calculateSalesRate() function relies on. Then we can update the signature of calculateSalesRate() to use this custom interface type as a parameter, instead of the concrete *ShopDB type. Like so: // File: main.go package main import ( "database/sql" "fmt" "log" "time" _ "github.com/lib/pq" ) // Create our own custom ShopModel interface. Notice that it is perfectly // fine for an interface to describe multiple methods, and that it should // describe input parameter types as well as return value types. type ShopModel interface { CountCustomers(time.Time) (int, error) CountSales(time.Time) (int, error) } // The ShopDB type satisfies our new custom ShopModel interface, because it // has the two necessary methods -- CountCustomers() and CountSales(). type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } // Swap this to use the ShopModel interface type as the parameter, instead of the // concrete *ShopDB type. func calculateSalesRate(sm ShopModel) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sm.CountSales(since) if err != nil { return "", err } customers, err := sm.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } With that done, it's straightforward for us to create a mock which satisfies our ShopModel interface. We can then use that mock during unit tests to test that the math logic in our calculateSalesRate() function works correctly. Like so: // File: main_test.go package main import ( "testing" "time" ) type MockShopDB struct{} func (m *MockShopDB) CountCustomers(_ time.Time) (int, error) { return 1000, nil } func (m *MockShopDB) CountSales(_ time.Time) (int, error) { return 333, nil } func TestCalculateSalesRate(t *testing.T) { // Initialize the mock. m := &MockShopDB{} // Pass the mock to the calculateSalesRate() function. sr, err := calculateSalesRate(m) if err != nil { t.Fatal(err) } // Check that the return value is as expected, based on the mocked // inputs. exp := "0.33" if sr != exp { t.Fatalf("got %v; expected %v", sr, exp) } } You could run that test now, everything should work fine. Application architecture In the previous examples, we've seen how interfaces can be used to decouple certain parts of your code from relying on concrete types. For instance, the calculateSalesRate() function is totally flexible about what you pass to it — the only thing that matters is that it satisfies the ShopModel interface. You can extend this idea to create decoupled 'layers' in larger projects. Let's say that you are building a web application which interacts with a database. If you create an interface that describes the exact methods for interacting with the database, you can refer to the interface throughout your HTTP handlers instead of a concrete type. Because the HTTP handlers only refer to an interface, this helps to decouple the HTTP layer and database-interaction layer. It makes it easier to work on the layers independently, and to swap out one layer in the future without affecting the other. I've written about this pattern in this previous blog post, which goes into more detail and provides some practical example code. What is the empty interface? If you've been programming with Go for a while, you've probably come across the empty interface type: interface{}. This can be a bit confusing, but I'll try to explain it here. At the start of this blog post I said: An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. The empty interface type essentially describes no methods. It has no rules. And because of that, it follows that any and every object satisfies the empty interface. Or to put it in a more plain-English way, the empty interface type interface{} is kind of like a wildcard. Wherever you see it in a declaration (such as a variable, function parameter or struct field) you can use an object of any type. Take a look at the following code: package main import "fmt" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 fmt.Printf("%+v", person) } In this code snippet we initialize a person map, which uses the string type for keys and the empty interface type interface{} for values. We've assigned three different types as the map values (a string, int and float32) — and that's OK. Because objects of any and every type satisfy the empty interface, the code will work just fine. You can give it a try here, and when you run it you should see some output which looks like this: map[age:21 height:167.64 name:Alice] But there's an important thing to point out when it comes to retrieving and using a value from this map. For example, let's say that we want to get the "age" value and increment it by 1. If you write something like the following code, it will fail to compile: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 person["age"] = person["age"] + 1 fmt.Printf("%+v", person) } And you'll get the following error message: invalid operation: person["age"] + 1 (mismatched types interface {} and int) This happens because the value stored in the map takes on the type interface{}, and ceases to have it's original, underlying, type of int. Because it's no longer an int type we cannot add 1 to it. To get around this this, you need to type assert the value back to an int before using it. Like so: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 age, ok := person["age"].(int) if !ok { log.Fatal("could not assert value to int") return } person["age"] = age + 1 log.Printf("%+v", person) } If you run this now, everything should work as expected: 2009/11/10 23:00:00 map[age:22 height:167.64 name:Alice] So when should you use the empty interface type in your own code? The answer is probably not that often. If you find yourself reaching for it, pause and consider whether using interface{} is really the right option. As a general rule it's clearer, safer and more performant to use concrete types — or non-empty interface types — instead. In the code snippet above, it would have been more appropriate to define a Person struct with relevant typed fields similar to this: type Person struct { Name string Age int Height float32 } But that said, the empty interface is useful in situations where you need to accept and work with unpredictable or user-defined types. You'll see it used in a a number of places throughout the standard library for that exact reason, such as in the gob.Encode, fmt.Print and template.Execute functions. Comman and useful types Lastly, here's a short list of some of the most common and useful interfaces in the standard library. If you're not familiar with them already, then I recommend taking out a bit of time to look at the relevant documentation for them. builtin.Error fmt.Stringer io.Reader io.Writer io.ReadWriteCloser http.ResponseWriter http.Handler There is also a longer and more comprehensive listing of standard libraries available in this gist.

• Streamline your Sublime Text + Go workflow
  Alex Edwards

  For the past couple of years I've used Sublime Text as my primary code editor, along with the GoSublime plugin to provide some extra IDE-like features. But I've recently swapped GoSublime for a more modular plugin setup and have been really happy with the way it's worked out. Although it took a while to configure, it's resulted in a coding environment that feels clearer to use and more streamlined than before. I've opted for: Tooling integration with the official sublime-build plugin. Automatic formatting with the Gofmt plugin and goimports. Code linting with the SublimeLinter plugin and gometalinter. Autocompletion with the gocode package. Code navigation with the GoGuru plugin. Snippet management with Sublime Text's inbuilt tool and the PackageResourceViewer plugin. In this post I'm going to run through the process of setting these up. If you haven't come across these plugins before, I recommend giving them a try! Prerequisites To work correctly some of these Sublime Text plugins need an explicit $GOPATH environment variable to be set. And if you're following along, you should also make sure that your workspace's bin directory is on your system path. Accordingly my bash ~/.profile configuration includes these lines: ... export GOPATH=/home/alex/Code/go export PATH=$PATH:$GOPATH/bin You'll also need to install Package Control, if you haven't already. In the latest version of Sublime Text the easiest way to do that by going to Tools > Install Package Control…. Tooling integration The official sublime-build plugin provides integrations so you can execute common go commands (like go run, go test and go get) without leaving your editor. You can install it like so: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Golang Build and hit Enter to install the package. After installation should see a bunch of new tools in your command palette. Their names are pretty self explanatory: Build With: Go Build With: Go - Clean Build With: Go - Install Build With: Go - Run Build With: Go - Test Build With: Go - Cross-Compile Go: Get Go: Open Terminal When you run these commands they will open and execute in a panel within Sublime Text. As an example, here's a screenshot of output from the Build With: Go - Test command: Automatic formatting For automatic formatting of .go files I've been using the Gofmt plugin. You can install it as follows: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Gofmt and hit Enter to install the package. By default this will run go fmt -s -e on the current file each time it is saved. I've customised this further to use the goimports tool. If you're not already familiar with goimports, it runs go fmt and fixes your import lines — adding missing packages and removing unreferenced ones as necessary. To set this up you'll need to install goimports and make sure it's available on your system path: $ go get golang.org/x/tools/cmd/goimports $ which goimports /home/alex/Code/go/bin/goimports When that's installed, you'll then need to change the Gofmt plugin settings in Sublime Text by opening Preferences > Package Settings > Gofmt > Settings - User and adding the following configuration settings: { "cmds": [ ["goimports"] ], "format_on_save": true } (You'll probably need to restart Sublime Text for this to take effect.) Each time you now save a .go file, you'll find that it gets automatically formatted and the import packages are updated. No more "imported and not used" errors! Code linting For linting of source code I'm using the SublimeLinter plugin. This plugin isn't a linter itself, but provides a framework for running linters and displaying error messages. You can install it like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter and hit Enter to install the package. The next step is to install an actual linter. I'm using gometalinter, which acts as a wrapper around a bunch of different linters and picks up more potential problems and inefficiencies than using go vet and golint alone. You can install it with the commands: $ go get github.com/alecthomas/gometalinter $ which gometalinter /home/alex/Code/go/bin/gometalinter $ gometalinter --install Once that's done, you'll need to install the SublimeLinter-contrib-gometalinter plugin. This acts as the bridge between SublimeLinter and gometalinter. Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter-contrib-gometalinter and hit Enter to install the package. By default the linter will run in the background as you type, and errors will be shown in the Sublime Text status bar at the bottom of the screen. But I've found suits me more to only lint when saving a file and to display all errors at once in a panel. If you want to do the same, go to Preferences > Package Settings > SublimeLinter > Settings and add the following settings to the SublimeLinter Settings - User file: { "show_panel_on_save": "window", "lint_mode": "save", } I should mention that the SublimeLinter-contrib-gometalinter plugin only executes the 'fast' linters included in gometalinter. You can see exactly which ones are run by checking the source code. Autocompletion For autocompletion I'm using the gocode package, which provides a deamon for code completion. You can install it like so: $ go get github.com/mdempsky/gocode $ which gocode /home/alex/Code/go/bin/gocode There isn't currently a gocode plugin available via Sublime Text package control (I might add one soon!)… but there is a plugin included in the subl3 directory within the gocode source itself. You should be able to copy it into your Sublime Text Packages directory with the following command: $ cp -r $GOPATH/src/github.com/mdempsky/gocode/subl3 ~/.config/sublime-text-3/Packages/gocode If you open the command palette and run Package Control: List Packages you should see a gocode entry in the list. By default Sublime Text will make autocomplete suggestions whenever a letter is pressed. But when working with Go I like also to display potential method names whenever I hit the . character. You can make that happen by going to Preferences > Settings and adding a new trigger in the Preferences.sublime-settings - User file: { ... "auto_complete_triggers": [ {"selector": "text.html", "characters": " You'll need to then restart Sublime Text for the settings to take effect. Once you have, you should have autocomplete working nicely and looking something like this: Code navigation To help with navigating code I use the guru tool, which you can install with the following command: $ go get golang.org/x/tools/cmd/guru $ which guru /home/alex/Code/go/bin/guru To integrate this with Sublime Text you'll also need to install the GoGuru plugin like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type GoGuru and hit Enter to install the package. To use the GoGuru tool, first place your cursor over the piece of code you're interested in. Then if you open the command palette and type the GoGuru prefix you'll see a list of available commands, including: GoGuru: callees – Show possible targets of selected function call GoGuru: callers – Show possible callers of selected function GoGuru: callstack – Show path from callgraph root to selected function GoGuru: definition – Show declaration of selected identifier GoGuru: describe – Describe selected syntax: definition, methods, etc GoGuru: freevars – Show free variables of selection GoGuru: implements – Show 'implements' relation for selected type or method GoGuru: jump to definition – Open the file at the declaration of selected identifier GoGuru: peers – Show send/receive corresponding to selected channel op GoGuru: pointsto – Show variables the selected pointer may point to GoGuru: referrers – Show all refs to thing denoted by selected identifier GoGuru: what – Show basic information about the selected syntax node GoGuru: whicherrs – Show possible values of the selected error variable You can find a detailed description these commands and their behaviour in this GoogleDoc. I don't use the GoGuru plugin as often as the others, but when working on a unfamiliar codebase it definitely makes navigating code and building up a mental map of how things work easier. I find the GoGuru: jump to definition and GoGuru: callers commands particularly useful, and easier to use than grepping or running Ctrl+F on the repository. As an illustration, here's a screenshot of running GoGuru: callers on the Sum function: Snippets Sublime Text ships with a pretty good workflow for creating and using custom snippets. If you're not already familiar with this Jimmy Zhang has written a great in-depth guide that I recommend reading. My most frequently-used snippet is probably this one for creating a HTTP handler function: hf source.go One thing that bugged me for a while was the built-in snippets for Go that Sublime Text ships with. In particular I didn't like the way that the main() snippet kept triggering whenever I wrote out "package main". If, like me, you want to edit these built-in snippets the easiest way is probably with the PackageResourceViewer plugin. You can install this as follows: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type PackageResourceViewer and hit Enter to install the package. Once installed you can open the command palette and run PackageResourceViewer: Open Resource which will list all packages on your system. If you navigate through Go > Snippets/ you should see a list of all the built-in snippets and you can open and edit them as you wish. Hint: You can also use PackageResourceViewer to edit your own custom snippets without leaving SublimeText. If – for example – your custom snippets are saved under your Packages/User directory, you can open them by running PackageResourceViewer: Open Resource and navigating to the User folder.

• Configuring sql.DB for better performance
  Alex Edwards

  There are a lot of good tutorials which talk about Go's sql.DB type and how to use it to execute SQL database queries and statements. But most of them gloss over the SetMaxOpenConns(), SetMaxIdleConns() and SetConnMaxLifetime() methods — which you can use to configure the behavior of sql.DB and alter its performance. In this post I'd like to explain exactly what these settings do and demonstrate the (positive and negative) impact that they can have. Open and idle connections I'll begin with a little background. A sql.DB object is a pool of many database connections which contains both 'in-use' and 'idle' connections. A connection is marked as in-use when you are using it to perform a database task, such as executing a SQL statement or querying rows. When the task is complete the connection is marked as idle. When you instruct sql.DB to perform a database task, it will first check if any idle connections are already available in the pool. If one is available then Go will reuse this existing connection and mark it as in-use for the duration of the task. If there are no idle connections in the pool when you need one, then Go will create an additional new additional connection. The SetMaxOpenConns method By default there's no limit on the number of open connections (in-use + idle) at the same time. But you can implement your own limit via the SetMaxOpenConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently open connections (in-use + idle) // to 5. Setting this to less than or equal to 0 will mean there is no // maximum limit (which is also the default setting). db.SetMaxOpenConns(5) In this example code the pool now has a maximum limit of 5 concurrently open connections. If all 5 connections are already marked as in-use and another new connection is needed, then the application will be forced to wait until one of the 5 connections is freed up and becomes idle. To illustrate the impact of changing MaxOpenConns I ran a benchmark test with the maximum open connections set to 1, 2, 5, 10 and unlimited. The benchmark executes parallel INSERT statements on a PostgreSQL database and you can find the code in this gist. Here's the results: BenchmarkMaxOpenConns1-8 500 3129633 ns/op 478 B/op 10 allocs/op BenchmarkMaxOpenConns2-8 1000 2181641 ns/op 470 B/op 10 allocs/op BenchmarkMaxOpenConns5-8 2000 859654 ns/op 493 B/op 10 allocs/op BenchmarkMaxOpenConns10-8 2000 545394 ns/op 510 B/op 10 allocs/op BenchmarkMaxOpenConnsUnlimited-8 2000 531030 ns/op 479 B/op 9 allocs/op PASS Edit: To make clear, the purpose of this benchmark is not to simulate 'real-life' behaviour of an application. It's solely to help illustrate how sql.DB behaves behind the scenes and the impact of changing MaxOpenConns on that behaviour. For this benchmark we can see that the more open connections that are allowed, the less time is taken to perform the INSERT on the database (3129633 ns/op with 1 open connection compared to 531030 ns/op for unlimited connections — about 6 times quicker). This is because the more open connections that are permitted, the more database queries can be performed concurrently. The SetMaxIdleConns method By default sql.DB allows a maximum of 2 idle connections to be retained in the connection pool. You can change this via the SetMaxIdleConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently idle connections to 5. Setting this // to less than or equal to 0 will mean that no idle connections are retained. db.SetMaxIdleConns(5) In theory, allowing a higher number of idle connections in the pool will improve performance because it makes it less likely that a new connection will need to be established from scratch — therefore helping to save resources. Lets take a look at the same benchmark with the maximum idle connections is set to none, 1, 2, 5 and 10 (and the number of open connections is unlimited): BenchmarkMaxIdleConnsNone-8 300 4567245 ns/op 58174 B/op 625 allocs/op BenchmarkMaxIdleConns1-8 2000 568765 ns/op 2596 B/op 32 allocs/op BenchmarkMaxIdleConns2-8 2000 529359 ns/op 596 B/op 11 allocs/op BenchmarkMaxIdleConns5-8 2000 506207 ns/op 451 B/op 9 allocs/op BenchmarkMaxIdleConns10-8 2000 501639 ns/op 450 B/op 9 allocs/op PASS When MaxIdleConns is set to none, a new connection has to be created from scratch for each INSERT and we can see from the benchmarks that the average runtime and memory usage is comparatively high. Allowing just 1 idle connection to be retained and reused makes a massive difference to this particular benchmark — it cuts the average runtime by about 8 times and reduces memory usage by about 20 times. Going on to increase the size of the idle connection pool makes the performance even better, although the improvements are less pronounced. So should you maintain a large idle connection pool? The answer is it depends on the application. It's important to realise that keeping an idle connection alive comes at a cost — it takes up memory which can otherwise be used for both your application and the database. It's also possible that if a connection is idle for too long then it may become unusable. For example, MySQL's wait_timeout setting will automatically close any connections that haven't been used for 8 hours (by default). When this happens sql.DB handles it gracefully. Bad connections will automatically be retried twice before giving up, at which point Go will remove the connection from the pool and create a new one. So setting MaxIdleConns too high may actually result in connections becoming unusable and more resources being used than if you had a smaller idle connection pool (with fewer connections that are used more frequently). So really you only want to keep a connection idle if you're likely to be using it again soon. One last thing to point out is that MaxIdleConns should always be less than or equal to MaxOpenConns. Go enforces this and will automatically reduce MaxIdleConns if necessary. The SetConnMaxLifetime method Let's now take a look at the SetConnMaxLifetime() method which sets the maximum length of time that a connection can be reused for. This can be useful if your SQL database also implements a maximum connection lifetime or if — for example — you want to facilitate gracefully swapping databases behind a load balancer. You use it like this: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum lifetime of a connection to 1 hour. Setting it to 0 // means that there is no maximum lifetime and the connection is reused // forever (which is the default behavior). db.SetConnMaxLifetime(time.Hour) In this example all our connections will 'expire' 1 hour after they were first created, and cannot be reused after they've expired. But note: This doesn't guarantee that a connection will exist in the pool for a whole hour; it's quite possible that the connection will have become unusable for some reason and been automatically closed before then. A connection can still be in use more than one hour after being created — it just cannot start to be reused after that time. This isn't an idle timeout. The connection will expire 1 hour after it was first created — not 1 hour after it last became idle. Once every second a cleanup operation is automatically run to remove 'expired' connections from the pool. In theory, the shorter ConnMaxLifetime is the more often connections will expire — and consequently — the more often they will need to be created from scratch. To illustrate this I ran the benchmarks with ConnMaxLifetime set to 100ms, 200ms, 500ms, 1000ms and unlimited (reused forever), with the default settings of unlimited open connections and 2 idle connections. These time periods are obviously much, much shorter than you'd use in most applications but they help illustrate the behaviour well. BenchmarkConnMaxLifetime100-8 2000 637902 ns/op 2770 B/op 34 allocs/op BenchmarkConnMaxLifetime200-8 2000 576053 ns/op 1612 B/op 21 allocs/op BenchmarkConnMaxLifetime500-8 2000 558297 ns/op 913 B/op 14 allocs/op BenchmarkConnMaxLifetime1000-8 2000 543601 ns/op 740 B/op 12 allocs/op BenchmarkConnMaxLifetimeUnlimited-8 3000 532789 ns/op 412 B/op 9 allocs/op PASS In these particular benchmarks we can see that memory usage was more than 3 times greater with a 100ms lifetime compared to an unlimited lifetime, and the average runtime for each INSERT was also slightly longer. If you do set ConnMaxLifetime in your code, it is important to bear in mind the frequency at which connections will expire (and subsequently be recreated). For example, if you have 100 total connections and a ConnMaxLifetime of 1 minute, then your application can potentially kill and recreate up to 1.67 connections (on average) every second. You don't want this frequency to be so great that it ultimately hinders performance, rather than helping it. Exceeding connection limits Lastly, this article wouldn't be complete without mentioning what happens if you exceed a hard limit on the number of database connections. As an illustration, I'll change my postgresql.conf file so only a total of 5 connections are permitted (the default is 100)... max_connections = 5 And then rerun the benchmark test with unlimited open connections... BenchmarkMaxOpenConnsUnlimited-8 --- FAIL: BenchmarkMaxOpenConnsUnlimited-8 main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already FAIL As soon as the hard limit of 5 connections is hit my database driver (pq) immediately returns a sorry, too many clients already error message instead of completing the INSERT. To prevent this error we need to set the total maximum of open connections (in-use + idle) in sql.DB to comfortably below 5. Like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the number of open connections (in-use + idle) to a maximum total of 3. db.SetMaxOpenConns(3) Now there will only ever be a maximum of 3 connections created by sql.DB at any moment in time, and the benchmark should run without any errors. But doing this comes with a big caveat: when the open connection limit is reached, any new database tasks that your application needs to execute will be forced to wait until a connection becomes free. In the context of a web application, for example, the user's HTTP request would appear to 'hang' and could potentially even timeout while waiting for the database task to be run. To mitigate this you should always pass in a context.Context object with a fixed, fast, timeout when making database calls, using the context-enabled methods like ExecContext(). An example can be seen in the gist here. Summary As a rule of thumb, you should explicitly set a MaxOpenConns value. This should be comfortably below any hard limits on the number of connections imposed by your database and infrastructure. In general, higher MaxOpenConns and MaxIdleConns values will lead to better performance. But the returns are diminishing, and you should be aware that having a too-large idle connection pool (with connections that are not re-used and eventually go bad) can actually lead to reduced performance. To mitigate the risk from point 2 above, you may want to set a relatively short ConnMaxLifetime. But you don't want this to be so short that leads to connections being killed and recreated unnecessarily often. MaxIdleConns should always be less than or equal to MaxOpenConns. For small-to-medium web applications I typically use the following settings as a starting point, and then optimize from there depending on the results of load-testing with real-life levels of throughput. db.SetMaxOpenConns(25) db.SetMaxIdleConns(25) db.SetConnMaxLifetime(5*time.Minute)