Reading

A reading list from the RSS feeds I follow (happy to get good recommendations)

See my feeds.yaml.

Dave Cheney

• The story of the one line fix
  Dec 15, 2020

  Picture yourself, an engineer working at the hottest distributed microservices de jour, assigned to fix a bug. You jump into an unfamiliar codebase and quickly locate the line where the problem occurred. The fix is simple, just return early or substitute a default value in the case that one cannot be determined from your input. […]

• How to dump the GOSSAFUNC graph for a method
  Jun 19, 2020

  The Go compiler’s SSA backend contains a facility to produce HTML debugging output of the compilation phases. This post covers how to print the SSA output for function and methods. Let’s start with a sample program which contains a function, a value method, and a pointer method: Control of the SSA debugging output is via […]

• Diamond interface composition in Go 1.14
  May 24, 2020

  Per the overlapping interfaces proposal, Go 1.14 now permits embedding of interfaces with overlapping method sets. This is a brief post explain what this change means: Let’s start with the definition of the three key interfaces from the io package; io.Reader, io.Writer, and io.Closer: Just as embedding a type inside a struct allows the embedded type’s […]

• Fatih’s question
  May 16, 2020

  A few days ago Fatih posted this question on twitter. I’m going to attempt to give my answer, however to do that I need to apply some simplifications as my previous attempts to answer it involved a lot of phrases like a pointer to a pointer, and other unhelpful waffling. Hopefully my simplified answer can […]

• Ensmallening Go binaries by prohibiting comparisons
  May 09, 2020

  Conventional wisdom dictates that the larger the number of types declared in a Go program, the larger the resulting binary. Intuitively this makes sense, after all, what’s the point in defining a bunch of types if you’re not going to write code that operates on them. However, part of the job of a linker is […]

• Mid-stack inlining in Go
  May 02, 2020

  In the previous post I discussed how leaf inlining allows the Go compiler to reduce the overhead of function calls and extend optimisation opportunities across function boundaries. In this post I’ll discuss the limits of inlining and leaf vs mid-stack inlining. The limits of inlining Inlining a function into its caller removes the call’s overhead […]

• Inlining optimisations in Go
  Apr 25, 2020

  This is a post about how the Go compiler implements inlining and how this optimisation affects your Go code. n.b. This article focuses on gc, the de facto Go compiler from golang.org. The concepts discussed apply broadly to other Go compilers like gccgo and tinygo but may differ in implementation and efficacy. What is inlining? […]

• go test -v streaming output
  Mar 10, 2020

  The testing package is one of my favourite packages in the Go standard library, not just because of its low noise approach to unit testing, but, over the lifetime of Go, it has received a steady stream of quality of life improvements driven by real world usage. The most recent example of this is, in […]

• Are large slices more expensive than smaller ones?
  Mar 01, 2020

  Programmers have a tendency to be superstitious. Particularly, when a programmer hears that copies are expensive, they start to see them everywhere, especially when they learn that, in Go, every assignment is a copy. Consider this code; x is three orders of magnitude larger than y, is the assignment of x to a more expensive […]

• The Zen of Go
  Feb 23, 2020

  This article was derived from my GopherCon Israel 2020 presentation. It’s also quite long. If you’d prefer a shorter version, head over to the-zen-of-go.netlify.com. A recording of the presentation is available on YouTube. How should I write good code? Something that I’ve been thinking about a lot recently, when reflecting on the body of my […]

• Dynamically scoped variables in Go
  Dec 08, 2019

  This is a thought experiment in API design. It starts with the classic Go unit testing idiom: func TestOpenFile(t *testing.T) { f, err := os.Open("notfound") if err != nil { t.Fatal(err) } // ... } What’s the problem with this code? The assertion. if err != nil { ... } is repetitive and in the […]

• Internets of Interest #15: The Queen of the Skies
  Dec 05, 2019

  If, like me, you’re a commercial aviation otaku, this walkthrough of an enthusiast built 747 cockpit simulator should be highly relevant to your interests.

• Complementary engineering indicators
  Dec 04, 2019

  Last year I had the opportunity to watch Cat Swetel’s presentation The Development Metrics You Should Use (but Don’t). The information that could be gleaned from just tracking the start and finish date of work items was eye opening. If you’re using an issue tracker this information is probably already (perhaps with some light data […]

• Internets of interest #14: UNIX v0
  Nov 17, 2019

  Read more over at the Living Computer Museum’s restoration page.

• Internets of interest: Warner Losh on the first ten years of UNIX
  Nov 16, 2019

  UNIX turns 50 this year which means 7th edition Research UNIX is that 40.

• Use internal packages to reduce your public API surface
  Oct 06, 2019

  In the beginning, before the go tool, before Go 1.0, the Go distribution stored the standard library in a subdirectory called pkg/ and the commands which built upon it in cmd/. This wasn’t so much a deliberate taxonomy but a by product of the original make based build system. In September 2014, the Go distribution […]

Antirez

• The open source paradox
  Oct 03, 2020

  A new idea is insinuating in social networks and programming communities. It’s the proportionality between the money people give you for coding something, and the level of demand for quality they can claim to have about your work. As somebody said, the best code is written when you are supposed to do something else [1]. Like a writer will do her best when writing that novel that, maybe, nobody will pay a single cent for, and not when doing copywriting work for a well known company, programmers are likely to spend more energies in their open source side projects than during office hours, while writing another piece of a project they feel stupid, boring, pointless. And, if the company is big enough, chances are it will be cancelled in six months anyway or retired one year after the big launch. Open source is different, it’s an artifact, it’s a transposition in code of what you really want to do, of what you feel software should be, or just of all your fun and joy, or even anger you are feeling while coding. And you want it to rock, to be perfect, and you can’t sleep at night if there is a fucking heisenbug. So if a user of your software is addressing you because some part of your code sucks, and is willing to work with you to do something about it, and is very demanding, don’t think they are abusing you because they are not paying you. It’s not about money. You can ignore bugs if you want, and ignore their complains, you can do that since you don’t have a contract to do otherwise, but they are helping you, they care about the same thing you care: your software quality, grandiosity, perfection. The real right you have, and often don’t exploit, is that you are the only one that can decide about the design of your software. So you are entitled to refuse a pull request, or a proposal to follow good practices, because you feel that what somebody is contributing does not fit in the big picture of what you are designing and building. But if you recognize that somebody is talking you about something that is, really, a defect in your software, don’t do the error of reducing the interaction to a vile matter of money. You are doing work for free, they are risking their asses deploying what you wrote, you both care about quality. EDIT: If you write OSS and you are upset about user demands, have you ever thought that maybe, at this point, your work is more similar to office work for some reason? EDIT 2: A HN user asked the reasons for such title. The paradox is that the OSS writer cares and is often willing to fix code she writes for free, more than the other paid work she does. [1] "The best programs are the ones written when the programmer is supposed to be working on something else." - Melinda Varian. https://twitter.com/CodeWisdom/status/1309470447667421189 Comments

• The end of the Redis adventure
  Jun 30, 2020

  When I started the Redis project more than ten years ago I was in one of the most exciting moments of my career. My co-founder and I had successfully launched two of the major web 2.0 services of the Italian web. In order to make them scalable we had to invent many new concepts, that were already known in the field most of the times, but we didn’t know, nor we cared to check. Problem? Let’s figure out a solution. We wanted to solve problems but we wanted, even more, to have fun. This was the playful environment where Redis was born. But now Redis is, incredibly, one of the main parts of so many things. And year after year my work changed from building this thing to making sure that it was also as useful as possible, as reliable as possible. And in recent years, what I do every day changed so much that most of my attention is spent in checking what other developers tell me about the Redis code, how to improve it, the changes it requires to be more correct or faster or more secure. However I never wanted to be a software maintainer. I write code in order to express myself, and I consider what I code an artifact, rather than just something useful to get things done. I would say that what I write is useful just as a side effect, but my first goal is to make something that is, in some way, beautiful. In essence, I would rather be remembered as a bad artist than a good programmer. Now I’m asked more and more, by the circumstances created by a project that became so important, to express myself less and to maintain the project more. And this is indeed exactly what Redis needs right now. But this is not what I want to do, and I stretched myself enough during the past years. So, dear Redis community, today I’m stepping back as the Redis maintainer. My new position will be, on one side, an “ideas” person at Redis Labs, in order to provide inputs for new Redis possibilities: I’ll continue to be part of the Redis Labs advisory board. On the other hand however my hands will be free, and I’ll do something else, that could be writing code or not, who knows, I don’t want to make plans for now. However I’m very skeptical about me not writing more code in the future. It’s just too much fun :D I leave Redis in the hands of the Redis community. I asked my colleagues Yossi Gottlieb and Oran Agra to continue to maintain the project starting from today: these are the people that helped me the most in recent years, and that tried hard, even when it was not “linear” to follow me in my very subjective point of views, to understand what my vision on Redis was. Since I don’t want to be part of how the new Redis development setup will be shaped (that is the most meta of the maintenance tasks, exactly what I want to avoid), I’ll just leave Yossi and Oran the task of understanding how to interface with the rest of the Redis developers to find a sustainable development model, you can hear directly from Yossi and Oran in this blog post: https://redislabs.com/blog/new-governance-for-redis/ I believe I’m not just leaving Redis in the hands of a community of expert programmers, but also in the hands of people who care about the legacy of the community spirit of Redis. In eleven years I hope I was able to provide a point of view that certain persons understood, about an alternative way to write software. I hope that such point of view will be taken into consideration in the evolution of Redis. Redis was the most stressful thing I did in my career, and probably also the most important. I don’t like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals. Thank you for your humanity and your help, and for what you taught me. You know who you are! I want to also say thank you to the companies and individuals inside such companies that allowed me to write open source every day for so many years, with the freedom to do what I believed to be correct for the user base. Redis Labs, VMware and Pivotal, thank you for your great help and generosity. As I said, I don’t really know what there is for me in my future, other than the involvement with the Redis advisory board. I guess that for some time, just look around is a good idea, without doing too many things. I would like to explore more a few hobbies of mine. Writing blog posts is also a major thing that I wanted to do but did less and less because of time concerns. Recently I published videos in Italian language explaining technological concepts to the general public, I had fun doing that and received good feedbacks, maybe I’ll do more of that as well. Anyway I guess some of you know that I’m active on Twitter as @antirez. If you are interested in what an old, strange programmer will do next, see you there. Comments

• Redis 6.0.0 GA is out!
  Apr 30, 2020

  Finally Redis 6.0.0 stable is out. This time it was a relatively short cycle between the release of the first release candidate and the final release of a stable version. It took about four months, that is not a small amount of time, but is not a lot compared to our past records :) So the big news are the ones announced before, but with some notable changes. The old stuff are: SSL, ACLs, RESP3, Client side caching, Threaded I/O, Diskless replication on replicas, Cluster support in Redis-benchmark and improved redis-cli cluster support, Disque in beta as a module of Redis, and the Redis Cluster Proxy (now at https://github.com/RedisLabs/redis-cluster-proxy). So what changed between RC1 and today, other than stability? 1. Client side caching was redesigned in certain aspects, especially the caching slot approach was discarded in favor of just using key names. After analyzing the alternatives, with the help of other Redis core team members, in the end this approach looks better. Other than that, finally the feature was completed with the things I had in the backlog for the feature, especially the “broadcasting mode”, that I believe will be one of the most popular usage modes of the feature. When broadcasting is used, the server no longer try to remember what keys each client requested. Instead clients subscribe to key prefixes: they’ll get notifications every time a key matching the prefix is modified. This means more messages (but only for the selected prefixes), but no memory effort in the server side. Moreover the opt-in / opt-out mode is now supported, so it is possible for clients not using the broadcasting mode, to exactly tell the server about what the client will cache, to reduce the number of invalidation messages. Basically the feature is now much better both when a low-memory mode is needed, and when a very selective (low-bandwidth) mode is needed. 2. This was an old request by many users. Now Redis supports a mode where RDB files used for replication are immediately deleted if no longer useful. In certain environments it is a good idea to never have the data around on disk, but just in memory. 3. ACLs are better in a few regards. First, there is a new ACL LOG command that allows to see all the clients that are violating the ACLs, accessing commands they should not, accessing keys they should not, or with failed authentication attempts. The log is actually in memory, so every external agent can call “ACL LOG” to see what’s going on. This is very useful in order to debug ACL problems. But my preferred feature is the reimplementation of ACL GENPASS. Now it uses SHA256 based HMAC, and accepts an optional argument to tell the server how many bits of unguessable pseudo random string you want to generate. Redis seeds an internal key at startup from /dev/urandom, and later uses the HMAC in counter mode in order to generate the other random numbers: this way you can abuse the API, and call it every time you want, since it will be very fast. Want to generate an unguessable session ID for your application? Just call ACL GENPASS. And so forth. 4. PSYNC2, the replication protocol, is now improved. Redis will be able to partially resynchronize more often, since now is able to trim the final PINGs in the protocol, to make more likely that replicas and masters can find a common offset. 5. Redis commands with timeouts are now much better: not only BLPOP and other commands that used to accept seconds, now accept decimal numbers, but the actual resolution was improved in order to never be worse than the current “HZ” value, regardless of the number of clients connected. 6. RDB files are now faster to load. You can expect a 20/30% improvement, depending on the file actual composition (larger or smaller values). INFO is also faster now when there are many clients connected, this was a long time problem that now is finally gone. 7. We have a new command, STRALGO, that implements complex string algorithms. For now the only one implemented is LCS (longest common subsequence), an important algorithm used, among the other things, in order to compare the RNA of the coronaviruses (and in general the DNA and RNA of other organisms). What is happening is too big, somewhat a trace inside Redis needed to remain. Redis 6 is the biggest release of Redis *ever*, so even if it is stable, handle it with care, test it for your workload before putting it in production. We never saw big issues so far, but make sure to be careful. As we collect bug reports, we will prepare to release Redis 6.0.1 ASAP. A big thank you to the many people that wrote code with me in this release, and to all the companies that sponsored both my work (Thanks Redis Labs), and the the work of the other contributors (Thanks other companies). Also a big thank you to the many that signaled bugs with care, sometimes following the boring process of reiterating after making some changes, or that suggested improvements of any kind. As usually you can find Redis 6 in different places: at https://redis.io as tarball, and in the Github repository tagged as “6.0.0”. Enjoy Redis 6, antirez Comments

• Redis 6 RC1 is out today
  Dec 19, 2019

  So it happened again, a new Redis version reached the release candidate status, and in a few months it will hit the shelves of most supermarkets. I guess this is the most “enterprise” Redis version to date, and it’s funny since I took quite some time in order to understand what “enterprise” ever meant. I think it’s word I genuinely dislike, yet it has some meaning. Redis is now everywhere, and it is still considerably able to “scale down”: you can still download it, compile it in 30 seconds, and run it without any configuration to start hacking. But being everywhere also means being in environments where things like encryption and ACLs are a must, so Redis, inevitably, and more than thanks to me, I would say, in spite of my extreme drive for simplicity, adapted. But what’s interesting is that, even additions may be done in very opinionated ways. Redis ACLs hardly resemble something you saw in other systems, and SSL support was written in a few iterations in order to finally pick the idea that was the most sounding, from the point of view of letting the core as clean as possible. I’m quite happy with the result. Redis 6 does not bring just ACLs and SSL, it is the largest release of Redis ever as far as I can tell, and the one where the biggest amount of people participated. So, let’s start with credits. Who made Redis 6? This is the list of contributors by commits (it’s a terrible metric, but the one I can easily generate), having at least two commits, and excluding merge commits. Also note that the number of commits in my case may be inflated a lot by the fact that I fix many small stuff here and there constantly. 685 antirez 81 zhaozhao.zz 76 Oran Agra 51 artix 28 Madelyn Olson 27 Yossi Gottlieb 15 David Carlier 14 Guy Benoish 14 Guy Korland 13 Itamar Haber 9 Angus Pearson 8 WuYunlong 8 yongman 7 vattezhang 7 Chris Lamb 5 Dvir Volk 5 meir@redislabs.com 5 chendianqiang 5 John Sully 4 dejun.xdj 4 Daniel Dai 4 Johannes Truschnigg 4 swilly22 3 Bruce Merry 3 filipecosta90 3 youjiali1995 2 James Rouzier 2 Andrey Bugaevskiy 2 Brad Solomon 2 Hamid Alaei 2 Michael Chaten 2 Steve Webster 2 Wander Hillen 2 Weiliang Li 2 Yuan Zhou 2 charsyam 2 hujie 2 jem 2 shenlongxing 2 valentino 2 zhudacai 00228490 2 喜欢兰花山丘 Thanks to all the above folks, it was a great team work ladies and gentlemen. The list of new features in the change log is the following: * Many new modules APIs. * Better expire cycle. * SSL * ACLs * RESP3 * Client side caching * Threaded I/O * Diskless replication on replicas * Redis-benchmark cluster support + Redis-cli improvements * Systemd support rewrite. * Redis Cluster proxy was released with Redis 6 (but different repository). * A Disque module was released with Redis 6 (but different repository). Many big things, as you can see. I’ll spend a few words on selected ones. RESP3 === After ten years we needed a new protocol, I talked extensively about it here http://antirez.com/news/125, but then changed my mind, so the RESP3 protocol in Redis 6 is “opt in”. The connection starts in RESP2 mode, and only if you do a handshake using the new HELLO command, you enter in the new protocol mode. Why a new protocol? Because the old one was not semantical enough. There are other features in RESP3, but the main idea was the ability to return complex data types from Redis directly, without the client having to know in what type to convert the flat arrays returned, or the numbers returned instead of proper boolean values, and so forth. Since RESP3 is not the only protocol supported I expect the adoption to be slower than expected, but maybe this is not a bad thing after all: we’ll have time to adapt. ACLs === The best introduction to Redis ACLs is the ACL documentation itself (https://redis.io/topics/acl), even if probably it needs some update to match the last minute changes. So it’s more interesting to talk about motivations here. Redis needed ACLs because people need ACLs in bigger environments, in order to control better which client can do certain operations. But another main point about adding ACLs to Redis was isolation in order to defend the data against application bugs. If your worker can only do BRPOPLPUSH, the chance of the new developer adding for debugging a FLUSHALL that ends in production code for error and creates a nightmare for 5 hours, is lower. ACLs in Redis are for free, both operationally, because if you don’t use them, you can avoid knowing they are supported at all, and from the point of view of performances, since the overhead is not measurable. I guess it’s a good deal to have them. Bonus point, we have a Redis modules interface for ACLs now, so you can write custom authentication methods. SSL === It’s 2019, almost 2020, and there are new regulations. The only problem was doing it right. And doing it right required doing it wrong, understanding the limitations, and then abstracting the Redis connections in order to do it right. This work was entirely performed without my help, which shows how the Redis development process changed in recent times. Client side caching === I blogged about it here http://antirez.com/news/130, however I think that right now this is the most immature feature of Redis 6. Yep, it’s cool that the server can assist you in caching client side values, but I want to improve this before Redis 6 GA is out. Especially it could be very good to add a new mode that requires the server to maintain no state about clients, or very little state at all, and trade this with more messages. Moreover right now the messages to expire certain “cache slots” can’t be compiled in a single one. There is more work to do in January about this feature, but it will be a good one. Disque as a module === Finally I did it :-) https://github.com/antirez/disque-module, and I’m very happy with the result. Disque as a module really shows how powerful is the Redis module system at this point. Cluster message bus APIs, ability to block and resume clients, timers, AOF and RDB control of module private data. If you don’t know what Disque is, check the repository: the README is quite extensive. Cluster Proxy === My colleague Fabio worked for months at this Redis Cluster proxy: https://github.com/artix75/redis-cluster-proxy. It is ages that I want to see it happening, the client landscape is very fragmented when the topic is Redis Cluster support, so now we have a (work in progress) proxy that can do many interesting things. The main one is to abstract the Redis Cluster for clients, like if they were talking to a single instance. Another one is to perform multiplexing, at least when it is simple and clients just use simple commands and features. When there is to block or to perform transactions, the proxy allocates a different set of connections for the client. The proxy is also completely threaded, so it can be a good way in order to maximize the CPU usage in case most of your CPU time is spent in I/O. Check the project README for status and give it a try! Modules === With Redis 6 the modules API is totally at a new level. This is one of the part of Redis that matured faster in our history, because Redis Labs used the modules system from day zero in order to develop very complex stuff, not just trivial examples. Some time ago I started the Disque port, and this also motivated to bring me new features to the modules system. The result is that Redis is really a framework in order to write systems as modules, without having to invent everything from scratch, and being BSD licensed, Redis is really an open platform to write systems. Internals === There are tons of improvements to the Redis internals: the way commands are replicated changed quite a bit, the expires are now using a different algorithm which is faster and more cache obvious. Status and ETA === Today we went RC1, and I hope that between end of March, or at worst, May, you’ll see the GA ready. Right now Redis 6 is definitely testable and the chance you run into a bug is very small. Yet it includes a ton of code changes, and the new features are composed of new code that nobody ran in production before. So if you find bad things, please report them in the issue system describing at your best what happened. Thanks everybody that made this release possible and that will work in the next months to bring it to a very stable state. Oh, I almost forgot! This is the LOLWUT command interactive art for version 6: img://antirez.com/misc/lolwut6.png Every run displays a different landscape that is randomly generated. Comments

• Client side caching in Redis 6
  Jul 04, 2019

  [Note: this post no longer describes the client side implementation in the final implementation of Redis 6, that changed significantly, see https://redis.io/topics/client-side-caching] The New York Redis day was over, I get up at the hotel at 5:30, still pretty in sync with the Italian time zone and immediately went walking on the streets of Manhattan, completely in love with the landscape and the wonderful feeling of being just a number among millions of other numbers. Yet I was thinking at the Redis 6 release with the feeling that, what was probably the most important feature at all, the new version of the Redis protocol (RESP3), was going to have a very slow adoption curve, and for good reasons: wise people avoid switching tools without very good reasons. After all why I wanted to improve the protocol so badly? For two reasons mainly, to provide clients with more semantical replies, and in order to open to new features that were hard to implement with the old protocol; one feature in particular was the most important to me: client side caching. Rewind back to about one year ago. I arrived at Redis Conf 2018, in San Francisco, with the firm idea that client side caching was the most important thing in the future of Redis. If we need fast stores and fast caches, then we need to store a subset of the information inside the client. It is a natural extension of the idea of serving data with small delays and at a big scale. Actually almost every very large company already does it, because it is the only way to survive to the load eventually. Yet Redis had no way to assist the client in such process. A fortunate coincidence wanted Ben Malec having a talk at Redis Conf exactly about client side caching [1], just using the tools that Redis provides and a number of very clever ideas. [1] https://www.youtube.com/watch?v=kliQLwSikO4 The approach taken by Ben really opened my imagination. There were two key ideas Ben used in order to make his design work. The first was to use the Redis Cluster idea of “hash slots” in order to divide keys into 16k groups. That way clients would not need to track the validity of each key, but could use a single metadata entry for a group of keys. Ben used Pub/Sub in order to send the notifications when keys where changed, so he needed some help by the application in all its parts, however the schema was very solid. Modify a key? Also publish a message invalidating it. On the client side, are you caching keys? Remember the timestamp at which you cache each key, and also when receiving invalidation messages, remember the invalidation time for each slot. When using a given cached key, do a lazy eviction, by checking if the key you cached has a timestamp which is older than the timestamp of the invalidation received for the slot this key belongs to: in that case the key is stale data, you have to ask the server again. After watching the talk, I realized that this was a great idea to be used inside the server, in order to allow Redis to do part of the work for the client, and make client side caching simpler and more effective, so I returned home and wrote a document describing my design [2]. [2] https://groups.google.com/d/msg/redis-db/xfcnYkbutDw/kTwCozpBBwAJ But to make my design working I had to focus on switching the Redis protocol to something better, so I started writing the specification and later the code for RESP3, and the other Redis 6 things like ACL and so forth, and client side caching joined the huge room of the many ideas for Redis that I abandoned in some way or the other for lack of time. Yet I was among the streets of New York thinking about this idea. Later went to lunch and coffee break with friends from the conference. When I returned to my hotel room I had all the evening left, and most of the next day before the flight, so I started writing the implementation of client side caching for Redis 6, closely following the proposal I wrote to the group one year ago: it still looked great. Redis server-assisted client side caching, finally called “tracking” (but I may change idea), is a very simple feature composed of just a few key ideas. The key space is split into “caching slots”, but they are a lot more than the hash slots used by Ben. We use 24 bits of the output of CRC64, so there are a bit more than 16 millions different slots. Why so much? Because I think you want to have a server with 100 millions of keys, and yet an invalidation message should not affect more than a few keys in the client side cache. The memory overhead inside Redis to take the invalidation table is 130 megabyte: an array of 8 bytes pointers to 16M entries. That’s ok with me, if you want the feature you are going to make a great use of all the memory you have in the clients, so to use 130MB server side is fine; what you win is a much more fine grained invalidation. Clients enable the feature in an “opt in” way, with a simple command: CLIENT TRACKING on The server replies the good old +OK, and starting from that moment, every command that is flagged as “read only” in the command table, will not just return the keys to the caller, it will also, as a side effect, remember the caching slots of all the keys the client requested so far (but only the ones using a read only command, that's the agreement between the server and the client). The way Redis stores this information is simple. Each Redis client has an unique ID, so if client ID 123 performs an MGET about keys hashing to the slot 1, 2, and 5, we’ll have the Invalidation Table with the following entry: 1 -> [123] 2 -> [123] 5 -> [123] But later also client ID 444 will ask about keys in the slot 5, so the table will be like: 5 -> [123, 444] Now some other client changes some key in the slot 5. What happens is that Redis will check the Invalidation Table, to find that both clients 123 and 444 may have cached keys on that slot. We’ll send an invalidation message to both clients, as a result they will be free to deal with it in any form: either remember with a timestamp the last time the slot was invalidate, and check later in a lazy way the timestamp (or incremental “epoch” if you like it more: it is safer) of the cached object, and evict it based on the comparison. Otherwise the client is free to reclaim the objects directly, by taking a table of what it cached about this specific slot. This approach with a 24 bit hash function is not an issue, because we’ll not have a very long list at all, even when caching tens of millions of keys. After sending the invalidation messages, we can remove the entries from the invalidation table, this way we'll no longer send invalidation messages to those clients until they don't read again keys for such slot. Note that clients are not forced to really use all the 24 bits of the hash function. They may just use, for instance, 20 bits, and then also shift the invalidation messages slots that Redis sends them. Not sure if there are many good reasons to do that, but in memory constrained systems may be an idea. If you followed closely what I said, you are thinking that the same connection receives both the normal client replies, and the invalidation messages. This is possible with RESP3, because invalidations are sent as “push” message types. Yet if the client is a blocking one, and not an event driven client, this starts to be complex: the application need some way to read new data from time to time, and that looks complex and fragile. It is perhaps, in that case, much better to use another application thread, and a different client connection, in order to receive the invalidation messages. So you are allowed to do the following: CLIENT TRACKING on REDIRECT 1234 Basically we can say that all the keys we get with the current connection, we want the invalidation message to be sent to client 1234 instead. Multiple clients may ask to have the invalidation messages redirected to a single client for instance, in case of connection pools. All you need to do is to create this special connection to receive the invalidation messages, call CLIENT ID to know which ID this client connection has, and later enable tracking. There is one problem left: what happens if we lose the connection with the server from the invalidation link? We may run into troubles since invalidation messages will no longer be received. Normally the application will detect the link is severed, and will reconnect again, flushing the current cache (or taking more soft resolutions, like putting all the timestamps for the slots a few seconds in the future to have some time to populate the cache while serving data that may be a few seconds stale). Yet it may be a better idea if the invalidation thread pings from time to time the connection to make sure it is alive. However in order to reduce the risk of stale data, Redis will also start to inform the clients that redirected the invalidation messages to some other client, that is now disconnected, about the situation, just using special push messages: at the next query performed the client will know. What I described was just merged into Redis unstable. Probably it’s not the final word, but we have more months before the first Redis 6 release candidate, there is time to change everything: just send me your feedbacks. I’m also looking at ways to enable the feature for RESP2. That would work only when redirection is enabled, and the client listening for messages should probably go into Pub/Sub mode so that we could send kinda of Pub/Sub messages. In this way old clients can be fully reused. I hope this was enough to stimulate your appetite: if we execute this inside Redis very well, and then document it for the client authors to know how to provide support, data may go a lot nearer to the application than ever, even in applications ran by small teams that so far avoided trying to implement client side caching. For large teams and very large applications doing this already, the overhead could be reduced, together with the complexity of the implementation. Comments

• The struggles of an open source maintainer
  May 16, 2019

  Months ago the maintainer of an OSS project in the sphere of system software, with quite a big and active community, wrote me an email saying that he struggles to continue maintaining his project after so many years, because of how much psychologically taxing such effort is. He was looking for advices from me, I’m not sure to be in the position of giving advices, however I told him I would write a blog post about what I think about the matter. Several weeks passed, and multiple times I started writing such post and stopped, because I didn’t had the time to process the ideas for enough time. Now I think I was able to analyze myself to find answers inside my own weakness, struggles, and desire of freedom, that inevitably invades the human minds when they do some task, that also has some negative aspect, for a prolonged amount of time. Maintaining an open source project is also a lot of joy and fun and these latest ten years of my professional life are surely memorable, even if not the absolute best (I had more fun during my startup times after all). However here I’ll focus on the negative side; simply make sure you don’t get the feeling it is just that, there is also a lot of good in it. Flood effect I don’t believe in acting fast, thinking fast, winning the competition on time and stuff like that. I don’t like the world of constant lack of focus we live in, because of social networks, chats, emails, and a schedule full of activities. So when I used to receive an email about Redis back in the early times of the project, when I still had plenty of time, I was able to focus on what the author of the message was trying to tell me. Then I could recall the relevant part of Redis we were discussing, and finally reply with my real thoughts, after considering the matter with care. I believe this is how most people should work regardless of what their job is. When a software project reaches the popularity Redis reached, and at the same time once the communications between individuals are made so simple by the new social tools, and by your attitude to be “there” for users, the amount of messages, issues, pull requests, suggestions the authors receive will grow exponentially. At the same time, at least in the case of Redis, but I believe this to be a common problem, the amount of very qualified people that can look at such inputs from the community grows very slowly. This creates an obvious congestion. Most people try to address it in the wrong way: using pragmatism. Let’s close the issue after two weeks of no original poster replies, after we ask some question. Close all the issues that are not very well specified. And other “inbox zero” solutions. The reality is that to process community feedbacks very well you have to take the time needed, otherwise you will just pretend your project has a small number of open issues. Having a lot of resources to hire core-level experts for each Redis subsystem, to work at OSS full time, would work but is not feasible. So what happens? That you start to prioritize more and more what to look at and what not. And you feel you are a piece of shit at ignoring so many things and people, and also the contributor believes you don’t care about what others have to give you. It’s a complex situation. Usually the end result is to develop an attitude to mostly address critical issues and disregard all the new stuff, since new stuff are yet not in the core, and who wants to have a larger code base with even more PRs and issues there? Maybe also written in a more convoluted way compared to your usual programming style, so, more complexity, and good luck when there is a critical bug there to track the root cause. Role shifting As a result of the “flood effect” problem exposed above, you suddenly also change job. Redis became popular because I supposedly am able to design and write software. And now instead most of the work I do is to look at issues and pull requests, and I also feel that I could do better many of the contributions I receive. Some will be better quality than I could do, because there are also better programmers than me contributing to Redis, but *most* for the nature of big numbers will be average contributions that are just written to solve a given problem that was contingent for the folks that submitted it. While, when I design for Redis, I tend to think at Redis as a whole, because it’s years I write this thing. So what you were good at, you have no longer time to do. This in turn means less organic big new features. My solution with that? Sometimes I just stop looking at issues and PRs for weeks, because I’m coding or designing: that is the work I really love and enjoy. However this in turn creates ways more pressure on me, psychologically. To do what I love and I can do well I’ve to feel like shit. Time There are two problems related to working at the same project for a prolonged amount of time, at least for me. First, before of the Redis experience I *never* worked every week day of my life. I could work one week, stop two, then work one month, then disappear for other two months. Always. People need to recharge, get new energy and ideas, to do creative work. And programming at high level is a fucking creative job. Redis itself was created like that for the first two years, that is, when the project evolved at the fastest speed. Because the sum of the productivity of me working just when I want is greater than the productivity I’ve when I’m forced to work every day in a steady way. However my work ethics allowed me to have a very discontinue schedule when I was working alone with my companies. Once I started to receive money to work at Redis, it was no longer possible for my ethics to have my past pattern, so I started to force myself to work under normal schedules. This for me is a huge struggle, for many years at this point. Moreover I’m sure I’m doing less than I could because of that, but this is how things work. I never found a way to solve this problem. I could say Redis Labs that I want to return to my old schedule, but that would not work because at this point I really “report” to the community, not to the company. Another problem is that working a lot at the same project is also a complex matter, mentally speaking. I used to change project every six months in the past. Now for ten years I did the same thing. In that regard I tried to save my sanity by having sub-projects inside Redis. One time I did Cluster, another time disk-storage (now abandoned), another was HyerLogLogs, and so forth. Basically things that bring value to the project but that, in isolation, are other stuff. But eventually you have to return back to the issues and PRs page and address the same things every day. “Replica is disconnecting because of a timeout” or whatever. Let’s investigate that again. Fear I always had some fear to lose the technological leadership of the project. Not because I think I’m not good enough at designing and evolving Redis, but because I know my ways are not aligned with: 1) what a sizable amount of users want. 2) what most people in IT believe software is. So I had to constantly balance between what I believe to be good design, set of features, speed of development (slow), size of the project (minimal), and what I was expected to deliver by most of the user base. Fortunately there is a percentage of Redis users that understand perfectly the Redis-way, so at least from time to time I can get some word of comfort. Frictions Certain people are total assholes. They are everywhere, it is natural and if you ask me, I even believe in programming there are a lot more nice people than in other fields. But yet you’ll always see a percentage of total jerks. As the leader of a popular OSS project, in one way or the other you’ll have to confront with these people, and that’s maybe one of the most stressful things I ever did in the course of the Redis development. Futileness Sometimes I believe that software, while great, will never be huge like writing a book that will survive for centuries. Note because it is not as great per-se, but because as a side effect it is also useful… and will be replaced when something more useful is around. I would like to have time to do other activities as well. So sometimes I believe all I’m doing is, in the end, futile. We’ll design and write systems, and new systems will emerge; but anyone that just stays in software, instead of staying in “software big ideas”, will ever set a new mark? From time to time I think I had potentially the ability to work at big ideas but because I focused on writing software instead of thinking about software, I was not able to use my potential in that regard. This is basically the contrary of the impostor syndrome, so I guess I’ve a big idea of myself: sorry for that I should be more humble. That said, I was able to work for many years doing things I really loved, that gave me friends, recognition, money, so I don’t want to say it was a bad deal. Yet I totally understand people struggling a lot to stay afloat once their projects start to be popular. This blog post is dedicated to them. Comments

• Redis streams as a pure data structure
  Mar 22, 2019

  The new Redis data structure introduced in Redis 5 under the name of “Streams” generated quite some interest in the community. Soon or later I want to run a community survey, talking with users having production use cases, and blogging about it. Today I want to address another issue: I’m starting to suspect that many users are only thinking at Streams as a way to solve Kafka(TM)-alike use cases. Actually the data structure was designed to *also* work in the context of messaging with producers and consumers, but to think that Redis Streams are just good for that is incredibly reductive. Streaming is a terrific pattern and “mental model” that can be applied when designing systems with great success, but Redis Streams, like most Redis data structures, are more general, and can be used to model dozen of different unrelated problems. So in this blog post I’ll focus on Streams as a pure data structure, completely ignoring its blocking operations, consumer groups, and all the messaging parts. ## Streams are CSV files on steroids If you want to log a series of structured data items and decided that databases are overrated after all, you may say something like: let’s just open a file in append only mode, and log every row as a CSV (Comma Separated Value) item: (open data.csv in append only) time=1553096724033,cpu_temp=23.4,load=2.3 time=1553096725029,cpu_temp=23.2,load=2.1 Looks simple and people did this for ages and still do: it’s a solid pattern if you know what you are doing. But what is the in-memory equivalent of that? Memory is more powerful than an append only file and can automagically remove the limitations of a CSV file like that: 1. It’s hard (inefficient) to do range queries here. 2. There is too much redundant information: the time is almost the same in every entry and the fields are duplicated. At the same time removing it will make the format less flexible, if I want to switch to a different set of fields. 3. Item offsets are just the byte offset in the file: if we change the file structure the offset will be wrong, so there is no actual true concept of primary ID here. Entries are basically not univocally addressed in some way. 4. I can’t remove entries, but only mark them as no longer valid without the ability of garbage collecting, if not by rewriting the log. Log rewriting usually sucks for several reasons and if it can be avoided, it’s good. Still such log of CSV entries is also great in some way: there is no fixed structure and fields may change, is trivial to generate, and after all is quite compact as well. The idea with Redis Streams was to retain the good things, but go over the limitations. The result is a hybrid data structure very similar to Redis Sorted Sets: they *feel like* a fundamental data structure, but to get such an effect, internally it uses multiple representations. ## Streams 101 (you may skip that if you know already Redis Stream basics) Redis Streams are represented as delta-compressed macro nodes that are linked together by a radix tree. The effect is to be able to seek to random entries in a very fast way, to obtain ranges if needed, remove old items to create a capped stream, and so forth. Yet our interface to the programmer is very similar to a CSV file: > XADD mystream * cpu-temp 23.4 load 2.3 "1553097561402-0" > XADD mystream * cpu-temp 23.2 load 2.1 "1553097568315-0" As you can see from the example above the XADD command auto generates and returns the entry ID, which is monotonically incrementing and has two parts: -. The time is in milliseconds and the counter increases for entries generated in the same milliseconds. So the first new abstraction on top of the “append only CSV file” idea is that, since we used the asterisk as the ID argument of XADD, we get the entry ID for free from the server. Such ID is not only useful to point to a specific item inside a stream, it’s also related to the time when the entry was added to the stream. In fact with XRANGE it is possible to perform range queries or fetch single items: > XRANGE mystream 1553097561402-0 1553097561402-0 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" In this case I used the same ID as the start and the stop of the range in order to identify a single element. However I can use any range, and a COUNT argument to limit the number of results. Similarly there is no need to specify full IDs as range, I can just use the millisecond unix time part of the IDs, to get elements in a given range of time: > XRANGE mystream 1553097560000 1553097570000 1) 1) "1553097561402-0" 2) 1) "cpu-temp" 2) "23.4" 3) "load" 4) "2.3" 2) 1) "1553097568315-0" 2) 1) "cpu-temp" 2) "23.2" 3) "load" 4) "2.1" For now there is no need to show you more Streams API, there is the Redis documentation for that. For now let’s just focus on that usage pattern: XADD to add stuff, XRANGE (but also XREAD) in order to fetch back ranges (depending on what you want to do), and let’s see why I claim Streams are so powerful as a data structure. However if you want to learn more about Redis Streams and their API, make sure to visit the tutorial here: https://redis.io/topics/streams-intro ## Tennis players A few days ago I was modeling an application with a friend of mine which is learning Redis those days: an app in order to keep track of local tennis courts, local players and matches. The way you model players in Redis is quite obvious, a player is a small object, so an Hash is all you need, with key names like player:. As you model the application data further, to use Redis as its primary, you immediately realize you need a way to track the games played in a given tennis club. If player:1 and player:2 played a game, and player 1 won, we could write the following entry in a stream: > XADD club:1234.matches * player-a 1 player-b 2 winner 1 "1553254144387-0" With this simple operation we have: 1. A unique identifier of the match: the ID in the stream. 2. No need to create an object in order to identify a match. 3. Range queries for free to paginate the matches, or check the matches played in a given moment in the past. Before Streams we needed to create a sorted set scored by time: the sorted set element would be the ID of the match, living in a different key as a Hash value. This is not just more work, it’s also an incredible amount of memory wasted. More, much more you could guess (see later). For now the point to show is that Redis Streams are kinda of a Sorted Set in append only mode, keyed by time, where each element is a small Hash. And in its simplicity this is a revolution in the context of modeling for Redis. ## Memory usage The above use case is not just a matter of a more solid pattern. The memory cost of the Stream solution is so different compared to the old approach of having a Sorted Set + Hash for every object that makes certain things that were not viable, now perfectly fine. Those are the numbers for storing one million of matches in the configurations exposed previously: Sorted Set + Hash memory usage = 220 MB (242 RSS) Stream memory usage = 16.8 MB (18.11 RSS) This is more than an order of magnitude difference (13 times difference exactly), and it means that use cases that yesterday were too costly for in-memory now are perfectly viable. The magic is all in the representation of Redis Streams: the macro nodes can contain several elements that are encoded in a data structure called listpack in a very compact way. Listpacks will take care, for instance, to encode integers in binary form even if they are semantically strings. On top of that, we then apply delta compression and same-fields compression. Yet we are able to seek by ID or time because such macro nodes are linked in the radix tree, which was also designed to use little memory. All these things together account for the low memory usage, but the interesting part is that semantically the user does not see any of the implementation details making Streams efficient. Now let’s do some simple math. If I can store 1 million entries in about 18 MB of memory, I can store 10 millions in 180 MB, and 100 millions in 1.8 GB. With just 18 GB of memory I can have 1 billion items. ## Time series One important thing to note is, in my opinion, how the usage above where we used a Stream to represent a tennis match was semantically *very different* than using a Redis Stream for a time series. Yes, logically we are still logging some kind of event, but one fundamental difference is that in one case we use the logging and the creation of entries in order to render objects. While in the case of time series, we are just metering something happening externally, that does not really represent an object. You may think that this difference is trivial but it’s not. It is important for Redis users to build the idea that Redis Streams can be used in order to create small objects that have a total order, and assign IDs to such objects. However even the most basic use case of time series is, obviously, a huge one here, because before Streams Redis was a bit hopeless in regard to such use case. The memory characteristics and flexibility of streams, plus the ability to have capped streams (see the XADD options), is a very important tool in the hands of the developer. ## Conclusions Streams are flexible and have lots of use cases, however I wanted to take this blog post short to make sure that there is a clear take-home message in the above examples and analysis of the memory usage. Perhaps this was already obvious to many readers, but talking with people in the last months gave me the feeling that there was a strong association between Streams and the streaming use case, like if the data structure was only good at that. That’s not the case :-) Comments

• Gopher: a present for Redis
  Feb 25, 2019

  Ten years ago Redis was announced on Hacker News, and I use this as virtual birthdate for the project, simply because it is more important when it was announced to the public than the actual date of the project first line of code (think at it conception VS actual birth in animals). I’ll use the ten years of Redis as an excuse to release something I played a bit in the previous days, thinking to use it for the 1st April fool: but such date is far and I want to talk to you about this project now… So, happy birthday Redis! Here it’s your present: a Gopher protocol implementation. [… here Redis tries to stop the tears, but the emotion is too strong and there are bits (I mean zeros and ones) on the floor …] WTF are you saying?! should be your automatic question. Gopher in 2019 sounds a bit strange. However it is not *just* a joke, while it is largely a joke. The implementation is just 100 lines of code after all, excluding the external tool to render the pages into Redis keys. But… the thing is that there is really an active community around Gopher, a very small one but one that is growing in the latest years and months. There are people that feel that internet is no longer what it used to be. There is too much control, companies tracking, comments, likes, retweets, to the point that the content is no longer the king. One writes new things for them to be popular for 5 hours and disappear. There is no longer a discussion that can survive more than a few minutes without becoming some kind of flame, unless all the parties self-censor every possible feeling, uneasy word, and belief, to the point to make the discussion quite useless. Finally to load a stupid page with 1k of text requires to load 50 javascript files, to see the screen flickering since client-side rendering is cool, and so forth. On the other hand Gopher is a text only protocol that is great to deliver text only documents where the stress is in what you write. But that would be fetichism, for me the silver bullet of Gopher is that it is UNCOOL. Uncool enough that it will be forever, AFAIK, an alternative reality where certain folks can decide to separate from the rest, to experience a different way to do things, more similar to the old times of BBSs or the first years of the internet. A place where most people will not want to go just to read nerdy stuff in an 80 columns fixed size font. What you do in Gopher is to create your Gopher hole, that is, your space inside the Gopher universe, like your web site on the internet basically. There was no shortage of tools to do that already, but Redis is quite nice for a few reasons: you can change the Redis keys to change the site content in real time, that’s handy. You can use replication in order to duplicate a site, and can even just save your RDB file to have an exact copy of the whole Gopher hole to archive for backup or historical reasons. This Redis Gopher concept was created with the collaboration of Freaknet, a historical hacking laboratory experience here in Catania. https://it.wikipedia.org/wiki/FreakNet. Those folks do a lot of interesting stuff, including a retrocomputing hardware museum project in Palazzolo Acreide here: https://museo.freaknet.org/en/. How it works? Well it’s trivial, I hijacked the inline protocol, and specifically two kind of inline requests that were anyway illegal: an empty request or any request that starts with "/" (there are no Redis commands starting with such a slash). Normal RESP2/RESP3 requests are completely out of the path of the Gopher protocol implementation and are served as usually as well. If you open a connection to Redis when Gopher is enabled and send it a string like "/foo", if there is a key named "/foo" it is served via the Gopher protocol. The whole implementation is 100 lines of code. Initially I thought about using data structures and have semantical transformations to Gopher types, but that’s just complex and useless. Instead what I did was to provide an authoring tool for Gopher over Redis, you can find it here: https://github.com/antirez/gopher2redis To see that example Gopher hole running on a Redis instance just go to gopher://gopher.antirez.com, and btw that will be the address of my Gopher hole once I’ll build one in the next days. P.S. I suggest using the Lynx text only web/gopher browser to access Gopher. The gopher support is disabled by default, to enable it use the Redis unstable branch and use the “gopher-enabled” option, setting it to yes. However MAKE SURE to also password protect Redis: the Gopher protocol will still serve content, but at the same time normal Redis commands will not be accessible. This way (and assuming you don’t have data other than your Gopher keys to expose in the instance) you could make the instance public, as a true Gopher server. Well, have fun with Gopher! I hope this Gopher thing will go forward, I really believe there are a few of us that need to create a community outside the chaos of the modern Internet. No, it will not be possible to have no interactions. For instance I’ve no plans to stop blogging or using Internet. But certain slower higher quality communications need a place to prosper. Comments

• An update about Redis developments in 2019
  Feb 20, 2019

  Yesterday a concerned Redis user wrote the following on Hacker News: — https://news.ycombinator.com/item?id=19204436 — I love Redis, but I'm a bit skeptical of some of the changes that are currently in development. The respv3 protocol has some features that, while they sound neat, also could significantly complicate client library code. There's also a lot of work going into a granular acl. I can't imagine why this would be necessary, or a higher priority than other changes like multi-thread support, better persistence model, data-types, etc. — end of user comment — I’ve the feeling she/he (not sure) is not the only one that looks at ACLs as some sort of feature imposed by the Redis Labs goals, because “enterprise users” or something like that. Also the other points in the comment are interesting, and I believe everything is very well worth addressing in order to communicate clearly with the Redis community what’s the road ahead. For simplicity I’ll split this blog post into sections addressing every single feature mentioned in the original comment. ## RESP3 The goal of RESP3, as I already blogged in these pages, is to actually simplify the clients landscape. Hopefully every client will have a lower layer that will not try to reinvent some kind of higher level interface: redis.call(“get”,”foo”). There is no longer need to orchestrate conversions because now the protocol is semantical enough to tell the client what a given reply should look like in the hand of the caller, nor any need to know beforehand the command fingerprint for the majority of commands. What I think the user is referring is RESP3 support for out of band communications, that is the reply “attributes”. I really believe that in the future of Redis “client side caching” will be a big thing. It’s the logical step in every scalable system. However without server assistance client side cache invalidation is a nightmare. This is the reason why RESP3 supports attributes in replies, mainly. However probably Redis 6 *will not implement any of that*. Redis unstable, that will become Redis 6, already has a RESP3 implementation that is almost complete, and there are no attributes. The clients implementing RESP3 can just decide to discard attributes if they are willing to be really future-proof, and likely attributes will not be sent at all anyway even for future Redis versions if the user did not activate some kind of special feature. For instance, for client side caching, the connection will have to be put in some special mode. Moreover, as you know, Redis 6 will be completely backward compatible with RESP2. Actually I’m starting to believe that RESP2 support will never be removed, because it is almost for free, and there is no good reason to break backward compatibility once we did the effort to implement the abstraction layer between RESP2 and RESP3. Normally I don’t like to change things without a good reason, however RESP2 limitations were having a strong effect on the client ecosystem. I would like to have a client landscape where users, going from one client to the other, will feel at home, and the API will be the Redis API, not the layer that the client author invented. I’m not against an *higher level* API in addition to the lower level one btw, but there should be a common ground, and clients should be able to send commands without knowing anything about such commands. ## ACLs The ACL specification was redacted by myself four years ago. I waited so much time in order to convince myself this was really the time to implement it: we went a long way without any ACL using just tricks, mainly command renaming. However don’t believe that ACLs main motivation is enterprise customers in need for security. As a side effect, ACLs also allow authentication of users for security purposes, but the main goal of the feature is *operational*. Let me show you an example. You have a Redis instance and you plan to use the instance to do a new thing: delayed jobs processing. You get a library from the internet, and it looks to work well. Now why on the earth such library, that you don’t know line by line, should be able to call “FLUSHALL” and flush away your database instantly? Maybe the library test will have such command inside and you realize it when it’s too late. Or maybe you just hired a junior developer that is keeping calling “KEYS *” on the Redis instance, while your company Redis policy is “No KEYS command”. Another scenario, cloud providers: they need to carefully rename the admin commands, and even to mask such commands from being leaked for some reason. More tricks: so MONITOR will not show the commands in the output for instance. With ACLs you can setup Redis so that default users, without some authentication, will be prevented to run anything that is administrative or dangerous. I think this will be a big improvement for operations. Moreover ACLs is one of the best code I wrote for Redis AFAIK. There is nearly no CPU cost at all, unless you se key patterns, but even so it’s small. The implementation is completely self contained inside the acl.c file, the rest of the core has a handful of calls to the ACL API. No complexity added to the system because it is completely modular. Actually the ACL code allowed to do some good refactoring around the AUTH command. ## Multi threading There are two possible multi threading supports that Redis could get. I believe the user is referring to “memcached alike” multithreading, that is the ability to scale a single Redis instance to multiple threads in order to increase the operations per second it can deliver in things like GET or SET and other simple commands. This involves making the I/O, command parsing and so forth multi threaded. So let’s call this thing “I/O threading”. Another multi threaded approach is to, instead, allow slow commands to be executed in a different thread, so that other clients are not blocked. We’ll call this threading model “Slow commands threading”. Well, that’s the plan: I/O threading is not going to happen in Redis AFAIK, because after much consideration I think it’s a lot of complexity without a good reason. Many Redis setups are network or memory bound actually. Additionally I really believe in a share-nothing setup, so the way I want to scale Redis is by improving the support for multiple Redis instances to be executed in the same host, especially via Redis Cluster. The things that will happen in 2019 about that are two: A) Redis Cluster multiple instances will be able to orchestrate to make a judicious use of the disk of the local instance, that is, let’s avoid an AOF rewrite at the same time. B) We are going to ship a Redis Cluster proxy as part of the Redis project, so that users are able to abstract away a cluster without having a good implementation of the Cluster protocol client side. Another thing to note is that Redis is not Memcached, but, like memcached, is an in-memory system. To make multithreaded an in-memory system like memcached, with a very simple data model, makes a lot of sense. A multi-threaded on-disk store is mandatory. A multi-threaded complex in-memory system is in the middle where things become ugly: Redis clients are not isolated, and data structures are complex. A thread doing LPUSH need to serve other threads doing LPOP. There is less to gain, and a lot of complexity to add. What instead I *really want* a lot is slow operations threading, and with the Redis modules system we already are in the right direction. However in the future (not sure if in Redis 6 or 7) we’ll get key-level locking in the module system so that threads can completely acquire control of a key to process slow operations. Now modules can implement commands and can create a reply for the client in a completely separated way, but still to access the shared data set a global lock is needed: this will go away. ## Better persistence Recently we did multiple efforts in order to improve this kind of fundamental functions of Redis. One of the best thing that was implemented lately is the RDB preamble inside the AOF file. Also a lot of work went both in Redis 4 and 5 about replication, that is now completely at another level compared to what it used to be. And yes, it is still one of my main focus to improve such parts. ## Data structures Now Redis has Streams, starting with Redis 5. For Redis 6 and 7 what is planned is, to start, to make what we have much more memory efficient by changing the implementations of certain things. However to add new data structures there are a lot of considerations to do. It took me years to realize how to fill the gap, with streams, between lists, pub/sub and sorted sets, in the context of time series and streaming. I really want Redis to be a set of orthogonal data structures that the user can put together, and not a set of *tools* that are ready to use. Streams are an abstract log, so I think it’s a very worthwhile addition. However other things I’m not completely sure if they are worth to be inside the core without a very long consideration. Anyway in the latest years there was definitely more stress in adding new data structures. HyperLogLogs, more advanced bit operations, streams, blocking sorted set operations (ZPOP* and BZPOP*), and streams are good examples. ## Conclusions I believe that the Redis community should be aware about why something is done and why something is instead postponed. I do the error to communicate a lot via Twitter like if everybody is there, but many people happen to have a life :-D and don’t care. The blog is a much better way to inform the community, I need to take the time to blog more. Incidentally I love to write posts, so it’s a win-win. An important thing to realize is that Redis has not a solid roadmap, over the years I found that opportunistic development is a huge win over having a roadmap. Something is demanded? I see the need? I’m in the mood to code it? It’s the right moment because there are no other huge priorities? There are a set of users that are helping the design process, giving hints, ideas, testing stuff? It’s the right moment, let’s do it. To have a solid roadmap for Redis is silly because the size of the OSS core team is small, sometimes I remain stuck with some random crash for weeks… Any fixed long term plan would not work. Moreover as the Redis community gives feedbacks my ideas change a lot, so I would rewrite the roadmap every month. Yet blogging is a good solution to at least show what is the current version of the priorities / ideas, and to show why other ideas were abandoned. A final note: the level of freedom I've with Redis Labs about what to put inside the open source project side is almost infinite. I think this is kinda of a miracle in the industry, or just the people I work with at Redis Labs are nice folks that understand that what we are doing originated from the open source movement and is wise to keep it going in that way. But it's not a common thing. If I do errors in the Redis roadmap they are surely my errors. Comments

• Why RESP3 will be the only protocol supported by Redis 6
  Nov 09, 2018

  [EDIT! I'm reconsidering all this because Marc Gravell from Stack Overflow suggested that we could just switch protocol for backward compatibility per-connection, sending a command to enable RESP3. That means no longer need for a global configuration that switches the behavior of the server. Put in that way it is a lot more acceptable for me, and I'm reconsidering the essence of the blog post] A few weeks after the release of Redis 5, I’m here starting to implement RESP3, and after a few days of work it feels very well to see this finally happening. RESP3 is the new client-server protocol that Redis will use starting from Redis 6. The specification at https://github.com/antirez/resp3 should explain in clear terms how this evolution of our old protocol, RESP2, should improve the Redis ecosystem. But let’s say that the most important thing is that RESP3 is more “semantic” than RESP2. For instance it has the concept of maps, sets (unordered lists of elements), attributes of the returned data, that may augment the reply with auxiliary information, and so forth. The final goal is to make new Redis clients have less work to do for us, that is, just deciding a set of fixed rules in order to convert every reply type from RESP3 to a given appropriate type of the client library programming language. In the future of Redis I see clients that are smarter under the hood, trying to do their best in order to handle connections, pipelining, and state, and apparently a lot more simpler in the user-facing side, to the point that the ideal Redis client is like: result = redis.call(“GET”,keyname); Of course on top of that you can build more advanced abstractions, but the bottom layer should look like that, and the returned reply should not require any filtering that is ad-hoc for specific commands: RESP3 return type should contain enough information to return an appropriate data type. So HGETALL will return a RESP3 “map”, while LRANGE will return an “array”, and EXISTS will return a RESP3 “boolean”. This also allows new commands to work as expected even if the client library was not *specifically* designed to handle it. With RESP2 instead what happened was that likely the command worked using mechanisms like "method missing" or similar, but later when the command was *really* implemented in the client library, the returned type changed, introducing a subtle incompatibility. However, while the new protocol is an incremental improvement over the old one, it will introduce breaking incompatibilities in the client-library side (of course) and *in the application layer* as well. Because for instance, ZSCORE will now return a double, and not a string, so application code should be updated, or, alternatively, client libraries could implement a compatibility option that will turn the RESP3 replies back to their original RESP2 types. Lua scripts will also no longer work if not modified for the new protocol, because also Lua will see more semantical types returned by the redis.call() command. Similarly Lua will be able to return all the new data types implemented in RESP3. Because of all that, people are scared about my decision: I’m going to ship Redis 6 with support for *only* RESP3. There will be no compatibility mode to switch a Redis 6 server to RESP2, so you either upgrade your client library and upgrade your application (or use the client library backward compatibility mode), or you cannot switch to Redis 6. I’ve good reasons to do so, and I want to explain why I’m taking this decision, and how I’m mitigating the problems for users and client library authors. Let’s start from the mitigations: * Redis 5 will be fully supported for 2 years after the release of Redis 6. Everything critical will be back-ported to Redis 5 and patch-level releases will be available constantly. * Redis 6 is expected to be released in about 1 or 1.5 years. However Redis 6 will be switched to RESP3 in about 1 month. So people will use, experiment, and deal with an unstable Redis version that uses the new protocol for a lot of time. Given that unlike many other softwares, Redis unstable has a lot of casual users, both because it’s the default branch on Github, and because traditionally Redis unstable is never really so unstable, this will grant a lot of prior exposure. * I’m still not 100% sure about that, but the Lua scripting engine may have a compatibility mode in order to return the same types as of Redis 5. The compatibility however will not be enabled by default, and will be opt-in for each script executed, by calling a special redis.resp2_compat() function before calling Redis commands. So every Redis 6 server will behave the same regardless of its configuration, as Redis always did in the last 10 years. Those are the mitigations. And this is, instead, why I’ll not have Redis 6 supporting both versions: 1) It is more or less completely useless. If people switch Redis 6 to RESP2 mode, they are still in the past and are just waiting for Redis 7 to go out without RESP2 support and break everything. In the meantime, when you deal with a Redis 6 installation, you never know *what it replies*, depending on how it is configured. So the same client library may return an Hash or an Array for the same command. 2) It’s more work and more complexity without a good reason (see “1”). Many commands will require a check for the old protocol in order to see in what format to reply. 3) By binding the new Redis 6 features together with a protocol change, we are giving good reasons to users to do the switch and port their clients and applications. At some point everything will be over and we can focus on new things. Otherwise we’ll have a set of Redis 6 users that switched to the new server for the new features but are still with the old protocol, and Redis 7 will be the same drama again. 4) If somebody tells you that adapting the client libraries is a terrible work, well, I’ll beg to differ. Yes, there is some change to do, but now that I’m implementing the server side, I see that it’s not so terrible. What is terrible instead is that most client work is not payed at all and happens just because of passion and willingness to share with others. I bet that we’ll see many implementations of RESP3 in short time. 5) RESP3 is designed so that clients can automatically detect if it’s RESP2 or RESP3, and switch, so new clients will work both with Redis Well that’s all. I hope it clarifies my point of view and the reasons behind it, and also at the same time the mitigations that will be enabled during the protocol switch may serve to convince users that it will not be a very “hard” breakage. Comments

• Writing system software: code comments.
  Oct 06, 2018

  For quite some time I’ve wanted to record a new video talking about code comments for my "writing system software" series on YouTube. However, after giving it some thought, I realized that the topic was better suited for a blog post, so here we are. In this post I analyze Redis comments, trying to categorize them. Along the way I try to show why, in my opinion, writing comments is of paramount importance in order to produce good code, that is maintainable in the long run and understandable by others and by the authors during modifications and debugging activities. Not everybody thinks likewise. Many believe that comments are useless if the code is solid enough. The idea is that when everything is well designed, the code itself documents what the code is doing, hence code comments are superfluous. I disagree with that vision for two main reasons: 1. Many comments don't explain what the code is doing. They explain what you can't understand just from what the code does. Often this missing information is *why* the code is doing a certain action, or why it’s doing something that is clear instead of something else that would feel more natural. 2. While it is not generally useful to document, line by line, what the code is doing, because it is understandable just by reading it, a key goal in writing readable code is to lower the amount of effort and the number of details the reader should take into her or his head while reading some code. So comments can be, for me, a tool for lowering the cognitive load of the reader. The following code snippet is a good example of the second point above. Note that all the code snippets in this blog post are obtained from the Redis source code. Every code snipped is presented prefixed by the file name it was extracted from. The branch used is the current "unstable" with hash 32e0d237. scripting.c: /* Initial Stack: array */ lua_getglobal(lua,"table"); lua_pushstring(lua,"sort"); lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ if (lua_pcall(lua,1,0,0)) { /* Stack: array, table, error */ /* We are not interested in the error, we assume that the problem is * that there are 'false' elements inside the array, so we try * again with a slower function but able to handle this case, that * is: table.sort(table, __redis__compare_helper) */ lua_pop(lua,1); /* Stack: array, table */ lua_pushstring(lua,"sort"); /* Stack: array, table, sort */ lua_gettable(lua,-2); /* Stack: array, table, table.sort */ lua_pushvalue(lua,-3); /* Stack: array, table, table.sort, array */ lua_getglobal(lua,"__redis__compare_helper"); /* Stack: array, table, table.sort, array, __redis__compare_helper */ lua_call(lua,2,0); } Lua uses a stack based API. A reader following each call in the function above, having also a Lua API reference at hand, will be able to mentally reconstruct the stack layout at every given moment. But why to force the reader to do such effort? While writing the code, the original author had to do that mental effort anyway. What I did there was just to annotate every line with the current stack layout after every call. Reading this code is now trivial, regardless of the fact the Lua API is otherwise non trivial to follow. My goal here is not just to offer my point of view on the usefulness of comments as a tool to provide a background that is not clearly available reading a local section of the source code. But also to also provide some evidence about the usefulness of the kind of comments that are historically considered useless or even dangerous, that is, comments stating *what* the code is doing, and not why. # Classification of comments The way I started this work was by reading random parts of the Redis source code, to check if and why comments were useful in different contexts. What quickly emerged was that comments are useful for very different reasons, since they tend to be very different in function, writing style, length and update frequency. I eventually turned the work into a classification task. During my research I identified nine types of comments: * Function comments * Design comments * Why comments * Teacher comments * Checklist comments * Guide comments * Trivial comments * Debt comments * Backup comments The first six are, in my opinion, mostly very positive forms of commenting, while the final three are somewhat questionable. In the next sections each type will be analyzed with examples from the Redis source code. FUNCTION COMMENTS The goal of a function comment is to prevent the reader from reading code in the first place. Instead, after reading the comment, it should be possible to consider some code as a black box that should obey certain rules. Normally function comments are at the top of functions definitions, but they may be at other places, documenting classes, macros, or other functionally isolated blocks of code that define some interface. rax.c: /* Seek the grestest key in the subtree at the current node. Return 0 on * out of memory, otherwise 1. This is an helper function for different * iteration functions below. */ int raxSeekGreatest(raxIterator *it) { ... Function comments are actually a form of in-line API documentation. If the function comment is written well enough, the user should be able most of the times to jump back to what she was reading (reading the code calling such API) without having to read the implementation of a function, a class, a macro, or whatever. Among all the kinds of comments, these are the ones most widely accepted by the programming community at large as needed. The only point to analyze is if it is a good idea to place comments that are largely API reference documentation inside the code itself. For me the answer is simple: I want the API documentation to exactly match the code. As the code is changed, the documentation should be changed. For this reason, by using function comments as a prologue of functions or other elements, we make the API documentation close to the code, accomplishing three results: * As the code is changed, the documentation can be easily changed at the same time, without the risk of making the API reference stale. * This approach maximizes the probability that the author of the change, that should be the one better understanding the change, will also be the author of the API documentation change. * Reading the code is handy to find the documentation of functions or methods directly where they are defined, so that the reader of the code can focus solely on the code, instead of context switching between code and documentation. DESIGN COMMENTS While a "function comment" is usually located at the start of a function, a design comment is more often located at the start of a file. The design comment basically states how and why a given piece of code uses certain algorithms, techniques, tricks, and implementation. It is an higher level overview of what you'll see implemented in the code. With such background, reading the code will be simpler. Moreover I tend to trust more code where I can find design notes. At least I know that some kind of explicit design phase happened, at some point, during the development process. In my experience design comments are also very useful in order to state, in case the solution proposed by the implementation looks a bit too trivial, what were the competing solutions and why a very simple solution was considered to be enough for the case at hand. If the design is correct, the reader will convince herself that the solution is appropriate and that such simplicity comes from a process, not from being lazy or only knowing how to code basic things. bio.c: * DESIGN * ------ * * The design is trivial, we have a structure representing a job to perform * and a different thread and job queue for every job type. * Every thread waits for new jobs in its queue, and process every job * sequentially. ... WHY COMMENTS Why comments explain the reason why the code is doing something, even if what the code is doing is crystal clear. See the following example from the Redis replication code. replication.c: if (idle > server.repl_backlog_time_limit) { /* When we free the backlog, we always use a new * replication ID and clear the ID2. This is needed * because when there is no backlog, the master_repl_offset * is not updated, but we would still retain our replication * ID, leading to the following problem: * * 1. We are a master instance. * 2. Our replica is promoted to master. It's repl-id-2 will * be the same as our repl-id. * 3. We, yet as master, receive some updates, that will not * increment the master_repl_offset. * 4. Later we are turned into a replica, connect to the new * master that will accept our PSYNC request by second * replication ID, but there will be data inconsistency * because we received writes. */ changeReplicationId(); clearReplicationId2(); freeReplicationBacklog(); serverLog(LL_NOTICE, "Replication backlog freed after %d seconds " "without connected replicas.", (int) server.repl_backlog_time_limit); } If I check just the function calls there is very little to wonder: if a timeout is reached, change the main replication ID, clear the secondary ID, and finally free the replication backlog. However what is not exactly clear is why we need to change the replication IDs when freeing the backlog. Now this is the kind of thing that happens continuously in software once it has reached a given level of complexity. Regardless of the code involved, the replication protocol has some level of complexity itself, so we need to do certain things in order to make sure that other bad things can't happen. Probably these kind of comments are, in some way, opportunities to reason about the system and check if it should be improved, so that such complexity is no longer needed, hence also the comment can be removed. However often making something simpler may make something else harder or is simply not viable, or requires future work breaking backward compatibility. Here is another one. replication.c: /* SYNC can't be issued when the server has pending data to send to * the client about already issued commands. We need a fresh reply * buffer registering the differences between the BGSAVE and the current * dataset, so that we can copy to other replicas if needed. */ if (clientHasPendingReplies(c)) { addReplyError(c,"SYNC and PSYNC are invalid with pending output"); return; } If you run SYNC while there is still pending output (from a past command) to send to the client, the command should fail because during the replication handshake the output buffer of the client is used to accumulate changes, and may be later duplicated to serve other replicas connecting while we are already creating the RDB file for the full sync with the first replica. This is the why we do that. What we do is trivial. Pending replies? Emit an error. Why is rather obscure without the comment. One may think that such comments are needed only when describing complex protocols and interactions, like in the case of replication. Is that the case? Let's change completely file and goals, and we see still such comments everywhere. expire.c: for (j = 0; j int expired; redisDb *db = server.db+(current_db % server.dbnum); /* Increment the DB now so we are sure if we run out of time * in the current DB we'll restart from the next. This allows to * distribute the time evenly across DBs. */ current_db++; ... That's an interesting one. We want to expire keys from different DBs, as long as we have some time. However instead of incrementing the “database ID” to process next at the end of the loop processing the current database, we do it differently: we select the current DB in the `db` variable, but then we immediately increment the ID of the next database to process (at the next call of this function). This way if the function terminates because too much effort was spent in a single call, we don't have the problem of restarting again from the same database, letting logically expired keys accumulating in the other databases since we are too focused in processing the same database again and again. With such comment we both explain why we increment at that stage, and that the next person going to modify the code, should preserve such quality. Note that without the comment the code looks completely harmless. Select, increment, go to do some work. There is no evident reason for not relocating the increment at the end of the loop where it could look more natural. Trivia: the loop increment was indeed at the end in the original code. It was moved there during a fix: at the same time the comment was added. So let's say this is kinda of a "regression comment". TEACHER COMMENTS Teacher comments don't try to explain the code itself or certain side effects we should be aware of. They teach instead the *domain* (for example math, computer graphics, networking, statistics, complex data structures) in which the code is operating, that may be one outside of the reader skills set, or is simply too full of details to recall all them from memory. The LOLWUT command in version 5 needs to display rotated squares on the screen (http://antirez.com/news/123). In order to do so it uses some basic trigonometry: despite the fact that the math used is simple, many programmers reading the Redis source code may not have any math background, so the comment at the top of the function explains what's going to happen inside the function itself. lolwut5.c: /* Draw a square centered at the specified x,y coordinates, with the specified * rotation angle and size. In order to write a rotated square, we use the * trivial fact that the parametric equation: * * x = sin(k) * y = cos(k) * * Describes a circle for values going from 0 to 2*PI. So basically if we start * at 45 degrees, that is k = PI/4, with the first point, and then we find * the other three points incrementing K by PI/2 (90 degrees), we'll have the * points of the square. In order to rotate the square, we just start with * k = PI/4 + rotation_angle, and we are done. * * Of course the vanilla equations above will describe the square inside a * circle of radius 1, so in order to draw larger squares we'll have to * multiply the obtained coordinates, and then translate them. However this * is much simpler than implementing the abstract concept of 2D shape and then * performing the rotation/translation transformation, so for LOLWUT it's * a good approach. */ The comment does not contain anything that is related to the code of the function itself, or its side effects, or the technical details related to the function. The description is only limited to the mathematical concept that is used inside the function in order to reach a given goal. I think teacher comments are of huge value. They teach something in case the reader is not aware of such concepts, or at least provide a starting point for further investigation. But this in turn means that a teacher comment increases the amount of programmers that can read some code path: writing code that can be read by many programmers is a major goal of mine. There are developers that may not have math skills but are very solid programmers that can contribute some wonderful fix or optimization. And in general code should be read other than being executed, since is written by humans for other humans. There are cases where teacher comments are almost impossible to avoid in order to write decent code. A good example is the Redis radix tree implementation. Radix trees are articulated data structures. The Redis implementation re-states the whole data structure theory as it implements it, showing the different cases and what the algorithm does to merge or split nodes and so forth. Immediately after each section of comment, we have the code implementing what was written before. After months of not touching the file implementing the radix tree, I was able to open it, fix a bug in a few minutes, and continue doing something else. There is no need to study again how a radix tree works, since the explanation is the same thing as the code itself, all mixed together. The comments are too long, so I'll just show certain snippets. rax.c: /* If the node we stopped at is a compressed node, we need to * split it before to continue. * * Splitting a compressed node have a few possible cases. * Imagine that the node 'h' we are currently at is a compressed * node contaning the string "ANNIBALE" (it means that it represents * nodes A -> N -> N -> I -> B -> A -> L -> E with the only child * pointer of this node pointing at the 'E' node, because remember that * we have characters at the edges of the graph, not inside the nodes * themselves. * * In order to show a real case imagine our node to also point to * another compressed node, that finally points at the node without * children, representing 'O': * * "ANNIBALE" -> "SCO" -> [] ... snip ... * 3a. IF $SPLITPOS == 0: * Replace the old node with the split node, by copying the auxiliary * data if any. Fix parent's reference. Free old node eventually * (we still need its data for the next steps of the algorithm). * * 3b. IF $SPLITPOS != 0: * Trim the compressed node (reallocating it as well) in order to * contain $splitpos characters. Change chilid pointer in order to link * to the split node. If new compressed node len is just 1, set * iscompr to 0 (layout is the same). Fix parent's reference. ... snip ... if (j == 0) { /* 3a: Replace the old node with the split node. */ if (h->iskey) { void *ndata = raxGetData(h); raxSetData(splitnode,ndata); } memcpy(parentlink,&splitnode,sizeof(splitnode)); } else { /* 3b: Trim the compressed node. */ trimmed->size = j; memcpy(trimmed->data,h->data,j); trimmed->iscompr = j > 1 ? 1 : 0; trimmed->iskey = h->iskey; trimmed->isnull = h->isnull; if (h->iskey && !h->isnull) { void *ndata = raxGetData(h); raxSetData(trimmed,ndata); } raxNode **cp = raxNodeLastChildPtr(trimmed); ... As you can see the description in the comment is then matched with the same labels in the code. It's hard to show it all in this form so if you want to get the whole idea just check the full file at: https://github.com/antirez/redis/blob/unstable/src/rax.c This level of commenting is not needed for everything, but things like radix trees are really full of little details and corner cases. They are hard to recall, and certain details are *specific* to a given implementation. Doing this for a linked list does not make much sense of course. It's a matter of personal sensibility to understand when it's worth it or not. CHECKLIST COMMENTS This is a very common and odd one: sometimes because of language limitations, design issues, or simply because of the natural complexity arising in systems, it is not possible to centralize a given concept or interface in one piece, so there are places in the code that tells you to remember to do things in some other place of the code. The general concept is: /* Warning: if you add a type ID here, make sure to modify the * function getTypeNameByID() as well. */ In a perfect world this should never be needed, but in practice sometimes there are no escapes from that. For example Redis types could be represented using an "object type" structure, and every object could link to the type the object it belongs, so you could do: printf("Type is %s\n", myobject->type->name); But guess what? It's too expensive for us, because a Redis object is represented like this: typedef struct redisObject { unsigned type:4; unsigned encoding:4; unsigned lru:LRU_BITS; /* LRU time (relative to global lru_clock) or * LFU data (least significant 8 bits frequency * and most significant 16 bits access time). */ int refcount; void *ptr; } robj; We use 4 bits instead of 64 to represent the type. This is just to show why sometimes things are not as centralized and natural as they should be. When the situation is like that, sometimes what helps is to use defensive commenting in order to make sure that if a given code section is touched, it reminds you to make sure to also modify other parts of the code. Specifically a checklist comment does one or both of the following things: * It tells you a set of actions to do when something is modified. * It warns you about the way certain changes should be operated. Another example in blocked.c, when a new blocking type is introduced. blocked.c: * When implementing a new type of blocking opeation, the implementation * should modify unblockClient() and replyToBlockedClientTimedOut() in order * to handle the btype-specific behavior of this two functions. * If the blocking operation waits for certain keys to change state, the * clusterRedirectBlockedClientIfNeeded() function should also be updated. The checklist comment is also useful in a context similar to when certain "why comments" are used: when it is not obvious why some code must be executed at a given place, after or before something. But while the why comment may tell you why a statement is there, the checklist comment used in the same case is more biased towards telling you what rules to follow if you want to modify it (in this case the rule is, follow a given ordering), without breaking the code behavior. cluster.c: /* Update our info about served slots. * * Note: this MUST happen after we update the master/replica state * so that CLUSTER_NODE_MASTER flag will be set. */ Checklist comments are very common inside the Linux kernel, where the order of certain operations is extremely important. GUIDE COMMENT I abuse guide comments at such a level that probably, the majority of comments in Redis are guide comments. Moreover guide comments are exactly what most people believe to be completely useless comments. * They don't state what is not clear from the code. * There are no design hints in guide comments. Guide comments do a single thing: they babysit the reader, assist him or her while processing what is written in the source code by providing clear division, rhythm, and introducing what you are going to read. Guide comments’ sole reason to exist is to lower the cognitive load of the programmer reading some code. rax.c: /* Call the node callback if any, and replace the node pointer * if the callback returns true. */ if (it->node_cb && it->node_cb(&it->node)) memcpy(cp,&it->node,sizeof(it->node)); /* For "next" step, stop every time we find a key along the * way, since the key is lexicographically smaller compared to * what follows in the sub-children. */ if (it->node->iskey) { it->data = raxGetData(it->node); return 1; } There is nothing that the comments are adding to the code above. The guide comments above will assist you reading the code, moreover they'll acknowledge you about the fact you are understanding it right. More examples. networking.c: /* Log link disconnection with replica */ if ((c->flags & CLIENT_SLAVE) && !(c->flags & CLIENT_MONITOR)) { serverLog(LL_WARNING,"Connection with replica %s lost.", replicationGetSlaveName(c)); } /* Free the query buffer */ sdsfree(c->querybuf); sdsfree(c->pending_querybuf); c->querybuf = NULL; /* Deallocate structures used to block on blocking ops. */ if (c->flags & CLIENT_BLOCKED) unblockClient(c); dictRelease(c->bpop.keys); /* UNWATCH all the keys */ unwatchAllKeys(c); listRelease(c->watched_keys); /* Unsubscribe from all the pubsub channels */ pubsubUnsubscribeAllChannels(c,0); pubsubUnsubscribeAllPatterns(c,0); dictRelease(c->pubsub_channels); listRelease(c->pubsub_patterns); /* Free data structures. */ listRelease(c->reply); freeClientArgv(c); /* Unlink the client: this will close the socket, remove the I/O * handlers, and remove references of the client from different * places where active clients may be referenced. */ unlinkClient(c); Redis is *literally* ridden of guide comments, so basically every file you open will contain plenty of them. Why bother? Of all the comment types I analyzed so far in this blog post, I'll admit that this is absolutely the most subjective one. I don't value code without such comments as less good, yet I firmly believe that if people regard the Redis code as readable, some part of the reason is because of all the guide comments. Guide comments have some usefulness other than the stated ones. Since they clearly divide the code in isolated sections, an addition to the code is very likely to be inserted in the appropriate section, instead of ending in some random part. To have related statements nearby is a big readability win. Also make sure to check the guide comment above before the unlinkClient() function is called. The guide comment briefly tells the reader what the function is going to do, avoiding the need to jump back into the function if you are only interested in the big picture. TRIVIAL COMMENTS Guide comments are very subjective tools. You may like them or not. I love them. However, a guide comment can degenerate into a a very bad comment: it can easily turn into a "trivial comment". A trivial comment is a guide comment where the cognitive load of reading the comment is the same or higher than just reading the associated code. The following form of trivial comment is exactly what many books will tell you to avoid. array_len++; /* Increment the length of our array. */ So if you write guide comments, make sure you avoid writing trivial ones. DEBT COMMENTS Debt comments are technical debts statements hard coded inside the source code itself: t_stream.c: /* Here we should perform garbage collection in case at this point * there are too many entries deleted inside the listpack. */ entries -= to_delete; marked_deleted += to_delete; if (entries + marked_deleted > 10 && marked_deleted > entries/2) { /* TODO: perform a garbage collection. */ } The snippet above is extracted from the Redis streams implementation. Redis streams allow to delete elements from the middle using the XDEL command. This may be useful in different ways, especially in the context of privacy regulations where certain data cannot be retained no matter what data structure or system you are using in order to store them. It is a very odd use case for a mostly append only data structure, but if users start to delete more than 50% of items in the middle, the stream starts to fragment, being composed of "macro nodes". Entries are just flagged as deleted, but are only reclaimed once all the entries in a given macro node are freed. So your mass deletions will change the memory behavior of streams. Right now, this looks like a non issue, since I don't expect users to delete most history in a stream. However it is possible that in the future we may want to introduce garbage collection: the macro node could be compacted once the ratio between the deleted entries and the existing entries reach a given level. Moreover nearby nodes may be glued together after the garbage collection. I was kind of afraid that later I would no longer remember what were the entry points to do the garbage collection, so I put TODO comments, and even wrote the trigger condition. This is probably not great. A better idea was instead to write, in the design comment at the top of the file, why we are currently not performing GC. And what are the entry points for GC, if we want to add it later. FIXME, TODO, XXX, "This is a hack", are all forms of debt comments. They are not great in general, I try to avoid them, but it's not always possible, and sometimes instead of forgetting forever about a problem, I prefer to put a node inside the source code. At least one should periodically grep for such comments, and see if it is possible to put the notes in a better place, or if the problem is no longer relevant or could be fixed right away. BACKUP COMMENTS Finally backup comments are the ones where the developer comments older versions of some code block or even a whole function, because she or he is insecure about the change that was operated in the new one. What is puzzling is that this still happens now that we have Git. I guess people have an uneasy feeling about losing that code fragment, considered more sane or stable, in some years old commit. But source code is not for making backups. If you want to save an older version of a function or code part, your work is not finished and cannot be committed. Either make sure the new function is better than the past one, or take it just in your development tree until you are sure. Backup comments end my classification. Let's try some conclusion. # Comments as an analysis tool. Comments are rubber duck debugging on steroids, except you are not talking with a rubber duck, but with the future reader of the code, which is more intimidating than a rubber duck, and can use Twitter. So in the process you really try to understand if what you are stating *is acceptable*, honorable, good enough. And if it is not, you make your homework, and come up with something more decent. It is the same process that happens while writing documentation: the writer attempts to provide the gist of what a given piece of code does, what are the guarantees, the side effects. This is often a bug hunting opportunity. It is very easy while describing something to find that it has holes... You can't really describe it all because you are not sure about a given behavior: such behavior is just emerging from complexity, at random. You really don't want that, so you go back and fix it all. I find this a splendid reason to write comments. # Writing good comments is harder than writing good code You may think that writing comments is a lesser noble form of work. After all you *can code*! However consider this: code is a set of statement and function calls, or whatever your programming paradigm is. Sometimes such statements do not make much sense, honestly, if the code is not good. Comments require always to have some design process ongoing, and to understand the code you are writing in a deeper sense. On top of that, in order to write good comments, you have to develop your writing skills. The same writing skills will assist you writing emails, documentation, design documents, blog posts, and commit messages. I write code because I have an urgent sense to share and communicate more than anything else. Comments coadiuvate the code, assist it, describe our efforts, and after all I love writing them as much as I love writing code itself. (Thanks to Michel Martens for giving feedbacks during the writing of this blog post) Comments

• LOLWUT: a piece of art inside a database command
  Sep 12, 2018

  The last few days have been quite intense. One of the arguments, about the dispute related to replacing or not the words used in Redis replication with different ones, was the following: is it worthwhile to do work that does not produce any technological result? As I was changing the Redis source code to get rid of a specific word where possible, I started to think that whatever my idea was about the work I was doing, I’m the kind of person that enjoys writing code that has no measurable technological effects. Replacing words is just annoying, even if, even there, there were a few worthwhile technological challenges. But there is some other kind of code that I believe has a quality called “hack value”. It may not solve any technological problem, yet it’s worth to write. Sometimes because the process of writing the code is, itself, rewarding. Other times because very technically advanced ideas are used to solve a not useful problem. Sometimes code is just written for artistic reasons. In some way the Twitter discussion of the last days, mostly uninformed, chaotic, heated, made me think that, at this point, we are very far from the first hackers in the 60s. As I get older I find that it is harder and harder to talk about technology with an hacking perspective, where there are no walls or pre-cooked ideas, and the limit is the exploration. For everything you say there is a best practice. For every idea there is a taboo. To this new setup I say LOLWUT, since I don’t feel represented by it, nor it represents hacking, at least in my vision. So the idea was to spend some technologically useless time in order to explore something of the 60s. My attention went immediately to one of the computer art pieces I love the most: Schotter, by Georg Nees (https://en.wikipedia.org/wiki/Georg_Nees). With the help of a plotter and ALGOL programs, Nees explored writing programs to generate art using caos (randomness) and repeating patterns. Schotter is remarkable because of the simplicity of the piece and the deep meaning that the observer can find looking at it. Under a surface of total calm and order, deep inside the disorder hides. Or, if you put it upside down it becomes like the sea during a tempest. However the surface may look impetuous, the deep sea remains calm. Is it possible to turn a piece of art into a database command? This was challenging because Redis is mostly used by a command line interface. Nowadays terminals are for sure fancier than the ones of the past, but yet to display decent graphics is hard. On the other side there is the huge advantage of the real time computation: a piece of art can be dynamic, changing every time it is generated. Before continuing, I want to show you the final result: img://antirez.com/misc/lolwut1.png While very low resolution the idea of the original piece is still there. To make this possible I used a trick that recently was used by multiple programs trying to display interesting things in a text console. It involves using the Braille unicode charset in order to create a pixel matrix which is more dense than the individual characters of the console. Specifically, for each character, it is possible to fit a 2x8 grid of pixels. The second part of the experiment was to make the art piece parametric: img://antirez.com/misc/lolwut2.png It is possible to generate different versions of the original piece, changing the number of squares and the output resolution. Finally the source code wanted to be an example of literate programming, being written in a form that resembles more a tutorial describing what everything does and why, instead of some opaque generator. You can find the code here: https://github.com/antirez/redis/blob/unstable/src/lolwut.c LOLWUT is also going to be a tradition starting from Redis 5. At each new major version of Redis what the command does will change completely, only a set of rules will be fixed: 1. It can’t do anything technologically useful. 2. It should be fast at doing what it does, so that it is safe to call LOLWUT on production instances. 3. The output should be entertaining in some way. I wrote the first one for Redis 5, for the next versions, if I find interest, I’ll ask somebody else that contributed to Redis to write the other LOLWUT versions, otherwise I’ll write it again myself (but I hope that’s not the case). LOLWUT should remember ourselves that the work we do, programming, did not start just in order to produce something useful. Initially it was mainly a matter of exploring possibilities. I hope that LOLWUT also will remind the Redis community that computers are about humans, and that it is not possible to reason in an aseptic way just thinking at the technological implications. There are people using systems, people building systems, and so forth. Comments

• On Redis master-slave terminology
  Sep 06, 2018

  Today it happened again. A developer, that we’ll call Mark to avoid exposing his real name, read the Redis 5.0 RC5 change log, and was disappointed to see that Redis still uses the “master” and “slave” terminology in order to identify different roles in Redis replication. I said that I was sorry he was disappointed about that, but at the same time, I don’t believe that terminology out of context is offensive, so if I use master-slave in the context of databases, and I’m not referring in any way to slavery. I originally copied the terms from MySQL, and now they are the way we call things in Redis, and since I do not believe in this battle (I’ll tell you later why), to change the documentation, deprecate the API and add a new one, change the INFO fields, just to make a subset of people that care about those things more happy, do not make sense to me. After it was clear that I was not interested in his argument, Mark accused me of being fascist. Now I’m Italian, and incidentally my grand grand father was put in jail for years by fascists because he was communist and was against the regime. He was released to die in a couple of months at home. The father of my mother instead went in the north of Italy for II World War, and was able to escape from the Nazis for a miracle. Stayed 5 years as a refugee, and eventually returned home to become the father of my mother. Mark do not care about the terminology he uses against other people, if the matter at hand is to make sure people that may potentially feel offended will not. Now, it’s time for you to know my political orientation, so that you can put in context my refuse to change terminology. I want my government to be more open to immigration, including economical immigration, I do not accept any racism and I was strongly in favor of “ius soli” law here in Italy. I do not just accept conceptually same-sex marriage, but I really love the beauty that there is in two men or women kissing, making sex, adopting a child. Every day in Facebook and with my social sphere I actively talk about politics in order to push equality. I believe in a systematic bias that our society perpetuates against women, and I’m proud to live in a country where women are free to not recognize the child as their own after giving birth, in order to have the same rights of the biological father that can go away: this was a big win of the European feminist movement in the 70s, together with the abortion right. I’m proud that in my country there is no death penalty like there is not in the rest of EU, that guns are mostly banned, that there is universal healthcare for free. I do not believe to be fascist or racist honestly, and I write almost daily about all this things on Facebook with my friends, talk at people on the street, and so forth. For years and years, since I was 16. So, what’s the problem with changing this terminology? The first problem is that every terminology is offensive in principle, and I don’t want to accept this idea that certain words that are problematic, especially for Americans to make peace with their past, should be banned. For example if I’m terminally ill, the “short living request” terminology may be offensive, it reminds me that I’m going to die, or that my father is going to die. Instead of banning every word out there, we should make the mental effort to do better than the political correctness movement that stops at the surface. So, let’s call it master-slave, and instead make a call for US, where a sizable black population is very poor, to have free healthcare, to have cops that are less biased against non-white people, to stop death penalty. This makes really a difference. For instance Europeans that are a lot less sensible to political correctness, managed to do a much better job on that stuff. There is more: I believe that political correctness has a puritan root. As such it focuses on formalities, but actually it has a real root of prejudice against others. For instance Mark bullied me because I was not complying with his ideas, showing problems at accepting differences in the way people think. I believe that this environment is making it impossible to have important conversations. For instance nobody at this point want to talk about women in tech and about the systematic bias of women in our society (to the point that recently in Japan it was discovered that women were systematically stopped from entering the best medical schools). People will go away once the discussion starts, because everybody knows that at this point to talk about this matters is a huge PR error, can cost you your job or company. Many, while reading this blog post, are thinking that I’m crazy at writing this stuff, even if they think likewise. Well, I don’t want that the people that did this to the our ability to have conversations will get a free pass to say what to do to others, because conversations is the only way we can make people that yet don't have an open vision to change ideas. Moreover I don't believe in the right to be offended, because it's a subjective thing. Different groups may feel offended by different things. To save the right of not being offended it becomes basically impossible to say or do anything in the long run. Is this the evolution of our society? A few days ago on Hacker News I discovered I can no longer say "fake news" for instance. I want a world of equity, opportunities, redistribution of wealth, and open borders. But the way I believe this world can be obtained is not by banning words, nor by stalking people on Twitter so that they comply with your ideology. I’ll continue my local political activity, and I’ll continue to write open source software. I hope that Mark, and the others like Mark, will let me live my life as I decided to do. Comments

• Redis is not "open core"
  Aug 25, 2018

  Human beings have a strong tendency to put new facts into pre-existing categories. This is useful to mentally and culturally classify similar events under the same logical umbrella, so when two days ago I clarified that the Redis core was still released under the vanilla BSD license, and only certain Redis modules developed by Redis Labs were going to change license, from AGPL to a different non open source license, people said “Ah! Ok you are going open core”. The simplification this time does not work if it is in your interest to capture the truth of what is happening here. An open core technology requires two things. One is that the system is modular, and the other is that parts of such system are made proprietary in order to create a product around an otherwise free software. For example providing a single node of a database into the open source, and then having the clustering logic and mechanism implemented in a different non-free layer, is an open core technology. Similarly is open core if I write a relational database with a modular storage system, but the only storage that is able to provide strong guarantees is non free. In an open core business model around an open source system it is *fundamental* that you take something useful out of the free software part. Now for some time Redis is a modular system. You can use Redis modules in order to write many things, including new distributed systems using the recently introduced cluster message bus API, or new data types that look native. However the reason to make Redis modular was not to remove something useful from the system and put a price tag on it. For instance one of the new data structures in Redis 5, the streams, are part of the core and are released under the BSD license. Streams were implemented when Redis was already a modular system. Redis modules started from a different observation. As a premise I should say that I’m a very conservative person about software development. I believe that Redis should be focused to address just things that, when operated with in-memory data structures, offer strong advantages over other ways to do the same thing. I don’t want Redis to do much more than it does, or to employ every possible consistency tradeoff. I want Redis to be Redis, that is, this general tool that the developer can use in different ways to solve certain problems. However at Redis Labs we observed multiple times that it’s a bit a shame that Redis cannot solve certain specific problems. For instance what about if Redis was a serious full text search engine? Also well, developers want so much JSON, what about having an API to talk directly JSON? And given that in-memory graphs if represented wisely can be so fast, what about having graph database abilities, with a rich query language? Redis Labs customers often asked directly for such things. And actually, such features could be cool, but it’s not Redis, I’m not interested, and the open source side of Redis does not have the development force to keep all this things going btw. And this is a major advantage both for Redis and for Redis Labs: it is relatively cheap to pay just me and a few more OSS development time internally, while allocating the rest of the resources to development of things that are useful for the Redis Labs business, like making sure the Redis enterprise SaaS and products are good. There is anyway a great deal of contributions arriving from the community. And I also keep saying “no” to all this fancy ideas that would keep Redis in other areas… which is also a problem. Still to have such things similar to Redis but outside the Redis scope, would be cool, because you know, while it’s not Redis mission, people may very well use a fast inverted index with full text search capabilities that you can feed in real time, while it serves a very good amount of queries per core at the same time. This is what Redis Labs is doing, it’s using the same Redis technology and approach to do more than what Redis wanted to do. Not just in the functionality areas, but also in other areas like consistency models. I’m very opinionated about certain things, and I think that, for instance, CRDTs while super cool in certain use cases, where not the right thing for Redis, to retain the same memory footprint, performance, simplicity, even at the cost of having a weaker consistency model. So Redis Labs, together with a top researcher in the area, did it (and this is a proprietary product without any source available). I can see how such feature can be tremendously useful for certain operations, but Redis was not there to solve everything, and Redis Labs did it. This is not open core. Redis Labs is doing things that you would never see from me: for bandwidth, and because I believe that not all the softwares must eventually become huge. So I think that calling this model “open core” is misleading, nothing is removed from the Redis table, just new things are explored, while trying to follow the “Redis way” in other areas otherwise not touched by the Redis project. Comments

• Redis will remain BSD licensed
  Aug 22, 2018

  Today a page about the new Common Clause license in the Redis Labs web site was interpreted as if Redis itself switched license. This is not the case, Redis is, and will remain, BSD licensed. However in the era of [edit] uncontrollable spreading of information, my attempts to provide the correct information failed, and I’m still seeing everywhere “Redis is no longer open source”. The reality is that Redis remains BSD, and actually Redis Labs did the right thing supporting my effort to keep the Redis core open as usually. What is happening instead is that certain Redis modules, developed inside Redis Labs, are now released under the Common Clause (using Apache license as a base license). This means that basically certain enterprise add-ons, instead of being completely closed source as they could be, will be available with a more permissive license. I think that Redis Labs Common Clause page did not provide a clear and complete information, but software companies often make communication errors, it happens. To me however, it looks more important that while running a system software business in the “cloud era” (LOL) is very challenging using an open source license, yet Redis Labs totally understood and supported the idea that the Redis core is an open source project, in the *most permissive license ever*, that is, BSD, and during the years provided a lot of funding to the project. The reason why certain modules developed internally at Redis Labs are switching license, is because they are added value that Redis Labs wants to be able to provide only to end users that are willing to compile and install the system themselves, or to the Redis Labs customers using their services. But it’s not ok to give away that value to everybody willing to resell it. An example of such module is RediSearch: it was AGPL and is now going to be Apache + Common Clause. About myself, I’ll keep writing BSD code for Redis. For Redis modules I’ll develop, such as Disque, I’ll pick AGPL instead, for similar reasons: we live in a “Cloud-poly”, so it’s a good idea to go forward with licenses that will force other SaaS companies to redistribute back their improvements. However this does not apply to Redis itself. Redis at this point is a 10 years collective effort, the base for many other things that we can do together, and this base must be as available as possible, that is, BSD licensed. We at Redis Labs are sorry for the confusion generated by the Common Clause page, and my colleagues are working to fix the page with better wording. Comments

• Redis Lua scripting: several security vulnerabilities fixed
  Jun 13, 2018

  A bit more than one month ago I received an email from the Apple Information Security team. During an auditing the Apple team found a security issue in the Redis Lua subsystem, specifically in the cmsgpack library. The library is not part of Lua itself, it is an implementation of MessagePack I wrote myself. In the course of merging a pull request improving the feature set, a security issue was added. Later the same team found a new issue in the Lua struct library, again such library was not part of Lua itself, at least in the release of Lua we use: we just embedded the source code inside our Lua implementation in order to provide some functionality to the Lua interpreter that is available to Redis users. Then I found another issue in the same struct package, and later the Alibaba team found many other issues in cmsgpack and other code paths using the Lua API. In a short amount of time I was sitting on a pile of Lua related vulnerabilities. Those vulnerabilities are mostly relevant in the specific case of providing managed Redis severs on the cloud, because it is very unlikely that the vulnerabilities discovered can be used without direct access to the Redis server: many Redis users don’t use the cmsgpack or the struct package at all, and who does will very unlikely feed them with untrusted input. However for cloud providers things are different: they have Redis instances, sometimes in multi tenancy setups, exposed to the user that subscribed for the service. She or he can send anything to such Redis instances, triggering the vulnerabilities, corrupting the memory, violating the Redis process, and potentially taking total control of the Redis process. For instance this simple Python program can crash Redis using one of the cmsgpack vunlerabilities [1]. [1] https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a However from the point of view of normal Redis users that control what is sent to their instances, the risk is limited to feeding untrusted data to a function like struct.unpack(), after selecting a particularly dangerous decoding format “bc0” in the format argument. # Coordinating the advisory Thanks to the cooperation and friendly communications between the Apple Information Security team, me, and the Redis cloud providers, I tried to coordinate the release of the vulnerability after contacting all the major Redis providers out there, so that they could patch their systems before the bug was published. I provided a single patch, so that the providers could easily apply it to their systems. Finally between yesterday and today I prepared new patch releases of Redis 3, 4 and 5, with the security fixes included. They are all already released if you are reading this blog post. Unfortunately I was not able to contact smaller or newer cloud providers. The effort to handle the communication with Redis Labs, Amazon, Alibaba, Microsoft, Google, Heroku, Open Redis and Redis Green was already massive, and the risk of leaks extending the information sharing with other subjects even higher (every company included many persons handling the process). I’m sorry if you are a Redis provider finding about this vulnerability just today, I tried to do my best. I want to say thank you to the Apple Information Security team and all the other providers for the hints and help about this issue. # The problem with Lua Honestly when the Redis Lua engine was designed, it was not conceived with this security model of the customer VS the cloud provider in mind. The assumption kinda was that you can trust who pokes with your Redis server. So in general the Lua libraries were not scrutinized for security. The feeling back then was, if you have access to Redis API, anyway you can do far worse. However later things evolved, and cloud providers restricted the API of Redis to expose to their customers, so that it was possible to provide managed Redis instances. However while things like the CONFIG or DEBUG commands were denied, you can’t really avoid exposing EVAL and EVALSHA. The Redis Lua scripting is one of the top used features in our community. So gradually, without me really noticing, the Lua libraries became also an attack vector in a security model that should instead be handled by Redis, because of the changing system in the way Redis is exposed and provided to the final user. As I said, in this model more than the Redis user, is the managed Redis “cloud” provider to be affected, but regardless it is a problem that must be handled. What we can do in order to improve the current state of cloud providers security, regarding the specific problem with Lua scripting? I identified a few things that I want to do in the next months. 1. Lua stack protection. It looks like Lua can be compiled, with some speed penalty, in a way that ensures that it is not possible to misuse the Lua stack API. To be fair, I think that the assumptions Lua makes about the stack are a bit too trivial, with the Lua library developer having to constantly check if there is enough space on the stack to push a new value. Other languages at the same level of abstraction have C APIs that don’t have this problem. So I’ll try to understand if the slowdown of applying more safeguards in the Lua low level C API is acceptable, and in that case, implement it. 2. Security auditing and fuzz testing. Even if my time was limited I already performed some fuzz testing in the Lua struct library. I’ll continue with an activity that will check for other bugs in this area. I’m sure there are much more issues, and the fact that we found just a given set of bugs is only due to the fact that there was no more time to investigate the scripting subsystem. So this is an important activity that is going to be performed. Again at the end of the activity, I’ll coordinate with the Redis vendors so that they could patch in time. 3. From the point of view of the Redis user, it is important that when some untrusted data is sent to the Lua engine, an HMAC is used in order to ensure that the data was not modified. For instance there is a popular pattern where the state of an user is stored in the user cookie itself, to be later decoded. Such data may later be used as input for Redis Lua functions. This is an example where an HMAC is absolutely needed in order to make sure that we read what we previously stored. 4. More Lua sandboxing. There should be plenty of literature and good practices about this topic. We already have some sandboxing implemented, but my feeling from my security days, is that sandboxing is ultimately always a mouse and cat game, and can never be executed in a perfect way. CPU / memory abuses for example may be too complex to track for the goals of Redis. However we should at least be sure that violations may result in a “graceful” abort without any memory content violation issue. 5. Maybe it’s time to upgrade the Lua engine? I’m not sure if newer versions of Lua are more advanced from the point of view of security, however we have the huge problem that upgrading Lua will result in old script potentially no longer working. A very big issue for the Redis community, especially since, for the kind of scripts Redis users normally develop, a more advanced Lua version is only marginally useful. # The issues The problems fixed are listed in the following commits: ce17f76b Security: fix redis-cli buffer overflow. e89086e0 Security: fix Lua struct package offset handling. 5ccb6f7a Security: more cmsgpack fixes by @soloestoy. 1eb08bcd Security: update Lua struct package for security. 52a00201 Security: fix Lua cmsgpack library stack overflow. The first commit is unrelated to this effort, and is a redis-cli buffer overflow that can be exploited only passing a long host argument in the command line. The other issues are the problems that we found on cmsgpack and the struct package. The two scripts to reproduce the issues are the following: https://gist.github.com/antirez/82445fcbea6d9b19f97014cc6cc79f8a and https://gist.github.com/antirez/bca0ad7a9c60c72e9600c7f720e9d035 Both authored by the Apple Information Security team. However the first was modified by me in order to make it more reliably causing the crash. # Versions affected Basically every Redis with Lua scripting is affected. The fixes are available as the following Github tags: 3.2.12 4.0.10 5.0-rc2 The stable release (4.0.10) is also available at http://download.redis.io as usually. Releases tarball hashes are available here: https://github.com/antirez/redis-hashes Please note that the versions released also include different other bugfixes, so it’s a good idea to also read the release notes to know what other things you are upgrading by switching to the new version. I hope to be back with a blog post in the future with the report of the security auditing that is planned for the Lua scripting subsystem in Redis. Comments

• Clarifications on the Incapsula Redis security report
  Jun 02, 2018

  A few days ago I started my day with my Twitter feed full of articles saying something like: “75% of Redis servers infected by malware”. The obvious misquote referred to a research by Incapsula where they found that 75% of the Redis instances left open on the internet, without any protection, on a public IP address, are infected [1]. [1] https://www.incapsula.com/blog/report-75-of-open-redis-servers-are-infected.html Many folks don’t need any clarification about all this, because if you have some grip on computer security and how Redis works, you can contextualize all this without much efforts. However I’m writing this blog post for two reasons. The obvious one is that it can help the press and other users that are not much into security and/or Redis to understand what’s going on. The second is that the exposed Redis instances are a case study about safe defaults that should be interesting for the security circles. The Incapsula report === Let’s start with the Incapsula report. What they did was to analyze exposed Redis instances on the internet. Instances that everybody from any place of the internet can access, because they are listening for connections in a public IP address, without any password protecting them. It is like if they were HTTP servers, but it’s Redis instead, that is not designed to be left exposed. This is far from a new story. Because of Redis popularity the number of total Redis installations is pretty huge, and a fraction of these installations are left exposed. It’s like that since the start basically. People spin a virtual machine in some cloud provider, install Redis, find that they cannot access it, open the port of the VM to anyone, and the instance is at this point running unprotected. They only thing that changed is that most of those instances in the past were left running unaffected in many cases. Maybe some script kiddie could connect and call “INFO” or a few more commands to check what there was inside, and that was it most of the times. Now the new crypto mining obsession is providing attackers a very good reason to break into other people’s systems. Actually the best of the reasons: money. So the same exposed instances are now cracked in order to install some software to mine some kind of crypto currency. Incapsula checked the percentage of instances that look violated. The way they collected the kind of attacks used against Redis instances was by running, on purpose, a set of exposed instances, to monitor how the attackers could target them. Many of the attacks look like variations on my own example attack [2]. [2] http://antirez.com/news/96 Also it is worth to note that to scan the IPv4 address space in order to find exposed instances of any kind of service is trivial. You could use, for instance, masscan. However you could do that 20 years ago using my own hping, and I remember doing this back then with success indeed. TLDR: there are open Redis instances on the internet because they are misconfigured. Attackers profit from them by installing some kind of mining software, or for other reasons. But there is more… keep reading. Protected mode =========== Security is a terrible field in my opinion. I worked in such field, and decided to go away once it started to be no longer an underground affair. People are very opinionated about security issues, while, at the same time, there is little real care about, for instance, performing security auditings on real world systems (something that Google project zero is changing a bit, btw). So after receiving for the Nth time some PGP encrypted email saying that Redis was vulnerable to a temp file creation attack, I wrote the blog post at [2], just to tell: “maybe you are missing the point of Redis security”. I basically published an attack against my own system that I could find in a few minutes of research, and that was very serious. The message was, again, Redis is not designed to be left exposed. And if you really want to play hack3rzzz look, there are more interesting things to do. However by doing that I put on the hands of script kiddies a great weapon to break into thousands of Redis instances open everywhere. In order to pay back from my fault, in an attempt to atone my sin, I started to think about what I could do to make the situation simpler. One problem of Redis 3.x was that by default it would listen to all the IP addresses, so it was simple to misconfigure: just open the port, or have no firewalling at all, and the instance is exposed. The security mantra about that was “run with safe defaults”. That is, in the case of Redis, a networked cache, to have a default configuration that basically more or less does not work for most real world users out of the box. And what was worse, people that don’t have a clue would just “bind *” to fix the problem, and we are back at the initial condition. So I tried to find something a bit smarter, a feature that I named “protected mode”, with great disappointment of a group of 386 processors that manifested in front of my houses for days. The idea of protected mode is to listen to every address, but if the connection is not local, the server replies with an error explaining *why* it is not working as expected, what to do, and the risks involved in just doing the silly thing of opening the instance to everybody by disabling protected mode. This is an example of how the feature works: $ nc 192.168.1.194 6379 -DENIED Redis is running in protected mode because protected mode is enabled, no bind address was specified, no authentication password is requested to clients. In this mode connections are only accepted from the loopback interface. If you want to connect from external computers to Redis you may adopt one of the following solutions: 1) Just disable protected mode sending the command 'CONFIG SET protected-mode no' from the loopback interface by connecting to Redis from the same host the server is running, however MAKE SURE Redis is not publicly accessible from internet if you do so. Use CONFIG REWRITE to make this change permanent. 2) Alternatively you can just disable the protected mode by editing the Redis configuration file, and setting the protected mode option to 'no', and then restarting the server. 3) If you started the server manually just for testing, restart it with the '--protected-mode no' option. 4) Setup a bind address or an authentication password. NOTE: You only need to do one of the above things in order for the server to start accepting connections from the outside. The advantage here is that: 1. The user knows what to do in order to fix this situation. It’s much better than “connection refused”. 2. We try to avoid that the user disables protected mode without understanding what is the risk involved, and that there are other solutions (setting a password or binding to certain interfaces). Disillusion ======== So my illusion was that, maybe, it could be more helpful than the safe defaults. But unfortunately you can’t fix the fact that a percentage of users just don’t care, that some don’t even know they are installing Redis, and that it’s there just as a side effect of being a dependency for something else, or is installed by a script, or is included in the ABI image you are running and so forth. Initially by grepping on Shodan it looked like Redis 4.0 protected mode was, actually, helping quite a lot! The open instances I could find replied “-DENIED” for the most part. That was great. However after the publication of [1] I asked Shodan (btw they rock and were super helpful on Twitter) if I could have the version breakdown of the exposed Redis instances. And the result is, there are still tons of Redis 4.0 instances exposed [3]. [3] https://asciinema.org/a/8heQvivQkFmUrisbb9FQLfMBj It is true that they are less than the 3.0 instances, but they are still a lot. By investigating more I was even told that there are VM images where Redis protected mode is *removed* by the installation script by default, since sometimes people are annoyed by security features. They want things to just work over the network out of the box, so this is what they do, remove the annoyance. Then such image becomes popular, and many folks install it without knowing what’s going on, and when Redis 4 is installed protected mode is off and the instance is exposed. I think this is a lesson about safe defaults that can be trivially disabled by users. It looks like that in some way they could help, but just in reducing by some percentage the number of incidents, not to make them a rare exception. What’s next? ========= One of the fundamental problems in the Redis security model is that the server can be reconfigured via the normal API, using special commands like the CONFIG command. This is a very valuable feature of Redis, but it makes much simpler to break into the instance once you have access to the Redis API, and normal applications using Redis don’t need this level of access. So in the course of Redis 6, what will happen is that we’ll introduce ACLs. Again, the way this feature will be introduced will try to be a “no pain” experience for existing users. If you connect without any credential, Redis will log in the client automatically using the “default” user, that can do everything applications normally do, but will deny all the administrative commands. Of course it will be possible to make it more strict by configuring things differently, create new users that can only call certain commands on keys matching a given pattern, and so forth. In the course of Redis 6 we plan to also merge support for SSL connections, while this is unlikely to have any impact on the issue discussed here, because by default Redis will run unencrypted and the feature will be opt-in, however SSL is also one step forward for a more secure Redis experience in certain environments. However my hopes are on ACLs, because it looks unlikely that the casual user will make the default account able to run the administrative commands. Especially because we plan to log connections originating from the local host as the “admin” user directly. If this goes as I hope, we’ll continue to see Redis 6 instances exposed, because it is inevitable, but at least those Redis 6 instances should make it harder to compromise the whole system. At least in theory: Redis EVAL command allows execution of Lua scripts, and such feature should be allowed by default since is a fundamental Redis feature. We try to have a kinda sandboxed Lua execution environment, but if you followed IT security for some time, you know that sandboxes are always imperfect, and more an exercise in finding how to escape them than a sealing solution. This time however I’ll avoid publishing an attack just to make a point. Comments

• A short tale of a read overflow
  Feb 07, 2018

  [This blog post is also experimentally available on Medium: https://medium.com/antirez/a-short-tale-of-a-read-overflow-b9210d339cff] When a long running process crashes, it is pretty uncool. More so if the process happens to take a lot of state in memory. This is why I love web programming frameworks that are able, without major performance overhead, to create a new interpreter and a new state for each page view, and deallocate every resource used at the end of the page generation. It is an inherently more reliable programming paradigm, where memory leaks, descriptor leaks, and even random crashes from time to time do not constitute a serious issue. However system software like Redis is at the other side of the spectrum, a side populated by things that should never crash. Months ago I received a crash report from my colleague Dvir Volk. He was developing his RediSearch Redis module, so it was not clear if the crash was due to a programming error inside the module, perhaps corrupting the heap, or a bug inside Redis. However it looked a lot like a real problem into the radix tree implementation: === REDIS BUG REPORT START: Cut & paste starting from here === # Redis 999.999.999 crashed by signal: 11 # Crashed running the instuction at: 0x7fceb6eb5af5 # Accessing address: 0x7fce9c400000 | Backtrace: | redis-server *:7016 [cluster](raxRemoveChild+0xd3)[0x49af53] | redis-server *:7016 [cluster](raxRemove+0x34f)[0x49b34f | redis-server *:7016 [cluster](slotToKeyUpdateKey+0x1ad)[0x4415dd] The radix tree is full of memmove() calls, and Redis crashed exactly trying to access a memory address that was oddly zero padded at the end: 0x7fce9c400000. My first thought was, I’m sure I’m doing some wrong memory movement here, and the address gets overwritten with zeroes, leading to the crash when the program attempts the deference the address. I’m pretty proud about my radix tree implementation. Not because of the implementation itself, while it’s a complex data structure to implement, it’s not rocket science. But because of the fuzz tester that comes with it and is able to cover the whole source code (trivial) and a lot of non trivial state (that’s definitely more interesting). The fuzz tester does not do fuzzing just to reach a crash, it compares the implementation of the radix tree dictionary and iterator with a reference implementation that uses an hash table and qsort, to have exactly the same semantics, but in a short and easy to audit implementation. After receiving the crash report I improved the fuzz tester, ran it for days, with and without Valgrind, wrote additional data models, created tests using 100 millions of keys, but despite the efforts I could not reproduce the crash. A few days ago I would discover that there was no bug in the implementation I was testing, but at the time I was not aware of that. I could simply never find a bug that was not there. So after failing to reproduce I gave up. One week ago I received two other additional bug reports that were almost the same. And again, the addresses were zero padded. Dvir crash: Accessing address: 0x7fce9c400000 Issue 4605: Accessing address: 0x7f2959e00000 Issue 4642: Accessing address: 0x7f0e9b800000 It was time to read every memmove, memcpy, realloccall inside the source code, trying to figure out if there was something wrong that for some reason the fuzz tester was not able to catch. I found nothing, but then inspecting the Redis crash report I noticed something funny. During crashes Redis reports the memory mapped areas of the process, things like the following: *** Preparing to test memory region 7f0e8c400000 (255852544 bytes) Now, if you add 255852544to 0x7f0e8c400000, the result is 0x7f0e9b800000 , which is exactly the accessed address in the crash reported in issue 4642. So the program was not crashing because the memory address was corrupted, but because it was accessing an address immediately after the end of the heap. I checked the other issues, and the same was true in all the instances. Basically the end of the heap, at the edge of the start of unmapped addresses, was acting as a memory guard in order to detect and crash when an access was performed outside the bounds. This is a common technique that certain C memory sanitation tools used to employ in the past. Such tools would provide a drop in replacement formalloc() that would return addresses allocated at the edge of a non accessible memory page. Every overflow would be immediately detected in this way. Because the program would crash only in that case, when deallocating a radix tree node at the end of the heap, it was simple to realize that the problem was a read overflow. You can never detect a read overflow otherwise: it will just access data outside your structure, but inside mapped memory, so the bug would be totally harmless and silent, with the exception of doing the same operations at the end of the mapped region. Finally I had a clear place where to look, this portion of C code: /* 3. Remove the edge and the pointer by memmoving the remaining children pointer and edge bytes one position before. */ int taillen = parent->size - (e - parent->data) - 1; debugf("raxRemoveChild tail len: %d\n", taillen); memmove(e,e+1,taillen); /* Since we have one data byte less, also child pointers start one byte before now. */ memmove(((char*)cp)-1,cp,(parent->size-taillen-1)*sizeof(raxNode**)); /* Move the remaining "tail" pointer at the right position as well. */ size_t valuelen = (parent->iskey && !parent->isnull) ? sizeof(void*) : 0; memmove(((char*)c)-1,c+1,taillen*sizeof(raxNode**)+valuelen); /* 4. Update size. */ parent->size--; I asked the user to send me the redis-server binary that produced the crash report, and reading the disassembled code it was clear that many CPU registers, also included in the Redis crash report, would be still populated with the variables above! Note that the CPU registers RDI, RSI, RDX are used in order to pass the first three arguments to memmove. In one of the crashes we had: parent = RBP = 7f2959dffff Checking RDI, RSI, RDX we extract the memmove() arguments: memmove(00007f2959dffff4,00007f2959dffffd,0000000000000008); The memmove will go out of bound accessing up to 7f2959e00004. So I had my proof. But there was more, checking other registers I could also reconstruct the node header, in order to understand how the memmove count argument was obtained. Something was definitely wrong there. Basically I was seeing that the disassembled executable did not match the C function I was reading. How was that possible? The read over the buffer should not happen, because the state was sane. Only the count was incorrectly computed by the crashing instance. It was night, and I had worked at this damn thing for two days no stop, so I decided to create a gist and post it on Twitter, to see if somebody could explain how such C would be turned into such assembler by the compiler. I was lucky enough that my friend Fedor Indutny of Node.js fame was willing to help. He quickly realized that it was very clear why the C and the assembler could not match: what I was analyzing was not the right C code… but a newer version of the same function. Fedor had a GCC 5.4.0 at hand, the same compiler used by the user reporting the bug, so he tried to compile an older version of the code with it, realizing that now the two versions of the emitted code matched perfectly. He pinged me about it, asking if I was sure that this was a recent Redis version. I was absolutely sure, it was Redis 4.0.6. But then I started to have a few doubts, and I took a diff of rax.c between the Redis unstable branch and Redis 4.0. What happened was that I fixed this bug about ten months ago, in the course of the Streams implementation. During the cherry picking activity, where bugfixes from unstable are backported to Redis 4.0, the fix was inside a commit about streams, so I was constantly skipping it. Everything was finally clear, I had debugged for days a bug that was not there in the version that I was testing. If it was just a lame mistake from me, why I did bother to write this blog post? Because, I believe, there are lessons to be learned from all this. The first lesson is that crash reports like the ones Redis is able to emit are a key asset in system software. They allow to reconstruct the state of bugs that you are not able to replicate, but happen very rarely on the wild. While the bug was already fixed, I was able to exactly understand what was happening just looking at the bug reports, register dumps, offending address, and call stack. The second lesson is that you should learn AMD64 assembler today, at least to read comfortably code emitted by the compiler and follow what is happening, if you want to get involved in system programming. This is often the only way to understand what is happening during an heisenbug. Debuggers won’t help much. GDB was claiming that the crash was happening in the instruction parent->size-- which is, of course impossible. But GDB is not to blame, modern compilers, with optimizations turned on, generate code that can hardly be matched back to the source code. Another lesson is how powerful well done fuzz testing is. The fuzz tester was immediately able to find the bug in the broken version. Similarly the fact that no radix tree crash was never observed other than this bug fixed a long time ago, speaks for itself. The radix tree implementation is very complex, yet thanks to fuzz testing there are apparently no bugs in the implementation which is so new and so complex. I want to stress the importance of doing fuzzing not just to find crashes: that’s good for security people that want to discover zero days. Fuzzing for system software should perform random operations according to a sane operation model, and compare the outcome with a reference implementation. And finally, there is a clear lesson for me: next time I need to be more carefully when working at a feature branch and making fixes that are not specific of the feature branch. I was working under the feeling that I would merge everything back into 4.0 ASAP. Then it was not the case, and putting the radix tree update into the same commit implementing Stream things was a fatal error. Well, bonus point, having smart friends help :-) I was a bit lost at that point, and the kind help of Fyodor helped realize the last part of the puzzle and quickly move forward. Don’t be afraid to ask for help. If you get involved in system software, remember that it is very different than other fields in programming. It’s not just that you do some work and you can bring things forward. You must be prepared to spend days trying to understand why a bug happened, a bug that often you are not able to reproduce, because users deserve more than software crashing like that. Comments

• An update on Redis Streams development
  Jan 25, 2018

  I saw multiple users asking me what is happening with Streams, when they’ll be ready for production uses, and in general what’s the ETA and the plan of the feature. This post will attempt to clarify a bit what comes next. To start, in this moment Streams are my main priority: I want to finish this work that I believe is very useful in the Redis community and immediately start with the Redis Cluster improvements plans. Actually the work on Cluster has already started, with my colleague Fabio Nicotra that is porting redis-trib, the Cluster management tool, inside the old and good redis-cli. This step involves translating the code from Ruby to C. In the meantime, a few weeks ago I finished writing the Streams core, and I deleted the “streams” feature branch, merging everything into the “unstable” branch. Later I reviewed again, several times actually, the specification for consumer groups. A few weeks ago I finally was happy with the result, so I started the implementation of this specification: https://gist.github.com/antirez/68e67f3251d10f026861be2d0fe0d2f4. Be aware that command names changed quite a bit… With the API being more like this: https://gist.github.com/antirez/4e7049ce4fce4aa61bf0cfbc3672e64d. I’m halfway, after the first 500 lines of code I today was able to see clients using the XREADGROUP command to see only their local history when fetching old messages. This was a milestone in the implementation. Also XACK is now available. Basically all the data structures supporting the streams consumer groups are working, but I need to finish all the commands implementations, support the blocking operations in XREADGROUP and so forth, handle replication, RDB and AOF persistence. More or less in one month I should have everything ready. Because of consumer groups, the RDB format of the Streams is going to change a lot compared to what “unstable” is using right now. And since there is to break compatibility, we’ll also add the ability of RDB files to persist information about keys last access, so that a Redis restart (or slave loading) involving RDB will get all the info needed in order to continue doing the eviction of keys in the correct way. Right now all that information would be lost instead. Because of this radical RDB changes the plan changed a bit: Streams are going to appear in Redis 5.0 and not 4.0, but 5.0 will be released as GA in about two months: this is possible because 5.0 will be just a 4.0 plus streams, no other major features inside, if not for the RDB changes… So TLDR: I’m working to Streams almost full time, if not for some time to check PRs / Issues for critical stuff, and in two months if you upgrade to Redis 5.0 you’ll have production ready streams with consumer groups fully implemented. There is also quite of a documentation effort to perform, I’ll write both the command manuals and an extensive introduction to streams once we go RC1, which is in about one month, so that people will have documentation to support their tests during the release candidate stage. I hope this clarifies what’s happening with Streams. See you soon with Redis 5.0 :-) Comments

• Redis PSYNC2 bug post mortem
  Dec 02, 2017

  Four days ago a user posted a critical issue in the Redis Github repository. The problem was related to the new Redis 4.0 PSYNC2 replication protocol, and was very critical. PSYNC2 brings a number of good things to Redis replication, including the ability to resynchronize just exchanging the differences, and not the whole data set, after a failover, and even after a slave controlled restart. The problem was about this latter feature: with PSYNC2 the RDB file is augmented with replication information. After a slave is restarted, the replication metadata is loaded back, and the slave is able to perform a PSYNC attempt, trying to handshake with the master and receive the differences since the last disconnection. All this is good news from the point of view of Redis operations, however while PSYNC2 was pretty solid since the introduction in Redis 4.0.0 stable, the feature involving a restarting slave was definitely lacking reliability. There were two problems with this feature: the first being that it was a last-minute addition to PSYNC2, and was not part of the original design document. It was more like an obvious extension of the work we did in PSYNC2, but was not scrutinized at the same level of the rest of the specification for potential bugs and issues. The second problem was due to the fact that the feature is more complex than it looks like initially, for the fact that it’s tricky to really restore *all* the state the replication has in the slave side after a restart. Moreover, failing to restore certain bits of the state, does not result in evident bugs most of the times, so they are hard to spot via integration testings. Only once specific conditions happen the lack of some state will result in problems. For instance failing to correctly reconstruct the currently selected DB in the slave replication state, will create problems only when there are writes happening in different Redis DBs, and such bug would not store correctly the currently selected DB only under special conditions. Thanks to the help of many Redis contributors, but especially thanks to the @soloestoy Github user, a Redis developer at Alibaba, recently we worked at improving a number of PSYNC2 potential issues. All the work done would finally end inside Redis 4.0.3 in a few days, after much testing of all the patches wrote so far. However once I received issue #4483, I understood I had to rush the release of Redis 4.0.3 because this was far more serious than the other Redis 4 PSYNC2 issues that we had discovered up to that point. The bug described in issue #4483 was fairly simple, yet very problematic. After a slave restart and a resulting reloading of the replication state from the RDB, the master replication backlog could contain Lua scripts executions in the form of EVALSHA commands. However the slave Lua scripting engine is flushed of all the scripts after the restart, so is not able to process such commands. This results in the slave not processing writes originated by Lua scripts, unless such scripts were using “commands replication”, which is not the default. The default is to replicate the script itself. I went a bit in panic mode… and wrote a few alternative patches that I submitted to the issue. At the end the one that did not require RDB incompatibilities with older versions was picked, so I went ahead and released Redis 4.0.3 ASAP. However I did a mistake… It’s a few weeks that I work from an office instead of working from home. Normally I do not work at night, but for 4.0.3, when I returned back home, I opened my home laptop and merged the patch into the 4.0 branch, and did some testing activity. The next day when I returned to the office, I continued working from another computer, but I did not realize that I was missing one commit that I merged at home, without pushing it to the repository. So I basically released a 4.0.3 that had all the PSYNC2 fixes minus the most important one for the replication bug. I released ASAP a new patch level version, Redis 4.0.4, including the replication fix. This was already not great: upgrading Redis is a scheduled activity, and nobody wants to upgrade two times because I make errors in preparing the release… But the worst was yet to happen. The fix that I added in 4.0.4, the one about scripts replication in restarting slaves performing PSYNC2, had an error that passed totally unnoticed through all the replication integration tests: the fix involved storing the Lua scripts in the slave memory directly into the RDB, in order to reload them later. However it was not considered that the function loading the scripts, would assert if the script was already in memory, so when a slave received a full synchronization from a master, it started to load the RDB file, crashing immediately because of some duplicated script that was already in memory. A user reported it ASAP via Twitter, and I fixed the problem in less than 45 minutes, from the reporting moment to the 4.0.5 availability, but this does not mitigate all the potential problems that this sequence of failures at delivering a fix caused. The above problems were caused by a number of reasons: 1. Redis 4.0 was delivered as stable too early, considering the complexity and the amount of code change in PSYNC2. The next time, even at the cost of delaying releases, I’ll wait more and will take the new major versions of Redis more time in the last release candidate, so that we can find such bugs before shipping a GA version. 2. I rushed too much trying to deliver a fix for issue #4483. Even if the bug was critical, it was better to spend some time in order to check what was happening, and the potential problems caused by the fix itself. Moreover I was so much in a hurry that I did not check the set of commits composing 4.0.3 with enough care, causing the lack of one of the fixes, and the need for a new release. 3. While Redis 4.0 has very strict PSYNC2 unit and integration tests, including tests that simulate continuous failovers checking data consistency at every step, because of this bug I discovered that the tests never try to mix PSYNC2 with replication of Lua scripts. This must be improved. Also the PSYNC2 replication after slave RDB restart must be enhanced as well. I’ll start with the improvements at “3”, and in the future I’ll consider the lessons at “1” and “2” before every new release. My sincere apologize to everybody that had to deal with my errors in this instance, and a big thank you to @soloestoy for an incredible amount of help and support. Comments

• Streams: a new general purpose data structure in Redis.
  Oct 02, 2017

  Until a few months ago, for me streams were no more than an interesting and relatively straightforward concept in the context of messaging. After Kafka popularized the concept, I mostly investigated their usefulness in the case of Disque, a message queue that is now headed to be translated into a Redis 4.2 module. Later I decided that Disque was all about AP messaging, which is, fault tolerance and guarantees of delivery without much efforts from the client, so I decided that the concept of streams was not a good match in that case. However, at the same time, there was a problem in Redis, that was not taking me relaxed about the data structures exported by default. There is some kind of gap between Redis lists, sorted sets, and Pub/Sub capabilities. You can kindly use all these tools in order to model a sequence of messages or events, but with different tradeoffs. Sorted sets are memory hungry, can’t model naturally the same message delivered again and again, clients can’t block for new messages. Because a sorted set is not a sequential data structure, it’s a set where elements can be moved around changing their scores: no wonder if it was not a good match for things like time series. Lists have different problems creating similar applicability issues in certain use cases: you cannot explore what is in the middle of a list because the access time in that case is linear. Moreover no fan-out is possible, blocking operations on list serve a single element to a single client. Nor there was a fixed element identifier in lists, in order to say: given me things starting from that element. For one-to-many workloads there is Pub/Sub, which is great in many cases, but for certain things you do not want fire-and-forget: to retain a history is important, not just to refetch messages after a disconnection, also because certain list of messages, like time series, are very important to explore with range queries: what were my temperature readings in this 10 seconds range? The way I tried to address the above problems, was planning a generalization of sorted sets and lists into a unique more flexible data structure, however my design attempts ended almost always in making the resulting data structure ways more artificial than the current ones. One good thing about Redis is that the data structures exported resemble more the natural computer science data structures, than, “this API that Salvatore invented”. So in the end, I stopped my attempts, and said, ok that’s what we can provide so far, maybe I’ll add some history to Pub/Sub, or some more flexibility to lists access patterns in the future. However every time an user approached me during a conference saying “how would you model time series in Redis?” or similar related questions, my face turned green. Genesis ======= After the introduction of modules in Redis 4.0, users started to see how to fix this problem themselves. One of them, Timothy Downs, wrote me the following over IRC: the module I'm planning on doing is to add a transaction log style data type - meaning that a very large number of subscribers can do something like pub sub without a lot of redis memory growth subscribers keeping their position in a message queue rather than having redis maintain where each consumer is up to and duplicating messages per subscriber This captured my imagination. I thought about it a few days, and realized that this could be the moment when we could solve all the above problems at once. What I needed was to re-imagine the concept of “log”. It is a basic programming element, everybody is used to it, because it’s just as simple as opening a file in append mode and writing data to it in some format. However Redis data structures must be abstract. They are in memory, and we use RAM not just because we are lazy, but because using a few pointers, we can conceptualize data structures and make them abstract, to allow them to break free from the obvious limits. For instance normally a log has several problems: the offset is not logical, but is an actual bytes offset, what if we want logical offsets that are related to the time an entry was inserted? We have range queries for free. Similarly, a log is often hard to garbage collect: how to remove old elements in an append only data structure? Well, in our idealized log, we just say we want at max this number of entries, and the old ones will go away, and so forth. While I was trying to write a specification starting from the seed idea of Timothy, I was working to a radix tree implementation that I was using for Redis Cluster, to optimize certain parts of its internals. This provided the ground in order to implement a very space efficient log, that was still accessible in logarithmic time to get ranges. At the same time I started reading about Kafka streams to get other interesting ideas that could fit well into my design, and this resulted into getting the concept of Kafka consumer groups, and idealizing it again for Redis and the in-memory use case. However the specification remained just a specification for months, at the point that after some time I rewrote it almost from scratch in order to upgrade it with many hints that I accumulated talking with people about this upcoming addition to Redis. I wanted Redis streams to be a very good use case for time series especially, not just for other kind of events and messaging applications. Let’s write some code ===================== Back from Redis Conf, during the summertime, I was implementing a library called “listpack”. This library is just the successor of ziplist.c, that is, a data structure that can represent a list of string elements inside a single allocation. It’s just a very specialized serialization format, with the peculiarity of being parsable also in reverse order, from right to left: something needed in order to substitute ziplists in all the use cases. Mixing radix trees + listpacks, it is possible to easily build a log that is at the same time very space efficient, and indexed, that means, allowing for random access by IDs and time. Once this was ready, I started to write the code in order to implement the stream data structure. I’m still finishing the implementation, however at this point, inside the Redis “streams” branch at Github, there is enough to start playing and having fun. I don’t claim that the API is 100% final, but there are two interesting facts: one is that at this point, only the consumer groups are missing, plus a number of less important commands to manipulate the stream, but all the big things are implemented already. The second is the decision to backport all the stream work back into the 4.0 branch in about two months, once everything looks stable. It means that Redis users will not have to wait for Redis 4.2 in order to use streams, they will be available ASAP for production usage. This is possible because being a new data structure, almost all the code changes are self-contained into the new code. With the exception of the blocking list operations: the code was refactored so that we share the same code for streams and lists blocking operations, with a great simplification of the Redis internals. Tutorial: welcome to Redis Streams ================================== In some way, you can think at streams as a supercharged version of Redis lists. Streams elements are not just a single string, they are more objects composed of fields and values. Range queries are possible and fast. Each entry in a stream has an ID, which is a logical offset. Different clients can blocking-wait for elements with IDs greater than a specified one. A fundamental command of Redis streams is XADD. Yes, all the Redis stream commands are prefixed by an “X”. > XADD mystream * sensor-id 1234 temperature 10.5 1506871964177.0 The XADD command will append the specified entry as a new element to the specified stream “mystream”. The entry, in the example above, has two fields: sensor-id and temperature, however each entry in the same stream can have different fields. Using the same field names will just lead to better memory usage. An interesting thing is also that the fields order is guaranteed to be retained. XADD returns the ID of the just inserted entry, because with the asterisk in the third argument, we asked the command to auto-generate the ID. This is almost always what you want, but it is possible also to force a specific ID, for instance in order to replicate the command to slaves and AOF files. The ID is composed of two parts: a millisecond time and a sequence number. 1506871964177 is the millisecond time, and is just a Unix time with millisecond resolution. The number after the dot, 0, is the sequence number, and is used in order to distinguish entries added in the same millisecond. Both numbers are 64 bit unsigned integers. This means that we can add all the entries we want in a stream, even in the same millisecond. The millisecond part of the ID is obtained using the maximum between the current local time of the Redis server generating the ID, and the last entry inside the stream. So even if, for instance, the computer clock jumps backward, the IDs will continue to be incremental. In some way you can think stream entry IDs as whole 128 bit numbers. However the fact that they have a correlation with the local time of the instance where they are added, means that we have millisecond precision range queries for free. As you can guess, adding two entries in a very fast way, will result in only the sequence number to be incremented. We can simulate the “fast insertion” simply with a MULTI/EXEC block: > MULTI OK > XADD mystream * foo 10 QUEUED > XADD mystream * bar 20 QUEUED > EXEC 1) 1506872463535.0 2) 1506872463535.1 The above example also shows how we can use different fields for different entries without having to specifying any schema initially. What happens however is that every first message of every block (that usually contains something in the range of 50-150 messages) is used as reference, and successive entries having the same fields are compressed with a single flag saying “same fields of the first entry in this block”. So indeed using the same fields for successive messages saves a lot of memory, even when the set of fields slowly change over time. In order to retrieve data from the stream there are two ways: range queries, that are implemented by the XRANGE command, and streaming, implemented by the XREAD command. XRANGE just fetches a range of items from start to stop, inclusive. So for instance I can fetch a single item, if I know its ID, with: > XRANGE mystream 1506871964177.0 1506871964177.0 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" However you can use the special start symbol of “-“ and the special stop symbol of “+” to signify the minimum and maximum ID possible. It’s also possible to use the COUNT option in order to limit the amount of entries returned. A more complex XRANGE example is the following: > XRANGE mystream - + COUNT 2 1) 1) 1506871964177.0 2) 1) "sensor-id" 2) "1234" 3) "temperature" 4) "10.5" 2) 1) 1506872463535.0 2) 1) "foo" 2) "10" Here we are reasoning in terms of ranges of IDs, however you can use XRANGE in order to get a specific range of elements in a given time range, because you can omit the “sequence” part of the IDs. So what you can do is to just specify times in milliseconds. The following means: “Give me 10 entries starting from the Unix time 1506872463”: 127.0.0.1:6379> XRANGE mystream 1506872463000 + COUNT 10 1) 1) 1506872463535.0 2) 1) "foo" 2) "10" 2) 1) 1506872463535.1 2) 1) "bar" 2) "20" A final important thing to note about XRANGE is that, given that we receive the IDs in the reply, and the immediately successive ID is trivially obtained just incrementing the sequence part of the ID, it is possible to use XRANGE to incrementally iterate the whole stream, receiving for every call the specified number of elements. After the *SCAN family of commands in Redis, that allowed iteration of Redis data structures *despite* the fact they were not designed for being iterated, I avoided to make the same error again. Streaming with XREAD: blocking for new data =========================================== XRANGE is perfect when we want to access our stream to get ranges by ID or time, or single elements by ID. However in the case of streams that different clients must consume as data arrives, this is not good enough and would require some form of pooling (that could be a good idea for *certain* applications that just connect from time to time to get data). The XREAD command is designed in order to read, at the same time, from multiple streams just specifying the ID of the last entry in the stream we got. Moreover we can request to block if no data is available, to be unblocked when data arrives. Similarly to what happens with blocking list operations, but here data is not consumed from the stream, and multiple clients can access the same data at the same time. This is a canonical example of XREAD call: > XREAD BLOCK 5000 STREAMS mystream otherstream $ $ And it means: get data from “mystream” and “otherstream”. If no data is available, block the client, with a timeout of 5000 milliseconds. After the STREAMS option we specify the keys we want to listen for, and the last ID we have. However a special ID of “$” means: assume I’ve all the elements that there are in the stream right now, so give me just starting from the next element arriving. If, from another client, I send the commnad: > XADD otherstream * message “Hi There” This is what happens on the XREAD side: 1) 1) "otherstream" 2) 1) 1) 1506935385635.0 2) 1) "message" 2) "Hi There" We get the key that received data, together with the data received. In the next call, we’ll likely use the ID of the last message received: > XREAD BLOCK 5000 STREAMS mystream otherstream $ 1506935385635.0 And so forth. However note that with this usage pattern, it is possible that the client will connect again after a very big delay (because it took time to process messages, or for any other reason). In such a case, in the meantime, a lot of messages could pile up, so it is wise to always use the COUNT option with XREAD, in order to make sure the client will not be flooded with messages and the server will not have to lose too much time just serving tons of messages to a single client. Capped streams ============== So far so good… however streams at some point have to remove old messages. Fortunately this is possible with the MAXLEN option of the XADD command: > XADD mystream MAXLEN 1000000 * field1 value1 field2 value2 This basically means, if the stream, after adding the new element is found to have more than 1 million messages, remove old messages so that the length returns back to 1 million elements. It’s just like using RPUSH + LTRIM with lists, but this time we have a built-in mechanism to do so. However note that the above means that every time we add a new message, we have also to incur in the work needed in order to remove a message from the other side of the stream. This takes some CPU, so it is possible to use the “~” symbol before the count in MAXLEN, in order to specify that we are not really demanding *exactly* 1 million messages, but if there are a few more it’s not a big problem: > XADD mystream MAXLEN ~ 1000000 * foo bar This way XADD will remove messages only when it can remove a whole node. This will make having the capped stream almost for free compared to vanilla XADD. Consumer groups (work in progress) ================================== This is the first of the features that is not already implemented in Redis, but is a work in progress. It is also the idea more clearly inspired by Kafka, even if implemented here in a pretty different way. The gist is that with XREAD, clients can also add a “GROUP ” option. Automatically all the clients in the same group will get *different* messages. Of course there could be multiple groups reading from the same stream, in such cases all groups will receive duplicates of the same messages arriving in the stream, but within each group, messages will not be repeated. An extension to groups is that it will be possible to specify a “RETRY ” option when groups are specified: in this case, if messages are not acknowledged for processing with XACK, they will be delivered again after the specified amount of milliseconds. This provides some best effort reliability to the delivering of the messages, in case the client has no private means to mark messages as processed. This part is a work in progress as well. Memory usage and saving loading times ===================================== Because of the design used to model Redis streams, the memory usage is remarkably low. It depends on the number of fields, values, and their lengths, but for simple messages we are at a few millions of messages for every 100 MB of used memory. Moreover, the format is conceived to need very minimal serialization: the listpack blocks that are stored as radix tree nodes, have the same representation on disk and in memory, so they are trivially stored and read. For instance Redis can read 5 million entries from the RDB file in 0.3 seconds. This makes replication and persistence of streams very efficient. It is planned to also allow deletion of items in the middle. This is only partially implemented, but the strategy is to mark entries as deleted in the entry flag, and when a given ratio between entries and deleted entires is reached, the block is rewritten to collect the garbage, and if needed it is glued to another adjacent block in order to avoid fragmentation. Conclusions end ETA =================== Redis streams will be part of Redis stable in the 4.0 series before the end of the year. I think that this general purpose data structure is going to put a huge patch in order for Redis to cover a lot of use cases that were hard to cover: that means that you had to be creative in order to abuse the current data structures to fix certain problems. One very important use case is time series, but my feeling is that also streaming of messages for other use cases via TREAD is going to be very interesting both as replacement for Pub/Sub applications that need more reliability than fire-and-forget, and for completely new use cases. For now, if you want to start to evaluate the new capabilities in the context of your problems, just fetch the “streams” branch at Github and start playing. After all bug reports are welcome :-) If you like videos, a real-time session showing streams is here: https://www.youtube.com/watch?v=ELDzy9lCFHQ Comments

• Doing the FizzleFade effect using a Feistel network
  Aug 29, 2017

  Today I read an interesting article about how the Wolfenstein 3D game implemented a fade effect using a Linear Feedback Shift Register. Every pixel of the screen is set red in a pseudo random way, till all the screen turns red (or other colors depending on the event happening in the game). The blog post describing the implementation is here and is a nice read: http://fabiensanglard.net/fizzlefade/index.php You may wonder why the original code used a LFSR or why I'm proposing a different approach, instead of the vanilla setPixel(rand(),rand()): doing this with a pseudo random generator, as noted in the blog post, is slow, but is also visually very unpleasant, since the more red pixels you have on the screen already, the less likely is that you hit a new yet-not-red pixel, so the final pixels take forever to turn red (I *bet* that many readers of this blog post tried it in the old times of the Spectum, C64, or later with QBASIC or GWBasic). In the final part of the blog post the author writes: "Because the effect works by plotting pixels individually, it was hard to replicate when developers tried to port the game to hardware accelerated GPU. None of the ports managed to replicate the fizzlefade except Wolf4SDL, which found a LFSR taps configuration to reach resolution higher than 320x200.” While not rocket science, it was possibly hard for other resolutions to find a suitable LFSR. However regardless of the real complexity of finding an appropriate LFSR for other resolutions, the authors of the port could use another technique, called a Feistel Network, to get exactly the same result in a trivial way. What is a Feistel Network? === It’s a building block typically used in cryptography: it creates a transformation between a sequence of bits and another sequence of bits, so that the transformation is always invertible, even if you use all the kind of non linear transformations inside the Feistel network. In practical terms the Feistel network can, for example, translate a 32 bit number A into another 32 bit number B, according to some function F(), so that you can always go from B to A later. Because the function is invertible, it implies that for every input value the Feistel network generates *a different* output value. This is a simple Feistel network in pseudo code: Split the input into L and R halves (Example: L = INPUT & 0xFF, R = INPUT >> 8) REPEAT for N rounds: next_L = R R = L XOR F(R) L = next_L END RETURN the value composing L and R again into a single sequence of bits: R So we basically split a (for example) 16 bit integer into two 8 bit integers L and R, perform some transformation for N rounds, and recompose them back into a 16 bit integer, which is our output. But how is this useful for our problem of implementing FizzleFade? Well you can imagine your 2D screen like a linear array of pixels. If the resolution is 320x200 like in the original game you have from pixel 0 to pixel 63999. So for every integer from 0 to 63999 we can generate a random looking pixel position just by counting and setting the pixel in the position returned by the Feistel network. The problem is that the Feistel network works in bits, so we can’t have exactly from 0 to 63999, we have to pick a power of two which is large enough. The nearest is 16 in this case: with 16 bits we have 65536 integer-to-integer transformations, a few cycles will not be used to set an actual pixel but is not a big waste. So, this is how our Feistel network looks like, in Javascript: /* Transforms the 16 bit input into another seemingly psenduo random number * in the same range. Every input 16 bit input will generate a different * 16 bit output. This is called a Feistel network. */ function feistelNet(input) { var l = input & 0xff; var r = input >> 8; for (var i = 0; i var nl = r; var F = (((r * 11) + (r >> 5) + 7 * 127) ^ r) & 0xff; r = l ^ F; l = nl; } return ((r} The non linear transformation “F” I’m using is just a few random multiplications and shifts, picked mostly at random. I’m using 8 rounds even if it is probably not needed with a better F function, but I want the effect to look random (coincidentally drawing random pixels is a decent way to visually spot trivial bad distribution properties). Implementing this using a Javascript canvas we need a few more functions, to get a 2D context and set a pixel. The final code is in this Gist: https://gist.github.com/antirez/6d58860b221a6ae5622ced8ccdddbe47 You can see the result here: http://antirez.com/misc/fizzlefade.html The original problem to explore in this article was to find a way to implement the effect in different resolutions, so even if it is a trivial extension of the 320x200 case, just to make an example, imagine you want to implement the same with 1024*768. There are 786432 pixels, so 2^20 will fit quite well with 1048576 possible integers. We’ll have to modify the Feistel network to have 20 bits input/output by using 10 bits L and R variables, otherwise everything is pretty much the same, but remember to also change the stop condition (that checks the number of frames). Actually Feistel networks one-to-one pseudo random mapping properties are very useful in other contexts as well. For instance I used it in my radix tree implementation tests (https://github.com/antirez/rax in case you are curious). A good tool to have in a programmer mental box. Comments

Alex Edwards

• Surprises and gotchas when working with JSON
  Sep 24, 2020

  This is a list of things about Go's encoding/json package which, over the years, have either confused or surprised me when I first encountered them. Many of these things are mentioned in the official package documentation if you read it carefully enough, so in theory they shouldn't come as a surprise. But a few of them aren't mentioned in the documentation at all — or at least, they aren't pointed out explicitly — and are worth being aware of! Map entries are sorted alphabetically Byte slices are encoded as base-64 strings Nil and empty slices are encoded differently Integer, time.Time and net.IP values can be used as map keys Angle brackets and ampersands in strings are escaped Trailing zeroes are removed from floats Using omitempty on an zero-valued struct doesn't work Using omitempty on a zero-value time.Time doesn't work There is a 'string' struct tag Non-ASCII punctuation characters aren't supported in struct tags Decoding a JSON number into an interface{} yields a float64 Don't use More() to check if there are remaining JSON objects in a stream String values returned by custom MarshalJSON() methods must be quoted Map entries are sorted alphabetically When encoding a Go map to JSON, the entries will be sorted alphabetically based on the map key. For example, the following map: m := map[string]int{ "z": 123, "0": 123, "a": 123, "_": 123, } Will be encoded to the JSON: {"0":123,"_":123,"a":123,"z":123} Byte slices are encoded as base-64 strings Any []byte slices will be converted to a base64-encoded string when encoding them to JSON. The base64 string uses padding and the standard encoding characters, as defined in RFC 4648. For example, the following map: m := map[string][]byte{ "foo": []byte("bar baz"), } Will be encoded to the JSON: {"foo":"YmFyIGJheg=="} Nil and empty slices are encoded differently Nil slices in Go will be encoded to the null JSON value. In contrast, an empty (but not nil) slice will be encoded as an empty JSON array. For example: var nilSlice []string emptySlice := []string{} m := map[string][]string{ "nilSlice": nilSlice, "emptySlice": emptySlice, } Will be encoded to the JSON: {"emptySlice":[],"nilSlice":null} Integer, time.Time and net.IP values can be used as map keys It's possible to encode a map which has integer values as the map keys. These integers will be automatically converted to strings in the resulting JSON (because the keys in a JSON object must always be strings). For example: m := map[int]string{ 123: "foo", 456_000: "bar", } Will be encoded to the JSON: {"123":"foo","456000":"bar"} In addition, Go allows you to encode maps with keys that implement the encoding.TextMarshaler interface. This means that you can also use time.Time and net.IP values as map keys out-of-the-box. For example: t1 := time.Now() t2 := t1.Add(24 * time.Hour) m := map[time.Time]string{ t1: "foo", t2: "bar", } Will be encoded to the JSON: {"2009-11-10T23:00:00Z":"foo","2009-11-11T23:00:00Z":"bar"} Note that trying to encode a map with any other type of key will result in a json.UnsupportedTypeError error. Angle brackets and ampersands in strings are escaped If a string contains angle brackets<> these will be escaped to \u003c and \u003e in the JSON output. Likewise the & character will be escaped to \u0026. This is to prevent some web browsers from accidentally interpreting the JSON as HTML. For example: s := []string{ "", "bar & baz", } Will be encoded to the JSON: ["\u003cfoo\u003e","bar \u0026 baz"] If you need to prevent these characters being escaped, you should use a json.Encoder instance and call SetEscapeHTML(false). An example is here. Trailing zeroes are removed from floats When encoding a floating-point number with a fractional part that ends in zero(es), any trailing zeroes will not appear in the JSON. For example: s := []float64{ 123.0, 456.100, 789.990, } Will be encoded to the JSON: [123,456.1,789.99] Using omitempty on an zero-valued struct doesn't work The omitempty directive never considers a struct type to be empty — even if all the struct fields have their zero value, and you use omitempty on those fields too. It will always appear as an object in the encoded JSON. For example: s := struct { Foo struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":{}} There’s a long-standing proposal which discusses changing this behavior, but the Go 1 compatibility promise means that it's unlikely to happen any time soon. Instead, you can get around this by making the field a pointer to a struct, which works because omitempty considers nil pointers to be empty. For example: s := struct { Foo *struct { Bar string `json:",omitempty"` } `json:",omitempty"` }{} Using omitempty on a zero-value time.Time doesn't work Using omitempty on a zero-value time.Time field won't hide it in the encoded JSON. This is because the time.Time type is a struct behind the scenes and, as mentioned above, omitempty never considers a struct type to be empty. Instead, the string "0001-01-01T00:00:00Z" will appear in the JSON (which is the value returned by calling the MarshalJSON() method on an zero-value time.Time. For example: s := struct { Foo time.Time `json:",omitempty"` }{} Will be encoded to the JSON: {"Foo":"0001-01-01T00:00:00Z"} There is a 'string' struct tag Go provides a string struct tag directive which forces the data in an individual field to be encoded as a string in the resulting JSON. For example, if you want to force an integer to be represented as a string instead of an JSON number you can use the string directive like so: s := struct { Foo int `json:",string"` }{ Foo: 123, } And this will be encoded to the JSON: {"Foo":"123"} Note that the string struct tag directive will only work on fields which contain float, integer or bool types. For any other type it will have no effect. Non-ASCII punctuation characters aren't supported in struct tags When using struct tags to change key names in JSON, any tags containing non-ASCII punctuation characters will be ignored. Notably this means that you can't use en or em dashes, or most currency signs, in struct tags. For example: s := struct { CostUSD string `json:"cost $"` // OK CostEUR string `json:"cost €"` // Contains the non-ASCII punctuation character €. Will be ignored. }{ CostUSD: "100.00", CostEUR: "100.00", } Will be encoded to the following JSON (notice that the struct tag renaming the CostEUR field has been ignored): {"cost $":"100.00","CostEUR":"100.00"} Likewise, any struct tags containing non-ASCII punctuation characters will be ignored when decoding values from a JSON object into a struct, and the struct field will be left with its zero value. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{} err := json.Unmarshal(js, &s) if err != nil { log.Fatal(err) } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:} This can be annoying in situations where you need to decode a JSON object that has keys containing non-ASCII characters, and you can't change the JSON. To work around this limitation, you can decode to a map as an intermediary step, and then copy the data from the map to the struct. For example the following code: js := []byte(`{"cost $":"100.00","cost €":"100.00"}`) var aux map[string]string err := json.Unmarshal([]byte(js), &aux) if err != nil { log.Fatal(err) } s := struct { CostUSD string `json:"cost $"` CostEUR string `json:"cost €"` }{ CostUSD: aux["cost $"], CostEUR: aux["cost €"], } fmt.Printf("%+v", s) Will print out: {CostUSD:100.00 CostEUR:100.00} Decoding a JSON number into an interface{} yields a float64 When decoding a JSON number into an interface{}, the value will have the underlying type float64 — even if it is an integer in the original JSON. If you want to get the value as an integer (instead of a float64) the most robust approach is to decode the JSON using a json.Decoder instance with the UseNumber() method set on it. This will decode all JSON numbers to the underlying type json.Number instead of float64, and you can then access the number as an integer using its Int64() method. For example: js := `{"foo": 123, "bar": true}` var m map[string]interface{} dec := json.NewDecoder(strings.NewReader(js)) dec.UseNumber() err := dec.Decode(&m) if err != nil { log.Fatal(err) } i, err := m["foo"].(json.Number).Int64() if err != nil { log.Fatal(err) } fmt.Printf("foo: %d", i) Will print: foo: 123 Don't use More() to check if there are remaining JSON objects in a stream When processing a stream of JSON objects with json.Decoder, don't use the More() method to check if there is a remaining object in the stream. Depsite its name, More() is not designed for this purpose†, and trying to use it in this way may cause some subtle problems. †The More() method is intended to be used in conjunction with Token(), and exists specifically to check if there is another element in the array or object currently being parsed. For example, if you use it when decoding an invalid JSON stream like {"name": "alice"}{"name": "bob"}] (notice the additional square bracket at the end) it won't result in an error (when it should!). Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { log.Fatal(err) } fmt.Printf("%v\n", user) // Don't do this! if !dec.More() { break } } This code will run without error and output: map[name:alice] map[name:bob] The correct technique to see if a stream contains another JSON object is to check for an io.EOF error, which will be returned when there are no more objects to process in the stream. Like so: js := `{"name": "alice"}{"name": "bob"}]` dec := json.NewDecoder(strings.NewReader(js)) for { var user map[string]string err := dec.Decode(&user) if err != nil { if errors.Is(err, io.EOF) { break } log.Fatal(err) } fmt.Printf("%v\n", user) } Running this will correctly result in an error, as we would expect given the invalid input: map[name:alice] map[name:bob] 2009/11/10 23:00:00 invalid character ']' looking for beginning of value String values returned by custom MarshalJSON() methods must be quoted If you are creating a custom MarshalJSON() method which returns a string value, you must wrap the string in double quotes before returning it, otherwise it won't be interpreted as a JSON string and will result in a runtime error. For example: type Age int func (age Age) MarshalJSON() ([]byte, error) { encodedAge := fmt.Sprintf("%d years", age) encodedAge = strconv.Quote(encodedAge) // Wrap the string in quotes before returning. return []byte(encodedAge), nil } func main() { users := map[string]Age{ "alice": 21, "bob": 84, } js, err := json.Marshal(users) if err != nil { log.Fatal(err) } fmt.Printf("%s", js) } Will result in the following JSON being printed: {"alice":"21 years","bob":"84 years"} If, in the code above, you didn't quote the return value from MarshalJSON() you will get the error: 2009/11/10 23:00:00 json: error calling MarshalJSON for type main.Age: invalid character 'y' after top-level value

• How to manage database timeouts and cancellations in Go
  Apr 20, 2020

  One of the great features of Go's database/sql package is that it's possible to cancel database queries while they are still running via a context.Context instance. On the face of it, usage of this functionality is quite straightforward (here's a basic example). But once you start digging into the details there's a lot a nuance and quite a few gotchas... especially if you are using this functionality in the context of a web application or API. So in this post I want to explain how to cancel database queries in a web application, what behavioral quirks and edge cases it is important to be aware of, and try to provide answers to the questions that you might have when working through all this. But first off, why would you want to cancel a database query? Two scenarios spring to mind: When a query is taking a lot longer to complete than expected. If this happens, it suggests a problem — either with that particular query or your database or application more generally. In this scenario, you would probably want to cancel the query after a set period of time (so that resources are freed-up and the database connection is returned to the sql.DB connection pool for reuse), log an error for further investigation, and return a 500 Internal Server Error response to the client. When a client goes away unexpectedly before the query completes. This could happen for a number of reasons, such as a user closing a browser tab or terminating a process. In this scenario, nothing has really gone 'wrong', but there is no client left to return a response to so you may as well cancel the query and free-up the resources. Mimicking a long-running query Let's start with the first scenario. To demonstrate this, I'll make a very basic web application with a handler that executes a SELECT pg_sleep(10) SQL query against a PostgreSQL database using the pq driver. The pg_sleep(10) function will make the query sleep for 10 seconds before returning, essentially mimicking a slow-running query. package main import ( "database/sql" "fmt" "log" "net/http" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery() error { _, err := db.Exec("SELECT pg_sleep(10)") return err } func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } if err = db.Ping(); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery() if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } func serverError(w http.ResponseWriter, err error) { log.Printf("ERROR: %s", err.Error()) http.Error(w, "Sorry, something went wrong", http.StatusInternalServerError) } If you were to run this code, then make a GET / request to the application you should find that the request hangs for 10 seconds before you finally get an "OK" response. Like so: $ curl -i localhost:5000/ HTTP/1.1 200 OK Date: Fri, 17 Apr 2020 07:46:40 GMT Content-Length: 3 Content-Type: text/plain; charset=utf-8 OK Note: The structure of the application code above is deliberately over-simplified. In a real project I would recommend using dependency injection to make the sql.DB connection pool and logger available to your handlers, instead of using global variables. Adding a context timeout OK, now that we've got some code that mimics a long-running query, let's enforce a timeout on the query so it is automatically canceled if it doesn't complete within 5 seconds. To do this we need to: Use the context.WithTimeout() function to create a context.Context instance with a 5-second timeout duration. Execute the SQL query using the ExecContext() method, passing the context.Context instance as a parameter. I'll demonstrate: package main import ( "context" // New import "database/sql" "fmt" "log" "net/http" "time" // New import _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { // Create a new child context with a 5-second timeout, using the // provided ctx parameter as the parent. ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() // Pass the child context (the one with the timeout) as the first // parameter to ExecContext(). _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { // Pass the request context to slowQuery(), so it can be used as the // parent context. err := slowQuery(r.Context()) if err != nil { serverError(w, err) return } fmt.Fprintln(w, "OK") } ... There are a few things about this that I'd like to emphasize and explain: Note that we pass r.Context() (the request context) to slowQuery() to use as the parent context. As we'll see in the next section, this is important because it means that any cancellation signal on the request context will be able to 'bubble down' to the context that we use in ExecContext(). The defer cancel() line is important because it ensures that the resources associated with our child context (the one with the timeout) will be released before the slowQuery() function returns. If we don't call cancel() it may cause a memory leak: the resources won't be released until either the parent r.Context() is canceled or the 5-second timeout is hit (whichever happens first). The timeout countdown begins from the moment that the child context is created using context.WithTimeout(). If you want more control over this you could use the alternative context.WithDeadline() function, which allows you to set an explicit time.Time value for when the context should timeout instead. OK, let's try this out. If you run the application again and make a GET / request, after a 5-second delay you should get a response like this: $ curl -i localhost:5000/ HTTP/1.1 500 Internal Server Error Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Fri, 17 Apr 2020 08:21:14 GMT Content-Length: 28 Sorry, something went wrong And if you go back to the terminal window running the application you should see a log message similar to this: $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request That log message might seem a bit odd... until you realize that the error message is actually coming from PostgreSQL. In that light it makes sense: our web application is the user and we're canceling the query after 5 seconds. So this is actually really good; things are working as we want. Specifically, after 5 seconds the context timeout is reached and the pq driver sends a cancellation signal to PostgreSQL†. PostgreSQL then terminates the running query (thereby freeing-up resources). The client is sent a 500 Internal Server Error response, and the error message is logged so we know that something has gone wrong. † More precisely, our child context (the one with the 5-second timeout) has a Done channel, and when the timeout is reached it will close the Done channel. While the SQL query is running, our database driver pq is also running a background goroutine which listens on this Done channel. If the channel is closed, then it sends a cancellation signal to PostgreSQL. PostgreSQL terminates the query, and then sends the error message that we see above as a response to the original pq goroutine. That error message is then returned to our slowQuery() function. Dealing with closed connections OK, let's try one more thing. Let's use curl to make a GET / request and then very quickly (within 5 seconds) press Ctrl+C to cancel the request. If you look at the logs for the application again, you should see another log line with exactly the same error message that we saw before. $ go run . 2020/04/17 10:21:07 Listening... 2020/04/17 10:21:14 ERROR: pq: canceling statement due to user request 2020/04/17 10:41:18 ERROR: pq: canceling statement due to user request So what's happening here? In this case, the request context (which we use as the parent in our code above) is canceled because the client closed the connection. From the net/http docs: For incoming server requests, the [request] context is canceled when the client's connection closes, the request is canceled (with HTTP/2), or when the ServeHTTP method returns. This cancellation signal bubbles down to our child context, it's Done channel is closed, and the pq driver terminates the running query in exactly the same way as before. With that in mind, it's not surprising that we see the same error message... From a PostgreSQL point of view exactly the same thing is happening as when the timeout was reached. But from the perspective of our web application the scenario is very different. A client connection being closed can happen for many different, innocuous, reasons. It's not really an error from our application's point of view, although it is probably sensible to log it as a warning (if we start to see elevated rates, it could be a sign that something is wrong). Fortunately, it's possible to tell these two scenarios apart by calling the ctx.Err() method on our child context. If the context was canceled (due to a client closing the connection), then ctx.Err() will return context.Canceled. If the timeout was reached, then it will return context.DeadlineExceeded. If both the deadline is reached and the context is canceled, then ctx.Err() will surface whichever happened first. There's another important thing to point out here: it's possible that a timeout/cancellation will happen before the PostgreSQL query even starts. For example you might have set MaxOpenConns() on your sql.DB connection pool, and if that open connection limit is reached and all connections are in-use, then the query will be 'queued' by sql.DB until a connection becomes available. In this scenario — or any other which causes a delay — it's quite possible that the timeout/cancellation will occur before a free database connection even becomes available. In this case ExecContext() will directly return the ctx.Err() value as the error response (instead of the "pq: canceling statement due to user request" error that we see above). If you're using the QueryContext() method then it's also possible that the timeout/cancellation will occur when processing the data with Scan(). If this happens, then Scan() will directly return the ctx.Err() value as an error. As far as I can see this behavior isn't mentioned in the database/sql docs, but I can confirm that this is the case with Go 1.14 and the comments on issue #28842 suggest that it is intentional. Putting all that together, a sensible approach is to check for the error "pq: canceling statement due to user request" and then wrap this with the error from ctx.Err() before returning from our slowQuery() function. Then in our handler, we can use the errors.Is() function to check if the error from slowQuery() is equal to (or wraps) context.Canceled and manage it accordingly. Like so: package main import ( "context" "database/sql" "errors" // New import "fmt" "log" "net/http" "time" _ "github.com/lib/pq" ) var db *sql.DB func slowQuery(ctx context.Context) error { ctx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() _, err := db.ExecContext(ctx, "SELECT pg_sleep(10)") // If we get a "pq: canceling statement..." error wrap it with the // context error before returning. if err != nil && err.Error() == "pq: canceling statement due to user request" { return fmt.Errorf("%w: %v", ctx.Err(), err) } return err } ... func exampleHandler(w http.ResponseWriter, r *http.Request) { err := slowQuery(r.Context()) if err != nil { // Check if the returned error equals or wraps context.Canceled and // record a warning if it does. switch { case errors.Is(err, context.Canceled): serverWarning(err) default: serverError(w, err) } return } fmt.Fprintln(w, "OK") } func serverWarning(err error) { log.Printf("WARNING: %s", err.Error()) } ... If you were to run this application again now and make two different GET / requests — one that times out, and the other that you cancel — you should see clearly different messages in the application log, like so: $ go run . 2020/04/17 13:09:25 Listening... 2020/04/17 13:09:45 ERROR: context deadline exceeded: pq: canceling statement due to user request 2020/04/17 13:09:47 WARNING: context canceled: pq: canceling statement due to user request Other context-aware methods The database/sql package provides context-aware variants for most actions on sql.DB, including PingContext(), QueryContext(), and QueryRowContext(). We can (and should!) update the main() function in the code above to use PingContext() instead of Ping(). In this case there is no request context to use as the parent, so we need to create an empty parent context with context.Background() instead. Like so: ... func main() { var err error db, err = sql.Open("postgres", "postgres://user:pa$$word@localhost/example_db") if err != nil { log.Fatal(err) } // Create a context with a 10-second timeout, using the empty // context.Background() as the parent. ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) defer cancel() // Use this when testing the connection pool. if err = db.PingContext(ctx); err != nil { log.Fatal(err) } mux := http.NewServeMux() mux.HandleFunc("/", exampleHandler) log.Print("Listening...") err = http.ListenAndServe(":5000", mux) if err != nil { log.Fatal(err) } } ... Can I set a global timeout for all requests? Sure, you could create and use some middleware on your routes which adds a timeout to the current request context, similar to this: func setTimeout(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ctx, cancel := context.WithTimeout(r.Context(), 5*time.Second) defer cancel() // This gives you a copy of the request with a the request context // changed to the new context with the 5-second timeout created // above. r = r.WithContext(ctx) next.ServeHTTP(w, r) }) } If you take this approach there are a few of things to be aware of: The timeout starts from the moment the context is created, so any code running in your handlers before the database query counts towards the timeout. If you have multiple queries being executed in a handler, then they all have to complete within that one time. The timeout will continue to apply even if you derive a child context with a different timeout duration. So while you can enforce an earlier timeout in a child context, you can't make it longer. What about http.TimeoutHandler? Go provides a http.TimeoutHandler() middleware function which you can use to wrap your handlers or router/servemux. This works similar to the middleware above in the sense that it sets a timeout on the request context... so the warnings above also apply when using this. However, http.TimeoutHandler() also sends the client a 503 Service Unavailable response and a HTML error message. So, if you're using this in your application, you shouldn't (or at least, you don't need to) send the client an error response yourself when encountering a context.DeadlineExceeded error. How about transactions? How does context work in those? The database/sql package provides a BeginTx() method which you can use to initiate a context-aware transaction. A code example can be seen here. It's important to understand that the context you provide to BeginTx() applies to the whole transaction. In the event of a timeout/cancellation on the context, then the queries in the transaction will automatically be rolled-back. It's perfectly fine to pass the same context as a parameter for all the queries in the transaction, in which case it ensures that they all (as a whole) complete before any timeout/cancellation . Alternatively, if you want per-query timeouts you can create different child contexts with different timeouts for each in the queries in the transaction. But you must derive these child contexts from the context you passed to BeginTX(). Otherwise there is a risk that the BeginTX() context timeout/cancellation occurs and the automatic rollback happens, but your code still may try to execute the query with a still-live context. If that happened you would receive the error "sql: transaction has already been committed or rolled back". What about background processing? When doing background-processing in a different goroutine, bear in mind that if a parent context is canceled, the cancellation signal 'bubbles down' to its children. And also bear in mind what I quoted earlier about request context cancellation: For incoming server requests, the [request] context is canceled ... when the ServeHTTP method returns. Combine those two things, and it means that if you use a context which is a child of the request context in the background-process, the background-process will get a cancellation signal when the HTTP response is sent for the initial request. If you don't want that to be the case (and you probably don't), then you should create a brand-new context for the background-process using context.Background() and copy over any values that you need... or just pass them as regular parameters instead. If a context is canceled, can I be confident that it's due to a closed connection? Yes — so long as it's within the main goroutine for the request, it's a child of the request context, and you haven't manually canceled it yourself yet using defer cancel(). Otherwise, no. Is the behavior the same with other databases and drivers? I'm not sure. I've only used these features extensively with PostgreSQL and the pq driver. I imagine that things will be roughly the same with other databases and drivers, but you'll need to check. Anything else I should know? Yep. This is a strange one and it's not officially documented yet, but if a client makes a request with a non-empty request body then closes the connection, the context won't be canceled until after you have read the request body. This doesn’t apply to requests without a request body, where the cancellation signal will be received immediately. You should also be aware of the WriteTimeout setting on your http.Server (if you have set one). Your context timeouts should always be shorter than your WriteTimeout value, otherwise the WriteTimeout will be hit first, the connection will be closed, and the client won’t get any response.

• How to parse a JSON request body in Go
  Oct 21, 2019

  Let's say that you're building a JSON API with Go. And in some of the handlers — probably as part of a POST or PUT request — you want to read a JSON object from the request body and assign it to a struct in your code. After a bit of research, there's a good chance that you'll end up with some code that looks similar to the personCreate handler here: // File: main.go package main import ( "encoding/json" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // Declare a new Person struct. var p Person // Try to decode the request body into the struct. If there is an error, // respond to the client with the error message and a 400 status code. err := json.NewDecoder(r.Body).Decode(&p) if err != nil { http.Error(w, err.Error(), http.StatusBadRequest) return } // Do something with the Person struct... fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } If you're putting together a quick prototype, or building an API for personal/internal use only, then the code in the personCreate handler is probably OK. But if you're building an API for public use in production then there are a few issues with this to be aware of, and things that can be improved. Not all errors returned by Decode() are caused by a bad request from the client. Specifically, Decode() can return a json.InvalidUnmarshalError error — which is caused by an unmarshalable target destination being passed to Decode(). If that happens, then it indicates a problem with our application — not the client request — so really the error should be logged and a 500 Internal Server Error response sent to the client instead. The error messages returned by Decode() aren't ideal for sending to a client. Some are arguably too detailed and expose information about the underlying program (like "json: cannot unmarshal number into Go struct field Person.Name of type string"). Others aren't descriptive enough (like "unexpected EOF") and some are just plain confusing (like "invalid character 'A' looking for beginning of object key string"). There also isn't consistency in the formatting or language used. A client can include extra unexpected fields in their JSON, and these fields will be silently ignored without the client receiving any error. We can fix this by using the decoder's DisallowUnknownFields() method. There's no upper limit on the size of the request body that will be read by the Decode() method. Limiting this would help prevent our server resources being wasted if a malcious client sends a very large request body, and it's something we can easily do by using the http.MaxBytesReader() function. There's no check for a Content-Type: application/json header in the request. Of course, this header may not always be present, and mistakes and malicious clients mean that it isn't a guarantee of the actual content type. But checking for an incorrect Content-Type header would allow us to 'fail fast' and send a helpful error message without spending unnecessary resources on parsing the body. The Decode() method is designed to decode streams of JSON objects. This means a request body like '{"Name": "Bob"}{"Name": "Carol": "Age": 54}' or '{"Name": "Dave"}{}' is considered valid and won't result in the client receiving an error message. But in each case, only the first JSON object in the request body will actually be parsed. There are two solutions here. We can either check the decoder's More() method after decoding to see if there any any other JSON objects in the request body. Or we could avoid using Decode() altogether and read the body into a byte slice and pass it to json.Unmarshal(), which would return an error if the body contains multiple JSON objects. The downside of using json.Unmarshal() is that there is no way to disallow extra unexpected fields in the JSON, so we can't address point 3 above. An Improved Handler Let's implement an alternative version of the personCreate handler which addresses all of these issues. You'll notice here that we're using the new errors.Is() and errors.As() functions, which have been introduced in Go 1.13, to help intercept the errors from Decode(). // File: main.go package main import ( "encoding/json" "errors" "fmt" "io" "log" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { // If the Content-Type header is present, check that it has the value // application/json. Note that we are using the gddo/httputil/header // package to parse and extract the value here, so the check works // even if the client includes additional charset or boundary // information in the header. if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" http.Error(w, msg, http.StatusUnsupportedMediaType) return } } // Use http.MaxBytesReader to enforce a maximum read of 1MB from the // response body. A request body larger than that will now result in // Decode() returning a "http: request body too large" error. r.Body = http.MaxBytesReader(w, r.Body, 1048576) // Setup the decoder and call the DisallowUnknownFields() method on it. // This will cause Decode() to return a "json: unknown field ..." error // if it encounters any extra unexpected fields in the JSON. Strictly // speaking, it returns an error for "keys which do not match any // non-ignored, exported fields in the destination". dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() var p Person err := dec.Decode(&p) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { // Catch any syntax errors in the JSON and send an error message // which interpolates the location of the problem to make it // easier for the client to fix. case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) http.Error(w, msg, http.StatusBadRequest) // In some circumstances Decode() may also return an // io.ErrUnexpectedEOF error for syntax errors in the JSON. There // is an open issue regarding this at // https://github.com/golang/go/issues/25956. case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") http.Error(w, msg, http.StatusBadRequest) // Catch any type errors, like trying to assign a string in the // JSON request body to a int field in our Person struct. We can // interpolate the relevant field name and position into the error // message to make it easier for the client to fix. case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by extra unexpected fields in the request // body. We extract the field name from the error message and // interpolate it in our custom error message. There is an open // issue at https://github.com/golang/go/issues/29035 regarding // turning this into a sentinel error. case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) http.Error(w, msg, http.StatusBadRequest) // An io.EOF error is returned by Decode() if the request body is // empty. case errors.Is(err, io.EOF): msg := "Request body must not be empty" http.Error(w, msg, http.StatusBadRequest) // Catch the error caused by the request body being too large. Again // there is an open issue regarding turning this into a sentinel // error at https://github.com/golang/go/issues/30715. case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" http.Error(w, msg, http.StatusRequestEntityTooLarge) // Otherwise default to logging the error and sending a 500 Internal // Server Error response. default: log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } // Check that the request body only contained a single JSON object. if dec.More() { msg := "Request body must only contain a single JSON object" http.Error(w, msg, http.StatusBadRequest) return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } The clear downside here is that this code is a lot more verbose, and IMO, a little bit ugly. Things aren't helped by the fact that there are quite a few open issues with json/encoding which are on hold pending a wider review of the package. But from a security and client perspective it's a lot better : ) The handler is now stricter about the content it will accept; we're reducing the amount of server resources used unnecessarily; and the client gets clear and consistent error messages that provide a decent amount of information without over-sharing. As a side note, you might have noticed that the json/encoding package contains some other error types (like json.UnmarshalFieldError) which aren't checked in the code above — but these have been deprecated and not used by Go 1.13. Making a Helper Function If you've got a few handlers that need to to process JSON request bodies, you probably don't want to repeat this code in all of them. A solution which I've found works well is to create a decodeJSONBody helper function, and have this return a custom malformedRequest error type which wraps the errors and relevant status codes. For example: // File: helpers.go package main import ( "encoding/json" "errors" "fmt" "io" "net/http" "strings" "github.com/golang/gddo/httputil/header" ) type malformedRequest struct { status int msg string } func (mr *malformedRequest) Error() string { return mr.msg } func decodeJSONBody(w http.ResponseWriter, r *http.Request, dst interface{}) error { if r.Header.Get("Content-Type") != "" { value, _ := header.ParseValueAndParams(r.Header, "Content-Type") if value != "application/json" { msg := "Content-Type header is not application/json" return &malformedRequest{status: http.StatusUnsupportedMediaType, msg: msg} } } r.Body = http.MaxBytesReader(w, r.Body, 1048576) dec := json.NewDecoder(r.Body) dec.DisallowUnknownFields() err := dec.Decode(&dst) if err != nil { var syntaxError *json.SyntaxError var unmarshalTypeError *json.UnmarshalTypeError switch { case errors.As(err, &syntaxError): msg := fmt.Sprintf("Request body contains badly-formed JSON (at position %d)", syntaxError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.ErrUnexpectedEOF): msg := fmt.Sprintf("Request body contains badly-formed JSON") return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.As(err, &unmarshalTypeError): msg := fmt.Sprintf("Request body contains an invalid value for the %q field (at position %d)", unmarshalTypeError.Field, unmarshalTypeError.Offset) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case strings.HasPrefix(err.Error(), "json: unknown field "): fieldName := strings.TrimPrefix(err.Error(), "json: unknown field ") msg := fmt.Sprintf("Request body contains unknown field %s", fieldName) return &malformedRequest{status: http.StatusBadRequest, msg: msg} case errors.Is(err, io.EOF): msg := "Request body must not be empty" return &malformedRequest{status: http.StatusBadRequest, msg: msg} case err.Error() == "http: request body too large": msg := "Request body must not be larger than 1MB" return &malformedRequest{status: http.StatusRequestEntityTooLarge, msg: msg} default: return err } } if dec.More() { msg := "Request body must only contain a single JSON object" return &malformedRequest{status: http.StatusBadRequest, msg: msg} } return nil } Once that's written, the code in your handlers can be kept really nice and compact: // File: main.go package main import ( "errors" "fmt" "log" "net/http" ) type Person struct { Name string Age int } func personCreate(w http.ResponseWriter, r *http.Request) { var p Person err := decodeJSONBody(w, r, &p) if err != nil { var mr *malformedRequest if errors.As(err, &mr) { http.Error(w, mr.msg, mr.status) } else { log.Print(err.Error()) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) } return } fmt.Fprintf(w, "Person: %+v", p) } func main() { mux := http.NewServeMux() mux.HandleFunc("/person/create", personCreate) log.Print("Starting server on :4000...") err := http.ListenAndServe(":4000", mux) log.Fatal(err) }

• Using PostgreSQL JSONB with Go
  May 14, 2019

  PostgreSQL provides two JSON-related data types that you can use — JSON and JSONB. The principal differences are: JSON stores an exact copy of the JSON input. JSONB stores a binary representation of the JSON input. This makes it slower to insert but faster to query. It may change the key order, and will remove whitespace and delete duplicate keys. JSONB also supports the ? (existence) and @> (containment) operators, whereas JSON doesn't. The PostgreSQL documentation recommends that you should generally use JSONB, unless you have a specific reason not too (like needing to preserve key order). Here's a cribsheet for the essential commands: -- Create a table with a JSONB column. CREATE TABLE items ( id SERIAL PRIMARY KEY, attrs JSONB ); -- You can insert any well-formed json input into the column. Note that only -- lowercase `true` and `false` spellings are accepted. INSERT INTO items (attrs) VALUES ('{ "name": "Pasta", "ingredients": ["Flour", "Eggs", "Salt", "Water"], "organic": true, "dimensions": { "weight": 500.00 } }'); -- Create an index on all key/value pairs in the JSONB column. CREATE INDEX idx_items_attrs ON items USING gin (attrs); -- Create an index on a specific key/value pair in the JSONB column. CREATE INDEX idx_items_attrs_organic ON items USING gin ((attrs->'organic')); -- The -> operator is used to get the value for a key. The returned value has -- the type JSONB. SELECT attrs->'dimensions' FROM items; SELECT attrs->'dimensions'->'weight' FROM items; -- Or you can use ->> to do the same thing, but this returns a TEXT value -- instead. SELECT attrs->>'dimensions' FROM items; -- You can use the returned values as normal, although you may need to type -- cast them first. SELECT * FROM items WHERE attrs->>'name' ILIKE 'p%'; SELECT * FROM items WHERE (attrs->'dimensions'->>'weight')::numeric 'dimensions' ? 'weight'; -- The ? operator can also be used to check for the existence of a specific -- text value in json arrays. SELECT * FROM items WHERE attrs->'ingredients' ? 'Salt'; -- Use @> to check if the JSONB column contains some specific json. This can -- be useful to filter for a specific key/value pair like so: SELECT * FROM items WHERE attrs @> '{"organic": true}'::jsonb; SELECT * FROM items WHERE attrs @> '{"dimensions": {"weight": 10}}'::jsonb; -- Note that @> looks for *containment*, not for an exact match. The -- followingquery will return records which have both "Flour" and "Water" -- as ingredients, rather than *only* "Flour" and "Water" as the ingredients. SELECT * FROM items WHERE attrs @> '{"ingredients": ["Flour", "Water"]}'::jsonb; For a full description of all available operators please see the official JSON Functions and Operators documentation. Using with Go If you're not familiar with the general patterns for working with SQL databases in Go, you might want to read my introduction to the database/sql package before continuing. Known JSON fields When the fields in a JSON/JSONB column are known in advance, you can map the contents of the JSON/JSONB column to and from a struct. To do this, you'll need make sure the struct implements: The driver.Valuer interface, such that it marshals the object into a JSON byte slice that can be understood by the database. The sql.Scanner interface, such that it unmarshals a JSON byte slice from the database into the struct fields. Here's a demonstration: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } // The Attrs struct represents the data in the JSON/JSONB column. We can use // struct tags to control how each field is encoded. type Attrs struct { Name string `json:"name,omitempty"` Ingredients []string `json:"ingredients,omitempty"` Organic bool `json:"organic,omitempty"` Dimensions struct { Weight float64 `json:"weight,omitempty"` } `json:"dimensions,omitempty"` } // Make the Attrs struct implement the driver.Valuer interface. This method // simply returns the JSON-encoded representation of the struct. func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } // Make the Attrs struct implement the sql.Scanner interface. This method // simply decodes a JSON-encoded value into the struct fields. func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } // Initialize a new Attrs struct and add some values. attrs := new(Attrs) attrs.Name = "Pesto" attrs.Ingredients = []string{"Basil", "Garlic", "Parmesan", "Pine nuts", "Olive oil"} attrs.Organic = false attrs.Dimensions.Weight = 100.00 // The database driver will call the Value() method and and marshall the // attrs struct to JSON before the INSERT. _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", attrs) if err != nil { log.Fatal(err) } // Similarly, we can also fetch data from the database, and the driver // will call the Scan() method to unmarshal the data to an Attr struct. item := new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } // You can then use the struct fields as normal... weightKg := item.Attrs.Dimensions.Weight / 1000 log.Printf("Item: %d, Name: %s, Weight: %.2fkg", item.ID, item.Attrs.Name, weightKg) } Unknown JSON fields The above pattern works great if you know in advance what keys and values your JSON/JSONB data will contain. And it has the major advantage of being type safe. For the times that you don't know this in advance (for example, the data contains user-generated keys and values) you can map the contents of the JSON/JSONB column to and from a map[string]interface{} instead. The big downside of this is that you will need to type assert any values that you retrieve from the database in order to use them. Here's the same example, but re-written to use a map[string]interface{}: package main import ( "database/sql" "database/sql/driver" "encoding/json" "errors" "log" _ "github.com/lib/pq" ) type Item struct { ID int Attrs Attrs } type Attrs map[string]interface{} func (a Attrs) Value() (driver.Value, error) { return json.Marshal(a) } func (a *Attrs) Scan(value interface{}) error { b, ok := value.([]byte) if !ok { return errors.New("type assertion to []byte failed") } return json.Unmarshal(b, &a) } func main() { db, err := sql.Open("postgres", "postgres://user:pass@localhost/db") if err != nil { log.Fatal(err) } item := new(Item) item.Attrs = Attrs{ "name": "Passata", "ingredients": []string{"Tomatoes", "Onion", "Olive oil", "Garlic"}, "organic": true, "dimensions": map[string]interface{}{ "weight": 250.00, }, } _, err = db.Exec("INSERT INTO items (attrs) VALUES($1)", item.Attrs) if err != nil { log.Fatal(err) } item = new(Item) err = db.QueryRow("SELECT id, attrs FROM items ORDER BY id DESC LIMIT 1").Scan(&item.ID, &item.Attrs) if err != nil { log.Fatal(err) } name, ok := item.Attrs["name"].(string) if !ok { log.Fatal("unexpected type for name") } dimensions, ok := item.Attrs["dimensions"].(map[string]interface{}) if !ok { log.Fatal("unexpected type for dimensions") } weight, ok := dimensions["weight"].(float64) if !ok { log.Fatal("unexpected type for weight") } weightKg := weight / 1000 log.Printf("%s: %.2fkg", name, weightKg) }

• An overview of Go's tooling
  Apr 15, 2019

  Occasionally I get asked “why do you like using Go?” And one of the things I often mention is the thoughtful tooling that exists alongside the language as part of the go command. There are some tools that I use everyday — like go fmt and go build — and others like go tool pprof that I only use to help solve a specific issue. But in all cases I appreciate the fact that they make managing and maintaining my projects easier. In this post I hope to provide a little background and context about the tools I find most useful, and importantly, explain how they can fit into the workflow of a typical project. I hope it'll give you a good start if you're new to Go. Or if you've been working with Go for a while, and that stuff's not applicable to you, hopefully you'll still discover a command or flag that you didn't know existed before : ) The information in this post is written for Go 1.12 and assumes that you're working on a project which has modules enabled. Installing Tooling Viewing Environment Information Development Running Code Fetching Dependencies Refactoring Code Viewing Go Documentation Testing Running Tests Profiling Test Coverage Stress Testing Testing all Dependencies Pre-Commit Checks Formatting Code Performing Static Analysis Linting Code Tidying and Verifying your Dependencies Build and Deployment Building an Executable Cross-Compilation Using Compiler and Linker Flags Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks Profiling and Tracing Checking for Race Conditions Managing Dependencies Upgrading to a New Go Release Reporting Bugs Installing Tooling In this post I'll mainly be focusing on tools that are a part of the go command. But there are a few I'll be mentioning which aren't part of the standard Go 1.12 release. To install these while using Go 1.12 you'll first need to make sure that you're outside of a module-enabled directory (I usually just change into /tmp). Then you can use the GO111MODULE=on go get command to install the tool. For example: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress This will download the relevant package and dependencies, build the executable and add it to your GOBIN directory. If you haven't explicitly set a GOBIN directory, then the executable will be added to your GOPATH/bin folder. Either way, you should make sure that the appropriate directory is on your system path. Note: This process is a bit clunky and will hopefully improve in future versions of Go. Issue 30515 is tracking the discussion about this. Viewing Environment Information You can use the go env tool to display information about your current Go operating environment. This can be particularly useful if you're working on an unfamiliar machine. $ go env GOARCH="amd64" GOBIN="" GOCACHE="/home/alex/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/alex/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build245740092=/tmp/go-build -gno-record-gcc-switches" If there are specific values that you're interested in, you can pass them as arguments to go env. For example: $ go env GOPATH GOOS GOARCH /home/alex/go linux amd64 To show documentation for all go env variables and values you can run: $ go help environment Development Running Code During development the go run tool is a convenient way to try out your code. It's essentially a shortcut that compiles your code, creates an executable binary in your /tmp directory, and then runs this binary in one step. $ go run . # Run the package in the current directory $ go run ./cmd/foo # Run the package in the ./cmd/foo directory Note: As of Go 1.11 you can pass the path of a package to go run, like we have above. This means that you no longer have to use workarounds like go run *.go wildcard expansion to run multiple files. I like this improvement a lot! Fetching Dependencies Assuming that you've got modules enabled, when you use go run (or go test or go build for that matter) any external dependencies will automatically (and recursively) be downloaded to fulfill the import statements in your code. By default the latest tagged release of the dependency will be downloaded, or if no tagged releases are available, then the dependency at the latest commit. If you know in advance that you need a specific version of a dependency (instead of the one that Go would fetch by default) you can use go get with the relevant version number or commit hash. For example: $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@8e1b8d3 If the dependency being fetched has a go.mod file, then its dependencies won't be listed in your go.mod file. In contrast, if the dependency you're downloading doesn't have a go.mod file, then it's dependencies will be listed in your go.mod file with an // indirect comment next to them. So that means your go.mod file doesn't necessarily show all the dependencies for your project in one place. Instead, you can view them all using the go list tool like so: $ go list -m all Sometimes you might wonder why is that a dependency? You can answer this with the go mod why command, which will show you the shortest path from a package in your main module to a given dependency. For example: $ go mod why -m golang.org/x/sys # golang.org/x/sys github.com/alexedwards/argon2id golang.org/x/crypto/argon2 golang.org/x/sys/cpu Note: The go mod why command will return an answer for most, but not all, dependencies. Issue 27900 is tracking this. If you're interested in analyzing or visualizing the dependencies for your application, then you might want to also check out the go mod graph tool. There's a great tutorial and example code for generating visualizations here. Lastly, downloaded dependencies are stored in the module cache located at GOPATH/pkg/mod. If you ever need to clear the module cache you can use the go clean tool. But be aware: this will remove the downloaded dependencies for all projects on your machine. $ go clean -modcache Refactoring Code Chances are you're probably familiar with using the gofmt tool to automatically format your code. But it also supports rewrite rules that you can use to help refactor your code. I'll demonstrate. Let's say that you have the following code and you want to change the foo variable to Foo so it is exported. var foo int func bar() { foo = 1 fmt.Println("foo") } To do this you can use gofmt with the -r flag to implement a rewrite rule, the -d flag to display a diff of the changes, and the -w flag to make the changes in place, like so: $ gofmt -d -w -r 'foo -> Foo' . -var foo int +var Foo int func bar() { - foo = 1 + Foo = 1 fmt.Println("foo") } Notice how this is smarter than a find-and-replace? The foo variable has been changed, but the "foo" string in the fmt.Println() statement has been left unchanged. Another thing to note is that the gofmt command works recursively, so the above command will run on all *.go files in your current directory and subdirectories. If you want to use this functionality, I recommend running rewrite rules without the -w flag first, and checking the diff first to make sure that the changes to the code are what you expect. Let's take a look at a slightly more complicated example. Say you want to update your code to use the new Go 1.12 strings.ReplaceAll() function instead of strings.Replace(). To make this change you can run: $ gofmt -w -r 'strings.Replace(a, b, c, -1) -> strings.ReplaceAll(a, b, c)' . In rewrite rules, single lowercase characters act as wildcards matching arbitrary expressions, and those expressions will be substituted-in in the replacement. Viewing Go Documentation You can view documentation for the standard library packages via your terminal using the go doc tool. I often use this during development to quickly check something — like the name or signature of a specific function. I find it faster than navigating the web-based documentation and it's always available offline too. $ go doc strings # View simplified documentation for the strings package $ go doc -all strings # View full documentation for the strings package $ go doc strings.Replace # View documentation for the strings.Replace function $ go doc sql.DB # View documentation for the database/sql.DB type $ go doc sql.DB.Query # View documentation for the database/sql.DB.Query method You can also include the -src flag to display the relevant Go source code. For example: $ go doc -src strings.Replace # View the source code for the strings.Replace function Testing Running Tests You can use the go test tool to run tests in your project like so: $ go test . # Run all tests in the current directory $ go test ./... # Run all tests in the current directory and sub-directories $ go test ./foo/bar # Run all tests in the ./foo/bar directory Typically I run my tests with Go's race detector enabled, which can help pick up some of the data races that might occur in real-life usage. Like so: $ go test -race ./... It's important to note that enabling the race detector will increase the overall running time of your tests. So if you're running tests very frequently part of a TDD workflow, you might prefer to save using this for a pre-commit test run only. Since 1.10, Go caches test results at the package-level. If a package hasn't changed between test runs — and you're using the same, cachable, flags for go test — then the cached test result will be displayed with a "(cached)" next to it. This is hugely helpful in speeding up the test runtime for large codebases. If you want force your tests to run in full (and avoid the cache) you can use the -count=1 flag, or clear all cached test results by using the go clean tool. $ go test -count=1 ./... # Bypass the test cache when running tests $ go clean -testcache # Delete all cached test results Note: Cached test results are stored alongside cached build results in your GOCACHE directory. Check go env GOCACHE if you're not sure where this is on your machine. You can limit go test to running specific tests (and sub-tests) by using the -run flag. This accepts a regular expression, and only tests which have names that match the regular expression will be run. I like to combine this with the -v flag to enable verbose mode, so the names of running tests and sub-tests are displayed. It's a useful way to make sure that I haven't screwed up the regexp and that the tests I expect are actually being run! $ go test -v -run=^TestFooBar$ . # Run the test with the exact name TestFooBar $ go test -v -run=^TestFoo . # Run tests whose names start with TestFoo $ go test -v -run=^TestFooBar$/^Baz$ . # Run the Baz subtest of the TestFooBar test only A couple more flags that it's good to be aware of are -short (which you can use to skip long-running tests) and -failfast (which will stop running further tests after the first failure). Note that -failfast will prevent test results from being cached. $ go test -short ./... # Skip long running tests $ go test -failfast ./... # Don't run further tests after a failure. Profiling Test Coverage You can enable coverage analysis when running tests by using the -cover flag. This will display the percentage of code covered by the tests in the output for each package, similar to this: $ go test -cover ./... ok github.com/alexedwards/argon2id 0.467s coverage: 78.6% of statements You can also generate a coverage profile using the -coverprofile flag and view it in your web browser by using the go tool cover -html command like so: $ go test -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out This will gives you a navigable listing of all the test files, with code covered by the tests displayed in green, and uncovered code in red. If you want you can go a step further and set the -covermode=count flag to make the coverage profile record the exact number of times that each statement is executed during the tests. $ go test -covermode=count -coverprofile=/tmp/profile.out ./... $ go tool cover -html=/tmp/profile.out When viewed in the browser, statements which are executed more frequently are shown in a more saturated shade of green, similar to this: Note: If you’re using the t.Parallel() command in any of your tests, then you should use the flag -covermode=atomic instead of -covermode=count instead to ensure an accurate count. Lastly, if you don't have a web browser available to view a coverage profile, you can see a breakdown of test coverage by function/method in your terminal with the command: $ go tool cover -func=/tmp/profile.out github.com/alexedwards/argon2id/argon2id.go:77: CreateHash 87.5% github.com/alexedwards/argon2id/argon2id.go:96: ComparePasswordAndHash 85.7% ... Stress Testing You can use the go test -count command to run a test multiple times in succession, which can be useful if you want to check for sporadic or intermittent failures. For example: $ go test -run=^TestFooBar$ -count=500 . In this example, the TestFooBar test will be repeated 500 times in a row. But it's important to note that the test will be repeated in serial — even if it contains a t.Parallel() instruction. So if your test is doing something relatively slow, like making a round trip to a database, hard disk or the internet, running a large number of tests can take quite a long time. In that case you might want to use the stress tool to repeat the same test multiple times in parallel instead. You can install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/stress To use the stress tool, you'll first need to compile a test binary for the specific package you want to test. You can do using the go test -c command. For example, to create a test binary for the package in your current directory: $ go test -c -o=/tmp/foo.test . In this example, the test binary will be outputted to /tmp/foo.test. You can then use the stress tool to execute a specific test in the test binary like so: $ stress -p=4 /tmp/foo.test -test.run=^TestFooBar$ 60 runs so far, 0 failures 120 runs so far, 0 failures ... Note: In the example above I've used the -p flag to restrict the number of parallel processes used by stress to 4. Without this flag, the tool will default to using a number of processes equal to runtime.NumCPU(). Testing all Dependencies Before you build an executable for release or deployment, or distribute your code publicly, you may want to run the go test all command: $ go test all This will run tests on all packages in your module and all dependencies — include testing test dependencies and the necessary standard library packages — and it can help validate that the exact versions of the dependencies being used are compatible with each other. This can take quite a long time to run, but the results cache well so any subsequent tests should be faster in the future. If you want, you could also use go test -short all to skip any long-running tests. Pre-Commit Checks Formatting Code Go provides two tools to automatically format your code according to the Go conventions: gofmt and go fmt. Using these helps keep your code consistent across your files and projects, and — if you use them before committing code — helps reduce noise when examining a diff between file versions. I like to use the gofmt tool with the following flags: $ gofmt -w -s -d foo.go # Format the foo.go file $ gofmt -w -s -d . # Recursively format all files in the current directory and sub-directories In these commands, the -w flag instructs the tool to rewrite files in place, the -s instructs the tool to apply simplifications to the code where possible, and the -d flag instructs the tool to output diffs of the changes (because I'm curious to see what is changed). If you want to only display the names of changed files, instead of diffs, you can swap this for the -l flag instead. Note: The gofmt command works recursively. If you pass it a directory like . or ./cmd/foo it'll format all .go files under the directory. The other formatting tool — go fmt — tool is a wrapper which essentially calls gofmt -l -w on a specified file or directory. You can use it like this: $ go fmt ./... Performing Static Analysis The go vet tool carries out static analysis of your code and warns you of things which might be wrong with your code but wouldn't be picked up by the compiler. Issues like unreachable code, unnecessary assignments and badly-formed build tags. You can use it like so: $ go vet foo.go # Vet the foo.go file $ go vet . # Vet all files in the current directory $ go vet ./... # Vet all files in the current directory and sub-directories $ go vet ./foo/bar # Vet all files in the ./foo/bar directory Behind the scenes, go vet runs a bunch of different analyzers which are listed here and you can disable specific ones on a case-by-case basis. For example to disable the composite analyzer you can use: $ go vet -composites=false ./... There are a couple of experimental analyzers in golang.org/x/tools which you might want to try: nilness (which checks for redundant or impossible nil comparisons) and shadow (which check for possible unintended shadowing of variables). If you want to use these, you'll need to install and run them separately. For example, to install nilness you would run: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/go/analysis/passes/nilness/cmd/nilness And you can then use it like so: $ go vet -vettool=$(which nilness) ./... Note: when the -vettool flag is used it will only run the specified analyzer — all the other go vet analyzers won't be run. As a side note, since Go 1.10 the go test tool automatically executes a small, high-confidence, subset of the go vet checks before running any tests. You can turn this behavior off when running tests like so: $ go test -vet=off ./... Linting Code You can use the golint tool to identify style mistakes in your code. Unlike go vet, this isn't concerned with correctness of the code, but helps you to align your code with the style conventions in Effective Go and the Go CodeReviewComments. It's not part of the standard library, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/lint/golint You can then run it as follows: $ golint foo.go # Lint the foo.go file $ golint . # Lint all files in the current directory $ golint ./... # Lint all files in the current directory and sub-directories $ golint ./foo/bar # Lint all files in the ./foo/bar directory Tidying and Verifying your Dependencies Before you commit any changes to your code I recommend running the following two commands to tidy and verify your dependencies: $ go mod tidy $ go mod verify The go mod tidy command will prune any unused dependencies from your go.mod and go.sum files, and update the files to include dependencies for all possible build tags/OS/architecture combinations (note: go run, go test, go build etc are ‘lazy' and will only fetch packages needed for the current build tags/OS/architecture). Running this before each commit will make it easier to determine which of your code changes were responsible for adding or removing which dependencies when looking at the version control history. I also recommend using the go mod verify command to check that the dependencies on your computer haven't accidentally (or purposely) been changed since they were downloaded and that they match the cryptographic hashes in your go.sum file. Running this helps ensure that the dependencies being used are the exact ones that you expect, and any build for that commit will be reproducible at a later point. Build and Deployment Building an Executable To compile a main package and create an executable binary you can use the go build tool. Typically I use it in conjunction with the -o flag, which let's you explicitly set the output directory and name of the binary like so: $ go build -o=/tmp/foo . # Compile the package in the current directory $ go build -o=/tmp/foo ./cmd/foo # Compile the package in the ./cmd/foo directory In these examples, go build will compile the specified package (and any dependent packages), then invoke the linker to generate an executable binary, and output this to /tmp/foo. It's important to note that, as of Go 1.10, the go build tool caches build output in the build cache. This cached output will be reused again in future builds where appropriate, which can significantly speed up the overall build time. This new caching behavior means that the old maxim of “prefer go install to go build to improve caching” no longer applies. If you're not sure where your build cache is, you can check by running the go env GOCACHE command: $ go env GOCACHE /home/alex/.cache/go-build Using the build cache comes with one important caveat — it does not detect changes to C libraries imported with cgo. So if your code imports a C library via cgo and you've made changes to it since the last build, you'll need to use the -a flag which forces all packages to be rebuilt. Alternatively, you could use go clean to purge the cache: $ go build -a -o=/tmp/foo . # Force all packages to be rebuilt $ go clean -cache # Remove everything from the build cache Note: Running go clean -cache will delete cached test results too. If you're interested in what go build is doing behind the scenes, you might like to use the following commands: $ go list -deps . | sort -u # List all packages that are used to build the executable $ go build -a -x -o=/tmp/foo . # Rebuild everything and show the commands that are run Finally, if you run go build on a non-main package, it will be compiled in a temporary location and again, the result will be stored in the build cache. No executable is produced. Cross-Compilation This is one of my favorite features of Go. By default go build will output a binary suitable for use on your current operating system and architecture. But it also supports cross-compilation, so you can generate a binary suitable for use on a different machine. This is particularly useful if you're developing on one operating system and deploying on another. You can specify the operating system and architecture that you want to create the binary for by setting the GOOS and GOARCH environment variables respectively. For example: $ GOOS=linux GOARCH=amd64 go build -o=/tmp/linux_amd64/foo . $ GOOS=windows GOARCH=amd64 go build -o=/tmp/windows_amd64/foo.exe . To see a list of all supported OS/architecture combinations you can run go tool dist list: $ go tool dist list aix/ppc64 android/386 android/amd64 android/arm android/arm64 darwin/386 darwin/amd64 ... Hint: You can use Go's cross-compilation to create WebAssembly binaries. For a bit more in-depth information about cross compilation I recommend reading this excellent post. Using Compiler and Linker Flags When building your executable you can use the -gcflags flag to change the behavior of the compiler and see more information about what it's doing. You can see a complete list of available compiler flags by running: $ go tool compile -help One flag that you might find interesting is -m, which triggers the printing of information about optimization decisions made during compilation. You can use it like this: $ go build -gcflags="-m -m" -o=/tmp/foo . # Print information about optimization decisions In the above example I used the -m flag twice to indicate that I want to print decision information two-levels deep. You can get simpler output by using just one. Also, as of Go 1.10, compiler flags only apply to the specific packages passed to go build — which in the example above is the package in the current directory (represented by .). If you want to print optimization decisions for all packages including dependencies can use this command instead: $ go build -gcflags="all=-m" -o=/tmp/foo . As of Go 1.11, you should find it easier to debug optimized binaries than before. However, you can still use the flags -N to disable optimizations and -l to disable inlining if you need to. For example: $ go build -gcflags="all=-N -l" -o=/tmp/foo . # Disable optimizations and inlining You can see a list of available linker flags by running: $ go tool link -help Probably the most well-known of these is the -X flag, which allows you to "burn in" a (string) value to a specific variable in your application. This is commonly used to add a version number or commit hash. For example: $ go build -ldflags="-X main.version=1.2.3" -o=/tmp/foo . For more information about the -X flag and some sample code see this StackOverflow question and this post and this post. You may also be interested in using the -s and -w flags to strip debugging information from the binary. This typically shaves about 25% off the final size. For example: $ go build -ldflags="-s -w" -o=/tmp/foo . # Strip debug information from the binary Note: If binary size is something that you need to optimize for you might want to use upx to compress it. See this post for more information. Diagnosing Problems and Making Optimizations Running and Comparing Benchmarks A nice feature of Go is that it makes it easy to benchmark your code. If you're not familiar with the general process for writing benchmarks there are good guides here and here. To run benchmarks you'll need to use the go test tool, with the -bench flag set to a regular expression that matches the benchmarks you want to execute. For example: $ go test -bench=. ./... # Run all benchmarks and tests $ go test -run=^$ -bench=. ./... # Run all benchmarks (and no tests) $ go test -run=^$ -bench=^BenchmarkFoo$ ./... # Run only the BenchmarkFoo benchmark (and no tests) I almost always run benchmarks using the -benchmem flag, which forces memory allocation statistics to be included in the output. $ go test -bench=. -benchmem ./... By default, each benchmark test will be run for a minimum of 1 second, once only. You can change this with the -benchtime and -count flags: $ go test -bench=. -benchtime=5s ./... # Run each benchmark test for at least 5 seconds $ go test -bench=. -benchtime=500x /.... # Run each benchmark test for exactly 500 iterations $ go test -bench=. -count=3 ./... # Repeat each benchmark test 3 times over If the code that you're benchmarking uses concurrency, you can use the -cpu flag to see the performance impact of changing your GOMAXPROCS value (essentially, the number of OS threads that can execute your Go code simultaneously). For example, to run benchmarks with GOMAXPROCS set to 1, 4 and 8: $ go test -bench=. -cpu=1,4,8 ./... To compare changes between benchmarks you might want to use the benchcmp tool. This isn't part of the standard go command, so you'll need to install it like so: $ cd /tmp $ GO111MODULE=on go get golang.org/x/tools/cmd/benchcmp You can then use it like this: $ go test -run=^$ -bench=. -benchmem ./... > /tmp/old.txt # make changes $ go test -run=^$ -bench=. -benchmem ./... > /tmp/new.txt $ benchcmp /tmp/old.txt /tmp/new.txt benchmark old ns/op new ns/op delta BenchmarkExample-8 21234 5510 -74.05% benchmark old allocs new allocs delta BenchmarkExample-8 17 11 -35.29% benchmark old bytes new bytes delta BenchmarkExample-8 8240 3808 -53.79% Profiling and Tracing Go makes it possible to create diagnostic profiles for CPU use, memory use, goroutine blocking and mutex contention. You can use these to dig a bit deeper and see exactly how your application is using (or waiting on) resources. There are three ways to generate profiles: If you have a web application you can import the net/http/pprof package. This will register some handlers with the http.DefaultServeMux which you can then use to generate and download profiles for your running application. This post provides a good explanation and some sample code. For other types of applications, you can profile your running application using the pprof.StartCPUProfile() and pprof.WriteHeapProfile() functions. See the runtime/pprof documentation for sample code. Or you can generate profiles while running benchmarks or tests by using the various -***profile flags like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -cpuprofile=/tmp/cpuprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -memprofile=/tmp/memprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -blockprofile=/tmp/blockprofile.out . $ go test -run=^$ -bench=^BenchmarkFoo$ -mutexprofile=/tmp/mutexprofile.out . Note: Using the -***profile flags when running benchmarks or tests will result in a test binary being outputted to your current directory. If you want to output this to an alternative location you should use the -o flag like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -o=/tmp/foo.test -cpuprofile=/tmp/cpuprofile.out . Whichever way you choose to create a profile, when profiling is enabled your Go program will stop about 100 times per second and take a snapshot at that moment in time. These samples are collected together to form a profile that you can analyze using the pprof tool. My favourite way to inspect a profile is to use the go tool pprof -http command to open it in a web browser. For example: $ go tool pprof -http=:5000 /tmp/cpuprofile.out This will default to displaying a graph showing the execution tree for the sampled aspects of your application, which makes it possible to quickly get a feel for any resource usage 'hotspots'. In the graph above, we can see that the hotspots in terms of CPU usage are two system calls originating from ioutil.ReadFile(). You can also navigate to other views of the profile including top usage by function and source code. If the amount of information is overwhelming, you might want to use the --nodefraction flag to ignore nodes that account for less than a certain percentage of samples. For example to ignore nodes that use appear in less than 10% of samples you can run pprof like so: $ go tool pprof --nodefraction=0.1 -http=:5000 /tmp/cpuprofile.out This makes the graph a lot less 'noisy' and if you zoom in on this screenshot, it's now much clearer to see and understand where the CPU usage hotspots are. Profiling and optimizing resource usage is big, nuanced, topic and I've barely scratched the surface here. If you're interested in knowing more then I encourage you to read the following blog posts: Profiling and optimizing Go web applications Debugging performance issues in Go programs Daily code optimization using benchmarks and profiling Profiling Go programs with pprof Another tool that you can use to help diagnose issues is the runtime execution tracer. This gives you a view of how Go is creating and scheduling goroutines to run, when the garbage collector is running, and information about blocking syscall/network/sync operations. Again, you can generate trace from your tests or benchmarks, or use net/http/pprof to create and download a trace for your web application. You can then use go tool trace to view the output in your web browser like so: $ go test -run=^$ -bench=^BenchmarkFoo$ -trace=/tmp/trace.out . $ go tool trace /tmp/trace.out Important: This is currently only viewable in Chrome/Chromium. For more information about Go's execution tracer and how to interpret the output please see Rhys Hiltner's dotGo 2016 talk and this excellent blog post. Checking for Race Conditions I talked earlier about enabling Go's race detector during tests by using go test -race. But you can also enable it for running programs when building a executable, like so: $ go build -race -o=/tmp/foo . It's critical to note that race-detector-enabled binaries will use more CPU and memory than normal, so you shouldn't use the -race flag when building binaries for production under normal circumstances. But you may want to deploy a race-detector-enabled binary on one server within a pool of many. Or use it to help track down a suspected race-condition by using a load-test tool to throw traffic concurrently at a race-detector-enabled binary. By default, if any races are detected while the binary is running a log will be written to stderr. You can change this by using the GORACE environment variable if necessary. For example, to run the binary located at /tmp/foo and output any race logs to /tmp/race. you can use: $ GORACE="log_path=/tmp/race" /tmp/foo Managing Dependencies You can use the go list tool to check whether a specific dependency has a newer version available like so: $ go list -m -u github.com/alecthomas/chroma github.com/alecthomas/chroma v0.6.2 [v0.6.3] This will output the dependency name and version that you're currently using, followed by the latest version in square brackets [], if a newer one exists. You can also use go list to check for updates to all dependencies (and sub-dependencies) like so: $ go list -m -u all You can upgrade (or downgrade) a dependency to the latest version, specific tagged-release or commit hash with the go get command like so: $ go get github.com/foo/bar@latest $ go get github.com/foo/bar@v1.2.3 $ go get github.com/foo/bar@7e0369f If the dependency you're updating has a go.mod file, then based on the information in this go.mod file, updates to any sub-dependencies will also be downloaded if necessary. If you use the go get -u flag, the contents of the go.mod file will be ignored and all sub-dependencies will be upgraded to their latest minor/patch version… even if the go.mod specifies a different version. After upgrading or downgrading any dependencies it's a good idea to tidy your modfiles. And you might also want to run the tests for all packages to help check for incompatibilities. Like so: $ go mod tidy $ go test all Occasionally you might want to use a local version of a dependency (for example, you need to use a local fork until a patch is merged upstream). To do this, you can use the go mod edit command to replace a dependency in your go.mod file with a local version. For example: $ go mod edit -replace=github.com/alexedwards/argon2id=/home/alex/code/argon2id This will add a replace rule to your go.mod file like so, and any future invocations of go run, go build etc will use the local version. File: go.mod module alexedwards.net/example go 1.12 require github.com/alexedwards/argon2id v0.0.0-20190109181859-24206601af6c replace github.com/alexedwards/argon2id => /home/alex/Projects/playground/argon2id Once it's no longer necessary, you can remove the replace rule with the command: $ go mod edit -dropreplace=github.com/alexedwards/argon2id You can use the same general technique to import packages that exist only on your own file system. This can be useful if you're working on multiple modules in development at the same time, one of which depends on the other. Note: If you don't want to use the go mod edit command, you can edit your go.mod file manually to make these changes. Either way will work. Upgrading to a New Go Release The go fix tool was originally released back in 2011 (when regular changes were still being made to Go's API) to help users automatically update their old code to be compatible with the latest version of Go. Since then, Go's compatibility promise means if you're upgrading from one Go 1.x version to a newer 1.x version everything should Just Work and using go fix should generally be unnecessary. However, there are a handful of very specific issues that it does deal with. You can see a summary of them by running go tool fix -help. If you decide that you want or need to run go fix after upgrading, you should you run the following command, then inspect a diff of the changes before you commit them. $ go fix ./... Reporting Bugs If you're confident that you've found an unreported issue with Go's standard library, tooling or documentation, you can use the go bug command to create a new Github issue. $ go bug This will open a browser window containing an issue pre-filled with your system information and reporting template.

• How to hash and verify passwords with Argon2 in Go
  Dec 10, 2018

  Thanks to Andreas Auernhammer, author of the golang.org/x/crypto/argon2 package, for checking over this post before publication. If you're planning to store user passwords it's good practice (essential really) to hash them using a computationally expensive key-derivation function (KDF) like Bcrypt, Scrypt or Argon2. Hashing and verifying passwords in Go with Bcrypt and Scrypt is already easy to do thanks to the golang.org/x/crypto/bcrypt package and Matt Silverlock's elithrar/simple-scrypt package. I recommend them both. If you want to use Argon2 — which is widely considered to be the best in class KDF for hashing passwords — then you've got a couple of choices. The tvdburgt/go-argon2 package provides Go bindings to the libargon2 C library, or you can implement a pure Go solution by wrapping the golang.org/x/crypto/argon2 package with helpers for hashing and verifying passwords. In the rest of this post I'm going to explain exactly how to use this pure Go approach. A Brief Introduction to Argon2 But first, a little bit of background. It's important to explain that the Argon2 algorithm has 3 variants which work slightly differently: Argon2d, Argon2i and Argon2id. In general, for password hashing you should use the Argon2id variant. This is essentially a hybrid of the Argon2d and Argon2i algorithms and uses a combination of data-independent memory access (for resistance against side-channel timing attacks) and data-depending memory access (for resistance against GPU cracking attacks). The Argon2 algorithm accepts a number of configurable parameters: Memory — The amount of memory used by the algorithm (in kibibytes). Iterations — The number of iterations (or passes) over the memory. Parallelism — The number of threads (or lanes) used by the algorithm. Salt length — Length of the random salt. 16 bytes is recommended for password hashing. Key length — Length of the generated key (or password hash). 16 bytes or more is recommended. The memory and iterations parameters control the computational cost of hashing the password. The higher these figures are, the greater the cost of generating the hash. It also follows that the greater the cost will be for any attacker trying to guess the password. But there's a balance that you need to strike. As you increase the cost, the time taken to generate the hash also increases. If you're generating the hash in response to a user action (like signing up or logging in to a website) then you probably want to keep the runtime to less than 500ms to avoid a negative user experience. If the Argon2 algorithm is running on a machine with multiple cores, then one way to decrease the runtime without reducing the cost is to increase the parallelism parameter. This controls the number of threads that the work is spread across. There's an important thing to note here though: changing the value of the parallelism parameter changes the output of the algorithm. So — for example — running Argon2 with a parallelism parameter of 2 will result in a different password hash to running it with a parallelism parameter of 4. Choosing Parameters Picking the right parameters for Argon2 depends heavily on the machine that the algorithm is running on, and you'll probably need to do some experimentation in order to set them appropriately. The recommended process for choosing the parameters can be paraphrased as follows: Set the parallelism and memory parameters to the largest amount you are willing to afford, bearing in mind that you probably don't want to max these out completely unless your machine is dedicated to password hashing. Increase the number of iterations until you reach your maximum runtime limit (for example, 500ms). If you're already exceeding the your maximum runtime limit with the number of iterations = 1, then you should reduce the memory parameter. Hashing Passwords Now that those explanations are out of the way let's jump into writing the code to hash a password with Argon2. First, you'll need to go get the golang.org/x/crypto/argon2 package which implements the Argon2 algorithm: $ go get golang.org/x/crypto/argon2 And you can use it to hash a specific password like so: File: main.go package main import ( "crypto/rand" "fmt" "log" "golang.org/x/crypto/argon2" ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { // Establish the parameters to use for Argon2. p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } // Pass the plaintext password and parameters to our generateFromPassword // helper function. hash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } fmt.Println(hash) } func generateFromPassword(password string, p *params) (hash []byte, err error) { // Generate a cryptographically secure random salt. salt, err := generateRandomBytes(p.saltLength) if err != nil { return nil, err } // Pass the plaintext password, salt and parameters to the argon2.IDKey // function. This will generate a hash of the password using the Argon2id // variant. hash = argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) return hash, nil } func generateRandomBytes(n uint32) ([]byte, error) { b := make([]byte, n) _, err := rand.Read(b) if err != nil { return nil, err } return b, nil } A quick note on terminology and naming. Formally, Argon2 is a key-derivation function and it produces a key derived from the provided password and salt. This derived key is our 'hashed password'. The other important thing to point out here is the generateRandomBytes() function. In this we're using Go's crypto/rand package to generate a cryptographically secure random salt, rather than using a fixed salt or a pseudo-random salt. If you run the program at this point it should print a slice containing the bytes of the hashed password, similar to this: $ go run main.go [9 18 35 54 101 221 120 189 57 241 229 248 140 1 102 58 93 211 115 49 131 162 24 50 167 142 227 198 85 186 200 248] Each time you run the program you'll see that it results in a completely different output for the same password, thanks to the addition of our random salt. Storing Passwords So, creating a hashed password with some specific parameters is straightforward enough. But in most cases you'll want to store the salt and specific parameters that you used alongside the hashed password, so that it can be reproducibly verified at a later point. The standard way to do this is to create an encoded representation of the hashed password which looks like this: $argon2id$v=19$m=65536,t=3,p=2$c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG Let's break down what this represents: $argon2id — the variant of Argon2 being used. $v=19 — the version of Argon2 being used. $m=65536,t=3,p=2 — the memory (m), iterations (t) and parallelism (p) parameters being used. $c29tZXNhbHQ — the base64-encoded salt, using standard base64-encoding and no padding. $c29tZXNhbHQ$RdescudvJCsgt3ub+b+dWRWJTmaaJObG — the base64-encoded hashed password (derived key), using standard base64-encoding and no padding. Let's update the generateHash() function so that it returns a string in this format: File: main.go package main import ( "crypto/rand" "encoding/base64" "fmt" "log" "golang.org/x/crypto/argon2" ) ... func generateFromPassword(password string, p *params) (encodedHash string, err error) { salt, err := generateRandomBytes(p.saltLength) if err != nil { return "", err } hash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Base64 encode the salt and hashed password. b64Salt := base64.RawStdEncoding.EncodeToString(salt) b64Hash := base64.RawStdEncoding.EncodeToString(hash) // Return a string using the standard encoded hash representation. encodedHash = fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s", argon2.Version, p.memory, p.iterations, p.parallelism, b64Salt, b64Hash) return encodedHash, nil } ... And if you run the code again now, the output should look similar to this: $ go run run.go $argon2id$v=19$m=65536,t=3,p=2$Woo1mErn1s7AHf96ewQ8Uw$D4TzIwGO4XD2buk96qAP+Ed2baMo/KbTRMqXX00wtsU Verifying Passwords The final aspect to cover is how to verify passwords. In most cases, you'll take the encoded password hash that we've just produced and store it in a database of some kind. Then at a later point, you'll want to check whether a plaintext password provided by a user matches the one represented by the encoded password hash. In essence, the steps to do this check are: Extract the salt and parameters from the encoded password hash stored in the database. Derive the hash of the plaintext password using the exact same Argon2 variant, version, salt and parameters. Check whether this new hash is the same as the original one. You can implement this like so: File: main.go package main import ( "crypto/rand" "crypto/subtle" "encoding/base64" "errors" "fmt" "log" "strings" "golang.org/x/crypto/argon2" ) var ( ErrInvalidHash = errors.New("the encoded hash is not in the correct format") ErrIncompatibleVersion = errors.New("incompatible version of argon2") ) type params struct { memory uint32 iterations uint32 parallelism uint8 saltLength uint32 keyLength uint32 } func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } match, err := comparePasswordAndHash("password123", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... func comparePasswordAndHash(password, encodedHash string) (match bool, err error) { // Extract the parameters, salt and derived key from the encoded password // hash. p, salt, hash, err := decodeHash(encodedHash) if err != nil { return false, err } // Derive the key from the other password using the same parameters. otherHash := argon2.IDKey([]byte(password), salt, p.iterations, p.memory, p.parallelism, p.keyLength) // Check that the contents of the hashed passwords are identical. Note // that we are using the subtle.ConstantTimeCompare() function for this // to help prevent timing attacks. if subtle.ConstantTimeCompare(hash, otherHash) == 1 { return true, nil } return false, nil } func decodeHash(encodedHash string) (p *params, salt, hash []byte, err error) { vals := strings.Split(encodedHash, "$") if len(vals) != 6 { return nil, nil, nil, ErrInvalidHash } var version int _, err = fmt.Sscanf(vals[2], "v=%d", &version) if err != nil { return nil, nil, nil, err } if version != argon2.Version { return nil, nil, nil, ErrIncompatibleVersion } p = &params{} _, err = fmt.Sscanf(vals[3], "m=%d,t=%d,p=%d", &p.memory, &p.iterations, &p.parallelism) if err != nil { return nil, nil, nil, err } salt, err = base64.RawStdEncoding.DecodeString(vals[4]) if err != nil { return nil, nil, nil, err } p.saltLength = uint32(len(salt)) hash, err = base64.RawStdEncoding.DecodeString(vals[5]) if err != nil { return nil, nil, nil, err } p.keyLength = uint32(len(hash)) return p, salt, hash, nil } If you run this code now, you should get a positive match when comparing the plaintext and hashed password and see output like this: $ go run main.go Match: true If you change the plaintext password used in one of the function calls, like so: File: main.go package main ... func main() { p := &params{ memory: 64 * 1024, iterations: 3, parallelism: 2, saltLength: 16, keyLength: 32, } encodedHash, err := generateFromPassword("password123", p) if err != nil { log.Fatal(err) } // Use a different password... match, err := comparePasswordAndHash("pa$$word", encodedHash) if err != nil { log.Fatal(err) } fmt.Printf("Match: %v\n", match) } ... Then running the code should result in a negative match: $ go run main.go Match: false The complete sample code for this post is available in this gist.

• HTTP Method spoofing in Go
  May 10, 2018

  A.K.A. HTTP method overriding. As a web developer you probably already know that HTML forms only support the GET and POST HTTP methods. If you want to send a PUT, PATCH or DELETE request you need to resort to either sending a XMLHttpRequest from JavaScript (where they are supported by most major browsers) or implement a workaround in your server-side application code to support 'spoofed' HTTP methods. The de-facto workaround — which you might be familiar with if you've used frameworks like Ruby on Rails, Laravel or Express — is to include a hidden _method input in your form containing the spoofed HTTP method. A bit like this: Submit Another common workaround is to send a spoofed HTTP method in a X-HTTP-Method-Override header. So how can we support these things in a Go application? MethodOverride Middleware Intercepting and dealing with spoofed HTTP methods is the perfect task for some custom middleware. We want the middleware to: Intercept POST requests before they reach any application handlers. Check for a spoofed HTTP method, either in a _method parameter of the request body or a X-HTTP-Method-Override header. If a spoofed method exists — and is equal to "PUT", "PATCH" or "DELETE" — the current http.Request.Method value should be updated accordingly. It's pretty quick to implement: package main import ( "net/http" ) func MethodOverride(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Only act on POST requests. if r.Method == "POST" { // Look in the request body and headers for a spoofed method. // Prefer the value in the request body if they conflict. method := r.PostFormValue("_method") if method == "" { method = r.Header.Get("X-HTTP-Method-Override") } // Check that the spoofed method is a valid HTTP method and // update the request object accordingly. if method == "PUT" || method == "PATCH" || method == "DELETE" { r.Method = method } } // Call the next handler in the chain. next.ServeHTTP(w, r) }) } You can then use the middleware in your application like so: package main import ( "html/template" "io" "log" "net/http" ) const form = ` Example field Submit ` func main() { mux := http.NewServeMux() mux.HandleFunc("/", formHandler) // Wrap the servemux with the MethodOverride middleware. err := http.ListenAndServe(":4000", MethodOverride(mux)) log.Print(err) } func formHandler(w http.ResponseWriter, r *http.Request) { switch r.Method { case "GET": t, err := template.New("form").Parse(form) if err != nil { http.Error(w, err.Error(), 500) } t.Execute(w, nil) case "PUT": io.WriteString(w, "This is a PUT request") default: http.Error(w, http.StatusText(405), 405) } }

• How to build a Serverless API with Go and AWS Lambda
  Apr 10, 2018

  Earlier this year AWS announced that their Lambda service would now be providing first-class support for the Go language, which is a great step forward for any gophers (like myself) who fancy experimenting with serverless technology. So in this post I'm going to talk through how to create a HTTPS API backed by AWS Lambda, building it up step-by-step. I found there to be quite a few gotchas in the process — especially if you're not familiar the AWS permissions system — and some rough edges in the way that Lamdba interfaces with the other AWS services. But once you get your head around these it works pretty well. There's a lot of content to cover in this tutorial, so I've broken it down into the following seven steps: Setting up the AWS CLI Creating and deploying an Lambda function Hooking it up to DynamoDB Setting up the HTTPS API Working with events Deploying the API Supporting multiple actions Throughout this post we'll work towards building an API with two actions: MethodPathAction GET/books?isbn=xxxDisplay information about a book with a specific ISBN POST/booksCreate a new book Where a book is a basic JSON record which looks like this: {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} I'm keeping the API deliberately simple to avoid getting bogged-down in application-specific code, but once you've grasped the basics it's fairly clear how to extend the API to support additional routes and actions. Setting up the AWS CLI Throughout this tutorial we'll use the AWS CLI (command line interface) to configure our lambda functions and other AWS services. Installation and basic usage instructions can be found here, but if you’re using a Debian-based system like Ubuntu you can install the CLI with apt and run it using the aws command: $ sudo apt install awscli $ aws --version aws-cli/1.11.139 Python/3.6.3 Linux/4.13.0-37-generic botocore/1.6.6 Next we need to set up an AWS IAM user with programmatic access permission for the CLI to use. A guide on how to do this can be found here. For testing purposes you can attach the all-powerful AdministratorAccess managed policy to this user, but in practice I would recommend using a more restrictive policy. At the end of setting up the user you'll be given a access key ID and secret access key. Make a note of these — you’ll need them in the next step. Configure the CLI to use the credentials of the IAM user you've just created using the configure command. You’ll also need to specify the default region and output format you want the CLI to use. $ aws configure AWS Access Key ID [None]: access-key-ID AWS Secret Access Key [None]: secret-access-key Default region name [None]: us-east-1 Default output format [None]: json (Throughout this tutorial I'll assume you're using the us-east-1 region — you'll need to change the code snippets accordingly if you're using a different region.) Creating and deploying an Lambda function Now for the exciting part: making a lambda function. If you're following along, go to your $GOPATH/src folder and create a books repository containing a main.go file. $ cd ~/go/src $ mkdir books && cd books $ touch main.go Next you'll need to install the github.com/aws-lambda-go/lambda package. This provides the essential libraries and types we need for creating a lambda function in Go. $ go get github.com/aws/aws-lambda-go/lambda Then open up the main.go file and add the following code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { bk := &book{ ISBN: "978-1420931693", Title: "The Republic", Author: "Plato", } return bk, nil } func main() { lambda.Start(show) } In the main() function we call lambda.Start() and pass in the show function as the lambda handler. In this case the handler simply initializes and returns a new book object. Lamdba handlers can take a variety of different signatures and reflection is used to determine exactly which signature you're using. The full list of supported forms is… func() func() error func(TIn) error func() (TOut, error) func(TIn) (TOut, error) func(context.Context) error func(context.Context, TIn) error func(context.Context) (TOut, error) func(context.Context, TIn) (TOut, error) … where the TIn and TOut parameters are objects that can be marshaled (and unmarshalled) by Go's encoding/json package. The next step is to build an executable from the books package using go build. In the code snippet below I'm using the -o flag to save the executable to /tmp/main but you can save it to any location (and name it whatever) you wish. $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books Important: as part of this command we're using env to temporarily set two environment variables for the duration for the command (GOOS=linux and GOARCH=amd64). These instruct the Go compiler to create an executable suitable for use with a linux OS and amd64 architecture — which is what it will be running on when we deploy it to AWS. AWS requires us to upload our lambda functions in a zip file, so let's make a main.zip zip file containing the executable we just made: $ zip -j /tmp/main.zip /tmp/main Note that the executable must be in the root of the zip file — not in a folder within the zip file. To ensure this I've used the -j flag in the snippet above to junk directory names. The next step is a bit awkward, but critical to getting our lambda function working properly. We need to set up an IAM role which defines the permission that our lambda function will have when it is running. For now let's set up a lambda-books-executor role and attach the AWSLambdaBasicExecutionRole managed policy to it. This will give our lambda function the basic permissions it need to run and log to the AWS cloudwatch service. First we have to create a trust policy JSON file. This will essentially instruct AWS to allow lambda services to assume the lambda-books-executor role: File: /tmp/trust-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } Then use the aws iam create-role command to create the role with this trust policy: $ aws iam create-role --role-name lambda-books-executor \ --assume-role-policy-document file:///tmp/trust-policy.json { "Role": { "Path": "/", "RoleName": "lambda-books-executor", "RoleId": "AROAIWSQS2RVEWIMIHOR2", "Arn": "arn:aws:iam::account-id:role/lambda-books-executor", "CreateDate": "2018-04-05T10:22:32.567Z", "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "lambda.amazonaws.com" }, "Action": "sts:AssumeRole" } ] } } } Make a note of the returned ARN (Amazon Resource Name) — you'll need this in the next step. Now the lambda-books-executor role has been created we need to specify the permissions that the role has. The easiest way to do this it to use the aws iam attach-role-policy command, passing in the ARN of AWSLambdaBasicExecutionRole permission policy like so: $ aws iam attach-role-policy --role-name lambda-books-executor \ --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole Note: you can find a list of other permission policies that might be useful here. Now we're ready to actually deploy the lambda function to AWS, which we can do using the aws lambda create-function command. This takes the following flags and can take a minute or two to run. --function-name Thethat name your lambda function will be called within AWS --runtime The runtime environment for the lambda function (in our case "go1.x") --role The ARN of the role you want the lambda function to assume when it is running (from step 6 above) --handler The name of the executable in the root of the zip file --zip-file Path to the zip file Go ahead and try deploying it: $ aws lambda create-function --function-name books --runtime go1.x \ --role arn:aws:iam::account-id:role/lambda-books-executor \ --handler main --zip-file fileb:///tmp/main.zip { "FunctionName": "books", "FunctionArn": "arn:aws:lambda:us-east-1:account-id:function:books", "Runtime": "go1.x", "Role": "arn:aws:iam::account-id:role/lambda-books-executor", "Handler": "main", "CodeSize": 2791699, "Description": "", "Timeout": 3, "MemorySize": 128, "LastModified": "2018-04-05T10:25:05.343+0000", "CodeSha256": "O20RZcdJTVcpEiJiEwGL2bX1PtJ/GcdkusIEyeO9l+8=", "Version": "$LATEST", "TracingConfig": { "Mode": "PassThrough" } } So there it is. Our lambda function has been deployed and is now ready to use. You can try it out by using the aws lambda invoke command (which requires you to specify an output file for the response — I've used /tmp/output.json in the snippet below). $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} If you're following along hopefully you've got the same response. Notice how the book object we initialized in our Go code has been automatically marshaled to JSON? Hooking it up to DynamoDB In this section we're going to add a persistence layer for our data which can be accessed by our lambda function. For this I'll use Amazon DynamoDB (it integrates nicely with AWS lambda and has a generous free-usage tier). If you're not familiar with DynamoDB, there's a decent run down of the basics here. The first thing we need to do is create a Books table to hold the book records. DynanmoDB is schema-less, but we do need to define the partion key (a bit like a primary key) on the ISBN field. We can do this in one command like so: $ aws dynamodb create-table --table-name Books \ --attribute-definitions AttributeName=ISBN,AttributeType=S \ --key-schema AttributeName=ISBN,KeyType=HASH \ --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 { "TableDescription": { "AttributeDefinitions": [ { "AttributeName": "ISBN", "AttributeType": "S" } ], "TableName": "Books", "KeySchema": [ { "AttributeName": "ISBN", "KeyType": "HASH" } ], "TableStatus": "CREATING", "CreationDateTime": 1522924177.507, "ProvisionedThroughput": { "NumberOfDecreasesToday": 0, "ReadCapacityUnits": 5, "WriteCapacityUnits": 5 }, "TableSizeBytes": 0, "ItemCount": 0, "TableArn": "arn:aws:dynamodb:us-east-1:account-id:table/Books" } } Then lets add a couple of items using the put-item command, which we'll use in the next steps. $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-1420931693"}, "Title": {"S": "The Republic"}, "Author": {"S": "Plato"}}' $ aws dynamodb put-item --table-name Books --item '{"ISBN": {"S": "978-0486298238"}, "Title": {"S": "Meditations"}, "Author": {"S": "Marcus Aurelius"}}' The next thing to do is update our Go code so that our lambda handler can connect to and use the DynamoDB layer. For this you'll need to install the github.com/aws/aws-sdk-go package which provides libraries for working with DynamoDB (and other AWS services). $ go get github.com/aws/aws-sdk-go Now for the code. To keep a bit of separation create a new db.go file in your books repository: $ touch ~/go/src/books/db.go And add the following code: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) // Declare a new DynamoDB instance. Note that this is safe for concurrent // use. var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { // Prepare the input for the query. input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } // Retrieve the item from DynamoDB. If no matching item is found // return nil. result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } // The result.Item object returned has the underlying type // map[string]*AttributeValue. We can use the UnmarshalMap helper // to parse this straight into the fields of a struct. Note: // UnmarshalListOfMaps also exists if you are working with multiple // items. bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } And then update the main.go to use this new code: File: books/main.gopackage main import ( "github.com/aws/aws-lambda-go/lambda" ) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show() (*book, error) { // Fetch a specific book record from the DynamoDB database. We'll // make this more dynamic in the next section. bk, err := getItem("978-0486298238") if err != nil { return nil, err } return bk, nil } func main() { lambda.Start(show) } Save the files, then rebuild and zip up the lambda function so it's ready to deploy: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main Re-deploying a lambda function is easier than creating it for the first time — we can use the aws lambda update-function-code command like so: $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip Let's try executing the lambda function now: $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200, "FunctionError": "Unhandled" } $ cat /tmp/output.json {"errorMessage":"AccessDeniedException: User: arn:aws:sts::account-id:assumed-role/lambda-books-executor/books is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:us-east-1:account-id:table/Books\n\tstatus code: 400, request id: 2QSB5UUST6F0R3UDSVVVODTES3VV4KQNSO5AEMVJF66Q9ASUAAJG","errorType":"requestError"} Ah. There's a slight problem. We can see from the output message that our lambda function (specifically, the lambda-books-executor role) doesn't have the necessary permissions to run GetItem on a DynamoDB instance. Let's fix that now. Create a privilege policy file that gives GetItem and PutItem privileges on DynamoDB like so: File: /tmp/privilege-policy.json{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "dynamodb:PutItem", "dynamodb:GetItem", ], "Resource": "*" } ] } And then attach it to the lambda-books-executor role using the aws iam put-role-policy command: $ aws iam put-role-policy --role-name lambda-books-executor \ --policy-name dynamodb-item-crud-role \ --policy-document file:///tmp/privilege-policy.json As a side note, AWS has some managed policies called AWSLambdaDynamoDBExecutionRole and AWSLambdaInvocation-DynamoDB which sound like they would do the trick. But neither of them actually provide GetItem or PutItem privileges. Hence the need to roll our own policy. Let's try executing the lambda function again. It should work smoothly this time and return information about the book with ISBN 978-0486298238. $ aws lambda invoke --function-name books /tmp/output.json { "StatusCode": 200 } $ cat /tmp/output.json {"isbn":"978-0486298238","title":"Meditations","author":"Marcus Aurelius"} Setting up the HTTPS API So our lambda function is now working nicely and communicating with DynamoDB. The next thing to do is set up a way to access the lamdba function over HTTPS, which we can do using the AWS API Gateway service. But before we go any further, it's worth taking a moment to think about the structure of our project. Let's say we have grand plans for our lamdba function to be part of a bigger bookstore API which deals with information about books, customers, recommendations and other things. There's three basic options for structuring this using AWS Lambda: Microservice style — Each lambda function is responsible for one action only. For example, there are 3 separate lambda functions for showing, creating and deleting a book. Service style — Each lambda function is responsible for a group of related actions. For example, one lambda function handles all book-related actions, but customer-related actions are kept in a separate lambda function. Monolith style — One lambda function manages all the bookstore actions. Each of these options is valid, and theres some good discussion of the pros and cons here. For this tutorial we'll opt for a service style, and have one books lambda function handle the different book-related actions. This means that we'll need to implement some form of routing within our lambda function, which I'll cover later in the post. But for now… Go ahead and create a bookstore API using the aws apigateway create-rest-api command like so: $ aws apigateway create-rest-api --name bookstore { "id": "rest-api-id", "name": "bookstore", "createdDate": 1522926250 } Note down the rest-api-id value that this returns, we'll be using it a lot in the next few steps. Next we need to get the id of the root API resource ("/"). We can retrieve this using the aws apigateway get-resources command like so: $ aws apigateway get-resources --rest-api-id rest-api-id { "items": [ { "id": "root-path-id", "path": "/" } ] } Again, keep a note of the root-path-id value this returns. Now we need to create a new resource under the root path — specifically a resource for the URL path /books. We can do this by using the aws apigateway create-resource command with the --path-part parameter like so: $ aws apigateway create-resource --rest-api-id rest-api-id \ --parent-id root-path-id --path-part books { "id": "resource-id", "parentId": "root-path-id", "pathPart": "books", "path": "/books" } Again, note the resource-id this returns, we'll need it in the next step. Note that it's possible to include placeholders within your path by wrapping part of the path in curly braces. For example, a --path-part parameter of books/{id} would match requests to /books/foo and /books/bar, and the value of id would be made available to your lambda function via an events object (which we'll cover later in the post). You can also make a placeholder greedy by postfixing it with a +. A common idiom is to use the parameter --path-part {proxy+} if you want to match all requests regardless of their path. But we're not doing either of those things. Let's get back to our /books resource and use the aws apigateway put-method command to register the HTTP method of ANY. This will mean that our /books resource will respond to all requests regardless of their HTTP method. $ aws apigateway put-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY \ --authorization-type NONE { "httpMethod": "ANY", "authorizationType": "NONE", "apiKeyRequired": false } Now we're all set to integrate the resource with our lambda function, which we can do using the aws apigateway put-integration command. This command has a few parameters that need a quick explanation: The --type parameter should be AWS_PROXY. When this is used the AWS API Gateway will send information about the HTTP request as an 'event' to the lambda function. It will also automatically transform the output from the lambda function to a HTTP response. The --integration-http-method parameter must be POST. Don't confuse this with what HTTP methods your API resource responds to. The --uri parameter needs to take the format: arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/your-lambda-function-arn/invocations With those things in mind, your command should look a bit like this: $ aws apigateway put-integration --rest-api-id rest-api-id \ --resource-id resource-id --http-method ANY --type AWS_PROXY \ --integration-http-method POST \ --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations { "type": "AWS_PROXY", "httpMethod": "POST", "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations", "passthroughBehavior": "WHEN_NO_MATCH", "cacheNamespace": "qtdn5h", "cacheKeyParameters": [] } Alright, let's give this a whirl. We can send a test request to the resource we just made using the aws apigateway test-invoke-method command like so: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 500, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:07:54 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:07:54 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:07:54 UTC 2018 : Method request path: {}[TRUNCATED]Thu Apr 05 11:07:54 UTC 2018 : Sending request to https://lambda.us-east-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:us-east-1:account-id:function:books/invocations\nThu Apr 05 11:07:54 UTC 2018 : Execution failed due to configuration error: Invalid permissions on Lambda function\nThu Apr 05 11:07:54 UTC 2018 : Method completed with status: 500\n", "latency": 39 } Ah. So that hasn't quite worked. If you take a look through the outputted log information you should see that the problem appears to be: Execution failed due to configuration error: Invalid permissions on Lambda function This is happening because our bookstore API gateway doesn't have permissions to execute our lambda function. The easiest way to fix that is to use the aws lambda add-permission command to give our API permissions to invoke it, like so: $ aws lambda add-permission --function-name books --statement-id a-GUID \ --action lambda:InvokeFunction --principal apigateway.amazonaws.com \ --source-arn arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/* { "Statement": "{\"Sid\":\"6d658ce7-3899-4de2-bfd4-fefb939f731\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"apigateway.amazonaws.com\"},\"Action\":\"lambda:InvokeFunction\",\"Resource\":\"arn:aws:lambda:us-east-1:account-id:function:books\",\"Condition\":{\"ArnLike\":{\"AWS:SourceArn\":\"arn:aws:execute-api:us-east-1:account-id:rest-api-id/*/*/*\"}}}" } Note that the --statement-id parameter needs to be a globally unique identifier. This could be a random ID or something more descriptive. Alright, let's try again: $ aws apigateway test-invoke-method --rest-api-id rest-api-id --resource-id resource-id --http-method "GET" { "status": 502, "body": "{\"message\": \"Internal server error\"}", "headers": {}, "log": "Execution log for request test-request\nThu Apr 05 11:12:53 UTC 2018 : Starting execution for request: test-invoke-request\nThu Apr 05 11:12:53 UTC 2018 : HTTP Method: GET, Resource Path: /books\nThu Apr 05 11:12:53 UTC 2018 : Method request path: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request query string: {}\nThu Apr 05 11:12:53 UTC 2018 : Method request headers: {}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"}\nThu Apr 05 11:12:53 UTC 2018 : Endpoint response headers: {X-Amz-Executed-Version=$LATEST, x-amzn-Remapped-Content-Length=0, Connection=keep-alive, x-amzn-RequestId=48d29098-38c2-11e8-ae15-f13b670c5483, Content-Length=74, Date=Thu, 05 Apr 2018 11:12:53 GMT, X-Amzn-Trace-Id=root=1-5ac604b5-cf29dd70cd08358f89853b96;sampled=0, Content-Type=application/json}\nThu Apr 05 11:12:53 UTC 2018 : Execution failed due to configuration error: Malformed Lambda proxy response\nThu Apr 05 11:12:53 UTC 2018 : Method completed with status: 502\n", "latency": 211 } So unfortunately there's still an error, but the message has now changed: Execution failed due to configuration error: Malformed Lambda proxy response And if you look closely at the output you'll see the information: Endpoint response body before transformations: {\"isbn\":\"978-0486298238\",\"title\":\"Meditations\",\"author\":\"Marcus Aurelius\"} So there's some definite progress here. Our API is talking to our lambda function and is receiving the correct response (a book object marshalled to JSON). It's just that the AWS API Gateway considers the response to be in the wrong format. This is because, when you're using the API Gateway's lambda proxy integration, the return value from the lambda function must be in the following JSON format: { "isBase64Encoded": true|false, "statusCode": httpStatusCode, "headers": { "headerName": "headerValue", ... }, "body": "..." } So to fix this it's time to head back to our Go code and make some alterations. Working with events The easiest way to provide the responses that the AWS API Gateway needs is to install the github.com/aws/aws-lambda-go/events package: go get github.com/aws/aws-lambda-go/events This provides a couple of useful types (APIGatewayProxyRequest and APIGatewayProxyResponse) which contain information about incoming HTTP requests and allow us to construct responses that the API Gateway understands. type APIGatewayProxyRequest struct { Resource string `json:"resource"` // The resource path defined in API Gateway Path string `json:"path"` // The url path for the caller HTTPMethod string `json:"httpMethod"` Headers map[string]string `json:"headers"` QueryStringParameters map[string]string `json:"queryStringParameters"` PathParameters map[string]string `json:"pathParameters"` StageVariables map[string]string `json:"stageVariables"` RequestContext APIGatewayProxyRequestContext `json:"requestContext"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } type APIGatewayProxyResponse struct { StatusCode int `json:"statusCode"` Headers map[string]string `json:"headers"` Body string `json:"body"` IsBase64Encoded bool `json:"isBase64Encoded,omitempty"` } Let's go back to our main.go file and update our lambda handler so that it uses the signature: func(events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) Essentially, the handler will accept a APIGatewayProxyRequest object which contains a bunch of information about the HTTP request, and return a APIGatewayProxyResponse object (which is marshalable into a JSON response suitable for the AWS API Gateway). File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { // Get the `isbn` query string parameter from the request and // validate it. isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } // Fetch the book record from the database based on the isbn value. bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } // The APIGatewayProxyResponse.Body field needs to be a string, so // we marshal the book record into JSON. js, err := json.Marshal(bk) if err != nil { return serverError(err) } // Return a response with a 200 OK status and the JSON book record // as the body. return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } // Add a helper for handling errors. This logs any error to os.Stderr // and returns a 500 Internal Server Error response that the AWS API // Gateway understands. func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } // Similarly add a helper for send responses relating to client errors. func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(show) } Notice how in all cases the error value returned from our lambda handler is now nil? We have to do this because the API Gateway doesn't accept error objects when you're using it in conjunction with a lambda proxy integration (they would result in a 'malformed response' errors again). So we need to manage errors fully within our lambda function and return the appropriate HTTP response. In essence, this means that the return parameter of error is superfluous, but we still need to include it to have a valid signature for the lambda function. Anyway, save the file and rebuild and redeploy the lambda function: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And if you test it again now it should work as expected. Give it a try with different isbn values in the query string: $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=978-1420931693" { "status": 200, "body": "{\"isbn\":\"978-1420931693\",\"title\":\"The Republic\",\"author\":\"Plato\"}", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60df0-0ea7a560337129d1fde588cd" }, "log": [TRUNCATED], "latency": 1232 } $ aws apigateway test-invoke-method --rest-api-id rest-api-id \ --resource-id resource-id --http-method "GET" \ --path-with-query-string "/books?isbn=foobar" { "status": 400, "body": "Bad Request", "headers": { "X-Amzn-Trace-Id": "sampled=0;root=1-5ac60e1c-72fad7cfa302fd32b0a6c702" }, "log": [TRUNCATED], "latency": 25 } As a side note, anything sent to os.Stderr will be logged to the AWS Cloudwatch service. So if you've set up an error logger like we have in the code above, you can query Cloudwatch for errors like so: $ aws logs filter-log-events --log-group-name /aws/lambda/books \ --filter-pattern "ERROR" Deploying the API Now that the API Gateway is working properly it's time to make it live. We can do this with the aws apigateway create-deployment command like so: $ aws apigateway create-deployment --rest-api-id rest-api-id \ --stage-name staging { "id": "4pdblq", "createdDate": 1522929303 } In the code above I've given the deployed API using the name staging, but you can call it anything that you wish. Once deployed your API should be accessible at the URL: https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging Go ahead and give it a try using curl. It should work as you expect: $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-1420931693 {"isbn":"978-1420931693","title":"The Republic","author":"Plato"} $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=foobar Bad Request Supporting multiple actions Let's add support for a POST /books action. We want this to read and validate a new book record (from a JSON HTTP request body) and then add it to the DynamoDB table. Now that the different AWS services are hooked up, extending our lambda function to support additional actions is perhaps the most straightforward part of this tutorial, as it can be managed purely within our Go code. First update the db.go file to include a new putItem function like so: File: books/db.gopackage main import ( "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/aws/session" "github.com/aws/aws-sdk-go/service/dynamodb" "github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute" ) var db = dynamodb.New(session.New(), aws.NewConfig().WithRegion("us-east-1")) func getItem(isbn string) (*book, error) { input := &dynamodb.GetItemInput{ TableName: aws.String("Books"), Key: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(isbn), }, }, } result, err := db.GetItem(input) if err != nil { return nil, err } if result.Item == nil { return nil, nil } bk := new(book) err = dynamodbattribute.UnmarshalMap(result.Item, bk) if err != nil { return nil, err } return bk, nil } // Add a book record to DynamoDB. func putItem(bk *book) error { input := &dynamodb.PutItemInput{ TableName: aws.String("Books"), Item: map[string]*dynamodb.AttributeValue{ "ISBN": { S: aws.String(bk.ISBN), }, "Title": { S: aws.String(bk.Title), }, "Author": { S: aws.String(bk.Author), }, }, } _, err := db.PutItem(input) return err } And then update the main.go function so that the lambda.Start() method calls a new router function, which does a switch on the HTTP request method to determine which action to take. Like so: File: books/main.gopackage main import ( "encoding/json" "fmt" "log" "net/http" "os" "regexp" "github.com/aws/aws-lambda-go/events" "github.com/aws/aws-lambda-go/lambda" ) var isbnRegexp = regexp.MustCompile(`[0-9]{3}\-[0-9]{10}`) var errorLogger = log.New(os.Stderr, "ERROR ", log.Llongfile) type book struct { ISBN string `json:"isbn"` Title string `json:"title"` Author string `json:"author"` } func router(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { switch req.HTTPMethod { case "GET": return show(req) case "POST": return create(req) default: return clientError(http.StatusMethodNotAllowed) } } func show(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { isbn := req.QueryStringParameters["isbn"] if !isbnRegexp.MatchString(isbn) { return clientError(http.StatusBadRequest) } bk, err := getItem(isbn) if err != nil { return serverError(err) } if bk == nil { return clientError(http.StatusNotFound) } js, err := json.Marshal(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: http.StatusOK, Body: string(js), }, nil } func create(req events.APIGatewayProxyRequest) (events.APIGatewayProxyResponse, error) { if req.Headers["Content-Type"] != "application/json" { return clientError(http.StatusNotAcceptable) } bk := new(book) err := json.Unmarshal([]byte(req.Body), bk) if err != nil { return clientError(http.StatusUnprocessableEntity) } if !isbnRegexp.MatchString(bk.ISBN) { return clientError(http.StatusBadRequest) } if bk.Title == "" || bk.Author == "" { return clientError(http.StatusBadRequest) } err = putItem(bk) if err != nil { return serverError(err) } return events.APIGatewayProxyResponse{ StatusCode: 201, Headers: map[string]string{"Location": fmt.Sprintf("/books?isbn=%s", bk.ISBN)}, }, nil } func serverError(err error) (events.APIGatewayProxyResponse, error) { errorLogger.Println(err.Error()) return events.APIGatewayProxyResponse{ StatusCode: http.StatusInternalServerError, Body: http.StatusText(http.StatusInternalServerError), }, nil } func clientError(status int) (events.APIGatewayProxyResponse, error) { return events.APIGatewayProxyResponse{ StatusCode: status, Body: http.StatusText(status), }, nil } func main() { lambda.Start(router) } Rebuild and zip up the lambda function, then deploy it as normal: $ env GOOS=linux GOARCH=amd64 go build -o /tmp/main books $ zip -j /tmp/main.zip /tmp/main $ aws lambda update-function-code --function-name books \ --zip-file fileb:///tmp/main.zip And now when you hit the API using different HTTP methods it should call the appropriate action: $ curl -i -H "Content-Type: application/json" -X POST \ -d '{"isbn":"978-0141439587", "title":"Emma", "author": "Jane Austen"}' \ https://rest-api-id.execeast-1.amazonaws.com/staging/books HTTP/1.1 201 Created Content-Type: application/json Content-Length: 7 Connection: keep-alive Date: Thu, 05 Apr 2018 14:55:34 GMT x-amzn-RequestId: 64262aa3-38e1-11e8-825c-d7cfe4d1e7d0 x-amz-apigw-id: E33T1E3eIAMF9dw= Location: /books?isbn=978-0141439587 X-Amzn-Trace-Id: sampled=0;root=1-5ac638e5-e806a84761839bc24e234c37 X-Cache: Miss from cloudfront Via: 1.1 a22ee9ab15c998bce94f1f4d2a7792ee.cloudfront.net (CloudFront) X-Amz-Cf-Id: wSef_GJ70YB2-0VSwhUTS9x-ATB1Yq8anWuzV_PRN98k9-DkD7FOAA== $ curl https://rest-api-id.execute-api.us-east-1.amazonaws.com/staging/books?isbn=978-0141439587 {"isbn":"978-0141439587","title":"Emma","author":"Jane Austen"}

• How to disable http.FileServer directory listings
  Mar 14, 2018

  A nice feature of Go's http.FileServer is that it automatically generates navigable directory listings, which look a bit like this: But for certain applications you might want to prevent this behavior and disable directory listings altogether. In this post I’m going to run through three different options for doing exactly that: Using index.html files Using middleware Using a custom filesystem Using index.html files Before http.FileServer generates a directory listing it checks for the existence of an index.html file in the directory root. If an index.html file exists, then it will respond with the contents of the file instead. So it follows that a simple way to disable directory listings is to add a blank index.html file to your root static file directory and all sub-directories, like so: . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ ├── index.html │ └── logo.png ├── index.html └── robots.txt If you've got a lot of sub-directories an easy way to do that is with a one-line command like this: $ find ./static/ -type d -exec touch {}/index.html \; Any requests for a directory should now result in an empty 200 OK response for the user, instead of a directory listing. For example: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 0 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:41:10 GMT Date: Tue, 13 Mar 2018 12:42:35 GMT Or without the trailing slash, the user should get a 301 Redirect like so: $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:43:13 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. This is a good-enough solution if you can't (or don't want to) make any changes to your Go application itself. But it's not perfect. You'll need to remember to add a blank index.html file for any new sub-directories in the future, and many people — myself included — would argue that a 403 Forbidden or 404 Not Found status would be more appropriate than sending the user an empty 200 OK response. Using middleware Both of these imperfections can be addressed if we take a different approach and implement some custom middleware to intercept requests before they reach the http.FileServer. Essentially, we want the middleware to check if the request URL ends with a / character, and if it does, return a 404 Not Found response instead of passing on the request to the http.FileServer. Here's a basic implementation: package main import ( "log" "net/http" "strings" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(http.Dir("./static")) mux.Handle("/static/", http.StripPrefix("/static", neuter(fileServer))) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } func neuter(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if strings.HasSuffix(r.URL.Path, "/") { http.NotFound(w, r) return } next.ServeHTTP(w, r) }) } This approach would result in a user getting responses like these: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:46:20 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 301 Moved Permanently Location: /static/img/ Date: Tue, 13 Mar 2018 12:46:55 GMT Content-Length: 43 Content-Type: text/html; charset=utf-8 Moved Permanently. To me, this feels like a cleaner and easier-to-maintain way to disable directory listings than using blank index.html files. But again, it's still not perfect. Firstly, requests for any directories without the trailing slash will be 301 redirected only to receive a 404 Not Found response. It's extra, unnecessary, requests for both the client and server to deal with. Secondly, if one of your directories does contain an index.html file then it won't ever be used. For example, if you have the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ... any request to http://localhost:4000/static/css/ will result in a 404 Not Found response instead of returning the contents of the /static/css/index.html file. $ curl -i http://localhost:4000/static/css/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 12:51:09 GMT Content-Length: 19 404 page not found Using a custom filesystem The final option we're going to look at is creating a custom filesystem and passing that to your http.FileServer. There are a couple of approaches described by Brad Fitzpatrick and George Armhold you might want to consider, but I would personally suggest doing something like this: package main import ( "log" "net/http" "path/filepath" ) func main() { mux := http.NewServeMux() fileServer := http.FileServer(neuteredFileSystem{http.Dir("./static")}) mux.Handle("/static", http.NotFoundHandler()) mux.Handle("/static/", http.StripPrefix("/static", fileServer)) err := http.ListenAndServe(":4000", mux) log.Fatal(err) } type neuteredFileSystem struct { fs http.FileSystem } func (nfs neuteredFileSystem) Open(path string) (http.File, error) { f, err := nfs.fs.Open(path) if err != nil { return nil, err } s, err := f.Stat() if s.IsDir() { index := filepath.Join(path, "index.html") if _, err := nfs.fs.Open(index); err != nil { closeErr := f.Close() if closeErr != nil { return nil, closeErr } return nil, err } } return f, nil } In this code we're creating a custom neuteredFileSystem type which embeds the standard http.FileSystem. We then implement an Open() method on it — which gets called each time our http.FileServer receives a request. In our Open() method we Stat() the requested file path and use the IsDir() method to check whether it's a directory or not. If it is a directory we then try to Open() any index.html file in it. If no index.html file exists, then this will return a os.ErrNotExist error (which in turn we return and it will be transformed into a 404 Not Found response by http.Fileserver). We also call Close() on the original file to avoid a file descriptor leak. Otherwise, we just return the file and let http.FileServer do its thing. Putting this to use with the directory structure... . ├── main.go └── static ├── css │ ├── index.html │ └── main.css ├── img │ └── logo.png └── robots.txt ...would result in responses like: $ curl -i http://localhost:4000/static/img/ HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:21 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/img HTTP/1.1 404 Not Found Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Tue, 13 Mar 2018 16:53:22 GMT Content-Length: 19 404 page not found $ curl -i http://localhost:4000/static/css/ HTTP/1.1 200 OK Accept-Ranges: bytes Content-Length: 37 Content-Type: text/html; charset=utf-8 Last-Modified: Tue, 13 Mar 2018 12:49:00 GMT Date: Tue, 13 Mar 2018 16:53:27 GMT This is my custom index page This is now working pretty nicely: All requests for directories (with no index.html file) return a 404 Not Found response, instead of a directory listing or a redirect. This works for requests both with and without a trailing slash. The default behavior of http.FileServer isn't changed any other way, and index.html files work as per the standard library documentation.

• How to rate limit HTTP requests in Go
  Dec 28, 2017

  If you're running a HTTP server and want to rate limit user requests, the go-to package to use is probably Tollbooth by Didip Kerabat. It's well maintained, has a good range of features and a clean and clear API. But if you want something simple and lightweight – or just want to learn – it's not too difficult to roll your own middleware to handle rate limiting. In this post I'll run through the essentials of how to do that by using the x/time/rate package, which provides a token bucket rate-limiter algorithm (note: this is also used by Tollbooth behind the scenes). If you would like to follow along, create a demo directory containing two files, limit.go and main.go, and initialize a new Go module. Like so: $ mkdir ratelimit-demo $ cd ratelimit-demo $ touch limit.go main.go $ go mod init example.com/ratelimit-demo Let's start by making a global rate limiter which acts on all the requests that a HTTP server receives. Open up the limit.go file and add the following code: File: ratelimit-demo/limit.go package main import ( "net/http" "golang.org/x/time/rate" ) var limiter = rate.NewLimiter(1, 3) func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } In this code we've used the rate.NewLimiter() function to initialize and return a new rate limiter. Its signature looks like this: func NewLimiter(r Limit, b int) *Limiter From the documentation: A Limiter controls how frequently events are allowed to happen. It implements a "token bucket" of size b, initially full and refilled at rate r tokens per second. Or to describe it another way – the limiter permits you to consume an average of r tokens per second, with a maximum of b tokens in any single 'burst'. So in the code above our limiter allows 1 token to be consumed per second, with a maximum burst size of 3. In the limit middleware function we call the global limiter's Allow() method each time the middleware receives a HTTP request. If there are no tokens left in the bucket Allow() will return false and we send the user a 429 Too Many Requests response. Otherwise, calling Allow() will consume exactly one token from the bucket and we pass on control to the next handler in the chain. It's important to note that the code behind the Allow() method is protected by a mutex and is safe for concurrent use. Let's put this to use. Open up the main.go file and setup a simple web server which uses the limit middleware like so: File: ratelimit-demo/main.go package main import ( "log" "net/http" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/", okHandler) // Wrap the servemux with the limit middleware. log.Print("Listening on :4000...") http.ListenAndServe(":4000", limit(mux)) } func okHandler(w http.ResponseWriter, r *http.Request) { w.Write([]byte("OK")) } Go ahead and run the application… $ go run . And if you make enough requests in quick succession, you should eventually get a response which looks like this: $ curl -i localhost:4000 HTTP/1.1 429 Too Many Requests Content-Type: text/plain; charset=utf-8 X-Content-Type-Options: nosniff Date: Thu, 21 Dec 2017 19:25:52 GMT Content-Length: 18 Too Many Requests Rate limiting per user While having a single, global, rate limiter is useful in some cases, another common scenario is implement a rate limiter per user, based on an identifier like IP address or API key. In this post we'll use IP address as the identifier. A conceptually straightforward way to do this is to create a map of rate limiters, using the identifier for each user as the map key. At this point you might think to reach for the sync.Map type that was introduced in Go 1.9. This essentially provides a concurrency-safe map, designed to be accessed from multiple goroutines without the risk of race conditions. But it comes with a note of caution: It is optimized for use in concurrent loops with keys that are stable over time, and either few steady-state stores, or stores localized to one goroutine per key.For use cases that do not share these attributes, it will likely have comparable or worse performance and worse type safety than an ordinary map paired with a read-write mutex. In our particular use-case the map keys will be the IP address of users, and so new keys will be added to the map each time a new user visits our application. We'll also want to prevent undue memory consumption by removing old entries from the map when a user hasn't been seen for a long period of time. So in our case the map keys won't be stable and it's likely that an ordinary map protected by a mutex will perform better. (If you're not familiar with the idea of mutexes or how to use them in Go, then this post has an explanation which you might want to read before continuing). Let's update the limit.go file to contain a basic implementation. I'll keep the code structure deliberately simple. File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "golang.org/x/time/rate" ) // Create a map to hold the rate limiters for each visitor and a mutex. var visitors = make(map[string]*rate.Limiter) var mu sync.Mutex // Retrieve and return the rate limiter for the current visitor if it // already exists. Otherwise create a new rate limiter and add it to // the visitors map, using the IP address as the key. func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() limiter, exists := visitors[ip] if !exists { limiter = rate.NewLimiter(1, 3) visitors[ip] = limiter } return limiter } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { // Get the IP address for the current user. ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } // Call the getVisitor function to retreive the rate limiter for // the current user. limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Removing old entries from the map There's one problem with this: as long as the application is running the visitors map will continue to grow unbounded. We can fix this fairly simply by recording the last seen time for each visitor and running a background goroutine to delete old entries from the map (and therefore free up memory as we go). File: ratelimit-demo/limit.go package main import ( "log" "net" "net/http" "sync" "time" "golang.org/x/time/rate" ) // Create a custom visitor struct which holds the rate limiter for each // visitor and the last time that the visitor was seen. type visitor struct { limiter *rate.Limiter lastSeen time.Time } // Change the the map to hold values of the type visitor. var visitors = make(map[string]*visitor) var mu sync.Mutex // Run a background goroutine to remove old entries from the visitors map. func init() { go cleanupVisitors() } func getVisitor(ip string) *rate.Limiter { mu.Lock() defer mu.Unlock() v, exists := visitors[ip] if !exists { limiter := rate.NewLimiter(1, 3) // Include the current time when creating a new visitor. visitors[ip] = &visitor{limiter, time.Now()} return limiter } // Update the last seen time for the visitor. v.lastSeen = time.Now() return v.limiter } // Every minute check the map for visitors that haven't been seen for // more than 3 minutes and delete the entries. func cleanupVisitors() { for { time.Sleep(time.Minute) mu.Lock() defer mu.Unlock() for ip, v := range visitors { if time.Now().Sub(v.lastSeen) > 3*time.Minute { delete(visitors, ip) } } } } func limit(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { ip, _, err := net.SplitHostPort(r.RemoteAddr) if err != nil { log.Print(err.Error()) http.Error(w, "Internal Server Error", http.StatusInternalServerError) return } limiter := getVisitor(ip) if limiter.Allow() == false { http.Error(w, http.StatusText(429), http.StatusTooManyRequests) return } next.ServeHTTP(w, r) }) } Some more improvements… For simple applications this code will work fine as-is, but you may want to adapt it further depending on your needs. For example, it might make sense to: Check the X-Forwarded-For or X-Real-IP headers for the IP address, if you are running your server behind a reverse proxy. Port the code to a standalone package. Make the rate limiter and cleanup settings configurable at runtime. Remove the reliance on global variables, so that different rate limiters can be created with different settings. Switch to a sync.RWMutex to help reduce contention on the map.

• Validation snippets for Go
  Aug 05, 2017

  Over the past few years I've built up a collection of snippets for validating inputs in Go. There's nothing new or groundbreaking here, but hopefully they might save you some time. The snippets assume that the data to validate is stored as strings in r.Form, but the principles are the same no matter where the data has come from. Required inputs Blank text Min and max length (bytes) Min and max length (number of characters) Starts with, ends with and contains Matches regular expression pattern Unicode character range Email validation URL validation Integers Floats Date Datetime-local Radio, Select and Datalist (one-in-set) Checkboxes (many-in-set) Single checkbox Required inputs If you have the HTML form: You can verify that a value for the "foo" field has been submitted with: if r.Form.Get("foo") == "" { fmt.Println("error: foo is required") } For checkbox and select inputs this will ensure that at least one item has been checked. Blank text If you have the HTML form: You can verify that a value for the "foo" field isn't blank (i.e. contains whitespace only) with the strings.TrimSpace function: import "strings" ··· if strings.TrimSpace(r.Form.Get("foo")) == "" { fmt.Println("error: foo must not be blank") } Min and max length (bytes) If you have the HTML form: You can verify that the "foo" field contains a certain number of bytes with the builtin len function: l := len(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 bytes long") } Min and max length (number of characters) If you have the HTML form: You can verify that the "foo" field contains a certain number of characters with the utf8.RuneCountInString function. This is subtly different to checking the number of bytes. For example, the string "Zoë" contains 3 characters but is 4 bytes long because of the accented character. import "unicode/utf8" ··· l := utf8.RuneCountInString(r.Form.Get("foo")) if l 10 { fmt.Println("error: foo must be between 5 and 10 characters long") } Starts with, ends with and contains If you have the HTML form: You can verify that the "foo" field starts with, ends with, or contains a particular string using the functions in the strings package: import "strings" ··· // Check that the field value starts with 'abc'. if !strings.HasPrefix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not start with 'abc'") } // Check that the field value ends with 'abc'. if !strings.HasSuffix(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not end with 'abc'") } // Check that the field value contains 'abc' anywhere in it. if !strings.Contains(r.Form.Get("foo"), "abc") { fmt.Println("error: foo does not contain 'abc'") } Matches regular expression pattern If you have the HTML form: You can verify that the "foo" field matches a particular regular expression using the regexp package. For example, to check that it matches the pattern ^[a-z]{4}\.[0-9]{2}$ (four lowercase letters followed by a period and two digits): import "regexp" ··· // Pre-compiling the regular expression and storing it in a variable is more efficient // if you're going to use it multiple times. The regexp.MustCompile function will // panic on failure. var rxPat = regexp.MustCompile(`^[a-z]{4}.[0-9]{2}$`) if !rxPat.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo does not match the required pattern") } Note that because the dot character has a special meaning in regular expressions, we escaped it using the \ character so it is interpreted as a literal period character instead. In the example above we also used a raw string for the regular expression. If you use an interpreted string (i.e. a string surrounded by double quotes), you need to escape the backslash too because that's the escape character for interpreted strings. So you would need to write: var rxPat = regexp.MustCompile("^[a-z]{4}\.[0-9]{2}$") If you're not familiar with regular expressions then this guide from Mozilla is a good explanation. Unicode character range If you have the HTML form: You can verify that the "foo" field only contains characters in a certain unicode range using the regexp package. For example, to check that it contains only Cyrillic characters in the two unicode blocks 0400 - 04FF (Cyrillic) and 0500 - 052F (Cyrillic Supplementary): import "regexp" ··· // Use an interpreted string and the \u escape notation to create a regular // expression matching the range of characters in the two unicode code blocks. var rxCyrillic = regexp.MustCompile("^[\u0400-\u04FF\u0500-\u052F]+$") if !rxCyrillic.MatchString(r.Form.Get("foo")) { fmt.Println("error: foo must only contain Cyrillic characters") } Email validation If you have the HTML form: You can sanity check that the "foo" field contains an email address using the regexp package. Choosing a regular expression to use for email validation is a contentious topic, but as a starting point I would suggest the pattern recommended by the W3C and Web Hypertext Application Technology Working Group. In addition, the email addresses have a practical limit of 254 bytes. Putting those together, a decent sanity check is: import "regexp" ··· var rxEmail = regexp.MustCompile("^[a-zA-Z0-9.!#$%&'*+\\/=?^_`{|}~-]+@[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?(?:\\.[a-zA-Z0-9](?:[a-zA-Z0-9-]{0,61}[a-zA-Z0-9])?)*$") e := r.Form.Get("foo") if len(e) > 254 || !rxEmail.MatchString(e) { fmt.Println("error: foo is not a valid email address") } Note that we have to use an interpreted string for the regular expression because it contains a backtick character (which means we can't use a raw string). URL validation If you have the HTML form: You can verify that the "foo" field contains a valid URL by first parsing it with the url.Parse function. This will take a URL string, break it into it's component pieces, and store it as a url.URL struct. You can then sanity check the component pieces as necessary. For instance, to check that a URL is absolute (i.e has both a scheme and host) and that the scheme is either http or https: import "net/url" ··· // If there are any major problems with the format of the URL, url.Parse() will // return an error. u, err := url.Parse(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo is not a valid URL") } else if u.Scheme == "" || u.Host == "" { fmt.Println("error: foo must be an absolute URL") } else if u.Scheme != "http" && u.Scheme != "https" { fmt.Println("error: foo must begin with http or https") } Integers If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.Atoi function. You can then sanity check the integer as necessary. For instance, to check that an integer is a multiple of 5 between 0 and 100: import "strconv" ··· n, err := strconv.Atoi(r.Form.Get("foo")) if err != nil { fmt.Println("error: foo must be an integer") } else if n 10 { fmt.Printf("error: foo must be between 0 and 100") } else if n%5 != 0 { fmt.Println("error: foo must be an multiple of 5") } Floats If you have the HTML form: You can verify that the "foo" field contains an integer by parsing it with the strconv.ParseFloat function. You can then sanity check the float as necessary. For instance, to check that an float is a between 0 and 1: import "strconv" n, err := strconv.ParseFloat(r.Form.Get("foo"), 64) if err != nil { fmt.Println("error: foo must be a float") } else if n 1 { fmt.Printf("error: foo must be between 0 and 1") } Date If you have the HTML form: You can verify that the "foo" field contains a valid date by parsing it with the time.Parse function. It will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. If you're not familiar with time.Parse, it converts strings with a given format into a time.Time object. You specify what the format is by passing the reference time (Mon Jan 2 15:04:05 -0700 MST 2006) as the first parameter, laid-out in the format you want. If you are expecting a date in the format YYYY-MM-DD then the reference time is 2006-01-02. You can then use the various functions in the time package to sanity check the date as necessary. For instance, to check that an date is valid and between 2017-01-01 and 2017-12-31: import "time" ··· d, err := time.Parse("2006-01-02", r.Form.Get("foo")) if err != nil { fmt.Printf("error: foo is not a valid date") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 and 2017-12-31") } Datetime-local If you have the HTML form: You can verify that the "foo" field contains a valid datetime for a specific location by parsing it with the time.ParseInLocation function. This will return an error if the date is not real: any day of month larger than 31 is rejected, as is February 29 in non-leap years, February 30, February 31, April 31, June 31, September 31, and November 31. For instance, to check that an datetime for the "Europe/Vienna" timezone is valid and between 2017-01-01 00:00 and 2017-12-31 23:59: import "time" ··· // Load the users local time zone. This accepts a location name corresponding // to a file in your IANA Time Zone database. loc, err := time.LoadLocation("Europe/Vienna") if err != nil { ··· } d, err := time.ParseInLocation("2006-01-02T15:04:05", r.Form.Get("foo"), loc) if err != nil { fmt.Printf("error: foo is not a valid datetime") } else if d.Year() != 2017 { fmt.Printf("error: foo is not between 2017-01-01 00:00:00 and 2017-12-31 23:59:00") } The time zone database needed by LoadLocation may not be present on all systems, especially non-Unix systems. LoadLocation looks in the directory or uncompressed zip file named by the ZONEINFO environment variable, if any, then looks in known installation locations on Unix systems, and finally looks in $GOROOT/lib/time/zoneinfo.zip. Radio, Select and Datalist (one-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble You can check that the submitted value for the "foo" field is one of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} if !set[r.Form.Get("foo")] { fmt.Printf("error: foo not match 'wibble', 'wobble' or 'wubble'") } Checkboxes (many-in-a-set) validation If you have the HTML form: Wibble Wobble Wubble To validate these, and make sure that all values sent by the form are either wibble, wobble or wubble, we need to access the underlying form data directly and range over each value: You can check that the submitted values for the "foo" field are part of a known set like this: set := map[string]bool{"wibble": true, "wobble": true, "wubble": true} for _, f := range r.Form["foo"] { if !set[f] { fmt.Printf("error: foo does not match 'wibble', 'wobble' or 'wubble'") break } } Single checkboxes Sometimes you might have a single checkbox, and you want to verify that it has been checked. A common example is an "I accept the terms" checkbox on a form. If you have the HTML form: I accept the terms You can verify that it has been checked like this: if r.Form.Get("foo") != "checked" { fmt.Println("foo must be checked") }

Bob Nystrom

• Crafting “Crafting Interpreters”
  Apr 05, 2020

  It took three years and 200,000 words more than I expected, but my second book, Crafting Interpreters, is complete. I finished the third draft of the last chapter today, marking the last of around 1,400 days of continuous writing. This book was much harder than my first book, along every axis. It’s larger, more technically complex, much more deeply intertwined, and it had the misfortune of aligning with a really difficult period in my life. Today feels less like coasting past the finish line at the Tour de France, arms raised in triumph, and more like dragging myself onto the beach, clutching sand in relief after a storm-thrashed ordeal at sea. Before I get into all that, I have a minor confession to make. When I finished my first book, I wrote a long post about how I cobbled together enough willpower to reach the end of the last page. Everything in there is true, but there is one fact I superstitiously omitted. Halfway through writing Game Programming Patterns, I discovered a new passion: programming languages. It had been a long time since a topic ignited my brain to the same degree, and I was on fire. I spent basically every free hour (and many not-so-free hours—sorry, family) designing and hacking on programming languages. I read every book I could get my hands on, went to conferences, blogged, I even dreamed about programming languages. This infatuation was the main reason I stopped working on my first book for two years. I have a personality quirk where when I’m excited about something I just have to teach it to other people. Hermione Granger, arm waving feverishly to get the teacher’s attention, is my spirit animal. It was inevitable that I would write something about interpreters. But I couldn’t just drop one half-finished book to start another. I have gigs of unfinished projects laying around, but—maybe because the completed chapters were already online—I couldn’t bear to abandon Game Programming Patterns. So I made a promise to myself. If I finished that book, then I would let myself write a second book on interpreters. In part because of that promise, I did manage to complete the chapters, and then the print and e-book editions. What I thought was merely a hobby and personal goal turned out to be a life-changing experience. My little self-published vanity project has hundreds of five-star reviews, and has been translated to Korean, Japanese, Chinese, German, and Polish. The book did so much better than I expected that I’m still not sure how to process it, beyond feeling immense gratitude to everyone who read it, bought a copy, or cheered me on. The seed of a book#the-seed-of-a-book Once I finished the print edition of Game Programming Patterns, I took some time off. But it didn’t take too long for that itch to write about interpreters to come back. I knew exactly what I was getting into with writing a book now, how hard the grind can be. At first, I just noodled around. I wasn’t committed to doing anything. It was more a sort of recreational intellectual exercise. If I were to do a book, what would it look like? You know, hypothetically speaking. The very first note I wrote to myself said: high-level goal: a *small* book that builds a complete, efficient interpreter. instead of a wide text about programming language*s*, it is a single path through the language space. aim for 60k words. My first book was about 90,000 words, and I didn’t want to hike a trail that long again. I also had a meta-goal to make programming languages more approachable, and I figured a short text would help. I had this vision of something you could literally hold in your hand or have open next to your laptop while you followed along. To make a small book, I needed a small language and a small implementation. One of my other side projects was a scripting language named Wren. Wren is written in C, with a simple single-pass bytecode compiler inspired by Lua. Building Wren taught me how much functionality you can pack into a few thousand lines of clean C code. For this hypothetical book, I figured a bytecode VM in C like that would be a great fit. It would also give me the chance to cover a bunch of really fun topics like stack-based VMs, object representation, and garbage collection. But Wren wasn’t the right language. I like Wren (obviously), but it has some design quirks that I think make it a better language for users but maybe not for teaching. For the book, I wanted a dynamically-typed scripting language in the vein of languages like JavaScript, Python, and Lua. I started tinkering on a new toy language, tentatively named “Vox”. The goal was to keep things as simple as possible without taking any shortcuts around the hard problems in implementing a language. I wanted a rich expression and statement syntax to cover parsing. First-class functions and closures because they are powerful and challenging to implement efficiently. Classes and methods because that paradigm is so prevalent but omitted by many compiler books. At some point, I realized that dropping readers straight into C was too unfriendly of an introduction. It’s hard to teach high-level concepts like parsing and name resolution while also tracking pointers and managing memory. OK, so we’ll build two interpreters. First, a simple one in a high-level language to focus on concepts. Then a second bytecode VM in C to focus on performance and low-level implementation techniques. Somehow, I didn’t notice that maybe this “handbook” wasn’t going to be as pocket-sized as I hoped. My first choice for the high-level implementation language was JavaScript. I implemented most of a Vox interpreter in JS, but never really liked it. I wanted to write the interpreter in an object-oriented style because there are techniques like the Visitor pattern for doing language stuff in OOP that aren’t covered well elsewhere. Doing OOP in JS means deciding whether to use classes or a prototypal style. The former is cleaner but infuriates some segment of readers. The latter is verbose and confusing to those not already steeped in prototypes. Also, I missed static types. People reading code in a book don’t get the luxury of seeing the code in a debugger where they can see what values are in various variables. Static type annotations in the code help. So I switched to Java. I don’t love Java but it seemed like the least biased choice for a statically typed object-oriented language. I found you can tame a lot of its infamous verbosity by simply not programming in 1990s enterprise Java style. Maybe it’s not idiomatic to have public fields, but it’s a hell of a lot shorter. In parallel, I started building the bytecode VM in C, porting over bits of Wren’s implementation and stripping out the Wren-specific stuff. I spent the spring and summer of 2016 circling between these three pieces—the design of Vox itself, the Java interpreter, and the C bytecode VM. This was a delightful, satisfying period of time. The three parts played off each other in challenging ways. Sometimes I would change the language to make one interpreter simpler, but find doing so made the other interpreter more complex. Other times I’d hit on some trick that made everything get smaller and cleaner. Getting back on the horse#getting-back-on-the-horse I remember the exact moment I committed to writing the book. I was stuck on a tricky language design problem: constructor syntax. I knew I wanted classes, which meant some way to construct instances. Adding a new keyword felt too special-purpose for my minimal language. I like Smalltalk and Ruby’s approach of making new be a method on the class object itself, but that requires metaclasses and a lot of other machinery. I was struggling to find a way to add instantiation without making the language much bigger. Then I remembered JavaScript’s thing where you can simply invoke a “class” as if it were a function to create new instances. That has all sorts of weird baggage in JavaScript because everything does in JS, but the concept and syntax were perfect. I already had first-class classes. And I already had closures which meant a function call syntax that could be applied to arbitrary expressions. So “constructors” just became what you got when you invoked a class. I felt like Vox had gelled, like it was a language now. And my two implementations were coming along well too. I was surprised by how few hacks or ugly corners I ran into. The codebases kind of fell together and the more I tweaked them, the nicer they got. It felt more like I had discovered them than that I had created them. It would be a shame to not write the book and put them out there into the world. They wanted me to. I committed to writing the book, and I restarted my rule of writing every single day. I had a few thousand lines of pretty Java and C code, but how do I turn that into a book that can be read in linear order? Compact codebases tend to be highly intertwined with many cyclic dependencies. I didn’t want readers to have to slog through ten chapters before they could even run main(). This was the real technical challenge of writing the book—how do I take two implementations of the same language, and break them into incremental pieces that I can build up a chapter at a time? I made this problem harder for myself because of the meta-goal I had. One reason I didn’t get into languages until later in my career was because I was intimidated by the reputation compilers have as being only for hardcore computer science wizard types. I’m a college dropout, so I felt I wasn’t smart enough, or at least wasn’t educated enough to hack it. Eventually I discovered that those barriers existed only in my mind and that anyone can learn it. My main overarching goal of the book is to pass on that feeling, to get readers to understand there’s no magic in there and nothing keeping them out. To nail that conceit, I wanted to include every single line of code used by the interpreters in the book. No parser generators, nothing left as an exercise for the reader. If you type in all of the code in the book, you get two complete, working interpreters. No tricks. So not only did I need to break these two interpreters into chapters, I needed to do it without any cheating. I wanted a hard guarantee that at the end of each chapter, you had a program that you could type in, compile, run, and do something with. I knew I wouldn’t be able to verify this manually, so it was time to create some tools. A bespoke build system#a-bespoke-build-system I wrote my first book in Markdown. I slapped together a tiny Python script that converts the Markdown to HTML and transcludes the code snippets which are stored in separate C++ files. When I started my second book, I took that script and started growing it. It evolved throughout writing the book, but in the end, here is how it works. All of the code for the interpreters are stored in separate source files. I have a Java project that contains the complete Java interpreter that you get by the end of that part of the book. Likewise, there’s a C project for the bytecode VM. I can edit and build those in an IDE, run tests, debug them, etc. They’re real programs. Meanwhile, the text of the book is authored in Markdown, one file per chapter, just like my first book. To include a snippet of code in the book, I put a tag in the Markdown like this: Which can be any of: ^code is-alpha Once we've found an identifier, we scan the rest of it using: Here, the ^code line says “look up the snippet named ‘is-alpha’ and insert it here.” When the build script generates the HTML for this chapter, it goes off and hunts through the code for that snippet. Over in the code, special comments delimit snippets. The one included here looks like this: //> Scanning on Demand is-alpha static bool isAlpha(char c) { return (c >= 'a' && c 'z') || (c >= 'A' && c 'Z') || c == '_'; } // The //> line begins the snippet and says what chapter the snippet appears in and the name of the snippet. The // line ends the snippet. Pretty straightforward. This let me build the book, but didn’t ensure that the thing I built actually worked. So I wrote a separate script that instead of building the book, builds programs. For each chapter, it collects all of the snippets that appear in that chapter and the previous ones and writes them out to separate source files. In other words, it produces a separate interpreter, one for each chapter, containing only the code that readers have seen so far. I put together a Makefile to build those per-chapter versions of each interpreter to make sure they compiled. Of course, compiling successfully doesn’t mean they do anything useful. Writing a single correct interpreter is hard. Writing thirty of them—there are thirty chapters in the book—is much harder. I had already harvested a little test runner from Wren and ported most of Wren’s tests over to be Lox tests. (I changed the name of the language in the book since there was already a language out there named “Vox.”). I took that test runner and extended it to be able to run the tests on each chapter’s version of the interpreters. Of course, the tests don’t all pass—the interpreters aren’t complete! So I added metadata to track which tests I expected to pass by which point in the book. With this in place, I could automatically verify that the code that I was showing readers did exactly what I expected. More complex snippets#more-complex-snippets The snippet markers look pretty straightforward, and in many cases they are. But reality tends to get messier and I didn’t allow myself to sweep any of that mess under nearby rugs. Some changes don’t just add code to the interpreter. I try to minimize it, but often you need to replace some existing code. A few lines of code may appear in chapter 5 and then later get superseded in chapter 9 by something more powerful. Obviously, I can’t jam both of those snippets into the same source file and expect it to compile. Remember, the source files that I hand author are themselves valid Java and C programs that I can build and run. If a function contained several versions of its body mixed together, odds are slim that the compiler will like what it sees. So, for any piece of code that later gets replaced—in other words code that is not part of the very final version of each interpreter—there is a different snippet syntax: static void concatenate() { /* Strings concatenate //> Garbage Collection concatenate-peek ObjString* b = AS_STRING(peek(0)); ObjString* a = AS_STRING(peek(1)); // int length = a->length + b->length; char* chars = ALLOCATE(char, length + 1); memcpy(chars, a->chars, a->length); memcpy(chars + a->length, b->chars, b->length); chars[length] = '\0'; } This block comment contains a snippet of code. The header indicates that this snippet is named “concatenate” and first appears in the “Strings” chapter. Then, later, it gets removed when the “concatenate-peek” snippet in the “Garbage Collection” chapter appears. In other words, that latter snippet replaces the previous two lines. By storing the code for this snippet inside a block comment, I ensure that the code as it is in the raw source file is still valid. In some places where the interpreter gets revised multiple times, the code can get pretty complex. Here is the main() function of the bytecode VM: int main(int argc, const char* argv[]) { //> A Virtual Machine main-init-vm initVM(); // /* Chunks of Bytecode main-chunk /* Chunks of Bytecode main-constant /* Chunks of Bytecode main-constant /* Chunks of Bytecode main-chunk-line /* A Virtual Machine main-chunk /* A Virtual Machine main-negate /* Chunks of Bytecode main-chunk /* Chunks of Bytecode main-chunk-line /* Chunks of Bytecode main-disassemble-chunk /* A Virtual Machine main-interpret //> Scanning on Demand args if (argc == 1) { repl(); } else if (argc == 2) { runFile(argv[1]); } else { fprintf(stderr, "Usage: clox [path]\n"); exit(64); } freeVM(); // /* A Virtual Machine main-free-vm /* Chunks of Bytecode main-chunk return 0; } Maintaining this is not super fun. But, thankfully, I have a build and test system to tell me when I break something. Slicing up the interpreters#slicing-up-the-interpreters So I had a tool that could let me split the interpreters across the chapters. If it was possible to break these interpreters into chapters at all, it would let me do so. Now I just had to figure out where to carve the seams. This was the most technically challenging part of the book writing process. I wasn’t sure if it was going to work at all. I spent several weeks sketching out potential lists of chapters, sprinkling snippet markers throughout the code, and seeing if the result built. I’d get a compile error because a snippet in an early chapter tried to call a function in some later chapter and I would have to go back and reorganize things. I hand-drew dependency graphs between language features and tried to untangle them. Here’s an example of how this process unfolded: To teach functions I want show that recursion works. But to have recursive functions I need control flow. Otherwise, every recursive function recurses infinitely without a base case. So control flow has to come before functions. For control flow, I need side effects so that I can show that a certain code path is not taken. The obvious way to do side effects is to have a print() function that displays output. But I don’t have functions yet. That’s a cycle. Crap. Sometimes I had to change the language itself to break cycles. The above example is why Lox has a built in print statement instead of a print function. Because that way we can introduce the print statement before control flow, which is in turn before functions. I had to break a couple of cycles like that but, eventually, to my surprise, I got it all sorted out. I had a complete list of chapters for both interpreters. Every line of code was sorted into a snippet that belonged to one of those chapters. I could build and run each chapter’s code. Best of all, each chapter had a reasonably coherent concept and a roughly similar amount of code. Before, I felt like I had a language and code that wanted to get out there into the world. Now I felt like I had a book. Or, at least, I had all of the code for a book. A chapter at a time#a-chapter-at-a-time I wrote my first book one chapter at a time. I drafted, edited, illustrated each chapter and put it online before moving to the next one. Serial publishing for the digital age. I really loved that process. It helped build an audience for the book and gave me incremental feedback which made the book better and kept me going. I don’t think I could write a whole book in the dark. I intended to publish this book the same way, but the deeply interconnected nature of the chapters made that much harder. I didn’t want to discover a problem with the code in chapter 28 that forced me to tweak things in an earlier chapter that readers had already read. I didn’t want to paint myself into a corner or invalidate any previously-published material. So the entire time I was designing the language, coding the interpreters, and splitting the codebases into chapters, I had not done any actual writing. I didn’t want to put down any prose until I knew the code was solid. So I spent the summer of 2016 just hacking on code. It was, honestly, a blast. The programming part is definitely the fun part, and it was a joy to tinker on the code and figure out how to break it into chapters. Sort of like making a jigsaw puzzle and solving it at the same time. After a few months, it was all there. Every single line of code for the entire book. A complete list of chapters. And I hadn’t written a single word of prose. In theory, “all” that remained was writing some text to explain the code I had already written along with some pictures. But, for me at least, English is a much more taxing language to write than C or Java. I had all of the difficult work ahead of me, and all of the fun was done. Illustrating by hand#illustrating-by-hand Well, not all of the fun. I did still have the illustrations to do. With my last book, I hand-drew little sketchy diagrams to show various bits of architecture. I wanted even more illustrations for this book to make the concepts less abstract, less opaque. Unlike a videogame, you can’t see a garbage collector doing its thing. Visual metaphors really help. I liked the hand-drawn look. It furthered my meta-goal of making the material more approachable, more human. But I wanted to up the quality. I wanted them to be more intricate and contain more information. I wanted the drawings to be more detailed. Less like margin doodles and more like, well, illustrations. Maybe even some lowercase letters. The ultimate goal for me is a print book, so I stuck with black and white ink. I wanted a tighter, more “spidery” style, so I got some technical pens. People often ask me what programs I used for the illustrations, assuming I did them all digitally. Here are the main tools I used: I went with Pigma Microns in 01 and 005. If I were doing it again, I think I'd do Faber-Castell Pitt pens. There are two kinds of illustrations in the books: diagram-like ones that show meaningful information, and drawings that are for metaphors or just to be silly jokes. The process is different for each. I draw each diagram in pencil on graph paper. That lets me erase and move things around until I get it where I like: All of the vertical and horizontal lines in the illustrations generally fall on the graph paper rules or halfway between them. Then I tape a piece of tracing paper on top and draw over it in ink: I make mistakes sometimes, usually when lettering like "upvaluels" here. I fix that in Photoshop after scanning. I hand letter everything. It takes a long time. I used to do graphic design, and I have this weird tic where any time I see something that looks handwritten, I look for multiple instances of the same letter to see if they are different or if the design just used a handwriting font. It’s almost always a handwriting font and I die a little inside to see the illusion evaporate. Well, this is my damned book and no reader will ever feel that disappointment. Every single fucking letter in every one of the illustrations was hand lettered and is unique. Here is the hand-lettered logotype for the book. Each "R" is different! Also, if that’s not obsessive enough, I spent time changing my own handwriting to better match the text font of the book. I taught myself to write double-story “a” and “g” letters and practiced by filling pages of paper with the same letter over and over. Look at the loop under the "g" in "filling" and the finial on the "a" in "apples". I also wanted to make sure that the illustrations and text matched each other across the book. To give the text a consistent size, I printed a little height guide: The dotted line indicates the x-height. I picked a ratio for that to match the fonts I use for text and code. I slid this paper under the tracing paper and lettered on top of those lines to keep the metrics the same across the book. To keep the diagram size and line thickness consistent, each illustration has a pair of registration marks a fixed distance apart: The little marks that the pencils are pointing at. I scan each illustration into Photoshop for clean up and processing. I use those marks when cropping to ensure that the image maintains the right size relative to other images. I recorded a video of the whole process if you want to see it in action: Even in timelapse, it takes a long time. Writing this all out makes me sound like a crazy person. What the hell am I doing with my life? Or, more importantly, what could I have been doing instead of doing all that? Too late now, I guess. The picture-like drawings have a different workflow since they don’t have a lot of straight lines or align to a grid. How will readers understand what a stack is without this helpful illustration? I draw those on regular sketch paper using a non-photo blue pencil. Then I ink on top of that. I scan the paper in RGB and use the blue channel, which mostly makes the blue pencil marks disappear. The sketch paper bleeds the ink more than I like but I didn't want to change paper partway through the book, so I stuck with it. It’s a lot of work for each image, and this doesn’t include all of the work after scanning it. And I wanted a lot of them. By the end, I had this stack of paper: Such a small image for so much work. I went through two full pads of tracing paper, two pads of graph paper, a sketch pad, and several pens. I drew 181 illustrations. Writing is suffering#writing-is-suffering I had the code, and I had a process for illustrations. The remaining work was just writing all the words and drawing all the pictures. So that’s what I did. I started at chapter one and started writing. For each chapter, I wrote an outline and then a first draft. I did an editing pass over that to fix all the major problems. Then a second pass where I read the whole chapter out loud to fix cadence and other stuff. This is the same process I used for the first book. I stumbled onto something that worked, so I wasn’t about to mess it up. I posted each chapter online, and then spent a day fixing bugs that readers noticed. Then I moved on to the next chapter. I wrote. And wrote. And wrote. Every single day. Every now and then I would have a trip or something where I couldn’t write. As with my first book, I would bank days by writing multiple sessions per day beforehand and then spend those banked days on days that I didn’t write. But for the most part, I wrote every day. In the blog post I wrote after my first book, I whined about how I had to write on days when I traveled for work, on holidays, when the kids had sniffles. At the time, it truly was one of the hardest things I’ve ever done. This time was something else entirely. I wrote the day my grandfather died (peacefully, unsurprisingly) and the day my aunt died (tragically, days after retiring). I wrote the day I found out my Mom had cancer and my children saw me cry for the first time. I was flying to Louisiana to keep my Mom company when I turned on my phone during the layover and discovered a dear friend had had a stroke. I wrote that evening. I woke up the next day and found out she had died. I wrote that morning sitting next to my brother in the waiting room of the hospital while my Mom got her PET scan. The morning of my friend’s memorial service, I wrote in the hotel. Later that day, I openly sobbed in front of a room full of people. The next day, my wife found out her aunt had terminal cancer. I wrote on the flight home. See that dog up there in my profile photo? That’s Ginny. She’s on the back cover of my first book. Her myriad health problems finally caught up with her last spring. People sometimes ask, “When did you know you were an adult?” For me it was the day I made the call to put my dog down. The hardest part was watching my kids say goodbye to her. I’m tearing up now writing about it. I ran my fingers through Ginny’s silky fur as the sedatives took her away. I only got through 59 words that afternoon. I wrote the day the US somehow elected a racist, abusive, corrupt demagogue, and every day afterwards as I saw my country and others turn towards hate and authoritarianism. I wrote while climate change and income inequality worsened. And now here I am writing at home on the same desk where I work now, quarantined like most of you all, hoping to survive the worst pandemic the world has seen in a century. This is not about how disciplined I was. Because during what have been some of the worst years of my life, a weird inversion happened. It’s not that I was going through that shit and still writing in addition to it. I had to keep writing. Writing was one thing I could still control in the face of many things I could not. If I could make it through the book, maybe I could make it through the other things too. If I had skipped a day it would have meant that the cancer or the deaths beat me that day, that they were stronger than me. I feared what it would mean to me to let go. I got through these four years and kept writing, but I paid a price. When I read the earlier chapters, they have a whimsy and light-heartedness that later chapters lack. We’re all going through dark times, and I don’t feel light. The past few years left a mark on me, and that mark shows up in the book. I miss the goofier person I used to be, sometimes. But I’d like to believe that maybe the person I am now is a little more honest. Maybe some of those jokes were a mask. And, thankfully, Mom is in remission. Psychological self examination aside, I did keep up the writing. Which is good because, man did I underestimate this book. I was aiming for 60,000 words and hoped to get it done in about a year. Here I am four years later sitting on a quarter of a million words. People sometimes ask what it’s like writing something that big. I’ve been asking myself that for the past couple of weeks. And the weird thing is, I don’t know. I’ve had my head down for the past four years and haven’t looked past the next paragraph or two the entire time. What does it feel like to write an email or draw a picture? Writing the book felt like that. I just happened to do it over and over again. I feel like a marathon runner who’s been watching his feet the whole time and didn’t even notice when he stumbled over the finish line. Now what?#now-what Crafting Interpreters is complete now. I had to stop here for a minute and look at that sentence. I’ve been working on this book every day for around 1,400 days. I can’t wait to take a break. So that’s the next step. My plan was to finish the book right before spring break and enjoy a week on the beach with family. That beach trip went the way of so many other plans in early 2020, but I still intend to take a long break. I don’t know if you noticed, but we all have a lot of other shit to deal with right now. I’m going to relax. Every morning since 2016, I’ve woken up with a task I had to do. Until I got my writing done for the day, it was on my mind, weighing me down. Writing left me drained. If you’ve ever had a newborn, you know the feeling of always having to carry the baby around. After a while, it’s like you forget what it’s like to have two free arms. I’ve been carrying this baby for four years, so I’m looking forward to having both arms for a while. Once I’m recharged, the real fun starts. Having the book online is important, but for me, Crafting Interpreters was always meant to be a book with pages and a cover. So after a long bout of editing and bug fixing, I’m going to get started doing the page layout for the print edition. I love graphic design, and I can’t wait to hold it in my hands. If you’d like to hold it in your hands when it comes out, I have a mailing list where I’ll let you know when the book is done. In the meantime, I think I’ve earned some rest.